Camera always pointing toward centre and camera movement doesnt effect result

Question

Im trying to make a 3d engine from scratch and normal perspectave projection works good but trying to implement the camera has been a pain.

Right now what I'm doing is using mous offset to change pitch and yaw:

float x_diff = (WIDTH / 2) - mousePos.x;
x_diff /= WIDTH;

float y_diff = (HEIGHT / 2) - mousePos.y;
y_diff /= HEIGHT;

yaw += x_diff;
pitch += y_diff;

Then using that and the position to create the view matrix:

ApplyView(vertex, cameraPosition, yaw, pitch);

Vector3 ApplyView(Vector3 vertex, Vector3 cameraPosition, float yaw, float pitch) {
    // adapted from https://www.3dgep.com/understanding-the-view-matrix/
    float cosPitch = cos(pitch);
    float sinPitch = sin(pitch);
    float cosYaw = cos(yaw);
    float sinYaw = sin(yaw);

    Vector3 xaxis(cosYaw, 0, -sinYaw);
    Vector3 yaxis(sinYaw * sinPitch, cosPitch, cosYaw * sinPitch);
    Vector3 zaxis(sinYaw * cosPitch, -sinPitch, cosPitch * cosYaw);


    float lookAt[4][4] = {{ xaxis.x,                            yaxis.x,                            zaxis.x,                           0},
                          { xaxis.y,                            yaxis.y,                            zaxis.y,                           0},
                          { xaxis.z,                            yaxis.z,                            zaxis.z,                           0},
                          {-DotProduct(xaxis, cameraPosition), -DotProduct(yaxis, cameraPosition), -DotProduct(zaxis, cameraPosition), 1}};

    float pos[4] = { vertex.x * lookAt[0][0] + vertex.y * lookAt[1][0] + vertex.z * lookAt[2][0] + 0 * lookAt[3][0],
                     vertex.x * lookAt[0][1] + vertex.y * lookAt[1][1] + vertex.z * lookAt[2][1] + 0 * lookAt[3][1],
                     vertex.x * lookAt[0][2] + vertex.y * lookAt[1][2] + vertex.z * lookAt[2][2] + 0 * lookAt[3][2],
                     0 };

    return Vector3(pos[0], pos[1], pos[2]);
}

Right now I can see the 3 cubes but when I move the mouse it rotates the cubes around the centre instead of rotationg the camera.