OpenGL first person camera - rotating directionVector

Question

I'm trying to implement a working fps camera. I read a lot of online tutorials and forum posts, but I still don't get it working properly. I'm not using glm or any other libraries for doing that, I want to do it by myself to understand it, hopefully.

I've got a camera class, which basically got and position vector, a target vector and an up vector.

I thought for myself, if I will rotate the camera's direction vector on the given mouse input, it should do the right thing. It does, but only for the case if my object is in the origin. If it's positioned let's say at x=10, then it seems that the cube will rotate around his own axes.. If I'm just moving around with W,A,S,D the view seems to be fine.

I've tried a lot, read about Camera moving vs world moving (which kinda confused me much more about all this).

Does anyone know where the problem can be?

Update function called in mainLoop:

void Camera::updatePosition(int viewport_width, int viewport_height) {

yaw += mouseSpeed * InputHandler::mouseDeltaX;
pitch += mouseSpeed * InputHandler::mouseDeltaY;

InputHandler::mouseDeltaX = 0;
InputHandler::mouseDeltaY = 0;

vector<float, 3> camDir = vector<float, 3>(-cos(yaw), sin(pitch), sin(yaw));
vector<float, 3> right = normalize(cross(_up, camDir));

movementSpeed = MOVEMENT_SPEED_MAX;

if (InputHandler::wPressed) {
    _pos += camDir * movementSpeed;
}
if (InputHandler::sPressed) {
    _pos -= camDir * movementSpeed;
}
if (InputHandler::dPressed) {
    _pos -= right * movementSpeed;
}
if (InputHandler::aPressed) {
    _pos += right * movementSpeed;
}
if (InputHandler::escPressed) {
    exit(0);
}

_target = _pos + camDir;

calcViewMatrix();   
}

ViewMatrix caculation:

void Camera::calcViewMatrix() {

vector<float, 3> w = normalize(_pos - _target);
vector<float, 3> u = normalize(cross(_up, w));
vector<float, 3> v = cross(w, u);

vector<float, 4> row1 = vector<float, 4>(u.x, v.x, w.x, 0);
vector<float, 4> row2 = vector<float, 4>(u.y, v.y, w.y, 0);
vector<float, 4> row3 = vector<float, 4>(u.z, v.z, w.z, 0);
vector<float, 4> row4 = vector<float, 4>(-dot(u, _pos), -dot(v, _pos), -dot(w, _pos), 1);

_viewMatrix = matrix<float, 4, 4>();
_viewMatrix = matrix<float, 4, 4>::from_rows(row1, row2, row3, row4);
}

Im loading the mvp matrix to the shader, calculated as follows:

mvp = P * V * M;

The modelMatrix was once an identity matrix for positioned at origin, or otherwise just with a x or z translational part.