Why I need the pixel format to decompress images with libjpeg turbo

Question

Background story: Right now I am working on an image encode and decode service. I will attach the service to a sensor (ex. camera) like to compress the image, send it over the network to receive it at the end with a client service, where the image is finaly decompressed and used. For this whole compression/decompression stuff I decided to use libJPEG Turbo, because it is the fastest library that I found.

Problem: In my mind a JPEG file is self describing and it should be possible to read it with every tool. But why I need for the compression and decompression with libJPEG Turbo additionally the pixel format (GRAY, RGB, BGR, RGBX, ...)? How can I extract/decompress the pixel format from the compressed JPEG (Byte Array)?

My code - compression (c++):

tjhandle _jpegCompressor = tjInitCompress();
tjCompress2(_jpegCompressor, buffer, width, 0, height, TJPF_RGB, &_compressedImage, &_jpegSize, TJSAMP_444, JPEG_QUALITY, TJFLAG_FASTDCT);
tjDestroy(_jpegCompressor);

My code - decompression (c++):

int width, height;
tjDecompressHeader(_jpegDecompressor, _compressedImage, _jpegSize, &width, &height);
size_t size = width*height*channels;
unsigned char* buffer = new unsigned char[size];
tjDecompress2(_jpegDecompressor, _compressedImage, _jpegSize, buffer, width, 0, height, TJPF_RGB, TJFLAG_FASTDCT);
tjDestroy(_jpegDecompressor);

Tests: I tested it out and the only importance is that I use for compression and decompression the same pixel format. If I use both times TJPF_BGR (or TJPF_RGB) it works, but if I use for decompression a different pixel format the channels are flipped.

Interestingly If I use the tjDecompress function (not tjDecompress2) I need to compress with TJ_RGB and can decompress with TJ_RGB or TJ_BGR to get a right result... If I start to compress with TJ_BGR the color is flipped at the end. What do I missunderstand?

tjDecompress(_jpegDecompressor, _compressedImage, _jpegSize, buffer, width, 0, height, 3, TJFLAG_FASTDCT);