I'm not an expert on streams and buffers, though I have learned an immense amount over the last month since I started tackling this problem.
This concerns boost::serialization and if I can get the binary archiving working, I'll save 50% of the storage space.
I've searched all over StackOverflow for the answer, and I've pieced together the following code that works, but only for text_iarchive. If I try to move to binary_iarchive, I get a segment fault with a message of "boost serialize allocate(size_t n) 'n' exceeds maximum supported size" or any other number of errors where it is obvious that there is a disconnect between the input stream/buffer and what binary_iarchive is expecting.
Like I said earlier, this works perfectly with text_iarchive. I can text_oarchive to an SQLite3 Blob, verify it in the database, retrieve it, text_iarchive it back in to the complex object, and it works perfectly.
Is there something wrong with the way I set up the input stream and buffer?
To not confuse everyone, I am NOT posting the structure of the object I am serializing and deserializing. There are many vector<double>, an Eigen Matrix, and a couple of basic objects. They work perfectly and are not part of the problem! (And yes, I delete the database records between tests to guard against reading a text_oarchive into a binary_iarchive.)
Here is the output archive section. This appears to work perfectly for text_oarchive OR binary_oarchive. The Blob shows up in the database and appears to be of the proper binary structure.
// BinaryData is a Typedef for std::vector<char>
BinaryData serializedDataStream;
bio::stream<bio::back_insert_device<BinaryData>> outbuf {serializedDataStream};
// I change the text_oarchive to binary_oarchive and uncomment the std::ios::binary parameter.
// when I'm attempting to move from text to binary
boost::archive::text_oarchive outStream(outbuf); //, std::ios::binary);
outStream << ssInputDataAndBestModel_->theModel_;
outbuf.flush();
// have to convert to unsigned char since that is the way sqlite3 expects to see
// a Blob object type
std::vector<unsigned char> buffer(serializedDataStream.begin(),serializedDataStream.end());
I then pass "buffer" to the SQLite3 processing object to store it in the Blob.
Here is the input archive section. The Blobs look identical storing and then retreiving from the DB whether it's text or binary. (But a text doesn't look like a binary, obviously.)
// this line is to get the blob out of SQLite3
currentModelDBRecPtr = cpp17::any_cast<dbo::ptr<Model>>(modelListModel);
if (!currentModelDBRecPtr->theModel.empty()) {
// have to convert to char since that is the way boost::serialize expects to see
// an archived object type (blob is vector of unsigned char)
std::vector<char> blobBuffer(currentModelDBRecPtr->theModel.begin(), currentModelDBRecPtr->theModel.end());
boost::iostreams::stream<boost::iostreams::array_source> membuf(blobBuffer.data(), blobBuffer.size());
std::istream &input_stream = membuf;
// Note: I change the following to binary_iarchive and uncomment the
// std::ios::binary flag to try to move from text_iarchive to binary_iarchive
boost::archive::text_iarchive input_archive(input_stream); //, std::ios::binary);
TheModel inputArchiveModel;
// it crashes on the next line, but it DOES successfully recreate half
// of the object before it randomly crashes.
input_archive >> inputArchiveModel;
}