Visitor pattern serialization memory bloat

70 Views Asked by At

I'm working on a serialization library for an embedded target. The library is based on a classic visitor pattern with a templated pack() method implemented in each struct that I want to serialize. Example:

struct softwareInfo
{
    uint32_t version;
    std::array<uint8_t, 40> commitHash;
    uint32_t buildDate;

    template <class T>
    void pack(T& archive)
    {
        archive.process(HVP(version));
        archive.process(HVP(commitHash));
        archive.process(HVP(buildDate));
    }
};

HVP macro packs the field with it's hashed name into a hashValuePair class that is used later on.

constexpr uint32_t compileTimeHash(const char* str, size_t index = 0, uint32_t hash = 0)
{
    return (str[index] == '\0') ? hash : compileTimeHash(str, index + 1, hash * 31 + str[index]);
}

template <typename T>
class hashValuePair
{
   public:
    hashValuePair(uint32_t hash, T* value) : hash(hash), value(value) {}
    uint32_t getHash() const { return hash; }
    T& getValueRef() { return *value; }

   private:
    uint32_t hash = 0;
    T* const value = nullptr;
};

#define HVP(var) hashValuePair(compileTimeHash(#var), &var)

There are two more modules:

  1. The PersistentStorage class that is used to register objects that can be serialized. It holds the following buffer:
std::array<StorageEntry, maxEntries> entries

where:

struct StorageEntry
{
    std::aligned_storage_t<sizeof(Entry<void>), alignof(Entry<void>)> buffer;
    EntryBase* object;
    struct Header
    {
        uint32_t hashName;
        uint32_t size;
    } header;
    uint32_t datacrc32;
    Status packStatus;
    Status unpackStatus;
};

class EntryBase
{
   public:
    virtual ~EntryBase() {}
    virtual void pack(Packer::Packer& packer) = 0;
    virtual void pack(Packer::Unpacker& unpacker) = 0;
};

template <typename T>
class Entry : public EntryBase
{
   public:
    explicit Entry(T* item) : item(item) {}
    void pack(Packer::Packer& packer) override { item->pack(std::forward<Packer::Packer&>(packer)); }
    void pack(Packer::Unpacker& unpacker) override { item->pack(std::forward<Packer::Unpacker&>(unpacker)); }

   private:
    T* const item;
};

It has two main methods - doRead() and doWrite() that take all the registered structs and packs/unpacks them one by one using the pack and unpack methods shown below.

  1. The second module is a pair of Packer/Unpacker classes that is used to serialize and deserialize structs by visiting every member listed using the process method.
namespace Packer
{

class Packer
{
   public:

    template <class T>
    void process(hashValuePair<T>&& hashValuePair)
    {
        packType(hashValuePair.getValueRef());
    }

   private:

    template <class T>
    void packType(T& value)
    { 
        (...)
        copyToBuf(value);
    }

    template <class T>
    void copyToBuf(T value)
    {
        it += serialize<T>(value, it);
    }
};

class Unpacker
{
   public:

    template <class T>
    void process(hashValuePair<T>&& hashValuePair)
    {
        //search for value hash
        unpackType(hashValuePair.getValueRef());
    }

   private:

    template <class T>
    void unpackType(T& value)
    {
        (...)
        tryUnpack(*it++, value);
    }

    template <typename T>
    void tryUnpack(uint8_t maybeType, T& value)
    {
        if (maybeType != getType(value))
            //error
        copyToValue(value);
    }

    template <typename T>
    void copyToValue(T& value)
    {
        value = deserialize<T>(it);
        it += sizeof(value);
    }
};

template <class PackableObject>
Status pack(PackableObject& obj, std::span<uint8_t>& buf, size_t& packedSize)
{
    Status status{};
    auto packer = Packer(buf, packedSize, &status);
    obj.pack(packer);
    return status;
}

template <class PackableObject>
Status unpack(PackableObject& obj, std::span<uint8_t>& buf)
{
    Status status{};
    auto unpacker = Unpacker(buf, &status);
    obj.pack(unpacker);
    return status;
}

} 

I intentionally omitted the rest of the implementation (error handling, searching for field hashes) to leave only the most basic flow.

Now, what I can see is that there is a relatively large memory footprint associated with the implementation provided above. I can imagine the templated methods have to be instantiated with types that are used, however i'm not sure I'm doing everything in the most optimal way. For example serializing a softwareInfo struct uses 464B of FLASH memory Packer FLASH footprint divided to functions. When I add another uint32_t field I get a 44B increase in size which feels quite large, especially considering I had already specialized this type with previous uint32_t field Packer FLASH footprint divided to functions with additional uint32_t field. The build type is release with -Os.

Can you see any obvious flaws in the current design? Could something like std::variant help in this situation, since its runtime polymorphism?

0

There are 0 best solutions below