Saving structured logs in RavenDB

Question

I’m writing .NET Core application and would like to use structured logging available in .NET Core Logging Extensions to write logs in more structured way. I want to save my logs in RavenDB and I’m wonder are there any best practices for use case like this? I’m new to Document DB’s and I cannot predict things in future like I’m used to in SQL relational databases. My main consideration are:

1. Should I save each log in separate documents or maybe better idea is to create one document per structured log template and save logs with same template inside. Second idea is tempting but I’m little worried it will blows up after some time?
2. It would be better to save all logs in one format (message, exception etc) and structured data in key value list attached to it or maybe it would be better to create separate document structure for each log type?

Answer 1

I think each log entry should go into a separated document for the following reasons:

• Log entries should expire, otherwise the database would be huge overtime. If you have several log entries in the same document, you cannot expire a log, you need to expire a group of logs in the same document.
• RavenDB doesn't like huge documents because they cause performance problems. Grouping logs into same documents by template would produce huge documents.
• Grouping documents would make more difficult to get the set of correlated logs.

I think structured data should go in classical JSON properties form. For example:

{
   "State": {
        "Service": "International Weather Forecast Service",
        "{OriginalFormat}": "Weather forecast called using {Service}"
    }

}

There is a project on GitHub that follows the grouping approach RavenDB.StructuredLog. But I think it is not the right one, so I created mine: Logging.Raven

Answer 2

What you can do is, use a model similar to this:

public class LogMessage<T>
{
    public string Message;
    public string Exception;

    public T Value;
}

Save each log message as a separate document in collection. If you will need to change things in the future you can change this model and use the patch feature to patch your old docs.

Also I think the Document Expiration feature would be useful to remove old log documents. https://ravendb.net/docs/article-page/4.2/Csharp/server/extensions/expiration