Create invalid Json with Newtonsoft - Allow invalid objects?

Question

I'm deliberately trying to create invalid JSON with Newtonsoft Json, in order to place an ESI include tag, which will fetch two more json nodes.

This is my JsonConverter's WriteJson method:

public override void WriteJson(JsonWriter writer, object value, JsonSerializer serializer)
{
    mApiResponseClass objectFromApi = (mApiResponseClass)value;

    foreach (var obj in objectFromApi.GetType().GetProperties())
    {
        if (obj.Name == "EsiObj")
        {
            writer.WriteRawValue(objectFromApi.EsiObj);
        }
        else
        {
            writer.WritePropertyName(obj.Name);
            serializer.Serialize(writer, obj.GetValue(value, null));
        }
    }
}

The EsiObj in mApiResponseClass is just a string, but it needs to be written into the JSON response to be interpretted without any property name - so that hte ESI can work.

This of course results in an exception with the Json Writer, with value:

Newtonsoft.Json.JsonWriterException: 'Token Undefined in state Object would result in an invalid JSON object. Path ''.'

Is there any way around this?

An ideal output from this would be JSON formatted, technically not valid, and would look like this:

{
value:7,
string1:"woohoo",
<esi:include src="/something" />
Song:["I am a small API","all i do is run","but from who?","nobody knows"]
}

Edit: Using ESI allows us to have varying cache lengths of a single response - i.e. we can place data that can be cached for a very long time in some parts of the JSON, and only fetch updated parts, such as those that rely on client-specific data. ESI is not HTML specific. (As some state below) It's being run via Varnish, which supports these tags. Unfortunately, it's required that we do only put out 1 file as a response, and require no further request from the Client. We cannot alter our response either - so i can't just add a JSON node specifically to contain the other nodes.

Edit 2: The "more json nodes" part is solved by ESI making a further request to our backend for user/client specific data, i.e. to another endpoint. The expected result is that we then merge the original JSON document and the later requested one together seamlessly. (This way, the original document can be old, and client-specific can be new)

Edit 3: The endpoint /something would output JSON-like fragments like:

teapots:[ {Id: 1, WaterLevel: 100, Temperature: 74, ShortAndStout: true}, {Id: 2, WaterLevel: 47, Temperature: 32, ShortAndStout: true} ],

For a total response of:

{
value:7,
string1:"woohoo",
teapots:[ {Id: 1, WaterLevel: 100, Temperature: 74, ShortAndStout: true}, {Id: 2, WaterLevel: 47, Temperature: 32, ShortAndStout: true} ],
Song:["I am a small API","all i do is run","but from who?","nobody knows"]
}

Answer 1

Your basic problem is that a JsonWriter is a state machine, tracking the current JSON state and validating transitions from state to state, thereby ensuring that badly structured JSON is not written. This is is tripping you up in two separate ways.

Firstly, your WriteJson() method is not calling WriteStartObject() and WriteEndObject(). These are the methods that write the { and } around a JSON object. Since your "ideal output" shows these braces, you should add calls to these methods at the beginning and end of your WriteJson().

Secondly, you are calling WriteRawValue() at a point where well-formed JSON would not allow a value to occur, specifically where a property name is expected instead. It is expected that this would cause an exception, since the documentation states:

Writes raw JSON where a value is expected and updates the writer's state.

What you can instead use is WriteRaw() which is documented as follows:

Writes raw JSON without changing the writer's state.

However, WriteRaw() won't do you any favors. In specific, you will need to take care of writing any delimiters and indentation yourself.

The fix would be to modify your converter to look something like:

public class EsiObjConverter<T> : JsonConverter
{
    const string EsiObjName = "EsiObj";

    public override void WriteJson(JsonWriter writer, object value, JsonSerializer serializer)
    {
        var contract = serializer.ContractResolver.ResolveContract(value.GetType()) as JsonObjectContract;
        if (contract == null)
            throw new JsonSerializationException(string.Format("Non-object type {0}", value));
        writer.WriteStartObject();
        int propertyCount = 0;
        bool lastWasEsiProperty = false;
        foreach (var property in contract.Properties.Where(p => p.Readable && !p.Ignored))
        {
            if (property.UnderlyingName == EsiObjName && property.PropertyType == typeof(string))
            {
                var esiValue = (string)property.ValueProvider.GetValue(value);
                if (!string.IsNullOrEmpty(esiValue))
                {
                    if (propertyCount > 0)
                    {
                        WriteValueDelimiter(writer);
                    }
                    writer.WriteWhitespace("\n");
                    writer.WriteRaw(esiValue);
                    // If it makes replacement easier, you could force the ESI string to be on its own line by calling
                    // writer.WriteWhitespace("\n");

                    propertyCount++;
                    lastWasEsiProperty = true;
                }
            }
            else
            {
                var propertyValue = property.ValueProvider.GetValue(value);

                // Here you might check NullValueHandling, ShouldSerialize(), ...

                if (propertyCount == 1 && lastWasEsiProperty)
                {
                    WriteValueDelimiter(writer);
                }
                writer.WritePropertyName(property.PropertyName);
                serializer.Serialize(writer, propertyValue);

                propertyCount++;
                lastWasEsiProperty = false;
            }
        }
        writer.WriteEndObject();
    }

    static void WriteValueDelimiter(JsonWriter writer)
    {
        var args = new object[0];
        // protected virtual void WriteValueDelimiter() 
        // https://www.newtonsoft.com/json/help/html/M_Newtonsoft_Json_JsonWriter_WriteValueDelimiter.htm
        // Since this is overridable by client code it is unlikely to be removed.
        writer.GetType().GetMethod("WriteValueDelimiter", BindingFlags.Instance | BindingFlags.Public | BindingFlags.NonPublic).Invoke(writer, args);
    }

    public override bool CanConvert(Type objectType)
    {
        return typeof(T).IsAssignableFrom(objectType);
    }

    public override bool CanRead { get { return false; } }

    public override object ReadJson(JsonReader reader, Type objectType, object existingValue, JsonSerializer serializer)
    {
        throw new NotImplementedException();
    }
}

And the serialized output would be:

{
  "value": 7,
  "string1": "woohoo",
<esi:include src="/something" />,
  "Song": [
    "I am a small API",
    "all i do is run",
    "but from who?",
    "nobody knows"
  ]
}

Now, in your question, your desired JSON output shows JSON property names that are not properly quoted. If you really need this and it is not just a typo in the question, you can accomplish this by setting JsonTextWriter.QuoteName to false as shown in this answer to Json.Net - Serialize property name without quotes by Christophe Geers:

var settings = new JsonSerializerSettings
{
    Converters = { new EsiObjConverter<mApiResponseClass>() },
};    
var stringWriter = new StringWriter();
using (var writer = new JsonTextWriter(stringWriter))
{
    writer.QuoteName = false;
    writer.Formatting = Formatting.Indented;
    writer.Indentation = 0;
    JsonSerializer.CreateDefault(settings).Serialize(writer, obj);
}

Which results in:

{
value: 7,
string1: "woohoo",
<esi:include src="/something" />,
Song: [
"I am a small API",
"all i do is run",
"but from who?",
"nobody knows"
]
}

This is almost what is shown in your question, but not quite. It includes a comma delimiter between the ESI string and the next property, but in your question there is no delimiter:

<esi:include src="/something" /> Song: [ ... ]

Getting rid of the delimiter turns out to be problematic to implement because JsonTextWriter.WritePropertyName() automatically writes a delimiter when not at the beginning of an object. I think, however, that this should be acceptable. ESI itself will not know whether it is replacing the first, last or middle property of an object, so it seems best to not include the delimiter in the replacement string at all.

Working sample .Net fiddle here.