Photo by Romain Vignes
Introduction
A couple of months ago we (more or less unknowingly) started to use System.Text.Json instead of Newtonsoft in one of our ASP.NET Core applications.
The application is responsible for sending emails, so it takes a payload, modifies it a bit and then passes it on to SendGrid. One property in the payload, templateParameters is a Dictionary<string, object> that we use for... template parameters. Basically, the caller can send in whatever they want and then use those parameters when creating the email template. By using a Dictionary<string, object>, we don't need to do any code changes when the email template changes. Obviously this is how Hello {{firstName}} errors happens but that's another story.
The bug
This had worked flawlessly until we received the following bug report from one of our customers
I got a really weird email, it looked like this:
Hello {"name": {"ValueKind":3},"data":{"ValueKind":1}}
Now, I recognized the ValueKind stuff straight away since I'd played around with the new System.Text.Json serializer before. And sure enough, it turns out that we are not using Newtonsoft anymore, we are now using System.Text.Json instead. Again, all the details of how this happened can be found in this GitHub issue.
Basically what happend for us was this:
- System.Text.Json was used instead of Newtonsoft when modelbinding occurred
- We serialized the Dictionary<string, object> (created by System.Text.Json) with Newtonsoft
- The following json output was produced:
{
"name": {
"ValueKind":3
},
"data":{
"ValueKind":1
}
}
Oooops.
It turns out that System.Text.Json and Newtonsoft differs in how they treat object by default. System.Text.Json tries to not make any assumptions at all of what type it should deserialize to when it comes to object.
Current functionality treats any object parameter as JsonElement when deserializing. The reason is that we don't know what CLR type to create, and decided as part of the design that the deserializer shouldn't "guess".
IMO that is fair enough but that caused us some problems (especially since we didn't knowingly enabled System.Text.Json :D ).
Solution
Prerequisite: Because of reasons, the serializer used for serializing the request towards SendGrid must use Newtonsoft for now. In the near future we will update that serializer to also use System.Text.Json, but for now, it can't be changed.
Newtonsoft solution
Now, the easiest solution would be to just switch over to Newtonsoft by adding a reference to Microsoft.AspNetCore.Mvc.NewtonsoftJson and then do this:
services.AddMvc().AddNewtonsoftJson();
So if you are not interested in using System.Text.Json, you can stop reading now.
System.Text.Json solution
So, we need to be able to replicate the behaviour of Newtonsoft when deserializing/model binding.
We have the following action method:
[HttpPost]
public IActionResult CustomProperty(MyInput data)
{
// Do some work here...
return new OkResult();
}
MyInput looks like this:
public class MyInput
{
public string Name { get; set; }
public Dictionary<string, object> Data { get; set; }
}
Now, if we send the following request to our API...
{
"name": "whatevs",
"data": {
"firstName": "Josef"
}
}
...and inspect it in the debugger, it will look like this:
As you can see, the items in the Dictionary<string, object> are wrapped in a JsonElement.
We need to customize the deserialization of Dictionary<string, object>.
We will do this by creating a custom JsonConverter. Fortunately, the documentation of how to do this provided by Microsoft is really good, so let's get into it.
public class DictionaryStringObjectJsonConverter : JsonConverter<Dictionary<string, object?>>
{
public override bool CanConvert(Type typeToConvert)
{
return typeToConvert == typeof(Dictionary<string, object>)
|| typeToConvert == typeof(Dictionary<string, object?>);
}
public override Dictionary<string, object?> Read(
ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options)
{
if (reader.TokenType != JsonTokenType.StartObject)
{
throw new JsonException($"JsonTokenType was of type {reader.TokenType}, only objects are supported");
}
var dictionary = new Dictionary<string, object?>();
while (reader.Read())
{
if (reader.TokenType == JsonTokenType.EndObject)
{
return dictionary;
}
if (reader.TokenType != JsonTokenType.PropertyName)
{
throw new JsonException("JsonTokenType was not PropertyName");
}
var propertyName = reader.GetString();
if (string.IsNullOrWhiteSpace(propertyName))
{
throw new JsonException("Failed to get property name");
}
reader.Read();
dictionary.Add(propertyName!, ExtractValue(ref reader, options));
}
return dictionary;
}
public override void Write(
Utf8JsonWriter writer, Dictionary<string, object?> value, JsonSerializerOptions options)
{
// We don't need any custom serialization logic for writing the json.
// Ideally, this method should not be called at all. It's only called if you
// supply JsonSerializerOptions that contains this JsonConverter in it's Converters list.
// Don't do that, you will lose performance because of the cast needed below.
// Cast to avoid infinite loop: https://github.com/dotnet/docs/issues/19268
JsonSerializer.Serialize(writer, (IDictionary<string, object?>)value, options);
}
private object? ExtractValue(ref Utf8JsonReader reader, JsonSerializerOptions options)
{
switch (reader.TokenType)
{
case JsonTokenType.String:
if (reader.TryGetDateTime(out var date))
{
return date;
}
return reader.GetString();
case JsonTokenType.False:
return false;
case JsonTokenType.True:
return true;
case JsonTokenType.Null:
return null;
case JsonTokenType.Number:
if (reader.TryGetInt64(out var result))
{
return result;
}
return reader.GetDecimal();
case JsonTokenType.StartObject:
return Read(ref reader, null!, options);
case JsonTokenType.StartArray:
var list = new List<object?>();
while (reader.Read() && reader.TokenType != JsonTokenType.EndArray)
{
list.Add(ExtractValue(ref reader, options));
}
return list;
default:
throw new JsonException($"'{reader.TokenType}' is not supported");
}
}
}
We start by implementing the Read and Write method. The Read method will be used for deserialization and the Write method is used when serializing.
Some notes:
- If it's an object, we recursively call
ExtractValue. - If it's an array, we create a list and read until the end of the array. For each item in the array we call
ExtractValue. - If the string can be parsed to a valid DateTime we return a
DateTime, otherwise we return the string as is. - If a number can be parsed to a valid
longwe return that, otherwise we return adecimal.
You get the point, we can choose exactly how we want to handle the different types of properties.
Time to test our custom converter.
There's a couple of different ways to register our custom converter. I will use the JsonConverterAttribute.
MyInput will now look like this:
public class MyInput
{
public string Name { get; set; }
[JsonConverter(typeof(DictionaryStringObjectJsonConverter))]
public Dictionary<string, object> Data { get; set; }
}
If we now send the same payload and inspect the input, it will look like this:
Great success :)
Let's try it with a more complex payload:
{
"name": "whatevs",
"data": {
"string": "string",
"int": 1,
"bool": true,
"date": "2020-01-23T00:01:02Z",
"decimal": 12.345,
"null": null,
"array": [1, 2, 3],
"objectArray": [{
"string": "string",
"int": 1,
"bool": true
},
{
"string": "string2",
"int": 2,
"bool": true
}
],
"object": {
"string": "string",
"int": 1,
"bool": true
}
}
}
In my case I didn't need to care about the Write method since we are using Newtonsoft for serializing the request towards SendGrid. But we still need to do something in that method so we just call the Serialize method on the JsonSerializer.
public class DictionaryStringObjectJsonConverter : JsonConverter<Dictionary<string, object>>
{
.......
public override void Write(Utf8JsonWriter writer, Dictionary<string, object> value, JsonSerializerOptions options)
{
JsonSerializer.Serialize(writer, value, options);
}
}
When doing that we rely on the default behaviour of System.Text.Json. If we want/need more control, we can do something like this:
public override void Write(Utf8JsonWriter writer, Dictionary<string, object> value, JsonSerializerOptions options)
{
writer.WriteStartObject();
foreach (var key in value.Keys)
{
HandleValue(writer, key, value[key]);
}
writer.WriteEndObject();
}
private static void HandleValue(Utf8JsonWriter writer, string key, object objectValue)
{
if (key != null)
{
writer.WritePropertyName(key);
}
switch (objectValue)
{
case string stringValue:
writer.WriteStringValue(stringValue);
break;
case DateTime dateTime:
writer.WriteStringValue(dateTime);
break;
case long longValue:
writer.WriteNumberValue(longValue);
break;
case int intValue:
writer.WriteNumberValue(intValue);
break;
case float floatValue:
writer.WriteNumberValue(floatValue);
break;
case double doubleValue:
writer.WriteNumberValue(doubleValue);
break;
case decimal decimalValue:
writer.WriteNumberValue(decimalValue);
break;
case bool boolValue:
writer.WriteBooleanValue(boolValue);
break;
case Dictionary<string, object> dict:
writer.WriteStartObject();
foreach (var item in dict)
{
HandleValue(writer, item.Key, item.Value);
}
writer.WriteEndObject();
break;
case object[] array:
writer.WriteStartArray();
foreach (var item in array)
{
HandleValue(writer, item);
}
writer.WriteEndArray();
break;
default:
writer.WriteNullValue();
break;
}
}
private static void HandleValue(Utf8JsonWriter writer, object value)
{
HandleValue(writer, null, value);
}
We do more or less the same thing as in the Read method, we choose how to handle the different types of properties.
Bonus - ModelBinder
If you want to use a Dictionary<string, object> without wrapping it in a model you can use a custom model binder like this:
[HttpPost]
public IActionResult Default([ModelBinder(typeof(DictionaryObjectJsonModelBinder))] Dictionary<string, object> data)
{
return new OkObjectResult(data);
}
public class DictionaryObjectJsonModelBinder : IModelBinder
{
private readonly ILogger<DictionaryObjectJsonModelBinder> _logger;
private static readonly JsonSerializerOptions DefaultJsonSerializerOptions = new JsonSerializerOptions(JsonSerializerDefaults.General)
{
Converters = { new DictionaryStringObjectJsonConverter()}
};
public DictionaryObjectJsonModelBinder(ILogger<DictionaryObjectJsonModelBinder> logger)
{
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
}
public async Task BindModelAsync(ModelBindingContext bindingContext)
{
if (bindingContext == null)
{
throw new ArgumentNullException(nameof(bindingContext));
}
if (bindingContext.ModelType != typeof(Dictionary<string, object>))
{
throw new NotSupportedException($"The '{nameof(DictionaryObjectJsonModelBinder)}' model binder should only be used on Dictionary<string, object>, it will not work on '{bindingContext.ModelType.Name}'");
}
try
{
var data = await JsonSerializer.DeserializeAsync<Dictionary<string, object>>(bindingContext.HttpContext.Request.Body, DefaultJsonSerializerOptions);
bindingContext.Result = ModelBindingResult.Success(data);
}
catch (Exception e)
{
_logger.LogError(e, "Error when trying to model bind Dictionary<string, object>");
bindingContext.Result = ModelBindingResult.Failed();
}
}
}
Bonus - Benchmarks
Deserialization
BenchmarkDotNet=v0.12.1, OS=Windows 10.0.19041.450 (2004/?/20H1)
Intel Core i7-8750H CPU 2.20GHz (Coffee Lake), 1 CPU, 12 logical and 6 physical cores
.NET Core SDK=5.0.100-preview.7.20366.6
[Host] : .NET Core 5.0.0 (CoreCLR 5.0.20.36411, CoreFX 5.0.20.36411), X64 RyuJIT
.NET Core 5.0 : .NET Core 5.0.0 (CoreCLR 5.0.20.36411, CoreFX 5.0.20.36411), X64 RyuJIT
Job=.NET Core 5.0 Runtime=.NET Core 5.0 InvocationCount=1
UnrollFactor=1
| Method | Mean | Error | StdDev | Median | Ratio | RatioSD | Gen 0 | Gen 1 | Gen 2 | Allocated |
|-------------------- |---------:|---------:|---------:|---------:|------:|--------:|------:|------:|------:|----------:|
| Default | 29.14 μs | 0.859 μs | 2.394 μs | 28.00 μs | 1.00 | 0.00 | - | - | - | 2.15 KB |
| CustomJsonConverter | 21.59 μs | 0.758 μs | 2.163 μs | 20.60 μs | 0.75 | 0.09 | - | - | - | 3.62 KB |
Our custom JsonConverter is faster than using the default behaviour, we allocate a bit more though.
Serialization
BenchmarkDotNet=v0.12.1, OS=Windows 10.0.19041.450 (2004/?/20H1)
Intel Core i7-8750H CPU 2.20GHz (Coffee Lake), 1 CPU, 12 logical and 6 physical cores
.NET Core SDK=5.0.100-preview.7.20366.6
[Host] : .NET Core 5.0.0 (CoreCLR 5.0.20.36411, CoreFX 5.0.20.36411), X64 RyuJIT
.NET Core 5.0 : .NET Core 5.0.0 (CoreCLR 5.0.20.36411, CoreFX 5.0.20.36411), X64 RyuJIT
Job=.NET Core 5.0 Runtime=.NET Core 5.0
| Method | Mean | Error | StdDev | Ratio | Gen 0 | Gen 1 | Gen 2 | Allocated |
|----------------------------------------------- |---------:|----------:|----------:|------:|-------:|------:|------:|----------:|
| Default | 3.107 μs | 0.0108 μs | 0.0101 μs | 1.00 | 0.2022 | - | - | 960 B |
| DictionaryStringObjectJsonConverter | 1.049 μs | 0.0083 μs | 0.0078 μs | 0.34 | 0.0935 | - | - | 440 B |
| DictionaryStringObjectJsonConverterCustomWrite | 1.052 μs | 0.0119 μs | 0.0111 μs | 0.34 | 0.0935 | - | - | 440 B |
Using the custom JsonConverter when serializing are both faster and allocates less than the default behaviour. There's no difference between the converters.
All code (together with benchmarks and tests) can be found on GitHub.