Unit Testing custom eager validation on top of lazy validation from IOptions .NET Core 1.1 and up

Question

This is not a question but a case study that was tried on by me where questions have not been asked. In case anyone else tries this kind of idiotic unit testing in the future, these are my findings:

While trying to implement eager validation, as it is not supported currently by .NET Core 3.1, but as the documentation states at the bottom of the section https://learn.microsoft.com/en-us/aspnet/core/fundamentals/configuration/options?view=aspnetcore-3.1#options-post-configuration:

Eager validation (fail fast at startup) is under consideration for a future release.

You cannot test programmatically the lazy validation from accessing the option in question if you've implemented custom eager validation.

This is what I did:

Created config class

public class TestOptions : IValidateObject // for eager validation config
{
    [Required]
    public string Prop { get; set; }

    public IEnumerable<ValidationResult> Validate(ValidationContext validationContext)
    {
        if (string.IsNullOrEmpty(this.Prop))
            yield return new ValidationResult($"{nameof(this.Prop)} is null or empty.");
    }
}

Added the configuration in my lib that I'm testing:

public static void AddConfigWithValidation(this IServiceCollection services, Action<TestOptions> options)
{
    var opt = new TestOptions();
    options(opt);

    // eager validation
    var validationErrors = opt.Validate(new ValidationContext(opt)).ToList();

    if (validationErrors.Any())
        throw new ApplicationException($"Found {validationErrors.Count} configuration error(s): {string.Join(',', validationErrors)}");

    // lazy validation with validate data annotations from IOptions
    services.AddOptions<TestOptions>()
        .Configure(o =>
        {
            o.Prop = opt.Prop
        })
        .ValidateDataAnnotations();
}

And the test looks like this

public class MethodTesting
{
    private readonly IServiceCollection _serviceCollection;

    public MethodTesting()
    {
        _serviceCollection = new ServiceCollection();
    }

    // this works as it should
    [Fact] 
    public void ServiceCollection_Eager_Validation()
    {
        var opt = new TestOptions { Prop = string.Empty };
        Assert.Throws<ApplicationException>(() => _serviceCollection.AddConfigWithValidation(o =>
        {
            o.Prop = opt.Prop
        });
    }

    // this does not work
    [Fact]
    public void ServiceCollection_Lazy_Validation_Mock_Api_Start()
    {
        var builder = new ConfigurationBuilder()
            .SetBasePath(Directory.GetCurrentDirectory())
            .AddJsonFile("settings.json", optional: false, reloadOnChange: true);

        _configuration = builder.Build();

        var opt = _configuration.GetSection(nameof(TestOptions)).Get<TestOptions>();

        _serviceCollection.AddConfigWithValidation(o =>
        {
            o.Prop = opt.Prop
        });

        // try to mock a disposable object, sort of how the API works on subsequent calls
        using (var sb = _serviceCollection.BuildServiceProvider())
        {
            var firstValue = sb.GetRequiredService<IOptionsSnapshot<TestOptions>>().Value;
            firstValue.Should().BeEquivalentTo(opt);
        }

        // edit the json file programmatically, trying to trigger a new IOptionsSnapshot<>
        var path = $"{Directory.GetCurrentDirectory()}\\settings.json";

        var jsonString = File.ReadAllText(path);

        var concreteObject = Newtonsoft.Json.JsonConvert.DeserializeObject<TestObject>(jsonString);

        concreteObject.TestObject.Prop = string.Empty;

        File.WriteAllText(path, Newtonsoft.Json.JsonConvert.SerializeObject(concreteObject));

        using (var sb = _serviceCollection.BuildServiceProvider())
        {
            // this does not work, as the snapshot is still identical to the first time it is pulled
            Assert.Throws<OptionsValidationException>(() => _serviceCollection.BuildServiceProvider().GetRequiredService<IOptionsSnapshot<TestOptions>>().Value);
        }
    }

    // this does not work as well
    [Fact]
    public void ServiceCollection_Lazy_Validation_Mock_Api_Start_With_Direct_Prop_Assignation()
    {
        var builder = new ConfigurationBuilder()
            .SetBasePath(Directory.GetCurrentDirectory())
            .AddJsonFile("settings.json", optional: false, reloadOnChange: true);

        _configuration = builder.Build();

        var opt = _configuration.GetSection(nameof(TestOptions)).Get<TestOptions>();

        _serviceCollection.AddConfigWithValidation(o =>
        {
            o.Prop = opt.Prop
        });

        using (var sb = _serviceCollection.BuildServiceProvider())
        {
            var firstValue = sb.GetRequiredService<IOptionsSnapshot<TestOptions>>().Value;
            firstValue.Should().BeEquivalentTo(opt);
        }

        var prop = _configuration["TestOptions:Prop"];

        _configuration["TestOptions:Prop"] = string.Empty;

        // this returns a new value
        var otherProp = _configuration["TestOptions:Prop"];

        using (var sb = _serviceCollection.BuildServiceProvider())
        {
            // this does not work, the snapshot is not yet modified, however, calling _configuration.GetSection(nameof(TestOptions)).Get<TestOptions>(); does return the new TestOptions.

            Assert.Throws<OptionsValidationException>(() => _serviceCollection.BuildServiceProvider().GetRequiredService<IOptionsSnapshot<TestOptions>>().Value);
        }

    }

    public class TestObject
    {
        public TestOptions TestOptions { get; set; }
    }

My settings.json looked like:

{
    "TestOptions": {
        "Prop": "something"
    }
}

A solution to get this up and running as a test, is to add an optional parameter or an overloaded method with an optional parameter that enforces or not eager validation and test that the lazy validation works properly when the eager is deactivated.

Please note that this is not perfect, but a method of test for people who want to test how the eager and lazy validation can be tested when the options provided are from a source that gets updated but the apps do not get restarted.

If you have suggestions, questions or want to discuss on the subject at hand, feel free to use the comment section

Answer 1

Looks like I found something that can satisfy the lazy validation parable that has eager validation on top of it. Please note that IValidatableObject vs IValidateOptions for eager validation does not make a difference, so please use whatever fits you best!

The solution:

public static void AddConfigWithValidation(this IServiceCollection services, IConfiguration config)
{
    // lazy validation
    services.Configure<TestOptions>(config.GetSection(nameof(TestOptions))).AddOptions<TestOptions>().ValidateDataAnnotations();

    var model = config.GetSection(nameof(TestOptions)).Get<TestOptions>();

    // eager validation
    var validationErrors = model.Validate(new ValidationContext(model)).ToList();

    if (validationErrors.Any())
        throw new ApplicationException($"Found {validationErrors.Count} configuration error(s): {string.Join(',', validationErrors)}");
}

And in test method:

[Fact]
public void ServiceCollection_Lazy_Validation_Mock_Api_Start()
{
    var builder = new ConfigurationBuilder()
        .SetBasePath(Directory.GetCurrentDirectory())
        .AddJsonFile("settings.json", optional: false, reloadOnChange: true);

    _configuration = builder.Build();

    var opt = _configuration.GetSection(nameof(TestOptions)).Get<TestOptions>();

    _serviceCollection.AddConfigWithValidation(_configuration);

    var firstValue = _serviceCollection.BuildServiceProvider().GetRequiredService<IOptionsSnapshot<TestOptions>>().Value;

    firstValue.Should().BeEquivalentTo(opt);

    // edit the json file programmatically, trying to trigger a new IOptionsSnapshot<>
    var path = $"{Directory.GetCurrentDirectory()}\\settings.json";

    var jsonString = File.ReadAllText(path);

    var concreteObject = Newtonsoft.Json.JsonConvert.DeserializeObject<TestObject>(jsonString);

    concreteObject.TestObject.Prop = string.Empty;

    File.WriteAllText(path, Newtonsoft.Json.JsonConvert.SerializeObject(concreteObject));

    _configuration = builder.Build(); // rebuild the config builder

    System.Threading.Thread.Sleep(1000); // let it propagate the change

    // error is thrown, lazy validation is triggered.
    Assert.Throws<OptionsValidationException>(() => _serviceCollection.BuildServiceProvider().GetRequiredService<IOptionsSnapshot<TestOptions>>().Value);
}

This now works correctly and the lazy validation is triggered.

Please note that I have tried to mimic their implementation for IConfiguration listening to change but it did not work.

Answer 2

For eager validation, I stumbled across this post on github (can't take any credit for it, but it seems to do the trick)

I use as follows...

    public static IServiceCollection AddOptionsWithEagerValidation<TOptions, TOptionsValidator>(this IServiceCollection services,
            Action<TOptions> configAction,
            ILogger<ServiceCollection>? logger = default)
        where TOptions : class, new()
        where TOptionsValidator : class, IValidator, new()
    {
        services
            .AddOptions<TOptions>()
            .Configure(configAction)
            .Validate(x =>
            {
                return ValidateConfigurationOptions<TOptions, TOptionsValidator>(x, logger);
            })
            .ValidateEagerly();

        return services;
    }

I do some custom stuff during Configure and then perform my own validation using Fluent Validation in Validate. ValidateEagerly causes the IStatupFilter to validate the options early.