TokenValidationParameters ignored if policy is added to Authorise attribute

Question

I'm using AspNetCore.All 2.0

I have a controller that uses the Authorize attribute like so:

[Authorize(AuthenticationSchemes = JwtBearerDefaults.AuthenticationScheme]

When I pass a JWT token it fails as expected because it's expired. However when I add a policy like below it always passes with a 200. I should still get a 401 because it's an expired token, right? What could I be doing wrong?

[Authorize(AuthenticationSchemes = JwtBearerDefaults.AuthenticationScheme, Policy = "Over21")]

Below are my token validation parameters. Please note that adding the policy doesn't just affect the expiration it could be any of the parameters

        .AddJwtBearer(cfg =>
        {
            cfg.RequireHttpsMetadata = false;
            cfg.SaveToken = true;

            cfg.TokenValidationParameters = new TokenValidationParameters()
            {
                RequireExpirationTime = true,
                ValidateLifetime = true,
                ValidateIssuerSigningKey = false,
                ValidateIssuer = false,
                ValidateAudience = false,
                ValidIssuer = "https://localhost",
                ValidAudience = "https://localhost",
                IssuerSigningKey = new SymmetricSecurityKey(Encoding.UTF8.GetBytes("someveryveryveryverylongkey")),                    
                ClockSkew = TimeSpan.Zero

            };
        });