Filter Application Insights events in ASP.NET Core

Application Insights ingests lots of data: requests, traces, events, metrics, etc. If your web site has lots of users, the amount of data can be huge. This mean you pay for this data. While Application Insights is cheap, you may want to reduce the bill. One way is to sample the data. In short, you send only xx% of the events. This is the simplest solution. The other solution is to filter low value events for your usage. For instance, if you use Application Insights to diagnose errors, you may want to filter success requests.

The Application Insights SDK provides an interface to manipulate the telemetry events: ITelemetryProcessor. This interface allows to change the events, for instance to add a property or remove sensible data. If also allow to filter the event. Telemetry processors are chained. Each telemetry processor handle the event and pass it to the next processor. A telemetry processor can also choose to discard the event by not sending it to the next processor. We'll use this to filter the events.

First, create a class that implements ITelemetryProcessor. This class will filter request events with status code 200.

public class MyTelemetryProcessor : ITelemetryProcessor
{
    private ITelemetryProcessor _next;

    public MyTelemetryProcessor(ITelemetryProcessor next)
    {
        // Next TelemetryProcessor in the chain
        _next = next;
    }

    public void Process(ITelemetry item)
    {
        if (item is RequestTelemetry request)
        {
            if (request.ResponseCode == "200")
            {
                // Filter the event
                return;
            }
        }

        // Send the item to the next TelemetryProcessor
        _next.Process(item);
    }
}

Then, you have to configure Application Insights to use the MyTelemetryProcessor. In the Startup.cs file, add the following code in the Configure method:

public void Configure(IApplicationBuilder app)
{
    var configuration = app.ApplicationServices.GetService<TelemetryConfiguration>();
    configuration.TelemetryProcessorChainBuilder.Use(next => new MyTelemetryProcessor(next));
    configuration.TelemetryProcessorChainBuilder.Build();

    // ...
}

You filter is now configured and active. Now you can customize the filter to remove other kind of events. For instance, you can remove some specific traces based on their category:

public void Process(ITelemetry item)
{
    if (item is TraceTelemetry trace)
    {
        if (trace.Properties.TryGetValue("CategoryName", out var category) &&
            category == "Microsoft.AspNetCore.StaticFiles.StaticFileMiddleware")
        {
            return;
        }
    }

    _next.Process(item);
}

If you have a public website, you may have lots of requests to the WordPress login page /wp-login.php. If you are not using a WordPress website, you can filter those events:

public void Process(ITelemetry item)
{
    if (item is RequestTelemetry request)
    {
        // The url is not handled, so 404
        if (string.Equals(request.ResponseCode, "404", StringComparison.Ordinal))
        {
            var path = request.Url?.AbsolutePath;
            if (path != null && path.EndsWith("/wp-login.php", StringComparison.OrdinalIgnoreCase))
            {
                return;
            }
        }
    }

    Next.Process(item);
}

Conclusion

The Application Insights SDK allows you to control the data sent to Azure. You can augment events with custom data, edit the events to remove sensitive data, or filter unneeded events. So, you can better detect & diagnose issues and understand usage for your web apps.

Testing an ASP.NET Core application using TestServer

When Microsoft has designed ASP.NET Core, testing was clearly part of the design. Using dependency injection, you can unit test your middlewares and controllers. But you can also use the TestServer to make integration tests. This means you can test your full web application without an IIS server or any external thing. It's a fully in process server. A simple example will be much clearer. Let's use the following controller:

public class Startup
{
    public void ConfigureServices(IServiceCollection services)
    {
        services.AddMvc();
    }

    public void Configure(IApplicationBuilder app, IHostingEnvironment env)
    {
        app.UseMvc();
    }
}

[Route("api/[controller]")]
public class ValuesController : Controller
{
    [HttpGet]
    public IEnumerable<string> Get()
    {
        return new string[] { "value1", "value2" };
    }
}
  1. Add a new MSTest project
  2. Add a reference to the web project
  3. Add the NuGet package: Microsoft.AspNetCore.TestHost
  4. Create a test
[TestClass]
public class Tests
{
    [TestMethod]
    public async Task TestMethod1()
    {
        var webHostBuilder =
              new WebHostBuilder()
                    .UseEnvironment("Test") // You can set the environment you want (development, staging, production)
                    .UseStartup<Startup>(); // Startup class of your web app project

        using (var server = new TestServer(webHostBuilder))
        using (var client = server.CreateClient())
        {
            string result = await client.GetStringAsync("/api/values");
            Assert.AreEqual("[\"value1\",\"value2\"]", result);
        }
    }
}

This was a basic example. In a more complex scenario, you may want to replace some service using DI. ASP.NET provides mecanisms to support multiple environments. For instance, you can create one Configure<Environment name>Services method per environment. If no method is defined for the environment, the ConfigureServices will be used. One way to personalize the DI for testing is to create a class that inherits from Startup and add the method ConfigureTestServices:

public class TestStartup : Startup
{
    public TestStartup(IHostingEnvironment env) : base(env)
    {
    }

    public void ConfigureTestServices(IServiceCollection services)
    {
        // Configure services for Test environment
    }
}

Then, use the TestStartup class to initialize the TestServer:

[TestMethod]
public async Task TestMethod1()
{
    var webHostBuilder =
            new WebHostBuilder()
                .UseEnvironment("Test")
                .UseStartup<TestStartup>();
    // ...
}

Test Explorer

This was just a quick introduction to integration testing with ASP.NET Core. You can find more resources in the documentation.

Validating user with cookie authentication in ASP.NET Core 2

In a previous post, I wrote about the cookie authentication in ASP.NET Core 2. The cookie authentication does 2 things:

  • Write a cookie with encrypted data when the user logs in
  • Read the cookie, decrypt it, and set the request identity (Request.User.Identity)

When it read the cookie and set the identity, it doesn't check the user actually exists. For instance, John logs in on browser A, then, he deletes his account on computer B. When he goes back to the website on browser A, the cookie is still valid, while his account doesn't exist anymore in the database. If you don't care, you may think the user is logged in, whereas it shouldn't.

Of course, the cookie authentication has a hook to validate the user and prevent this kind of situation. Let's see how to use it.

First, you need to handle the OnValidatePrincipal method in the ConfigureServices method:

public void ConfigureServices(IServiceCollection services)
{
    services.AddAuthentication(options =>
    {
        options.DefaultSignInScheme = CookieAuthenticationDefaults.AuthenticationScheme;
        options.DefaultAuthenticateScheme = CookieAuthenticationDefaults.AuthenticationScheme;
        options.DefaultChallengeScheme = CookieAuthenticationDefaults.AuthenticationScheme;
    })
        .AddCookie(options =>
        {
            options.Events.OnValidatePrincipal = PrincipalValidator.ValidateAsync;
        });
}

Then, you can check the principal corresponds to an actual user. In case the user is invalid, you can prevent the user from being authenticated by calling the RejectPrincipal method. The validation logic is depend of your application. So, the following code is just an example:

public static class PrincipalValidator
{
    public static async Task ValidateAsync(CookieValidatePrincipalContext context)
    {
        if (context == null) throw new System.ArgumentNullException(nameof(context));

        var userId = context.Principal.Claims.FirstOrDefault(claim => claim.Type == ClaimTypes.NameIdentifier)?.Value;
        if (userId == null)
        {
            context.RejectPrincipal();
            return;
        }

        // Get an instance using DI
        var dbContext = context.HttpContext.RequestServices.GetRequiredService<IdentityDbContext>();
        var user = await dbContext.Users.FindByIdAsync(userId);
        if (user == null)
        {
            context.RejectPrincipal();
            return;
        }
    }
}

The cookie authentication is very easy to use. However, authentication is not as easy as you think. It's very easy to forget something, and get unexpected behaviors.

Use brotli compression with ASP.NET Core

Brotli is a compression algorithm. In general, it provides better compression results than gzip. For instance, the size of the default page of an ASP.NET Core project using gzip is 1797 bytes, and using Brotli, it's only 1333 bytes. So it's a 25% reduction!

You can use Brotli with most of the major browsers (source):

Brotli browser support

So, it's time to support Brotli in addition to gzip!

I've already written about the gzip compression in ASP.NET Core using the ReponseCompression package: Enabling gzip compression with ASP.NET Core. Today, we'll see how to provide your own compression method, and support brotli.

Implementation

If you check the code in the CoreFX Lab repository on GitHub regulary, you may have notice there is a new project: System.IO.Compression.Brotli. This project contains an API to compress and decompress data using the Brotli algorithm. Of course, this package is a preview, and is not supported yet.

CoreFx Lab projects are not published to NuGet.org. Instead, there are published to a public MyGet repository. So, to get this package, you must declare the feed in Visual Studio:

https://dotnet.myget.org/F/dotnet-corefxlab/api/v3/index.json

Or, you can create a nuget.config at the root of the solution with the following content:

<configuration>
  <packageSources>
    <add key="dotnet-corefxlab" value="https://dotnet.myget.org/F/dotnet-corefxlab/" />
  </packageSources>
</configuration>

Then, you can add the 2 following packages:

  • System.IO.Compression.Brotli
  • Microsoft.AspNetCore.ResponseCompression

The response compression middleware look at the headers of the request and check if a compression provider can handle one of the accepted encoding. By default, only the gzip algorithm is supported. You can support custom encodings by implementing a custom compression provider. A custom provider must implement the ICompressionProvider interface. This interface contains a property with the name of the encoding, and a method to create a compression stream. The brotli package provide a BrotliStream class, so the implementation is very easy:

public class BrotliCompressionProvider : ICompressionProvider
{
    public string EncodingName => "br";
    public bool SupportsFlush => true;

    public Stream CreateStream(Stream outputStream)
    {
        return new BrotliStream(
            outputStream,
            CompressionMode.Compress,
            leaveOpen: false,
            bufferSize: 65520,
            quality: CompressionLevel.Optimal); // very costly, may not be appropriate for a web server
    }
}

Finally, you can use the custom provider:

public class Startup
{
    public void ConfigureServices(IServiceCollection services)
    {
        services.AddMvc();

        services.AddResponseCompression(options =>
        {
            options.Providers.Add(new BrotliCompressionProvider());
        });
    }

    public void Configure(IApplicationBuilder app)
    {
        app.UseResponseCompression();
        app.UseStaticFiles();
        app.UseMvcWithDefaultRoute();
    }
}

You can check the server compress the response using brotli by looking at the response header:

Conclusion

The Brotli compression package is in preview, and not ready to go in production. So, use it with caution. However, this is another example of how easy it is to extend ASP.NET Core 😃 Plus, you should really monitor the corefx repository on GitHub. You may find some gems such as the Brotli compression project.

Cookie authentication in ASP.NET Core 2 without ASP.NET Identity

The default ASP.NET Core 2 web template provides lots of code to authenticate users. My screen is not big enough to display all the files in the solution explorer. The template offers lots of functionalities: users can log in using username/password or using an external provider such as Google or Microsoft. You can use two-factor authentication. And so on.

You can rely on this code and use it as is. Or, if you just want to use cookie authentication, you can delete all these files, and create the sign up and log in pages by yourself. As you'll see in this post, the cookie authentication is very easy to use 😃

Register the authentication middlewares

First, add the NuGet package Microsoft.AspNetCore.Authentication.Cookies. Then, you must configure and register the authentication middleware.

public void ConfigureServices(IServiceCollection services)
{
    services.AddAuthentication(options =>
    {
        options.DefaultSignInScheme = CookieAuthenticationDefaults.AuthenticationScheme;
        options.DefaultAuthenticateScheme = CookieAuthenticationDefaults.AuthenticationScheme;
        options.DefaultChallengeScheme = CookieAuthenticationDefaults.AuthenticationScheme;
    })
        .AddCookie();

    services.AddMvc();
}

public void Configure(IApplicationBuilder app, IHostingEnvironment env)
{
    app.UseAuthentication();
    app.UseMvc();
}

You are now ready to use the authentication functionalities 😃

Create the LogIn model

public class LogInModel
{
    [Required]
    public string Username { get; set; }

    [Required]
    [DataType(DataType.Password)]
    public string Password { get; set; }

    public bool RememberMe { get; set; }
}

Create the LogIn page

ASP.NET Core 2 has a new way to create page: Razor Pages. Razor pages provide a way to create pages in a more understandable way. But you can easily transpose this code to use a Controller by just copying the code in the controller.

I create a new page Pages/login.cshtml. The page contains a form with two fields username and password and some bootstrap classes. A Razor Page starts with @page.

@page
<form method="post">
    <div asp-validation-summary="All" class="text-danger"></div>

    <div class="form-group">
        <label asp-for="LogInModel.Username" class="col-md-2 control-label"></label>
        <div class="col-md-10">
            <input asp-for="LogInModel.Username" class="form-control" />
            <span asp-validation-for="LogInModel.Username" class="text-danger"></span>
        </div>
    </div>

    <div class="form-group">
        <label asp-for="LogInModel.Password" class="col-md-2 control-label"></label>
        <div class="col-md-10">
            <input asp-for="LogInModel.Password" class="form-control" />
            <span asp-validation-for="LogInModel.Password" class="text-danger"></span>
        </div>
    </div>

    <div class="form-group">
        <div class="col-md-offset-2 col-md-10">
            <div class="checkbox">
                <label asp-for="RememberMe">
                    <input asp-for="RememberMe" />
                    @Html.DisplayNameFor(m => m.RememberMe)
                </label>
            </div>
        </div>
    </div>

    <div class="form-group">
        <div class="col-md-offset-2 col-md-10">
            <button type="submit" class="btn btn-default">Log In</button>
        </div>
    </div>
</form>

Let's add some code to handle the button click. You can add the code directly in the page or create a PageModel class. The code is very short, so I'll add it in the page.

@page

@using Microsoft.AspNetCore.Authentication;
@using Microsoft.AspNetCore.Authentication.Cookies;
@using System.Security.Claims;

@functions {
    [BindProperty] // Bind on Post
    public LogInModel LogInModel { get; set; }

    public async Task<IActionResult> OnPostAsync()
    {
        if (ModelState.IsValid)
        {
            var isValid = true; // TODO Validate the username and the password with your own logic
            if (!isValid)
            {
                ModelState.AddModelError("", "username or password is invalid");
                return Page();
            }

            // Create the identity from the user info
            var identity = new ClaimsIdentity(CookieAuthenticationDefaults.AuthenticationScheme, ClaimTypes.Name, ClaimTypes.Role);
            identity.AddClaim(new Claim(ClaimTypes.NameIdentifier, LogInModel.Username));
            identity.AddClaim(new Claim(ClaimTypes.Name, LogInModel.Username));

            // Authenticate using the identity
            var principal = new ClaimsPrincipal(identity);
            await HttpContext.SignInAsync(CookieAuthenticationDefaults.AuthenticationScheme, principal, new AuthenticationProperties { IsPersistent = LogInModel.RememberMe });

            return RedirectToPage("Index");
        }

        return Page();
    }
}

<form>
    ...
</form>

As you can see, it's very simple 😃 Indeed, you just need to create an identity with the user data and call HttpContext.SignInAsync(...) with the user data.

Now you can sign out using a single line of code:

await HttpContext.SignOutAsync(CookieAuthenticationDefaults.AuthenticationScheme);

Conclusion

The cookie authentication is very easy to use. If you don't need to use all the functionalities provided by the default template, you can consider implement them by yourself.