Photo by Reuben

Introduction

We have the following Polly policy:

public static IAsyncPolicy<HttpResponseMessage> MyRetryPolicy(
        int attempts = 3, Func<int, TimeSpan>? sleepDurationProvider = null)
{
    sleepDurationProvider ??= (retryAttempt => TimeSpan.FromSeconds(Math.Pow(2, retryAttempt)));
    return HttpPolicyExtensions
           .HandleTransientHttpError()
           .WaitAndRetryAsync(retryAttempts, sleepDurationProvider);
}

It's pretty basic, we say that we will retry failing calls a maximum of three times with an exponential wait between the calls. All parameters are optional to allow for easy customization.

The HandleTransientHttpError method handles the following scenarios:

Network failures (as HttpRequestException)
HTTP 5XX status codes (server errors)
HTTP 408 status code (request timeout)

To use the policy, we can configure any HttpClient like this:

services.AddHttpClient<MyClient>(c =>
  c.BaseAddress = new Uri("https://josef.codes"))
.AddPolicyHandler(HttpClientPolicies.MyRetryPolicy())

Testing our policy

We have the following requirements for our tests:

• We want to test that the code retries failing requests.
• We want the test to be fast, we don't want to wait 10+ seconds for all the retries to complete.

Since the policy is configured as an extension method on IHttpClientBuilder, we need to create our own IServiceCollection and use it in our test. I will show you a complete test and then I'll explain the different "concepts" used in the test.

[Fact]
public async Task ShouldRetryFailingRequestsThreeTimes()
{
    var services = new ServiceCollection();
    var fakeHttpDelegatingHandler = new FakeHttpDelegatingHandler(
        _ => Task.FromResult(new HttpResponseMessage(HttpStatusCode.GatewayTimeout)));
    var policy =
        HttpClientPolicies.DefaultRetryPolicy(sleepDurationProvider: _ => TimeSpan.FromMilliseconds(1));
    services.AddHttpClient("my-httpclient", client =>
    {
        client.BaseAddress = new Uri("http://any.localhost");
    })
    .AddPolicyHandler(policy)
    .AddHttpMessageHandler(() => fakeHttpDelegatingHandler);
    var serviceProvider = services.BuildServiceProvider();
    using var scope = serviceProvider.CreateScope();
    var sut = scope.ServiceProvider.GetRequiredService<IHttpClientFactory>().CreateClient("my-httpclient");
    var request = new HttpRequestMessage(HttpMethod.Get, "/any");

    var result = await sut.SendAsync(request);

    result.StatusCode.ShouldBe(HttpStatusCode.GatewayTimeout);
    fakeHttpDelegatingHandler.Attempts.ShouldBe(3);
}

FakeHttpDelegatingHandler

This class is used to prevent the actual sending of the request. Instead of sending the request, it will run the supplied responseFactory func and increment the Attempts property. This allows us to keep track of how many times this method has been called == how many retries has been made.

public class FakeHttpDelegatingHandler : DelegatingHandler
{
    private readonly Func<int, Task<HttpResponseMessage>> _responseFactory;
    public int Attempts { get; private set; }

    public FakeHttpDelegatingHandler(Func<int, Task<HttpResponseMessage>> responseFactory)
    {
        _responseFactory = responseFactory ?? throw new ArgumentNullException(nameof(responseFactory));
    }

    protected override async Task<HttpResponseMessage> SendAsync(
        HttpRequestMessage request, CancellationToken cancellationToken)
    {
        return await _responseFactory.Invoke(++Attempts);
    }
}

In our test we've configured the FakeHttpDelegatingHandler to just return a Gateway Timeout:

var fakeHttpDelegatingHandler = new FakeHttpDelegatingHandler(
        _ => Task.FromResult(new HttpResponseMessage(HttpStatusCode.GatewayTimeout)));

We then use a custom sleepDurationProvider to set the wait time between retries to 1ms (instead of the default value of multiple seconds):

var policy =
    HttpClientPolicies.DefaultRetryPolicy(sleepDurationProvider: _ => TimeSpan.FromMilliseconds(1));

We then configure a named http client using HttpClientFactory like this:

services.AddHttpClient("my-httpclient", client =>
{
    client.BaseAddress = new Uri("http://any.localhost");
})
.AddPolicyHandler(policy) // We add our policy here...
.AddHttpMessageHandler(() => fakeHttpDelegatingHandler); // ...and our fake delegating handler here

Then, we build a ServiceProvider, create a scope and resolve our http client we just registered like this:

var serviceProvider = services.BuildServiceProvider();
using var scope = serviceProvider.CreateScope();
var sut = scope.ServiceProvider.GetRequiredService<IHttpClientFactory>().CreateClient("my-httpclient");

We then "send" the request and assert that we indeed retried the call three times.

var request = new HttpRequestMessage(HttpMethod.Get, "/any");

var result = await sut.SendAsync(request);

result.StatusCode.ShouldBe(HttpStatusCode.GatewayTimeout);
fakeHttpDelegatingHandler.Attempts.ShouldBe(3);

Since our FakeHttpDelegatingHandler accepts a func, we can test more advanced scenarios as well. Let's write another test that will verify that the retry pipeline will stop retrying when it encounters a successful status code.

[Fact]
public async Task ShouldStopRetryingIfSuccessStatusCodeIsEncountered()
{
    var services = new ServiceCollection();
    var fakeHttpDelegatingHandler = new FakeHttpDelegatingHandler(
        attempt =>
        {
            return attempt switch
            {
                2 => Task.FromResult(new HttpResponseMessage(HttpStatusCode.OK)),
                _ => Task.FromResult(new HttpResponseMessage(HttpStatusCode.GatewayTimeout))
            };
    });
    var policy =
        HttpClientPolicies.DefaultRetryPolicy(sleepDurationProvider: _ => TimeSpan.FromMilliseconds(1));
    services.AddHttpClient("test-httpclient", client =>
    {
        client.BaseAddress = new Uri("http://dummy.localhost");
    })
    .AddPolicyHandler(policy)
    .AddHttpMessageHandler(() => fakeHttpDelegatingHandler);
    var serviceProvider = services.BuildServiceProvider();
    using var scope = serviceProvider.CreateScope();
    var sut = scope.ServiceProvider.GetRequiredService<IHttpClientFactory>().CreateClient("test-httpclient");
    var request = new HttpRequestMessage(HttpMethod.Get, "/any");

    var result = await sut.SendAsync(request, HttpCompletionOption.ResponseHeadersRead);

    result.StatusCode.ShouldBe(HttpStatusCode.OK);
    fakeHttpDelegatingHandler.Attempts.ShouldBe(2);
}

Now we're saying that our FakeHttpDelegatingHandler should return 504 Gateway Timeout for all calls except the second call. When it receives the second call, it should return 200 OK.

This means that the FakeHttpDelegatingHandler should only receive two calls, and, as our assertions shows, it works perfectly :).

As you can see, using Polly for retries is really simple to setup, and not that hard to test.