Using http/2 with ASP.NET Core - Traefik/Kubernetes/container example

Intro

When dealing with TLS and containers, a common pattern is to terminate the TLS connection in a load balancer and then speak http/1 between the loadbalancer and the container. It has a number of benefits, mainly that you don't need to deal with certificates in your container.

But sometimes you need to speak http/2 "all the way" to the container (when using grpc for example).

I'll show you my setup. I use Traefik in my Kubernetes cluster but that's not important, it will only passthrough the request to the ASP.NET container anyway, you can use whatever load balancer you want.

My setup

ASP.NET Core

I have a simple ASP.NET Core application running ASP.NET Core 8 called JOS.Echo. It's configured to listen on port 443.

Dockerfile

FROM mcr.microsoft.com/dotnet/sdk:8.0 as builder
# https://learn.microsoft.com/en-us/nuget/reference/cli-reference/cli-ref-environment-variables
ARG informationalVersion
ARG version
ENV NUGET_XMLDOC_MODE=none
COPY . app/
WORKDIR /app

RUN dotnet build -c Release /p:AssemblyVersion=$version /p:InformationalVersion=$informationalVersion
RUN dotnet test -c Release --no-build
RUN dotnet publish src/JOS.Echo -c Release -o published --no-build

FROM mcr.microsoft.com/dotnet/aspnet:8.0

COPY --from=builder /app/published/ /app/
WORKDIR /app

RUN apt-get update && apt-get -y install libcap2-bin
RUN setcap 'cap_net_bind_service=+ep' /usr/share/dotnet/dotnet

USER 1000

ENV ASPNETCORE_URLS https://+:443
EXPOSE 443/tcp

ENTRYPOINT ["dotnet", "JOS.Echo.dll"]

Since I wanted the application to listen on port 443 inside the container, I had to use setcap because I don't want to run the container as root. If you run the container as a non-root user, you are not allowed to bind to a port lower than 1024. setcap enables the dotnet binary to bind to a port lower than 1024 as a non-root user, 443 in this case.

Traefik

IngressRoute

apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: jos-echo-ingress-route
  annotations:
    kubernetes.io/ingress.class: traefik-internal
spec:
  entryPoints:
    - websecure
  routes:
    - match: Host(`echo.local.josef.guru`)
      kind: Rule
      services:
        - name: jos-echo-service
          port: 443

Kubernetes

Vanilla Kuberentes. I use cert-manager for dealing with certificates. More on that later.
Service

apiVersion: v1
kind: Service
metadata:
  name: jos-echo-service
spec:
  ports:
    - name: http
      port: 443
      targetPort: http-web-svc
  selector:
    app: jos-echo

Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: jos-echo-deployment
spec:
  replicas: 3
  selector:
    matchLabels:
      app: jos-echo
  template:
    metadata:
      labels:
        app: jos-echo
    spec:
      imagePullSecrets:
        - name: dockerconfig-ghcr.io
      containers:
        - name: jos-echo-container
          image: ghcr.io/joseftw/jos-echo:1.0.30-ci.g83055582fe
          env:
            - name: "Server__Port"
              value: "443"
            - name: "TLS__Certificate__CertificateFile"
              value: "/certs/tls.crt"
            - name: "TLS__Certificate__KeyFile"
              value: "/certs/tls.key"
          volumeMounts:
            - name: certs-volume
              mountPath: /certs
              readOnly: true
          ports:
            - containerPort: 443
              name: http-web-svc
      volumes:
        - name: certs-volume
          secret:
            secretName: local-josef-guru-tls

Kubernetes

Since http/2 requires the use of https, I thought it made sense to quickly show how I handle certificates in my cluster.
I'm using cert-manager.
My local domain is local.josef.guru so I've created a wildcard certificate with cert-manager.
Two custom resources are needed, a ClusterIssuer and a Certificate.

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-production
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: josef@josef.guru
    privateKeySecretRef:
      name: letsencrypt-production
    solvers:
      - dns01:
          cloudflare:
            email: josef@josef.guru
            apiTokenSecretRef:
              name: cloudflare-token-secret
              key: cloudflare-token
        selector:
          dnsZones:
            - "josef.guru"

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: local-josef-guru
  namespace: default
spec:
  secretName: local-josef-guru-tls
  issuerRef:
    name: letsencrypt-production
    kind: ClusterIssuer
  commonName: "*.local.josef.guru"
  dnsNames:
    - "local.josef.guru"
    - "*.local.josef.guru"
  secretTemplate:
    annotations:
      reflector.v1.k8s.emberstack.com/reflection-allowed: "true"
      reflector.v1.k8s.emberstack.com/reflection-allowed-namespaces: ""
      reflector.v1.k8s.emberstack.com/reflection-auto-enabled: "true"
      reflector.v1.k8s.emberstack.com/reflection-auto-namespaces: ""

Cert-manager will create a secret named local-josef-guru-tls that contains the certificate. As you can see, the certificate will be created in the default namespace, so I'm using reflector to make it possible to access the secret from other namespaces.

If you want to know more about cert-manager I highly recommend this video by Techo Tim.

Traefik

Since I don't want to terminate the tls connection in Traefik, I don't need to do any additional configuration of Traefik, it will happily forward the request to my container without intervention.

JOS.Echo - The ASP.NET Core portion

"Mount" the certificate

In the deployment.yml file, the following can be found:

....
containers:
  ....
  volumeMounts:
    - name: certs-volume
      mountPath: /certs
      readOnly: true
volumes:
  - name: certs-volume
    secret:
      secretName: local-josef-guru-tls

Here, I'm mounting the contents of the local-josef-guru-tls secret (the certificate) to /certs. This will enable us to access the certificate inside our container.

One nice thing with doing it this way is that if the certificate gets updated, it will be reflected immediately inside the container.

Since the certificate can be updated while the container is running, I don't want to cache it at startup. But at the same time, I don't want to read the certificate from disk on every request either.

That's why I'm making use of ServerCertificateSelector. This code will run on every request.

public class CachingMountedCertificateReader
{
    private static readonly SemaphoreSlim Semaphore;
    private readonly MountedCertificateReader _certificateReader;
    private static DateTime? LastRefreshed;
    private static X509Certificate2? _certificate;

    static CachingMountedCertificateReader()
    {
        Semaphore = new SemaphoreSlim(1, 1);
    }

    public CachingMountedCertificateReader(MountedCertificateReader certificateReader)
    {
        _certificateReader = certificateReader ?? throw new ArgumentNullException(nameof(certificateReader));
    }

    public X509Certificate2 Read()
    {
        if (!NeedsRefresh(_certificate))
        {
            return _certificate!;
        }

        Semaphore.Wait();

        if (!NeedsRefresh(_certificate))
        {
            Semaphore.Release(1);
            return _certificate!;
        }

        _certificate = _certificateReader.Read();
        LastRefreshed = DateTime.UtcNow;
        Semaphore.Release(1);
        return _certificate;
    }

    private static bool NeedsRefresh(X509Certificate2? certificate)
    {
        return certificate is null || certificate.NotAfter <= DateTime.UtcNow || (DateTime.UtcNow - LastRefreshed!.Value).TotalMinutes >= 60;
    }
}

public class MountedCertificateReader
{
    private readonly TlsConfiguration _tlsConfiguration;

    public MountedCertificateReader(TlsConfiguration tlsConfiguration)
    {
        _tlsConfiguration = tlsConfiguration ?? throw new ArgumentNullException(nameof(tlsConfiguration));
    }

    public X509Certificate2 Read()
    {
        if (!_tlsConfiguration.HasCertificate)
        {
            throw new Exception("No certificate has been configured");
        }

        var certificate = _tlsConfiguration.Certificate!;
        if (!certificate.HasKeyFile)
        {
            return X509Certificate2.CreateFromPemFile(certificate.CertificateFile);
        }

        if (certificate.HasPassword)
        {
            return X509Certificate2.CreateFromEncryptedPemFile(
                certificate.CertificateFile,
                certificate.Password,
                certificate.KeyFile);
        }

        return X509Certificate2.CreateFromPemFile(
            certificate.CertificateFile,
            certificate.KeyFile);
    }
}

It works like this:

Check if the certificate needs refresh (refresh in this case just means "read from disk")
If it doesn't, return the cached certificate.
If it does -> read the certificate from disk and cache it.

Putting it all together

To enable Kestrel to use http/2, you'll need to do the following:

builder.WebHost.ConfigureKestrel((_, options) =>
{
    options.ListenAnyIP(serverConfiguration.Port, listenOptions =>
    {
        if (tlsConfiguration.HasCertificate)
        {
            listenOptions.Protocols = HttpProtocols.Http2 | HttpProtocols.Http3;
            listenOptions.UseHttps(httpsOptions =>
            {
                httpsOptions.ServerCertificateSelector = (_, _) => certificateReader.Read();
            });
        }
        else
        {
            listenOptions.Protocols = HttpProtocols.Http1;
        }
    });
});

As you can see, if a certificate is supplied, Kestrel will enable http2 AND http3 (http3 will be a separate post in the future).

I've also created a simple endpoint that just echoes the request information back in the response body.

A call to / would return a bunch of details regarding the request, but the only thing we care about in this post is the following:

{
    ...
    "request": {
        "protocol": "HTTP/2"
    }
    ...
}

This means that http/2 works "all the way" from the users browser -> Traefik -> Kestrel.

All code for JOS.Echo can be found on GitHub.

Josef Ottosson

Freelancing Developer · Microsoft MVP

Using http/2 with ASP.NET Core - Traefik/Kubernetes/container example

Intro

My setup

ASP.NET Core

Dockerfile

Traefik

Kubernetes

Kubernetes

Traefik

JOS.Echo - The ASP.NET Core portion

"Mount" the certificate

Putting it all together

Testing your Polly policies

GitOps with GitHub Actions and Argo CD