As it was provided in a web deploy package (WDP), making small config changes like integrating with Azure AD was straightforward. With Sitecore moving to containers, and SI thus being provided as container image, making these same config changes becomes more complex… unless you’re willing to throw some Kubernetes features at it.

Sitecore Identity is an OpenID Connect compliant security token service (STS), based on IdentityServer4. Its default setup is an OAuth wrapper around good old Sitecore Membership:

• As an unauthenticated user that tries to access Sitecore CM, you are redirected to SI.
• On SI you can authenticate with your Sitecore credentials (username and password, stored in the security database).
• On successful authentication, SI redirects you back to Sitecore CM with a security token.
• With this security token, you can now access Sitecore CM (and potential other applications behind Sitecore security).

Because it’s based on IdentityServer4, you can use SI as a gateway to one or more external identity providers (or subproviders, sometimes also called inner providers). For example, Azure AD:

• As an unauthenticated user that tries to access Sitecore CM, you are redirected to SI.
• Instead of using Sitecore Membership, SI now offloads authentication to Azure AD: you are redirected to Azure, where you authenticate with your Azure AD account.
• On successful authentication, SI redirects you back to Sitecore CM with a security token, now based on information from your Azure AD account.
• With this security token, you can now access Sitecore CM (and potential other applications behind Sitecore security).

This allows business users to log into Sitecore CM using their corporate (Azure AD) account.

Configuring SI to integrate with Azure AD is supported out-of-the-box. You need to

1. Register Sitecore as an Application in your Azure AD tenant

2. Create groups in your Azure AD for the users you want to give access to the Sitecore back-end

3. Configure SI to connect to that Azure AD application and add a user mapping that specifies the role (e.g. Sitecore\admin) for a specified Azure AD group

You find the official documentation on Use the Sitecore Identity server as a federation gateway, but I recommend Setting Up Azure Active Directory Integration with Sitecore Identity Server / Sitecore 9.1 (derekc.net) for a more detailed walkthrough of this setup.

Deploying SI with a Web Deploy Package

Before containers, changing the configuration was straightforward: SI is provided by Sitecore as a WDP (web deploy package — see Sitecore Downloads: Sitecore Identity 600). A structured folder containing DLLs, config XML files, … that you can deploy to IIS (virtual machine, Azure App Service, …) using Web Deploy.

Change the config files, deploy the updated package and you’re done.

Deploying SI with a container image

Next to WDP, SI is also provided as a container image. For example, scr.sitecore.com/sxp/sitecore-id:10.2-ltsc2019

Compared to changing a WDP zip file, creating a custom container image requires more skills and infrastructure:

• you need to create a DockerFile that takes an official Sitecore ID image as a base image and specify which custom files (the modified config files) you will patch over it to create your own custom Sitecore ID
• you need to build the image (build server?) and push it to a container registry (acr?)
• you need to change your deployment (k8s specs) to use the modified image

None of this is rocket science, but it’s quite some work to change a simple config file.

Configuring SI with Kubernetes ConfigMap

With the custom container image approach, you first copy modified files over an original image. Then these modifications are baked into a new image that you deploy to Kubernetes. Kubernetes uses an abstraction to run this image on actual hardware.

Thanks to the abstraction layer, it’s possible for Kubernetes to hijack the files an image tries to access while it’s running: in the id.yaml k8s spec file provided by Sitecore, you add a volumeMount, which tells the runtime that when image wants a file from /Identity/sitecore/Sitecore.Plugin.IdentityProvider.AzureAd/Config/ it should first be search in a volume called config. The config volume forwards this to a configMap called sitecore-id-azuread-config (an alternative would be an actual storage).

A Configmaps is an API object in Kubernetes used to store data in:

A ConfigMap allows you to decouple environment-specific configuration from your container images, so that your applications are easily portable.

In our case, we can simply put the whole custom configuration file in it:

With this setup it’s possible to run Sitecore’s official SI image with a custom Identity/sitecore/Sitecore.Plugin.IdentityProvider.AzureAd/Config/Sitecore.Plugin.IdentityProvider.AzureAd.xml file, containing your project specific Azure AD configuration. No need to create a custom image for it!

Tags: #kubernetes, #sitecoreidentity

Where’s the error?

The logical thing to do is start looking for errors. Unfortunately, the nginx-ingress-ingress-nginx-controller pods did not show any (they did not show anything at all). Perhaps running them in debug mode could have told me more, but my experience with nginx is limited (close to none actually :)), so I postponed figuring out how to do that.

The only pointer I had was my browser. The HTTP response did not show anything special, but the insecure https indication was odd. I used a self-signed certificate that worked in the past without any issues. Inspecting the details shows a strange issuer: Kubernetes Ingress Controller Fake Certificate. That can’t be right.

After a quick google, I landed on a Microsoft documentation site (Use your TLS certificates for ingress – Azure Kubernetes Service | Microsoft Docs), telling me the certificate and configured ingress route are not compatible in a way.

But, as said, I was simply using the ingress configuration provided by Sitecore.

What changed in my scripts?

The scripts I used had been working before, so it had to be in the details. One thing I explicitly changed in my script was setting AKS to a more recent version of Kubernetes.

Sitecore’s Installation Guide for Production Environment with Kubernetes (Sitecore 10.1) does not specify an upper bound of the version, so this is an acceptable change:

An AKS cluster configured with the latest stable release of Kubernetes – version 1.16.x or later.

For startup probes to check whether the Sitecore software container has started successfully, Kubernetes version 1.18.x or later is required.

The most popular GitHub examples, which I based myself on for previous -working- setups, use an older version:

• Sitecore-Symposium-2020-Containers-AKS/2.CreateAKS.ps1 at main · bplasmeijer/Sitecore-Symposium-2020-Containers-AKS (github.com)
• paas-to-aks/create-aks.ps1 at main · robhabraken/paas-to-aks (github.com)

One thing that always bothered during previous setups was the warning / usage of ingress beta:

Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress

According to the mentioned versions, it should still work. But with experienced 404 / certificate issues, I thought I try to upgrade it and see what happens.

Upgrading ingress to solve the issue (sort of)

Upgrading is rather straightforward (see end of post for full spec).

First you change the apiVersion from extensions/v1beta1to networking.k8s.io/v1.

And make the specs a bit more structured / verbose.

spec:
  rules:
  - host: cd.globalhost
    http:
      paths:
      - path: /
        backend:
          serviceName: cd
          servicePort: 80

becomes

spec:
  rules:
  - host: cd.globalhost
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: cd
            port: 
              number: 80

And, the most crucial part of all, you need to add ingressClassName: nginx to the spec (e.g. right below rules).

See Basic usage – NGINX Ingress Controller (kubernetes.github.io) for full details.

After making the changes, Sitecore URLs resolve as expected.

Conclusion

The ingressClassName: nginx change turned out to be the root cause of my problem. According to its documentation, it’s required for K8s >= 1.19, which is exactly the change I made.

Note that adding this to Sitecore’s ingress spec (apiVersion extensions/v1beta1) fixes the issue as well. So the warning unavailable in v1.22+ is correct, but there are extra versioning dependencies to take into account.

Tags: #aks, #kubernetes, #ngnix.

Full spec

For those interested, this is updated ingress-nginx/ingress.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: sitecore-ingress
  annotations:
    nginx.ingress.kubernetes.io/proxy-buffer-size: "32k"
    nginx.ingress.kubernetes.io/affinity: "cookie"
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/proxy-connect-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-body-size: "512m"
spec:
  rules:
  - host: cd.globalhost
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: cd
            port: 
              number: 80
  - host: cm.globalhost
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: cm
            port:
              number: 80
  - host: id.globalhost
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: id
            port:
              number: 80
  ingressClassName: nginx
  tls:
  - secretName: global-cd-tls
    hosts:
    - cd.globalhost
  - secretName: global-cm-tls
    hosts:
    - cm.globalhost
  - secretName: global-id-tls
    hosts:
    - id.globalhost

When repeating the setup on the client’s subscription, deployment went well but we failed to log in into Sitecore:

• We try to access /sitecore.
• Since we’re not logged in, Sitecore CM redirects us to Sitecore ID.
• On Sitecore ID, we enter our credentials.
• Everything seems fine and we are redirected back to Sitecore CM.
• We see the browser redirecting, but the page failed to load: the browser fails with a timeout (IIRC).

A quick look at the log files did not reveal any errors or failures.

In my setup, everything worked as expected, so what’s different for our client? Since this is a real project setup, of course the URLs (and TLS) are going to be customer specific (so not globalhost). Typo’s are easily made, but not this time: double checking the config did not reveal any mistakes.

With no traces in the logs we had to dig a little bit deeper. Since single-sign on is all about redirection and passing along the required information in the process, taking a closer look at the network traffic made sense.

(see Virtual Developer Day 2020 – Getting to Know Sitecore Identity – George Chang for a good introduction on Sitecore ID)

Security best practices for the win

A useful tool to do this is fiddler (fiddler classic in my case): a web debugging proxy to successfully log, inspect, and alter HTTP(s) network requests and server responses.

I was already thinking bad tokens, encryption issues, claim mismatches, … Luckily the issue was not as complex as I feared. Comparing network traffic from my (working) setup with traffic from the client setup revealed they had one HTTP header missing: Strict-Transport-Security, also known as HSTS.

If a website accepts a connection through HTTP and redirects to HTTPS, visitors may initially communicate with the non-encrypted version of the site before being redirected, if, for example, the visitor types http://www.foo.com/ or even just foo.com. This creates an opportunity for a man-in-the-middle attack. The redirect could be exploited to direct visitors to a malicious site instead of the secure version of the original site.

The HTTP Strict Transport Security header informs the browser that it should never load a site using HTTP and should automatically convert all attempts to access the site using HTTP to HTTPS requests instead.

https://developer.mozilla.org/en-US/docs/Glossary/HSTS

This is exactly what happens in my fiddler trace: even though the server tells my browser to go to HTTP (using the Location: header), my browser ignores this and connects over HTTPS directly (line 52).

Why would Sitecore redirect us to HTTP? No idea. As far as I can tell all our config is set correctly. I can only assume this is leftover code from a time where local development was often done on HTTP. With Sitecore on containers, development setup is a lot closer to production setup, including running on HTTPS.

Ingress

So why did I did not face this issue on my setup? An important part of a Kubernetes setup is managing how external traffic can access the services in your cluster (HTTP, HTTPS, TLS, …). This is done with the Ingress API object. Out of the box, Sitecore suggests you to use the NGINX Ingress controller for this.

NGINX is a fine default, but my client has a track record in Kubernetes (not on Azure) and for their networking requirements they standardize on Istio, a service mesh. To use it, we had to switch NGINX with Istio Ingress.

Turns out HSTS is enabled by default in NGINX (see https://kubernetes.github.io/ingress-nginx/user-guide/tls/#http-strict-transport-security), but not in Istio. Enabling it is quite easy though (in case you might be interested): https://www.wagner-dev.com/istio-configure-strict-transport-security-hsts.html

In a development setup, network traffic is typically handled with traefik. Sitecore provides you with docker-compose files that force this behavior.

So why didn’t we spot this right away? The browser we were using (it’s name starts with a ‘C’) doesn’t show the protocol in the URL bar, so we simply didn’t spot it. That we had to test this through a screen share session probably didn’t help neither (but that’s a different story).

In summary

• Sitecore login depends on HSTS.
• HSTS is enabled by default for NGNIX ingress, but that might not be the case for other ingress controllers (e.g. Isitio).
• You can’t see what you can’t see.

Tags: #aks, #kubernetes, #ngnix, #istio.