Where’s the error?

The logical thing to do is start looking for errors. Unfortunately, the nginx-ingress-ingress-nginx-controller pods did not show any (they did not show anything at all). Perhaps running them in debug mode could have told me more, but my experience with nginx is limited (close to none actually :)), so I postponed figuring out how to do that.

The only pointer I had was my browser. The HTTP response did not show anything special, but the insecure https indication was odd. I used a self-signed certificate that worked in the past without any issues. Inspecting the details shows a strange issuer: Kubernetes Ingress Controller Fake Certificate. That can’t be right.

After a quick google, I landed on a Microsoft documentation site (Use your TLS certificates for ingress – Azure Kubernetes Service | Microsoft Docs), telling me the certificate and configured ingress route are not compatible in a way.

But, as said, I was simply using the ingress configuration provided by Sitecore.

What changed in my scripts?

The scripts I used had been working before, so it had to be in the details. One thing I explicitly changed in my script was setting AKS to a more recent version of Kubernetes.

Sitecore’s Installation Guide for Production Environment with Kubernetes (Sitecore 10.1) does not specify an upper bound of the version, so this is an acceptable change:

An AKS cluster configured with the latest stable release of Kubernetes – version 1.16.x or later.

For startup probes to check whether the Sitecore software container has started successfully, Kubernetes version 1.18.x or later is required.

The most popular GitHub examples, which I based myself on for previous -working- setups, use an older version:

• Sitecore-Symposium-2020-Containers-AKS/2.CreateAKS.ps1 at main · bplasmeijer/Sitecore-Symposium-2020-Containers-AKS (github.com)
• paas-to-aks/create-aks.ps1 at main · robhabraken/paas-to-aks (github.com)

One thing that always bothered during previous setups was the warning / usage of ingress beta:

Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress

According to the mentioned versions, it should still work. But with experienced 404 / certificate issues, I thought I try to upgrade it and see what happens.

Upgrading ingress to solve the issue (sort of)

Upgrading is rather straightforward (see end of post for full spec).

First you change the apiVersion from extensions/v1beta1to networking.k8s.io/v1.

And make the specs a bit more structured / verbose.

spec:
  rules:
  - host: cd.globalhost
    http:
      paths:
      - path: /
        backend:
          serviceName: cd
          servicePort: 80

becomes

spec:
  rules:
  - host: cd.globalhost
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: cd
            port: 
              number: 80

And, the most crucial part of all, you need to add ingressClassName: nginx to the spec (e.g. right below rules).

See Basic usage – NGINX Ingress Controller (kubernetes.github.io) for full details.

After making the changes, Sitecore URLs resolve as expected.

Conclusion

The ingressClassName: nginx change turned out to be the root cause of my problem. According to its documentation, it’s required for K8s >= 1.19, which is exactly the change I made.

Note that adding this to Sitecore’s ingress spec (apiVersion extensions/v1beta1) fixes the issue as well. So the warning unavailable in v1.22+ is correct, but there are extra versioning dependencies to take into account.

Tags: #aks, #kubernetes, #ngnix.

Full spec

For those interested, this is updated ingress-nginx/ingress.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: sitecore-ingress
  annotations:
    nginx.ingress.kubernetes.io/proxy-buffer-size: "32k"
    nginx.ingress.kubernetes.io/affinity: "cookie"
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/proxy-connect-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-body-size: "512m"
spec:
  rules:
  - host: cd.globalhost
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: cd
            port: 
              number: 80
  - host: cm.globalhost
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: cm
            port:
              number: 80
  - host: id.globalhost
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: id
            port:
              number: 80
  ingressClassName: nginx
  tls:
  - secretName: global-cd-tls
    hosts:
    - cd.globalhost
  - secretName: global-cm-tls
    hosts:
    - cm.globalhost
  - secretName: global-id-tls
    hosts:
    - id.globalhost

When repeating the setup on the client’s subscription, deployment went well but we failed to log in into Sitecore:

• We try to access /sitecore.
• Since we’re not logged in, Sitecore CM redirects us to Sitecore ID.
• On Sitecore ID, we enter our credentials.
• Everything seems fine and we are redirected back to Sitecore CM.
• We see the browser redirecting, but the page failed to load: the browser fails with a timeout (IIRC).

A quick look at the log files did not reveal any errors or failures.

In my setup, everything worked as expected, so what’s different for our client? Since this is a real project setup, of course the URLs (and TLS) are going to be customer specific (so not globalhost). Typo’s are easily made, but not this time: double checking the config did not reveal any mistakes.

With no traces in the logs we had to dig a little bit deeper. Since single-sign on is all about redirection and passing along the required information in the process, taking a closer look at the network traffic made sense.

(see Virtual Developer Day 2020 – Getting to Know Sitecore Identity – George Chang for a good introduction on Sitecore ID)

Security best practices for the win

A useful tool to do this is fiddler (fiddler classic in my case): a web debugging proxy to successfully log, inspect, and alter HTTP(s) network requests and server responses.

I was already thinking bad tokens, encryption issues, claim mismatches, … Luckily the issue was not as complex as I feared. Comparing network traffic from my (working) setup with traffic from the client setup revealed they had one HTTP header missing: Strict-Transport-Security, also known as HSTS.

If a website accepts a connection through HTTP and redirects to HTTPS, visitors may initially communicate with the non-encrypted version of the site before being redirected, if, for example, the visitor types http://www.foo.com/ or even just foo.com. This creates an opportunity for a man-in-the-middle attack. The redirect could be exploited to direct visitors to a malicious site instead of the secure version of the original site.

The HTTP Strict Transport Security header informs the browser that it should never load a site using HTTP and should automatically convert all attempts to access the site using HTTP to HTTPS requests instead.

https://developer.mozilla.org/en-US/docs/Glossary/HSTS

This is exactly what happens in my fiddler trace: even though the server tells my browser to go to HTTP (using the Location: header), my browser ignores this and connects over HTTPS directly (line 52).

Why would Sitecore redirect us to HTTP? No idea. As far as I can tell all our config is set correctly. I can only assume this is leftover code from a time where local development was often done on HTTP. With Sitecore on containers, development setup is a lot closer to production setup, including running on HTTPS.

Ingress

So why did I did not face this issue on my setup? An important part of a Kubernetes setup is managing how external traffic can access the services in your cluster (HTTP, HTTPS, TLS, …). This is done with the Ingress API object. Out of the box, Sitecore suggests you to use the NGINX Ingress controller for this.

NGINX is a fine default, but my client has a track record in Kubernetes (not on Azure) and for their networking requirements they standardize on Istio, a service mesh. To use it, we had to switch NGINX with Istio Ingress.

Turns out HSTS is enabled by default in NGINX (see https://kubernetes.github.io/ingress-nginx/user-guide/tls/#http-strict-transport-security), but not in Istio. Enabling it is quite easy though (in case you might be interested): https://www.wagner-dev.com/istio-configure-strict-transport-security-hsts.html

In a development setup, network traffic is typically handled with traefik. Sitecore provides you with docker-compose files that force this behavior.

So why didn’t we spot this right away? The browser we were using (it’s name starts with a ‘C’) doesn’t show the protocol in the URL bar, so we simply didn’t spot it. That we had to test this through a screen share session probably didn’t help neither (but that’s a different story).

In summary

• Sitecore login depends on HSTS.
• HSTS is enabled by default for NGNIX ingress, but that might not be the case for other ingress controllers (e.g. Isitio).
• You can’t see what you can’t see.

Tags: #aks, #kubernetes, #ngnix, #istio.