Recently, I was refactoring my Azure DevOps CI/CD scripts that setup and deploy Sitecore 10.1 on a newly created AKS. Deployment went smooth, until I tried to browse to the deployed environments. Nginx, configured with sitecore-ingress provided by Sitecore, returned a 404 Not Found. What gives?

Where’s the error?

The logical thing to do is start looking for errors. Unfortunately, the nginx-ingress-ingress-nginx-controller pods did not show any (they did not show anything at all). Perhaps running them in debug mode could have told me more, but my experience with nginx is limited (close to none actually :)), so I postponed figuring out how to do that.

The only pointer I had was my browser. The HTTP response did not show anything special, but the insecure https indication was odd. I used a self-signed certificate that worked in the past without any issues. Inspecting the details shows a strange issuer: Kubernetes Ingress Controller Fake Certificate. That can’t be right.

After a quick google, I landed on a Microsoft documentation site (Use your TLS certificates for ingress – Azure Kubernetes Service | Microsoft Docs), telling me the certificate and configured ingress route are not compatible in a way.

But, as said, I was simply using the ingress configuration provided by Sitecore.

What changed in my scripts?

The scripts I used had been working before, so it had to be in the details. One thing I explicitly changed in my script was setting AKS to a more recent version of Kubernetes.

Sitecore’s Installation Guide for Production Environment with Kubernetes (Sitecore 10.1) does not specify an upper bound of the version, so this is an acceptable change:

An AKS cluster configured with the latest stable release of Kubernetes – version 1.16.x or later.

For startup probes to check whether the Sitecore software container has started successfully, Kubernetes version 1.18.x or later is required.

The most popular GitHub examples, which I based myself on for previous -working- setups, use an older version:

• Sitecore-Symposium-2020-Containers-AKS/2.CreateAKS.ps1 at main · bplasmeijer/Sitecore-Symposium-2020-Containers-AKS (github.com)
• paas-to-aks/create-aks.ps1 at main · robhabraken/paas-to-aks (github.com)

One thing that always bothered during previous setups was the warning / usage of ingress beta:

Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress

According to the mentioned versions, it should still work. But with experienced 404 / certificate issues, I thought I try to upgrade it and see what happens.

Upgrading ingress to solve the issue (sort of)

Upgrading is rather straightforward (see end of post for full spec).

First you change the apiVersion from extensions/v1beta1to networking.k8s.io/v1.

And make the specs a bit more structured / verbose.

spec:
  rules:
  - host: cd.globalhost
    http:
      paths:
      - path: /
        backend:
          serviceName: cd
          servicePort: 80

becomes

spec:
  rules:
  - host: cd.globalhost
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: cd
            port: 
              number: 80

And, the most crucial part of all, you need to add ingressClassName: nginx to the spec (e.g. right below rules).

See Basic usage – NGINX Ingress Controller (kubernetes.github.io) for full details.

After making the changes, Sitecore URLs resolve as expected.

Conclusion

The ingressClassName: nginx change turned out to be the root cause of my problem. According to its documentation, it’s required for K8s >= 1.19, which is exactly the change I made.

Note that adding this to Sitecore’s ingress spec (apiVersion extensions/v1beta1) fixes the issue as well. So the warning unavailable in v1.22+ is correct, but there are extra versioning dependencies to take into account.

Tags: #aks, #kubernetes, #ngnix.

Full spec

For those interested, this is updated ingress-nginx/ingress.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: sitecore-ingress
  annotations:
    nginx.ingress.kubernetes.io/proxy-buffer-size: "32k"
    nginx.ingress.kubernetes.io/affinity: "cookie"
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/proxy-connect-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-body-size: "512m"
spec:
  rules:
  - host: cd.globalhost
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: cd
            port: 
              number: 80
  - host: cm.globalhost
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: cm
            port:
              number: 80
  - host: id.globalhost
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: id
            port:
              number: 80
  ingressClassName: nginx
  tls:
  - secretName: global-cd-tls
    hosts:
    - cd.globalhost
  - secretName: global-cm-tls
    hosts:
    - cm.globalhost
  - secretName: global-id-tls
    hosts:
    - id.globalhost