SSC API key is required. Pass with 'sc_apikey' query string or HTTP header.

The code that caused the issue was a textbook GraphQL request:

const graphQLClient = new GraphQLRequestClient(config.graphQLEndpoint, {
    apiKey: config.sitecoreApiKey,
});

Turns out nginx (that connects incoming AKS traffic to the actual Sitecore containers) does not allow underscores in header keys by default:

http://nginx.org/en/docs/http/ngx_http_core_module.html#underscores_in_headers

This case (different use case though) is documented in SitecoreXP1020ProductionDeploymentWithKubernetes-en.pdf, see

4.6.3. When I request SSC, problems occur if there are underscores in header names

but since that document is not indexed by Google, finding it was not that easy.

Luckily somebody had a similar issue when running Sitecore on Amazon: https://github.com/Sitecore/jss/issues/1060.

The solution for k8s is straightforward. Simply add

enable-underscores-in-headers: "true"

to ingress-nginx\configuration.yaml (the configmap used by nginx).

Note that if you install this with kubectl -k ingress-nginx this requires nginx and Sitecore to run in the same namespace. If not, the config map is created in the Sitecore namespace, but won’t be picked up by nginx.

As it was provided in a web deploy package (WDP), making small config changes like integrating with Azure AD was straightforward. With Sitecore moving to containers, and SI thus being provided as container image, making these same config changes becomes more complex… unless you’re willing to throw some Kubernetes features at it.

Sitecore Identity is an OpenID Connect compliant security token service (STS), based on IdentityServer4. Its default setup is an OAuth wrapper around good old Sitecore Membership:

• As an unauthenticated user that tries to access Sitecore CM, you are redirected to SI.
• On SI you can authenticate with your Sitecore credentials (username and password, stored in the security database).
• On successful authentication, SI redirects you back to Sitecore CM with a security token.
• With this security token, you can now access Sitecore CM (and potential other applications behind Sitecore security).

Because it’s based on IdentityServer4, you can use SI as a gateway to one or more external identity providers (or subproviders, sometimes also called inner providers). For example, Azure AD:

• As an unauthenticated user that tries to access Sitecore CM, you are redirected to SI.
• Instead of using Sitecore Membership, SI now offloads authentication to Azure AD: you are redirected to Azure, where you authenticate with your Azure AD account.
• On successful authentication, SI redirects you back to Sitecore CM with a security token, now based on information from your Azure AD account.
• With this security token, you can now access Sitecore CM (and potential other applications behind Sitecore security).

This allows business users to log into Sitecore CM using their corporate (Azure AD) account.

Configuring SI to integrate with Azure AD is supported out-of-the-box. You need to

1. Register Sitecore as an Application in your Azure AD tenant

2. Create groups in your Azure AD for the users you want to give access to the Sitecore back-end

3. Configure SI to connect to that Azure AD application and add a user mapping that specifies the role (e.g. Sitecore\admin) for a specified Azure AD group

You find the official documentation on Use the Sitecore Identity server as a federation gateway, but I recommend Setting Up Azure Active Directory Integration with Sitecore Identity Server / Sitecore 9.1 (derekc.net) for a more detailed walkthrough of this setup.

Deploying SI with a Web Deploy Package

Before containers, changing the configuration was straightforward: SI is provided by Sitecore as a WDP (web deploy package — see Sitecore Downloads: Sitecore Identity 600). A structured folder containing DLLs, config XML files, … that you can deploy to IIS (virtual machine, Azure App Service, …) using Web Deploy.

Change the config files, deploy the updated package and you’re done.

Deploying SI with a container image

Next to WDP, SI is also provided as a container image. For example, scr.sitecore.com/sxp/sitecore-id:10.2-ltsc2019

Compared to changing a WDP zip file, creating a custom container image requires more skills and infrastructure:

• you need to create a DockerFile that takes an official Sitecore ID image as a base image and specify which custom files (the modified config files) you will patch over it to create your own custom Sitecore ID
• you need to build the image (build server?) and push it to a container registry (acr?)
• you need to change your deployment (k8s specs) to use the modified image

None of this is rocket science, but it’s quite some work to change a simple config file.

Configuring SI with Kubernetes ConfigMap

With the custom container image approach, you first copy modified files over an original image. Then these modifications are baked into a new image that you deploy to Kubernetes. Kubernetes uses an abstraction to run this image on actual hardware.

Thanks to the abstraction layer, it’s possible for Kubernetes to hijack the files an image tries to access while it’s running: in the id.yaml k8s spec file provided by Sitecore, you add a volumeMount, which tells the runtime that when image wants a file from /Identity/sitecore/Sitecore.Plugin.IdentityProvider.AzureAd/Config/ it should first be search in a volume called config. The config volume forwards this to a configMap called sitecore-id-azuread-config (an alternative would be an actual storage).

A Configmaps is an API object in Kubernetes used to store data in:

A ConfigMap allows you to decouple environment-specific configuration from your container images, so that your applications are easily portable.

In our case, we can simply put the whole custom configuration file in it:

With this setup it’s possible to run Sitecore’s official SI image with a custom Identity/sitecore/Sitecore.Plugin.IdentityProvider.AzureAd/Config/Sitecore.Plugin.IdentityProvider.AzureAd.xml file, containing your project specific Azure AD configuration. No need to create a custom image for it!

Tags: #kubernetes, #sitecoreidentity

Where’s the error?

The logical thing to do is start looking for errors. Unfortunately, the nginx-ingress-ingress-nginx-controller pods did not show any (they did not show anything at all). Perhaps running them in debug mode could have told me more, but my experience with nginx is limited (close to none actually :)), so I postponed figuring out how to do that.

The only pointer I had was my browser. The HTTP response did not show anything special, but the insecure https indication was odd. I used a self-signed certificate that worked in the past without any issues. Inspecting the details shows a strange issuer: Kubernetes Ingress Controller Fake Certificate. That can’t be right.

After a quick google, I landed on a Microsoft documentation site (Use your TLS certificates for ingress – Azure Kubernetes Service | Microsoft Docs), telling me the certificate and configured ingress route are not compatible in a way.

But, as said, I was simply using the ingress configuration provided by Sitecore.

What changed in my scripts?

The scripts I used had been working before, so it had to be in the details. One thing I explicitly changed in my script was setting AKS to a more recent version of Kubernetes.

Sitecore’s Installation Guide for Production Environment with Kubernetes (Sitecore 10.1) does not specify an upper bound of the version, so this is an acceptable change:

An AKS cluster configured with the latest stable release of Kubernetes – version 1.16.x or later.

For startup probes to check whether the Sitecore software container has started successfully, Kubernetes version 1.18.x or later is required.

The most popular GitHub examples, which I based myself on for previous -working- setups, use an older version:

• Sitecore-Symposium-2020-Containers-AKS/2.CreateAKS.ps1 at main · bplasmeijer/Sitecore-Symposium-2020-Containers-AKS (github.com)
• paas-to-aks/create-aks.ps1 at main · robhabraken/paas-to-aks (github.com)

One thing that always bothered during previous setups was the warning / usage of ingress beta:

Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress

According to the mentioned versions, it should still work. But with experienced 404 / certificate issues, I thought I try to upgrade it and see what happens.

Upgrading ingress to solve the issue (sort of)

Upgrading is rather straightforward (see end of post for full spec).

First you change the apiVersion from extensions/v1beta1to networking.k8s.io/v1.

And make the specs a bit more structured / verbose.

spec:
  rules:
  - host: cd.globalhost
    http:
      paths:
      - path: /
        backend:
          serviceName: cd
          servicePort: 80

becomes

spec:
  rules:
  - host: cd.globalhost
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: cd
            port: 
              number: 80

And, the most crucial part of all, you need to add ingressClassName: nginx to the spec (e.g. right below rules).

See Basic usage – NGINX Ingress Controller (kubernetes.github.io) for full details.

After making the changes, Sitecore URLs resolve as expected.

Conclusion

The ingressClassName: nginx change turned out to be the root cause of my problem. According to its documentation, it’s required for K8s >= 1.19, which is exactly the change I made.

Note that adding this to Sitecore’s ingress spec (apiVersion extensions/v1beta1) fixes the issue as well. So the warning unavailable in v1.22+ is correct, but there are extra versioning dependencies to take into account.

Tags: #aks, #kubernetes, #ngnix.

Full spec

For those interested, this is updated ingress-nginx/ingress.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: sitecore-ingress
  annotations:
    nginx.ingress.kubernetes.io/proxy-buffer-size: "32k"
    nginx.ingress.kubernetes.io/affinity: "cookie"
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/proxy-connect-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-body-size: "512m"
spec:
  rules:
  - host: cd.globalhost
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: cd
            port: 
              number: 80
  - host: cm.globalhost
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: cm
            port:
              number: 80
  - host: id.globalhost
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: id
            port:
              number: 80
  ingressClassName: nginx
  tls:
  - secretName: global-cd-tls
    hosts:
    - cd.globalhost
  - secretName: global-cm-tls
    hosts:
    - cm.globalhost
  - secretName: global-id-tls
    hosts:
    - id.globalhost

Docker is a great tool, except when it’s not.

Yesterday I picked up on container development after some inactivity. Before I could start, quite some updates were pushed (in case you missed it: in a recent change, only paid plans are allowed to skip updates). Eventually everything worked as expected

Today, after a reboot, I faced following error:

Resetting to factory defaults did not help (it did help clean out my hard drive since resetting deletes all your downloaded images).

Reinstalling did help a bit: docker succeeded to start up in Linux mode. However, switching to Windows mode threw the same exception.

Since the exception is not very detailed, I had to dig a little deeper: good old Event Viewer to the rescue. Turns out something is wrong with the panic log (love the name).

After inspecting the file it turns out it’s marked as readonly:

Unchecking the checkbox and restarting docker solves the issue.

Tags: #docker

Since it’s a SaaS tool, you can setup a free account within minutes: simply register on https://ordercloud.io to get your sandbox environment (no credit card needed :)). The getting started guide gives you a good understanding of the basic concepts.

Here are my first impressions on it after browsing through the playground environment and the API reference.

What’s an OrderCloud?

The Sitecore Acquisition of Boxever and Four51 FAQ page frames it as

1. Microservices-based, so individual pieces of business functionality can be independently developed, deployed, and managed.

2. API-first, with all functionality exposed through an API to allow simple, seamless integration.

3. Cloud-native SaaS to leverage the full capabilities of the cloud, including elastic scaling and automatic upgrades.

4. Headless, where the front-end user experience is completely decoupled from the back-end logic, allowing for complete design freedom to create the user interface and connect to other channels and devices.

It comes with following core concepts:

• Sellers
• Buyers
• Suppliers
• Product Catalogs
• Orders and Fulfillment

Instances of all of these are grouped in a marketplace. OrderCloud allows you to setup multiple marketplaces under the same account. Marketplaces are completely separated from each other (multi-tenant principle).

Postman friendly

OrderCloud is basically an API in the cloud that you call (GET / POST / … over HTTPS, SDKs available for multiple languages) to setup and run your store site.

The API is exposed with OpenApi, a standard, language-agnostic interface to RESTful APIs which allows both humans and computers to discover and understand the capabilities of the service without access to source code, documentation, or through network traffic inspection maintained by swagger.

Simply import https://api.ordercloud.io/v1/openapi/v3 into Postman to get a collection with all available routes:

The core concepts and their relations are easily managed using the standard GET / POST / PUT / … HTTP verbs. See the api reference to get a feeling of what’s possible.

So what about the back-end?

In a headless scenario, the front-end is something you build yourself of course. But what about the back-end? Some products offer you a straightforward user interface where you can see and edit the core concepts (e.g. change values), but not order cloud. The only back-end you have in the cloud is developer-oriented. Basically it’s PostMan in your browser:

Cool, but not something you want to give (most of) your business users.

In its feature guide, Sitecore mentions HeadStart as (among other things) a tool for commerce management (basically, a back-end). Unfortunately, the only info I found on it is an example implementation on GitHub (https://github.com/ordercloud-api/headstart). It illustrates to possibilities to do your own implementation, but -in my opinion-, it’s not something you can convince a customer with as an out-of-the-box solution. The screenshots in the guide look totally different from the demo environment, so more to come on this I hope.

It’s all about me

An interesting concept in OrderCloud that will certainly speed up your front-end store implementation is building up context around your current shopper, simply called me. The idea is that, in OrderCloud, you

• model a buyer organization. This defines the catalog, products, prices, …. This can range from a straightforward (single buyer / seller) to complex (lots of different buyers each having their specific sellers, prices, rules, …) setups.
• assign one or more buyer users to a buyer organization. Basically, this is somebody who can buy the configured catalog / product / prices. These can be known / registered users or a default anonymous user.

On your front-end store implementation, you

• identify the current shopper using an external login or as anonymous
• get an oauth/token for this shopper from OrderCloud (identified as a buyer user)

With this token you call the /me endpoint to get back context specific information for this user:

• /me/addresses
• /me/catalogs
• /me/products
• /me/orders
• /me/promotions
• …

The endpoints support all kinds of operations: GET by ID, filter by propery, text search, …

In my opinion, this is where the power of OrderCloud lays: the core concepts are very flexible to set up in terms of variations and dependencies.

This allows you to setup a back-end integration process, responsible for syncing data from various back-end systems into the OrderCloud format. Enforcing the logic business requires you to.

On the front-end, all of this complexity is abstracted away behind the authentication token and uniform endpoints. Making it very easy to offer a personalized shopping experience (in terms of products, prices, …) for your customers.

What’s next?

A first investigation of OrderCloud looks promising. You can clearly see this product has grown from real world experience, having all those possibilities for modelling buyer / seller relations and personalizing offerings. I definitely see a potential to start building with it for some of my customers. Hopefully I can share you some more hand’s on experience later.

OrderCloud has been acquired quite recently by Sitecore, so it doesn’t have much Sitecore in it (yet). The claim of HeadStart in the feature guide hints we can expect something in the near future (Horizon style tool / interface maybe?). It’s definitely something business users will be happy with.

And what about integrating with other Sitecore products? Importing products from Sitecore Product Content Management, incorporating the API with the headless SDK or streamlining it Boxever CDP with to name just a few.

Tags: #ordercloud

When repeating the setup on the client’s subscription, deployment went well but we failed to log in into Sitecore:

• We try to access /sitecore.
• Since we’re not logged in, Sitecore CM redirects us to Sitecore ID.
• On Sitecore ID, we enter our credentials.
• Everything seems fine and we are redirected back to Sitecore CM.
• We see the browser redirecting, but the page failed to load: the browser fails with a timeout (IIRC).

A quick look at the log files did not reveal any errors or failures.

In my setup, everything worked as expected, so what’s different for our client? Since this is a real project setup, of course the URLs (and TLS) are going to be customer specific (so not globalhost). Typo’s are easily made, but not this time: double checking the config did not reveal any mistakes.

With no traces in the logs we had to dig a little bit deeper. Since single-sign on is all about redirection and passing along the required information in the process, taking a closer look at the network traffic made sense.

(see Virtual Developer Day 2020 – Getting to Know Sitecore Identity – George Chang for a good introduction on Sitecore ID)

Security best practices for the win

A useful tool to do this is fiddler (fiddler classic in my case): a web debugging proxy to successfully log, inspect, and alter HTTP(s) network requests and server responses.

I was already thinking bad tokens, encryption issues, claim mismatches, … Luckily the issue was not as complex as I feared. Comparing network traffic from my (working) setup with traffic from the client setup revealed they had one HTTP header missing: Strict-Transport-Security, also known as HSTS.

If a website accepts a connection through HTTP and redirects to HTTPS, visitors may initially communicate with the non-encrypted version of the site before being redirected, if, for example, the visitor types http://www.foo.com/ or even just foo.com. This creates an opportunity for a man-in-the-middle attack. The redirect could be exploited to direct visitors to a malicious site instead of the secure version of the original site.

The HTTP Strict Transport Security header informs the browser that it should never load a site using HTTP and should automatically convert all attempts to access the site using HTTP to HTTPS requests instead.

https://developer.mozilla.org/en-US/docs/Glossary/HSTS

This is exactly what happens in my fiddler trace: even though the server tells my browser to go to HTTP (using the Location: header), my browser ignores this and connects over HTTPS directly (line 52).

Why would Sitecore redirect us to HTTP? No idea. As far as I can tell all our config is set correctly. I can only assume this is leftover code from a time where local development was often done on HTTP. With Sitecore on containers, development setup is a lot closer to production setup, including running on HTTPS.

Ingress

So why did I did not face this issue on my setup? An important part of a Kubernetes setup is managing how external traffic can access the services in your cluster (HTTP, HTTPS, TLS, …). This is done with the Ingress API object. Out of the box, Sitecore suggests you to use the NGINX Ingress controller for this.

NGINX is a fine default, but my client has a track record in Kubernetes (not on Azure) and for their networking requirements they standardize on Istio, a service mesh. To use it, we had to switch NGINX with Istio Ingress.

Turns out HSTS is enabled by default in NGINX (see https://kubernetes.github.io/ingress-nginx/user-guide/tls/#http-strict-transport-security), but not in Istio. Enabling it is quite easy though (in case you might be interested): https://www.wagner-dev.com/istio-configure-strict-transport-security-hsts.html

In a development setup, network traffic is typically handled with traefik. Sitecore provides you with docker-compose files that force this behavior.

So why didn’t we spot this right away? The browser we were using (it’s name starts with a ‘C’) doesn’t show the protocol in the URL bar, so we simply didn’t spot it. That we had to test this through a screen share session probably didn’t help neither (but that’s a different story).

In summary

• Sitecore login depends on HSTS.
• HSTS is enabled by default for NGNIX ingress, but that might not be the case for other ingress controllers (e.g. Isitio).
• You can’t see what you can’t see.

Tags: #aks, #kubernetes, #ngnix, #istio.