Add support for the AWS Load Balancer mTLS

[slaskawi]

Summary

As highlighted in keycloak/keycloak#25579, Keycloak doesn't support AWS Load Balancer (ALB) mTLS certificate proxying.

Supporting such a certificate handling requires reading the x-amzn-mtls-clientcert that contains URL Encoded, Base64 encoded certificate (so the implementation needs to first, URL decode it to a String and then perform base64 decoding.

This ticket contains the following subtasks:

• Establish path forward for this issue with the Keycloak Team. Potential solutions are:
  • Implementing this as a Keycloak Extension
  • Implementing this directly in Keycloak, potentially extending the NginxProxySslClientCertificateLookupFactory implementation with base64 decoding and overriding headers
• Agree with @eddiezane and his team on the path forward
• Proceed with the implementation and test it on EKS

Existing workaround for Istio

There is a workaround for this issue today, which requires using Envoy Filter. @eddiezane was kind enough to share the implementation:

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: alb-clientcert-parse
  namespace: keycloak
spec:
  workloadSelector:
    labels:
      app.kubernetes.io/instance: keycloak
      app.kubernetes.io/name: keycloak
  configPatches:
  - applyTo: HTTP_FILTER
    match:
      context: SIDECAR_INBOUND
      listener:
        filterChain:
          filter:
            name: envoy.filters.network.http_connection_manager
            subFilter:
              name: envoy.filters.http.router
        portNumber: 8080
    patch:
      operation: INSERT_BEFORE
      value:
        name: envoy.filters.http.lua
        typed_config:
          '@type': type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua
          defaultSourceCode:
            inlineString: |
              function url_decode(str)
                str = string.gsub(str, "%%(%x%x)", function(h) return string.char(tonumber(h,16)) end)
                str = string.gsub(str, "\r\n", "\n")
                return str
              end

              function envoy_on_request(request_handle)
                local raw_cert = request_handle:headers():get("x-amzn-mtls-clientcert")
                if raw_cert ~= nil and raw_cert ~= "" then
                  local decoded = url_decode(raw_cert)
                  local encoded = request_handle:base64Escape(decoded)
                  request_handle:headers():add("x-client-cert", encoded)
                end
              end

[slaskawi]

The prototype might be found in #483. There's also an ongoing conversation on different approaches.

Blocking this ticket until we hear back from Mission Heroes

[bburky]

When implementing this, please validate or address the following issues which may allow spoofing users by sending another user's public key in the x-amzn-mtls-clientcert header (assume public keys are indeed public, I think they're in some databases you can query for all users).

1. What does AWS do if an incoming request has an existing x-amzn-mtls-clientcert header? I would hope it strips it and replaces it, and doesn't perhaps give you duplicate headers or something. Make sure to check that AWS/Istio/Keycloak all use the same normalization (such as case insensitivity)

   • The intent is to make sure an external user can't send a malicious request with a x-amzn-mtls-clientcert header.

2. This header is really only safe to trust if the HTTP request was handled by an AWS Load Balancer. But it's also possible to reach Keycloak via in-cluster only paths like the k8s LoadBalancer directly (access the pod IP), via the istio-ingress Gateway or via the Keycloak Pod directly.

   • The intent is to prevent a compromised/untrusted cluster-internal workload (e.g. GitLab CI) from connecting to Keycloak in-cluster and spoofing the x-amzn-mtls-clientcert header.
   • A possible option is an AuthoriztionPolicy to check source IP at the ingress gateway, and an AuthorizationPolicy at the workload to check the source is an istio ingress gateway.
     • Configuring source IP sounds brittle and hard to configure well though (you can't allow all of AWS IPs, an adversary might make a malicious load balancer and get an AWS IP of their own)
     • Is there a better way than source IP to validate that the request came from the AWS load balancer? Can AWS perhaps initiate mTLS itself when connecting to the ingress-gateway and we could check a client cert? Is there any AWS recommended way to authenticate the request here?

When we implemented Istio ingress-gateway TLS termination with KC_SPI_X509CERT_LOOKUP_PROVIDER=nginx we had this concern too and addressed it by:

1. making sure the Istio ingress gateway (via VirtualService configuration) drops any existing header and populates a new one
   https://github.com/defenseunicorns/uds-core/blob/v0.42.0/src/keycloak/chart/templates/uds-package.yaml#L168-L173
2. ensuring only the istio gateways are allowed to set this header (via AuthorizationPolicy configuration) https://github.com/defenseunicorns/uds-core/blob/v0.42.0/src/keycloak/chart/templates/istio-admin.yaml#L51-L68

[slaskawi]

Sent a meeting request to @mjnagel and @eddiezane. We need to clarify what is the expected level of support we deliver here - is it a simple Delivery add-on or do we need to provide the full support for it.

[slaskawi]

This is the recent change:

request_handle:headers():remove("x-client-cert")

[slaskawi]

Other notable changes:

• HA Proxy Provider (Seb to look at the differences between HAProxy and NGINX Providers)