[Bug]: X5C provisioner doesn't work behind caddy

[stulle123]

Steps to Reproduce

Hey there!

I'm using the X5C provisioner to issue SSH host certificates. Works fine without a reverse proxy, with caddy it doesn't.

Reaching /root/*, /roots.pem, /provisioners, /version, /ssh/*, etc. behind caddy works fine.

Here are the steps to reproduce:

• Setup caddy with the config below
• Add X5C provisioner: $ step ca provisioner add x5c --type X5C --x5c-root $MY-CUSTOM-ROOT-CERT --admin-password-file $STEPPATH/provisioner_password
• Grab host cert: $ STEPDEBUG=1 step ssh certificate --host --x5c-cert $MY-CLIENT-CERT --x5c-key $MY-CLIENT-KEY $HOSTNAME $MY-SSH-KEY --ca-url "https://caddy:9595"

With caddy I'm getting this error message:

Client:

✔ Provisioner: x5c (X5C)
✔ CA: https://caddy:9595
The request lacked necessary authorization to be completed. Please see the certificate authority logs for more info.

CA:

With caddy enabled:

authority.Authorize: authority.authorizeSSHSign: provisioner not found or invalid audience (https://caddy:9595/1.0/ssh/sign#x5c/x5c)

Full log:

step-ca  | time="2025-05-20T12:05:10Z" level=warning duration="378.502µs" duration-ns=378502 error="authority.Authorize: authority.authorizeSSHSign: provisioner not found or invalid audience (https://caddy:9595/1.0/ssh/sign#x5c/x5c)" fields.time="2025-05-20T12:05:10Z" method=POST name=ca ott=<long-ott-value> path=/ssh/sign protocol=HTTP/2.0 referer= remote-address=172.20.0.5 request-id=f1d9909f-c0e9-4196-9b7f-f312749ea2d3 size=144 status=401 user-agent="Smallstep CLI/0.28.6 (linux/arm64)" user-id=

No caddy:

step-ca  | time="2025-05-20T12:06:10Z" level=info certificate=<my-cert-foo> certificate-type="[email protected] host certificate" duration=5.241085ms duration-ns=5241085 fields.time="2025-05-20T12:06:10Z" method=POST name=ca ott=<long-ott-value> path=/ssh/sign principals="<foo-host-bar>" protocol=HTTP/2.0 public-key="SHA256:ZEDUxJ+thwkw/xupELWHDGryL8aHncDQVfAi5PyWZec (ECDSA-CERT)" referer= remote-address=172.20.0.6 request-id=9647602c-be7d-4aed-89e6-06b8b7ef10b7 serial=17348826345035799124 size=647 status=201 user-agent="Smallstep CLI/0.28.6 (linux/arm64)" user-id= valid-from="2025-05-20T12:05:10Z" valid-to="2025-06-01T09:08:43Z"

Caddy config:

https://caddy:9595 {
	tls {
		issuer acme {
			dir https://step-ca:9000/acme/acme/directory
			email [email protected]
			trusted_roots /step-ca-certs/root_ca.crt
			disable_tlsalpn_challenge
		}
	}
	@ca {
		# Allows clients to bootstrap with "step ca bootstrap"
		path /root/*
		# For downloading the root certificate
		path /roots.pem
		# Exposes provisioner configuration
		path /provisioners
		path /version
		# For issuing SSH certificates
		path /ssh/*
		# For JWK certificate issuance
		path /1.0/*
	}
	handle @ca {
		reverse_proxy https://step-ca:9000 {
			transport http {
				tls_trust_pool file {
					pem_file /step-ca-certs/root_ca.crt
				}
			}
		}
	}
}

Cheers,

stulle123

Your Environment

• OS - smallstep/step-ca:0.28.3 Docker image
• step-ca Version - Step CA v0.28.3 (25-03-18)
• step cli Version Smallstep CLI/0.28.6 (linux/arm64)

Expected Behavior

Response from step-ca should be:

step-ca  | time="2025-05-20T12:06:10Z" level=info certificate=<my-cert-foo> certificate-type="[email protected] host certificate" duration=5.241085ms duration-ns=5241085 fields.time="2025-05-20T12:06:10Z" method=POST name=ca ott=<long-ott-value> path=/ssh/sign principals="<foo-host-bar>" protocol=HTTP/2.0 public-key="SHA256:ZEDUxJ+thwkw/xupELWHDGryL8aHncDQVfAi5PyWZec (ECDSA-CERT)" referer= remote-address=172.20.0.6 request-id=9647602c-be7d-4aed-89e6-06b8b7ef10b7 serial=17348826345035799124 size=647 status=201 user-agent="Smallstep CLI/0.28.6 (linux/arm64)" user-id= valid-from="2025-05-20T12:05:10Z" valid-to="2025-06-01T09:08:43Z"

Actual Behavior

authority.Authorize: authority.authorizeSSHSign: provisioner not found or invalid audience (https://caddy:9595/1.0/ssh/sign#x5c/x5c)

Additional Context

No response

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[stulle123]

Not sure if this is something I missed in the caddy config or if this an actual step-ca bug.

[hslatman]

I believe it's due to providing --ca-url "https://caddy:9595", which then ends up in the JWT that's sent to the CA in the aud claim. The CA will then check that to be one of the valid audiences based on the hostnames its configured for. What happens if you add caddy to the dnsNames in the configuration: https://smallstep.com/docs/step-ca/configuration/#example-configuration?

[stulle123]

Omitting --ca-url "https://caddy:9595" wasn't it, but adding caddy to the dnsNames fixed it indeed! Thanks!

[hslatman]

Yeah, I worded it a bit confusing. --ca-url is necessary at all times, but it is often read from a configuration file after bootstrapping with step ca bootstrap. It will always be used to determine which hostname to connect to, and is also used to determine the audience for the JWT.

[hslatman]

I've opened a PR to include this in our docs: smallstep/docs#414. Closing this issue now.