Replace kube-rbac-proxy with controller-runtime's built-in TLS capabilities #437
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
frobware
force-pushed
the
drop-kube-rbac-proxy
branch
from
0b9c314 to
c3b8925
Compare
Test Script to verify bpfman-{agent, operator} metrics are being scraped#!/usr/bin/env bash
set -euo pipefail
echo "=== Controller-manager ServiceMonitor endpoint ==="
oc get servicemonitor bpfman-controller-manager-metrics-monitor \
-n bpfman -o yaml \
| yq e '.spec.endpoints[0]' -
echo
echo "=== Agent ServiceMonitor endpoint ==="
oc get servicemonitor bpfman-agent-metrics-monitor \
-n bpfman -o yaml \
| yq e '.spec.endpoints[0]' -
echo
echo "=== Prometheus controller targets (bpfman namespace) ==="
oc exec -n openshift-monitoring prometheus-k8s-0 -c prometheus -- \
curl -s 'http://localhost:9090/api/v1/targets?state=active' \
| jq '
.data.activeTargets[]
| select(.discoveredLabels["__meta_kubernetes_namespace"] == "bpfman")
| select(.scrapePool | test("controller-manager-metrics-monitor"))
| { job: .scrapePool, url: .scrapeUrl, health: .health }
'
echo
echo "=== Prometheus agent targets (bpfman namespace) ==="
oc exec -n openshift-monitoring prometheus-k8s-0 -c prometheus -- \
curl -s 'http://localhost:9090/api/v1/targets?state=active' \
| jq '
.data.activeTargets[]
| select(.discoveredLabels["__meta_kubernetes_namespace"] == "bpfman")
| select(.scrapePool | test("agent-metrics-monitor"))
| { job: .scrapePool, url: .scrapeUrl, health: .health }
' |
% ~/verify-bpfman-metrics.sh
=== Controller-manager ServiceMonitor endpoint ===
bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
path: /metrics
port: https-metrics
scheme: https
tlsConfig:
caFile: /etc/prometheus/configmaps/serving-certs-ca-bundle/service-ca.crt
insecureSkipVerify: false
serverName: bpfman-controller-manager-metrics-service.bpfman.svc
=== Agent ServiceMonitor endpoint ===
bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
path: /metrics
port: https-metrics
scheme: https
tlsConfig:
caFile: /etc/prometheus/configmaps/serving-certs-ca-bundle/service-ca.crt
insecureSkipVerify: false
serverName: bpfman-agent-metrics-service.bpfman.svc
=== Prometheus controller targets (bpfman namespace) ===
{
"job": "serviceMonitor/bpfman/bpfman-controller-manager-metrics-monitor/0",
"url": "https://10.130.0.26:8443/metrics",
"health": "up"
}
=== Prometheus agent targets (bpfman namespace) ===
{
"job": "serviceMonitor/bpfman/bpfman-agent-metrics-monitor/0",
"url": "https://192.168.7.108:8443/metrics",
"health": "up"
}
{
"job": "serviceMonitor/bpfman/bpfman-agent-metrics-monitor/0",
"url": "https://192.168.7.109:8443/metrics",
"health": "up"
}
{
"job": "serviceMonitor/bpfman/bpfman-agent-metrics-monitor/0",
"url": "https://192.168.7.110:8443/metrics",
"health": "up"
} |
|
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #437 +/- ##
===========================
===========================
☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
frobware
changed the title
|
@frobware, this pull request is now in conflict and requires a rebase. |
This commit removes the kube-rbac-proxy sidecar container that was previously used to secure metrics. Metrics are temporarily disabled by setting metrics-bind-address=0, which prevents the metrics server from starting (as per controller-runtime implementation). Future commits will reintroduce metrics with native TLS support from controller-runtime, following the approach recommended in kubernetes-sigs/kubebuilder#3907. Removed kube-rbac-proxy component labels from metrics service definition and ClusterRole. The app.kubernetes.io/component label will be updated to 'metrics' in a subsequent commit that implements TLS-enabled metrics with controller-runtime's built-in capabilities. Signed-off-by: Andrew McDermott <[email protected]>
Add Go dependencies required for upcoming TLS metrics implementation: - sigs.k8s.io/controller-runtime/pkg/certwatcher - sigs.k8s.io/controller-runtime/pkg/metrics/filters This commit includes only the vendored dependencies via: 1. go get sigs.k8s.io/controller-runtime/pkg/certwatcher 2. go get sigs.k8s.io/controller-runtime/pkg/metrics/filters 3. go mod tidy 4. go mod vendor Upcoming changes will use these dependencies to implement TLS-based metrics in the operator without requiring kube-rbac-proxy. Signed-off-by: Andrew McDermott <[email protected]>
Implement controller-runtime's built-in TLS support for secure metrics, following the earlier removal of kube-rbac-proxy: 1. Updated main.go with controller-runtime's TLS capabilities: - Added certificate watcher for rotation - Support for self-signed and provided certificates - RBAC authentication and authorization 2. Added RBAC resources: - metrics_reader_role with GET /metrics permission - auth_delegator_binding for authentication - metrics_reader_rolebinding for service accounts 3. Updated manifests: - Metrics service on port 8443 - Updated metrics-bind-address configuration - Fixed namespace configuration The implementation generates self-signed certificates when needed and follows Kubebuilder's recommended approach without requiring a sidecar. Signed-off-by: Andrew McDermott <[email protected]>
frobware
force-pushed
the
drop-kube-rbac-proxy
branch
from
c3b8925 to
963f233
Compare
frobware
force-pushed
the
drop-kube-rbac-proxy
branch
2 times, most recently
from
fdee32d to
2233100
Compare
|
Testing cert rotation: Then delete the TLS secret: And notice the cert-watcher inject a new TLS cert: |
Signed-off-by: Andrew McDermott <[email protected]>
frobware
pushed a commit
to frobware/bpfman-operator
that referenced
this pull request
May 6, 2025
…s/component-update-ocp-bpfman-operator-bundle chore(deps): update ocp-bpfman-operator-bundle to f56e565
1 task
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR removes the kube-rbac-proxy sidecar container and implements native TLS support for metrics endpoints.
Generic changes
/tmp/k8s-webhook-server/serving-certswith the controller-runtime watching for changesGeneric RBAC Changes
FilterProvider: filters.WithAuthenticationAndAuthorizationto the metrics server, which requires API server delegation for authentication validation.OpenShift-specific Changes
service.beta.openshift.io/serving-cert-secret-name)insecureSkipVerify: false) with proper CA paths and server name validation/etc/prometheus/configmaps/serving-certs-ca-bundle/service-ca.crt)