Decouple metrics from hostNetwork using proxy DaemonSet

[frobware]

The bpfman-agent DaemonSet requires hostNetwork: true for eBPF operations such as loading XDP programs and accessing host network interfaces. It also exposes Prometheus metrics via TCP port 8443 on the host network.

In #437, the metrics service was updated to use TCP port 8443 by default. This aligned with controller-runtime’s default for secure metrics endpoints and is configurable via the bpfman ConfigMap.

However, 8443 is a commonly used port and may plausibly be claimed by other host-level services or privileged containers. In clusters where hostNetwork: true is used, this increases the risk of port binding conflicts. The underlying issue is not the specific port but the need to bind to any TCP port on the host network.

Additionally, in PR #437, the metrics Service was not marked clusterIP: None, so it was not headless. As a result, Prometheus would scrape the Service’s cluster IP, which performs round-robin load balancing across pods. This is not suitable for a DaemonSet, where metrics must be scraped from each pod individually (e.g., to collect per-node data). A headless Service is required to expose the full set of pod endpoints for proper per-pod scraping.

In cloud environments with restrictive security groups, the use of hostNetwork: true introduces additional operational complexity. Since hostNetwork pods are assigned node IPs rather than pod IPs, Prometheus scraping across nodes may fail (does fail) unless explicit firewall rules or security group exceptions are configured. This creates cloud-provider-specific coupling, requires coordination with infrastructure teams, and increases deployment friction - particularly in environments like AWS where inter-node traffic to arbitrary ports is not allowed by default.

This PR proposes an architectural change to eliminate the requirement to bind a host port for metrics entirely.

Options considered:

1. Do not expose metrics from bpfman-agent.

   • Simple
   • But eliminates observability

2. Leave metrics on hostNetwork and document the need to open firewall ports in environments that restrict inter-node traffic (e.g. cloud platforms using security groups). This is not typically required in libvirt or bare-metal environments.

   • Minimal change (would still want a headless service, for example)
   • Still subject to TCP port conflicts and blocked metrics

3. Introduce a metrics-proxy DaemonSet (chosen).

   • Avoids host-level TCP port binding
   • Enables per-pod scraping over HTTPS
   • Works across clouds without firewall changes
   • Small resource overhead

Implementation:

• Add a metrics-proxy DaemonSet that runs in the container network.
• It mounts a shared volume with bpfman-agent to access a Unix domain socket.
• Proxies metrics over HTTPS using controller-runtime’s metrics server.
• bpfman-agent now serves metrics only over the socket, no longer binds TCP ports.
• The associated Service is marked clusterIP: None to enable per-pod scraping.
• ServiceMonitor now targets /proxy-metrics.
• Proxy HTTP client timeout is set to 8 seconds to fail before Prometheus’s 10-second default scrape timeout.

Outcome:

• Port conflicts are eliminated.
• Prometheus can scrape all pods without infrastructure changes.
• No need to modify AWS security groups or cloud firewall rules.
• Metrics are cleanly separated from core eBPF operations.

Cons:

• Another daemonset, more resource usage
• Requires privileged: true to access the host-mounted Unix socket, but:
  • This does not widen the security surface - the original bpfman-agent pod already ran with privileged: true
  • The metrics-proxy pod inherits only the minimum required permissions
  • The shared volume containing the Unix socket is mounted read-only
  • The proxy only reads metrics; it does not interact with eBPF or mutate state