Resolve CVE-2023-44487

[ahrtr]

What would you like to be added?

Refer to GHSA-qppj-fm5r-hxr3

Bump golang.org/x/net:

• 0.17.0 for 3.4/3.5/main
  • bump golang.org/x/net to v0.17.0, google.golang.org/grpc to v1.58.3 #16739
  • [3.5] Bump golang.org/x/net to v0.17.0 #16743
  • [3.4] Bump golang.org/x/net to v0.17.0 #16746

Bump golang:

• 1.21.3 for main: Bump go to 1.21.3 #16741
• 1.20.10 for 3.4/3.5
  • [3.5] Bump go to v1.20.10 #16745
  • [3.4] bump golang version to 1.20.10 #16744

Bump grpc-go:

• 1.58.3 for main: bump golang.org/x/net to v0.17.0, google.golang.org/grpc to v1.58.3 #16739
• 1.56.3 or 1.57.1 or 1.58.3 for 3.4/3.5.
  • [3.5] Bump grpc-go to1.47 (and fix the connection-string format) #16625
  • [3.5] Upgrade gRPC-go to v1.52.0 #16781
  • [3.5] Upgrade gRPC-go to 1.58.3 #16790
  • [3.4] Upgrade grpc-go to 1.29.1 and Backport "Introduce grpc-1.30+ compatible client/v3/naming API." #16795
  • [3.4] Backport clientv3 naming implementation #16800
  • [3.4] Backport clientv3:get AuthToken gracefully without dialing gRPC with balancer API to get extra connection #16826
  • [3.4] Backport clientv3: Replace balancer with upstream grpc solution #16827
    • Backport [3.4] clientV3: simplify grpc dialer usage. Remove workaround #11184 after bumping grpc to 1.26.0. #16842
    • [3.4] Backport #12671 clientv3: Replace balancer with upstream grpc solution #16844
    • [3.4] backport client: call .Endpoints() in dial() in client/v3/client.go instead of accessing cfg.Endpoints directly #16857
    • [3.4] backport #13359 Fix http2 authority header in single endpoint scenario #16988
  • [3.4] Upgrade grpc to 1.52.0 #16997
  • [3.4] Upgrade grpc 1.58.3 #16999

Clarification

• https://github.com/grpc/grpc-go/pull/6706/files#r1354612788

Why is this needed?

resolve CVE

[tjungblu]

The fix in apiserver seems a little bit more involved than just bumping dependencies:
https://github.com/kubernetes/kubernetes/pull/121120/files

@ahrtr already wrote it down on the OP - sorry!

[ahrtr]

The fix in apiserver seems a little bit more involved than just bumping dependencies:
https://github.com/kubernetes/kubernetes/pull/121120/files

Thanks @tjungblu for pointing this out. (I am not an api-server expert) based on the PR, it seems api-server made two changes:

• In (r *rudimentaryErrorBackoff) OnError(error), do not sleep with the lock, so as not to block all callers;
• Do not allow unauthenticated clients to keep the connections open;

I don't think etcd has similar problems (note K8s doesn't enable auth at all).

Another thing... I noticed that Kubernetes hasn't bumped grpc-go version yet (note latest grpc-go versions have fix to CVE-2023-44487), I guess probably it's because api-server doesn't start grpc server at all.

@liggitt

[chaochn47]

Bump grpc-go:
1.56.3 or 1.57.1 or 1.58.3 for 3.4/3.5.

Just took a quick peak at the gRPC dependency upgrade history on the difference of 3.4 and 3.5:

1. tools: Update tools & go.mod for protos (gogo/protobuf 1.3.1, grpc 1.29.1, grpc-gateway 1.14.6) #12397
2. Update grpc dependency to 1.32. #12709
3. Update dep: github.com/golang/protobuf v1.3.5 ->  v1.5.1, grpc: 1.32->1.36 #12801
4. Add initial Tracing with OpenTelemetry #12919
5. backport-3.5: Update of dependencies:zap, bbolt, grpc, ... #13079
6. [3.5] etcdserver: bump OpenTelemetry to 1.0.1 and gRPC to 1.41.0 #14312

I think goal will be let 3.4 catch up with 3.5 first, ignore other unrelated dependency upgrade and eventually upgrade to 1.58.3, correct?

Is there any major call outs this upgrade could potentially break 3.4 release? @ahrtr @serathius @fuweid

[ahrtr]

I think goal will be let 3.4 catch up with 3.5 first, ignore other unrelated dependency upgrade and eventually upgrade to 1.58.3, correct?

I am good as long as eventually we can resolve the CVE for both 3.5 and 3.4.

Is there any major call outs this upgrade could potentially break 3.4 release? @ahrtr @serathius @fuweid

No immediate call out in my head for now. We can have more discussion if we run into any issue during the upgrade. thx

[fuweid]

Just quick bump the version in my local for v3.4. It might introduce other changes, such as #12675. We might need to try to bump and get the changes result and then evaluate it.

[ahrtr]

Just quick bump the version in my local for v3.4. It might introduce other changes, such as #12675.

I would suggest not to make such changes if they aren't necessary. Another such example: #12671

[chaochn47]

Just quick bump the version in my local for v3.4. It might introduce other changes, such as #12675.

I would suggest not to make such changes if they aren't necessary. Another such example: #12671

https://github.com/grpc/grpc-go/releases/tag/v1.30.0

1. This release removed the deprecated naming package that etcd 3.4 depends on in https://github.com/etcd-io/etcd/blob/release-3.4/clientv3/naming/grpc.go#L25. etcd client won't compile without backport Cleanup grpc client/v3/naming API #12675.
2. etcd 3.4 balancer is built on top of the removed API balancer.ClientConn.UpdateBalancerState.

I guess we have to check in #12675 and #12671 in release-3.4 in order to bump up gRPC-go version instead of forking the gRPC usage in another release branch to fix the CVE.

[ahrtr]

1. This release removed the deprecated naming package

OK, then we have to backport #12675 to 3.4 in order to bum grpc-go version

2. etcd 3.4 balancer is built on top of the removed API balancer.ClientConn.UpdateBalancerState.

Is there a simple solution to just replace the removed balancer.ClientConn.UpdateBalancerState, instead of replacing the whole client side load balancer solution? Note the grpc.Resolver is still an experimental package.

[chaochn47]

Is there a simple solution to just replace the removed balancer.ClientConn.UpdateBalancerState, instead of replacing the whole client side load balancer solution? Note the grpc.Resolver is still an experimental package.

I need to investigate more to fully understand before adapting any code changes to the release.34 client based on the gRPC update since v1.26.0. ETA is end of this week.

Based on #12124 (comment) and #12652, if we still maintain release-3.4 for a prolonged time, it will bite us again since it's forked.

[fuweid]

etcd 3.4 balancer is built on top of the removed API balancer.ClientConn.UpdateBalancerState

This one seems easy to fix. We just need to replace the API with new type input.

Remove the original balancer plugin API, based on grpc.Balancer, and all related functionality.

From https://github.com/grpc/grpc-go/releases/tag/v1.30.0
REF: grpc/grpc-go#3180

But for the balancer v2 API, grpc/grpc-go#3431 has changed the Balancer API.
I think we can't rewrite all the things to adapt the new API because it needs more test cases to ensure the functionality is working.
We can just backport this clientv3: Replace balancer with upstream grpc solution and then bump the grpc.

---

I try to backport the following pull requests in my local (without test code and testing) and then with their changes, we can bump grpc to v1.52.0 easier. However, each change needs more eyes to check it.

• clientv3: replace deprecated UpdateBalancerState/Picker API (Part of update grpc-go version to v1.32.0 which has some breaking api changes #12398)
• Introduce grpc-1.30+ compatible client/v3/naming API. 5d7c1db
• endpoints: implement Update method for EndpointManager. 571ed50
• *: implement Endpoint Watch and new Resolver 8feb55f
• Get rid of legacy client/v3/naming API. Update grpcproxy to use the new abstractions. a836a80
• clientv3:get AuthToken gracefully without extra connection. (clientv3:get AuthToken gracefully without extra connection. #12165) 8050881
• Bump to grpc-1.29.1
• Replace client/v3/balancer with standard components: resolver + round_robin LB 0b75fed

[ahrtr]

Oops, we haven't bumped grpc-go version yet:(

[ahrtr]

All done!

[ahrtr]

Thanks all again, especially @chaochn47 and @fuweid !