Kube-Router - v2.0.0

Summary

kube-router v2.0.0 is a major release that the project has been working on for almost a year. We're very excited to finally enable some long-requested features.

Its worth noting, that almost 50% of the existing code base had to be modified in order to bring this release to fruition. We've done testing via the several release candidates that we've released over the last few months (huge thanks to all of the users that engaged that by submitting fixes or reporting bugs), but I'm sure that there are still a few out there that we haven't caught yet.

The project recommends that users approach deploying this release to their production critical clusters with caution. If you do find a bug, please open a new GitHub Issue with the Report a v2.0.0 Release Issue issue type so that we can resolve it as soon as possible.

Major New Features

Dual-Stack implemented in all kube-router controllers (e.g. --run-firewall, --run-router, --run-loadbalancer, & --run-service-proxy)
IPAM management for LoadBalancer service types (previously kube-router would only perform BGP, firewall, and proxy functionality and users would need to supplement kube-router with something like MetalLB to get IPs from a pool for their LoadBalancer services)
FoU tunneling capability added for pod networking (previously only ipip was supported)

Backward Compatibility Breaks

You'll also notice that this has been tagged as a major release version. This is due to the fact that there are breaking changes in this version of kube-router that are not backwards compatible with previous versions of kube-router.

Tunnel names have changed, so if you run kube-router with an overlay network (ipip tunnels), you'll want to deploy this update carefully. At this point, the project recommends doing a rolling-reboot of nodes after the application (or as part of) of this release to ensure that unused tunnels are properly cleaned up and don't have a negative impact on traffic flows.

As part of this work, kube-router now needs to be able to get, list, and watch EndpointSlice objects. Please make sure to update your ClusterRole RBAC before deploying this release if you use kube-router with the --run-service-proxy functionality enabled or you'll get an error. For examples of this RBAC please see the daemonset examples.

Other Notes

Dual-stack handling is fairly complex and it comes with a few gotchas. Users that want to enable dual-stack features should carefully read the IPv6 / Dual-Stack Support in kube-router documentation to ensure that they fully understand the feature set, requirements, and limitations.

If you want to run the new load balancer controller (e.g. --run-loadbalancer) kube-router will need some additional RBAC and requires a few environment variables set. See load-balancer allocation docs for more information.

For more information on enabling or migrating to FoU tunneling and any caveats on doing so, please see the Tunnels in kube-router documentation.

Now that kube-router has dual-stack capability, it doesn't make sense to have an annotation that can only represent a single pod CIDR any longer. As such, with this release we are announcing the deprecation of the kube-router.io/pod-cidr annotation in favor of the new kube-router.io/pod-cidrs annotation see the IPv6 / Dual-Stack Support in kube-router documentation for more information.

Major Dependency Updates

This release also updates:

Alpine base container from 3.17 to 3.18
Go from v1.19 to v1.20
Kubernetes from v1.25 to v1.27
GoBGP from v3.11.0 to v3.17.0.

Contributions

A release this big would not be possible without an awesome community. Thanks so much to all of the following people that helped play a part in this release: @brandond, @iggy, @whooo, @BoleynSu, @k-raval, @rwagoner, @vadorovsky, and @thomasferrandiz

Changelog (from v1.6.0 - v2.0.0)

1c89be87 - feat(go): upgrade 1.20.7 -> 1.20.9 <Aaron U'Ren>
3db03ccf - feat(dep): upgrade all dependencies <Aaron U'Ren>
d2c3858c - feat(goreleaser): upgrade v1.14.1 -> v1.21.2 <Aaron U'Ren>
1a4896f4 - feat(lint): upgrade golangci-lint v1.50.1 -> v1.54.2 <Aaron U'Ren>
678b7129 - fix(ecmp_vip.go): non-local service advertisement <Aaron U'Ren>
1a891c33 - fix(dsr): add family specific link inside pod <Aaron U'Ren>
514a8af7 - fix(dsr): add family for fwmark <Aaron U'Ren>
c92f76aa - fix(service_endpoints_sync.go): use save command <Aaron U'Ren>
9abe20d5 - fix(NSC): compare all pod IPs for endpoint check <Aaron U'Ren>
9f23cf5a - fix(linux_networking.go): add better error messages <Aaron U'Ren>
7ce09a64 - fix(linux_networking.go): don't return err on warn <Aaron U'Ren>
9d63cc68 - feat(debug): add some extra debug at level 3 <Aaron U'Ren>
4c6e19f2 - feat(ipset): consolidate ipset usage across controllers <Aaron U'Ren>
c62e1b79 - feat(linux_networking.go): add more logging info <Aaron U'Ren>
da73dea6 - feat(NSC): use EndpointSlice instead of Endpoints <Aaron U'Ren>
15cd4eb0 - feat(nsc): add more insight into sync steps <Aaron U'Ren>
81bc9e20 - fix(nsc): don't modify netmask during flag setup <Aaron U'Ren>
903466b7 - fix(nsc): fail fast during init <Aaron U'Ren>
25ecb098 - feat(nsc): add dualstack capabilities <Aaron U'Ren>
f397a1f0 - feat: increase log level for save/restore msgs <Aaron U'Ren>
a6cd5b00 - lint(metrics_controller.go): reduce line length <Aaron U'Ren>
68a7d03b - fix: take family metrics out of defer <Aaron U'Ren>
301e856a - fix(NPC): remove redundant assign <Aaron U'Ren>
b06b4f05 - Move ipset restore outside policy loop <Brad Davidson>
e34ef29f - Add additional save/restore metrics <Brad Davidson>
aa107d63 - Make metrics registerer/gathererer replacable <Brad Davidson>
e6f668cb - fix: syntax updates for Go 1.20.X and k8s 1.27 <Aaron U'Ren>
5cf1265f - fix(NRC): prevent adding routes with mixed families <Aaron U'Ren>
bab0d4ff - feat(bgp_policies.go): don't override-nexthop for internal peers <Aaron U'Ren>
afdf553f - add loadbalancer address allocator <Erik Larsson>
7699d165 - doc(tunnels.md): add info about changing live clusters <Aaron U'Ren>
ddf857de - doc(tunnel): add information about tunnels <Aaron U'Ren>
944ab917 - fix(FoU): make more robust <Aaron U'Ren>
bac4ae62 - fix(FoU): add docs, sanity checking, and logic reduction <Aaron U'Ren>
2a57d6c1 - Adding FoU encapsulation over IPIP tunnel : added checks for restart and multi-node cases <Kartik Raval>
6ce37e61 - Support for FoU encapsulation for IPIP tunnel <Kartik Raval>
48610217 - fix(NPC): update IPBlocks to be ipFamily specific <Aaron U'Ren>
f0d7f1e1 - netpol: Fix ipset only containing one IP when port name is used. <Boleyn Su>
384ed97a - fix(bgp_policy): allow for statement add / remove <Aaron U'Ren>
1d5c9ce2 - fix(ecmp_vip): update VIPs based on svc change <Aaron U'Ren>
f5ac980b - fix(bgp_policies.go): return -> continue on family set evaluation <Aaron U'Ren>
3387f5c1 - use JoinHostPort for GRPC listen address <Erik Larsson>
76ffcbdb - add generation of router id based on hash of primary IP <Erik Larsson>
57c9b086 - fix(ecmp_vip.go): ClusterIP -> ClusterIPs <Aaron U'Ren>
fe939782 - feat(bgp_policies_test.go): use different IP ranges <Aaron U'Ren>
31c22ff6 - fix(bgp_policies.go): don't get BGP peers twice <Aaron U'Ren>
06f5f8ba - feat(go): update package version to /v2 <Aaron U'Ren>
e51ee3ae - fix(NPC): add warning for unsupported family <Aaron U'Ren>
4e1679f0 - fix(NPC): don't add chains for missing family <Aaron U'Ren>
a2bb2ba8 - doc(bgp.md): clean up grammar and syntax <Aaron U'Ren>
0ecb51de - fix(NPC/pod): check drop policy on ipv4 & ipv6 <Aaron U'Ren>
367aedf8 - fix(bgp_policies): add empty DS set checking <Aaron U'Ren>
aeb51ba6 - fact(bgp_policies): rename clusterIPPrefixSet -> serviceVIPIPPrefixSet <Aaron U'Ren>
6e038360 - fact(bgp_policies): abstract get DS for GoBGP <Aaron U'Ren>
67254ad2 - fix(ecmp_vip): handle ipv4 & ipv6 protocols <Aaron U'Ren>
5f952e0f - test(bgp_policies_test): add local address <Aaron U'Ren>
ec12fda8 - fix(node): do nil checking on FindBestIP util funcs <Aaron U'Ren>
5d7f62c5 - fix(NRC): ensure local addr IP is bindable early <Aaron U'Ren>
67abc4b8 - fix(bgp_peers): adv. AfiSafi based on capabability <Aaron U'Ren>
c491bcb4 - fix(bgp_peers): do peer only if IP protos match <Aaron U'Ren>
0023dedc - fix(NRC): error when nec. host IP not found <Aaron U'Ren>
4f284be5 - fix(NRC): add IPv6 logic to bgp-local-addresses <Aaron U'Ren>
51f46964 - feat(ci): run CI on version prep branches and MRs <Aaron U'Ren>
b3e07682 - fix(options): make clusterIP specification similar to other options <Aaron U'Ren>
a31511d9 - fix(NPC): actually separate chain indices for ipv4 / ipv6 <Aaron U'Ren>
096da81f - fact(NPC): pluralize newIPTablesHandler <Aaron U'Ren>
ddb0e63c - feat(NRC): make NRC dual stack <Aaron U'Ren>
01f2ff2a - fact(NRC): convert BGP set names to const <Aaron U'Ren>
85cecb6e - feat(pod_cidr): handle multiple pod CIDRs <Aaron U'Ren>
c18d811f - fix(kube-router.go): metric message -> not error <Aaron U'Ren>
3db482be - fix(NPC): separate chain indices for ipv4 / ipv6 <Aaron U'Ren>
9d2b3c72 - fix(node.go): make node address errors more helpful <Aaron U'Ren>
d7e2a146 - fix golangci issues <Thomas Ferrandiz>
12561f9f - fix test compilation error <Thomas Ferrandiz>
e5f272e9 - go mod <Thomas Ferrandiz>
76e5d20c - use createGenericHashIPSet <Thomas Ferrandiz>
b3dcaa08 - rename utilsnet import to netutils <Thomas Ferrandiz>
4256a607 - syncPodFirewallChains: loop on all NodeIp to find the pods running on a given Node - Load PodIp in podInfo struct and use it instead of pod.ips[0].IP <Thomas Ferrandiz>
92e91df9 - refactor whitelisting of cluster IP Range <Thomas Ferrandiz>
6fea9c2d - Validate that ClusterIP service range type matches the configuration and update documentation <Thomas Ferrandiz>
16d3cd18 - godoc update <Thomas Ferrandiz>
d1f3839b - remove redundant default value <Thomas Ferrandiz>
d6dcf767 - rename Adresses <Thomas Ferrandiz>
26d06c40 - Turn IPTablesSaveRestore into an interface <Thomas Ferrandiz>
3839ec1d - init iptablesCmdHandlers and ipSetHandlers inside NewNetworkPolicyController <Thomas Ferrandiz>
5b7da83c - disable ipv6 by default <Thomas Ferrandiz>
5d04a9fd - netpol: Add dual-stack support <Michal Rostecki>
08f05a80 - fix(moq): chown generated moqs <Aaron U'Ren>
ee85441c - fix(moq): remove previous moqs before generation <Aaron U'Ren>
4556aa3b - fix(lint): convert sh -> bash <Aaron U'Ren>
3c16d3cd - feat(close_stale.yml): increase operations 30 -> 100 <Aaron U'Ren>
0d12e617 - feat(close_stale.yml): add workflow for closing stale issues & PRs <Aaron U'Ren>
b084c2ae - fix: available typo <guoguangwu>
82cd7c97 - build(deps): bump github.com/aws/aws-sdk-go from 1.44.334 to 1.45.1 <dependabot[bot]>
16474cf9 - build(deps): bump github.com/aws/aws-sdk-go from 1.44.332 to 1.44.334 <dependabot[bot]>
0110d1d3 - Partially revert riscv64 support <Manuel Rüger>
9805c79b - Makefile: Bump go + alpine as well <Manuel Rüger>
ade98d6b - Bump go 1.20 + k8s 1.27 <Manuel Rüger>
e7a521a0 - feat(ci.yml): add ppc64le and riscv64 to ci <Aaron U'Ren>
8b76e9c5 - Add RISC-V 64 support <Iggy Jackson>
70defa4d - build(deps): bump github.com/aws/aws-sdk-go from 1.44.313 to 1.44.332 <dependabot[bot]>
8de9b641 - build(deps): bump golang.org/x/net from 0.12.0 to 0.14.0 <dependabot[bot]>
d6a4045d - doc(ipv6): add differences in --override-nexthop <Aaron U'Ren>
85bcde85 - build(deps): bump github.com/coreos/go-iptables from 0.6.0 to 0.7.0 <dependabot[bot]>
099664a7 - build(deps): bump github.com/aws/aws-sdk-go from 1.44.309 to 1.44.313 <dependabot[bot]>
9372d620 - build(deps): bump github.com/osrg/gobgp/v3 from 3.16.0 to 3.17.0 <dependabot[bot]>
c50bdfc3 - build(deps): bump github.com/onsi/gomega from 1.27.7 to 1.27.10 <dependabot[bot]>
98479d8d - build(deps): bump github.com/aws/aws-sdk-go from 1.44.308 to 1.44.309 <dependabot[bot]>
64784c40 - build(deps): bump google.golang.org/grpc from 1.56.2 to 1.57.0 <dependabot[bot]>
b900acbf - .github: Add github-actions dependabot config <Manuel Rüger>
fb37a647 - build(deps): bump golang.org/x/net from 0.10.0 to 0.12.0 <dependabot[bot]>
a42466a8 - build(deps): bump google.golang.org/grpc from 1.56.1 to 1.56.2 <dependabot[bot]>
371a3a83 - build(deps): bump github.com/osrg/gobgp/v3 from 3.14.0 to 3.16.0 <dependabot[bot]>
09940db7 - build(deps): bump github.com/aws/aws-sdk-go from 1.44.273 to 1.44.308 <dependabot[bot]>
5d39fe05 - build(deps): bump github.com/prometheus/client_golang <dependabot[bot]>
68e0fe5b - build(deps): bump k8s.io/cri-api from 0.27.2 to 0.27.4 <dependabot[bot]>
1701f9c7 - build(deps): bump github.com/stretchr/testify from 1.8.2 to 1.8.4 <dependabot[bot]>
d5fcc784 - build(deps): bump github.com/docker/docker <dependabot[bot]>
7a3a495a - build(deps): bump google.golang.org/protobuf from 1.30.0 to 1.31.0 <dependabot[bot]>
dd3b75a0 - build(deps): bump github.com/docker/docker <dependabot[bot]>