Kube-Router - v2.1.0

Summary

The major theme of this kube-router release is to make kube-router fully conformant with the end-to-end upstream sig-network tests. These are tests that the Kubernetes project provides for testing significant networking features and becoming conformant with the tests means that the networking functionality of kube-router is more consistent with other network providers like kube-proxy, Calico, and Cilium.

Major New Features / Fixes

kube-router's init container now installs CNI plugins when they are missing. The location for these plugins can be controlled by setting the environment variable HOST_BIN_PATH within the init container, but will default to /opt/cni/bin. This is something that all major Kubernetes networking providers do, but kube-router has not done in the past. For more information on CNI plugins, please see: https://github.com/containernetworking/plugins
kube-router now implements .spec.healthCheckNodePort which has long been a part of the Kubernetes service specification, but kube-router hasn't implemented it until now. This port is meant to give visibility about whether or not an endpoint for a service exists on a node to workloads outside the Kubernetes cluster. If a service endpoint is on the node, then this port returns a 200 HTTP response, otherwise it returns a 503 HTTP response if no endpoint exists on the node. By choosing a node that contains an endpoint, a client can ensure source IP preservation.
Hairpinning in kube-router is now fixed. kube-router now has a dedicated controller to ensuring that the veth interface inside the container is in promiscuous mode. This used to be handled by kubelet and then it was handled by docker-shim and then it was removed entirely. As far as I can tell, it was never ported to containerd or cri-o. Without this functionality, return traffic ends up getting black-holed before it is routed outside of the container. For more details see: https://github.com/cloudnativelabs/kube-router/commit/0f3714b9b758f24de0b1911c148bdba8d87de9b6
Users can now specify --metrics-addr to choose the IP address that kube-router listens on when providing Prometheus metrics
Users can now specify custom protocol timeouts for IPVS services exposed by kube-router
--service-tcp-timeout - (default: 0s preserves system value, typically 900 seconds)
--service-tcpfin-timeout - (default: 0s preserves system value, typically 120 seconds)
--service-udp-timeout - (default: 0s preserves system value, typically 300 seconds)
kube-router now abides by the service label service.kubernetes.io/service-proxy-name. Setting this label to something other than kube-router will result in kube-router ignoring the service
kube-router now honors spec.internalTrafficPolicy and implements spec.externalTrafficPolicy correctly. For more information see: https://kubernetes.io/docs/reference/networking/virtual-ips/#traffic-policies for more information

Backward Compatibility Breaks

kube-router now serves traffic to endpoints that that have a status of either ready or serving. See: https://kubernetes.io/docs/concepts/services-networking/endpoint-slices/#conditions for more information on the differences between these conditions. Before this release, kube-router only served traffic to ready endpoints, which improperly ignored pods that were in Terminating status. This change makes kube-router compliant with the upstream standards.

Other Notes

If you're running kube-router in a container and use hairpinning, make sure that hostPID: true is set on your kube-router pod. This is the only way that kube-router is able to properly set the veth interface into promiscuous mode. This attribute has now been added to all of the official kube-router DeaemonSet YAML files

Major Dependency Updates

This release also updates:

Alpine base container from 3.18 to 3.19
Go from v1.20 to v1.21
Kubernetes from v1.28 to v1.29

Contributions

Thanks to:

@nextsux - For improving overlay tunnel device binding
@jpiper - For adding the ability to configure an IP to bind metrics to
@mrueg - For always keeping our dependencies up to date
@AlexisDucastel - For testing the CNI plugin installation functionality
@hakman - For helping us improve the upstream e2e tests and providing kOps support and feedback

Changelog (from v2.0.1 - v2.1.0)

a7d7ac5d - feat(alpine): update 3.18 -> 3.19 <Aaron U'Ren>
5b56e6c6 - feat(goreleaser): update v1.21.2 -> v1.24.0 <Aaron U'Ren>
a6488004 - feat(moq): update v0.3.2 -> v0.3.4 <Aaron U'Ren>
46a1b179 - feat(go): upgrade 1.20.13 -> 1.21.7 + dep update <Aaron U'Ren>
c3562897 - feat(lint): update markdownlint v0.37.0 -> 0.39.0 <Aaron U'Ren>
47fe189f - feat(lint): update golangci-lint and fix lint errors <Aaron U'Ren>
9a136c1b - feat(NSC): implement NodePort Health Check <Aaron U'Ren>
7aec8d04 - doc(NSC): add comment for hairpin controller <Aaron U'Ren>
959022fd - feat(NSC): add endpoint statuses to internal struct <Aaron U'Ren>
16daa08c - feat(NSC): add endpoints that are ready or serving <Aaron U'Ren>
efddb2ea - Bump dependencies <Manuel Rüger>
3a1efa0e - dual_stack_issue_report.md: remove issue report template <Aaron U'Ren>
dda7ce61 - doc(cni): CNI requirements and install process <Aaron U'Ren>
a57a2261 - feat(cni): add binaries to host when missing <Aaron U'Ren>
98eea791 - doc(user-guide.md): add info about traffic policies <Aaron U'Ren>
fcd21b47 - feat: fully support service traffic policies <Aaron U'Ren>
84042603 - feat: increase unit test coverage <Aaron U'Ren>
24505f03 - fact(service_endpoints_sync.go): standardize error handling <Aaron U'Ren>
d3cf4d13 - feat(NSC): add / clarify log messages <Aaron U'Ren>
d757f49d - feat(NSC): honor headless label <Aaron U'Ren>
8afdee87 - fact(NSC): differentiate headless services <Aaron U'Ren>
30d37695 - fact(NSC): update Errorf syntax <Aaron U'Ren>
a0fe844a - feat(NSC): honor service-proxy-name label <Aaron U'Ren>
ced5102d - feat(NSC): add IPVS service timeouts <Aaron U'Ren>
47290a7e - build(deps): bump k8s.io/client-go from 0.28.3 to 0.28.4 <dependabot[bot]>
27f2283b - build(deps): bump k8s.io/cri-api from 0.28.3 to 0.28.4 <dependabot[bot]>
a6de8cb9 - build(deps): bump github.com/aws/aws-sdk-go from 1.48.10 to 1.48.16 <dependabot[bot]>
eb462bae - feat(linux_networking.go): add more error info <Aaron U'Ren>
eb262887 - fix(manifests): add hostPID for hairpin <Aaron U'Ren>
aebaa48e - fix(NSC): handle endpoint slice ready nil <Aaron U'Ren>
0f3714b9 - fix(hairpin): set hairpin_mode for veth iface <Aaron U'Ren>
3ea293d5 - build(deps): bump github.com/aws/aws-sdk-go from 1.47.9 to 1.48.10 <dependabot[bot]>
4b863e90 - build(deps): bump k8s.io/api from 0.28.3 to 0.28.4 <dependabot[bot]>
f575d318 - build(deps): bump github.com/osrg/gobgp/v3 from 3.20.0 to 3.21.0 <dependabot[bot]>
724588c6 - build(deps): bump golang.org/x/net from 0.18.0 to 0.19.0 <dependabot[bot]>
448f8b6f - build(deps): bump github.com/aws/aws-sdk-go from 1.45.24 to 1.47.9 <dependabot[bot]>
0209ac56 - build(deps): bump google.golang.org/grpc from 1.58.3 to 1.59.0 <dependabot[bot]>
b00a997e - build(deps): bump github.com/onsi/gomega from 1.29.0 to 1.30.0 <dependabot[bot]>
595d3e46 - build(deps): bump golang.org/x/net from 0.17.0 to 0.18.0 <dependabot[bot]>
fcf0ad91 - prometheus metrics: add option to specify listen address <Jason Piper>
961c63b0 - build(deps): bump github.com/onsi/gomega from 1.27.10 to 1.29.0 <dependabot[bot]>
b13cedc5 - build(deps): bump k8s.io/client-go from 0.28.2 to 0.28.3 <dependabot[bot]>
49314ab6 - build(deps): bump k8s.io/cri-api from 0.28.2 to 0.28.3 <dependabot[bot]>
e0861fc9 - build(deps): bump github.com/osrg/gobgp/v3 from 3.19.0 to 3.20.0 <dependabot[bot]>
44d81088 - build(deps): bump github.com/docker/docker <dependabot[bot]>
66890d5f - feat: Disable binding overlay tunnels to specific device <Martin -nexus- Mlynář>
f41defce - build(deps): bump google.golang.org/grpc from 1.58.2 to 1.58.3 <dependabot[bot]>
9d6b6474 - doc(user-guide): remove bootkube, add k3s <Aaron U'Ren>
e8be179e - doc(RELEASE.md): add more information <Aaron U'Ren>
a00778e7 - feat(.goreleaser.yml): add prerelease and header to release <Aaron U'Ren>
eb30de91 - lint: add markdownlint to linting <Aaron U'Ren>
dad4991d - doc: update and format all documentation <Aaron U'Ren>
64426896 - doc(user-guide.md): update formatting <Aaron U'Ren>
dbd741a5 - doc(introduction.md): update and format <Aaron U'Ren>
43791045 - doc(generic.md): update and format <Aaron U'Ren>
1957907b - doc(kubeadm.md): update and format <Aaron U'Ren>