Kubernetes - v1.22.0-beta.1

Changelog since v1.22.0-beta.0

Urgent Upgrade Notes

(No, really, you MUST read this before you upgrade)

• The flag --experimental-patches is now deprecated and will be removed in a future release. You can migrate to using the new flag --patches. Add a new field {Init|Join}Configuration.patches.directory that can be used for the same purpose. For "init" and "join" it is now recommended that you migrate to configure patches via {Init|Join}Configuration.patches.directory. For the time being, these flags can be mixed with --config, but that might change in the future. On a command line, the last *patches flag takes precedence over previous flags and the value in config. "kubeadm upgrade" --patches will continue to be the only available option, since "upgrade" does not support a configuration file yet. (#103063, @neolit123) [SIG Cluster Lifecycle]

Changes by Kind

Deprecation

• The feature Dynamic Kubelet Configuration is deprecated and kubelet will report warning when the flag --dynamic-config-dir is used. Feature gate DynamicKubeletConfig is disabled out of the box and needs to be explicitly enabled. (#102966, @SergeyKanzhelev) [SIG Cloud Provider, Instrumentation and Node]
• The in-tree azure and gcp auth plugins have been deprecated. The https://github.com/Azure/kubelogin and gcloud commands serve as out-of-tree replacements via the kubectl/client-go credential plugin mechanism. (#102181, @enj) [SIG API Machinery and Auth]

API Change

• A new score extension for NodeResourcesFit plugin that merges the functionality of NodeResourcesLeastAllocated,NodeResourcesMostAllocated,RequestedToCapacityRatio plugins, which are marked as deprecated as of v1beta2. In v1beta1, the three plugins can still be used in v1beta1 but not at the same time with the score extension of NodeResourcesFit
  ```

#### Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

(#101822, @yuzhiquan) [SIG API Machinery, Apps, Auth, Cloud Provider, Instrumentation, Node, Scheduling, Storage and Testing]
- Add DataSourceRef alpha field to PVC spec, which allows contents other than PVCs and VolumeSnapshots to be data sources. (#103276, @bswartz) [SIG API Machinery, Apps and Storage]
- Add PersistentVolumeClaimDeletePoilcy to StatefulSet API. (#99378, @mattcary) [SIG API Machinery and Apps]
- Add distributed tracing to the kube-apiserver. It is can be enabled with the feature gate: APIServerTracing=true (#94942, @dashpole) [SIG API Machinery, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Node, Storage and Testing]
- Added new kubelet alpha feature SeccompDefault. This feature enables falling back to
the RuntimeDefault (former runtime/default) seccomp profile if nothing else is specified
in the pod/container SecurityContext or the pod annotation level. To use the feature, enable
the feature gate as well as set the kubelet configuration option SeccompDefault
(--seccomp-default) to true. (#101943, @saschagrunert) [SIG Node]
- Adds the ReadWriteOncePod access mode for PersistentVolumes and PersistentVolumeClaims. Restricts volume access to a single pod on a single node. (#102028, @chrishenzie) [SIG Apps, CLI, Node, Scheduling and Storage]
- Alpha swap support can now be enabled on Kubernetes nodes with the NodeSwapEnabled feature flag. See for details. (#102823, @ehashman) [SIG Node]
- CSIServiceAccountToken is GA. (#103001, @zshihang) [SIG Auth and Storage]
- Enforce the ReadWriteOncePod PVC access mode during scheduling (#103082, @chrishenzie) [SIG Apps, CLI, Node, Scheduling and Storage]
- Improved parsing of label selectors (#102188, @alculquicondor) [SIG API Machinery]
- Kube API server accepts Impersonate-Uid header to impersonate a user with a specific UID, in the same way that you can currently use Impersonate-User, Impersonate-Group and Impersonate-Extra (#99961, @margocrawf) [SIG API Machinery, Auth and Testing]
- Kube-scheduler: a plugin enabled in a v1beta2 configuration file takes precedence over the default configuration for that plugin; this simplifies enabling default plugins with custom configuration without needing to explicitly disable those default plugins. (#99582, @chendave) [SIG Scheduling]
- Scheduler could be configured to consider new resources beside CPU and memory, GPU for example, for the score plugin of NodeResourcesBalancedAllocation. (#101946, @chendave) [SIG Scheduling]
- Suspend Job feature graduated to beta
Added the "action" label to Job controller sync metrics job_sync_total and job_sync_duration_seconds (#102022, @adtac) [SIG Apps, Instrumentation and Testing]
- TerminationGracePeriodSeconds on pod specs and container probes should not be negative.
Negative values of TerminationGracePeriodSeconds will be treated as the value 1s on the delete path.
Immutable field validation will be relaxed in order to update negative values.
In a future release, negative values will not be permitted. (#98866, @wzshiming) [SIG API Machinery, Apps and Node]
- The API documentation for the DaemonSet's spec.updateStrategy.rollingUpdate.maxUnavailable field was corrected to state that the value is rounded up. (#101296, @Miciah) [SIG Apps and CLI]
- The CertificateSigningRequest.certificates.k8s.io API supports an optional expirationSeconds field to allow the client to request a particular duration for the issued certificate. The default signer implementations provided by the Kubernetes controller manager will honor this field as long as it does not exceed the --cluster-signing-duration flag. (#99494, @enj) [SIG API Machinery, Apps, Auth, CLI, Instrumentation, Node, Security and Testing]
- The ServiceAppProtocol feature gate has been removed. It reached GA in Kubernetes 1.20. (#103190, @robscott) [SIG Network]

Feature

• (alpha feature) If the CSI driver supports the NodeServiceCapability VOLUME_MOUNT_GROUP and the DelegateFSGroupToCSIDriver feature gate is enabled, kubelet will delegate applying FSGroup to the driver by passing it to NodeStageVolume and NodePublishVolume, regardless of what other FSGroup policies are set. (#103244, @verult) [SIG Apps and Storage]
• API Server tracing can now trace re-entrant api requests. (#103218, @dashpole) [SIG API Machinery, CLI, Cloud Provider, Cluster Lifecycle and Instrumentation]
• Add unified map on CRI to support cgroup v2. Refer to https://github.com/opencontainers/runtime-spec/blob/master/config-linux.md#unified. (#102578, @payall4u) [SIG Node]
• Added field-selector option for kubectl top pod (#102155, @lauchokyip) [SIG CLI]
• Added new metrics about API Priority and Fairness. Each one has a label priority_level. The last two also have a label bound taking values min and `max.
• apiserver_flowcontrol_current_r: R(the time of the last change in state of the queues)
• apiserver_flowcontrol_dispatch_r: R(the time of the latest request dispatch)
• apiserver_flowcontrol_latest_s: S(the request last dispatched) = R(when that request starts executing in the virtual world)
• apiserver_flowcontrol_next_s_bounds: min and max next S among non-empty queues
• apiserver_flowcontrol_next_discounted_s_bounds: min and max next S - (sum [over requests executing] width * estimatedDuration) among non-empty queues (#102859, @MikeSpreitzer) [SIG API Machinery and Instrumentation]
• Adding --restart-kubelet flag on E2E Node test suite (#97028, @knabben) [SIG Node and Testing]
• Adds feature gate KubeletInUserNamespace which enables support for running kubelet in a user namespace.

The user namespace has to be created before running kubelet.
All the node components such as CRI need to be running in the same user namespace.

When the feature gate is enabled, kubelet ignores errors that happens during setting the following sysctl values: vm.overcommit_memory, vm.panic_on_oom, kernel.panic, kernel.panic_on_oops, kernel.keys.root_maxkeys, kernel.keys.root_maxbytes. (These sysctl values for the host, not for the containers)

kubelet also ignores an error during opening /dev/kmsg.
This feature gate also allows kube-proxy to ignore an error during setting RLIMIT_NOFILE.

This feature gate is especially useful for running Kubernetes inside Rootless Docker/Podman with kind or minikube. (#92863, @AkihiroSuda) [SIG Network, Node and Testing]
- Adds metrics for the delegated authenticator used by extension APIs that delegate authentication logic to the Kube API server. (#99364, @p0lyn0mial) [SIG API Machinery, Auth, Instrumentation and Node]
- Adds metrics for the delegated authorizer used by extension APIs that delegate authorization logic to the Kube API server. (#100339, @p0lyn0mial) [SIG API Machinery, Auth, Instrumentation and Node]
- Client-go credential plugins are now GA (#102890, @ankeesler) [SIG API Machinery, Auth and Testing]
- Deprecated warning message for igonre-errors flag. (#102677, @yuzhiquan) [SIG CLI]
- Endpoints that have more than 1000 endpoints will be truncated and the endpoints.kubernetes.io/over-capacity annotation on the Endpoints resource will be set to truncated. (#103520, @swetharepakula) [SIG Apps and Network]
- Expose /debug/flags/v to allow dynamically setting log level for kube-proxy. (#98306, @borgerli) [SIG Network]
- Expose container start time as container_start_time_seconds in kubelet /metrics/resource endpoint (#102444, @sanwishe) [SIG Instrumentation, Node and Testing]
- Extended resources defined in LeastAllocated, MostAllocated and RequestedToCapacityRatio plugin argument are bypassed by the scheduler if the incoming Pod doesn't request them in the pod spec. (#103169, @Huang-Wei) [SIG Scheduling]
- Features gates EndpointSliceProxying & WindowsEndpointSliceProxying are now GA and cannot be disabled. Kube-proxy will use EndpointSlices for endpoint information. (#103451, @swetharepakula) [SIG Network]
- IngressClassNamespacedParams feature gate has graduated to beta and is enabled by default. This means IngressClass resource will now have two new fields - spec.paramters.namespace and spec.parameters.scope. (#101711, @hbagdi) [SIG Network]
- JSON logging format is no longer available by default in non-core Kubernetes Components and require owners to opt in. (#102869, @mengjiao-liu) [SIG API Machinery, Cluster Lifecycle and Instrumentation]
- Kube-apiserver: the alpha PodSecurity feature can be enabled by passing --feature-gates=PodSecurity=true, and enables controlling allowed pods using namespace labels. See https://git.k8s.io/enhancements/keps/sig-auth/2579-psp-replacement for more details. (#103099, @liggitt) [SIG API Machinery, Auth, Instrumentation, Release, Security and Testing]
- Kube-proxy uses V1 EndpointSlices. (#103306, @swetharepakula) [SIG Network]
- LogarithmicScaleDown is now Beta and enabled by default (#101767, @damemi) [SIG Apps and Instrumentation]
- Make kubectl command headers default for beta. (#103238, @seans3) [SIG CLI]
- Mark net.ipv4.ip_unprivileged_port_start as safe sysctl (#103326, @pacoxu) [SIG Auth, Network, Node and Security]
- Move the memory manager feature to the beta that will make it enabled by default with the None policy. (#101947, @cynepco3hahue) [SIG Node and Testing]
- Promote the ServiceInternalTrafficPolicy feature to beta, which enables the internalTrafficPolicy field of Service by default. (#103462, @andrewsykim) [SIG Apps and Network]
- Promote the feature ServiceLBNodePortControl to Beta (#100412, @hanlins) [SIG API Machinery, Apps, Architecture, Network and Testing]
- Scheduler nows provides an option for plugin developers to move Pods to activeQ proactively. (#103383, @Huang-Wei) [SIG Scheduling and Testing]
- Services with externalTrafficPolicy: Local now support graceful termination when using
the iptables or ipvs mode of kube-proxy with EndpointSlices enabled. Specifically, if a
connection for such a service arrives on a node when there are no "Ready" endpoints
for the service, but there is at least one Terminating pod for that service on the node,
then kube-proxy will send the traffic to the Terminating pod rather than dropping it. This
patches up a race condition between when a pod is killed and when the external load
balancer notices that it has been killed. (#97238, @andrewsykim) [SIG Network, Scalability and Windows]
- Shell completion has been migrated to Cobra's go solution. kubectl is now smarter about disabling file completion when it does not apply. Furthermore, completion for the cp command does not show all files unless the user has started typing something. (#96087, @marckhouzam) [SIG CLI]
- The HugePageStorageMediumSize feature has been promoted to GA, allowing unconditional usage of multiple sizes huge page resources on a container level. (#99144, @bart0sh) [SIG Apps, Node and Storage]
- Update the system-validators library to v1.5.0. Includes validation for seccomp and fixes a stdout/stderr problem in the Docker validator. (#103390, @ironyman) [SIG Cluster Lifecycle]
- Updates the following images to pick up CVE fixes:
- debian to v1.8.0
- debian-iptables to v1.6.5
- setcap to v2.0.3 (#103235, @thejoycekung) [SIG API Machinery, Release and Testing]
- kubectl debug is able to create ephemeral containers in pre-1.22 clusters with the EphemeralContainers feature enabled. Note that versions of kubectl prior to 1.22 are unable to create ephemeral containers in clusters version 1.22 and greater due to an API change. (#103292, @verb) [SIG CLI]

Documentation

• kubectl debug will now print a warning message when using the --target option since many container runtimes do not support this yet. (#101074, @verb) [SIG CLI]

Bug or Regression

• Cut strings at 100 chars or first new line in kubectl get (#103514, @soltysh) [SIG CLI]
• Disruption controller shouldn't error while syncing for unmanaged pods. (#103414, @ravisantoshgudimetla) [SIG Apps and Testing]
• Fix kubectl alpha debug node does not work on tainted(NoExecute) nodes and tolerate everything. (#98431, @wawa0210) [SIG CLI]
• Fix performance regression for update and apply operations on large CRDs. (#103318, @jpbetz) [SIG API Machinery, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation and Storage]
• Fixed SELinux relabeling of CSI volumes after CSI driver failure. (#103154, @jsafrane) [SIG Node and Storage]
• Kubeadm: fix a bug where "kubeadm join" for control plane nodes would download certificates and keys from the cluster, but would not write publicly readable certificates and public keys with mode 0644 and instead use mode 0600. (#103313, @neolit123) [SIG Cluster Lifecycle]
• Kubelet: the returned value for PodIPs is the same in the Downward API and in the pod.status.PodIPs field (#103307, @aojea) [SIG Network and Node]
• Metrics Server will use Addon Manager 1.8.3 (#103541, @jbartosik) [SIG Cloud Provider and Instrumentation]
• Pods that are known to the kubelet to have previously been Running should not revert to Pending state; the kubelet will now infer a termination. (#102821, @ehashman) [SIG Node]
• Switch scheduler to generate the merge patch on pod status instead of the full pod (#103133, @marwanad) [SIG Scheduling]
• The current behavior for Services that IPFamilyPolicy set as PreferDualstack. The current behavior when the cluster is upgraded to dual-stack is:
• Services that have been set to IPFamilyPolicy = PreferDualstack will be upgraded when the service object is updated. e.g., when a user change a label.

This behavior will change to:
- Services that have been set IPFamilyPolicy = PreferDualstack will not be upgraded when the service object is updated. User can still change policy, type etc and existing behaviors remain the same. (#102898, @khenidak) [SIG Network and Testing]
- Treat VSphere "File (vmdk path here) was not found" errors as success during volume deletion (#92372, @breunigs) [SIG Cloud Provider and Storage]
- Upgrades functionality of kubectl kustomize as described at
https://github.com/kubernetes-sigs/kustomize/releases/tag/kustomize%2Fv4.2.0 (#103419, @natasha41575) [SIG CLI]

Other (Cleanup or Flake)

• Featuregate ServiceLoadBalancerClass will be promoted to beta, this enables the Service loadBalancerClass field by default. (#103129, @XudongLiuHarold) [SIG Testing]
• Increased CSINodeIDMaxLength from 128 bytes to 192 bytes. Prepare to increase the length limit to 256 bytes in 1.23 release. (#101256, @Jiawei0227) [SIG Storage]
• Kubeadm: move the BootstrapToken* API and related utilities from v1beta3 to a separate API group/version - bootstraptoken/v1. (#102964, @neolit123) [SIG Cluster Lifecycle]
• Remove balanced attached node volumes feature (#102443, @ravisantoshgudimetla) [SIG Scheduling]
• Replace go-bindata with //go:embed (#99829, @palnabarun) [SIG Architecture, CLI, Node and Testing]
• The deprecated RunAsGroup feature-gate has been removed, since the RunAsGroup feature graduated to GA in 1.21 and was unconditionally enabled. (#101581, @carlory) [SIG API Machinery and Node]
• Update golang.org/x/net to v0.0.0-20210520170846-37e1c6afe023 (#103176, @CaoDonghui123) [SIG API Machinery, Auth, CLI, Cloud Provider, Cluster Lifecycle, Node and Storage]
• Updated command descriptions and examples for grammar and punctuation consistency. (#103524, @bergerhoffer) [SIG Auth and CLI]

Dependencies

Added

• github.com/felixge/httpsnoop: v1.0.1
• go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp: v0.20.0

Changed

• github.com/auth0/go-jwt-middleware: 5493cab → v1.0.1
• github.com/gopherjs/gopherjs: 0766667 → fce0ec3
• github.com/heketi/heketi: v10.2.0+incompatible → v10.3.0+incompatible
• github.com/mattn/go-runewidth: v0.0.9 → v0.0.7
• github.com/olekukonko/tablewriter: v0.0.5 → v0.0.4
• github.com/smartystreets/assertions: b2de0cb → v1.1.0
• golang.org/x/net: a5a99cb → 37e1c6a
• k8s.io/system-validators: v1.4.0 → v1.5.0
• k8s.io/utils: da69540 → 4b05e18
• sigs.k8s.io/apiserver-network-proxy/konnectivity-client: v0.0.20 → v0.0.21
• sigs.k8s.io/kustomize/api: v0.8.10 → v0.8.11
• sigs.k8s.io/kustomize/cmd/config: v0.9.12 → v0.9.13
• sigs.k8s.io/kustomize/kustomize/v4: v4.1.3 → v4.2.0
• sigs.k8s.io/kustomize/kyaml: v0.10.20 → v0.11.0
• sigs.k8s.io/structured-merge-diff/v4: v4.1.1 → v4.1.2

Removed

• github.com/etcd-io/gofail: ad7f989
• github.com/go-bindata/go-bindata: v3.1.1+incompatible
• go.etcd.io/etcd/etcdutl/v3: v3.5.0
• go.etcd.io/etcd/tests/v3: v3.5.0