Kubernetes - v1.28.0-beta.0

Changelog since v1.28.0-alpha.4

Changes by Kind

Deprecation

• Changed kubectl version default output to be identical to what kubectl version --short printed,
  and remove --short flag entirely. (#116720, @soltysh) [SIG CLI and Testing]
• Deprecated support for CSI migration of Ceph RBD volumes.

Users who were relying on Kubernetes' ability to migrate to an out-of-tree storage driver should complete
that migration before the support for it is removed. (#118303, @carlory) [SIG Storage]
- KMSv1 is deprecated and will only receive security updates going forward. Use KMSv2 instead. Set --feature-gates=KMSv1=true to use the deprecated KMSv1 feature. (#119007, @aramase) [SIG API Machinery and Auth]
- The deprecated flag --lock-object-namespace and --lock-object-name have been removed from kube-scheduler. Please use --leader-elect-resource-namespace and --leader-elect-resource-name or ComponentConfig instead to configure those parameters. (#119130, @SataQiu) [SIG Scheduling]

API Change

• A CDIDevice field is includes in the Device Plugin's ContainerAllocateResponse. This field maps to the CDIDevice field in the CRI protocol. (#118254, @elezar) [SIG Node and Testing]
• Add IP mode field to loadbalancer status ingress (#118895, @RyanAoh) [SIG API Machinery, Apps, Cloud Provider, Network and Testing]
• Add new annotation batch.kubernetes.io/cronjob-scheduled-timestamp to Job objects scheduled from CronJobs. (#118137, @helayoty) [SIG Apps]
• Add podReplacementPolicy and terminating field to job api (#119301, @kannon92) [SIG API Machinery and Apps]
• Added fields reason and fieldPath into CRD validation rules to allow users to specify reason and field path when validation failed. (#118041, @cici37) [SIG API Machinery]
• Added namespace access support to the CEL expressions of ValidatingAdmissionPolicy via a namespaceObject
  variable with expressions. (#118267, @cici37) [SIG API Machinery and Testing]
• Adds new CRDValidationRatcheting alpha feature. During a PATCH or UPDATE Validation Ratcheting discards errors thrown by unchanged portions of the resource from most OpenAPI schema validations. (#118990, @alexzielenski) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node and Storage]
• Adds new namespaceParamRef to admissionregistration.k8s.io/v1alpha1.ValidatingAdmissionPolicy (#119215, @alexzielenski) [SIG API Machinery and Testing]
• Extend the Job API for alpha version of BackoffLimitPerIndex (#119294, @mimowo) [SIG API Machinery and Apps]
• Graduate AdmissionWebhookMatchCondition feature to beta (#119380, @a-hilaly) [SIG API Machinery]
• In the API Priority and Fairness feature, priority levels that are exempt from limitation can now be given a nominal and a lendable concurrency and their dispatching borrows from the concurrency limits of the other priority levels. For details see https://github.com/kubernetes/enhancements/tree/master/keps/sig-api-machinery/1040-priority-and-fairness#dispatching . (#118782, @MikeSpreitzer) [SIG API Machinery]
• Indexed Job pods now have the pod completion index set as a pod label. (#118883, @danielvegamyhre) [SIG Apps]
• Kube-proxy: add '--logging-format' flag to support structured logging (#117800, @cyclinder) [SIG API Machinery, Architecture, Instrumentation and Network]
• Registered_metric_total, disabled_metric_total, hidden_metric_total & kubernetes_feature_enabled are promoted to BETA stability. (#119264, @logicalhan) [SIG API Machinery, Architecture, Cluster Lifecycle and Instrumentation]
• Removed resizeStatus enum from pvc.Status and replaced with AllocatedResourceStatus (#116335, @gnufied) [SIG API Machinery, Apps, Auth, Node, Storage and Testing]
• StatefulSet pods now have the pod index set as a pod label statefulset.kubernetes.io/pod-index. (#119232, @danielvegamyhre) [SIG Apps]
• Support BackoffLimitPerIndex in Jobs (#118009, @mimowo) [SIG API Machinery, Apps and Testing]
• Support for proxying a request to a peer kube-apiserver if the local apiserver is not able to serve it due to version skew or in the case the requested api is disabled on the local apiserver (#117740, @Richabanker) [SIG API Machinery, Apps, Auth, Cloud Provider, Network, Node and Testing]
• The IPTablesOwnershipCleanup feature (KEP-3178) is now GA; kubelet no longer
  creates the KUBE-MARK-DROP chain (which has been unused for several releases)
  or the KUBE-MARK-MASQ chain (which is now only created by kube-proxy). (#119374, @danwinship) [SIG API Machinery, Network and Node]
• The names of ResourceClaims generated from ResourceClaimTemplate are now generated. The base name is still <pod>-<claim name>, but a random suffix will avoid name collisions. (#117351, @pohly) [SIG API Machinery, Apps, Auth, Node, Scheduling and Testing]
• The new feature gate "SidecarContainers" is now available. This feature introduces sidecar containers, a new type of init container that starts before other containers but remains running for the full duration of the pod's lifecycle and will not block pod termination. (#116429, @gjkim42) [SIG API Machinery, Apps, Node, Scheduling and Testing]

Feature

• A ValidatingAdmissionPolicy now has its messageExpression field checked against resolved types. (#119209, @jiahuif) [SIG API Machinery]
• Add ConsistentListFromCache feature gate that allows apiserver to serve consistent lists from cache (#118508, @serathius) [SIG API Machinery, Instrumentation and Testing]
• Add full cgroup v2 swap support for both Limited and Unlimited swap.

When LimitedSwap is enabled the swap limit would be automatically calculated for
Burstable QoS pods. For Best-Effort / Guaranteed QoS pods, swap would be disabled.

Containers with memory requests equal to their memory limits also won't have
swap access, and it is a way to opt-out of swap for a single container.

The formula for the swap limit for Burstable QoS pods is:
(<memory-request>/<node-memory-capacity>)*<node-swap-capacity>.

Support for cgroup v1 is removed. (#118764, @iholder101) [SIG Node and Testing]
- Add handling for pods in podgc for PodReplacementPolicy or PodDisruption (#118772, @kannon92) [SIG Apps and Testing]
- Add reason to metric attachdetach_controller_forced_detaches in the attach detach controller. (#119185, @xing-yang) [SIG Apps and Storage]
- Add swap to stats to Summary API and Prometheus endpoints (stats/summary and /metrics/resource). (#118865, @iholder101) [SIG Node and Testing]
- Added a new command line argument --interactive to kubectl. The new command line argument lets a user confirm deletion requests per resource interactively. (#114530, @ardaguclu) [SIG CLI and Testing]
- Added a new feature gate, SchedulerQueueingHints (enabled by default).
The new feature gate activates a framework for fine-grained filtering of events related to scheduler plugins.
In this release, no default scheduling plugins make use of the hinting framework, so you should not expect any behavior changes. (#119328, @sanposhiho) [SIG Scheduling]
- Adds apiserver_admission_match_condition_evaluation_seconds and apiserver_admission_match_condition_exclusions_total metrics (#119311, @ivelichkovich) [SIG API Machinery]
- Bump distroless-iptables to 0.2.6 based on Go 1.20.6 (#119365, @xmudrii) [SIG Testing]
- CEL authorizer checks no longer raise runtime errors. Calls to "check" will always return a decision object and the authorization error (if any) can be accessed within expressions using the new decision methods "errored" and "error". (#118804, @benluddy) [SIG API Machinery]
- CRI: expose commit memory bytes in container stats specific to Windows (#119238, @kiashok) [SIG Node and Windows]
- Cloud controller manager's node controller now emits timing metrics for initial Node synchronization. These metrics measure the delay between the creation of a new Node and the node controller's initial management actions, such as removing the cloud provider taint. These metrics should be consulted when setting cloud controller manager's --concurrent-node-syncs flag. (#119241, @cartermckinnon) [SIG Cloud Provider and Instrumentation]
- Faster scheduling when ResourceClaims are involved (#119078, @pohly) [SIG Node and Scheduling]
- Graduate the ProbeTerminationGracePeriod feature gate to GA (#114307, @rphillips) [SIG Apps and Node]
- Hashing of KeyID in Logs

This release adds a feature to hash the KeyID values in the logs. The KeyID values are sensitive information that should not be exposed in plain text in the logs. By hashing the KeyID values, we can protect the confidentiality of the data while still being able to log the necessary information. (#118988, @nilekhc) [SIG API Machinery, Auth and Testing]
- Implement alpha support for a drop-in kubelet configuration directory (#119390, @sohankunkerkar) [SIG Node]
- In the course of admitting a single request, the ValidatingAdmissionPolicy plugin will perform no more than one authorization check per unique authorizer expression. All evaluations of identical authorizer expressions will produce the same decision. (#116443, @benluddy) [SIG API Machinery and Testing]
- Kube-controller-manager: the dynamic resource controller steps in when a pod got created such that the scheduler ignores it (i.e. spec.nodeName is set) and then takes care of triggering delayed resource claim allocation and/or reserving a claim for the pod. (#118209, @pohly) [SIG API Machinery, Apps, Auth, Node and Testing]
- Kube-proxy service health returns http header "X-Load-Balancing-Endpoint-Weight" with number of local endpoints. The same information is still available in response body JSON payload.LocalEndpoints. (#118999, @cezarygerard) [SIG Network]
- Kubelet: plugins for dynamic resource allocation may use the v1alpha3 API instead of v1alpha2 if they want to do prepare/unprepare operations in batches. (#119012, @pohly) [SIG Node and Testing]
- Kubelet: security of dynamic resource allocation was enhanced by limiting node access to those objects that are needed on the node. (#116254, @pohly) [SIG Auth and Testing]
- Kubernetes is now built with Go 1.20.6 (#119324, @xmudrii) [SIG API Machinery, Auth, Cloud Provider, Release and Testing]
- Migrate pkg/controller/endpoint to contextual logging (#116755, @my-git9) [SIG Apps, Instrumentation and Network]
- Migrated the EndpointSlice and EndpointSliceMirroring controllers (within kube-controller-manager) to use contextual logging. (#115295, @Namanl2001) [SIG API Machinery, Apps, Network and Testing]
- Move non-graceful node shutdown to GA. (#118228, @carlory) [SIG Apps, Storage and Testing]
- New CEL Library functions to support Kubernetes Quantities. (#118803, @alexzielenski) [SIG API Machinery]
- New Metrics Added for Encryption Configuration Controller

This release adds new metrics to the Encryption Configuration Controller to help monitor the automatic reloading of encryption configuration. The new metrics include:

• apiserver_encryption_config_controller_automatic_reload_failures_total: Total number of failed automatic reloads of encryption configuration.
• apiserver_encryption_config_controller_automatic_reload_success_total: Total number of successful automatic reloads of encryption configuration.
• apiserver_encryption_config_controller_automatic_reload_last_timestamp_seconds: Timestamp of the last successful or failed automatic reload of encryption configuration.

These metrics can be used to monitor the health of the Encryption Configuration Controller and to troubleshoot any issues that may arise during automatic reloading of encryption configuration. (#119008, @nilekhc) [SIG API Machinery, Auth and Instrumentation]
- New staging repo has been created for the EndpointSlice reconciler. (#118953, @mskrocki) [SIG Apps, Network and Release]
- Promote the following apiserver flowcontrol metrics to Beta:

apiserver_flowcontrol_request_wait_duration_seconds
apiserver_flowcontrol_current_executing_seats
apiserver_flowcontrol_nominal_limit_seats
apiserver_flowcontrol_rejected_requests_total
apiserver_flowcontrol_dispatched_requests_total
apiserver_flowcontrol_current_inqueue_requests
apiserver_flowcontrol_current_executing_requests (#119110, @andrewsykim) [SIG API Machinery and Instrumentation]
- Replace apiserver_storage_db_total_size_in_bytes with apiserver_storage_size_bytes metric (#118812, @serathius) [SIG API Machinery, Instrumentation and Testing]
- The apiserver debug endpoint /debug/api_priority_and_fairness/dump_requests has been extended to dump executing requests as well as queued ones. A column for StartTime has been added to the returned table, with the queued requests having a StartTime of "0001-01-01T00:00:00Z". The executing requests have a RequestIndexInQueue of -1, and the QueueIndex is also -1 for priority levels without queues. (#119009, @MikeSpreitzer) [SIG API Machinery]
- The scheduler skips the PodTopologySpread Score plugin when nothing to do with the Pod.
It will affect some metrics values related to the PodTopologySpread Score plugin. (#118608, @utam0k) [SIG Scheduling]
- TopologyManagerPolicyOptions feature-flag is promoted to beta and enabled by default. (#118816, @PiotrProkop) [SIG Node]
- Update kube-apiserver's priority & fairness work estimator such that 'max seats' is MIN(0.15 x nominalCL, nomincalCL / handSize) (#118601, @andrewsykim) [SIG API Machinery]
- ValidatingAdmissionPolicy type checking now correctly handles authorizer variable. (#118540, @jiahuif) [SIG API Machinery]
- With the KubeletCgroupDriverFromCRI feature gate enabled and sufficiently new version of a container
runtime, kubelet automatically detects the cgroup driver config from the container runtime, eliminating
the need to specify the cgroupDriver configuration option (or --cgroup-driverflag) of kubelet. ([#118770](https://github.com/kubernetes/kubernetes/pull/118770), [@marquiz](https://github.com/marquiz)) [SIG Node] - [Kube-proxy]: implement connection draining for terminating nodes, KEP-3836 ([#116470](https://github.com/kubernetes/kubernetes/pull/116470), [@alexanderConstantinescu](https://github.com/alexanderConstantinescu)) [SIG Network] -force_delete_pods_total andforce_delete_pod_errors_total ` metrics count all pod deletion behaviors. (#118480, @carlory) [SIG Apps]

Failing Test

• Switched back to debian-base instead of distroless for conformance image. (#119422, @saschagrunert) [SIG Architecture, Release and Testing]

Bug or Regression

• Add warning for dup ports update/patching in pod's container ports and service ports (#113245, @pacoxu) [SIG Network]
• Bump cadvisor version to v0.47.3 (#119225, @iholder101) [SIG Node and Testing]
• Dynamic Resource Allocation: log a error and submit an event when Kubelet fails to prepare dynamic resources (#118578, @bart0sh) [SIG Node]
• Fix computing backoff delay when using Job pod failure policy, by including in the backoff delay calculation pod failures ignored from the backoffLimit counter (#119434, @mimowo) [SIG Apps]
• Fix discoverability of apiregistration.k8s.io in openapi/v3 (#118879, @atiratree) [SIG API Machinery]
• Fixed a bug where kubectl port-forward, when used with a Deployment, could connect to a terminating pod even when a running pod is also available. (#119256, @brianpursley) [SIG CLI]
• Fixed kubelet startup getting stuck with NewVolumeManagerReconstruction feature enabled and a CSI volume present in /var/lib/kubelet/pods. (#117804, @jsafrane) [SIG Node and Storage]
• Kubeadm: the limitation that the 'ignorePreflightErrors' field can not be set to 'all' in kubeadm config file has been removed (#119351, @SataQiu) [SIG Cluster Lifecycle]
• Only declare Job as finished after removing all Pod finalizers to avoid orphan Pods (#119159, @alculquicondor) [SIG Apps and Testing]
• Reduces CPU and memory consumption of kube-apiserver if OpenAPI V2 is not accessed by any client. Also improves performance of the apiserver on installation of many CRDs. (#118212, @Jefftree) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node and Storage]
• The kube-proxy sync_proxy_rules_iptables_total metric has now reverted back
  to its pre-1.27 behavior of tracking the total number of iptables rules that
  kube-proxy is responsible for, rather than only counting the number of rules
  that it re-synced on the last sync. The new sync_proxy_rules_iptables_last
  metric now gives the latter number. (#119140, @danwinship) [SIG Network]
• The metric apiserver_flowcontrol_request_concurrency_limit has been deprecated and will be removed in a future release. It is a duplicate of apiserver_flowcontrol_nominal_limit_seats (introduced in release 1.26) but has an outdated name and had an outdated HELP string. (#118959, @MikeSpreitzer) [SIG API Machinery]
• [Dual-stack] Fix generateAPIPodStatus() of kubelet handling Secondary IP. hostIPs order may not be be consistent. If secondary IP is before primary one, current logic adds primary IP twice into PodIPs, which leads to error: "may specify no more than one IP for each IP family". (#116879, @lzhecheng) [SIG Node]

Other (Cleanup or Flake)

• Migrated the disruption controller (within kube-controller-manager) to use contextual logging. (#119147, @mengjiao-liu) [SIG API Machinery, Apps, Instrumentation and Testing]
• Migrated the podgc controller and some other remaining log calls within kube-controller-manager to use contextual logging. kube-controller-manager is now converted completely. (#119250, @pohly) [SIG API Machinery, Apps, Cloud Provider, Instrumentation, Network, Storage and Testing]
• Remove KUBECTL_EXPLAIN_OPENAPIV3 which is already redundant (#119286, @ardaguclu) [SIG CLI]
• Revised OpenAPI v2 fetching for CustomResourceDefinitions. CRDs are now aggregated lazily,
  which improves resource usage during installation of many CRDs. As a result, the first request
  to fetch the OpenAPI may be slower. (#118808, @Jefftree) [SIG API Machinery and Testing]
• Shrink the OpenAPI v2 spec by more than 50%, especially for less CPU resource consumption. (#118204, @sttts) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node and Storage]
• The GetAllocatableResources podresources API endpoint is now GA (#118973, @ffromani) [SIG Node and Testing]
• Updated debian-base image to bookworm-v1.0.0. (#119095, @saschagrunert) [SIG API Machinery, Architecture, Release and Testing]
• Updated setcap image to debian bookworm v1.0.0. (#119247, @saschagrunert) [SIG Release]

Dependencies

Added

• github.com/xhit/go-str2duration/v2: v2.1.0

Changed

• github.com/alecthomas/kingpin/v2: v2.3.1 → v2.3.2
• github.com/google/cadvisor: v0.47.2 → v0.47.3
• github.com/prometheus/client_model: v0.3.0 → v0.4.0
• github.com/prometheus/common: v0.42.0 → v0.44.0
• github.com/rogpeppe/go-internal: v1.6.1 → v1.10.0
• golang.org/x/crypto: v0.6.0 → v0.11.0
• golang.org/x/net: v0.9.0 → v0.12.0
• golang.org/x/oauth2: v0.6.0 → v0.8.0
• golang.org/x/sys: v0.8.0 → v0.10.0
• golang.org/x/term: v0.7.0 → v0.10.0
• golang.org/x/text: v0.9.0 → v0.11.0
• k8s.io/kube-openapi: 7562a10 → 2695361

Removed

• github.com/xhit/go-str2duration: v1.2.0