Kubernetes - v1.28.0-alpha.4

Changelog since v1.28.0-alpha.3

Urgent Upgrade Notes

(No, really, you MUST read this before you upgrade)

• Action required for the custom scheduler plugin developers.
  Here's the breaking change in EnqueueExtension in the scheduling framework.
  The EventsToRegister in EnqueueExtension changed the return value from ClusterEvent to ClusterEventWithHint. ClusterEventWithHint allows each plugin to filter out more useless events via the callback function named QueueingHintFn.
  When the scheduling queue receives a cluster event, before moving each Pod from unschedulable pod pool to activeQ/backoffQ, it will call QueueingHintFn of plugins that rejected each Pod in the previous scheduling cycle.
  Depending on the value returned from QueueingHintFn, the scheduling queue changes how it queues each Pod:
• if more than one QueueingHintFn returns QueueImmediately, it queues Pod to activeQ.
• If no QueueingHintFn returns QueueImmediately and more than one plugin returns QueueAfterBackoff, it queues Pod to backoffQ if Pod is backing off, or to activeQ if Pod's backoff has already finished.
• If all QueueingHintFn return QueueSkip, it puts this pod back to the unschedulable pod pool

Having appropriate QueueingHintFn contributes to reducing useless retries and thus improves the overall scheduler's performance.

How can I migrate?

For backward compatibility, nil QueueingHintFn is treated as always returning QueueAfterBackoff.
So, if you want to just keep the existing behavior, you can register ClusterEventWithHint with no QueueingHintFn in it.
But, registering appropriate QueueingHintFn is, of course, better from a scheduling performance perspective. (#118551, @sanposhiho) [SIG Node, Scheduling, Storage and Testing]
- RBD volume plugin ( kubernetes.io/rbd) has been deprecated in this release and will be removed in a subsequent release. Alternative is to use RBD CSI driver (https://github.com/ceph/ceph-csi/) in your Kubernetes Cluster. (#118552, @humblec) [SIG Storage]

Changes by Kind

Deprecation

• KMSv1 is deprecated and will only receive security updates going forward. Use KMSv2 instead. Set --feature-gates=KMSv1=true to use the deprecated KMSv1 feature. (#119007, @aramase) [SIG API Machinery and Auth]

API Change

• Add ServedVersions field to StorageVersion API (#118386, @Richabanker) [SIG API Machinery and Testing]
• Component-base/logs is now more strict about not applying configurations multiple times and will return an error when that is attempted. Can be overridden by binaries which need to do that. (#117108, @pohly) [SIG API Machinery, Architecture, Cloud Provider, Instrumentation, Scheduling and Testing]

Feature

• "plugin_evaluation_total" metric supports prescore/score extension point.
  The metric doesn't get incremented when the prescore/score plugin has nothing to do with an incoming Pod. (#118025, @AxeZhan) [SIG Scheduling]
• Add --concurrency flag to configure the concurrency of kubectl diff execution, defaults to 1 (#118810, @brancz) [SIG CLI]
• AdvancedAuditing feature gate that graduated to GA in 1.12 and was unconditionally enabled has been removed in v1.28. (#118763, @Shubham82) [SIG API Machinery and Auth]
• Allow to monitor client-go DNS resolver latencies via rest_client_dns_resolution_duration_seconds Prometheus metric (#115357, @mfojtik) [SIG API Machinery, Architecture and Instrumentation]
• Dynamic resource allocation: when a claim uses "wait for first consumer" allocation (the default), then it will now get deallocated after it was used by a pod. That ensures that the next pod isn't affected by previous scheduling decision and that resources are not kept allocated unless really needed. If keeping a claim allocated is desired, use "immediate allocation". (#118936, @pohly) [SIG Apps, Node and Testing]
• Kubeadm: add the --allow-experimental-api flag to "kubeadm config migrate/validate" commands. It can be used to migrate or validate WIP / experimental APIs in the future. (#118866, @neolit123) [SIG Cluster Lifecycle]
• Kubeadm: generate CA certificates with a start time that is offset 5 minutes in the past relative to the current system time to workaround cases of clock desync.
  client-go: allow to set NotBefore in NewSelfSignedCACert() (#118922, @champtar) [SIG API Machinery, Auth and Cluster Lifecycle]
• Migrated controller functions to use contextual logging. (#116930, @fatsheep9146) [SIG API Machinery, Apps, Network, Node, Storage and Testing]
• Migrated the certificate controller (within kube-controller-manager) to use contextual logging. (#113994, @mengjiao-liu) [SIG API Machinery, Apps, Auth, Instrumentation and Testing]
• Now it is possible to use pods with volumes and user namespaces. The feature gate was renamed from UserNamespacesStatelessPodsSupport to UserNamespacesSupport (#118691, @giuseppe) [SIG Apps, Node and Testing]
• RetroactiveDefaultStorageClass feature is stable and enabled by default. (#118102, @RomanBednar) [SIG Apps, Storage and Testing]
• Schedular now waits for handlers to finish syncing before the scheduling cycles start. (#116729, @AxeZhan) [SIG Apps, Scheduling and Testing]
• The "value" part in the wait --for=jsonpath='{expression}'[=value] is now optional. If the value is not provided i.e. the command looks like wait --for=jsonpath='{expression}' then the wait condition is interpreted as matched when the expression returns any single JSON value like object or a literal. (#118160, @minherz) [SIG CLI and Testing]
• Updated cAdvisor to v0.47.2 - Fix metrics in cri-o when a container restarts (#118774, @harche) [SIG Node]
• When a pod is done or not going to run, then ResourceClaims for it can be reused by other pods or deleted. (#118817, @pohly) [SIG API Machinery, Apps, Auth, Node and Testing]

Bug or Regression

• Added a new event FailedToRetrieveImagePullSecret which will be generated when a pod references an ImagePullSecret that doesn't exist. (#117927, @kaisoz) [SIG Node]
• As in Kubernetes 1.26 and 1.27, resource claims do not get prepared by kubelet when no container uses them. This was changed accidentally in v1.28.0-alpha.1. (#118786, @pohly) [SIG Node and Testing]
• Faster StatefulSet creation when Parallel mode is enabled. (#117865, @aleksandra-malinowska) [SIG Apps]
• Fix cronjob controller handling of complex schedules, like "30 6-16/4 * * 1-5", for example (#118724, @soltysh) [SIG Apps]
• Fix deletion of non-admissible pods that are deleted during Kubelet restart (#118497, @mimowo) [SIG Node and Testing]
• Fix discoverability of apiregistration.k8s.io in openapi/v3 (#118879, @atiratree) [SIG API Machinery]
• Kubectl explain should correctly work for all resources (#118876, @atiratree) [SIG CLI]
• Kubectl expose supports the creation of different protocol service on the same port (#114909, @aimuz) [SIG CLI]
• The Daemonset controller creates replacements for terminal Pods, which can appear during VM preemptions or when using Pod finalizers (#118716, @alculquicondor) [SIG Apps, Node and Testing]
• The pod_scheduling_duration_seconds metrics won't consider the time when a Pod fails PreEnqueue (like being gated). (#118049, @helayoty) [SIG Scheduling]
• Update apiserver metric request_filter_duration_seconds to include a 10s, 15s and 30s bucket.
• Update apiserver metric request_wait_duration_seconds to include a 15s bucket. (#118945, @andrewsykim) [SIG API Machinery, Instrumentation and Testing]
• Users will no longer see an error for failed events caused due to terminating namespace. (#114849, @padlar) [SIG API Machinery]
• Wait.PollUntilContextTimeout function, if immediate is true, the condition will be invoked before waiting and guarantees that the condition is invoked at least once, regardless of whether the context has been cancelled. (#118686, @aojea) [SIG API Machinery]

Other (Cleanup or Flake)

• Kube-controller-manager and cloud-controller-manager have changed the name of controllers that can be turned off/on that are passed to the --controllers flag (eg pod-garbage-collector-controller ). The old names (eg podgc) are also accepted and aliased to the new names (#115813, @atiratree) [SIG API Machinery and Cloud Provider]
• Kube-proxy will now warn at startup if the configuration seems inconsistent
  with respect to IP families. (For example, if you have an IPv4 node IP, but
  --cluster-cidr is IPv6.) (#119003, @danwinship) [SIG Network]
• Promote kubernetes_healthcheck and kubernetes_healthchecks_total to BETA stability level. (#118986, @logicalhan) [SIG Architecture, Instrumentation and Testing]
• Reduce delay when processing jobs after a transient API error (#118759, @mimowo) [SIG Apps]
• The NetworkPolicyLegacy test suite (deprecated in v1.21) has now officially been removed in favor of the new table driven e2e tests. (#118915, @astoycos) [SIG Network and Testing]
• The feature gates CSIMigrationGCE is graduated to GA and were unconditionally enabled have been removed in v1.25, and the entire gcepd package has been removed. (#117055, @cyclinder) [SIG API Machinery, Node, Scheduling and Storage]
• The metric apiserver_flowcontrol_current_executing_seats has been introduced as a duplicate of apiserver_flowcontrol_request_concurrency_in_use because the latter has a confusing name and will be removed in a later release. (#118960, @MikeSpreitzer) [SIG API Machinery]
• Use table-driven test for TestPerPodSchedulingMetrics (#118842, @helayoty) [SIG Scheduling]

Dependencies

Added

Nothing has changed.

Changed

• github.com/google/cadvisor: v0.47.1 → v0.47.2

Removed

Nothing has changed.