Changelog since v1.28.0

Important Security Information

This release contains changes that address the following vulnerabilities:

CVE-2023-3955: Insufficient input sanitization on Windows nodes leads to privilege escalation

A security issue was discovered in Kubernetes where a user that can create pods on Windows nodes may be able to escalate to admin privileges on those nodes. Kubernetes clusters are only affected if they include Windows nodes.

Affected Versions:
- kubelet <= v1.28.0
- kubelet <= v1.27.4
- kubelet <= v1.26.7
- kubelet <= v1.25.12
- kubelet <= v1.24.16

Fixed Versions:
- kubelet v1.28.1
- kubelet v1.27.5
- kubelet v1.26.8
- kubelet v1.25.13
- kubelet v1.24.17

This vulnerability was discovered by James Sturtevant @jsturtevant and Mark Rossetti @marosset during the process of fixing CVE-2023-3676 (that original CVE was reported by Tomer Peled @tomerpeled92)

CVSS Rating: High (8.8) CVSS:3.1/AV:N/AC:L/PR:L/UI:N/S:U/C:H/I:H/A:H

CVE-2023-3676: Insufficient input sanitization on Windows nodes leads to privilege escalation

A security issue was discovered in Kubernetes where a user that can create pods on Windows nodes may be able to escalate to admin privileges on those nodes. Kubernetes clusters are only affected if they include Windows nodes.

Affected Versions:
- kubelet <= v1.28.0
- kubelet <= v1.27.4
- kubelet <= v1.26.7
- kubelet <= v1.25.12
- kubelet <= v1.24.16

Fixed Versions:
- kubelet v1.28.1
- kubelet v1.27.5
- kubelet v1.26.8
- kubelet v1.25.13
- kubelet v1.24.17

This vulnerability was reported by Tomer Peled @tomerpeled92

CVSS Rating: High (8.8) CVSS:3.1/AV:N/AC:L/PR:L/UI:N/S:U/C:H/I:H/A:H

Changes by Kind

Other (Cleanup or Flake)

• Fixes ability to build 1.28 without network access (#119982, @liggitt) [SIG Testing]

Dependencies

Added

Nothing has changed.

Changed

Nothing has changed.

Removed

Nothing has changed.

v1.28.0

Documentation

Downloads for v1.28.0

Source Code

filename | sha512 hash
-------- | -----------
kubernetes.tar.gz | 9aaf7cc004d09297dc7bbc1f0149d6424d85717b0f31658997bf9a1eee7343ad1ede25e506e1b85956f6b08393d5c7b58e59de860c2f880d97544fd79dfae9da
kubernetes-src.tar.gz | 8e9071210316caac1762535d9437a1e7dcbf644ae8852f4d5babd19a743810c3a2ff2b30f009ba47d28a50e7f5691e56f9b273444bc387dbe95be56c71f2c9d4

Client Binaries

filename | sha512 hash
-------- | -----------
kubernetes-client-darwin-amd64.tar.gz | e9282dc96a73431ed50ef0c515be2a7953b4a243d42ce817e282006aa0431f4f3909971701c4847ea2dd9f268de13fbca40424eff6316f7697faebc2bc0fcff6
kubernetes-client-darwin-arm64.tar.gz | d0c24710717979494d32e9e518435e0803c297a369a1e5022fb522df6a2d4cf91112ad2a3e583abc85080c8ead9644dbb42a6387518a834e5f3d93ca097d0977
kubernetes-client-linux-386.tar.gz | 8f55c1fa60e50c40a81551ab2d6fdf9327a57c445e281105f426ff686395097219766869edef1935b299feb58ab01c9612a1efe3cd1ef06bdc09fc2f93f2ba3a
kubernetes-client-linux-amd64.tar.gz | 0506d82a49794089137e0f240deb3bbaf48be6e61b5b242af02a0d2a0f94fb1df55fcb87570cf40f9abec6b2d6bf11d40fa5d66ab0829ea43169448bda2609d1
kubernetes-client-linux-arm.tar.gz | 3db12e083be8f7a7555a391311f78064a0ffa51f5cfdb6509ee71f33ee5bf56d986e687c5eb39e01c9def7b2154b0298a41c0b960c1fc76228b99c39546529c9
kubernetes-client-linux-arm64.tar.gz | c907332aa6b38ecd82ed7c16741d80e8f23857b49ebff05bdb89692ab286955e03537cc2aba631e932c56bcbcc2aafee2a9b69c5bb6496b869d31b771dc93759
kubernetes-client-linux-ppc64le.tar.gz | fccda39ca81d171ff8bf30a27596ea6e13a6fca7113c3b46f1c2915d50a367b96b6db2e7d8e27fd76c5b3b00f3d447b1da4d1a70fbaf652a7b2b2c4aae71853f
kubernetes-client-linux-s390x.tar.gz | f177677ba4c7e59901ba5cbc10d23384e69cabbbe7f924d0bf0267534eefca4ed0b855ad7193533f5034b080d9894278393b9012b008dd17056d9684aa36e7d3
kubernetes-client-windows-386.tar.gz | bc3cc07e5101c1d1b4484f28b748b03083ad8b1a74c51d67b7c9d628c65a9db07d20f0695f458508a88a259d1f4396b2008d898476716998a32d74dd84901320
kubernetes-client-windows-amd64.tar.gz | 6969e153438cfc3165530562b4bb4cb620588a8b59f1f28bb6a369d7e4ba5f636613d3cab2925d1b00385239ac82bb46bffa00cbbadc3aed1ab54ed620909de2
kubernetes-client-windows-arm64.tar.gz | 3955501ba210e0af25df1431e8b9f641b6d746d0fb882192a24b1d2b18b55ddb48e0c3ab086a0f6e0bf2156a38e11b979f78d5d7b898e24dc34e10f1d558abb0

Server Binaries

filename | sha512 hash
-------- | -----------
kubernetes-server-linux-amd64.tar.gz | 36476c4c64f23b3a48d6e79885ddc764102eefd2ab25b1d721386bdfc15c954d7940ab905615bec357dc5530610ecf7b8e640790206cb5a8da8312cea46db97a
kubernetes-server-linux-arm64.tar.gz | 42c17eb2229f4210521bb78ff7712de9ac4a3e483b503e4a4f9a889a4aea3015890360242d88e9f2dbcb3d4d645bbfc37b31a5d2f151023b594392c23d1b0154
kubernetes-server-linux-ppc64le.tar.gz | e8929e915e61cb9380f6e48abf21f6dfd00b52d7afc1867f8999f55fdf8c60404397dcec51f6feda7bd281fdb15035516a3187162349320460574b14c2a63f25
kubernetes-server-linux-s390x.tar.gz | 88bce78171ff9b12796ccc3e6781a09afb05b7412dcf154aa1b9a9da151a01de07e8ef3a35e305822a67fbf0dfd482bc7320d0b7a39c6384f74139153b0644f8

Node Binaries

filename | sha512 hash
-------- | -----------
kubernetes-node-linux-amd64.tar.gz | fed9ba533e54a4dd6cf26788c27d4f41534ee4f6cf22ee75b183afc45764273e8ac008f06297608342797bc9463c82603947800dce37155424489e20987d3dea
kubernetes-node-linux-arm64.tar.gz | a500906699bc25a8c0825fd38e92b1ce5fddbb3bfc09561e21cfd04bab0cefe6430231b9f347835c8e1d06ffb926b72ae272c119eb72d64580b318d7fcacad20
kubernetes-node-linux-ppc64le.tar.gz | 16bb9a4762fdb4f5140cf518a93d812ddb04c08cc98f0447d1c540d290648a8a050d2d6133e244b40645bb25813d149a96a313de5af178ed30a5dab2919fa845
kubernetes-node-linux-s390x.tar.gz | fb2cf3f819f8c3329fbfc13588a8b206bb16e3b4e351680ed03e3a74cc34b42341743f8f913941e25ed3ca2d7779bf331f31f30821787b1f8cb916f58f183ab4
kubernetes-node-windows-amd64.tar.gz | 60e964a33f10cf0b361f50199aa19f0d89dd82073b31377e7d61b6fb761ef779cc3577bf407edb448c104333185d25eba8d8953e917f52cb62b504ea85121ac4

Container Images

All container images are available as manifest lists and support the described
architectures. It is also possible to pull a specific architecture directly by
adding the "-$ARCH" suffix to the container image name.
name | architectures
---- | -------------
registry.k8s.io/conformance:v1.28.0 | amd64, arm64, ppc64le, s390x
registry.k8s.io/kube-apiserver:v1.28.0 | amd64, arm64, ppc64le, s390x
registry.k8s.io/kube-controller-manager:v1.28.0 | amd64, arm64, ppc64le, s390x
registry.k8s.io/kube-proxy:v1.28.0 | amd64, arm64, ppc64le, s390x
registry.k8s.io/kube-scheduler:v1.28.0 | amd64, arm64, ppc64le, s390x
registry.k8s.io/kubectl:v1.28.0 | amd64, arm64, ppc64le, s390x

Changelog since v1.27.0

Urgent Upgrade Notes

(No, really, you MUST read this before you upgrade)

• Action required for the custom scheduler plugin developers.
  Here's the breaking change in EnqueueExtension in the scheduling framework.
  The EventsToRegister in EnqueueExtension changed the return value from ClusterEvent to ClusterEventWithHint. ClusterEventWithHint allows each plugin to filter out more useless events via the callback function named QueueingHintFn.
  When the scheduling queue receives a cluster event, before moving each Pod from unschedulable pod pool to activeQ/backoffQ, it will call QueueingHintFn of plugins that rejected each Pod in the previous scheduling cycle.
  Depending on the value returned from QueueingHintFn, the scheduling queue changes how it queues each Pod:
• if more than one QueueingHintFn returns QueueImmediately, it queues Pod to activeQ.
• If no QueueingHintFn returns QueueImmediately and more than one plugin returns QueueAfterBackoff, it queues Pod to backoffQ if Pod is backing off, or to activeQ if Pod's backoff has already finished.
• If all QueueingHintFn return QueueSkip, it puts this pod back to the unschedulable pod pool

Having appropriate QueueingHintFn contributes to reducing useless retries and thus improves the overall scheduler's performance.

How can I migrate?

For backward compatibility, nil QueueingHintFn is treated as always returning QueueAfterBackoff.
So, if you want to just keep the existing behavior, you can register ClusterEventWithHint with no QueueingHintFn in it.
But, registering appropriate QueueingHintFn is, of course, better from a scheduling performance perspective. (#118551, @sanposhiho) [SIG Node, Scheduling, Storage and Testing]
- CephFS volume plugin (kubernetes.io/cephfs) has been deprecated in this release and will be removed in a subsequent release. The alternative is to use the CephFS CSI driver (https://github.com/ceph/ceph-csi/) in your Kubernetes cluster. (#118143, @humblec)
- Deprecated support for CSI migration of Ceph RBD volumes. Users who were relying on Kubernetes' ability
to migrate to an out-of-tree storage driver should complete that migration before the support for it is removed. (#118303, @carlory)
- RBD volume plugin (kubernetes.io/rbd) has been deprecated in this release
and will be removed in a subsequent release. Alternative is to use RBD CSI driver
(https://github.com/ceph/ceph-csi/) in your Kubernetes Cluster. (#118552, @humblec)

Changes by Kind

Deprecation

• Changed kubectl version default output to be identical to what kubectl version --short printed,
  and removed --short flag entirely. (#116720, @soltysh)
• Kube-controller-manager deprecate --volume-host-cidr-denylist and --volume-host-allow-local-loopback flags. (#118128, @carlory) [SIG API Machinery, Apps, Network, Node, Storage and Testing]
• Kubelet: The --azure-container-registry-config flag has been deprecated and will be removed in a future release, please use --image-credential-provider-config and --image-credential-provider-bin-dir to setup acr credential provider instead. (#118596, @SataQiu) [SIG Node]
• Removed tracking annotation from validation and defaulting. (#117633, @kannon92)
• Removed withdrawn feature NetworkPolicyStatus. (#115843, @rikatz)
• The deprecated flag --lock-object-namespace and --lock-object-name have been removed from kube-scheduler. Please use --leader-elect-resource-namespace and --leader-elect-resource-name or ComponentConfig instead to configure those parameters. (#119130, @SataQiu) [SIG Scheduling]
• KMSv1 is deprecated and will only receive security updates going forward. Use KMSv2 instead. In a future release, Set --feature-gates=KMSv1=true to use the deprecated KMSv1 feature. (#119007, @aramase)

API Change

• A CDIDevice field is included in the Device Plugin's ContainerAllocateResponse. This field maps to the CDIDevice field in the CRI protocol. (#118254, @elezar) [SIG Node and Testing]
• ACTION_REQUIRED
  When an Indexed Job has a number of completions higher than 10^5 and parallelism higher than 10^4, and a big number of Indexes fail, Kubernetes might not be able to track the termination of the Job. Kubernetes now emits a warning, at Job creation, when the Job manifest exceeds both of these limits. (#118420, @alculquicondor) [SIG Apps]
• Added ServedVersions field to StorageVersion API. (#118386, @Richabanker)
• Added IP mode field to loadbalancer status ingress. (#118895, @RyanAoh)
• Added podReplacementPolicy and terminating field to job api. (#119301, @kannon92)
• Added a new namespaceParamRef field to admissionregistration.k8s.io/v1alpha1.ValidatingAdmissionPolicy. (#119215, @alexzielenski) [SIG API Machinery and Testing]
• Added a warning that TLS 1.3 ciphers are not configurable. (#115399, @3u13r) [SIG API Machinery and Node]
• Added error handling for seccomp localhost configurations that do not properly set a localhostProfile. (#117020, @cji)
• Added fields reason and fieldPath into CRD validation rules to allow users to specify reason and field path when validation failed. (#118041, @cici37) [SIG API Machinery]
• Added namespace access support to the CEL expressions of ValidatingAdmissionPolicy via a namespaceObject
  variable with expressions. (#118267, @cici37) [SIG API Machinery and Testing]
• Added new CRDValidationRatcheting alpha feature. During a PATCH or UPDATE Validation Ratcheting discards errors thrown by unchanged portions of the resource from most OpenAPI schema validations. (#118990, @alexzielenski)
• Added new annotation batch.kubernetes.io/cronjob-scheduled-timestamp to Job objects scheduled from CronJobs. (#118137, @helayoty)
• Added new config option delayCacheUntilActive to KubeSchedulerConfiguration that can provide a tradeoff between memory efficiency and scheduling speed when their leadership is updated in kube-scheduler (#115754, @linxiulei) [SIG API Machinery and Scheduling]
• Changed how KMS v2 encryption at rest can generate data encryption keys.
  When you enable the KMSv2KDF feature gate (off by default), KMS v2 uses a key derivation function to generate single use data encryption keys from a secret seed combined with some random data. This eliminates the need for a counter based nonce while avoiding nonce collision concerns associated with AES-GCM's 12 byte nonce. (#118828, @enj)
• Exposed rest.DefaultServerUrlFor function. (#118055, @timofurrer)
• Extended the Job API for alpha version of BackoffLimitPerIndex. (#119294, @mimowo)
• Graduated AdmissionWebhookMatchCondition feature to beta. (#119380, @a-hilaly)
• If using cgroups v2, then the cgroup aware OOM killer will be enabled for container cgroups via memory.oom.group . This causes processes within the cgroup to be treated as a unit and killed simultaneously in the event of an OOM kill on any process in the cgroup. (#117793, @tzneal) [SIG Apps, Node and Testing]
• In the API Priority and Fairness feature, priority levels that are exempt from limitation can now be given a nominal and a lendable concurrency and their dispatching borrows from the concurrency limits of the other priority levels. For details see https://github.com/kubernetes/enhancements/tree/master/keps/sig-api-machinery/1040-priority-and-fairness#dispatching . (#118782, @MikeSpreitzer) [SIG API Machinery]
• Indexed Job pods now have the pod completion index set as a pod label. (#118883, @danielvegamyhre) [SIG Apps]
• Kube-proxy: added --logging-format flag to support structured logging. (#117800, @cyclinder)
• NodeVolumeLimits implement the PreFilter extension point for skipping the Filter phase if the Pod doesn't use volumes with limits. (#115398, @tangwz) [SIG Scheduling]
• PersistentVolumes have a new LastPhaseTransitionTime field which holds a timestamp of when the volume last transitioned its phase. (#116469, @RomanBednar)
• Pods which set hostNetwork: true and declare ports, get the hostPort field set automatically. Previously this would happen in the PodTemplate of a Deployment, DaemonSet or other workload API. Now hostPort will only be set when an actual Pod is being created. If this presents a problem, setting the feature gate "DefaultHostNetworkHostPortsInPodTemplates" to true will revert this behavior. Please file a kubernetes bug if you need to do this. (#117696, @thockin) [SIG Apps]
• Promoted API groups ValidatingAdmissionPolicy and ValidatingAdmissionPolicyBinding to v1beta1. (#118644, @alexzielenski) [SIG API Machinery, Apps and Testing]
• Promoted the feature gate ValidtaingAdmissionPolicy to beta, and it is turned off by default. (#119409, @alexzielenski)
• Registered_metric_total, disabled_metric_total, hidden_metric_total & kubernetes_feature_enabled are promoted to BETA stability. (#119264, @logicalhan) [SIG API Machinery, Architecture, Cluster Lifecycle and Instrumentation]
• Removed resizeStatus enum from pvc.Status and replaced with AllocatedResourceStatus. (#116335, @gnufied) [SIG API Machinery, Apps, Auth, Node, Storage and Testing]
• Removed WindowsHostProcessContainers feature-gate. (#117570, @marosset) [SIG API Machinery, Apps, Auth, Node and Windows]
• Revised the comment about the feature-gate level for PodFailurePolicy from alpha to beta. (#117802, @kerthcet) [SIG API Machinery and Apps]
• StatefulSet pods now have the pod index set as a pod label statefulset.kubernetes.io/pod-index. (#119232, @danielvegamyhre) [SIG Apps]
• Support for proxying a request to a peer kube-apiserver if the local apiserver is not able to serve it due to version skew or in the case the requested api is disabled on the local apiserver (#117740, @Richabanker) [SIG API Machinery, Apps, Auth, Cloud Provider, Network, Node and Testing]
• Supported BackoffLimitPerIndex in Jobs. (#118009, @mimowo)
• The IPTablesOwnershipCleanup feature (KEP-3178) is now GA; kubelet no longer
  creates the KUBE-MARK-DROP chain (which has been unused for several releases)
  or the KUBE-MARK-MASQ chain (which is now only created by kube-proxy). (#119374, @danwinship)
• The SelfSubjectReview API is promoted to authentication.k8s.io/v1 and the kubectl auth whoami command is GA. (#117713, @nabokihms) [SIG API Machinery, Architecture, Auth, CLI and Testing]
• The names of ResourceClaims generated from ResourceClaimTemplate are now generated. The base name is still <pod>-<claim name>, but a random suffix will avoid name collisions. (#117351, @pohly) [SIG API Machinery, Apps, Auth, Node, Scheduling and Testing]
• The new feature gate "SidecarContainers" is now available. This feature introduces sidecar containers, a new type of init container that starts before other containers but remains running for the full duration of the pod's lifecycle and will not block pod termination. (#116429, @gjkim42) [SIG API Machinery, Apps, Node, Scheduling and Testing]
• Updated the comment about the feature-gate level for PodFailurePolicy from alpha to beta (#118278, @mimowo)
• client-go: Improved memory use of reflector caches when watching large numbers
  of objects which do not change frequently. (#113362, @sxllwx)
• component-base/logs is now stricter about not applying configurations multiple
  times and will return an error when that is attempted. Can be overridden by binaries
  which need to do that. (#117108, @pohly)
• kube-controller-manager: The LegacyServiceAccountTokenCleanUp feature gate
  is now available as alpha (off by default). When enabled, the legacy-service-account-token-cleaner
  controller loop removes service account token secrets that have not been used
  in the time specified by --legacy-service-account-token-clean-up-period (defaulting
  to one year), and are referenced from the .secrets list of a ServiceAccount
  object, and are not referenced from pods. (#115554, @yt2985)
• kube-scheduler component config (KubeSchedulerConfiguration) kubescheduler.config.k8s.io/v1beta2
  is removed in v1.28. Migrate kube-scheduler configuration files to kubescheduler.config.k8s.io/v1. (#117649, @SataQiu)

Feature

• A ValidatingAdmissionPolicy now has its messageExpression field checked against resolved types. (#119209, @jiahuif) [SIG API Machinery]
• Added '--concurrent-cron-job-syncs' flag for kube-controller-manager to set the number of workers for cron job controller. (#117550, @borgerli)
• Added '--concurrent-job-syncs' flag for kube-controller-manager to set the number of job controller workers. (#117138, @tosi3k)
• Added --concurrency flag to configure the concurrency of kubectl diff execution, defaults to 1. (#118810, @brancz)
• Added ConsistentListFromCache feature gate that allows apiserver to serve consistent lists from cache. (#118508, @serathius)
• Added DisruptionTarget condition to the pod preempted by kubelet to make room for a critical pod. (#117586, @mimowo)
• Added apiserver_admission_match_condition_evaluation_seconds and apiserver_admission_match_condition_exclusions_total metrics. (#119311, @ivelichkovich)
• Added a container image for kubectl at registry.k8s.io/kubectl across the same architectures as other images (linux/amd64 linux/arm64 linux/s390x linux/ppc64le) (#116672, @dims) [SIG Architecture and Release]
• Added a new command line argument --interactive to kubectl. The new command line argument lets a user confirm deletion requests per resource interactively. (#114530, @ardaguclu) [SIG CLI and Testing]
• Added a new feature gate, SchedulerQueueingHints (enabled by default).
  The new feature gate activates a framework for fine-grained filtering of events related to scheduler plugins.
  In this release, no default scheduling plugins make use of the hinting framework, so you should not expect any behavior changes. (#119328, @sanposhiho) [SIG Scheduling]
• Added full cgroup v2 swap support for both Limited and Unlimited swap.

When LimitedSwap is enabled the swap limit would be automatically calculated for
Burstable QoS pods. For Best-Effort/Guaranteed QoS pods, swap would be disabled.

Containers with memory requests equal to their memory limits also won't have
swap access, and it is a way to opt-out of swap for a single container.

The formula for the swap limit for Burstable QoS pods is:
(<memory-request>/<node-memory-capacity>)*<node-swap-capacity>.

Support for cgroup v1 is removed. (#118764, @iholder101)
- Added handling for pods in podgc for PodReplacementPolicy or PodDisruption. (#118772, @kannon92)
- Added reason to metric attachdetach_controller_forced_detaches in the attach detach controller. (#119185, @xing-yang)
- Added support for pod hostNetwork field selector (#110477, @halfcrazy) [SIG Apps and Node]
- Added swap to stats to Summary API and Prometheus endpoints (stats/summary and /metrics/resource). (#118865, @iholder101)
- Added the implementation for PodRecreationPolicy to wait for the creation of pods once the existing ones are fully terminated. (#117015, @kannon92)
- Allow to monitor client-go DNS resolver latencies via rest_client_dns_resolution_duration_seconds Prometheus metric. (#115357, @mfojtik)
- Apiserver adds two new metrics etcd_requests_total and etcd_request_errors_total that allow users to monitor requests to etcd storage, split by operation and resource type. (#117222, @iyear) [SIG API Machinery]
- Bumped distroless-iptables to 0.2.6 based on Go 1.20.6. (#119365, @xmudrii)
- Bumped metrics-server to v0.6.3. (#117120, @dgrisonnet)
- CEL authorizer checks no longer raise runtime errors. Calls to "check" will always return a decision object and the authorization error (if any) can be accessed within expressions using the new decision methods "errored" and "error". (#118804, @benluddy) [SIG API Machinery]
- CRI: exposed commit memory bytes in container stats specific to Windows (#119238, @kiashok)
- Client-go now exposes two new metrics to monitor the client-go logic that
generate http.Transports for the clients.

• rest_client_transport_cache_entries is a gauge metric
  with the number of existing entries in the internal cache

• rest_client_transport_create_calls_total is a counter
  that increments each time a new transport is created, storing
  the result of the operation needed to generate it: hit, miss
  or uncacheable. (#117295, @aojea)

• Cloud controller manager's node controller now emits timing metrics for initial Node synchronization. These metrics measure the delay between the creation of a new Node and the node controller's initial management actions, such as removing the cloud provider taint. These metrics should be consulted when setting cloud controller manager's --concurrent-node-syncs flag. (#119241, @cartermckinnon) [SIG Cloud Provider and Instrumentation]
• Dynamic resource allocation: when a claim uses "wait for first consumer" allocation (the default), then it will now get deallocated after it was used by a pod. That ensures that the next pod isn't affected by previous scheduling decision and that resources are not kept allocated unless really needed. If keeping a claim allocated is desired, use "immediate allocation." (#118936, @pohly)
• Enabled use of pods with volumes and user namespaces. The feature gate was renamed from UserNamespacesStatelessPodsSupport to UserNamespacesSupport. (#118691, @giuseppe)
• External credential provider plugins will now have their standard error output logged by kubelet upon failures. (#117448, @cartermckinnon)
• Faster scheduling when ResourceClaims are involved. (#119078, @pohly)
• Fixed the alpha CloudDualStackNodeIPs feature. (#118329, @danwinship)
• Graduated the LegacyServiceAccountTokenTracking feature gate to GA. The usage of auto-generated secret-based service account token now produces warnings, and relevant Secrets are labeled with a last-used timestamp (label key kubernetes.io/legacy-token-last-used). (#117591, @zshihang) [SIG API Machinery, Auth and Testing]
• Graduated the ProbeTerminationGracePeriod feature gate to GA. (#114307, @rphillips)
• Hashing of KeyID in Logs

This release adds a feature to hash the KeyID values in the logs. The KeyID values are sensitive information that should not be exposed in plain text in the logs. By hashing the KeyID values, we can protect the confidentiality of the data while still being able to log the necessary information. (#118988, @nilekhc) [SIG API Machinery, Auth and Testing]
- Implemented alpha support for a drop-in kubelet configuration directory. (#119390, @sohankunkerkar)
- In the course of admitting a single request, the ValidatingAdmissionPolicy plugin will perform no more than one authorization check per unique authorizer expression. All evaluations of identical authorizer expressions will produce the same decision. (#116443, @benluddy) [SIG API Machinery and Testing]
- Introduce support for CEL optionals (see CEL spec proposal 246).
This feature will not be fully enabled until a future Kubernetes release (likely to be v1.29), but is added in v1.28 to enable
safe rollback on downgrade. (#118339, @jpbetz) [SIG API Machinery, Auth, Cloud Provider and Testing]
- Kube-controller-manager: the dynamic resource controller steps in when a pod got created such that the scheduler ignores it (i.e. spec.nodeName is set) and then takes care of triggering delayed resource claim allocation and/or reserving a claim for the pod. (#118209, @pohly) [SIG API Machinery, Apps, Auth, Node and Testing]
- Kube-proxy handles Terminating EndpointSlices conditions and enables zero downtime deployments for Services with ExternalTrafficPolicy=Local author: @andrewsykim (#117718, @aojea) [SIG Network, Testing and Windows]
- Kube-proxy service health returns http header X-Load-Balancing-Endpoint-Weight with number of local endpoints. The same information is still available in response body JSON payload.LocalEndpoints. (#118999, @cezarygerard)
- Kubelet: plugins for dynamic resource allocation may use the v1alpha3 API instead of v1alpha2 if they want to do prepare/unprepare operations in batches. (#119012, @pohly)
- Kubelet: security of dynamic resource allocation was enhanced by limiting node access to those objects that are needed on the node. (#116254, @pohly) [SIG Auth and Testing]
- Kubelet: un-deprecated --provider-id flag. (#116530, @pacoxu)
- Kubernetes is now built with Go 1.20.4. (#117744, @xmudrii) [SIG Release and Testing]
- Kubernetes is now built with Go 1.20.5. (#118507, @jeremyrickard)
- Kubernetes is now built with Go 1.20.6. (#119324, @xmudrii)
- Metric scheduler_scheduler_goroutines is removed. Use scheduler_goroutines instead. (#117727, @kerthcet) [SIG Scheduling]
- Migrated pkg/controller/endpoint to contextual logging. (#116755, @my-git9)
- Migrated pkg/scheduler/framework/preemption to use contextual logging. (#116835, @mengjiao-liu)
- Migrated pod-security-admission to use contextual logging. (#114471, @Namanl2001) [SIG Apps and Auth]
- Migrated controller functions to use contextual logging. (#116930, @fatsheep9146) [SIG API Machinery, Apps, Network, Node, Storage and Testing]
- Migrated the Job controller (within kube-controller-manager) to use contextual logging. (#116910, @fatsheep9146) [SIG API Machinery, Apps and Testing]
- Migrated the EndpointSlice and EndpointSliceMirroring controllers (within kube-controller-manager) to use contextual logging. (#115295, @Namanl2001) [SIG API Machinery, Apps, Network and Testing]
- Migrated the certificate controller (within kube-controller-manager) to use contextual logging. (#113994, @mengjiao-liu) [SIG API Machinery, Apps, Auth, Instrumentation and Testing]
- Migrated the noderesources scheduler plugin to use contextual logging. (#116748, @mengjiao-liu)
- Migrated the podtopologyspread scheduler plugins to use contextual logging. (#116797, @mengjiao-liu) [SIG Instrumentation and Scheduling]
- Moved non-graceful node shutdown to GA. (#118228, @carlory)
- New CEL Library functions to support Kubernetes Quantities. (#118803, @alexzielenski) [SIG API Machinery]
- New Metrics Added for Encryption Configuration Controller

This release adds new metrics to the Encryption Configuration Controller to help monitor the automatic reloading of encryption configuration. The new metrics include:

• apiserver_encryption_config_controller_automatic_reload_failures_total: Total number of failed automatic reloads of encryption configuration.
• apiserver_encryption_config_controller_automatic_reload_success_total: Total number of successful automatic reloads of encryption configuration.
• apiserver_encryption_config_controller_automatic_reload_last_timestamp_seconds: Timestamp of the last successful or failed automatic reload of encryption configuration.

These metrics can be used to monitor the health of the Encryption Configuration Controller and to troubleshoot any issues that may arise during automatic reloading of encryption configuration. (#119008, @nilekhc)
- New staging repo has been created for the EndpointSlice reconciler. (#118953, @mskrocki)
- Promoted ServiceNodePortStaticSubrange feature gate to beta, and it will be enabled by default. (#117877, @xuzhenglun)
- Promoted the following apiserver flowcontrol metrics to Beta:

• apiserver_flowcontrol_request_wait_duration_seconds
• apiserver_flowcontrol_current_executing_seats
• apiserver_flowcontrol_nominal_limit_seats
• apiserver_flowcontrol_rejected_requests_total
• apiserver_flowcontrol_dispatched_requests_total
• apiserver_flowcontrol_current_inqueue_requests
• apiserver_flowcontrol_current_executing_requests (#119110, @andrewsykim)
• Renamed PodHasNetwork to PodReadyToStartContainers. (#117702, @kannon92) [SIG Node and Testing]
• Replaced apiserver_storage_db_total_size_in_bytes with apiserver_storage_size_bytes metric. (#118812, @serathius)
• Scheduler now waits for handlers to finish syncing before the scheduling cycles start. (#116729, @AxeZhan)
• Set metrics-server's metric-resolution to 15s. (#117121, @dgrisonnet) [SIG Cloud Provider and Instrumentation]
• SubjectAccessReview requests sent to webhook authorizers now default spec.resourceAttributes.version to * if unset. (#116937, @AxeZhan) [SIG Apps and Auth]
• Supported specifying a custom retry period for cloud load-balancer operations. (#94021, @timoreimann)
• The "value" part in the wait --for=jsonpath='{expression}'[=value] is now
  optional. If the value is not provided i.e., the command looks like wait --for=jsonpath='{expression}'
  then the wait condition is interpreted as matched when the expression returns
  any single JSON value like object or a literal. (#118160, @minherz)
• The Kubernetes apiserver now emits a warning message for Pods with a null labelSelector in podAffinity or topologySpreadConstraints. The null labelSelector means "match none". Using it in podAffinity or topologySpreadConstraint could lead to unintended behavior. (#117025, @sanposhiho) [SIG Scheduling]
• The AdvancedAuditing feature gate that graduated to GA in v1.12 (and was unconditionally
  enabled) has been removed. (#118763, @Shubham82)
• The ExpandedDNSConfig feature has graduated to GA. 'ExpandedDNSConfig' feature was locked to default value and will be removed in v1.30. If you were setting this feature gate explicitly, please remove it now. (#116741, @gjkim42) [SIG Apps, Network and Node]
• The apiserver debug endpoint /debug/api_priority_and_fairness/dump_requests has been extended to dump executing requests as well as queued ones. A column for StartTime has been added to the returned table, with the queued requests having a StartTime of "0001-01-01T00:00:00Z". The executing requests have a RequestIndexInQueue of -1, and the QueueIndex is also -1 for priority levels without queues. (#119009, @MikeSpreitzer) [SIG API Machinery]
• The helping message of commands which have sub-commands is now clearer and more instructive. It will show the full command instead of kubectl <command> --help ...

Changed kubectl create secret --help description. There will be a short introduction to the three secret types and clearer guidance on how to use the command. (#117930, @LronDC)
- The scheduler skips the InterPodAffinity Score plugin when nothing to do with the Pod.
It will affect some metrics values related to the InterPodAffinity Score plugin. (#117794, @utam0k) [SIG Scheduling]
- The scheduler skips the PodTopologySpread Filter plugin if no spread constraints.
It will affect some metrics values related to the PodTopologySpread Filter plugin. (#117683, @utam0k)
- The scheduler skips the PodTopologySpread Score plugin when nothing to do with the Pod.
It will affect some metrics values related to the PodTopologySpread Score plugin. (#118608, @utam0k)
- The short names vwc and mwc were introduced for the resources validatingwebhookconfigurations and mutatingwebhookconfigurations. (#117535, @hysyeah)
- Updated etcd image to 3.5.9-0. (#117999, @kkkkun) [SIG API Machinery]
- Updated cAdvisor to v0.47.2 and fixed metrics in cri-o when a container restarts. (#118774, @harche)
- Updated distroless I-tables to use registry.k8s.io/build-image/distroless-iptables:v0.2.5 (#118541, @jeremyrickard) [SIG Testing]
- Updated distroless iptables to use released image registry.k8s.io/build-image/distroless-iptables:v0.2.4 (#117746, @xmudrii) [SIG Testing]
- Updated the scheduler interface and cache methods to use contextual logging. (#116849, @mengjiao-liu)
- ValidatingAdmissionPolicy type checking now correctly handles authorizer variable. (#118540, @jiahuif) [SIG API Machinery]
- When a pod is done or not going to run, then ResourceClaims for it can be reused by other pods or deleted. (#118817, @pohly)
- With the KubeletCgroupDriverFromCRI feature gate enabled and sufficiently new version of a container
runtime, kubelet automatically detects the cgroup driver config from the container runtime, eliminating
the need to specify the cgroupDriver configuration option (or --cgroup-driver flag) of kubelet. (#118770, @marquiz)
- [Kube-proxy]: Implemented connection draining for terminating nodes. (#116470, @alexanderConstantinescu)
- --version=v1.X.Y... can now be used to set the prerelease and buildID portions of the version reported by components (#117688, @liggitt) [SIG API Machinery, Architecture and Release]
- RetroactiveDefaultStorageClass feature made stable and enabled by default. (#118102, @RomanBednar)
- TopologyManagerPolicyOptions feature-flag is promoted to beta and enabled by default. (#118816, @PiotrProkop)
- force_delete_pods_total and force_delete_pod_errors_total metrics count all pod deletion behaviors. (#118480, @carlory)
- klog text output now uses JSON as encoding for structs, maps and slices. (#117687, @pohly)
- kube-proxy in iptables mode will now have separate sync_full_proxy_rules_duration_seconds\nand
sync_partial_proxy_rules_duration_seconds (in addition to the existing\nsync_proxy_rules_duration_seconds),
giving better information about the duration of each \nsync type, rather than
only giving a weighted average of the two sync types together. (#117787, @danwinship)
- kubeadm: added a new "kubeadm config validate" command that can be used to
validate any input config file. Use the --config flag to pass a config file
to it. See the command --help screen for more information. As a result of adding
this new command, enhance the validation capabilities of the existing "kubeadm
config migrate" command. For both commands unknown APIs or fields will throw errors. (#118013, @neolit123)
- kubeadm: added the --allow-experimental-api flag to "kubeadm config migrate/validate" commands. It can be used to migrate or validate WIP/experimental APIs in the future. (#118866, @neolit123)
- kubeadm: generate CA certificates with a start time that is offset 5
minutes in the past relative to the current system time to workaround cases of
clock desync. (#118922, @champtar)
- plugin_evaluation_total metric supports prescore/score extension point.
The metric doesn't get incremented when the prescore/score plugin has nothing to do with an incoming pod. (#118025, @AxeZhan)

Documentation

• Enhanced clarity in error messaging when waiting for volume creation (#118262, @torredil) [SIG Apps and Storage]

Failing Test

• Allowed Azure Disk e2es to use newer topology labels if available from nodes. (#117216, @gnufied)
• Fixed nil pointer in test AfterEach volumeperf.go for sidecar release. (#117368, @sunnylovestiramisu)
• Switched back to debian-base instead of distroless for conformance image. (#119422, @saschagrunert)

Bug or Regression

• Added a new event FailedToRetrieveImagePullSecret which will be generated when a pod references an ImagePullSecret that doesn't exist. (#117927, @kaisoz) [SIG Node]
• Added additional validation for endpoint IP configuration while iterating through queried endpoint list. (#116749, @princepereira)
• Added warning for dup ports update/patching in pod's container ports and service ports. (#113245, @pacoxu)
• As in Kubernetes v1.26 and v1.27, resource claims do not get prepared by kubelet when no container uses them. This was changed accidentally in v1.28.0-alpha.1. (#118786, @pohly)
• Bumped cadvisor version to v0.47.3. (#119225, @iholder101)
• CI job ci-kubernetes-node-arm64-ubuntu-serial will test node e2e on arm64, use-dockerized-build and target-build-arch are required to run this job. (#118567, @chendave)
• CVE-2023-27561 CVE-2023-25809 CVE-2023-28642: Bump fix runc v1.1.4 -> v1.1.5 (#117095, @PushkarJ) [SIG Architecture, Node and Security]
• Code blocks in kubectl {$COMMAND}--help will move right by 3 indentation. (#118029, @ardaguclu)
• Compute the backoff delay more accurately for deleted pods (#118413, @mimowo) [SIG Apps]
• Declare Job as finished only after removing all Pod finalizers to avoid orphan Pods. (#119159, @alculquicondor)
• During device plugin allocation, resources requested by the pod can only be allocated if the device plugin has registered itself to kubelet AND healthy devices are present on the node to be allocated. If these conditions are not sattsfied, the pod would fail with UnexpectedAdmissionError error. (#116376, @swatisehgal) [SIG Node and Testing]
• Dynamic Resource Allocation: logged an error and submitted an event when Kubelet failed to prepare dynamic resources. (#118578, @bart0sh)
• Ensure Job status updates are batched by 1s. This fixes an unlikely scenario when a sequence of immediately
  completing pods could trigger a sequence of non-batched Job status updates. (#118470, @mimowo) [SIG Apps]
• Faster StatefulSet creation when Parallel mode is enabled. (#117865, @aleksandra-malinowska)
• Fixed a data race in TopologyCache when AddHints and SetNodes are called concurrently. (#117249, @tnqn) [SIG Apps and Network]
• Fixed a race condition in kube-proxy when using LocalModeNodeCIDR, to avoid dropping Services traffic if the object node is recreated when kube-proxy is starting. (#118499, @aojea)
• Fixed bug where listOfStrings.join() in CEL expressions resulted in an unexpected internal error. (#117593, @jpbetz) [SIG API Machinery]
• Fixed incorrect calculation for ResourceQuota with PriorityClass as its scope. (#117677, @Huang-Wei) [SIG API Machinery]
• Fix: After a Node is down and take some time to get back to up again, the mount point of the evicted Pods cannot be cleaned up successfully. (#111933) Meanwhile Kubelet will print the log Orphaned pod "xxx" found, but error not a directory occurred when trying to remove the volumes dir every 2 seconds. (#105536) (#116134, @cvvz) [SIG Node and Storage]
• Fix: The volume is not detached after the pod and PVC objects are deleted. (#116138, @cvvz) [SIG Storage]
• Fixed Cronjob status.lastSuccessfulTime not populated by a manually triggered (#118530, @carlory)
• Fixed Topology Aware Hints not working when the topology.kubernetes.io/zone label is added after Node creation. (#117245, @tnqn)
• Fixed creationTimestamp: null causing unnecessary writes to etcd. (#116865, @alexzielenski)
• Fixed vSphere cloud provider not to skip detach volumes from nodes at kube-controller-startup. (#117243, @jsafrane)
• Fixed a bug at kube-apiserver start where APIService objects for custom resources could be deleted and recreated. (#118104, @liggitt)
• Fixed a bug that unintentionally overrides custom Accept headers in http (live-/readiness)-probes if the header is in lowercase. (#114606, @tuunit)
• Fixed a bug where kubectl port-forward, when used with a Deployment, could connect to a terminating pod even when a running pod is also available. (#119256, @brianpursley) [SIG CLI]
• Fixed a bug where pv recycler failed to scrub volume with too many files in the directory due to hitting ARG_MAX limit with rm command (#117189). (#117283, @defo89) [SIG Cloud Provider and Storage]
• Fixed a memory leak in the Kubernetes API server that occurs during APIService processing. (#117258, @enj) [SIG API Machinery]
• Fixed a race condition between Run() and SetTransform() and SetWatchErrorHandler() in shared informers. (#117870, @howardjohn) [SIG API Machinery]
• Fixed a race condition serving OpenAPI content (#117705, @Jefftree)
• Fixed a regression in 1.27.0 that resulted in missing metadata in converted object errors when modifying objects for multi-version custom resource definitions with a conversion strategy of None. (#117301, @ncdc)
• Fixed a regression in kubectl and client-go discovery when configured with a server URL other than the root of a server (#117495, @ardaguclu)
• Fixed an issue where the API server did not send impersonated UID to authentication webhooks. (#116681, @stlaz) [SIG API Machinery and Auth]
• Fixed bug that caused a resource to include patch directives when using strategic merge patch against a non-existent field. (#117568, @alexzielenski)
• Fixed bug to correctly report ErrRegistryUnavailable on pulling container images for remote CRI runtimes. (#117612, @saschagrunert) [SIG Node]
• Fixed bug where explain was not properly respecting jsonpaths. (#115694, @mpuckett159)
• Fixed bug where using the $deleteFromPrimitiveList directive in a strategic merge patch of certain fields would remove the other values from the list instead of the values specified. (#110472, @brianpursley) [SIG API Machinery]
• Fixed component status calling etcd health endpoint over http which exposed kubernetes to the risk of complete watch starvation and is inconsistent with other etcd probing done by kube-apiserver. (#118460, @serathius)
• Fixed computing backoff delay when using Job pod failure policy, by including in the backoff delay calculation pod failures ignored from the backoffLimit counter. (#119434, @mimowo)
• Fixed cronjob controller handling of complex schedules, like 30 6-16/4 * * 1-5, for example. (#118724, @soltysh)
• Fixed deletion of non-admissible pods that are deleted during Kubelet restart. (#118497, @mimowo)
• Fixed issue where kubectl-convert would fail when encountering resources that could not be converted to the specified api version. New behavior is to warn the user of the failed conversions and continue to convert the remaining resources. (#117002, @gxwilkerson33)
• Fixed issue where there was no response or error from kubectl rollout status when there were no resources of specified kind. (#117884, @gxwilkerson33) [SIG CLI]
• Fixed kubelet startup getting stuck with NewVolumeManagerReconstruction feature enabled and a CSI volume present in /var/lib/kubelet/pods. (#117804, @jsafrane) [SIG Node and Storage]
• Fixed performance regression in scheduler caused by frequent metric lookup on critical code path. (#117594, @tosi3k)
• Fixed restricted debug profile. (#117543, @mochizuki875)
• Fixed the preStop hook. This will now block the pod termination grace period. (#115835, @HirazawaUi)
• Fixed the discoverability of apiregistration.k8s.io in openapi/v3 (#118879, @atiratree)
• If kubeadm reset finds no etcd member ID for the peer it removes during the remove-etcd-member phase, it continues immediately to other phases, instead of retrying the phase for up to 3 minutes before continuing. (#117724, @dlipovetsky) [SIG Cluster Lifecycle]
• Improved exponential backoff in Reflector, significantly reducing the load on Kubernetes apiserver in case of throttling of requests. (#118132, @marseel) [SIG API Machinery and Scalability]
• Known issue: fixed that the PreEnqueue plugins aren't executed for Pods proceeding to activeQ through backoffQ. (#117194, @sanposhiho) [SIG Release and Scheduling]
• Kubeadm: the limitation that the ignorePreflightErrors field can not be set to all in kubeadm config file has been removed. (#119351, @SataQiu)
• Kubelet terminates pods correctly upon restart, fixing an issue where pods may have not been fully terminated if the kubelet was restarted during pod termination. (#117019, @bobbypage) [SIG Node and Testing]
• Kubelet will now skip pod resource checks when the request is zero. (#116408, @ChenLingPeng)
• Number of errors reported to the metric storage_operation_duration_seconds_count for emptyDir decreased significantly because previously one error was reported for each projected volume created. (#117022, @mpatlasov) [SIG Storage]
• Pod termination will be faster when the pod has a missing volume reference. (#117412, @smarterclayton) [SIG Node and Testing]
• Recording timing traces had a race condition. Impact in practice was probably low. (#117139, @pohly) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node and Storage]
• Reduced CPU and memory consumption of kube-apiserver if OpenAPI V2 will not be accessed by any client. Also improved performance of the apiserver on installation of many CRDs. (#118212, @Jefftree)
• Removed leading zeros from the etcd member ID in kubeadm log messages. (#117919, @dlipovetsky) [SIG Cluster Lifecycle]
• Resolves a spurious "Unknown discovery response content-type" error in client-go discovery requests by tolerating extra content-type parameters in API responses (#117571, @seans3) [SIG API Machinery]
• Reverted NewVolumeManagerReconstruction and SELinuxMountReadWriteOncePod feature gates to disabled by default to resolve a regression of volume reconstruction on kubelet/node restart (#117751, @liggitt) [SIG Storage]
• Setting a mirror pod's phase to Succeeded or Failed can prevent the corresponding static pod from restarting due mutation of a Kubelet cache. (#116482, @smarterclayton) [SIG Node]
• Show a warning when volume.beta.kubernetes.io/storage-class annotation is used in pv or pvc (#117036, @haoruan) [SIG Storage]
• Static pods were taking extra time to be restarted after being updated. Static pods that are waiting to restart were not correctly counted in kubelet_working_pods. (#116995, @smarterclayton) [SIG Node]
• The Daemonset controller creates replacements for terminal Pods, which can appear during VM preemptions or when using Pod finalizers. (#118716, @alculquicondor)
• The pod_scheduling_duration_seconds metrics won't consider the time when a pod fails PreEnqueue (like being gated). (#118049, @helayoty)
• The kube-proxy sync_proxy_rules_iptables_total metric has now reverted back
  to its pre-1.27 behavior of tracking the total number of iptables rules that
  kube-proxy is responsible for, rather than only counting the number of rules
  that it re-synced on the last sync. The new sync_proxy_rules_iptables_last
  metric now gives the latter number. (#119140, @danwinship) [SIG Network]
• The metric apiserver_flowcontrol_request_concurrency_limit has been deprecated and will be removed in a future release. It is a duplicate of apiserver_flowcontrol_nominal_limit_seats (introduced in release 1.26) but has an outdated name and had an outdated HELP string. (#118959, @MikeSpreitzer) [SIG API Machinery]
• Updated etcd version to 3.5.8. (#117335, @kkkkun)
• Updated apiserver metric request_filter_duration_seconds to include a 10s, 15s and 30s bucket.
  Updated apiserver metric request_wait_duration_seconds to include a 15s bucket. (#118945, @andrewsykim)
• Updated kube-apiserver's priority & fairness work estimator such that 'max seats' is MIN(0.15 x nominalCL, nominalCL / handSize)

This fixes a bug where clients with requests using hand size x max seats greater than the nominal concurrency limit can starve other requests in the same priority level. (#118601, @andrewsykim)
- Updated static pods are restarted 2s faster by correcting a safe but non-optimal ordering bug. (#116690, @smarterclayton) [SIG Node]
- Users will no longer see an error for failed events caused due to terminating namespace. (#114849, @padlar) [SIG API Machinery]
- [Dual-stack] Fixed generateAPIPodStatus() of kubelet handling Secondary IP. hostIPs order may not be consistent. If secondary IP is before primary one, current logic adds primary IP twice into PodIPs, which leads to error: "may specify no more than one IP for each IP family". (#116879, @lzhecheng)
- [KCCM] service controller: change the cloud controller manager to make providerID a predicate when synchronizing nodes. This change allows load balancer integrations to ensure that the providerID is set when configuring
load balancers and targets. (#117388, @alexanderConstantinescu) [SIG Cloud Provider and Network]
- kube-apiserver will now always remove its endpoint from Kubernetes service during
graceful shutdown (even if it's the only/last one). (#116685, @nayihz)
- kubeadm: fixed a bug where the static pod changes detection logic is inconsistent
with kubelet. (#118069, @SataQiu)
- kubeadm: crictl pull should use -i to set the image service endpoint. (#117835, @pacoxu)
- kubeadm: fixed a bug where file copy(backup) could not be executed correctly
on Windows platform during upgrade. (#117861, @SataQiu)
- kubeadm: speedup init by 0s or 20s. kubelet-start phase is now after etcd
and control-plane phases, removing a race condition between kubelet looking for
static pod manifests and kubeadm writing them. (#117984, @champtar)
- kubeadm: will now throw warnings instead of errors for deprecated feature gates. (#118270, @pacoxu)
- kubectl events --for will also support fully qualified names such as replicasets.apps,
etc. (#117034, @ardaguclu)
- kubectl explain should correctly work for all resources. (#118876, @atiratree)
- kubectl expose supports the creation of different protocol services on the same port. (#114909, @aimuz)
- kubelet will ensure /etc/hosts file is mode 0644 regardless of umask. (#113209, @luozhiwenn)
- kubelet: print sorted volumes message in events. (#117079, @qingwave)
- wait.PollUntilContextTimeout function, if immediate is true, the condition
will be invoked before waiting and guarantees that the condition is invoked at
least once, regardless of whether the context has been cancelled. (#118686, @aojea)

Other (Cleanup or Flake)

• A v2-level info log will be added, which will output the details of the pod being preempted, including victim and preemptor. (#117214, @HirazawaUi)
• Allowed container runtimes to use ErrSignatureValidationFailed as possible image pull failure. (#117717, @saschagrunert)
• Deprecated genericclioptions.IOStreams and used genericiooptions.IOStreams. (#117102, @ardaguclu)
• E2e framework: the node-role.kubernetes.io/master taint has been removed from the default value of --non-blocking-taints flag. You may need to set --non-blocking-taints explicitly if the cluster to be tested has nodes with the deprecated node-role.kubernetes.io/master taint. (#118510, @SataQiu) [SIG Testing]
• Enabled the node-local kubelet podresources API endpoint on windows, alongside unix. (#115133, @ffromani)
• Fixed dra e2e image build on non-amd64 architectures. (#117912, @bart0sh) [SIG Node and Testing]
• Kube-apiserver adds two new alpha metrics conversion_webhook_request_total and conversion_webhook_duration_seconds that allow users to monitor requests to CRD conversion webhooks, split by result, and failure_type (In case of failure). (#118292, @cchapla) [SIG API Machinery, Architecture and Instrumentation]
• Kube-proxy will now warn at startup if the configuration seems inconsistent
  with respect to IP families. (For example, if you have an IPv4 node IP, but
  --cluster-cidr is IPv6.) (#119003, @danwinship) [SIG Network]
• Kube-proxy: removed log warning about not using config file. (#118115, @TommyStarK) [SIG Network]
• Made Job controller batching of syncJob invocations enabled unconditionally (it was conditional on JobReadyPods feature before).
  Also, Job controller's constants for default backoff and maximal backoff are lowered down to 1s (from 10s) and 1min (from 6min), respectively. These constants are used to determine the backoff delay for the next Job controller sync in case of a request failure. (#118615, @mimowo) [SIG Apps and Testing]
• Marked the feature gate ExperimentalHostUserNamespaceDefaulting as deprecated.
  Enabling the feature gate already had no effect; the deprecation allows for removing the feature gate in a future release. (#116723, @SergeyKanzhelev) [SIG Node]
• Migrated pkg/scheduler/framework/runtime to use contextual logging. (#116842, @mengjiao-liu) [SIG Instrumentation and Scheduling]
• Migrated the disruption controller (within kube-controller-manager) to use contextual logging. (#119147, @mengjiao-liu) [SIG API Machinery, Apps, Instrumentation and Testing]
• Migrated the interpodaffinity scheduler plugin to use contextual logging. (#116635, @mengjiao-liu) [SIG Instrumentation and Scheduling]
• Migrated the podgc controller and some other remaining log calls within kube-controller-manager to use contextual logging. kube-controller-manager is now converted completely. (#119250, @pohly) [SIG API Machinery, Apps, Cloud Provider, Instrumentation, Network, Storage and Testing]
• Migrated the volumezone scheduler plugin to use contextual logging. (#116829, @mengjiao-liu) [SIG Instrumentation and Scheduling]
• Moved k8s.io/kubernetes/pkg/kubelet/cri/streaming package to k8s.io/kubelet/pkg/cri/streaming. (#118253, @saschagrunert) [SIG Node, Release and Security]
• OpenAPI proto deserializations should use gnostic-models instead of the gnostic library. (#118384, @Jefftree) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Instrumentation, Node, Storage and Testing]
• Projects which use k8s.io/code-generator and invoke generate-groups or generate-internal-groups.sh have a new, simpler script (kube_codegen.sh) they can use. The old scripts are deprecated but remain intact. (#117262, @thockin) [SIG API Machinery and Instrumentation]
• Promoted kubernetes_healthcheck and kubernetes_healthchecks_total to BETA stability level. (#118986, @logicalhan)
• Reduced delay when processing jobs after a transient API error. (#118759, @mimowo)
• Removed GA'ed feature gate DelegateFSGroupToCSIDriver. (#117655, @carlory)
• Removed GA'ed feature gate DevicePlugins. (#117656, @carlory)
• Removed GA'ed feature gate KubeletCredentialProviders. (#116901, @pacoxu)
• Removed GA'ed feature gates: MixedProtocolLBService, ServiceInternalTrafficPolicy,
  ServiceIPStaticSubrange, and EndpointSliceTerminatingCondition. (#117237, @yulng)
• Removed KUBECTL_EXPLAIN_OPENAPIV3 which is already redundant. (#119286, @ardaguclu)
• Removed the deprecated azureFile in-tree storage plugin. (#118236, @andyzhangx)
• Revised OpenAPI v2 fetching for CustomResourceDefinitions. CRDs are now aggregated lazily,
  which improves resource usage during installation of many CRDs. As a result, the first request
  to fetch the OpenAPI may be slower. (#118808, @Jefftree)
• Shrank the OpenAPI v2 spec by more than 50%, especially for less CPU resource consumption. (#118204, @sttts)
• Structured logging of NamespacedName was inconsistent with klog.KObj. Now both will use lower case field names and namespace is optional. (#117238, @pohly)
• The GetAllocatableResources podresources API endpoint is now GA. (#118973, @ffromani)
• The NetworkPolicyLegacy test suite (deprecated in v1.21) has now officially been removed in favor of the new table driven e2e tests. (#118915, @astoycos)
• The generate_groups.sh and generate_internal_groups.sh scripts from the k8s.io/code-generator repo are deprecated (but still work) in favor of kube_codegen.sh in that same repo. Projects which use the old scripts are encouraged to look at adopting the new one. (#117897, @thockin) [SIG API Machinery]
• The feature gate CSIStorageCapacity have been removed and must no longer be referenced in --feature-gates flags. (#118018, @humblec)
• The feature gates CSIMigrationGCE is graduated to GA and were unconditionally enabled have been removed in v1.25, and the entire gcepd package has been removed. (#117055, @cyclinder)
• The feature gates DisableAcceleratorUsageMetrics and PodSecurity that graduated to GA and were unconditionally enabled have been removed in v1.28. (#114068, @cyclinder) [SIG API Machinery, Node, Scheduling and Storage]
• The kubelet podresources endpoint is GA and always enabled. (#116525, @ffromani) [SIG Node]
• The metric apiserver_flowcontrol_current_executing_seats has been introduced as a duplicate of apiserver_flowcontrol_request_concurrency_in_use because the latter has a confusing name and will be removed in a later release. (#118960, @MikeSpreitzer) [SIG API Machinery]
• Updated Cluster Autosaler to version 1.26.1. (#116526, @pacoxu) [SIG Autoscaling and Cloud Provider]
• Updated cri-tools to v1.27.0. (#117545, @saschagrunert)
• Updated setcap image to debian bookworm v1.0.0. (#119247, @saschagrunert)
• Updated cri-tools to v1.26.1. (#116649, @saschagrunert) [SIG Architecture and Release]
• Updated debian-base image to bookworm-v1.0.0. (#119095, @saschagrunert)
• Use table-driven test for TestPerPodSchedulingMetrics. (#118842, @helayoty)
• When retrieving event resources, the reportingController and reportingInstance fields in the event will contain values. (#116506, @HirazawaUi) [SIG API Machinery and Instrumentation]
• [KCCM] drop filtering nodes for the providerID when syncing load balancers, but have changes to the field trigger a re-sync of load balancers. This should ensure that cloud providers which don't specify providerID, can still use the service controller implementation to provision load balancers. (#117602, @alexanderConstantinescu) [SIG Cloud Provider and Network]
• kube-apiserver added two new metrics authorization_attempts_total and authorization_duration_seconds
  that allow users to monitor requests to authorization webhooks, split by result. (#117211, @HirazawaUi)
• kube-apiserver: Improved memory use when performing GetList on the cache. (#116327, @sxllwx)
• kube-controller-manager and cloud-controller-manager have changed the
  name of controllers that can be turned on/off that are passed to the --controllers
  flag (e.g., pod-garbage-collector-controller). The old names (eg podgc) are
  also accepted and aliased to the new names. (#115813, @atiratree)
• kubeadm: Introduced a new feature gate UpgradeAddonsBeforeControlPlane to
  fix a kube-proxy skew policy misalignment. Its default value is false. Upgrade
  of the CoreDNS and kube-proxy addons will now trigger after all the control plane
  instances have been upgraded, unless the fearure gate is set to true. This feature
  gate will be removed in a future release. (#117660, @pacoxu)

Dependencies

Added

• cloud.google.com/go/accessapproval: v1.6.0
• cloud.google.com/go/accesscontextmanager: v1.7.0
• cloud.google.com/go/aiplatform: v1.37.0
• cloud.google.com/go/analytics: v0.19.0
• cloud.google.com/go/apigateway: v1.5.0
• cloud.google.com/go/apigeeconnect: v1.5.0
• cloud.google.com/go/apigeeregistry: v0.6.0
• cloud.google.com/go/appengine: v1.7.1
• cloud.google.com/go/area120: v0.7.1
• cloud.google.com/go/artifactregistry: v1.13.0
• cloud.google.com/go/asset: v1.13.0
• cloud.google.com/go/assuredworkloads: v1.10.0
• cloud.google.com/go/automl: v1.12.0
• cloud.google.com/go/baremetalsolution: v0.5.0
• cloud.google.com/go/batch: v0.7.0
• cloud.google.com/go/beyondcorp: v0.5.0
• cloud.google.com/go/billing: v1.13.0
• cloud.google.com/go/binaryauthorization: v1.5.0
• cloud.google.com/go/certificatemanager: v1.6.0
• cloud.google.com/go/channel: v1.12.0
• cloud.google.com/go/cloudbuild: v1.9.0
• cloud.google.com/go/clouddms: v1.5.0
• cloud.google.com/go/cloudtasks: v1.10.0
• cloud.google.com/go/compute/metadata: v0.2.3
• cloud.google.com/go/compute: v1.19.0
• cloud.google.com/go/contactcenterinsights: v1.6.0
• cloud.google.com/go/container: v1.15.0
• cloud.google.com/go/containeranalysis: v0.9.0
• cloud.google.com/go/datacatalog: v1.13.0
• cloud.google.com/go/dataflow: v0.8.0
• cloud.google.com/go/dataform: v0.7.0
• cloud.google.com/go/datafusion: v1.6.0
• cloud.google.com/go/datalabeling: v0.7.0
• cloud.google.com/go/dataplex: v1.6.0
• cloud.google.com/go/dataproc: v1.12.0
• cloud.google.com/go/dataqna: v0.7.0
• cloud.google.com/go/datastream: v1.7.0
• cloud.google.com/go/deploy: v1.8.0
• cloud.google.com/go/dialogflow: v1.32.0
• cloud.google.com/go/dlp: v1.9.0
• cloud.google.com/go/documentai: v1.18.0
• cloud.google.com/go/domains: v0.8.0
• cloud.google.com/go/edgecontainer: v1.0.0
• cloud.google.com/go/errorreporting: v0.3.0
• cloud.google.com/go/essentialcontacts: v1.5.0
• cloud.google.com/go/eventarc: v1.11.0
• cloud.google.com/go/filestore: v1.6.0
• cloud.google.com/go/functions: v1.13.0
• cloud.google.com/go/gaming: v1.9.0
• cloud.google.com/go/gkebackup: v0.4.0
• cloud.google.com/go/gkeconnect: v0.7.0
• cloud.google.com/go/gkehub: v0.12.0
• cloud.google.com/go/gkemulticloud: v0.5.0
• cloud.google.com/go/gsuiteaddons: v1.5.0
• cloud.google.com/go/iam: v0.13.0
• cloud.google.com/go/iap: v1.7.1
• cloud.google.com/go/ids: v1.3.0
• cloud.google.com/go/iot: v1.6.0
• cloud.google.com/go/kms: v1.10.1
• cloud.google.com/go/language: v1.9.0
• cloud.google.com/go/lifesciences: v0.8.0
• cloud.google.com/go/logging: v1.7.0
• cloud.google.com/go/longrunning: v0.4.1
• cloud.google.com/go/managedidentities: v1.5.0
• cloud.google.com/go/maps: v0.7.0
• cloud.google.com/go/mediatranslation: v0.7.0
• cloud.google.com/go/memcache: v1.9.0
• cloud.google.com/go/metastore: v1.10.0
• cloud.google.com/go/monitoring: v1.13.0
• cloud.google.com/go/networkconnectivity: v1.11.0
• cloud.google.com/go/networkmanagement: v1.6.0
• cloud.google.com/go/networksecurity: v0.8.0
• cloud.google.com/go/notebooks: v1.8.0
• cloud.google.com/go/optimization: v1.3.1
• cloud.google.com/go/orchestration: v1.6.0
• cloud.google.com/go/orgpolicy: v1.10.0
• cloud.google.com/go/osconfig: v1.11.0
• cloud.google.com/go/oslogin: v1.9.0
• cloud.google.com/go/phishingprotection: v0.7.0
• cloud.google.com/go/policytroubleshooter: v1.6.0
• cloud.google.com/go/privatecatalog: v0.8.0
• cloud.google.com/go/pubsublite: v1.7.0
• cloud.google.com/go/recaptchaenterprise/v2: v2.7.0
• cloud.google.com/go/recommendationengine: v0.7.0
• cloud.google.com/go/recommender: v1.9.0
• cloud.google.com/go/redis: v1.11.0
• cloud.google.com/go/resourcemanager: v1.7.0
• cloud.google.com/go/resourcesettings: v1.5.0
• cloud.google.com/go/retail: v1.12.0
• cloud.google.com/go/run: v0.9.0
• cloud.google.com/go/scheduler: v1.9.0
• cloud.google.com/go/secretmanager: v1.10.0
• cloud.google.com/go/security: v1.13.0
• cloud.google.com/go/securitycenter: v1.19.0
• cloud.google.com/go/servicedirectory: v1.9.0
• cloud.google.com/go/shell: v1.6.0
• cloud.google.com/go/spanner: v1.45.0
• cloud.google.com/go/speech: v1.15.0
• cloud.google.com/go/storagetransfer: v1.8.0
• cloud.google.com/go/talent: v1.5.0
• cloud.google.com/go/texttospeech: v1.6.0
• cloud.google.com/go/tpu: v1.5.0
• cloud.google.com/go/trace: v1.9.0
• cloud.google.com/go/translate: v1.7.0
• cloud.google.com/go/video: v1.15.0
• cloud.google.com/go/videointelligence: v1.10.0
• cloud.google.com/go/vision/v2: v2.7.0
• cloud.google.com/go/vmmigration: v1.6.0
• cloud.google.com/go/vmwareengine: v0.3.0
• cloud.google.com/go/vpcaccess: v1.6.0
• cloud.google.com/go/webrisk: v1.8.0
• cloud.google.com/go/websecurityscanner: v1.5.0
• cloud.google.com/go/workflows: v1.10.0
• github.com/alecthomas/kingpin/v2: v2.3.2
• github.com/antlr/antlr4/runtime/Go/antlr/v4: 8188dc5
• github.com/google/gnostic-models: v0.6.8
• github.com/googleapis/enterprise-certificate-proxy: v0.2.3
• github.com/xhit/go-str2duration/v2: v2.1.0
• go.etcd.io/gofail: v0.1.0
• google.golang.org/genproto/googleapis/api: dd9d682
• google.golang.org/genproto/googleapis/rpc: 28d5490

Changed

• cloud.google.com/go/bigquery: v1.8.0 → v1.50.0
• cloud.google.com/go/datastore: v1.1.0 → v1.11.0
• cloud.google.com/go/firestore: v1.1.0 → v1.9.0
• cloud.google.com/go/pubsub: v1.3.1 → v1.30.0
• cloud.google.com/go: v0.97.0 → v0.110.0
• github.com/Azure/azure-sdk-for-go: v55.0.0+incompatible → v68.0.0+incompatible
• github.com/Azure/go-autorest/autorest/adal: v0.9.20 → v0.9.23
• github.com/Azure/go-autorest/autorest/validation: v0.1.0 → v0.3.1
• github.com/Azure/go-autorest/autorest: v0.11.27 → v0.11.29
• github.com/Microsoft/go-winio: v0.4.17 → v0.6.0
• github.com/alecthomas/units: f65c72e → b94a6e3
• github.com/cenkalti/backoff/v4: v4.1.3 → v4.2.1
• github.com/census-instrumentation/opencensus-proto: v0.2.1 → v0.4.1
• github.com/cespare/xxhash/v2: v2.1.2 → v2.2.0
• github.com/cilium/ebpf: v0.7.0 → v0.9.1
• github.com/cncf/udpa/go: 04548b0 → c52dc94
• github.com/cncf/xds/go: cb28da3 → 06c439d
• github.com/cockroachdb/datadriven: bf6692d → v1.0.2
• github.com/container-storage-interface/spec: v1.7.0 → v1.8.0
• github.com/containerd/cgroups: v1.0.1 → v1.1.0
• github.com/containerd/ttrpc: v1.1.0 → v1.2.2
• github.com/coredns/caddy: v1.1.0 → v1.1.1
• github.com/coreos/go-oidc: v2.1.0+incompatible → v2.2.1+incompatible
• github.com/coreos/go-semver: v0.3.0 → v0.3.1
• github.com/coreos/go-systemd/v22: v22.4.0 → v22.5.0
• github.com/docker/distribution: v2.8.1+incompatible → v2.8.2+incompatible
• github.com/dustin/go-humanize: v1.0.0 → v1.0.1
• github.com/envoyproxy/go-control-plane: 49ff273 → v0.10.3
• github.com/envoyproxy/protoc-gen-validate: v0.1.0 → v0.9.1
• github.com/evanphx/json-patch: v4.12.0+incompatible → v5.6.0+incompatible
• github.com/frankban/quicktest: v1.11.3 → v1.14.0
• github.com/fvbommel/sortorder: v1.0.1 → v1.1.0
• github.com/go-kit/log: v0.2.0 → v0.2.1
• github.com/go-logr/logr: v1.2.3 → v1.2.4
• github.com/go-openapi/jsonreference: v0.20.1 → v0.20.2
• github.com/go-task/slim-sprig: 348f09d → 52ccab3
• github.com/gofrs/uuid: v4.0.0+incompatible → v4.4.0+incompatible
• github.com/golang-jwt/jwt/v4: v4.4.2 → v4.5.0
• github.com/google/cadvisor: v0.47.1 → v0.47.3
• github.com/google/cel-go: v0.12.6 → v0.16.0
• github.com/google/gofuzz: v1.1.0 → v1.2.0
• github.com/googleapis/gax-go/v2: v2.1.1 → v2.7.1
• github.com/inconshreveable/mousetrap: v1.0.1 → v1.1.0
• github.com/kr/pretty: v0.3.0 → v0.3.1
• github.com/matttproud/golang_protobuf_extensions: v1.0.2 → v1.0.4
• github.com/mitchellh/go-wordwrap: v1.0.0 → v1.0.1
• github.com/mitchellh/mapstructure: v1.4.1 → v1.1.2
• github.com/onsi/ginkgo/v2: v2.9.1 → v2.9.4
• github.com/onsi/gomega: v1.27.4 → v1.27.6
• github.com/opencontainers/runc: v1.1.4 → v1.1.7
• github.com/prometheus/client_golang: v1.14.0 → v1.16.0
• github.com/prometheus/client_model: v0.3.0 → v0.4.0
• github.com/prometheus/common: v0.37.0 → v0.44.0
• github.com/prometheus/procfs: v0.8.0 → v0.10.1
• github.com/seccomp/libseccomp-golang: f33da4d → v0.10.0
• github.com/spf13/cobra: v1.6.0 → v1.7.0
• github.com/stretchr/testify: v1.8.1 → v1.8.2
• github.com/vishvananda/netns: v0.0.2 → v0.0.4
• github.com/xlab/treeprint: v1.1.0 → v1.2.0
• go.etcd.io/bbolt: v1.3.6 → v1.3.7
• go.etcd.io/etcd/api/v3: v3.5.7 → v3.5.9
• go.etcd.io/etcd/client/pkg/v3: v3.5.7 → v3.5.9
• go.etcd.io/etcd/client/v2: v2.305.7 → v2.305.9
• go.etcd.io/etcd/client/v3: v3.5.7 → v3.5.9
• go.etcd.io/etcd/pkg/v3: v3.5.7 → v3.5.9
• go.etcd.io/etcd/raft/v3: v3.5.7 → v3.5.9
• go.etcd.io/etcd/server/v3: v3.5.7 → v3.5.9
• go.opencensus.io: v0.23.0 → v0.24.0
• go.starlark.net: 8dd3e2e → a134d8f
• go.uber.org/atomic: v1.7.0 → v1.10.0
• go.uber.org/multierr: v1.6.0 → v1.11.0
• golang.org/x/crypto: v0.1.0 → v0.11.0
• golang.org/x/exp: 6cc2880 → a9213ee
• golang.org/x/mod: v0.9.0 → v0.10.0
• golang.org/x/net: v0.8.0 → v0.13.0
• golang.org/x/oauth2: ee48083 → v0.8.0
• golang.org/x/sync: v0.1.0 → v0.2.0
• golang.org/x/sys: v0.6.0 → v0.10.0
• golang.org/x/term: v0.6.0 → v0.10.0
• golang.org/x/text: v0.8.0 → v0.11.0
• golang.org/x/time: 90d013b → v0.3.0
• golang.org/x/tools: v0.7.0 → v0.8.0
• google.golang.org/api: v0.60.0 → v0.114.0
• google.golang.org/genproto: c8bf987 → 0005af6
• google.golang.org/grpc: v1.51.0 → v1.54.0
• google.golang.org/protobuf: v1.28.1 → v1.30.0
• gopkg.in/gcfg.v1: v1.2.0 → v1.2.3
• gopkg.in/natefinch/lumberjack.v2: v2.0.0 → v2.2.1
• gopkg.in/warnings.v0: v0.1.1 → v0.1.2
• k8s.io/klog/v2: v2.90.1 → v2.100.1
• k8s.io/kube-openapi: 15aac26 → 2695361
• k8s.io/utils: a36077c → d93618c
• sigs.k8s.io/apiserver-network-proxy/konnectivity-client: v0.1.1 → v0.1.2
• sigs.k8s.io/kustomize/api: v0.13.2 → 6ce0bf3
• sigs.k8s.io/kustomize/cmd/config: v0.11.1 → v0.11.2
• sigs.k8s.io/kustomize/kustomize/v5: v5.0.1 → 6ce0bf3
• sigs.k8s.io/kustomize/kyaml: v0.14.1 → 6ce0bf3

Removed

• github.com/antlr/antlr4/runtime/Go/antlr: v1.4.10
• github.com/certifi/gocertifi: 2c3bb06
• github.com/cockroachdb/errors: v1.2.4
• github.com/cockroachdb/logtags: eb05cc2
• github.com/docopt/docopt-go: ee0de3b
• github.com/getsentry/raven-go: v0.2.0
• github.com/google/gnostic: v0.5.7-v3refs

v1.28.0-rc.1

Downloads for v1.28.0-rc.1

Source Code

filename | sha512 hash
-------- | -----------
kubernetes.tar.gz | efc4ec914eb4e1147cdbadb9a5dccc4608a983ba6308c85d4c2e8e1c984f35c12e04b027d0a0f6e07c2371fae9aa4879b4831158e7cfe77887da7e20778e717b
kubernetes-src.tar.gz | cfdd470979b447dab1678c03bd9bc7745ef7d6907a043d5180e96494d4a5b91b4d8a08b09726e15cda4668437cc296528df646f5f58c870af8134312cf8851ba

Client Binaries

filename | sha512 hash
-------- | -----------
kubernetes-client-darwin-amd64.tar.gz | ca12f940ef00fc38152dae75d8f817b03be4d59c7a8d64a80a515fbbf78d526a2b98311efbc9a8d34361b70ba0b07156cff7fbb6c19dc503c7c16e0dfc8e3ec0
kubernetes-client-darwin-arm64.tar.gz | 9d1d0aaedd698a4c5ad5ed2903e8492e52e6f16f858b8d186129edb5c9b199d4352e443c8aba67f58db7fcf950162ffbb4f7211570a0c4be684656ad5ada42bf
kubernetes-client-linux-386.tar.gz | 59e1c0692821eb64b59147baf43985226ae2223fff66981a2a13d5d60d5e102faf7c5cecfa5b8ec1c5a48c9a9fed35223f8ebbc93ac972ea949f2a3096f64672
kubernetes-client-linux-amd64.tar.gz | a348ad33f936e075083e2ffec4405a726984df8c522e10d34ad65b64eee53902fb6483903581c30b047872fd130cf24f6cdc193458fcf7d5774364bf78c1c982
kubernetes-client-linux-arm.tar.gz | 3069b3000445218a98b7d11bd196cbfb84aed16dcbd16adff88935bb01a87f8da29cc4824de4c8af7ddc2050134e2c3467408218fc7209700c0e1c0aec2d3ced
kubernetes-client-linux-arm64.tar.gz | d59ebea7b7d78ee1ef59317073a5a4b9e513b9f43026ffc1b7745211e8cd9de738a05d1fa2c29501d3cee24252732c3348f109f2b7c1e7425a4eff46cf1b4654
kubernetes-client-linux-ppc64le.tar.gz | 2490266ee44469c56bd85f86774668bc9a6d6f2d4f21ddeb95da8eb638f743501e9ed3646c817db0c746730f93b381b6c5ae46d25d9dbadc78d3ca8f89eccfd6
kubernetes-client-linux-s390x.tar.gz | a843bc8df85594ab5fca9e1e17997e14cdf8e45ef2e74165222b6dc8d26a8f85d85a972a5c3e1740f3eb6d3647b81e3dfb66787cdac6dcd42a59c2f5507f6031
kubernetes-client-windows-386.tar.gz | 1ce7d2802cb9ebbc61f68ae3a8380056fc039e9959ea999f3d7da1254b33225809932a9679b2a0f96ca4adad73aa4aedba3ec9f20182899f62ff59133e48f4d1
kubernetes-client-windows-amd64.tar.gz | 85986399b437aa48d276301a7a06b17e4004d5423dec2faf699a3a377dd28f3e734b0655848168407fb25b6898389daf45f0ad695519e1f3f31586e9a8586531
kubernetes-client-windows-arm64.tar.gz | ac18cf32f0273470841024a762d7ded78993128bada439340d0f4c604af6d7001971f3075437e65471ab7ae89d15fd82f2689b6d47da681dcc8779c277a9cea5

Server Binaries

filename | sha512 hash
-------- | -----------
kubernetes-server-linux-amd64.tar.gz | 6b5ebfe42050e48f108da43275cdfb0b5ec652867d12a632bf5ed4b00482efd2470184028bf94b36a1f05c5a70ad1057f334483461f9212bdb48dcac6b169600
kubernetes-server-linux-arm64.tar.gz | 09b1a44182ab30a4740b703cae5f46ce4cb4af9716ed1f2cb2a4dc59c9baa2eaa402b01167b04c6801b550035ba9f939d4d1209689363daffe870dd2f44e4528
kubernetes-server-linux-ppc64le.tar.gz | 09ba87bfb42f7f710b446a588d2047fe6aea26df171aacc3157c3fa4c9e718856ad3efc45b0050d35a9153e94d5da81c632ddaec71663d30c5d43284292b305f
kubernetes-server-linux-s390x.tar.gz | 42ae7f05cb279e57ef89570b1596759cc771663ceb72f358a9119c91b981b99335b46a887f59f8a8727303366c3111bd4696817343ddbee3ee02811bd6022e4c

Node Binaries

filename | sha512 hash
-------- | -----------
kubernetes-node-linux-amd64.tar.gz | 4a0c7ae5ce52698087eaca1ec108ca5c1333ddc607a5fcb63d5e65cde17e3c8e64037905da02656e4a663037be1b00441754af4563c5eed1ec8ab57bf692c4ed
kubernetes-node-linux-arm64.tar.gz | aba8ddbc9cd9cf0e2fd5eaac8bee2490135c31fcd43751dd5e4438eca813372104e99f34517acfdd2abddf3a28cffef4cc42eb9bfecd76b50d89adff5675f32b
kubernetes-node-linux-ppc64le.tar.gz | d16d79c2f5680d31aa6fcc659de210632d0ed761b6c4c067ab5976fe41360a9cc7a75cad545e04831bf9d2b8669523dd9cf4c756337c5328ff10a8d61a5301bc
kubernetes-node-linux-s390x.tar.gz | f7bb37de732b35db011f1d2f52ac461c7f912de39bb16c13a638b2dcd160876c6d1e278d36fdee07d8598b69b30e33f7c9bc980b6b25651e4b74cf3517514371
kubernetes-node-windows-amd64.tar.gz | 86114fbfa8980e678bddb3d01290b5b3158d92ffdc92b970d1a224fff1f7914415c9adb3b663f5f036d5abb0aad95c96c1a819b309e157aba9392a6c77e65ff5

Container Images

All container images are available as manifest lists and support the described
architectures. It is also possible to pull a specific architecture directly by
adding the "-$ARCH" suffix to the container image name.

name | architectures
---- | -------------
registry.k8s.io/conformance:v1.28.0-rc.1 | amd64, arm64, ppc64le, s390x
registry.k8s.io/kube-apiserver:v1.28.0-rc.1 | amd64, arm64, ppc64le, s390x
registry.k8s.io/kube-controller-manager:v1.28.0-rc.1 | amd64, arm64, ppc64le, s390x
registry.k8s.io/kube-proxy:v1.28.0-rc.1 | amd64, arm64, ppc64le, s390x
registry.k8s.io/kube-scheduler:v1.28.0-rc.1 | amd64, arm64, ppc64le, s390x
registry.k8s.io/kubectl:v1.28.0-rc.1 | amd64, arm64, ppc64le, s390x

Changelog since v1.28.0-rc.0

Changes by Kind

API Change

• Aggregated discovery now returns responseKind: {} for resources which are missing group/version/kind information, to ensure compatibility with v0.26.0-v0.26.3 clients. (#119835, @liggitt) [SIG API Machinery and Testing]
• Fix CustomResourceDefinition status.storedVersions validation error messages. (#119653, @sttts) [SIG API Machinery]
• Kube-proxy in Kubernetes >= 1.28 up until v1.28.0-beta.0 ignored the -v command line flag when combined with --config. (#119867, @pohly) [SIG Network]

Feature

• Bump distroless-iptables to 0.2.7 based on Go 1.20.7 (#119818, @jeremyrickard) [SIG Testing]
• Kubernetes is now built with Go 1.20.7 (#119804, @jeremyrickard) [SIG Release and Testing]

Bug or Regression

• Fixes issue https://github.com/kubernetes-sigs/cloud-provider-azure/issues/4230 and removes the additional filtering on NotReady nodes by the azure cloud provider code (#119128, @alexanderConstantinescu) [SIG Cloud Provider]
• Kube-scheduler: Fine-grained tracking of events (introduced in 1.28) suffered from a data race when binding fails. (#119729, @pohly) [SIG Scheduling]
• Revert kubelet prober metrics pod tag to include actual pod name (#118549, @a7i) [SIG Node]
• When the cluster size is small and the scheduler doesn't get unscheduled Pods frequently, the scheduler doesn't try to reschedule Pods in some cases. (#119784, @sanposhiho) [SIG Scheduling and Testing]

Dependencies

Added

Nothing has changed.

Changed

• golang.org/x/net: v0.12.0 → v0.13.0

Removed

Nothing has changed.

v1.28.0-rc.0

Downloads for v1.28.0-rc.0

Source Code

filename | sha512 hash
-------- | -----------
kubernetes.tar.gz | bd3feaf924371be8fe3e2130c6cac5fd5fd3c90d42be383e076c16160c95ec48668b5b330e0742d562a3b0eae18eda71bab76dff5e2aebad61513c2be6b251b6
kubernetes-src.tar.gz | 9f3fbb4c624124bf82473028e9c21a123f525e9dc8a224ede4f00fbf0630ae812ba58d56be69aa45a2b039da1deb4ce9052061b40699945f9fb88bb59fcb3977

Client Binaries

filename | sha512 hash
-------- | -----------
kubernetes-client-darwin-amd64.tar.gz | fabc769eef82c242d905cee81d7e876280adeffa95208ddd4d1c0de69e82d775afef984cc9b3b375ee2072e8dceb846da105c76c94e6d323f9778c9e9c0b49ba
kubernetes-client-darwin-arm64.tar.gz | 0a321eaa366d73feab93bab77e56dbd8ee2fde298766a8f7e37b98f6053c4e86ae77f64561bd083cd5f3b4f61d727de013d6ef74a0bd0c35b7afb8cb110a063c
kubernetes-client-linux-386.tar.gz | a8720f87b0eafefca413547e3f603660d147b9198eabac03ac59a57a70fc011c48daaef4cf1d63a974578e8b20f98e6f3dca7997f6feaa009944e16ec47ea8c9
kubernetes-client-linux-amd64.tar.gz | 918dc97380ebe56c16de8131d35f9ddc21b2196f8b6b0b24361fa4a23a2cf56c75edf4555eccbad3453663c007bd51d147e0a589f933e0759410879e2aeadfd8
kubernetes-client-linux-arm.tar.gz | c1f87fff085884632cb1b60c72f4cf168079bff4150270e67855b1211a1240e3252e6791dfb61672a2ffbe4314b360b917657608b3c65d661871852ba84e8ca3
kubernetes-client-linux-arm64.tar.gz | 2f33cb523472e162dc206edee4e33903a4550d3f73adbe327c7c34f8084b44dd0ed6b6b28c9d85eca4e6e2a1d1124c9dbc0d8a4a14abc7b810a7e4085f5e97bc
kubernetes-client-linux-ppc64le.tar.gz | 9edc3618d12e4480ccb448e928017bd9c3f1e8ad16bc83cddea4a73c81ab2d7a5085bbedf2a0324a9377d17faa7168e2d5c27a7de5bed8e07809b1227c4b9079
kubernetes-client-linux-s390x.tar.gz | 53a31035114aaa7d837dcdc7663a9ee91e6d49d549eea6c7337a6f3a4325b34a6931e65ce471a758bed152a92adb434581084e2810b16c8143582501b48e4363
kubernetes-client-windows-386.tar.gz | 8158c3947c5838fef84b6427fb27e6cf2375344f6d27bd1a2d0563d3a242bc445278968bcf36b7657a4db7322b2a9d5aad028480c6bb5fbf2faf3a2dddad931e
kubernetes-client-windows-amd64.tar.gz | aa809425d557fc9323748a0484f9b59f9f6e089ab6256da5690014efbda9a9c1a96110a3511f930e4e2714315005bd803ea059b1a5221a825f109a69d6c60967
kubernetes-client-windows-arm64.tar.gz | f72acfa3ac61cbf7fc2b612a9c8c9d19de42488752120e6b4d69fafa35beda00bfbe9dc839430b3987ff5e9737930d0a9fae867ab35824f0a0eef47b6496404d

Server Binaries

filename | sha512 hash
-------- | -----------
kubernetes-server-linux-amd64.tar.gz | a3ca2b91e204f6bcf84e1d54412ca6c908a1e02a48b8022da821bfd4fe136b565a980da3fef2270a34ca637ff4ff306cd3b09760556602db15a7b8a3dfead0d1
kubernetes-server-linux-arm64.tar.gz | 40e6aea2c1fa0f9f6c135363991216d80b927c6aabcf30fc3288512c38fd75b8fff868770209cde8016a2a33dee7ad862709840c7a040f6203240b06bc2e5c5d
kubernetes-server-linux-ppc64le.tar.gz | da41491b39fb8a1a2f7619f14ff1165ff0d467bef82348e71bfd31d481678dff27b03a01d3fa2deccdf04227a0fe7c9593d8d7b9e7745070afcb53dfd70b2bc6
kubernetes-server-linux-s390x.tar.gz | 6deb9a3625f510cdd5dea4ced9648fe563595cddfd2f9223e4acbf3fe74cd2b4283011984105f79794731e81b7b3725a16c964b70f25a4bad76b60b75f54372d

Node Binaries

filename | sha512 hash
-------- | -----------
kubernetes-node-linux-amd64.tar.gz | d4178a311495afe6e9d4a5c4fa9b1d0d17c7a0918305b782406ea5f5fbfcd8c4f60073e02b9aff3c37eb9f4fd331844177e48650b4e489b0a5430da5e00a33a2
kubernetes-node-linux-arm64.tar.gz | 8ee5e87248caca033fafea7ea1470fcc282cc402f6591d9120fd87c4e533bade19e125c1a840d1ebf503fb0eda21096a047571ffde79a6b0263494799bdf042f
kubernetes-node-linux-ppc64le.tar.gz | 022f4804d408788d426176f5dda8046005084261c426766476b429e2d62f60e22509d66e9199350ba501b17f6e6cb72e7e4bd2581b950c85e5c5a05efcb0139c
kubernetes-node-linux-s390x.tar.gz | d9e95ce6e0f886d8a89df8d53f2533c9be2ef733be25a7455cfc89028ee5ed5b7fbdfb81f7c3e452b405dc8b3d4c252f74039d300a0fd01c7b75e7b5f6c0c551
kubernetes-node-windows-amd64.tar.gz | 6bd4809f6b8d1bea8da4565ddda294d39eb3ab424dd47e01a5919e974b33316c64fc21bc8b2c62a05f14db71621951ecc82c5988da11b8190a6e1ee2eec79cd5

Container Images

All container images are available as manifest lists and support the described
architectures. It is also possible to pull a specific architecture directly by
adding the "-$ARCH" suffix to the container image name.

name | architectures
---- | -------------
registry.k8s.io/conformance:v1.28.0-rc.0 | amd64, arm64, ppc64le, s390x
registry.k8s.io/kube-apiserver:v1.28.0-rc.0 | amd64, arm64, ppc64le, s390x
registry.k8s.io/kube-controller-manager:v1.28.0-rc.0 | amd64, arm64, ppc64le, s390x
registry.k8s.io/kube-proxy:v1.28.0-rc.0 | amd64, arm64, ppc64le, s390x
registry.k8s.io/kube-scheduler:v1.28.0-rc.0 | amd64, arm64, ppc64le, s390x
registry.k8s.io/kubectl:v1.28.0-rc.0 | amd64, arm64, ppc64le, s390x

Changelog since v1.28.0-beta.0

Changes by Kind

API Change

• PersistentVolumes have a new LastPhaseTransitionTime field which holds a timestamp of when the volume last transitioned its phase. (#116469, @RomanBednar) [SIG API Machinery, Apps, Auth, Node, Release, Storage and Testing]
• Promoted API groups ValidatingAdmissionPolicy and ValidatingAdmissionPolicyBinding to v1beta1. (#118644, @alexzielenski) [SIG API Machinery, Apps and Testing]
• Promoted the feature gate ValidtaingAdmissionPolicy to beta and it is turned off by default. (#119409, @alexzielenski) [SIG API Machinery, Apps, Auth, Instrumentation, Node, Release, Storage and Testing]
• Changed how KMS v2 encryption at rest can generate data encryption keys. When you enable the KMSv2KDF feature gate (off by default), KMS v2 uses a key derivation function to generate single use data encryption keys from a secret seed combined with some random data. This eliminates the need for a counter based nonce while avoiding nonce collision concerns associated with AES-GCM's 12 byte nonce. (#118828, @enj) [SIG API Machinery, Auth and Testing]

Feature

• Add implementation for PodRecreationPolicy to wait for creation of pods once the existing ones are fully terminated. (#117015, @kannon92) [SIG API Machinery, Apps and Testing]

Dependencies

Added

Nothing has changed.

Changed

Nothing has changed.

Removed

Nothing has changed.

v1.28.0-beta.0

Downloads for v1.28.0-beta.0

Source Code

filename | sha512 hash
-------- | -----------
kubernetes.tar.gz | 5a4575157380ff5dd66fd87e7045f2f57ed0db59513bfef57ee768a7a98f855faa06503a7480e77cdf5128fe66461a6c91f0705f8148347f903342f45b65f8da
kubernetes-src.tar.gz | 8efebf779daf168929dd2798d2e52750d09339a17256091b4315b1de82167b26388725a60bc3525468b1a23656932faf1c45ba6957df0bd2b3f48bafc8b62138

Client Binaries

filename | sha512 hash
-------- | -----------
kubernetes-client-darwin-amd64.tar.gz | 956bd780cb36815ae8969e345d4f6004740167f3e5e3a1d1b1deda254ac2b167371b7c9e79497bd01e3f11d2e2f8e1c35f8fdc3114f08324470635cff1efab20
kubernetes-client-darwin-arm64.tar.gz | 68ebd65edb40e23c5e70b6666fea34a774ccd66313dc884e28a8032cfd8c166c1a2dc66c635a61b0d568fe825b208bfd32a040e3eded536f71617acac625f3ea
kubernetes-client-linux-386.tar.gz | 0291c01019e47af1631f3f90f46aa92e4572301fe2411586fb2d69291fdf113a8ff78531f51530d05c6113e28e0e69e23f1d2e2143f832f6b8f77a133b09e493
kubernetes-client-linux-amd64.tar.gz | bd9a98bd35ed9bb113399e50456da7008629e060381a7f6b9071fd7bcd498cb252da2824376662a4d4d4cae637feb90f3768414751e7ec0339c6c9711f6aceda
kubernetes-client-linux-arm.tar.gz | 311b3c1a42d68fa337f6e8b2290817196c97024535ee94f2180971e09ca78c9037b41fca48c1eda1fa75079631ea8805f0c6a173e35fe4a9762d13bf3bdf1c58
kubernetes-client-linux-arm64.tar.gz | 039491fddc63e279821ff5287cada111b5c94dc0389d697adc2503f77905a74cda1a67dd52d4b42be0141896d04cfb18f3dfc0c387620d9a9fb3a7f2aa5b399e
kubernetes-client-linux-ppc64le.tar.gz | 70e4011dce6777c511a69872a1e015b3a2137cea100cd6b988946ea227bd05827f4223a44c9d24433043cc414d3746465603dc4de2e84128e7689712dd29b00d
kubernetes-client-linux-s390x.tar.gz | 51ea51fbcdcf7f2c9c4c97b6f28ceb42753e52ccf69318c05befa77b94d9c17e871061b4d03dbc12632663efed161424b3dba97ac6df46cf27d6deb1a0c011fb
kubernetes-client-windows-386.tar.gz | e069121289664d53a6570d90ac2d66911d18cf8c47461f5e8e2cd52ddd651a0e119327d23455b9c273e979378379d27e5cf102c7f8b8c98871c9ef9c7e790e52
kubernetes-client-windows-amd64.tar.gz | f19a06ad641a282b9895673a0628cc937222b53d9b852fd7543de01294d403100d0bcd4659fbf73bf8cdc55a8e3c7f494991db2b9f4d8bc63446b7810232d3aa
kubernetes-client-windows-arm64.tar.gz | 7bd6a6d6ae7b487aa35c162532e9b59e184aac0fb6b65c61b2af06217007f78aeb5d0bed039983c534358152155739a6e30f533488c49a13d3c81d54eea6b8c6

Server Binaries

filename | sha512 hash
-------- | -----------
kubernetes-server-linux-amd64.tar.gz | 7cd83dc6e7013ef8638965fb3e30f8ba122f045d987029da345c185662bea824fc6a3fae34ff549c457638daf703833312893180538552d194fcc7f4fb0642fd
kubernetes-server-linux-arm64.tar.gz | 07e9304f6864f4334b9710ad1a130044c9b1349bf9e47e5c9857b688322abe5babf45a95cdbad3f2650b5447c11864edc3d50fa86de5d485e84730260efbdc8e
kubernetes-server-linux-ppc64le.tar.gz | 84f7c2b7021bd136d0c9013b349d63102433f7485552a0f19cb61f4630f256aef1f99a54f9da14b5d6c242778488c5539e0c358b9e421aafaf746ce783773e9b
kubernetes-server-linux-s390x.tar.gz | 80263d622d219ae687ea24d9b552c5e6fcb920edc6adc73fe5c742cbb34db08a045f52babb0e6c5acfab98616b9f9f2c87150db9a04f5799836b8c8fd0709f31

Node Binaries

filename | sha512 hash
-------- | -----------
kubernetes-node-linux-amd64.tar.gz | bb24b87e1971fde06882d4a2b91b2663ee9daad1e1f8f31f457eaac235b26466ec2413d947d06803b8fc9c356e56e77f7ed31b1f021eae0fdb3df426bc610717
kubernetes-node-linux-arm64.tar.gz | 91aa0771cf8f6615655aeda2484d967f53dbe10cee7ac724be23570d5ca60e3fe11e354cd8b715b882ba20534dab67fee505cb2cd6df1c90d124f778eadff67b
kubernetes-node-linux-ppc64le.tar.gz | e30e0efdd8ee02b3ee9c4da1e106a41ffdff7606401f44647e6bd03ebf4ef1900bf7c7d5a2382412a0e2b4bd7013e04e3baa08637d4dbe2b2993fcdab7e2378d
kubernetes-node-linux-s390x.tar.gz | e011016917d19bc5b84a76899a3eb6d7e2a6bf270e2d799ba77a9c90daabaf2055655ed0b6a62f2b9e5edfbd4b902e6e4a2408d2dc5c63a19a706d3f838a3864
kubernetes-node-windows-amd64.tar.gz | b9f365607de7112b2e62458462f1261905b1731d40c125db228bbc5aa48cc108872552db3c873702df0607bc0eb1996a1dca16cd27526f002bf1842e2eefc4ef

Container Images

All container images are available as manifest lists and support the described
architectures. It is also possible to pull a specific architecture directly by
adding the "-$ARCH" suffix to the container image name.

name | architectures
---- | -------------
registry.k8s.io/conformance:v1.28.0-beta.0 | amd64, arm64, ppc64le, s390x
registry.k8s.io/kube-apiserver:v1.28.0-beta.0 | amd64, arm64, ppc64le, s390x
registry.k8s.io/kube-controller-manager:v1.28.0-beta.0 | amd64, arm64, ppc64le, s390x
registry.k8s.io/kube-proxy:v1.28.0-beta.0 | amd64, arm64, ppc64le, s390x
registry.k8s.io/kube-scheduler:v1.28.0-beta.0 | amd64, arm64, ppc64le, s390x
registry.k8s.io/kubectl:v1.28.0-beta.0 | amd64, arm64, ppc64le, s390x

Changelog since v1.28.0-alpha.4

Changes by Kind

Deprecation

• Changed kubectl version default output to be identical to what kubectl version --short printed,
  and remove --short flag entirely. (#116720, @soltysh) [SIG CLI and Testing]
• Deprecated support for CSI migration of Ceph RBD volumes.

Users who were relying on Kubernetes' ability to migrate to an out-of-tree storage driver should complete
that migration before the support for it is removed. (#118303, @carlory) [SIG Storage]
- KMSv1 is deprecated and will only receive security updates going forward. Use KMSv2 instead. In the future, set --feature-gates=KMSv1=true to use the deprecated KMSv1 feature. (#119007, @aramase) [SIG API Machinery and Auth]
- The deprecated flag --lock-object-namespace and --lock-object-name have been removed from kube-scheduler. Please use --leader-elect-resource-namespace and --leader-elect-resource-name or ComponentConfig instead to configure those parameters. (#119130, @SataQiu) [SIG Scheduling]

API Change

• A CDIDevice field is includes in the Device Plugin's ContainerAllocateResponse. This field maps to the CDIDevice field in the CRI protocol. (#118254, @elezar) [SIG Node and Testing]
• Add new annotation batch.kubernetes.io/cronjob-scheduled-timestamp to Job objects scheduled from CronJobs. (#118137, @helayoty) [SIG Apps]
• Add podReplacementPolicy and terminating field to job api (#119301, @kannon92) [SIG API Machinery and Apps]
• Added fields reason and fieldPath into CRD validation rules to allow users to specify reason and field path when validation failed. (#118041, @cici37) [SIG API Machinery]
• Added namespace access support to the CEL expressions of ValidatingAdmissionPolicy via a namespaceObject
  variable with expressions. (#118267, @cici37) [SIG API Machinery and Testing]
• Adds new CRDValidationRatcheting alpha feature. During a PATCH or UPDATE Validation Ratcheting discards errors thrown by unchanged portions of the resource from most OpenAPI schema validations. (#118990, @alexzielenski) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node and Storage]
• Adds new namespaceParamRef to admissionregistration.k8s.io/v1alpha1.ValidatingAdmissionPolicy (#119215, @alexzielenski) [SIG API Machinery and Testing]
• Extend the Job API for alpha version of BackoffLimitPerIndex (#119294, @mimowo) [SIG API Machinery and Apps]
• Graduate AdmissionWebhookMatchCondition feature to beta (#119380, @a-hilaly) [SIG API Machinery]
• In the API Priority and Fairness feature, priority levels that are exempt from limitation can now be given a nominal and a lendable concurrency and their dispatching borrows from the concurrency limits of the other priority levels. For details see https://github.com/kubernetes/enhancements/tree/master/keps/sig-api-machinery/1040-priority-and-fairness#dispatching . (#118782, @MikeSpreitzer) [SIG API Machinery]
• Indexed Job pods now have the pod completion index set as a pod label. (#118883, @danielvegamyhre) [SIG Apps]
• Kube-proxy: add '--logging-format' flag to support structured logging (#117800, @cyclinder) [SIG API Machinery, Architecture, Instrumentation and Network]
• Registered_metric_total, disabled_metric_total, hidden_metric_total & kubernetes_feature_enabled are promoted to BETA stability. (#119264, @logicalhan) [SIG API Machinery, Architecture, Cluster Lifecycle and Instrumentation]
• Removed resizeStatus enum from pvc.Status and replaced with AllocatedResourceStatus (#116335, @gnufied) [SIG API Machinery, Apps, Auth, Node, Storage and Testing]
• StatefulSet pods now have the pod index set as a pod label statefulset.kubernetes.io/pod-index. (#119232, @danielvegamyhre) [SIG Apps]
• Support BackoffLimitPerIndex in Jobs (#118009, @mimowo) [SIG API Machinery, Apps and Testing]
• Support for proxying a request to a peer kube-apiserver if the local apiserver is not able to serve it due to version skew or in the case the requested api is disabled on the local apiserver (#117740, @Richabanker) [SIG API Machinery, Apps, Auth, Cloud Provider, Network, Node and Testing]
• The IPTablesOwnershipCleanup feature (KEP-3178) is now GA; kubelet no longer
  creates the KUBE-MARK-DROP chain (which has been unused for several releases)
  or the KUBE-MARK-MASQ chain (which is now only created by kube-proxy). (#119374, @danwinship) [SIG API Machinery, Network and Node]
• The names of ResourceClaims generated from ResourceClaimTemplate are now generated. The base name is still <pod>-<claim name>, but a random suffix will avoid name collisions. (#117351, @pohly) [SIG API Machinery, Apps, Auth, Node, Scheduling and Testing]
• The new feature gate "SidecarContainers" is now available. This feature introduces sidecar containers, a new type of init container that starts before other containers but remains running for the full duration of the pod's lifecycle and will not block pod termination. (#116429, @gjkim42) [SIG API Machinery, Apps, Node, Scheduling and Testing]

Feature

• A ValidatingAdmissionPolicy now has its messageExpression field checked against resolved types. (#119209, @jiahuif) [SIG API Machinery]
• Add ConsistentListFromCache feature gate that allows apiserver to serve consistent lists from cache (#118508, @serathius) [SIG API Machinery, Instrumentation and Testing]
• Add full cgroup v2 swap support for both Limited and Unlimited swap.

When LimitedSwap is enabled the swap limit would be automatically calculated for
Burstable QoS pods. For Best-Effort / Guaranteed QoS pods, swap would be disabled.

Containers with memory requests equal to their memory limits also won't have
swap access, and it is a way to opt-out of swap for a single container.

The formula for the swap limit for Burstable QoS pods is:
(<memory-request>/<node-memory-capacity>)*<node-swap-capacity>.

Support for cgroup v1 is removed. (#118764, @iholder101) [SIG Node and Testing]
- Add handling for pods in podgc for PodReplacementPolicy or PodDisruption (#118772, @kannon92) [SIG Apps and Testing]
- Add reason to metric attachdetach_controller_forced_detaches in the attach detach controller. (#119185, @xing-yang) [SIG Apps and Storage]
- Add swap to stats to Summary API and Prometheus endpoints (stats/summary and /metrics/resource). (#118865, @iholder101) [SIG Node and Testing]
- Added a new command line argument --interactive to kubectl. The new command line argument lets a user confirm deletion requests per resource interactively. (#114530, @ardaguclu) [SIG CLI and Testing]
- Added a new feature gate, SchedulerQueueingHints (enabled by default).
The new feature gate activates a framework for fine-grained filtering of events related to scheduler plugins.
In this release, no default scheduling plugins make use of the hinting framework, so you should not expect any behavior changes. (#119328, @sanposhiho) [SIG Scheduling]
- Adds apiserver_admission_match_condition_evaluation_seconds and apiserver_admission_match_condition_exclusions_total metrics (#119311, @ivelichkovich) [SIG API Machinery]
- Bump distroless-iptables to 0.2.6 based on Go 1.20.6 (#119365, @xmudrii) [SIG Testing]
- CEL authorizer checks no longer raise runtime errors. Calls to "check" will always return a decision object and the authorization error (if any) can be accessed within expressions using the new decision methods "errored" and "error". (#118804, @benluddy) [SIG API Machinery]
- CRI: expose commit memory bytes in container stats specific to Windows (#119238, @kiashok) [SIG Node and Windows]
- Cloud controller manager's node controller now emits timing metrics for initial Node synchronization. These metrics measure the delay between the creation of a new Node and the node controller's initial management actions, such as removing the cloud provider taint. These metrics should be consulted when setting cloud controller manager's --concurrent-node-syncs flag. (#119241, @cartermckinnon) [SIG Cloud Provider and Instrumentation]
- Faster scheduling when ResourceClaims are involved (#119078, @pohly) [SIG Node and Scheduling]
- Graduate the ProbeTerminationGracePeriod feature gate to GA (#114307, @rphillips) [SIG Apps and Node]
- Hashing of KeyID in Logs

This release adds a feature to hash the KeyID values in the logs. The KeyID values are sensitive information that should not be exposed in plain text in the logs. By hashing the KeyID values, we can protect the confidentiality of the data while still being able to log the necessary information. (#118988, @nilekhc) [SIG API Machinery, Auth and Testing]
- Implement alpha support for a drop-in kubelet configuration directory (#119390, @sohankunkerkar) [SIG Node]
- In the course of admitting a single request, the ValidatingAdmissionPolicy plugin will perform no more than one authorization check per unique authorizer expression. All evaluations of identical authorizer expressions will produce the same decision. (#116443, @benluddy) [SIG API Machinery and Testing]
- Kube-controller-manager: the dynamic resource controller steps in when a pod got created such that the scheduler ignores it (i.e. spec.nodeName is set) and then takes care of triggering delayed resource claim allocation and/or reserving a claim for the pod. (#118209, @pohly) [SIG API Machinery, Apps, Auth, Node and Testing]
- Kube-proxy service health returns http header "X-Load-Balancing-Endpoint-Weight" with number of local endpoints. The same information is still available in response body JSON payload.LocalEndpoints. (#118999, @cezarygerard) [SIG Network]
- Kubelet: plugins for dynamic resource allocation may use the v1alpha3 API instead of v1alpha2 if they want to do prepare/unprepare operations in batches. (#119012, @pohly) [SIG Node and Testing]
- Kubelet: security of dynamic resource allocation was enhanced by limiting node access to those objects that are needed on the node. (#116254, @pohly) [SIG Auth and Testing]
- Kubernetes is now built with Go 1.20.6 (#119324, @xmudrii) [SIG API Machinery, Auth, Cloud Provider, Release and Testing]
- Migrate pkg/controller/endpoint to contextual logging (#116755, @my-git9) [SIG Apps, Instrumentation and Network]
- Migrated the EndpointSlice and EndpointSliceMirroring controllers (within kube-controller-manager) to use contextual logging. (#115295, @Namanl2001) [SIG API Machinery, Apps, Network and Testing]
- Move non-graceful node shutdown to GA. (#118228, @carlory) [SIG Apps, Storage and Testing]
- New CEL Library functions to support Kubernetes Quantities. (#118803, @alexzielenski) [SIG API Machinery]
- New Metrics Added for Encryption Configuration Controller

This release adds new metrics to the Encryption Configuration Controller to help monitor the automatic reloading of encryption configuration. The new metrics include:

• apiserver_encryption_config_controller_automatic_reload_failures_total: Total number of failed automatic reloads of encryption configuration.
• apiserver_encryption_config_controller_automatic_reload_success_total: Total number of successful automatic reloads of encryption configuration.
• apiserver_encryption_config_controller_automatic_reload_last_timestamp_seconds: Timestamp of the last successful or failed automatic reload of encryption configuration.

These metrics can be used to monitor the health of the Encryption Configuration Controller and to troubleshoot any issues that may arise during automatic reloading of encryption configuration. (#119008, @nilekhc) [SIG API Machinery, Auth and Instrumentation]
- New staging repo has been created for the EndpointSlice reconciler. (#118953, @mskrocki) [SIG Apps, Network and Release]
- Promote the following apiserver flowcontrol metrics to Beta:

apiserver_flowcontrol_request_wait_duration_seconds
apiserver_flowcontrol_current_executing_seats
apiserver_flowcontrol_nominal_limit_seats
apiserver_flowcontrol_rejected_requests_total
apiserver_flowcontrol_dispatched_requests_total
apiserver_flowcontrol_current_inqueue_requests
apiserver_flowcontrol_current_executing_requests (#119110, @andrewsykim) [SIG API Machinery and Instrumentation]
- Replace apiserver_storage_db_total_size_in_bytes with apiserver_storage_size_bytes metric (#118812, @serathius) [SIG API Machinery, Instrumentation and Testing]
- The apiserver debug endpoint /debug/api_priority_and_fairness/dump_requests has been extended to dump executing requests as well as queued ones. A column for StartTime has been added to the returned table, with the queued requests having a StartTime of "0001-01-01T00:00:00Z". The executing requests have a RequestIndexInQueue of -1, and the QueueIndex is also -1 for priority levels without queues. (#119009, @MikeSpreitzer) [SIG API Machinery]
- The scheduler skips the PodTopologySpread Score plugin when nothing to do with the Pod.
It will affect some metrics values related to the PodTopologySpread Score plugin. (#118608, @utam0k) [SIG Scheduling]
- TopologyManagerPolicyOptions feature-flag is promoted to beta and enabled by default. (#118816, @PiotrProkop) [SIG Node]
- Update kube-apiserver's priority & fairness work estimator such that 'max seats' is MIN(0.15 x nominalCL, nomincalCL / handSize) (#118601, @andrewsykim) [SIG API Machinery]
- ValidatingAdmissionPolicy type checking now correctly handles authorizer variable. (#118540, @jiahuif) [SIG API Machinery]
- With the KubeletCgroupDriverFromCRI feature gate enabled and sufficiently new version of a container
runtime, kubelet automatically detects the cgroup driver config from the container runtime, eliminating
the need to specify the cgroupDriver configuration option (or --cgroup-driverflag) of kubelet. ([#118770](https://github.com/kubernetes/kubernetes/pull/118770), [@marquiz](https://github.com/marquiz)) [SIG Node] - [Kube-proxy]: implement connection draining for terminating nodes, KEP-3836 ([#116470](https://github.com/kubernetes/kubernetes/pull/116470), [@alexanderConstantinescu](https://github.com/alexanderConstantinescu)) [SIG Network] -force_delete_pods_total andforce_delete_pod_errors_total ` metrics count all pod deletion behaviors. (#118480, @carlory) [SIG Apps]

Failing Test

• Switched back to debian-base instead of distroless for conformance image. (#119422, @saschagrunert) [SIG Architecture, Release and Testing]

Bug or Regression

• Add warning for dup ports update/patching in pod's container ports and service ports (#113245, @pacoxu) [SIG Network]
• Bump cadvisor version to v0.47.3 (#119225, @iholder101) [SIG Node and Testing]
• Dynamic Resource Allocation: log a error and submit an event when Kubelet fails to prepare dynamic resources (#118578, @bart0sh) [SIG Node]
• Fix computing backoff delay when using Job pod failure policy, by including in the backoff delay calculation pod failures ignored from the backoffLimit counter (#119434, @mimowo) [SIG Apps]
• Fix discoverability of apiregistration.k8s.io in openapi/v3 (#118879, @atiratree) [SIG API Machinery]
• Fixed a bug where kubectl port-forward, when used with a Deployment, could connect to a terminating pod even when a running pod is also available. (#119256, @brianpursley) [SIG CLI]
• Fixed kubelet startup getting stuck with NewVolumeManagerReconstruction feature enabled and a CSI volume present in /var/lib/kubelet/pods. (#117804, @jsafrane) [SIG Node and Storage]
• Kubeadm: the limitation that the 'ignorePreflightErrors' field can not be set to 'all' in kubeadm config file has been removed (#119351, @SataQiu) [SIG Cluster Lifecycle]
• Only declare Job as finished after removing all Pod finalizers to avoid orphan Pods (#119159, @alculquicondor) [SIG Apps and Testing]
• Reduces CPU and memory consumption of kube-apiserver if OpenAPI V2 is not accessed by any client. Also improves performance of the apiserver on installation of many CRDs. (#118212, @Jefftree) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node and Storage]
• The kube-proxy sync_proxy_rules_iptables_total metric has now reverted back
  to its pre-1.27 behavior of tracking the total number of iptables rules that
  kube-proxy is responsible for, rather than only counting the number of rules
  that it re-synced on the last sync. The new sync_proxy_rules_iptables_last
  metric now gives the latter number. (#119140, @danwinship) [SIG Network]
• The metric apiserver_flowcontrol_request_concurrency_limit has been deprecated and will be removed in a future release. It is a duplicate of apiserver_flowcontrol_nominal_limit_seats (introduced in release 1.26) but has an outdated name and had an outdated HELP string. (#118959, @MikeSpreitzer) [SIG API Machinery]
• [Dual-stack] Fix generateAPIPodStatus() of kubelet handling Secondary IP. hostIPs order may not be be consistent. If secondary IP is before primary one, current logic adds primary IP twice into PodIPs, which leads to error: "may specify no more than one IP for each IP family". (#116879, @lzhecheng) [SIG Node]

Other (Cleanup or Flake)

• Migrated the disruption controller (within kube-controller-manager) to use contextual logging. (#119147, @mengjiao-liu) [SIG API Machinery, Apps, Instrumentation and Testing]
• Migrated the podgc controller and some other remaining log calls within kube-controller-manager to use contextual logging. kube-controller-manager is now converted completely. (#119250, @pohly) [SIG API Machinery, Apps, Cloud Provider, Instrumentation, Network, Storage and Testing]
• Remove KUBECTL_EXPLAIN_OPENAPIV3 which is already redundant (#119286, @ardaguclu) [SIG CLI]
• Revised OpenAPI v2 fetching for CustomResourceDefinitions. CRDs are now aggregated lazily,
  which improves resource usage during installation of many CRDs. As a result, the first request
  to fetch the OpenAPI may be slower. (#118808, @Jefftree) [SIG API Machinery and Testing]
• Shrink the OpenAPI v2 spec by more than 50%, especially for less CPU resource consumption. (#118204, @sttts) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node and Storage]
• The GetAllocatableResources podresources API endpoint is now GA (#118973, @ffromani) [SIG Node and Testing]
• Updated debian-base image to bookworm-v1.0.0. (#119095, @saschagrunert) [SIG API Machinery, Architecture, Release and Testing]
• Updated setcap image to debian bookworm v1.0.0. (#119247, @saschagrunert) [SIG Release]

Dependencies

Added

• github.com/xhit/go-str2duration/v2: v2.1.0

Changed

• github.com/alecthomas/kingpin/v2: v2.3.1 → v2.3.2
• github.com/google/cadvisor: v0.47.2 → v0.47.3
• github.com/prometheus/client_model: v0.3.0 → v0.4.0
• github.com/prometheus/common: v0.42.0 → v0.44.0
• github.com/rogpeppe/go-internal: v1.6.1 → v1.10.0
• golang.org/x/crypto: v0.6.0 → v0.11.0
• golang.org/x/net: v0.9.0 → v0.12.0
• golang.org/x/oauth2: v0.6.0 → v0.8.0
• golang.org/x/sys: v0.8.0 → v0.10.0
• golang.org/x/term: v0.7.0 → v0.10.0
• golang.org/x/text: v0.9.0 → v0.11.0
• k8s.io/kube-openapi: 7562a10 → 2695361

Removed

• github.com/xhit/go-str2duration: v1.2.0

v1.28.0-alpha.4

Downloads for v1.28.0-alpha.4

Source Code

filename | sha512 hash
-------- | -----------
kubernetes.tar.gz | 05a404e2a5a526cb4713a9cab1cfcadb03cbeb065663a8ccec9c7eaf60277e1c69bea422716fc3b805ca569effb036b2d88adc752409b4f6103f10111f620736
kubernetes-src.tar.gz | 0707c72499098c2eb8ca3cffd1baf0cb4da553ded8acbf6bc1b725461484a75ba5baf277ccbe318cdb5df0c970cb31bf8afe3df0130acdd23c35b8a2fbc8a15f

Client Binaries

filename | sha512 hash
-------- | -----------
kubernetes-client-darwin-amd64.tar.gz | 5fc07afae149003d53d8bc72f9d3bbf578efd7ec7c36fda46a436774f4471cd457317dada967fc3596d369783219bdca1974d62f47c09fb8b2d158a78d48aebe
kubernetes-client-darwin-arm64.tar.gz | a025567c86d8af69e34d589f36f079eeff85105e342047f5d74b1ba749b9b857d19ffadd280910fd58926dfae54eb7eb8203009ac96362877a05e3cb88c49e4b
kubernetes-client-linux-386.tar.gz | b17a16d8f3ce78e92b2988e726a9c818f0f0f36b8ae22809f4db2568c1746b585888820cb3cb276d00b76781e75bb10d1e9a19887d438ffee58c435775f114f0
kubernetes-client-linux-amd64.tar.gz | 8133aaf1c3a964d32666fa7616917235ca62ed57b879915cae3acc4867db46e35b127cc302d1a3fa7fff143ade6f73c0c1667b45eb1debef052b2b69f5c407f1
kubernetes-client-linux-arm.tar.gz | 84f30f2d113f9003207e547dcb4c3467e17b93b05dade5c6b8cd847bd040e7b21574c1d75d923d8f1d3906a4793ab8a78ab477cb16ccd72a98221c0edd394ce9
kubernetes-client-linux-arm64.tar.gz | e7cad9b40187afa63168e40ec96128d4f2469115049cd0642e3a255d25b325c662fd99c1866dd6798a634d2de179493be9c05de11372f86f6d31329b24b8c283
kubernetes-client-linux-ppc64le.tar.gz | 3597df23599a6f9da7886601c0ff7e49fc23b0817b6463aa426e7191f23955f772986494d9b8926b9b9dcb1f7f2f75054739b3d25a893f4d65e3f58d567eeb2b
kubernetes-client-linux-s390x.tar.gz | a5630f732ef831722c778484742368a3768276bd4e443bcedcbd2c02b1164265e7a70fb55b6e7560558b0a3d4eaca3cbc7c7ded436c19024e6826224d73b4ef6
kubernetes-client-windows-386.tar.gz | 0be37194c9d1fc75877346eb1ab1f612286068732558d59f862c1901c3217b91a7e758f41aaba2308142aab4170bb4e5f4e7291fde7717062a2fb4ca91b159bf
kubernetes-client-windows-amd64.tar.gz | f53a4216ac8e959b40d10257bc01044b1c1e430c5da02b61b6ed5184f0acb7317d19f36946e60de0d168d45d71d404bd615a2beb9b70b6495b2db0182fde375c
kubernetes-client-windows-arm64.tar.gz | 0bce42d19642c6dd794ce7944274b0dd997df5984fa8a22f72f07551f00533f8f018bd209994c69615ca33d4f8a41a873b364e5c3d996b2edccf849f94621236

Server Binaries

filename | sha512 hash
-------- | -----------
kubernetes-server-linux-amd64.tar.gz | 593409f6e3accafbd448da6d6b775cc1f85cc4b787acb7b7580e94221a528c6b805e73d5fe16fc36a9c2838da6bad3928b18c3771ec95c5bbd0efd19d404d8e2
kubernetes-server-linux-arm64.tar.gz | 3e8f26b51f85b61ad249f54255077a8f4ffaf80c55935cfa2f490f56eea112eb2df569882a7f486cd19371e41c1f65c43aa2bffeece3e35269b67c19ff9e7ae4
kubernetes-server-linux-ppc64le.tar.gz | 3e4d8f4ebed1632878a78936e62331973fa57e8b394c79c262f98316a81f460b6bd8ee4cf9dd74d77df289ea2cf3716e58a431a5f52c610c916a7f45cee80bdd
kubernetes-server-linux-s390x.tar.gz | 6aa22b0fa568b70dd9c34408dfb70b60d09b2fd65429671e23d0becfc83aac75d082818c21737105e4e2485752fd9e5d5f1d92e8dbbc46b257d269237155a85a

Node Binaries

filename | sha512 hash
-------- | -----------
kubernetes-node-linux-amd64.tar.gz | 7f0a902031857e8ec8189dd37c52e788c36e02c5e19439ec24602822e95466b008064419961f728f3c02b661dfc23d89fcc0bca15b085f9ed3d001b4cd94adb8
kubernetes-node-linux-arm64.tar.gz | fa564b5fe5d69f2c31b453da9024ab505adfb62373722fe3d6d3852eb13e938d271f5a90542982ea9ea4fd2182f67a720be0b8c77f2e7353a6bbda3ff16e34a5
kubernetes-node-linux-ppc64le.tar.gz | 14ea08bf2ad7f177de8f59b4ac44bd7ba451a9b6493c1b6a3fbc00337e2c7865824b790e30d3a4dbb6cbce9ecbfa62a4e12fb7da04049e2bdf718d273131fff2
kubernetes-node-linux-s390x.tar.gz | a1119b19f271a5ddb90b71f4b3e844b3d46889340349fcbf297c2ffaae253538e303535f7b180a1107d189a7fc66fdf3f029da90db761f1ca5faf52aedfd0c64
kubernetes-node-windows-amd64.tar.gz | a3b03111ff946bb21722d1f140cfa483b9692381aaadc3cf7726d2e53b717b573887f3a541ab42cb3244a95f023817d2fefd5c4314d1b5fe30ecd68643709295

Container Images

All container images are available as manifest lists and support the described
architectures. It is also possible to pull a specific architecture directly by
adding the "-$ARCH" suffix to the container image name.

name | architectures
---- | -------------
registry.k8s.io/conformance:v1.28.0-alpha.4 | amd64, arm64, ppc64le, s390x
registry.k8s.io/kube-apiserver:v1.28.0-alpha.4 | amd64, arm64, ppc64le, s390x
registry.k8s.io/kube-controller-manager:v1.28.0-alpha.4 | amd64, arm64, ppc64le, s390x
registry.k8s.io/kube-proxy:v1.28.0-alpha.4 | amd64, arm64, ppc64le, s390x
registry.k8s.io/kube-scheduler:v1.28.0-alpha.4 | amd64, arm64, ppc64le, s390x
registry.k8s.io/kubectl:v1.28.0-alpha.4 | amd64, arm64, ppc64le, s390x

Changelog since v1.28.0-alpha.3

Urgent Upgrade Notes

(No, really, you MUST read this before you upgrade)

• Action required for the custom scheduler plugin developers.
  Here's the breaking change in EnqueueExtension in the scheduling framework.
  The EventsToRegister in EnqueueExtension changed the return value from ClusterEvent to ClusterEventWithHint. ClusterEventWithHint allows each plugin to filter out more useless events via the callback function named QueueingHintFn.
  When the scheduling queue receives a cluster event, before moving each Pod from unschedulable pod pool to activeQ/backoffQ, it will call QueueingHintFn of plugins that rejected each Pod in the previous scheduling cycle.
  Depending on the value returned from QueueingHintFn, the scheduling queue changes how it queues each Pod:
• if more than one QueueingHintFn returns QueueImmediately, it queues Pod to activeQ.
• If no QueueingHintFn returns QueueImmediately and more than one plugin returns QueueAfterBackoff, it queues Pod to backoffQ if Pod is backing off, or to activeQ if Pod's backoff has already finished.
• If all QueueingHintFn return QueueSkip, it puts this pod back to the unschedulable pod pool

Having appropriate QueueingHintFn contributes to reducing useless retries and thus improves the overall scheduler's performance.

How can I migrate?

For backward compatibility, nil QueueingHintFn is treated as always returning QueueAfterBackoff.
So, if you want to just keep the existing behavior, you can register ClusterEventWithHint with no QueueingHintFn in it.
But, registering appropriate QueueingHintFn is, of course, better from a scheduling performance perspective. (#118551, @sanposhiho) [SIG Node, Scheduling, Storage and Testing]
- RBD volume plugin ( kubernetes.io/rbd) has been deprecated in this release and will be removed in a subsequent release. Alternative is to use RBD CSI driver (https://github.com/ceph/ceph-csi/) in your Kubernetes Cluster. (#118552, @humblec) [SIG Storage]

Changes by Kind

Deprecation

• KMSv1 is deprecated and will only receive security updates going forward. Use KMSv2 instead. Set --feature-gates=KMSv1=true to use the deprecated KMSv1 feature. (#119007, @aramase) [SIG API Machinery and Auth]

API Change

• Add ServedVersions field to StorageVersion API (#118386, @Richabanker) [SIG API Machinery and Testing]
• Component-base/logs is now more strict about not applying configurations multiple times and will return an error when that is attempted. Can be overridden by binaries which need to do that. (#117108, @pohly) [SIG API Machinery, Architecture, Cloud Provider, Instrumentation, Scheduling and Testing]

Feature

• "plugin_evaluation_total" metric supports prescore/score extension point.
  The metric doesn't get incremented when the prescore/score plugin has nothing to do with an incoming Pod. (#118025, @AxeZhan) [SIG Scheduling]
• Add --concurrency flag to configure the concurrency of kubectl diff execution, defaults to 1 (#118810, @brancz) [SIG CLI]
• AdvancedAuditing feature gate that graduated to GA in 1.12 and was unconditionally enabled has been removed in v1.28. (#118763, @Shubham82) [SIG API Machinery and Auth]
• Allow to monitor client-go DNS resolver latencies via rest_client_dns_resolution_duration_seconds Prometheus metric (#115357, @mfojtik) [SIG API Machinery, Architecture and Instrumentation]
• Dynamic resource allocation: when a claim uses "wait for first consumer" allocation (the default), then it will now get deallocated after it was used by a pod. That ensures that the next pod isn't affected by previous scheduling decision and that resources are not kept allocated unless really needed. If keeping a claim allocated is desired, use "immediate allocation". (#118936, @pohly) [SIG Apps, Node and Testing]
• Kubeadm: add the --allow-experimental-api flag to "kubeadm config migrate/validate" commands. It can be used to migrate or validate WIP / experimental APIs in the future. (#118866, @neolit123) [SIG Cluster Lifecycle]
• Kubeadm: generate CA certificates with a start time that is offset 5 minutes in the past relative to the current system time to workaround cases of clock desync.
  client-go: allow to set NotBefore in NewSelfSignedCACert() (#118922, @champtar) [SIG API Machinery, Auth and Cluster Lifecycle]
• Migrated controller functions to use contextual logging. (#116930, @fatsheep9146) [SIG API Machinery, Apps, Network, Node, Storage and Testing]
• Migrated the certificate controller (within kube-controller-manager) to use contextual logging. (#113994, @mengjiao-liu) [SIG API Machinery, Apps, Auth, Instrumentation and Testing]
• Now it is possible to use pods with volumes and user namespaces. The feature gate was renamed from UserNamespacesStatelessPodsSupport to UserNamespacesSupport (#118691, @giuseppe) [SIG Apps, Node and Testing]
• RetroactiveDefaultStorageClass feature is stable and enabled by default. (#118102, @RomanBednar) [SIG Apps, Storage and Testing]
• Schedular now waits for handlers to finish syncing before the scheduling cycles start. (#116729, @AxeZhan) [SIG Apps, Scheduling and Testing]
• The "value" part in the wait --for=jsonpath='{expression}'[=value] is now optional. If the value is not provided i.e. the command looks like wait --for=jsonpath='{expression}' then the wait condition is interpreted as matched when the expression returns any single JSON value like object or a literal. (#118160, @minherz) [SIG CLI and Testing]
• Updated cAdvisor to v0.47.2 - Fix metrics in cri-o when a container restarts (#118774, @harche) [SIG Node]
• When a pod is done or not going to run, then ResourceClaims for it can be reused by other pods or deleted. (#118817, @pohly) [SIG API Machinery, Apps, Auth, Node and Testing]

Bug or Regression

• Added a new event FailedToRetrieveImagePullSecret which will be generated when a pod references an ImagePullSecret that doesn't exist. (#117927, @kaisoz) [SIG Node]
• As in Kubernetes 1.26 and 1.27, resource claims do not get prepared by kubelet when no container uses them. This was changed accidentally in v1.28.0-alpha.1. (#118786, @pohly) [SIG Node and Testing]
• Faster StatefulSet creation when Parallel mode is enabled. (#117865, @aleksandra-malinowska) [SIG Apps]
• Fix cronjob controller handling of complex schedules, like "30 6-16/4 * * 1-5", for example (#118724, @soltysh) [SIG Apps]
• Fix deletion of non-admissible pods that are deleted during Kubelet restart (#118497, @mimowo) [SIG Node and Testing]
• Fix discoverability of apiregistration.k8s.io in openapi/v3 (#118879, @atiratree) [SIG API Machinery]
• Kubectl explain should correctly work for all resources (#118876, @atiratree) [SIG CLI]
• Kubectl expose supports the creation of different protocol service on the same port (#114909, @aimuz) [SIG CLI]
• The Daemonset controller creates replacements for terminal Pods, which can appear during VM preemptions or when using Pod finalizers (#118716, @alculquicondor) [SIG Apps, Node and Testing]
• The pod_scheduling_duration_seconds metrics won't consider the time when a Pod fails PreEnqueue (like being gated). (#118049, @helayoty) [SIG Scheduling]
• Update apiserver metric request_filter_duration_seconds to include a 10s, 15s and 30s bucket.
• Update apiserver metric request_wait_duration_seconds to include a 15s bucket. (#118945, @andrewsykim) [SIG API Machinery, Instrumentation and Testing]
• Users will no longer see an error for failed events caused due to terminating namespace. (#114849, @padlar) [SIG API Machinery]
• Wait.PollUntilContextTimeout function, if immediate is true, the condition will be invoked before waiting and guarantees that the condition is invoked at least once, regardless of whether the context has been cancelled. (#118686, @aojea) [SIG API Machinery]

Other (Cleanup or Flake)

• Kube-controller-manager and cloud-controller-manager have changed the name of controllers that can be turned off/on that are passed to the --controllers flag (eg pod-garbage-collector-controller ). The old names (eg podgc) are also accepted and aliased to the new names (#115813, @atiratree) [SIG API Machinery and Cloud Provider]
• Kube-proxy will now warn at startup if the configuration seems inconsistent
  with respect to IP families. (For example, if you have an IPv4 node IP, but
  --cluster-cidr is IPv6.) (#119003, @danwinship) [SIG Network]
• Promote kubernetes_healthcheck and kubernetes_healthchecks_total to BETA stability level. (#118986, @logicalhan) [SIG Architecture, Instrumentation and Testing]
• Reduce delay when processing jobs after a transient API error (#118759, @mimowo) [SIG Apps]
• The NetworkPolicyLegacy test suite (deprecated in v1.21) has now officially been removed in favor of the new table driven e2e tests. (#118915, @astoycos) [SIG Network and Testing]
• The feature gates CSIMigrationGCE is graduated to GA and were unconditionally enabled have been removed in v1.25, and the entire gcepd package has been removed. (#117055, @cyclinder) [SIG API Machinery, Node, Scheduling and Storage]
• The metric apiserver_flowcontrol_current_executing_seats has been introduced as a duplicate of apiserver_flowcontrol_request_concurrency_in_use because the latter has a confusing name and will be removed in a later release. (#118960, @MikeSpreitzer) [SIG API Machinery]
• Use table-driven test for TestPerPodSchedulingMetrics (#118842, @helayoty) [SIG Scheduling]

Dependencies

Added

Nothing has changed.

Changed

• github.com/google/cadvisor: v0.47.1 → v0.47.2

Removed

Nothing has changed.

v1.28.0-alpha.3

Downloads for v1.28.0-alpha.3

Source Code

filename | sha512 hash
-------- | -----------
kubernetes.tar.gz | 1479e19873837e09f08f4c14d4b7587b1bc40d0b0d3214637311b63068301d34a63663f5d13b8ec62c81095a30eef1e8589633c630fe613eb825eb4afa0ddeb9
kubernetes-src.tar.gz | 1f5d3486f15574220d3c5a5d7c7a2b7412347418650deeda326eb513ba2bb43c197e089999756eee09ac4b55dcafc18eef96c6e15e20fcc9b91183a35f224cc1

Client Binaries

filename | sha512 hash
-------- | -----------
kubernetes-client-darwin-amd64.tar.gz | 4e5139d5a5a98343ded5b506451c3b9b052803a2e2ebb4e2328e17edbbde56dde749407ef8fc816283a1a1b1f80939a76b7c64a09a9496d4448ec47fe34cd95b
kubernetes-client-darwin-arm64.tar.gz | c58fdf32d3f1b411e10a7ede56115020c7a4c50912a899f06cf94d2f06a5e24a21df1deda6eb0e87f70d88afec46186f64dd18bb1b26b94b24b01059aef88b2c
kubernetes-client-linux-386.tar.gz | 26d3d2663de49063d02c682557dce616f5630b69a1ca219024ccd3405081193732bee42d24d9e5f5cf9fcd214da50defd64c141775a6aea372dbe3e9793e4547
kubernetes-client-linux-amd64.tar.gz | f9c29ea075c4b842c39587e911b83e2c798d6d69ed5f4601ce17ce3c33bb9db8f511b3d2149dfe3d86576763e92e0b6eb6d3aa1b12bc04868fb6c8a013a2fdd1
kubernetes-client-linux-arm.tar.gz | be984144ebe8c2f3156b3d13aaebd362543399f4c91f8f18b1936999327468ccb5af7d95723280000b627e2f81fb119d6e175cc305391ffd83b3b632eb3c30df
kubernetes-client-linux-arm64.tar.gz | bfc44ed472351dee434984e49be2cc9959df2f98048beeb1ad85a50bfe833a5396bb3f5c7f81ced2048de269bd60f6d4cf55944c0d44d288fb858b5552354389
kubernetes-client-linux-ppc64le.tar.gz | d7ee2633c172442a04d317d4d935b55aa892131f716308b78d0f7b3cb5d2bfa9069f7b55ec837e4ff9a260717313e7fc4c7ca4a931f9f2b36768a9c54593ba45
kubernetes-client-linux-s390x.tar.gz | 4621ed7a77990430cbb03a14f85780f059a408a567c26c14eb7c7127300f4e1ac1b2013206f00a92e35da9f1142909745262ad43a8026a9ebe31cf423dbb3b90
kubernetes-client-windows-386.tar.gz | 6526de9f4ccd6f036152d73ebbf3a2b626ca7291c85d5b46977e8a8643ff80616665f4bd8b009310cc887f7d5e1827045a0b2489dd19621338e727e18cca4097
kubernetes-client-windows-amd64.tar.gz | 784e8d07b1db0331f454faac34b279bffba16c11f98ba25aa54f64eb6a0d9d55f0e8221828aa4e6e4d0d6bfb66a3c22a0d8a66d6287b37a33890a960d25fc54e
kubernetes-client-windows-arm64.tar.gz | 523d2809d4ff4616ffd94e0868ac15ee10f87bee65760741b09ba2a04626e3ce226a5fc44a1bddb32835403ee374a6c12afa74ec0d2b9441304a11d923f43cee

Server Binaries

filename | sha512 hash
-------- | -----------
kubernetes-server-linux-amd64.tar.gz | 5c2716137752b514fe7a8b23291cc7a661984deaf642d111e65690e928ecfbb6b5168b1f043847f3d4689cafbbdf5bb6fb5c97d2fd6f5f83c5b6cc701a4f388f
kubernetes-server-linux-arm64.tar.gz | 814e4ec6b46960e90ccce6ef1c234666f372b2b41583ead530b4163b829c4bb9001df822daf55493e54ad749076ab5391ebde7261d73f4e524f6cace402d49c3
kubernetes-server-linux-ppc64le.tar.gz | 8a7749282ba4d3df877097dc07843112e815879ee911379d02ed33d5aab59cb60ffd27127ebd94879bef45d9534470d6aa7e48d71ecb455e3055c2dbb169fcdb
kubernetes-server-linux-s390x.tar.gz | 031a55294a06e5773d3c277dcab6b1e32c5e6409448015282275cb62e5bafb6a10845f0de3d37d501067f3f4176f48d04cafdd4eb96f6f7e30c7274ce3adaec6

Node Binaries

filename | sha512 hash
-------- | -----------
kubernetes-node-linux-amd64.tar.gz | 488d426c09f92a6f30d77352af7154b9fda8758a946566b31d855a4c30f82ccdb071d40708f96e78f9bf5ca961be27fa67ecc833101818e43da19bc967c9eb28
kubernetes-node-linux-arm64.tar.gz | e5f6914bd6e393915ea814401c09f4b0accc3454087091e60b01ea7c87aa64edd1a0af954cd1a90a3dc83c5559772d1ba5a8ff04134638e911d972947e3e94a0
kubernetes-node-linux-ppc64le.tar.gz | 459453ad83c9cdd2568e43ed988ec2635b5413b435e587e8a6394feb7d49c550622c491f337c48f90ad32cc02b7ef47e24069f09737817a1c37f1a8930d6c5c2
kubernetes-node-linux-s390x.tar.gz | e5a6ff5bb58275720c18b6181ffb9d135dcfaf05a229f87787538775e0280543db43e7218168f0a2de8a74acca6899dccd1cce88780b086ce6a96fb3e0368870
kubernetes-node-windows-amd64.tar.gz | 7cc51923e34cfeb00681e7c2d26dff5c442b39d0000f11ad30ec5c843d80847a0353b3b4e6e5a228ad5cae7f33dfc0d4c38bf4b843b98b8115d28c3dc683dd9e

Container Images

All container images are available as manifest lists and support the described
architectures. It is also possible to pull a specific architecture directly by
adding the "-$ARCH" suffix to the container image name.

name | architectures
---- | -------------
registry.k8s.io/conformance:v1.28.0-alpha.3 | amd64, arm64, ppc64le, s390x
registry.k8s.io/kube-apiserver:v1.28.0-alpha.3 | amd64, arm64, ppc64le, s390x
registry.k8s.io/kube-controller-manager:v1.28.0-alpha.3 | amd64, arm64, ppc64le, s390x
registry.k8s.io/kube-proxy:v1.28.0-alpha.3 | amd64, arm64, ppc64le, s390x
registry.k8s.io/kube-scheduler:v1.28.0-alpha.3 | amd64, arm64, ppc64le, s390x
registry.k8s.io/kubectl:v1.28.0-alpha.3 | amd64, arm64, ppc64le, s390x

Changelog since v1.28.0-alpha.2

Changes by Kind

Deprecation

• Kube-controller-manager deprecate --volume-host-cidr-denylist and --volume-host-allow-local-loopback flags. (#118128, @carlory) [SIG API Machinery, Apps, Network, Node, Storage and Testing]
• Kubelet: The --azure-container-registry-config flag has been deprecated and will be removed in a future release, please use --image-credential-provider-config and --image-credential-provider-bin-dir to setup acr credential provider instead. (#118596, @SataQiu) [SIG Node]

API Change

• ACTION_REQUIRED
  When an Indexed Job has a number of completions higher than 10^5 and parallelism higher than 10^4, and a big number of Indexes fail, Kubernetes might not be able to track the termination of the Job. Kubernetes now emits a warning, at Job creation, when the Job manifest exceeds both of these limits. (#118420, @alculquicondor) [SIG Apps]
• Expose rest.DefaultServerUrlFor function (#118055, @timofurrer) [SIG API Machinery]
• If using cgroups v2, then the cgroup aware OOM killer will be enabled for container cgroups via memory.oom.group . This causes processes within the cgroup to be treated as a unit and killed simultaneously in the event of an OOM kill on any process in the cgroup. (#117793, @tzneal) [SIG Apps, Node and Testing]
• Update the comment about the feature-gate level for PodFailurePolicy from alpha to beta (#118278, @mimowo) [SIG Apps]

Feature

• Add '--concurrent-cron-job-syncs' flag for kube-controller-manager to set the number of workers for cron job controller (#117550, @borgerli) [SIG Apps]
• Client-go: make generated CA certificates valid 1 hour in the past (NewSelfSignedCACert). Applies to CA certificates and other certificates generated by kubeadm. (#118631, @champtar) [SIG Auth]
• Fixes the alpha CloudDualStackNodeIPs feature. (#118329, @danwinship) [SIG Network and Node]
• Kubelet: un-deprecate --provider-id flag (#116530, @pacoxu) [SIG Node]
• Migrated the Job controller (within kube-controller-manager) to use contextual logging. (#116910, @fatsheep9146) [SIG API Machinery, Apps and Testing]
• Rename PodHasNetwork to PodReadyToStartContainers (#117702, @kannon92) [SIG Node and Testing]

Bug or Regression

• CI job ci-kubernetes-node-arm64-ubuntu-serial will test node e2e on arm64, use-dockerized-build and target-build-arch are required to run this job. (#118567, @chendave) [SIG Node and Testing]
• Fix Cronjob status.lastSuccessfulTime not populated by a manually triggered job (#118530, @carlory) [SIG CLI]
• Fix component status calling etcd health endpoint over http which exposed kubernetes to the risk of complete watch starvation and is inconsistent with other etcd probing done by kube-apiserver. (#118460, @serathius) [SIG API Machinery]
• Fixed the preStop hook will block the pod termination grace period (#115835, @HirazawaUi) [SIG Node and Testing]
• Users will no longer see an error for failed events caused due to terminating namespace. (#114849, @padlar) [SIG API Machinery]

Other (Cleanup or Flake)

• Kube-proxy: remove log warning about not using config file (#118115, @TommyStarK) [SIG Network]
• Make Job controller batching of syncJob invocations enabled unconditionally (it was conditional on JobReadyPods feature before).

Also, Job controller's constants for default backoff and maximal backoff are lowered down to 1s (from 10s) and 1min (from 6min), respectively. These constants are used to determine the backoff delay for the next Job controller sync in case of a request failure. (#118615, @mimowo) [SIG Apps and Testing]
- Migrated the interpodaffinity scheduler plugin to use contextual logging. (#116635, @mengjiao-liu) [SIG Instrumentation and Scheduling]

Dependencies

Added

• github.com/alecthomas/kingpin/v2: v2.3.1
• github.com/xhit/go-str2duration: v1.2.0

Changed

• github.com/alecthomas/units: f65c72e → b94a6e3
• github.com/go-kit/log: v0.2.0 → v0.2.1
• github.com/kr/pretty: v0.3.0 → v0.3.1
• github.com/matttproud/golang_protobuf_extensions: v1.0.2 → v1.0.4
• github.com/prometheus/client_golang: v1.14.0 → v1.16.0
• github.com/prometheus/common: v0.37.0 → v0.42.0
• github.com/prometheus/procfs: v0.8.0 → v0.10.1
• golang.org/x/sync: v0.1.0 → v0.2.0

Removed

Nothing has changed.

v1.28.0-alpha.2

Downloads for v1.28.0-alpha.2

Source Code

filename | sha512 hash
-------- | -----------
kubernetes.tar.gz | 3ef803c3df0a722e4fdaa045fa7aca8c39892916b0788e0c3216747397fe4e6e8fc02483c7e1a7280ed9f2716324c067cd806037dbf9b635e268f8b62db43841
kubernetes-src.tar.gz | c7594dfdefc92fabd6ffeef08fb98e6686c6bbf8dc952197e6629838e645c83d4b7d375e078dc3f61f6cdbd425ca68eb88f0ffdbb1e3d2e034af9efeb8f7f34b

Client Binaries

filename | sha512 hash
-------- | -----------
kubernetes-client-darwin-amd64.tar.gz | 66e80a10b94d81ca08bd3eb94168afa429c3bc2b036e80bd55e4d0c93b4eb9e645d575ff520e6961368d2559e9bec786fbf115d0d902af7a3b0eae43213fb9e2
kubernetes-client-darwin-arm64.tar.gz | 0654d9ec7234a042d08db30cf01b69f66889451219a57a9e8a1dda2e11e7767e0ea1a171c2f16957bc6fedf1777bea455a03f453b9b06046d1588824ddb72627
kubernetes-client-linux-386.tar.gz | cdb3fdaacad12e8910f7b3dc26cd4bb684fcfbf8e9d0e594a1dda3da8ca804a9f69d0d030893b15cdd4c0daf613b3efdf8c2be54aaa8406764aaa17550df2fb8
kubernetes-client-linux-amd64.tar.gz | ceb64baa175b6444f719311f76b5de7cc1cfcad13650d3a3dc50fa5fbb6f5c724aa7fc63aa4df27fc1e3b58c419a3d1b3aaa75fb415c3709bd4f4c7253b4d99a
kubernetes-client-linux-arm.tar.gz | c9a9f307fe69ce45a1f31ff8f68a158c08de202676952fc243cba4bca4f66f020fc68c7f36191d731ac536f9ff96a0e8c75a7ecc2884eac59468b92af7f72c5c
kubernetes-client-linux-arm64.tar.gz | b5ee31bfb316559a9bec2bf7c5c2e555d1cfc07a307e1bfd2f4022b3a9988f2b6ae2e1189079a81dac0d3cc46d8982e25eafb2aaad5500be1c872313427e3544
kubernetes-client-linux-ppc64le.tar.gz | d91e713944e21cd7729933fcb13118a3aae1a59e9809f7c8c23c047a55ceb9428459758b27d1496be3909d5097cdf5bcdc9089f6dbbfca6154a763ede0fc38d1
kubernetes-client-linux-s390x.tar.gz | b34d9111b967d295092a57c9838f97400f526c9ad8086c562e090b3e92d171159dcece2576dfd0345c59c3adab06f5b01ff51a840e5997764b5bdc15df44c00f
kubernetes-client-windows-386.tar.gz | 82dc590e72c866e561617aed045b4377b90e4eb42574e998c71fe287202acf37ff0a3f1d55545d3465b2410108e0b4a4ab921a0fd749a99efc36723fe945c1f6
kubernetes-client-windows-amd64.tar.gz | a7aba1527252e942bb3a88df21a0c7a6c9f9fdbfbb1e5b2e6e28c431a413fbb51d1e50682f05a353fc6f997ed7b5e7b229f08aea0d12b92631c1b8152b029fce
kubernetes-client-windows-arm64.tar.gz | 728574089c01b253a1e81da19680ffaf58a88780653583d1af35ad68ee50c05cbbc21a7ca1b33c293d6db0eaa1de04988b8540467f3b4dfa22e48d5ca7a93604

Server Binaries

filename | sha512 hash
-------- | -----------
kubernetes-server-linux-amd64.tar.gz | fe373323bd6940fc1801d04f3fd1f574a85d103ce6526f58bc894cf0bbbbe387a26f4a4f248249aec36f88bc15c1eceb67b85af9dc876b139bad245b0551f219
kubernetes-server-linux-arm64.tar.gz | 4e76b003da0dedc9b457cb47fbcb4e8f719144aced497b6ebaaf9c515c8125a72e72c3a059757de3cdfc177dd4e3b18368ba47fd2fefb9cb2e4bc6a23c73f802
kubernetes-server-linux-ppc64le.tar.gz | c17f57f13d1393d4cf32a4ac0c122b6ebe400d88a26b3bd2eae9c6b4b4ed9f26a23ab99032847fb758bc4cbfa09a4dc7b843603d81659d0b79d42953efeca15b
kubernetes-server-linux-s390x.tar.gz | 36ce4946d3879c1e4b5e6e928be8da0eb9f146c96633d1bcb7d6541fa158f5a176d8174fba113d17bacff0db2debcedad73aed2d4054e3cc65bbf29256e62942

Node Binaries

filename | sha512 hash
-------- | -----------
kubernetes-node-linux-amd64.tar.gz | 34274b82a809fea6bdbfc1602dc66929efa71da0b3e510880a7b498253aa379a5ddc1fb1875666d2e6b5bbdd6473a7471a6b80e46526d57358ae9f97ec0e1904
kubernetes-node-linux-arm64.tar.gz | 610dc57e34222170c57b8c31ae15392cb960c2bff750f57acf0925d1597b54326c2f78b610c0f79f9762ba67a3af077f985b0ed8ef275e0d81794775b6448d58
kubernetes-node-linux-ppc64le.tar.gz | 2b65b45735680dab34e462f45c34c64c809bd8e80b2a45297446b1489dc974a54f8cd11364192e090e98edd987b4d7fb81b81d439987b1131d64a943d94ebb8f
kubernetes-node-linux-s390x.tar.gz | b3cba42fa9532bb9c7d80e31b1be8cb904bad1d32d6f3c651a690be1435d7f8d610bb203c89c420e96997a11a51b3e46f05fe626d2818fcc1bd3cef7d5a15b92
kubernetes-node-windows-amd64.tar.gz | 0b00eac88f2220dbb7f6243f9d4de41789bf3f38c14eec1c44bd48b9a067a43ef4cc9c468dc5869598d9f5ba06828b8bc5b7ea5eaa42faabaf584a8bdcb9ea29

Container Images

All container images are available as manifest lists and support the described
architectures. It is also possible to pull a specific architecture directly by
adding the "-$ARCH" suffix to the container image name.

name | architectures
---- | -------------
registry.k8s.io/conformance:v1.28.0-alpha.2 | amd64, arm64, ppc64le, s390x
registry.k8s.io/kube-apiserver:v1.28.0-alpha.2 | amd64, arm64, ppc64le, s390x
registry.k8s.io/kube-controller-manager:v1.28.0-alpha.2 | amd64, arm64, ppc64le, s390x
registry.k8s.io/kube-proxy:v1.28.0-alpha.2 | amd64, arm64, ppc64le, s390x
registry.k8s.io/kube-scheduler:v1.28.0-alpha.2 | amd64, arm64, ppc64le, s390x
registry.k8s.io/kubectl:v1.28.0-alpha.2 | amd64, arm64, ppc64le, s390x

Changelog since v1.28.0-alpha.1

Urgent Upgrade Notes

(No, really, you MUST read this before you upgrade)

• CephFS volume plugin ( kubernetes.io/cephfs) has been deprecated in this release and will be removed in a subsequent release. Alternative is to use CephFS CSI driver (https://github.com/ceph/ceph-csi/) in your Kubernetes Cluster. (#118143, @humblec) [SIG Storage]

Changes by Kind

Feature

• Introduce support for CEL optionals (see CEL spec proposal 246).
  This feature will not be fully enabled until a future Kubernetes release (likely to be v1.29), but is added in v1.28 to enable
  safe rollback on downgrade. (#118339, @jpbetz) [SIG API Machinery, Auth, Cloud Provider and Testing]
• Kubernetes is now built with Go 1.20.5 (#118507, @jeremyrickard) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node, Release, Storage and Testing]
• Promote ServiceNodePortStaticSubrange to beta and it will be enabled by default (#117877, @xuzhenglun) [SIG Network]
• The ExpandedDNSConfig feature has graduated to GA. 'ExpandedDNSConfig' feature was locked to default value and will be removed in v1.30. If you were setting this feature gate explicitly, please remove it now. (#116741, @gjkim42) [SIG Apps, Network and Node]
• The helping message of commands which have sub-commands is now clearer and more instructive. It will show the full command instead of 'kubectl --help ...'

Changed 'kubectl create secret --help' description. There will be a short introduction to the three secret types and clearer guidance on how to use the command. (#117930, @LronDC) [SIG CLI and Testing]
- Updated distroless I-tables to use registry.k8s.io/build-image/distroless-iptables:v0.2.5 (#118541, @jeremyrickard) [SIG Testing]

Bug or Regression

• Compute the backoff delay more accurately for deleted pods (#118413, @mimowo) [SIG Apps]
• Ensure Job status updates are batched by 1s. This fixes an unlikely scenario when a sequence of immediately
  completing pods could trigger a sequence of non-batched Job status updates. (#118470, @mimowo) [SIG Apps]
• Fix a race condition in kube-proxy when using LocalModeNodeCIDR to avoid dropping Services traffic if the object node is recreated when kube-proxy is starting (#118499, @aojea) [SIG Network]
• Fixed a race condition between Run() and SetTransform() and SetWatchErrorHandler() in shared informers. (#117870, @howardjohn) [SIG API Machinery]
• Fixes bug where explain was not properly respecting jsonpaths (#115694, @mpuckett159) [SIG CLI]
• Kubelet: print sorted volumes message in events (#117079, @qingwave) [SIG Node]

Other (Cleanup or Flake)

• E2e framework: the node-role.kubernetes.io/master taint has been removed from the default value of --non-blocking-taints flag. You may need to set --non-blocking-taints explicitly if the cluster to be tested has nodes with the deprecated node-role.kubernetes.io/master taint. (#118510, @SataQiu) [SIG Testing]
• Kube-apiserver adds two new alpha metrics conversion_webhook_request_total and conversion_webhook_duration_seconds that allow users to monitor requests to CRD conversion webhooks, split by result, and failure_type (In case of failure). (#118292, @cchapla) [SIG API Machinery, Architecture and Instrumentation]
• Moved k8s.io/kubernetes/pkg/kubelet/cri/streaming package to k8s.io/kubelet/pkg/cri/streaming. (#118253, @saschagrunert) [SIG Node, Release and Security]
• OpenAPI proto deserializations should use gnostic-models instead of the gnostic library (#118384, @Jefftree) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Instrumentation, Node, Storage and Testing]
• [KCCM] drop filtering nodes for the providerID when syncing load balancers, but have changes to the field trigger a re-sync of load balancers. This should ensure that cloud providers which don't specify providerID, can still use the service controller implementation to provision load balancers. (#117602, @alexanderConstantinescu) [SIG Cloud Provider and Network]

Dependencies

Added

• github.com/antlr/antlr4/runtime/Go/antlr/v4: 8188dc5
• github.com/google/gnostic-models: v0.6.8

Changed

• github.com/dustin/go-humanize: v1.0.0 → v1.0.1
• github.com/evanphx/json-patch: v4.12.0+incompatible → v5.6.0+incompatible
• github.com/go-openapi/jsonreference: v0.20.1 → v0.20.2
• github.com/google/cel-go: v0.12.6 → v0.16.0
• github.com/mitchellh/mapstructure: v1.4.1 → v1.1.2
• go.starlark.net: 8dd3e2e → a134d8f
• golang.org/x/exp: 6cc2880 → a9213ee
• golang.org/x/sys: v0.7.0 → v0.8.0
• k8s.io/kube-openapi: 7828149 → 7562a10
• sigs.k8s.io/kustomize/api: v0.13.2 → 6ce0bf3
• sigs.k8s.io/kustomize/cmd/config: v0.11.1 → v0.11.2
• sigs.k8s.io/kustomize/kustomize/v5: v5.0.1 → 6ce0bf3
• sigs.k8s.io/kustomize/kyaml: v0.14.1 → 6ce0bf3

Removed

• github.com/antlr/antlr4/runtime/Go/antlr: v1.4.10
• github.com/docopt/docopt-go: ee0de3b
• github.com/google/gnostic: v0.5.7-v3refs