Kubernetes - v1.29.1
Changelog since v1.29.0
Changes by Kind
API Change
- Fixes accidental enablement of the new alpha
optionalOldSelf
API field in CustomResourceDefinition validation rules, which should only be allowed to be set when the CRDValidationRatcheting feature gate is enabled. Existing CustomResourceDefinition objects which have the field set will retain it on update, but new CustomResourceDefinition objects will not be permitted to set the field while the CRDValidationRatcheting feature gate is disabled. (#122343, @jpbetz) [SIG API Machinery]
Feature
Bug or Regression
- Allow deletion of pods that use raw block volumes on node reboot (#122211, @gnufied) [SIG Node and Storage]
- Fix an issue where kubectl apply could panic when imported as a library (#122559, @Jefftree) [SIG CLI]
- Fix: Mount point may become local without calling NodePublishVolume after node rebooting. (#119923, @cvvz) [SIG Node and Storage]
- Fixed a regression since 1.24 in the scheduling framework when overriding MultiPoint plugins (e.g. default plugins).
The incorrect loop logic might lead to a plugin being loaded multiple times, consequently preventing any Pod from being scheduled, which is unexpected. (#122366, @caohe) [SIG Scheduling] - Fixed migration of in-tree vSphere volumes to the CSI driver. (#122341, @jsafrane) [SIG Storage]
- QueueingHint implementation for NodeAffinity is reverted because we found potential scenarios where events that make Pods schedulable could be missed. (#122327, @sanposhiho) [SIG Scheduling]
- QueueingHint implementation for NodeUnschedulable is reverted because we found potential scenarios where events that make Pods schedulable could be missed. (#122326, @sanposhiho) [SIG Scheduling]
Other (Cleanup or Flake)
- Reverts the EventedPLEG feature (beta, but disabled by default) back to alpha for a known issue (#122718, @pacoxu) [SIG Node]
Dependencies
Added
Nothing has changed.
Changed
- golang.org/x/crypto: v0.14.0 → v0.16.0
- golang.org/x/mod: v0.12.0 → v0.14.0
- golang.org/x/net: v0.17.0 → v0.19.0
- golang.org/x/sync: v0.3.0 → v0.5.0
- golang.org/x/sys: v0.13.0 → v0.15.0
- golang.org/x/term: v0.13.0 → v0.15.0
- golang.org/x/text: v0.13.0 → v0.14.0
- golang.org/x/tools: v0.12.0 → v0.16.1
Removed
Nothing has changed.
v1.29.0
Downloads for v1.29.0
Source Code
filename | sha512 hash
-------- | -----------
kubernetes.tar.gz | f07879916d7c4c7f8059ff9fd3c0006ce9bceb540874e183268a2bf2936df2632c4a3878a613cf2d695a80796e6c3eb52de5e3d83a73c91cb9a0bb5627091bae
kubernetes-src.tar.gz | a37a7927224785625e9863c1e2dcbc88943593d003b8d126fee63770e6b8eff122004d0f80e1301de34e8a2d6ce208ec6fa55cad3bbe8631b92e5469f45bd00d
Client Binaries
filename | sha512 hash
-------- | -----------
kubernetes-client-darwin-amd64.tar.gz | 22da1d2a217a8de91c1a8c393d17eb5ca81e243a1a3e509f3a40fb91d623670ace4ee87a09218a184aaa2eec4ca9c5478b992b8c6f136c568767d6e9dea493bf
kubernetes-client-darwin-arm64.tar.gz | cbc0cafecae18a50f98aaa8b508b1808a50b7a477638dc8699830a9dae7ffa83641f9fdb9f53616b32ebc8df84835fc847ea252c5ebe647c7d3462029a63b7a0
kubernetes-client-linux-386.tar.gz | f7ace756a3b6c56f2620d0ea6236fb94328c0a928094e4be7fbb78990a5771e8628bd93eac34017f3c33505c0248e8a64f933724a5fec6b322cf54dc30901985
kubernetes-client-linux-amd64.tar.gz | 6ff15bed6030c47e2ce90723500f08fa9968413f5b858456d4395bc67ab529b0b523ad0521e03be37664965e2fa588680aa0a5180054bc5cb3bafeef1497029b
kubernetes-client-linux-arm.tar.gz | bafe1ca945c41ae671029d5398e564bac0753400ee3a50dc0b4979284c0a905e8c77575d8b64b303e9c776d09c919d27f1f99847390d4e2e1c43be826a8dc1a4
kubernetes-client-linux-arm64.tar.gz | f3bca520625eaf6e6dd9af4cc709ff20bfce4da298a03e0be8835013a95fe0d6a25693d7702a4739c9477f9d49d2492d739718245ff91716fff90f60279ff376
kubernetes-client-linux-ppc64le.tar.gz | e6ea574272cefe9fd6e8eea2bddd89e1d67d0cb560089813e7429f3fb6d98be0c6601f33c8a0b2364d3becfb93c0904c171096ed6cafc4071e08851566d70d82
kubernetes-client-linux-s390x.tar.gz | b67dd572d84382e3f713d56bfb371de379807dca52cc4a1e082d6f4720a12770354ef2c9eac93bfc73bc0ea5f4be293db3b6c03328b94a797c2da17b9c40d9f3
kubernetes-client-windows-386.tar.gz | 0cf4b665f46e36616452916d744367b0ae2238098705b32de79559d06ea551173ab95190a26e87bebc03e67a75dc6a65699be3ef3db12aef82f32b66fd5afb0e
kubernetes-client-windows-amd64.tar.gz | 69cbe2b3942ba7d9c66e99f819adca94a9c7b420ad72cfd74407954c23ad70a4e7e76296824c4899f88232cabffe08d364c96af83bdaa538f29fa1303bcda2fa
kubernetes-client-windows-arm64.tar.gz | 44b0d1a7904bc2bf754abecb9b43a9efdc7cf700ab18f2564d95d98b4e38fe6d91f066943db7105baea964f86d77ade3b1acd57c7aaf1cdf689660f0d4422960
Server Binaries
filename | sha512 hash
-------- | -----------
kubernetes-server-linux-amd64.tar.gz | 651a8bf34acb6d61c39cc67ae23d9ef18204f95b309561d31f49da26c0c6a1b7585e7d7c2ac2f1522b2c326470a4e1ec9aa0dcf3bb1f66e1a41e6a2286e0aa5f
kubernetes-server-linux-arm64.tar.gz | 7f1f58b05c923d860f2daa6d31906faf834584b1560f4eda01ba5499338d07a7f183030ab625557b1f5df50a5f0ea30d97d487e2571c85260e5b88fc3519cd43
kubernetes-server-linux-ppc64le.tar.gz | 3ca2af4a7d68c0d84ef65e69190daeb2392946c87c6b8e84ff8d5cf917c979f0778fc00040d4b471e71b8474ca57ac8fdf786f006260d4403b53f59a203a48f1
kubernetes-server-linux-s390x.tar.gz | dfa172456f98210e614a9a538b9027ba211cc19f6eec22a42d5e89ce12d7f5e7e58dfd3229bb974ecba31ffafdf1a5361aef18b9610a45614a181918d87500db
Node Binaries
filename | sha512 hash
-------- | -----------
kubernetes-node-linux-amd64.tar.gz | 8057197e9354e2e0f48aab18c0ce87e4ea39c1682cfd4c491c2bc83f8881787b09cb0c9b9f4d7bef8fbe53cc4056f5381745dbfde7f7474bb76a2358b8b3953e
kubernetes-node-linux-arm64.tar.gz | 70d086c71f6258b1667bcb1efe60c15810b5b76848fdf26781c5a90efb8a78030e9ffb230bb0fd52d994f02b13c0b558c8e8ad3a42b601a0f9440a71cf91be2d
kubernetes-node-linux-ppc64le.tar.gz | 2740f6ac0dfeebbe4ba8804b43ec5968997d9137de9a9432861c3e71e614cb84b309da31bde3554f896f829a570c21b833f0af241659ad326fa753a80f185ec4
kubernetes-node-linux-s390x.tar.gz | 9877d5a6cc84569efe30256ba5e8095f38bfa0b11c28892499a12b577b467b516880a33022d88f65263c7ffa2a9a3687ef52cb85fa611e95b14ae0c5b7a79c5c
kubernetes-node-windows-amd64.tar.gz | 66b264de5e810bff31c4cf7cc575c3c57fed491fa4e21de7035dad76127e17d5fc88aff9f65277adf0826b255bf9b983f61c91bff2f8386d950f87509db6ec6b
Container Images
All container images are available as manifest lists and support the described
architectures. It is also possible to pull a specific architecture directly by
adding the "-$ARCH" suffix to the container image name.
name | architectures
---- | -------------
registry.k8s.io/conformance:v1.29.0 | amd64, arm64, ppc64le, s390x
registry.k8s.io/kube-apiserver:v1.29.0 | amd64, arm64, ppc64le, s390x
registry.k8s.io/kube-controller-manager:v1.29.0 | amd64, arm64, ppc64le, s390x
registry.k8s.io/kube-proxy:v1.29.0 | amd64, arm64, ppc64le, s390x
registry.k8s.io/kube-scheduler:v1.29.0 | amd64, arm64, ppc64le, s390x
registry.k8s.io/kubectl:v1.29.0 | amd64, arm64, ppc64le, s390x
Changelog since v1.28.0
Urgent Upgrade Notes
(No, really, you MUST read this before you upgrade)
- Stopped accepting component configuration for
kube-proxy
andkubelet
duringkubeadm upgrade plan --config
. This was a legacy behavior that was not well supported for upgrades and could be used only at the plan stage to determine if the configuration for these components stored in the cluster needs manual version migration. In the future,kubeadm
will attempt alternative component config migration approaches. (#120788, @chendave) kubeadm
: a separate "super-admin.conf" file is now deployed. The User inadmin.conf
is now bound to a new RBAC Groupkubeadm:cluster-admins
that hascluster-admin
ClusterRole
access. The User insuper-admin.conf
is now bound to thesystem:masters
built-in super-powers / break-glass Group that can bypass RBAC. Before this change, the defaultadmin.conf
was bound tosystem:masters
Group, which was undesired. Executingkubeadm init phase kubeconfig all
or justkubeadm init
will now generate the newsuper-admin.conf
file. The cluster admin can then decide to keep the file present on a node host or move it to a safe location.kubadm certs renew
will renew the certificate insuper-admin.conf
to one year if the file exists; if it does not exist a "MISSING" note will be printed.kubeadm upgrade apply
for this release will migrate this particular node to the two file setup. Subsequent kubeadm releases will continue to optionally renew the certificate insuper-admin.conf
if the file exists on disk and if renew on upgrade is not disabled.kubeadm join --control-plane
will now generate only anadmin.conf
file that has the less privileged User. (#121305, @neolit123)
Changes by Kind
Deprecation
-
Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:
(#119495, @bzsuni) [SIG API Machinery]
- Creation of new CronJob
objects containing TZ
or CRON_TZ
in .spec.schedule
, accidentally enabled in v1.22
, is now disallowed. Use the .spec.timeZone
field instead, supported in v1.25+
clusters in default configurations. See https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/#unsupported-timezone-specification for more information. (#116252, @soltysh)
- Removed the networking alpha
API ClusterCIDR
. (#121229, @aojea)
API Change
- '
kube-apiserver
: adds--authentication-config
flag for readingAuthenticationConfiguration
files.--authentication-config
flag is mutually exclusive with the existing--oidc-*
flags.' (#119142, @aramase) - '
kube-scheduler
component config (KubeSchedulerConfiguration
)kubescheduler.config.k8s.io/v1beta3
is removed inv1.29
. Migratedkube-scheduler
configuration files tokubescheduler.config.k8s.io/v1
.' (#119994, @SataQiu) - A new sleep action for the
PreStop
lifecycle hook was added, allowing containers to pause for a specified duration before termination. (#119026, @AxeZhan) - Added CEL expressions to
v1alpha1 AuthenticationConfiguration
. (#121078, @aramase) - Added Windows support for InPlace Pod Vertical Scaling feature. (#112599, @fabi200123) [SIG Autoscaling, Node, Scalability, Scheduling and Windows]
- Added
ImageMaximumGCAge
field to Kubelet configuration, which allows a user to set the maximum age an image is unused before it's garbage collected. (#121275, @haircommander) - Added
UserNamespacesPodSecurityStandards
feature gate to enable user namespace support for Pod Security Standards.
Enabling this feature will modify all Pod Security Standard rules to allow setting:spec[.*].securityContext.[runAsNonRoot,runAsUser]
.
This feature gate should only be enabled if all nodes in the cluster support the user namespace feature and have it enabled.
The feature gate will not graduate or be enabled by default in future Kubernetes releases. (#118760, @saschagrunert) [SIG API Machinery, Auth, Node and Release] - Added
optionalOldSelf
tox-kubernetes-validations
to support ratcheting CRD schema constraints. (#121034, @alexzielenski) - Added a new
ServiceCIDR
type that allows to dynamically configure the cluster range used to allocateService ClusterIPs
addresses. (#116516, @aojea) - Added a new
ipMode
field to the.status
of Services wheretype
is set toLoadBalancer
.
The new field is behind theLoadBalancerIPMode
feature gate. (#119937, @RyanAoh) [SIG API Machinery, Apps, Cloud Provider, Network and Testing] - Added options for configuring
nf_conntrack_udp_timeout
, andnf_conntrack_udp_timeout_stream
variables of netfilter conntrack subsystem. (#120808, @aroradaman) - Added support for CEL expressions to
v1alpha1 AuthorizationConfiguration
webhookmatchConditions
. (#121223, @ritazh) - Added support for projecting
certificates.k8s.io/v1alpha1
ClusterTrustBundle objects into pods. (#113374, @ahmedtd) - Added the
DisableNodeKubeProxyVersion
feature gate. IfDisableNodeKubeProxyVersion
is enabled, thekubeProxyVersion
field is not set. (#120954, @HirazawaUi) - Fixed a bug where CEL expressions in CRD validation rules would incorrectly compute a high estimated cost for functions that return strings, lists or maps.
The incorrect cost was evident when the result of a function was used in subsequent operations. (#119800, @jpbetz) [SIG API Machinery, Auth and Cloud Provider] - Fixed the API comments for the Job
Ready
field in status. (#121765, @mimowo) - Fixed the API comments for the
FailIndex
Job pod failure policy action. (#121764, @mimowo) - Go API: the
ResourceRequirements
struct was replaced withVolumeResourceRequirements
for use with volumes. (#118653, @pohly) - Graduated
Job BackoffLimitPerIndex
feature tobeta
. (#121356, @mimowo) - Marked the
onPodConditions
field as optional inJob
's pod failure policy. (#120204, @mimowo) - Promoted
PodReadyToStartContainers
condition tobeta
. (#119659, @kannon92) - The
flowcontrol.apiserver.k8s.io/v1beta3
FlowSchema
andPriorityLevelConfiguration
APIs has been promoted toflowcontrol.apiserver.k8s.io/v1
, with the following changes: PriorityLevelConfiguration
: the.spec.limited.nominalConcurrencyShares
field defaults to30
only if the field is omitted (v1beta3 also defaulted an explicit0
value to30
). Specifying an explicit0
value is not allowed in thev1
version in v1.29 to ensure compatibility withv1.28
API servers. Inv1.30
, explicit0
values will be allowed in this field in thev1
API.
Theflowcontrol.apiserver.k8s.io/v1beta3
APIs are deprecated and will no longer be served in v1.32. All existing objects are available via thev1
APIs. Transition clients and manifests to use thev1
APIs before upgrading tov1.32
. (#121089, @tkashem)- The
kube-proxy
command-line documentation was updated to clarify that
--bind-address
does not actually have anything to do with binding to an
address, and you probably don't actually want to be using it. (#120274, @danwinship) - The
kube-scheduler
selectorSpread
plugin has been removed, please use thepodTopologySpread
plugin instead. (#117720, @kerthcet) - The
matchLabelKeys/mismatchLabelKeys
feature is introduced to the hard/softPodAffinity/PodAntiAffinity
. (#116065, @sanposhiho) - When updating a CRD, per-expression cost limit check are now skipped for
x-kubernetes-validations
rules of versions that are not mutated. (#121460, @jiahuif) CSINodeExpandSecret
feature has been promoted toGA
in this release and is enabled
by default. The CSI drivers can make use of thesecretRef
values passed inNodeExpansion
request optionally sent by the CSI Client from this release onwards. (#121303, @humblec)NodeStageVolume
calls will now be retried if the CSI node driver is not running. (#120330, @rohitssingh)PersistentVolumeLastPhaseTransitionTime
is now beta and enabled by default. (#120627, @RomanBednar)ValidatingAdmissionPolicy
type checking now supports CRDs and API extensions types. (#119109, @jiahuif)kube-apiserver
: added--authorization-config
flag for reading a configuration file containing anapiserver.config.k8s.io/v1alpha1 AuthorizationConfiguration
object. The--authorization-config
flag is mutually exclusive with--authorization-modes
and--authorization-webhook-*
flags. Thealpha
StructuredAuthorizationConfiguration
feature flag must be enabled for--authorization-config
to be specified. (#120154, @palnabarun)-
kube-proxy
now has a new nftables-based mode, available by runningkube-proxy --feature-gates NFTablesProxyMode=true --proxy-mode nftables
This is currently an alpha-level feature and while it probably will not
eat your data, it may nibble at it a bit. (It passes e2e testing but has
not yet seen real-world use.)
At this point it should be functionally mostly identical to the iptables
mode, except that it does not (and will not) support Service NodePorts on
127.0.0.1. (Also note that there are currently no command-line arguments
for the nftables-specific config; you will need to use a config file if
you want to set the equivalent of any of the --iptables-xxx
options.)
As this code is still very new, it has not been heavily optimized yet;
while it is expected to eventually have better performance than the
iptables backend, very little performance testing has been done so far. (#121046, @danwinship)
- kube-proxy
: Added an option/flag for configuring the nf_conntrack_tcp_be_liberal
sysctl (in the kernel's netfilter conntrack subsystem). When enabled, kube-proxy
will not install the DROP
rule for invalid conntrack states, which currently breaks users of asymmetric routing. (#120354, @aroradaman)
Feature
-
Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:
(#119517, @sanposhiho) [SIG Node, Scheduling and Testing]
- 'kubeadm
: added validation to verify that the CertificateKey
is a valid hex
encoded AES key.' (#120064, @SataQiu)
- A customizable OrderedScoreFuncs()
function was introduced. Out-of-tree plugins
that used the scheduler's preemption interface could implement this function
for custom preemption preferences or return nil to keep the current behavior. (#121867, @lianghao208)
- Added apiextensions_apiserver_update_ratcheting_time
metric for tracking time taken during requests by feature CRDValidationRatcheting
. (#121462, @alexzielenski)
- Added apiserver_envelope_encryption_dek_cache_filled
to measure number of records in data encryption key (DEK) cache. (#119878, @ritazh)
- Added apiserver_watch_list_duration_seconds
metrics which will measure response latency distribution in seconds for watchlist requests broken by group, version, resource and scope. (#120490, @p0lyn0mial)
- Added job_pods_creation_total
metrics for tracking Pods created by the Job controller labeled by events which triggered the Pod creation. (#121481, @dejanzele)
- Added kubectl node drain
helper callbacks OnPodDeletionOrEvictionStarted
and OnPodDeletionOrEvictionFailed
; people extending kubectl
can use these
new callbacks for more granularity. Deprecated the OnPodDeletedOrEvicted
node drain helper callback. (#117502, @adilGhaffarDev)
- Added a new --init-only
command line flag to kube-proxy
. Setting the flag makes kube-proxy
perform its initial configuration that requires privileged mode, and then exit. The --init-only
mode is intended to be executed in a privileged init container, so that the main container may run with a stricter securityContext
. (#120864, @uablrek) [SIG Network and Scalability]
- Added a new scheduler metric, pod_scheduling_sli_duration_seconds
, and started the deprecation for pod_scheduling_duration_seconds
. (#119049, @helayoty)
- Added a return value to QueueingHint
to indicate an error. If QueueingHint
returns an error,
the scheduler logs it and treats the event as a QueueAfterBackoff
so that
the Pod won't be stuck in the unschedulable pod pool. (#119290, @carlory)
- Added apiserver identity to the following metrics:
apiserver_envelope_encryption_key_id_hash_total
, apiserver_envelope_encryption_key_id_hash_last_timestamp_seconds
, apiserver_envelope_encryption_key_id_hash_status_last_timestamp_seconds
, apiserver_encryption_config_controller_automatic_reload_failures_total
, apiserver_encryption_config_controller_automatic_reload_success_total
, apiserver_encryption_config_controller_automatic_reload_last_timestamp_seconds
.
Fixed bug to surface events for the following metrics: apiserver_encryption_config_controller_automatic_reload_failures_total
, apiserver_encryption_config_controller_automatic_reload_last_timestamp_seconds
, apiserver_encryption_config_controller_automatic_reload_success_total
. (#120438, @ritazh)
- Added container filesystem to the ImageFsInfoResponse
. (#120914, @kannon92)
- Added multiplication functionality to Quantity
. (#117411, @tenzen-y)
- Added new feature gate called RuntimeClassInImageCriApi
to address kubelet
changes needed for KEP 4216.
Noteable changes:
1. Populate new RuntimeHandler
field in CRI's ImageSpec
struct during image pulls from container runtimes.
2. Pass runtimeHandler
field in RemoveImage()
call to container runtime in kubelet
's image garbage collection. (#121456, @kiashok)
- Added support for split image filesystem in kubelet. (#120616, @kannon92)
- Bumped cel-go
to v0.17.7
and introduced set ext
library with new options. (#121577, @cici37)
- Bumped distroless-iptables
to 0.3.2
based on Go 1.21.1
. (#120527, @cpanato)
- Bumped distroless-iptables
to 0.3.3
based on Go 1.21.2
. (#121073, @cpanato)
- Bumped distroless-iptables
to 0.4.1
based on Go 1.21.3
. (#121216, @cpanato)
- Bumped distroless-iptables to 0.4.1 based on Go 1.21.3
. (#121871, @cpanato)
- CEL can now correctly handle a CRD openAPIV3Schema
that has neither Properties
nor AdditionalProperties
. (#121459, @jiahuif)
- CEL cost estimator no longer treats enums as unbounded strings when determining its length. Instead, the length is set to the longest possible enum value. (#121085, @jiahuif) [SIG API Machinery]
- CRI: image pull per runtime class is now supported. (#121121, @kiashok)
- Certain requestBody
parameters in the OpenAPI v3
are now correctly marked as required. (#120735, @Jefftree)
- Changed kubectl help
to display basic details for subcommands from plugins. (#116752, @xvzf)
- Changed the KMSv2KDF
feature gate to be enabled by default. (#120433, @enj) [SIG API Machinery, Auth and Testing]
- Client-side apply will now use OpenAPI v3
by default. (#120707, @Jefftree)
- Decoding etcd's response now respects the timeout context. (#121614, @HirazawaUi)
- Decoupled TaintManager
from NodeLifeCycleController
(KEP-3902). (#119208, @atosatto)
- Enabled traces for KMSv2 encrypt/decrypt operations. (#121095, @aramase)
- Fixed kube-proxy
panicking on exit when the Node
object changed its PodCIDR
. (#120375, @pegasas)
- Fixed bugs in handling of server-side apply, create, and update API requests for objects containing duplicate items in keyed lists.
- A create
or update
API request with duplicate items in a keyed list no longer wipes out managedFields. Examples include env var entries with the same name, or port entries with the same containerPort in a pod spec.
- A server-side apply request that makes unrelated changes to an object which has duplicate items in a keyed list no longer fails, and leaves the existing duplicate items as-is.
- A server-side apply request that changes an object which has duplicate items in a keyed list, and modifies the duplicated item removes the duplicates and replaces them with the single item contained in the server-side apply request. (#121575, @apelisse)
- Fixed overriding default KubeletConfig
fields in drop-in configs if not set. (#121193, @sohankunkerkar)
- Graduated API List chunking (aka pagination) feature to stable
. (#119503, @wojtek-t)
- Graduated the ReadWriteOncePod
feature gate to GA
. (#121077, @chrishenzie)
- Graduated the following kubelet resource metrics to general availability:
- container_cpu_usage_seconds_total
- container_memory_working_set_bytes
- container_start_time_seconds
- node_cpu_usage_seconds_total
- node_memory_working_set_bytes
- pod_cpu_usage_seconds_total
- pod_memory_working_set_bytes
- resource_scrape_error
Deprecated (renamed) scrape_error
in favor of resource_scrape_error
(#116897, @Richabanker) [SIG Architecture, Instrumentation, Node and Testing]
- Implemented API for streaming for the etcd
store implementation.
When sendInitialEvents ListOption
is set together with watch=true
, it begins the watch stream with synthetic init events followed by a synthetic Bookmark
, after which the server continues streaming events. (#119557, @p0lyn0mial)
- Improved memory usage of kube-scheduler
by dropping the .metadata.managedFields
field that kube-scheduler
doesn't require. (#119556, @linxiulei)
- In a scheduler with Permit
plugins, when a Pod is rejected during WaitOnPermit
, the scheduler records the plugin.
The scheduler will use the record to honor cluster events and queueing hints registered
for the plugin, to inform whether to retry the pod. (#119785, @sanposhiho)
- In-tree cloud providers are now switched off by default. Please use DisableCloudProviders
and DisableKubeletCloudCredentialProvider
feature flags if you still need this functionality. (#117503, @dims)
- Introduced new apiserver metric apiserver_flowcontrol_current_inqueue_seats
. This metric is analogous to apiserver_flowcontrol_current_inqueue_requests
, but tracks the total number of seats, as each request can take more than one seat. (#119385, @andrewsykim)
- Introduced the job_finished_indexes_total
metric for the BackoffLimitPerIndex
feature. (#121292, @mimowo)
- Kubeadm: supported updating certificate organization during kubeadm certs renew
operation. (#121841, @SataQiu)
- Kubernetes is now built with Go 1.21.0
. (#118996, @cpanato)
- Kubernetes is now built with Go 1.21.1
. (#120493, @cpanato)
- Kubernetes is now built with Go 1.21.2
. (#121021, @cpanato)
- Kubernetes is now built with Go 1.21.4
. (#121808, @cpanato)
- Kubernetes is now built with Go v1.21.3
. (#121149, @cpanato)
- List of metric labels can now be configured by supplying a manifest using the --allow-metric-labels-manifest
flag. (#118299, @rexagod)
- Listed the pods using <PVC>
as an ephemeral storage volume in "Used by:" part of the output of kubectl describe pvc <PVC>
command. (#120427, @MaGaroo)
- Migrated the nodevolumelimits
scheduler plugin to use contextual logging. (#116884, @mengjiao-liu)
- Migrated the volumebinding scheduler plugins
to use contextual logging. (#116803, @mengjiao-liu)
- Priority and Fairness feature is now stable
, the feature gate will be removed in v1.31
. (#121638, @tkashem)
- Promoted PodHostIPs
condition to beta
. (#120257, @wzshiming)
- Promoted PodHostIPs
condition to beta
. (#121477, @wzshiming)
- Promoted PodReplacementPolicy
to beta
. (#121491, @dejanzele)
- Promoted ServiceNodePortStaticSubrange
to stable and lock to default. (#120233, @xuzhenglun)
- Promoted plugin subcommand resolution feature to beta
. (#120663, @ardaguclu)
- Removed /livez
livezchecks for KMS v1 and v2 to ensure KMS health does not cause kube-apiserver
restart. KMS health checks are still in place as a healthz and readiness checks. (#120583, @ritazh)
- Restartable init containers resource in pod autoscaler are now calculated. (#120001, @qingwave)
- Sidecar termination is now serialized and each sidecar container will receive a SIGTERM
after all main containers and later starting sidecar containers have terminated. (#120620, @tzneal)
- The CRD validation rule with feature gate CustomResourceValidationExpressions
was promoted to GA
. (#121373, @cici37)
- The KMSv2 features with feature gates KMSv2
and KMSv2KDF
are promoted to GA
. The KMSv1
feature gate is now disabled by default. (#121485, @ritazh)
- The --interactive
flag in kubectl delete
is now visible to all users by default. (#120416, @ardaguclu)
- The CloudDualStackNodeIPs
feature is now beta
, meaning that when using
an external cloud provider that has been updated to support the feature,
you can pass comma-separated dual-stack --node-ips
to kubelet
and have
the cloud provider take both IPs into account. (#120275, @danwinship)
- The Dockerfile
for the kubectl image has been updated with the addition of a specific base image and essential utilities (bash and jq). (#119592, @rayandas)
- The SidecarContainers
feature has graduated to beta
and is enabled by default. (#121579, @gjkim42)
- The kube-apiserver
will now expose four new metrics to inform about errors on the clusterIP and nodePort allocation logic. (#120843, @aojea)
- The volume_zone
plugin will consider beta
labels as GA
labels during the scheduling
process. Therefore, if the values of the labels are the same, PVs with beta
labels
can also be scheduled to nodes with GA
labels. (#118923, @AxeZhan)
- Updated the generic apiserver library to produce an error if a new API server is configured with support for a data format other than JSON, YAML, or Protobuf. (#121325, @benluddy) [SIG API Machinery]
- Use of secret-based service account tokens now adds an authentication.k8s.io/legacy-token-autogenerated-secret
or authentication.k8s.io/legacy-token-manual-secret
audit annotation containing the name of the secret used. (#118598, @yuanchen8911) [SIG Auth, Instrumentation and Testing]
- --sync-frequency
will not affect the update interval of volumes that use ConfigMaps
or Secrets
when the configMapAndSecretChangeDetectionStrategy
is set to Cache
.
The update interval is only affected by node.alpha.kubernetes.io/ttl
node annotation." (#120255, @likakuli)
- CRDValidationRatcheting
: added support for ratcheting x-kubernetes-validations
in schema. (#121016, @alexzielenski)
- DevicePluginCDIDevices
feature has been graduated to beta
and enabled by default in the kubelet. (#121254, @bart0sh)
- ValidatingAdmissionPolicy
now preserves types of composition variables, and raises type-related errors early. (#121001, @jiahuif)
- cluster/gce
: added webhook to replace PersistentVolumeLabel
admission controller. (#121628, @andrewsykim)
- dra
: the scheduler plugin now avoids additional scheduling attempts in some cases by falling back to SSA after a conflict. (#120534, @pohly)
- etcd
: image is now based on v3.5.9
. (#121567, @mzaian)
- kube-apiserver
added:
- alpha
support (guarded by the ServiceAccountTokenJTI
feature gate) for adding a jti
(JWT ID) claim to service account tokens it issues, adding an authentication.kubernetes.io/credential-id
audit annotation in audit logs when the tokens are issued, and authentication.kubernetes.io/credential-id
entry in the extra user info when the token is used to authenticate.
- alpha
support (guarded by the ServiceAccountTokenPodNodeInfo
feature gate) for including the node name (and uid, if the node exists) as additional claims in service account tokens it issues which are bound to pods, and authentication.kubernetes.io/node-name
and authentication.kubernetes.io/node-uid
extra user info when the token is used to authenticate.
- alpha
support (guarded by the ServiceAccountTokenNodeBinding
feature gate) for allowing TokenRequests
that bind tokens directly to nodes, and (guarded by the ServiceAccountTokenNodeBindingValidation feature gate) for validating the node name and uid still exist when the token is used. (#120780, @munnerz)
- kube-controller-manager
: The LegacyServiceAccountTokenCleanUp
feature gate is now beta
and enabled by default. When enabled, legacy auto-generated service account token secrets are auto-labeled with a kubernetes.io/legacy-token-invalid-since
label if the credentials have not been used in the time specified by --legacy-service-account-token-clean-up-period
(defaulting to one year), and are referenced from the .secrets
list of a ServiceAccount object, and are not referenced from pods. This label causes the authentication layer to reject use of the credentials. After being labeled as invalid, if the time specified by --legacy-service-account-token-clean-up-period
(defaulting to one year) passes without the credential being used, the secret is automatically deleted. Secrets labeled as invalid which have not been auto-deleted yet can be re-activated by removing the kubernetes.io/legacy-token-invalid-since
label. (#120682, @yt2985)
- kube-proxy
will only install the DROP
rules for invalid conntrack
states if
the nf_conntrack_tcp_be_liberal
is not set. (#120412, @aojea)
- kube-scheduler
implemented scheduling hints for the NodeUnschedulable
plugin.
The scheduling hints allow the scheduler to only retry scheduling a Pod
that was previously rejected by the NodeSchedulable
plugin if a new Node
or a Node
update sets .spec.unschedulable
to false. (#119396, @wackxu)
- kube-scheduler
implements scheduling hints for the NodeAffinity
plugin.
The scheduling hints allow the scheduler to only retry scheduling a Pod
that was previously rejected by the NodeAffinity
plugin if a new Node
or a Node
update matches the Pod
's node affinity. (#119155, @carlory)
- kubeadm
: promoted feature gate EtcdLearnerMode
to beta
. Learner mode for
joining etcd
members is now enabled by default. (#120228, @pacoxu)
- kubeadm
: turned on feature gate MergeCLIArgumentsWithConfig
to merge the config from flag and config file, otherwise, if the flag --ignore-preflight-errors
is set from the CLI, then the value from config file will be ignored. (#119946, @chendave)
- kubeadm
: will now allow deploying a kubelet that is 3 versions older than the version of kubeadm
(N-3). This aligns with the recent change made by SIG Architecture that extends the support skew between the control plane and kubelets. Tolerate this new kubelet skew for the commands init
, join
and upgrade
. Note that if the kubeadm
user applies a control plane version that is older than the kubeadm
version (N-1 maximum) then the skew between the kubelet and control plane would become a maximum of N-2. (#120825, @pacoxu)
- kubelet
, when using --cloud-provider=external
, will now initialize the node addresses with the value of --node-ip
, if it exists, or waits for the cloud provider to assign the addresses. (#121028, @aojea)
- kubelet
allows pods to use the net.ipv4.tcp_fin_timeout
, “net.ipv4.tcp_keepalive_intvl”
and “net.ipv4.tcp_keepalive_probes“ sysctl by default; Pod Security Admission
allows this sysctl in v1.29+
versions of the baseline and restricted policies. (#121240, @HirazawaUi)
- kubelet
now allows pods to use the net.ipv4.tcp_keepalive_time
sysctl by default
and the minimal kernel version is 4.5; Pod Security Admission allows this sysctl
in v1.29+
versions of the baseline and restricted policies. (#118846, @cyclinder)
- kubelet
now emits a metric for end-to-end pod startup latency, including image pull. (#121041, @ruiwen-zhao)
- kubelet
now exposes latency metrics of different stages of the node startup. (#118568, @qiutongs)
Documentation
- Added descriptions and examples for the situation of using
kubectl rollout restart
without specifying a particular deployment. (#120118, @Ithrael) - When the kubelet fails to assign CPUs to a Pod because there less available CPUs than the Pod requests, the error message changed from
not enough cpus available to satisfy request
tonot enough cpus available to satisfy request: <num_requested> requested, only <num_available> available
. (#121059, @matte21)
Failing Test
- Added mock framework support for unit tests for Windows in
kubeproxy
. (#120105, @princepereira) - DRA: when the scheduler had to deallocate a claim after a node became unsuitable for a pod, it might have needed more attempts than really necessary. This was fixed by first disabling allocations. (#120428, @pohly)
- E2e framework: retrying after intermittent
apiserver
failures was fixed inWaitForPodsResponding
(#120559, @pohly) - KCM specific args can be passed with
/cluster
script, without affecting CCM. New variable name:KUBE_CONTROLLER_MANAGER_TEST_ARGS
. (#120524, @jprzychodzen) [SIG Cloud Provider] k8s.io/dynamic-resource-allocation
: DRA drivers updating to this release are compatible with Kubernetesv1.27
andv1.28
. (#120868, @pohly)
Bug or Regression
- '
kubeadm
: printing the default component configs forreset
andjoin
is now
unsupported.' (#119346, @chendave) - '
kubeadm
: removedsystem:masters
organization frometcd/healthcheck-client
certificate.' (#119859, @SataQiu) - Added
CAP_NET_RAW
to netadmin debug profile and removed privileges when debugging nodes. (#118647, @mochizuki875) - Added a check on a user attempting to create a static pod via the
kubelet
without specifying a name. They will now get a visible validation error. (#119522, @YTGhost) - Added a redundant process to remove tracking finalizers from Pods that belong to Jobs. The process kicks in after the control plane marks a Job as finished. (#119944, @Sharpz7)
- Added more accurate requeueing in scheduling queue for Pods rejected by the temporal failure (e.g., temporal failure on
kube-apiserver
). (#119105, @sanposhiho) - Allowed specifying
ExternalTrafficPolicy
forServices
withExternalIPs
. (#119150, @tnqn) - Changed kubelet logs from
error
toinfo
for uncached partitions when using CRI stats provider. (#100448, @saschagrunert) - Empty values are no longer assigned to undefined resources (CPU or memory) when storing the resources allocated to the pod in checkpoint. (#117615, @aheng-ch)
- Fixed CEL estimated cost of
replace()
to handle a zero length replacement string correctly.
Previously this would cause the estimated cost to be higher than it should be. (#120097, @jpbetz) [SIG API Machinery] - Fixed OpenAPI v3 not being cleaned up after deleting
APIServices
. (#120108, @tnqn) - Fixed 121094 by re-introducing the readiness predicate for
externalTrafficPolicy: Local
services. (#121116, @alexanderConstantinescu) - Fixed
kubectl events
not filtering events byGroupVersion
for resources with a full name. (#120119, @Ithrael) - Fixed
systemLogQuery
service name matching. (#120678, @rothgar) - Fixed a
1.27
scheduling regression thatPostFilter
plugin may not function if previousPreFilter
plugins returnSkip
. (#119769, @Huang-Wei) - Fixed a
v1.26
regression scheduling bug by ensuring that preemption is skipped when aPreFilter
plugin returnsUnschedulableAndUnresolvable
. (#119778, @sanposhiho) - Fixed a
v1.28.0
regression wherekube-controller-manager
can crash whenStatefulSet
withParallel
policy and PVC labels are scaled up. (#121142, @aleksandra-malinowska) - Fixed a
v1.28
regression around restarting init containers in the right order relative to normal containers. (#120281, @gjkim42) - Fixed a
v1.28
regression handling negative index json patches. (#120327, @liggitt) - Fixed a
v1.28
regression in scheduler: a pod with concurrent events could incorrectly get moved to the unschedulable queue where it could get stuck until the next periodic purging after 5 minutes, if there was no other event for it. (#120413, @pohly) - Fixed a bug around restarting init containers in the right order relative to normal containers with
SidecarContainers
feature enabled. (#120269, @gjkim42) - Fixed a bug in the cronjob controller where already created jobs might be missing from the status. (#120649, @andrewsykim)
- Fixed a bug where
Services
using finalizers may hold ontoClusterIP
and/orNodePort
allocated resources for longer than expected if the finalizer is removed using the status subresource. (#120623, @aojea) - Fixed a bug where an API group's path was not unregistered from the API server's root paths when the group was deleted. (#121283, @tnqn) [SIG API Machinery and Testing]
- Fixed a bug where containers would not start on
cgroupv2
systems whereswap
is disabled. (#120784, @elezar) - Fixed a bug where the CPU set allocated to an init container, with containerRestartPolicy of
Always
, were erroneously reused by a regular container. (#119447, @gjkim42) [SIG Node and Testing] - Fixed a bug where the device resources allocated to an init container, with
containerRestartPolicy
ofAlways
, were erroneously reused by a regular container. (#120461, @gjkim42) - Fixed a bug where the memory resources allocated to an init container, with containerRestartPolicy of
Always
, were erroneously reused by a regular container. (#120715, @gjkim42) [SIG Node] - Fixed a concurrent map access in
TopologyCache
'sHasPopulatedHints
method. (#118189, @Miciah) - Fixed a regression (
CLIENTSET_PKG: unbound variable
) when invoking deprecatedgenerate-groups.sh
script. (#120877, @soltysh) - Fixed a regression in
kube-proxy
where it might refuse to start if given
single-stackIPv6
configuration options on a node that has bothIPv4
and
IPv6
IPs. (#121008, @danwinship) - Fixed a regression in default configurations, which enabled
PodDisruptionConditions
by default, that prevented the control plane's pod garbage collector from deleting
pods that contained duplicated field keys (environmental variables with repeated keys or
container ports). (#121103, @mimowo) - Fixed a regression in the default
v1.27
configurations inkube-apiserver
: fixed theAggregatedDiscoveryEndpoint
feature (beta
inv1.27+
) to successfully fetch discovery information from aggregated API servers that do not checkAccept
headers when serving the/apis
endpoint. (#119870, @Jefftree) - Fixed a regression in the kubelet's behavior while creating a container when the
EventedPLEG
feature gate is enabled. (#120942, @sairameshv) - Fixed a regression since
v1.27.0
in the scheduler framework when running score plugins.
TheskippedScorePlugins
number might be greater thanenabledScorePlugins
,
so when initializing a slice thecap(len(skippedScorePlugins) - len(enabledScorePlugins))
is negative,
which is not allowed. (#121632, @kerthcet) - Fixed a situation when, sometimes, the scheduler incorrectly placed a pod in the
unschedulable
queue instead of thebackoff
queue. This happened when some plugin previously declared the pod asunschedulable
and then in a later attempt encounters some other error. Scheduling of that pod then got delayed by up to five minutes, after which periodic flushing moved the pod back into theactive
queue. (#120334, @pohly) - Fixed an issue related to not draining all the pods in a namespace when an empty selector, i.e., "{}," is specified in a Pod Disruption Budget (PDB). (#119732, @sairameshv)
- Fixed an issue where
StatefulSet
might not restart a pod after eviction or node failure. (#120398, @aleksandra-malinowska) - Fixed an issue where a
CronJob
could fail to clean up Jobs when theResourceQuota
forJobs
had been reached. (#119776, @ASverdlov) - Fixed an issue where a
StatefulSet
might not restart a pod after eviction or node failure. (#121389, @aleksandra-malinowska) - Fixed an issue with the
garbagecollection
controller registering duplicate event handlers if discovery requests failed. (#117992, @liggitt) - Fixed attaching volumes after detach errors. Now volumes that failed to detach are not treated as attached. Kubernetes will make sure they are fully attached before they can be used by pods. (#120595, @jsafrane)
- Fixed bug that kubelet resource metric
container_start_time_seconds
had timestamp equal to container start time. (#120518, @saschagrunert) [SIG Instrumentation, Node and Testing] - Fixed inconsistency in the calculation of number of nodes that have an image, which affect the scoring in the
ImageLocality
plugin. (#116938, @olderTaoist) - Fixed issue with incremental id generation for
loadbalancer
andendpoint
inkubeproxy
mock test framework. (#120723, @princepereira) - Fixed panic in Job controller when
podRecreationPolicy: Failed
is used, and the number of terminating pods exceeds parallelism. (#121147, @kannon92) - Fixed regression with adding aggregated
APIservices
panicking and affected health check introduced in releasev1.28.0
. (#120814, @Jefftree) - Fixed some invalid and unimportant log calls. (#121249, @pohly) [SIG Cloud Provider, Cluster Lifecycle and Testing]
- Fixed stale SMB mount issue when SMB file share is deleted and then unmounted. (#121851, @andyzhangx)
- Fixed the bug where images that were pinned by the container runtime could be garbage collected by
kubelet
. (#119986, @ruiwen-zhao) - Fixed the bug where kubelet couldn't output logs after log file rotated when
kubectl logs POD_NAME -f
is running. (#115702, @xyz-li) - Fixed the calculation of the requeue time in the cronjob controller, resulting in proper handling of failed/stuck jobs. (#121327, @soltysh)
- Fixed the issue where pod with ordinal number lower than the rolling partitioning number was being deleted. It was coming up with updated image. (#120731, @adilGhaffarDev)
- Fixed tracking of terminating Pods in the Job status. The field was not updated unless there were other changes to apply. (#121342, @dejanzele)
- Forbidden sysctls for pod sharing the respective namespaces with the host are now checked when creating or updating pods without such sysctls. (#118705, @pacoxu)
- If a watch with the
progressNotify
option set is to be created, and the registry hasn't provided anewFunc
, return an error. (#120212, @p0lyn0mial) [SIG API Machinery] - Improved handling of jsonpath expressions for
kubectl wait --for
. It is now possible to use simple filter expressions which match on a field's content. (#118748, @andreaskaris) - In the
wait.PollUntilContextTimeout
function, ifimmediate
is true, the condition will now be invoked before waiting, guaranteeing that the condition is invoked at least once and then wait a interval before executing again. (#119762, @AxeZhan) - Incorporating feedback on PR #119341 (#120087, @divyasri537) [SIG API Machinery]
- KCCM: fixed transient node addition and removal caused by #121090 while syncing load balancers on large clusters with a lot of churn. (#121091, @alexanderConstantinescu)
- Kubeadm: changed the "system:masters" Group in the apiserver-kubelet-client.crt certificate Subject to be "kubeadm:cluster-admins" which is a less privileged Group. (#121837, @neolit123)
- Metric buckets for
pod_start_duration_seconds
were changed to{0.5, 1, 2, 3, 4, 5, 6, 8, 10, 20, 30, 45, 60, 120, 180, 240, 300, 360, 480, 600, 900, 1200, 1800, 2700, 3600}
. (#120680, @ruiwen-zhao) - Mitigated http/2 DOS vulnerabilities for
CVE-2023-44487
andCVE-2023-39325
for the API server when the client is unauthenticated. The mitigation may be disabled by setting theUnauthenticatedHTTP2DOSMitigation
feature gate tofalse
(it is enabled by default). An API server fronted by an L7 load balancer that already mitigates these http/2 attacks may choose to disable the kube-apiserver mitigation to avoid disrupting load balancer -> kube-apiserver connections if http/2 requests from multiple clients share the same backend connection. An API server on a private network may opt to disable the kube-apiserver mitigation to prevent performance regressions for unauthenticated clients. Authenticated requests rely on the fix in golang.org/x/netv0.17.0
alone. https://issue.k8s.io/121197 tracks further mitigation of http/2 attacks by authenticated clients. (#121120, @enj) - No-op and GC related updates to cluster trust bundles no longer require attest authorization when the
ClusterTrustBundleAttest
plugin is enabled. (#120779, @enj) - Registered metric
apiserver_request_body_size_bytes
to track the size distribution of requests byresource
andverb
. (#120474, @YaoC) [SIG API Machinery and Instrumentation] - Revised the logic for
DaemonSet
rolling update to exclude nodes if scheduling constraints are not met. This eliminates the problem of rolling updates to aDaemonSet
getting stuck around tolerations. (#119317, @mochizuki875) - Scheduler: in 1.29 pre-releases, enabling contextual logging slowed down pod scheduling. (#121715, @pohly) [SIG Instrumentation and Scheduling]
- Service Controller: will now update load balancer hosts after node's
ProviderID
is
updated. (#120492, @cezarygerard) - Setting the
status.loadBalancer
of a Service whosespec.type
is notLoadBalancer
was previously allowed, but any update to themetadata
orspec
would wipe that field. Setting this field is no longer permitted unlessspec.type
isLoadBalancer
. In the very unlikely event that this has unexpected impact, you can enable theAllowServiceLBStatusOnNonLB
feature gate, which will restore the previous behavior. If you do need to set this, please file an issue with the Kubernetes project to help contributors understand why you need it. (#119789, @thockin) - The
--bind-address
parameter in kube-proxy is misleading, no port is opened with this address. Instead it is translated internally to "nodeIP". The nodeIPs for both families are now taken from the Node object if--bind-address
is unspecified or set to the "any" address (0.0.0.0 or ::). It is recommended to leave--bind-address
unspecified, and in particular avoid to set it to localhost (127.0.0.1 or ::1) (#119525, @uablrek) [SIG Network and Scalability] - Updated
kube-openapi
to remove invalid defaults: OpenAPI spec no longer includes default of{}
for certain fields where it did not make sense. (#120757, @alexzielenski) - Updated the CRI-O socket path, so users who configure kubelet to use a location like
/run/crio/crio.sock
don't see strange behaviour from CRI stats provider. (#118704, @dgl) - Volume attach or publish operation will not fail at
kubelet
if target path directory already exists on the node. (#119735, @akankshapanse) cluster-bootstrap
: improved the security of the functions responsible for generation and validation of bootstrap tokens. (#120400, @neolit123)etcd
: updated tov3.5.10
. (#121566, @mzaian)k8s.io/dynamic-resource-allocation/controller:
UnsuitableNodes
can now handle a mix of allocated and unallocated claims correctly. (#120338, @pohly)k8s.io/dynamic-resource-allocation/controller
:ResourceClaimParameters
andResourceClassParameters
validation errors are now visible onResourceClaim
,ResourceClass
andPod
. (#121065, @byako)k8s.io/dynamic-resource-allocation
: can now handle aselected
node which isn't listed
aspotential
node. (#120871, @pohly)kube-proxy
now reports its health more accurately in dual-stack clusters when there are problems with only one IP family. (#118146, @aroradaman)kubeadm
: Fixed the bug where it always did CRI detection when--config
was passed, even if it is not required by the subcommand. (#120828, @SataQiu)kubeadm
: fixednil
pointer whenetcd
member is already removed. (#119753, @pacoxu)kubeadm
: fixed the bug where--image-repository
flag is missing for some init
phase sub-commands. (#120072, @SataQiu)kubeadm
: improved the logic that checks whether asystemd
service exists. (#120514, @fengxsong)kubeadm
: will now use universal deserializer to decode static pod. (#120549, @pacoxu)kubectl prune v2
: Switched annotation fromcontains-group-resources
tocontains-group-kinds
,
because this is what we defined in the KEP and is clearer to end-users. Although the functionality is
inalpha
, we will recognize the prior annotation. This migration support will be removed inbeta
/GA
. (#118942, @justinsb)kubectl
will not print events if--show-events=false
argument is passed to
describe PVC subcommand. (#120380, @MaGaroo)scheduler
: Fixed missing fieldapiVersion
from events reported by the taint
manager. (#114095, @aimuz)
Other (Cleanup or Flake)
- Added automatic download of the CNI binary in
local-up-cluster.sh
, facilitating local debugging. (#120312, @HirazawaUi) - Added context to
caches populated
log messages. (#119796, @sttts) - Changed behavior of
kube-proxy
by allowing to setsysctl
values lower than the existing one. (#120448, @aroradaman) - Cleaned up
kube-apiserver
HTTP logs for impersonated requests. (#119795, @sttts) - Deprecated the
--cloud-provider
and--cloud-config
CLI parameters in kube-apiserver.
These parameters will be removed in a future release. (#120903, @dims) [SIG API Machinery] - Dynamic resource allocation: will now avoid creating a new gRPC connection for every call of prepare/unprepare resource(s). (#118619, @TommyStarK)
- E2E storage tests: setting test tags like
[Slow]
via theDriverInfo.FeatureTag
field is no longer supported. (#121391, @pohly) - Fixed an issue where the
vsphere
cloud provider would not trust a certificate if: - The issuer of the certificate was unknown (
x509.UnknownAuthorityError
) - The requested name did not match the set of authorized names (
x509.HostnameError
) - The error surfaced after attempting a connection contained one of the substrings: "certificate is not trusted" or "certificate signed by unknown authority". (#120736, @MadhavJivrajani)
- Fixed bug where
Adding GroupVersion
log line was constantly repeated without any group version changes. (#119825, @Jefftree) - Generated
ResourceClaim
names are now more readable because of an additional hyphen before the random suffix (<pod name>-<claim name>-<random suffix>
). (#120336, @pohly) - Graduated
JobReadyPods
tostable
. The feature gate can no longer be disabled. (#121302, @stuton) - Improved memory usage of
kube-controller-manager
by dropping the.metadata.managedFields
field thatkube-controller-manager
doesn't require. (#118455, @linxiulei) - Lower and upper case feature flag values are now allowed, but the name still has to match. (#121441, @soltysh)
- Makefile and scripts now respect
GOTOOLCHAIN
and otherwise ensure./.go-version
is used. (#120279, @BenTheElder) - Migrated the remainder of the scheduler to use contextual logging. (#120933, @mengjiao-liu) [SIG Instrumentation, Scheduling and Testing]
- Optimized
NodeUnschedulable
Filter to avoid unnecessary calculations. (#119399, @wackxu) - Previous versions of Kubernetes on Google Cloud required that workloads (e.g. Deployments, DaemonSets, etc.) which used
PersistentDisk
volumes were using them in read-only mode. This validation provided very little value at relatively host implementation cost, and will no longer be validated. If this is a problem for a specific use-case, please set theSkipReadOnlyValidationGCE
gate to false to re-enable the validation, and file a Kubernetes bug with details. (#121083, @thockin) - Previously, the pod name and namespace were eliminated in the event log message. This PR attempts to add the preemptor pod UID in the preemption event message logs for easier debugging and safer transparency. (#119971, @kwakubiney) [SIG Scheduling]
- Promoted to conformance a test that verified that
Services
only forward traffic on the port and protocol specified. (#120069, @aojea) - Removed
GA
feature gate aboutCSIMigrationvSphere
. (#121291, @bzsuni) - Removed
GA
feature gate aboutProbeTerminationGracePeriod
. (#121257, @bzsuni) - Removed
GA
feature gate forJobTrackingWithFinalizers
inv1.28
. (#119100, @bzsuni) - Removed
GA
ed feature gateTopologyManager
. (#121252, @tukwila) - Removed
GA
ed feature gatesOpenAPIV3
. (#121255, @tukwila) - Removed
GA
ed feature gatesSeccompDefault
. (#121246, @tukwila) - Removed ephemeral container legacy server support for the server versions prior to
1.22
. (#119537, @ardaguclu) - Removed the
CronJobTimeZone
feature gate (the feature is stable and always enabled) - Removed the
JobMutableNodeSchedulingDirectives
feature gate (the feature is stable and always enabled) - Removed the
LegacyServiceAccountTokenNoAutoGeneration
feature gate (the feature is stable and always enabled) (#120192, @SataQiu) [SIG Apps, Auth and Scheduling] - Removed the
DownwardAPIHugePages
feature gate (the feature is stable and always enabled) (#120249, @pacoxu) [SIG Apps and Node] - Removed the
GRPCContainerProbe
feature gate (the feature is stable and always enabled). (#120248, @pacoxu) - Renamed
apiserver_request_body_sizes
metric toapiserver_request_body_size_bytes
. (#120503, @dgrisonnet) - Set the resolution for the
job_controller_job_sync_duration_seconds
metric from4ms
to1min
. (#120577, @alculquicondor) - The
horizontalpodautoscaling
andclusterrole-aggregation
controllers now assume theautoscaling/v1
andrbac.authorization.k8s.io/v1
APIs are available. If you disable those APIs and do not want to run those controllers, exclude them by passing--controllers=-horizontalpodautoscaling
or--controllers=-clusterrole-aggregation
tokube-controller-manager
. (#117977, @liggitt) [SIG API Machinery and Cloud Provider] - The metrics controlled by the
ComponentSLIs
feature-gate and served at/metrics/slis
are now GA and unconditionally enabled. The feature-gate will be removed inv1.31
. (#120574, @logicalhan) - Updated CNI plugins to
v1.3.0
. (#119969, @saschagrunert) - Updated
cri-tools
tov1.28.0
. (#119933, @saschagrunert) - Updated
distroless-iptables
to useregistry.k8s.io/build-image/distroless-iptables:v0.3.1
. (#120352, @saschagrunert) - Updated runc to
1.1.10
. (#121739, @ty-dc) - Upgraded
coredns
tov1.11.1
. (#120116, @tukwila) EnqueueExtensions
from plugins other thanPreEnqueue
,PreFilter
,Filter
,Reserve
andPermit
are now ignored.
It reduces the number of kinds of cluster events the scheduler needs to subscribe/handle. (#121571, @sanposhiho)GetPodQOS(pod *core.Pod)
function now returns the stored value fromPodStatus.QOSClass
, if set. To compute/evaluate the value ofQOSClass
from scratch,ComputePodQOS(pod*core.Pod)
must be used. (#119665, @vinaykul)RetroactiveDefaultStorageClass
feature gate that graduated to GA inv1.28
and was unconditionally enabled has been removed inv1.29
. (#120861, @RomanBednar)Statefulset
now waits for new replicas in tests when removing.start.ordinal
. (#119761, @soltysh)ValidatingAdmissionPolicy
andValidatingAdmissionPolicyBinding
objects are
persisted inetcd
using thev1beta1
version. Either remove alpha objects, or disable the
alphaValidatingAdmissionPolicy
feature in av1.27
server before upgrading to a
v1.28
server with the beta feature and API enabled. (#120018, @liggitt)client-go
:k8s.io/client-go/tools
events and record packages now have new APIs for specifying a context and logger. (#120729, @pohly)kube-controller-manager
help now includes controllers behind a feature gate in--controllers
flag. (#120371, @atiratree)kubeadm
: removedsystem:masters
organization fromapiserver-etcd-client
certificate. (#120521, @SataQiu)kubeadm
: removed leftover disclaimer that could be seen in thekubeadm init phase certs
command help screen, since the "certs" phase of "init" is no longer alpha. (#121172, @SataQiu)kubeadm
: updated warning message when swap space is detected. When swap is
active on Linux,kubeadm
explains that swap is supported for cgroup v2 only and
is beta but disabled by default. (#120198, @pacoxu)kubectl
will not support the/swagger-2.0.0.pb-v1
endpoint that has been long deprecated. (#119410, @Jefftree)scheduler
: handling of unschedulable pods because aResourceClass
is missing
is a bit more efficient and no longer relies on periodic retries. (#120213, @pohly)
Dependencies
Added
- cloud.google.com/go/dataproc/v2: v2.0.1
- github.com/danwinship/knftables: v0.0.13
- github.com/distribution/reference: v0.5.0
- github.com/google/s2a-go: v0.1.7
- google.golang.org/genproto/googleapis/bytestream: e85fd2c
Changed
- cloud.google.com/go/accessapproval: v1.6.0 → v1.7.1
- cloud.google.com/go/accesscontextmanager: v1.7.0 → v1.8.1
- cloud.google.com/go/aiplatform: v1.37.0 → v1.48.0
- cloud.google.com/go/analytics: v0.19.0 → v0.21.3
- cloud.google.com/go/apigateway: v1.5.0 → v1.6.1
- cloud.google.com/go/apigeeconnect: v1.5.0 → v1.6.1
- cloud.google.com/go/apigeeregistry: v0.6.0 → v0.7.1
- cloud.google.com/go/appengine: v1.7.1 → v1.8.1
- cloud.google.com/go/area120: v0.7.1 → v0.8.1
- cloud.google.com/go/artifactregistry: v1.13.0 → v1.14.1
- cloud.google.com/go/asset: v1.13.0 → v1.14.1
- cloud.google.com/go/assuredworkloads: v1.10.0 → v1.11.1
- cloud.google.com/go/automl: v1.12.0 → v1.13.1
- cloud.google.com/go/baremetalsolution: v0.5.0 → v1.1.1
- cloud.google.com/go/batch: v0.7.0 → v1.3.1
- cloud.google.com/go/beyondcorp: v0.5.0 → v1.0.0
- cloud.google.com/go/bigquery: v1.50.0 → v1.53.0
- cloud.google.com/go/billing: v1.13.0 → v1.16.0
- cloud.google.com/go/binaryauthorization: v1.5.0 → v1.6.1
- cloud.google.com/go/certificatemanager: v1.6.0 → v1.7.1
- cloud.google.com/go/channel: v1.12.0 → v1.16.0
- cloud.google.com/go/cloudbuild: v1.9.0 → v1.13.0
- cloud.google.com/go/clouddms: v1.5.0 → v1.6.1
- cloud.google.com/go/cloudtasks: v1.10.0 → v1.12.1
- cloud.google.com/go/compute: v1.19.0 → v1.23.0
- cloud.google.com/go/contactcenterinsights: v1.6.0 → v1.10.0
- cloud.google.com/go/container: v1.15.0 → v1.24.0
- cloud.google.com/go/containeranalysis: v0.9.0 → v0.10.1
- cloud.google.com/go/datacatalog: v1.13.0 → v1.16.0
- cloud.google.com/go/dataflow: v0.8.0 → v0.9.1
- cloud.google.com/go/dataform: v0.7.0 → v0.8.1
- cloud.google.com/go/datafusion: v1.6.0 → v1.7.1
- cloud.google.com/go/datalabeling: v0.7.0 → v0.8.1
- cloud.google.com/go/dataplex: v1.6.0 → v1.9.0
- cloud.google.com/go/dataqna: v0.7.0 → v0.8.1
- cloud.google.com/go/datastore: v1.11.0 → v1.13.0
- cloud.google.com/go/datastream: v1.7.0 → v1.10.0
- cloud.google.com/go/deploy: v1.8.0 → v1.13.0
- cloud.google.com/go/dialogflow: v1.32.0 → v1.40.0
- cloud.google.com/go/dlp: v1.9.0 → v1.10.1
- cloud.google.com/go/documentai: v1.18.0 → v1.22.0
- cloud.google.com/go/domains: v0.8.0 → v0.9.1
- cloud.google.com/go/edgecontainer: v1.0.0 → v1.1.1
- cloud.google.com/go/essentialcontacts: v1.5.0 → v1.6.2
- cloud.google.com/go/eventarc: v1.11.0 → v1.13.0
- cloud.google.com/go/filestore: v1.6.0 → v1.7.1
- cloud.google.com/go/firestore: v1.9.0 → v1.11.0
- cloud.google.com/go/functions: v1.13.0 → v1.15.1
- cloud.google.com/go/gkebackup: v0.4.0 → v1.3.0
- cloud.google.com/go/gkeconnect: v0.7.0 → v0.8.1
- cloud.google.com/go/gkehub: v0.12.0 → v0.14.1
- cloud.google.com/go/gkemulticloud: v0.5.0 → v1.0.0
- cloud.google.com/go/gsuiteaddons: v1.5.0 → v1.6.1
- cloud.google.com/go/iam: v0.13.0 → v1.1.1
- cloud.google.com/go/iap: v1.7.1 → v1.8.1
- cloud.google.com/go/ids: v1.3.0 → v1.4.1
- cloud.google.com/go/iot: v1.6.0 → v1.7.1
- cloud.google.com/go/kms: v1.10.1 → v1.15.0
- cloud.google.com/go/language: v1.9.0 → v1.10.1
- cloud.google.com/go/lifesciences: v0.8.0 → v0.9.1
- cloud.google.com/go/longrunning: v0.4.1 → v0.5.1
- cloud.google.com/go/managedidentities: v1.5.0 → v1.6.1
- cloud.google.com/go/maps: v0.7.0 → v1.4.0
- cloud.google.com/go/mediatranslation: v0.7.0 → v0.8.1
- cloud.google.com/go/memcache: v1.9.0 → v1.10.1
- cloud.google.com/go/metastore: v1.10.0 → v1.12.0
- cloud.google.com/go/monitoring: v1.13.0 → v1.15.1
- cloud.google.com/go/networkconnectivity: v1.11.0 → v1.12.1
- cloud.google.com/go/networkmanagement: v1.6.0 → v1.8.0
- cloud.google.com/go/networksecurity: v0.8.0 → v0.9.1
- cloud.google.com/go/notebooks: v1.8.0 → v1.9.1
- cloud.google.com/go/optimization: v1.3.1 → v1.4.1
- cloud.google.com/go/orchestration: v1.6.0 → v1.8.1
- cloud.google.com/go/orgpolicy: v1.10.0 → v1.11.1
- cloud.google.com/go/osconfig: v1.11.0 → v1.12.1
- cloud.google.com/go/oslogin: v1.9.0 → v1.10.1
- cloud.google.com/go/phishingprotection: v0.7.0 → v0.8.1
- cloud.google.com/go/policytroubleshooter: v1.6.0 → v1.8.0
- cloud.google.com/go/privatecatalog: v0.8.0 → v0.9.1
- cloud.google.com/go/pubsub: v1.30.0 → v1.33.0
- cloud.google.com/go/pubsublite: v1.7.0 → v1.8.1
- cloud.google.com/go/recaptchaenterprise/v2: v2.7.0 → v2.7.2
- cloud.google.com/go/recommendationengine: v0.7.0 → v0.8.1
- cloud.google.com/go/recommender: v1.9.0 → v1.10.1
- cloud.google.com/go/redis: v1.11.0 → v1.13.1
- cloud.google.com/go/resourcemanager: v1.7.0 → v1.9.1
- cloud.google.com/go/resourcesettings: v1.5.0 → v1.6.1
- cloud.google.com/go/retail: v1.12.0 → v1.14.1
- cloud.google.com/go/run: v0.9.0 → v1.2.0
- cloud.google.com/go/scheduler: v1.9.0 → v1.10.1
- cloud.google.com/go/secretmanager: v1.10.0 → v1.11.1
- cloud.google.com/go/security: v1.13.0 → v1.15.1
- cloud.google.com/go/securitycenter: v1.19.0 → v1.23.0
- cloud.google.com/go/servicedirectory: v1.9.0 → v1.11.0
- cloud.google.com/go/shell: v1.6.0 → v1.7.1
- cloud.google.com/go/spanner: v1.45.0 → v1.47.0
- cloud.google.com/go/speech: v1.15.0 → v1.19.0
- cloud.google.com/go/storagetransfer: v1.8.0 → v1.10.0
- cloud.google.com/go/talent: v1.5.0 → v1.6.2
- cloud.google.com/go/texttospeech: v1.6.0 → v1.7.1
- cloud.google.com/go/tpu: v1.5.0 → v1.6.1
- cloud.google.com/go/trace: v1.9.0 → v1.10.1
- cloud.google.com/go/translate: v1.7.0 → v1.8.2
- cloud.google.com/go/video: v1.15.0 → v1.19.0
- cloud.google.com/go/videointelligence: v1.10.0 → v1.11.1
- cloud.google.com/go/vision/v2: v2.7.0 → v2.7.2
- cloud.google.com/go/vmmigration: v1.6.0 → v1.7.1
- cloud.google.com/go/vmwareengine: v0.3.0 → v1.0.0
- cloud.google.com/go/vpcaccess: v1.6.0 → v1.7.1
- cloud.google.com/go/webrisk: v1.8.0 → v1.9.1
- cloud.google.com/go/websecurityscanner: v1.5.0 → v1.6.1
- cloud.google.com/go/workflows: v1.10.0 → v1.11.1
- cloud.google.com/go: v0.110.0 → v0.110.6
- github.com/alecthomas/template: fb15b89 → a0175ee
- github.com/cncf/xds/go: 06c439d → e9ce688
- github.com/coredns/corefile-migration: v1.0.20 → v1.0.21
- github.com/cyphar/filepath-securejoin: v0.2.3 → v0.2.4
- github.com/docker/docker: v20.10.21+incompatible → v20.10.24+incompatible
- github.com/emicklei/go-restful/v3: v3.9.0 → v3.11.0
- github.com/envoyproxy/go-control-plane: v0.10.3 → v0.11.1
- github.com/envoyproxy/protoc-gen-validate: v0.9.1 → v1.0.2
- github.com/evanphx/json-patch: v5.6.0+incompatible → v4.12.0+incompatible
- github.com/fsnotify/fsnotify: v1.6.0 → v1.7.0
- github.com/go-logr/logr: v1.2.4 → v1.3.0
- github.com/godbus/dbus/v5: v5.0.6 → v5.1.0
- github.com/golang/glog: v1.0.0 → v1.1.0
- github.com/google/cadvisor: v0.47.3 → v0.48.1
- github.com/google/cel-go: v0.16.0 → v0.17.7
- github.com/google/go-cmp: v0.5.9 → v0.6.0
- github.com/googleapis/gax-go/v2: v2.7.1 → v2.11.0
- github.com/gorilla/websocket: v1.4.2 → v1.5.0
- github.com/grpc-ecosystem/grpc-gateway/v2: v2.7.0 → v2.16.0
- github.com/ishidawataru/sctp: 7c296d4 → 7ff4192
- github.com/konsorten/go-windows-terminal-sequences: v1.0.3 → v1.0.1
- github.com/mrunalp/fileutils: v0.5.0 → v0.5.1
- github.com/onsi/ginkgo/v2: v2.9.4 → v2.13.0
- github.com/onsi/gomega: v1.27.6 → v1.29.0
- github.com/opencontainers/runc: v1.1.7 → v1.1.10
- github.com/opencontainers/selinux: v1.10.0 → v1.11.0
- github.com/spf13/afero: v1.2.2 → v1.1.2
- github.com/stretchr/testify: v1.8.2 → v1.8.4
- github.com/vmware/govmomi: v0.30.0 → v0.30.6
- go.etcd.io/bbolt: v1.3.7 → v1.3.8
- go.etcd.io/etcd/api/v3: v3.5.9 → v3.5.10
- go.etcd.io/etcd/client/pkg/v3: v3.5.9 → v3.5.10
- go.etcd.io/etcd/client/v2: v2.305.9 → v2.305.10
- go.etcd.io/etcd/client/v3: v3.5.9 → v3.5.10
- go.etcd.io/etcd/pkg/v3: v3.5.9 → v3.5.10
- go.etcd.io/etcd/raft/v3: v3.5.9 → v3.5.10
- go.etcd.io/etcd/server/v3: v3.5.9 → v3.5.10
- go.opentelemetry.io/contrib/instrumentation/github.com/emicklei/go-restful/otelrestful: v0.35.0 → v0.42.0
- go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc: v0.35.0 → v0.42.0
- go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp: v0.35.1 → v0.44.0
- go.opentelemetry.io/contrib/propagators/b3: v1.10.0 → v1.17.0
- go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc: v1.10.0 → v1.19.0
- go.opentelemetry.io/otel/exporters/otlp/otlptrace: v1.10.0 → v1.19.0
- go.opentelemetry.io/otel/metric: v0.31.0 → v1.19.0
- go.opentelemetry.io/otel/sdk: v1.10.0 → v1.19.0
- go.opentelemetry.io/otel/trace: v1.10.0 → v1.19.0
- go.opentelemetry.io/otel: v1.10.0 → v1.19.0
- go.opentelemetry.io/proto/otlp: v0.19.0 → v1.0.0
- golang.org/x/crypto: v0.11.0 → v0.14.0
- golang.org/x/mod: v0.10.0 → v0.12.0
- golang.org/x/net: v0.13.0 → v0.17.0
- golang.org/x/oauth2: v0.8.0 → v0.10.0
- golang.org/x/sync: v0.2.0 → v0.3.0
- golang.org/x/sys: v0.10.0 → v0.13.0
- golang.org/x/term: v0.10.0 → v0.13.0
- golang.org/x/text: v0.11.0 → v0.13.0
- golang.org/x/tools: v0.8.0 → v0.12.0
- google.golang.org/api: v0.114.0 → v0.126.0
- google.golang.org/genproto/googleapis/api: dd9d682 → 23370e0
- google.golang.org/genproto/googleapis/rpc: 28d5490 → b8732ec
- google.golang.org/genproto: 0005af6 → f966b18
- google.golang.org/grpc: v1.54.0 → v1.58.3
- google.golang.org/protobuf: v1.30.0 → v1.31.0
- k8s.io/gengo: c0856e2 → 9cce18d
- k8s.io/klog/v2: v2.100.1 → v2.110.1
- k8s.io/kube-openapi: 2695361 → 2dd684a
- k8s.io/utils: d93618c → 3b25d92
- sigs.k8s.io/apiserver-network-proxy/konnectivity-client: v0.1.2 → v0.28.0
- sigs.k8s.io/structured-merge-diff/v4: v4.2.3 → v4.4.1
Removed
- cloud.google.com/go/dataproc: v1.12.0
- cloud.google.com/go/gaming: v1.9.0
- github.com/blang/semver: v3.5.1+incompatible
- github.com/jmespath/go-jmespath/internal/testify: v1.5.1
- go.opentelemetry.io/otel/exporters/otlp/internal/retry: v1.10.0
v1.29.0-rc.2
Downloads for v1.29.0-rc.2
Source Code
filename | sha512 hash
-------- | -----------
kubernetes.tar.gz | daaabe57e2da16a076380072bb0e4a178400f045bb82d44d338efa90641bb4e35b590764d9ab4f365219634149588526da57d1aaabdb1ed805ee0ccd9aed63b6
kubernetes-src.tar.gz | c4a3ea15db8a7d0696f2ef4a2f3d1e65b89a931074043957fa59be2bb0fac04b9967e8eff1037b4c649fcdd34a3ad2b717d129b2ce2f45691675bbef95710833
Client Binaries
filename | sha512 hash
-------- | -----------
kubernetes-client-darwin-amd64.tar.gz | 3b22ba3aa4f778f0086e739999f81f8ca040f5a5b9b88a8e71c9cd94dc728ee9090f0c388a0020f458a6c13716c614b37e1be11b7e5402cc00380595ce3be21f
kubernetes-client-darwin-arm64.tar.gz | ed2d2f866f28b5f157b1ca39ae2cb832cc934c2383f8cb90b5e79781a4835579dd99ebe2f34245c4a764a49ebcb343499fded8f180ac1a084acbfa2bfc38ef37
kubernetes-client-linux-386.tar.gz | cdf10ab26e223742a882d19a36a932a3230a061e026514fd3cbe3d763f7de50372903e04ea9b85c2554980310d8fa8b8c140bb623e491a6c646af18499e14354
kubernetes-client-linux-amd64.tar.gz | 7aa3da03d393b9da31ec3fd0c5c9694ab3a3e1bc7340108375238a5b132da8955db318559f2d6f7968f141802373fa3462a980932e5bb9d59b553816fc1bc2d4
kubernetes-client-linux-arm.tar.gz | c24722f8f20f86a842f04e047041c631975ee4a8800da9cee1ff5415055842eb381b7c918b7da2c7421369d718b13e182008e08c2f5f2a5c5a1782c481851ddf
kubernetes-client-linux-arm64.tar.gz | 1d4df5d1bb6fd5fdb8b9dc3b0bb7c8b7f3c155dab019222d9611706d5140206ac9a7eda6673a2a912e1738f02f8707184e4e711d4f82f5680a8098ce59ad9f74
kubernetes-client-linux-ppc64le.tar.gz | c7187e7834f690958ea2b3f7aeac987f6efcd76b859487b391972a25daf347c87fc60e9e9b9a440a2c4df529c8f467b94790e6f8992cdbbac87f73d1373f03c0
kubernetes-client-linux-s390x.tar.gz | e758559d18ef1510d50fa57177e2023ffc4bc19d3a4e302e984bef35701e9a47894e7c7d80589742bd24a337d75b35e83faf1c097202401deee01e0f5fe31829
kubernetes-client-windows-386.tar.gz | 0ef788e96ea786b0d62de7dcc1315800a4106b367381e02731ca384bc89397aa7a1de5b678a0532df590c1f8d448206a236623ed729bc5c30b0e63317aec6a6c
kubernetes-client-windows-amd64.tar.gz | 719c3c1f9b7beb199bd9d0a0b5c85d99b76bd462397a3d4b37fe8b5970e30e69e26f30815f97bb9a4049b28c7aaf0b384d1378a14e8273774d65f3549cbd3083
kubernetes-client-windows-arm64.tar.gz | 24916863e604c14939ccd1574f754306215d603ff2efdd4dd00fa667923932aff88f8046fc8a4b7dea47f97ea5bfc53d4193fb293fa48fdefb4cd65c301e32ab
Server Binaries
filename | sha512 hash
-------- | -----------
kubernetes-server-linux-amd64.tar.gz | 88e7a746f69b3070908fc32213d6046a10c5df139be566f42b135fb344f9681f7d527e5d8b65214f3a5abc4564962d4d1f8391d6ce62ca3620e538bb0e836055
kubernetes-server-linux-arm64.tar.gz | 69b9bf487dabf7ae466cb9cf0e569d04908c359dea2772ed2a0daa0c5be2d1389d09cac3eebc7c350af0fd0e0f30e2cfc86c4ede423a0cd8d00f62909e13fac4
kubernetes-server-linux-ppc64le.tar.gz | 5925a756d4ab2be13141391c7c18888f2b7337aa3db05ac0cb0ed25ad66a6d290b38731e389b829571bb5fcb95b6b9f5b4d054058818adad66d1f39e45fa9356
kubernetes-server-linux-s390x.tar.gz | 82e4aa613bffc8658e8a10c53269a7977219fd38b985fb8a4a4df78a8fa876521c1925ab927a74737e3a57c415963c13717720d106b6f086294c717bdc5f02c9
Node Binaries
filename | sha512 hash
-------- | -----------
kubernetes-node-linux-amd64.tar.gz | fd7be4f743a641f22c04d32749825d509b6c9ff095207997db40dc39cb1647bd03f47b71287339a02f303b2903ffe6bd0910bd051f20f44c03e7d853c2e794b5
kubernetes-node-linux-arm64.tar.gz | 12454bf94d86282cbf04db188f504aed6d8c6eeda8f32b0001152aa7ed52c65c1ec0db90c99077b30987d40a4ccd6c707dcda1ce6be84c746f4c972af0b277f4
kubernetes-node-linux-ppc64le.tar.gz | 1eb0d47ca00231df825daf2c39b203c438a7c1b1d9ec1e7bcefd63c16e89771316d9a2cf66c89ad6653e70fed3ccc38a650e83be106f2738c5ae97bbc055de7f
kubernetes-node-linux-s390x.tar.gz | ff7f4da7e71859ebea3a70410575f7ffa52c476f4f6d76958366ca5e4ec7b315fdd4b5b8cbd48faaa800ffcfb64f9af85b6605582480242ae336c3fea84f8733
kubernetes-node-windows-amd64.tar.gz | 8fc83a3735d163866e370df3c6bccbe15d2ce478b95bc1fe71b199952db68c95f31d8967514b0930c2ad7bc2734c2a4ab5d8cdc738611365215f7d88a19bc2d6
Container Images
All container images are available as manifest lists and support the described
architectures. It is also possible to pull a specific architecture directly by
adding the "-$ARCH" suffix to the container image name.
name | architectures
---- | -------------
registry.k8s.io/conformance:v1.29.0-rc.2 | amd64, arm64, ppc64le, s390x
registry.k8s.io/kube-apiserver:v1.29.0-rc.2 | amd64, arm64, ppc64le, s390x
registry.k8s.io/kube-controller-manager:v1.29.0-rc.2 | amd64, arm64, ppc64le, s390x
registry.k8s.io/kube-proxy:v1.29.0-rc.2 | amd64, arm64, ppc64le, s390x
registry.k8s.io/kube-scheduler:v1.29.0-rc.2 | amd64, arm64, ppc64le, s390x
registry.k8s.io/kubectl:v1.29.0-rc.2 | amd64, arm64, ppc64le, s390x
Changelog since v1.29.0-rc.1
Changes by Kind
Feature
- Bump distroless-iptables to v0.4.3 (#122206, @xmudrii) [SIG Release and Testing]
- Kubernetes is now built with Go 1.21.5 (#122201, @xmudrii) [SIG Release and Testing]
Dependencies
Added
Nothing has changed.
Changed
Nothing has changed.
Removed
Nothing has changed.
v1.29.0-rc.1
Downloads for v1.29.0-rc.1
Source Code
filename | sha512 hash
-------- | -----------
kubernetes.tar.gz | e44677de7af6634c31b86672dc6755d97ae145fd0497229c08b156d11dcdbc922f15c715fd878b585d21a6c7dd10fde0b43135f0b6f7e77a9f957f2280a32018
kubernetes-src.tar.gz | 63e197478e315a64dae6282c1e4ce2b672f0a3941bea9920094b703d44a09aaed74228a7c29fbec4051a4cb832f6e791d54f5e76e006aada1216a502c6d2e744
Client Binaries
filename | sha512 hash
-------- | -----------
kubernetes-client-darwin-amd64.tar.gz | 167d931f6b9540b9fbd8501e3dd9d2ea032ead96f57cf4a929020fff3c4efb0456e3bc5eee2a8436670589ef18b48018dabf57081ff13393542007e9d0c8cb72
kubernetes-client-darwin-arm64.tar.gz | 6623d8b08b69beea8832a1130c50624f248464a01fec4fce720700ccebdd3ed440af664f5beb49a294557db3c4ff7a8fecfd3cf48f9dabcc48b9bda2d791c08e
kubernetes-client-linux-386.tar.gz | 11f6c0d6f0954938c4217436536a67713c403b1f3c2b988d26944374fab6f70c385cb9356e6aa51b480fdcd07539de4667be2e72fa8128ba792a607bb388254f
kubernetes-client-linux-amd64.tar.gz | ee7605531629a2e320f299ef49bfb87566b73245f16dee79a40b824d637bd97ca8bd7f81a177320c2e4bbb82acd7f5d840359dd79c3062c2136e0b5f04eeb90e
kubernetes-client-linux-arm.tar.gz | baa1af4d932c3d36ff084f2fc4c7676e76a2f0e4c4c746495f932c56a583b4390376b0b631b13f6dc3b03bd874d84c20f82e71d2483940b37ea439bc7d21dadf
kubernetes-client-linux-arm64.tar.gz | 26a20dfbeca7abb73a73f1cfb5337b4af71bb3d2810d053f9d94e4a3709d282e515a542f02502a7e86adf3b32ac52e7b434d1604fa9664729682d083a55b314d
kubernetes-client-linux-ppc64le.tar.gz | eca5ca3028b64ea44138b08831e998ac85bd054785099366e295b88b8b568c455a54f4fd110b568208169b9a3aef918c7f6caf8e05f9e73f85c26f973a589e2d
kubernetes-client-linux-s390x.tar.gz | 5bcde8b36b8dc1d3ed83337322fc260311238e9f067301838c5002bb0dc63153f82c2602d8c9a28c1ae5dd85b0eecc4d3e8c31dcf75f4653c6b9bebd3a564321
kubernetes-client-windows-386.tar.gz | d183c3183bfac0878377eaa8adf00e6ecdb3f252ed47180b8f9231d757b44c30931620b04da52c3470a1a278fa5a76e99f0b7c587b696f517caec5ff16103480
kubernetes-client-windows-amd64.tar.gz | 7a51ed89ad5f850bfc94e4175294d944eae9628c281fea2c18939417f84c438b82246d262e645bea9fd257deb60b51c05b1c1ebd321b35aafeb87d1b4f83ebe6
kubernetes-client-windows-arm64.tar.gz | 2efc1fd75461ed5e0bceba78681804567e731a545a801b28e8291d2f3cc8e2c8c22d3e418888a666dc1e40754681acbcbfc64fc81772088b8524fde7c55e7e3a
Server Binaries
filename | sha512 hash
-------- | -----------
kubernetes-server-linux-amd64.tar.gz | 28eeda8ab821891ca445b4a884ae70028146f6d4264e364a7ca88c819ea80bd95653c7f538af6c5f660921f140026f3d61c469afb0109c2f5111245592439fd7
kubernetes-server-linux-arm64.tar.gz | d786685de63f060910181f0de66511fe8f8f0fa66f00ee0c76957246a62e3de3208ec009908c6700ff83dc7c5e5c24f3c1c3118d06b07568fcb004190d1e5fe6
kubernetes-server-linux-ppc64le.tar.gz | b9d14fdcf282b4f9e523d81f71cd82d64bfb9bf9ba4affb7d6dbcfa8191f9b0d238f7091c65c1f1d516c6bcaa13f68affdd3bfda1ec759f35d505284a87494e1
kubernetes-server-linux-s390x.tar.gz | f84c315e3d3ab6e3124106783e43b18d407d5c4ef09910641ae51c034b550fba0581515181ae4d355eea5b75eb0688460a891141c47d13be902e06156908d26d
Node Binaries
filename | sha512 hash
-------- | -----------
kubernetes-node-linux-amd64.tar.gz | b2a974c505878c757bc643c64cf24208ed92bfe7943f6e9e50d215f027e4fc2f4a578d3975721f53ff6a06e6560dec362887a236052213784222695c916a59fb
kubernetes-node-linux-arm64.tar.gz | 7f7b6a3bcdef051ae0126a811d02b327f6f45b13050934dd4d90219a23efd5aeabb945e040a9aa57a24a4151222b558a4e2a996bad5e5f7ef508c26e6cca85ee
kubernetes-node-linux-ppc64le.tar.gz | 6245a8645a6b52c2bd40e1104d4fbba015bf583f50478b2686ecf0c3c8fdc05c6729da76dc6301ca0f2f0d63f1a5df84a0871fb4793fc3aa9e8e2b6cdae37d34
kubernetes-node-linux-s390x.tar.gz | fe530554696b84db43372ca48e95a01c46cf3a09dd51fb569b1792a341827a428bfbb04c3bab59d6aea095687eafe144534a0d98df8852d469e8bdc69a8d2d1a
kubernetes-node-windows-amd64.tar.gz | 0a80378b4037f8d325fdbdc065ce5fd841b0b66242d64ec5c6b8ad6af0e430fc020d4fb0d624909b0725761d2c249a8f75c91eae22af03cefe4a4338b57d3d29
Container Images
All container images are available as manifest lists and support the described
architectures. It is also possible to pull a specific architecture directly by
adding the "-$ARCH" suffix to the container image name.
name | architectures
---- | -------------
registry.k8s.io/conformance:v1.29.0-rc.1 | amd64, arm64, ppc64le, s390x
registry.k8s.io/kube-apiserver:v1.29.0-rc.1 | amd64, arm64, ppc64le, s390x
registry.k8s.io/kube-controller-manager:v1.29.0-rc.1 | amd64, arm64, ppc64le, s390x
registry.k8s.io/kube-proxy:v1.29.0-rc.1 | amd64, arm64, ppc64le, s390x
registry.k8s.io/kube-scheduler:v1.29.0-rc.1 | amd64, arm64, ppc64le, s390x
registry.k8s.io/kubectl:v1.29.0-rc.1 | amd64, arm64, ppc64le, s390x
Changelog since v1.29.0-rc.0
Dependencies
Added
Nothing has changed.
Changed
Nothing has changed.
Removed
Nothing has changed.
v1.29.0-rc.0
Downloads for v1.29.0-rc.0
Source Code
filename | sha512 hash
-------- | -----------
kubernetes.tar.gz | 6a5b027a35b96d1cf8495efce0f9f518499b94e63e1d11058876d1b364d0bba42ccedac4612082771eb38cd54be0d8868a808de05c7e9077b8644f15a5c6f413
kubernetes-src.tar.gz | d92897e5e28a14f0fbd3f03e9016e9c86f30bf097c4e709e6dba74b1a9897ce016e3c3a44aed9d5f851af1f5d5bd0ea2240efe8d8d12d7893b7f9cff66caff55
Client Binaries
filename | sha512 hash
-------- | -----------
kubernetes-client-darwin-amd64.tar.gz | a4e8dd4e65158024a46843701ed24082eefde5d407c6d6a191b7b7f690413ea65c5422ba578e2813cd6624ac7327174554d879dbbbb324b56fbfe99892eb8d80
kubernetes-client-darwin-arm64.tar.gz | ef14378eaa3a35a34ab5e9b06c9856ff46165bbee2a4efc1b8512de47e8f584449d94155665978eca6264e23f131e31d072f9333117c11a3e92aadeea367b8e5
kubernetes-client-linux-386.tar.gz | 686a8b69525e8e1494cdc890e8023ba60f86e41ceb28cb5df7e33f152ecc3ac8c62b0b1d24fa6c8198278a9d585bfd8962d058daf7f27dfa658580598b45cafe
kubernetes-client-linux-amd64.tar.gz | 7ebe8d866f8fd1dccd6761be0ad5096cb861e5fb20bdea0ac65a3d63230d9a7d47df16a6933fcf4069cf819ab90d12eaf87ec53873eacd88c3feab009e85e430
kubernetes-client-linux-arm.tar.gz | f8a336b48c27819f979336fff3ffa7eeb5512330f3eafe7a3b85ea65a4d94b213ed20d4d35d6fe3f92cb557037a051023eb47fd6e6dcaf3e0a6fe88a5c6cd632
kubernetes-client-linux-arm64.tar.gz | a72602ac48b13c6a97883c34170fd64095539d4f9a3900367ff628a195aa931c27d7c9582f864c669332bbc58b4883d5e41bc65d5ac83337bdf7066e538deceb
kubernetes-client-linux-ppc64le.tar.gz | 002b2e685758ad6fa2a18d7706a335249f55a786a4315d3f2cab8e34d38a01302af91063742729de664c7ab06bd656b388166f82010af36e43e931e8ddd93752
kubernetes-client-linux-s390x.tar.gz | 21da21e1f7ba24b6967b5e22abb62e1c1691cd7cc15eb5ecd9777fa51d788a7a132f31c04306d3a59e5cee96bb58b9d1838630de1bcbf168cabe8f4afb514501
kubernetes-client-windows-386.tar.gz | 21494c5fe65e6a9aaf2f7f11996219155ed85a4f54d048b64df05de1adbd925af40ee51d4119801333143364902b9805cefceafac8d407f62eef1e7f07b686ee
kubernetes-client-windows-amd64.tar.gz | eaaddeac2e0a69a618f606574044eec8b41f4c3d4f6cf0045e4456ad57d44c865d1f183b6e0929f6913e28febe67b178662dfce3e40395c6d97180985b4fb48d
kubernetes-client-windows-arm64.tar.gz | dad6a73bf2530c0c2f58b8e77956ec444b6795c9882b0f2b960998fbd9e22720fc6fff114af3b0ad10655e9e1d627f70bc6f67fdd388d0e995aeb9bd4bf9bea2
Server Binaries
filename | sha512 hash
-------- | -----------
kubernetes-server-linux-amd64.tar.gz | c64651213144ef4696fa11da0ec93c6fd7540798bfc28df8e69ee8bdb35dbc7114ee043cd38dc86c75a3dbff5e45ed4474be22ce74b8c4b3206030a10cd20f8d
kubernetes-server-linux-arm64.tar.gz | 0724ce02d551d72f39c7ac6b29c78dcaeae7878126b33cde7a949d0b9be0b35b3977f5494ab48f02a382bf83a70a8ad035f4962b0644a6fedc084068b525ddf9
kubernetes-server-linux-ppc64le.tar.gz | 7f527bb02e046308b2720a99d8f6ac13e1daee23b44e77603b75aa5569a9b4baf29a7b19f3076219ff94f296ce8316fdaadb9cabaf1c58173a7e3719e94f3917
kubernetes-server-linux-s390x.tar.gz | 0285e04f2834bdbb66b46193f54724e6f9264ff992b10dbaa3694abbab297f5e1f4e95ade14f7dcb41f856d9e3a292f1af16f3ed59e2b02961451973d4972f1d
Node Binaries
filename | sha512 hash
-------- | -----------
kubernetes-node-linux-amd64.tar.gz | b4c5e4a0e818eb9f88128e2a051591b4955a858e400489d04b75cdfb68eb3a7d004ced839c2916bf5ca885d7ae496fb68c0620b2c3352cb5435c435756b0a70a
kubernetes-node-linux-arm64.tar.gz | e525decd637860b9621ec7ec8c42913c419bb81577a1a359e752e7628507b6e9b1a82889ef0ba17ca975aa8630edba12a87d38b75ea3f9e213493873036b92c8
kubernetes-node-linux-ppc64le.tar.gz | 4282286b775a5bdaab753c911fa0f351476d89070a34569bedb104cb2c56a408d125d44c4895cc28fcb8cd5c12585f3cbecccfe045880b546766202113a703b1
kubernetes-node-linux-s390x.tar.gz | af88dac8622e10e336e5d79f9d4511de3eceed384da210dda83223c7b6582133acaa7d8f6b361cf213a4ca3ea51379bd828816c991ed7b4c62cf6fd9830f0c30
kubernetes-node-windows-amd64.tar.gz | 7b322df6a7e9e0b0b881f99d7ef76b3eda0f856345eb888efa62daf1f3638f88a630fc40626800075db90215c68a956fec7ce274381e06a173d494e4b03b4f49
Container Images
All container images are available as manifest lists and support the described
architectures. It is also possible to pull a specific architecture directly by
adding the "-$ARCH" suffix to the container image name.
name | architectures
---- | -------------
registry.k8s.io/conformance:v1.29.0-rc.0 | amd64, arm64, ppc64le, s390x
registry.k8s.io/kube-apiserver:v1.29.0-rc.0 | amd64, arm64, ppc64le, s390x
registry.k8s.io/kube-controller-manager:v1.29.0-rc.0 | amd64, arm64, ppc64le, s390x
registry.k8s.io/kube-proxy:v1.29.0-rc.0 | amd64, arm64, ppc64le, s390x
registry.k8s.io/kube-scheduler:v1.29.0-rc.0 | amd64, arm64, ppc64le, s390x
registry.k8s.io/kubectl:v1.29.0-rc.0 | amd64, arm64, ppc64le, s390x
Changelog since v1.29.0-alpha.3
Changes by Kind
API Change
- Added support for projecting certificates.k8s.io/v1alpha1 ClusterTrustBundle objects into pods. (#113374, @ahmedtd) [SIG API Machinery, Apps, Auth, Node, Storage and Testing]
- Adds
optionalOldSelf
tox-kubernetes-validations
to support ratcheting CRD schema constraints (#121034, @alexzielenski) [SIG API Machinery] - Fix API comment for the Job Ready field in status (#121765, @mimowo) [SIG API Machinery and Apps]
- Fix API comments for the FailIndex Job pod failure policy action. (#121764, @mimowo) [SIG API Machinery and Apps]
Feature
- A customizable OrderedScoreFuncs() function is introduced. Out-of-tree plugins that use scheduler's preemption interface can implement this function for custom preemption preferences, or return nil to keep current behavior. (#121867, @lianghao208) [SIG Scheduling]
- Bump distroless-iptables to 0.4.1 based on Go 1.21.3 (#121871, @cpanato) [SIG Testing]
- Fix overriding default KubeletConfig fields in drop-in configs if not set (#121193, @sohankunkerkar) [SIG Node and Testing]
- KEP-4191- add support for split image filesystem in kubelet (#120616, @kannon92) [SIG Node and Testing]
- Kubeadm: support updating certificate organization during 'kubeadm certs renew' (#121841, @SataQiu) [SIG Cluster Lifecycle]
- Kubernetes is now built with Go 1.21.4 (#121808, @cpanato) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node, Release, Storage and Testing]
Bug or Regression
- Fix: statle smb mount issue when smb file share is deleted and then unmount (#121851, @andyzhangx) [SIG Storage]
- KCCM: fix transient node addition + removal caused by #121090 while syncing load balancers on large clusters with a lot of churn (#121091, @alexanderConstantinescu) [SIG Cloud Provider, Network and Testing]
- Kubeadm: change the "system:masters" Group in the apiserver-kubelet-client.crt certificate Subject to be "kubeadm:cluster-admins" which is a less privileged Group. (#121837, @neolit123) [SIG Cluster Lifecycle]
- Scheduler: in 1.29 pre-releases, enabling contextual logging slowed down pod scheduling. (#121715, @pohly) [SIG Instrumentation and Scheduling]
Other (Cleanup or Flake)
Dependencies
Added
Nothing has changed.
Changed
- github.com/mrunalp/fileutils: v0.5.0 → v0.5.1
- github.com/opencontainers/runc: v1.1.9 → v1.1.10
Removed
Nothing has changed.
v1.29.0-alpha.3
Downloads for v1.29.0-alpha.3
Source Code
filename | sha512 hash
-------- | -----------
kubernetes.tar.gz | 998a680aee880601d65c14cf43a8ace13aacb3d693ac2f32c40ddc5c0a567fd4cc5627f397bf5612ed83d6b37ef568260f2700d46592bbc74174e155bf8f0606
kubernetes-src.tar.gz | ca46836dabd989a8dc6ee61032ab7f73747a5e2ef3bc11437e4036d95cbfbb9574f647b1672a098625729b62f1ff663726fcaf2dc3ea472e7b27d6b373d8afa9
Client Binaries
filename | sha512 hash
-------- | -----------
kubernetes-client-darwin-amd64.tar.gz | b82008b54b2a90e3640e786782cc20cf3a7d6a5011974f6710d418770541b53edb7d9d4ccd9489d4d81fcf7df7db38a3766db19898c86381b6fcfd7b261bc06a
kubernetes-client-darwin-arm64.tar.gz | b389eece6ea7ba07fdff76a6acdf36e77ed81e474277b62ef40b91ccc0d00c37f6f7c1194cacb14df844b1ea4dc66895b1c73bfedba570c71b72c6ab9a697861
kubernetes-client-linux-386.tar.gz | 5fa044082a1d2fb9d0b428fd2ba913196b4891f4c0571a7061ef1b6fee19ff820ff2b67506edad27fe0ddc735630d7398f66160836620188a527a8f3dbeb6b09
kubernetes-client-linux-amd64.tar.gz | 782d262f696e9b706de195870e5589fe3a0c4c11698574709668d4f60fcfe3cfb0137e86fb2d43a27a297bf88c29552216d30ac72255b4a757525f0b7e2385a1
kubernetes-client-linux-arm.tar.gz | cd9038cd3fa938aac9a0b462f7c6822d031f4e05e2529df378b4069f2d69d362236e5fc6e464d20cc42549f84f283f302b6cb33eb4a128ab91dcaa1cf04552e8
kubernetes-client-linux-arm64.tar.gz | d0fdf61def1be6c3b9e5259c13e8dbd764af44ed3dcdeb83c6a7d6cfe87b2293cbd88ceaad0aac87b448fe4766635b6b9eca40bc1a302f717d1bc0e26dac60ed
kubernetes-client-linux-ppc64le.tar.gz | c8b148404eecdff20939f0bec92024be58cb9629802c3768085834998e97b82e87f37443a867dfbb73e9922aada038d308ef02ec52a078aefb1f76360220c77b
kubernetes-client-linux-s390x.tar.gz | fb7070ef9d610fae614eadf9ec7fcfe68958143010b709586f0339309336e87f06d03ff8df5108b340ea063082e7b1d393b519ee6a4b4ed302427fd66e896295
kubernetes-client-windows-386.tar.gz | c0021a7668504a0a2be408cb2d1754bd20fc9afeeeb31dfb11f11787eb7047540c544888058300a83662dab845c726c7001562ee712b4c1d485ff0c3a88827f9
kubernetes-client-windows-amd64.tar.gz | 5678ab6523345ec38ba49a81c945223112228e75b82d0b459f0c9d6c37d3a0c93af4b82f05226e2d1110c1315ae5ed7ed3ad0bb085afad7f376b9715fe8fee75
kubernetes-client-windows-arm64.tar.gz | 2d7ac1add995683b29396fbbc06b15bf81ca62338ba0ed6d4738283752560fa167d6b9ffc44aeb4ed9b1bae6bc2ed8fe1d6346436c34a2acd3a5467ac68041bb
Server Binaries
filename | sha512 hash
-------- | -----------
kubernetes-server-linux-amd64.tar.gz | a948e26c77fb7cef3c50543c0b92c1ec1085516c4f16f9cd6695a02e1d88e44d5cd1f6fcc13bda48a580a26d024d1665ca2a960b37b31f0a7f0186e941a21e7e
kubernetes-server-linux-arm64.tar.gz | 14538ca02dc149a57c1cb30206b281248f9a84b024dad777e0326a9c6dc6c74228211e1a49f0480e2b7f825b12891d176489bcafcfab0fa05b18acc33c5044f8
kubernetes-server-linux-ppc64le.tar.gz | d347d6072f5a4c6c14ddb9418eadcc075824fa2dd15e49bfb79ee3fab7b2cc0efdd18021d7baed9024a4b83b4f9b800cedaeb8fa3917bfd47c4e5935146fc9fc
kubernetes-server-linux-s390x.tar.gz | 210ed1f933ba611cc3a828382ad15b02d2e35e74e99baed7077c21708249c5a561963e25f2582562773f1ea8f3eccb89b9fa25ca58da6eec9a516652efd432a5
Node Binaries
filename | sha512 hash
-------- | -----------
kubernetes-node-linux-amd64.tar.gz | 6d9b9e382a3137a2622a4631162b5c6a0c0c709fe95b76a7d5af610aec2a292b2f5a0b3378ddf8243450d774f6c1cd2ca16cbf240aed7109d819cd366b7abda9
kubernetes-node-linux-arm64.tar.gz | 0951c701155c914a0578dab9c8d584d32e8260ca923e1efccd0739db2065bf3d37d5d1b6584bc38f7873e20d164e57603e79abfecfbbe89e5386b6d7738d521b
kubernetes-node-linux-ppc64le.tar.gz | afbbedb58bd8344608e1fe047666914874419aef7f31c057a992e0dc24acae6151a7b0c53c2cfc8144ab8e0e914ee8b3a2f11adbb3791fb3b412172ade67439d
kubernetes-node-linux-s390x.tar.gz | 10c73d669dde0841078e5cee9158fa1a551c8bfe668d07beadf316386f815979f8729b89a0bed7e9e76350e82f6fd94d204187b9c3fe6e2bc1aabb2e580fee87
kubernetes-node-windows-amd64.tar.gz | c94d9f4979aeebfae9e66e029ea99ef6e349209f641d853032f87bb9cb646e885b995a512be3eef8e8cf2def76418e70086a03c00121e22c082b67b45562a6c5
Container Images
All container images are available as manifest lists and support the described
architectures. It is also possible to pull a specific architecture directly by
adding the "-$ARCH" suffix to the container image name.
name | architectures
---- | -------------
registry.k8s.io/conformance:v1.29.0-alpha.3 | amd64, arm64, ppc64le, s390x
registry.k8s.io/kube-apiserver:v1.29.0-alpha.3 | amd64, arm64, ppc64le, s390x
registry.k8s.io/kube-controller-manager:v1.29.0-alpha.3 | amd64, arm64, ppc64le, s390x
registry.k8s.io/kube-proxy:v1.29.0-alpha.3 | amd64, arm64, ppc64le, s390x
registry.k8s.io/kube-scheduler:v1.29.0-alpha.3 | amd64, arm64, ppc64le, s390x
registry.k8s.io/kubectl:v1.29.0-alpha.3 | amd64, arm64, ppc64le, s390x
Changelog since v1.29.0-alpha.2
Urgent Upgrade Notes
(No, really, you MUST read this before you upgrade)
- Kubeadm: deploy a separate "super-admin.conf" file. The User in "admin.conf" is now bound to a new RBAC Group "kubeadm:cluster-admins" that have "cluster-admin" ClusterRole access. The User in "super-admin.conf" is bound to the "system:masters" built-in super-powers / break-glass Group that can bypass RBAC. Before this change the default "admin.conf" was bound to "system:masters" Group which was undesired. Executing "kubeadm init phase kubeconfig all" or just "kubeadm init" will now generate the new "super-admin.conf" file. The cluster admin can then decide to keep the file present on a node host or move it to a safe location. "kubadm certs renew" will renew the certificate in "super-admin.conf" to one year if the file exists. If it does not exist a "MISSING" note will be printed. "kubeadm upgrade apply" for this release will migrate this particular node to the two file setup. Subsequent kubeadm releases will continue to optionally renew the certificate in "super-admin.conf" if the file exists on disk and if renew on upgrade is not disabled. "kubeadm join --control-plane" will now generate only an "admin.conf" file that has the less privileged User. (#121305, @neolit123) [SIG Cluster Lifecycle]
- Stop accepting component configuration for kube-proxy and kubelet during
kubeadm upgrade plan --config
. This is a legacy behavior that is not well supported for upgrades and can be used only at the plan stage to determine if the configuration for these components stored in the cluster needs manual version migration. In the future, kubeadm will attempt alternative component config migration approaches. (#120788, @chendave) [SIG Cluster Lifecycle]
Changes by Kind
Deprecation
- Creation of new CronJob objects containing
TZ
orCRON_TZ
in.spec.schedule
, accidentally enabled in 1.22, is now disallowed. Use the.spec.timeZone
field instead, supported in 1.25+ clusters in default configurations. See https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/#unsupported-timezone-specification for more information. (#116252, @soltysh) [SIG Apps] - Remove the networking alpha API ClusterCIDR (#121229, @aojea) [SIG Apps, CLI, Cloud Provider, Network and Testing]
API Change
- A new sleep action for the PreStop lifecycle hook is added, allowing containers to pause for a specified duration before termination. (#119026, @AxeZhan) [SIG API Machinery, Apps, Node and Testing]
- Add ImageMaximumGCAge field to Kubelet configuration, which allows a user to set the maximum age an image is unused before it's garbage collected. (#121275, @haircommander) [SIG API Machinery and Node]
- Add a new ServiceCIDR type that allows to dynamically configure the cluster range used to allocate Service ClusterIPs addresses (#116516, @aojea) [SIG API Machinery, Apps, Auth, CLI, Network and Testing]
- Add the DisableNodeKubeProxyVersion feature gate. If DisableNodeKubeProxyVersion is enabled, the kubeProxyVersion field is not set. (#120954, @HirazawaUi) [SIG API Machinery, Apps and Node]
- Added Windows support for InPlace Pod Vertical Scaling feature. (#112599, @fabi200123) [SIG Autoscaling, Node, Scalability, Scheduling and Windows]
- Added
UserNamespacesPodSecurityStandards
feature gate to enable user namespace support for Pod Security Standards.
Enabling this feature will modify all Pod Security Standard rules to allow setting:spec[.*].securityContext.[runAsNonRoot,runAsUser]
.
This feature gate should only be enabled if all nodes in the cluster support the user namespace feature and have it enabled.
The feature gate will not graduate or be enabled by default in future Kubernetes releases. (#118760, @saschagrunert) [SIG API Machinery, Auth, Node and Release] - Added options for configuring nf_conntrack_udp_timeout, and nf_conntrack_udp_timeout_stream variables of netfilter conntrack subsystem. (#120808, @aroradaman) [SIG API Machinery and Network]
- Adds CEL expressions to v1alpha1 AuthenticationConfiguration. (#121078, @aramase) [SIG API Machinery, Auth and Testing]
- Adds support for CEL expressions to v1alpha1 AuthorizationConfiguration webhook matchConditions. (#121223, @ritazh) [SIG API Machinery and Auth]
- CSINodeExpandSecret feature has been promoted to GA in this release and enabled by default. The CSI drivers can make use of the
secretRef
values passed in NodeExpansion request optionally sent by the CSI Client from this release onwards. (#121303, @humblec) [SIG API Machinery, Apps and Storage] - Graduate Job BackoffLimitPerIndex feature to Beta (#121356, @mimowo) [SIG Apps]
- Kube-apiserver: adds --authorization-config flag for reading a configuration file containing an apiserver.config.k8s.io/v1alpha1 AuthorizationConfiguration object. --authorization-config flag is mutually exclusive with --authorization-modes and --authorization-webhook-* flags. The alpha StructuredAuthorizationConfiguration feature flag must be enabled for --authorization-config to be specified. (#120154, @palnabarun) [SIG API Machinery, Auth and Testing]
-
Kube-proxy now has a new nftables-based mode, available by running
kube-proxy --feature-gates NFTablesProxyMode=true --proxy-mode nftables
This is currently an alpha-level feature and while it probably will not
eat your data, it may nibble at it a bit. (It passes e2e testing but has
not yet seen real-world use.)
At this point it should be functionally mostly identical to the iptables
mode, except that it does not (and will not) support Service NodePorts on
127.0.0.1. (Also note that there are currently no command-line arguments
for the nftables-specific config; you will need to use a config file if
you want to set the equivalent of any of the --iptables-xxx
options.)
As this code is still very new, it has not been heavily optimized yet;
while it is expected to eventually have better performance than the
iptables backend, very little performance testing has been done so far. (#121046, @danwinship) [SIG API Machinery and Network]
- Kube-proxy: Added an option/flag for configuring the nf_conntrack_tcp_be_liberal
sysctl (in the kernel's netfilter conntrack subsystem). When enabled, kube-proxy will not install the DROP rule for invalid conntrack states, which currently breaks users of asymmetric routing. (#120354, @aroradaman) [SIG API Machinery and Network]
- PersistentVolumeLastPhaseTransitionTime is now beta, enabled by default. (#120627, @RomanBednar) [SIG Storage]
- Promote PodReadyToStartContainers condition to beta. (#119659, @kannon92) [SIG Node and Testing]
- The flowcontrol.apiserver.k8s.io/v1beta3 FlowSchema and PriorityLevelConfiguration APIs has been promoted to flowcontrol.apiserver.k8s.io/v1, with the following changes:
- PriorityLevelConfiguration: the .spec.limited.nominalConcurrencyShares
field defaults to 30
only if the field is omitted (v1beta3 also defaulted an explicit 0
value to 30
). Specifying an explicit 0
value is not allowed in the v1
version in v1.29 to ensure compatibility with 1.28 API servers. In v1.30, explicit 0
values will be allowed in this field in the v1
API.
The flowcontrol.apiserver.k8s.io/v1beta3 APIs are deprecated and will no longer be served in v1.32. All existing objects are available via the v1
APIs. Transition clients and manifests to use the v1
APIs before upgrading to v1.32. (#121089, @tkashem) [SIG API Machinery and Testing]
- The kube-proxy command-line documentation was updated to clarify that
--bind-address
does not actually have anything to do with binding to an
address, and you probably don't actually want to be using it. (#120274, @danwinship) [SIG Network]
- The matchLabelKeys/mismatchLabelKeys feature is introduced to the hard/soft PodAffinity/PodAntiAffinity. (#116065, @sanposhiho) [SIG API Machinery, Apps, Cloud Provider, Scheduling and Testing]
- ValidatingAdmissionPolicy Type Checking now supports CRDs and API extensions types. (#119109, @jiahuif) [SIG API Machinery, Apps, Auth and Testing]
- When updating a CRD, per-expression cost limit check is skipped for x-kubernetes-validations rules of versions that are not mutated. (#121460, @jiahuif) [SIG API Machinery]
Feature
-
Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:
(#119517, @sanposhiho) [SIG Node, Scheduling and Testing]
- --interactive flag in kubectl delete will be visible to all users by default. (#120416, @ardaguclu) [SIG CLI and Testing]
- Add container filesystem to the ImageFsInfoResponse. (#120914, @kannon92) [SIG Node and Testing]
- Add job_pods_creation_total metrics for tracking Pods created by the Job controller labeled by events which triggered the Pod creation (#121481, @dejanzele) [SIG Apps and Testing]
- Add multiplication functionality to Quantity. (#117411, @tenzen-y) [SIG API Machinery]
- Added a new --init-only
command line flag to kube-proxy
. Setting the flag makes kube-proxy
perform its initial configuration that requires privileged mode, and then exit. The --init-only
mode is intended to be executed in a privileged init container, so that the main container may run with a stricter securityContext
. (#120864, @uablrek) [SIG Network and Scalability]
- Added new feature gate called "RuntimeClassInImageCriApi" to address kubelet changes needed for KEP 4216.
Noteable changes:
1. Populate new RuntimeHandler field in CRI's ImageSpec struct during image pulls from container runtimes.
2. Pass runtimeHandler field in RemoveImage() call to container runtime in kubelet's image garbage collection (#121456, @kiashok) [SIG Node and Windows]
- Adds apiextensions_apiserver_update_ratcheting_time
metric for tracking time taken during requests by feature CRDValidationRatcheting
(#121462, @alexzielenski) [SIG API Machinery]
- Bump cel-go to v0.17.7 and introduce set ext library with new options. (#121577, @cici37) [SIG API Machinery, Auth and Cloud Provider]
- Bump distroless-iptables to 0.4.1 based on Go 1.21.3 (#121216, @cpanato) [SIG Testing]
- CEL can now correctly handle a CRD openAPIV3Schema that has neither Properties nor AdditionalProperties. (#121459, @jiahuif) [SIG API Machinery and Testing]
- CEL cost estimator no longer treats enums as unbounded strings when determining its length. Instead, the length is set to the longest possible enum value. (#121085, @jiahuif) [SIG API Machinery]
- CRDValidationRatcheting: Adds support for ratcheting x-kubernetes-validations
in schema (#121016, @alexzielenski) [SIG API Machinery]
- CRI: support image pull per runtime class (#121121, @kiashok) [SIG Node and Windows]
- Calculate restartable init containers resource in pod autoscaler (#120001, @qingwave) [SIG Apps and Autoscaling]
- Certain requestBody params in the OpenAPI v3 are correctly marked as required (#120735, @Jefftree) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node and Storage]
- Client-side apply will use OpenAPI V3 by default (#120707, @Jefftree) [SIG API Machinery and CLI]
- Cluster/gce: add webhook to replace PersistentVolumeLabel admission controller (#121628, @andrewsykim) [SIG Cloud Provider]
- Decouple TaintManager from NodeLifeCycleController (KEP-3902) (#119208, @atosatto) [SIG API Machinery, Apps, Instrumentation, Node, Scheduling and Testing]
- DevicePluginCDIDevices feature has been graduated to Beta and enabled by default in the Kubelet (#121254, @bart0sh) [SIG Node]
- Dra: the scheduler plugin avoids additional scheduling attempts in some cases by falling back to SSA after a conflict (#120534, @pohly) [SIG Node, Scheduling and Testing]
- Enable traces for KMSv2 encrypt/decrypt operations. (#121095, @aramase) [SIG API Machinery, Architecture, Auth, Instrumentation and Testing]
- Etcd: build image for v3.5.9 (#121567, @mzaian) [SIG API Machinery]
- Fixes bugs in handling of server-side apply, create, and update API requests for objects containing duplicate items in keyed lists.
- A create
or update
API request with duplicate items in a keyed list no longer wipes out managedFields. Examples include env var entries with the same name, or port entries with the same containerPort in a pod spec.
- A server-side apply request that makes unrelated changes to an object which has duplicate items in a keyed list no longer fails, and leaves the existing duplicate items as-is.
- A server-side apply request that changes an object which has duplicate items in a keyed list, and modifies the duplicated item removes the duplicates and replaces them with the single item contained in the server-side apply request. (#121575, @apelisse) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node, Storage and Testing]
- Graduate the ReadWriteOncePod
feature gate to GA (#121077, @chrishenzie) [SIG Apps, Node, Scheduling, Storage and Testing]
- Introduce the job_finished_indexes_total metric for BackoffLimitPerIndex feature (#121292, @mimowo) [SIG Apps and Testing]
- KEP-4191- add support for split image filesystem in kubelet (#120616, @kannon92) [SIG Node and Testing]
- Kube-apiserver adds alpha support (guarded by the ServiceAccountTokenJTI feature gate) for adding a jti
(JWT ID) claim to service account tokens it issues, adding an authentication.kubernetes.io/credential-id
audit annotation in audit logs when the tokens are issued, and authentication.kubernetes.io/credential-id
entry in the extra user info when the token is used to authenticate.
- kube-apiserver adds alpha support (guarded by the ServiceAccountTokenPodNodeInfo feature gate) for including the node name (and uid, if the node exists) as additional claims in service account tokens it issues which are bound to pods, and authentication.kubernetes.io/node-name
and authentication.kubernetes.io/node-uid
extra user info when the token is used to authenticate.
- kube-apiserver adds alpha support (guarded by the ServiceAccountTokenNodeBinding feature gate) for allowing TokenRequests that bind tokens directly to nodes, and (guarded by the ServiceAccountTokenNodeBindingValidation feature gate) for validating the node name and uid still exist when the token is used. (#120780, @munnerz) [SIG API Machinery, Apps, Auth, CLI and Testing]
- Kube-controller-manager: The LegacyServiceAccountTokenCleanUp
feature gate is now beta and enabled by default. When enabled, legacy auto-generated service account token secrets are auto-labeled with a kubernetes.io/legacy-token-invalid-since
label if the credentials have not been used in the time specified by --legacy-service-account-token-clean-up-period
(defaulting to one year), and are referenced from the .secrets
list of a ServiceAccount object, and are not referenced from pods. This label causes the authentication layer to reject use of the credentials. After being labeled as invalid, if the time specified by --legacy-service-account-token-clean-up-period
(defaulting to one year) passes without the credential being used, the secret is automatically deleted. Secrets labeled as invalid which have not been auto-deleted yet can be re-activated by removing the kubernetes.io/legacy-token-invalid-since
label. (#120682, @yt2985) [SIG Apps, Auth and Testing]
- Kube-scheduler implements scheduling hints for the NodeAffinity plugin.
The scheduling hints allow the scheduler to only retry scheduling a Pod
that was previously rejected by the NodeAffinity plugin if a new Node or a Node update matches the Pod's node affinity. (#119155, @carlory) [SIG Scheduling]
- Kubeadm: Turn on FeatureGate MergeCLIArgumentsWithConfig
to merge the config from flag and config file, otherwise, If the flag --ignore-preflight-errors
is set from CLI, then the value from config file will be ignored. (#119946, @chendave) [SIG Cluster Lifecycle]
- Kubeadm: allow deploying a kubelet that is 3 versions older than the version of kubeadm (N-3). This aligns with the recent change made by SIG Architecture that extends the support skew between the control plane and kubelets. Tolerate this new kubelet skew for the commands "init", "join" and "upgrade". Note that if the kubeadm user applies a control plane version that is older than the kubeadm version (N-1 maximum) then the skew between the kubelet and control plane would become a maximum of N-2. (#120825, @pacoxu) [SIG Cluster Lifecycle]
- Kubelet allows pods to use the net.ipv4.tcp_fin_timeout
, “net.ipv4.tcp_keepalive_intvl” and “net.ipv4.tcp_keepalive_probes“ sysctl by default; Pod Security admission allows this sysctl in v1.29+ versions of the baseline and restricted policies. (#121240, @HirazawaUi) [SIG Auth and Node]
- Kubelet allows pods to use the net.ipv4.tcp_keepalive_time
sysctl by default and the minimal kernel version is 4.5; Pod Security admission allows this sysctl in v1.29+ versions of the baseline and restricted policies. (#118846, @cyclinder) [SIG Auth, Network and Node]
- Kubelet emits a metric for end-to-end pod startup latency including image pull. (#121041, @ruiwen-zhao) [SIG Node]
- Kubernetes is now built with Go 1.21.3 (#121149, @cpanato) [SIG Release and Testing]
- Make decoding etcd's response respect the timeout context. (#121614, @HirazawaUi) [SIG API Machinery]
- Priority and Fairness feature is stable in 1.29, the feature gate will be removed in 1.31 (#121638, @tkashem) [SIG API Machinery and Testing]
- Promote PodHostIPs condition to beta. (#120257, @wzshiming) [SIG Network, Node and Testing]
- Promote PodHostIPs condition to beta. (#121477, @wzshiming) [SIG Network and Testing]
- Promote PodReplacementPolicy to beta. (#121491, @dejanzele) [SIG Apps and Testing]
- Promotes plugin subcommand resolution feature to beta (#120663, @ardaguclu) [SIG CLI and Testing]
- Sidecar termination is now serialized and each sidecar container will receive a SIGTERM after all main containers and later starting sidecar containers have terminated. (#120620, @tzneal) [SIG Node and Testing]
- The CRD validation rule with feature gate CustomResourceValidationExpressions
is promoted to GA. (#121373, @cici37) [SIG API Machinery and Testing]
- The KMSv2 feature with feature gates KMSv2
and KMSv2KDF
are promoted to GA. The KMSv1
feature gate is now disabled by default. (#121485, @ritazh) [SIG API Machinery, Auth and Testing]
- The SidecarContainers feature has graduated to beta and is enabled by default. (#121579, @gjkim42) [SIG Node]
- Updated the generic apiserver library to produce an error if a new API server is configured with support for a data format other than JSON, YAML, or Protobuf. (#121325, @benluddy) [SIG API Machinery]
- ValidatingAdmissionPolicy now preserves types of composition variables, and raise type-related errors early. (#121001, @jiahuif) [SIG API Machinery and Testing]
Documentation
- When the Kubelet fails to assign CPUs to a Pod because there less available CPUs than the Pod requests, the error message changed from
"not enough cpus available to satisfy request" to "not enough cpus available to satisfy request:requested, only available". (#121059, @matte21) [SIG Node]
Failing Test
- K8s.io/dynamic-resource-allocation: DRA drivers updating to this release are compatible with Kubernetes 1.27 and 1.28. (#120868, @pohly) [SIG Node]
Bug or Regression
- Add CAP_NET_RAW to netadmin debug profile and remove privileges when debugging nodes (#118647, @mochizuki875) [SIG CLI and Testing]
- Add a check: if a user attempts to create a static pod via the kubelet without specifying a name, they will get a visible validation error. (#119522, @YTGhost) [SIG Node]
- Bugfix: OpenAPI spec no longer includes default of
{}
for certain fields where it did not make sense (#120757, @alexzielenski) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node and Storage] - Changed kubelet logs from error to info for uncached partitions when using CRI stats provider (#100448, @saschagrunert) [SIG Node]
- Do not assign an empty value to the resource (CPU or memory) that not defined when stores the resources allocated to the pod in checkpoint (#117615, @aheng-ch) [SIG Node]
- Etcd: Update to v3.5.10 (#121566, @mzaian) [SIG API Machinery, Cloud Provider, Cluster Lifecycle, Etcd and Testing]
- Fix 121094 by re-introducing the readiness predicate for externalTrafficPolicy: Local services. (#121116, @alexanderConstantinescu) [SIG Cloud Provider and Network]
- Fix panic in Job controller when podRecreationPolicy: Failed is used, and the number of terminating pods exceeds parallelism. (#121147, @kannon92) [SIG Apps]
- Fix systemLogQuery service name matching (#120678, @rothgar) [SIG Node]
- Fixed a 1.28.0 regression where kube-controller-manager can crash when StatefulSet with Parallel policy and PVC labels is scaled up. (#121142, @aleksandra-malinowska) [SIG Apps]
- Fixed a bug around restarting init containers in the right order relative to normal containers with SidecarContainers feature enabled. (#120269, @gjkim42) [SIG Node and Testing]
- Fixed a bug where an API group's path was not unregistered from the API server's root paths when the group was deleted. (#121283, @tnqn) [SIG API Machinery and Testing]
- Fixed a bug where the CPU set allocated to an init container, with containerRestartPolicy of
Always
, were erroneously reused by a regular container. (#119447, @gjkim42) [SIG Node and Testing] - Fixed a bug where the device resources allocated to an init container, with containerRestartPolicy of
Always
, were erroneously reused by a regular container. (#120461, @gjkim42) [SIG Node and Testing] - Fixed a bug where the memory resources allocated to an init container, with containerRestartPolicy of
Always
, were erroneously reused by a regular container. (#120715, @gjkim42) [SIG Node] - Fixed a regression in default configurations, which enabled PodDisruptionConditions by default,
that prevented the control plane's pod garbage collector from deleting pods that contained duplicated field keys (env. variables with repeated keys or container ports). (#121103, @mimowo) [SIG Apps, Auth, Node, Scheduling and Testing] - Fixed a regression in the Kubelet's behavior while creating a container when the
EventedPLEG
feature gate is enabled (#120942, @sairameshv) [SIG Node] - Fixed a regression since 1.27.0 in scheduler framework when running score plugins.
TheskippedScorePlugins
number might be greater thanenabledScorePlugins
,
so when initializing a slice the cap(len(skippedScorePlugins) - len(enabledScorePlugins)) is negative,
which is not allowed. (#121632, @kerthcet) [SIG Scheduling] - Fixed bug that kubelet resource metric
container_start_time_seconds
had timestamp equal to container start time. (#120518, @saschagrunert) [SIG Instrumentation, Node and Testing] - Fixed inconsistency in the calculation of number of nodes that have an image, which affect the scoring in the ImageLocality plugin (#116938, @olderTaoist) [SIG Scheduling]
- Fixed some invalid and unimportant log calls. (#121249, @pohly) [SIG Cloud Provider, Cluster Lifecycle and Testing]
- Fixed the bug that kubelet could't output logs after log file rotated when kubectl logs POD_NAME -f is running. (#115702, @xyz-li) [SIG Node]
- Fixed the issue where pod with ordinal number lower than the rolling partitioning number was being deleted it was coming up with updated image. (#120731, @adilGhaffarDev) [SIG Apps and Testing]
- Fixed tracking of terminating Pods in the Job status. The field was not updated unless there were other changes to apply (#121342, @dejanzele) [SIG Apps and Testing]
- Fixes an issue where StatefulSet might not restart a pod after eviction or node failure. (#121389, @aleksandra-malinowska) [SIG Apps and Testing]
- Fixes calculating the requeue time in the cronjob controller, which results in properly handling failed/stuck jobs (#121327, @soltysh) [SIG Apps]
- Forbid sysctls for pod sharing the respective namespaces with the host when creating and update pod without such sysctls (#118705, @pacoxu) [SIG Apps and Node]
- K8s.io/dynamic-resource-allocation/controller: ResourceClaimParameters and ResourceClassParameters validation errors were not visible on ResourceClaim, ResourceClass and Pod. (#121065, @byako) [SIG Node]
- Kube-proxy now reports its health more accurately in dual-stack clusters when there are problems with only one IP family. (#118146, @aroradaman) [SIG Network and Windows]
- Metric buckets for pod_start_duration_seconds are changed to {0.5, 1, 2, 3, 4, 5, 6, 8, 10, 20, 30, 45, 60, 120, 180, 240, 300, 360, 480, 600, 900, 1200, 1800, 2700, 3600} (#120680, @ruiwen-zhao) [SIG Instrumentation and Node]
- Mitigates http/2 DOS vulnerabilities for CVE-2023-44487 and CVE-2023-39325 for the API server when the client is unauthenticated. The mitigation may be disabled by setting the
UnauthenticatedHTTP2DOSMitigation
feature gate tofalse
(it is enabled by default). An API server fronted by an L7 load balancer that already mitigates these http/2 attacks may choose to disable the kube-apiserver mitigation to avoid disrupting load balancer → kube-apiserver connections if http/2 requests from multiple clients share the same backend connection. An API server on a private network may opt to disable the kube-apiserver mitigation to prevent performance regressions for unauthenticated clients. Authenticated requests rely on the fix in golang.org/x/net v0.17.0 alone. https://issue.k8s.io/121197 tracks further mitigation of http/2 attacks by authenticated clients. (#121120, @enj) [SIG API Machinery] - Registered metric
apiserver_request_body_size_bytes
to track the size distribution of requests byresource
andverb
. (#120474, @YaoC) [SIG API Machinery and Instrumentation] - Update the CRI-O socket path, so users who configure kubelet to use a location like
/run/crio/crio.sock
don't see strange behaviour from CRI stats provider. (#118704, @dgl) [SIG Node] - Wait.PollUntilContextTimeout function, if immediate is true, the condition will be invoked before waiting and guarantees that the condition is invoked at least once and then wait a interval before executing again. (#119762, @AxeZhan) [SIG API Machinery]
Other (Cleanup or Flake)
- Allow using lower and upper case feature flag value, the name has to match still (#121441, @soltysh) [SIG CLI]
- E2E storage tests: setting test tags like
[Slow]
via the DriverInfo.FeatureTag field is no longer supported. (#121391, @pohly) [SIG Storage and Testing] - EnqueueExtensions from plugins other than PreEnqueue, PreFilter, Filter, Reserve and Permit are ignored.
It reduces the number of kinds of cluster events the scheduler needs to subscribe/handle. (#121571, @sanposhiho) [SIG Scheduling] - GetPodQOS(pod core.Pod) function now returns the stored value from PodStatus.QOSClass, if set. To compute/evaluate the value of QOSClass from scratch, ComputePodQOS(pod core.Pod) must be used. (#119665, @vinaykul) [SIG API Machinery, Apps, CLI, Node, Scheduling and Testing]
- Graduate JobReadyPods to stable. The feature gate can no longer be disabled. (#121302, @stuton) [SIG Apps and Testing]
- Kube-controller-manager's help will include controllers behind a feature gate in
--controllers
flag (#120371, @atiratree) [SIG API Machinery] - Kubeadm: remove leftover ALPHA disclaimer that can be seen in the "kubeadm init phase certs" command help screen. The "certs" phase of "init" is not ALPHA. (#121172, @SataQiu) [SIG Cluster Lifecycle]
- Migrated the remainder of the scheduler to use contextual logging. (#120933, @mengjiao-liu) [SIG Instrumentation, Scheduling and Testing]
- Previous versions of Kubernetes on Google Cloud required that workloads (e.g. Deployments, DaemonSets, etc.) which used PersistentDisk volumes were using them in read-only mode. This validation provided very little value at relatively host implementation cost, and will no longer be validated. If this is a problem for a specific use-case, please set the
SkipReadOnlyValidationGCE
gate to false to re-enable the validation, and file a kubernetes bug with details. (#121083, @thockin) [SIG Apps] - Remove GA featuregate about CSIMigrationvSphere in 1.29 (#121291, @bzsuni) [SIG API Machinery, Node and Storage]
- Remove GA featuregate about ProbeTerminationGracePeriod in 1.29 (#121257, @bzsuni) [SIG Node and Testing]
- Remove GA featuregate for JobTrackingWithFinalizers in 1.28 (#119100, @bzsuni) [SIG Apps]
- Remove GAed feature gates OpenAPIV3 (#121255, @tukwila) [SIG API Machinery and Testing]
- Remove GAed feature gates SeccompDefault (#121246, @tukwila) [SIG Node]
- Remove GAed feature gates TopologyManager (#121252, @tukwila) [SIG Node]
- Removed the
CronJobTimeZone
feature gate (the feature is stable and always enabled) - Removed the
JobMutableNodeSchedulingDirectives
feature gate (the feature is stable and always enabled) - Removed the
LegacyServiceAccountTokenNoAutoGeneration
feature gate (the feature is stable and always enabled) (#120192, @SataQiu) [SIG Apps, Auth and Scheduling] - Removed the
DownwardAPIHugePages
feature gate (the feature is stable and always enabled) (#120249, @pacoxu) [SIG Apps and Node] - Removed the
GRPCContainerProbe
feature gate (the feature is stable and always enabled) (#120248, @pacoxu) [SIG API Machinery, CLI and Node] - Rename apiserver_request_body_sizes metric to apiserver_request_body_size_bytes (#120503, @dgrisonnet) [SIG API Machinery]
- RetroactiveDefaultStorageClass feature gate that graduated to GA in 1.28 and was unconditionally enabled has been removed in v1.29. (#120861, @RomanBednar) [SIG Storage]
Dependencies
Added
- cloud.google.com/go/dataproc/v2: v2.0.1
- github.com/danwinship/knftables: v0.0.13
- github.com/google/s2a-go: v0.1.7
- google.golang.org/genproto/googleapis/bytestream: e85fd2c
Changed
- cloud.google.com/go/accessapproval: v1.6.0 → v1.7.1
- cloud.google.com/go/accesscontextmanager: v1.7.0 → v1.8.1
- cloud.google.com/go/aiplatform: v1.37.0 → v1.48.0
- cloud.google.com/go/analytics: v0.19.0 → v0.21.3
- cloud.google.com/go/apigateway: v1.5.0 → v1.6.1
- cloud.google.com/go/apigeeconnect: v1.5.0 → v1.6.1
- cloud.google.com/go/apigeeregistry: v0.6.0 → v0.7.1
- cloud.google.com/go/appengine: v1.7.1 → v1.8.1
- cloud.google.com/go/area120: v0.7.1 → v0.8.1
- cloud.google.com/go/artifactregistry: v1.13.0 → v1.14.1
- cloud.google.com/go/asset: v1.13.0 → v1.14.1
- cloud.google.com/go/assuredworkloads: v1.10.0 → v1.11.1
- cloud.google.com/go/automl: v1.12.0 → v1.13.1
- cloud.google.com/go/baremetalsolution: v0.5.0 → v1.1.1
- cloud.google.com/go/batch: v0.7.0 → v1.3.1
- cloud.google.com/go/beyondcorp: v0.5.0 → v1.0.0
- cloud.google.com/go/bigquery: v1.50.0 → v1.53.0
- cloud.google.com/go/billing: v1.13.0 → v1.16.0
- cloud.google.com/go/binaryauthorization: v1.5.0 → v1.6.1
- cloud.google.com/go/certificatemanager: v1.6.0 → v1.7.1
- cloud.google.com/go/channel: v1.12.0 → v1.16.0
- cloud.google.com/go/cloudbuild: v1.9.0 → v1.13.0
- cloud.google.com/go/clouddms: v1.5.0 → v1.6.1
- cloud.google.com/go/cloudtasks: v1.10.0 → v1.12.1
- cloud.google.com/go/compute: v1.19.0 → v1.23.0
- cloud.google.com/go/contactcenterinsights: v1.6.0 → v1.10.0
- cloud.google.com/go/container: v1.15.0 → v1.24.0
- cloud.google.com/go/containeranalysis: v0.9.0 → v0.10.1
- cloud.google.com/go/datacatalog: v1.13.0 → v1.16.0
- cloud.google.com/go/dataflow: v0.8.0 → v0.9.1
- cloud.google.com/go/dataform: v0.7.0 → v0.8.1
- cloud.google.com/go/datafusion: v1.6.0 → v1.7.1
- cloud.google.com/go/datalabeling: v0.7.0 → v0.8.1
- cloud.google.com/go/dataplex: v1.6.0 → v1.9.0
- cloud.google.com/go/dataqna: v0.7.0 → v0.8.1
- cloud.google.com/go/datastore: v1.11.0 → v1.13.0
- cloud.google.com/go/datastream: v1.7.0 → v1.10.0
- cloud.google.com/go/deploy: v1.8.0 → v1.13.0
- cloud.google.com/go/dialogflow: v1.32.0 → v1.40.0
- cloud.google.com/go/dlp: v1.9.0 → v1.10.1
- cloud.google.com/go/documentai: v1.18.0 → v1.22.0
- cloud.google.com/go/domains: v0.8.0 → v0.9.1
- cloud.google.com/go/edgecontainer: v1.0.0 → v1.1.1
- cloud.google.com/go/essentialcontacts: v1.5.0 → v1.6.2
- cloud.google.com/go/eventarc: v1.11.0 → v1.13.0
- cloud.google.com/go/filestore: v1.6.0 → v1.7.1
- cloud.google.com/go/firestore: v1.9.0 → v1.11.0
- cloud.google.com/go/functions: v1.13.0 → v1.15.1
- cloud.google.com/go/gkebackup: v0.4.0 → v1.3.0
- cloud.google.com/go/gkeconnect: v0.7.0 → v0.8.1
- cloud.google.com/go/gkehub: v0.12.0 → v0.14.1
- cloud.google.com/go/gkemulticloud: v0.5.0 → v1.0.0
- cloud.google.com/go/gsuiteaddons: v1.5.0 → v1.6.1
- cloud.google.com/go/iam: v0.13.0 → v1.1.1
- cloud.google.com/go/iap: v1.7.1 → v1.8.1
- cloud.google.com/go/ids: v1.3.0 → v1.4.1
- cloud.google.com/go/iot: v1.6.0 → v1.7.1
- cloud.google.com/go/kms: v1.10.1 → v1.15.0
- cloud.google.com/go/language: v1.9.0 → v1.10.1
- cloud.google.com/go/lifesciences: v0.8.0 → v0.9.1
- cloud.google.com/go/longrunning: v0.4.1 → v0.5.1
- cloud.google.com/go/managedidentities: v1.5.0 → v1.6.1
- cloud.google.com/go/maps: v0.7.0 → v1.4.0
- cloud.google.com/go/mediatranslation: v0.7.0 → v0.8.1
- cloud.google.com/go/memcache: v1.9.0 → v1.10.1
- cloud.google.com/go/metastore: v1.10.0 → v1.12.0
- cloud.google.com/go/monitoring: v1.13.0 → v1.15.1
- cloud.google.com/go/networkconnectivity: v1.11.0 → v1.12.1
- cloud.google.com/go/networkmanagement: v1.6.0 → v1.8.0
- cloud.google.com/go/networksecurity: v0.8.0 → v0.9.1
- cloud.google.com/go/notebooks: v1.8.0 → v1.9.1
- cloud.google.com/go/optimization: v1.3.1 → v1.4.1
- cloud.google.com/go/orchestration: v1.6.0 → v1.8.1
- cloud.google.com/go/orgpolicy: v1.10.0 → v1.11.1
- cloud.google.com/go/osconfig: v1.11.0 → v1.12.1
- cloud.google.com/go/oslogin: v1.9.0 → v1.10.1
- cloud.google.com/go/phishingprotection: v0.7.0 → v0.8.1
- cloud.google.com/go/policytroubleshooter: v1.6.0 → v1.8.0
- cloud.google.com/go/privatecatalog: v0.8.0 → v0.9.1
- cloud.google.com/go/pubsub: v1.30.0 → v1.33.0
- cloud.google.com/go/pubsublite: v1.7.0 → v1.8.1
- cloud.google.com/go/recaptchaenterprise/v2: v2.7.0 → v2.7.2
- cloud.google.com/go/recommendationengine: v0.7.0 → v0.8.1
- cloud.google.com/go/recommender: v1.9.0 → v1.10.1
- cloud.google.com/go/redis: v1.11.0 → v1.13.1
- cloud.google.com/go/resourcemanager: v1.7.0 → v1.9.1
- cloud.google.com/go/resourcesettings: v1.5.0 → v1.6.1
- cloud.google.com/go/retail: v1.12.0 → v1.14.1
- cloud.google.com/go/run: v0.9.0 → v1.2.0
- cloud.google.com/go/scheduler: v1.9.0 → v1.10.1
- cloud.google.com/go/secretmanager: v1.10.0 → v1.11.1
- cloud.google.com/go/security: v1.13.0 → v1.15.1
- cloud.google.com/go/securitycenter: v1.19.0 → v1.23.0
- cloud.google.com/go/servicedirectory: v1.9.0 → v1.11.0
- cloud.google.com/go/shell: v1.6.0 → v1.7.1
- cloud.google.com/go/spanner: v1.45.0 → v1.47.0
- cloud.google.com/go/speech: v1.15.0 → v1.19.0
- cloud.google.com/go/storagetransfer: v1.8.0 → v1.10.0
- cloud.google.com/go/talent: v1.5.0 → v1.6.2
- cloud.google.com/go/texttospeech: v1.6.0 → v1.7.1
- cloud.google.com/go/tpu: v1.5.0 → v1.6.1
- cloud.google.com/go/trace: v1.9.0 → v1.10.1
- cloud.google.com/go/translate: v1.7.0 → v1.8.2
- cloud.google.com/go/video: v1.15.0 → v1.19.0
- cloud.google.com/go/videointelligence: v1.10.0 → v1.11.1
- cloud.google.com/go/vision/v2: v2.7.0 → v2.7.2
- cloud.google.com/go/vmmigration: v1.6.0 → v1.7.1
- cloud.google.com/go/vmwareengine: v0.3.0 → v1.0.0
- cloud.google.com/go/vpcaccess: v1.6.0 → v1.7.1
- cloud.google.com/go/webrisk: v1.8.0 → v1.9.1
- cloud.google.com/go/websecurityscanner: v1.5.0 → v1.6.1
- cloud.google.com/go/workflows: v1.10.0 → v1.11.1
- cloud.google.com/go: v0.110.0 → v0.110.6
- github.com/alecthomas/template: fb15b89 → a0175ee
- github.com/cncf/xds/go: 06c439d → e9ce688
- github.com/cyphar/filepath-securejoin: v0.2.3 → v0.2.4
- github.com/docker/distribution: v2.8.1+incompatible → v2.8.2+incompatible
- github.com/docker/docker: v20.10.21+incompatible → v20.10.24+incompatible
- github.com/envoyproxy/go-control-plane: v0.10.3 → v0.11.1
- github.com/envoyproxy/protoc-gen-validate: v0.9.1 → v1.0.2
- github.com/fsnotify/fsnotify: v1.6.0 → v1.7.0
- github.com/go-logr/logr: v1.2.4 → v1.3.0
- github.com/godbus/dbus/v5: v5.0.6 → v5.1.0
- github.com/golang/glog: v1.0.0 → v1.1.0
- github.com/google/cadvisor: v0.47.3 → v0.48.1
- github.com/google/cel-go: v0.17.6 → v0.17.7
- github.com/google/go-cmp: v0.5.9 → v0.6.0
- github.com/googleapis/gax-go/v2: v2.7.1 → v2.11.0
- github.com/grpc-ecosystem/grpc-gateway/v2: v2.7.0 → v2.16.0
- github.com/ishidawataru/sctp: 7c296d4 → 7ff4192
- github.com/konsorten/go-windows-terminal-sequences: v1.0.3 → v1.0.1
- github.com/onsi/gomega: v1.28.0 → v1.29.0
- github.com/spf13/afero: v1.2.2 → v1.1.2
- github.com/stretchr/testify: v1.8.2 → v1.8.4
- go.etcd.io/bbolt: v1.3.7 → v1.3.8
- go.etcd.io/etcd/api/v3: v3.5.9 → v3.5.10
- go.etcd.io/etcd/client/pkg/v3: v3.5.9 → v3.5.10
- go.etcd.io/etcd/client/v2: v2.305.9 → v2.305.10
- go.etcd.io/etcd/client/v3: v3.5.9 → v3.5.10
- go.etcd.io/etcd/pkg/v3: v3.5.9 → v3.5.10
- go.etcd.io/etcd/raft/v3: v3.5.9 → v3.5.10
- go.etcd.io/etcd/server/v3: v3.5.9 → v3.5.10
- go.opentelemetry.io/contrib/instrumentation/github.com/emicklei/go-restful/otelrestful: v0.35.0 → v0.42.0
- go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc: v0.35.0 → v0.42.0
- go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp: v0.35.1 → v0.44.0
- go.opentelemetry.io/contrib/propagators/b3: v1.10.0 → v1.17.0
- go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc: v1.10.0 → v1.19.0
- go.opentelemetry.io/otel/exporters/otlp/otlptrace: v1.10.0 → v1.19.0
- go.opentelemetry.io/otel/metric: v0.31.0 → v1.19.0
- go.opentelemetry.io/otel/sdk: v1.10.0 → v1.19.0
- go.opentelemetry.io/otel/trace: v1.10.0 → v1.19.0
- go.opentelemetry.io/otel: v1.10.0 → v1.19.0
- go.opentelemetry.io/proto/otlp: v0.19.0 → v1.0.0
- golang.org/x/crypto: v0.12.0 → v0.14.0
- golang.org/x/net: v0.14.0 → v0.17.0
- golang.org/x/oauth2: v0.8.0 → v0.10.0
- golang.org/x/sys: v0.12.0 → v0.13.0
- golang.org/x/term: v0.11.0 → v0.13.0
- golang.org/x/text: v0.12.0 → v0.13.0
- google.golang.org/api: v0.114.0 → v0.126.0
- google.golang.org/genproto/googleapis/api: dd9d682 → 23370e0
- google.golang.org/genproto/googleapis/rpc: 28d5490 → b8732ec
- google.golang.org/genproto: 0005af6 → f966b18
- google.golang.org/grpc: v1.54.0 → v1.58.3
- k8s.io/klog/v2: v2.100.1 → v2.110.1
- k8s.io/kube-openapi: d090da1 → 2dd684a
- sigs.k8s.io/structured-merge-diff/v4: v4.3.0 → v4.4.1
Removed
- cloud.google.com/go/dataproc: v1.12.0
- cloud.google.com/go/gaming: v1.9.0
- github.com/blang/semver: v3.5.1+incompatible
- github.com/jmespath/go-jmespath/internal/testify: v1.5.1
- go.opentelemetry.io/otel/exporters/otlp/internal/retry: v1.10.0
v1.29.0-alpha.2
Downloads for v1.29.0-alpha.2
Source Code
filename | sha512 hash
-------- | -----------
kubernetes.tar.gz | 138f47b2c53030e171d368d382c911048ce5d8387450e5e6717f09ac8cf6289b6c878046912130d58d7814509bbc45dbc19d6ee4f24404321ea18b24ebab2a36
kubernetes-src.tar.gz | 73ab06309d6f6cbcb8a417c068367b670a04dcbe90574a7906201dd70b9c322cd052818114b746a4d61b7bce6115ae547eaafc955c41053898a315c968db2f36
Client Binaries
filename | sha512 hash
-------- | -----------
kubernetes-client-darwin-amd64.tar.gz | c9604fbb9e848a4b3dc85ee2836f74b4ccd321e4c72d22b2d4558eb0f0c3833bff35d0c36602c13c5c5c79e9233fda874bfa85433291ab3484cf61c9012ee515
kubernetes-client-darwin-arm64.tar.gz | fed42ecbfc20b5f63ac48bbb9b73abc4b72aca76ac8bdd51b9ea6af053b1fc6a8e63b5e11f9d14c4814f03b49531da2536f1342cda2da03514c44ccf05c311b0
kubernetes-client-linux-386.tar.gz | 93c61229d7b07a476296b5b800c853c8e984101d5077fc19a195673f7543e7d2eb2599311c1846c91ef1f7ae29c3e05b6f41b873e92a3429563e3d83900050da
kubernetes-client-linux-amd64.tar.gz | 4260b49733f6b0967c504e2246b455b2348b487e84f7a019fda8b4a87d43d27a03e7ed55b505764c14f2079c4c3d71c68d77f981b604e13e7210680f45ee66e3
kubernetes-client-linux-arm.tar.gz | 4e837fd2f55cbb5f93cdf60235511a85635485962f00e0378a95a7ff846eb86b7bf053203ab353b294131b2e2663d0e783dae79c18601d4d66f98a6e5152e96e
kubernetes-client-linux-arm64.tar.gz | 6f3954d2adc289879984d18c2605110a7d5f0a5f6366233c25adf3a742f8dc1183e8a4d4747de8077af1045a259b150e0e86b27e10d683aa8decdc760ac6279b
kubernetes-client-linux-ppc64le.tar.gz | 741b76827ff9e810e490d8698eb7620826a16e978e5c7744a1fa0e65124690cfc9601e7f1c8f50e77f25185ba3176789ddcb7d5caaddde66436c31658bacde1d
kubernetes-client-linux-s390x.tar.gz | 0c635883e2f9caca03bcf3b42ba0b479f44c8cc2a3d5dd425b0fee278f3e884bef0e897fe51cbf00bb0bc061371805d9f9cbccf839477671f92e078c04728735
kubernetes-client-windows-386.tar.gz | ebddbb358fd2d817908069eb66744dc62cae56ad470b1e36c6ebd0d2284e79ae5b9a5f8a86fef365f30b34e14093827ad736814241014f597e2ac88788102cf4
kubernetes-client-windows-amd64.tar.gz | 01a451a809cd45e7916a3e982e2b94d372accab9dfe20667e95c10d56f9194b997721c0c219ff7ff97828b6466108eec6e57dcb33e3e3b0c5f770af1514a9f1a
kubernetes-client-windows-arm64.tar.gz | 473ba648ffde41fd5b63374cc1595eb43b873808c6b0cc5e939628937f3f7fb36dba4b7c7c8ef03408d557442094ec22e12c03f40be137f9cc99761b4cc1a1f8
Server Binaries
filename | sha512 hash
-------- | -----------
kubernetes-server-linux-amd64.tar.gz | c3f7abcee3fdcf6f311b5de0bfe037318e646641c1ce311950d920252623cca285d1f1cef0e2d936c0f981edc1c725897a42aa9e03b77fe5f76f1090665d967f
kubernetes-server-linux-arm64.tar.gz | 17614842df6bb528434b8b063b1d1c3efc8e4eff9cbc182f049d811f68e08514026fbb616199a3dee97e62ce2fd1eda0b9778d8e74040e645c482cfe6a18a8b4
kubernetes-server-linux-ppc64le.tar.gz | 2f818035ef199a7745e24d2ce86abf6c52e351d7922885e264c5d07db3e0f21048c32db85f3044e01443abd87a45f92df52fda44e8df05000754b03f34132f2f
kubernetes-server-linux-s390x.tar.gz | 96a34c768f347f23c46f990a8f6ddf3127b13f7a183453b92eb7bc27ce896767f31b38317a6ae5a11f2d4b459ec9564385f8abe61082a4165928edfee0c9765e
Node Binaries
filename | sha512 hash
-------- | -----------
kubernetes-node-linux-amd64.tar.gz | 66845cf86e32c19be9d339417a4772b9bcf51b2bf4d1ef5acc2e9eb006bbd19b3c036aa3721b3d8fe08b6fb82284ba25a6ecb5eb7b84f657cc968224d028f22c
kubernetes-node-linux-arm64.tar.gz | 98902ee33242f9e78091433115804d54eafde24903a3515f0300f60c0273c7c0494666c221ce418d79e715f8ecf654f0edabc5b69765da26f83a812e963b5afb
kubernetes-node-linux-ppc64le.tar.gz | 82f1213b5942c5c1576afadb4b066dfa1427c7709adf6ba636b9a52dfdb1b20f62b1cc0436b265e714fbee08c71d8786295d2439c10cc05bd58b2ab2a87611d4
kubernetes-node-linux-s390x.tar.gz | 7cb8cb65195c5dd63329d02907cdbb0f5473066606c108f4516570f449623f93b1ca822d5a00fad063ec8630e956fa53a0ab530a8487bccb01810943847d4942
kubernetes-node-windows-amd64.tar.gz | 1222e2d7dbaf7920e1ba927231cc7e275641cf0939be1520632353df6219bbcb3b49515d084e7f2320a2ff59b2de9fee252d8f5e9c48d7509f1174c6cb357b66
Container Images
All container images are available as manifest lists and support the described
architectures. It is also possible to pull a specific architecture directly by
adding the "-$ARCH" suffix to the container image name.
name | architectures
---- | -------------
registry.k8s.io/conformance:v1.29.0-alpha.2 | amd64, arm64, ppc64le, s390x
registry.k8s.io/kube-apiserver:v1.29.0-alpha.2 | amd64, arm64, ppc64le, s390x
registry.k8s.io/kube-controller-manager:v1.29.0-alpha.2 | amd64, arm64, ppc64le, s390x
registry.k8s.io/kube-proxy:v1.29.0-alpha.2 | amd64, arm64, ppc64le, s390x
registry.k8s.io/kube-scheduler:v1.29.0-alpha.2 | amd64, arm64, ppc64le, s390x
registry.k8s.io/kubectl:v1.29.0-alpha.2 | amd64, arm64, ppc64le, s390x
Changelog since v1.29.0-alpha.1
Changes by Kind
Feature
- Adds
apiserver_watch_list_duration_seconds
metrics. Which will measure response latency distribution in seconds for watch list requests broken by group, version, resource and scope (#120490, @p0lyn0mial) [SIG API Machinery and Instrumentation] - Allow-list of metric labels can be configured by supplying a manifest using the --allow-metric-labels-manifest flag (#118299, @rexagod) [SIG Architecture and Instrumentation]
- Bump distroless-iptables to 0.3.3 based on Go 1.21.2 (#121073, @cpanato) [SIG Testing]
- Implements API for streaming for the etcd store implementation
When sendInitialEvents ListOption is set together with watch=true, it begins the watch stream with synthetic init events followed by a synthetic "Bookmark" after which the server continues streaming events. (#119557, @p0lyn0mial) [SIG API Machinery]
- Kubelet, when using cloud provider external, initializes temporary the node addresses using the --node-ip flag values if set, until the cloud provider overrides it. (#121028, @aojea) [SIG Cloud Provider and Node]
- Kubernetes is now built with Go 1.21.2 (#121021, @cpanato) [SIG Release and Testing]
- Migrated the volumebinding scheduler plugins to use contextual logging. (#116803, @mengjiao-liu) [SIG Instrumentation, Scheduling and Storage]
- The kube-apiserver exposes four new metrics to inform about errors on the clusterIP and nodePort allocation logic (#120843, @aojea) [SIG Instrumentation and Network]
Failing Test
- K8s.io/dynamic-resource-allocation: DRA drivers updating to this release are compatible with Kubernetes 1.27 and 1.28. (#120868, @pohly) [SIG Node]
Bug or Regression
- Cluster-bootstrap: improve the security of the functions responsible for generation and validation of bootstrap tokens (#120400, @neolit123) [SIG Cluster Lifecycle and Security]
- Do not fail volume attach or publish operation at kubelet if target path directory already exists on the node. (#119735, @akankshapanse) [SIG Storage]
- Fix regression with adding aggregated apiservices panicking and affected health check introduced in release v1.28.0 (#120814, @Jefftree) [SIG API Machinery and Testing]
- Fixed a bug where containers would not start on cgroupv2 systems where swap is disabled. (#120784, @elezar) [SIG Node]
- Fixed a regression in kube-proxy where it might refuse to start if given
single-stack IPv6 configuration options on a node that has both IPv4 and
IPv6 IPs. (#121008, @danwinship) [SIG Network] - Fixed attaching volumes after detach errors. Now volumes that failed to detach are not treated as attached, Kubernetes will make sure they are fully attached before they can be used by pods. (#120595, @jsafrane) [SIG Apps and Storage]
- Fixes a regression (CLIENTSET_PKG: unbound variable) when invoking deprecated generate-groups.sh script (#120877, @soltysh) [SIG API Machinery]
- K8s.io/dynamic-resource-allocation/controller: UnsuitableNodes did not handle a mix of allocated and unallocated claims correctly. (#120338, @pohly) [SIG Node]
- K8s.io/dynamic-resource-allocation: handle a selected node which isn't listed as potential node (#120871, @pohly) [SIG Node]
- Kubeadm: fix the bug that kubeadm always do CRI detection when --config is passed even if it is not required by the subcommand (#120828, @SataQiu) [SIG Cluster Lifecycle]
Other (Cleanup or Flake)
- Client-go: k8s.io/client-go/tools events and record packages have new APIs for specifying a context and logger (#120729, @pohly) [SIG API Machinery and Instrumentation]
- Deprecated the
--cloud-provider
and--cloud-config
CLI parameters in kube-apiserver.
These parameters will be removed in a future release. (#120903, @dims) [SIG API Machinery]
Dependencies
Added
Nothing has changed.
Changed
- github.com/emicklei/go-restful/v3: v3.9.0 → v3.11.0
- github.com/onsi/ginkgo/v2: v2.9.4 → v2.13.0
- github.com/onsi/gomega: v1.27.6 → v1.28.0
- golang.org/x/crypto: v0.11.0 → v0.12.0
- golang.org/x/mod: v0.10.0 → v0.12.0
- golang.org/x/net: v0.13.0 → v0.14.0
- golang.org/x/sync: v0.2.0 → v0.3.0
- golang.org/x/sys: v0.10.0 → v0.12.0
- golang.org/x/term: v0.10.0 → v0.11.0
- golang.org/x/text: v0.11.0 → v0.12.0
- golang.org/x/tools: v0.8.0 → v0.12.0
Removed
Nothing has changed.
Security
Details
- 🔍View and search all Kubernetes releases.
- 🛠️Create and share lists to track your tools.
- 🚨Setup notifications for major, security, feature or patch updates.
- 🚀Much more coming soon!