Changelog since v1.27.0

Changes by Kind

Bug or Regression

• Fixes a regression in 1.27.0 that resulted in "missing metadata in converted object" errors when modifying objects for multi-version custom resource definitions with a conversion strategy of None. (#117305, @ncdc) [SIG API Machinery]
• Known issue: fixed that the PreEnqueue plugins aren't executed for Pods proceeding to activeQ through backoffQ. (#117194, @sanposhiho) [SIG Release and Scheduling]
• Setting a mirror pod's phase to Succeeded or Failed can prevent the corresponding static pod from restarting due mutation of a Kubelet cache. (#116482, @smarterclayton) [SIG Node]

Dependencies

Added

Nothing has changed.

Changed

Nothing has changed.

Removed

Nothing has changed.

v1.27.0

Documentation

Downloads for v1.27.0

Source Code

filename | sha512 hash
-------- | -----------
kubernetes.tar.gz | 78dbb72f270ab70d0ad70d2da6727eed64bdc54a11892fd6c2157882865f93ab41fedf5fced2f3e71dc0eda5679d06884c262a7960277face4510eed30a3678e
kubernetes-src.tar.gz | 4080d2452ff4fd316a823c1c495e7e9a39d364e24225020a91bf0bc0289c3ef90ade746ef5a05172d6e355af9014cbddf144ca71839ec65fc57f3eaf553fb7ab

Client Binaries

filename | sha512 hash
-------- | -----------
kubernetes-client-darwin-amd64.tar.gz | faa0e340f1829ba694326c6ff71f8527249af03d8d78f784289be4122b6ceb0829fa70ee1eab25f64bbb9f5972ae30f3cfdfefe617ce3360b2897d4f6259bd81
kubernetes-client-darwin-arm64.tar.gz | 9c4fe911e41ab9c355d39b21d77372bd2a070cc376fdfceac362eb6cc3e8f616754cc61593ea140030a81961b40fa6344b7628d7a4edf7e6dcdef29711bbd064
kubernetes-client-linux-386.tar.gz | ba522302624ac7b3a9e5c1a5c80857bdde4c47b44394dbfa8da597ee07b2e1975409e8eac514516329826f593fa82d143a03185ef3c30a97cb1f8011ffb96060
kubernetes-client-linux-amd64.tar.gz | 3ea3b4a866815cc08f1897771d63bf4e4f75b481e1d70417e34581d079a58b647b077382a264224acb52e6a76474d6e92efd22a0d4f7fdfde0c244006beef76a
kubernetes-client-linux-arm.tar.gz | 5fd69b567ab835b35b8156c66eec02ee109f731acf7d68250b05a1f43a56458be68654f95107cd28859b4b8e73d5f64c78aca2f4b1dc74fff3ca8d942c60d2db
kubernetes-client-linux-arm64.tar.gz | f20e579ab71b1cdace22bec0a11314ec44534f0e7040a436c63eb18a47d839e070e5134917ef2b531fe7b8bfee12133fa14de4dac7c0ac7798b4d9fa5679f193
kubernetes-client-linux-ppc64le.tar.gz | c56a2d021b1a99fde0871bbe8e71427b8c4f03847e2bf6cbf526a71f6d7d1060481bd0f00d7dce2bd8afa1c969e02422ac1a2283ab58facd3db43f0713c10212
kubernetes-client-linux-s390x.tar.gz | 4ad879e2ab2b952cc0fdfcd738b6264db60b72174057947737ab07f40dd0c4c727fb042c24323be3accaf8fbc320973821c915fd1bb3c4ea8a22eb16c03ce4a3
kubernetes-client-windows-386.tar.gz | befab85193ce017c647b391606d45d3626e71bf7ea6bbca7f955985e0f505a9c8ca27898ee41c4f3124b7a3788b4a4eab602994415b24b8b0bfc154b938c547e
kubernetes-client-windows-amd64.tar.gz | 0fbba06f00713c32c74d9b62733dfb83a597e3a33ee62bfb3a93de7cd883c460a0c56f25cd1577dd7923ef73312788d9b805020297fcf784722783ac1890253f
kubernetes-client-windows-arm64.tar.gz | b7edbb25dbbf5b0bd9839f93d43f08262cf5f6e138599c034da0ff402c763a0cff18c1e9b42631d250389ff6b865dec4aa35b577fac75a51e65c825ec8efe234

Server Binaries

filename | sha512 hash
-------- | -----------
kubernetes-server-linux-amd64.tar.gz | 9726ba173084adade1c1b0de014ccedc5dc5317a80076cbf20d15fdcd6296dd1e9efcf1b1349456757a5c186fa52293f60411397cb6c79765adff335391add9e
kubernetes-server-linux-arm64.tar.gz | 657726cd4ae93a9696371717a280689af76c488586c49273086bef4e712228025c6e179c2a5c93b8a33640ac42347dd821053485659f9383dbb1b3e2a17f022c
kubernetes-server-linux-ppc64le.tar.gz | 2ff2464453ca8ca2e9e4a024ad730c12fa506379b4a7bd749431fe64ddb13c2dccea05c37dba119799940eac2dc57635e9d70b908d1786a3cdc031a5b70504a5
kubernetes-server-linux-s390x.tar.gz | 44ce8faa8710832593b656e3b053207e05def556ad821b8e08e0c2f33b73f280a455fbef933ea70e9efbe8a085ef7deb47139d4e9af43417d8242029a2b60c35

Node Binaries

filename | sha512 hash
-------- | -----------
kubernetes-node-linux-amd64.tar.gz | 812f5adfafe778200558678af6510f9f315f75b46f7bb4482e92b57d1bed08c4b7f236850bf8e4dcac7018879736d614fc482e3641da06c6f8d0554af4f4ef45
kubernetes-node-linux-arm64.tar.gz | a5e353205a93ebaade50dfd652ee5623b28ee4f6fd8ca949fb2303d708468026ba66c10b70f1761f4099706baad8959993a9ec0053259b94b5f4793aeda27adb
kubernetes-node-linux-ppc64le.tar.gz | b5dbcf8131bad7ef897c64ac482599ac3bedc99e5c211d189e0566543a13c89d0812d7a7b1e4e9655d8d884ee24dc553616c96cf74df19f4d2cce0ea552015ce
kubernetes-node-linux-s390x.tar.gz | 81662d6e14a7500bf2714ae3c0b9070031ea5ef2628c84aaf2a8fe96fed07a52c3677babc74247160cb71ce1fc77b728f549f2c18dfc7dc6a65dfadb7ec17cd7
kubernetes-node-windows-amd64.tar.gz | a8b1a53ba6ee416fb9939961d8290ae1f5e0c21117f1cd6cebbc9ba01cafa235730a2887fd91b92552d07eec78aeb65aefac92899eaf9b2f4f195c61f20d05d7

Container Images

All container images are available as manifest lists and support the described
architectures. It is also possible to pull a specific architecture directly by
adding the "-$ARCH" suffix to the container image name.
name | architectures
---- | -------------
registry.k8s.io/conformance:v1.27.0 | amd64, arm64, ppc64le, s390x
registry.k8s.io/kube-apiserver:v1.27.0 | amd64, arm64, ppc64le, s390x
registry.k8s.io/kube-controller-manager:v1.27.0 | amd64, arm64, ppc64le, s390x
registry.k8s.io/kube-proxy:v1.27.0 | amd64, arm64, ppc64le, s390x
registry.k8s.io/kube-scheduler:v1.27.0 | amd64, arm64, ppc64le, s390x

Changelog since v1.26.0

Known Issues

The PreEnqueue extension point doesn't work for Pods going to activeQ through backoffQ

In v1.26.0, we've found the bug that the PreEnqueue extension point doesn't work for Pods going to activeQ through backoffQ.
It doesn't affect any of the vanilla Kubernetes behavior, but, may break custom PreEnqueue plugins.

The cause PR is reverted by v1.26.1.

Urgent Upgrade Notes

(No, really, you MUST read this before you upgrade)

• 'The IPv6DualStack feature gate for external cloud providers was removed.
  (The feature became GA in 1.23 and the gate was removed for all other
  components several releases ago.) If you were still manually
  enabling it you must stop now.' (#116255, @danwinship)
• Give terminal phase correctly to all pods that will not be restarted.

In particular, assign Failed phase to pods which are deleted while pending. Also, assign a terminal
phase (Succeeded or Failed, depending on the exit statuses of the pod containers) to pods which
are deleted while running.

This fixes the issue for jobs using pod failure policy (with JobPodFailurePolicy and PodDisruptionConditions
feature gates enabled) that their pods could get stuck in the pending phase when deleted.

Users who maintain controllers which relied on the fact that pods with RestartPolicy=Always
never enter the Succeeded phase may need to adapt their controllers. This is because as a consequence of
the change pods which use RestartPolicy=Always may end up in the Succeeded phase in two scenarios: pod
deletion and graceful node shutdown. (#115331, @mimowo) [SIG Cloud Provider, Node and Testing]

Changes by Kind

Deprecation

• Added a warning response when handling requests that set the deprecated spec.externalID field for a Node. (#115944, @SataQiu) [SIG Node]
• Added warnings to the Services API. Kubernetes now warns for Services in the case of:
• IPv4 addresses with leading zeros
• IPv6 address in non-canonical format (RFC 5952) (#114505, @aojea)
• Support for the alpha seccomp annotations seccomp.security.alpha.kubernetes.io/pod and container.seccomp.security.alpha.kubernetes.io were deprecated since v1.19, now have been completely removed. The seccomp fields are no longer auto-populated when pods with seccomp annotations are created. Pods should use the corresponding pod or container securityContext.seccompProfile field instead. (#114947, @saschagrunert)
• The SecurityContextDeny admission plugin is going deprecated and will be removed in future versions. (#115879, @mtardy)

API Change

• A fix in the resource.k8s.io/v1alpha1/ResourceClaim API avoids harmless (?) ".status.reservedFor: element 0: associative list without keys has an element that's a map type" errors in the apiserver. Validation now rejects the incorrect reuse of the same UID in different entries. (#115354, @pohly)
• A terminating pod on a node that is not caused by preemption no longer prevents kube-scheduler from preempting pods on that node
• Rename PreemptionByKubeScheduler to PreemptionByScheduler (#114623, @Huang-Wei)
• API: resource.k8s.io/v1alpha1.PodScheduling was renamed to resource.k8s.io/v1alpha2.PodSchedulingContext. (#116556, @pohly) [SIG API Machinery, Apps, Auth, CLI, Node, Scheduling and Testing]
• Added CEL runtime cost calculation into ValidatingAdmissionPolicy, matching the evaluation cost
  restrictions that already apply to CustomResourceDefinition.
  If rule evaluation uses more compute than the limit, the API server aborts the evaluation and the
  admission check that was being performed is aborted; the failurePolicy for the ValidatingAdmissionPolicy
  determines the outcome. (#115747, @cici37)
• Added auditAnnotations to ValidatingAdmissionPolicy, enabling CEL to be used to add audit annotations to request audit events.
  Added validationActions to ValidatingAdmissionPolicyBinding, enabling validation failures to be handled by any combination of the warn, audit and deny enforcement actions. (#115973, @jpbetz)
• Added messageExpression field to ValidationRule. (#115969, @DangerOnTheRanger)
• Added messageExpression to ValidatingAdmissionPolicy, to set custom failure message via CEL expression. (#116397, @jiahuif) [SIG API Machinery]
• Added a new IPAddress object kind
• Added a new ClusterIP allocator. The new allocator removes previous Service CIDR block size limitations for IPv4, and limits IPv6 size to a /64 (#115075, @aojea) [SIG API Machinery, Apps, Auth, CLI, Cluster Lifecycle, Network and Testing]
• Added a new alpha API: ClusterTrustBundle (certificates.k8s.io/v1alpha1).
  A ClusterTrustBundle may be used to distribute X.509 trust anchors to workloads within the cluster. (#113218, @ahmedtd) [SIG API Machinery, Auth and Testing]
• Added authorization check support to the CEL expressions of ValidatingAdmissionPolicy via a authorizer
  variable with expressions. The new variable provides a builder that allows expressions such authorizer.group('').resource('pods').check('create').allowed(). (#116054, @jpbetz) [SIG API Machinery and Testing]
• Added matchConditions field to ValidatingAdmissionPolicy and enabled support for CEL based custom match criteria. (#116350, @maxsmythe)
• Added new option to the InterPodAffinity scheduler plugin to ignore existing
  podspreferred inter-pod affinities if the incoming pod has no preferred inter-pod affinities. This option can be used as an optimization for higher scheduling throughput (at the cost of an occasional pod being scheduled non-optimally/violating existing pods preferred inter-pod affinities). To enable this scheduler option, set theInterPodAffinityscheduler plugin argignorePreferredTermsOfExistingPods: true` (#114393, @danielvegamyhre)
• Added the MatchConditions field to ValidatingWebhookConfiguration and MutatingWebhookConfiguration for the v1beta and v1 apis.

The AdmissionWebhookMatchConditions featuregate is now in Alpha (#116261, @ivelichkovich) [SIG API Machinery and Testing]
- Added validation to ensure that if service.kubernetes.io/topology-aware-hints and service.kubernetes.io/topology-mode annotations are both set, they are set to the same value.Also Added deprecation warning if service.kubernetes.io/topology-aware-hints annotation is used. (#116612, @robscott)
- Added warnings about workload resources (Pods, ReplicaSets, Deployments, Jobs, CronJobs, or ReplicationControllers) whose names are not valid DNS labels. (#114412, @thockin)
- Adds feature gate NodeLogQuery which provides cluster administrators with a streaming view of logs using kubectl without them having to implement a client side reader or logging into the node. (#96120, @LorbusChris)
- Api: validation of a PodSpec now rejects invalid ResourceClaim and ResourceClaimTemplate names. For a pod, the name generated for the ResourceClaim when using a template also must be valid. (#116576, @pohly)
- Bump default API QPS limits for Kubelet. (#116121, @wojtek-t)
- Enabled the StatefulSetStartOrdinal feature gate in beta (#115260, @pwschuurman)
- Enabled usage of kube-proxy, kube-scheduler and kubelet HTTP APIs for changing the logging
verbosity at runtime for JSON output. (#114609, @pohly)
- Encryption of API Server at rest configuration now allows the use of wildcards in the list of resources. For example, . can be used to encrypt all resources, including all current and future custom resources. (#115149, @nilekhc)
- Extended the kubelet's PodResources API to include resources allocated in ResourceClaims via DynamicResourceAllocation. Additionally, added a new Get() method to query a specific pod for its resources. (#115847, @moshe010) [SIG Node]
- Forbid to set matchLabelKeys when labelSelector is not set in topologySpreadConstraints (#116535, @denkensk)
- GCE does not support LoadBalancer Services with ports with different protocols (TCP and UDP) (#115966, @aojea) [SIG Apps and Cloud Provider]
- GRPC probes are now a GA feature. GRPCContainerProbe feature gate was locked to default value and will be removed in v1.29. If you were setting this feature gate explicitly, please remove it now. (#116233, @SergeyKanzhelev)
- Graduated Kubelet Topology Manager to GA. (#116093, @swatisehgal)
- Graduated KubeletTracing to beta, which means that the feature gate is now enabled by default. (#115750, @saschagrunert)
- Graduated seccomp profile defaulting to GA.

Set the kubelet --seccomp-default flag or seccompDefault kubelet configuration field to true to make pods on that node default to using the RuntimeDefault seccomp profile.

Enabling seccomp for your workload can have a negative performance impact depending on the kernel and container runtime version in use.

Guidance for identifying and mitigating those issues is outlined in the Kubernetes seccomp tutorial. (#115719, @saschagrunert) [SIG API Machinery, Node, Storage and Testing]
- Graduated the container resource metrics feature on HPA to beta. (#116046, @sanposhiho)
- Implemented API streaming for the watch-cache

When sendInitialEvents ListOption is set together with watch=true, it begins the watch stream with synthetic init events followed by a synthetic "Bookmark" after which the server continues streaming events. (#110960, @p0lyn0mial)
- Introduced API for streaming.

Added SendInitialEvents field to the ListOptions. When the new option is set together with watch=true, it begins the watch stream with synthetic init events followed by a synthetic "Bookmark" after which the server continues streaming events. (#115402, @p0lyn0mial)
- Introduced a breaking change to the resource.k8s.io API in its AllocationResult struct. This change allows a kubelet plugin for the DynamicResourceAllocation feature to service allocations from multiple resource driver controllers. (#116332, @klueska)
- Introduces new alpha functionality to the reflector, allowing user to enable API streaming.

To activate this feature, users can set the ENABLE_CLIENT_GO_WATCH_LIST_ALPHA environmental variable.
It is important to note that the server must support streaming for this feature to function properly.
If streaming is not supported by the server, the reflector will revert to the previous method
of obtaining data through LIST/WATCH semantics. (#110772, @p0lyn0mial) [SIG API Machinery]
- K8s.io/client-go/tools/record.EventBroadcaster: after Shutdown() is called, the broadcaster now gives up immediately after a failure to write an event to a sink. Previously it tried multiple times for 12 seconds in a goroutine. (#115514, @pohly) [SIG API Machinery]
- K8s.io/component-base/logs: usage of the pflag values in a normal Go flag set led to panics when printing the help message (#114680, @pohly) [SIG Instrumentation]
- Kubeadm: explicitly set priority for static pods with priorityClassName: system-node-critical (#114338, @champtar) [SIG Cluster Lifecycle]
- Kubelet: a "maxParallelImagePulls" field can now be specified in the kubelet configuration file to control how many image pulls the kubelet can perform in parallel. (#115220, @ruiwen-zhao) [SIG API Machinery, Node and Scalability]
- Kubelet: changed MemoryThrottlingFactor default value to 0.9 and formulas to calculate memory.high (#115371, @pacoxu)
- Kubernetes components that perform leader election now only support using Leases for this. (#114055, @aimuz)
- Migrated the DaemonSet controller (within kube-controller-manager) to use contextual logging (#113622, @249043822)
- New service.kubernetes.io/topology-mode annotation has been introduced as a replacement for the service.kubernetes.io/topology-aware-hints annotation.
- service.kubernetes.io/topology-aware-hints annotation has been deprecated.
- kube-proxy now accepts any value that is not "disabled" for these annotations, enabling custom implementation-specific and/or future built-in heuristics to be used. (#116522, @robscott) [SIG Apps, Network and Testing]
- Pods owned by a Job now uses the labels batch.kubernetes.io/job-name and batch.kubernetes.io/controller-uid.
The legacy labels job-name and controller-uid are still added for compatibility. (#114930, @kannon92)
- Promoted CronJobTimeZone feature to GA (#115904, @soltysh)
- Promoted SelfSubjectReview to Beta (#116274, @nabokihms) [SIG API Machinery, Auth, CLI and Testing]
- Relaxed API validation to allow pod node selector to be mutable for gated pods (additions only, no deletions or mutations). (#116161, @danielvegamyhre)
- Remove kubernetes.io/grpc standard appProtocol (#116866, @LiorLieberman) [SIG API Machinery and Apps]
- Remove deprecated --enable-taint-manager and --pod-eviction-timeout CLI (#115840, @atosatto)
- Removed support for the v1alpha1 kubeletplugin API of DynamicResourceManagement. All plugins must be updated to v1alpha2 in order to function properly. (#116558, @klueska)
- The API server now re-uses data encryption keys while the kms v2 plugin key ID is stable. Data encryption keys are still randomly generated on server start but an atomic counter is used to prevent nonce collisions. (#116155, @enj)
- The PodDisruptionBudget spec.unhealthyPodEvictionPolicy field has graduated to beta and is enabled by default. On servers with the feature enabled, this field may be set to AlwaysAllow to always allow unhealthy pods covered by the PodDisruptionBudget to be evicted. (#115363, @ravisantoshgudimetla) [SIG Apps, Auth and Node]
- The DownwardAPIHugePages kubelet feature graduated to stable / GA. (#115721, @saschagrunert) [SIG Apps and Node]
- The following feature gates for volume expansion GA features have now been removed and must no longer be referenced in --feature-gates flags: ExpandCSIVolumes, ExpandInUsePersistentVolumes, ExpandPersistentVolumes (#113942, @mengjiao-liu)
- The list-type of the alpha resourceClaims field introduced to Pods in 1.26.0 was modified from set to map, resolving an incompatibility with use of this schema in CustomResourceDefinitions and with server-side apply. (#114585, @JoelSpeed)
- Updated API reference for Requests, specifying they must not exceed limits (#115434, @ehashman)
- Updated KMSv2 to beta (#115123, @aramase)
- Updated: Redefine AppProtocol field description and add new standard values (#115433, @LiorLieberman) [SIG API Machinery, Apps and Network]
- /metrics/slis is now available for control plane components allowing you to scrape health check metrics. (#114997, @Richabanker)
- APIServerTracing feature gate is now enabled by default. Tracing in the API
Server is still disabled by default, and requires a config file to enable. (#116144, @dashpole)
- NodeResourceFit and NodeResourcesBalancedAllocation implement the PreScore
extension point for a more performant calculation. (#115655, @tangwz)
- PodSchedulingReadiness is graduated to beta. (#115815, @Huang-Wei)
- PodSpec.Container.Resources became mutable for CPU and memory resource types.
- PodSpec.Container.ResizePolicy (new object) gives users control over how their containers are resized.
- PodStatus.Resize status describes the state of a requested Pod resize.
- PodStatus.ResourcesAllocated describes node resources allocated to Pod.
- PodStatus.Resources describes node resources applied to running containers by CRI.
- UpdateContainerResources CRI API now supports both Linux and Windows. (#102884, @vinaykul)
- SELinuxMountReadWriteOncePod graduated to Beta. (#116425, @jsafrane)
- StatefulSetAutoDeletePVC feature gate promoted to beta. (#116501, @mattcary)
- StatefulSet names must be DNS labels, rather than subdomains. Any StatefulSet
which took advantage of subdomain validation (by having dots in the name) can't
possibly have worked, because we eventually set pod.spec.hostname from the StatefulSetName,
and that is validated as a DNS label. (#114172, @thockin)
- ValidatingAdmissionPolicy now provides a status field that contains results of type checking the validation expression.
The type checking is fully informational, and the behavior of the policy is unchanged. (#115668, @jiahuif)
- cacheSize field in EncryptionConfiguration is not supported for KMSv2 provider (#113121, @aramase)
- k8s.io/component-base/logs now also supports adding command line flags to a flag.FlagSet. (#114731, @pohly)
- kubelet: migrated --container-runtime-endpoint and --image-service-endpoint
to kubelet config (#112136, @pacoxu)
- resource.k8s.io/v1alpha1 was replaced with resource.k8s.io/v1alpha2. Before
upgrading a cluster, all objects in resource.k8s.io/v1alpha1 (ResourceClaim, ResourceClaimTemplate,
ResourceClass, PodScheduling) must be deleted. The changes are internal, so
YAML files which create pods and resource claims don't need changes except for
the newer apiVersion. (#116299, @pohly)
- volumes: resource.claims is now cleared for PVC specs during create or update of a pod spec with inline PVC template or of a PVC because it has no effect. (#115928, @pohly)

Feature

• A new client side metric rest_client_request_retries_total has been added that tracks the number of retries sent to the server, partitioned by status code, verb and host (#108396, @tkashem)

• A new feature was enabled to improve the performance of the iptables mode of kube-proxy in large clusters. No action was required, however:

• If you experienced problems with Services not syncing to iptables correctly, you can disable the feature by passing --feature-gates=MinimizeIPTablesRestore=false to kube-proxy (and file a bug if this fixes it). (This might also be detected by seeing the value of kube-proxy's sync_proxy_rules_iptables_partial_restore_failures_total metric rising.)

• If you were previously overriding the kube-proxy configuration for performance reasons, this may no longer be necessary. See https://kubernetes.io/docs/reference/networking/virtual-ips/#optimizing-iptables-mode-performance. (#115138, @danwinship)
• API validation relaxed allowing Indexed Jobs to be scaled up/down by changing parallelism and completions in tandem, such that parallelism == completions. (#115236, @danielvegamyhre) [SIG Apps and Testing]
• Added "general", "baseline", and "restricted" debugging profiles for kubectl debug. (#114280, @sding3) [SIG CLI]
• Added "netadmin" debugging profiles for kubectl debug. (#115712, @wedaly) [SIG CLI]
• Added --output plaintext-openapiv2 argument to kubectl explain to use old openapiv2 explain implementation. (#115480, @alexzielenski)
• Added NewVolumeManagerReconstruction feature gate and enabled it by default to enable updated discovery of mounted volumes during kubelet startup. Please watch for kubelet getting stuck at startup and / or not unmounting volumes from deleted Pods and report any issues in this area. (#115268, @jsafrane)
• Added kubelet Topology Manager metrics to track admission requests processed and occured admission errors. (#115137, @swatisehgal)
• Added apiserver_envelope_encryption_invalid_key_id_from_status_total to measure number of times an invalid keyID is returned by the Status RPC call. (#115846, @ritazh) [SIG API Machinery and Auth]
• Added apiserver_envelope_encryption_kms_operations_latency_seconds metric to measure the KMSv2 grpc calls latency. (#115649, @aramase) [SIG API Machinery, Auth and Testing]
• Added e2e test to node expand volume with secret (#115451, @zhucan)
• Added e2e tests for kubectl --subresource for beta graduation (#116590, @MadhavJivrajani)
• Added kubelet Topology Manager metric to measure topology manager admission latency. (#115590, @swatisehgal)
• Added logging-format option to CCMs based on k8s.io/cloud-provider (#108984, @LittleFox94)
• Added metrics for volume reconstruction during kubelet startup. (#115965, @jsafrane) [SIG Node and Storage]
• Added new -f flag into debug command to be used passing pod or node files instead explicit names. (#111453, @ardaguclu)
• Added new feature gate ServiceNodePortStaticSubrange, to enable the new strategy in the NodePort Service port allocators, so the node port range is subdivided and dynamic allocated NodePort port for Services are allocated preferentially from the upper range. (#114418, @xuzhenglun)
• Added scheduler preemption support for pods using ReadWriteOncePod PVCs (#114051, @chrishenzie)
• Added the applyconfiguration generator to the code-generator script that generates server-side apply configuration and client APIs (#114987, @astefanutti)
• Added the ability to host webhooks in the cloud controller manager. (#108838, @nckturner)
• Apiserver_storage_transformation_operations_total metric has been updated to include labels transformer_prefix and status. (#115394, @ritazh) [SIG API Machinery, Auth, Instrumentation and Testing]
• By enabling the UserNamespacesStatelessPodsSupport feature gate in kubelet, you can now run a stateless pod in a separate user namespace (#116377, @giuseppe) [SIG Apps, Node and Storage]
• By enabling the alpha CloudNodeIPs feature gate in kubelet and the cloud
  provider, you can now specify a dual-stack --node-ip value (when using an
  external cloud provider that supports that functionality). (#116305, @danwinship) [SIG API Machinery, Cloud Provider, Network and Node]
• Changed kubectl --subresource flag to beta (#116595, @MadhavJivrajani)
• Changed metrics for aggregated discovery to publish new time series (alpha). (#115630, @Jefftree) [SIG API Machinery and Testing]
• Dynamic Resource Allocation framework can be used for network devices (#114364, @bart0sh) [SIG Node]
• Enable external plugins can be used as subcommands for kubectl create command if subcommand does not exist as builtin only when KUBECTL_ENABLE_CMD_SHADOW environment variable is exported. (#116293, @ardaguclu)
• GRPC probes now set a linger option of 1s to improve the TIME-WAIT state. (#115321, @rphillips) [SIG Network and Node]
• Graduated CRI Events driven Pod LifeCycle Event Generator (Evented PLEG) to Beta (#115967, @harche)
• Graduated matchLabelKeys in podTopologySpread to Beta (#116291, @denkensk)
• Graduated the CSINodeExpandSecret feature to Beta. This feature facilitates passing secrets to CSI driver as part of Node Expansion CSI operation. (#115621, @humblec)
• Graduated the LegacyServiceAccountTokenTracking feature gate to Beta. The usage of auto-generated secret-based service account token now produces warnings by default, and relevant Secrets are labeled with a last-used timestamp (label key kubernetes.io/legacy-token-last-used). (#114523, @zshihang) [SIG API Machinery and Auth]
• HPA controller exposes the following metrics from the kube-controller-manager.
• metric_computation_duration_seconds: Number of metric computations.
• metric_computation_total: The time(seconds) that the HPA controller takes to calculate one metric. (#116326, @sanposhiho) [SIG Apps, Autoscaling and Instrumentation]
• HPA controller starts to expose metrics from the kube-controller-manager.\n- reconciliations_total: Number of reconciliation of HPA controller. \n- reconciliation_duration_seconds: The time(seconds) that the HPA controller takes to reconcile once. (#116010, @sanposhiho)
• Kube-up now includes CoreDNS version v1.9.3 (#114279, @pacoxu)
• Kubeadm: added the experimental (alpha) feature gate EtcdLearnerMode that allows etcd members to be joined as learner and only then promoted as voting members (#113318, @pacoxu)
• Kubectl will now display SeccompProfile for pods, containers and ephemeral containers, if values were set. (#113284, @williamyeh)
• Kubectl: added e2e test for default container annotation (#115046, @pacoxu)
• Kubelet TCP and HTTP probes are now more effective using networking resources:
  conntrack entries, sockets. This is achieved by reducing the TIME-WAIT state
  of the connection to 1 second, instead of the defaults 60 seconds. This allows
  kubelet to free the socket, and free conntrack entry and ephemeral port associated. (#115143, @aojea)
• Kubelet allows pods to use the net.ipv4.ip_local_reserved_ports sysctl by default and the minimal kernel version is 3.16; Pod Security admission allows this sysctl in v1.27+ versions of the baseline and restricted policies. (#115374, @pacoxu) [SIG Auth, Network and Node]
• Kubelet config file will be backed up to /etc/kubernetes/tmp/ folder with kubeadm-kubelet-config append with a random suffix as the filename (#114695, @chendave) [SIG Cluster Lifecycle]
• Kubernetes is now built with Go 1.19.5 (#115010, @cpanato)
• Kubernetes is now built with go 1.20 (#114502, @cpanato) [SIG Release and Testing]
• Kubernetes is now built with go 1.20.1 (#115828, @cpanato) [SIG Release and Testing]
• Kubernetes is now built with go 1.20.2 (#116404, @cpanato) [SIG Release and Testing]
• Locked CSIMigrationvSphere feature gate. (#116610, @xing-yang)
• Made apiextensions-apiserver binary linking static (also affects the deb and rpm packages). (#114226, @saschagrunert)
• Made kube-aggregator binary linking static (also affects the deb and rpm packages). (#114227, @saschagrunert)
• Made kubectl-convert binary linking static (also affects the deb and rpm packages). (#114228, @saschagrunert)
• Migrated controller helper functions to use contextual logging. (#115049, @fatsheep9146)
• Migrated the ResourceQuota controller (within kube-controller-manager) to use contextual logging. (#113315, @ncdc) [SIG API Machinery, Apps and Testing]
• Migrated the StatefulSet controller (within kube-controller-manager) to use contextual logging (#113840, @249043822)
• Migrated the ClusterRole aggregation controller (within kube-controller-manager) to use contextual logging. (#113910, @mengjiao-liu)
• Migrated the Deployment controller (within kube-controller-manager) to use contextual logging (#113525, @249043822)
• Migrated the ReplicaSet controller (within kube-controller-manager) to use contextual logging. (#114871, @Namanl2001)
• Migrated the bootstrap signer controller and the token cleaner controller (within kube-controller-manager) to use contextual logging. (#113464, @mengjiao-liu) [SIG API Machinery, Apps and Instrumentation]
• Migrated the defaultbinder scheduler plugin to use contextual logging. (#116571, @mengjiao-liu) [SIG Instrumentation and Scheduling]
• Migrated the main kube-controller-manager binary to use contextual logging. (#116529, @pohly)
• Migrated the namespace controller (within kube-controller-manager) to support contextual logging. (#113443, @yangjunmyfm192085)
• Migrated the service-account controller (within kube-controller-manager) to use contextual logging. (#114918, @Namanl2001) [SIG API Machinery, Apps, Auth, Instrumentation and Testing]
• Migrated the volume attach/detach controller (within kube-controller-manager) to use contextual logging.
  Migrated the PersistentVolumeClaim protection controller (within kube-controller-manager) to use contextual logging.
  Migrated the PersistentVolume protection controller (within kube-controller-manager) to use contextual logging. (#113584, @yangjunmyfm192085)
• Migrated the “TTL after finished” controller (within kube-controller-manager)to use contextual logging. (#113916, @songxiao-wang87)
• NONE (#113428, @mengjiao-liu)
• New plugin_evaluation_total is added to the scheduler.This metric counts how many times the specific plugin affects the scheduling result. The metric does not get incremented when the plugin has nothing to do with an incoming Pod. (#115082, @sanposhiho)
• Node ipam controller now exposes metrics cidrset_cidrs_max_total and multicidrset_cidrs_max_total with information about the max number of CIDRs that can be allocated. (#112260, @aryan9600)
• Performance improvements in klog (#115277, @pohly)
• Pod template schedulingGates are now mutable for Jobs that are suspended and have never been started (#115940, @ahg-g) [SIG Apps]
• Pods which have an invalid negative spec.terminationGracePeriodSeconds value will now be treated as having a terminationGracePeriodSeconds of 1 (#115606, @wzshiming)
• Profiling can now be served on a unix-domain socket by using the --profiling-path option (when profiling is enabled) for security purposes. (#114191, @apelisse) [SIG API Machinery]
• Promote aggregated discovery endpoint to beta and it will be enabled by default (#116108, @Jefftree)
• Promoted OpenAPIV3 to GA (#116235, @Jefftree)
• Promoted whoami kubectl command. (#116510, @nabokihms)
• Scheduler no longer runs the plugin's Filter method when its PreFilter method returned a Skip status.
  In other words, your PreFilter/Filter plugin can return a Skip status in PreFilter if the plugin does nothing in Filter for that Pod.
  Scheduler skips NodeAffinity Filter plugin when NodeAffinity Filter plugin has nothing to do with a Pod.
  It may affect some metrics values related to the NodeAffinity Filter plugin. (#114125, @sanposhiho)
• Scheduler now skips InterPodAffinity Filter plugin when InterPodAffinity Filter plugin has nothing to do with a Pod.
  It may affect some metrics values related to the InterPodAffinity Filter plugin. (#114889, @sanposhiho)
• Scheduler volumebinding: leveraged PreFilterResult to reduce down to only
  eligible node(s) for pod with bound claim(s) to local PersistentVolume(s) (#109877, @yibozhuang)
• Scheduling cycle now terminates immediately when any scheduler plugin returns an
  unschedulableAndUnresolvable status in PostFilter. (#114699, @kerthcet)
• Since Kubernetes v1.5, kubectl apply has had an alpha-stage --prune flag to support deleting previously applied objects that have been removed from the input manifest. This feature has remained in alpha ever since due to performance and correctness issues inherent in its design. This PR exposes a second, independent pruning alpha powered by a new standard named ApplySets. An ApplySet is a server-side object (by default, a Secret; ConfigMaps are also allowed) that kubectl can use to accurately and efficiently track set membership across apply operations. The format used for ApplySet is set out in KEP 3659 as a low-level specification. Other tools in the ecosystem can also build on this specification for improved interoperability. To try the ApplySet-based pruning alpha, set KUBECTL_APPLYSET=true and use the flags --prune --applyset=secret-name with kubectl apply. (#116205, @justinsb)
• Switched kubectl explain to use OpenAPIV3 information published by the server. OpenAPIV2 backend can still be used with the --output plaintext-openapiv2 argument (#116390, @alexzielenski) [SIG API Machinery, CLI and Testing]
• The Pod API field .spec.schedulingGates[*].name now requires qualified names (like example.com/mygate), matching validation for names of .spec.readinessGates[*].name. Any uses of the alpha scheduling gate feature prior to 1.27 that do not match that validation must be renamed or deleted before upgrading to 1.27. (#115821, @lianghao208) [SIG Apps and Scheduling]
• The Scheduler did not run the plugin Score method when its PreScore method returned a Skip status. In other words, the PreScore/Score plugin could return a Skip status in PreScore if the plugin did nothing in Score for that Pod. (#115652, @AxeZhan)
• The AdvancedAuditing feature gate was locked to true in v1.27, and will be removed completely in v1.28 (#115163, @SataQiu) [SIG API Machinery]
• The JobMutableNodeSchedulingDirectives feature gate has graduated to GA. (#116116, @ahg-g) [SIG Apps, Scheduling and Testing]
• The ReadWriteOncePod feature gate has been graduated to beta. (#114494, @chrishenzie)
• The bug which caused the status of Indexed Jobs to only update when new indexes were completed was fixed. Now, completed indexes are updated even if the .status.completedIndexes values are outside the [0, .spec.completions> range. (#115349, @danielvegamyhre)
• The go version defined in .go-version is now fetched when invoking test, build, and code generation targets if the current go version does not match it. Set $FORCE_HOST_GO=y while testing or building to skip this behavior, or set $GO_VERSION to override the selected go version. (#115377, @liggitt) [SIG Testing]
• The job controller back-off logic is now decoupled from workqueue. In case of parallelism > 1, if there are multiple new failures in a reconciliation cycle, all the failures are taken into account to compute the back-off. Previously, the back-off kicked in for all types of failures; with this change, only pod failures are taken into account. If the back-off limits exceeds, the job is marked as failed immediately; before this change, the job is marked as failed in the next back-off. (#114768, @sathyanarays) [SIG Apps and Testing]
• The mount-utils mounter now provides an option to limit the number of concurrent format operations. (#115379, @artemvmin) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node and Storage]
• The scheduler's metric plugin_execution_duration_seconds now records PreEnqueue plugins execution seconds. (#116201, @sanposhiho)
• Two changes to the /debug/api_priority_and_fairness/dump_priority_levels endpoint of API Priority and Fairness: added total number of dispatched, timed-out, rejected and cancelled requests; output now sorted by PriorityLevelName. (#112393, @borgerli)
• Unlocked the CSIMigrationvSphere feature gate.
  The change allow users to continue using the in-tree vSphere driver,pending a vSphere
  CSI driver release that has with GA support for Windows, XFS, and raw block access. (#116342, @msau42) [SIG Storage]
• Updated cAdvisor to v0.47.0 (#114883, @bobbypage)
• Updated kube-apiserver SLO/SLI latency metrics to exclude priority & fairness queue wait times (#116420, @andrewsykim)
• Updated distroless iptables to use released image registry.k8s.io/build-image/distroless-iptables:v0.2.2
• Updated setcap to use released image registry.k8s.io/build-image/setcap:bullseye-v1.4.2 (#116509, @cpanato) [SIG Testing]
• Updated distroless iptables to use released image registry.k8s.io/distroless-iptables:v0.2.1 (#115905, @cpanato) [SIG Testing]
• Upgrades functionality of kubectl kustomize as described at
  https://github.com/kubernetes-sigs/kustomize/releases/tag/kustomize%2Fv5.0.0 and https://github.com/kubernetes-sigs/kustomize/releases/tag/kustomize%2Fv5.0.1.

This is a new major release of kustomize, so there are a few backwards-incompatible changes, most of which are rare use cases, bug fixes with side effects, or things that have been deprecated for multiple releases already:

• https://github.com/kubernetes-sigs/kustomize/pull/4911: Drop support for a very old, legacy style of patches. patches used to be allowed to be used as an alias for patchesStrategicMerge in kustomize v3. You now have to use patchesStrategicMerge explicitly, or update to the new syntax supported by patches. See examples in the PR description of https://github.com/kubernetes-sigs/kustomize/pull/4911.
• https://github.com/kubernetes-sigs/kustomize/issues/4731: Remove a potential build-time side-effect in ConfigMapGenerator and SecretGenerator, which loaded values from the local environment under some circumstances, breaking kustomize build's side-effect-free promise. While this behavior was never intended, we deprecated it and are announcing it as a breaking change since it existed for a long time. See also the Eschewed Features documentation.
• https://github.com/kubernetes-sigs/kustomize/pull/4985: If you previously included .git in an AWS or Azure URL, we will no longer automatically remove that suffix. You may need to add an extra / to replace the .git for the URL to properly resolve.
• https://github.com/kubernetes-sigs/kustomize/pull/4954: Drop support for using gh: as a host (e.g. gh:kubernetes-sigs/kustomize). We were unable to find any usage of or basis for this and believe it may have been targeting a custom gitconfig shorthand syntax. (#116598, @natasha41575) [SIG CLI]
• When an unsupported PodDisruptionBudget configuration is found, an event and log will be emitted to inform users of the misconfiguration. (#115861, @JayKayy) [SIG Apps]
• [E2E] Pods spawned by E2E tests can now pull images from the private registry using the new --e2e-docker-config-file flag (#114625, @Divya063) [SIG Node and Testing]
• [alpha: kubectl apply --prune --applyset] Enabled certain custom resources (CRs) to be used as ApplySet parent objects. To enable this for a given CR, apply the label applyset.kubernetes.io/is-parent-type: true to the CustomResourceDefinition (CRD) that defines it. (#116353, @KnVerey)
• Kubelet no longer creates certain legacy iptables rules by default.
  It is possible that this will cause problems with some third-party components
  that improperly depended on those rules. If this affects you, you can run
  kubelet with --feature-gates=IPTablesOwnershipCleanup=false, but a bug should also be filed against the third-party component. (#114472, @danwinship)
• MinDomainsInPodTopologySpread feature gate is enabled by default as a
  Beta feature in 1.27. (#114445, @mengjiao-liu)
• Secret of kubernetes.io/tls type now verifies that the private key matches the cert (#113581, @aimuz)
• StorageVersionGC (within kube-controller-manager) to use contextual logging. (#113986, @songxiao-wang87)
• client-go: sharedInformerFactory now waits for goroutines during shutdown for metadatainformer and dynamicinformer. (#114434, @howardjohn)
• kube-proxy now accepts the ContextualLogging, LoggingAlphaOptions,
  LoggingBetaOptions (#115233, @pohly)
• kube-scheduler: Optimized implementation of null labelSelector in topology spreading. (#116607, @alculquicondor)
• kubeadm: now shows a warning message when detecting that the sandbox image of the container runtime is inconsistent with that used by kubeadm (#115610, @SataQiu)
• kubectl now uses HorizontalPodAutoscaler v2 by default. (#114886, @a7i)
• Kubernetes is now built with Go 1.20.3 (#117125, @xmudrii) [SIG Release and Testing]
• Updated distroless iptables to use released image registry.k8s.io/build-image/distroless-iptables:v0.2.3 (#117126, @xmudrii) [SIG Testing]

Documentation

• Documented the reason field in CRI API to ensure it equals OOMKilled for the containers terminated by OOM killer (#112977, @mimowo)
• Error message for Pods with requests exceeding limits will have a limit value printed. (#112925, @SergeyKanzhelev)
• The change affects the following CLI command:

kubectl create rolebinding -h (#107124, @ptux) [SIG CLI]

Failing Test

• Deflaked a preemption test that may patch Nodes incorrectly. (#114350, @Huang-Wei)
• Fixed panic in vSphere e2e tests. (#115863, @jsafrane) [SIG Storage and Testing]
• Setting the Kubelet config option --resolv-conf=Host on Windows will now result in Kubelet applying the Pod DNS Policies as intended. (#110566, @claudiubelu)

Bug or Regression

• Added (dry run) and (server dry run) suffixes to kubectl scale command when dry-run is passed (#114252, @ardaguclu)
• Applied configurations can be generated for types with non-builtin map fields (#114920, @astefanutti)
• Changed the error message of kubectl rollout restart when subsequent kubectl rollout restart commands are executed within a second (#113040, @ardaguclu)
• Changed the error message to cannot exec into multiple objects at a time when file passed to kubectl exec contains multiple resources (#114249, @ardaguclu)
• Client-go: fixed potential data races retrying requests using a custom io.Reader body; with this fix, only requests with no body or with string / []byte / runtime.Object bodies can be retried (#113933, @liggitt)
• Describing the CRs will now hide .metadata.managedFields (#114584, @soltysh)
• Discovery document will correctly return the resources for aggregated apiservers that do not implement aggregated disovery (#115770, @Jefftree)
• Excluded preemptor pod metadata in the event message (#114923, @mimowo)
• Expanded the partial fix for https://github.com/kubernetes/kubernetes/issues/111539
  which was already started in https://github.com/kubernetes/kubernetes/pull/109706
  Specifically, we will now reduce the amount of syncs for ETP=local services even
  further in the CCM and avoid re-configuring LBs to an even greater extent. (#111658, @alexanderConstantinescu)
• File content check for IPV4 is now enabled by default, and the check of IPV4 or IPV6 is done for kubeadm init or kubeadm join only in case the user intends to create a cluster to support that kind of IP address family (#115420, @chendave)
• Fixed CSI PersistentVolumes to allow Secrets names longer than 63 characters. (#114776, @jsafrane)
• Fixed Route controller to update routes when NodeIP changes (#108095, @lzhecheng)
• Fixed DaemonSet to update the status even if it fails to create a pod. (#113787, @gjkim42)
• Fixed SELinux label for host path volumes created by host path provisioner (#112021, @mrunalp)
• Fixed StatefulSetAutoDeletePVC feature when OwnerReferencesPermissionEnforcement admission plugin is enabled. (#114116, @jsafrane)
• Fixed a bug on the EndpointSlice mirroring controller that generated multiple slices in some cases for custom endpoints in non canonical format. (#114155, @aojea)
• Fixed a bug that caused the apiserver to panic when trying to allocate a Service with a dynamic ClusterIP and was configured with Service CIDRs with a /28 mask for IPv4 and a /124 mask for IPv6 (#115322, @aojea)
• Fixed a bug where Kubernetes would apply a default StorageClass to a PersistentVolumeClaim,
  even when the deprecated annotation volume.beta.kubernetes.io/storage-class was set. (#116089, @cvvz) [SIG Apps and Storage]
• Fixed a bug where events/v1 Events with similar event type and reporting instance were not aggregated by client-go. (#112365, @dgrisonnet)
• Fixed a bug where when emitting similar Events consecutively, some were rejected by the apiserver. (#114237, @dgrisonnet)
• Fixed a data race when emitting similar Events consecutively (#114236, @dgrisonnet)
• Fixed a log line in scheduler that inaccurately implies that volume binding has finalized (#116018, @TommyStarK)
• Fixed a rare race condition in kube-apiserver that could lead to missing events when a watch API request was created at the same time kube-apiserver was re-initializing its internal watch. (#116172, @wojtek-t)
• Fixed a regression in the pod binding subresource to honor the metadata.uid precondition.
  This allows kube-scheduler to ensure it is assigns node names to the same instances of pods it made scheduling decisions for. (#116550, @alculquicondor)
• Fixed a regression that the scheduler always goes through all Filter plugins. (#114518, @Huang-Wei)
• Fixed an EndpointSlice Controller hashing bug that could cause EndpointSlices to incorrectly handle Pods with duplicate IP addresses. For example this could happen when a new Pod reused an IP that was also assigned to a Pod in a completed state. (#115907, @qinqon) [SIG Apps and Network]
• Fixed an issue where a CSI migrated volume may be prematurely detached when the CSI driver is not running on the node.
  If CSI migration is enabled on the node, even the csi-driver is not up and ready, we will still add this volume to DSW. (#115464, @sunnylovestiramisu)
• Fixed an issue where failed pods associated with a job with parallelism = 1 are recreated by the job controller honoring exponential backoff delay again. However, for jobs with parallelism > 1, pods might be created without exponential backoff delay. (#114516, @nikhita)
• Fixed an issue with Winkernel Proxier - ClusterIP Loadbalancers missing if the ExternalTrafficPolicy is set to Local and the available endpoints are all remoteEndpoints. (#115919, @princepereira)
• Fixed bug in CRD Validation Rules (beta) and ValidatingAdmissionPolicy (alpha) where all admission requests could result in internal error: runtime error: index out of range [3] with length 3 evaluating rule: <rule name> under certain circumstances. (#114857, @jpbetz)
• Fixed bug in beta aggregated discovery endpoint which caused CRD discovery information to be temporarily missing when an Aggregated APIService with the same GroupVersion is deleted (and vice versa). (#116770, @alexzielenski)
• Fixed bug in reflector that couldn't recover from Too large resource version errors with API servers before 1.17.0. (#115093, @xuzhenglun)
• Fixed clearing of rate-limiter for the queue of checks for cleaning stale pod disruption conditions. The bug could result in the PDB synchronization updates firing too often or the pod disruption cleanups taking too long to happen. (#114770, @mimowo)
• Fixed data race in kube-scheduler when preemption races with a Pod update. (#116395, @alculquicondor) [SIG Scheduling]
• Fixed file permission issues that happened during update of Secret/ConfigMap/projected volume when fsGroup is used. The problem caused a race condition where application gets intermittent permission denied error when reading files that were just updated, before the correct permissions were applied. (#114464, @tsaarni)
• Fixed incorrect watch events when a watch is initialized simultanously with a reinitializing watchcache. (#116436, @wojtek-t)
• Fixed issue in Winkernel Proxier - Unexpected active TCP connection drops while horizontally scaling the endpoints for a LoadBalancer Service with Internal Traffic Policy: Local (#113742, @princepereira)
• Fixed issue on Windows when calculating cpu limits on nodes with more than 64 logical processors (#114231, @mweibel)
• Fixed issue with Winkernel Proxier - IPV6 load balancer policies were missing when service was configured with ipFamilyPolicy: RequireDualStack (#115503, @princepereira)
• Fixed issue with Winkernel Proxier - IPV6 load balancer policies were missing when service was configured with ipFamilyPolicy: RequireDualStack (#115577, @princepereira)
• Fixed issue with Winkernel Proxier - No ingress load balancer rules with endpoints to support load balancing when all the endpoints are terminating. (#113776, @princepereira)
• Fixed missing delete events on informer re-lists to ensure all delete events were correctly emitted and using the latest known object state, so that all event handlers and stores always reflect the actual apiserver state as best as possible (#115620, @odinuge)
• Fixed nil pointer error in NodeVolumeLimits csi logging (#115179, @sunnylovestiramisu)
• Fixed panic validating custom resource definition schemas that set multipleOf to 0 (#114869, @liggitt)
• Fixed performance regression in scheduler caused by frequent metric lookup on critical code path. (#116428, @mborsz) [SIG Scheduling]
• Fixed stuck apiserver if an aggregated apiservice returned 304 Not Modified for aggregated discovery information (#114459, @alexzielenski)
• Fixed the problem Pod terminating stuck because of trying to umount not actual mounted dir. (#115769, @mochizuki875)
• Fixed the regression that introduced 34s timeout for DELETECOLLECTION calls (#115341, @tkashem)
• Fixed two regressions introduced by the PodDisruptionConditions feature (on by default in 1.26):
• pod eviction API calls returned spurious precondition errors and required a second evict API call to succeed
• dry-run eviction API calls persisted a DisruptionTarget condition into the pod being evicted (#116554, @atiratree)
• Fixes #115825. Kube-proxy will now include the healthz state in its response to the LB HC as to avoid indicating to the LB that it should use the node in question when Kube-proxy is not healthy. (#111661, @alexanderConstantinescu) [SIG Network]
• Flag --concurrent-node-syncs has been added to cloud node controller which defines how many workers in parallel will be initialising and synchronising nodes. (#113104, @pawbana) [SIG API Machinery, Cloud Provider and Scalability]
• Force deleted pods may fail to terminate until the kubelet is restarted when the container runtime returns an error during termination. We have strengthened testing for runtime failures and now perform a more rigorous reconciliation to ensure static pods (especially those that use fixed UIDs) are restarted. As a side effect of these changes static pods will be restarted with lower latency than before (2s vs 4s, on average) and rapid updates to pod configuration should take effect sooner.

A new metric kubelet_known_pods has been added at ALPHA stability to report the number of pods a Kubelet is tracking in a number of internal states. Operators may use the metrics to track an excess of pods in the orphaned state that may not be completing. (#113145, @smarterclayton) [SIG API Machinery, Auth, Cloud Provider, Node and Testing]
- From now on, the HPA controller will return an error for the container resource metrics when the feature gate HPAContainerMetrics is disabled. As a result, HPA with a container resource metric performs no scale-down and performs only. (#116043, @sanposhiho)
- IPVS: Any ipvs scheduler can now be configured. If a un-usable scheduler is configured kube-proxy will re-start and the logs must be checked (same as before but different log printouts). (#114878, @uablrek)
- If a user attempts to add an ephemeral container to a static pod, they will now get a visible validation error. (#114086, @xmcqueen)
- Ingress with ingressClass annotation and IngressClassName both set can be created now. (#115447, @AxeZhan)
- Kube-apiserver: errors decoding objects in etcd are now recorded in an apiserver_storage_decode_errors_total counter metric (#114376, @baomingwang) [SIG API Machinery and Instrumentation]
- Kube-apiserver: regular expressions specified with the --cors-allowed-origins option are now validated to match the entire hostname inside the Origin header of the request and
must contain '^' or the '//' prefix to anchor to the start, and '$' or the port separator ':' to anchor to
the end. (#112809, @tkashem) [SIG API Machinery]
- Kube-apiserver: removed N^2 behavior loading webhook configurations. (#114794, @lavalamp) [SIG API Machinery, Architecture, CLI, Cloud Provider and Node]
- Kubeadm: fixed an etcd learner-mode bug by preparing an etcd static pod manifest before promoting (#115038, @tobiasgiese)
- Kubeadm: fixed the bug where kubeadm always does CRI detection even if it is not required by a phase subcommand (#114455, @SataQiu)
- Kubeadm: improved retries when updating node information, in case kube-apiserver is temporarily unavailable (#114176, @QuantumEnergyE)
- Kubeadm: modified--configflag from required to optional forkubeadm kubeconfig usercommand ([#116074](https://github.com/kubernetes/kubernetes/pull/116074), [@SataQiu](https://github.com/SataQiu)) - Kubectl: enabled usage of label selector for filtering out resources when pruning for kubectl diff ([#114863](https://github.com/kubernetes/kubernetes/pull/114863), [@danlenar](https://github.com/danlenar)) - Kubelet startup now fails CRI connection if service or image endpoint is throwing any error ([#115102](https://github.com/kubernetes/kubernetes/pull/115102), [@saschagrunert](https://github.com/saschagrunert)) - Kubelet: fix recording issue when pulling image did finish ([#114904](https://github.com/kubernetes/kubernetes/pull/114904), [@TommyStarK](https://github.com/TommyStarK)) [SIG Node] - Kubelet: fixed a bug in kubelet that stopped rendering the ConfigMaps when fsquota monitoring is enabled (#112624, @pacoxu)
- Messages of DisruptionTarget condition now excludes preemptor pod metadata (#114914, @mimowo)
- Optimized LoadBalancer creation with the help of attribute Internal Traffic Policy: Local (#114407, @princepereira)
- PVCs will automatically be recreated if they are missing for a pending Pod. (#113270, @rrangith) [SIG Apps and Testing]
- PersistentVolume API objects which set NodeAffinities using beta Kubernetes labels for OS, architecture, zone, region, and instance type may now be modified to use the stable Kubernetes labels. (#115391, @haoruan)
- Potentially breaking change - Updating the polling interval for Windows stats collection from 1 second to 10 seconds (#116546, @marosset) [SIG Node and Windows]
- Relaxed API validation for usage key encipherment and kubelet uses requested usages accordingly (#111660, @pacoxu)
- Removed scheduler names from preemption event messages. (#114980, @mimowo)
- Shared informers now correctly propagate whether they are synced or not. Individual informer handlers may now check if they are synced or not (new HasSynced method). Library support is added to assist controllers in tracking whether their own work is completed for items in the initial list (AsyncTracker). (#113985, @lavalamp)
- The Kubernetes API server now correctly detects and closes existing TLS connections when its client certificate file for kubelet authentication has been rotated. (#115315, @enj) [SIG API Machinery, Auth, Node and Testing]
- Total test spec is now available by ProgressReporter, it will be reported before test suite got executed. (#114417, @chendave)
- Updated the Event series starting count when emitting isomorphic events from 1 to 2. (#112334, @dgrisonnet)
- When GCing pods, kube-controller-manager will delete Evicted pods first. (#116167, @borgerli)
- When describing deployments, OldReplicaSets now always shows all replicasets controlled the deployment, not just those that still have replicas available. (#113083, @llorllale) [SIG CLI]
- Windows CPU usage node stats are now correctly calculated for nodes with multiple Processor Groups. (#110864, @claudiubelu) [SIG Node, Testing and Windows]
- LabelSelectors specified in topologySpreadConstraints were validated to ensure that pods are scheduled as expected. Existing pods with invalid LabelSelectors could be updated, but new pods were required to specify valid LabelSelectors. (#111802, @maaoBit)
- PodGC for pods which are in terminal phase now do not add the DisruptionTarget condition. (#115056, @mimowo)
- Service of type ExternalName do not create an Endpoint anymore. (#114814, @panslava)
- cacher: If ResourceVersion is unset, the watch is now served from the underlying storage as documented. (#115096, @MadhavJivrajani)
- client-go: fixed the wait time for trying to acquire the leader lease (#114872, @Iceber)
- etcd: Updated to v3.5.7 (#115310, @mzaian)
- golang.org/x/net updated to v0.7.0 to fix CVE-2022-41723 (#115786, @liggitt)
- kube-controller-manager will not run nodeipam controller when allocator type
is CloudAllocator and the cloud provider is not enabled. (#114596, @andrewsykim)
- kube-controller-manager: fixed a bug that the kubeconfig field of kubecontrollermanager.config.k8s.io configuration is not populated correctly (#116219, @SataQiu)
- kube-proxy with --proxy-mode=ipvs can be used with statically linked kernels.
The reseved IPv4 range TEST-NET-2 in rfc5737 MUST NOT be used for ClusterIP or loadBalancerIP since address 198.51.100.0 is used for probing. (#114669, @uablrek)
- kubeadm: fixed a bug where the uploaded kubelet configuration in kube-system/kubelet-config ConfigMap does not respect user patch (#115575, @SataQiu)
- kubeadm: now respects user provided kubeconfig during discovery process (#113998, @SataQiu)
- kubectl port-forward now exits with exit code 1 when remote connection is
lost (#114460, @brianpursley)
- nodeName being set along with non-empty schedulingGates is now enforced. (#115569, @Huang-Wei)
- node_stage_path is now set whenever available for expansion during mount (#115346, @gnufied)
- statefulset status will now be consistent on API errors (#113834, @atiratree)
- tryUnmount now respects mounter.withSafeNotMountedBehavior (#114736, @andyzhangx)
- The encryption response from KMS v2 plugins is now validated earlier at DEK generation time instead of waiting until an encryption is performed. (#116877, @enj) [SIG API Machinery and Auth]
- Recreate DaemonSet pods completed with Succeeded phase (#117073, @mimowo) [SIG Apps and Testing]

Other (Cleanup or Flake)

• Added basic Denial Of Service prevention for the the node-local kubelet podresource API (#116459, @ffromani) [SIG Node and Testing]
• Callers of wait.ExponentialBackoffWithContext now must pass a ConditionWithContextFunc to be consistent with the signature and avoid creating a duplicate context. If your condition does not need a context you can use the ConditionFunc.WithContext() helper to ignore the context, or use ExponentialBackoff directly. (#115113, @smarterclayton)
• Changed docs for --contention-profiling flag to reflect it performed block profiling (#114490, @MadhavJivrajani)
• E2e framework: added --report-complete-ginkgo and --report-complete-junit parameters. They work like ginkgo --json-report <report dir>/ginkgo/report.json --junit-report <report dir>/ginkgo/report.xml. (#115678, @pohly) [SIG Testing]
• Fixed incorrect log information in the iptables utility. (#110723, @yangjunmyfm192085)
• Improved FormatMap: Improves performance by about 4x, or nearly 2x in the worst case (#112661, @aimuz) [SIG Node]
• Improved misleading message, in case of no metrics received for the HPA controlled pods. (#114740, @kushagra98)
• Introduced new metrics removing the redundant subsystem in kube-apiserver pod logs metrics and deprecate the original ones:
• kube_apiserver_pod_logs_pods_logs_backend_tls_failure_total becomes kube_apiserver_pod_logs_backend_tls_failure_total
• kube_apiserver_pod_logs_pods_logs_insecure_backend_total becomes kube_apiserver_pod_logs_insecure_backend_total (#114497, @dgrisonnet)
• Kubeadm: removed the deprecated v1beta2 API. kubeadm 1.26's config migrate
  command can be used to migrate a v1beta2 configuration file to v1beta3 (#114540, @pacoxu)
• Kubelet: remove deprecated flag --container-runtime (#114017, @calvin0327) [SIG Cloud Provider and Node]
• Kubelet: the deprecated --master-service-namespace flag is removed in v1.27 (#116015, @SataQiu)
• Linux/arm will not ship in Kubernetes 1.27 as we are running into issues with building artifacts using golang 1.20.2 (please see issue #116492) (#115742, @dims) [SIG Architecture, Release and Testing]
• Migrated pkg/controller/nodeipam/ipam/cloud_cidr_allocator.go, pkg/controller/nodeipam/ipam/multi_cidr_range_allocator.go pkg/controller/nodeipam/ipam/range_allocator.go pkg/controller/nodelifecycle/node_lifecycle_controller.go to structured logging (#112670, @yangjunmyfm192085)
• Migrated the Kubernetes object garbage collector (within kube-controller-manager) to use contextual logging. (#113471, @ncdc)
• Migrated the ttlafterfinished controller (within kube-controller-manager) to use contextual logging. (#115332, @obaranov1) [SIG Apps]
• Migrated the “sample-controller” controller to use contextual logging. (#113879, @pchan) [SIG API Machinery and Instrumentation]
• Promoted pod resource limit/request metrics to stable. (#115454, @dgrisonnet)
• Removed AWS kubelet credential provider. Please use the external kubelet credential provider binary named ecr-credential-provider instead. (#116329, @dims) [SIG Node, Storage and Testing]
• Removed Azure disk in-tree storage plugin (#116301, @andyzhangx)
• Removed flag master-service-namespace from api-server arguments (#114446, @lengrongfu)
• Removed the following deprecated metrics:
• node_collector_evictions_number replaced by node_collector_evictions_total
• scheduler_e2e_scheduling_duration_seconds replaced by scheduler_scheduling_attempt_duration_seconds (#115209, @dgrisonnet)
• Removed unused rule for nodes/spec from ClusterRole system:kubelet-api-admin (#113267, @hoskeri)
• Renamed API server identity Lease labels to use the key apiserver.kubernetes.io/identity (#114586, @andrewsykim) [SIG API Machinery, Apps, Cloud Provider and Testing]
• Storage.k8s.io/v1beta1 API version of CSIStorageCapacity will no longer be served (#116523, @pacoxu) [SIG API Machinery]
• The CSIMigrationAzureFile feature gate (for the feature which graduated to GA in v1.26) is now unconditionally enabled and will be removed in v1.28. (#114953, @enj)
• The ControllerManagerLeaderMigration feature, GA since 1.24, is now unconditionally enabled and the feature gate option has been removed. (#113534, @pacoxu)
• The WaitFor and WaitForWithContext functions in the wait package have now been marked private. Callers should use the equivalent Poll* method with a zero duration interval. (#115116, @smarterclayton)
• The wait.Poll* and wait.ExponentialBackoff* functions have been deprecated and will be removed in a future release. Callers should switch to using wait.PollUntilContextCancel, wait.PollUntilContextTimeout, or wait.ExponentialBackoffWithContext as appropriate.

PollWithContext(Cancel|Deadline) will no longer return ErrWaitTimeout - use the Interrupted(error) bool helper to replace checks for err == ErrWaitTimeout, or compare specifically to context errors as needed. A future release will make the ErrWaitTimeout error private and callers must use Interrupted() instead. If you are returning ErrWaitTimeout from your own methods, switch to creating a location specific cause err and pass it to the new method wait.ErrorInterrupted(cause) error which will ensure Interrupted() returns true for your loop.

The wait.NewExponentialBackoffManager and wait.NewJitteringBackoffManager functions have been marked as deprecated. Callers should switch to using the Backoff{...}.DelayWithReset(clock, resetInterval) method and must set the Steps field when using Factor. As a short term change, callers may use the Timer() method on the BackoffManager until the backoff managers are deprecated and removed. Please see the godoc of the deprecated functions for examples of how to replace usage of this function. (#107826, @smarterclayton) [SIG API Machinery, Auth, Cloud Provider, Storage and Testing]
- The feature gates CSIInlineVolume, CSIMigration, DaemonSetUpdateSurge, EphemeralContainers, IdentifyPodOS, LocalStorageCapacityIsolation, NetworkPolicyEndPort and StatefulSetMinReadySeconds that graduated to GA in v1.25 and were unconditionally enabled have been removed in v1.27 (#114410, @SataQiu) [SIG Node]
- Upgraded coredns to v1.10.1 (#115603, @pacoxu)
- Upgraded go-jose to v2.6.0 (#115893, @mgoltzsche)
- [KCCM - service controller]: enabled connection draining for terminating pods upon node downscale by the cluster autoscaler. This is done by not reacting to the taint used by the cluster autoscaler to indicate that the node is going away soon, thus keeping the node referenced by the load balancer until the VM has been completely deleted. (#115204, @alexanderConstantinescu)
- apiserver_admission_webhook_admission_duration_seconds buckets have been expanded, 25s is now the largest bucket size to match the webhook default timeout. (#115802, @logicalhan) [SIG API Machinery and Instrumentation]
- wait.ContextForChannel() now implements the context.Context interface and
does not return a cancellation function. (#115140, @smarterclayton)

Dependencies

Added

• github.com/a8m/tree: 10a5fd5
• github.com/dougm/pretty: 2ee9d74
• github.com/rasky/go-xdr: 4930550
• github.com/vmware/vmw-guestinfo: 25eff15
• sigs.k8s.io/kustomize/kustomize/v5: v5.0.1

Changed

• github.com/Microsoft/hcsshim: v0.8.22 → v0.8.25
• github.com/aws/aws-sdk-go: v1.44.116 → v1.35.24
• github.com/coredns/corefile-migration: v1.0.17 → v1.0.20
• github.com/coreos/go-systemd/v22: v22.3.2 → v22.4.0
• github.com/creack/pty: v1.1.11 → v1.1.18
• github.com/docker/docker: v20.10.18+incompatible → v20.10.21+incompatible
• github.com/go-errors/errors: v1.0.1 → v1.4.2
• github.com/go-openapi/jsonpointer: v0.19.5 → v0.19.6
• github.com/go-openapi/jsonreference: v0.20.0 → v0.20.1
• github.com/go-openapi/swag: v0.19.14 → v0.22.3
• github.com/golang-jwt/jwt/v4: v4.2.0 → v4.4.2
• github.com/golang/protobuf: v1.5.2 → v1.5.3
• github.com/google/cadvisor: v0.46.0 → v0.47.1
• github.com/google/cel-go: v0.12.5 → v0.12.6
• github.com/google/uuid: v1.1.2 → v1.3.0
• github.com/kr/pretty: v0.2.1 → v0.3.0
• github.com/mailru/easyjson: v0.7.6 → v0.7.7
• github.com/moby/ipvs: v1.0.1 → v1.1.0
• github.com/moby/term: 39b0c02 → 1aeaba8
• github.com/onsi/ginkgo/v2: v2.4.0 → v2.9.1
• github.com/onsi/gomega: v1.23.0 → v1.27.4
• github.com/opencontainers/runtime-spec: 1c3f411 → 494a5a6
• github.com/rogpeppe/go-internal: v1.3.0 → v1.10.0
• github.com/sirupsen/logrus: v1.8.1 → v1.9.0
• github.com/stretchr/objx: v0.4.0 → v0.5.0
• github.com/stretchr/testify: v1.8.0 → v1.8.1
• github.com/tmc/grpc-websocket-proxy: e5319fd → 673ab2c
• github.com/vishvananda/netns: db3c7e5 → v0.0.2
• github.com/vmware/govmomi: v0.20.3 → v0.30.0
• go.etcd.io/etcd/api/v3: v3.5.5 → v3.5.7
• go.etcd.io/etcd/client/pkg/v3: v3.5.5 → v3.5.7
• go.etcd.io/etcd/client/v2: v2.305.5 → v2.305.7
• go.etcd.io/etcd/client/v3: v3.5.5 → v3.5.7
• go.etcd.io/etcd/pkg/v3: v3.5.5 → v3.5.7
• go.etcd.io/etcd/raft/v3: v3.5.5 → v3.5.7
• go.etcd.io/etcd/server/v3: v3.5.5 → v3.5.7
• go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp: v0.35.0 → v0.35.1
• go.uber.org/goleak: v1.2.0 → v1.2.1
• golang.org/x/mod: v0.6.0 → v0.9.0
• golang.org/x/net: 1e63c2f → v0.8.0
• golang.org/x/sync: 886fb93 → v0.1.0
• golang.org/x/sys: v0.3.0 → v0.6.0
• golang.org/x/term: v0.3.0 → v0.6.0
• golang.org/x/text: v0.5.0 → v0.8.0
• golang.org/x/tools: v0.2.0 → v0.7.0
• golang.org/x/xerrors: 5ec99f8 → 04be3eb
• google.golang.org/grpc: v1.49.0 → v1.51.0
• gopkg.in/check.v1: 8fa4692 → 10cb982
• gopkg.in/square/go-jose.v2: v2.2.2 → v2.6.0
• k8s.io/klog/v2: v2.80.1 → v2.90.1
• k8s.io/kube-openapi: 172d655 → 15aac26
• k8s.io/utils: 1a15be2 → a36077c
• sigs.k8s.io/apiserver-network-proxy/konnectivity-client: v0.0.33 → v0.1.1
• sigs.k8s.io/json: f223a00 → bc3834c
• sigs.k8s.io/kustomize/api: v0.12.1 → v0.13.2
• sigs.k8s.io/kustomize/cmd/config: v0.10.9 → v0.11.1
• sigs.k8s.io/kustomize/kyaml: v0.13.9 → v0.14.1

Removed

• github.com/PuerkitoBio/purell: v1.1.1
• github.com/PuerkitoBio/urlesc: de5bf2a
• github.com/elazarl/goproxy: 947c36d
• github.com/form3tech-oss/jwt-go: v3.2.3+incompatible
• github.com/mattn/go-runewidth: v0.0.7
• github.com/mindprince/gonvml: 9ebdce4
• github.com/niemeyer/pretty: a10e7ca
• github.com/olekukonko/tablewriter: v0.0.4
• sigs.k8s.io/kustomize/kustomize/v4: v4.5.7

v1.27.0-rc.1

Downloads for v1.27.0-rc.1

Source Code

filename | sha512 hash
-------- | -----------
kubernetes.tar.gz | f6a57401347cb6c6329f4334d0f5b9408125784c13cac8c69288e49fac9fcf057ba9b38340e170c9ee24840bfd9c6e63df5706760837e321efc2ce4da795d6cb
kubernetes-src.tar.gz | 62cca03a925930f58083070e3877df2d3de0fc5a2a96ce4079931fab77c77f10cdd739d1ac9f64c16c3dac107f075c6f112d8ed063e3f466d662e55271487e10

Client Binaries

filename | sha512 hash
-------- | -----------
kubernetes-client-darwin-amd64.tar.gz | 3cf168843bd207f0c277465560c900e09487a3ff5eca6877afc5ca947bf164b7fc20f33c5379783ddf55380ca8370a44e400b37216229496a89309242a6f9bbf
kubernetes-client-darwin-arm64.tar.gz | 1b6c7eab5bb7cfe437400049c1d5bb7320828f975c63f3ac07d9e0ef943799f5fc7e0cd283aa6486da5ada075367447963b1346c96ebdb49632535e2f90dd664
kubernetes-client-linux-386.tar.gz | 7d4e84fd2fa4a06890c9a5a8fc2af2e01c1f3516eb8d164ef4c97a554536f845993f096828469e845392d39576baa7ba5fa125dabeb17c049f8556ff07d941e6
kubernetes-client-linux-amd64.tar.gz | f300c80e465f32b3c365e9c036c02aaaa3e713fa9e7318ce9e426406e07be6f3e6166664d9d940e2c3f073c3044f2d677f32a14e87a1e40acd973625634baf59
kubernetes-client-linux-arm.tar.gz | 5152fddf3a01b83c89070403d37c43b46daad2ccf566f484f9f9eaa776b3d022672d902dbef1edd7b8f4241c035208f83d7049c2cf6b76ef1da8d7ffb41d86b3
kubernetes-client-linux-arm64.tar.gz | 955c4019d6e52475bb6cf1df7d3db70ea6dbe226b54e145260f4470c53a0c12128e747ac0fae9843b2ed9d598eb3fbe964df2999a999f89726abd63feadb7ccf
kubernetes-client-linux-ppc64le.tar.gz | feed26863b79c6a67c3ab7842e11663fa4861982666b451ff13d89c7659482ed08c174b2d70517ef542ba17423e5aff172e30806e17e473815c05d5f6d1c431e
kubernetes-client-linux-s390x.tar.gz | 6d3704b1edc07ff244b85e9a387f283524cc9975a527a23c567af0554ebe61f564d10068fe6af5635d71ed2d9e01ca1dc79716177c449493cbbd32903b4c11fd
kubernetes-client-windows-386.tar.gz | 4abff8190b3e843b603fae1b6236815c7f15510735ed25f24ffe2ed31a0e904495431b9314366a042eadd1312a3c2e7d97ab4748ac0b67bdb0d0c3b0322a4b36
kubernetes-client-windows-amd64.tar.gz | f903c4b8397954a11c10463e1fde15c95805fc048988e66c5d59cba5155e244c12cea98e0502365705c3cae138d251a1a87c2bb29da0e1762e84d08ff225ccc4
kubernetes-client-windows-arm64.tar.gz | 4ac1bcba49774c9eba9a562ebc007bbe5963575c9e55917128171b955cc95bc2447352668fe768130b3013fbc425dfc86ed17e6767d59441e3566a456d6356c7

Server Binaries

filename | sha512 hash
-------- | -----------
kubernetes-server-linux-amd64.tar.gz | 3c726e0583497813ba93953e17ec93805d18129401ad3e851e44c74d4a2ecb45c2340bf82b53e28c4e480222e7cf85fa4270506103233fbad7f38ac751b16c2d
kubernetes-server-linux-arm64.tar.gz | 17ef3d22f86328c2faea3df3b476f5b949e0bab2964f878e2b7b61965381ce3dfe43702aea2cfb9ed191e4e3d41ebbe780eccaa57b943d019cabb2adddafe458
kubernetes-server-linux-ppc64le.tar.gz | d35ba5854c853d095f7588372677f1da2a5b3ef5fde846b8cce2925e7ff4fca21b9f3c9afc8eb82a973027a35918887f2d838cf5c3ec29fb2c7af576a2493efb
kubernetes-server-linux-s390x.tar.gz | 79568358644e993126aaaf14a4126de5abae4ba2cbf59f264de57045189b53c1ead4ce8bdc75592164537ed6b836261feab1f2f5aca07f218ba5c2db1c80318c

Node Binaries

filename | sha512 hash
-------- | -----------
kubernetes-node-linux-amd64.tar.gz | dd1feed73c4d0ea7be79338500ae413dc5f55a69da0e2a27cf8e44d14f49a514b262c69b6069d1cd709564fac6e030cef581204b3d48e5208b9967a9d42b69af
kubernetes-node-linux-arm64.tar.gz | 23421b4f6c4b21168302fa22ddf193f9d2db6a582edfded4cbf88dfa646b1f5f65817165f49bf251cb5c32aaf51a92e425cd92051745f890bfd314a82e292573
kubernetes-node-linux-ppc64le.tar.gz | 9149c0bd4765484351e703d074e1554ca0735bed13bc7da3ba511ba8b6e547337f2c2c904e6db8256306b8089018ac66857a212bcb555034487111c07170c5b6
kubernetes-node-linux-s390x.tar.gz | 1fc2f78314fc7781f5296ebe82cce905eb841bb551919261571f549f3d39f53dd8b9546f060f31209e129ff9a4dafbb50ebfc68741525b01d82639ef8a48e12c
kubernetes-node-windows-amd64.tar.gz | 3ab964ee2fa017e96adf606ce8e8540bc3b79323141548247415e5865dd7447dca4146706cf21a1e36512d6f7950ed866351f9e6427e27c20b27118f81bf5d6c

Container Images

All container images are available as manifest lists and support the described
architectures. It is also possible to pull a specific architecture directly by
adding the "-$ARCH" suffix to the container image name.

name | architectures
---- | -------------
registry.k8s.io/conformance:v1.27.0-rc.1 | amd64, arm64, ppc64le, s390x
registry.k8s.io/kube-apiserver:v1.27.0-rc.1 | amd64, arm64, ppc64le, s390x
registry.k8s.io/kube-controller-manager:v1.27.0-rc.1 | amd64, arm64, ppc64le, s390x
registry.k8s.io/kube-proxy:v1.27.0-rc.1 | amd64, arm64, ppc64le, s390x
registry.k8s.io/kube-scheduler:v1.27.0-rc.1 | amd64, arm64, ppc64le, s390x

Changelog since v1.27.0-rc.0

Changes by Kind

Feature

• Kubernetes is now built with Go 1.20.3 (#117125, @xmudrii) [SIG Release and Testing]
• Updated distroless iptables to use released image registry.k8s.io/build-image/distroless-iptables:v0.2.3 (#117126, @xmudrii) [SIG Testing]

Bug or Regression

• Recreate DaemonSet pods completed with Succeeded phase (#117073, @mimowo) [SIG Apps and Testing]
• The encryption response from KMS v2 plugins is now validated earlier at DEK generation time instead of waiting until an encryption is performed. (#116877, @enj) [SIG API Machinery and Auth]

Dependencies

Added

Nothing has changed.

Changed

• github.com/rogpeppe/go-internal: v1.9.0 → v1.10.0

Removed

Nothing has changed.

v1.27.0-rc.0

Downloads for v1.27.0-rc.0

Source Code

filename | sha512 hash
-------- | -----------
kubernetes.tar.gz | 00c1377aacf2540f9dd92538e95e4d676bb77839edf59645dec6be96d4988d64b79f0a2f4a3e604a42a8c06710a88efc86d7c4bdab8d11269aadbbeaa8f02cc0
kubernetes-src.tar.gz | e39d9fe4d1426ad35db1593c75d9bcc1e64178fac46a4c759aeb24cf37e061e1e559ab2fe8d3c4f7f66e813a11aa730aefc06649ba5ff3e8a7ac5b4db79db278

Client Binaries

filename | sha512 hash
-------- | -----------
kubernetes-client-darwin-amd64.tar.gz | 811e5f52ee5f000bbac5ef5f45a7266da96ec56054056e793fab1b39dfcea2c872b6c464d163f9ff445e928750ac7fe04539b38751646c06ec204c10968d3114
kubernetes-client-darwin-arm64.tar.gz | 4a1b6ee903132f23d153369bdf97b40eeb7d111c67f60fa6367909c0bb6dd82ec7ad017ced4cb381bdbe272db514f5e6f12576b8f4b5f384a4cbd544a39268f5
kubernetes-client-linux-386.tar.gz | a422191ad2118c8f1debd73834d5a963d1992441c4c0a917ebfd64818f59038d1971589b9a6f8ba7252949b5bfde62dbfab60da8783502733550c5f65cfae592
kubernetes-client-linux-amd64.tar.gz | aca2b4ff673b381da5a979266395b4eea61c48ba59b6ae81555da21394fd535bb4b408ba584586a1336c02bf0706a90ed53ee9b6d0712519612d60b1c1f7b59c
kubernetes-client-linux-arm.tar.gz | dd5d9d5adc928e114a9a35d1f845fa98b2b236a3937f056adce5c7280f91d7024f013329e3a9a17329f1247c42a1af7c011d37bfb7830937af2cf14babec3dc4
kubernetes-client-linux-arm64.tar.gz | 0dff54bcb39c82a7142dea53ccec384c12637b9b3f261d67338d7ab1508a32897baa29657ffeeaf9bf8e65d11feb25b83349749671c7fb2a38e1a44491f50716
kubernetes-client-linux-ppc64le.tar.gz | 28538ade0567bfd90f3b3975ba4e1e5c986fc2a0e7e02d4cd5f93b22a9f7ec9a0b82226938a0512a1b95149e5d801c16b46c9181aff33cf4b02ec958fefdcd73
kubernetes-client-linux-s390x.tar.gz | 39cbbed42e2e53955e0b76306046f7ad06a88a800902c76584fe9a8a2741349c1cd74ebf4e9d0c7711e5721dd2b028df3f882eeee7242596c747b39f7ddadf87
kubernetes-client-windows-386.tar.gz | 6f5947e9d4760bb00393d9bd8a2a5d389306cb51e0cc46012becaaa2ebbc4f6d0b63d239a7b99f24136ec6ce50ea51032659a68daedd79b8a494acc2cc966e09
kubernetes-client-windows-amd64.tar.gz | 5436de39bcb3edbc7db16e8628e161a0c1fea9cf501ec6969622f3980c2b009f62137f9a60bff77b33751d3188f24812b13bafe4b397e341cd3e8979459f9972
kubernetes-client-windows-arm64.tar.gz | 9390bc41779539e2475e992969a6f41eb68f82082d75abb793a8c9b832b86f4de42be6e313fb963b9e50ec9fd3265d9b8c8b223f7999d823a74062baaf536406

Server Binaries

filename | sha512 hash
-------- | -----------
kubernetes-server-linux-amd64.tar.gz | 652e12dd953d3dbd0fc11c02d145cfe608c41639077a056cc8ecef8d427a73c937d5fc341b8d7e4adb9a68bb6babc3165315106fe554a7ea961f1e1a45cf9566
kubernetes-server-linux-arm64.tar.gz | c2d93520b9a7e6554207477527f408698d1b086c9ee6c6c5716f42e007367b15d1d137c723768d0d8b55cc4f1b32e5f2c57bb05b589c3a2ffe71af9c5626e303
kubernetes-server-linux-ppc64le.tar.gz | f388a0b93b722814e51c0de14dd332fbe4517549a717237fa3d2d75503d5f0db7d73db154e07bc958e40ca4bea11f88312b5c8b334b77d9494637b0da4046b2e
kubernetes-server-linux-s390x.tar.gz | 2ff2af56a4306b807e6792524489c4e2e04b2e9b661b767fe9efa34ede151ff6281b30d2a86332ea9ad69ef3031d96aa072bd099a893ad8db0be81c5c1215d98

Node Binaries

filename | sha512 hash
-------- | -----------
kubernetes-node-linux-amd64.tar.gz | 2fbe286b501a1d84bad54c035fd5daecff6268b21adec003ad90b5c6c813964d00853e8014ea8d4fc9e748586801f3231ad5fa405f213c1fc15b5b0e1819eda8
kubernetes-node-linux-arm64.tar.gz | 1531ac12836e354269f2c0df254593d37f5860352a408c931594735212aa3504fd0ba21bee6ac8df0081d9502d307f093d6932536bd6a48124884e8925a2a76d
kubernetes-node-linux-ppc64le.tar.gz | 87e3599e471a33e922235d6c54eb76c477100c04b185d897896b2942396b609492e851f5537de46b08973732e7518ccdbf5fe88c015b1479bf899e6e8c2549b4
kubernetes-node-linux-s390x.tar.gz | 4c67645f9658809cf4357f94c0a8363a9ef4535ee07e7212565443f5e35a3037d87293a57e4b5fdc7cdca4ef946f8f0e1db09f4db87d747230da21df5f59bd8b
kubernetes-node-windows-amd64.tar.gz | e5bd3034027e38bfe9ffd81f85a3d4c5f5a3e07542d5d718aa4e89d4d4662530316ac7c3b17bbaeb514ecdcc553117688c58899d067b89ffc0e8f3a282e119c2

Container Images

All container images are available as manifest lists and support the described
architectures. It is also possible to pull a specific architecture directly by
adding the "-$ARCH" suffix to the container image name.

name | architectures
---- | -------------
registry.k8s.io/conformance:v1.27.0-rc.0 | amd64, arm64, ppc64le, s390x
registry.k8s.io/kube-apiserver:v1.27.0-rc.0 | amd64, arm64, ppc64le, s390x
registry.k8s.io/kube-controller-manager:v1.27.0-rc.0 | amd64, arm64, ppc64le, s390x
registry.k8s.io/kube-proxy:v1.27.0-rc.0 | amd64, arm64, ppc64le, s390x
registry.k8s.io/kube-scheduler:v1.27.0-rc.0 | amd64, arm64, ppc64le, s390x

Changelog since v1.27.0-beta.0

Changes by Kind

API Change

• Added a new alpha API: ClusterTrustBundle (certificates.k8s.io/v1alpha1).
  A ClusterTrustBundle may be used to distribute X.509 trust anchors to workloads within the cluster. (#113218, @ahmedtd) [SIG API Machinery, Auth and Testing]
• Remove kubernetes.io/grpc standard appProtocol (#116866, @LiorLieberman) [SIG API Machinery and Apps]

Feature

• Give terminal phase correctly to all pods that will not be restarted.

In particular, assign Failed phase to pods which are deleted while pending. Also, assign a terminal
phase (Succeeded or Failed, depending on the exit statuses of the pod containers) to pods which
are deleted while running.

This fixes the issue for jobs using pod failure policy (with JobPodFailurePolicy and PodDisruptionConditions
feature gates enabled) that their pods could get stuck in the pending phase when deleted. (#115331, @mimowo) [SIG Cloud Provider, Node and Testing]

Bug or Regression

• Fixed two regressions introduced by the PodDisruptionConditions feature (on by default in 1.26):
• pod eviction API calls returned spurious precondition errors and required a second evict API call to succeed
• dry-run eviction API calls persisted a DisruptionTarget condition into the pod being evicted (#116554, @atiratree) [SIG API Machinery and Testing]
• Fixes a regression in the pod binding subresource to honor the metadata.uid precondition.
  This allows kube-scheduler to ensure it is assigns node names to the same instances of pods it made scheduling decisions for. (#116550, @alculquicondor) [SIG API Machinery and Testing]
• Fixes bug in beta aggregated discovery endpoint which caused CRD discovery information to be temporarily missing when an Aggregated APIService with the same GroupVersion is deleted (and vice versa). (#116770, @alexzielenski) [SIG API Machinery and Testing]

Dependencies

Added

Nothing has changed.

Changed

Nothing has changed.

Removed

Nothing has changed.

v1.27.0-beta.0

Downloads for v1.27.0-beta.0

Source Code

filename | sha512 hash
-------- | -----------
kubernetes.tar.gz | a648cbc81d762e1b37f673871906ebe7f3b871f0a3c527d0dcfb5d20a9f4eff519354155d6a2cec8deabc2f0e9db8bb4b6ac2215597a11caad396e9d31461944
kubernetes-src.tar.gz | 2cb02e63a58590dc65962f42a6be484b804595adbecb1bcbfaf94186004bb3f9e0000aa8be9e1fb270de89733ea3baa0853211673e8c2f76d6be436782bba5dd

Client Binaries

filename | sha512 hash
-------- | -----------
kubernetes-client-darwin-amd64.tar.gz | 957d1abe4282ae6bba75732b83f858b5c3a61de4148c947862bbc90f0ecf290a3cd94eb267da2127bb2ff28237a50c0b913c261c014e06580a766f69e4b45d5b
kubernetes-client-darwin-arm64.tar.gz | 5827723ec6bc6f0d96cd20046bd736a3045f168cbf78a9064645f0e94653f3e751bcca6d18836aa038cb726ab991a48b1451fcc00bd0e751eb0af30d7bf002aa
kubernetes-client-linux-386.tar.gz | bdacf9b42269238e97b6301a975c4accd7363a05a63a35305d0d74916c138c70985491ac9d13a152d0b10609f265aede4a910ebed61bdf1b8a37264773dffd3b
kubernetes-client-linux-amd64.tar.gz | e139daa8df28d13ad8625c819ba94e6e4dd7805c89dd2a0bba6ce478a2bc7d9b52a3fccc18de08c13dca1b98c693d50d37599e8a3b34b7a1f39401098dea2df5
kubernetes-client-linux-arm.tar.gz | 3f669851c6317d67bbcae591056ee9cfda6e9bca3eeac02cc41eae35db3448e745e123ab75da8b9dbb546172b07d625bf821da3b0a1b6420d41140eb7b96b474
kubernetes-client-linux-arm64.tar.gz | c8f394650db292a117e1db5a76775541087ab0da9b3d43041d50f3126ef47a0dcb65ebbe61d8be9bdb67adce1c43d5f7a695ff0b9909c8c9461d6937ebe9160f
kubernetes-client-linux-ppc64le.tar.gz | 83b51c787f57b698584c3c585a772470819260008808a2102a9e765ef1458d9bb536aeb3e2587d391c6efb06d56326f1c8b47f12ab98069d1605ef210ecd6e8c
kubernetes-client-linux-s390x.tar.gz | d79766f56263a78549d7e2bc8f93977d8730435beeb7fe9413686d09ac6a6edc8a868621023623656782272e518fa7955275ab0d4aecb8a71cb4ba544dd5f77d
kubernetes-client-windows-386.tar.gz | 399741ba92a59c0c3640f4d4d0c961b63bd24ba8a5ce036f4a82dcd040a0d2873e7e3237af10da1b2982af5ca6ae8edb2a4d023db3af87dfae6c90528a487de3
kubernetes-client-windows-amd64.tar.gz | 3eba7adbb6c7c386d04bdddcd6d66ca7f5799789680c7fbb9216a0520884264dc5fdb35a0417d03d77955097c6341a30e3e07d077266c2ed2f96d1765f344e39
kubernetes-client-windows-arm64.tar.gz | 158475196f75764dd115e187a5fb27894367a8a2ddad755e3d542e5f225fe9bad476f592c0b7fad2a3dded4638ccec2a1f717eec4d04c8e510334a3a410e0541

Server Binaries

filename | sha512 hash
-------- | -----------
kubernetes-server-linux-amd64.tar.gz | fb9caa627e77d1bd39b11106dd95c9dd008c5d418234636a0beddd48e59c980d4924ed3006133e20d2ac0715a4353d14a90f7ebc5345804f24160a13efb7a2b5
kubernetes-server-linux-arm64.tar.gz | 34f61cfeba8adf7fd3dd83599e34ed36d5942a41904f0430a7b8a5078d306283a4dd7eec40716c8aa6f4ff87dea1faa588fff66a2c388aac8c7b461a64366c33
kubernetes-server-linux-ppc64le.tar.gz | c37a226fa7b6d35b32420c13e67482820f4b23cd9dc9c23820d8f3024bf969d2acc96dd31267a964a73e3a4a61a046c778ab3443598b111eccbf20a682b93f40
kubernetes-server-linux-s390x.tar.gz | b2f29641f5756bb77b048cd336997e89ae50236fb32a7b425c348fab1f077534facce6c90ad9650dd2db5b708bff1ddabb478e29fc69f32b59e5ded247665840

Node Binaries

filename | sha512 hash
-------- | -----------
kubernetes-node-linux-amd64.tar.gz | 305ee41682bb222e040134e75aefeda6cad1f81f4af761c514bb5d66fe83d42dd993c0a118c178a9e8abd6d2ae3fdb7b70c0509f1134f032c2ef2ef2bc103d81
kubernetes-node-linux-arm64.tar.gz | 6bc84fb35f278742734ac0c6265d6f2d654a7d57d65e98d597ba4c438b7ea20033e0431515f120fbcbf2fb6e99d3f50d4b4ecfc88e3705d08fc949b7f42c3776
kubernetes-node-linux-ppc64le.tar.gz | 638ad423ddbc52179320fe497f775d50c210745044aca9cea00c674dc1e710e979b7fca564811ccae99b801582e075194b09a00548f789740e0e6c4791309bdc
kubernetes-node-linux-s390x.tar.gz | 4874d3e34145c19973aa130c3f2c4eb5b01991142eb9bbf7391378bb6f83179a163659c80b3e45526cf334f7c63868502381afce18205ab92c521f4c911e3179
kubernetes-node-windows-amd64.tar.gz | 5d3e9e88577e5be11d56e65d76cec6ab931811f106fd1683551d9b2514ec8edf21f39c6512adc3ce901862f015b28237fc1774b0ccfaf771f106237a2ed599c6

Container Images

All container images are available as manifest lists and support the described
architectures. It is also possible to pull a specific architecture directly by
adding the "-$ARCH" suffix to the container image name.

name | architectures
---- | -------------
registry.k8s.io/conformance:v1.27.0-beta.0 | amd64, arm64, ppc64le, s390x
registry.k8s.io/kube-apiserver:v1.27.0-beta.0 | amd64, arm64, ppc64le, s390x
registry.k8s.io/kube-controller-manager:v1.27.0-beta.0 | amd64, arm64, ppc64le, s390x
registry.k8s.io/kube-proxy:v1.27.0-beta.0 | amd64, arm64, ppc64le, s390x
registry.k8s.io/kube-scheduler:v1.27.0-beta.0 | amd64, arm64, ppc64le, s390x

Changelog since v1.27.0-alpha.3

Urgent Upgrade Notes

(No, really, you MUST read this before you upgrade)

• The IPv6DualStack feature gate for external cloud providers was removed.
  (The feature became GA in 1.23 and the gate was removed for all other
  components several releases ago.) If you were still manually
  enabling it you must stop now. (#116255, @danwinship) [SIG API Machinery, Cloud Provider and Network]

Changes by Kind

Deprecation

• The alpha SecurityContextDeny admission plugin is deprecated and now requires enabling the alpha SecurityContextDeny feature gate to use. It will be removed in a future version. (#115879, @mtardy) [SIG Auth]

API Change

• API: resource.k8s.io/v1alpha1.PodScheduling was renamed to resource.k8s.io/v1alpha2.PodSchedulingContext. (#116556, @pohly) [SIG API Machinery, Apps, Auth, CLI, Node, Scheduling and Testing]
• APIServerTracing feature gate is now enabled by default. Tracing in the API Server is still disabled by default, and requires a config file to enable. (#116144, @dashpole) [SIG API Machinery and Testing]
• Added CEL runtime cost calculation into ValidatingAdmissionPolicy, matching the evaluation cost
  restrictions that already apply to CustomResourceDefinition.
  If rule evaluation uses more compute than the limit, the API server aborts the evaluation and the
  admission check that was being performed is aborted; the failurePolicy for the ValidatingAdmissionPolicy
  determines the outcome. (#115747, @cici37) [SIG API Machinery]
• Added messageExpression to ValidatingAdmissionPolicy, to set custom failure message via CEL expression. (#116397, @jiahuif) [SIG API Machinery]
• Added a new IPAddress object kind
• Added a new ClusterIP allocator. The new allocator removes previous Service CIDR block size limitations for IPv4, and limits IPv6 size to a /64 (#115075, @aojea) [SIG API Machinery, Apps, Auth, CLI, Cluster Lifecycle, Network and Testing]
• Added a new alpha API: ClusterTrustBundle (certificates.k8s.io/v1alpha1).
  A ClusterTrustBundle may be used to distribute X.509 trust anchors to workloads within the cluster. (#113218, @ahmedtd) [SIG API Machinery, Auth and Testing]
• Added authorization check support to the CEL expressions of ValidatingAdmissionPolicy via a authorizer
  variable with expressions. The new variable provides a builder that allows expressions such authorizer.group('').resource('pods').check('create').allowed(). (#116054, @jpbetz) [SIG API Machinery and Testing]
• Added matchConditions field to ValidatingAdmissionPolicy, enabled support for CEL based custom match criteria. (#116350, @maxsmythe) [SIG API Machinery and Testing]
• Added messageExpression field to ValidationRule. (#115969, @DangerOnTheRanger) (#115969, @DangerOnTheRanger) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Instrumentation, Node and Testing]
• Added the MatchConditions field to ValidatingWebhookConfiguration and MutatingWebhookConfiguration for the v1beta and v1 apis.

The AdmissionWebhookMatchConditions featuregate is now in Alpha (#116261, @ivelichkovich) [SIG API Machinery and Testing]
- Added validation to ensure that if service.kubernetes.io/topology-aware-hints and service.kubernetes.io/topology-mode annotations are both set, they are set to the same value.
- Added deprecation warning if service.kubernetes.io/topology-aware-hints annotation is used. (#116612, @robscott) [SIG Apps, Network and Testing]
- Adds auditAnnotations to ValidatingAdmissionPolicy, enabling CEL to be used to add audit annotations to request audit events.
Adds validationActions to ValidatingAdmissionPolicyBinding, enabling validation failures to be handled by any combination of the warn, audit and deny enforcement actions. (#115973, @jpbetz) [SIG API Machinery and Testing]
- Adds feature gate NodeLogQuery which provides cluster administrators with a streaming view of logs using kubectl without them having to implement a client side reader or logging into the node. (#96120, @LorbusChris) [SIG API Machinery, Apps, CLI, Node, Testing and Windows]
- Api: validation of a PodSpec now rejects invalid ResourceClaim and ResourceClaimTemplate names. For a pod, the name generated for the ResourceClaim when using a template also must be valid. (#116576, @pohly) [SIG Apps]
- Bump default API QPS limits for Kubelet. (#116121, @wojtek-t) [SIG API Machinery and Node]
- Enable the "StatefulSetStartOrdinal" feature gate in beta (#115260, @pwschuurman) [SIG API Machinery and Apps]
- Extended the kubelet's PodResources API to include resources allocated in ResourceClaims via DynamicResourceAllocation. Additionally, added a new Get() method to query a specific pod for its resources. (#115847, @moshe010) [SIG Node]
- Forbid to set matchLabelKeys when labelSelector isn’t set in topologySpreadConstraints (#116535, @denkensk) [SIG API Machinery, Apps and Scheduling]
- GCE does not support LoadBalancer Services with ports with different protocols (TCP and UDP) (#115966, @aojea) [SIG Apps and Cloud Provider]
- GRPC probes are now a GA feature. GRPCContainerProbe feature gate was locked to default value and will be removed in v1.29. If you were setting this feature gate explicitly, please remove it now. (#116233, @SergeyKanzhelev) [SIG API Machinery, Apps and Node]
- Graduate Kubelet Topology Manager to GA. (#116093, @swatisehgal) [SIG API Machinery, Node and Testing]
- Graduate KubeletTracing to beta, which means that the feature gate is now enabled by default. (#115750, @saschagrunert) [SIG Instrumentation and Node]
- Graduate the container resource metrics feature on HPA to beta. (#116046, @sanposhiho) [SIG Autoscaling]
- Introduced a breaking change to the resource.k8s.io API in its AllocationResult struct. This change allows a kubelet plugin for the DynamicResourceAllocation feature to service allocations from multiple resource driver controllers. (#116332, @klueska) [SIG API Machinery, Apps, CLI, Node, Scheduling and Testing]
- Introduces new alpha functionality to the reflector, allowing user to enable API streaming.

To activate this feature, users can set the ENABLE_CLIENT_GO_WATCH_LIST_ALPHA environmental variable.
It is important to note that the server must support streaming for this feature to function properly.
If streaming is not supported by the server, the reflector will revert to the previous method
of obtaining data through LIST/WATCH semantics. (#110772, @p0lyn0mial) [SIG API Machinery]
- Kubelet: change MemoryThrottlingFactor default value to 0.9 and formulas to calculate memory.high (#115371, @pacoxu) [SIG API Machinery, Apps and Node]
- Migrated the DaemonSet controller (within kube-controller-manager) to use [contextual logging](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging) ([#113622](https://github.com/kubernetes/kubernetes/pull/113622), [@249043822](https://github.com/249043822)) [SIG API Machinery, Apps, Instrumentation and Testing] - Newservice.kubernetes.io/topology-modeannotation has been introduced as a replacement for theservice.kubernetes.io/topology-aware-hintsannotation. -service.kubernetes.io/topology-aware-hintsannotation has been deprecated. - kube-proxy now accepts any value that is not "disabled" for these annotations, enabling custom implementation-specific and/or future built-in heuristics to be used. ([#116522](https://github.com/kubernetes/kubernetes/pull/116522), [@robscott](https://github.com/robscott)) [SIG Apps, Network and Testing] - NodeResourceFit and NodeResourcesBalancedAllocation implement the PreScore extension point for a more performant calculation. ([#115655](https://github.com/kubernetes/kubernetes/pull/115655), [@tangwz](https://github.com/tangwz)) [SIG Scheduling] - Pods owned by a Job will now use the labelsbatch.kubernetes.io/job-nameandbatch.kubernetes.io/controller-uid. The legacy labelsjob-nameandcontroller-uidare still added for compatibility. ([#114930](https://github.com/kubernetes/kubernetes/pull/114930), [@kannon92](https://github.com/kannon92)) [SIG Apps] - Promote CronJobTimeZone feature to GA ([#115904](https://github.com/kubernetes/kubernetes/pull/115904), [@soltysh](https://github.com/soltysh)) [SIG API Machinery and Apps] - PromotedSelfSubjectReviewto Beta ([#116274](https://github.com/kubernetes/kubernetes/pull/116274), [@nabokihms](https://github.com/nabokihms)) [SIG API Machinery, Auth, CLI and Testing] - Relax API validation to allow pod node selector to be mutable for gated pods (additions only, no deletions or mutations). ([#116161](https://github.com/kubernetes/kubernetes/pull/116161), [@danielvegamyhre](https://github.com/danielvegamyhre)) [SIG Apps, Scheduling and Testing] - Remove deprecated--enable-taint-managerand--pod-eviction-timeoutCLI flags ([#115840](https://github.com/kubernetes/kubernetes/pull/115840), [@atosatto](https://github.com/atosatto)) [SIG API Machinery, Apps, Node and Testing] - Resource.k8s.io/v1alpha1 was replaced with resource.k8s.io/v1alpha2. Before upgrading a cluster, all objects in resource.k8s.io/v1alpha1 (ResourceClaim, ResourceClaimTemplate, ResourceClass, PodScheduling) must be deleted. The changes will be internal, so YAML files which create pods and resource claims don't need changes except for the newerapiVersion`. (#116299, @pohly) [SIG API Machinery, Apps, CLI, Node, Scheduling and Testing]
- SELinuxMountReadWriteOncePod graduated to Beta. (#116425, @jsafrane) [SIG Storage and Testing]
- StatefulSetAutoDeletePVC feature gate promoted to beta. (#116501, @mattcary) [SIG Apps, Auth and Testing]
- The API server now re-uses data encryption keys while the kms v2 plugin's key ID is stable. Data encryption keys are still randomly generated on server start but an atomic counter is used to prevent nonce collisions. (#116155, @enj) [SIG API Machinery, Auth and Testing]
- The API server's encryption at rest configuration now allows the use of wildcards in the list of resources. For example, '.' can be used to encrypt all resources, including all current and future custom resources. (#115149, @nilekhc) [SIG API Machinery, Auth and Testing]
- Update KMSv2 to beta (#115123, @aramase) [SIG API Machinery, Auth and Testing]
- Updated: Redefine AppProtocol field description and add new standard values (#115433, @LiorLieberman) [SIG API Machinery, Apps and Network]
- ValidatingAdmissionPolicy now provides a status field that contains results of type checking the validation expression.
The type checking is fully informational, and the behavior of the policy is unchanged. (#115668, @jiahuif) [SIG API Machinery, Auth, Cloud Provider and Testing]
- We have removed support for the v1alpha1 kubeletplugin API of DynamicResourceManagement. All plugins must update to v1alpha2 in order to function properly going forward. (#116558, @klueska) [SIG API Machinery, Apps, CLI, Node, Scheduling and Testing]

Feature

• Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

(#113428, @mengjiao-liu) [SIG API Machinery, Apps, Instrumentation and Testing]
- Add e2e test to node expand volume with secret (#115451, @zhucan) [SIG Storage and Testing]
- Added NewVolumeManagerReconstruction feature gate and enable it by default to enable updated discovery of mounted volumes during kubelet startup. Please watch for kubelet getting stuck at startup and / or not unmounting volumes from deleted Pods and report any issues in this area. (#115268, @jsafrane) [SIG Node and Storage]
- Added metrics for volume reconstruction during kubelet startup. (#115965, @jsafrane) [SIG Node and Storage]
- Added the ability to host webhooks in the cloud controller manager. (#108838, @nckturner) [SIG API Machinery, Cloud Provider and Testing]
- Adding e2e tests for kubectl --subresource for beta graduation (#116590, @MadhavJivrajani) [SIG CLI and Testing]
- Adds --output plaintext-openapiv2 argument to kubectl explain to use old openapiv2 explain implementation. (#115480, @alexzielenski) [SIG Architecture, Auth, CLI, Cloud Provider and Node]
- By enabling the UserNamespacesStatelessPodsSupport feature gate in kubelet, you can now run a stateless pod in a separate user namespace (#116377, @giuseppe) [SIG Apps, Node and Storage]
- By enabling the alpha CloudNodeIPs feature gate in kubelet and the cloud
provider, you can now specify a dual-stack --node-ip value (when using an
external cloud provider that supports that functionality). (#116305, @danwinship) [SIG API Machinery, Cloud Provider, Network and Node]
- Change kubectl --subresource flag to beta (#116595, @MadhavJivrajani) [SIG CLI]
- Changed metrics for aggregated discovery to publish new time series (alpha). (#115630, @Jefftree) [SIG API Machinery and Testing]
- Graduate CRI Events driven Pod LifeCycle Event Generator (Evented PLEG) to Beta (#115967, @harche) [SIG Node]
- Graduated matchLabelKeys in podTopologySpread to Beta (#116291, @denkensk) [SIG Scheduling]
- Graduates the CSINodeExpandSecret feature to Beta. This feature facilitates passing secrets to CSI driver as part of Node Expansion CSI operation. (#115621, @humblec) [SIG Storage]
- HPA controller exposes the following metrics from the kube-controller-manager.
- metric_computation_duration_seconds: Number of metric computations.
- metric_computation_total: The time(seconds) that the HPA controller takes to calculate one metric. (#116326, @sanposhiho) [SIG Apps, Autoscaling and Instrumentation]
- HPA controller starts to expose metrics from the kube-controller-manager.
- reconciliations_total: Number of reconciliation of HPA controller.
- reconciliation_duration_seconds: The time(seconds) that the HPA controller takes to reconcile once. (#116010, @sanposhiho) [SIG Apps, Autoscaling and Instrumentation]
- Kube-scheduler: Optimized implementation of null labelSelector in topology spreading. (#116607, @alculquicondor) [SIG Scheduling]
- Kubelet allows pods to use the net.ipv4.ip_local_reserved_ports sysctl by default and the minimal kernel version is 3.16; Pod Security admission allows this sysctl in v1.27+ versions of the baseline and restricted policies. (#115374, @pacoxu) [SIG Auth, Network and Node]
- Kubernetes is now built with go 1.20.2 (#116404, @cpanato) [SIG Release and Testing]
- Locks CSIMigrationvSphere feature gate. (#116610, @xing-yang) [SIG Storage]
- Make apiextensions-apiserver binary linking static (also affects the deb and rpm packages). (#114226, @saschagrunert) [SIG API Machinery and Release]
- Make kube-aggregator binary linking static (also affects the deb and rpm packages). (#114227, @saschagrunert) [SIG API Machinery and Release]
- Migrated controller helper functions to use contextual logging. (#115049, @fatsheep9146) [SIG Apps]
- Migrated the ClusterRole aggregation controller (within kube-controller-manager) to use contextual logging. (#113910, @mengjiao-liu) [SIG API Machinery, Apps and Instrumentation]
- Migrated the Deployment controller (within kube-controller-manager) to use [contextual logging](https://k8s.io/docs/concepts/cluster-administration/system-logs/#contextual-logging) ([#113525](https://github.com/kubernetes/kubernetes/pull/113525), [@249043822](https://github.com/249043822)) [SIG API Machinery, Apps, Instrumentation and Testing] - Migrated the StatefulSet controller (withinkube-controller-manager) to use contextual logging (#113840, @249043822) [SIG API Machinery, Apps, Instrumentation and Testing]
- Migrated the bootstrap signer controller and the token cleaner controller (within kube-controller-manager) to use contextual logging. (#113464, @mengjiao-liu) [SIG API Machinery, Apps and Instrumentation]
- Migrated the defaultbinder scheduler plugin to use contextual logging. (#116571, @mengjiao-liu) [SIG Instrumentation and Scheduling]
- Migrated the main kube-controller-manager binary to use contextual logging. (#116529, @pohly) [SIG API Machinery, Apps, Auth and Node]
- Migrated the replicaset controller (within kube-controller-manager) to use contextual logging. (#114871, @Namanl2001) [SIG API Machinery, Apps, Instrumentation and Testing]
- Migrated the service-account controller (within kube-controller-manager) to use contextual logging. (#114918, @Namanl2001) [SIG API Machinery, Apps, Auth, Instrumentation and Testing]
- Migrated the volume attach/detach controller (within kube-controller-manager) to use contextual logging.
Migrated the PersistentVolumeClaim protection controller (within kube-controller-manager) to use contextual logging.
Migrated the PersistentVolume protection controller (within kube-controller-manager) to use contextual logging. (#113584, @yangjunmyfm192085) [SIG API Machinery, Apps, Instrumentation, Node, Scheduling, Storage and Testing]
- Migrated the “TTL after finished” controller (within kube-controller-manager) to use contextual logging. (#113916, @songxiao-wang87) [SIG API Machinery, Apps, Instrumentation and Testing]
- New "plugin_evaluation_total" is added to the scheduler.
This metric counts how many times the specific plugin affects the scheduling result. The metric doesn't get incremented when the plugin has nothing to do with an incoming Pod. (#115082, @sanposhiho) [SIG Instrumentation and Scheduling]
- Promote whoami kubectl command. (#116510, @nabokihms) [SIG Auth and CLI]
- Promote aggregated discovery endpoint to beta and it will be enabled by default (#116108, @Jefftree) [SIG API Machinery]
- Promoted OpenAPIV3 to GA (#116235, @Jefftree) [SIG API Machinery]
- StorageVersionGC (within kube-controller-manager) to use contextual logging. (#113986, @songxiao-wang87) [SIG API Machinery, Apps and Testing]
- Switched kubectl explain to use OpenAPIV3 information published by the server. OpenAPIV2 backend can still be used with the --output plaintext-openapiv2 argument (#116390, @alexzielenski) [SIG API Machinery, CLI and Testing]
- The job controller back-off logic is now decoupled from workqueue. In case of parallelism > 1, if there are multiple new failures in a reconciliation cycle, all the failures are taken into account to compute the back-off. Previously, the back-off kicked in for all types of failures; with this change, only pod failures are taken into account. If the back-off limits exceeds, the job is marked as failed immediately; before this change, the job is marked as failed in the next back-off. (#114768, @sathyanarays) [SIG Apps and Testing]
- The scheduler's metric "plugin_execution_duration_seconds" now records PreEnqueue plugins execution seconds. (#116201, @sanposhiho) [SIG Scheduling]
- Unlocked the CSIMigrationvSphere feature gate.
The change allow users to continue using the in-tree vSphere driver,pending a vSphere
CSI driver release that has with GA support for Windows, XFS, and raw block access. (#116342, @msau42) [SIG Storage]
- Update kube-apiserver SLO/SLI latency metrics to exclude priority & fairness queue wait times (#116420, @andrewsykim) [SIG API Machinery]
- Updated distroless iptables to use released image registry.k8s.io/build-image/distroless-iptables:v0.2.2
- Updated setcap to use released image registry.k8s.io/build-image/setcap:bullseye-v1.4.2 (#116509, @cpanato) [SIG Testing]
- Upgrades functionality of kubectl kustomize as described at
https://github.com/kubernetes-sigs/kustomize/releases/tag/kustomize%2Fv5.0.0 and https://github.com/kubernetes-sigs/kustomize/releases/tag/kustomize%2Fv5.0.1.

This is a new major release of kustomize, so there are a few backwards-incompatible changes, most of which are rare use cases, bug fixes with side effects, or things that have been deprecated for multiple releases already:

• https://github.com/kubernetes-sigs/kustomize/pull/4911: Drop support for a very old, legacy style of patches. patches used to be allowed to be used as an alias for patchesStrategicMerge in kustomize v3. You now have to use patchesStrategicMerge explicitly, or update to the new syntax supported by patches. See examples in the PR description of https://github.com/kubernetes-sigs/kustomize/pull/4911.
• https://github.com/kubernetes-sigs/kustomize/issues/4731: Remove a potential build-time side-effect in ConfigMapGenerator and SecretGenerator, which loaded values from the local environment under some circumstances, breaking kustomize build's side-effect-free promise. While this behavior was never intended, we deprecated it and are announcing it as a breaking change since it existed for a long time. See also the Eschewed Features documentation.
• https://github.com/kubernetes-sigs/kustomize/pull/4985: If you previously included .git in an AWS or Azure URL, we will no longer automatically remove that suffix. You may need to add an extra / to replace the .git for the URL to properly resolve.
• https://github.com/kubernetes-sigs/kustomize/pull/4954: Drop support for using gh: as a host (e.g. gh:kubernetes-sigs/kustomize). We were unable to find any usage of or basis for this and believe it may have been targeting a custom gitconfig shorthand syntax. (#116598, @natasha41575) [SIG CLI]
• When an unsupported PodDisruptionBudget configuration is found, an event and log will be emitted to inform users of the misconfiguration. (#115861, @JayKayy) [SIG Apps]
• [alpha: kubectl apply --prune --applyset] Enables certain custom resources (CRs) to be used as ApplySet parent objects. To enable this for a given CR, apply the label applyset.k8s.io/is-parent-type: true to the CustomResourceDefinition (CRD) that defines it . (#116353, @KnVerey) [SIG CLI]

Documentation

• The change affects the following CLI command:

kubectl create rolebinding -h (#107124, @ptux) [SIG CLI]

Failing Test

• Setting the Kubelet config option --resolv-conf=Host on Windows will now result in Kubelet applying the Pod DNS Policies as intended. (#110566, @claudiubelu) [SIG Network, Node, Testing and Windows]

Bug or Regression

• Expands the partial fix for https://github.com/kubernetes/kubernetes/issues/111539 which was already started in https://github.com/kubernetes/kubernetes/pull/109706 Specifically, we will now reduce the amount of syncs for ETP=local services even further in the CCM and avoid re-configuring LBs to an even greater extent. (#111658, @alexanderConstantinescu) [SIG Cloud Provider and Network]
• Fix the problem Pod terminating stuck because of trying to umount not actual mounted dir. (#115769, @mochizuki875) [SIG Node and Storage]
• Fixed a rare race condition in kube-apiserver that could lead to missing events when a watch API request was created at the same time kube-apiserver was re-initializing its internal watch. (#116172, @wojtek-t) [SIG API Machinery]
• Fixed data race in kube-scheduler when preemption races with a Pod update. (#116395, @alculquicondor) [SIG Scheduling]
• Fixed incorrect watch events when a watch is initialized simultanously with a reinitializing watchcache. (#116436, @wojtek-t) [SIG API Machinery]
• Fixed performance regression in scheduler caused by frequent metric lookup on critical code path. (#116428, @mborsz) [SIG Scheduling]
• Fixes #115825. Kube-proxy will now include the healthz state in its response to the LB HC as to avoid indicating to the LB that it should use the node in question when Kube-proxy is not healthy. (#111661, @alexanderConstantinescu) [SIG Network]
• Force deleted pods may fail to terminate until the kubelet is restarted when the container runtime returns an error during termination. We have strengthened testing for runtime failures and now perform a more rigorous reconciliation to ensure static pods (especially those that use fixed UIDs) are restarted. As a side effect of these changes static pods will be restarted with lower latency than before (2s vs 4s, on average) and rapid updates to pod configuration should take effect sooner.

A new metric kubelet_known_pods has been added at ALPHA stability to report the number of pods a Kubelet is tracking in a number of internal states. Operators may use the metrics to track an excess of pods in the orphaned state that may not be completing. (#113145, @smarterclayton) [SIG API Machinery, Auth, Cloud Provider, Node and Testing]
- From now on, the HPA controller will return an error for the container resource metrics when the feature gate "HPAContainerMetrics" is disabled. As a result, HPA with a container resource metric performs no scale-down and performs only scale-up based on other metrics. (#116043, @sanposhiho) [SIG API Machinery, Apps and Autoscaling]
- Ingress with ingressClass annotation and IngressClassName both set can be created now. (#115447, @kidddddddddddddddddddddd) [SIG Network]
- Kube-controller-manager: fix a bug that the "kubeconfig" field of "kubecontrollermanager.config.k8s.io" configuration is not populated correctly (#116219, @SataQiu) [SIG API Machinery and Cloud Provider]
- Kubelet: fix recording issue when pulling image did finish (#114904, @TommyStarK) [SIG Node]
- PVCs will automatically be recreated if they are missing for a pending Pod. (#113270, @rrangith) [SIG Apps and Testing]
- PersistentVolume API objects which set NodeAffinities using beta Kubernetes labels for OS, architecture, zone, region, and instance type may now be modified to use the stable Kubernetes labels. (#115391, @haoruan) [SIG Apps and Storage]
- Potentially breaking change - Updating the polling interval for Windows stats collection from 1 second to 10 seconds (#116546, @marosset) [SIG Node and Windows]
- Update the Event series starting count when emitting isomorphic events from 1 to 2. (#112334, @dgrisonnet) [SIG API Machinery and Testing]
- When GCing pods, kube-controller-manager will delete Evicted pods first. (#116167, @borgerli) [SIG Apps]
- Windows CPU usage node stats are now correctly calculated for nodes with multiple Processor Groups. (#110864, @claudiubelu) [SIG Node, Testing and Windows]

Other (Cleanup or Flake)

• Added basic Denial Of Service prevention for the the node-local kubelet podresource API (#116459, @ffromani) [SIG Node and Testing]
• Introduce new metrics removing the redundant subsystem in kube-apiserver pod logs metrics and deprecate the original ones:
• kube_apiserver_pod_logs_pods_logs_backend_tls_failure_total becomes kube_apiserver_pod_logs_backend_tls_failure_total
• kube_apiserver_pod_logs_pods_logs_insecure_backend_total becomes kube_apiserver_pod_logs_insecure_backend_total (#114497, @dgrisonnet) [SIG API Machinery]
• Kubelet: remove deprecated flag --container-runtime (#114017, @calvin0327) [SIG Cloud Provider and Node]
• Kubelet: the deprecated --master-service-namespace flag is removed in v1.27 (#116015, @SataQiu) [SIG Node]
• Linux/arm will not ship in Kubernetes 1.27 as we are running into issues with building artifacts using golang 1.20.2 (please see issue #116492) (#115742, @dims) [SIG Architecture, Release and Testing]
• Migrate pkg/controller/nodeipam/ipam/cloud_cidr_allocator.go, pkg/controller/nodeipam/ipam/multi_cidr_range_allocator.go pkg/controller/nodeipam/ipam/range_allocator.go pkg/controller/nodelifecycle/node_lifecycle_controller.go to structured logging (#112670, @yangjunmyfm192085) [SIG API Machinery, Apps, Architecture, Cloud Provider, Instrumentation, Network and Testing]
• Migrated the Kubernetes object garbage collector (within kube-controller-manager) to use contextual logging. (#113471, @ncdc) [SIG API Machinery, Apps and Testing]
• Migrated the ttlafterfinished controller (within kube-controller-manager) to use contextual logging. (#115332, @obaranov1) [SIG Apps]
• Migrated the “sample-controller” controller to use contextual logging. (#113879, @pchan) [SIG API Machinery and Instrumentation]
• Remove Azure disk in-tree storage plugin (#116301, @andyzhangx) [SIG API Machinery, Cloud Provider, Node, Scheduling, Storage and Testing]
• Remove the following deprecated metrics:
• node_collector_evictions_number replaced by node_collector_evictions_total
• scheduler_e2e_scheduling_duration_seconds replaced by scheduler_scheduling_attempt_duration_seconds (#115209, @dgrisonnet) [SIG Apps and Scheduling]
• Removed AWS kubelet credential provider. Please use the external kubelet credential provider binary named ecr-credential-provider instead. (#116329, @dims) [SIG Node, Storage and Testing]
• Storage.k8s.io/v1beta1 API version of CSIStorageCapacity will no longer be served (#116523, @pacoxu) [SIG API Machinery]
• The wait.Poll* and wait.ExponentialBackoff* functions have been deprecated and will be removed in a future release. Callers should switch to using wait.PollUntilContextCancel, wait.PollUntilContextTimeout, or wait.ExponentialBackoffWithContext as appropriate.

PollWithContext(Cancel|Deadline) will no longer return ErrWaitTimeout - use the Interrupted(error) bool helper to replace checks for err == ErrWaitTimeout, or compare specifically to context errors as needed. A future release will make the ErrWaitTimeout error private and callers must use Interrupted() instead. If you are returning ErrWaitTimeout from your own methods, switch to creating a location specific cause err and pass it to the new method wait.ErrorInterrupted(cause) error which will ensure Interrupted() returns true for your loop.

The wait.NewExponentialBackoffManager and wait.NewJitteringBackoffManager functions have been marked as deprecated. Callers should switch to using the Backoff{...}.DelayWithReset(clock, resetInterval) method and must set the Steps field when using Factor. As a short term change, callers may use the Timer() method on the BackoffManager until the backoff managers are deprecated and removed. Please see the godoc of the deprecated functions for examples of how to replace usage of this function. (#107826, @smarterclayton) [SIG API Machinery, Auth, Cloud Provider, Storage and Testing]
- Upgrade coredns to v1.10.1 (#115603, @pacoxu) [SIG Cloud Provider and Cluster Lifecycle]
- [KCCM - service controller]: enable connection draining for terminating pods upon node downscale by the cluster autoscaler. This is done by not reacting to the taint used by the cluster autoscaler to indicate that the node is going away soon, thus keeping the node referenced by the load balancer until the VM has been completely deleted. (#115204, @alexanderConstantinescu) [SIG API Machinery, Cloud Provider, Instrumentation and Network]

Dependencies

Added

• sigs.k8s.io/kustomize/kustomize/v5: v5.0.1

Changed

• github.com/aws/aws-sdk-go: v1.44.147 → v1.35.24
• github.com/coreos/go-systemd/v22: v22.3.2 → v22.4.0
• github.com/go-errors/errors: v1.0.1 → v1.4.2
• github.com/golang/protobuf: v1.5.2 → v1.5.3
• github.com/onsi/ginkgo/v2: v2.7.0 → v2.9.1
• github.com/onsi/gomega: v1.26.0 → v1.27.4
• golang.org/x/mod: v0.7.0 → v0.9.0
• golang.org/x/net: v0.7.0 → v0.8.0
• golang.org/x/sys: v0.5.0 → v0.6.0
• golang.org/x/term: v0.5.0 → v0.6.0
• golang.org/x/text: v0.7.0 → v0.8.0
• golang.org/x/tools: v0.4.0 → v0.7.0
• k8s.io/kube-openapi: 1cb3ae2 → 15aac26
• sigs.k8s.io/json: f223a00 → bc3834c
• sigs.k8s.io/kustomize/api: v0.12.1 → v0.13.2
• sigs.k8s.io/kustomize/cmd/config: v0.10.9 → v0.11.1
• sigs.k8s.io/kustomize/kyaml: v0.13.9 → v0.14.1

Removed

• github.com/PuerkitoBio/purell: v1.1.1
• github.com/PuerkitoBio/urlesc: de5bf2a
• github.com/mattn/go-runewidth: v0.0.7
• github.com/niemeyer/pretty: a10e7ca
• github.com/olekukonko/tablewriter: v0.0.4
• sigs.k8s.io/kustomize/kustomize/v4: v4.5.7

v1.27.0-alpha.3

Downloads for v1.27.0-alpha.3

Source Code

filename | sha512 hash
-------- | -----------
kubernetes.tar.gz | 86bdc8dfcb5ce47ef6f57917a9deed3dbd800669411e494f565bcb0fc6caf2982026d995205cac614e608be2cf240804668fc9f90579bef0872ee5e5ef33f4d8
kubernetes-src.tar.gz | c8395e5693aa148b0b326477b78d1067ff4368f34c755c003938fd88a777ae2303b102d2da240e762cf40cf171cc7a70746a4ddee4c6a35a16bd0eb9265877af

Client Binaries

filename | sha512 hash
-------- | -----------
kubernetes-client-darwin-amd64.tar.gz | 677ee66b49d137335a16ec3acc7fef33bcf1cee094edc0370487ac8060d728ed009c92b3438dbeb675c113320991ea9bf703579dbaf929883a82874222534760
kubernetes-client-darwin-arm64.tar.gz | d1dbdc0d3f9ad5772aa33276a097f0848f445072c4a3cad1902c3fa015952aa0b78656d144a792beccbfe1bf7b78c0257b4507edcda00fb960fd2249b2cab5fd
kubernetes-client-linux-386.tar.gz | 8f85dfe2f157921b6b3250c744549c2ed4ccd491279b16eb0eb45402ad8f72e73a368eeaa46a4cc309fc5079c333b6f810d5904f87bb282b07b1d0abb0e22586
kubernetes-client-linux-amd64.tar.gz | a058c582c63da65fd9a4b95ea80e67367d0a98e5181c8733f6fe1023c356a85c87594c1ade300eb00f3ef3ef6a32f13c44eb8394f862f512217b4d437b6dbb63
kubernetes-client-linux-arm.tar.gz | 403e8083abc8ee509e636d0b58a013b35c7b36f10fb5e75cccbefbd6bc9a2760dfbd1ad40845b7dc0f5d78d6c93d4a6527e119a4e5f74f5dfa71281bda6a64bb
kubernetes-client-linux-arm64.tar.gz | a4defcf0bb8684cfe49352016abf1c720c36cc2cec8a599f987b164669ce67f581b23134a4194f4a2fb58ac489fd5b7c99ad5539efffa5a7ac5aaa2a31c79f65
kubernetes-client-linux-ppc64le.tar.gz | fdf33f56239537311b1839f22de6db9a60f515bdcfadd5e470f5c459ab1a1d1fcdd67e27abdd4d6fcc5b356a8188794e7605d9633dd9364a4cccc01bce027357
kubernetes-client-linux-s390x.tar.gz | 8475beee121129cb7bf68763ba2cba816f9ad0daec055b564074d9364674b254dc5b368f5acf648ae32b00f0f925b7d8c0403c9ca15754fabf635c272e26a64a
kubernetes-client-windows-386.tar.gz | 7a03a2918722fb94e4a2cee827f6abf13184479274e9c246573cc5515bd9eb4cae46c62988bf9072ae0572086d4689986b78ee5349fcefa60a62e20f17c28d45
kubernetes-client-windows-amd64.tar.gz | 52e8d2da2ec5e3f51f79a2f946c49f57ab0c22321d721914a91656100a721f7278baf419d6aa629c9e8247be60177cc8cb077499b692159a6496506b56e8e17a
kubernetes-client-windows-arm64.tar.gz | efa2d76d57b6e3d9eb897a6e8d9812d67d4d57b9368754e22efeaa4a031d34e1ba4e8db0272b5f2ae9ee6d5615ab078201bc35fa4762f1e32c6b67c8fb7b6c8d

Server Binaries

filename | sha512 hash
-------- | -----------
kubernetes-server-linux-amd64.tar.gz | a1602f5be2abcbc67c763533c7352ac067fe9f2e778b3b2eeb72ee39c885e299edb53f01889420199111cd53465b463290097b8a4f8879c91b912ffbf1aa7dde
kubernetes-server-linux-arm.tar.gz | e129be174e855f48a5de6cf07731c9897edb360c9df1f842915323a8abf2bf704be1c275df04c93a878edbb51ea3316eeb37779ba3ebf3dc19887c97c4b5bc23
kubernetes-server-linux-arm64.tar.gz | f890256c2e5b4096ee944c095df2ff5fd38056ed09deabcbe7501f6b9b5665267e775d1db081f94735dba4e9423f0bebca4b275a2b38389119513bbe2d2b53c2
kubernetes-server-linux-ppc64le.tar.gz | 878ebfa012b184c93505f62e30ada071420bd199ef22d4690c28dff6cbc48de6048978d41d7a6f518d26564261c9da4b7fefce8f0da1e5d274ac871696bb7b93
kubernetes-server-linux-s390x.tar.gz | 217bcec3012f3aefa612178e9842779825cf8d50dc7e1d6a84239206f70976d939c1089d177851f3657a1f7f763ebe6a31b3ff3d99c1120105036cf37650900c

Node Binaries

filename | sha512 hash
-------- | -----------
kubernetes-node-linux-amd64.tar.gz | c15b9a152ceed976aff0b64c2d601985db4e58785b1a756fac6e4c83be27c70e96a277692448d75fccc69b82a5d2a872fa2ab3b5fe4b4145eacbab3d0fa03086
kubernetes-node-linux-arm.tar.gz | 6c0f7d3b97532aacd7aaba931bf4fb858cf49559bf51ea0e154efc4ee26de8d01f7f39f017462e136afe5faaa8d86ee49e954c3a9051751ce8dfa8312cd95f06
kubernetes-node-linux-arm64.tar.gz | fb2934282018539abb5ba0fde67225d5131d08c5798ed54dc93081739a803210c9aaf73412c56af7aa9999b497aac073be42ec3a7677255e5d4e39783330bd86
kubernetes-node-linux-ppc64le.tar.gz | 84078c93f7661c5aedf276964cc5866e1eba9b8cbc9e1c40b9e5bfe76ad115a632f4f591b177dc5fa926cb65a66c6847fb243b82ad1f880796d1d7cfe061b498
kubernetes-node-linux-s390x.tar.gz | dff5a35a352e5a17abad8eaa51b9447ca0a8a52df4b22e325ac7acf70efe3d0fe9bef958e8bb5a1e90d43c4586551d456e2dcf42cd01074605c07d69a52fc5b7
kubernetes-node-windows-amd64.tar.gz | a2da87f8a7bd25d6efc8c3b7af79e268809e743ce36c187cdc0af3ff423d54f95a5bd03b50494aea8b0c394ca716e5c4225e7e2d49f40f08db6cd302d894eb3b

Container Images

All container images are available as manifest lists and support the described
architectures. It is also possible to pull a specific architecture directly by
adding the "-$ARCH" suffix to the container image name.

name | architectures
---- | -------------
registry.k8s.io/conformance:v1.27.0-alpha.3 | amd64, arm, arm64, ppc64le, s390x
registry.k8s.io/kube-apiserver:v1.27.0-alpha.3 | amd64, arm, arm64, ppc64le, s390x
registry.k8s.io/kube-controller-manager:v1.27.0-alpha.3 | amd64, arm, arm64, ppc64le, s390x
registry.k8s.io/kube-proxy:v1.27.0-alpha.3 | amd64, arm, arm64, ppc64le, s390x
registry.k8s.io/kube-scheduler:v1.27.0-alpha.3 | amd64, arm, arm64, ppc64le, s390x

Changelog since v1.27.0-alpha.2

Changes by Kind

Deprecation

• Added a warning response when handling requests that set the deprecated spec.externalID field for a Node. (#115944, @SataQiu) [SIG Node]

API Change

• Graduated seccomp profile defaulting to GA.

Set the kubelet --seccomp-default flag or seccompDefault kubelet configuration field to true to make pods on that node default to using the RuntimeDefault seccomp profile.

Enabling seccomp for your workload can have a negative performance impact depending on the kernel and container runtime version in use.

Guidance for identifying and mitigating those issues is outlined in the Kubernetes seccomp tutorial. (#115719, @saschagrunert) [SIG API Machinery, Node, Storage and Testing]
- Implements API for streaming for the watch-cache

When sendInitialEvents ListOption is set together with watch=true, it begins the watch stream with synthetic init events followed by a synthetic "Bookmark" after which the server continues streaming events. (#110960, @p0lyn0mial) [SIG API Machinery]
- Introduce API for streaming.

Add SendInitialEvents field to the ListOptions. When the new option is set together with watch=true, it begins the watch stream with synthetic init events followed by a synthetic "Bookmark" after which the server continues streaming events. (#115402, @p0lyn0mial) [SIG API Machinery]
- Kubelet: a "maxParallelImagePulls" field can now be specified in the kubelet configuration file to control how many image pulls the kubelet can perform in parallel. (#115220, @ruiwen-zhao) [SIG API Machinery, Node and Scalability]
- PodSchedulingReadiness is graduated to beta. (#115815, @Huang-Wei) [SIG API Machinery, Apps, Scheduling and Testing]
- In-place resize feature for Kubernetes Pods
- Changed the Pod API so that the resources defined for containers are mutable for cpu and memory resource types.
- Added resizePolicy for containers in a pod to allow users control over how their containers are resized.
- Added allocatedResources field to container status in pod status that describes the node resources allocated to a pod.
- Added resources field to container status that reports actual resources applied to running containers.
- Added resize field to pod status that describes the state of a requested pod resize.

For details, see KEPs below. (#102884, @vinaykul) [SIG API Machinery, Apps, Instrumentation, Node, Scheduling and Testing]
- The PodDisruptionBudget spec.unhealthyPodEvictionPolicy field has graduated to beta and is enabled by default. On servers with the feature enabled, this field may be set to AlwaysAllow to always allow unhealthy pods covered by the PodDisruptionBudget to be evicted. (#115363, @ravisantoshgudimetla) [SIG Apps, Auth and Node]
- The DownwardAPIHugePages kubelet feature graduated to stable / GA. (#115721, @saschagrunert) [SIG Apps and Node]
- Volumes: resource.claims gets cleared for PVC specs during create or update of a pod spec with inline PVC template or of a PVC because it has no effect. (#115928, @pohly) [SIG API Machinery, Apps and Storage]

Feature

• API validation relaxed allowing Indexed Jobs to be scaled up/down by changing parallelism and completions in tandem, such that parallelism == completions. (#115236, @danielvegamyhre) [SIG Apps and Testing]
• Add kubelet Topology Manager metric to measure topology manager admission latency. (#115590, @swatisehgal) [SIG Node and Testing]
• Added "netadmin" debugging profiles for kubectl debug. (#115712, @wedaly) [SIG CLI]
• Added apiserver_envelope_encryption_invalid_key_id_from_status_total to measure number of times an invalid keyID is returned by the Status RPC call. (#115846, @ritazh) [SIG API Machinery and Auth]
• Apiserver_storage_transformation_operations_total metric has been updated to include labels transformer_prefix and status. (#115394, @ritazh) [SIG API Machinery, Auth, Instrumentation and Testing]
• Client-go: metadatainformer and dynamicinformer SharedInformerFactorys now supports waiting for goroutines during shutdown (#114434, @howardjohn) [SIG API Machinery]
• Graduate the ReadWriteOncePod feature gate to beta (#114494, @chrishenzie) [SIG Scheduling, Storage and Testing]
• Kubeadm: show a warning message when detecting that the sandbox image of the container runtime is inconsistent with that used by kubeadm (#115610, @SataQiu) [SIG Cluster Lifecycle]
• Kubernetes is now built with go 1.20.1 (#115828, @cpanato) [SIG Release and Testing]
• Performance improvements in klog (#115277, @pohly) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Node, Storage and Testing]
• Pod template schedulingGates are now mutable for Jobs that are suspended and have never been started (#115940, @ahg-g) [SIG Apps]
• Pods which have an invalid negative spec.terminationGracePeriodSeconds value will be treated as having terminationGracePeriodSeconds of 1 (#115606, @wzshiming) [SIG Apps, Node and Testing]
• The Pod API field .spec.schedulingGates[*].name now requires qualified names (like example.com/mygate), matching validation for names of .spec.readinessGates[*].name. Any uses of the alpha scheduling gate feature prior to 1.27 that do not match that validation must be renamed or deleted before upgrading to 1.27. (#115821, @lianghao208) [SIG Apps and Scheduling]
• The JobMutableNodeSchedulingDirectives feature gate has graduated to GA. (#116116, @ahg-g) [SIG Apps, Scheduling and Testing]
• Two changes to the /debug/api_priority_and_fairness/dump_priority_levels endpoint of API Priority and Fairness: add total number of dispatched, timed-out, rejected and cancelled requests; output now sorted by PriorityLevelName. (#112393, @borgerli) [SIG API Machinery]
• Updated distroless iptables to use released image registry.k8s.io/distroless-iptables:v0.2.1 (#115905, @cpanato) [SIG Testing]
• [E2E] Pods spawned by E2E tests can now pull images from the private registry using the new --e2e-docker-config-file flag (#114625, @Divya063) [SIG Node and Testing]

Documentation

• Document the reason field in CRI API to ensure it equals OOMKilled for the containers terminated by OOM killer (#112977, @mimowo) [SIG Node]

Failing Test

• Fixed panic in vSphere e2e tests. (#115863, @jsafrane) [SIG Storage and Testing]

Bug or Regression

• Cacher: If RV is unset, the watch is now served from the underlying storage as documented. (#115096, @MadhavJivrajani) [SIG API Machinery]
• Client-go: fix the wait time for trying to acquire the leader lease (#114872, @Iceber) [SIG API Machinery]
• File content check for IPV4 is not enabled by default, and the check of IPV4 or IPV6 is done for kubeadm init or kubeadm join only in case the user intends to create a cluster to support that kind of IP address family (#115420, @chendave) [SIG Cluster Lifecycle and Network]
• Fix log line in scheduler that inaccurately implies that volume binding has finalized (#116018, @TommyStarK) [SIG Scheduling and Storage]
• Fix missing delete events on informer re-lists to ensure all delete events are correctly emitted and using the latest known object state, so that all event handlers and stores always reflect the actual apiserver state as best as possible (#115620, @odinuge) [SIG API Machinery]
• Fixed a bug where Kubernetes would apply a default StorageClass to a PersistentVolumeClaim,
  even when the deprecated annotation volume.beta.kubernetes.io/storage-class was set. (#116089, @cvvz) [SIG Apps and Storage]
• Fixed an EndpointSlice Controller hashing bug that could cause EndpointSlices to incorrectly handle Pods with duplicate IP addresses. For example this could happen when a new Pod reused an IP that was also assigned to a Pod in a completed state. (#115907, @qinqon) [SIG Apps and Network]
• Fixing issue with Winkernel Proxier - ClusterIP Loadbalancers are missing if the ExternalTrafficPolicy is set to Local and the available endpoints are all remoteEndpoints. (#115919, @princepereira) [SIG Network and Windows]
• Golang.org/x/net updates to v0.7.0 to fix CVE-2022-41723 (#115786, @liggitt) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Node and Storage]
• Kubeadm: fix a bug where the uploaded kubelet configuration in kube-system/kubelet-config ConfigMap does not respect user patch (#115575, @SataQiu) [SIG Cluster Lifecycle]
• Kubeadm: modify '--config' flag from required to optional for 'kubeadm kubeconfig user' command (#116074, @SataQiu) [SIG Cluster Lifecycle]
• Yes, discovery document will correctly return the resources for aggregated apiservers that do not implement aggregated disovery (#115770, @Jefftree) [SIG API Machinery]

Other (Cleanup or Flake)

• Improved FormatMap: Improves performance by about 4x, or nearly 2x in the worst case (#112661, @aimuz) [SIG Node]
• Upgrade go-jose to v2.6.0 (#115893, @mgoltzsche) [SIG API Machinery, Auth, Cluster Lifecycle and Testing]
• apiserver_admission_webhook_admission_duration_seconds buckets have been expanded, 25s is now the largest bucket size to match the webhook default timeout. (#115802, @logicalhan) [SIG API Machinery and Instrumentation]

Dependencies

Added

Nothing has changed.

Changed

• github.com/coredns/corefile-migration: v1.0.18 → v1.0.20
• github.com/golang-jwt/jwt/v4: v4.2.0 → v4.4.2
• go.etcd.io/etcd/api/v3: v3.5.5 → v3.5.7
• go.etcd.io/etcd/client/pkg/v3: v3.5.5 → v3.5.7
• go.etcd.io/etcd/client/v2: v2.305.5 → v2.305.7
• go.etcd.io/etcd/client/v3: v3.5.5 → v3.5.7
• go.etcd.io/etcd/pkg/v3: v3.5.5 → v3.5.7
• go.etcd.io/etcd/raft/v3: v3.5.5 → v3.5.7
• go.etcd.io/etcd/server/v3: v3.5.5 → v3.5.7
• go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp: v0.35.0 → v0.35.1
• golang.org/x/net: v0.5.0 → v0.7.0
• golang.org/x/sys: v0.4.0 → v0.5.0
• golang.org/x/term: v0.4.0 → v0.5.0
• golang.org/x/text: v0.6.0 → v0.7.0
• gopkg.in/square/go-jose.v2: v2.2.2 → v2.6.0
• k8s.io/klog/v2: v2.80.1 → v2.90.1

Removed

• github.com/form3tech-oss/jwt-go: v3.2.3+incompatible

v1.27.0-alpha.2

Downloads for v1.27.0-alpha.2

Source Code

filename | sha512 hash
-------- | -----------
kubernetes.tar.gz | 5420d881db6412c1c1e55044aea61f310ef42d7809d4d90113b2a80ae0d1446f3e7988a8205100c476a313182a0c8b2d1605ad3000eee3b45fec4034d17f2ac2
kubernetes-src.tar.gz | 3b9693bd03ed7f5aee3257a167e431b9de4c576f8843f1441f81cbcdfc6be607c84ca703bd2e7ca4bb5f3b9dee9fbb8645cdf49c1921d796e1a3f027c8f23162

Client Binaries

filename | sha512 hash
-------- | -----------
kubernetes-client-darwin-amd64.tar.gz | ce9875156a7c80452dc3303177b0a137cfc6ae398b66a32b1436768ab77771b000287dba0702510da239c056a697e624416f6126a6205c3c65e78ff6d7d4635b
kubernetes-client-darwin-arm64.tar.gz | bc0791295f926f285f18163bef7faf893162918d75e8de0aa46704d2ac665bbff641a7332d5a3d112d93dd5e14087f8e7333e39b4cc44ab71330e059b0abe4bb
kubernetes-client-linux-386.tar.gz | 76abdb1dbb8886c554628ba634449c42f0c61eb47e168d1cb7bb1eabb0354b37474738879955805165e52a4cbbe39c57cabe63a2b8dbbaed00e14a0da5a7419f
kubernetes-client-linux-amd64.tar.gz | 5ced29d3f8411f34ba9dbd115aab7e45d542b34f7feebba6bfe8dcd394abd9fe127daa6c36460c2d8b35ca6386729a3e644b23b5631fe4d81ae3ae0cf1297e67
kubernetes-client-linux-arm.tar.gz | fff2ce7f24f9fa6c5240d69d81a0c363742b17963265ce744e3366d05d149bce005cfe96fb1dd20b7c5faceed481225da0715dee8b2743ee3ff21391c742a1a0
kubernetes-client-linux-arm64.tar.gz | 54da883e07f1a6e6bb9ca29ca4b5bedb2d24485cd07c8ba03da90b063a07e01271d0ad3b58d20fc3370a40486134b7b6144ad2d18049d7e3a38600ad14d84f8f
kubernetes-client-linux-ppc64le.tar.gz | 50be6728b20612ea3d422e3346150c4cece1cd42356446cf8fd2f9164a40ac997188d840536dc45deab5acf12143233d36b76d9ef12165bb0884024f1725f28c
kubernetes-client-linux-s390x.tar.gz | fa76f8655266fb9b64c450185c175f6854ac1d569140f6d62c383d829111091b79a2f266929ca10f642a989e9ada066988845666621fe13c75cbcfa971f5aa0d
kubernetes-client-windows-386.tar.gz | 67c36d790cd5de91e0241cee3800fcdb49db3f3a9e91e087937686367149d7b06c489e62919ef6fdcb8ec29974ead4a64bab0eb3278f404188cf8fffe89baba9
kubernetes-client-windows-amd64.tar.gz | 49e73490b58576237627cc8015f84eda36aa3af02b8e80b251b07d294fe161e90815ee4149f3b8605fad7c43b278f7b0f631ae3d51fa344ad326abcd480d781d
kubernetes-client-windows-arm64.tar.gz | 27ac0573663d5e45585b205c84cb0e5a7f16282654e30445ad4115a148c20b548d0c3520e45327a4443768c83b68c96e8754cdfb34f17fa0ebf875e4eec2eb48

Server Binaries

filename | sha512 hash
-------- | -----------
kubernetes-server-linux-amd64.tar.gz | 0a2fa2de60af23f722a27479ee0721551561a6bf947ec66c9548b0d14410745de2db3f69c6536768768ffeec4f6afe3af3bd336aeccef67391c4cdaca4a427a6
kubernetes-server-linux-arm.tar.gz | c5ab1da7a7e19acebdba7107c27954e522d33c245ea04556347b601e2cc0f40595b9ca5159661b134a090d5505a76d967da3161b3a409b2a4d9c0f36e1b4d7b1
kubernetes-server-linux-arm64.tar.gz | 3a5cad9ae0f4a1086a238e2fb44f59733361d9dea206390c73825daf25dc8b333fce166f5c5e6c0e1ca3be80b303afb1b6b6c8e9dc13666446c2a70b5b7bc1cb
kubernetes-server-linux-ppc64le.tar.gz | 411be709cde53aa27bca30e7d5ab7523f4dea192c85a1aa810985b23a41cdd00c6969e9b9614a193618be94d346900c2f8e9211c95927a12398142268db4ce5e
kubernetes-server-linux-s390x.tar.gz | 0ac65f78a5cad3506649ca40912f328b2af747ae6367f0ca16ba741af20aeecd73c27ea216921ab7a28da8a61b58f0760e9211c65953e70304fec4b940a39440

Node Binaries

filename | sha512 hash
-------- | -----------
kubernetes-node-linux-amd64.tar.gz | 3a0559b2305136a15cd43104ce728f7651b4fcde13db69f565d66e117ad7f8f30a017d3ea6be92811e4ab880273033c089688675559912bfb6d2aa2c92d60225
kubernetes-node-linux-arm.tar.gz | ac7597cfab9eb93dd9c0f1cd088dd08d120991bc94a718fa89ddd9b8fa12a9f6b9987eaaee66b8aafbb055c836c289ca7ca415b57f61bd8f9159045025026100
kubernetes-node-linux-arm64.tar.gz | 395d65f26b4f482cd1d8be49b846ce80f536ca825ae8ce25d10fe746d95e4297c31512247d22caefe632d2236a33616e2650ed385811ced24c3e6338a5eda36d
kubernetes-node-linux-ppc64le.tar.gz | 7f92e5fdfba981ac80b71fdc00e84b4eb661604861f5602e5fa489f13a10ac699e6e5795cc3654ff95e1b5f7fd51df7773bd5ae511ace9b861a87b6fb1465cc7
kubernetes-node-linux-s390x.tar.gz | 75ec78e900a4df4819c899893fe98fe32b6fa8ae000318dcfed8972d356cc1c5e0a3875885681375c080b0770c377164c03c100a6c45b7d025363e174a00af00
kubernetes-node-windows-amd64.tar.gz | d7630730d547414bdb2b245e1b444a5949cecead751d6c243db72e8f20782ba85e1c06dfab499c13e1b529a74a5d4acef4a9b1a6ca571faf41d3253b1bf74773

Container Images

All container images are available as manifest lists and support the described
architectures. It is also possible to pull a specific architecture directly by
adding the "-$ARCH" suffix to the container image name.

name | architectures
---- | -------------
registry.k8s.io/conformance:v1.27.0-alpha.2 | amd64, arm, arm64, ppc64le, s390x
registry.k8s.io/kube-apiserver:v1.27.0-alpha.2 | amd64, arm, arm64, ppc64le, s390x
registry.k8s.io/kube-controller-manager:v1.27.0-alpha.2 | amd64, arm, arm64, ppc64le, s390x
registry.k8s.io/kube-proxy:v1.27.0-alpha.2 | amd64, arm, arm64, ppc64le, s390x
registry.k8s.io/kube-scheduler:v1.27.0-alpha.2 | amd64, arm, arm64, ppc64le, s390x

Changelog since v1.27.0-alpha.1

Changes by Kind

API Change

• A fix in the resource.k8s.io/v1alpha1/ResourceClaim API avoids harmless (?) ".status.reservedFor: element 0: associative list without keys has an element that's a map type" errors in the apiserver. Validation now rejects the incorrect reuse of the same UID in different entries. (#115354, @pohly) [SIG API Machinery]
• CacheSize field in EncryptionConfiguration is not supported for KMSv2 provider (#113121, @aramase) [SIG API Machinery, Auth and Testing]
• K8s.io/client-go/tools/record.EventBroadcaster: after Shutdown() is called, the broadcaster now gives up immediately after a failure to write an event to a sink. Previously it tried multiple times for 12 seconds in a goroutine. (#115514, @pohly) [SIG API Machinery]
• K8s.io/component-base/logs now also supports adding command line flags to a flag.FlagSet. (#114731, @pohly) [SIG Architecture]
• Update API reference for Requests, specifying they must not exceed limits (#115434, @ehashman) [SIG Architecture, Docs and Node]
• /metrics/slis is made available for control plane components allowing you to scrape health check metrics. (#114997, @Richabanker) [SIG API Machinery, Apps, Architecture, Auth, Autoscaling, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node, Release, Scheduling, Storage and Testing]

Feature

• A new client side metric rest_client_request_retries_total has been added that tracks
  the number of retries sent to the server, partitioned by status code, verb, and host (#108396, @tkashem) [SIG API Machinery, Architecture and Instrumentation]

• A new feature has been enabled to improve the performance of the iptables mode of kube-proxy in large clusters. You do not need to take any action, however:

• If you experience problems with Services not syncing to iptables correctly, you can disable the feature by passing --feature-gates=MinimizeIPTablesRestore=false to kube-proxy (and file a bug if this fixes it). (This might also be detected by seeing the value of kube-proxy's sync_proxy_rules_iptables_partial_restore_failures_total metric rising.)

• If you were previously overriding the kube-proxy configuration for performance reasons, this may no longer be necessary. See https://kubernetes.io/docs/reference/networking/virtual-ips/#optimizing-iptables-mode-performance. (#115138, @danwinship) [SIG Network]
• Add kubelet Topology Manager metrics to track admission requests processed by it and occured admission errors. (#115137, @swatisehgal) [SIG Node and Testing]
• Add logging-format option to CCMs based on k8s.io/cloud-provider (#108984, @LittleFox94) [SIG Cloud Provider and Instrumentation]
• Add new -f flag into debug command to be used passing pod or node files instead explicit names. (#111453, @ardaguclu) [SIG CLI and Testing]
• Added "general", "baseline", and "restricted" debugging profiles for kubectl debug. (#114280, @sding3) [SIG CLI]
• Added apiserver_envelope_encryption_kms_operations_latency_seconds metric to measure the KMSv2 grpc calls latency. (#115649, @aramase) [SIG API Machinery, Auth and Testing]
• Adds scheduler preemption support for pods using ReadWriteOncePod PVCs (#114051, @chrishenzie) [SIG Scheduling, Storage and Testing]
• Adds the applyconfiguration generator to the code-generator script that generates server-side apply configuration and client APIs (#114987, @astefanutti) [SIG API Machinery]
• Dynamic Resource Allocation framework can be used for network devices (#114364, @bart0sh) [SIG Node]
• Fixed bug which caused the status of Indexed Jobs to only be updated when there are newly completed indexes. The completed indexes are now updated if the .status.completedIndexes has values outside of the [0, .spec.completions> range (#115349, @danielvegamyhre) [SIG Apps]
• GRPC probes now set a linger option of 1s to improve the TIME-WAIT state. (#115321, @rphillips) [SIG Network and Node]
• Kubelet config file will be backed up to /etc/kubernetes/tmp/ folder with kubeadm-kubelet-config append with a random suffix as the filename (#114695, @chendave) [SIG Cluster Lifecycle]
• Kubelet no longer creates certain legacy iptables rules by default.
  It is possible that this will cause problems with some third-party components
  that improperly depended on those rules. If this affects you, you can run
  kubelet with --feature-gates=IPTablesOwnershipCleanup=false, but you should
  also file a bug against the third-party component. (#114472, @danwinship) [SIG Network]
• Kubernetes is now built with go 1.20 (#114502, @cpanato) [SIG Release and Testing]
• Migrated the ResourceQuota controller (within kube-controller-manager) to use contextual logging. (#113315, @ncdc) [SIG API Machinery, Apps and Testing]
• New feature gate, ServiceNodePortStaticSubrange, to enable the new strategy in the NodePort Service port allocators, so the node port range is subdivided and dynamic allocated NodePort port for Services are allocated preferentially from the upper range. (#114418, @xuzhenglun) [SIG Network]
• Scheduler doesn't run plugin's Score method when its PreScore method returned a Skip status. In other words, your PreScore/Score plugin can return a Skip status in PreScore if the plugin does nothing in Score for that Pod. (#115652, @kidddddddddddddddddddddd) [SIG Scheduling]
• The go version defined in .go-version is now fetched when invoking test, build, and code generation targets if the current go version does not match it. Set $FORCE_HOST_GO=y while testing or building to skip this behavior, or set $GO_VERSION to override the selected go version. (#115377, @liggitt) [SIG Testing]
• The mount-utils mounter now provides an option to limit the number of concurrent format operations. (#115379, @artemvmin) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node and Storage]

Bug or Regression

• Apply configurations can be generated for types with non-builtin map fields (#114920, @astefanutti) [SIG API Machinery]
• Enforce nodeName cannot be set along with non-empty schedulingGates (#115569, @Huang-Wei) [SIG Apps and Scheduling]
• Etcd: Update to v3.5.7 (#115310, @mzaian) [SIG API Machinery, Cloud Provider, Cluster Lifecycle and Testing]
• Fix a bug that caused to panic the apiserver when trying to allocate a Service with a dynamic ClusterIP and it has been configured with Service CIDRs with a /28 mask for IPv4 and a /124 mask for IPv6 (#115322, @aojea) [SIG Testing]
• Fix an issue where a CSI migrated volume may be prematurely detached when the CSI driver is not running on the node.
  If CSI migration is enabled on the node, even the csi-driver is not up and ready, we will still add this volume to DSW. (#115464, @sunnylovestiramisu) [SIG Apps and Storage]
• Fix nil pointer error in nodevolumelimits csi logging (#115179, @sunnylovestiramisu) [SIG Scheduling]
• Fix the regression that introduced 34s timeout for DELETECOLLECTION calls (#115341, @tkashem) [SIG API Machinery]
• Fixing issue with Winkernel Proxier - IPV6 load balancer policies are missing when service is configured with ipFamilyPolicy: RequireDualStack (#115503, @princepereira) [SIG Network and Windows]
• Fixing issue with Winkernel Proxier - IPV6 load balancer policies are missing when service is configured with ipFamilyPolicy: RequireDualStack (#115577, @princepereira) [SIG Network and Windows]
• Flag workerCount has been added to cloud node controller which defines how many workers will be synchronizing nodes. (#113104, @pawbana) [SIG API Machinery, Cloud Provider and Scalability]
• Kube-apiserver: errors decoding objects in etcd are now recorded in an apiserver_storage_decode_errors_total counter metric (#114376, @baomingwang) [SIG API Machinery and Instrumentation]
• Kube-apiserver: regular expressions specified with the --cors-allowed-origins option are now validated to match the entire hostname inside the Origin header of the request and
  must contain '^' or the '//' prefix to anchor to the start, and '$' or the port separator ':' to anchor to
  the end. (#112809, @tkashem) [SIG API Machinery]
• Kubeadm: fix an etcd learner-mode bug by preparing an etcd static pod manifest before promoting (#115038, @tobiasgiese) [SIG Cluster Lifecycle]
• Kubelet: fix a bug of stoping rendering configmap when enabling fsquota monitoring (#112624, @pacoxu) [SIG Node and Storage]
• Set device stage path whenever available for expansion during mount (#115346, @gnufied) [SIG Storage and Testing]
• The Kubernetes API server now correctly detects and closes existing TLS connections when its client certificate file for kubelet authentication has been rotated. (#115315, @enj) [SIG API Machinery, Auth, Node and Testing]

Other (Cleanup or Flake)

• Changes docs for --contention-profiling flag to reflect it performs block profiling (#114490, @MadhavJivrajani) [SIG API Machinery, Cloud Provider, Docs, Node and Scheduling]
• E2e framework: added --report-complete-ginkgo and --report-complete-junit parameters. They work like ginkgo --json-report <report dir>/ginkgo/report.json --junit-report <report dir>/ginkgo/report.xml. (#115678, @pohly) [SIG Testing]
• Promote pod resource limit/request metrics to stable. (#115454, @dgrisonnet) [SIG Instrumentation and Scheduling]
• The ControllerManagerLeaderMigration feature, GA since 1.24, is unconditionally enabled and the feature gate option has been removed. (#113534, @pacoxu) [SIG API Machinery and Cloud Provider]

Dependencies

Added

Nothing has changed.

Changed

• github.com/onsi/gomega: v1.24.2 → v1.26.0
• go.uber.org/goleak: v1.2.0 → v1.2.1
• golang.org/x/net: v0.4.0 → v0.5.0
• golang.org/x/sys: v0.3.0 → v0.4.0
• golang.org/x/term: v0.3.0 → v0.4.0
• golang.org/x/text: v0.5.0 → v0.6.0
• k8s.io/kube-openapi: 3758b55 → 1cb3ae2
• k8s.io/utils: 1a15be2 → a36077c

Removed

Nothing has changed.