Kubernetes - v1.30.0-beta.0

Changelog since v1.30.0-alpha.3

Changes by Kind

API Change

• A new (alpha) field, trafficDistribution, has been added to the Service spec.
  This field provides a way to express preferences for how traffic is distributed to the endpoints for a Service.
  It can be enabled through the ServiceTrafficDistribution feature gate. (#123487, @gauravkghildiyal) [SIG API Machinery, Apps and Network]
• Add alpha-level support for the SuccessPolicy in Jobs (#123412, @tenzen-y) [SIG API Machinery, Apps and Testing]
• Added (alpha) support for the managedBy field on Jobs. Jobs with a custom value of this field - any
  value other than kubernetes.io/job-controller - are skipped by the job controller, and their
  reconciliation is delegated to an external controller, indicated by the value of the field. Jobs that
  don't have this field at all, or where the field value is the reserved string kubernetes.io/job-controller,
  are reconciled by the built-in job controller. (#123273, @mimowo) [SIG API Machinery, Apps and Testing]
• Added a alpha feature, behind the RelaxedEnvironmentVariableValidation feature gate.
  When that gate is enabled, Kubernetes allows almost all printable ASCII characters to be used in the names
  of environment variables for containers in Pods. (#123385, @HirazawaUi) [SIG Apps, Node and Testing]
• Added alpha support for field selectors on custom resources.
  Provided that the CustomResourceFieldSelectors feature gate is enabled, the CustomResourceDefinition
  API now lets you specify selectableFields. Listing a field there allows filtering custom resources for that
  CustomResourceDefinition in list or watch requests. (#122717, @jpbetz) [SIG API Machinery]
• Added support for configuring multiple JWT authenticators in Structured Authentication Configuration. The maximum allowed JWT authenticators in the authentication configuration is 64. (#123431, @aramase) [SIG Auth and Testing]
• Aggregated discovery supports both v2beta1 and v2 types and feature is promoted to GA (#122882, @Jefftree) [SIG API Machinery and Testing]
• Allowing container runtimes to fix an image garbage collection bug by adding an image_id field to the CRI Container message. (#123508, @saschagrunert) [SIG Node]
• AppArmor profiles can now be configured through fields on the PodSecurityContext and container SecurityContext.
• The beta AppArmor annotations are deprecated.
• AppArmor status is no longer included in the node ready condition (#123435, @tallclair) [SIG API Machinery, Apps, Auth, Node and Testing]
• Conflicting issuers between JWT authenticators and service account config are now detected and fail on API server startup. Previously such a config would run but would be inconsistently effective depending on the credential. (#123561, @enj) [SIG API Machinery and Auth]
• Dynamic Resource Allocation: DRA drivers may now use "structured parameters" to let the scheduler handle claim allocation. (#123516, @pohly) [SIG API Machinery, Apps, Auth, CLI, Cluster Lifecycle, Instrumentation, Node, Release, Scheduling, Storage and Testing]
• Graduated pod scheduling gates to general availability.
  The PodSchedulingReadiness feature gate no longer has any effect, and the
  .spec.schedulingGates field is always available within the Pod and PodTemplate APIs. (#123575, @Huang-Wei) [SIG API Machinery, Apps, Node, Scheduling and Testing]
• Graduated support for minDomains in pod topology spread constraints, to general availability.
  The MinDomainsInPodTopologySpread feature gate no longer has any effect, and the field is
  always available within the Pod and PodTemplate APIs. (#123481, @sanposhiho) [SIG API Machinery, Apps, Scheduling and Testing]
• JWT authenticator config set via the --authentication-config flag is now dynamically reloaded as the file changes on disk. (#123525, @enj) [SIG API Machinery, Auth and Testing]
• Kube-apiserver: the AuthenticationConfiguration type accepted in --authentication-config files has been promoted to apiserver.config.k8s.io/v1beta1. (#123696, @aramase) [SIG API Machinery, Auth and Testing]
• Kube-apiserver: the AuthorizationConfiguration type accepted in --authorization-config files has been promoted to apiserver.config.k8s.io/v1beta1. (#123640, @liggitt) [SIG Auth and Testing]
• Kubelet should fail if NodeSwap is used with LimitedSwap and cgroupv1 node. (#123738, @kannon92) [SIG API Machinery, Node and Testing]
• Kubelet: a custom root directory for pod logs (instead of default /var/log/pods) can be specified using the podLogsDir
  key in kubelet configuration. (#112957, @mxpv) [SIG API Machinery, Node, Scalability and Testing]
• Kubelet: the .memorySwap.swapBehavior field in kubelet configuration accepts a new value NoSwap and makes this the default if unspecified; the previously accepted UnlimitedSwap value has been dropped. (#122745, @kannon92) [SIG API Machinery, Node and Testing]
• OIDC authentication will now fail if the username asserted based on a CEL expression config is the empty string. Previously the request would be authenticated with the username set to the empty string. (#123568, @enj) [SIG API Machinery, Auth and Testing]
• PodSpec API: remove note that hostAliases are not supported on hostNetwork Pods. The feature has been supported since v1.8. (#122422, @neolit123) [SIG API Machinery and Apps]
• Promote AdmissionWebhookMatchConditions to GA. The feature is now stable and the feature gate is now locked to default. (#123560, @ivelichkovich) [SIG API Machinery and Testing]
• Structured Authentication Configuration now supports DiscoveryURL.
  discoveryURL if specified, overrides the URL used to fetch discovery information.
  This is for scenarios where the well-known and jwks endpoints are hosted at a different
  location than the issuer (such as locally in the cluster). (#123527, @aramase) [SIG API Machinery, Auth and Testing]
• Support Recursive Read-only (RRO) mounts (KEP-3857) (#123180, @AkihiroSuda) [SIG API Machinery, Apps, Node and Testing]
• The StructuredAuthenticationConfiguration feature is now beta and enabled by default. (#123719, @enj) [SIG API Machinery and Auth]
• The StorageVersionMigration API, which was previously available as a Custom Resource Definition (CRD), is now a built-in API in Kubernetes. (#123344, @nilekhc) [SIG API Machinery, Apps, Auth, CLI and Testing]
• The kubernetes repo now uses Go workspaces. This should not impact end users at all, but does have impact for developers of downstream projects. Switching to workspaces caused some breaking changes in the flags to the various k8s.io/code-generator tools. Downstream consumers should look at staging/src/k8s.io/code-generator/kube_codegen.sh to see the changes. (#123529, @thockin) [SIG API Machinery, Apps, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node, Release, Storage and Testing]
• ValidatingAdmissionPolicy is promoted to GA and will be enabled by default. (#123405, @cici37) [SIG API Machinery, Apps, Auth and Testing]
• When configuring a JWT authenticator:

If username.expression uses 'claims.email', then 'claims.email_verified' must be used in
username.expression or extra[].valueExpression or claimValidationRules[].expression.
An example claim validation rule expression that matches the validation automatically
applied when username.claim is set to 'email' is 'claims.?email_verified.orValue(true)'. (#123737, @enj) [SIG API Machinery and Auth]

Feature

• Added access_mode label to volume_manager_selinux_* metrics. (#123667, @jsafrane) [SIG Node, Storage and Testing]
• Added client-go support for upgrading subresource fields from client-side to server-side management (#123484, @erikgb) [SIG API Machinery]
• Added apiserver_watch_cache_read_wait metric to measure watch cache impact on request latency. (#123190, @padlar) [SIG API Machinery and Instrumentation]
• Adds new flag, namely custom, in kubectl debug to let users customize pre-defined profiles. (#120346, @ardaguclu) [SIG CLI]
• Bump cAdvisor to v0.49.0 (#123599, @bobbypage) [SIG Node]
• Embed Node information into Pod-bound service account tokens as additional metadata
• Set the 'JTI' field in issued service account tokens, and embed this information as authentication.kubernetes.io/credential-id in user's ExtraInfo (#123135, @munnerz) [SIG API Machinery, Auth and Testing]
• Feature gates for RemoteCommand (kubectl exec, cp, and attach) over WebSockets are now enabled by default (Beta).
• Server-side feature gate: TranslateStreamCloseWebsocketRequests
• Client-side (kubectl) feature gate: KUBECTL_REMOTE_COMMAND_WEBSOCKETS
• To turn off RemoteCommand over WebSockets for kubectl, the environment variable feature gate must be explicitly set - KUBECTL_REMOTE_COMMAND_WEBSOCKETS=false (#123281, @seans3) [SIG API Machinery, CLI and Testing]
• Graduated HorizontalPodAutoscaler support for per-container metrics to stable. (#123482, @sanposhiho) [SIG API Machinery, Apps and Autoscaling]
• Graduated forensic container checkpointing KEP #2008 from Alpha to Beta. (#123215, @adrianreber) [SIG Node and Testing]
• In the Pod API, setting the alpha procMount field to Unmasked in a container now requires setting spec.hostUsers=false as well. (#123520, @haircommander) [SIG Apps, Auth and Testing]
• InitContainer's image location will be considered in scheduling when prioritizing nodes. (#123366, @kerthcet) [SIG Scheduling]
• It is possible to configure the IDs that the Kubelet uses to create user namespaces.

User namespaces support is a Beta feature now. (#123593, @giuseppe) [SIG Node]
- Kube-apiserver now reports latency metric for JWT authenticator authenticate token decisions in the apiserver_authentication_jwt_authenticator_latency_seconds metric, labeled by jwtIssuer hash and result. (#123225, @aramase) [SIG API Machinery and Auth]
- Kube-apiserver now reports the following metrics for authorization webhook match conditions:
- apiserver_authorization_match_condition_evaluation_errors_total counter metric labeled by authorizer type and name
- apiserver_authorization_match_condition_exclusions_total counter metric labeled by authorizer type and name
- apiserver_authorization_match_condition_evaluation_seconds histogram metric labeled by authorizer type and name (#123611, @ritazh) [SIG API Machinery, Auth and Testing]
- Kube-apiserver: Authorization webhooks now report the following metrics:
- apiserver_authorization_webhook_evaluations_total
- apiserver_authorization_webhook_duration_seconds
- apiserver_authorization_webhook_evaluations_fail_open_total (#123639, @liggitt) [SIG API Machinery, Auth and Testing]
- Kube-apiserver: JWT authenticator now report the following metrics:
- apiserver_authentication_config_controller_automatic_reloads_total
- apiserver_authentication_config_controller_automatic_reload_last_timestamp_seconds (#123793, @aramase) [SIG API Machinery, Auth and Testing]
- Kube-apiserver: the StructuredAuthorizationConfiguration feature gate is promoted to beta and allows using the --authorization-configuration flag (#123641, @liggitt) [SIG API Machinery and Auth]
- Kube-controller-manager: increase the global level for broadcaster's logging to 3 so that users can ignore event messages by lowering the logging level. It reduces information noise. (#122293, @mengjiao-liu) [SIG API Machinery, Apps, Autoscaling, Network, Node, Scheduling, Storage and Testing]
- Kubeadm: add the WaitForAllControlPlaneComponents feature gate. It can be used to tell kubeadm to wait for all control plane components to be ready when running "kubeadm init" or "kubeadm join --control-plane". Currently kubeadm only waits for the kube-apiserver. The "kubeadm join" workflow now includes a new experimental phase called "wait-control-plane". This phase will be marked as non-experimental when WaitForAllControlPlaneComponents becomes GA. Accordingly a "kubeadm init" phase "wait-control-plane" will also be available once WaitForAllControlPlaneComponents becomes GA. These phases can be skipped if the user prefers to not wait for the control plane components. (#123341, @neolit123) [SIG Cluster Lifecycle]
- Kubeadm: print all the kubelets and nodes that need to be upgraded on "upgrade plan". (#123578, @carlory) [SIG Cluster Lifecycle]
- Kubectl port-forward over websockets (tunneling SPDY) can be enabled using an Alpha feature flag environment variable: KUBECTL_PORT_FORWARD_WEBSOCKETS=true. The API Server being communicated to must also have an Alpha feature flag enabled: PortForwardWebsockets. (#123413, @seans3) [SIG API Machinery, CLI, Node and Testing]
- Kubernetes is now built with go 1.22.1 (#123750, @cpanato) [SIG Release and Testing]
- Node podresources API now includes init containers with containerRestartPolicy of Always when SidecarContainers feature is enabled. (#120718, @gjkim42) [SIG Node and Testing]
- Promote ImageMaximumGCAge feature to beta (#123424, @haircommander) [SIG Node and Testing]
- Promote PodHostIPs condition to GA and lock to default. (#122870, @wzshiming) [SIG Apps, Network, Node and Testing]
- Target drop-in kubelet configuration dir feature to Beta (#122907, @sohankunkerkar) [SIG Node and Testing]
- The Kubelet rejects creating the pod if hostUserns=false and the CRI runtime does not support user namespaces. (#123216, @giuseppe) [SIG Node]
- The watch cache waits until it is at least as fresh as given requestedWatchRV if sendInitialEvents was requested. (#122830, @p0lyn0mial) [SIG API Machinery, Network and Testing]
- ValidatingAdmissionPolicy now exclude TokenReview, SelfSubjectReview, LocalSubjectAccessReview, and SubjectAccessReview from all versions of authentication.k8s.io and authorization.k8s.io group. (#123543, @jiahuif) [SIG API Machinery and Testing]
- kubectl get job now displays the status for the listed jobs. (#123226, @ivanvc) [SIG Apps and CLI]

Bug or Regression

• Adds the namespace when using 'kubectl logs ' and the pod is not found. Previously the message returned would be 'Error from server (NotFound): pods "my-pod-name" not found'. This has been updated to reflect the namespace in the message as follows: 'Error from server (NotFound): pods "my-pod-name" not found in namespace "default"' (#120111, @newtondev) [SIG CLI]
• DRA: ResourceClaim and PodSchedulingContext status updates no longer allow changing object meta data. (#123730, @pohly) [SIG Node]
• Fix CEL estimated cost to for expressions that perform operations on the result of map()
  operations, (e.g. .map(...).exists(...) ) to have the correct estimated instead of an unbounded
  cost. (#123562, @jpbetz) [SIG API Machinery, Auth and Cloud Provider]
• Fix node lifecycle controller panic when conditionType ready is been patch nil by mistake (#122874, @fusida) [SIG Apps, Network and Node]
• Fix non-recursive list returning "resource version too high" error when consistent list from cache is enabled (#123674, @serathius) [SIG API Machinery]
• Fixed a bug that an init container with containerRestartPolicy with Always cannot update its state from terminated to non-terminated for the pod with restartPolicy with Never or OnFailure. (#123323, @gjkim42) [SIG Apps and Node]
• Fixed incorrect syncCronJob error logging. (#122493, @mengjiao-liu) [SIG Apps]
• Fixed the disruption controller's PDB status synchronization to maintain all PDB conditions during an update. (#122056, @dhenkel92) [SIG Apps]
• Fixes bug where providing a fieldpath to a CRD Validation Rule would erroneously affect the reported field path of other unrelated CRD Validation Rules on the same schema (#123475, @alexzielenski) [SIG API Machinery]
• JWTs used in service account and OIDC authentication are now strictly parsed to confirm that they use compact serialization. Other encodings were not previously accepted, but would result in different unspecific errors. (#123540, @enj) [SIG API Machinery and Auth]
• Kubeadm: in the new output API "output.kubeadm.k8s.io/v1alpha3" modify the UpgradePlan structure that is used when calling "kubeadm upgrade plan ... -o yaml|json", to include a list of multiple available upgrades. (#123461, @carlory) [SIG Cluster Lifecycle]
• Kubeadm: avoid uploading a defaulted flag value "--authorization-mode=Node,RBAC" for the kube-apiserver in the ClusterConfiguration stored in the "kube-system/kubeadm-config" ConfigMap. "Node,RBAC" are already the kubeadm defaults for this flag, so this action is redundant. (#123555, @neolit123) [SIG Cluster Lifecycle]
• OpenAPI V2 will no longer publish aggregated apiserver OpenAPI for group-versions not matching the APIService specified group version (#123570, @Jefftree) [SIG API Machinery]
• Prevent watch cache starvation by moving its watch to separate RPC and add a SeparateCacheWatchRPC feature flag to disable this behavior (#123532, @serathius) [SIG API Machinery]
• The initialization of nodes using external cloud-providers now waits for the providerID value to be available before declaring the node ready. This is required because previously, if there were errors of communication with the cloud-provider on the cloud-controller-manager, nodes may have been declared Ready without having this field or the zone labels, and the information was never reconciled. The providerID and the zone labels are required for integrations like loadbalancers to work correctly. Users still can opt-out to this new behavior by setting the feature flag OptionalProviderID in the cloud-controller-manager. (#123331, @aojea) [SIG API Machinery, Cloud Provider and Testing]
• The initialization of nodes using external cloud-providers now waits for the providerID value to be available before untainting it. This is required because , if there are communication errors with the cloud-provider on the cloud-controller-manager, nodes may have been declared Ready without having this field or the zone labels, and this information was never reconciled. The providerID and the zone labels are required for integrations like loadbalancers to work correctly. Cloud providers that does not implement the GetInstanceProviderID method will not require the providerID to be set and will not fail to initialize the node for backward compatibility issues. (#123713, @aojea) [SIG Cloud Provider]
• Updates google.golang.org/protobuf to v1.33.0 to resolve CVE-2024-24786 (#123758, @liggitt) [SIG API Machinery, Architecture, Auth, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node and Storage]
• [kubeadam][structured authz] avoid setting default --authorization-mode when --authorization-config is provided (#123654, @LiorLieberman) [SIG Cluster Lifecycle]

Other (Cleanup or Flake)

• Accept zero as a default value for kubectl create token duration (#123565, @ah8ad3) [SIG CLI]
• Update kubedns and nodelocaldns to v1.23.0 (#123310, @bzsuni) [SIG Cloud Provider]

Dependencies

Added

• github.com/pkg/diff: 20ebb0f
• golang.org/x/telemetry: b75ee88
• k8s.io/gengo/v2: 51d4e06

Changed

• github.com/docker/docker: v20.10.24+incompatible → v20.10.27+incompatible
• github.com/golang/protobuf: v1.5.3 → v1.5.4
• github.com/google/cadvisor: v0.48.1 → v0.49.0
• github.com/google/cel-go: v0.17.7 → v0.17.8
• golang.org/x/mod: v0.14.0 → v0.15.0
• golang.org/x/net: v0.19.0 → v0.21.0
• golang.org/x/sync: v0.5.0 → v0.6.0
• golang.org/x/tools: v0.16.1 → v0.18.0
• google.golang.org/protobuf: v1.31.0 → v1.33.0
• k8s.io/kube-openapi: 778a556 → 70dd376

Removed

• k8s.io/gengo: 9cce18d