This is the first beta of the 1.29 release.

Significant changes

Deferred deletion / pruning phase

Some infrastructure changes are potentially disruptive to the continued
operation of the cluster. For the most disruptive operations, particularly
those that break rolling-update of the cluster, we have started to use deferred
deletion to minimize the impact. For example, on AWS we create a second NLB
during the kops update phase when we cannot change the NLB directly.
kops update will report that a --prune is needed. To minimize disruption,
we recommend you perform this after a rolling-update, for example:

kops update $MYCLUSTER --yes --admin
kops rolling-update $MYCLUSTER --yes
kops update $MYCLUSTER --yes --admin --prune # NEW!

Deferred deletion is currently used to safely introduce security groups for NLBs on AWS,
and to move to an internal load balancer for kops-controller on GCP.

Initial OpenTelemetry Support

We are starting to add (experimental) support for OpenTelemetry,
in particular Tracing support. Setting OTEL_EXPORTER_OTLP_TRACES_FILE
will write a trace file which can then be read by the traceserver program.
More information and options are described in docs/opentelemetry.md.
The tracing data is not expected to be particularly useful for end-users in
this release; the (non-standard) recording approach is instead intended to
work well with our Prow end-to-end testing system so that developers can
optimize kOps.

Please note: this is not telemetry in the "phone-home" sense.
The kOps project does not collect data from your machine. As an
open-source project we do not even want to collect any of your data.
Currently the only OpenTelemetry backend supported is writing to a
filesystem (and it is opt-in). In future you will be able to configure
other OpenTelemetry backends, but this data will only be sent if
you enable OpenTelemetry, and only sent to where you configure.

AWS

• Network Load Balancers in front of the Kubernetes API and bastion hosts now
  have a security group attached. These security groups are used for security group rules
  allowing incoming traffic to the NLBs as well as traffic between the NLBs and their target
  instances.

• Posts event data to URL upon instance interruption action in aws-node-termination-handler with WEBHOOK_URL.

GCP

• As of Kubernetes version 1.29, credentials for private GCR/AR repositories will be handled by the out-of-tree credential provider. This is an additional binary that each instance downloads from the assets repository.

• We now use a private load-balancer for in-cluster traffic on GCP, which allows us
  to use network tags to restrict access only to the cluster nodes.

Breaking changes

• kops toolbox dump limits the number of nodes dumped to 500 by default. Use --max-nodes to override.

• Support for Kubernetes version 1.23 has been removed.

Known Issues

• The Amazon VPC CNI is now compatible with Ubuntu 22.04. Fix applied via kubernetes/kops#16313.

Deprecations

• Support for Kubernetes version 1.24 is deprecated and will be removed in kOps 1.30.

• Support for Kubernetes version 1.25 is deprecated and will be removed in kOps 1.31.

• Support for AWS Classic Load Balancer for API is deprecated and should not be used for newly created clusters.

• All legacy addons are deprecated in favor of managed addons, including the metrics server addon and the autoscaler addon.

What's Changed

• kops validate cluster improvements by @upodroid in https://github.com/kubernetes/kops/pull/16187
• gce: Remove custom resolver by @hakman in https://github.com/kubernetes/kops/pull/16189
• skip_regex.go: kube-router add back in service afinity test by @aauren in https://github.com/kubernetes/kops/pull/16188
• chore(channels): bump k8s and ubuntu ami versions in alpha channel by @moshevayner in https://github.com/kubernetes/kops/pull/16190
• Update Calico to v3.27.0 by @hakman in https://github.com/kubernetes/kops/pull/16192
• Disable Statefulsets provisioning from CL2 Load Tests by @hakuna-matatah in https://github.com/kubernetes/kops/pull/16172
• Add cert-manager resource types to kubetest2-kops artifacts by @rifelpet in https://github.com/kubernetes/kops/pull/16193
• Parallelize k8s resource dumps with kops toolbox dump by @rifelpet in https://github.com/kubernetes/kops/pull/16196
• Include pod logs in toolbox dump by @rifelpet in https://github.com/kubernetes/kops/pull/16198
• Update k8s.io/* to v0.29.0 by @hakman in https://github.com/kubernetes/kops/pull/16199
• Update dependencies by @github-actions in https://github.com/kubernetes/kops/pull/16201
• test: Print the create cluster command by @hakman in https://github.com/kubernetes/kops/pull/16202
• scale-test: Add feature flag for creating a single nodes instance group by @hakman in https://github.com/kubernetes/kops/pull/16203
• Dump previous and current container logs separately by @hakman in https://github.com/kubernetes/kops/pull/16200
• aws: Set AWS_REGION env var for ebs-csi-node and ebs-csi-driver by @hakman in https://github.com/kubernetes/kops/pull/16206
• aws: Add option for setting QPS and Burst for EBS CSI Driver by @hakman in https://github.com/kubernetes/kops/pull/16207
• Spotinst: Bump controller version to 1.0.97 by @yehielnetapp in https://github.com/kubernetes/kops/pull/16208
• feat: add us-west zone for hetzner by @finzzz in https://github.com/kubernetes/kops/pull/16209
• Check if kubeconfig exists before dumping resources by @hakman in https://github.com/kubernetes/kops/pull/16205
• Promote alpha to stable by @moshevayner in https://github.com/kubernetes/kops/pull/16210
• aws: Use instance metadata to get warm pool state by @rifelpet in https://github.com/kubernetes/kops/pull/16213
• Dump and redact secrets by @rifelpet in https://github.com/kubernetes/kops/pull/16211
• Update to Cilium 1.14.5 by @hakman in https://github.com/kubernetes/kops/pull/16214
• Allow override of the DNS domain used by the tests. by @ameukam in https://github.com/kubernetes/kops/pull/16217
• aws: Retrieve instance info only when max pods is not set by @hakman in https://github.com/kubernetes/kops/pull/16216
• Add permission needed for service-linked role creation by @ameukam in https://github.com/kubernetes/kops/pull/16219
• Remove kube-system cert-manager webhook exclusion by @rifelpet in https://github.com/kubernetes/kops/pull/16221
• Jaeger tracing visualizer improvements by @rifelpet in https://github.com/kubernetes/kops/pull/16220
• Update dependencies by @github-actions in https://github.com/kubernetes/kops/pull/16224
• aws: Set provider ID when starting kubelet by @hakman in https://github.com/kubernetes/kops/pull/16223
• scale-test: Reduce validation count and interval by @hakman in https://github.com/kubernetes/kops/pull/16225
• aws: Update EBS CSI driver to v1.26.0 by @hakman in https://github.com/kubernetes/kops/pull/16227
• Add option for setting CCM ConcurrentNodeSyncs by @hakman in https://github.com/kubernetes/kops/pull/16228
• aws: Skip deleting ASG instances without volumes by @hakman in https://github.com/kubernetes/kops/pull/16229
• Make cluster deletion configurable by @hakman in https://github.com/kubernetes/kops/pull/16231
• Bump actions/dependency-review-action from 3.1.4 to 3.1.5 by @dependabot in https://github.com/kubernetes/kops/pull/16232
• scale-test: Use single nodes instance group for AWS by @hakman in https://github.com/kubernetes/kops/pull/16204
• Bump kubetest2 by @ameukam in https://github.com/kubernetes/kops/pull/16234
• test: Improve cluster deletion defaults by @hakman in https://github.com/kubernetes/kops/pull/16236
• Replace k8s.io/utils/strings/slices with golang.org/x/exp/slices by @hakman in https://github.com/kubernetes/kops/pull/16238
• aws: Update EBS CSI driver by @hakman in https://github.com/kubernetes/kops/pull/16239
• aws: Use domain instead of vpc when rendering aws_eip by @hakman in https://github.com/kubernetes/kops/pull/16237
• Bump GCP terraform provider to latest by @rifelpet in https://github.com/kubernetes/kops/pull/16242
• docs: fix broken example command by @markusleh in https://github.com/kubernetes/kops/pull/16243
• Update dependencies by @github-actions in https://github.com/kubernetes/kops/pull/16244
• Revert "aws: Skip deleting ASG instances without volumes" by @hakman in https://github.com/kubernetes/kops/pull/16246
• Update Go to v1.21.6 by @hakman in https://github.com/kubernetes/kops/pull/16245
• Prefer external endpoints when building kubeconfig by @justinsb in https://github.com/kubernetes/kops/pull/16248
• aws: Terminate ASG instances in batches of 100 instances by @hakman in https://github.com/kubernetes/kops/pull/16251
• aws: Ignore InvalidRouteTableID.NotFound errors during cluster deletion by @hakman in https://github.com/kubernetes/kops/pull/16252
• aws: fix maxPods when cilium ipam=eni is used by @argusua in https://github.com/kubernetes/kops/pull/16253
• Update containerd to v1.7.12 by @hakman in https://github.com/kubernetes/kops/pull/16257
• Switch to GCS url for upgrades tests by @ameukam in https://github.com/kubernetes/kops/pull/16258
• Use dns=none for newly created clusters including for AWS and GCE by @hakman in https://github.com/kubernetes/kops/pull/16262
• Update aws-sdk-go to v1.49.24 by @ameukam in https://github.com/kubernetes/kops/pull/16263
• test: Set num-nodes flag by @upodroid in https://github.com/kubernetes/kops/pull/16176
• Refactor: Replace ForAPIServer with WellKnownServices by @justinsb in https://github.com/kubernetes/kops/pull/15829
• gce: fix nlb firewall rules, operations and alias network subnets by @upodroid in https://github.com/kubernetes/kops/pull/16265
• build(deps): bump actions/dependency-review-action from 3.1.5 to 4.0.0 by @dependabot in https://github.com/kubernetes/kops/pull/16267
• openstack: Include kube-apiserver controlplane ports in dns=none by @zetaab in https://github.com/kubernetes/kops/pull/16271
• Increase CCM workers to speed up node bootstrap process by @hakuna-matatah in https://github.com/kubernetes/kops/pull/16256
• Add 1.28 release notes to docs menu by @yurrriq in https://github.com/kubernetes/kops/pull/16275
• Add support to configure HPA Controller concurrent syncs flag in HPA/KCM Controller by @hakuna-matatah in https://github.com/kubernetes/kops/pull/16277
• Add support to configure Job Controller concurrent syncs flag in Job… by @hakuna-matatah in https://github.com/kubernetes/kops/pull/16280
• Refactor: Plumb context through GCE firewallRule methods by @justinsb in https://github.com/kubernetes/kops/pull/16281
• Fix dumping logs for GCE scale tests by @upodroid in https://github.com/kubernetes/kops/pull/16266
• Add boskos-resource-type flag to use different GCE projects for scale/gpu testing by @upodroid in https://github.com/kubernetes/kops/pull/16268
• OpenStack: update CSI images by @zetaab in https://github.com/kubernetes/kops/pull/16283
• toolbox dump: output correct type for target groups by @justinsb in https://github.com/kubernetes/kops/pull/16285
• chore(channels): bump k8s and ubuntu ami versions in alpha channel by @moshevayner in https://github.com/kubernetes/kops/pull/16284
• Fix: support comparison of int types in dry-run by @justinsb in https://github.com/kubernetes/kops/pull/16290
• refactor: NetworkLoadBalancer Name should match Name tag by @justinsb in https://github.com/kubernetes/kops/pull/16288
• tweak: Set Scheme on NLB tasks for public load balancers by @justinsb in https://github.com/kubernetes/kops/pull/16289
• refactor: Introduce runTests helper method into aws tests by @justinsb in https://github.com/kubernetes/kops/pull/16292
• Refactor: Move NLB listing function into awsup by @justinsb in https://github.com/kubernetes/kops/pull/16295
• chore(networking): bump aws cni to 1.16.2 by @moshevayner in https://github.com/kubernetes/kops/pull/16297
• Revert "Don't set LimitNoFile for containerd systemd unit file" by @zetaab in https://github.com/kubernetes/kops/pull/16300
• Update runc & containerd by @zetaab in https://github.com/kubernetes/kops/pull/16302
• chore(channels): promote alpha to stable by @moshevayner in https://github.com/kubernetes/kops/pull/16306
• refactor: wait for load balancer readiness using a private field by @justinsb in https://github.com/kubernetes/kops/pull/16294
• Add GCE scale testing on kops by @upodroid in https://github.com/kubernetes/kops/pull/16181
• fix(nodeup): set MACAddressPolicy=none when using AWS VPC CNI by @moshevayner in https://github.com/kubernetes/kops/pull/16313
• Upgrade AWS Load Balancer Controller to v2.7.0 by @yurrriq in https://github.com/kubernetes/kops/pull/16316
• Update to cilium 1.15 by @zadjadr in https://github.com/kubernetes/kops/pull/16315
• feat: added image minimum and maximum gc age by @Lerentis in https://github.com/kubernetes/kops/pull/16318
• build(deps): bump peter-evans/create-pull-request from 5.0.2 to 6.0.0 by @dependabot in https://github.com/kubernetes/kops/pull/16322
• Update dependencies by @hakman in https://github.com/kubernetes/kops/pull/16323
• Skip gen-cli-docs on depup by @hakman in https://github.com/kubernetes/kops/pull/16321
• Refactor: Split out NLB Listener into its own task by @justinsb in https://github.com/kubernetes/kops/pull/16299
• refactor: Drop TargetGroups from NetworkLoadBalancer task by @justinsb in https://github.com/kubernetes/kops/pull/16324
• Dont set -num-nodes on karpenter-managed clusters by @rifelpet in https://github.com/kubernetes/kops/pull/16325
• docs: Remove warning about Amazon VPC CNI not being compatible with Ubuntu 22.04 by @moshevayner in https://github.com/kubernetes/kops/pull/16326
• Set LimitNOFILE to 1048576 instead of infinity by @dims in https://github.com/kubernetes/kops/pull/16329
• azure: Migrate to the new SDK version by @hakman in https://github.com/kubernetes/kops/pull/16286
• Update dependencies by @github-actions in https://github.com/kubernetes/kops/pull/16331
• Set KUBECONFIG for LBC's ginkgo tests by @rifelpet in https://github.com/kubernetes/kops/pull/16334
• Docs: fix typos in office hours page by @justinsb in https://github.com/kubernetes/kops/pull/16337
• clockmock: Add more methods that take a context by @justinsb in https://github.com/kubernetes/kops/pull/16338
• Move DNS topology setup earlier in cluster creation by @rifelpet in https://github.com/kubernetes/kops/pull/16342
• deletion: tolerate concurrent SQS queue deletion by @justinsb in https://github.com/kubernetes/kops/pull/16341
• Cleanup import of the same package in tests by @justinsb in https://github.com/kubernetes/kops/pull/16343
• validation: Allow overlap of pod/node CIDR and service CIDR by @justinsb in https://github.com/kubernetes/kops/pull/16344
• Include /etc/hosts coredns mounts for dns=none clusters by @rifelpet in https://github.com/kubernetes/kops/pull/16347
• azure: Replace lb.ForAPIServer with lb.WellKnownServices by @hakman in https://github.com/kubernetes/kops/pull/16348
• Add support for AL2023 AMI to use Amazon VPC CNI by @dims in https://github.com/kubernetes/kops/pull/16350
• aws: Post event data to URL upon instance interruption action by @voriol in https://github.com/kubernetes/kops/pull/16009
• Refactor IAM Policy Builder by @rifelpet in https://github.com/kubernetes/kops/pull/16351
• create command: remove example docs say is not implemented yet. by @jrabbit in https://github.com/kubernetes/kops/pull/16308
• target group: refactor discovery into awsup by @justinsb in https://github.com/kubernetes/kops/pull/16339
• Use IAM Policy Builder for SQS Queue Policy by @rifelpet in https://github.com/kubernetes/kops/pull/16353
• refactor: Introduce DeletionProcessingMode by @justinsb in https://github.com/kubernetes/kops/pull/16293
• Update Go to v1.22.0 by @hakman in https://github.com/kubernetes/kops/pull/16346
• Update dependencies by @github-actions in https://github.com/kubernetes/kops/pull/16357
• azure: Avoid spurious changes in VirtualNetwork by @hakman in https://github.com/kubernetes/kops/pull/16358
• Generate revisions of NLB objects, and introduce cleanup phase by @justinsb in https://github.com/kubernetes/kops/pull/16356
• gce: Update GCE storage service scope to DevstorageFullControlScope to resolve permission error. by @sl1pm4t in https://github.com/kubernetes/kops/pull/16355
• add support for devcontainer by @remyleone in https://github.com/kubernetes/kops/pull/16186
• azure: Mark a few tasks as implementing HasAddress by @justinsb in https://github.com/kubernetes/kops/pull/16359
• Set --dns=none on upgrade tests from older kops versions by @rifelpet in https://github.com/kubernetes/kops/pull/16360
• build(deps): bump actions/dependency-review-action from 4.0.0 to 4.1.0 by @dependabot in https://github.com/kubernetes/kops/pull/16361
• Update Cilium to v1.15.1 by @hakman in https://github.com/kubernetes/kops/pull/16362
• Fix bash conditional pattern matching in upgrade script by @rifelpet in https://github.com/kubernetes/kops/pull/16364
• devcontainer: update go version, use features by @justinsb in https://github.com/kubernetes/kops/pull/16365
• Skip known-failing test on most e2e jobs by @rifelpet in https://github.com/kubernetes/kops/pull/16368
• aws: Update EBS CSI driver to v1.28.0 by @hakman in https://github.com/kubernetes/kops/pull/16369
• doc/aws: Add space before the k8s slack url by @tungbq in https://github.com/kubernetes/kops/pull/16370
• Skip hostname test for all aws jobs by default by @rifelpet in https://github.com/kubernetes/kops/pull/16373
• Migrate many-addons e2e template to dns=none by @rifelpet in https://github.com/kubernetes/kops/pull/16374
• Update dependencies by @github-actions in https://github.com/kubernetes/kops/pull/16375
• gce: match IP addresses including subnet where relevant by @justinsb in https://github.com/kubernetes/kops/pull/16380
• chore: update dependencies in submodules by @justinsb in https://github.com/kubernetes/kops/pull/16372
• GCE: Use internal load balancer for node to control-plane traffic by @justinsb in https://github.com/kubernetes/kops/pull/16379
• Skip hostname e2e test on digitalocean by @rifelpet in https://github.com/kubernetes/kops/pull/16381
• build(deps): bump actions/dependency-review-action from 4.1.0 to 4.1.3 by @dependabot in https://github.com/kubernetes/kops/pull/16384
• gce: Limit health check names to 63 chars by @hakman in https://github.com/kubernetes/kops/pull/16385
• gce: Limit backend names to 63 chars by @hakman in https://github.com/kubernetes/kops/pull/16386
• Update NVIDIA Container Toolkit URL by @elezar in https://github.com/kubernetes/kops/pull/16387
• Install nerdctl and crictl on nodes by @h3poteto in https://github.com/kubernetes/kops/pull/16383
• Continue attemps to dump artifacts in toolbox dump by @rifelpet in https://github.com/kubernetes/kops/pull/16389
• chore: update boilerplate.py to recognize new build tags by @justinsb in https://github.com/kubernetes/kops/pull/16390
• Add validation to help users move from usePolicyConfigMap by @hakman in https://github.com/kubernetes/kops/pull/16391
• Experimental limited support for cluster-api by @justinsb in https://github.com/kubernetes/kops/pull/15522
• Update dependencies by @github-actions in https://github.com/kubernetes/kops/pull/16392
• build(deps): bump peter-evans/create-pull-request from 6.0.0 to 6.0.1 by @dependabot in https://github.com/kubernetes/kops/pull/16393
• e2e tests: When upgrading, wait for the new configuration by @justinsb in https://github.com/kubernetes/kops/pull/16395
• Fix shellcheck warnings for bootstrap script by @hakman in https://github.com/kubernetes/kops/pull/16394
• Update dependencies by @github-actions in https://github.com/kubernetes/kops/pull/16397
• Use github.com/go-viper/mapstructure/v2 by @ameukam in https://github.com/kubernetes/kops/pull/16402
• docs: Update relnotes for 1.29 for deferred deletion by @justinsb in https://github.com/kubernetes/kops/pull/16404
• aws: Expose port 8443 when using NLB with a custom certificate by @justinsb in https://github.com/kubernetes/kops/pull/16403
• gce: Change default storage class to balanced-csi by @sl1pm4t in https://github.com/kubernetes/kops/pull/16269
• gce: Set node IP Alias range to match NodeCIDRMaskSize by @sl1pm4t in https://github.com/kubernetes/kops/pull/16272
• Release 1.29.0-beta.1 by @justinsb in https://github.com/kubernetes/kops/pull/16406

New Contributors

• @finzzz made their first contribution in https://github.com/kubernetes/kops/pull/16209
• @markusleh made their first contribution in https://github.com/kubernetes/kops/pull/16243
• @argusua made their first contribution in https://github.com/kubernetes/kops/pull/16253
• @Lerentis made their first contribution in https://github.com/kubernetes/kops/pull/16318
• @voriol made their first contribution in https://github.com/kubernetes/kops/pull/16009
• @jrabbit made their first contribution in https://github.com/kubernetes/kops/pull/16308
• @tungbq made their first contribution in https://github.com/kubernetes/kops/pull/16370

Full Changelog: https://github.com/kubernetes/kops/compare/v1.29.0-alpha.3...v1.29.0-beta.1