Kubernetes - v1.26.0

Documentation

Changelog since v1.25.0

Urgent Upgrade Notes

(No, really, you MUST read this before you upgrade)

• Deprecated beta APIs scheduled for removal in v1.26 are no longer served. See https://kubernetes.io/docs/reference/using-api/deprecation-guide/#v1-26 for more information. (#111973, @liggitt)

Changes by Kind

Deprecation

• CLI flag pod-eviction-timeout is deprecated and will be removed together with enable-taint-manager in v1.27. (#113710, @kerthcet)
• Kube-apiserver: the unused --master-service-namespace flag was deprecated and will be removed in v1.27. (#112797, @SataQiu)
• The gcp and azure auth plugins have been removed from client-go and kubectl. See kubelogin and Kubectl Auth Changes in GKE for details about the cloud-specific replacements. (#112341, @enj)

API Change

• 'A new preEnqueue extension point was added to scheduler's component config
  v1beta2/v1beta3/v1.'
  (#113275, @Huang-Wei)
• 'Added a ResourceClaim API (in the resource.k8s.io/v1alpha1 API group and
  behind the DynamicResourceAllocation feature gate).
  The new API is now more flexible than the existing Device Plugins feature of Kubernetes because it
  allows Pods to request (claim) special kinds of resources, which can be available at node level, cluster
  level, or following any other model you implement.' (#111023, @pohly)
• 'Container preStop and postStart lifecycle handlers using httpGet now
  honor the specified scheme and headers fields. This enables setting custom
  headers and changing the scheme to HTTPS, consistent with container
  startup/readiness/liveness probe capabilities. Lifecycle handlers configured
  with scheme: HTTPS that encounter errors indicating the endpoint is actually
  using HTTP fall back to making the request over HTTP for compatibility with
  previous releases. When this happens, a LifecycleHTTPFallback event is recorded
  in the namespace of the pod and a kubelet_lifecycle_handler_http_fallbacks_total
  metric in the kubelet is incremented. Cluster administrators can opt out of the
  expanded lifecycle handler capabilities by setting
  --feature-gates=ConsistentHTTPGetHandlers=false in kubelet.'
  (#86139, @jasimmons)
• 'Graduated JobTrackingWithFinalizers to stable.
  Jobs created before the feature was enabled are still tracked without finalizers.
  Jobs tracked with finalizers have the annotation batch.kubernetes.io/job-tracking.
  If the annotation is present and the user attempts to remove it, the control plane adds it back.
  The annotation batch.kubernetes.io/job-tracking is now deprecated.
  The control plane will ignore it and stop adding it for new Jobs in v1.27.' (#113510, @alculquicondor)
• 'Kubelet added the following Pod failure conditions:
• DisruptionTarget (graceful node shutdown, node pressure eviction)' (#112360, @mimowo)
• 'Priority and Fairness has introduced a new feature called borrowing that allows an API priority level
  to borrow a number of seats from other priority level(s). As a cluster operator, you can enable borrowing
  for a certain priority level configuration object via the two newly introduced fields lendablePercent, and
  borrowingLimitPercent located under the .spec.limited field of the designated priority level.
  This change added the following metrics:
  • apiserver_flowcontrol_nominal_limit_seats: Nominal number of execution seats configured for each priority level
  • apiserver_flowcontrol_lower_limit_seats: Configured lower bound on number of execution seats available to each priority level
  • apiserver_flowcontrol_upper_limit_seats: Configured upper bound on number of execution seats available to each priority level
  • apiserver_flowcontrol_demand_seats: Observations, at the end of every nanosecond, of (the number of seats each priority level could use) / (nominal number of seats for that level)
  • apiserver_flowcontrol_demand_seats_high_watermark: High watermark, over last adjustment period, of demand_seats
  • apiserver_flowcontrol_demand_seats_average: Time-weighted average, over last adjustment period, of demand_seats
  • apiserver_flowcontrol_demand_seats_stdev: Time-weighted standard deviation, over last adjustment period, of demand_seats
  • apiserver_flowcontrol_demand_seats_smoothed: Smoothed seat demands
  • apiserver_flowcontrol_target_seats: Seat allocation targets
  • apiserver_flowcontrol_seat_fair_frac: Fair fraction of server's concurrency to allocate to each priority level that can use it
  • apiserver_flowcontrol_current_limit_seats: current derived number of execution seats available to each priority level
    The possibility of borrowing means that the old metric apiserver_flowcontrol_request_concurrency_limit can no longer mean both the configured concurrency limit and the enforced concurrency limit. Henceforth it means the configured concurrency limit.' (#113485, @MikeSpreitzer)
• 'NodeInclusionPolicy in podTopologySpread plugin is now enabled by default.'
  (#113500, @kerthcet)
• 'PodDisruptionBudget now adds an alpha spec.unhealthyPodEvictionPolicy field.
  When the PDBUnhealthyPodEvictionPolicy feature-gate is enabled in kube-apiserver,
  setting this field to "AlwaysAllow" allows pods to be evicted if they do not
  have a ready condition, regardless of whether the PodDisruptionBudget is currently
  healthy.'
  (#113375, @atiratree)
• 'metav1.LabelSelectors specified in API objects are now validated to ensure
  they do not contain invalid label values that will error at time of use. Existing
  invalid objects can be updated, but new objects are required to contain valid
  label selectors.'
  (#113699, @liggitt)
• Add percentageOfNodesToScore as a scheduler profile level parameter to API version v1. When a profile percentageOfNodesToScore is set, it will override global percentageOfNodesToScore. (#112521, @yuanchen8911)
• Add auth API to get self subject attributes (new selfsubjectreviews API is added).
  The corresponding command for kubctl is provided - kubectl auth whoami. (#111333, @nabokihms) [SIG API Machinery, Auth, CLI and Testing]
• Added kubernetes_feature_enabled metric series to track whether each active feature gate is enabled. (#112690, @logicalhan)
• Added a --topology-manager-policy-options flag to the kubelet to support fine tuning the topology manager policies. The first policy option, prefer-closest-numa-nodes, allows these policies to favor sets of NUMA nodes with shorter distance between nodes when making admission decisions. (#112914, @PiotrProkop)
• Added a feature that allows a StatefulSet to start numbering replicas from an arbitrary non-negative ordinal, using the .spec.ordinals.start field. (#112744, @pwschuurman)
• Added a kube-proxy flag (--iptables-localhost-nodeports, default true) to allow disabling NodePort services on loopback addresses. Note: this only applies to iptables mode and ipv4. (#108250, @cyclinder)
• Added a new namespace alpha field to DataSourceRef field in PersistentVolumeClaim API. (#113186, @ttakahashi21)
• Aggregated discovery will be alpha and can be toggled with the AggregatedDiscoveryEndpoint feature flag. (#113171, @Jefftree)
• Clarified the CFS quota as 100ms in the code comments and set the minimum cpuCFSQuotaPeriod to 1ms to match Linux kernel expectations. (#112123, @paskal)
• Component-base: make the validation logic about LeaderElectionConfiguration consistent between component-base and client-go (#111758, @SataQiu) [SIG API Machinery and Scheduling]
• Deprecated the apiserver_request_slo_duration_seconds metric for v1.27 in favor of apiserver_request_sli_duration_seconds for naming consistency purposes with other SLI-specific metrics and to avoid any confusion between SLOs and SLIs. (#112679, @dgrisonnet)
• Enable the "Retriable and non-retriable pod failures for jobs" feature into beta. (#113360, @mimowo)
• Enabled kube-controller-manager to support '--concurrent-horizontal-pod-autoscaler-syncs' flag to set the number of horizontal pod autoscaler controller workers. (#108501, @zroubalik)
• Fixed spurious field is immutable errors validating updates to Event API objects via the events.k8s.io/v1 API. (#112183, @liggitt)
• Graduated ServiceInternalTrafficPolicy feature to GA. (#113496, @avoltz)
• In 'kube-proxy`: The "userspace" proxy mode (deprecated for over a year) is no
  longer supported on either Linux or Windows. Users should use "iptables" or "ipvs"
  on Linux, or "kernelspace" on Windows.
  (#112133, @knabben)
• Introduce v1beta3 for Priority and Fairness with the following changes to the API spec:
• rename 'assuredConcurrencyShares' (located under `spec.limited') to 'nominalConcurrencyShares'.
• apply strategic merge patch annotations to 'Conditions' of flowschemas and prioritylevelconfigurations. (#112306, @tkashem)
• Introduced v1alpha1 API for validating admission policies, enabling extensible admission control via CEL expressions (KEP 3488: CEL for Admission Control). To use, enable the ValidatingAdmissionPolicy feature gate and the admissionregistration.k8s.io/v1alpha1 API via --runtime-config. (#113314, @cici37)
• KMS: added validation for duplicate kms config name when auto reload is enabled. If you enabled automatic reload of encryption configuration with API server flag --encryption-provider-config-automatic-reload, ensure all the KMS provider names (v1 and v2) in the encryption configuration are unique. (#113697, @aramase)
• Kubelet external Credential Provider feature is moved to GA. Credential Provider Plugin and Credential Provider Config APIs updated from v1beta1 to v1 with no API changes. (#111616, @ndixita)
• Legacy klog flags are no longer available. Only -v and -vmodule are still supported. (#112120, @pohly) [SIG Architecture, CLI, Instrumentation, Node and Testing]
• Moved MixedProtocolLBService from beta to GA. (#112895, @janosi)
• New Pod API field .spec.schedulingGates is introduced to enable users to control when to mark a Pod as scheduling ready. (#113274, @Huang-Wei)
• Protobuf serialization of metav1.MicroTime timestamps (used in Lease and Event API objects) has been corrected to truncate to microsecond precision, to match the documented behavior and JSON/YAML serialization. Any existing persisted data is truncated to microsecond when read from etcd. (#111936, @haoruan)
• Removed feature gates ServiceLoadBalancerClass and ServiceLBNodePortControl. These feature gates were enabled (and locked) since v1.24. (#112577, @andrewsykim)
• Reverted regression that prevented client-go latency metrics to be reported with a template URL to avoid label cardinality. (#111752, @aanm)
• The EndpointSliceTerminatingCondition feature gate was graduated to GA. The gate is now locked and will be removed in v1.28. (#113351, @andrewsykim)
• DynamicKubeletConfig feature gate has been removed from the API server.
  Dynamic kubelet reconfiguration now can't be used even when older nodes are still
  attempting to rely on it. This is aligned with the Kubernetes version skew policy.
  (#112643, @SergeyKanzhelev)
• kubectl wait command with jsonpath flag will wait for target path until timeout.
  (#109525, @jonyhy96)

Feature

• 'Added selector validation to HorizontalPodAutoscaler: when multiple HPAs
  select the same set of Pods, scaling now will be disabled for those HPAs with
  the reason AmbiguousSelector. This change also covers a case when multiple HPAs
  point to the same deployment.'
  (#112011, @pbeschetnov)
• 'Pod Security admission: the pod-security warn level will now default to
  the enforce level.'
  (#113491, @tallclair)
• 'Promoted the APIServerIdentity feature to Beta. By default, each kube-apiserver
  will now create a Lease in the kube-system namespace. These lease objects can
  be used to identify the number of active API servers in the cluster, and may also
  be used for future features such as the Storage Version API.'
  (#113629, @andrewsykim)
• 'The iptables kube-proxy backend now process service/endpoint changes
  more efficiently in very large clusters.' (#110268, @danwinship)
• 'CSIMigrationvSphere was upgraded to GA and locked to true. Do not upgrade to K8s
  1.26 if you need Windows, or XFS, or raw block support until vSphere CSI Driver
  adds support for them in a version post v2.7.x.'
  (#113336, @divyenpatel)
• 'DelegateFSGroupToCSIDriver feature is GA.'
  (#113225, @bertinatto)
• 'NodeOutOfServiceVolumeDetach is now beta.'
  (#113511, @xing-yang)
• 'RetroactiveDefaultStorageClass feature is now beta.'
  (#113329, @RomanBednar)
• 'registered_metric_total will now report the number of metrics broken down by
  stability level and deprecated version.'
  (#112907, @logicalhan)
• A new DisableCompression field (default = false) has been added to kubeconfig under cluster info. When set to true, clients using the kubeconfig opt out of response compression for all requests to the apiserver. This can help improve list call latencies significantly when client-server network bandwidth is ample (>30MB/s) or if the server is CPU-constrained. (#112309, @shyamjvs)
• A new pod_status_sync_duration_seconds histogram is reported at alpha metrics stability that estimates how long the Kubelet takes to write a pod status change once it is detected. (#107896, @smarterclayton) [SIG Apps, Architecture, CLI, Cloud Provider, Cluster Lifecycle, Instrumentation, Network, Node, Release, Scheduling, Storage and Testing]
• API Server Tracing now includes a variety of new spans and span events. (#113172, @dashpole) [SIG API Machinery, Architecture, Auth, Instrumentation, Network, Node and Scheduling]
• API Server tracing now includes the latency of authorization, priorityandfairness, impersonation, audit, and authentication filters. (#113217, @dashpole)
• API Server tracing root span name for opentelemetry is changed from KubernetesAPI to HTTP GET. (#112545, @dims)
• Added --disable-compression flag to kubectl (default = false). When true, it opts out of response compression for all requests to the apiserver. This can help improve list call latencies significantly when client-server network bandwidth is ample (>30MB/s) or if the server is CPU-constrained. (#112580, @shyamjvs)
• Added a method StreamWithContext to remotecommand.Executor to support cancelable SPDY executor stream. (#103177, @arkbriar)
• Added a new feature gate CelValidatingAdmissionExtensibility to enable expression validation for Admission Control. (#112792, @cici37) [SIG API Machinery]
• Added alpha support for WindowsHostNetworking feature. (#112961, @marosset)
• Added alpha support for returning container and pod metrics from CRI, instead of cAdvsior. (#113609, @haircommander)
• Added categories column to the kubectl api-resources command's wide output (-o wide).
  Added --categories flag to the kubectl api-resources command, which can be used to filter the output to show only resources belonging to one or more categories. (#111096, @brianpursley) [SIG CLI]
• Added kubelet metrics to track the cpumanager cpu allocation and pinning (#112855, @fromanirh)
• Added new Golang runtime-related metrics to Kubernetes components:
• go_gc_cycles_automatic_gc_cycles_total
• go_gc_cycles_forced_gc_cycles_total
• go_gc_cycles_total_gc_cycles_total
• go_gc_heap_allocs_by_size_bytes
• go_gc_heap_allocs_bytes_total
• go_gc_heap_allocs_objects_total
• go_gc_heap_frees_by_size_bytes
• go_gc_heap_frees_bytes_total
• go_gc_heap_frees_objects_total
• go_gc_heap_goal_bytes
• go_gc_heap_objects_objects
• go_gc_heap_tiny_allocs_objects_total
• go_gc_pauses_seconds
• go_memory_classes_heap_free_bytes
• go_memory_classes_heap_objects_bytes
• go_memory_classes_heap_released_bytes
• go_memory_classes_heap_stacks_bytes
• go_memory_classes_heap_unused_bytes
• go_memory_classes_metadata_mcache_free_bytes
• go_memory_classes_metadata_mcache_inuse_bytes
• go_memory_classes_metadata_mspan_free_bytes
• go_memory_classes_metadata_mspan_inuse_bytes
• go_memory_classes_metadata_other_bytes
• go_memory_classes_os_stacks_bytes
• go_memory_classes_other_bytes
• go_memory_classes_profiling_buckets_bytes
• go_memory_classes_total_bytes
• go_sched_goroutines_goroutines
• go_sched_latencies_seconds (#111910, @tosi3k)
• Added new metric job_controller_terminated_pods_tracking_finalizer which can be used to monitor whether the job controller is removing Pod finalizers from terminated Pods after accounting them in Job status. (#113176, @alculquicondor)
• Added publishing events when enabling/disabling TopologyAwareHints. (#113544, @LiorLieberman)
• Added reconstruction of SELinux mount context after kubelet restart. Feature SELinuxMountReadWriteOncePod is now fully implemented and kubelet does not lose its cache of SELinux contexts after kubelet process restart. (#113596, @jsafrane)
• Added support for Evented PLEG feature gate. (#111384, @harche)
• Added the metric pod_start_sli_duration_seconds to kubelet. (#111930, @azylinski)
• Added validation for the --container-runtime-endpoint flag of kubelet to be non-empty. (#112542, @astraw99)
• Adds alpha --output plaintext protected by environment variable KUBECTL_EXPLAIN_OPENAPIV3 (#113146, @alexzielenski) [SIG CLI]
• Adds metrics force_delete_pods_total and force_delete_pod_errors_total in the Pod GC Controller. (#113519, @xing-yang) [SIG Apps]
• Azure File CSI migration is now GA. (#113160, @andyzhangx)
• Changed preemption_victims metric bucket from LinearBuckets to ExponentialBuckets. (#112939, @lengrongfu)
• Exposed health check SLI metrics on metrics/slis for apiserver. (#112741, @logicalhan)
• Extend the job job_finished_total metric by newreason` label and introduce a new job metric to count pod failures
  handled by pod failure policy with respect to the action applied. (#113324, @mimowo) [SIG Apps and Testing]
• Graduate ServiceIPStaticSubrange feature to GA. (#112163, @aojea)
• Graduated Kubelet CPU Manager to GA. (#113018, @fromanirh)
• Graduated Kubelet Device Manager to GA. (#112980, @swatisehgal)
• If ComponentSLIs feature gate is enabled, then /metrics/slis becomes available on kubelet, allowing you to scrape health check metrics. (#113030, @Richabanker) [SIG Node]
• If ComponentSLIs feature gate is enabled, then /metrics/slis now becomes available on cloud-controller-manager allowing you to scrape health check metrics. (#113340, @Richabanker)
• If more than one StorageClass is designated as default (via the "storageclass.kubernetes.io/is-default-class" annotation), choose the newest one instead of throwing an error.' (#110559, @danishprakash)
• In 'client-goSharedInformerFactory` will now support waiting for goroutines during shutdown.
  (#112200, @pohly)
• In kubeadm, command kubeadm join phase control-plane-prepare certs now supports to run with dry-run mode on it's own. (#113005, @chendave)
• Kube-apiserver: gzip compression switched from level 4 to level 1 to improve large list call latencies in exchange for higher network bandwidth usage (10-50% higher). This increases the headroom before very large unpaged list calls exceed request timeout limits. (#112299, @shyamjvs)
• Kubeadm: added show-join-command as a new separate phase at the end of kubeadm init. You can skip printing the join information by using kubeadm init --skip-phases=show-join-command. Executing only this phase on demand will throw an error because the phase needs dependencies such as bootstrap tokens to be pre-populated. (#111512, @SataQiu)
• Kubeadm: added the "--cleanup-tmp-dir" flag for kubeadm reset. It will cleanup the contents of /etc/kubernetes/tmp. The flag is off by default. (#112172, @chendave)
• Kubeadm: now supports image repository format validation. (#112732, @SataQiu)
• Kubeadm: sub-phases are now able to support the dry-run mode, e.g. kubeadm reset phase cleanup-node --dry-run (#112945, @chendave) [SIG Cluster Lifecycle]
• Kubeadm: tried to load CA cert from external CertificateAuthority file when CertificateAuthorityData is empty for existing kubeconfig. (#111783, @SataQiu)
• Kubectl shell completions for the bash shell now include descriptions. (#113636, @marckhouzam)
• Kubernetes is now built with Go 1.19.1 (#112287, @palnabarun) [SIG Release and Testing]
• Kubernetes is now built with Go 1.19.2 (#112900, @xmudrii) [SIG Release and Testing]
• Kubernetes is now built with Go 1.19.3. (#113550, @xmudrii)
• Logs of requests that were timed out by a timeout handler no longer contain a statusStack and logging error output fields. (#112374, @Argh4k)
• Metrics for RetroactiveDefaultStorageClass feature are now available. To see an attempt count for updating PVC retroactively with a default StorageClass see retroactive_storageclass_total metric and for total numer of errors see retroactive_storageclass_errors_total. (#113323, @RomanBednar)
• Promoted kubectl alpha events to kubectl events. (#113819, @soltysh)
• Promoting WindowsHostProcessContainers to stable. (#113476, @marosset)
• Scheduler now retries updating a pod's status on ServiceUnavailable and InternalError errors, in addition to net.ConnectionRefused error. (#111809, @Huang-Wei)
• Shell completion now shows plugin names when appropriate. Furthermore, shell completion will work for plugins that provide such support. (#105867, @marckhouzam)
• Switched kubectl to use github.com/russross/blackfriday/v2 (#112731, @pacoxu)
• The ExpandedDNSConfig feature has graduated to beta and is enabled by default. Note that this feature requires container runtime support. (#112824, @gjkim42) [SIG Network and Testing]
• The LegacyServiceAccountTokenNoAutoGeneration feature gate was promoted to GA. (#112838, @zshihang)
• The ProxyTerminatingEndpoints feature is now Beta and enabled by default. When enabled, kube-proxy will attempt to route traffic to terminating pods when the traffic policy is Local and there are only terminating pods remaining on a node. (#113363, @andrewsykim)
• The goroutines metric is newly added in the scheduler.
  It replaces scheduler_goroutines metric and it counts the number of goroutine in more places than scheduler_goroutine does. (#112003, @sanposhiho) [SIG Instrumentation and Scheduling]
• Updated cAdvisor to v0.46.0. (#113769, @bobbypage)
• Updated the Lease identity naming format for the APIServerIdentity feature to use a persistent name. (#113307, @andrewsykim)
• When ComponentSLIs feature gate is enabled, /metrics/slis becomes available on kube-scheduler, allowing you to scrape health check metrics. (#113026, @Richabanker)
• When ComponentSLIs feature gate is enabled, then /metrics/slis becomes available on kube-proxy allowing you to scrape health check metrics. (#113057, @Richabanker)
• When ComponentSLIs feature gate is enabled, then /metrics/slis becomes available on kube-controller-manager, allowing you to scrape health check metrics. (#112978, @logicalhan)
• When the alpha LegacyServiceAccountTokenTracking feature gate is enabled, secret-based service account tokens will have a kubernetes.io/legacy-token-last-used applied to them containing the date they were last used. (#108858, @zshihang) [SIG API Machinery, Auth and Testing]
• CSRDuration feature gate that graduated to GA in 1.24 and is unconditionally
  enabled now removed in v1.26.
  (#112386, @Shubham82)
• kubectl config view now automatically redacts any secret fields marked with a datapolicy tag. (#109189, @mpuckett159)

Documentation

• Clarified the default CFS quota period as being 100µs and not 100ms. (#111554, @paskal) [SIG Node]

Bug or Regression

• Added back unused flags on kubectl run command, which did not go through the required deprecation period before being removed. (#112243, @brianpursley)
• Added support for RSA and ECDSA format keys in preflight check on kubeadm. (#112508, @SataQiu)
• Allowed Label section in vSphere e2e cloud provider configuration. (#112427, @gnufied)
• Apiserver /healthz/etcd endpoint rate limits the number of forwarded health check requests to the etcd backends, answering with the last known state if the rate limit is exceeded. The rate limit is based on 1/2 of the timeout configured, with no burst allowed. (#112046, @aojea)
• Apiserver: used the correct error when logging errors updating managedFields. (#113711, @andrewsykim)
• Avoided propagating hosts search . into containers in /etc/resolv.conf. (#112157, @dghubble)
• Bump golang.org/x/net to v0.1.1-0.20221027164007-c63010009c80. (#112693, @aimuz)
• Bump runc to v1.1.4. (#113719, @pacoxu)
• Callers using DelegatingAuthenticationOptions can now use DisableAnonymous to disable Anonymous authentication. (#112181, @xueqzhan)
• Changed error message when resource is not supported by given patch type in kubectl patch. (#112556, @ardaguclu)
• Correct the calculating error in podTopologySpread plugin to avoid unexpected scheduling results. (#112507, @kerthcet)
• Etcd: Updated to v3.5.5. (#112489, @dims)
• Fixed Admission controllers that caused unnecessary significant load on `apiserver'. (#112696, @aimuz)
• Fixed DaemonSet to update the status even if it fails to create a pod. (#112127, @gjkim42)
• Fixed a bug where a change in the appProtocol for a Service did not trigger a load balancer update. (#112785, @MartinForReal) [SIG Cloud Provider and Network]
• Fixed a bug where the kubelet choose the wrong container by its name when running kubectl exec. (#113041, @saschagrunert)
• Fixed an ephemeral port exhaustion bug caused by improper connection management that occurred when a large number of objects were handled by kubectl while exec auth was in use. (#112017, @enj)
• Fixed an issue in winkernel proxier that causes proxy rules to leak anytime service backends are modified. (#112837, @daschott)
• Fixed bug in kubectl rollout history where only the latest revision was displayed when a specific revision was requested and an output format was specified. (#111093, @brianpursley)
• Fixed bug where dry run message was not printed when running kubectl label with --dry-run flag. (#111571, @brianpursley)
• Fixed code to ensure that pods running on nodes tainted with NoExecute continue to run when the PodDisruptionConditions feature gate is enabled. (#112518, @mimowo)
• Fixed cost estimation of token creation request for service account in Priority and Fairness. (#113206, @marseel)
• Fixed issue where the APIServer would panic on startup if an egress selector without a controlplane configuration is specified when using APIServerTracing. (#112979, @dashpole)
• Fixed list cost estimation in Priority and Fairness for list requests with metadata.name specified. (#112557, @marseel)
• Fixed race condition in GCE between containerized mounter setup in the kubelet. (#112195, @mattcary)
• Fixed relative CPU priority for pods where containers explicitly request zero cpu by giving the lowest priority instead of falling back to the cpu limit to avoid possible cpu starvation of other pods. (#108832, @waynepeking348)
• Fixed that disruption controller used to change the status of a stale disruption condition after 2 min when the PodDisruptionConditions feature gate is enabled. (#113580, @mimowo)
• Fixed the PodAndContainerStatsFromCRI feature, instead of supplementing with stats from cAdvisor. (#113291, @mengjiao-liu)
• Fixed the occasional double-counting of the job_finished_total metric. (#112948, @mimowo)
• For kubectl, --server-side now migrates ownership of all fields used by client-side-apply to the specified --fieldmanager. This prevents fields previously specified using kubectl from being able to live outside of server-side-apply's management and become undeleteable. (#112905, @alexzielenski)
• For raw block CSI volumes on Kubernetes, kubelet was incorrectly calling CSI NodeStageVolume for every single "map" (i.e. raw block "mount") operation for a volume already attached to the node. This change modified that behavior to ensure it is only called once per volume per node. (#112403, @akankshakumari393)
• Improved kubectl display of invalid request errors returned by the API server. (#112150, @liggitt)
• In 'kube-apiserver,x-kubernetes-list-type` validation is now enforced when updating status of custom resources.' (#111866, @pacoxu)
• In kube-apiserver, custom resources can now be specified in the --encryption-provider-config file and can be encrypted in etcd. (#113015, @ritazh)
• Increased the maximum backoff delay of the endpointslice controller to match the expected sequence of delays when syncing Services. (#112353, @dgrisonnet)
• Known issue: Job field .spec.podFailurePolicy.rules[*].onExitCode might be ignored if the Pod is deleted before it terminates. (#113856, @alculquicondor)
• Kube-apiserver: DELETECOLLECTION API requests are now recorded in metrics with the correct verb. (#113133, @sxllwx)
• Kube-apiserver: redirect responses are no longer returned from backends by default. Set --aggregator-reject-forwarding-redirect=false to continue forwarding redirect responses. (#112193, @jindijamie) [SIG API Machinery and Testing]
• Kube-apiserver: redirects from backend API servers are no longer followed when checking availability with requests to /apis/$group/$version (#112772, @liggitt) [SIG API Machinery and Testing]
• Kube-apiserver: resolved a regression that treated 304 Not Modified responses from aggregated API servers as internal errors. (#112526, @liggitt)
• Kube-proxy no longer falls back from ipvs mode to iptables mode if you ask it to do ipvs but the system is not correctly configured. Instead, it will just exit with an error. (#111806, @danwinship) [SIG Network]
• Kube-scheduler: added taints filtering logic consistent with TaintToleration plugin for PodTopologySpread plugin. (#112357, @SataQiu)
• Kubeadm will cleanup the stale data on best effort basis. Stale data will be removed when each reset phase are executed, default etcd data directory will be cleanup when the remove-etcd-member phase are executed. (#110972, @chendave) [SIG Cluster Lifecycle]
• Kubeadm: fixed a bug when performing validation on ClusterConfiguration networking fields. (#112751, @SataQiu)
• Kubeadm: when a sub command is needed but not provided for a kubeadm command, print a help screen instead of showing a short message. (#111277, @chymy)
• Kubectl apply: warning that kubectl will ignore no-namespaced resource pv & namespace in a future release if the namespace is specified and allowlist is not specified. (#110907, @pacoxu)
• Kubectl: fixed a bug where kubectl convert did not pick the right API version (#112700, @SataQiu)
• Kubelet now cleans up the Node's cloud node IP annotation correctly if you
  stop using --node-ip. (In particular, this fixes the problem where people who
  were unnecessarily using --node-ip with an external cloud provider in 1.23,
  and then running into problems with 1.24, could not fix the problem by just
  removing the unnecessary --node-ip from the kubelet arguments, because
  that wouldn't remove the annotation that caused the problems.) (#112184, @danwinship) [SIG Network and Node]
• Kubelet: Fixed a startup crash in devicemanager. (#113021, @rphillips)
• Kubelet: fixed log spam from kubelet_getters.go Path does not exist. (#112650, @rphillips)
• Kubelet: fixed nil pointer in reflector start for standalone mode. (#113501, @pacoxu)
• Kubelet: when there are multi option lines in /etc/resolv.conf, merge all options into one line in a pod with the Default DNS policy. (#112414, @pacoxu) [SIG Network and Node]
• Log messages and metrics for the watch cache are now keyed by <resource>.<group> instead of go struct type. This means e.g. that *v1.Pod becomes pods. Additionally, resources that come from CustomResourceDefinitions are displayed as the correct resource and group, instead of *unstructured.Unstructured. (#111807, @ncdc)
• Moved LocalStorageCapacityIsolationFSQuotaMonitoring back to Alpha. (#112076, @rphillips)
• NOTE (#113749, @jpbetz) [SIG API Machinery]
• Nested MountPoints are now grouped correctly on all cases. (#112571, @claudiubelu)
• Pod failed in scheduling due to expected error will be updated with the reason of SchedulerError rather than Unschedulable. (#111999, @kerthcet)
• Pod logs using --timestamps are not broken up with timestamps anymore. (#113481, @rphillips)
• Removed of raising an error when setting an annotation with the same value, just ignore it. (#109505, @zigarn)
• Resolved an issue that caused winkernel proxier to treat stale VIPs as valid. (#113521, @daschott)
• The ResourceVersion returned in objects from delete responses is now consistent with the ResourceVersion contained in the delete watch event. (#113369, @wojtek-t)
• The kube-scheduler and kube-controller-manager now use server side apply to set conditions related to pod disruption. (#113304, @mimowo) [SIG API Machinery, Apps and Scheduling]
• The errors in k8s.io/apimachinery/pkg/api/meta now support for the stdlibs errors.Is matching, including when wrapped. (#111808, @alvaroaleman)
• The metrics etcd_request_duration_seconds and etcd_bookmark_counts now differentiate by group resource instead of object type, allowing unique entries per CustomResourceDefinition, instead of grouping them all under *unstructured.Unstructured. (#112042, @ncdc)
• The pod admission error message was improved for usability. (#112644, @vitorfhc) [SIG Node]
• The time duration of a failed or unschedulable scheduling attempt will be longer, it
  now includes the time duration of the unreserve operation. (#113113, @kerthcet)
• Updated kube-proxy to restart in case it detects that the Node assigned pod.Spec.PodCIDRs have changed. (#111344, @aojea)
• Updated creation of LoadBalancer services, for there to be fewer AWS security group rules in most cases. (#112267, @sjenning)
• Updated the system-validators library to v1.8.0 (#112026, @pacoxu)
• Updates golang.org/x/text`` tov0.3.8`` to fix CVE-2022-32149 (#112989, @ameukam)
• Volume mount cleanup now considers only plugin directory and not the entire kubelet root (#112607, @mattcary)
• kubectl now escapes terminal special characters in output. This fixes CVE-2021-25743.
  (#112553, @dgl)

Other (Cleanup or Flake)

• 'Promoted cronjob_job_creation_skew metric to stable to follow the cronjob v2 controller,
  the following metrics had their name updated to match metrics API guidelines:
  • cronjob_job_creation_skew_duration_seconds -> job_creation_skew_duration_seconds.' (#113008, @soltysh)
• 'Promoted job-related metrics to stable to follow IndexedJobs GA. The following metrics have their name updated to match metrics API guidelines:
• job_sync_total -> job_syncs_total
• job_finished_total -> jobs_finished_total' (#113010, @soltysh)
• 'kubelet_kubelet_credential_provider_plugin_duration was renamed to kubelet_credential_provider_plugin_duration
  and kubelet_kubelet_credential_provider_plugin_errors was renamed to kubelet_credential_provider_plugin_errors.'
  (#113754, @logicalhan)
• A new API server flag --encryption-provider-config-automatic-reload was added to control when the encryption config should be automatically reloaded without needing to restart the server. All KMS plugins are now merged into a single healthz check at /healthz/kms-providers when reload is enabled, or when only KMS v2 plugins are used. (#113529, @enj)
• Added a --prune-allowlist flag that can be used with kubectl apply --prune. This flag now replaces and functions the same as the --prune-whitelist flag, which has been deprecated. (#113116, @brianpursley)
• Added a kubernetes_feature_enabled metric which will tell you if a feature is enabled. (#112652, @logicalhan)
• Deprecated the following kubectl run flags, which are ignored if set: --cascade, --filename, --force, --grace-period, --kustomize, --recursive, --timeout, --wait. (#112261, @brianpursley)
• Dropped support for the Container Runtime Interface (CRI) version v1alpha2, which means that container runtimes just have to implement v1. (#110618, @saschagrunert)
• E2e: tests can now register callbacks with ginkgo.BeforeEach, ginkgo.AfterEach or ginkgo.DeferCleanup directly after creating a framework instance and are guaranteed that their code is called after the framework is initialized and before it gets cleaned up. ginkgo.DeferCleanup replaces f.AddAfterEach and AddCleanupAction which got removed to simplify the framework. (#111998, @pohly)
• Introduce ComponentSLIs alpha feature-gate for component SLIs metrics endpoint. (#112884, @logicalhan) [SIG API Machinery]
• Kube scheduler Component Config release version v1beta3 is deprecated in v1.26 and will be removed in v1.29,
  also v1beta2 will be removed in v1.28. (#112257, @kerthcet) [SIG Scheduling]
• Kube-scheduler: the DefaultPodTopologySpread, NonPreemptingPriority, PodAffinityNamespaceSelector and PreferNominatedNode feature gates that graduated to GA in v1.24 and were unconditionally enabled have been removed in v1.26. (#112567, @SataQiu)
• Kubeadm: removed the UnversionedKubeletConfigMap feature gate. The feature has been GA and locked to enabled since v1.25. (#113448, @pacoxu)
• Kubeadm: removed the toleration for the node-role.kubernetes.io/master taint from the CoreDNS deployment of kubeadm. With the 1.25 release of kubeadm the taint node-role.kubernetes.io/master is no longer applied to control plane nodes and the toleration for it can be removed with the release of 1.26. You can also perform the same toleration removal from your own addon manifests. (#112008, @pacoxu)
• Kubeadm: removed the usage of the --container-runtime=remote flag for the kubelet during kubeadm init/join/upgrade. The flag value remote had been the only possible value since dockershim was removed from the kubelet. (#112000, @pacoxu)
• Locked ServerSideApply feature gate to true with the feature already being GA. (#112748, @wojtek-t)
• Refactored test/e2e/framework so that the core framework is smaller. Optional functionality like resource monitoring, log size monitoring, metrics gathering and debug information dumping must be imported by specific e2e test suites. Init packages are provided which can be imported to re-enable the functionality that traditionally was in the core framework. If you have code that no longer compiles because of this PR, you can use the script from a commit message to update that code. (#112043, @pohly)
• Release-note (#111708, @yangjunmyfm192085) [SIG Apps, Instrumentation and Network]
• Removed PodOverhead feature gate as the feature is in GA since v1.24. (#112579, @SergeyKanzhelev)
• Removing Windows Server, Version 20H2 flavors from various container images. (#112924, @marosset)
• Renamed the feature gate for CEL in Admission Control to ValidatingAdmissionPolicy. (#113735, @cici37)
• Reworded log message upon image garbage collection failure to be more clear. (#112631, @tzneal) [SIG Node]
• Scheduler dumper now exposes a summary to indicate the number of pending pods in each internal queue. (#111726, @Huang-Wei) [SIG Scheduling and Testing]
• Service session affinity timeout tests are no longer required for Kubernetes network plugin conformance due to variations in existing implementations. New conformance tests will be developed to better express conformance in future releases. (#112806, @dcbw) [SIG Architecture, Network and Testing]
• The IndexedJob and SuspendJob feature gates that graduated to GA in 1.24 and were unconditionally enabled have been removed in v1.26. (#112589, @SataQiu)
• The e2e.test binary no longer emits JSON structs to document progress. (#113212, @pohly)
• The in-tree cloud provider for OpenStack (and the cinder volume provider) has been removed. Please use the external cloud provider and csi driver from cloud-provider-openstack instead. (#67782, @dims)
• The metric etcd_db_total_size_in_bytes is renamed to apiserver_storage_db_total_size_in_bytes. (#113310, @logicalhan) [SIG API Machinery]
• Updated cri-tools to [v1.25.0(https://github.com/kubernetes-sigs/cri-tools/releases/tag/v1.25.0) (#112058, @saschagrunert)
• GlusterFS in-tree storage driver which was deprecated in kubernetes 1.25 release is now removed entirely in 1.26. (#112015, @humblec)

Dependencies

Added

• cloud.google.com/go/datastore: v1.1.0
• cloud.google.com/go/firestore: v1.1.0
• cloud.google.com/go/pubsub: v1.3.1
• github.com/OneOfOne/xxhash: v1.2.2
• github.com/alecthomas/template: fb15b89
• github.com/alecthomas/units: f65c72e
• github.com/armon/consul-api: eb2c6b5
• github.com/armon/go-metrics: f0300d1
• github.com/armon/go-radix: 7fddfc3
• github.com/bgentry/speakeasy: v0.1.0
• github.com/bketelsen/crypt: 5cbc8cc
• github.com/cenkalti/backoff/v4: v4.1.3
• github.com/cespare/xxhash: v1.1.0
• github.com/client9/misspell: v0.3.4
• github.com/coreos/bbolt: v1.3.2
• github.com/coreos/etcd: v3.3.13+incompatible
• github.com/coreos/go-systemd: 95778df
• github.com/coreos/pkg: 399ea9e
• github.com/dgrijalva/jwt-go: v3.2.0+incompatible
• github.com/dgryski/go-sip13: e10d5fe
• github.com/fatih/color: v1.7.0
• github.com/go-gl/glfw: e6da0ac
• github.com/go-logr/stdr: v1.2.2
• github.com/google/martian: v2.1.0+incompatible
• github.com/grpc-ecosystem/grpc-gateway/v2: v2.7.0
• github.com/hashicorp/consul/api: v1.1.0
• github.com/hashicorp/consul/sdk: v0.1.1
• github.com/hashicorp/errwrap: v1.0.0
• github.com/hashicorp/go-cleanhttp: v0.5.1
• github.com/hashicorp/go-immutable-radix: v1.0.0
• github.com/hashicorp/go-msgpack: v0.5.3
• github.com/hashicorp/go-multierror: v1.0.0
• github.com/hashicorp/go-rootcerts: v1.0.0
• github.com/hashicorp/go-sockaddr: v1.0.0
• github.com/hashicorp/go-syslog: v1.0.0
• github.com/hashicorp/go-uuid: v1.0.1
• github.com/hashicorp/go.net: v0.0.1
• github.com/hashicorp/golang-lru: v0.5.1
• github.com/hashicorp/hcl: v1.0.0
• github.com/hashicorp/logutils: v1.0.0
• github.com/hashicorp/mdns: v1.0.0
• github.com/hashicorp/memberlist: v0.1.3
• github.com/hashicorp/serf: v0.8.2
• github.com/jpillora/backoff: v1.0.0
• github.com/jstemmer/go-junit-report: v0.9.1
• github.com/kr/logfmt: b84e30a
• github.com/kr/pty: v1.1.1
• github.com/magiconair/properties: v1.8.1
• github.com/mattn/go-colorable: v0.0.9
• github.com/mattn/go-isatty: v0.0.3
• github.com/miekg/dns: v1.0.14
• github.com/mitchellh/cli: v1.0.0
• github.com/mitchellh/go-homedir: v1.1.0
• github.com/mitchellh/go-testing-interface: v1.0.0
• github.com/mitchellh/gox: v0.4.0
• github.com/mitchellh/iochan: v1.0.0
• github.com/oklog/ulid: v1.3.1
• github.com/pascaldekloe/goe: 57f6aae
• github.com/pelletier/go-toml: v1.2.0
• github.com/posener/complete: v1.1.1
• github.com/prometheus/tsdb: v0.7.1
• github.com/ryanuber/columnize: 9b3edd6
• github.com/sean-/seed: e2103e2
• github.com/shurcooL/sanitized_anchor_name: v1.0.0
• github.com/spaolacci/murmur3: f09979e
• github.com/spf13/cast: v1.3.0
• github.com/spf13/jwalterweatherman: v1.0.0
• github.com/spf13/viper: v1.7.0
• github.com/subosito/gotenv: v1.2.0
• github.com/ugorji/go: v1.1.4
• github.com/xordataexchange/crypt: b2862e3
• go.opentelemetry.io/contrib/propagators/b3: v1.10.0
• go.opentelemetry.io/otel/exporters/otlp/internal/retry: v1.10.0
• go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc: v1.10.0
• go.opentelemetry.io/otel/exporters/otlp/otlptrace: v1.10.0
• gopkg.in/ini.v1: v1.51.0
• gopkg.in/resty.v1: v1.12.0
• rsc.io/binaryregexp: v0.2.0
• rsc.io/quote/v3: v3.1.0
• rsc.io/sampler: v1.3.0

Changed

• dmitri.shuralyov.com/gpu/mtl: 28db891 → 666a987
• github.com/antlr/antlr4/runtime/Go/antlr: f25a4f6 → v1.4.10
• github.com/aws/aws-sdk-go: v1.38.49 → v1.44.116
• github.com/container-storage-interface/spec: v1.6.0 → v1.7.0
• github.com/containerd/ttrpc: v1.0.2 → v1.1.0
• github.com/cpuguy83/go-md2man/v2: v2.0.1 → v2.0.2
• github.com/dnaeon/go-vcr: v1.0.1 → v1.2.0
• github.com/docker/docker: v20.10.17+incompatible → v20.10.18+incompatible
• github.com/docker/go-units: v0.4.0 → v0.5.0
• github.com/emicklei/go-restful/v3: v3.8.0 → v3.9.0
• github.com/felixge/httpsnoop: v1.0.1 → v1.0.3
• github.com/fsnotify/fsnotify: v1.4.9 → v1.6.0
• github.com/go-kit/log: v0.1.0 → v0.2.0
• github.com/go-logfmt/logfmt: v0.5.0 → v0.5.1
• github.com/go-openapi/jsonreference: v0.19.5 → v0.20.0
• github.com/google/cadvisor: v0.45.0 → v0.46.0
• github.com/google/cel-go: v0.12.4 → v0.12.5
• github.com/google/go-cmp: v0.5.6 → v0.5.9
• github.com/google/pprof: 94a9f03 → 4bb14d4
• github.com/gopherjs/gopherjs: fce0ec3 → 0766667
• github.com/inconshreveable/mousetrap: v1.0.0 → v1.0.1
• github.com/karrick/godirwalk: v1.16.1 → v1.17.0
• github.com/konsorten/go-windows-terminal-sequences: v1.0.2 → v1.0.3
• github.com/matttproud/golang_protobuf_extensions: v1.0.1 → v1.0.2
• github.com/moby/sys/mountinfo: v0.6.0 → v0.6.2
• github.com/moby/term: 3f7ff69 → 39b0c02
• github.com/onsi/ginkgo/v2: v2.1.4 → v2.4.0
• github.com/onsi/gomega: v1.19.0 → v1.23.0
• github.com/opencontainers/runc: v1.1.3 → v1.1.4
• github.com/prometheus/client_golang: v1.12.1 → v1.14.0
• github.com/prometheus/client_model: v0.2.0 → v0.3.0
• github.com/prometheus/common: v0.32.1 → v0.37.0
• github.com/prometheus/procfs: v0.7.3 → v0.8.0
• github.com/smartystreets/assertions: v1.1.0 → b2de0cb
• github.com/spf13/afero: v1.6.0 → v1.2.2
• github.com/spf13/cobra: v1.4.0 → v1.6.0
• github.com/stretchr/objx: v0.2.0 → v0.4.0
• github.com/stretchr/testify: v1.7.0 → v1.8.0
• go.etcd.io/etcd/api/v3: v3.5.4 → v3.5.5
• go.etcd.io/etcd/client/pkg/v3: v3.5.4 → v3.5.5
• go.etcd.io/etcd/client/v2: v2.305.4 → v2.305.5
• go.etcd.io/etcd/client/v3: v3.5.4 → v3.5.5
• go.etcd.io/etcd/pkg/v3: v3.5.4 → v3.5.5
• go.etcd.io/etcd/raft/v3: v3.5.4 → v3.5.5
• go.etcd.io/etcd/server/v3: v3.5.4 → v3.5.5
• go.opentelemetry.io/contrib/instrumentation/github.com/emicklei/go-restful/otelrestful: v0.20.0 → v0.35.0
• go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc: v0.20.0 → v0.35.0
• go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp: v0.20.0 → v0.35.0
• go.opentelemetry.io/otel/metric: v0.20.0 → v0.31.0
• go.opentelemetry.io/otel/sdk: v0.20.0 → v1.10.0
• go.opentelemetry.io/otel/trace: v0.20.0 → v1.10.0
• go.opentelemetry.io/otel: v0.20.0 → v1.10.0
• go.opentelemetry.io/proto/otlp: v0.7.0 → v0.19.0
• go.uber.org/goleak: v1.1.10 → v1.2.0
• golang.org/x/crypto: 3147a52 → v0.1.0
• golang.org/x/exp: 85be41e → 6cc2880
• golang.org/x/mobile: e6ae53a → d2bd2a2
• golang.org/x/mod: 86c51ed → v0.6.0
• golang.org/x/net: a158d28 → 1e63c2f
• golang.org/x/oauth2: d3ed0bb → ee48083
• golang.org/x/sys: 8c9f86f → v0.3.0
• golang.org/x/term: 03fcf44 → v0.3.0
• golang.org/x/text: v0.3.7 → v0.5.0
• golang.org/x/tools: v0.1.12 → v0.2.0
• google.golang.org/grpc: v1.47.0 → v1.49.0
• google.golang.org/protobuf: v1.28.0 → v1.28.1
• k8s.io/gengo: c02415c → c0856e2
• k8s.io/klog/v2: v2.70.1 → v2.80.1
• k8s.io/kube-openapi: 67bda5d → 172d655
• k8s.io/system-validators: v1.7.0 → v1.8.0
• k8s.io/utils: ee6ede2 → 1a15be2
• sigs.k8s.io/apiserver-network-proxy/konnectivity-client: v0.0.32 → v0.0.33
• sigs.k8s.io/yaml: v1.2.0 → v1.3.0

Removed

• github.com/ajstarks/svgo: 644b8db
• github.com/auth0/go-jwt-middleware: v1.0.1
• github.com/boltdb/bolt: v1.3.1
• github.com/fogleman/gg: 0403632
• github.com/getkin/kin-openapi: v0.76.0
• github.com/go-ozzo/ozzo-validation: v3.5.0+incompatible
• github.com/golang/freetype: e2365df
• github.com/gophercloud/gophercloud: v0.1.0
• github.com/gorilla/mux: v1.8.0
• github.com/heketi/heketi: v10.3.0+incompatible
• github.com/heketi/tests: f3775cb
• github.com/jung-kurt/gofpdf: 24315ac
• github.com/kr/fs: v0.1.0
• github.com/lpabon/godbc: v0.1.1
• github.com/mvdan/xurls: v1.1.0
• github.com/pkg/sftp: v1.10.1
• github.com/remyoudompheng/bigfft: 52369c6
• github.com/russross/blackfriday: v1.5.2
• github.com/urfave/negroni: v1.0.0
• go.opentelemetry.io/contrib/propagators: v0.20.0
• go.opentelemetry.io/contrib: v0.20.0
• go.opentelemetry.io/otel/exporters/otlp: v0.20.0
• go.opentelemetry.io/otel/oteltest: v0.20.0
• go.opentelemetry.io/otel/sdk/export/metric: v0.20.0
• go.opentelemetry.io/otel/sdk/metric: v0.20.0
• gonum.org/v1/gonum: v0.6.2
• gonum.org/v1/netlib: 7672324
• gonum.org/v1/plot: e2840ee
• modernc.org/cc: v1.0.0
• modernc.org/golex: v1.0.0
• modernc.org/mathutil: v1.0.0
• modernc.org/strutil: v1.0.0
• modernc.org/xc: v1.0.0
• rsc.io/pdf: v0.1.1