Kubernetes Annual Report 2021

This is a summary of the Kubernetes project’s contributor community and activities.

This report documents both quantitative measures of community health (project milestones and snapshot) as well as qualitative measures of the community as reported by community leaders and contributors to the project.

Authors: The Kubernetes Steering Committee
Editor: Tim Bannister, @sftim, SIG Docs Tech Lead

Accolades

On behalf of the project, we’d like to say thanks to the following contributors, community groups, and ecosystem for the following highlights. As always, give praise to an effort in #shoutouts on Kubernetes slack.

Feature Maturity and Stability

Thanks to our groups for continuing the efforts from 2020, many SIGs continue to drive long standing beta features to graduate to stable.

Several features that graduated to stable or made notable progress include:

• CSI Plugins on Windows Nodes graduated to stable in v1.22 (SIG Windows)
• Generic ephemeral inline volumes graduated to stable in v1.23 (SIG Storage)
• IPv4/IPv6 dual-stack graduated to stable in v1.23 (SIG Network)
• Metrics stability framework graduated to stable in v1.21 (SIG Instrumentation)
• Server-side Apply graduated to stable in v1.22 (SIG API Machinery)
• Client credential plugins graduated to stable in v1.22 (SIG Auth)
• Kubetest2 is maturing (SIG Testing)
• CSI migration has been an effort that has been going on for several releases. It involves SIG Storage, SIG Cloud Provider, and contributors across many cloud providers and storage vendors to work together and move in-tree volume plugins to out-of-tree CSI drivers.

Other project processes are maturing, too, and not just the code. A new way to cast votes in elections (like Steering Committee and more) runs via Elekto. The Kubernetes Monthly Community meeting was rebooted to include discussions and not just presentations.

Showing up and sticking around

Climbing the contributor ladder is a trust-building exercise as much as it is a skills one. Sticking around, chopping wood, and carrying water is the main formula for growing OWNERs and leaders on the project.

An example of an intentional contributor ladder growth effort happened in SIG Docs by growing its contributor and reviewer base in 2021. They introduced a shadow program for PR Wrangling and dedicated more time to being active in the #sig-docs Slack channel, helping grow the community. SIG Docs also worked on a leadership transition strategy to bring community members into leadership roles via a specialized six-month group mentorship program. They were able to cultivate leaders for the SIG and some of its subgroups, adding new co-chairs and tech leads.

SIG CLI deserves another great shoutout for having long-standing Chairs and Tech Leads take the emeritus route while growing new leaders into the roles. Thanks for your service and great job, team!

Amping up Kubernetes security

Every group in Kubernetes has a responsibility to make sure we are putting our best foot forward with supply chain security. Accolades to all of SIG Release, SIG Auth, and SIG Security for their sustained efforts in this area that include:

• generating SBOMs,
• compliance with SLSA 3 standards,
• artifact signing,
• rearchitecting release process from bash to Go,
• and adding new features, tests and checks to the release process – these were missing from the original anago tooling (binary verification, CVE disclosure, building from custom branches and repositories).

Alongside those improvements specifically to supply-chain security, we’ve seen:

• improvements to end-user security documentation.
• Pod credentials are auto-revoked when pods complete or are deleted (1.22+)
• CSI drivers can use pod-scoped credentials using [Service Account Token for CSI Driver] (1.22+)
• Certificates can be requested with shorter lifetimes (1.22+)
• Pods can listen on low ports without requiring a root user or expanded capabilities (1.22+)
• Pod Security admission has graduated to beta and is enabled by default (1.23+)

Things that no longer spark joy

There are plenty of processes, tools, and policy that are put together in a project lifecycle that eventually need to be phased out for whatever reason. A contributor painpoint that we’ve had with a codebase this large is bazel. The crews in SIG Testing and SIG Release put in a lot of time and attention on removing bazel from kubernetes/kubernetes. There are some pieces left in kubernetes/test-infra but needless to say, we are on the road to moving on in our build processes.

Growing Windows support

Thanks to the SIG Windows team and surrounding groups for their efforts in growing the support in this space! A true testament to the power of the ecosystem. They have more upcoming work to do and we are looking forward to seeing their growth in 2022 and beyond.

Details:

• Implemented hostProcess container support in Kubernetes (now in beta) and promoted adoption in multiple open source communities
• Defined the kubectl subcommand for fetching node-level logs.
• Made the developer UX for Windows transparent with sig-windows-dev-tools.
• Defined operational readiness standards for Windows.
• Defined the pod OS field.

Themes / Trends

Prioritizing Quality

The project saw an increase in regression-related backports in the two most recent releases (1.22 and 1.23). Many of these regressions were related to a couple types of changes:

• Changes to add features or fix unrelated bugs in areas that are complex and undertested
• Changes that were intended to be mechanical refactors that accidentally modified behavior

What have we done?

Adjustments are being made in several areas throughout the release cycle to reverse this trend:

• Encouraging SIG and component leads to track and consider the health of existing code/components when planning and accepting new feature proposals.
• Guiding proposal authors to provide more specific test plans, and reminding them that stabilizing or improving the existing health of the area they want to change may be required before their proposal can proceed.
• Clarifying the standards reviewers and approvers should apply during implementation.
• Improving test signal by cleaning up unowned or permanently failing CI jobs, to give better visibility to test flakes or failures introduced during a development cycle.
• Adjusting release schedules to ensure time for at least two release candidate builds, and giving time for feedback on those builds. Thanks to reports from users testing pre-release builds, regressions were fixed before both the 1.23.0 and 1.24.0 releases!

Independent contributors play a critical role on the project

A misconception is that this project is just cloud providers maintaining it; however, one of our biggest contributor bases are “independent” that is, not affliated with an organization.

There is space for everyone here.

What have we done?

Connect folks to jobs! While not all indie contributors are looking for employment, many are. This year we worked with CNCF to add a feature to the [cncf.jobs.io site], which allows employers to indicate a percentage of time that they would support upstream activities. The Kubernetes project needs more contributors with employer-backed time, and this was a great step toward that goal. Aligning contributors with the right incentives is the sweet spot for lasting contributions.

Areas to research?

As part of upcoming surveys, we will poll the indpedenet contributors on various topics and how we can support them more. As always, we welcome feedback via SIG Contributor Experience or for high level governance matters, the Steering Committee.

Niche contributor documentation /help-wanted

With one of the largest decentralized distributed open-source projects out there, expect our contribution guides to be in-depth and extensive. k8s.dev/guide is our primary guide; no matter where you contribute to the project, you start there. But because the project is so large, some groups have other style guides, code review processes, and more that define how they do business and operationalize. This is an important part of our [values]. Same thing at big employers: everyone gets the standard onboarding docs, but your department might have an additional “here’s how to get work done” document floating around.

Many of our groups reported in that they have a hard time keeping this information up to date, if they even have this kind of documentation at all. This is a great way to get involved if you are new to a group! Want to become an OWNER? Set someone up for success behind you by creating documentation for your area.

What have we done?

In late 2020, SIG Leads were tasked with auditing their area specific documentation, with many removing out-dated information and creating follow-up items calling out things things that should be documented. These audits made it easy for companies to bring on Tech Writers to help shore up this needed documentation.

Additional processes have been put in place, such as a documentation review as part of the annual report process should ensure that project contributing docs remain (relatively) up-to-date.

Areas to research

Updating documentation is usually a good onboarding path for interns and new contributors but this can get murky with some of the complexities of the code and doc set. It can take up to 3 months to onboard on to the project before suggestion and submitting changes. Is there a program that SIGs could create as an onboarding path towards OWNERship here?

Burnout

The topics of burnout and workload management are frequent in our Leads and group meetings, Steering Committee, and even the growing voices at ecosystem level during talks and events at KubeCon/CloudNativeCons. This is an industry wide problem that we need to solve together. With a mix of reasons why contributors are burning out, there is no one “solve all” solution here. Aligning incentives to grow OWNERs seems to be one of the main challenges in this space.

What have we done

• Reducing the release cadence. While this wasn’t the only reason for having 3 and not 4 releases in a year, it factored in. Check out this blog for more on the release cadence change.
• Continue to talk about how we can do better in our monthly Chairs and Tech Leads and keeping our doors open for contributors to have these conversations with us.
• Educating contributors and OWNERs on the use of “Emeritus” and why it’s ok to take breaks.

Growth Areas

This section represents an area of the project that we’ve identified as having a growth opportunity or need.

What’s project health anyway?

Some of the more mature groups like SIG Instrumentation or those with industry open-source veterans can quickly identify areas of their components that need help and tell stories about what’s flourishing. Yet, it can be challenging to establish universal indicators of “project health” in a project as large and diverse as Kubernetes. We need to develop these indicators to provide signal to the leads so that they may detect, pre-empt, or bubble up this information to keep their area healthy.

Every group needs more reviewers

If you’ve been watching open source news over the last year, supply chain security has made headlines. According to OpenSSF and other security groups, code reviews are an important piece to putting prioritization on security. Kubernetes strives for two reviews (a reviewer and approver) to match our values of quality and high-trust. Reviewers are a key part of our success in quality code and documentation changes upstream. Reviewer is the next step on our contributor ladder post Org Member; however, you don’t need to be in an OWNERs file as a Reviewer to review. Anyone can leave a comment or an “/lgtm” (looks good to me!). As we try and grow top level approvers, sticking around and getting into an OWNERs file as a Reviewer is crucial for our sustainability as a project to keep pace.

As a data point, in 1.19, the average daily PR reviewers were down -15% across Kubernetes Orgs and -24% in the Kubernetes/Kubernetes repo while still receiving the same amount of issues and PRs. If you are trying to get a change into Kubernetes, this will effect you.

The 9 to 5 contributor is almost over and we have to adjust

Only a handful of our OWNERs, some of our most active contributors, will tell you that they work 80-100% upstream. These folks know the codebase and docs extensively and are some of our most experienced reviewer eyes. But anecdotally, the number of experienced and very active core folks able to contribute has decreased in recent years. Ensuring continuity and growing more people into senior roles is becoming critical for the project to continue to deliver a robust and reliable releases.

In 2022 we have started discussion the CNCF Governing Board to see how we can tackle long term strategies together.

• How can we incentivize growth in this area of sustainers?
• How can we surface areas of risk that require investment to keep going?
• Are there additional actions we might take in the short term?

This reporting process and its summary

This process takes us 6 months. This is both not sustainable and not helpful. Between our groups being heads down shipping reliable and stable enhancements, societal challenges and atrocities that affect us such the war in Ukraine, not to mention a global pandemic, we have a lot of leniency for groups getting this together. Our contributors live all over the world, have day jobs, and might have their own challenges that they are living through.

With the theme of burnout, how can we support groups without bogging them down with paperwork? How can we communicate our needs at a level that hears and takes action on them? We need to build more tooling in this area and will be putting out a call for interns soon. Have other advice for us?

Help Wanted

SIG API Machinery

• Client libraries
• Triage
• Sticking around and growing into contributor ladder roles

SIG Apps

SIG Apps is looking to grow their pool of [reviewers and appprovers]. Contributors looking at growing into these roles can join the SIG Apps / SIG CLI Review club.

SIG Auth

SIG Auth keeps a running list of KEPs that need help and tracks their progress on their SIG Auth project board. They are also looking for help in enhancing their own onboarding guide and PR review guidance.

Specifically SIG Auth is looking for help in these initatives:

• KMS-Plugin: Improvements
• Specifying multiple webhooks in the kube-apiserver authorization chain
• Structured config for OIDC authentication
• Audit logging improvements
• Renaming the system:masters group

SIG CLI

SIG CLI has three areas where they’re looking for more help:

• Optimizing kubectl memory usage.
• Contributors that can dedicate time and grow into maintainer roles (reviewer / approver) for Kustomize.
• SIG CLI’s docs for both kubectl and kustomize need additional support. They are built off cli-experimental, are outdated, need SEO improvements and migrated to the new kustomize.io and kubectl.io domains. Alignment with k8s.io docs would be useful too.

SIG Cloud Provider

SIG Cloud Provider needs more support from cloud providers to extract the provider specific code from the main Kubernetes repo. Spinning them out will create a smaller and more secure core, while enabling the Cloud Providers to release and update their components on their own cadence.

SIG Contributor Experience

The SIG is looking for a full time community manager. Also, there are three subprojects where SIG Contributor Experience could use assistance.

• GitHub Administration
  • The GitHub Admin team needs more new membership coordinators. These coordinators are current contributors that help serve as a friendly face to newer, prospective community members, guiding them through the process to request membership to a Kubernetes GitHub organization.
• Community Management Automation
  • Auto upload recordings from Zoom to YouTube
    • Every community group (SG/WG/Committee) records and publishes their meetings for transparency. The current process is frought with manual work and toil, frequently leading to recordings being published in batches long after the meeting was held.
  • Workspace Automation
    • The Kubernetes project as a whole relies heavily on Google Workspace, mailing lists, calendars and docs. There is an ongoing effort to streamline these processes and bring them under a single domain for central management.
• Mentoring Program Management and new Roles
  • Group Mentoring Coordinator
    • SIG Contributor Experience facilitates and aids other groups with their in-project mentoring initatives. With increased interest in mentoring from other SIGs and WGs, there is a need for a dedicated coordinator to spin up and manage these initatives.
  • 3rd Party Mentoring Coordinator
    • SIG Contributor Experiences works with a number of external mentorship programs such as Outreachy, Google Summer of Code, LFX and more. As there are a number of external parties with a variety of deadlines and requirements, the SIG is looking for a dedicated person(s) to manage and facilicate working with these external mentorship programs.

SIG Docs

There are two initatives where SIG Docs could use assistance.

The blog subproject is particularly short on resources and attention. At the moment a very small pool of active editors are the constraint / most critical resource for article publication. One editor is involved in the majority of published articles; other editors are perhaps even more stretched with other Kubernetes contributions and involvement with other SIGs.

The Ukrainian localization team is primarily worked on by people based in Ukraine, where the ongoing and intensifying conflict creates challenges that take priority over open source contribution.

SIG Instrumentation

The Prometheus Adapter subproject is in need of additional contributors that can grow and commit to becoming reviewer/approvers. It currently only has one active approver and is used a number of endusers.

SIG K8s Infra

SIG K8s Infra is looking for engineers to help build tools to automate more of the project’s infrastructure and to help migrate more tests to community owned resources. Please show up to #sig-k8s-infra on Slack to help with this important group. (You can get an invitation to Slack from https://slack.k8s.io/)

SIG Release

SIG Release is looking for more contributors in a number of subprojects

• kubernetes-sigs/bom – A utility to generate SPDX-compliant Bill of Materials manifests
• kubernetes-sigs/downloadkubernetes – The tool that generates the site downloadkubernetes.com, making it easier to download Kubernetes release artifacts
• kubernetes-sigs/mdtoc – A small utility that generates a Table of Contents in Markdown.
• kubernetes-sigs/release-notes – Generator for Kubernetes release notes
• kubernetes-sigs/zeitgeist – language-agnostic dependency checker
• kubernetes/repo-infra – A collection of common Kubernetes repo project tools

SIG Scalability

SIG Scalability is looking to grow their contributors base across all their subprojects. Good entry points for new scalability contributors are the Scalability Test Framework and Performance Tests & Validaiton subproject.

SIG Scheduling

The Scheduler Simulator, a project that allows for simulating and testing of scheduling profiles/plugins needs more reviewers and approvers.

SIG Security

The SIG Security docs subproject is always looking for security-minded contributors of all experience levels to share their learning and knowledge with the community. This subproject has consistently been a place where people merge their first Kubernetes PRs. There’s always room for continuous improvement in our documentation, and contributing to this provides an opportunity to learn more about Kubernetes security while helping everyone run their clusters more safely. We’re really proud of the way Docs encourages and welcomes new contributors, and we’d love to encourage you to become a part of it!

SIG Storage

SIG Storage is broadly looking for more help fixing bugs and growing reviewers across the board.

Full time contributors in the following areas:

• Write more tests and monitor test grid health
• Improve out of tree test framework
• Enhance CSI release tools
• Improve docs on CSI and general storage architecture
• Help with initial PR triage

SIG Testing

SIG Testing is broadly looking for more contributors that can become reviewers / approvers.

Looking for help in the following projects:

• Boskos– Resource management service used by Kubernetes CI that provides reservation and lifecycle management
• Kubetest2 – Framework for launching and running end-to-end tests on Kubernetes.
• Prow – Main Kubernetes CI system
  • Cannot continue to maintain https://monitoring.prow.k8s.io due to Grafana license change. Kubernetes has switched to using Google Cloud Monitoring, but cannot make the dashboards publicly visible.
• Triage – Tool for gathering and reporting similar test failures across all CI jobs
• Kettle – Tool that collections CI job information and loads it into BigQuery for analysis

SIG Windows

SIG Windows has several areas it is looking for support, the largest being related to Windows Storage support/CSI Proxy.

Looking for full time contributors to help with:

• Testing hostProcess implementations on several windows apps
• Improving Windows dev tools to help grow the Windows contributor community
• Hardening the CSI proxy and CSI support ecosystem
• Performance testing Kubernetes on Windows

WG API Expression

No Report

WG Data Protection

• End users come to meetings and contribute to design/implementation of the features we are working on

WG IoT/Edge

Spinning down inside of Kubernetes and heading to CNCF level

WG Multitenancy

No specific help needed! Contributions are still welcome.

WG Structured Logging

• Graduate Contextual Logging to Beta and GA
• Graduate Deprecation of klog specific flags to GA
• Graduated Structured Logging to GA
• All code in kubernetes/kubernetes repository is migrated to Structured Logging API

Initiatives

SIG API Machinery

API Machinery is evaluating the potential for generics in go1.19. There are a number of other initiatives.

SIG Apps

• Significant improvements were made to the Job API, along with finally driving CronJobs to stable and introduced several long-desired features. This work is expected to continue through 2022 to finish rounding out the Job API.
  • CronJobs promoted to stable (1.21)
  • Indexed Job promoted to beta (1.22)
  • Suspend Job promoted to beta (1.22)
  • Job tracking without lingering Pods promoted to beta (1.23)
• Stability and availability improvements were made across several controllers, with larger improvements being made to both DaemonSets and StatefulSets.
  • minReadySeconds for StatefulSets promoted to beta (1.22)
  • Allow DaemonSets to surge during update like Deployments promoted to beta (1.22)
• Additional improvements have been made to conformance testing promotions.

Kubernetes Enhancements:

• Stable
  • 19 – CronJob to Stable – 1.21
  • 85 – PodDisruptionBudget to GA – 1.22
  • 592 – TTL After Finished – 1.23
• Beta
  • 2185 – Random Pod Selection on ReplicaSet Downscale – 1.22
  • 1591 – Allow DaemonSets to surge during update like Deployments – 1.22
  • 2214 – Indexed Job – 1.22
  • 2232 – Suspend Job – 1.22
  • 2255 – ReplicaSet Pod Deletion Cost – 1.22
  • 2307 – Job tracking without lingering Pods – 1.23
  • 2599 – minReadySeconds for StatefulSets – 1.23
  • 2926 – Mutable Node Scheduling Directives for Jobs – 1.23
• Alpha
  • 2185 – Random Pod Selection on ReplicaSet Downscale – 1.21
  • 1591 – Allow DaemonSets to surge during update like Deployments – 1.21
  • 2214 – Indexed Job – 1.21
  • 2232 – Suspend Job – 1.21
  • 2255 – ReplicaSet Pod Deletion Cost – 1.21
  • 2307 – Job tracking without lingering Pods – 1.22
  • 2599 – minReadySeconds for StatefulSets – 1.22
  • 1847 – Auto delete PVCs created by StatefulSet – 1.23
  • 2879 – Track ready Pods in Job status – 1.23

SIG Auth

• Pod Security admission has graduated to beta and is enabled by default. The admission configuration version has been promoted to pod-security.admission.config.k8s.io/v1beta1 in v1.23.
• The PodSecurityPolicy API is deprecated in v1.21, and will no longer be served starting in v1.25.
• Marking audit.k8s.io/v1[alpha|beta]1 versions as deprecated and warning if a version other than audit.k8s.io/v1 was passed to the kube-apiserver flags --audit-log-version and --audit-webhook-version in v1.21.
• PodSecurityPolicy only stores “generic” as allowed volume type if the GenericEphemeralVolume feature gate is enabled
• RunAsGroup feature for Containers in a Pod graduates to GA in v1.21
• RootCAConfigMap feature graduates to GA in v1.21
• The ServiceAccountIssuerDiscovery feature has graduated to GA, and is unconditionally enabled in v1.21.
• CSIServiceAccountToken graduates to GA in 1.22
• Mark net.ipv4.ip_unprivileged_port_start as safe sysctl in v1.22
• BoundServiceAccountTokenVolume graduates to GA in v1.22
• Kubernetes client credential plugins feature graduates to stable in v1.22. The GA feature set includes improved support for plugins that provide interactive login flows. The in-tree Azure and GCP authentication plugins have been deprecated in favor of out-of-tree implementations.
• Kube-apiserver --service-account-issuer can be specified multiple times now, to enable non-disruptive change of issuer starting v1.22
• The CertificateSigningRequest.certificates.k8s.io API supports an optional expirationSeconds field to allow the client to request a particular duration for the issued certificate. The default signer implementations provided by the Kubernetes controller manager will honor this field as long as it does not exceed the --cluster-signing-duration flag starting v1.22.
• Aggregate write permissions on events to edit and admin role starting v1.22
• The kubelet now reports distinguishes log messages about certificate rotation for its client cert and server cert separately to make debugging problems with one or the other easier.starting v1.22
• A new field omitManagedFields has been added to both audit.Policy and audit.PolicyRule so cluster operators can opt in to omit managed fields of the request and response bodies from being written to the API audit log starting v1.23
• Adds --as-uid flag to kubectl to allow uid impersonation in the same way as user and group impersonation starting v1.23
• Stable
  • 1205-bound-service-account-tokens – 1.22
  • 1393-oidc-discovery – 1.21
  • 2907-secrets-store-csi-driver – 1.0.0
  • 541-external-credential-providers – 1.22
  • 1687-hierarchical-namespaces-subproject – stable
• Beta
  • 2579-psp-replacement – 1.23
  • 2784-csr-duration – 1.22

SIG CLI

SIG CLI made progress on a number of initiatives in 2021:

• kubectl events alpha command.
• KRM Functions subproject started.
• New changes to leadership.
  • @KnVerey brought on as new Co-Chair and Tech Lead.
  • @soltysh stepped down from Co-Chair to focus on Tech Lead.
  • @pwittrock moved to emeritus.
  • @monopole moved to emeritus for Kustomize.
• Started a new monthly Kustomize bug scrub.
• Upgraded the version of Kustomize that ships with kubectl.
• Implemented native Go shell completions.
• Replicated donated kubectl.io and kustomize.io to the project.
• IBM donated the Kui project.
• The Kustomize Roadmap
• Refactoring old kubectl commands

Kubernetes Enhancements

• Stable
  • KEP-555 – Server-side apply – 1.22
• Beta
  • KEP-1441 – kubectl debug – 1.20, continued to evolve the beta through the year
  • KEP-859 – kubectl command metadata in http request headers – 1.22
• Alpha
  • KEP-1440 – kubectl events – 1.23
  • KEP-2227 – Default container annotation to be used by kubectl – 1.21
• Pre-alpha
  • KEP-2985 – Public KRM functions registry
  • KEP-2953 – Kustomize Plugin Graduation
• Rejected
  • KEP-2229 – Use XDG Base Directory Specification

SIG Cloud Provider

No Report

SIG Contributor Experience

During 2021, SIG Contributor Experience continued to provide a number of services to the project and it’s 75,000 contributors. Some achievements include the migration of the large public kubernetes-dev mailing list to to managed a project owned Google workspace, developing Elekto, a replacement for the CIVS voting system, and the seamless migration of the CLA system to EasyCLA.

SIG Contributor Experience also ran the North America Contributor Summit, the end of year Contributor Celebration, ran three successful mentoring cohorts, and the [Contributor Comms team] automated and started using the @k8scontributors twitter account to reach 5700 follows with a number of them being contributors.

Contributor Experience (“ContribEx”) is a service and program orientated SIG. Most of its initiatives cover long term services for the Kubernetes project.

SIG Docs

• SIG Docs put meaningful effort into growing its contributor and reviewer base in 2021, introducing a shadow program for PR Wrangling as well as dedicating more time to being active via our Slack community channel.
• Ahead of the dockershim removal in the Kubernetes 1.24 release, SIG Docs has been collaborating with various community members and the CNCF towards ensuring updation and creation of content in the form of documentation, blog posts etc. With weekly meetings and a project board to track progress, this enabled SIG Docs to invite contributors across experience levels to help us keep the Kubernetes website updated and relevant ahead of the major change.
• Alongside growing our contributor base, SIG Docs also worked on a leadership transition strategy to bring community members into leadership roles. Via a specialized six month mentorship program expertly led by Steering Committee member Paris Pittman, SIG Docs was able to grow its leadership cohort for the main SIG, as well as some of its subgroups, adding new co-chairs and tech leads.
  • SIG Docs google group
  • Call for help sent to dev@kubernetes.io, kubernetes-sig-leads, kubernetes-sig-docs
  • Announcement of new roles and leadership nominations
• Localization Subproject: SIG Docs is working on formalizing the localization work that has been ongoing for some time, with appointed leads of this initiative as well as recognizing the contributions of various community members across the different languages the Kubernetes website has been translated into. This subproject will be finalized by Q1 2022, with all active localizations informed and updated.
• New Contributor Ambassador Program: As a continued focus to grow the SIG Docs contributor base, they’re working on a specalized role that aims to support new and would-be contributors get up to speed with our processes and workflows. This role would be capped at six months for it to be shared amongst the community, with this feeding into a possible reviewer funnel as contributors get more comfortable with providing feedback to others.

Kubernetes Enhancements:

• 1326 – Doc policies for third party content

SIG Instrumentation

SIG Instrumentation had several large accomplishments in 2021.

• Formed WG Structured Logging. Successfully migrated multiple components to structured logs and graduated feature to beta
• Added tracing support to the Kubernetes API server and began work on Kubelet tracing
• Graduated the metrics stability framework
• Put into practice Bi-weekly triage meeting

Kubernetes Enhancements:

• Stable
  • 1209 – Metrics Stability – 1.21
  • 1933 – Prevent logging secrets via static analysis – 1.23
• Beta
  • 1602 – Structured Logging – 1.23
  • 1748 – Pod resource requests/limits metrics – 1.22
• Alpha
  • 2305 – Metrics Cardinality Enforcement – 1.21
  • 647 – API Server Tracing – 1.22
  • 2845 – Deprecate klog-specific flags in k8s components – 1.23
• Pre-alpha
  • 2831 – Kubelet OpenTelemetry Tracing – alpha in 1.24

SIG Release

After finalizing the rewrite of the release process from bash into golang, the release engineering team focused its efforts on two main areas:

• Improving the release automation on two fronts:
  • Adding new features, tests and checks to the release process which were missing from the original release tooling (binary verification, CVE disclosure, building from custom branches and repositories).
  • Consolidating the codebases of new repositories which SIG Release brought under its responsibility. The range of new repositories it is consolidating go from critical projects (like the image promoter) to less important repositories such as https://downloadkubernetes.com.
• Hardening the Kubernetes Supply Chain via key efforts:
  • SBOM Generation
  • SLSA 3 compliance
  • Artifact signing

Kubernetes Enhancements

• Alpha
  • KEP-3027 – SLSA Level 3 Compliance in the Kubernetes Release Process – v1.23

SIG Scalability

SIG Scalability spent significant effort on validating the scalability and reliability impact of many Kubernetes features across 2021; growing the scalability tests of large services to cover 1000+ pods. Additional work was put into adding support for modules in tests, measuring the availability of the api-server and adding support for measuring cilium propagation delay & dns latency.

Kubernetes Enhancements:

• Beta
  • 1040 – Priority and Fairness for API Server Requests – 1.23
• Alpha
  • 647 – APIServer Tracing – 1.22
  • 1669 – Proxy Terminating Endpoints – 1.22
  • 2464 – Kubetest2 CI Migration – 1.21

SIG Scheduling

During 2021, SIG Scheduling focused on improving the overall performance of the scheduler, some highlights include:

• Efficient re-queueing of pods, significantly cutting the number of failed scheduling cycles
• Improvements to preemption performance
• Simplified plugin configuration in component config
• Created the Scheduler simulator
• Performance improvements and benchmarking
• Code refactorings and cleanups
• Enhancements to node resource-based scoring (see 101946 and 101822)

Kubernetes Enhancements:

• Stable
  • 2249 – Multi-scheduling Profiles – 1.22
  • 1845 – Prioritization on Volume Capacity – 1.22
• Beta
  • 2249 – Namespace Selector for Pod Affinity – 1.22
  • 1923 – Prefer Nominated Node – 1.22
  • 2458 – Resource Fit Scoring Strategy – 1.22
  • 2891 – Simplified Scheduler Config – 1.22
  • 785 – Scheduler Component Config API – 1.23
  • 2926 – Job Mutable Scheduling Directives – 1.23

SIG Security

Most of SIG Security’s initiatives are out of scope for KEPs, and instead are largelty service and process oriented.

In 2021 they had several notable achievements:

• Kickstarted the security self-assessment project aimed at providing guidance and a framework for Kubernetes subprojects to perform their own security self-assessment.
• Implemented vulnerability scanning for build-time dependences in container images.
• Scoped the work and went through the RFP process to select a vendor to perform the project’s second external third-party audit.
• Bootstrapped the Security Docs subproject aimed at improving the security content in Kubernetes documentation.

Kubernetes Enhancements:

• Stable
  • 1933 – Defend against logging secrets via static analysis – 1.23
• Beta
  • 2579 – PSP Replacement Policy – 1.23
  • 1933 – Defend against logging secrets via static analysis – 1.21
• Alpha
  • 2579 – PSP Replacement Policy – 1.22
• Pre-alpha
  • 2763 – Ambient Capabilities

SIG Storage

In addition to a number of KEPs, SIG Storage has been working on CBT (Change Blocking Tracking)] in conjunction with the Data Protection WG

Kubernetes Enhancements:

• Stable
  • 1412 – Immutable Secrets and ConfigMaps – v1.21
  • 1682 – Skip Volume Ownership Change – v1.23
  • 1698 – generic ephemeral inline volumes – v1.23
  • 1855 – Service Account Token for CSI Driver – v1.22
  • 1122 – CSI Windows – v1.22
• Beta
  • 1472 – Storage Capacity Constraints for Pod Scheduling – v1.21
  • 1885 – In-tree Storage Plugin to CSI Migration – Azurefile – v1.21
  • 2317 – Provide fsgroup of pod to CSI driver on mount – v1.23
• Alpha
  • 1432 – Volume Health Monitor – v1.21
  • 1790 – Recover from volume expansion failure – v1.23
  • 2485 – ReadWriteOncePod PersistentVolume AccessMode – v1.22
  • 2589 – In-tree Storage Plugin to CSI Migration – Portworx – v1.23
  • 2644 – Honor Persistent Volume Reclaim Policy – v1.23
  • 2923 – In-tree Storage Plugin to CSI Migration – Ceph RBD – v1.23
• Pre-alpha
  • Object Storage API (COSI)

SIG Testing

SIG Testing is largely service-oriented and their initatives are not often tracked as KEPs, yet they have had a number of achievements in the past year improving testing infrastructure and features.

Highlights of some of these initiatives include:

• kubetest2 is feature-complete and stable
• Automated secret syncing for ProwJob secrets
• Developed GitHub App support for Prow
• Improved job config validation (strict field checks, build cluster existence)
• Improved in-repo Prow config support and performance
• Added support for Prow config file sharding to better manage approval permissions
• Developed new monitoring stack solution that doesn’t rely on Grafana (GKE Workload Metrics + Cloud Monitoring)
• Added OSS-Fuzz integration
• Developed private repo multitenancy (multiple private front ends)
• Completed the removal of Bazel from kubernetes/kubernetes
• Removed most of Bazel from the kubernetes/test-infra repo

Kubernetes Enhancements

• Stable
  • KEP 2420 – Reducing Kubernetes Build Maintenance – 1.23
• Beta
  • KEP 2539 – Continuously Deploy K8s Prow – 1.21
  • KEP 2464 – kubetest2 CI migration – 1.23

SIG Windows

SIG Windows has made progress on a number of lower level features. They implemented hostProcess container support (now in beta) which has now been adopted by a number of other OSS Projects. Other achievements include better node-level logging, improving the Windows Kubernetes developer experience with sig-windows-dev-tools, defining a set of operational readiness standards, and removed Dockershim from Windows nodes.

Kubernetes Enhancements

• Stable
  • 1122 – windows-csi-support – v1.22
• Beta
  • 1981 – Windows Privileged Container Support – v1.23
  • 2802 -Identify Windows pods at API admission level authoritatively – v1.23
• Alpha
  • 1981 – Windows Privileged Container Support – v1.22
  • 2802 -Identify Windows pods at API admission level authoritatively – v1.23
• Pre-alpha (Targeting 1.24)
  • 2578 – Windows Operational Readiness

WG API Expression

• Server-side Apply went GA in 1.22
• Started new initiatives around OpenAPI v3
• Enum for built-in types in OpenAPI
• Server-side field validation

WG Data Protection

The Data Protection WG identified the missing building blocks for supporting data protection in Kubernetes and published in their whitepaper. Features such as Volume Backups, Change Block Tracking, Volume Populator, Volume Group Group Snapshot, and Backup Repositories are owned by SIG Storage. Features such as Quiesce and Unquiesce Hooks are owned by SIG Node, with SIG Storage and SIG Apps participating. Features such as Application Snapshots and Backups are owned by SIG Apps, with SIG Storage participating. We will continue to work on them until all the missing pieces are available in Kubernetes.

The following items have been under development and have not yet been captured in a KEP:

• Change Block Tracking (CBT) API design
• Volume Replication
• Data Protection for Managed Services Presentation
• Snapshot policy (immutable snapshot
• Volume Snapshot GA phases
• Kubernetes Data Protection with Velero

WG IoT/Edge

The IoT/Edge Working Group is moving to the CNCF ecosystem.

WG Structured Logging

In 2021 The structured logging WG migrated kubelet, kube-scheduler, kube-proxy to the new standard format.

Kubernetes Enhancements Beta:

• Structured Logging v1.23 Alpha:
• Deprecation of klog specific flags v1.23