Standardizing Continuous Delivery

Software has a complicated relationship with standards. The motto "move fast and break things" embodies developer desire to get their creation to the world as quickly as possible. This has meant favoring workarounds and local optimization at the expense of permanent design practices. Standards may be adopted just to solve a specific problem, a temporary choice plucked from a vast sea of competing paradigms.

Deployment teams are in an unenviable position. They must ensure working and secure releases while keeping up with new features. So they've also adopted tool after tool to match pace while complying with security, legal, and business requirements.

But tools only address the how of implementation, not that the what and why of a business process. A standard is an accepted pattern for doing something. It's a what (and sometimes a why), in a world laser focused on the how.

Standards avoidance is understandable. Standards restrict by design, and are exercises in calculated risk. Engineers might feel less innovative in the near term, but the investment is in the ability to scale and ensure security for the business over the long run. However, well-meaning efforts to shift left has sometimes meant release faster at all costs. Continuous delivery efforts that focus exclusively on delivery speed is tempted to avoid standards entirely. That's a recipe for catastrophe.

How do teams balance standards with continuous innovation? Here are a few pointers, followed by practical steps for platform teams.

Good standards are outcomes, not specific tools

Don't confuse standards with tools. A standard is like a specification; it describes a result. Teams can then be flexible in their implementation.

Engineers use technology standards every day. The Internet protocol defines the attributes required for communication over a network. But you can use any software or hardware you want, as long as it adheres to the standard. Similarly, an international working group standardizes the C programming language, but there are dozens of compilers, with some orgs famously writing their own.

Because standards are results driven, teams can reduce complexity to the limit of standard adherence, while also pursuing a north star for automation. For example, a QA organization implementing SQuaRE now has a framework to replace manual tests with automated unit, integration, and security testing.

Good standards are shared and enforced

Standards are useful when teams know and understand them. Sure, you can create standards in a smoke-filled room, keep the details under lock and key, and simply report whether workflows pass or fail. But then teams tend to throw processes at the wall hoping something will stick. Instead, document and templatize your standards, share them, and then enforce them via automated policies. A couple examples:

• The security org wants to proactively prevent High and Critical CWE's from ending up in the product. They work with engineering to write pipeline templates for static vulnerability analysis to be included during CI. The templates' inclusion is enforced by automated policy. Both the templates and policies are maintained in a shared repository maintained by the security org.
• The release org requires a manual sign-off before promoting services from staging to production. The team creates a policy requiring pipelines to include a manual approval from a release team member before deploying to prod. Meanwhile, attribute-based access control grants read-only permissions to production resources for anyone outside the release or infra team.

The takeaway is that templates help enforce the standards and standardize the enforcement. Defaulting to transparency broadens team member knowledge, and even allows crowdsourced iteration of the standards themselves (e.g. an engineer can submit a PR to improve a security policy).

Finally, as organizations grow, they'll find...

Good standards trade overhead now for significant ROI later

Standards are an investment with an upfront cost. That cost can mean educating team members and refactoring workflows to comply with the standards. There is also near-term opportunity cost where teams feel less productive due the overhead of needing to adapt to the new requirements.

Over time, though, properly chosen standards should always produce a return on investment. For engineering standards, it would be improved scalability, reusability, and performance that the standard enforces. For security and compliance, it would be reduced security incidences and legal liability.

Engineers already understand this as individual developers. Variable consistency, documentation, and unit tests all take more work initially, but improve developer productivity in the long run. Likewise, well-chosen standards at the team or org level is similar to raising the bumpers on a bowling lane; they're guardrails keeping teams headed toward the goals set by the business.

A Practical Example

Consider this scenario:

"Each deployment to production must include sign-off by a member of the release team. These sign-offs must be auditable and reportable to management."

This example is an outcome-driven standard. The outcome is a documented human approval of every prod deployment. Given this standard, there are many possible implementations. A low-tech example could be a release engineer hand signing and dating a clipboard before pressing "Go" to launch the deployment pipeline.

A more modern and maintainable implementation would leverage templates, policies, and RBAC/ABAC to document and enforce the standard.

• Templates are configurations that are defined once and shared among teams. Here, a template might be a deployment pipeline configuration adhering to the standard: manual approval before prod deploy.
• Policies are software rules enforcing compliance. Here a policy could be a ruleset specifying that the deployment template must be used for production environments, otherwise the pipeline fails. There maybe additional policies forcing failure if anything circumvents the approval.
• RBAC/ABAC are role and attribute-based access control, respectively. They are fine-grained permissions determining who does what in the deployment platform, and which resources they can use. Here, there may be a user or service account granted exclusive access to the templates, pipelines, and infrastructure needed to approve and perform the deployment, following principles of least privilege.

Every deployment platform worth its salt should at minimum support the above. Here's how the three work together in Harness.

Templates

Harness supports shareable pipeline templates at multiple scopes. The release team might create a template for a "deploy to prod stage", which includes the required approval and then the deployment action.

The template has an underlying text configuration for version control and auditing changes. For example, the approval may look as follows.

# Standalone or part of a larger deployment template
         - step:
              type: HarnessApproval
              name: Prod Approval
              identifier: Prod_Approval
              spec:
                approvalMessage: Please review the following information and approve deployment to production.
                includePipelineExecutionHistory: true
                isAutoRejectEnabled: false
                approvers:
                  userGroups:
                    - release_admin
                  minimumCount: 1
                  disallowPipelineExecutor: false
                approverInputs: []
              timeout: 1d

This helps accomplish the what of the standard's implementation.

Policies

If templates are the what, policies are the how. Policies ensure the template is actually used. Harness has a rich policy mechanism leveraging Open Policy Agent, allowing teams to write very granular rules. There's also a library of reusable sample policies. We also highly recommend this tutorial to get some practice creating and enforcing policies.

For instance, a policy might enforce that any pipeline running against environment "prod" uses the DeployProd template. Alternatively, you might have blanket enforcement of templates for all pipelines, and the DeployProd template can only target environment prod.

If templates represent the implementation's what, and policies the enforcement how, access control handles the who.

RBAC/ABAC

Harness supports RBAC with ABAC as an extension. Teams group entities and components into resource groups (ABAC), whose access is then granted to users, user groups, or service accounts. In our scenario, there might be a resource group defined with only the resources needed for a production deployment.

That resource group is then assigned to a dedicated service account, and only that account, in line with least privilege practices.

Keeping well-architected standards

Standards exist to help teams communicate good ideas and enforce good practices. When standards exist purely as bureaucratic inertia, innovation slows down and the business can be threatened by teams resorting to unsafe workarounds. Well designed standards are outcome driven and delegate implementation to individual teams and contributors.

Harness as a platform is designed to address these concerns by balancing rapid release automation with granular compliance enforcement. Teams can therefore maintain a standard of deploying faster, with visibility and control.