CD and the Modern Software Shop

My first job after college was as a field tech for a local IT provider. I reviewed firewall logs, and repaired printers, and deployed servers. I reassured worried users that their computer is likely safe despite scary popups.

I wasn’t at an innovative SaaS startup, rather it was more like Office Space. At the time I wasn't familiar with modern software toolchains like containers, developer portals, CI/CD, and end-to-end monitoring. I certainly didn't understand the benefits of continuous delivery.

But what the job did provide me was time with users in the "real" world. That is, employees and business owners who want software to just work so they can do their jobs and go home. We're talking trainers strapping on heart rate monitors, factory employees clocking in and out, and support reps updating a ticket status.

The typical user is generally not the type to spend 30 hours trying to port their favorite emacs package to Rust. Rather, it’s a specialist owner doing focused work. Their paycheck depends on the network staying up and their spreadsheet app not crashing. The following conclusion, then, reflects what I experienced at that MSP and in each job since.

People want their software tools to just work, with minimal friction and cognitive overhead.

This is utterly non-controversial for consumer software. Companies design smartphones for to be immediately useful upon powering them on. Personal devices and apps are walled gardens of pleasure and convenience, making choices and even "thinking" for their end users.

Now think of the enterprise; does the previous sentence still apply? Consumers desire guidance to a world of pure imagination. But organizations typically want complete control over their processes, tools, and data.

The result is layers of runbooks and delivery processes, where friction and cognitive overhead are kind of the point. It proves the organization is precise and cautious!

A great contradiction in the software trade has been the creation of delightful software using incredibly undelightful development practices and deployment tools. How did we get here? Are engineers as happy with the tools they use to deliver software as users are with the software itself? And have we made any progress toward this ideal?

I've noticed that the organizations that do manage to approach this utopian ideal engage in the following:

Everything is automated.

Everything is version controlled.

Everything is immutable.

Everything is observable.

To better understand how the above have become best practices, let's discuss how we arrived here.

Lessons from the physical world

The current software release process derives from the principles used in manufacturing physical products. This approach originated from Frederick Taylor's concepts on work management during the early 1900s. Taylor believed that business is like solving an engineering problem. He thought that the management should have control over everything needed to produce the best results at the lowest cost.

Taylorism heavily standardized and specialized production methods. On the other hand, firms implementing Taylorism became paternalistic. As supply chains got more complicated, having a small group of managers making decisions became a problem. The scientific bureaucracy slowed down innovation.

What’s the analog to developing and deploying software? Probably the 1968 NATO Software Engineering Conference. The meeting aimed to predict the difficulties in treating computer science as a mature engineering field. They coined the phrase “the software crisis," which might be summarized as implementing large systems is hard, and changing them without breaking things is even harder.

The meeting attendees advocated deployment method we'd consider quaint today. Computer experts would completely design the software specification before writing any code. They'd then hand the specification to a programmer, who'd then code without any further decision making. It was scientific management through and through.

We now know that’s not a recipe for success. Deployment environments aren't fully closed systems. For any reasonably complex system, the deployment action will impact the software performance itself.

Yet firms continued to embrace Taylorism and created extensive decision trees and playbooks to manage the division of labor. We’re talking literal books of every possible decision a system administrator should make for any eventuality.

Back in the physical world, lean manufacturing took off in Japan in the 1950s. Companies like Toyota implemented a pull system that was laser focused on minimizing waste and excess inventory. In the 1980s, the book The Goal popularized these methods, and western companies used them to compete with affordable Asian vehicles.

Still, software remained in Waterfall-land. Sequential development took precedence over real-time user demands.

The infrastructure needed for iteration and modern version control just wasn’t in place yet (CVS wasn't first released until 1986). Feedback loops were long, and tools comprised proprietary stacks of compilers, linkers, and databases. Development teams worked on centralized mainframes or minicomputers, with manual builds, testing, and deployment.

Early and often

Things finally changed in the early 1990s. NFSNET upgrades allowed TCP/IP and opened the doors to the World Wide Web.

Communities and software tools began to flourish and make use of this new communication backbone. In 1995, JUnit came out as a unit test framework to help automate testing. Apache HTTP server helped dot-com companies start using practices similar to today's continuous integration (CI) and CD.

Key to this innovation was a Cambrian explosion of open source software. It was all developed in view of the public, accompanied by a culture of rapid experimentation. Eric S. Raymond’s The Cathedral and the Baazar presented a mantra of “Release early, release often. And listen to your customers.”

As companies started adding open source to own tech stacks, the culture followed. Extreme programming and the Agile Manifesto brought in new project management styles. These styles emphasized user feedback loops instead of top-down directed flows like Waterfall. “Hacker culture” had now become a business model, and one that worked.

Moving up the stack

After the dot-com crash, Web 2.0 changed how developers built and released software. The internet became a network of platforms for users to generate and post content. Software was now a service in and of itself, rather than simply a supporting communication medium. Downtime was no longer acceptable and operation teams achieved their moment in the sun.

Marc Andreessen proclaimed that software was now eating the world, and would affect all areas of industry, digital and physical. Books like The Phoenix Project served as sequels to The Goal, but applied to software delivery and operations.

Cloud computing arrived when Amazon started opening its internal tools to external customers (thereby launching Amazon Web Services). Organization realized they could innovate faster by focusing on their core competencies while outsourcing lower layers of the stack. All companies could now be software companies, and tools became more abstract as "plug and play" models became common.

Teams began shifting-left to identify constraints as early as possible in the development process. In practice this means treating everything as software-defined.

Think of how infrastructure has undergone significant changes in the past thirty years. It has evolved from mainframes to servers, to VMs, to containers, to serverless setups and now AI-generated configs. Now think of the tools that have sprung up to support and maintain these abstractions. They include configuration management, continuous integration, infrastructure as code, and large language models.

Every problem is now a software problem, and modern continuous delivery requires thinking as such. This brings us back to the four principles identified earlier.

Everything is Automated

Scientific management, lean manufacturing, and agile all put a heavy premium on documentation to standardize release cycles. Before computers, this was just a means for later workers to have a standard process to follow. Now, computers can implement runbooks and deployment playbooks as coded specifications.

In other words, program or automate it if possible. Runbooks become configuration management and infrastructure as code. Deployment playbooks become imperative or declarative pipelines. Best practices involve using a range of tools, both from vendors and open source, along with their reference architectures.

Because we’re thinking of operations as software development, is also means…

Everything is version controlled

Change management is a mature set of approaches with roots in Taylorism and lean manufacturing. ITIL standards brought it to technology in the 1980s. Today, once we represent deployment procedures as software, we can store everything in version control.

We document our versioned changes via commits in a code repo. Branches, code review, and pull requests represent our change management procedure.

We can go a step further and implement GitOps. This enforces that our version controlled codebase is our declared source of truth. Our production deployment must look like the HEAD of our main branch, for example. Our chain of commits representing our version history then moves us from mutable, difficult to measure deployment environments, to…

Everything is Immutable

IImmutable means it can’t be changed. Immutable deployments mean we can’t change a deployment after it happens; we must do a new one. If that sounds uncomfortable, think about how manufacturing companies fix product defects. Make small changes in the process that improves the next product off the line.

Another way to think of this is the pets vs cattle analogy. We think of deployments and infrastructure as livestock - interchangeable and subject to iterative improvements.

Remember version control is our source of truth. A change means a new commit on a branch in the repo. Kicking off a new deployment pipeline means parsing the full commit. The new deployment is then the delta between the new commit and the previous one.

We’re swapping out cattle, not trying to apply ad hoc changes outside our automated source of truth.

Immutable therefore implies that our changes are more traceable. And because we fix our environments in place after deployment, we can move toward…

Everything is Observable

If we representing our deployment process as code, we can only document (and thereby automate) what we have know. Observability provides continuous feedback loops and data to drive our automation choices. Tools like Prometheus provide a core metrics exporter we can filter into dashboards or organize into DORA metrics.

Full observability isn’t a panacea. Reams of data are only useful if they are interpretable and actionable. DataOps integrates the interpreting and decision making around the software-driven practices we’ve already described. In CD, metrics like deployment frequency and mean time to resolution directly connect to business performance, if not survival.

In Summary

CD is still a young discipline. The industry has learned from the challenges faced by producing physical goods. It has therefore developed a set of evolving best practices summarized as: automate everything you can around your deployments to immutable and observable environments, with underlying configurations maintained in a version controlled source of truth.

‍Harness is among the platforms that seek to untangle the complexity and provide a deployment solution that just works, while encapsulating the principles articulated in this article. We encourage you to sign up for free and start deploying today. You can also check out the Harness Community projects on GitHub to see practical examples of modern CD pipelines built around the principles discussed here.