.svg)
With software being intangible when compared to physical hardware, a common exercise most organizations go through is a build vs. buy analysis when leveraging software to meet needed goals. Unlike something physical infrastructure-related, software’s only perceived material is time and effort.
A value calculation vs. pure time and effort can be difficult for organizations to calculate. Even physical hardware today with the push to the public cloud is under the same amount of build vs. buy analysis; build/expand a data center or fire up cloud resources.
Value calculations not only take in time and material, but also take into account total cost of ownership and opportunity cost; the cost to the business having key resources tied up not delivering core functionality to external customers.
For most organizations, building a CI/CD platform is not core to the organization. Building substantial functionality and tooling to get features functionality into production is not a value add to an organization. The adage, “your external customers don’t care how you do something,” ironically is not the same for your internal customers. CI/CD systems directly impact your internal customers and they very much will care how the goal is achieved. CI/CD platforms can take on a life similar to external customer/business value-driving applications. Organizations that want to embrace CI/CD do have a few prerequisites.
In the DevOps trifecta of “people, process, and technology,” achieving Continuous Integration and Continuous Delivery requires investments and culture shifts in all three pillars. A CI/CD platform by itself does not automatically mean your organization is achieving full Continuous Integration and Continuous Delivery. A well-architected and adopted CI/CD platform allows for opinion to be leveraged in the platform and provides guardrails and expert opinion from various teams to be a guiding light.
The biggest prerequisite is understanding that a CI/CD platform is an evolving piece of technology that should be treated as a product instead of a project. Technologies that these platforms need to support continue to evolve and items continue to shift left/become automated that need to be included in the CI/CD pipeline.
From a technical standpoint, for the Continuous Integration piece of CI/CD, some sort of source code management (SCM) solution that can fire off events to an external system and an artifact repository to capture and store the builds are foundational pieces. For the Continuous Delivery piece, movements towards codification/automation to support confidence and infrastructure the application needs. These can be the first steps into having automated test coverage and some sort of infrastructure automation/planning to have infrastructure created at deployment time or have at the ready to deploy to (for example, spinning up cloud resources or deploying to an awaiting Kubernetes Cluster). Like any software undertaking, painting with a wide paintbrush needs to be constrained by scope.
Teams tasked with engineering efficiency look to include as much of the organization as possible with the platforms they are building. Most enterprises have heterogeneous technology stacks and some parts of the stacks have been around for potentially decades. Building automation to support 100% of the enterprise can take several years and be an ongoing battle as certain technologies are deprecated or being brought onboard. This is why scope is important.
A common approach when building a platform is to try to tackle items that would have the most impact or support the most internal customers. The first step would be taking a census on the technology choices in the organization. Planning and due diligence are important when raking over technology choices. For those who dislike ITIL/ITSM, this might be the one time for those processes to shine, especially at larger enterprises, giving a concise picture of technology choices. Part of what we do at Harness when partnering with a customer or prospect is to help run a Continuous Delivery Capability Assessment (CDCA) to catalog and measure maturity. Part of this catalog is also taking inventory of technology choices.
When building out the minimum viable product (MVP), the two radio dials to balance would be where would be the easiest success for your customers vs. wide adoption and impact. To achieve both would be fantastic, but more often than not, you’d need to pick one path. The good news is that incremental success builds success and momentum, and if you start to build easier functionality, you gain expertise in the platform as time goes on. Both CI and CD have different requirements.
The purpose of Continuous Integration is build automation. A Continuous Integration platform has three main pillars: repeatability, consistency, and availability of the builds. The modern approach is that any sort of source code management - even a commit/merge - should be expected to kick off a build.
As Continuous Integration became mainstream, an ironic bottleneck in the software delivery process started to be CI systems. The number of concurrent builds increases with the number of engineers and new packaging. Such as creating a Docker Image would be compute-intensive, CI systems would be strained and suffer from long queues. A core requirement is that a CI system should be designed to scale to meet the needs of the enterprise. This can be in the form of distributed build runners that spin up and down when needed or not.
Building on an external system from a dependency standpoint is not different than building on your local machine. Language and package-specific build tools need to be distributed across the CI system’s build runners. If on your local build there are other gates, such as code quality or unit test requirements, these too need to be distributed across everywhere the build takes place. After several builds take place, a viable release candidate is made/needing to be deployed, thus making the transition to Continuous Delivery.
When compared to Continuous Integration being more technology-heavy in the DevOps trifecta of “people, process, and technology,” Continuous Delivery represents the other two points. The goal of Continuous Delivery is to automate the safe deployment of changes into production. Traditionally, confidence in safety would be brought about by orchestrating approvals by people and several different testing and verification processes.
There is a difference between Deployment and Delivery; Continuous Deployment vs. Continuous Delivery. Deployment focuses on the actual act of distributing/installing the application binaries and typically takes the path of least resistance. Delivery focuses on safety and confidence-building culminating in production. A hallmark of Continuous Delivery is the canary release, which is an incremental release. Canary releases are technically more complicated to pull off, because they represent several incremental releases and judgment calls as the canary marches towards taking on full traffic.
The challenge with Continuous Delivery is building for the non-happy path. Failure is bound to occur, and how to gracefully restart, recover, or debug a process can be challenging. At a minimum, when building your own CD platform, a workflow should be the first piece of functionality created. Having the ability to systematically record all of the steps from artifact to production, even if some of those steps are manual today, is crucial for identifying repeated bottlenecks and eventually ushering in automation. Building any piece of software requires time, ownership, commitment, and iteration.
Creating and building software takes a village. For those who are coming from an operational background and still remember the days of ticket-based resolutions, creating a CI/CD platform is more than a ticket to act on.
Software, and especially your CI/CD platform, has a lifecycle. If you have been through the software development lifecycle (SDLC) before, leveraging several open-source projects, potentially in conjunction with commercial tools to create functionality, would be very familiar.
To garner longevity and adoption, several resources are potentially needed. Taking a product mindset for your CI/CD platform requires product-level resources, from a product manager to several types of engineers to write, run, and perform quality tasks, to internal champions needed to sell the solution.
Focusing on only the needed fully-burdened costs of development and quality assurance resources needed, depending on company size, you can see how the costs and add up quickly - and that doesn’t even take into account the opportunity costs.
Like the applications that the CI/CD platform supports, a bulk share of costs with any application is after the initial release. This can be in the form of continuing enhancements to the not-too-exciting maintenance/patching phase of the SDLC. Once the initial surge of getting the new platform is out the door, resources and investments are still required to maintain the CI/CD platform.
Making sure that the platform has longevity is core to maintaining the CI/CD platform. Pillars to maintaining the CI/CD platform are: ensuring growth with new features to match technology trends, providing scalable infrastructure, educating on best practices, patching/upgrades/break fixes, and administrative tasks such as onboarding/offboarding applications and users.
Not only is expertise required in a potentially bespoke solution, but all of the other various tasks and investments are required to further drive adoption, stability, and further efficiencies in the platform. Experienced software engineers can shake out what an initial project would look like, but there are certainly unknowns - especially into efficiency.
The technology adage is true: with infinite time and resources, you could build anything. Even if you had the most wonderful platform built, encouraging antipatterns can cost the business money.
If the platform is too specific and does not include enough of the organization, it can fit the category of “just one more tool.” Running through a deployment or onboarding a new service to the CI/CD platform costs money.
If deployments are supposed to happen all through the day, how much prep work does a team have to do to submit to the CI/CD platform? The more manual tests and approvals there are, the more time is needed from the maintainers of the CI/CD platform, and of the internal customers as well.
Non-functional requirements such as security, reliability, scalability, and verification can eat any software platform alive. Like any piece of software, there needs to be conformance to enterprise standards. As CI/CD platforms become even more critical in the enterprise, uptime and scalability requirements can start to look a lot like an external customer-facing application that has the full support of development and operations teams. Another adage that transcends life is, “If you are good at something you should not do it for free.” There are certainly costs to free open-source from projects looking for monetization to just general overhead building and maintaining for your organization.
Open-source is a research and development methodology. In industry, there is also the assumption that open-source is free; this meaning free of license costs. On the contrary, there are hundreds of potential Open-Source Software (OSS) licenses out there, which can take into account charging for intellectual property.
In the CI/CD space, open-source projects tend to be more engine-focused. An uneasy topic to talk about with open-source purists is the monetization strategy behind the projects. Popular CI/CD projects can be sponsored by commercial organizations selling enterprise renditions and services behind the projects. With a car analogy, if you get the engine for free, you would need seats, a body, and a host of safety features, like airbags, to complete the car. Typically, items that enterprises need, such as security hardening, specific patching, and even support, are relegated to paid enterprise editions of OSS CI/CD projects.
This completing, maintaining, and upgrading of the car is exactly what using free open-source projects costs: the sunk costs. If changes are needed, there are only two options. One is to build the needed functionality or changes yourself, or go and lobby the open-source projects to hear out your needed functionality and hope someone builds the functionality in due time. Open-source projects also have varying release and patch schedules; leveraging several projects means several times the maintenance.
Buying a platform from a firm that is focus driven on CI/CD - like Harness - and takes all the orthogonal concerns of the platform into account, has a great value proposition.
As an engineering organization, buying a solution can be a knock on your ego. As an engineer or engineering leader, clearly you are highly skilled and have the ability to build what is needed; we wouldn’t be engineers if we did not like building things and solving complex problems. The main driver for buying is if the time and effort for getting features into production are worth keeping significant engineering resources on the self-built platform.
Value and focus are drivers when considering buying a CI/CD platform. The decision becomes even more difficult when an existing investment has been made previously in building a CI/CD platform. If starting from scratch can backwards engineer the time and effort it would take to leverage a commercial vs. open-source solution.
A two-fold challenge is solved with Harness: making sure that the platform continues to be inclusive of technology of the past, present, and future, and keeping up with all of the demands such as scalability and security the modern enterprise requires. Harness has learned from a wide set of customers and continuously improves the platform on a daily basis.
Harness is on a mission to standardize deployments and allow every firm to deploy as sophisticatedly as a FAANG (Facebook, Apple, Amazon, Netflix, Google) organization as easily as using one of your favorite consumer applications. Where the most tribal knowledge about deployments is locked (what is normal vs. not) is unlocked and democratized with AI/ML inside the Harness Platform.
The Harness Platform is pretty flexible, allowing for as much or as little automation as your organization or team sees fit. The Harness Platform is ubiquitous where you can deploy and is opinionated to encourage best practices and discourage antipatterns. Harness is an expert in CI/CD adoption, UX, and helping firms move towards more frequent, stable, and safe deployments. Ironically, deployments have a cost associated with them, with the internal customers taking resources to handhold and validate deployments. Part of partnering with Harness is helping you calculate the value with our Continuous Delivery Capability Assessment.
At Harness in 2020, we published an inaugural study around the CI/CD journey organizations are taking. These organizations were in varying stages of their CI/CD journey. Prior to partnering with Harness, part of the value engineering that Harness helps provide is taking a census of the current lay of the land. We have completed hundreds of Continuous Delivery Capability Assessments, and in 2020, we aggregated the results.
Listing out several of the challenges that organizations face, there is still significant effort going into deployment. The median deployment frequency, as we surveyed firms in 2020, was ten days. On average, each of those ten-day deployments took eight hours of lead time, meaning one full day of prep to even start the deployment. The validation of the in-flight, then completed deployment, on average took 25 hours of engineering burden. That is more than half a week of a full-time employee’s work.
Doing some math, if you are curious what engineers cost, taking a look at Glassdoor, Levels, or Blind would give you a good indication of the market rate today. Adding the human cost for building the platform and the human cost per deployment, multiplied by how many deployments on average your firm does, would give you the actual cost of getting features into production. What is not accounted for is the opportunity cost for not focusing on core capabilities, and if there was a failure during a deployment, those hours of impact could be significantly higher. Harness is here to take the consternation out of software delivery.
From day one, we have been on a mission to democratize software delivery. The platform is constantly expanding capabilities and making software delivery easy for everyone. There is a treasure trove of expertise at Harness, committed to making your CI/CD goals - and beyond - possible. Feel free to engage with us and if you have not already, sign up for a Harness Account.
Cheers!
-Ravi
Sign up for our free plan, start building and deploying with Harness, take your software delivery to the next level.
Learn intelligent software delivery at your own pace. Step-by-step tutorials, videos, and reference docs to help you deliver customer happiness.
.svg){kind=small}
Learn intelligent software delivery at your own pace. Step-by-step tutorials, videos, and reference docs to help you deliver customer happiness.
Enjoyed reading this blog post or have questions or feedback?
Share your thoughts by creating a new topic in the Harness community forum.
.png)
