Learn parallel step execution in workflows, secrets management, and environment variables.
Welcome back! If you’ve made it this far, you must’ve read the first two parts of this series. If you have not, I highly recommend that you read the first and second articles. Today, I’m going to introduce you to three concepts:
- Parallel step execution in workflows
- Secrets management
- Environment Variables
I’ve learned a lot since joining Harness earlier this year and so I’ve designed this series to help you understand everything I’ve learned about Continuous Delivery. Keep in mind that the CI/CD space is exploding. While CI has been solved for and is an extremely mature market, the CD space is going through its growth spurt at the moment.
What’s holding you back?
Companies around the world are quickly starting to realize that building a custom CD solution is as silly as building a custom CI solution. Why would you do that when there is already proven, tested, and widely adopted software being used by the largest software deployments in the world? These are enterprise environments experiencing challenges in scale from both customer engagement but also application complexity. So, if they’re already using Harness, I can’t help but ask: why aren’t you?
Perhaps, you’re unsure if you need a CD solution?
Well, if you’re deploying, you’re already doing CD, so there is no debate here.
Perhaps, you think you’ve already solved for CD with a custom solution?
I can only imagine the cost behind the building and maintenance of custom stack. Even if you did build it, how many engineers have you hired to maintain this behemoth of a problem?
Perhaps you’re ready to begin exploring a true CD platform but are unsure of what exactly you’re buying and the concepts behind it?
This is why I built this introductory series. You should catch up on the CD 101 blog, which introduces you to the fundamentals of Continuous Delivery. Then, read the CD 102 blog which introduces advanced concepts like Continuous Verification and release strategies. Finally, we’ll jump into a few more concepts in this blog to continue our understanding of the CD space.
On my CD 101 post, I introduced you to the concept of a pipeline, stages, and workflows. Remember: a pipeline is an umbrella term of multiple stages stitched together, and each stage has its own workflow. A workflow is, basically, a series of steps that have to happen in a stage in order to deploy a service/artifact.
Usually, each step in a workflow is sequential, so it is dependent on the previous step. Imagine you’re:
- conducting a deployment of an artifact using a canary release strategy,
- then verifying your deployment using Continuous Verification,
- Then rolling forward with the entire deployment across your entire infrastructure
Each of these steps depends on the prior. Very simple. But, what if you have steps that don’t depend on each other? If this is the case, you can execute those steps in parallel. Executing a step in parallel is essentially just that: executing workflow steps in parallel that have no dependencies on each other. This is in contrast to steps that are sequential. A popular example of this is deploying multiple services to multiple hosts within the same stage of a pipeline.
As a developer, your application goes through various environments within a pipeline as it eventually reaches production. One of the obvious challenges of this conveyor belt process is that while your application business logic remains the same, the environment does not. Thus, you use environment variables throughout your application in order to support the variability of your stages.
Naturally, this raises the question: how do you access those environment variables? I really hope you’re not hard coding the variables into your application (yeah, I’m guilty of this, when I was maybe 13). A proper CD platform exposes environment and custom variables to applications in order to deploy successfully through each stage of a pipeline.
“A proper CD platform exposes environment and custom variables to applications in order to deploy successfully through each stage of a pipeline.“
The way that we solve this with Harness is extremely simple. We have two types of variables:
- Harness built-in variables
- User-created variables
When configuring your pipelines in Harness, you are naturally going to begin creating information that your application will require to access in order to execute properly. These variables are immediately exposed and available by Harness, including:
- Environment name
- Emails (where to send emails e.g., QA testing)
These are just some examples. For the entire list, take a look here.
A common reason for environment variables is secret login credentials. Perhaps you’re already using a secrets manager, such as Hashicorp Vault or Amazon Secrets Manager.
Or, maybe you’re keeping your variables within your Git repository? I’m totally kidding.
Actually, I’m not.
I really hope you’re not doing this today. In 2014, Uber was hacked, and data was stolen on 50,000 Uber drivers. This happened because a developer kept login credentials as plain text within a file in their Github repository. You should be shaking your head in disappointment, just like I am. In 2016, Uber engineers did the same thing again by checking in credentials into Github.
Seriously? Insane, I know.
The proper way to access environment-dependent secret variables is to use a proper secrets manager. Harness includes a built-in secrets management feature, and if you aren’t already using one. Detailed information may be found here.
If you’re using an existing solution, you can easily add your secrets manager to Harness, including:
- AWS KMS
- HashCorp Vault
- AWS Secrets Manager
We’ve covered some more advanced Continuous Delivery concepts to expand your understanding of what a true CD platform should contain. You’re now well versed in understanding:
- Workflow steps executing in sequence or parallel
- Environment variables
Luckily, there is Harness, and I can proudly say that I’ve barely scratched the surface with these blogs of what is capable with our platform. No matter what your deployment type and configuration, cloud provider, or application stack, Harness has proven to be the industry’s leading CD platform.
Take Harness for a spin with a free trial!