Streamlining CI/CD and Optimizing AWS Cloud Spend
When companies adopt DevOps methodologies, software delivery becomes more sophisticated and teams are able to deploy more frequently.
When this occurs…
How do you keep track of the software delivery lifecycle?
How do you manage and report on Kubernetes?
How do you map out your cloud spend?
The cloud is a consumption-based model that can either be extremely efficient, or extremely costly. The below is how Linedata answered the above questions and reduced cloud spend.
About Linedata
With 20 years of experience and 700 clients in 50 countries, Linedata’s 1100 employees in 20 offices provide global humanized technology solutions and services for the asset management and credit industries that help their clients to evolve and operate at the highest levels. Headquartered in France, Linedata achieved revenues of EUR 161.0 million in 2020.
The company is in the middle of a change from monolithic to microservices, and is implementing DevOps methodologies to drive efficiency.
DevOps at Linedata
Many companies attempt to implement DevOps by creating a separate DevOps team that’s responsible for CI/CD, infrastructure, etc. However, this normally creates a permanent centralized team that has to maintain and troubleshoot the company’s pipelines. If this team is not scaled properly, it quickly becomes the bottleneck for the entire organization.
Linedata decided to take a different approach. Instead of forcing the global organization to rely on a centralized team, Linedata created a self-service system that allowed teams to be autonomous.
They built upon the SRE model developed by Google. A global platform engineering team built secure infrastructure as code that could be reused by everyone else in the organization. The autonomous teams used and iterated on these templates while also providing feedback to the platform engineering team. The autonomous teams weren’t completely self-reliant due to the high compliance needed to operate in Linedata’s industry.
Andrey leads the platform team and acts as an internal DevOps Evangelist. He helps different teams within the company adopt new technologies. One of the technologies Andrey leverages is Continuous Infrastructure as Code.
Developers shouldn’t have to spin up and destroy infrastructure themselves. This is manual labor/scripting, and traditional scripting does not equate to Continuous Delivery.
As such, Andrey leveraged a modular approach where each team used the module as a part of their own process. Modules were layered on top of each other to handle the large amounts of configuration needed to deploy. This replaced the need to hardcode variables into the pipeline. Developers simply target an output file to create their infrastructure. By default, every cluster they created had the same configuration, which then created more security.
Linedata defined this level of automation as “maturity institutionalized.” Automation wasn’t operating in only a few pockets within the organization, it was prevalent throughout.
All of this was possible thanks to Harness and Terraform.
FinOps
Linedata deploys 150 to 200 times per day, equating to roughly 28,000 instances being deployed in the last 30 days. Their developers move really fast. But at what cost?
The goal of DevOps is to be fast and efficient. Just going fast could inadvertently lead to high spend. Take one look at Twitter and you can see hundreds of developers who forgot to turn off their environments, which sent cloud costs skyrocketing. Maybe they ran an EC2 instance and forgot to stop it, or maybe they didn’t size their resources properly.
Engineers forget to go back and turn resources off and since the cloud is consumption-based, companies are charged for all of it.
Developers are worried about speed. But Finance is worried about spend.
FinOps is a new model for cloud management that focuses on speed and efficiency. This, in turn, leads to higher ROI. It involves the accurate allocation of cloud spend, optimizing environments, rightsizing resources, and turning unused resources off.
Summary
With both of these methodologies, Linedata completely automated the provisioning lifecycle. All of their engineers interact with Harness through Git, everything is auditable, and there is a continuous feedback loop.
From an organizational perspective, they cut their cloud bill by roughly 50%.