Modern Software Delivery Best Practices & Software Delivery Management
Marc Andreessen famously said “software is eating the world,” in that software is no longer just static websites and a few programs running on your computer. Modern software runs your car, makes healthcare better, and even cleans up after my cats. Yeah, that last one wasn’t the future predicted in movies, but here we are. I digress. This article will have two main parts: modern software delivery, and software delivery best practices. Let’s dive right in.
Modern Software Delivery
With software as a critical component of daily life, the release cycle must be faster, software quality must be higher, and security (especially data security) can’t be an afterthought. The news is full of failures on each of these fronts on a weekly basis, and the businesses that can adapt will thrive in this new world.
These demands on modern software are non-negotiable, and it’s forcing a shift in how business is done. Longstanding technical organizations are completely redesigning their software development life cycle, companies where IT was a cost center to the business are investing in software, and hundreds of startup hopefuls are flooding the market looking to bring software up against some of the longest standing challenges in everything from healthcare to space exploration.
That’s a long way of saying that everyone needs to improve how they deliver software. Best practices from as little as five years ago are already outdated. Technological shifts, innovation, and market shifts have teams on their toes at all times to just keep up with demand. Movement to the cloud, digital transformation, DevOps, DevSecOps, chaos engineering - That’s just a shortlist of what’s dominated the attention of development teams in recent years. Let’s talk about how we can tackle these challenges and make the world a better place.
Start With Goals
Before you dive into data, or best practices, you need to know what success looks like. Every organization has different ambitions based on the needs of their customer base. At Harness, we speak to hundreds of companies every year. What we’ve found is that most common goals generally break down into one or more of these four categories. And often, they hit all of them.
Velocity
Delivering software faster is front of mind for almost every organization I talk to. The market is moving fast and they can’t wait weeks or months to get changes out into the hands of their customers. Moreover, long delivery cycles mean more time and money is being invested in new features that aren’t getting real world feedback.
Governance
Most organizations have audits to deal with, whether it’s security audits, or industry-specific regulation. Including considerations for traceability, documentation, and logging in the design of your software delivery process reduces headaches down the road when you realize compiling all your audit data by hand is an approach that doesn’t scale with growth and velocity.
Quality
Not every organization is obsessed with speed, is highly regulated, or has a workload that demands optimization in order to scale. But, quality is the one goal that crosses all industries and practices. People remember bad experiences more than they do good ones, so quality has to be baked into every aspect of your software development, not something done right before your end users see your latest feature.
Efficiency
Efficiency often goes hand in hand with speed, but as you add other considerations mentioned above, lack of efficiency can often cut into speed. Processes that get features to market rapidly often involve trade offs, and scaling these processes can amplify the worst effects of that balancing act. These challenges can include manual toil, cost considerations in your cloud spend, maintenance, and more.
Software Delivery Best Practices
Take a Data-Driven Approach
Once you know what you’re looking to achieve, start looking at ways to quantify that success. There’s a whole litany of literature on effective software delivery and metrics, with the choice of this author being Accelerate by Jez Humble, Gene Kim, and Nicole Forsgren. A quick summary of Accelerate: High performing organizations have a lot of things in common, and the data proves it out. As an aside, we also have an article on Accelerate metrics that may be of interest!
A quantitative approach to manufacturing in the real world unlocked higher levels of productivity, and software is no different. While there is a fierce debate of what metric shows how productive a software team is, there is no dispute over measuring this data and working to improve processes.
Use the Agile Approach
The same principles you use to evaluate and improve your software development process should be applied to your software. Manageable, measurable goals that you iterate quickly make you more responsive to your customers. Delivering software is a constant process and feedback is not something that should be reserved for the end.
Delivering new features starts with planning deliverables that you can provide feedback on quickly. Spending months building out a product before you solicit feedback leads to waste as requirements are not always met as intended, corner cases don’t get surfaced, and technical debt can be created by bad tradeoffs that were unnecessary.
Depending on the data you look at, somewhere around 2/3s of organizations are practicing agile. With that, there’s a good chance you already are too. So how can you level this up? For one, there’s a whole lot of well-researched articles on this topic.
What else? I’m glad you asked. Enter the humble feature flag. Homegrown systems have existed for years, but in recent years it has gone from a wild west hack to a full-blown ecosystem of tools with a user base that spans developers, DevOps teams, and product management. Feature flagging tools allow for you to iterate faster, merge your code in smaller commits, and gather feedback on alpha and beta features in production, without the risks inherent in testing in production.
Ditch the Monolith
Microservices hit peak buzzword a few years ago, but the hype is real. Having distinct services that are loosely coupled allows teams to deliver software faster and with better quality (in fact, I talked about the journey from monolith to microservices on this very blog). In a microservices architecture your service is responsible for fulfilling a role, no more, no less. Other services and the rest of the organization is not concerned with how you achieve the ends they need, just that you fulfill it in a timely manner, correct manner.
This approach frees up software development teams from commitments to specific languages, frameworks, deployment strategies, and other patterns that don’t fit right for every single service.
Source Control Process
Your code is your product. Everything downstream of source control will be affected by decisions you make in your branching process, code reviews, and merge process. The more software developers are touching a given codebase, the likelihood of merge conflicts and other logistical issues goes up.
Review process - Be deliberate in establishing a culture of peer review. Validating changes and providing feedback is not a chore, it’s a quality gate. It provides a learning mechanism for newer developers.
Merge frequently - Avoid long lived branches. Merge conflicts and lack of proper integration slow down delivery velocity, and can deter quality. Consider using trunk-based development as a strategy to minimize your lead time to production.
Testing
It goes without saying that robust testing is the cornerstone of delivering quality software. In the current year, it’s still not uncommon that the bulk of testing applications happens in the QA phase of delivery. As a part of the greater shift left mentality, writing and running tests is just as important to software engineering as writing your application source code.
Frequent production pushes rely on a rigorous testing regimen. During development, you build out unit tests, integration tests as you pull the different pieces together, and functional tests to validate the end-user experience is as expected.
The benefits of automated testing are obvious, though a new challenge has emerged from its adoption at scale. Mainly, it takes up increasing amounts of resources and time. Harness introduced Test Intelligence to address this exact problem.
Finally, testing in production is no longer a third rail. The aforementioned feature flag has changed the paradigm in the level of control when trying out new things. Seeing how a feature works in the wild allows for greater refinement ahead of general release.
Continuous Integration
Continuous Integration processes allow for multiple developers to contribute to one codebase, while maintaining a consistent, well-tested end product. A shortlist of what your build process should include is unit testing, security scans, dependency management, compilation/build, and packaging.
Continuous Deployment / Continuous Delivery / Continuous Verification
Software delivery has long been plagued by manual toil, scripting, and wasted time in getting your code to production. The release process will generally involve a long list of tasks such as provisioning infrastructure, vetting in the test environment, sign-offs/approvals by QA, and ultimately deploying to the production environment. Automated software delivery makes your process repeatable and sustainable while avoiding liabilities come audit time. It’s 2022, so it’s time to ditch the scripts for delivery. Modern software delivery tools provide extensive tooling, integrations, and options to automate your entire delivery process.
To achieve both your speed and quality goals, automated verification of deployments is a must. This means more than just validating your app starts, or that your health endpoint gave you a quick HTTP 200. Verification is tying your metrics into your pipeline and evaluating the health of your app for a period of time to make sure you don’t miss adverse events that happen soon after deployment. This is in addition to your normal monitoring regimens, though Harness Continuous Verification provides you with a way to use those existing tools in your pipelines, with AI/ML analysis.
It’s worth noting that while it cooperates with CI, CD is a distinct problem being solved and we have spilled much ink on this lovely blog explaining the finer nuances of what is CI and CD.
Security
Just as testing is not the sole responsibility of the QA team, security is the responsibility of everyone. Security begins at developing source code, is tracked throughout the process including artifact management, and ultimately through each and every deployment to environments.
Basic measures any organization can take include requiring security scans in CI/CD pipelines, code analysis, managing SBOMs (software bill of materials), and well designed standards enforced throughout the release process.
Shift Left
With the agile approach, feedback and responsiveness are paramount. The idea of shifting left is to move control over every key aspect of the end product into the hands of the software team. With security as an example, think about how much more secure your end product is likely to be if teams are made aware of potentially unsafe code, or dependencies while building the software product, instead of well after the fact when they may have moved on to another initiative.
If teams spend a small amount of time upfront to manage security, testing, and infrastructure as code, the business saves resources in the long run with happier customers and better agility to meet business goals.
Efficient Management of Software Development
Tools
The acceleration of software development has resulted in a multitude of great tools to meet the challenges. Open-source projects such as Jenkins paved the way for modern CI/CD solutions, products such as GitLab brought a first class CI experience to version control, the Jira Board is a staple in software delivery, and lest we forget Harness bringing the first modern CD solution to market.
Modern software solutions for software development range anywhere from minimum viable product (MVP) solutions to white glove solutions, and full product suites. There is a constant push and pull in arguments over costs, build versus buy, developer experience, and more. The best organizations invest in appropriate tools because every developer working on a tooling problem is not working on a customer problem.
Culture
Software is still a people business, but this can often get lost in the storm of technologies, tools, and trends. Software is written by human beings with families and interests outside of work. To manage a productive team there must be a culture of respect for people and their time. Moreover, with the rapid expansion in jobs and opportunities, many developers will leave a job than deal with a toxic culture, a bad manager, or poor development tooling/processes. Invest time and resources into career development, team building that isn’t just dropping a ping pong table and a keg in the middle of the office, and listening to feedback.
Continuous Improvement
Perhaps the most important point here is that this is a journey. No organization has solved everything, regardless of how many smart and creative people they hire. Measure your success against the industry at large, against your peers, and most importantly against yourself. Your software development process should be better today than it was yesterday.
Closing Thoughts
At Harness, we partner with a diverse array of customers; everyone from the biggest banks in the world to early stage startups. When we partner with these organizations a big part of what we are doing is helping them find the right ways to optimize their development process and bring it in line with their delivery goals. Successful software delivery is one part planning, one part culture, and one part tools.
Do your tools support your software delivery process, or do they slow you down? Come have a chat with us about our software delivery platform, whether you’re looking at Continuous Integration, Continuous Delivery, feature flags, or something else.