Deployment pipelines are complex with lots of spinning plates. Visibility into these pipelines is a major challenge for any organization with a CI/CD platform.
According to Wikipedia, Analytics is the discovery, interpretation, and communication of meaningful patterns in data.
Vendors often confuse this term with “pie charts.”
Analytics is about communicating information that is contextual, understood and relevant for a specific user for a specific use case, In this case–Continuous Delivery.
Continuous Delivery is hard. Automation is one thing; understanding its impact or status is another. A major retailer yesterday told me, “We have deployment scripts but we lack any sort of visibility once those scripts are executing across our applications and environments.” Visibility is tough when your CI/CD platform is 100+ deployment scripts.
Our Harness engineering team has spent the last six months working with customers to design analytics from scratch to help them manage their CD pipelines.
Here’s a quick summary of what we delivered:
Deployment Velocity Analytics
Is the customer’s deployment velocity increasing, decreasing or flat over time?
In the example above, you can see that deployment velocity increased significantly over the last two weeks. Failures also decreased the past week. Notice nobody works on the weekend 🙂
Service Velocity Analytics
Which of the customer’s services are deployed the most frequently, and how many of those deployment pipelines actually succeed vs. fail?
A glance at the above instantly tells the customer which services are deployed the most, and more importantly, which pipelines succeed or fail the most. CD isn’t just about deployment numbers: Having failed pipelines is a good thing, because it shows you’re actually catching things before your end users. Remember, the purpose of a deployment pipeline is to kill the release candidate.
Service Deployment Analytics
What is the deployment status of every service right now? What environments are they running in? How big are these environments? What versions are currently deployed?
What is the current deployment status for a given service?
What is the status of each deployment pipeline, and did all pipeline stages complete successfully? How long did each pipeline stage take to execute?
Deployment Workflow Analytics
What is the status of each stage and deployment step within a deployment pipeline?
The above visual is an animated real-time view of a deployment workflow that allows development teams to watch their deployment. Clicking on any workflow step will provide granular details on the state and execution (e.g console logs).
Deployment Verification Analytics
Was each deployment successful?
What regressions or anomalies did each deployment introduce to the service? The screenshot below shows that a deployment introduced 5 new error regressions. The one highlighted was caused by an invalid Docker Registry credential that had expired. Harness uses unsupervised machine learning techniques to generate this analytics and insight.
What was the business impact of each deployment? The below visual shows that a new deployment introduced 3 new performance regressions to the customer’s service. Again, we’re applying unsupervised machine learning to 3rd party monitoring data to understand what normal and abnormal performance look like.
Security & Compliance Analytics
Who executed What, When and Where? How about a complete audit trail of every deployment pipeline, stage, workflow, step, and execution with complete access to console and log files? Automation still requires control and standard security processes, especially for large enterprises.
What secrets are being managed, used and changed across all deployment pipelines? Our customers wanted a complete report of all secrets being used and changed across their deployment pipelines.
Analytics Isn’t About Being Sexy
You can make any product analytics look good, but the real value is down to how quickly users can understand and act on the information being shown to them. Simple visuals with concise insight are significantly better than complex visuals and information overload.
You don’t have to answer multiple questions at once with analytics–just the questions that get asked most often. When our engineers sat down with customers, we stack ranked the top 10 questions that were asked the most when it came to Continuous Delivery. What you see above is the start and we’ve got way more to deliver in 2018!
If you’d like to evaluate Harness, sign up for our free trial.