Leveraging Data in Your CICD Pipeline: A Comprehensive Guide

In this blog, we'll explore several use cases that showcase how to integrate data and databases into your CI/CD pipeline. This blog provides a high-level overview of this end-to-end story and will link to more detailed blogs on this topic as we add them to the series. Here’s a quick demo of the use cases we are talking about.

Use Case 1: Migrating Data Between Kubernetes Clusters

It can be a challenge for CI/CD pipelines to ensure that the testing environments closely mirror the production environment. Many products struggle with representative test data. When the environments don’t match, it often results in outages and incidents in production that could have been caught in staging. This demo shows how to use a CI/CD pipeline to replicate production data in a test environment. When doing so it is critical the test environment meets the same production security safeguards. By copying production data into a pre-production environment, you can test your entire migration process using real-world data. This approach uncovers patterns and issues that might not be evident in test data, thereby reducing the risk of unexpected failures when deploying to production. This copying is accomplished by running a Portworx migration.

Use Case 2: Database Schema Changes with Zero Downtime

Introducing changes to your database schema without causing downtime is a critical requirement for many applications. In this demo, the scenario involves adding a new warehouse location to a fictional barbecue restaurant chain. The solution involves two key steps:

1. Adding New Columns: Introducing new columns for city and state while keeping the existing location column intact.

2. Setting Up Triggers: Ensuring that updates to either column are synchronized with the other set.

This method ensures that both the old and new versions of the application can work against the updated schema, providing a smooth transition without any downtime. Meanwhile, the triggers ensure referential data integrity and consistency between the data in both versions of the schema.

Details of this schema change are available in git here.

Use Case 3: Deploying Application Updates

Deploying a new version of an application that depends on recent database changes can be challenging. The demo leverages Harness CD to deploy the new version of the application and Harness Database DevOps to deploy changes to the database. By integrating database schema changes and application updates within the same pipeline, you can ensure that both are deployed in a coordinated manner. Similarly, if either needs to roll back, they can roll back together. This approach minimizes the risk of compatibility issues and ensures your application remains stable and functional.

Use Case 4: Monitoring Database Health

Monitoring the health of your database throughout the CI/CD process is crucial for identifying and addressing performance issues. The schema migration deployed in this demo refactors a query so that it starts joining two different database tables on unindexed columns. This slows down the database which is detected by DBmarlin, a database monitoring solution that integrates views for analyzing database changes. DBmarlin provides visibility into any changes applied by Harness DB DevOps or Harness CD and correlates those changes with monitoring data. This allows you to:

• Detect Performance Issues: Identify any negative performance impacts caused by database schema changes or application updates.
• Optimize Queries: Use DBmarlin's optimization guidance to improve query performance.
• Compare Performance: Run time comparison reports to analyze the impact of changes on query execution times.

After this issue is detected, the demo deploys some updated indexes to fix it. The details of the index fix are in git here.

Learn More

This blog is the first in a planned series that will span Portworx, DBmarlin, and Harness discussing these areas of DevOps for the database in greater detail. As additional blogs post, we will be linking them here, to help you learn more about how you can benefit from leveraging data within your CI/CD pipeline.

Related Resources:

• https://www.harness.io/blog/how-to-safely-move-a-db-column - details how the schema change used in this demo works.

‍

By integrating tools like Portworx, Harness Database DevOps, Harness CD, and DBmarlin, you can create a robust and efficient pipeline that ensures smooth transitions, minimizes downtime, and maintains optimal performance. By adopting these best practices and tools, you can significantly improve the reliability and efficiency of your development and deployment processes. If you're interested in learning more about how we can help you leverage data within your CICD pipeline, please reach out to Harness.

‍