The system release(Applicable for Continuous Delivery, Continuous Integration, Chaos Engineering and Platform) train cycle will commence on Monday through a scheduled automated branch cut job. A cron job will pick up the latest SHA from the Harness repos, initiate the builds, deploy it to the QA environment for regression testing, and address any open tickets.
Developers can submit Alpha fixes between noon on Tuesday and Thursday, subject to approval from the respective service owner. By Thursday EOD, stakeholders and QA must provide their signoff.
On Friday Morning IST time, the system's Prod 0 cluster deployment, known as the beta rollout, will begin. It will soak there till Next Monday, during which beta fixes are allowed with service owner approval.
On the following Monday, the Prod 1 deployment will commence, followed by Prod 2 on Wednesday and Prod 3 after 7th day. Prod 3 is deployed first, followed by EU1, and then Prod 4 with a day apart. In case any hotfix comes in between the Prod 1 and Prod 3 rollouts e will thoroughly investigate and apply the production fix called Hotfixes and release it to customers. Any hotfix in Prod 4 gets applied to all the clusters deployed before Prod 4 and likewise.
As we expand our footprint and roll out more Prod clusters, production rollouts flow from Prod0 to all Prod clusters can be pictured below:
Note: Modules that are not part of above system release follow an independent on-demand release cycle.
Rollback and Hotfix Process
One of the critical decisions of any deployment is whether to roll back the deployment or fix it forward. Rolling back the code is a business decision that needs to be weighed in the context of the impact on our customers. In the following scenarios, we usually do the roll-back
If the deployment has a performance impact across the product line.
If the deployment has introduced regressions to most of our customers, the product will be unusable from the customer’s perspective.
As mentioned before, every roll-back decision is carefully considered. If we decide to do a roll-back, we roll it back to the previous build and follow the same deployment process mentioned above.
Managing Breaking Changes
Changes in existing behavior have a high potential to cause disruptions. We actively manage this, and ensure that we both minimize these changes and give customers appropriate notice when appropriate.
We use a notification framework where once a change is identified as breaking change during the release sign-off process, it goes through a breaking change notification process where we identify the user set that can be impacted by that change and proactively notify them via our in-app or email or both to notify about the impact of that change and related remedy to avoid any impact on existing flow.
This breaking change is then rolled out to Prod after 6 weeks of notification to these users.
SMP release process
Harness releases the Self-Managed Enterprise Edition every end of the month. Additionally, periodic hotfixes are released as needed. Self-Managed Enterprise Edition takes a branch cut of the Harness SaaS release. It creates a Release Candidate which goes through the iteration of testing and bug fixing process after which the release candidate is released.
.jpeg)