Campspot Reduces Outage Risk by 78%

Authors:

Table of Contents

About

Campspot crafts reservation software for resort owners that delivers their customers the perfect camping experience. They are a team of motivated people who use their talents to solve tough problems and build tested software.

Founded in early 2015, Campspot is looking to grow with people who embody their culture of continuous learning, innovation, and respectful collaboration. They’re made up of smart and innovative individuals who work together. Every individual adds value to their product with proven skills, ingenuity, and creativity, and they expand these qualities in a cycle of teaching and learning.

AWS Code Deploy != Applications With Multiple Ports

Campspot originally deployed all of its applications using AWS Code Deploy, but as the company scaled, Code Deploy didn’t scale with it. Campspot started adding multiple ports for its applications and Code Deploy’s blue-green deploys didn’t have native support for this structure. The team’s only option was to add layers of custom scripted workarounds to deploy updates. These custom scripts solved the problem in the short term but in the long-term introduced three issues: complexity, large troubleshooting efforts, and slow manual rollbacks.

The deployment process went from straightforward to a complex nightmare. New developers had to read through long wiki descriptions to learn how to deploy. Click this button, then navigate over here and click this button, then go back and click this other button. Something often went wrong during this process.

“It’s not our developers’ fault that the tool was clunky. They shouldn’t have even needed to access the AWS console to finish deploys in the first place.”

Kristin Smith, DevOps Services Team Lead

This complexity meant Campspot’s application could only be deployed once a week in the middle of the night. Developers were tired of staying up late to deploy their software.

“When dealing with B2C customers, it's important to get features and fixes to them as soon as possible - we needed to be able to deploy when code was ready.”

Kristin Smith, DevOps Services Team Lead

This complexity meant the DevOps team spent large amounts of time troubleshooting failed deployments. The team got three or four pings every week about failed deployments. Each ping resulted in an hour of lost work time, which led to frustration by both the developers and DevOps teams.

“The original architect actively regrets writing this custom solution in hindsight. He knows he could have gone with a different tool and saved tons of headaches.”

Kristin Smith, DevOps Services Team Lead

The most concerning issue with the custom scripted Code Deploy solution was the downtime risk. Rollbacks took roughly 45 minutes to resolve, and bugs often went unnoticed for several hours. Campspot experienced some sort of issue every two weeks. This culminated into back-to-back outages in the summer of 2021. These outages each lasted for 15 minutes and caused large disturbances to users.

This was the final straw. Campspot needed a better, safer way to deploy.

Consistency and Safety

Campspot turned to Harness for reliable deployments with less outage risk.

The complexity that caused so much developer confusion is all but a memento from the past. Teams now have the capability to deploy every day and don’t have to worry about exclusively deploying at night. Developers don’t have to read pages of wiki notes, they simply press a button with Harness.

The DevOps team freed up time to work on other mission-critical projects, and developers no longer have to wait for the DevOps team to troubleshoot all of their deployment issues.

However, the following example showcases Campspot’s largest improvement:

“Last Thursday was the perfect storm. Our developers accidentally released a really bad loop into production. This P1 issue was going to cause 500 errors across the board. But someone caught the issue in their APM console. They then used Harness to roll back before the issue affected any customers.”