The Top Chaos Engineering Tools

Businesses are increasingly turning to cloud-native deployments (i.e., those based on Kubernetes) versus traditional deployment methods for a variety of reasons, one being the need to increase deployment velocity. The challenge site reliability engineers (SREs) and development teams now face is that cloud-native systems can fail in more ways than traditional deployments. 

Unplanned downtime can have significant business financial, brand, and reputational impacts. The costs of unplanned downtime plus this increase in systems-level complexity have created a heightened need to evolve how we test cloud-native systems. Chaos engineering provides the mechanism by which systems-level software testing happens to reveal weak points and helps teams deliver more reliable systems.

Today's highly intricate software systems must be tested for potential weaknesses and faults. Chaos engineering, as the name implies, is a process that involves testing a software's ability to handle failures without affecting systematic functionality. By testing a software's resiliency, development teams can identify failures and proactively address them.

Chaos testing can be performed as a means of proactively experimenting on a software's infrastructure. Inducing failures can help improve organizational confidence if systems are able to overcome and mitigate turbulent conditions and outages. 

Do your systems have the real-world capabilities needed to overcome network latency and infrastructure performance issues? Testing your system's capability is imperative for ensuring your software can withstand any issues that come your way. With these principles in mind, we've reviewed some of the top chaos engineering tools on the market today. 

Why Use Chaos Engineering Tools?

Chaos engineering tools are a relatively new approach to traditional testing methods used to establish confidence in systems. Software platforms will inevitably fail, and therefore it's critical to pinpoint weaknesses and fix them before they negatively impact business operations.

Top tech organizations such as Amazon, Netflix, and Microsoft utilize chaos engineering to achieve a better understanding of internal systematic behavior and flaws. The principles of this approach are predicated on the idea of testing system architectures through various hypotheses and performance-based metrics. Through the deployment of assumptions and successful chaos experiments, chaos engineering tools can provide a roadmap for uncovering infrastructural failures or unresponsive systems.

Chaos engineering follows a general set of guidelines that includes each of these steps:

• Creating a steady-state hypothesis: Think of potential system issues that could occur. Set up failure injection chaos testing protocols and predict various potential outcomes.
• Simulate real-world scenarios: Create a set of tests that will determine how systems react to different variables. Use an experimental group to test various conditions and factors.
• Review system metrics: Review system outcomes related to system performance and metrics. Determine failure rates against hypotheses and figure out a path forward to correct and fix recurring issues.
• Implement changes as needed: Upon conclusion of chaos experiments, you should be able to ascertain what the best course of action is. Attempt to fix any issues and repeat the process until systems are operating with little to no errors.
• Automate chaos experiments: Once your system has been verified that it is resilient to the failure mode, the next step is to create chaos experiments and automate them in your software delivery pipeline to ensure continuous validation through the dynamic system configuration changes occurring in the environment. If the experiment fails, you can be notified or automate the rollback of a change that introduced the failure. Your system won’t experience an outage from a known failure mode that you are protected against.

Creating an effective and well-rounded practice can help your organization test resiliency and discover potential fault tolerances. Let's take a look at some of the popular chaos engineering tools that can be utilized to optimize your systems.

Chaos Mesh

Pros

• Easy-to-use functionality and automation
• The user interface supports many different configurations
• Experiments can be paused and resumed at will

Cons

• Experiments run indefinitely as there is no ability to schedule attacks
• Node-level attacks cannot be run
• Cannot control user access within the dashboard; as a result, there are increased security risks

Chaos Mesh is an open-source cloud-native tool. Using various fault simulations, Chaos Mesh helps organizations determine system abnormalities that may occur during various portions of the development, testing process, and production stages.

As an open-source chaos tool that's created with a web user interface known as the Chaos Dashboard, Chaos Mesh can be added to DevOps workflows to spot potential areas of weakness and timeouts. To ensure resiliency, Chaos Mesh utilizes chaos experiments within Kubernetes environments. It's able to use various types of scenarios related to fault simulations within a distributed system.

Chaos Mesh is able to deploy attacks that test network latency, system time manipulation, resource utilization, and more. The Chaos Dashboard can be used to modify and manage various forms of experiments within set timeframes.

Key Features

• Chaos Mesh uses a Kubernetes-based interface that's supported with full automation and graphical capabilities used in the testing of high visibility distribution systems such as Apache APISIX and RabbitMQ
• Chaos Mesh technology is able to test various scenarios using event-driven fault simulations
• Chaos Mesh provides the ability to design experiments on the platform using different variables and status checks

Cost

As an open-source chaos tool, Chaos Mesh is free to use without a commercial license.

Should I use Chaos Mesh?

Chaos Mesh offers an open-source technology that can be used in Kubernetes to design and manage automated experiments. However, be wary of certain limitations to the technology. Predicting failures can be a cumbersome task due to the complexities in cloud operations. Unreliable functions and outages can result in a downgraded reputation and a loss of consumer trust. 

Chaos Monkey

Pros

• Configurable technology allows for easy monitoring and scheduling of attacks
• Open-source software has no licensing costs
• Extensive development history

Cons

• Can only perform one type of experiment
• Attacks are randomized and users have limited control of the blast radius
• Requires writing custom code

Netflix’s Chaos Monkey is an open-source chaos engineering tool originally created by Netflix developers. It was developed to help test their system reliability and resiliency after moving to the AWS cloud. The software functions by implementing continuous unpredictable attacks. Chaos Monkey uses the basic fundamental approach of terminating one or more virtual machine instances

The configurability of Chaos Monkey allows for easy scheduling and close monitoring. The technology is easily replicable but can cause headaches if users are unprepared for the aftermath of attacks. Users can check for outages prior to deployment but must be able to write and edit custom Go code.

Key Features

Chaos Monkey was one of the first chaos engineering tools and the first open-source technology to help initiate the movement. After its inception, Netflix later developed additional fault injection tools collectively known as the Simian Army. 

Key features of Chaos Monkey include:

• Detects systems bottlenecks to help limit disruption to production environments
• The ability to test resiliency and availability of applications at an infra level
• Tests can be scheduled during certain timeframes
• Allows for easy monitoring

Cost

As open-source software, Chaos Monkey is free to use without a commercial license.

Should I use Chaos Monkey?

Chaos Monkey is a popular chaos engineering tool. While it may have revolutionized the open-source community, its contemporary application is far less practical today. Chaos Monkey is useful to an extent, but users must take into account its limitations and arduous deployment capabilities.

Gremlin

Pros

• Easy-to-use UI allows for various attacks and tests to onboard teams
• Support with API for creating manual integrations
• Evaluates reliability based on a variety of different factors

Cons

• Software is not customizable
• Challenging to integrate experiment JSON files into the software delivery pipeline 
• Minimal reporting capabilities

Gremlin is the first hosted chaos engineering platform designed to improve web-based reliability. Offered as software-as-a-service (SaaS), Gremlin is able to test system resiliency using multiple attack types. Users provide system inputs as a means of determining which type of attack will provide the most optimal results. Tests can be performed in conjunction with one another as a means of facilitating comprehensive infrastructural assessments.

Key Features

Features of Gremlin include:

• Controlling failures in a precise and controlled manner
• Custom scenarios that include multi-levels of system attacks
• Testing process for memory leaks, latency injections, disk fill-ups, and more
• GameDay feature
• Reliability score based on predefined tests

Cost

Gremlin's pricing has fluctuated over the years ranging from per-agent pricing to attacks per target to support the frequency of testing required by a team.

Should I use Gremlin?

As the world's first managed enterprise chaos engineering technology, Gremlin provides users with the ability to launch dozens of attack vectors, stop and roll back attacks, and improve system reliability. Designed with the mission of creating a sustainable and reliable internet, Gremlin pinpoints software weaknesses to minimize revenue loss and negative systematic impacts.

Harness Chaos Engineering Powered by Litmus

Pros

• A variety of experiments to cover multiple reliability failure modes offered through an Enterprise ChaosHub
• Chaos probes that enable automation of experiments while providing safety and control to abort with automatic recovery of the system
• Automation in CI/CD and GitOps through event-driven chaos experiments that can be triggered based on environment changes enabling continuous validation of reliability

Harness Chaos Engineering is a solution for both engineering and reliability teams. The tool enables DevOps and SRE teams to collaborate and run chaos tests to identify reliability issues in their deployments. These scenarios go beyond traditional unit, integration, and system tests, more closely representing failures in a production environment. 

Teams gain insight into how systems behave under defined failure scenarios, enabling them to understand weaknesses that exist in the applications and infrastructure, and proactively create reliability to prevent costly downtime. Harness Chaos Engineering was created to help enterprises adopt, scale, and automate software reliability best practices.

The capabilities provided by this Harness enable a proactive application reliability testing approach, which reduces the risk of failures getting into production and greatly decreases application downtime associated with those failures.

Key Features

Harness Chaos Engineering was created for SREs and developers to easily run chaos experiments. Designed for cloud-native systems, the software can easily be added to CI/CD pipelines for continuous reliability validation to protect production environments from downtime.

Features of Harness Chaos Engineering include:

• Ability to orchestrate chaos engineering experiments automatically in your software delivery pipeline
• Enterprise ChaosHub, a catalog of advanced experiments with coverage across multiple cloud providers and platforms
• Private deployments to implement chaos engineering securely using self-hosted, on-premises, or air-gapped deployments
• A complete SaaS platform that enables you to automatically integrate with other modules that provide automated deployment rollbacks, service level objectives, feature flags, and cloud cost management
• Roll out to the entire enterprise through GitOps and event-triggered chaos testing
• A resiliency score to measure improvements but also to automate experiment analysis and results
• Custom integrations with application performance monitoring and observability tools
• Enterprise dashboards, analytics, and reports to ensure alignment of key metrics
• A GameDay feature to enable teams to run repeatable events with multiple experiments organized for ease of running

Cost

Harness Chaos Engineering has simple-to-understand pricing based on experiments run with full enterprise support brought to you by the team that built the open-source tool, LitmusChaos.

Should I Use Harness Chaos Engineering?

Harness Chaos Engineering has a large array of chaos experiments that enable developers to test the reliability of many cloud providers and platforms. Private deployments make it an easy tool to adopt and approve through security. Enterprise-grade features and professional support help an enterprise scale this practice immediately rather than team by team over a long period of time. 

LitmusChaos

Pros

• Centralized repository containing a variety of experiments available
• Reoccurring system health checks
• Automated error detection and resiliency scores

Cons

• Starting with LitmusChaos can be difficult depending on the user's background
• Complicated administrative tasks require setting up service accounts and annotations for each namespace
• Permissions can be difficult to manage and track

LitmusChaos is an open-source platform designed for cloud-native infrastructures and applications. It assists teams with identifying system deficiencies and outages by performing controlled chaos tests. LitmusChaos uses a cloud-native strategy to closely control and manage chaos practices.

Developers use LitmusChaos as a set of tools to create, facilitate, and analyze chaos within Kubernetes. LitmusChaos allows developers to develop chaos experiments, find errors, and remediate them prior to reaching full-scale production. The LitmusChaos technology allows users to deploy a variety of experiments to the Kubernetes cluster as a means of preparing for future use.

Key Features

LitmusChaos was created as an open-source tool used within Kubernetes. Designed to pinpoint bugs and deficiencies in Kubernetes.

Features of Litmus include:

• The ability to perform both chaos and functional tests
• Allows users to run test suites, perform log capturing, and generate reports  
• The ability to monitor application health before, during, and upon conclusion of an experiment

Cost

As open-source software, LitmusChaos is free to use without a commercial license. Enterprise support to quickly scale and build a practice is offered by Harness.

Should I Use LitmusChaos?

LitmusChaos is a Kubernetes-native tool that facilitates experiments ranging from testing docker containers to specific Pods. As a versatile tool with a variety of monitoring capabilities through Prometheus, LitmusChaos is useful but requires a significant depth of knowledge prior to getting started.

Explore Harness as Your Top Chaos Engineering Tool

Interested in learning more about how your organization can leverage Harness Chaos Engineering? Request a demo today!