Harness Blog

• Harness IaCM introduces native Terragrunt support, enabling true enterprise-grade orchestration at scale.
• Teams can now manage Terraform, OpenTofu, and Terragrunt in a single platform without fragmented tooling.
• Built-in governance, policy enforcement, and approvals streamline secure infrastructure operations.
• End-to-end visibility and drift detection improve reliability across complex, multi-environment deployments.
• The launch marks a major step toward a unified, multi-IaC control plane for modern infrastructure teams.

‍

Bringing First-Class Terragrunt Support to IaCM

“We’ve been operating in a hybrid environment with both OpenTofu and Terragrunt, and Harness has made it much easier to bring those workflows together into a single, consistent platform with IaCM. The addition of Terragrunt support is a valuable step toward simplifying how we manage infrastructure at scale.”

— Lead Platform Engineer, Enterprise Customer

Infrastructure as Code is now a standard for modern cloud operations, with most enterprises using IaC to provision and manage environments. However, as adoption grows, so does complexity. Teams are no longer managing a handful of environments. They are operating across multiple regions, accounts, and services, often at massive scale.

This is where traditional approaches begin to fall short.

As organizations scale their infrastructure, Terraform alone is often not enough. Teams adopt Terragrunt to manage complex, multi-environment deployments, but they are often forced to stitch together fragmented tooling that lacks visibility, governance, and consistency.

At Harness, we are changing that.

Today, we are excited to announce native Terragrunt support in Harness IaCM, bringing it to full parity with Terraform and OpenTofu while delivering capabilities that go beyond what is available in standalone tooling. This is more than support. It is about making Terragrunt a first-class platform for enterprise infrastructure management.

With Harness IaCM, teams can now:

• Orchestrate complex Terragrunt environments with full visibility across all units
• Apply cost estimation, approvals, and policy enforcement natively
• Detect and manage drift across environments with granular insights
• View infrastructure changes at the resource level across orchestrated deployments

Terragrunt has become a critical layer for managing infrastructure at scale because it simplifies how teams structure and reuse configurations across environments. Harness builds on that foundation with deep, native integration, enabling platform teams to operate with both flexibility and control.

This is especially important for enterprises where a single deployment spans multiple environments and services. Harness abstracts that complexity while maintaining governance, auditability, and consistency.

Extending IaCM to a Multi-IaC Future

Terragrunt is part of a broader shift toward multi-tool infrastructure strategies.

Modern teams are no longer standardized on a single IaC tool. Instead, they operate across:

• Terraform and OpenTofu for provisioning
• Terragrunt for orchestration
• CDK for developer-driven infrastructure
• Ansible for configuration and automation

This creates challenges around consistency, visibility, and governance. Harness IaCM is built for this reality. We are evolving IaCM into a unified control plane for multi-IaC workflows, where teams can manage different frameworks with a consistent experience, shared policies, and centralized visibility.

This means:

• Eliminating fragmented pipelines across tools
• Standardizing governance across environments
• Gaining full visibility into infrastructure state and changes

Instead of managing infrastructure in silos, teams can now operate from a single platform across the entire lifecycle.

What’s Next for Infrastructure as Code?

The next phase of Infrastructure as Code is not just about supporting more tools. It is about making infrastructure systems more intelligent and automated.

We are investing in two key areas:

Expanded IaC Support

We are continuing to support modern frameworks like AWS CDK, enabling developer-centric infrastructure workflows alongside provisioning, configuration, and orchestration tools.

AI-Driven Automation

We are introducing intelligence into IaC workflows to simplify tasks such as drift management and optimization. This helps teams reduce manual effort and operate more efficiently at scale.

Together, these investments move IaCM toward a unified, multi-IaC platform that combines flexibility, governance, and automation. Terragrunt has become essential for managing infrastructure at scale but until now, it hasn’t had a platform that truly supports it. As infrastructure continues to grow in complexity, our focus remains the same. Helping teams move faster, reduce risk, and scale with confidence no matter which IaC tools they use.

The release of Anthropic Mythos and Project Glasswing marks an exciting and pivotal new chapter in software development. As the industry advances, the speed and economics of vulnerability exploitation have fundamentally shifted. What once took weeks of manual reconnaissance can now be scaled rapidly through automated models. However, this is not just a security problem to solve. It is a massive engineering opportunity to build cleaner, more robust systems. By leaning into AI-accelerated defense, engineering teams are uniquely positioned to lead the charge and redesign the landscape of modern software architecture.

Breaking Down Silos and Establishing Shared Accountability

To succeed in this new era, the traditional silos separating security and engineering must fall. Defense at machine speed requires a unified front.

• Organizations need a shared roadmap and accountability model across Engineering, Infrastructure, and Security.
• These roadmaps must be crafted jointly with clear responsibility assigned per action item.
• Every executive and their corresponding team will be affected and accountable for changing the way work is done.
• Preparations for these improvements should be treated exactly like new product features.
• Savvy customers will start to pay attention to companies who are responding to Mythos, turning your proactive resilience into a highly visible competitive advantage.

Core Engineering Imperatives

The foundation of AI-accelerated defense relies on sound, proactive engineering practices. Developers must take ownership of architectural hygiene from the ground up.

• Accelerate velocity: Teams must focus heavily on shortening patch and change cycles (such as with Harness CI and CD). The single most important metric is how quickly you can safely make changes.
• Shift left completely: You must find bugs before you ship code. Achieve this by integrating SAST, SCA, and auto-pen testing into a secure pipeline, and prefer using memory safe code languages.
• Design for resilience: Always build with breach assumed. In practice, this means implementing zero-trust, isolating services by identity, and using short lived tokens by default.
• Simplify the architecture: As you engineer and build for resilience and simplicity , take time to audit your current code base to reduce dependencies and standardize on known good services and libraries. Additionally, actively reduce and inventory what you expose.
• Pay attention runtime: Aside from bugs, engineering teams haven’t traditionally paid attention to the run-time security of their applications. Aside from the functional insights developers can glean from runtime security tools, understanding how a system is attacked can help you make better architectural and functionality decisions.

Planning for the Unexpected

Even with the best architecture, unexpected friction will occur. Resilient engineering means planning comprehensively for your ecosystem.

• Ensure you know your software dependencies and precisely who to contact in emergencies.
• Engineering teams should build technical work-arounds for times when providers or internal systems experience issues.
• Organizations must establish a surge defense capability. When faced with a severe situation, have a SWAT team established with pre-approved authority, budget, and standard operating procedures across domains and outside help.
• At the company level, pre-position high-visibility incident response. This includes having pre-approved and crafted messaging triggered by established conditions.

Security as an AI-Powered Partner

To keep pace with the increased velocity of engineering teams, Security teams must also evolve their operational models.

• Security needs to leverage AI to de-toil high calorie activities.
• Practical applications include putting a model in front of your alert queue and testing it regularly.
• AI should also handle the triage and prioritization of scan findings alongside ticket ops automation.
• It is crucial to automate the technical incident response pipeline.
• By automating the bookkeeping around incidents, human decisions should be made with assistance at most.
• The ultimate goal is to find places to leverage AI and accelerate the time between incident and resolution.

Leading the Charge

Engineering leaders and developers are in the perfect position to navigate this industry inflection point. By taking ownership of these structural changes today, you ensure the long-term viability of your products and the enduring strength of your codebase. Bring your security, infrastructure, and engineering teams together into the same room and start building your shared roadmap today.

What happens when your Infrastructure as Code management strategy works perfectly in dev, scales reasonably well in staging, and then quietly fractures across seventeen production workspaces because nobody documented which Terragrunt wrapper goes with which AWS account? You spend Friday afternoon reverse-engineering DRY patterns that made sense six months ago, wondering why your team is managing three different IaC execution engines with four incompatible workflow philosophies.

This scenario isn't hypothetical. It's the reality of organizations that adopted IaC incrementally, layer by layer, without a unified management approach. One team standardized on OpenTofu for new infrastructure. Another maintained legacy Terraform configurations because migration felt risky. A third discovered Terragrunt and used it to wrangle complexity across AWS regions, but now those wrappers exist outside any centralized governance model. Each decision was rational in isolation. Together, they created an orchestration problem masquerading as a tooling problem.

The actual challenge isn't choosing between Terraform, OpenTofu, or Terragrunt. It's managing their outputs, enforcing policy consistently across execution contexts, and ensuring that infrastructure changes don't outpace your ability to understand what's deployed.

The Hidden Complexity of Multi-IaC Environments

Most platform teams don't set out to run multiple IaC tools simultaneously. They inherit Terraform state from acquisitions, adopt OpenTofu for licensing predictability, and introduce Terragrunt because someone needed to stop copying backend configurations across 40 AWS accounts. The tools themselves aren't the problem. The problem is that each tool introduces its own state management assumptions, module resolution logic, and workflow expectations.

Terragrunt, for instance, exists specifically to solve Terraform's verbosity problem. It lets you define backend configurations once and reference them across environments. It supports dependency graphs so you can deploy a VPC before attempting to create subnets. These capabilities are valuable, but they also mean your actual infrastructure logic now spans two layers: the Terraform or OpenTofu code that defines resources, and the Terragrunt configuration that orchestrates execution.

When you lack centralized Infrastructure as Code management, those layers drift independently. Someone updates a Terragrunt dependency graph without realizing it breaks a downstream workspace. Another engineer modifies an OpenTofu module but forgets that three different Terragrunt configurations depend on its output structure. You don't discover these issues until a deployment fails in production, and the postmortem reveals that nobody had visibility into the full dependency chain.

Why Workflow Sprawl Defeats Governance at Scale

The typical response to multi-IaC complexity is to standardize on one tool and deprecate the others. That works if you're early in your IaC journey. It's impractical if you're managing hundreds of workspaces across regulated environments where compliance audits expect immutable infrastructure definitions and audit trails for every state change.

Here's what actually happens: platform teams create custom CI/CD pipelines for each tool. Terraform runs in Jenkins. OpenTofu runs in GitHub Actions. Terragrunt configurations use a shell script someone wrote during an incident. Each pipeline implements drift detection differently. Policy enforcement exists as scattered OPA rules that don't share a common evaluation context. When an auditor asks, "How do you prevent unapproved infrastructure changes?", the honest answer is, "We run some checks in some places, and we hope teams remember to use them."

This isn't negligence. It's what emerges when Infrastructure as Code management tooling doesn't natively support the reality of polyglot IaC environments. Teams need a system that treats OpenTofu, Terraform, and Terragrunt as execution details, not architectural boundaries. The workflow layer—plan generation, policy evaluation, approval gates, state locking—should remain consistent regardless of which engine interprets the configuration.

Infrastructure Automation Tools Need Orchestration, Not Just Execution

Running `terragrunt apply` successfully doesn't mean your infrastructure is well-managed. It means Terragrunt successfully invoked OpenTofu or Terraform and applied a configuration. The actual management work—validating inputs, enforcing cost policies, detecting drift, promoting changes through environments—exists outside the execution layer.

This is where most homegrown solutions collapse under their own weight. You build a wrapper script that runs Terragrunt with the right flags. Then you add pre-commit hooks for policy checks. Then you integrate Sentinel or OPA, but only for workspaces that someone remembered to configure. Then you add Slack notifications so people know when drift occurs, but the notifications don't include enough context to act on them. Eventually, you have a Rube Goldberg machine that works until it doesn't, and debugging requires institutional knowledge that exists in one person's head.

The fundamental issue is that IaC workflow optimization requires thinking beyond execution engines. You need orchestration that understands module dependencies, workspace relationships, and policy boundaries. You need variable management that doesn't require copying YAML files between repositories. You need drift detection that runs automatically and surfaces meaningful deltas, not raw Terraform output dumped into a log file.

Terragrunt Support as a First-Class Workflow Primitive

Treating Terragrunt as an afterthought—something teams bolt onto existing Terraform or OpenTofu pipelines—misses its architectural intent. Terragrunt exists because managing backend configurations, passing outputs between modules, and orchestrating multi-account deployments shouldn't require copying boilerplate across dozens of directories. When Infrastructure as Code management platforms support Terragrunt natively, they acknowledge this reality: the DRY principle applies to infrastructure orchestration, not just resource definitions.

Native Terragrunt support means the platform understands dependency graphs without requiring custom parsing logic. It means workspace templates can reference Terragrunt configurations directly, rather than forcing teams to flatten everything into monolithic Terraform modules. It means policy enforcement applies before Terragrunt invokes the underlying execution engine, catching invalid configurations before they generate failed plans.

This matters most in organizations running multi-region or multi-cloud architectures. A typical pattern: one Terragrunt configuration defines networking across AWS regions, another manages Kubernetes clusters, a third provisions databases. Each configuration depends on outputs from the others. Without native orchestration, teams either write brittle shell scripts to sequence these dependencies or accept that deployments sometimes fail halfway through because someone applied changes out of order.

Multi-IaC Tools Require Unified State and Policy Management

The real test of an Infrastructure as Code management platform isn't whether it runs OpenTofu or Terraform. It's whether it provides consistent state visibility, policy enforcement, and audit trails across both. If your platform requires separate workflows for each execution engine, you've automated the mechanics but not the governance.

Consider policy evaluation. A reasonable security requirement: no S3 buckets should allow public read access. With fragmented tooling, you implement this rule multiple times. Once for Terraform workspaces using Sentinel. Again for OpenTofu configurations using OPA. A third time for Terragrunt-managed infrastructure, where you're not sure which policy engine applies because Terragrunt is just orchestrating calls to Terraform or OpenTofu. When an audit occurs, you can't prove consistent enforcement because there's no unified policy evaluation layer.

The same fragmentation affects drift detection. Terraform Cloud detects drift for Terraform-managed resources. Your OpenTofu workspaces might run scheduled reconciliation jobs, or they might not—it depends on whether someone configured them. Terragrunt configurations drift silently unless you've built custom tooling to periodically run `terragrunt plan` and parse the output. The result: partial visibility across your infrastructure estate, where "managed by IaC" becomes aspirational rather than descriptive.

OpenTofu Integration and Terraform Alternatives in Practice

Organizations exploring Terraform alternatives often focus on licensing or community governance. Those considerations matter, but they don't address the operational question: how do you manage infrastructure deployed with multiple execution engines without creating parallel workflow systems?

OpenTofu integration means more than "we can run OpenTofu commands." It means workspaces provisioned for OpenTofu behave identically to Terraform workspaces at the orchestration layer. Variable sets apply consistently. Policy evaluation uses the same rule sets. Drift detection runs on the same schedule. Approval workflows follow the same governance model. The execution engine becomes an implementation detail, not a workflow boundary.

This distinction matters during migrations. Teams don't flip entire infrastructure estates from Terraform to OpenTofu overnight. They migrate incrementally, starting with non-critical workspaces and expanding as confidence grows. If your Infrastructure as Code management platform treats each engine as a separate silo, you're managing two parallel systems during the transition. If the platform abstracts execution details behind a unified orchestration layer, the migration becomes a configuration change, not an architectural overhaul.

IaC Orchestration Beyond Engine Selection

The hard problems in infrastructure management aren't technical; they're organizational. How do you ensure that 40 engineers across six teams follow the same approval process for production changes? How do you enforce cost policies without blocking legitimate deployments? How do you maintain audit trails that satisfy compliance requirements without turning every infrastructure change into a bureaucratic ordeal?

IaC orchestration platforms solve these problems by decoupling policy from execution. Instead of embedding governance rules in CI/CD pipelines—where they're invisible, untestable, and easy to bypass—you define them once at the platform level. Instead of writing custom scripts to sequence Terragrunt dependencies, you describe the dependency graph declaratively and let the platform handle execution order. Instead of building bespoke drift detection logic, you configure detection schedules and let the platform surface meaningful deltas.

This approach doesn't eliminate complexity. It consolidates complexity into a layer designed to manage it. Your IaC configurations remain simple: modules that define resources, Terragrunt wrappers that eliminate boilerplate, workspace configurations that specify execution context. The orchestration platform handles everything else: state locking, policy evaluation, approval workflows, audit logging, drift remediation.

Harness IaCM: Orchestration for Multi-IaC Environments

Harness Infrastructure as Code Management approaches these challenges by treating the execution engine as a deployment detail, not an architectural constraint. Whether you're running OpenTofu, Terraform, or Terragrunt, the orchestration layer remains consistent: standardized pipelines for plan generation and apply operations, unified policy enforcement across all workspaces, centralized drift detection that surfaces actionable insights.

• Native Terragrunt support means dependency graphs defined in Terragrunt configurations are understood and respected during execution. You don't need custom scripts to sequence deployments across modules or AWS accounts. The platform interprets dependencies and orchestrates execution accordingly, applying changes in the correct order while maintaining state consistency.

• The Module and Provider Registry provides a centralized source of truth for infrastructure components, whether they're Terraform modules, OpenTofu modules, or Terragrunt configurations. Variable Sets and Workspace Templates eliminate configuration duplication, letting you define environment-specific values once and reference them across workspaces. Default plan and apply pipelines ensure consistent execution patterns without requiring custom CI/CD configurations for each workspace.

• Policy enforcement happens before execution, not after. Whether you're using Terraform, OpenTofu, or Terragrunt, policies evaluate against the generated plan before any infrastructure changes occur. Drift detection runs automatically, comparing deployed infrastructure against IaC definitions and surfacing discrepancies through a unified interface. Audit trails capture every state change, policy evaluation, and approval decision, providing the compliance evidence required in regulated environments.

For teams managing infrastructure across multiple clouds, regions, or execution engines, Harness IaCM provides the orchestration layer that makes polyglot IaC environments manageable. The platform doesn't force you to standardize on a single tool. It provides governance, visibility, and workflow consistency regardless of which engine interprets your configurations.

Governance Enables Velocity in Multi-IaC Environments

The promise of Infrastructure as Code—reproducible deployments, version-controlled infrastructure, collaborative development—only materializes when you have consistent orchestration across execution engines. Running Terraform in one pipeline, OpenTofu in another, and Terragrunt through a shell script doesn't scale. It creates workflow fragmentation that defeats governance and slows teams down.

Effective Infrastructure as Code management platforms abstract execution details behind unified workflows. They treat Terragrunt as a first-class orchestration primitive, not an afterthought. They provide native support for OpenTofu alongside Terraform, recognizing that organizations migrate gradually, not overnight. Most importantly, they enforce policy, detect drift, and maintain audit trails consistently across all workspaces, regardless of which engine runs the actual infrastructure changes.

The technical lesson: orchestration complexity belongs in platforms designed to manage it, not scattered across custom scripts and fragmented CI/CD pipelines. The operational lesson: governance doesn't slow teams down when it's embedded in the workflow rather than bolted on afterward. Multi-IaC environments are manageable when you have the right orchestration layer. Without it, you're just running tools in parallel and hoping they don't conflict.

Explore how Harness Infrastructure as Code Management handles multi-IaC orchestration, or review the technical documentation or implementation details. The product roadmap outlines upcoming capabilities for workflow optimization and policy enforcement.

The Gap: Modern DevOps Stops at the Database

Most development teams today build everything around Git, and deploy with GitOps principles.

Code sits in version controlled environments, changes go through PRs, and deployments are handled through modern CI/CD. That part is pretty standard at this point, especially when using a modern DevOps platform like Harness.

MongoDB fits into that developer world and workflow pretty naturally. Data is stored in documents that look a lot like JSON, the format many developers already use in application code and APIs. Under the hood, MongoDB stores those documents as BSON, which is essentially a binary form of JSON that supports additional data types like dates, object IDs, and binary data. That means developers get a familiar model to work with, while MongoDB gets a format that is efficient for storing and querying application data. 

Looks just like JSON, with native types like ObjectId and dates powered by BSON.

The tradeoff is that structure isn’t always defined upfront. Schemas change over time, and not always in a clean or consistent way.

Collections can contain documents with different shapes. Index changes can directly impact performance. These aren’t problems on their own, but they require discipline to manage safely.

MongoDB changes are often handled outside the standard development workflow, whether that’s by developers, platform teams, or database teams.

Teams rely on application-level updates or one-off scripts to backfill data, modify structures, or create indexes. These approaches work, but they’re not always consistently versioned in Git. Execution can vary across environments, and review or validation is often informal.

The result is limited visibility into what changed, when it changed, and how it was applied. Over time, that leads to inconsistencies between environments and increased risk during deployment. 

Flexibility is powerful, but without proper controls it introduces risk.

To solve this, teams need to bring MongoDB changes into the same workflow they already trust for application code: Git-driven, reviewable, and automated.

What GitOps for MongoDB Actually Looks Like

GitOps for MongoDB isn’t about changing how Mongo works. It’s about changing how changes are managed.

Instead of handling updates through scripts or application logic alone, database changes are treated like application code. Index creation, schema validation rules, and migration scripts are all defined in Git and tracked over time. This includes MongoDB’s native schema validation rules, which can be versioned and applied consistently across environments.

Changes need to go through pull requests, just like any other code change. This allows developers, platform teams, and DBAs to review what’s being modified before anything runs in an environment.

From there, pipelines handle the validation and deployment. Changes are applied consistently across environments, rather than being run manually and potentially differently each time.

In practice, this means a new field, an index, or a backfill isn’t just a script someone runs once. It’s a versioned change that can be reviewed, tested, and repeated.

This isn’t about forcing rigid schemas onto MongoDB. It’s about making changes visible, consistent, and easier to manage as systems grow.

Harness DB DevOps provides the structure to do this. With Harness, we define changes as changesets, store them in Git, and deploy them through pipelines with built-in validation and policy checks.

To demonstrate how this works, we will walk through a practical MongoDB change from start to finish.

Example: Managing a MongoDB Change End-to-End

Here’s a simple example: A team needs to add a new userPreferences field to the users collection and create an index to support a new query.

Instead of writing a script and running it manually, we define the change and commit it to Git.

1. Define the change in Git
A developer creates the update as a changeset. That includes the logic to add or backfill the new field, along with the createIndex operation needed for performance. The change is committed alongside application code, like any other update.

2. Open a pull request
From there, the change goes through a pull request. Other developers or DBAs can review what’s being changed before anything runs. If something looks off, it gets caught here instead of in production.

3. Let the pipeline take over
Once the change is approved, the pipeline takes over.

The Pipeline

Before anything gets applied, the change is validated and previewed against the target environment. This helps catch issues early, whether it’s a conflict, a bad query pattern, or something that could impact performance.

This is especially important for heavy operations like index creation on massive collections, where resource contention and performance degradation are real risks. Instead of running those changes manually, pipelines can enforce safe rollout strategies like rolling index creations across replica sets, without manual intervention.

Policies are enforced as part of that same process, with required approvals, environment rules, and other guardrails checked automatically so teams aren’t relying on someone to manually verify every step.

Once everything passes, the change is deployed through the pipeline and applied consistently across environments, moving from dev to staging to production in a controlled way. No one is logging into a database to run scripts by hand.

Now, everything is tracked. You can see what was applied, where it was deployed, when it happened, and who approved it, with a full history available if something needs to be reviewed or rolled back later.

Sound familiar? This workflow should sound a lot like application delivery, where changes are versioned, reviewed, validated before deployment, and visible after.

From Manual Oversight to Guardrails

Traditionally, database changes have been tightly controlled by DBAs. They review scripts, approve changes, and sometimes execute them manually in each environment. That model helps reduce risk, but it doesn’t scale as teams grow and release more frequently.

With a GitOps approach, that control doesn’t disappear, it moves earlier in the process.

Instead of reviewing every individual change, database teams define policies and standards up front. Those rules are then enforced automatically through pipelines. Every change must pass the same checks before it reaches an environment, without requiring manual intervention each time.

In practice, this means:

• Required approvals are enforced automatically
• Deployment rules are validated before execution
• Production environments can have stricter controls than lower environments
• Policies are applied consistently across every change

The role of the database team evolves from gatekeeper to system designer. Rather than being involved in every deployment, they define the guardrails that ensure every deployment is safe.

Developers still move quickly, but now within a controlled, repeatable system.

What Teams Actually Gain

Bringing MongoDB into a Git-driven workflow changes how teams ship.

• Faster delivery
  Developers don’t wait on manual reviews or execution. Changes move through the same workflow as application code.
• Consistent environments
  Changes are applied the same way every time, reducing drift between dev, staging, and production.
• More predictable deployments
  Validation and policy checks happen before execution, catching issues earlier and reducing surprises.
• Built-in auditability
  Every change is versioned, reviewed, and tracked, with full visibility into what changed, when, and why.

MongoDB's flexibility doesn't eliminate the need for structure - it just shifts the responsibility for maintaining consistency from the database itself to your development processes.

If your application is managed through Git, your database should be too.

Schedule a demo

If you've ever run an ALTER TABLE on a busy MySQL table in production, you know the feeling. The change is small. The risk isn't. Long-running table locks, queued writes, application timeouts, replication lag, a five-minute migration that turns into a half-hour incident review.

We're shipping an integration that takes that anxiety out of the loop. Harness Database DevOps now supports Percona Toolkit for MySQL as part of Liquibase-based schema management. Flip a checkbox at schema creation, and eligible changes execute through pt-online-schema-change instead of native MySQL DDL.

Why it matters

Native ALTER TABLE on MySQL can lock tables for as long as the change takes to apply. On a large or hot table, that means writes pile up, dependent services start timing out, and replicas fall behind.

Percona Toolkit handles the same change very differently. pt-online-schema-change creates a shadow table with the new schema, copies your data over in small chunks, uses triggers to keep the original and shadow tables in sync, then performs an atomic swap with minimal lock time. The practical upside: schema changes you can run during business hours, not at 2 AM with a runbook open.

How to turn it on

The integration is enabled per schema. When you create a Database Schema in Harness DB DevOps:

1. Fill in your MySQL connection details.
2. Select Liquibase as your schema configuration.
3. Check the box labeled Use Percona Toolkit.
4. Save and continue with your normal workflow.

That's it. With the box unchecked (the default), Harness DB DevOps applies your changelogs using native MySQL operations through Liquibase, exactly as before. Check it, and eligible changes route through Percona Toolkit instead.

Things worth knowing before you flip the switch

Percona Toolkit isn't a silver bullet for every DDL. A few cases need extra thought.

Adding or dropping foreign keys can break during the table swap, so plan those changes carefully or apply them outside the toolkit. Tables without a primary key or unique index won't migrate safely either, since pt-online-schema-change needs one to chunk data deterministically. And a handful of specific operations sit outside the safe-change envelope: dropping a primary key, complex column reordering, and some storage engine swaps.

You'll also want to give the database user the right privileges: ALTER, SELECT, INSERT, and UPDATE on the target table, plus CREATE and DROP on the database for shadow table management.

The full list of supported patterns, edge cases, and required permissions is in the Harness DB DevOps docs.

Try it on your next migration

If you're already running Harness DB DevOps for MySQL, the next schema you create is a good place to try this. Turn it on against a non-critical environment first, watch how it behaves on your workload, and the path to using it in production gets a lot shorter.

For teams running MySQL at scale, that's one fewer reason to schedule schema changes around your customers' sleep.

If you aren't already using Database devops, speak with our experts to discuss how you can achieve zero downtime database schema migrations.

• Mean Time to Failure (MTTF) turns repeated failures into a signal that can be predicted, allowing platform teams to plan for the number of incidents, the amount of capacity they need, and the SLOs they can actually meet.
  
• To get an accurate Mean Time to Failure, you need to have clean data, clear definitions of failure, and separate short-lived components from long-lived services instead of averaging everything together.
  
• Connecting Mean Time to Failure with MTTR, SLOs, and AI-powered automation from Harness changes how you think about failures from random chaos to an automated control loop that runs reliability.

‍

Your production problems aren't just random. If a Kubernetes node fails every 72 hours or your CI runners crash every 4 builds, that's a clear pattern. Mean Time to Failure (MTTF) turns these failures into data that you can control, plan for, and improve over time.

MTTF should not be a decoration on a dashboard for platform engineering leaders; it should be a decision-making tool. With the right calculations, you can set realistic SLOs, plan capacity, and cut down on developer work by focusing on the parts that break the most often. You'll get exact formulas for distributed systems, data collection patterns that avoid common mistakes, and a playbook to turn reliability improvements into measurable ROI through automated resilience practices alongside faster recovery metrics.

Stop letting unpredictable failures drain your team's time and budget. With Harness Continuous Integration and Continuous Delivery, you can turn MTTF insights into concrete pipeline changes, progressive delivery strategies, and guardrails that keep reliability improving release after release.

What Is Mean Time to Failure (MTTF)?

Mean Time to Failure (MTTF) is the average operating time of non-repairable components before failure across a population.

At a basic level:

MTTF = total operating time ÷ number of failures

If 100 CI runners each run for 50 hours during a week (5,000 runner‑hours total) and 20 runners experience at least one hard failure, then:

MTTF = 5,000 ÷ 20 = 250 hours

Historically, MTTF is used for physical assets you replace instead of fix (light bulbs, disks, sealed devices). In software, the same concept fits ephemeral resources such as:

• Short‑lived containers and pods.
  
• CI/CD job runners and build agents.
  
• Batch jobs and serverless functions that are recreated after failure.

MTTF tells you how long things run, on average, before they fail and must be replaced. MTTF is an approximation, not a strict reliability model.

MTTF vs MTTR vs MTBF in DevOps Workflows

Three reliability metrics show up in every platform review:

• Mean Time to Failure (MTTF):  Measures how long non‑repairable or ephemeral components run before failing and being replaced.
  
• Mean Time to Repair/Restore (MTTR): Measures how long it takes to restore a service to healthy operation after a failure.
  
• Mean Time Between Failures (MTBF): Measures how much uptime you get between failures for systems that are repaired and returned to service.

Use them to answer different questions:

• MTTF: How often do our ephemeral components fail?
  
• MTTR: How quickly can we restore user‑visible services when they do?
  
• MTBF: How stable are our long‑lived systems between failures?

For example:

• Track MTTF for pod lifetimes or CI runner stability so you can forecast incident volume and choose when to add redundancy or auto‑healing.
• Track MTTR for customer‑facing services and tie it to SLOs and incident response.
• Track MTBF for databases or API gateways so you can plan maintenance windows and capacity around real behavior instead of theoretical SLAs.

Your platform scorecards should display all three together, alongside SLO health and error budget burn, so teams see the full reliability picture instead of optimizing a single metric in isolation.

When MTTF Applies to Software

The theoretical rules around MTTF and MTBF are straightforward; the ambiguity comes when you apply them to real cloud‑native stacks. Concrete examples help.

Where MTTF is Most Meaningful

These components typically behave like non‑repairable items:

• Pods and containers that are recreated on failure.
  
• CI runners and ephemeral build agents that are provisioned per job or per batch.
  
• Batch jobs and workers that run to completion and are never “repaired” mid‑flight.
  
• Serverless functions are triggered on demand and recreated automatically.
  

For each of these, you can treat a single lifecycle (from start to failure/termination) as one observation in your MTTF dataset.

Where MTBF (Plus MTTR) is A Better Fit

These components behave more like classic repairable systems:

• Databases and stateful services that are brought back through restart, patch, or failover.
  
• API gateways, control planes, and load balancers that remain logically continuous across many incidents.
  
• Long‑running worker pools that survive node or pod replacements.
  

For these, you care more about how much uptime you get between failures (MTBF) and how quickly you can restore full health (MTTR).

The Common Pitfall: Equating Infra MTTF With User Reliability

It is tempting to say “our nodes have an MTTF of 720 hours, so our service is very reliable.” That is only true if your architecture masks those failures from users. User‑facing reliability lives at the service boundary, measured via SLOs and error budgets; component MTTF is an input that helps you:

• Find noisy dependencies.
  
• Plan redundancy and failover.
  
• Decide where to add automation or chaos tests.
  

MTTF helps you understand where things break; SLOs and MTTR tell you how much that matters to customers.

How to Calculate MTTF for Distributed Systems

The MTTF calculation is trivial. The work is in collecting honest data across a distributed system without losing important details.

1. Define Specific Failure Events

For each component type, decide exactly what counts as “failed,” for example:

• Container exits with a non‑zero status.
  
• Pod enters CrashLoopBackOff or is evicted for resource pressure.
  
• CI pipeline reaches a terminal “failed” state (not just retried).
  
• Node is in NotReady for longer than an agreed threshold.

Document these in your platform taxonomy so every team logs and reports failures the same way.

2. Track Lifecycle Start and End for Each Instance

For each instance in the population you’re measuring, capture:

• Start time – when the instance began its lifecycle.
  
• End time – either the failure time or the end of your observation window.
  
• Instance identifier – pod name, job ID, runner ID, etc.

Then compute:

MTTF = total operating time across all instances ÷ number of failed instances

This gives you MTTF for that class (e.g., “Linux GPU runners in prod”).

3. Segment by Workload Class and Aggregate With Weights

Never pool dissimilar components into a single MTTF number. Instead:

• Group instances into workload classes (service, environment, hardware profile, or usage pattern).
  
• Compute MTTF for each class independently.
  
• When needed, roll up with weighted exposure hours, not a simple average.

Example:

• Class A: 1,000 hours, 5 failures → MTTF_A = 200 hours.
  
• Class B: 100 hours, 1 failure → MTTF_B = 100 hours.

Fleet MTTF (weighted) = (1,000 + 100) ÷ (5 + 1) ≈ 183 hours, not the naive (200 + 100) ÷ 2.

4. Include Right‑censored Data

Some instances will still be running when you take the snapshot. If you drop them:

• You shrink total operating time.
  
• You keep the same number of failures.
  
• Your MTTF ends up artificially low.
  

When censored samples are common, use basic survival analysis (like Kaplan–Meier) so that "still running" instances add to the exposure instead of being thrown away. If you give them clear timestamps and labels, observability tools and data teams can usually take care of this for you.

Using MTTF to Set SLOs and Reduce Toil

MTTF becomes strategically important when you use it to shape SLOs, error budgets, and reliability investments, not just track uptime.

1. Project Incident Volume From MTTF

If a class of components has an MTTF of 72 hours, a single instance will fail about:

8,760 hours/year ÷ 72 ≈ 121 failures/year

With multiple instances and redundancy, not every failure becomes a user‑visible incident, but you can still estimate:

• Roughly how many failures will your platform team see?
  
• How often will those failures stress specific SLOs or error budgets?

2. Prioritize Components That Generate the Most Toil

MTTF highlights which components generate excessive manual work:

• A CI runner class with low MTTF forces engineers to babysit builds.
  
• A particular pod type that fails often might drive a disproportionate share of pages.

Use this to:

• Rank components by failure frequency × human touch time.
  
• Focus on resilience and automation work where you retire the most pages and tickets per unit effort.

3. Translate MTTF improvements into business outcomes

Because MTTF underpins incident rates, any improvement can be tied to measurable gains:

• Fewer incidents and on‑call pages.
  
• Less time lost to debugging failures that do not reach customers.
  
• Less need for excess capacity added as a buffer against frequent failures.

Treat MTTF as a leading indicator: when you raise it on critical components, you should see downstream improvements in SLO attainment and delivery cadence.

Practical Ways to Improve Mean Time to Failure

Once you know which components have the lowest MTTF and the highest operational cost, you can systematically improve them. In modern delivery pipelines, four patterns tend to pay off quickly.

1. Stabilize CI Pipelines and Build Infrastructure

Flaky CI is one of the most common sources of low MTTF and wasted engineering time.

You can improve CI‑related MTTF by:

• Reducing test‑driven failures with Harness Test Intelligence, which selects only tests relevant to a change and helps you isolate flaky tests instead of letting them hammer your MTTF.
  
• Identifying failure hot spots using Harness CI analytics and insights to see which repos, branches, or services correlate with most build failures.
  
• Shortening build exposure windows using incremental builds, which reuse previous build outputs, reducing the time during which build infra can fail.

Result: higher MTTF for pipelines and runners, fewer broken builds, and fewer interruptions for developers.

2. Use Progressive Delivery and Rollback to Protect Service MTTF

You cannot prevent every bad change, but you can limit how many become full‑blown incidents that count against your service‑level MTTF.

Key tactics:

• Design for redundancy across environments with Harness “deploy anywhere”, so a single node or zone failure does not reset service health.
  
• Configure canary, blue‑green, or rolling strategies in Harness “powerful pipelines” so new versions are exposed gradually and can be rolled back quickly.
  
• Wire in verification steps so bad versions are caught early rather than after a significant SLO hit.

This keeps effective MTTF for user‑facing services higher, even if underlying components still fail regularly.

3. Enforce Reliability Guardrails With Pipeline Governance

Many MTTF regressions start as “just one more config change” that slips past informal reviews. Prevent those with:

• Policy‑as‑code in Harness DevOps pipeline governance, where you define rules around high‑risk changes, mandatory checks, and SLO states.
  
• Conditional approvals for changes that touch historically fragile services or dependencies.
  
• Blocks on deployments when key reliability gates, like SLO burn or unresolved incidents, are breached.

This ensures the MTTF gains you’ve earned are not eroded by ad‑hoc changes and one‑off exceptions.

4. Continuously Validate Resilience With Chaos Engineering

To sustainably raise MTTF, you need confidence that your architecture and runbooks can handle real failures, not just happy‑path tests.

By running targeted chaos experiments on the components with the lowest MTTF, you can:

• Discover previously hidden failure modes.
  
• Fix them before they cause production incidents.
  
• Demonstrate to stakeholders how your changes improve resilience over time.

Monitoring and Improving MTTF with AI‑Powered Automation

When failures happen, MTTF tells you how often they occur. AI‑powered automation helps you decide what to do next—fast—so more failures stay under control and never become major incidents.

AI‑assisted Detection and Rollback

Harness AI‑assisted deployment verification analyzes metrics and logs during and after each deployment:

• It learns what “normal” looks like for each service.
  
• It flags anomalies in latency, error rates, or custom SLO indicators.
  
• It can recommend or trigger rollback directly through your CD pipelines.
  

The result is fewer deployments turning into user‑visible failures and a higher effective MTTF for your services, because many problematic changes are automatically rolled back before customers notice.

On the CI side, AI‑driven analysis works with Test Intelligence and analytics to:

• Cluster failures by likely root cause.
  
• Surface patterns, such as a particular dependency, test suite, or environment causing repeated failures.
  
• Guide you toward changes that will increase MTTF for builds and jobs.

SLO‑driven Guardrails Instead of Ad‑hoc Decisions

SLOs and error budgets turn raw data into rules. Instead of making teams watch dashboards and make decisions on their own, you can:

• Use SRM to set SLOs for important services and keep an eye on how much of the error budget is being used.
  
• Set up pipeline rules that automatically slow down or stop deployments when SLOs are in danger.
• Direct engineering efforts toward services with a decreasing MTTF and an increasing error-budget burn.

This completes the cycle: MTTF informs SLO design. Guardrails are based on SLOs, and AI-powered verification and rollbacks work on those guardrails at machine speed.

Want to turn MTTF insights into automated reliability improvements?

Explore Harness CI/CD to reduce failure rates, enforce guardrails, and improve SLO performance.

MTTF: Frequently Asked Questions (FAQs)

MTTF can feel abstract until you have to justify reliability decisions or explain incident patterns to stakeholders. These FAQs break down the most common questions practitioners ask about MTTF and how it relates to other reliability metrics.

How is MTTF different from MTBF?

MTTF is the average time it takes for a group of parts, like pods or temporary CI runners, to fail in a way that can't be fixed. MTBF tells you how long systems you fix and put back into service, like databases or long-running services, are up and running before they break down again.

When should I use MTTF instead of MTTR?

When you need to know how often failures happen so you can plan for redundancy or auto-healing, use MTTF. Use MTTR to find out how quickly you can fix services that users can see after they go down. Both metrics work together and are usually used to help make decisions about SLOs and error budgets.

Can I trust MTTF if I only have a few failures?

MTTF estimates are very uncertain when there aren't many failures. To make the number more reliable, put similar workloads together, add up the exposure hours for each class, and think of MTTF as a range or trend instead of a single point. If a part didn't fail in your window, don't assume that it will never fail; instead, treat that as incomplete data.

What data quality issues most often skew MTTF?

Most of the time, MTTF is skewed by dropping instances that are still running when the measurement is taken (right-censoring), combining environments (staging, load, and production) into one metric, and having different or unclear definitions of failure across teams. Fixing these problems usually makes MTTF more useful than any other advanced statistical method.

When is MTTF the wrong metric for modern platforms?

MTTF doesn't work when failures are very similar or when you're measuring systems that are fixed instead of replaced. In those cases, MTBF and MTTR, when looked at through SLOs and error budgets, usually give better advice than just one MTTF value.

How does MTTF connect to business outcomes?

When the MTTF is higher on important parts, there are fewer problems, fewer pages, and less time lost by developers fixing them. You can link improvements directly to faster safe release velocity, lower downtime risk, and lower operational costs when you combine MTTF with SLOs, error budgets, chaos engineering, and AI-powered automation.

Modern application security goes from code to runtime. Vulnerabilities are found at every stage of the software development lifecycle (SDLC) - in the code developers write, open source packages they pull in, container images they build, and cloud infrastructure where it all runs. But finding vulnerabilities is no longer enough. With attack surfaces sprawling across pipelines, registries, and production environments, the harder problem is fixing the vulnerabilities that actually matter.

Understanding what’s important increasingly depends on correlating multiple data points. A critical CVE buried in a dependency looks very different depending on whether the vulnerable function is actually reachable, the library is used in production, or the affected service is internet-facing. Without runtime context, security and development teams are often left triaging noise instead of actually reducing risk. And fixing vulnerabilities discovered in production can be challenging without being able to follow the trail back to the repo and line of code where the vulnerability can be found.

No matter where application security lives in your organization - and increasingly, it lives in more than one place - Harness and Wiz are working together to make sure you're covered. Whether your team is shifting left from cloud security or pushing right from the development pipeline, integrating Harness and Wiz brings code and runtime findings together so you always have the context you need to act.

Shift Left, Protect Right. Who Owns Application Security?

Application security used to have a clear owner. The AppSec team ran the scanners, triaged findings, and created tickets for developers. But "shift left" has been pushing security earlier into the development process and ownership has been migrating toward the teams that actually write and ship code. Today, the DevSecOps or platform engineering team owns application security tooling in many organizations. They're the ones who know exactly where a vulnerability lives in code, who owns it, and how to get developers to fix them.

But as applications move to the cloud, cloud security and infrastructure teams have a stake in application security outcomes as well. They're the ones with visibility into what's actually running in production - what's internet-facing, what's over-privileged, what's actively being exploited. Cloud security platforms have expanded their focus from purely infrastructure and runtime back through the SDLC to code. For many cloud teams, application security isn't a handoff; it feeds into their cloud risk picture.

The result is that application security now has multiple stakeholders with different vantage points. DevSecOps teams see risk through the lens of the CI/CD pipeline and the developer workflow. Cloud security teams see it through the lens of the deployed environment and the blast radius of a breach. Neither view is complete on its own. The good news is that these teams don't have to choose between their tools or their workflows. They need integration that lets each team work in their context while sharing the signals that make both more effective.

Integrating Wiz Code into CI/CD Pipelines with Harness STO

DevSecOps teams need to expand right. SAST and SCA tools often generate more findings than any team can fix. Runtime context helps separate signal from noise. Knowing that a vulnerable service is actively internet-facing or that a dependency with a critical CVE is actually loaded in production changes how a team prioritizes. Without it, developers are left triaging based on CVSS scores alone. With it, they can focus effort where exposure is real and the risk in production is highest.

Harness Security Testing Orchestration (STO) makes it easy to orchestrate Wiz Code across your CI/CD pipelines. With a pre-built integration, you can deploy Wiz Code in just a few clicks instead of needing to create a custom integration or write custom scripts. Harness orchestrates Wiz Code alongside all your other scanners so you know your pipelines always get the required security tests, without needing to manually coordinate multiple tools.

Once Wiz Code is integrated, STO aggregates findings with other scanners in your pipeline, automatically deduplicating vulnerabilities so teams aren't triaging the same issue twice. The consolidated view means developers and security engineers can see the full picture in one place, understanding pipeline-level risk and assigning tickets to developers. In addition, Harness Policy as Code lets teams define and take action at the pipeline level instead of tool by tool, so decisions about what to fail a build on, what to flag for review, and what to pass through are applied consistently and holistically across every scan and pipeline.

Feeding Harness SAST and SCA Findings Into Wiz

Cloud security is pushing left - past runtime, past containers, all the way back to the code and open source packages that vulnerabilities originate from. The driver is enabling action. A misconfigured cloud resource or a vulnerable container image is more actionable when you can tie it back to the specific dependency introduced in a pull request, the developer who owns the code, and the pipeline that shipped it. Runtime findings without code context are just alerts. With code context, they become actionable work items that can be routed to the right person and fixed at the source.

Wiz Application Security Posture Management (ASPM) is designed to aggregate findings from across the SDLC and correlate them with runtime context - what's deployed, what's exposed, and what's actually at risk. By integrating Harness SAST and SCA scanner findings directly into Wiz, cloud security teams can connect the dots between a vulnerable open source package or insecure code pattern and the running workloads it affects. That correlation is what turns a list of CVEs into a prioritized risk picture that reflects what's actually happening in production.

For cloud security teams already working in Wiz, this integration means Harness SAST and SCA become part of their existing workflow rather than a separate tool to check. Code-level findings surface alongside runtime signals in the same platform where cloud risk is already being managed, analyzed, and acted on. Teams get broader coverage without adding friction, and the context that makes those findings meaningful - reachability, exposure, business criticality - is already there when they need it.

AppSec Is a Team Sport. Your Tools Should Be Too.

DevSecOps and cloud security teams are not generally not competing - they're looking at risk from different angles. One team lives in the development pipeline; the other lives in the cloud. Both need visibility into what the other sees to do their jobs well. When those views are siloed, findings get duplicated, priorities diverge, and the vulnerabilities that matter most fall through the cracks between teams.

Harness and Wiz close that gap from both directions. DevSecOps teams get runtime signals from Wiz Code inside the pipeline context where they already work, so they can prioritize fixes based on real-world exposure. Cloud security teams get code-level findings from Harness SAST and SCA inside the risk context where they already work, so they can trace production risk back to its source. Each team keeps their workflow. Both teams get the full picture.

The right combination of these integrations depends on how your organization is structured, where application security ownership sits today, and where you want it to go. If you're a Wiz customer evaluating how Harness SAST and SCA fit into your security program, or a Harness customer looking to bring runtime context into your pipelines, contact your Harness account team to understand how you can map the integrations to your specific environment.

Application security testing tools promise coverage and accuracy, but teams often struggle just to get started. One of the biggest friction points in dynamic application security testing is configuring authentication correctly so a scanner can even access a target application, let alone API endpoints that power the functionality. 

Whether it’s API keys, bearer tokens, or custom auth flows, setting up authentication for scans frequently requires trial-and-error and engineering support. This reality of scanning configuration slows down security validations, delays insights, and makes it difficult to integrate with AI-driven tooling that depends on fast, accurate access to API endpoints.

Today, we’re excited to introduce AI-Powered Custom Authentication Generation—a new capability designed to eliminate this friction and help teams move from setup to security insights faster than ever.

What’s New: AI-Powered Authentication Generation

With this release, teams can now generate and refine authentication configurations using natural language and LLMs. Instead of manually configuring authentication logic or relying on additional support, users can simply describe their requirements and let AI handle the rest.

The average time to configure authentication for API security testing is measured in seconds, whereas older manual approaches can take hours and require extensive trial-and-error. 

Here are a few highlights:

• Use Natural Language to Create Auth Scripts: Generate fully functional authentication configurations by describing your needs in plain English.
• Support for Common Auth Types: Easily create configurations for API keys, JWTs, Bearer tokens, and more.
• Iterative Refinement with AI: Update and fine-tune authentication logic by prompting the system instead of time-consuming scripting or manual adjustments.
• Inline Visibility and Change Tracking: Steps taken by AI to generate auth flows are audited, and testing runs are logged fully to maintain transparency and control.

Why is Authentication Setup Difficult Historically?

Authentication setup has long been one of the most frustrating parts of security scanning. Access control mechanisms are already complex due to security hardening used to protect applications and APIs. Successfully automating authentication flows so a machine can access an app or endpoint raises the bar substantially. 

Some of the common pain paints include:

• Manual configuration slows down work: Teams often need to consult different documentation, dashboards, and code to define authentication flows correctly. Even small mistakes can cause scans to fail silently or produce incomplete results.
• Lack of standardization creates confusion: Each application or API may use slightly varied access control mechanisms and authentication material, including headers, tokens, cookies, and custom flows. Practitioners must rebuild the logic from scratch each time due to inconsistent implementation.
• Iteration is painful and time-consuming: If something doesn’t work, fixing it usually requires manually editing scripts, rerunning scans, and debugging errors. Oftentimes, additional help from support or engineering teams is needed.
• Limited visibility makes troubleshooting harder: When authentication fails, it’s not always clear why. Without clear feedback or context, teams spend valuable time diagnosing issues rather than improving their security posture.

What should be a simple prerequisite, gaining authenticated context into an application, becomes a major bottleneck to dynamic application security testing. 

How AI-Powered Authentication Changes the Game

The new AI-powered authentication feature in Harness API Testing removes these barriers entirely by reworking how authentication config is created and managed.

Generate Authentication in Seconds

Users can navigate to the authentication configuration page, select the custom option, and simply describe what they need. For example:

“Generate an API key-based authentication hook where the token <token> is injected into the request header <authorization>.”

With a single click on “Generate with AI,” the system produces a complete, ready-to-use authentication script. This functionality eliminates the need to write or stitch together configurations manually.

Support for Multiple Authentication Types

The feature supports a range of common authentication mechanisms, including:

• API key-based authentication
• JWT-based authentication
• Bearer token authentication

This flexibility ensures teams can quickly configure access regardless of how their application or API is secured. Learn more details about the supported authentication types. 

Refine Without Rewriting

Authentication requirements often evolve. Instead of starting over, users can iteratively refine their configurations using natural language prompts.

For example, if you want to change how credentials are injected into the auth flow, you can simply say:

“Change the injection type to header name.”

By selecting “Refine with AI,” the system updates the existing configuration accordingly—no manual edits required.

Built-In Transparency

Every AI-generated or modified configuration includes inline comments that explain what changed. These comments make it easier for teams to:

• Understand how authentication is implemented
• Review updates confidently
• Maintain control over their configurations

Additionally, no credentials are stored in logs or persisted in prompts. Any sensitive authentication material is masked and encrypted at rest.

Faster, More Reliable Scans

By reducing setup errors and simplifying authentication configuration, this Harness API Testing feature directly improves scan success rates. Teams can spend less time troubleshooting authentication issues and more time analyzing real security findings.

Unlocking Faster, Smarter Security Workflows

This release is more than just a usability improvement. It’s a foundational step towards enabling AI-driven security workflows.

By removing the friction of authentication setup, teams can:

• Onboard new applications for testing faster
• Integrate seamlessly with AI-powered testing and analysis tools
• Reduce dependency on manual scripting and support interventions
• Achieve more consistent and reliable scan coverage

Ultimately, this translates to a faster time-to-value and a more scalable approach to dynamic application security testing.

Get Started Today

AI-Powered Custom Authentication Generation is available immediately with your existing Harness subscription. You can find related technical documentation here. 

Current Customers: Log in to your dashboard today to start exploring your threat data in a whole new dimension.

New to the Platform? If you aren't yet protected, contact us to schedule a personalized demo.

Request a demo

There is a version of the Legal team that exists in most companies: thorough, careful, and quietly overwhelmed. Good lawyers are spending their days on tasks that really should not require a lawyer at all.

We decided early on that this was not the team we wanted to be.

At Harness, the AI-first approach is not just saved for the engineering team. It is how every team operates, including Legal. That means we stopped asking “should we use AI?” a long time ago and started asking “how do we build with AI?” The result is a Legal team that does not just use AI tools. It develops them. We close faster, advise smarter, and frankly, have more fun doing it!

Our AI legal stack

Every tool in our stack has a job. Here is what that looks like in practice:

• Contract review and redlining: This is where we are headed this year. The vision: we work from our own templates, so AI already knows what good looks like. It runs the first review, benchmarks the contract against our standards, and surfaces only the deviations worth our attention. Not every deviation is a problem. We focus on the relevant ones.
• Legal research: Finding the relevant law (the correct statute, regulation, or court decision) for a specific situation is the real challenge, not just finding the general law itself. Our AI tools are trained on comprehensive legal datasets to quickly filter out irrelevancies, ensuring we arrive at the precise answer faster and are supported by verifiable sources.
• Knowledge management: We are building a system where years of institutional know-how live in searchable, AI-powered playbooks, giving  teams across Harness answers to common legal questions instantly, without filing a request or waiting for a callback.
• Data discovery and privacy compliance: Before you can be compliant, you need to know what you are actually dealing with. We use AI to discover and map all data currently being processed across the business: what it is, where it lives, and how it flows. That visibility makes it possible to track, manage, and meet our legal obligations, including privacy requirements for personal data.
• M&A project management: An AI-assisted platform provides a secure, structured home for our M&A work. When the pace picks up and the details multiply, nothing falls through the cracks.
• Regulatory compliance intake: Every new product goes through a single AI-assisted form before launch. Privacy, AI regulation, cybersecurity, accessibility: one entry point, no gaps.
• Building AI agents: This is the part most people do not expect. Our lawyers are not just using AI. They are learning to build with it, developing agents that handle specific legal tasks end-to-end. We are not waiting for the legal tech market to catch up with what we need. We are building it ourselves.

What it actually changes

The honest answer: the relationship between Legal and the rest of the business.

Turnarounds that used to take days take hours. Quality has gone up, not down. And because teams can self-serve answers to routine questions through our Legal Playbooks, the requests that do reach us are the ones that genuinely need us. We spend less time being a checkpoint and more time being a partner. That is a different job, and a more meaningful one.

We take the guardrails seriously

Moving fast with AI does not mean being reckless about it:

• Every output gets reviewed by a qualified attorney before it goes anywhere. 
• We only use approved, enterprise-grade tools. 
• Sensitive data never touches a public AI model. 
• And we treat every AI-generated output as a starting point, not an answer, checking it against real sources before we rely on it.

What makes this more than a policy is the culture around it. We run regular sessions where the team shares what they are learning: tools worth trying, prompting approaches that actually work for legal drafting, and ways to get more out of what we already use. When one person figures something out, everyone benefits. That collective curiosity is what stops this from becoming shelfware and keeps it genuinely evolving.

If this is what Legal looks like at Harness, imagine the rest.

Every team here operates this way. Not because they are told to, but because it is genuinely a better way to work. If you are looking for a company where AI is woven into how things actually get done, not just what gets announced, we are hiring!

Explore open roles at Harness →

• Setting up Terraform securely and consistently is essential for scaling your infrastructure and staying compliant.
• To achieve strong security, keep Terraform runs isolated, encrypt your state files and plans, and use lock files and trusted provider sources to ensure integrity.
• By automating Terraform setup and management in your CI/CD and GitOps pipelines, you keep versions consistent, speed up delivery, and stay compliant across many services.
• For enterprise teams, managing Terraform through an Infrastructure as Code Management tool, rather than standalone execution, ensures consistency, governance, and scalability.

If your Terraform install is insecure or inconsistent, it can quickly slow down your delivery. A single compromised file or a misconfigured backend can stop deployments for many services. Teams that set up Terraform correctly from the start can scale easily and avoid compliance issues.

The answer is to install Terraform with strong security measures right from the beginning. Use verified binaries, encrypt your state, and set up automated CI/CD integration from day one. This method includes OS-specific setup, security checklists, GitOps alignment, and governance that can grow with your company. Want to speed up secure infrastructure automation? Harness Infrastructure as Code Management offers AI-powered pipelines with built-in governance for enterprises.

Install Terraform Correctly on Every Platform: OS Prerequisites, Packages, and Verification

One misconfigured Terraform install can cause hours of pipeline failures across many services. When setting up Terraform on development machines, build agents, and production, focus on consistency and security for reliable automation. Start with verified binaries, pinned versions, and automated checks to keep your infrastructure stable.

Use Vendor-Backed Packages to Eliminate Supply Chain Risk

Always get Terraform from HashiCorp’s official repositories, not from third-party mirrors or unofficial packages. For macOS, use the official Homebrew tap (brew tap hashicorp/tap && brew install hashicorp/tap/terraform).

On Linux, add HashiCorp’s GPG-signed package repository instead of using versions from your distribution, which may be outdated. Windows users should download signed binaries directly from releases.hashicorp.com. This helps keep your infrastructure safe from compromised or outdated packages.

Pin Exact Versions and Verify SHA256 Checksums

To make builds reproducible, control the exact Terraform version in every environment. Download the specific version you need, such as from https://releases.hashicorp.com/terraform/1.6.0/terraform_1.6.0_linux_amd64.zip, and check the SHA256 checksum against HashiCorp’s signed SHASUMS file before extracting.

Keep your version-pinned install scripts in your infrastructure repository so teams can create identical environments. If you use Terraform with Harness, delegates manage versions for you, but local development still needs consistent versioning.

Automate Post-Install Verification to Prevent Configuration Drift

After installing Terraform, run terraform version to make sure the right version is active and in your PATH. Set up the plugin cache directory (TF_PLUGIN_CACHE_DIR) to avoid repeated provider downloads and check that you have write permissions.

Write a simple script to check the Terraform binary location, version, and basic provider setup. Run this script automatically in your CI/CD pipelines, container builds, and onboarding workflows to catch problems before they affect deployments. While local installation is useful for development, enterprise teams should standardize Terraform execution through an IaCM platform. This ensures consistent environments across developers, CI/CD pipelines, and production systems without relying on manual setup

Enterprise Security Hardening for Terraform Installation

Installing Terraform is only the beginning. In enterprise settings where you manage important infrastructure and need to meet regulations, hardening your Terraform setup turns a basic install into a system ready for production and governance. These controls are significantly easier to enforce when Terraform is managed through an IaCM platform that centralizes execution, credentials, and policy enforcement.

Credential Management and Execution Isolation:

• Run Terraform on dedicated build agents or through an IaCM platform that provides isolated, governed execution environments. Avoid putting long-term cloud credentials in developer profiles or local setups.
• Encrypt your Terraform plans by storing them in secrets managers instead of plain text files. This keeps sensitive infrastructure changes protected during the approval process.

Provider Security and Integrity:

• Use dependency lock files and private provider mirrors to control provider integrity. These steps enforce SHA256 checksum checks and stop unauthorized plugin downloads from public registries.
• Set up curated provider repositories with the Terraform providers mirror command. This creates trusted, version-controlled sources for your team, so you don’t have to rely on external registries in production.

State Management and Backend Security:

• Store your remote state in encrypted backends like S3 with KMS encryption. Set up bucket policies to limit access and turn on audit logging for all state changes.
• Set secure CLI defaults in your Terraform config files. This limits provider downloads to approved sources and blocks access to untrusted registries during runtime.

Automate Terraform Installation in CI/CD and GitOps Pipelines

Make your Terraform CI/CD setup consistent by including the binary in versioned container images or reusable templates that all services use. This prevents differences between developer machines, build agents, and production. This approach can become even more scalable when implemented through an IaCM tool integrated with your CI/CD platform where Terraform execution, policy checks, and governance are built into reusable workspaces and modules.

When updating Terraform versions or security patches, make changes in your template library instead of updating each pipeline one by one.  We recommend this version-controlled method for enterprise customers.

Use Policy as Code checks to enforce governance by validating Terraform versions, approved modules, and provider rules before running any plans. OPA can review Terraform plans in your CI/CD pipeline, automatically approving safe changes and flagging risky ones for manual review.

Pair this with GitOps workflows, where pull requests start plans and approved merges triggers applies. This creates clear audit trails for compliance and keeps developers moving quickly. Instead of treating Terraform as a standalone CLI step, IaC tools allow you to manage infrastructure workflows as first-class citizens within your delivery pipelines.

Terraform Install FAQs: Security, Compliance, and Scale

DevOps teams running hundreds of services need Terraform installation methods that scale and stay secure and compliant. Here are practical answers to common questions from teams in regulated settings.

How do you install Terraform securely in an enterprise environment?

Start with package repositories that include GPG verification rather than direct binary downloads to prevent compromised or malicious software packages. Install from official HashiCorp repositories with signed packages, verify SHA256 checksums, and run Terraform from isolated build environments with limited-access credentials that only provide necessary permissions. Keep your state files in encrypted, secure storage with access controls and comprehensive audit logging.

What are the best practices for integrating Terraform installation with CI/CD pipelines?

Include Terraform in your container images with specific versions, or use custom binaries to keep all pipeline runs consistent. Pin exact builds in your pipeline templates and use policy-as-code to allow only approved releases before running plans. This keeps development and production in sync and maintains clear compliance records.

How can you automate Terraform installation for large-scale DevOps teams while meeting compliance requirements?

Make reusable install scripts that check checksums and pin builds, then share them through central config management or container registries. Use remote execution on dedicated infrastructure for security and audit trails. Apply OPA policies to control which Terraform releases and providers your teams can use.

Should you use local or remote Terraform execution for enterprise deployments?

Running Terraform remotely on dedicated infrastructure gives you better security and audit trails. Running it locally on developer machines can cause compliance and credential issues. Use isolated build environments or cloud-managed services that run Terraform plans with proper authentication and detailed logs for production. Even better, IaC platforms standardize this by enforcing remote execution with built-in security, auditability, and role-based access controls.

How do you maintain Terraform version consistency across multiple teams and services?

Set up golden path templates with pinned Terraform installs that update all services automatically. Distribute approved releases using container images or package managers, or use platforms that handle governance for you. IaC platforms automate this by centrally managing Terraform versions and enforcing them across all pipelines and environments.

From Installation to Enterprise Scale: Govern, Automate, and Evolve

Standardizing how you install Terraform sets the stage for everything else. Pinning versions, using verified binaries, and securing remote state help your teams work quickly and stay compliant. These best practices are the base for templates that scale to hundreds of services.

Once you have this foundation, the real benefits come when your install standards connect to automated pipelines and GitOps workflows. Using centralized templates and modules for Terraform means security updates are spread automatically, and developers keep their flexibility. Policy-as-code makes sure every deployment meets enterprise needs without slowing things down. At this stage, adopting an IaC Platform approach becomes the recommended path. By managing Terraform through platforms like Harness, teams can standardize execution, enforce governance, and scale infrastructure delivery without increasing operational overhead.”

Are you ready to move from manual installs to enterprise-level automation and governance? Harness Infrastructure as Code Management offers AI-powered templates, a central control plane, and automated checks to make your Terraform setup a real advantage.

• Integrating A/B testing and feature flags directly into CI/CD pipelines empowers developers with self-service experimentation, while maintaining enterprise governance and security.
• Standardizing experimentation workflows—including flag management, guardrail metrics, and automated verification—reduces operational bottlenecks and technical debt across large engineering teams.
• AI-powered automation enable platform teams to scale safe experimentation. Intelligent tooling provides portfolio-level visibility and ROI measurement without sacrificing control or compliance.

With the acceleration of AI-assisted coding, spurring the velocity of software releases, the challenge of ensuring stable deployments is heightened, and platform teams are feeling the hit.  The State of AI-assisted Software Development DORA report measured a negative impact on software delivery stability: “an estimated 7.2% reduction for every 25% increase in AI adoption.” 

The DORA report advises:

Considered together, our data suggest that improving the development process does not automatically improve software delivery—at least not without proper adherence to the basics of successful software delivery, like small batch sizes and robust testing mechanisms.

A robust testing mechanism rapidly gaining momentum is testing in production. Let’s take a closer look at how this practice boosts software delivery stability and supports the software development lifecycle (SDLC). We’ll also consider how to make testing in production, specifically A/B testing at scale, work for you.

What is testing in production?

Testing in production (TIP) means testing new software code on live traffic in active real-world environments. TIP is complementary to pre-production testing and does not replace it. It does, however, carry tangible benefits: 

• Real-World Validation. Tests new features in the actual production environment, measuring performance under real-world conditions and detecting issues that staging (due to lack of data variety) cannot catch.

• Speed & Efficiency. Eliminate the need to create and maintain expensive, multi-layered staging environments for heavy testing.

• Improved User Experience. Allows teams to quickly iterate on features where they create confirmed value, by means of tight feedback loops on real user data.

• Early Issue Detection: Equips your team to spot issues early on, by testing features in production in small amounts or with limited visibility. You can resolve the errors before they escalate or affect more customers.

Feature flags are instrumental in the practice of safe testing in production because they decouple deployment and exposure at the most granular level. By means of feature flags, you implement incremental feature release techniques and unlock progressive experimentation. With carefully crafted A/B testing, you empower rapid feedback loops that confirm real feature value, validate high quality software, and increase team productivity and satisfaction. 

These testing and verification capabilities are crucial as never before in this “AI moment” where AI-assisted coding enjoys wide adoption and funding.

How A/B Testing Works

A/B testing is the process of simultaneously testing two different versions of a web page or product feature in order to optimize a behavioral or performance metric, while ensuring guardrail metrics are not negatively impacted. A/B testing spans the whole spectrum of software verification: you can safely carry out architectural validation on fundamental architectural changes or gather behavioral analytics on UI variations.

Progressive experimentation with feature flags lets you roll out changes to a small slice of users first, catch problems early, and expand only when the data looks good.

The key is keeping deployment and release separate. You decouple deployment and release by delivering new features in a dormant state. Code goes out behind a flag. You validate it with real traffic.

Why A/B Testing Belongs in Your CI/CD Pipeline

A/B testing built into your CI/CD pipeline means you're making data-driven decisions based on observed metrics. Advanced feature flagging correlates statistical data, with pinpoint precision, to the actual feature variation causing the impact. Even when multiple features are rolled out concurrently, an enterprise-grade feature management platform will effectively parse the data, alert you to the impactful variant, and enable you to roll back any negative feature in seconds. The time/cost savings and safety benefits are astounding.

A/B testing provides a great experience for both marketing teams and engineers:

• Marketers can enjoy the freedom to boldly conduct business experiments and conclusively determine the features that drive key performance indicators (KPIs) and return on investment (ROI). 

• Engineers can confidently improve architecture and perform code refactoring, knowing these changes will be safely measured against guardrail metrics for real-world engineering verification.

An enterprise-level platform like Harness, provides Feature Management and Experimentation, bringing flags, monitoring, and full experimentation freedom into a finely-tuned, seamless end-to-end software delivery tech stack for your platform team. Integrating A/B testing and feature flags directly into CI/CD pipelines empowers your teams with self-service experimentation while maintaining enterprise governance and security.

Progressive exposure limits the blast radius 

Bundling features into cliff-jump releases put every user account at risk simultaneously. A progressive ramp—starting with just 1 or 2% of traffic, and gradually increasing—means a bug in your checkout flow only affects a fraction of users before you catch it. Progressive delivery validates that SLOs are holding before exposure expands. p95 latency spiking? Error rate creeping up? You catch it when a tiny fraction of users are affected—not thousands—and Harness CD integrates cleanly with Jenkins, GitLab, or GitHub Actions.

The deploy-and-hold pattern is the keystone. Ship code in the "off" state behind a feature flag and nothing changes for users until you're ready. Deploy at 11 AM on a Tuesday instead of 1 AM on a Sunday. No change windows, no dashboard babysitting. Code is in production, the feature is dark, and you flip the switch when you're ready to monitor it. That's the freedom of progressive experimentation with feature flags in practice.

AI-assisted verification handles the noise

Raw telemetry is information in theory and chaos in practice. AI-powered monitoring watches flag-level metrics—not just "something is slower," but "checkout button variant B is adding 43ms of p95 latency." That specificity matters. When you have six active experiments running, your engineers are not flipping through dashboards trying to isolate which one broke something. The system tells you.

From Feature Flags to Experiments: Architecture That Works Seamlessly

If your team is already running feature flags with health monitoring, you're closer to a full experimentation platform than you might think. Targeting logic, rollout percentages, kill switches—that's already experiment infrastructure. What's missing is experiment tracking, statistical analysis, and deterministic assignment.

To implement experiments with your feature flagging:

• Build on existing flag infrastructure. Targeting, rollout percentages, and kill switches are already there. Add experiment tracking on top rather than running parallel systems. Harness Feature Management handles both in one place.
• Use stable user IDs for deterministic assignment. Consistent hashing keeps users in the same variant across sessions and devices. No drift, no contaminated data.
• Evaluate flags on the SDK side. Toggle decisions should be fast and deterministic. Local evaluation avoids remote call latency and keeps user data secure.
• Route alerts to features, not just systems. "Checkout variant B caused a +43ms p95 increase" is actionable; "Latency is up" isn't. Release monitoring with flag context makes rollback decisions take on surgical precision, and proper experimentation systems prevent sample ratio mismatch and bot traffic from skewing results.

An experimentation system built on top of your feature flagging makes A/B testing a cinch and eliminates operational bottlenecks and technical debt for your platform team.

CI/CD Workflow: Six Stages for Safe Experimentation

A/B testing doesn't have to be complicated. It can run as part of a structured rollout with automated KPI metrics and guardrails:

The seven stages are built into your pipeline and completed with minimal human intervention:

1. Build and test. Catch issues before they ship. Pre-production unit, regression, smoke, and integration testing is incorporated into code builds and repository merges.
2. Deploy, but keep new features inactive. Launch new features but keep them dark behind conditional statements that evaluate a feature flag. New features are in production but hidden.
3. Canary at 1%. Enable the flag for a small slice of real traffic. This achieves engineering verification of new code under production traffic and processing loads. At this point, the blast radius of these issues is minimal and they are quickly isolated and resolved.
4. Check guardrails. Automated monitoring reports error rate, latency, performance, and business KPIs. A powerful statistical analysis engine measures metrics for feature flag variants against the baseline, detects statistical significance, and alerts on positive or negative impact at the feature level.
5. Auto-ramp. Expand the audience for each feature by progressively increasing real-user exposure as you continue to monitor guardrails, for example 1% → 5% → 25% → 50% → 75% → 100%. A deterministic feature flag assignment algorithm controls the gradual increase in audience size and ensures that users aren’t repeatedly flipped between feature variants.
6. A/B Analysis. Automatically calculate metrics and generate data analysis charts for feature variant comparison. Executive dashboards show experiment velocity, win rates, and KPI lift to demonstrate ROI and guide strategic decisions.
7. Auto-rollback or promote. Thresholds crossed? The system reverts a feature without waiting for a human to notice, while you sleep.

Sample sizes matter more than most teams realize

A common mistake is ramping too fast and drawing conclusions from thin data. If your sample size is too low, your experiment will be underpowered, and you will be unlikely to detect a reasonably-sized impact. Calculate that you have a large enough sample to be able to detect impacts of the size that are important to you. 

Progressive experimentation requires patience. Premature conclusions produce unreliable results, and unreliable results produce bad decisions.

Governance isn't overhead, it's insurance 

Every experiment should have a documented hypothesis, defined success metrics, blast radius assessment, and rollback plan before it touches production. Feature flag lifecycle management also keeps technical debt from quietly accumulating—flags that never get retired are toggle debt and a production surprise waiting to happen.

Turn Every Release Into a Measured Experiment

The goal isn't just fewer 3 a.m. incidents, though that's a welcome side effect. The real win is replacing gut feel with data at every stage of delivery.

With modern testing in production: feature flags decouple deploy from release, progressive ramps limit blast radius, AI-powered guardrails catch regressions before they spread, and centralized analytics replace the multi-tool sprawl that makes experimentation feel expensive.

Every time you release a feature you can ramp gradually up to 100% using percentage-based rollouts, alert on specific pre-decided latency increases, and enforce minimum sample sizes before promotion. Let every release become a decision backed by actual evidence, not optimism.

Harness Feature Management & Experimentation consolidates flags, release monitoring, and A/B testing, so every deployment is a controlled experiment—not a gamble.

Safe A/B Testing in Production: Frequently Asked Questions

How do you pick guardrail metrics without blocking every release?

Start with your existing SLO metrics and be conservative. Grafana's SLO guidance recommends event-based SLIs over percentiles for cleaner signals. Focus on business-critical user journeys first.

What's a practical ramp schedule for a mid-sized SaaS team?

Every team has slightly different criteria to consider before safely ramping up. Release monitoring with automated guardrails removes the need for someone to manually review metrics at each stage—which is the only way this actually scales.

How do you handle sample ratio mismatch?

Monitor assignment ratios continuously using chi-squared tests. Harness FME’s attribution and exclusion algorithm is honed to ensure accurate samples. In addition, FME reassesses experiment health in real-time, including sample ratio.

Filter bot traffic early too. Microsoft's bot detection research shows bots can skew conversion rates by 15–30%. Behavioral signals like sub-10-second session duration or unusual referrer patterns are a practical starting point for exclusion algorithms.

Should you A/B test infrastructure changes or just product features?

A/B testing works best for user-facing changes where behavior matters. Infrastructure changes are better suited to progressive rollouts with guardrail monitoring—different changes, different success metrics. Performance and reliability for engineering experiments; conversion and engagement for growth. Keep the tooling integrated in your pipeline either way.

How do you maintain consistent user experiences across devices and services?

Deterministic hashing on stable user IDs. Hash user ID plus experiment name to generate consistent assignments and make sure the same user sees the same variant whether they're on mobile, desktop, or clearing cookies every 20 minutes. Avoid session-based bucketing—it creates flickering experiences, causes re-bucketing, and erodes trust in experiment data. Lean on SDK-side evaluation for consistency that holds across your entire stack.

At 2 am, your migration goes live. By 2:07, error rates spike, and rollback isn’t an option. Cloud migrations, API rewrites, and architecture transformations rarely fail because of bad code. They fail because of how that code is released.

Most teams still rely on a “big bang” cutover where infrastructure, services, and user-facing changes go live at once. This concentrates risk into a single moment. When something breaks, rollback is slow, visibility is limited, and the blast radius is large.

This is not just anecdotal. According to BCG, more than half of transformation efforts fail to achieve their intended outcomes within three years.

The difference between success and failure is not the migration itself. It is the release strategy.

Cloud Migration Is Not a Single Change

“Cloud migration” sounds simple, but in practice, it is a layered transformation.

Most migrations combine several of the following:

• Monolith to microservices
• API or data pipeline rewrites
• Frontend or UI rebuilds
• On-prem to cloud infrastructure moves
• Introduction of new service layers

These rarely happen in isolation. Teams often try to ship them together in a single coordinated release. That coupling increases complexity and multiplies risk.

Before your next migration, list every system involved. If they are all released together, you are carrying unnecessary risk.

The Core Anti-Pattern: Big Bang Releases

The failure mode is consistent:

• A new service is deployed
• Infrastructure flips to the cloud
• A redesigned UI is released
• All at once

There is no safe way to validate behavior in production. There is no gradual exposure. Rollback often requires redeploying an old stack that may no longer be compatible.

Even worse, teams lack a reliable baseline. They cannot answer simple questions:

• Is performance better?
• Is the cost lower?
• Is reliability improved?

Without that, migration becomes guesswork.

Decoupling Deployment from Release

Modern teams are adopting a different model:

• Deploy code anytime. Release it gradually.

Feature flags provide a control layer that separates deployment from exposure. Code can exist in production without being active for all users.

This enables:

• Controlled rollout by percentage, region, or cohort
• Instant rollback without redeployment
• Real-time measurement tied to specific changes

Start by putting one service behind a feature flag and releasing it to internal users first.

Progressive Delivery for Migrations

Replace Cutovers with Progressive Rollouts

Instead of switching everything at once:

1. Deploy the new system alongside the old one
2. Route a small percentage of traffic
3. Observe behavior
4. Increase exposure gradually

If something fails, you reduce traffic or revert instantly.

This shifts migration from a single high-risk event to a series of measurable steps.

The Strangler Fig Pattern in Practice

A common migration strategy is the strangler fig pattern.

• Build new functionality alongside the legacy system
• Gradually route traffic to the new components
• Retire legacy code over time

Feature flags make this executable in production by controlling routing and exposure. But to make this work in practice, you need a control layer that can manage traffic in real time.

How Progressive Migration Actually Works

Below is a simplified view of how feature flags act as a control plane during migration:

Fig: Feature-flag–driven progressive traffic routing during migration

Two things matter here:

1. Routing control: traffic can be shifted gradually
2. Measurement: metrics are tied to each variation

This is not just a toggle. It is a runtime decision and an observability layer.

Measure What Actually Matters

A successful migration is not defined by deployment success. It is defined by outcomes.

Key metrics include:

• Latency and throughput
• Error rates
• Infrastructure cost
• Output correctness

These metrics are not theoretical. They are what teams use to validate migrations in real production environments.

Real Example: Dual Pipeline Migration

In the Beyond the Toggle ebook, a legacy Spark batch pipeline was replaced with a streaming architecture, with a progressive rollout rather than a cutover.

• Both pipelines ran in production
• A feature flag routed traffic between them
• Metrics were compared in real time

The new system showed faster processing and lower costs before the full rollout.

From the webinar, teams often go further:

• Run both systems simultaneously
• Compare outputs for correctness
• Measure performance differences per request

This allows validation of both performance and data integrity before committing.

Define your baseline metrics before migration. If you cannot measure improvement, you cannot prove success.

Staging Lies. Production Doesn’t.

Staging environments cannot replicate production conditions. They lack:

• Real traffic patterns
• Data scale
• Edge cases

Feature flags enable safe production testing through controlled exposure.

Common Patterns

• Canary releases by percentage or region
• Cohort-based rollouts (geo, customer segment)
• Dual execution for validation

Not all canary releases are percentage-based. Some teams roll out by country or user segment first, then expand globally.

Guardrails

To make this safe:

• Automated rollback based on thresholds
• Feature-level observability
• Access control and audit logs

Decision Making: Continuous Go / No-Go

A migration is a sequence of decisions, not a single moment.

At each stage:

1. Define the metric
2. Measure impact
3. Decide to expand or roll back

In one example from the webinar:

• A rollout reached 30% traffic
• Error rates increased
• Traffic was reduced to 20%
• The issue was isolated and fixed
• Rollout continued safely

This approach removes pressure from a single “launch moment” and distributes risk across stages.

Advanced Considerations for Developers

Feature Flag Performance and Reliability

Modern flag systems avoid becoming a bottleneck:

• Evaluations happen locally via SDKs
• Configurations are cached
• Systems continue operating even if the flag service is unavailable

This ensures minimal latency and high reliability.

Handling Complex Systems

Not all migrations are equal.

• Data pipelines and database paths require more planning
• Read and write paths may need staged transitions
• Flags still apply, but design complexity increases

The key is incremental transition, not avoidance.

Managing Flag Lifecycle and Tech Debt

Feature flags are temporary by design.

If left unmanaged, they accumulate and create complexity. Teams need:

• Visibility into flag state and usage
• Defined lifecycle policies
• Cleanup after full rollout

Emerging approaches include automation that detects stale flags and generates pull requests to remove them.

This Is a Delivery Strategy Change

Adopting progressive delivery is not just a tooling decision. It changes how teams release software.

Key considerations:

• Align with existing change management processes
• Integrate flags into CI/CD pipelines
• Maintain governance and auditability

Feature flags do not bypass controls. They enhance them by adding visibility and control at runtime.

What to Look for in a Feature Flag Platform

For migration use cases, a Feature Flag platform should provide:

• Tight integration with CI/CD pipelines
• Built-in experimentation and metrics
• Governance, approvals, and audit logs
• Developer-friendly SDKs and workflows

Flags should not feel like a bolt-on. They should be part of how software is built and released.

Conclusion: From Risk Event to Controlled Process

The biggest mistake teams make is treating migration as a moment.

It is not.

It is a controlled progression of changes, each validated in production under real conditions.

Feature flags enable this by:

• Decoupling deployment from release
• Enabling gradual exposure
• Providing real-time measurement
• Allowing instant rollback

The result is simple:

Migrations become reversible, observable, and data-driven.

Want a deeper breakdown of these patterns and real-world examples? Read the full ebook or see a demo.

Businesses today run on computers, cloud systems, and digital tools. One big failure can stop everything. A cyber attack, a power outage, or a software glitch can shut down operations for hours or days. Disaster recovery testing is how you prove you can restore critical services when the unexpected happens. 


In 2026, with hybrid and multi-cloud estates, distributed data, and tighter oversight, this is not a once-a-year fire drill. It is a continuous discipline that validates plans, uncovers weak links before they cause outages, and gives leaders confidence that customer-facing and internal systems can bounce back on demand.

Disaster recovery testing is a simple way to practice getting your systems back online after something goes wrong. It checks if your backup plans actually work before a real problem hits. This blog gives you a clear, step-by-step look at what it is, why it is essential right now, and how to get started. 

What Is Disaster Recovery Testing?

Disaster recovery testing is a structured way to confirm that systems, data, and services can be restored to meet defined recovery goals after a disruption. The mandate is simple: verify that recovery works as designed and within the time and data loss thresholds the business requires. Effective programs test more than technology. They exercise people, processes, communications, and third-party dependencies end to end. The goal is to prove you can bring back data, apps, and services quickly with little loss.

A strong disaster recovery test plan typically covers:

• Clear recovery time objectives (RTOs) and recovery point objectives (RPOs) for each application tier.
• A current asset and application inventory with criticality tiers and upstream/downstream dependencies.
• Documented runbooks and playbooks for failover and failback, including decision criteria.
• Data protection strategies such as backups, replication, and snapshots with defined retention and immutability.
• Communication plans for internal teams, executives, customers, and partners.
• Roles and responsibilities, escalation paths, and an incident command structure.
• Third-party and vendor recovery commitments, service level agreements, and contact procedures.
• Metrics, governance, and reporting for audits and continuous improvement.

Without regular tests, even the best plan stays unproven. Many companies learn this the hard way when an outage lasts longer than expected.

Types of Disaster Recovery Tests

Different systems require different levels of validation based on their criticality, risk, and business impact. A layered testing strategy helps teams build confidence gradually starting with low-risk discussions and moving toward full-scale failovers.

By combining multiple types of tests, organizations can validate both technical recovery and team readiness without unnecessary disruption.

Tabletop Exercises:

Tabletop exercises are discussion-based sessions where stakeholders walk through a hypothetical disaster scenario step by step. These are typically the starting point for any disaster recovery program, as they help clarify roles, responsibilities, and decision-making processes. While they do not involve actual system changes, they are highly effective in identifying communication gaps and aligning teams on escalation paths.

Simulations:

Simulations introduce more realism by creating scenario-driven drills with staged alerts and mocked dependencies. Teams respond as if a real incident is happening, but without impacting production systems. This type of testing is useful for validating how teams react under pressure and ensuring that tools, alerts, and workflows function as expected in a controlled environment.

Operational Walkthroughs:

Operational walkthroughs involve executing recovery runbooks step by step to verify that all prerequisites such as permissions, tooling, and sequencing are in place. These tests are more hands-on than simulations and are often conducted before attempting partial or full failovers. They help reduce surprises by ensuring that recovery procedures are practical and executable.

Partial Failovers:

Partial failovers test the recovery of specific services, components, or regions, usually during off-peak hours. This approach allows teams to validate critical dependencies and recovery workflows without risking the entire system. It is especially useful for building confidence in complex environments where a full failover may be too risky or costly to perform frequently.

Full Failovers:

Full failovers are the most comprehensive form of disaster recovery testing, where production systems are completely switched to a secondary site or region. After validation, systems are failed back to the primary environment. These tests provide the strongest proof of resilience, as they validate end-to-end recovery, including performance and data integrity, but they require careful planning due to their potential impact.

Automated Validations:

Automated validations use codified workflows or pipelines to continuously test recovery processes. These tests can automatically spin up recovery environments, validate configurations, and run health checks. They are ideal for frequent, low-risk testing and help reduce human error while providing fast and consistent feedback. Over time, automation becomes a key driver for maintaining continuous assurance in disaster recovery readiness.

Here’s the table outlines the primary types of disaster recovery testing and where they fit.

If you are building a disaster recovery testing checklist, include a mix of these types of disaster recovery testing and map each to the systems they protect. Over time, increase the frequency of automated validations and reserve full failovers for the highest-value services.

Why Disaster Recovery Testing Matters in 2026

The world is more connected than ever. Companies rely on cloud services, remote teams, and AI tools. At the same time, threats keep growing. Cyber attacks like ransomware are more common. Natural events and supply chain problems add extra risk. Cloud systems can fail without warning.

Recent studies show the cost of downtime keeps rising. For many large companies, one hour of downtime can cost more than 300,000 dollars. Some industries see losses climb into the millions per hour. Smaller businesses lose thousands per minute in lost sales and unhappy customers.

In 2026, experts note that most organizations still test their recovery plans only once or twice a year. That is not enough. Systems change fast. New software updates, new cloud setups, and new team members can break old plans.

Regular testing gives you confidence. It cuts recovery time and protects revenue. It also helps meet rules from banks, healthcare groups, and government agencies that require proof of preparedness.

How Modern Tools Make Disaster Recovery Testing Easier

Traditional testing took weeks of manual work. Today, platforms combine different testing methods in one place. This approach saves time and gives better results.

For example, Harness recently released its Resilience Testing module. It brings together chaos testing (to inject real-world failures safely), load testing (to check performance under stress), and disaster recovery testing. You can run everything inside your existing pipelines. This means you can test recovery steps automatically, validate failovers, and spot risks early.

Teams using this kind of integrated platform report faster recovery times and fewer surprises. It fits right into daily development work instead of feeling like an extra project.

The Role of AI in Disaster Recovery Testing

Artificial intelligence is making disaster recovery testing much smarter in 2026. It turns testing from a once-a-year chore into something fast, ongoing, and more accurate.

AI helps teams spot problems early by analyzing system data and predicting where failures might happen, allowing issues to be fixed before they cause real damage. It also enables continuous and automated testing, running scenarios in the background without interrupting normal business operations. Instead of manually creating test plans, AI can generate and recommend the most relevant scenarios based on your actual system setup, saving time and improving coverage.

Another major advantage is how quickly AI can analyze results. It processes test outcomes in real time and clearly points out what needs to be fixed, removing the guesswork. Over time, it learns from every test run and continuously improves your disaster recovery strategy, making it more reliable with each iteration.

Overall, AI helps teams recover faster and with fewer mistakes. Rather than relying on assumptions, teams get clear, data-driven insights to strengthen their systems. Tools like the Resilience Testing module from Harness already bring these capabilities into practice by combining chaos testing, load testing, and disaster recovery testing. With AI built into the platform, it can recommend the right tests, automate execution, and provide simple, actionable steps to improve system resilience.

Conclusion

Disaster recovery testing is not a one-time task. It is an ongoing habit that protects your business in 2026 and beyond. The companies that test regularly recover faster, lose less money, and keep customer trust.

Take a moment now to review your current plan. Pick one critical system and schedule a simple test this quarter. If you want a modern way to make the process simple and powerful, look at solutions like the Resilience Testing module from Harness. It helps you combine multiple testing types and use AI so you stay ready no matter what comes next.

Your business depends on technology. Make sure that technology can bounce back when it counts. Start testing today and build the confidence your team needs for whatever 2026 brings.

The design of the Harness MCP (Model Context Protocol) server is driven by a pattern that keeps reappearing across systems that scale well: small, stable interfaces with most of the complexity pushed behind a dispatch layer. The central idea is this: the agent loop behaves like an operating system boundary. The LLM is the reasoning engine, the context window is working memory, tool calls act like syscalls, and the MCP server serves as a kernel that mediates access to underlying systems. This isn’t a literal equivalence, but it’s a useful design lens. It forces you to think in terms of memory pressure, interface stability, and clean I/O contracts.

We built the Harness MCP server to make Harness agent-native. In practice, that means exposing the platform through a runtime-discoverable, schema-driven interface that agents can inspect, select from, and compose without hardcoded knowledge of the domain. Today, that interface consists of 10 generic tools that dispatch to 30 toolsets covering 140+ resource types across the platform, along with 57 Knowledge Graph views for cross-module analytics.

Those numbers matter less than the constraint behind them: tool count stays constant while capability scales through data and dispatch. The goal is to keep the agent’s context focused on reasoning, not on parsing a large menu of endpoints.

Before getting into the architecture, though, it’s worth asking a simpler question: why does Claude feel so capable when you give it nothing more than a bash shell? 

Why Claude Feels Powerful with Just Bash

The Contrast

Give Claude access to a terminal. Just bash. No APIs, no SDKs, no custom tools. It can navigate an unfamiliar codebase, find a bug across 50 files, refactor code, run tests, and commit end-to-end.

Now give an LLM access to a hundred perfectly-documented REST endpoints. It gets confused by the tool count, picks the wrong endpoint, and loses track of multi-step operations.

The difference isn't the tools themselves. It's the shape of the interface. The point isn’t that shell text streams are superior to structured APIs, but that agents perform better with interfaces that have a small, consistent grammar and are easy to compose.

What Makes Bash Work

Bash provides three properties that matter enormously for agent reasoning:

Composability. Every Unix tool does one thing and communicates through a uniform interface: text streams. grep | sort | uniq -c | head is four tools composed into an analytical pipeline. The agent doesn't need to know about a special "count-unique-matches" API. It composes primitives.

Uniform interface. Every tool takes text in and produces text out. There's no per-tool protocol, no per-tool authentication, no per-tool response schema. The contract is always the same: stdin, stdout, and exit code.

Introspection. ls, find, file, cat, head — the agent can discover what exists at runtime. It doesn't need to memorize the file system layout. It explores, then acts.

These three properties mean the agent doesn't need to hold 200 tool schemas in its context window. It learns a small set of verbs and composes them. The intelligence isn't in any single tool. It's in the loop that decides what to call next.

The Deeper Insight

Watch what actually happens when Claude debugs with bash:

1. Observe:     ls src/                    → see the project structure
2. Hypothesize: "error likely in auth module"
3. Act:         grep -r "token" src/auth/
4. Observe:     see the grep output
5. Refine:      "ah, token expiry not handled"
6. Act:         cat src/auth/session.ts
7. Observe:     read the file
8. Fix:         edit the file
9. Verify:      npm test

This is not "call the right API." This is a reasoning loop — observe, hypothesize, act, verify. The bash commands are just I/O. The reasoning happens between them.

This loop is the program. The tools are the I/O. And the design of the tools determines how efficiently the loop can run.

The Agent Loop — What's Actually Happening

The Event Loop

Every agent, whether it's Claude in a terminal, Cursor with MCP tools, or a custom orchestrator, runs some version of this loop:

while (!task_done) {    
context   = observe(environment)       // tool outputs, previous results    
plan      = reason(context, goal)      // LLM inference    
action    = select_tool(plan)          // tool selection    
result    = execute(action)            // tool call    
environment.update(result)             // state change
}

This is an event loop. The LLM is the scheduler (the scheduling behavior is an emergent property of the loop, not an intrinsic property of the LLM). The tools are I/O operations. The context window is working memory. Each iteration, the agent observes the current state, reasons about what to do next, selects a tool, executes it, and incorporates the result into its context.

Where the Intelligence Lives

The critical insight: the intelligence is in the loop, not in the tools. The tools just move information in and out. The loop is what plans, backtracks, retries, composes, and converges.

This means the quality of the agent's output depends on two things:

1. The loop's ability to reason, which is a function of the LLM's capability and the available context
2. The quality of I/O, which is a function of tool design (clear, structured, context-efficient)

If the tools are well-designed (few, composable, self-describing, context-efficient), the loop can reason clearly. If the tools are poorly designed (many, verbose, opaque), the loop spends its context budget parsing menus and payloads instead of thinking.

The MCP Server's Role

Our MCP server does not implement the agent loop. The loop lives in the MCP host: Cursor, Claude Desktop, or whatever IDE/agent framework the user is running. Our server is stateless at the request level: each tool call arrives as a JSON-RPC message, runs an async handler, and returns a structured response. Task-level state lives in the MCP host and in the underlying Harness systems.

We implement the kernel that the loop dispatches into. Our job is to make each dispatch fast, clean, and context-efficient.

Systems That Just Click: We've Seen This Pattern Before

Before drawing the OS analogy, it's worth stepping back. The properties that make bash work for agents, composability, uniform interface, and runtime discovery, aren't unique to Unix. They show up in every long-lived system that engineers describe as "just working."

Linux: The syscall ABI has been stable for decades. The VFS (Virtual File System) is a dispatch table: open(), read(), write(), close() work against ext4, NFS, procfs, sysfs, and any backend. New filesystem? Write a driver, load it at runtime. The interface never changes. /proc and /sys let the kernel describe itself through runtime introspection.

Git: Content-addressable blobs plus a handful of verbs. Branches are just pointers. The plumbing/porcelain split gives you a tiny, stable core with everything else built through composition. The transport protocol is uniform: push/fetch work the same over HTTP, SSH, or a local filesystem.

Kubernetes: Declare desired state. Controllers reconcile. kubectl get, apply, describe work on any resource kind: Pods, Services, your custom CRDs. New capability = new CRD, not a new CLI.

SQL: Small grammar: SELECT, JOIN, WHERE, GROUP BY. Works against any schema. The engine optimizes. You declare intent. The grammar has been stable for 40 years.

The Shared DNA

These systems share five properties:

1. Small, stable interface: few verbs, many nouns
2. Uniform contract: same interaction pattern regardless of backend
3. Runtime discovery: the system describes itself
4. Dispatch layer: a central mechanism that routes requests to domain-specific handlers
5. Extend without changing the core interaction model: new capabilities don’t require changing the existing interface.

This is the design target for agent infrastructure.

The Gap in REST: What MCP Had to Fill

REST APIs answered two questions well:

• WHAT: what resources exist (/pipelines, /services, /executions)
• HOW: how to operate on them (GET, POST, PUT, DELETE)

For programs, code written by humans who already understood the domain, this was enough. 

The developer read the docs, wrote the integration, and deployed it. The logic was pre-written.

An agent encounters your API at runtime, with no prior knowledge. It needs a third answer:

WHY: 

• Why would I use this tool? 
• What does it return? 
• When should I prefer it over another tool? 
• What are the side effects?

This "why" lived in documentation, READMEs, and developers' heads. It was never machine-readable. MCP fills this gap by making tools carry their own intent — descriptions, hints (readOnlyHint, destructiveHint), schemas, and metadata that the agent reads at runtime to decide what to call.

The difference between REST and MCP isn't the transport. It's the audience. REST APIs are typically optimized for pre-written integrations. MCP tool surfaces are optimized for runtime selection and composition by an agent.

The OS Analogy: A Systems Engineer's Guide to Agent Platforms

The mapping between operating systems and agent platforms is more than metaphorical — parts of it are structural, and the rest provide a useful design vocabulary. The same engineering constraints apply, and the same design principles solve them.

The Full Mapping

Context Is RAM

This is the most important mapping, and it has direct engineering consequences.

The context window is finite. Every token you put in is a token that can't be used for something else. Verbose API responses, unnecessary fields, large tool schemas: these are all memory allocations. If you fill the context with data, the agent can't reason.

The OS parallels are precise:

The #1 job of an agent platform is to keep the context window clean for reasoning. Every architectural decision should be evaluated through this lens: does this consume more or less of the context budget?

Syscalls, Not Raw I/O

Programs don't write directly to disk. They call write(), and the kernel handles buffering, permissions, journaling, and device-specific quirks. This abstraction is what makes programs portable and reliable.

The same applies to agents. An agent shouldn't construct HTTP requests with auth headers, manage pagination cursors, handle retry backoff, or parse nested response wrappers. It should call a tool — a syscall — and the MCP server (the kernel) handles all of that.

‍

The tool is the syscall. The MCP server is the kernel. Same contract every time. The agent never has to think about x-api-key headers, accountIdentifier query parameters, or exponential backoff on HTTP 429.

Virtual Memory = Schema Discovery

An OS doesn't load every file into RAM upfront. It uses virtual memory — a large address space backed by on-demand paging. Hot pages stay in RAM; cold pages live on disk until needed.

Our MCP server applies the same pattern to domain knowledge. The agent's "address space" covers 140+ resource types. But at any given moment, only the relevant metadata occupies context:

• Need pipeline fields? → harness_describe(resource_type='pipeline') loads ~500 tokens
• Need execution details? → harness_describe(resource_type='pipeline_execution') loads ~3K tokens of type catalog
• Need pipeline schema? → harness_schema(resource_type=’pipeline’), loaded only when constructing pipeline, and enables progressive discovery

This is demand paging for domain knowledge. The agent discovers what it needs, when it needs it, and the rest stays "on disk" (available but not occupying context).

Our Architecture Through This Lens

The Three-Layer Stack

The MCP server has three layers, each corresponding to a layer in the OS model:

Layer 1 — MCP Tool Surface (syscall table). Ten generic tools that accept a resource_type parameter and dispatch through the registry. These are registered with the MCP SDK using Zod schemas for input validation. Each tool handler is a thin wrapper: normalize inputs → call registry → format response.

Layer 2 — Registry (kernel). The Registry class in src/registry/index.ts is the core dispatch engine. It holds a Map<string, ResourceDefinition> populated from 30 toolset files. When a tool handler calls registry.dispatch(client, resourceType, operation, input), the registry resolves the ResourceDefinition, looks up the EndpointSpec, and calls executeSpec() — the single execution pipeline that handles path templating, scope injection, query parameter building, body construction, auth header interpolation, HTTP dispatch, response extraction, and deep link generation.

Layer 3 — HarnessClient (block device driver). The raw HTTP client in src/client/harness-client.ts. Handles fetch() with the x-api-key auth header, accountIdentifier injection, retry with exponential backoff on 429/5xx, client-side rate limiting, timeouts, and response parsing.

The Syscall Table

The agent learns 10 verbs. They work against every domain in Harness.

Why Tool Count Matters

Every tool the agent "sees" costs tokens:

Our approach keeps this at ~1.2%. Tool count stays O(1). Capabilities grow O(n). This is the core design invariant.

The Dispatch Table (vtable)

The registry is a vtable — a dispatch table that maps (resource_type, operation) to an EndpointSpec and executes it through a unified pipeline. One execution path. Every resource type. Every operation.

Design Patterns

Pattern 1: Generic Verbs + Type Dispatch

Principle: Don't create a tool per API endpoint. Create generic verbs that dispatch by resource type through a registry.

This is the same insight behind REST (uniform interface + varying resources) and Unix (uniform file interface + varying devices). The agent learns the grammar once — list, get, create, execute. New nouns (resource types) are just data in the registry.

Pattern 2: Declarative Over Imperative

Principle: Resource definitions are data structures, not handler functions.

Each API mapping is expressed as an EndpointSpec — a declarative object that describes the HTTP method, path, path parameters, query parameter mappings, body builder, response extractor, and metadata. The registry's executeSpec() reads this spec and handles execution.

This means:

• Auth, retry, pagination, rate limiting, deep links — all handled once in the registry
• New resource types get these behaviors automatically
• Changes to the execution pipeline propagate to every resource type

Pattern 3: Fix Once in the Kernel, Every Driver Benefits

Principle: Centralized dispatch creates compounding returns on infrastructure investment.

Features that propagate everywhere through the registry:

Pattern 4: Validate Before Execute

Move error detection left. Validate agent-generated inputs before spending API budget on execution.

When an agent tries to create or execute something, validate the inputs before committing. If the agent provides a malformed pipeline YAML or references a nonexistent service, catch it at the schema level — before the API call burns tokens on a 400 error and the agent has to parse the response to figure out what went wrong.

harness_execute(
resource_type='pipeline', action='run', 
inputs={branch: 'main', service: 'payment-svc'}
) 
← Error: input 'service' is not a valid runtime input for this pipeline.    
Valid inputs: branch, environment, tag. Did you mean 'environment'?
// Agent retries with corrected inputs. Typically converges in one retry.

Validation is cheap — milliseconds. Wrong answers are expensive — broken trust, bad decisions. This is compile-time checking for agent-generated operations.

Pattern 5: Runtime Reflection Over Static Documentation

Principle: Let agents discover your domain model at runtime. Self-describing systems don't need documentation updates.

Add a new toolset → the agent discovers it immediately. Add a new resource type → the agent can query it immediately. This is introspection — ls for your platform. The same thing that makes bash work for agents.

Anti-Patterns

Anti-Pattern 1: One Tool Per Endpoint

Problem: Creating list_pipelines, get_pipeline, list_services, etc. Each tool costs ~150 tokens. At 50 tools, that's 7,500 tokens of menu.

Fix: Generic verbs with type dispatch.

Anti-Pattern 2: Raw API Passthrough

Problem: Returning the full Harness API response — 50+ fields, nested wrappers.

Fix: Use responseExtractor to return clean, relevant fields. Treat context tokens like memory allocations.

Anti-Pattern 3: Hardcoded Domain Knowledge

Problem: Embedding field lists or API shapes in tool descriptions. They go stale immediately.

Fix: Keep tool descriptions generic. Point to harness_describe() for runtime discovery.

Anti-Pattern 4: Context-Window Aggregation

Problem: Agents often fetch large datasets and aggregate in context, leading to extremely high token usage and degraded accuracy.

Fix: Routing aggregation to the Knowledge Graph dramatically reduces token usage and improves answer reliability.

Anti-Pattern 5: Bloating Server Instructions

Problem: Adding per-resource docs to instructions in src/index.ts.

Fix: Keep instructions under ~20 lines. Put resource-specific guidance in description, diagnosticHint, executeHint, and bodySchema.description on the EndpointSpec.

Conclusion

The agent loop is the new operating system. That’s not a rhetorical flourish. It’s a constraint with real engineering consequences.

Every design decision in the Harness MCP server follows from a single principle: the context window is RAM, and RAM is finite. Verbose responses trash it. Oversized tool menus fragment it. Redundant schemas waste it. The agent’s ability to reason, to observe, hypothesize, act, and verify, degrades in direct proportion to how much of that budget gets consumed by infrastructure noise instead of domain signal.

The patterns described here, generic verbs with type dispatch, declarative resource definitions, demand-paged schema discovery, and centralized kernel dispatch, aren’t novel. They’re the same patterns that made Unix, Git, Kubernetes, and SQL endure for decades: small, stable interfaces, uniform contracts, runtime introspection, and the ability to extend without changing the core interaction model.

What’s different is the audience. Those systems were designed for programs. This one is designed for reasoning systems operating at runtime.

If you're building agent infrastructure, the questions to ask are the same ones OS designers asked in the 1970s: Does this abstraction compose? Does it describe itself? Does it keep the critical resource, then RAM, now context, available for the work that actually matters?

A useful test for any tool, schema, or abstraction is simple: does it reduce the amount of information the agent has to hold in working memory, or increase it? If it increases it, it’s probably making the system worse.

—

If you found this useful, follow and subscribe to the Harness Engineering blog for more deep dives on building agent-native systems and modern developer infrastructure.

‍