Harness Blog

TLDR: Today, Harness is introducing the Harness Cursor Plugin, bringing the power of the Harness AI-native software delivery platform directly into Cursor. This integration, along with the Harness Secure AI Coding hook for Cursor, allows developers and AI agents to move from code changes to vulnerability detection, CI/CD execution, security validation, approvals, deployments, and operational insight without leaving the editor.

AI has completely changed how we write code. You can spin up functions, refactor entire files, and generate tests in seconds. The inner loop, writing and iterating on code, has never been faster. But the moment you try to ship that code, everything slows down. This is what we call the AI Velocity Paradox.

You are suddenly back to juggling pipelines, waiting on approvals, checking security scans, debugging failed runs, and bouncing between tools just to get a change into production.

That gap, between fast code and slow delivery, is what we kept running into. So we built something to fix it.

Today, we are introducing the Harness Plugin for Cursor, a way to go from PR to production without leaving your editor.

AI Made Coding Faster, But Delivery Did Not Catch Up

If you are using agentic coding tools, such as Cursor, you have probably felt this.

You can:

• Generate code instantly
• Understand unfamiliar repos faster
• Fix bugs and open PRs in minutes

But shipping still depends on everything outside your editor:

• CI/CD pipelines
• Security checks
• Approval flows
• Policy enforcement
• Deployment tooling
• Monitoring and debugging

And none of that got simpler just because AI showed up. In fact, AI makes the problem more obvious.

Now you can create changes faster than your delivery process can safely handle. And if those controls are not tight, you are introducing a whole new category of risk. Fast-moving code with fragmented governance.

AI did not break software delivery. It exposed how disconnected it already was.

What If You Could Just Ask

Instead of jumping between tools, what if you could just tell your editor what you want to happen?

Something like:

“Deploy PR #4821 to staging once the security scan passes, and Slack me if anything fails.”

That is the idea behind the Harness Cursor Plugin.

It connects Cursor directly to Harness, so you can trigger and manage your entire delivery workflow using natural language, right inside Cursor.

No tab switching. No manual orchestration. No guessing what is happening in the pipeline.

Some Sample Use Cases

Once connected, you can use Cursor to interact with your delivery system just as you do with your code.

For example, you can:

This builds on what we introduced last month, Secure AI Coding, which integrates directly with Cursor and scans code at the moment of generation rather than waiting for a PR review. Developers see inline vulnerability warnings with the option to send flagged code back to the agent for remediation, without leaving their workflow. Under the hood, it leverages Harness's Code Property Graph (CPG) to trace data flows across the entire codebase, surfacing complex vulnerabilities that simpler linting tools would miss.

The key thing is that you are no longer just interacting with code. You are interacting with the entire delivery system from the same place.

The Important Part: This Is Not Skipping Control

One of the biggest concerns with AI in delivery is obvious:

“Are we about to let agents push code to production without guardrails?”

No.

With Harness, everything runs through the controls that you can rely on:

• Granular RBAC permissions
• OPA policies
• Approval gates
• Audit logs

Instead of being manual checkpoints spread across tools, they are enforced automatically as part of the workflow while you stay in flow.

So AI can help move things faster, but it cannot bypass the governance that matters.

Why We Built It This Way

Most integrations today expose APIs or bolt AI onto existing systems. That is not what we wanted to do.

We designed the Harness Cursor Plugin specifically for how AI agents actually work:

• It is built around actions and workflows, not raw endpoints
• It spans the full delivery lifecycle, not just one step
• It gives agents enough context to reason about what to do next

Because shipping software is not a single action. It is a chain of decisions across CI, CD, security, approvals, and operations. If AI is going to help here, it needs access to that full picture. That’s where the Harness Software Delivery Knowledge Graph comes into play. It provides the necessary context for AI to take actions for you.

The knowledge graph models the relationships between services, pipelines, environments, policies, and operational signals in real time. Instead of treating each step in delivery as an isolated task, it creates a connected system of record that AI can reason over. This allows agents to understand not just what to do, but when and why to do it, based on dependencies, risk signals, and historical behavior.

In practice, this means smarter automation: deployments that adapt to context, approvals that are triggered based on policy and impact, and faster root cause analysis because the system already understands how everything is connected.

This Changes How Ideas Move To Prod

This is not just about convenience. It is a shift in how software actually moves from idea to production.

Instead of:

• Writing code in one place
• Managing delivery somewhere else
• And stitching it all together manually

You get a single, connected workflow:

• Code to pipeline to validation to deployment to operations

All accessible from your editor. Cursor accelerates the building. Harness governs the shipping. And the handoff between the two disappears.

Watch the demo:

Getting Started

If you want to try it:

• Install the Harness Cursor Plugin from the Cursor Marketplace
• Authenticate with Harness using OAuth. No API keys or setup headaches
• Start using natural language to run pipelines, debug issues, and manage deployments

For example:

“Run the CI pipeline for this branch, check if the security scan passed, and promote to staging if it did.”

That is it.

AI is not just changing how we write code. It is changing expectations for how fast we should be able to ship it. But speed without control does not work in real environments. What we are building toward is something simpler:

A world where every step, from PR to production, is:

• Fast
• Governed
• Observable
• Auditable

Without forcing developers to leave their flow. This plugin is one step in that direction.

The question for enterprise AI in 2026 is no longer just which model. It’s which harness.

An agent harness is the system around the model. It decides what the agent remembers, what context it sees, what tools it can call, what it is allowed to do, and what happens when it is wrong.

The model provides intelligence. The harness provides control.

This is where the real engineering is happening. When Claude Code's source was accidentally exposed earlier this year, reports put it at more than half a million lines. None of that was the model. All of it was the system around the model.

The model gets you started. The harness gets you to production.

In software engineering

Software engineering is one of the first places this plays out. AI coding tools are writing and editing code. Autonomous agents are starting to deploy, operate, and respond to incidents. These are not suggestions anymore. They are changes to running software, made by agents acting on their own.

And one harness is not enough.

Two loops, two harnesses

Software engineering has two halves at the level that matters for agent harness design. Software development, where code gets written. Software delivery, where code becomes running software.

The inner loop is software development. Code gets written, edited, tested, and reviewed. Coding agents work here, close to the developer and bounded by the repository. Whether they live in an IDE, a terminal, a background session, or a web workspace doesn’t change what they do. They help one person write better code faster.

The outer loop is software delivery. Code becomes software that is built, tested, secured, deployed, verified, operated, and sometimes rolled back. That includes CI, security scans, deployments, infrastructure, feature flags, incidents, and approvals.

The two are loops different. The inner loop is about individual productivity. The outer loop is about organizational execution under risk. It crosses teams, touches production, uses secrets, enforces policy, and leaves an audit trail.

An agent delivering software can’t be a coding assistant with API access. It has to run inside a system that enforces the organization’s rules.

What goes wrong without the right harness

The stakes are easier to see by starting with what breaks.

Security. An agent with broad access to deploy, provision, and push config changes is a new attack surface. Prompt injection through a PR description, a poisoned dependency, or a malicious issue comment can turn an autonomous agent into the most privileged insider threat in the company. It acts under its own identity, with its own scoped credentials, doing exactly what it’s authorized to do. The attacker just redirects the authorization. Without an identity model and governed execution, every action the agent can take becomes a potential action path for an attacker.

Compliance. An agent that ships code without the same policy gates, approvals, and audit trails humans use creates a parallel path that regulators and auditors will challenge. A single deployment that skipped EU data residency review can trigger a finding that takes quarters to close. Cyber insurers are starting to scrutinize AI governance, and some are exploring exclusions or tighter terms for poorly governed AI. Within a year or two, “we have autonomous agents deploying code without an evidence trail” will be impossible to defend. Autonomous delivery without verification is autonomous liability.

Confident bad decisions. An agent with partial context looks like it’s working. It deploys during a change freeze. It rolls out a config change that breaks an upstream service. It enables a feature flag during an incident. Each failure is locally reasonable and globally wrong. Without the full knowledge graph, the agent keeps making the wrong call.

AI-specific failure modes. Autonomous agents fail in ways that deterministic automation doesn’t. They hallucinate actions, generating and deploying a Kubernetes manifest that doesn’t match reality. They get stuck in loops, rolling back and redeploying the same change until a human kills the process. They’re confidently wrong, proposing a fix that passes a weak policy gate and breaks production an hour later. No attacker involved. Without verification strong enough to catch them, errors reach production.

All of this has happened with deterministic automation, one mistake at a time. With autonomous agents, errors happen in parallel. A coding agent with bad context can push 10 broken PRs in 10 minutes. A delivery agent without verification can deploy 20 services before anyone notices.

Speed used to be the feature. With autonomous agents, speed is also the damage multiplier.

What a software delivery agent actually needs

A software delivery agent needs four things: memory, context, tools, and verification. The shape and stakes of each element are distinct.

Suppose a team is shipping a new version of a retailer’s checkout service on Thursday. Checkout depends on payments, inventory, fraud, and identity.

Memory: a graph of how your company ships

A Software Delivery Knowledge Graph is a connected map of services, teams, pipelines, deployments, incidents, policies, scorecards, and artifacts. Nodes and edges show how they all relate.

To answer “Is checkout safe to ship Thursday?”, the agent has to know which services checkout depends on, what their scorecards look like, whether any have open critical CVEs, whether there’s a change freeze, and who’s on call Thursday night.

Tha’is a graph query. If the agent doesn’t have the graph, it’s guessing.

Context: the live signal

Memory is the durable map. Context is the live signal. Memory tells the agent how the delivery system is connected. Context tells it what’s happening now.

Back to checkout. The agent sees that a chaos experiment last week showed payments fail when its Redis cache is unavailable. It sees that yesterday’s security scan flagged a critical CVE in a library fraud detection depends on. It sees that the new version changes the same config flag that caused an incident two weeks ago.

None of this is in the pull request. All of it matters.

Context isn’t something you assemble from scratch at runtime. It accumulates in the harness long before the agent is asked to act.

Tools: governed execution

People often assume “tools” means function calls to APIs. For a software delivery agent, it means something different. The agent can deploy to Kubernetes, run a database migration, apply a feature flag, trigger a security scan, run a chaos experiment, open and close an incident. Real actions, inside your network, using your credentials, under your policies, with full audit logging.

At Harness, every action runs through a Delegate: a lightweight worker inside your environment. Your VPC, your Kubernetes cluster, your data center. The agent issues an instruction. The Delegate executes it inside your perimeter and returns the result.

Secrets are decrypted inside the Delegate. Never in the agent’s context window, never in a model provider's memory, never in an audit log.

An agent with arbitrary production access is dangerous. An agent constrained by governed execution is governable.

Verification: proving the action was safe

This is the pillar coding and personal productivity agents don’t need at this depth. Software delivery agents do.

Three mechanisms make it concrete:

• Scorecards grade services against rules the organization defines. Test coverage, SLO compliance, library currency, critical CVEs. Every rule measurable. Every score live. Thresholds set by the organization.
• Policy gates block actions until conditions are met. “No deployment without a passing scorecard.” “No EU infrastructure change without a named EU approver.” The gate sits in the pipeline. The agent can’t route around it.
• Evidence is cryptographically signed proof that each action met its policy. When an auditor asks, “prove last Tuesday’s deployment passed security testing,” the system returns a tamper-evident record.

For checkout, the Thursday release is blocked unless the scorecard passes, no critical CVEs are open, no change freeze applies, and an EU compliance approver signs off. If any of those fail, the agent cannot deploy. If they all pass, the deployment runs through a Delegate and an evidence record is written.

The rules of the organization are enforced in the harness. The agent operates inside them.

The foundation is already built

I mentioned that an agent needs memory, context, tools, and verification. The good news: a modern software delivery platform like Harness already has the foundations, because truly automated delivery has always needed those four things.

A note on our name. We called the company Harness in 2017 because the original thesis was a safety harness for code: let developers move fast without breaking things. Pipelines, policies, approvals, rollbacks, evidence. The scaffolding that lets speed and safety coexist.

That thesis hasn’t changed. The mover has. Developers are still moving fast. AI agents are moving fast too, and faster. The harness has to hold both.

Pipelines aren’t agents. Pipelines are the harness that lets agents safely act. They’re the control plane where agent actions are evaluated, constrained, and executed under policy.

The word “pipeline” carries baggage. Many people hear “script runner.” That isn’t what we mean. Harness pipelines are production orchestration engines: loops, matrix runs, parallel stages, conditions, approvals, OPA gates, rollback, retries, and deterministic-plus-agentic step-chaining.

An agent step can run inside a loop. A deterministic step can pass output to an agent, then to a policy gate, an approval, another agent, and a deployment. The agent isn’t replacing the pipeline. The agent is one kind of step the pipeline already knows how to run.

Harness pipelines execute hundreds of millions of runs a year across enterprise production systems. That isn’t a theoretical runtime for agents. It’s a runtime already hardened at scale, on real delivery, under real policy, with real rollback. That’s the difference between a script runner and a production harness for autonomous action.

The rest of the foundation maps the same way. The Delegate is how actions reach your infrastructure. The Software Delivery Knowledge Graph is the memory. Our platform modules are the tools. Scorecards, policy gates, and signed evidence are the verification. Harness AI, the intelligence layer on top, uses all four of these elements.

We didn’t set out to build an agent harness. We set out to build a software delivery platform with AI at its core. It turns out those two things are the same.

Why coding agents are a different harness

Coding agents (IDE copilots, background agents, terminal-based assistants, cloud coding sessions) are built for a different job. They know your codebase, your style, your recent commits. That’s a real harness, bounded by the repository and the developer. A software delivery harness has different scope, memory, risks, and accountability.

A coding agent’s memory is the repository. A software delivery agent’s memory is the organization.

The context gap. Ask your coding assistant: “Is it safe to deploy this checkout change to production tonight?” It can’t answer. It doesn’t know the current scorecard, the change freeze status, last week’s chaos test results, or who’s on call. None of that lives inside the developer's workspace. A coding agent can write a change. It can’t know if the change is safe to ship.

The blast radius gap. A coding agent’s bad change usually gets caught before it hurts anything: in review, in CI, in a security scan, on a policy gate. Fifteen minutes wasted, not a production incident. A software delivery agent’s worst day is customer data exposure, a production outage, or a regulatory incident. Same agent paradigm, radically different blast radius.

The safety-net gap. Both kinds of agents are moving toward less human oversight. The difference is what catches them when they’re wrong. A coding agent mistake gets caught downstream: by CI, by security scans, by policy gates, by the delivery harness itself. A delivery agent mistake has nothing downstream. It is the downstream.

The control-plane gap. Could a coding agent call Harness as a backend? Of course. It should. But the caller isn’t the control plane. The software delivery harness decides whether the request is allowed, how it executes, and what evidence is retained.

The preference gap. Developers are going to pick their own coding agents. Most enterprises already run two or three: Cursor on some teams, Claude Code on others, Copilot on others, whatever ships next year on yet other teams. That’s healthy. Software development is distributed by design. Software delivery is the opposite: it’s centralized. One company, one delivery control plane. One set of policies, one audit trail, one source of evidence, one place where credentials are held.

The winning pattern is the two meeting cleanly: whichever coding agent the developer picks, the deployment passes through the same delivery harness.

Why model providers aren’t the delivery harness today

Managed agents. Stateful APIs. Server-side memory. Model providers are extending into harness territory, and for many use cases, that works. For software delivery specifically, the architecture runs into a different set of constraints.

The credentials problem. Every software delivery action requires production credentials: cloud admin roles, Kubernetes service accounts, database passwords, secrets manager keys. The most sensitive assets in the company. Enterprises spend years building the controls around them: vaults, rotation, scoped access, audit trails. A model-provider-hosted agent loop would require those credentials to flow through the model provider’s infrastructure on every action. Few CISOs will approve it. Few auditors will sign off. In regulated industries, it’s often a non-starter.

The inversion. A model can be hosted anywhere. Any provider, any cloud. Execution has to happen inside the enterprise, using credentials that never leave. The model stays outside. The control plane runs inside. Intelligence can live anywhere. The control plane can’t.

The live-state problem. A software delivery agent’s answer to “Is this safe to ship?” depends on a state that changes every minute. The current change freeze. The latest incident. The newest CVE. Who’s on call right now. Whether the deployment window just closed. A model provider can reason about what you put in the prompt. It doesn’t naturally own the current state of your delivery system. A model provider knows the world. The harness has to know your world, right now.

The accountability problem. When a delivery agent does something wrong, the model provider isn’t on the incident bridge. The on-call engineer is. The platform lead is. The CTO is. The company is the one that has to explain the outage to customers, the finding to regulators, the miss to the board. Accountability can’t be outsourced. The harness that constrains the agent can’t be either.

A model provider can be the brain. It can’t be the harness for delivery.

AI for everything after code

More and more code will be written by AI. The bottleneck is shifting from code generation to safe delivery.

Coding agents help developers write code. Software delivery agents help teams safely deliver and operate it. Two harnesses. Two categories. Two sets of winners.

The foundation for software delivery is ready. The agents that need it are arriving now. The category now has a name.

We’ve always called it Harness. The idea just got bigger.

• Harness IaCM introduces native Terragrunt support, enabling true enterprise-grade orchestration at scale.
• Teams can now manage Terraform, OpenTofu, and Terragrunt in a single platform without fragmented tooling.
• Built-in governance, policy enforcement, and approvals streamline secure infrastructure operations.
• End-to-end visibility and drift detection improve reliability across complex, multi-environment deployments.
• The launch marks a major step toward a unified, multi-IaC control plane for modern infrastructure teams.

‍

Bringing First-Class Terragrunt Support to IaCM

“We’ve been operating in a hybrid environment with both OpenTofu and Terragrunt, and Harness has made it much easier to bring those workflows together into a single, consistent platform with IaCM. The addition of Terragrunt support is a valuable step toward simplifying how we manage infrastructure at scale.”

— Lead Platform Engineer, Enterprise Customer

Infrastructure as Code is now a standard for modern cloud operations, with most enterprises using IaC to provision and manage environments. However, as adoption grows, so does complexity. Teams are no longer managing a handful of environments. They are operating across multiple regions, accounts, and services, often at massive scale.

This is where traditional approaches begin to fall short.

As organizations scale their infrastructure, Terraform alone is often not enough. Teams adopt Terragrunt to manage complex, multi-environment deployments, but they are often forced to stitch together fragmented tooling that lacks visibility, governance, and consistency.

At Harness, we are changing that.

Today, we are excited to announce native Terragrunt support in Harness IaCM, bringing it to full parity with Terraform and OpenTofu while delivering capabilities that go beyond what is available in standalone tooling. This is more than support. It is about making Terragrunt a first-class platform for enterprise infrastructure management.

With Harness IaCM, teams can now:

• Orchestrate complex Terragrunt environments with full visibility across all units
• Apply cost estimation, approvals, and policy enforcement natively
• Detect and manage drift across environments with granular insights
• View infrastructure changes at the resource level across orchestrated deployments

Terragrunt has become a critical layer for managing infrastructure at scale because it simplifies how teams structure and reuse configurations across environments. Harness builds on that foundation with deep, native integration, enabling platform teams to operate with both flexibility and control.

This is especially important for enterprises where a single deployment spans multiple environments and services. Harness abstracts that complexity while maintaining governance, auditability, and consistency.

Extending IaCM to a Multi-IaC Future

Terragrunt is part of a broader shift toward multi-tool infrastructure strategies.

Modern teams are no longer standardized on a single IaC tool. Instead, they operate across:

• Terraform and OpenTofu for provisioning
• Terragrunt for orchestration
• CDK for developer-driven infrastructure
• Ansible for configuration and automation

This creates challenges around consistency, visibility, and governance. Harness IaCM is built for this reality. We are evolving IaCM into a unified control plane for multi-IaC workflows, where teams can manage different frameworks with a consistent experience, shared policies, and centralized visibility.

This means:

• Eliminating fragmented pipelines across tools
• Standardizing governance across environments
• Gaining full visibility into infrastructure state and changes

Instead of managing infrastructure in silos, teams can now operate from a single platform across the entire lifecycle.

What’s Next for Infrastructure as Code?

The next phase of Infrastructure as Code is not just about supporting more tools. It is about making infrastructure systems more intelligent and automated.

We are investing in two key areas:

Expanded IaC Support

We are continuing to support modern frameworks like AWS CDK, enabling developer-centric infrastructure workflows alongside provisioning, configuration, and orchestration tools.

AI-Driven Automation

We are introducing intelligence into IaC workflows to simplify tasks such as drift management and optimization. This helps teams reduce manual effort and operate more efficiently at scale.

Together, these investments move IaCM toward a unified, multi-IaC platform that combines flexibility, governance, and automation. Terragrunt has become essential for managing infrastructure at scale but until now, it hasn’t had a platform that truly supports it. As infrastructure continues to grow in complexity, our focus remains the same. Helping teams move faster, reduce risk, and scale with confidence no matter which IaC tools they use.

The release of Anthropic Mythos and Project Glasswing marks an exciting and pivotal new chapter in software development. As the industry advances, the speed and economics of vulnerability exploitation have fundamentally shifted. What once took weeks of manual reconnaissance can now be scaled rapidly through automated models. However, this is not just a security problem to solve. It is a massive engineering opportunity to build cleaner, more robust systems. By leaning into AI-accelerated defense, engineering teams are uniquely positioned to lead the charge and redesign the landscape of modern software architecture.

Breaking Down Silos and Establishing Shared Accountability

To succeed in this new era, the traditional silos separating security and engineering must fall. Defense at machine speed requires a unified front.

• Organizations need a shared roadmap and accountability model across Engineering, Infrastructure, and Security.
• These roadmaps must be crafted jointly with clear responsibility assigned per action item.
• Every executive and their corresponding team will be affected and accountable for changing the way work is done.
• Preparations for these improvements should be treated exactly like new product features.
• Savvy customers will start to pay attention to companies who are responding to Mythos, turning your proactive resilience into a highly visible competitive advantage.

Core Engineering Imperatives

The foundation of AI-accelerated defense relies on sound, proactive engineering practices. Developers must take ownership of architectural hygiene from the ground up.

• Accelerate velocity: Teams must focus heavily on shortening patch and change cycles (such as with Harness CI and CD). The single most important metric is how quickly you can safely make changes.
• Shift left completely: You must find bugs before you ship code. Achieve this by integrating SAST, SCA, and auto-pen testing into a secure pipeline, and prefer using memory safe code languages.
• Design for resilience: Always build with breach assumed. In practice, this means implementing zero-trust, isolating services by identity, and using short lived tokens by default.
• Simplify the architecture: As you engineer and build for resilience and simplicity , take time to audit your current code base to reduce dependencies and standardize on known good services and libraries. Additionally, actively reduce and inventory what you expose.
• Pay attention runtime: Aside from bugs, engineering teams haven’t traditionally paid attention to the run-time security of their applications. Aside from the functional insights developers can glean from runtime security tools, understanding how a system is attacked can help you make better architectural and functionality decisions.

Planning for the Unexpected

Even with the best architecture, unexpected friction will occur. Resilient engineering means planning comprehensively for your ecosystem.

• Ensure you know your software dependencies and precisely who to contact in emergencies.
• Engineering teams should build technical work-arounds for times when providers or internal systems experience issues.
• Organizations must establish a surge defense capability. When faced with a severe situation, have a SWAT team established with pre-approved authority, budget, and standard operating procedures across domains and outside help.
• At the company level, pre-position high-visibility incident response. This includes having pre-approved and crafted messaging triggered by established conditions.

Security as an AI-Powered Partner

To keep pace with the increased velocity of engineering teams, Security teams must also evolve their operational models.

• Security needs to leverage AI to de-toil high calorie activities.
• Practical applications include putting a model in front of your alert queue and testing it regularly.
• AI should also handle the triage and prioritization of scan findings alongside ticket ops automation.
• It is crucial to automate the technical incident response pipeline.
• By automating the bookkeeping around incidents, human decisions should be made with assistance at most.
• The ultimate goal is to find places to leverage AI and accelerate the time between incident and resolution.

Leading the Charge

Engineering leaders and developers are in the perfect position to navigate this industry inflection point. By taking ownership of these structural changes today, you ensure the long-term viability of your products and the enduring strength of your codebase. Bring your security, infrastructure, and engineering teams together into the same room and start building your shared roadmap today.

What happens when your Infrastructure as Code management strategy works perfectly in dev, scales reasonably well in staging, and then quietly fractures across seventeen production workspaces because nobody documented which Terragrunt wrapper goes with which AWS account? You spend Friday afternoon reverse-engineering DRY patterns that made sense six months ago, wondering why your team is managing three different IaC execution engines with four incompatible workflow philosophies.

This scenario isn't hypothetical. It's the reality of organizations that adopted IaC incrementally, layer by layer, without a unified management approach. One team standardized on OpenTofu for new infrastructure. Another maintained legacy Terraform configurations because migration felt risky. A third discovered Terragrunt and used it to wrangle complexity across AWS regions, but now those wrappers exist outside any centralized governance model. Each decision was rational in isolation. Together, they created an orchestration problem masquerading as a tooling problem.

The actual challenge isn't choosing between Terraform, OpenTofu, or Terragrunt. It's managing their outputs, enforcing policy consistently across execution contexts, and ensuring that infrastructure changes don't outpace your ability to understand what's deployed.

The Hidden Complexity of Multi-IaC Environments

Most platform teams don't set out to run multiple IaC tools simultaneously. They inherit Terraform state from acquisitions, adopt OpenTofu for licensing predictability, and introduce Terragrunt because someone needed to stop copying backend configurations across 40 AWS accounts. The tools themselves aren't the problem. The problem is that each tool introduces its own state management assumptions, module resolution logic, and workflow expectations.

Terragrunt, for instance, exists specifically to solve Terraform's verbosity problem. It lets you define backend configurations once and reference them across environments. It supports dependency graphs so you can deploy a VPC before attempting to create subnets. These capabilities are valuable, but they also mean your actual infrastructure logic now spans two layers: the Terraform or OpenTofu code that defines resources, and the Terragrunt configuration that orchestrates execution.

When you lack centralized Infrastructure as Code management, those layers drift independently. Someone updates a Terragrunt dependency graph without realizing it breaks a downstream workspace. Another engineer modifies an OpenTofu module but forgets that three different Terragrunt configurations depend on its output structure. You don't discover these issues until a deployment fails in production, and the postmortem reveals that nobody had visibility into the full dependency chain.

Why Workflow Sprawl Defeats Governance at Scale

The typical response to multi-IaC complexity is to standardize on one tool and deprecate the others. That works if you're early in your IaC journey. It's impractical if you're managing hundreds of workspaces across regulated environments where compliance audits expect immutable infrastructure definitions and audit trails for every state change.

Here's what actually happens: platform teams create custom CI/CD pipelines for each tool. Terraform runs in Jenkins. OpenTofu runs in GitHub Actions. Terragrunt configurations use a shell script someone wrote during an incident. Each pipeline implements drift detection differently. Policy enforcement exists as scattered OPA rules that don't share a common evaluation context. When an auditor asks, "How do you prevent unapproved infrastructure changes?", the honest answer is, "We run some checks in some places, and we hope teams remember to use them."

This isn't negligence. It's what emerges when Infrastructure as Code management tooling doesn't natively support the reality of polyglot IaC environments. Teams need a system that treats OpenTofu, Terraform, and Terragrunt as execution details, not architectural boundaries. The workflow layer—plan generation, policy evaluation, approval gates, state locking—should remain consistent regardless of which engine interprets the configuration.

Infrastructure Automation Tools Need Orchestration, Not Just Execution

Running `terragrunt apply` successfully doesn't mean your infrastructure is well-managed. It means Terragrunt successfully invoked OpenTofu or Terraform and applied a configuration. The actual management work—validating inputs, enforcing cost policies, detecting drift, promoting changes through environments—exists outside the execution layer.

This is where most homegrown solutions collapse under their own weight. You build a wrapper script that runs Terragrunt with the right flags. Then you add pre-commit hooks for policy checks. Then you integrate Sentinel or OPA, but only for workspaces that someone remembered to configure. Then you add Slack notifications so people know when drift occurs, but the notifications don't include enough context to act on them. Eventually, you have a Rube Goldberg machine that works until it doesn't, and debugging requires institutional knowledge that exists in one person's head.

The fundamental issue is that IaC workflow optimization requires thinking beyond execution engines. You need orchestration that understands module dependencies, workspace relationships, and policy boundaries. You need variable management that doesn't require copying YAML files between repositories. You need drift detection that runs automatically and surfaces meaningful deltas, not raw Terraform output dumped into a log file.

Terragrunt Support as a First-Class Workflow Primitive

Treating Terragrunt as an afterthought—something teams bolt onto existing Terraform or OpenTofu pipelines—misses its architectural intent. Terragrunt exists because managing backend configurations, passing outputs between modules, and orchestrating multi-account deployments shouldn't require copying boilerplate across dozens of directories. When Infrastructure as Code management platforms support Terragrunt natively, they acknowledge this reality: the DRY principle applies to infrastructure orchestration, not just resource definitions.

Native Terragrunt support means the platform understands dependency graphs without requiring custom parsing logic. It means workspace templates can reference Terragrunt configurations directly, rather than forcing teams to flatten everything into monolithic Terraform modules. It means policy enforcement applies before Terragrunt invokes the underlying execution engine, catching invalid configurations before they generate failed plans.

This matters most in organizations running multi-region or multi-cloud architectures. A typical pattern: one Terragrunt configuration defines networking across AWS regions, another manages Kubernetes clusters, a third provisions databases. Each configuration depends on outputs from the others. Without native orchestration, teams either write brittle shell scripts to sequence these dependencies or accept that deployments sometimes fail halfway through because someone applied changes out of order.

Multi-IaC Tools Require Unified State and Policy Management

The real test of an Infrastructure as Code management platform isn't whether it runs OpenTofu or Terraform. It's whether it provides consistent state visibility, policy enforcement, and audit trails across both. If your platform requires separate workflows for each execution engine, you've automated the mechanics but not the governance.

Consider policy evaluation. A reasonable security requirement: no S3 buckets should allow public read access. With fragmented tooling, you implement this rule multiple times. Once for Terraform workspaces using Sentinel. Again for OpenTofu configurations using OPA. A third time for Terragrunt-managed infrastructure, where you're not sure which policy engine applies because Terragrunt is just orchestrating calls to Terraform or OpenTofu. When an audit occurs, you can't prove consistent enforcement because there's no unified policy evaluation layer.

The same fragmentation affects drift detection. Terraform Cloud detects drift for Terraform-managed resources. Your OpenTofu workspaces might run scheduled reconciliation jobs, or they might not—it depends on whether someone configured them. Terragrunt configurations drift silently unless you've built custom tooling to periodically run `terragrunt plan` and parse the output. The result: partial visibility across your infrastructure estate, where "managed by IaC" becomes aspirational rather than descriptive.

OpenTofu Integration and Terraform Alternatives in Practice

Organizations exploring Terraform alternatives often focus on licensing or community governance. Those considerations matter, but they don't address the operational question: how do you manage infrastructure deployed with multiple execution engines without creating parallel workflow systems?

OpenTofu integration means more than "we can run OpenTofu commands." It means workspaces provisioned for OpenTofu behave identically to Terraform workspaces at the orchestration layer. Variable sets apply consistently. Policy evaluation uses the same rule sets. Drift detection runs on the same schedule. Approval workflows follow the same governance model. The execution engine becomes an implementation detail, not a workflow boundary.

This distinction matters during migrations. Teams don't flip entire infrastructure estates from Terraform to OpenTofu overnight. They migrate incrementally, starting with non-critical workspaces and expanding as confidence grows. If your Infrastructure as Code management platform treats each engine as a separate silo, you're managing two parallel systems during the transition. If the platform abstracts execution details behind a unified orchestration layer, the migration becomes a configuration change, not an architectural overhaul.

IaC Orchestration Beyond Engine Selection

The hard problems in infrastructure management aren't technical; they're organizational. How do you ensure that 40 engineers across six teams follow the same approval process for production changes? How do you enforce cost policies without blocking legitimate deployments? How do you maintain audit trails that satisfy compliance requirements without turning every infrastructure change into a bureaucratic ordeal?

IaC orchestration platforms solve these problems by decoupling policy from execution. Instead of embedding governance rules in CI/CD pipelines—where they're invisible, untestable, and easy to bypass—you define them once at the platform level. Instead of writing custom scripts to sequence Terragrunt dependencies, you describe the dependency graph declaratively and let the platform handle execution order. Instead of building bespoke drift detection logic, you configure detection schedules and let the platform surface meaningful deltas.

This approach doesn't eliminate complexity. It consolidates complexity into a layer designed to manage it. Your IaC configurations remain simple: modules that define resources, Terragrunt wrappers that eliminate boilerplate, workspace configurations that specify execution context. The orchestration platform handles everything else: state locking, policy evaluation, approval workflows, audit logging, drift remediation.

Harness IaCM: Orchestration for Multi-IaC Environments

Harness Infrastructure as Code Management approaches these challenges by treating the execution engine as a deployment detail, not an architectural constraint. Whether you're running OpenTofu, Terraform, or Terragrunt, the orchestration layer remains consistent: standardized pipelines for plan generation and apply operations, unified policy enforcement across all workspaces, centralized drift detection that surfaces actionable insights.

• Native Terragrunt support means dependency graphs defined in Terragrunt configurations are understood and respected during execution. You don't need custom scripts to sequence deployments across modules or AWS accounts. The platform interprets dependencies and orchestrates execution accordingly, applying changes in the correct order while maintaining state consistency.

• The Module and Provider Registry provides a centralized source of truth for infrastructure components, whether they're Terraform modules, OpenTofu modules, or Terragrunt configurations. Variable Sets and Workspace Templates eliminate configuration duplication, letting you define environment-specific values once and reference them across workspaces. Default plan and apply pipelines ensure consistent execution patterns without requiring custom CI/CD configurations for each workspace.

• Policy enforcement happens before execution, not after. Whether you're using Terraform, OpenTofu, or Terragrunt, policies evaluate against the generated plan before any infrastructure changes occur. Drift detection runs automatically, comparing deployed infrastructure against IaC definitions and surfacing discrepancies through a unified interface. Audit trails capture every state change, policy evaluation, and approval decision, providing the compliance evidence required in regulated environments.

For teams managing infrastructure across multiple clouds, regions, or execution engines, Harness IaCM provides the orchestration layer that makes polyglot IaC environments manageable. The platform doesn't force you to standardize on a single tool. It provides governance, visibility, and workflow consistency regardless of which engine interprets your configurations.

Governance Enables Velocity in Multi-IaC Environments

The promise of Infrastructure as Code—reproducible deployments, version-controlled infrastructure, collaborative development—only materializes when you have consistent orchestration across execution engines. Running Terraform in one pipeline, OpenTofu in another, and Terragrunt through a shell script doesn't scale. It creates workflow fragmentation that defeats governance and slows teams down.

Effective Infrastructure as Code management platforms abstract execution details behind unified workflows. They treat Terragrunt as a first-class orchestration primitive, not an afterthought. They provide native support for OpenTofu alongside Terraform, recognizing that organizations migrate gradually, not overnight. Most importantly, they enforce policy, detect drift, and maintain audit trails consistently across all workspaces, regardless of which engine runs the actual infrastructure changes.

The technical lesson: orchestration complexity belongs in platforms designed to manage it, not scattered across custom scripts and fragmented CI/CD pipelines. The operational lesson: governance doesn't slow teams down when it's embedded in the workflow rather than bolted on afterward. Multi-IaC environments are manageable when you have the right orchestration layer. Without it, you're just running tools in parallel and hoping they don't conflict.

Explore how Harness Infrastructure as Code Management handles multi-IaC orchestration, or review the technical documentation or implementation details. The product roadmap outlines upcoming capabilities for workflow optimization and policy enforcement.

The Gap: Modern DevOps Stops at the Database

Most development teams today build everything around Git, and deploy with GitOps principles.

Code sits in version controlled environments, changes go through PRs, and deployments are handled through modern CI/CD. That part is pretty standard at this point, especially when using a modern DevOps platform like Harness.

MongoDB fits into that developer world and workflow pretty naturally. Data is stored in documents that look a lot like JSON, the format many developers already use in application code and APIs. Under the hood, MongoDB stores those documents as BSON, which is essentially a binary form of JSON that supports additional data types like dates, object IDs, and binary data. That means developers get a familiar model to work with, while MongoDB gets a format that is efficient for storing and querying application data. 

Looks just like JSON, with native types like ObjectId and dates powered by BSON.

The tradeoff is that structure isn’t always defined upfront. Schemas change over time, and not always in a clean or consistent way.

Collections can contain documents with different shapes. Index changes can directly impact performance. These aren’t problems on their own, but they require discipline to manage safely.

MongoDB changes are often handled outside the standard development workflow, whether that’s by developers, platform teams, or database teams.

Teams rely on application-level updates or one-off scripts to backfill data, modify structures, or create indexes. These approaches work, but they’re not always consistently versioned in Git. Execution can vary across environments, and review or validation is often informal.

The result is limited visibility into what changed, when it changed, and how it was applied. Over time, that leads to inconsistencies between environments and increased risk during deployment. 

Flexibility is powerful, but without proper controls it introduces risk.

To solve this, teams need to bring MongoDB changes into the same workflow they already trust for application code: Git-driven, reviewable, and automated.

What GitOps for MongoDB Actually Looks Like

GitOps for MongoDB isn’t about changing how Mongo works. It’s about changing how changes are managed.

Instead of handling updates through scripts or application logic alone, database changes are treated like application code. Index creation, schema validation rules, and migration scripts are all defined in Git and tracked over time. This includes MongoDB’s native schema validation rules, which can be versioned and applied consistently across environments.

Changes need to go through pull requests, just like any other code change. This allows developers, platform teams, and DBAs to review what’s being modified before anything runs in an environment.

From there, pipelines handle the validation and deployment. Changes are applied consistently across environments, rather than being run manually and potentially differently each time.

In practice, this means a new field, an index, or a backfill isn’t just a script someone runs once. It’s a versioned change that can be reviewed, tested, and repeated.

This isn’t about forcing rigid schemas onto MongoDB. It’s about making changes visible, consistent, and easier to manage as systems grow.

Harness DB DevOps provides the structure to do this. With Harness, we define changes as changesets, store them in Git, and deploy them through pipelines with built-in validation and policy checks.

To demonstrate how this works, we will walk through a practical MongoDB change from start to finish.

Example: Managing a MongoDB Change End-to-End

Here’s a simple example: A team needs to add a new userPreferences field to the users collection and create an index to support a new query.

Instead of writing a script and running it manually, we define the change and commit it to Git.

1. Define the change in Git
A developer creates the update as a changeset. That includes the logic to add or backfill the new field, along with the createIndex operation needed for performance. The change is committed alongside application code, like any other update.

2. Open a pull request
From there, the change goes through a pull request. Other developers or DBAs can review what’s being changed before anything runs. If something looks off, it gets caught here instead of in production.

3. Let the pipeline take over
Once the change is approved, the pipeline takes over.

The Pipeline

Before anything gets applied, the change is validated and previewed against the target environment. This helps catch issues early, whether it’s a conflict, a bad query pattern, or something that could impact performance.

This is especially important for heavy operations like index creation on massive collections, where resource contention and performance degradation are real risks. Instead of running those changes manually, pipelines can enforce safe rollout strategies like rolling index creations across replica sets, without manual intervention.

Policies are enforced as part of that same process, with required approvals, environment rules, and other guardrails checked automatically so teams aren’t relying on someone to manually verify every step.

Once everything passes, the change is deployed through the pipeline and applied consistently across environments, moving from dev to staging to production in a controlled way. No one is logging into a database to run scripts by hand.

Now, everything is tracked. You can see what was applied, where it was deployed, when it happened, and who approved it, with a full history available if something needs to be reviewed or rolled back later.

Sound familiar? This workflow should sound a lot like application delivery, where changes are versioned, reviewed, validated before deployment, and visible after.

From Manual Oversight to Guardrails

Traditionally, database changes have been tightly controlled by DBAs. They review scripts, approve changes, and sometimes execute them manually in each environment. That model helps reduce risk, but it doesn’t scale as teams grow and release more frequently.

With a GitOps approach, that control doesn’t disappear, it moves earlier in the process.

Instead of reviewing every individual change, database teams define policies and standards up front. Those rules are then enforced automatically through pipelines. Every change must pass the same checks before it reaches an environment, without requiring manual intervention each time.

In practice, this means:

• Required approvals are enforced automatically
• Deployment rules are validated before execution
• Production environments can have stricter controls than lower environments
• Policies are applied consistently across every change

The role of the database team evolves from gatekeeper to system designer. Rather than being involved in every deployment, they define the guardrails that ensure every deployment is safe.

Developers still move quickly, but now within a controlled, repeatable system.

What Teams Actually Gain

Bringing MongoDB into a Git-driven workflow changes how teams ship.

• Faster delivery
  Developers don’t wait on manual reviews or execution. Changes move through the same workflow as application code.
• Consistent environments
  Changes are applied the same way every time, reducing drift between dev, staging, and production.
• More predictable deployments
  Validation and policy checks happen before execution, catching issues earlier and reducing surprises.
• Built-in auditability
  Every change is versioned, reviewed, and tracked, with full visibility into what changed, when, and why.

MongoDB's flexibility doesn't eliminate the need for structure - it just shifts the responsibility for maintaining consistency from the database itself to your development processes.

If your application is managed through Git, your database should be too.

Schedule a demo

If you've ever run an ALTER TABLE on a busy MySQL table in production, you know the feeling. The change is small. The risk isn't. Long-running table locks, queued writes, application timeouts, replication lag, a five-minute migration that turns into a half-hour incident review.

We're shipping an integration that takes that anxiety out of the loop. Harness Database DevOps now supports Percona Toolkit for MySQL as part of Liquibase-based schema management. Flip a checkbox at schema creation, and eligible changes execute through pt-online-schema-change instead of native MySQL DDL.

Why it matters

Native ALTER TABLE on MySQL can lock tables for as long as the change takes to apply. On a large or hot table, that means writes pile up, dependent services start timing out, and replicas fall behind.

Percona Toolkit handles the same change very differently. pt-online-schema-change creates a shadow table with the new schema, copies your data over in small chunks, uses triggers to keep the original and shadow tables in sync, then performs an atomic swap with minimal lock time. The practical upside: schema changes you can run during business hours, not at 2 AM with a runbook open.

How to turn it on

The integration is enabled per schema. When you create a Database Schema in Harness DB DevOps:

1. Fill in your MySQL connection details.
2. Select Liquibase as your schema configuration.
3. Check the box labeled Use Percona Toolkit.
4. Save and continue with your normal workflow.

That's it. With the box unchecked (the default), Harness DB DevOps applies your changelogs using native MySQL operations through Liquibase, exactly as before. Check it, and eligible changes route through Percona Toolkit instead.

Things worth knowing before you flip the switch

Percona Toolkit isn't a silver bullet for every DDL. A few cases need extra thought.

Adding or dropping foreign keys can break during the table swap, so plan those changes carefully or apply them outside the toolkit. Tables without a primary key or unique index won't migrate safely either, since pt-online-schema-change needs one to chunk data deterministically. And a handful of specific operations sit outside the safe-change envelope: dropping a primary key, complex column reordering, and some storage engine swaps.

You'll also want to give the database user the right privileges: ALTER, SELECT, INSERT, and UPDATE on the target table, plus CREATE and DROP on the database for shadow table management.

The full list of supported patterns, edge cases, and required permissions is in the Harness DB DevOps docs.

Try it on your next migration

If you're already running Harness DB DevOps for MySQL, the next schema you create is a good place to try this. Turn it on against a non-critical environment first, watch how it behaves on your workload, and the path to using it in production gets a lot shorter.

For teams running MySQL at scale, that's one fewer reason to schedule schema changes around your customers' sleep.

If you aren't already using Database devops, speak with our experts to discuss how you can achieve zero downtime database schema migrations.

• Mean Time to Failure (MTTF) turns repeated failures into a signal that can be predicted, allowing platform teams to plan for the number of incidents, the amount of capacity they need, and the SLOs they can actually meet.
  
• To get an accurate Mean Time to Failure, you need to have clean data, clear definitions of failure, and separate short-lived components from long-lived services instead of averaging everything together.
  
• Connecting Mean Time to Failure with MTTR, SLOs, and AI-powered automation from Harness changes how you think about failures from random chaos to an automated control loop that runs reliability.

‍

Your production problems aren't just random. If a Kubernetes node fails every 72 hours or your CI runners crash every 4 builds, that's a clear pattern. Mean Time to Failure (MTTF) turns these failures into data that you can control, plan for, and improve over time.

MTTF should not be a decoration on a dashboard for platform engineering leaders; it should be a decision-making tool. With the right calculations, you can set realistic SLOs, plan capacity, and cut down on developer work by focusing on the parts that break the most often. You'll get exact formulas for distributed systems, data collection patterns that avoid common mistakes, and a playbook to turn reliability improvements into measurable ROI through automated resilience practices alongside faster recovery metrics.

Stop letting unpredictable failures drain your team's time and budget. With Harness Continuous Integration and Continuous Delivery, you can turn MTTF insights into concrete pipeline changes, progressive delivery strategies, and guardrails that keep reliability improving release after release.

What Is Mean Time to Failure (MTTF)?

Mean Time to Failure (MTTF) is the average operating time of non-repairable components before failure across a population.

At a basic level:

MTTF = total operating time ÷ number of failures

If 100 CI runners each run for 50 hours during a week (5,000 runner‑hours total) and 20 runners experience at least one hard failure, then:

MTTF = 5,000 ÷ 20 = 250 hours

Historically, MTTF is used for physical assets you replace instead of fix (light bulbs, disks, sealed devices). In software, the same concept fits ephemeral resources such as:

• Short‑lived containers and pods.
  
• CI/CD job runners and build agents.
  
• Batch jobs and serverless functions that are recreated after failure.

MTTF tells you how long things run, on average, before they fail and must be replaced. MTTF is an approximation, not a strict reliability model.

MTTF vs MTTR vs MTBF in DevOps Workflows

Three reliability metrics show up in every platform review:

• Mean Time to Failure (MTTF):  Measures how long non‑repairable or ephemeral components run before failing and being replaced.
  
• Mean Time to Repair/Restore (MTTR): Measures how long it takes to restore a service to healthy operation after a failure.
  
• Mean Time Between Failures (MTBF): Measures how much uptime you get between failures for systems that are repaired and returned to service.

Use them to answer different questions:

• MTTF: How often do our ephemeral components fail?
  
• MTTR: How quickly can we restore user‑visible services when they do?
  
• MTBF: How stable are our long‑lived systems between failures?

For example:

• Track MTTF for pod lifetimes or CI runner stability so you can forecast incident volume and choose when to add redundancy or auto‑healing.
• Track MTTR for customer‑facing services and tie it to SLOs and incident response.
• Track MTBF for databases or API gateways so you can plan maintenance windows and capacity around real behavior instead of theoretical SLAs.

Your platform scorecards should display all three together, alongside SLO health and error budget burn, so teams see the full reliability picture instead of optimizing a single metric in isolation.

When MTTF Applies to Software

The theoretical rules around MTTF and MTBF are straightforward; the ambiguity comes when you apply them to real cloud‑native stacks. Concrete examples help.

Where MTTF is Most Meaningful

These components typically behave like non‑repairable items:

• Pods and containers that are recreated on failure.
  
• CI runners and ephemeral build agents that are provisioned per job or per batch.
  
• Batch jobs and workers that run to completion and are never “repaired” mid‑flight.
  
• Serverless functions are triggered on demand and recreated automatically.
  

For each of these, you can treat a single lifecycle (from start to failure/termination) as one observation in your MTTF dataset.

Where MTBF (Plus MTTR) is A Better Fit

These components behave more like classic repairable systems:

• Databases and stateful services that are brought back through restart, patch, or failover.
  
• API gateways, control planes, and load balancers that remain logically continuous across many incidents.
  
• Long‑running worker pools that survive node or pod replacements.
  

For these, you care more about how much uptime you get between failures (MTBF) and how quickly you can restore full health (MTTR).

The Common Pitfall: Equating Infra MTTF With User Reliability

It is tempting to say “our nodes have an MTTF of 720 hours, so our service is very reliable.” That is only true if your architecture masks those failures from users. User‑facing reliability lives at the service boundary, measured via SLOs and error budgets; component MTTF is an input that helps you:

• Find noisy dependencies.
  
• Plan redundancy and failover.
  
• Decide where to add automation or chaos tests.
  

MTTF helps you understand where things break; SLOs and MTTR tell you how much that matters to customers.

How to Calculate MTTF for Distributed Systems

The MTTF calculation is trivial. The work is in collecting honest data across a distributed system without losing important details.

1. Define Specific Failure Events

For each component type, decide exactly what counts as “failed,” for example:

• Container exits with a non‑zero status.
  
• Pod enters CrashLoopBackOff or is evicted for resource pressure.
  
• CI pipeline reaches a terminal “failed” state (not just retried).
  
• Node is in NotReady for longer than an agreed threshold.

Document these in your platform taxonomy so every team logs and reports failures the same way.

2. Track Lifecycle Start and End for Each Instance

For each instance in the population you’re measuring, capture:

• Start time – when the instance began its lifecycle.
  
• End time – either the failure time or the end of your observation window.
  
• Instance identifier – pod name, job ID, runner ID, etc.

Then compute:

MTTF = total operating time across all instances ÷ number of failed instances

This gives you MTTF for that class (e.g., “Linux GPU runners in prod”).

3. Segment by Workload Class and Aggregate With Weights

Never pool dissimilar components into a single MTTF number. Instead:

• Group instances into workload classes (service, environment, hardware profile, or usage pattern).
  
• Compute MTTF for each class independently.
  
• When needed, roll up with weighted exposure hours, not a simple average.

Example:

• Class A: 1,000 hours, 5 failures → MTTF_A = 200 hours.
  
• Class B: 100 hours, 1 failure → MTTF_B = 100 hours.

Fleet MTTF (weighted) = (1,000 + 100) ÷ (5 + 1) ≈ 183 hours, not the naive (200 + 100) ÷ 2.

4. Include Right‑censored Data

Some instances will still be running when you take the snapshot. If you drop them:

• You shrink total operating time.
  
• You keep the same number of failures.
  
• Your MTTF ends up artificially low.
  

When censored samples are common, use basic survival analysis (like Kaplan–Meier) so that "still running" instances add to the exposure instead of being thrown away. If you give them clear timestamps and labels, observability tools and data teams can usually take care of this for you.

Using MTTF to Set SLOs and Reduce Toil

MTTF becomes strategically important when you use it to shape SLOs, error budgets, and reliability investments, not just track uptime.

1. Project Incident Volume From MTTF

If a class of components has an MTTF of 72 hours, a single instance will fail about:

8,760 hours/year ÷ 72 ≈ 121 failures/year

With multiple instances and redundancy, not every failure becomes a user‑visible incident, but you can still estimate:

• Roughly how many failures will your platform team see?
  
• How often will those failures stress specific SLOs or error budgets?

2. Prioritize Components That Generate the Most Toil

MTTF highlights which components generate excessive manual work:

• A CI runner class with low MTTF forces engineers to babysit builds.
  
• A particular pod type that fails often might drive a disproportionate share of pages.

Use this to:

• Rank components by failure frequency × human touch time.
  
• Focus on resilience and automation work where you retire the most pages and tickets per unit effort.

3. Translate MTTF improvements into business outcomes

Because MTTF underpins incident rates, any improvement can be tied to measurable gains:

• Fewer incidents and on‑call pages.
  
• Less time lost to debugging failures that do not reach customers.
  
• Less need for excess capacity added as a buffer against frequent failures.

Treat MTTF as a leading indicator: when you raise it on critical components, you should see downstream improvements in SLO attainment and delivery cadence.

Practical Ways to Improve Mean Time to Failure

Once you know which components have the lowest MTTF and the highest operational cost, you can systematically improve them. In modern delivery pipelines, four patterns tend to pay off quickly.

1. Stabilize CI Pipelines and Build Infrastructure

Flaky CI is one of the most common sources of low MTTF and wasted engineering time.

You can improve CI‑related MTTF by:

• Reducing test‑driven failures with Harness Test Intelligence, which selects only tests relevant to a change and helps you isolate flaky tests instead of letting them hammer your MTTF.
  
• Identifying failure hot spots using Harness CI analytics and insights to see which repos, branches, or services correlate with most build failures.
  
• Shortening build exposure windows using incremental builds, which reuse previous build outputs, reducing the time during which build infra can fail.

Result: higher MTTF for pipelines and runners, fewer broken builds, and fewer interruptions for developers.

2. Use Progressive Delivery and Rollback to Protect Service MTTF

You cannot prevent every bad change, but you can limit how many become full‑blown incidents that count against your service‑level MTTF.

Key tactics:

• Design for redundancy across environments with Harness “deploy anywhere”, so a single node or zone failure does not reset service health.
  
• Configure canary, blue‑green, or rolling strategies in Harness “powerful pipelines” so new versions are exposed gradually and can be rolled back quickly.
  
• Wire in verification steps so bad versions are caught early rather than after a significant SLO hit.

This keeps effective MTTF for user‑facing services higher, even if underlying components still fail regularly.

3. Enforce Reliability Guardrails With Pipeline Governance

Many MTTF regressions start as “just one more config change” that slips past informal reviews. Prevent those with:

• Policy‑as‑code in Harness DevOps pipeline governance, where you define rules around high‑risk changes, mandatory checks, and SLO states.
  
• Conditional approvals for changes that touch historically fragile services or dependencies.
  
• Blocks on deployments when key reliability gates, like SLO burn or unresolved incidents, are breached.

This ensures the MTTF gains you’ve earned are not eroded by ad‑hoc changes and one‑off exceptions.

4. Continuously Validate Resilience With Chaos Engineering

To sustainably raise MTTF, you need confidence that your architecture and runbooks can handle real failures, not just happy‑path tests.

By running targeted chaos experiments on the components with the lowest MTTF, you can:

• Discover previously hidden failure modes.
  
• Fix them before they cause production incidents.
  
• Demonstrate to stakeholders how your changes improve resilience over time.

Monitoring and Improving MTTF with AI‑Powered Automation

When failures happen, MTTF tells you how often they occur. AI‑powered automation helps you decide what to do next—fast—so more failures stay under control and never become major incidents.

AI‑assisted Detection and Rollback

Harness AI‑assisted deployment verification analyzes metrics and logs during and after each deployment:

• It learns what “normal” looks like for each service.
  
• It flags anomalies in latency, error rates, or custom SLO indicators.
  
• It can recommend or trigger rollback directly through your CD pipelines.
  

The result is fewer deployments turning into user‑visible failures and a higher effective MTTF for your services, because many problematic changes are automatically rolled back before customers notice.

On the CI side, AI‑driven analysis works with Test Intelligence and analytics to:

• Cluster failures by likely root cause.
  
• Surface patterns, such as a particular dependency, test suite, or environment causing repeated failures.
  
• Guide you toward changes that will increase MTTF for builds and jobs.

SLO‑driven Guardrails Instead of Ad‑hoc Decisions

SLOs and error budgets turn raw data into rules. Instead of making teams watch dashboards and make decisions on their own, you can:

• Use SRM to set SLOs for important services and keep an eye on how much of the error budget is being used.
  
• Set up pipeline rules that automatically slow down or stop deployments when SLOs are in danger.
• Direct engineering efforts toward services with a decreasing MTTF and an increasing error-budget burn.

This completes the cycle: MTTF informs SLO design. Guardrails are based on SLOs, and AI-powered verification and rollbacks work on those guardrails at machine speed.

Want to turn MTTF insights into automated reliability improvements?

Explore Harness CI/CD to reduce failure rates, enforce guardrails, and improve SLO performance.

MTTF: Frequently Asked Questions (FAQs)

MTTF can feel abstract until you have to justify reliability decisions or explain incident patterns to stakeholders. These FAQs break down the most common questions practitioners ask about MTTF and how it relates to other reliability metrics.

How is MTTF different from MTBF?

MTTF is the average time it takes for a group of parts, like pods or temporary CI runners, to fail in a way that can't be fixed. MTBF tells you how long systems you fix and put back into service, like databases or long-running services, are up and running before they break down again.

When should I use MTTF instead of MTTR?

When you need to know how often failures happen so you can plan for redundancy or auto-healing, use MTTF. Use MTTR to find out how quickly you can fix services that users can see after they go down. Both metrics work together and are usually used to help make decisions about SLOs and error budgets.

Can I trust MTTF if I only have a few failures?

MTTF estimates are very uncertain when there aren't many failures. To make the number more reliable, put similar workloads together, add up the exposure hours for each class, and think of MTTF as a range or trend instead of a single point. If a part didn't fail in your window, don't assume that it will never fail; instead, treat that as incomplete data.

What data quality issues most often skew MTTF?

Most of the time, MTTF is skewed by dropping instances that are still running when the measurement is taken (right-censoring), combining environments (staging, load, and production) into one metric, and having different or unclear definitions of failure across teams. Fixing these problems usually makes MTTF more useful than any other advanced statistical method.

When is MTTF the wrong metric for modern platforms?

MTTF doesn't work when failures are very similar or when you're measuring systems that are fixed instead of replaced. In those cases, MTBF and MTTR, when looked at through SLOs and error budgets, usually give better advice than just one MTTF value.

How does MTTF connect to business outcomes?

When the MTTF is higher on important parts, there are fewer problems, fewer pages, and less time lost by developers fixing them. You can link improvements directly to faster safe release velocity, lower downtime risk, and lower operational costs when you combine MTTF with SLOs, error budgets, chaos engineering, and AI-powered automation.

Modern application security goes from code to runtime. Vulnerabilities are found at every stage of the software development lifecycle (SDLC) - in the code developers write, open source packages they pull in, container images they build, and cloud infrastructure where it all runs. But finding vulnerabilities is no longer enough. With attack surfaces sprawling across pipelines, registries, and production environments, the harder problem is fixing the vulnerabilities that actually matter.

Understanding what’s important increasingly depends on correlating multiple data points. A critical CVE buried in a dependency looks very different depending on whether the vulnerable function is actually reachable, the library is used in production, or the affected service is internet-facing. Without runtime context, security and development teams are often left triaging noise instead of actually reducing risk. And fixing vulnerabilities discovered in production can be challenging without being able to follow the trail back to the repo and line of code where the vulnerability can be found.

No matter where application security lives in your organization - and increasingly, it lives in more than one place - Harness and Wiz are working together to make sure you're covered. Whether your team is shifting left from cloud security or pushing right from the development pipeline, integrating Harness and Wiz brings code and runtime findings together so you always have the context you need to act.

Shift Left, Protect Right. Who Owns Application Security?

Application security used to have a clear owner. The AppSec team ran the scanners, triaged findings, and created tickets for developers. But "shift left" has been pushing security earlier into the development process and ownership has been migrating toward the teams that actually write and ship code. Today, the DevSecOps or platform engineering team owns application security tooling in many organizations. They're the ones who know exactly where a vulnerability lives in code, who owns it, and how to get developers to fix them.

But as applications move to the cloud, cloud security and infrastructure teams have a stake in application security outcomes as well. They're the ones with visibility into what's actually running in production - what's internet-facing, what's over-privileged, what's actively being exploited. Cloud security platforms have expanded their focus from purely infrastructure and runtime back through the SDLC to code. For many cloud teams, application security isn't a handoff; it feeds into their cloud risk picture.

The result is that application security now has multiple stakeholders with different vantage points. DevSecOps teams see risk through the lens of the CI/CD pipeline and the developer workflow. Cloud security teams see it through the lens of the deployed environment and the blast radius of a breach. Neither view is complete on its own. The good news is that these teams don't have to choose between their tools or their workflows. They need integration that lets each team work in their context while sharing the signals that make both more effective.

Integrating Wiz Code into CI/CD Pipelines with Harness STO

DevSecOps teams need to expand right. SAST and SCA tools often generate more findings than any team can fix. Runtime context helps separate signal from noise. Knowing that a vulnerable service is actively internet-facing or that a dependency with a critical CVE is actually loaded in production changes how a team prioritizes. Without it, developers are left triaging based on CVSS scores alone. With it, they can focus effort where exposure is real and the risk in production is highest.

Harness Security Testing Orchestration (STO) makes it easy to orchestrate Wiz Code across your CI/CD pipelines. With a pre-built integration, you can deploy Wiz Code in just a few clicks instead of needing to create a custom integration or write custom scripts. Harness orchestrates Wiz Code alongside all your other scanners so you know your pipelines always get the required security tests, without needing to manually coordinate multiple tools.

Once Wiz Code is integrated, STO aggregates findings with other scanners in your pipeline, automatically deduplicating vulnerabilities so teams aren't triaging the same issue twice. The consolidated view means developers and security engineers can see the full picture in one place, understanding pipeline-level risk and assigning tickets to developers. In addition, Harness Policy as Code lets teams define and take action at the pipeline level instead of tool by tool, so decisions about what to fail a build on, what to flag for review, and what to pass through are applied consistently and holistically across every scan and pipeline.

Feeding Harness SAST and SCA Findings Into Wiz

Cloud security is pushing left - past runtime, past containers, all the way back to the code and open source packages that vulnerabilities originate from. The driver is enabling action. A misconfigured cloud resource or a vulnerable container image is more actionable when you can tie it back to the specific dependency introduced in a pull request, the developer who owns the code, and the pipeline that shipped it. Runtime findings without code context are just alerts. With code context, they become actionable work items that can be routed to the right person and fixed at the source.

Wiz Application Security Posture Management (ASPM) is designed to aggregate findings from across the SDLC and correlate them with runtime context - what's deployed, what's exposed, and what's actually at risk. By integrating Harness SAST and SCA scanner findings directly into Wiz, cloud security teams can connect the dots between a vulnerable open source package or insecure code pattern and the running workloads it affects. That correlation is what turns a list of CVEs into a prioritized risk picture that reflects what's actually happening in production.

For cloud security teams already working in Wiz, this integration means Harness SAST and SCA become part of their existing workflow rather than a separate tool to check. Code-level findings surface alongside runtime signals in the same platform where cloud risk is already being managed, analyzed, and acted on. Teams get broader coverage without adding friction, and the context that makes those findings meaningful - reachability, exposure, business criticality - is already there when they need it.

AppSec Is a Team Sport. Your Tools Should Be Too.

DevSecOps and cloud security teams are not generally not competing - they're looking at risk from different angles. One team lives in the development pipeline; the other lives in the cloud. Both need visibility into what the other sees to do their jobs well. When those views are siloed, findings get duplicated, priorities diverge, and the vulnerabilities that matter most fall through the cracks between teams.

Harness and Wiz close that gap from both directions. DevSecOps teams get runtime signals from Wiz Code inside the pipeline context where they already work, so they can prioritize fixes based on real-world exposure. Cloud security teams get code-level findings from Harness SAST and SCA inside the risk context where they already work, so they can trace production risk back to its source. Each team keeps their workflow. Both teams get the full picture.

The right combination of these integrations depends on how your organization is structured, where application security ownership sits today, and where you want it to go. If you're a Wiz customer evaluating how Harness SAST and SCA fit into your security program, or a Harness customer looking to bring runtime context into your pipelines, contact your Harness account team to understand how you can map the integrations to your specific environment.

Application security testing tools promise coverage and accuracy, but teams often struggle just to get started. One of the biggest friction points in dynamic application security testing is configuring authentication correctly so a scanner can even access a target application, let alone API endpoints that power the functionality. 

Whether it’s API keys, bearer tokens, or custom auth flows, setting up authentication for scans frequently requires trial-and-error and engineering support. This reality of scanning configuration slows down security validations, delays insights, and makes it difficult to integrate with AI-driven tooling that depends on fast, accurate access to API endpoints.

Today, we’re excited to introduce AI-Powered Custom Authentication Generation—a new capability designed to eliminate this friction and help teams move from setup to security insights faster than ever.

What’s New: AI-Powered Authentication Generation

With this release, teams can now generate and refine authentication configurations using natural language and LLMs. Instead of manually configuring authentication logic or relying on additional support, users can simply describe their requirements and let AI handle the rest.

The average time to configure authentication for API security testing is measured in seconds, whereas older manual approaches can take hours and require extensive trial-and-error. 

Here are a few highlights:

• Use Natural Language to Create Auth Scripts: Generate fully functional authentication configurations by describing your needs in plain English.
• Support for Common Auth Types: Easily create configurations for API keys, JWTs, Bearer tokens, and more.
• Iterative Refinement with AI: Update and fine-tune authentication logic by prompting the system instead of time-consuming scripting or manual adjustments.
• Inline Visibility and Change Tracking: Steps taken by AI to generate auth flows are audited, and testing runs are logged fully to maintain transparency and control.

Why is Authentication Setup Difficult Historically?

Authentication setup has long been one of the most frustrating parts of security scanning. Access control mechanisms are already complex due to security hardening used to protect applications and APIs. Successfully automating authentication flows so a machine can access an app or endpoint raises the bar substantially. 

Some of the common pain paints include:

• Manual configuration slows down work: Teams often need to consult different documentation, dashboards, and code to define authentication flows correctly. Even small mistakes can cause scans to fail silently or produce incomplete results.
• Lack of standardization creates confusion: Each application or API may use slightly varied access control mechanisms and authentication material, including headers, tokens, cookies, and custom flows. Practitioners must rebuild the logic from scratch each time due to inconsistent implementation.
• Iteration is painful and time-consuming: If something doesn’t work, fixing it usually requires manually editing scripts, rerunning scans, and debugging errors. Oftentimes, additional help from support or engineering teams is needed.
• Limited visibility makes troubleshooting harder: When authentication fails, it’s not always clear why. Without clear feedback or context, teams spend valuable time diagnosing issues rather than improving their security posture.

What should be a simple prerequisite, gaining authenticated context into an application, becomes a major bottleneck to dynamic application security testing. 

How AI-Powered Authentication Changes the Game

The new AI-powered authentication feature in Harness API Testing removes these barriers entirely by reworking how authentication config is created and managed.

Generate Authentication in Seconds

Users can navigate to the authentication configuration page, select the custom option, and simply describe what they need. For example:

“Generate an API key-based authentication hook where the token <token> is injected into the request header <authorization>.”

With a single click on “Generate with AI,” the system produces a complete, ready-to-use authentication script. This functionality eliminates the need to write or stitch together configurations manually.

Support for Multiple Authentication Types

The feature supports a range of common authentication mechanisms, including:

• API key-based authentication
• JWT-based authentication
• Bearer token authentication

This flexibility ensures teams can quickly configure access regardless of how their application or API is secured. Learn more details about the supported authentication types. 

Refine Without Rewriting

Authentication requirements often evolve. Instead of starting over, users can iteratively refine their configurations using natural language prompts.

For example, if you want to change how credentials are injected into the auth flow, you can simply say:

“Change the injection type to header name.”

By selecting “Refine with AI,” the system updates the existing configuration accordingly—no manual edits required.

Built-In Transparency

Every AI-generated or modified configuration includes inline comments that explain what changed. These comments make it easier for teams to:

• Understand how authentication is implemented
• Review updates confidently
• Maintain control over their configurations

Additionally, no credentials are stored in logs or persisted in prompts. Any sensitive authentication material is masked and encrypted at rest.

Faster, More Reliable Scans

By reducing setup errors and simplifying authentication configuration, this Harness API Testing feature directly improves scan success rates. Teams can spend less time troubleshooting authentication issues and more time analyzing real security findings.

Unlocking Faster, Smarter Security Workflows

This release is more than just a usability improvement. It’s a foundational step towards enabling AI-driven security workflows.

By removing the friction of authentication setup, teams can:

• Onboard new applications for testing faster
• Integrate seamlessly with AI-powered testing and analysis tools
• Reduce dependency on manual scripting and support interventions
• Achieve more consistent and reliable scan coverage

Ultimately, this translates to a faster time-to-value and a more scalable approach to dynamic application security testing.

Get Started Today

AI-Powered Custom Authentication Generation is available immediately with your existing Harness subscription. You can find related technical documentation here. 

Current Customers: Log in to your dashboard today to start exploring your threat data in a whole new dimension.

New to the Platform? If you aren't yet protected, contact us to schedule a personalized demo.

Request a demo

There is a version of the Legal team that exists in most companies: thorough, careful, and quietly overwhelmed. Good lawyers are spending their days on tasks that really should not require a lawyer at all.

We decided early on that this was not the team we wanted to be.

At Harness, the AI-first approach is not just saved for the engineering team. It is how every team operates, including Legal. That means we stopped asking “should we use AI?” a long time ago and started asking “how do we build with AI?” The result is a Legal team that does not just use AI tools. It develops them. We close faster, advise smarter, and frankly, have more fun doing it!

Our AI legal stack

Every tool in our stack has a job. Here is what that looks like in practice:

• Contract review and redlining: This is where we are headed this year. The vision: we work from our own templates, so AI already knows what good looks like. It runs the first review, benchmarks the contract against our standards, and surfaces only the deviations worth our attention. Not every deviation is a problem. We focus on the relevant ones.
• Legal research: Finding the relevant law (the correct statute, regulation, or court decision) for a specific situation is the real challenge, not just finding the general law itself. Our AI tools are trained on comprehensive legal datasets to quickly filter out irrelevancies, ensuring we arrive at the precise answer faster and are supported by verifiable sources.
• Knowledge management: We are building a system where years of institutional know-how live in searchable, AI-powered playbooks, giving  teams across Harness answers to common legal questions instantly, without filing a request or waiting for a callback.
• Data discovery and privacy compliance: Before you can be compliant, you need to know what you are actually dealing with. We use AI to discover and map all data currently being processed across the business: what it is, where it lives, and how it flows. That visibility makes it possible to track, manage, and meet our legal obligations, including privacy requirements for personal data.
• M&A project management: An AI-assisted platform provides a secure, structured home for our M&A work. When the pace picks up and the details multiply, nothing falls through the cracks.
• Regulatory compliance intake: Every new product goes through a single AI-assisted form before launch. Privacy, AI regulation, cybersecurity, accessibility: one entry point, no gaps.
• Building AI agents: This is the part most people do not expect. Our lawyers are not just using AI. They are learning to build with it, developing agents that handle specific legal tasks end-to-end. We are not waiting for the legal tech market to catch up with what we need. We are building it ourselves.

What it actually changes

The honest answer: the relationship between Legal and the rest of the business.

Turnarounds that used to take days take hours. Quality has gone up, not down. And because teams can self-serve answers to routine questions through our Legal Playbooks, the requests that do reach us are the ones that genuinely need us. We spend less time being a checkpoint and more time being a partner. That is a different job, and a more meaningful one.

We take the guardrails seriously

Moving fast with AI does not mean being reckless about it:

• Every output gets reviewed by a qualified attorney before it goes anywhere. 
• We only use approved, enterprise-grade tools. 
• Sensitive data never touches a public AI model. 
• And we treat every AI-generated output as a starting point, not an answer, checking it against real sources before we rely on it.

What makes this more than a policy is the culture around it. We run regular sessions where the team shares what they are learning: tools worth trying, prompting approaches that actually work for legal drafting, and ways to get more out of what we already use. When one person figures something out, everyone benefits. That collective curiosity is what stops this from becoming shelfware and keeps it genuinely evolving.

If this is what Legal looks like at Harness, imagine the rest.

Every team here operates this way. Not because they are told to, but because it is genuinely a better way to work. If you are looking for a company where AI is woven into how things actually get done, not just what gets announced, we are hiring!

Explore open roles at Harness →

• Setting up Terraform securely and consistently is essential for scaling your infrastructure and staying compliant.
• To achieve strong security, keep Terraform runs isolated, encrypt your state files and plans, and use lock files and trusted provider sources to ensure integrity.
• By automating Terraform setup and management in your CI/CD and GitOps pipelines, you keep versions consistent, speed up delivery, and stay compliant across many services.
• For enterprise teams, managing Terraform through an Infrastructure as Code Management tool, rather than standalone execution, ensures consistency, governance, and scalability.

If your Terraform install is insecure or inconsistent, it can quickly slow down your delivery. A single compromised file or a misconfigured backend can stop deployments for many services. Teams that set up Terraform correctly from the start can scale easily and avoid compliance issues.

The answer is to install Terraform with strong security measures right from the beginning. Use verified binaries, encrypt your state, and set up automated CI/CD integration from day one. This method includes OS-specific setup, security checklists, GitOps alignment, and governance that can grow with your company. Want to speed up secure infrastructure automation? Harness Infrastructure as Code Management offers AI-powered pipelines with built-in governance for enterprises.

Install Terraform Correctly on Every Platform: OS Prerequisites, Packages, and Verification

One misconfigured Terraform install can cause hours of pipeline failures across many services. When setting up Terraform on development machines, build agents, and production, focus on consistency and security for reliable automation. Start with verified binaries, pinned versions, and automated checks to keep your infrastructure stable.

Use Vendor-Backed Packages to Eliminate Supply Chain Risk

Always get Terraform from HashiCorp’s official repositories, not from third-party mirrors or unofficial packages. For macOS, use the official Homebrew tap (brew tap hashicorp/tap && brew install hashicorp/tap/terraform).

On Linux, add HashiCorp’s GPG-signed package repository instead of using versions from your distribution, which may be outdated. Windows users should download signed binaries directly from releases.hashicorp.com. This helps keep your infrastructure safe from compromised or outdated packages.

Pin Exact Versions and Verify SHA256 Checksums

To make builds reproducible, control the exact Terraform version in every environment. Download the specific version you need, such as from https://releases.hashicorp.com/terraform/1.6.0/terraform_1.6.0_linux_amd64.zip, and check the SHA256 checksum against HashiCorp’s signed SHASUMS file before extracting.

Keep your version-pinned install scripts in your infrastructure repository so teams can create identical environments. If you use Terraform with Harness, delegates manage versions for you, but local development still needs consistent versioning.

Automate Post-Install Verification to Prevent Configuration Drift

After installing Terraform, run terraform version to make sure the right version is active and in your PATH. Set up the plugin cache directory (TF_PLUGIN_CACHE_DIR) to avoid repeated provider downloads and check that you have write permissions.

Write a simple script to check the Terraform binary location, version, and basic provider setup. Run this script automatically in your CI/CD pipelines, container builds, and onboarding workflows to catch problems before they affect deployments. While local installation is useful for development, enterprise teams should standardize Terraform execution through an IaCM platform. This ensures consistent environments across developers, CI/CD pipelines, and production systems without relying on manual setup

Enterprise Security Hardening for Terraform Installation

Installing Terraform is only the beginning. In enterprise settings where you manage important infrastructure and need to meet regulations, hardening your Terraform setup turns a basic install into a system ready for production and governance. These controls are significantly easier to enforce when Terraform is managed through an IaCM platform that centralizes execution, credentials, and policy enforcement.

Credential Management and Execution Isolation:

• Run Terraform on dedicated build agents or through an IaCM platform that provides isolated, governed execution environments. Avoid putting long-term cloud credentials in developer profiles or local setups.
• Encrypt your Terraform plans by storing them in secrets managers instead of plain text files. This keeps sensitive infrastructure changes protected during the approval process.

Provider Security and Integrity:

• Use dependency lock files and private provider mirrors to control provider integrity. These steps enforce SHA256 checksum checks and stop unauthorized plugin downloads from public registries.
• Set up curated provider repositories with the Terraform providers mirror command. This creates trusted, version-controlled sources for your team, so you don’t have to rely on external registries in production.

State Management and Backend Security:

• Store your remote state in encrypted backends like S3 with KMS encryption. Set up bucket policies to limit access and turn on audit logging for all state changes.
• Set secure CLI defaults in your Terraform config files. This limits provider downloads to approved sources and blocks access to untrusted registries during runtime.

Automate Terraform Installation in CI/CD and GitOps Pipelines

Make your Terraform CI/CD setup consistent by including the binary in versioned container images or reusable templates that all services use. This prevents differences between developer machines, build agents, and production. This approach can become even more scalable when implemented through an IaCM tool integrated with your CI/CD platform where Terraform execution, policy checks, and governance are built into reusable workspaces and modules.

When updating Terraform versions or security patches, make changes in your template library instead of updating each pipeline one by one.  We recommend this version-controlled method for enterprise customers.

Use Policy as Code checks to enforce governance by validating Terraform versions, approved modules, and provider rules before running any plans. OPA can review Terraform plans in your CI/CD pipeline, automatically approving safe changes and flagging risky ones for manual review.

Pair this with GitOps workflows, where pull requests start plans and approved merges triggers applies. This creates clear audit trails for compliance and keeps developers moving quickly. Instead of treating Terraform as a standalone CLI step, IaC tools allow you to manage infrastructure workflows as first-class citizens within your delivery pipelines.

Terraform Install FAQs: Security, Compliance, and Scale

DevOps teams running hundreds of services need Terraform installation methods that scale and stay secure and compliant. Here are practical answers to common questions from teams in regulated settings.

How do you install Terraform securely in an enterprise environment?

Start with package repositories that include GPG verification rather than direct binary downloads to prevent compromised or malicious software packages. Install from official HashiCorp repositories with signed packages, verify SHA256 checksums, and run Terraform from isolated build environments with limited-access credentials that only provide necessary permissions. Keep your state files in encrypted, secure storage with access controls and comprehensive audit logging.

What are the best practices for integrating Terraform installation with CI/CD pipelines?

Include Terraform in your container images with specific versions, or use custom binaries to keep all pipeline runs consistent. Pin exact builds in your pipeline templates and use policy-as-code to allow only approved releases before running plans. This keeps development and production in sync and maintains clear compliance records.

How can you automate Terraform installation for large-scale DevOps teams while meeting compliance requirements?

Make reusable install scripts that check checksums and pin builds, then share them through central config management or container registries. Use remote execution on dedicated infrastructure for security and audit trails. Apply OPA policies to control which Terraform releases and providers your teams can use.

Should you use local or remote Terraform execution for enterprise deployments?

Running Terraform remotely on dedicated infrastructure gives you better security and audit trails. Running it locally on developer machines can cause compliance and credential issues. Use isolated build environments or cloud-managed services that run Terraform plans with proper authentication and detailed logs for production. Even better, IaC platforms standardize this by enforcing remote execution with built-in security, auditability, and role-based access controls.

How do you maintain Terraform version consistency across multiple teams and services?

Set up golden path templates with pinned Terraform installs that update all services automatically. Distribute approved releases using container images or package managers, or use platforms that handle governance for you. IaC platforms automate this by centrally managing Terraform versions and enforcing them across all pipelines and environments.

From Installation to Enterprise Scale: Govern, Automate, and Evolve

Standardizing how you install Terraform sets the stage for everything else. Pinning versions, using verified binaries, and securing remote state help your teams work quickly and stay compliant. These best practices are the base for templates that scale to hundreds of services.

Once you have this foundation, the real benefits come when your install standards connect to automated pipelines and GitOps workflows. Using centralized templates and modules for Terraform means security updates are spread automatically, and developers keep their flexibility. Policy-as-code makes sure every deployment meets enterprise needs without slowing things down. At this stage, adopting an IaC Platform approach becomes the recommended path. By managing Terraform through platforms like Harness, teams can standardize execution, enforce governance, and scale infrastructure delivery without increasing operational overhead.”

Are you ready to move from manual installs to enterprise-level automation and governance? Harness Infrastructure as Code Management offers AI-powered templates, a central control plane, and automated checks to make your Terraform setup a real advantage.

• Integrating A/B testing and feature flags directly into CI/CD pipelines empowers developers with self-service experimentation, while maintaining enterprise governance and security.
• Standardizing experimentation workflows—including flag management, guardrail metrics, and automated verification—reduces operational bottlenecks and technical debt across large engineering teams.
• AI-powered automation enable platform teams to scale safe experimentation. Intelligent tooling provides portfolio-level visibility and ROI measurement without sacrificing control or compliance.

With the acceleration of AI-assisted coding, spurring the velocity of software releases, the challenge of ensuring stable deployments is heightened, and platform teams are feeling the hit.  The State of AI-assisted Software Development DORA report measured a negative impact on software delivery stability: “an estimated 7.2% reduction for every 25% increase in AI adoption.” 

The DORA report advises:

Considered together, our data suggest that improving the development process does not automatically improve software delivery—at least not without proper adherence to the basics of successful software delivery, like small batch sizes and robust testing mechanisms.

A robust testing mechanism rapidly gaining momentum is testing in production. Let’s take a closer look at how this practice boosts software delivery stability and supports the software development lifecycle (SDLC). We’ll also consider how to make testing in production, specifically A/B testing at scale, work for you.

What is testing in production?

Testing in production (TIP) means testing new software code on live traffic in active real-world environments. TIP is complementary to pre-production testing and does not replace it. It does, however, carry tangible benefits: 

• Real-World Validation. Tests new features in the actual production environment, measuring performance under real-world conditions and detecting issues that staging (due to lack of data variety) cannot catch.

• Speed & Efficiency. Eliminate the need to create and maintain expensive, multi-layered staging environments for heavy testing.

• Improved User Experience. Allows teams to quickly iterate on features where they create confirmed value, by means of tight feedback loops on real user data.

• Early Issue Detection: Equips your team to spot issues early on, by testing features in production in small amounts or with limited visibility. You can resolve the errors before they escalate or affect more customers.

Feature flags are instrumental in the practice of safe testing in production because they decouple deployment and exposure at the most granular level. By means of feature flags, you implement incremental feature release techniques and unlock progressive experimentation. With carefully crafted A/B testing, you empower rapid feedback loops that confirm real feature value, validate high quality software, and increase team productivity and satisfaction. 

These testing and verification capabilities are crucial as never before in this “AI moment” where AI-assisted coding enjoys wide adoption and funding.

How A/B Testing Works

A/B testing is the process of simultaneously testing two different versions of a web page or product feature in order to optimize a behavioral or performance metric, while ensuring guardrail metrics are not negatively impacted. A/B testing spans the whole spectrum of software verification: you can safely carry out architectural validation on fundamental architectural changes or gather behavioral analytics on UI variations.

Progressive experimentation with feature flags lets you roll out changes to a small slice of users first, catch problems early, and expand only when the data looks good.

The key is keeping deployment and release separate. You decouple deployment and release by delivering new features in a dormant state. Code goes out behind a flag. You validate it with real traffic.

Why A/B Testing Belongs in Your CI/CD Pipeline

A/B testing built into your CI/CD pipeline means you're making data-driven decisions based on observed metrics. Advanced feature flagging correlates statistical data, with pinpoint precision, to the actual feature variation causing the impact. Even when multiple features are rolled out concurrently, an enterprise-grade feature management platform will effectively parse the data, alert you to the impactful variant, and enable you to roll back any negative feature in seconds. The time/cost savings and safety benefits are astounding.

A/B testing provides a great experience for both marketing teams and engineers:

• Marketers can enjoy the freedom to boldly conduct business experiments and conclusively determine the features that drive key performance indicators (KPIs) and return on investment (ROI). 

• Engineers can confidently improve architecture and perform code refactoring, knowing these changes will be safely measured against guardrail metrics for real-world engineering verification.

An enterprise-level platform like Harness, provides Feature Management and Experimentation, bringing flags, monitoring, and full experimentation freedom into a finely-tuned, seamless end-to-end software delivery tech stack for your platform team. Integrating A/B testing and feature flags directly into CI/CD pipelines empowers your teams with self-service experimentation while maintaining enterprise governance and security.

Progressive exposure limits the blast radius 

Bundling features into cliff-jump releases put every user account at risk simultaneously. A progressive ramp—starting with just 1 or 2% of traffic, and gradually increasing—means a bug in your checkout flow only affects a fraction of users before you catch it. Progressive delivery validates that SLOs are holding before exposure expands. p95 latency spiking? Error rate creeping up? You catch it when a tiny fraction of users are affected—not thousands—and Harness CD integrates cleanly with Jenkins, GitLab, or GitHub Actions.

The deploy-and-hold pattern is the keystone. Ship code in the "off" state behind a feature flag and nothing changes for users until you're ready. Deploy at 11 AM on a Tuesday instead of 1 AM on a Sunday. No change windows, no dashboard babysitting. Code is in production, the feature is dark, and you flip the switch when you're ready to monitor it. That's the freedom of progressive experimentation with feature flags in practice.

AI-assisted verification handles the noise

Raw telemetry is information in theory and chaos in practice. AI-powered monitoring watches flag-level metrics—not just "something is slower," but "checkout button variant B is adding 43ms of p95 latency." That specificity matters. When you have six active experiments running, your engineers are not flipping through dashboards trying to isolate which one broke something. The system tells you.

From Feature Flags to Experiments: Architecture That Works Seamlessly

If your team is already running feature flags with health monitoring, you're closer to a full experimentation platform than you might think. Targeting logic, rollout percentages, kill switches—that's already experiment infrastructure. What's missing is experiment tracking, statistical analysis, and deterministic assignment.

To implement experiments with your feature flagging:

• Build on existing flag infrastructure. Targeting, rollout percentages, and kill switches are already there. Add experiment tracking on top rather than running parallel systems. Harness Feature Management handles both in one place.
• Use stable user IDs for deterministic assignment. Consistent hashing keeps users in the same variant across sessions and devices. No drift, no contaminated data.
• Evaluate flags on the SDK side. Toggle decisions should be fast and deterministic. Local evaluation avoids remote call latency and keeps user data secure.
• Route alerts to features, not just systems. "Checkout variant B caused a +43ms p95 increase" is actionable; "Latency is up" isn't. Release monitoring with flag context makes rollback decisions take on surgical precision, and proper experimentation systems prevent sample ratio mismatch and bot traffic from skewing results.

An experimentation system built on top of your feature flagging makes A/B testing a cinch and eliminates operational bottlenecks and technical debt for your platform team.

CI/CD Workflow: Six Stages for Safe Experimentation

A/B testing doesn't have to be complicated. It can run as part of a structured rollout with automated KPI metrics and guardrails:

The seven stages are built into your pipeline and completed with minimal human intervention:

1. Build and test. Catch issues before they ship. Pre-production unit, regression, smoke, and integration testing is incorporated into code builds and repository merges.
2. Deploy, but keep new features inactive. Launch new features but keep them dark behind conditional statements that evaluate a feature flag. New features are in production but hidden.
3. Canary at 1%. Enable the flag for a small slice of real traffic. This achieves engineering verification of new code under production traffic and processing loads. At this point, the blast radius of these issues is minimal and they are quickly isolated and resolved.
4. Check guardrails. Automated monitoring reports error rate, latency, performance, and business KPIs. A powerful statistical analysis engine measures metrics for feature flag variants against the baseline, detects statistical significance, and alerts on positive or negative impact at the feature level.
5. Auto-ramp. Expand the audience for each feature by progressively increasing real-user exposure as you continue to monitor guardrails, for example 1% → 5% → 25% → 50% → 75% → 100%. A deterministic feature flag assignment algorithm controls the gradual increase in audience size and ensures that users aren’t repeatedly flipped between feature variants.
6. A/B Analysis. Automatically calculate metrics and generate data analysis charts for feature variant comparison. Executive dashboards show experiment velocity, win rates, and KPI lift to demonstrate ROI and guide strategic decisions.
7. Auto-rollback or promote. Thresholds crossed? The system reverts a feature without waiting for a human to notice, while you sleep.

Sample sizes matter more than most teams realize

A common mistake is ramping too fast and drawing conclusions from thin data. If your sample size is too low, your experiment will be underpowered, and you will be unlikely to detect a reasonably-sized impact. Calculate that you have a large enough sample to be able to detect impacts of the size that are important to you. 

Progressive experimentation requires patience. Premature conclusions produce unreliable results, and unreliable results produce bad decisions.

Governance isn't overhead, it's insurance 

Every experiment should have a documented hypothesis, defined success metrics, blast radius assessment, and rollback plan before it touches production. Feature flag lifecycle management also keeps technical debt from quietly accumulating—flags that never get retired are toggle debt and a production surprise waiting to happen.

Turn Every Release Into a Measured Experiment

The goal isn't just fewer 3 a.m. incidents, though that's a welcome side effect. The real win is replacing gut feel with data at every stage of delivery.

With modern testing in production: feature flags decouple deploy from release, progressive ramps limit blast radius, AI-powered guardrails catch regressions before they spread, and centralized analytics replace the multi-tool sprawl that makes experimentation feel expensive.

Every time you release a feature you can ramp gradually up to 100% using percentage-based rollouts, alert on specific pre-decided latency increases, and enforce minimum sample sizes before promotion. Let every release become a decision backed by actual evidence, not optimism.

Harness Feature Management & Experimentation consolidates flags, release monitoring, and A/B testing, so every deployment is a controlled experiment—not a gamble.

Safe A/B Testing in Production: Frequently Asked Questions

How do you pick guardrail metrics without blocking every release?

Start with your existing SLO metrics and be conservative. Grafana's SLO guidance recommends event-based SLIs over percentiles for cleaner signals. Focus on business-critical user journeys first.

What's a practical ramp schedule for a mid-sized SaaS team?

Every team has slightly different criteria to consider before safely ramping up. Release monitoring with automated guardrails removes the need for someone to manually review metrics at each stage—which is the only way this actually scales.

How do you handle sample ratio mismatch?

Monitor assignment ratios continuously using chi-squared tests. Harness FME’s attribution and exclusion algorithm is honed to ensure accurate samples. In addition, FME reassesses experiment health in real-time, including sample ratio.

Filter bot traffic early too. Microsoft's bot detection research shows bots can skew conversion rates by 15–30%. Behavioral signals like sub-10-second session duration or unusual referrer patterns are a practical starting point for exclusion algorithms.

Should you A/B test infrastructure changes or just product features?

A/B testing works best for user-facing changes where behavior matters. Infrastructure changes are better suited to progressive rollouts with guardrail monitoring—different changes, different success metrics. Performance and reliability for engineering experiments; conversion and engagement for growth. Keep the tooling integrated in your pipeline either way.

How do you maintain consistent user experiences across devices and services?

Deterministic hashing on stable user IDs. Hash user ID plus experiment name to generate consistent assignments and make sure the same user sees the same variant whether they're on mobile, desktop, or clearing cookies every 20 minutes. Avoid session-based bucketing—it creates flickering experiences, causes re-bucketing, and erodes trust in experiment data. Lean on SDK-side evaluation for consistency that holds across your entire stack.

At 2 am, your migration goes live. By 2:07, error rates spike, and rollback isn’t an option. Cloud migrations, API rewrites, and architecture transformations rarely fail because of bad code. They fail because of how that code is released.

Most teams still rely on a “big bang” cutover where infrastructure, services, and user-facing changes go live at once. This concentrates risk into a single moment. When something breaks, rollback is slow, visibility is limited, and the blast radius is large.

This is not just anecdotal. According to BCG, more than half of transformation efforts fail to achieve their intended outcomes within three years.

The difference between success and failure is not the migration itself. It is the release strategy.

Cloud Migration Is Not a Single Change

“Cloud migration” sounds simple, but in practice, it is a layered transformation.

Most migrations combine several of the following:

• Monolith to microservices
• API or data pipeline rewrites
• Frontend or UI rebuilds
• On-prem to cloud infrastructure moves
• Introduction of new service layers

These rarely happen in isolation. Teams often try to ship them together in a single coordinated release. That coupling increases complexity and multiplies risk.

Before your next migration, list every system involved. If they are all released together, you are carrying unnecessary risk.

The Core Anti-Pattern: Big Bang Releases

The failure mode is consistent:

• A new service is deployed
• Infrastructure flips to the cloud
• A redesigned UI is released
• All at once

There is no safe way to validate behavior in production. There is no gradual exposure. Rollback often requires redeploying an old stack that may no longer be compatible.

Even worse, teams lack a reliable baseline. They cannot answer simple questions:

• Is performance better?
• Is the cost lower?
• Is reliability improved?

Without that, migration becomes guesswork.

Decoupling Deployment from Release

Modern teams are adopting a different model:

• Deploy code anytime. Release it gradually.

Feature flags provide a control layer that separates deployment from exposure. Code can exist in production without being active for all users.

This enables:

• Controlled rollout by percentage, region, or cohort
• Instant rollback without redeployment
• Real-time measurement tied to specific changes

Start by putting one service behind a feature flag and releasing it to internal users first.

Progressive Delivery for Migrations

Replace Cutovers with Progressive Rollouts

Instead of switching everything at once:

1. Deploy the new system alongside the old one
2. Route a small percentage of traffic
3. Observe behavior
4. Increase exposure gradually

If something fails, you reduce traffic or revert instantly.

This shifts migration from a single high-risk event to a series of measurable steps.

The Strangler Fig Pattern in Practice

A common migration strategy is the strangler fig pattern.

• Build new functionality alongside the legacy system
• Gradually route traffic to the new components
• Retire legacy code over time

Feature flags make this executable in production by controlling routing and exposure. But to make this work in practice, you need a control layer that can manage traffic in real time.

How Progressive Migration Actually Works

Below is a simplified view of how feature flags act as a control plane during migration:

Fig: Feature-flag–driven progressive traffic routing during migration

Two things matter here:

1. Routing control: traffic can be shifted gradually
2. Measurement: metrics are tied to each variation

This is not just a toggle. It is a runtime decision and an observability layer.

Measure What Actually Matters

A successful migration is not defined by deployment success. It is defined by outcomes.

Key metrics include:

• Latency and throughput
• Error rates
• Infrastructure cost
• Output correctness

These metrics are not theoretical. They are what teams use to validate migrations in real production environments.

Real Example: Dual Pipeline Migration

In the Beyond the Toggle ebook, a legacy Spark batch pipeline was replaced with a streaming architecture, with a progressive rollout rather than a cutover.

• Both pipelines ran in production
• A feature flag routed traffic between them
• Metrics were compared in real time

The new system showed faster processing and lower costs before the full rollout.

From the webinar, teams often go further:

• Run both systems simultaneously
• Compare outputs for correctness
• Measure performance differences per request

This allows validation of both performance and data integrity before committing.

Define your baseline metrics before migration. If you cannot measure improvement, you cannot prove success.

Staging Lies. Production Doesn’t.

Staging environments cannot replicate production conditions. They lack:

• Real traffic patterns
• Data scale
• Edge cases

Feature flags enable safe production testing through controlled exposure.

Common Patterns

• Canary releases by percentage or region
• Cohort-based rollouts (geo, customer segment)
• Dual execution for validation

Not all canary releases are percentage-based. Some teams roll out by country or user segment first, then expand globally.

Guardrails

To make this safe:

• Automated rollback based on thresholds
• Feature-level observability
• Access control and audit logs

Decision Making: Continuous Go / No-Go

A migration is a sequence of decisions, not a single moment.

At each stage:

1. Define the metric
2. Measure impact
3. Decide to expand or roll back

In one example from the webinar:

• A rollout reached 30% traffic
• Error rates increased
• Traffic was reduced to 20%
• The issue was isolated and fixed
• Rollout continued safely

This approach removes pressure from a single “launch moment” and distributes risk across stages.

Advanced Considerations for Developers

Feature Flag Performance and Reliability

Modern flag systems avoid becoming a bottleneck:

• Evaluations happen locally via SDKs
• Configurations are cached
• Systems continue operating even if the flag service is unavailable

This ensures minimal latency and high reliability.

Handling Complex Systems

Not all migrations are equal.

• Data pipelines and database paths require more planning
• Read and write paths may need staged transitions
• Flags still apply, but design complexity increases

The key is incremental transition, not avoidance.

Managing Flag Lifecycle and Tech Debt

Feature flags are temporary by design.

If left unmanaged, they accumulate and create complexity. Teams need:

• Visibility into flag state and usage
• Defined lifecycle policies
• Cleanup after full rollout

Emerging approaches include automation that detects stale flags and generates pull requests to remove them.

This Is a Delivery Strategy Change

Adopting progressive delivery is not just a tooling decision. It changes how teams release software.

Key considerations:

• Align with existing change management processes
• Integrate flags into CI/CD pipelines
• Maintain governance and auditability

Feature flags do not bypass controls. They enhance them by adding visibility and control at runtime.

What to Look for in a Feature Flag Platform

For migration use cases, a Feature Flag platform should provide:

• Tight integration with CI/CD pipelines
• Built-in experimentation and metrics
• Governance, approvals, and audit logs
• Developer-friendly SDKs and workflows

Flags should not feel like a bolt-on. They should be part of how software is built and released.

Conclusion: From Risk Event to Controlled Process

The biggest mistake teams make is treating migration as a moment.

It is not.

It is a controlled progression of changes, each validated in production under real conditions.

Feature flags enable this by:

• Decoupling deployment from release
• Enabling gradual exposure
• Providing real-time measurement
• Allowing instant rollback

The result is simple:

Migrations become reversible, observable, and data-driven.

Want a deeper breakdown of these patterns and real-world examples? Read the full ebook or see a demo.