DevOps & Automation | Harness Blog Primary Category

Technical

The pipeline that never reached production

How to improve CI/CD governance using template-driven pipelines, Git controls, and policy enforcement to protect production.

Anmol Pandey

Stefano Mazzone

April 2, 2026

Time to read

How template-driven CD prevents governance drift

Modern CI/CD platforms allow engineering teams to ship software faster than ever before.

Pipelines complete in minutes. Deployments that once required carefully coordinated release windows now happen dozens of times per day. Platform engineering teams have succeeded in giving developers unprecedented autonomy, enabling them to build, test, and deploy their services with remarkable speed.

Yet in highly regulated environments-especially in the financial services sector-speed alone cannot be the objective.

Control matters. Consistency matters. And perhaps most importantly, auditability matters.

In these environments, the real measure of a successful delivery platform is not only how quickly code moves through a pipeline. It is also how reliably the platform ensures that production changes are controlled, traceable, and compliant with governance standards.

Sometimes the most successful deployment pipeline is the one that never reaches production.

This is the story of how one enterprise platform team redesigned their delivery architecture to ensure that production pipelines remained governed, auditable, and secure by design.

The subtle risk in fast CI/CD platforms

A large financial institution had successfully adopted Harness for CI and CD across multiple engineering teams.

From a delivery perspective, the transformation looked extremely successful. Developers were productive, teams could create pipelines quickly, and deployments flowed smoothly through various non-production environments used for integration testing and validation. From the outside, the platform appeared healthy and efficient.

But during a platform architecture review, a deceptively simple question surfaced:

“What prevents someone from modifying a production pipeline directly?”

There had been no incidents. No production outages had been traced back to pipeline misconfiguration. No alarms had been raised by security or audit teams.

However, when the platform engineers examined the system more closely, they realized something concerning.

Production pipelines could still be modified manually.

In practice this meant governance relied largely on process discipline rather than platform enforcement. Engineers were expected to follow the right process, but the platform itself did not technically prevent deviations. In regulated industries, that is a risky place to be.

The architecture shift: separate authoring from execution

The platform team at the financial institution decided to rethink the delivery architecture entirely. Their redesign was guided by a simple but powerful principle:

Pipelines should be authored in a non-prod organization and executed in the production organization. And, if additional segregation was needed due to compliance, the team could decide to split into two separate accounts.

Authoring and experimentation should happen in a safe environment. Execution should occur in a controlled one.

Instead of creating additional tenants or separate accounts, the platform team decided to go with a dedicated non-prod organization within the same Harness account. This organization effectively acted as a staging environment for pipeline design and validation.

Architecture diagram

This separation introduced a clear lifecycle for pipeline evolution.

The non-prod organization became the staging environment where pipeline templates could be developed, tested, and refined. Engineers could experiment safely without impacting production governance.

The production organization, by contrast, became an execution environment. Pipelines there were not designed or modified freely. They were consumed from approved templates.

Guardrail #1: production pipelines must use templates

The first guardrail introduced by the platform team was straightforward but powerful.

Production pipelines must always be created from account-level templates.

Handcrafted pipelines were no longer allowed. Project-level template shortcuts were also prohibited, ensuring that governance could not be bypassed unintentionally.

This rule was enforced directly through OPA policies in Harness.

Example policy

package harness.cicd.pipeline

deny[msg] {
  template_scope := input.pipeline.template.scope
  template_scope != "account"
  msg = "pipeline can only be created from account level pipeline template"
}

This policy ensured that production pipelines were standardized by design. Engineers could not create or modify arbitrary pipelines inside the production organization. Instead, they were required to build pipelines by selecting from approved templates that had been validated by the platform team.

As a result, production pipelines ceased to be ad-hoc configurations. They became governed platform artifacts.

Guardrail #2: governance starts in the non-prod organization

Blocking unsafe pipelines in production was only part of the solution.

The platform team realized it would be even more effective to prevent non-compliant pipelines earlier in the lifecycle.

To accomplish this, they implemented structural guardrails within the non-prod organization used for pipeline staging. Templates could not even be saved unless they satisfied specific structural requirements defined by policy.

For example, templates were required to include mandatory stages, compliance checkpoints, and evidence collection steps necessary for audit traceability.

Example policy

package harness.ci_cd

deny[msg] {
  input.templates[_].stages == null
  msg = "Template must have necessary stages defined"
}

deny[msg] {
  some i
  stages := input.templates[i].stages
  stages == [Evidence_Collection]
  msg = "Template must have necessary stages defined"
}

These guardrails ensured that every template contained required compliance stages such as Evidence Collection, making it impossible for teams to bypass mandatory governance steps during pipeline design.

Governance, in other words, became embedded directly into the pipeline architecture itself.

The source of truth: Git

The next question the platform team addressed was where the canonical version of pipeline templates should reside.

The answer was clear: Git must become the source of truth.

Every template intended for production usage lived inside a repository where the main branch represented the official release line.

Direct pushes to the main branch were blocked. All changes required pull requests, and pull requests themselves were subject to approval workflows that mirrored enterprise change management practices.

Governance flow

This model introduced peer review, immutable change history, and a clear traceability chain connecting pipeline changes to formal change management records.

For auditors and platform leaders alike, this was a significant improvement.

The promotion workflow

Once governance mechanisms were in place, the promotion workflow itself became predictable and repeatable.

Engineers first authored and validated templates within the non-prod organization used for pipeline staging. There they could test pipelines using real deployments in controlled non-production environments.

The typical delivery flow followed a familiar sequence:

After validation, the template definition was committed to Git through a branch and promoted through a pull request. Required approvals ensured that platform engineers, security teams, and change management authorities could review the change before it reached the release line.

Once merged into main, the approved template became available for pipelines running in the production organization. Platform administrators ensured that naming conventions and version identifiers remained consistent so that teams consuming the template could easily track its evolution.

Finally, product teams created their production pipelines simply by selecting the approved template. Any attempt to bypass the template mechanism was automatically rejected by policy enforcement

The day the model proved its value

Several months after the new architecture had been implemented, an engineer attempted to modify a deployment pipeline directly inside the production organization.

Under the previous architecture, that change would have succeeded immediately.

But now the platform rejected it. The pipeline violated the OPA rule because it was not created from an approved account-level template.

Instead of modifying the pipeline directly, the engineer followed the intended process: updating the template within the non-prod organization, submitting a pull request, obtaining the necessary approvals, merging the change to Git main, and then consuming the updated template in production.

The system had behaved exactly as intended. It prevented uncontrolled change in production.

Why this model works

The architecture introduced by the large financial institution delivered several key guarantees.

Production pipelines are standardized because they originate only from platform-approved templates. Governance is preserved because Git main serves as the official release line for pipeline definitions. Auditability improves dramatically because every pipeline change can be traced back to a pull request and associated change management approval. Finally, platform administrators retain the ability to control how templates evolve and how they are consumed in production environments.

The lesson for platform teams

Pipelines are often treated as simple automation scripts.

In reality they represent critical production infrastructure.

They define how code moves through the delivery system, how security scans are executed, how compliance evidence is collected, and ultimately how deployments reach production environments. If pipeline creation is uncontrolled, the entire delivery system becomes fragile.

The financial institution solved this problem with a remarkably simple model. Pipelines are built in the non-prod staging organization. Templates are promoted through Git governance workflows. Production pipelines consume those approved templates.

Nothing more. Nothing less.

Final takeaway

Modern CI/CD platforms have dramatically accelerated the speed of software delivery.

But in regulated environments, the true achievement lies elsewhere. It lies in building a platform where developers move quickly, security remains embedded within the delivery workflow, governance is enforced automatically, and production environments remain protected from uncontrolled change.

That is not just CI/CD. That is platform engineering done right.

Technical

Introducing Zero Trust Architecture for Software Delivery

Move beyond RBAC and gates. Discover how to create a "last line of defense" for your SDLC by validating every automated task at the runner level.

Eric Minick

Pranay Kaikini

April 2, 2026

Time to read

For the world’s largest financial institutions, places like Citi and National Australia Bank, shipping code fast is just part of the job. But at that scale, speed is nothing without a rock-solid security foundation. It’s the non-negotiable starting point for every release.

Most Harness users believe they are fully covered by our fine-grained Role-Based Access Control (RBAC) and Open Policy Agent (OPA). These are critical layers, but they share a common assumption: they trust the user or the process once the initial criteria are met. If you let someone control and execute a shell script, you’ve trusted them to a great extent.

But what happens when the person with the "right" permissions decides to go rogue? Or when a compromised account attempts to inject a malicious script into a trusted pipeline?

Harness is changing the security paradigm by moving beyond Policy as Code to a true Zero Trust model for your delivery infrastructure.

The Challenge: When Permissions Aren't Enough

Traditional security models focus on the "Front Door." Once an employee is authenticated and their role is verified, the system trusts their actions. In a modern CI/CD environment, this means an engineer with "Edit" and "Execute" rights can potentially run arbitrary scripts on your infrastructure.

If that employee goes rogue or their credentials are stolen, RBAC won't stop them. OPA can control whether shell scripts are allowed at all, but it often struggles to parse the intent of a custom shell script in real-time.

The reality is that verify-at-the-door is a legacy mindset. We need to verify at execution time. CI/CD platforms are a supply-chain target that are often targeted. The recent attack against the Checkmarx GitHub Action has been a painful reminder of the lesson the Solarwinds fiasco should have taught the industry.

Introducing Harness Zero Trust

Harness Zero Trust is a new architectural layer that acts as a mandatory "interruption" service at the most critical point: the Harness Delegate (our lightweight runner in your infrastructure).

Instead of the Delegate simply executing tasks authorized by the control plane, it now operates on a "Never Trust, Always Verify" basis.

How It Works: The Final Line of Defense

When Zero Trust is enabled, the Harness Delegate pauses before executing any task. It sends the full execution context to a Zero Trust Validator, a service hosted and controlled by your security team.

This context includes:

User Identity: Who triggered the action?
Task Specifics: Exactly what is the Delegate being asked to do?
Script Content: The full body of any shell scripts.
Environment Variables: The inputs and secrets being injected into the task.

The Delegate waits a moment. Only if the validator returns a "True" signal does the task proceed. If the signal is "False," the execution is killed instantly.

Why This Matters for Enterprise DevSecOps

By moving validation to the Delegate level, we provide a "Last Line of Defense" that hits several key enterprise requirements:

Rogue Employee Protection: Even if a user has the rights to run a pipeline, your security service can flag suspicious patterns (like a script attempting to delete a production database or exfiltrate data) and stop it before it starts.
Architectural Superiority: While competitors struggle with stability and baseline security, Harness is doubling down on a hardened architecture. RBAC protects the door; OPA governs the "what"; and Zero Trust validates the "how."
Custom Judgement: Because Harness sends the complete task details, you can point this validator at a customer algorithm or an AI-powered security tool to judge the "safety" of scripts in real-time. Essentially, you are peer-reviewing every line of automation at execution time.

The Takeaway

We built this capability alongside some of the world's most regulated institutions to ensure it doesn't become a bottleneck. It’s designed to be a silent guardian. It shuts down the 1% of rogue actions while the other 99% of your engineers continue to innovate at high velocity.

The bottom line: at Harness, we believe that the promise of AI-accelerated coding must be met with an equally advanced delivery safety net. We’re building out that safety net every day. Zero Trust is the next piece.

Technical

Defeating Context Rot: Mastering the Flow of AI Sessions

AI agents fail due to context rot—growing context degrades accuracy. Use plan, execute, reset, and checkpoints to keep outputs consistent and reliable.

Dewan Ahmed

Shreyas Nagaraj

April 1, 2026

Time to read

In Part 1, we argued that most dev teams start in the wrong place. They obsess over prompts, when the real problem is structural: agents are dropped into repositories that were never designed for them. The solution was to make the repository itself agent-native through a standardized instruction layer like AGENTS.md.

But even after you fix the environment, something still breaks.

The agent starts strong. It understands the problem, follows instructions, seems super intelligent

Then, somewhere along the way, things begin to drift. The code still compiles, but the logic gets inconsistent. Small mistakes creep in. Constraints are ignored. Assumptions mutate.

Nothing fails loudly. Everything just gets slightly worse.

This is the second failure mode of AI systems: context rot.

Context Rot Is Not a Bug — It’s a Property

There is a persistent assumption in the industry that more context leads to better performance. If a model can handle large context windows, then giving it more information should improve accuracy.

In practice, the opposite is often true.

Recent research from Chroma shows that LLM performance degrades as input length increases, even when the model is operating well within its maximum context window. Similar observations are echoed across independent analyses, including this breakdown of why models deteriorate in longer sessions and practical explorations of how context mismanagement impacts production systems.

This is not an edge case. It is a structural limitation.

Models do not “understand” context in a hierarchical way. They distribute attention across tokens. As context grows, signal competes with noise. Important instructions lose weight. Irrelevant details gain influence. Conflicts accumulate.

What looks like a reasoning failure is often just context degradation.

Why Long Sessions Break Down

If you’ve worked with AI coding agents for more than a few hours, you’ve already seen this pattern.

A session starts with clear instructions and aligned reasoning. Over time, it fills with partial implementations, outdated assumptions, repeated instructions, and exploratory dead-ends. The model doesn’t forget earlier information, it simply cannot prioritize it effectively anymore.

Detailed guides on context management highlight this exact failure mode: as sessions grow, models become increasingly sensitive to irrelevant or redundant tokens, which degrade output quality. Platform-level documentation also reinforces the same principle - effective systems explicitly control how context is introduced, retained, and pruned.

In practice, this shows up as inconsistency. But underneath, it’s the predictable outcome of unmanaged context growth.

The Link Between Context Rot and Hallucination

This is where teams often misdiagnose the issue.

When agents hallucinate, the instinct is to blame the model. But hallucination is often downstream of context rot.

OpenAI’s work on hallucinations explains that models are optimized to produce plausible outputs even under uncertainty. When context degrades, uncertainty increases. The model fills gaps with statistically likely answers.

So the failure chain looks like this:

Context degradation → ambiguity → confident guessing → hallucination

In other words, hallucination is not always a knowledge problem.
It is often a context management problem.

Sessions Are Not Conversations

Most developers interact with AI through chat, so they treat sessions like conversations.

That mental model breaks at scale.

A long-running AI session is not a conversation. It is a stateful system.

And like any stateful system, it degrades without control.

Letting context accumulate indefinitely is equivalent to running a system without memory management. Eventually, performance collapses—not because the system is incapable, but because it is overloaded.

The Plan → Execute → Reset Discipline

Once you accept that context degrades, the solution becomes straightforward: you don’t try to out-prompt the problem. You control how context evolves.

Across production teams, a consistent pattern emerges:

Plan → Execute → Reset

This is not a trick. It is operational discipline.

Planning before execution

The most common mistake is asking the agent to write code immediately. This forces premature decisions and locks the model into an approach before it has fully understood the problem.

Instead, enforce a planning phase.

Have the model break down the task, identify dependencies, and surface uncertainties before implementation. This aligns closely with best practices in production-grade prompt engineering, where structured reasoning is prioritized over immediate generation.

Planning reduces unnecessary context growth and prevents incorrect assumptions from propagating.

Stepwise execution

Once the plan is validated, execution should be incremental.

Large, monolithic prompts create large, monolithic contexts—and those degrade fastest.

Stepwise execution keeps the working context focused. Each step introduces only the information required for that step. Errors are caught early, before they spread across the system.

This is not about slowing down development. It is about maintaining signal integrity.

Resetting the session

Even with disciplined execution, context will eventually degrade.

The only reliable solution is to reset.

This may feel inefficient, but in practice, it is one of the highest-leverage actions you can take. A fresh session restores clarity, removes noise, and re-establishes correct prioritization of instructions.

Modern context management approaches consistently emphasize this: keep context bounded, and reintroduce only what is necessary for the task at hand.

Meta-Prompting: Forcing the Model to Think First

One of the most effective techniques for preventing context rot is meta-prompting.

Instead of telling the model what to do, you tell it how to approach the task.

You explicitly require it to:

identify assumptions
highlight uncertainties
ask clarifying questions

This interrupts the model’s default behavior of immediate generation.

Why does this work?

Because hallucinations are often driven by premature certainty. Meta-prompting introduces friction at exactly the right point—before incorrect assumptions become embedded in the context.

Checkpoints: Turning Drift into Signal

Context rot is dangerous because it is gradual and often invisible.

Checkpoints make it observable.

At key moments, you force the model to validate its output against:

the original task
repository constraints (AGENTS.md)
architectural expectations

This transforms hidden drift into explicit feedback.

Instead of discovering problems at the end, you correct them continuously.

The Connection to Part 1

Part 1 of the series solved the problem of what the agent sees.

Part 2 addresses what happens over time.

AGENTS.md provides structure.
Session discipline preserves that structure.

Without AGENTS.md, the agent guesses.
Without discipline, the agent drifts.

You need both to achieve reliable outcomes.

Why This Matters Now

As teams move from experimentation to production, sessions become longer and more complex. Agents interact with more systems, touch more code, and accumulate more context.

This is where most failures emerge.

Not because the model is incapable, but because the workflow is uncontrolled.

Context rot is the primary bottleneck in real-world AI engineering today.

Before You Scale, You Stabilize

In Part 3, we turn to a different problem.

So far, we have focused on a single agent operating within a controlled session. That constraint makes it possible to reason about context, to reset it, and to keep it aligned with the task.

But most real systems do not stay within that boundary.

As soon as you introduce multiple agents, external tools, or retrieval systems, the problem changes. Context is no longer contained in a single session. It becomes distributed across components that do not share the same state or assumptions.

At that point, failures become harder to trace. Drift is no longer local. It propagates.

This is where orchestration becomes necessary, but also where it becomes risky.

Part 3 explores how to build these systems in a way that preserves the guarantees established here. We will look at how to introduce MCPs, subagents, and external integrations without losing control over context, consistency, or behavior.

Technical

Operationalizing Production Data Testing with Harness Database DevOps

Automate production data testing with Harness DB DevOps. Validate schema changes, rollback safety, and reduce deployment risk in CI/CD pipelines.

Animesh Pathak

April 1, 2026

Time to read

Testing database changes against production-like data removes risk from your delivery process but to be effective, it must be orchestrated, governed, and automated. Manual scripts and ad-hoc checks lack the repeatability and auditability required for modern delivery practices.

Harness Database DevOps provides a framework to embed production data testing into your CI/CD pipelines, enabling you to manage database schema changes with the same rigor as application code. Harness DB DevOps is designed to bridge development, operations, and database teams by bringing visibility, governance, and standardized execution to database changes.

Instead of treating testing with production data as an afterthought, you can define it as a pipeline stage that executes reliably across environments.

A Pipeline-Driven Reference Workflow

To incorporate production data testing into your delivery process, you define a Harness Database DevOps pipeline with structured, repeatable steps. The result is a governed testing model that captures evidence of correctness before any change ever reaches production.

Stage 1: Environment Provisioning and Data Preparation

In Harness Database DevOps, you begin by configuring the necessary database instances and schemas:

DB Schemas are defined in Git, using Liquibase or Flyway changelogs or script-based SQL files under version control.
DB Instances connect to your target environments with credentials and Delegate access configured.

For production data testing, you provision two isolated instances seeded with a snapshot of production data (secured and masked as needed). These instances are not customer-facing; they serve as ephemeral test targets.

This structure sets up identical baselines for controlled experimentation.

Stage 2: Schema Application Within a Harness Pipeline

Harness Database DevOps lets you define a deployment pipeline that incorporates database and application changes in the same workflow:

Pipelines unify database deployments with application delivery.
You add stages such as “Apply Schema Changes” that reference your DB Schema and DB Instance.

Using Liquibase or Flyway via Harness, the pipeline applies schema changes to Instance A while Instance B remains the baseline.

This step executes the migration in a real, production-scale context, capturing performance, constraint behaviors, and other runtime characteristics.

Stage 3: Automated Rollback and Undo Migration Testing

A powerful capability of Harness Database DevOps is automated rollback testing within the pipeline:

The pipeline can execute undo migrations to revert schema changes.
Pipelines track execution results and rollback outcomes, enabling teams to validate that undo logic works reliably.

Testing rollback paths removes the assumption that reversal will work in production, a key risk often untested in traditional workflows.

Stage 4: Comparison Against Baseline and Validation

After rollback, you compare Instance A (post-rollback) with Instance B (untouched):

At this point, they should be identical in schema and data state.
Tools designed for database comparisons (e.g., DiffKit) can be integrated to perform row-level and schema-level verification, highlighting hidden drifts or silent mutations.

If disparities are detected, the pipeline can fail early, prompting review and remediation before production deployment.

This approach builds evidence rather than assumptions about the quality and safety of database changes.

How This Aligns with Harness Capabilities?

The updated workflow aligns with the documented capabilities of Harness Database DevOps:

Orchestration of database changes as part of CI/CD pipelines with visibility across environments.
Governance and approvals, so schema changes are reviewed and compliant.
Integrated rollback support for automated undo migrations.
Unified interface and audit trails to track which changes ran where and when.

Importantly, the workflow does not assume native data cloning features within Harness itself. Instead, it positions data-centric operations (cloning and validation) as composable steps in a broader automation pipeline.

Strategic Outcomes for Engineering Teams

Embedding production data testing inside Harness Database DevOps pipelines delivers measurable outcomes:

Reduced risk of production incidents by catching issues early.
Repeatable, auditable delivery practices, aligning database changes with application code flows.
Clear rollback evidence, not just theoretical promise.
Improved collaboration between developers and DBAs through pipeline transparency.

This integrated, pipeline-oriented approach elevates database change management into a disciplined engineering practice rather than a set of isolated tasks.

Conclusion

Database changes do not fail because teams lack skill or intent. They fail because uncertainty is tolerated too late in the delivery cycle when production data, scale, and history finally collide with untested assumptions.

Testing with production data, when executed responsibly, shifts database delivery from hope-based validation to evidence-based confidence. It allows teams to validate not just that a migration applies, but that it performs, rolls back cleanly, and leaves no hidden drift behind. That distinction is the difference between routine releases and high-severity incidents.

By operationalizing this workflow through Harness Database DevOps, organizations gain a governed, repeatable way to:

Treat database changes as first-class citizens in CI/CD
Validate forward and rollback paths against real data
Produce auditable proof of correctness before production
Scale database delivery without scaling risk

This is not about adding more processes. It is about removing uncertainty from the most irreversible layer of your system.

Explore a Harness Database DevOps to see how production-grade database testing, rollback validation, and governed pipelines can fit seamlessly into your existing workflows The fastest teams don’t just deploy quickly, they deploy with confidence.

Technical

The Dangerous Myth: “We Have SCA, So We’re Covered”

SCA alone isn’t enough. Secure your supply chain by governing pipelines, artifacts, containers, and AI across build, promotion, and runtime stages.

Bri Strozewski

April 1, 2026

Time to read

SCA provides visibility into open source risk but leaves artifacts, pipelines, containers, and AI components exposed.
CI/CD pipelines are privileged infrastructure and must be treated as zero-trust environments.
Build systems are a critical trust boundary; artifact attestation and registry gating are essential.
Container images, third-party integrations, and AI models are now active supply chain attack vectors.
True supply chain security requires control at ingest, build, promotion, and runtime — not just scanning.

Most organizations begin their software supply chain journey the same way: they implement Software Composition Analysis (SCA) to manage open source risk. They scan for vulnerabilities, generate SBOMs, and remediate CVEs. For many teams, that feels like progress—and it is.

But it is not enough.

As discussed in the webinar conversation with Information Security Media Group, open-source visibility is necessary, but it is not sufficient. Modern applications are no longer just collections of third-party libraries. They are built, packaged, signed, stored, promoted, deployed, and increasingly augmented by AI systems. Each stage introduces new dependencies and new trust boundaries.

Events like Log4j made open source risk impossible to ignore. However, the evolution of the threat landscape has demonstrated that attackers are no longer limited to exploiting known vulnerabilities in libraries. They are targeting the mechanics of delivery itself—ingestion, build pipelines, artifact storage, and CI/CD automation. Organizations that stop at SCA are securing one layer of a much broader system.

Where Modern Supply Chains Actually Get Compromised

Artifact Integrity and Registry Blind Spots

Artifacts are the final outputs of the build process—container images, binaries, scripts, Helm charts, JAR files. They are what actually run in production. Yet many organizations treat artifact management as operational plumbing rather than a security control point.

The webinar highlighted how focusing exclusively on source code can obscure the reality that artifacts may be tampered with after build, altered during promotion, or stored in misconfigured registries. Visibility into open source dependencies does not automatically guarantee artifact integrity. An attacker who compromises a registry or intercepts promotion workflows can distribute malicious artifacts at scale.

The key risk lies in assuming that once an artifact is built, it is trustworthy. Without signing, provenance tracking, and gating at the registry level, artifacts become one of the most exploitable surfaces in the supply chain.

CI/CD Pipelines as Privileged Infrastructure

CI/CD systems hold credentials, secrets, deployment paths, and signing keys. They connect development directly to production. As one of the speakers noted during the discussion, pipelines should be treated as privileged infrastructure and assumed to be potential targets of compromise.

A compromised runner can publish malicious artifacts, exfiltrate secrets, or promote unauthorized builds. This is not theoretical. Attacks involving poisoned GitHub Actions and manipulated build systems demonstrate how easily the pipeline itself can become the distribution mechanism.

Security must therefore extend beyond scanning artifacts to enforcing strict governance within pipelines. This includes least privilege access, ephemeral credentials, audit trails, and Policy as Code enforcement to ensure required security checks cannot be bypassed.

Containers and Upstream Image Poisoning

Container ecosystems introduce additional risk vectors. Malicious images uploaded to public registries, typosquatting packages, and compromised upstream components can all infiltrate environments through seemingly legitimate pulls.

Organizations that implicitly trust external registries transfer vendor risk into their own infrastructure. Without upstream proxy controls, cool-off periods, or quarantine mechanisms, container ingestion becomes another blind spot.

The supply chain does not stop at internal code repositories. It extends to every external source that feeds into the build.

Vendor and Third-Party Integration Risk

Modern delivery pipelines integrate numerous third-party services. Vendors often have privileged access to environments or automated integrations into CI/CD workflows. If a vendor is compromised, that risk propagates downstream.

The webinar discussion emphasized that the supply chain must be viewed as a “trust fabric.” Pipelines, registries, and vendors are all part of that fabric. A weakness in any one node can cascade across the system.

Build Systems Are the New Trust Boundary

Build systems represent one of the most underestimated attack surfaces in modern software delivery. A source tree may pass review and scanning, yet small modifications in the build process can fundamentally alter the resulting artifact.

Examples discussed during the session included pre-install hooks, registry overrides, runtime installers, or seemingly minor shell script changes that introduce malicious behavior before artifacts are signed or scanned. These changes can bypass traditional SCA tools because the underlying source code appears clean.

This is why build integrity must be verifiable. Provenance should be recorded and tied to specific systems and identities. Build steps should be signed and attested. Promotion gates should require verification of those attestations before artifacts move forward.

Trust must be anchored in the build output, not assumed from the source input.

Why Pipelines Must Be Treated as Untrusted

CI/CD pipelines are often viewed as automation tools, but they are in fact highly privileged systems that bridge development and production. They hold secrets, manage deployment logic, and often operate with broad permissions across infrastructure.

The webinar discussion stressed that pipelines must be treated as untrusted environments by default. This does not imply mistrust of developers, but rather recognition that any high-privilege system is an attractive target.

Policy-as-code frameworks, strict RBAC, auditability, and enforcement of mandatory security checks ensure that controls cannot be disabled under pressure to ship. Developers may unintentionally bypass safeguards when under deadlines. Governance mechanisms must therefore be systemic, not optional.

In complex environments where multiple tools—GitHub Actions, Jenkins, Docker, Kubernetes—are integrated together, misconfiguration becomes another source of risk. Each tool has its own security model. Without centralized governance, complexity compounds vulnerability.

AI Is Expanding the Supply Chain Attack Surface

As if artifacts, pipelines, and containers were not enough, AI-native applications are adding an entirely new dimension to supply chain security.

Modern applications increasingly rely on Large Language Models, prompt libraries, embeddings, model weights, and training datasets. These components influence runtime behavior in ways that traditional code does not. Yet they are rarely tracked or governed with the same rigor as open-source dependencies.

The concept of an “AI Bill of Materials” is emerging, but no standardized framework currently exists. Organizations are integrating AI features faster than governance standards can keep pace.

The risks differ from traditional CVEs. Poisoned training data can subtly manipulate model behavior. Backdoored model weights can introduce hidden functionality. Prompt injection attacks can trick systems into exposing sensitive information. Shadow AI systems may be deployed without formal oversight.

Unlike deterministic software, AI systems produce probabilistic outputs. Static security testing does not fully address this unpredictability. Security teams must now consider model provenance, data lineage, vendor trust, and runtime behavior monitoring as part of the supply chain equation.

Even the build-versus-buy decision for LLMs becomes a supply chain governance choice. Building offers control but introduces operational burden and long-term responsibility. Buying accelerates deployment but increases trust dependency on external vendors. In both cases, AI components extend the trust fabric and must be governed accordingly.

A Practical Framework for End-to-End Supply Chain Security

Moving beyond SCA requires structured controls across the full lifecycle of delivery.

Ingest-Time Controls ensure that risky packages and images are prevented from entering developer workflows in the first place through dependency firewalls, upstream proxy governance, and vendor controls.

Build-Time Integrity requires signed environments, provenance attestations, and enforcement of SLSA-style compliance so that artifacts can be cryptographically tied to verified build processes.

Promotion-Time Governance introduces artifact registry gating, quarantine workflows, and policy enforcement to prevent unauthorized or tampered artifacts from advancing to production.

Runtime Verification ensures continuous monitoring of deployment health, secret usage, and, increasingly, AI behavior to detect anomalous activity after release.

This layered approach transforms supply chain security from a reactive scanning function into an operational control system embedded directly into software delivery workflows.

From Visibility to Control

Software supply chain security has evolved.

It is no longer an open-source vulnerability problem alone. It is a trust management challenge spanning artifacts, pipelines, containers, vendors, and AI components.

Organizations that succeed will not stop at generating reports. They will enforce policy at every stage. They will treat CI/CD as privileged infrastructure. They will require attestations before promotion. They will govern ingestion and monitor runtime behavior. They will extend security controls into AI-native systems.

Supply chain security must move beyond visibility. It must deliver enforceable control across the entire software delivery lifecycle. Software supply chain security isn’t about scanning more. It’s about governing every stage of software delivery from code to artifact to pipeline to production to AI.

Ready to see how Harness embeds supply chain security directly into CI/CD, artifact governance, and AI-powered verification?

Explore Harness Software Supply Chain Security solutions and secure your delivery pipeline end-to-end.

FAQs

Is SCA enough for software supply chain security?

No. SCA provides visibility into open-source vulnerabilities but does not protect CI/CD pipelines, artifact integrity, container ecosystems, or AI components.

Why are CI/CD pipelines considered high-risk?

Pipelines hold credentials, signing keys, and deployment paths. A compromised runner can inject malicious artifacts or exfiltrate secrets.

What is artifact governance?

Artifact governance includes registry gating, quarantine workflows, attestation verification, and policy enforcement before artifacts are promoted or deployed.

What is an AI Bill of Materials (AI-BOM)?

An AI-BOM would catalog AI components such as models, prompts, embeddings, and training data. Standards are still emerging.

How do supply chain attacks bypass SCA?

They exploit ingestion workflows, build steps, compromised pipelines, or malicious container images — rather than known CVEs.

Should organizations build or buy LLMs?

Build offers control but high cost and operational burden. Buy offers speed and vendor expertise but introduces trust and governance risks.

Engineering Blog

Your Repo Is a Knowledge Graph. You Just Don't Query It Yet.

SCM is evolving into Source Context Management, enabling AI agents with rich, precomputed context to improve accuracy, speed, and reliability in software development.

Ompragash Viswanathan

April 1, 2026

Time to read

Why Source Code Management must become Source Context Management in the age of AI agents

The Premise

For decades, SCM has meant one thing: Source Code Management. Git commits, branches, pull requests, and version history. The plumbing of software delivery. But as AI agents show up in every phase of the software development lifecycle, from writing a spec to shipping code to reviewing a PR, the acronym is quietly undergoing its most important transformation yet.

SCM is becoming Source Context Management.

And this isn't a rebrand. It's a rethinking of what a source repository is, what it stores, and what it serves, not just to developers, but to the agents working alongside them.

The Context Crisis in Agentic SDLC

AI agents in software development are powerful but contextually blind by default. Ask a coding agent to implement a feature and it will reach out and read files, one by one, directory by directory, until it has assembled enough context to act. Ask a code review agent to assess a PR and it will crawl through the codebase to understand what changed and why it matters.

Anthropic's 2026 Agentic Coding Trends Report documents this shift in detail: the SDLC is changing dramatically as single agents evolve into coordinated multi-agent teams operating across planning, coding, review, and deployment. The report projects the AI agents market to grow from $7.84 billion in 2025 to $52.62 billion by 2030. But as agents multiply across the lifecycle, so does their hunger for codebase context, and so does the cost of getting that context wrong.

This approach has two brutal failure modes:

Context window bloat. Feeding raw source files to an LLM is expensive, slow, and lossy. A 300,000-line codebase doesn't fit in any context window. The agent is forced to guess what's relevant, and it often guesses wrong.
Semantic blindness. Reading files doesn't tell an agent why code is structured the way it is, what modules depend on what, which functions are high-risk, or what the design philosophy behind a component is. Text is not meaning.

The result? Agents that hallucinate implementations because they missed a key abstraction three directories away. Code reviewers that flag style issues but miss architectural regressions. PRD generators that know the syntax of your codebase but not its soul.

The bottleneck is not the model. It is the absence of a pre-computed, semantically rich, always-available representation of the entire codebase: a context engine.

A Tale of Two Agents

Consider a simple task: "Add rate limiting to the /checkout endpoint."

Without a context engine, a coding agent opens checkout.go, reads the handler function, and writes a token-bucket rate limiter inline at the top of the handler. The code compiles. The tests pass. The PR looks clean.

The agent missed three things:

The service already uses a middleware-based rate limiting pattern in middleware/ratelimit.go for every other endpoint. The agent created a second, inconsistent approach.
A shared RateLimitConfig interface exists that all rate limiters must implement for centralized configuration management. The agent's inline implementation ignores it.
Every rate-limit event flows through a centralized metrics.Emit() call for observability dashboards. The agent's version remains invisible to ops.

The code works. The team that maintains it finds it wrong in every way that matters. A senior engineer catches these issues in review, requests changes, and the cycle restarts. Multiply this by every agent-generated PR across every team, every day.

With a context engine, the same agent queries before writing code: "How is rate limiting implemented in this service?" The context engine returns:

The existing middleware pattern in middleware/ratelimit.go
The RateLimitConfig interface it must implement
The metrics.Emit() integration point for observability
The test conventions in middleware/ratelimit_test.go

The agent writes a new rate limiter that follows the established pattern, implements the shared interface, emits metrics through the standard pipeline, and includes tests that match the existing style. The PR wins approval on the first pass.

The difference is context quality, not model quality.

Beyond LSP: From Interactive Intelligence to Agentic Intelligence

The Language Server Protocol (LSP) transformed developer tooling in the past decade. By standardizing the interface between editors and language-aware backends, LSP gave every IDE, from VS Code to Neovim, access to autocomplete, go-to-definition, hover documentation, and real-time diagnostics. LSP was designed to serve a specific consumer: a human developer, working interactively, in a single file at a time. That design made the right trade-offs for its era:

Interactive response optimization. Servers pre-compute and cache indices for low-latency, cursor-anchored queries rather than producing complete semantic snapshots of entire repositories on demand
Position orientation. Most queries anchor to a file and cursor position, perfect for an editor but limiting for full-repo semantic traversal
Session binding. Requires an active language server process, tightly coupled to an open editor session
Single-client design. The protocol assumes one client per server instance, not built for concurrent multi-agent access

For interactive development, these are strengths. LSP excels at what it was built to do.

Agents are a different class of consumer. They don't sit in a file waiting for cursor events. They operate across entire repositories, across SDLC phases, often in parallel. They need the full semantic picture before they start, not incrementally as they navigate.

Agents need not a replacement for LSP, but a complement: something pre-built, always available, queryable at repo scale, and semantically complete, ready before anyone opens a file.

Enter LST: The Foundation of Source Context Management

Lossless Semantic Trees (LST), pioneered by the OpenRewrite project (born at Netflix, commercialized by Moderne), take a different approach to code representation.

Unlike the traditional Abstract Syntax Tree (AST), an LST:

Preserves formatting. Whitespace, comments, style decisions are retained, enabling round-trip code transformation without destructive rewrites
Is fully type-attributed. Every node in the tree knows the full type of every symbol, including fields defined in external binary dependencies
Is pre-computed and cacheable. LSTs are generated once, stored, and queried repeatedly without needing a live language server
Scales to entire repositories. Moderne's platform has demonstrated querying and transforming hundreds of millions of lines of code in seconds using pre-stored LSTs

This is the first layer of a Source Context Management system. Not raw files. Not a running language server. A pre-indexed semantic tree of the entire codebase, queryable by agents at any time.

The Three-Layer Architecture of a Context Engine

A proper Source Context Management system is not a single component. It is a three-layer stack that turns a repository from a file store into something agents can actually reason over.

Layer 1: Semantic Indexing (LST + Embeddings)

Every file in the repository is parsed into an LST and simultaneously embedded into a vector representation. This creates two complementary indices:

Structural index (LST): knows types, dependencies, call hierarchies, inheritance chains
Semantic index (vectors): knows meaning, intent, similar patterns, conceptual proximity

Layer 2: Code Graph

The LST and semantic indices are projected into a code knowledge graph, a property graph where nodes are functions, classes, modules, interfaces, and comments, and edges are relationships: calls, imports, inherits, implements, modifies, tests.

This graph enables queries like:

"What is the blast radius if I change this interface?"
"Which modules have never been touched by the team that owns this service?"
"What are all the callers of this deprecated function across microservices?"
"Which code paths are covered by zero tests?"

Layer 3: Agentic Integration (MCP / API)

The context engine exposes itself through a Model Context Protocol (MCP) server or REST API, so any agent (whether a coding agent, a review agent, a risk assessment agent, or a documentation agent) can query the context engine directly, retrieving precisely the subgraph or semantic chunk it needs, without ever touching the raw file system.

The key insight: agents never read files. They query the context engine.

One Context Engine, Entire SDLC

A single context engine can serve every phase of the software development lifecycle.

Product Requirements (PRD Generation)

A PRD agent queries the context engine to understand existing capabilities, technical constraints, and module boundaries before generating a requirements document. It produces specs grounded in what the system actually is, not what someone thinks it is.

Technical Specification

A spec agent traverses the code graph to identify affected components, surface similar prior implementations, flag integration points, and propose an architecture, all without reading a single file directly.

Implementation (Coding)

A coding agent retrieves the precise subgraph surrounding the feature area: the types it needs to implement, the interfaces it must satisfy, the patterns used in adjacent modules, the test conventions for this package. It writes code that fits the codebase, not just code that compiles.

Pull Request & Code Review

A review agent queries the context engine to understand the semantic diff, not just what lines changed, but what that change means for the rest of the system. It can immediately surface:

Functions that are now unreachable
Breaking changes to downstream consumers
Regressions in design patterns
Missing test coverage for the changed blast radius

Risk Assessment

A risk agent scores every PR against the code graph, identifying high-centrality nodes (code that many things depend on), historically buggy modules, and changes that cross team ownership boundaries. No DORA metrics spreadsheet required.

Documentation & Design Principles

A documentation agent can traverse the code graph to generate living documentation (architecture diagrams, module dependency maps, API contracts) that updates automatically as the codebase evolves. Design principles can be encoded as graph constraints and validated on every merge.

Incident Response

When a production incident occurs, an on-call agent queries the context engine with the failing component and gets an immediate blast-radius map, the last 10 changes to that subgraph, the owners, and the test coverage status. Time-to-understanding drops from hours to seconds.

The Business Imperative

The business case is simple:

Developer productivity. Agents with accurate context write correct code on the first pass. Fewer review cycles, fewer reverted commits, fewer rollbacks.
Delivery velocity. Pre-computed context means agents don't spend half their time reading the codebase. Tasks that take minutes of agent compute today can take seconds.
Risk reduction. A code graph makes the blast radius of every change visible before it merges. Risk moves left, from production incidents to pre-merge awareness.
Institutional memory. The context engine captures why code is structured the way it is, not just what it does. New engineers (and new agents) onboard against the graph, not against tribal knowledge.

The Open Source Ecosystem Is Already Here

This is not a theoretical architecture. Tools exist today:

OpenRewrite / Moderne. LST generation and large-scale codebase transformation
tree-sitter. Universal parser for building ASTs across 150+ languages
CodeRAG. Graph-based code analysis for AI-assisted development
ArangoDB / FalkorDB / Neo4j / Memgraph. Graph databases well-suited for code relationship storage
Chroma / Qdrant / Milvus. Vector databases for semantic code embeddings
MCP (Model Context Protocol). Anthropic's open protocol for agent-to-tool communication
Context-aware code review engines. Platforms that leverage semantic code understanding to power AI-assisted review beyond surface-level linting

The missing piece is not any individual component. It is the platform that assembles them into a unified, repo-attached context engine that every agent in the SDLC can query through a single interface.

The Challenges to Solve

Source Context Management faces real engineering challenges:

Security and access control. A context engine is a pre-analyzed, queryable understanding of the codebase: dependency chains, blast-radius maps, test coverage gaps, ownership boundaries. In the wrong hands, this becomes a penetration testing roadmap. Agents querying the context engine must respect the same repo-level permissions that developers have, enforced at the graph query level, not just the API boundary. Context leakage across team or tenant boundaries is the single highest-severity threat vector this architecture introduces. Any serious deployment must treat threat modeling as a first-class architectural concern. Anthropic's 2026 Agentic Coding Trends Report makes the same call, listing "security-first architecture" as one of eight defining trends for agentic coding.
Polyglot repos. Most enterprise codebases span multiple languages. The context engine must support unified graph construction across Java, Python, TypeScript, Go, and more simultaneously.
Index freshness. The context engine must update incrementally on every commit, much like Git's own index, which uses content-addressable hashing and stat caching to detect exactly what changed without re-reading every file. A context engine that rebuilds from scratch on every push will not scale; one that recomputes only the affected subgraph on each commit will. A stale graph is worse than no graph, because it gives agents false confidence.
Graph scale. A 10-million-line monorepo produces a graph with hundreds of millions of edges. Query performance at this scale requires dedicated graph infrastructure.
Evaluation. How do you measure whether an agent's output improved because the context engine was accurate? Building evals for context quality is an unsolved problem.

The Reframe: What Is a Repository?

This is the shift:

A repository is not a collection of files. A repository is a knowledge graph with a version history attached.

Git's job is to version that knowledge. The context engine's job is to make it queryable. The agent's job is to act on it.

Follow this model and the consequences are concrete. Every CI/CD pipeline should include a context engine update step, as natural as running tests. Every developer platform should expose a context engine API alongside its code hosting API. Every AI coding tool should be evaluated not just on model quality but on context engine quality.

Source code repositories that don't invest in their context layer will produce agents that are fast but wrong. Repositories with rich, well-maintained context engines will produce agents that feel like senior engineers, because they have the same depth of understanding of the codebase that a senior engineer carries in their head.

Conclusion: The Next Infrastructure Primitive

The LSP gave us IDE intelligence. Git gave us version control. Docker gave us portable environments. Kubernetes gave us cluster orchestration. Each of these was an infrastructure primitive that unlocked a new generation of developer tooling.

The Source Context Engine is the next infrastructure primitive.

It is the prerequisite for every agentic SDLC capability worth building. And like every infrastructure primitive before it, the teams and platforms that build it first will be hard to catch.

SCM is no longer just about managing source code. It's about managing the context that makes the source code understandable.

Technical

Terraform Vendor Lock-In: How to Escape It

Terraform vendor lock-in is real. Learn how OpenTofu and Harness IaCM help you escape it and manage multi-IaC environments. Explore now.

Richard Black

March 31, 2026

Time to read

Did ecTerraform vendor lock-in just become your biggest operational risk without you noticing? When HashiCorp changed Terraform's license from MPL to BSL in August 2023, legal terms were not the only alteration. They fundamentally shifted the operational landscape for thousands of platform teams who built their infrastructure automation around what they believed was an open, community-driven tool. If your organization runs Terraform at scale, you're now facing a strategic decision that wasn't on your roadmap six months ago.

The uncomfortable truth is that most teams didn't architect for IaC portability. Why would they? Terraform was open source. It was the standard. And now, many organizations find themselves in a position they swore they'd never be in again after the Kubernetes wars: locked into a single vendor's roadmap, pricing model, and strategic priorities.

This isn't theoretical; it’s the very serious reality platform engineers are dealing with right now!

Why Terraform Lock-In Became Real Overnight

Terraform lock-in wasn't always a concern. For years, Terraform represented the opposite of vendor lock-in. It was open source, cloud-agnostic, and community-driven. Teams built entire operational models around it. They trained engineers, standardized on HCL, built module libraries, and integrated Terraform deeply into CI/CD pipelines. You’ve got to hand it to them; these aspects were very desirable.

Then HashiCorp moved to the Business Source License. Suddenly, the "open" in "open source" came with conditions. The BSL restricts certain commercial uses, and while many organizations technically fall outside those restrictions, the change introduced uncertainty.

Could HashiCorp tighten the license further?
Would future versions include features only available under commercial terms?
What happens when your infrastructure automation depends on a tool whose strategic direction you no longer control?

The deeper problem is architectural. Most teams didn't design for IaC engine portability because they didn't need to. Terraform state files, provider interfaces, and workflow patterns became embedded assumptions. Module libraries assumed Terraform syntax. Pipelines called `terraform plan` and `terraform apply` directly. When every workflow is tightly coupled to a single tool's CLI and API, switching becomes expensive.

This is classic vendor lock-in, even if it happened gradually and without malice.

The Hidden Costs of Staying Locked In

The immediate cost of Terraform lock-in isn't the license itself, but rather related to what you can't do when you're locked in.

You can't experiment with alternative IaC engines without rewriting modules.
You can't adopt tools that might better suit specific use cases.
You can't negotiate from a position of strength because your entire infrastructure automation stack depends on one vendor's roadmap.

If HashiCorp decides to sunset features, deprecate APIs, or introduce breaking changes, you either adapt or do without; stuck on an outdated version with mounting technical debt.

The operational risk compounds over time. When you're locked into a single IaC tool, you're also locked into its limitations. If drift detection isn't native, you build workarounds. If policy enforcement is bolted on, you maintain custom integrations. If the state backend causes performance issues at scale, you optimize around the bottleneck rather than solving the root problem.

And then there's the talent risk. If your team only knows Terraform, and the industry shifts toward other IaC paradigms, you're either retraining everyone or competing for a shrinking talent pool. Monocultures are fragile.

How to Escape Terraform Lock-In Without Rewriting Everything

The good news is that escaping Terraform lock-in doesn't require a full rewrite. It requires a deliberate strategy to introduce portability into your IaC architecture.

Start with OpenTofu as Your Baseline

OpenTofu emerged as the open-source fork of Terraform immediately after the license change. It's MPL-licensed, community-governed through the Linux Foundation, and API-compatible with Terraform 1.5.x. For most teams, OpenTofu migration is the lowest-friction path to regaining control over your IaC engine.

Migrating to OpenTofu doesn't mean abandoning your existing Terraform workflows. Because OpenTofu maintains compatibility with Terraform's core primitives, you can run OpenTofu side-by-side with Terraform during a transition. This lets you validate behavior, test edge cases, and build confidence before committing fully.

The strategic advantage of OpenTofu is not just licensing, optionality. Once you're no longer tied to HashiCorp's roadmap, you can evaluate IaC engines based on technical merit rather than sunk cost.

Decouple Your Workflows from Engine-Specific Assumptions

The harder part of escaping IaC vendor lock-in is decoupling your operational workflows from Terraform-specific patterns. This means abstracting your pipelines so they don't hardcode `terraform plan` and `terraform apply`. It means designing module interfaces that could theoretically support multiple engines. It means treating the IaC engine as an implementation detail rather than the foundation of your architecture.

This is where infrastructure as code portability becomes a design principle. If your pipelines call a generic "plan" and "apply" interface, switching engines becomes a simple configuration change, not a migration project.

Adopt Multi-IaC Tool Management from the Start

The reality is that most large organizations will eventually run multiple IaC tools. Some teams will use OpenTofu. Others will stick with Terraform for compatibility with existing state. New projects might adopt Terragrunt for DRY configurations or Pulumi for type-safe infrastructure definitions.

Fighting this diversity creates friction. Embracing it requires tooling that supports multi-IaC environments without forcing everyone into a lowest-common-denominator workflow. You need a platform that treats OpenTofu, Terraform, and other engines as first-class citizens, not as competing standards.

How Harness IaCM Enables Vendor-Neutral Infrastructure Management

Harness Infrastructure as Code Management was built to solve the multi-IaC problem that most teams are only now realizing they have. It doesn't force you to pick a single engine. It doesn't assume Terraform is the default. It treats OpenTofu and Terraform as equally supported engines, with workflows that abstract away engine-specific details while preserving the flexibility to use either.

This matters because escaping Terraform lock-in isn't just about switching tools. It's about building infrastructure automation that doesn't collapse the next time a vendor changes direction.

Harness IaCM supports OpenTofu and Terraform natively, which means you can run both engines in the same platform without maintaining separate toolchains. You get unified drift detection, policy enforcement, and workspace management across engines. If you're migrating from Terraform to OpenTofu, you can run both during the transition and compare results side-by-side.

The platform also supports Terragrunt, which means teams that have invested in DRY Terraform configurations don't have to throw away that work to gain vendor neutrality. You can keep your existing module structure while gaining the operational benefits of a managed IaC platform.

Beyond engine support, Harness IaCM addresses the systemic problems that make IaC vendor lock-in so painful. The built-in Module and Provider Registry means you're not dependent on third-party registries that could introduce their own lock-in. Variable Sets and Workspace Templates let you enforce consistency without hardcoding engine-specific logic into every pipeline. Default plan and apply pipelines abstract away the CLI layer, so switching engines doesn't require rewriting every workflow.

Drift detection runs continuously, which means you catch configuration drift before it becomes an incident. Policy enforcement happens at plan time, which means violations are blocked before they reach production. These aren't afterthoughts or plugins. They're native platform capabilities that work the same way regardless of which IaC engine you're using.

And because Harness IaCM is part of the broader Harness Platform, you can integrate IaC workflows with CI/CD, feature flags, and policy governance without duct-taping together disparate tools. This is the architectural model that makes multi-IaC tool management practical at scale.

Explore the Harness IaCM product or dive into the technical details in the IaCM docs.

The Path Forward: Portability as a Design Principle

Escaping Terraform lock-in is not about abandoning Terraform everywhere tomorrow. It's about regaining strategic control over your infrastructure automation. It's about designing for portability so that future licensing changes, roadmap shifts, or technical limitations don't force another painful migration.

The teams that will navigate this transition successfully are the ones that treat IaC engines as interchangeable components in a larger platform architecture. They're the ones that build workflows that abstract away engine-specific details. They're the ones that invest in tooling that supports multi-IaC environments without creating operational chaos.

If your organization is still locked into Terraform, now is the time to architect for optionality. Start by evaluating OpenTofu migration paths. Decouple your pipelines from engine-specific CLI calls. Adopt a platform that treats IaC engines as implementation details, not strategic dependencies.

Because the next time a vendor changes their license, you want to be in a position to evaluate your options, not scramble for a migration plan.

Technical

Regression Testing in CI/CD: Deliver Faster Without the Fear

Learn how intelligent regression testing in CI/CD uses test impact analysis, flaky test detection, and automated rollbacks to ship faster without risk.

Chinmay Gaikwad

March 31, 2026

Time to read

A financial services company ships code to production 47 times per day across 200+ microservices. Their secret isn't running fewer tests; it's running the right tests at the right time.

Modern regression testing must evolve beyond brittle test suites that break with every change. It requires intelligent test selection, process parallelization, flaky test detection, and governance that scales with your services.

Harness Continuous Integration brings these capabilities together: using machine learning to detect deployment anomalies and automatically roll back failures before they impact customers. This framework covers definitions, automation patterns, and scale strategies that turn regression testing into an operational advantage. Ready to deliver faster without fear?

What Is Regression Testing? (A Real-world Example)

Managing updates across hundreds of services makes regression testing a daily reality, not just a testing concept. Regression testing in CI/CD ensures that new code changes don’t break existing functionality as teams ship faster and more frequently. In modern microservices environments, intelligent regression testing is the difference between confident daily releases and constant production risk.

The Simple Definition: Regression testing is the practice of re-running existing tests after code changes to ensure nothing that previously worked is unintentionally broken. Instead of validating new features, it safeguards stable functionality across your application.
When Small Changes Create Big Problems: Even “low-risk” tweaks, like changing a payments API header, can silently break downstream jobs and critical flows like checkout. Regression tests catch these integration issues before production, protecting revenue and user experience.
How This Fits Into Modern CI/CD: In modern CI/CD, regression tests run continuously on pull requests, main branch merges, and staged rollouts like canaries. In each case, the tests ensure the application continues to work as expected.

Regression Testing vs. Retesting

These terms often get used interchangeably, but they serve different purposes in your pipeline. Understanding the distinction helps you avoid both redundant test runs and dangerous coverage gaps.

Retesting validates a specific fix. When a bug is found and patched, you retest that exact functionality to confirm the fix works. It's narrow and targeted.
Regression testing protects everything else. After that fix goes in, regression tests verify the change didn't break existing functionality across dependent services.

In practice, you run them sequentially: retest the fix first, then run regression suites scoped to the affected services. For microservices environments with hundreds of interdependent services, this sequencing prevents cascade failures without creating deployment bottlenecks.

The challenge is deciding which regression tests to run. A small change to one service might affect three downstream dependencies, or even thirty. This is where governance rules help. You can set policies that automatically trigger retests on pull requests and broader regression suites at pre-production gates, scoping coverage based on change impact analysis rather than gut feel.

To summarize, Regression testing checks that existing functionality still works after a change. Retesting verifies that a specific bug fix works as intended. Both are essential, but they serve different purposes in CI/CD pipelines.

Where Regression Fits in the CI/CD Pipeline

The regression testing process works best when it matches your delivery cadence and risk tolerance. Smart timing prevents bottlenecks while catching regressions before they reach users.

Run targeted regression subsets on every pull request to catch breaking changes within developer workflows. Keep these under 10 minutes for fast feedback.
Execute broader suites on main branch merges using parallelization and cloud resources to compress full regression cycles from hours to minutes.
Gate pre-production deployments with end-to-end smoke tests and contract validation before progressive rollout begins.
Monitor live metrics during canary releases and feature experiments to detect regressions under real traffic patterns that test environments can't replicate.
Combine synthetic monitoring with AI-powered automated rollback triggers to validate actual user impact and revert within seconds when thresholds are breached.

This layered approach balances speed with safety. Developers get immediate feedback while production deployments include comprehensive verification. Next, we'll explore why this structured approach becomes even more critical in microservices environments where a single change can cascade across dozens of services.

Why Regression Testing Matters for Microservices, Risk, and Compliance

Modern enterprises managing hundreds of microservices face three critical challenges: changes that cascade across dependent systems, regulatory requirements demanding complete audit trails, and operational pressure to maintain uptime while accelerating delivery.

Microservices Amplify Blast Radius Across Dependent Services

A single API change can break dozens of downstream services you didn't know depended on it.

Cascade failures are the norm, not the exception. A payment schema change that seems harmless in isolation can break reconciliation jobs, notification services, and reporting pipelines across 47 dependent services. Resilience or “Chaos” testing can help you assess your exposure to cascading failures.
Loosely coupled doesn't mean independent. NIST guidance confirms that cloud-native architectures consist of multiple components where individual changes can have system-wide impact.
Higher deployment frequency requires higher automation. Research demonstrates that microservices require automated testing integration into CD pipelines to maintain reliability at scale.

Regulated Environments Demand Complete Audit Trails

Financial services, healthcare, and government sectors require documented proof that tests were executed and passed for every promotion.

Compliance requires traceability. The DoD Cyber DT&E Guidebook mandates traceable test evidence for continuous authorization, noting that minor software changes can significantly impact system risk posture.
Policy-as-code turns testing into a compliance enabler. Harness governance features enforce required test gates and generate comprehensive audit logs of every approval and pipeline execution.
Auditors expect timestamped evidence on demand. Automated frameworks with audit logging eliminate scrambling when validation requests arrive.

Pre-Production Detection Reduces Operational Costs and MTTR

Catching regressions before deployment saves exponentially more than fixing them during peak traffic.

The math is simple. A failed regression test costs developer time; a production incident costs customer trust, revenue, and weekend firefighting.
Automated gates prevent breaks from reaching users. Research confirms that regression testing in pipelines stops new changes from introducing functionality failures.
AI verification adds a final safety net. Harness detects anomalies post-deployment and triggers automated rollbacks within seconds, eliminating expensive emergency responses.

With the stakes clear, the next question is which techniques to apply.

Types of Regression Testing Techniques You'll Actually Use

Once you've established where regression testing fits in your pipeline, the next question is which techniques to apply. Modern CI/CD demands regression testing that balances thoroughness with velocity. The most effective techniques fall into three categories: selective execution, integration safety, and production validation.

Types of Regression Testing Techniques You'll Actually Use

Once you've established where regression testing fits in your pipeline, the next question is which techniques to apply. Modern CI/CD demands regression testing that balances thoroughness with velocity. The most effective techniques fall into three categories: selective execution, integration safety, and production validation—with a few pragmatic variants you’ll use day-to-day.

Full regression suites rerun your critical end-to-end and high-value scenarios before major releases or architectural changes. They’re slower, but essential for high‑risk changes and compliance-heavy environments.
Smoke and sanity regression focus on a small, fast set of tests that validate core flows (login, checkout, core APIs) on every commit or deployment. These suites act as your “always on” safety net.
Unit-level regression runs targeted unit tests around recently changed modules. This is your fastest feedback loop, catching logic regressions before they ever hit cross-service integration or UI layers.
Selective regression and test impact analysis run only the suites that exercise changed code paths, using dependency mapping to cut execution time without sacrificing confidence.
Contract testing enforces backward compatibility through consumer-driven contracts like Pact, preventing integration failures between teams and services.
API/UI regression testing locks in behavior at the interaction layer—REST, GraphQL, or UI flows—so refactors behind the scenes don’t break user-visible behavior.
Performance and scalability regression ensure that latency, throughput, and resource usage don’t degrade between releases, especially for high-traffic or revenue-critical paths.
Progressive delivery verification combines canary deployments with real-time metrics and error signals to surface regressions under actual traffic, with automated halt/rollback when thresholds are breached.

These approaches work because they target specific failure modes. Smart selection outperforms broad coverage when you need both reliability and rapid feedback.

How to Automate Regression Testing Across Your Pipeline

Managing regression testing across 200+ microservices doesn't require days of bespoke pipeline creation. Harness Continuous Integration provides the building blocks to transform testing from a coordination nightmare into an intelligent safety net that scales with your architecture.

Step 1: Generate pipelines with context-aware AI. Start by letting Harness AI build your pipelines based on industry best practices and the standards within your organization. The approach is interactive, and you can refine the pipelines with Harness as your guide. Ensure that the standard scanners are run.

Step 2: Codify golden paths with reusable templates. Create Harness pipeline templates that define when and how regression tests execute across your service ecosystem. These become standardized workflows embedding testing best practices while giving developers guided autonomy. When security policies change, update a single template and watch it propagate to all pipelines automatically.

Step 3: Enforce governance with Policy as Code. Use OPA policies in Harness to enforce minimum coverage thresholds and required approvals before production promotions. This ensures every service meets your regression standards without manual oversight.

With automation in place, the next step is avoiding the pitfalls that derail even well-designed pipelines.

Best Practices and Common Challenges (And How to Fix Them)

Regression testing breaks down when flaky tests erode trust and slow suites block every pull request. These best practices focus on governance, speed optimization, and data stability.

Quarantine flaky tests automatically using policy enforcement and require test owners before suite re-entry. Research shows flaky tests reproduce only 17-43% of the time, making governance more effective than debugging individual failures.
Parallelize and shard test execution across multiple agents to keep PR feedback under 5 minutes.
Apply test impact analysis to run only tests affected by code changes, reducing unnecessary execution.
Provision ephemeral test environments with seeded datasets to eliminate data drift between runs.
Use contract-backed mocks for external dependencies to ensure consistent test behavior.
Add AI-powered verification as a final backstop to catch regressions that slip past test suites.

Turn Regression Testing Into a Safety Net, Not a Speed Bump

Regression testing in CI/CD enables fast, confident delivery when it’s selective, automated, and governed by policy. Regression testing transforms from a release bottleneck into an automated protection layer when you apply the right strategies. Selective test prioritization, automated regression gates, and policy-backed governance create confidence without sacrificing speed.

The future belongs to organizations that make regression testing intelligent and seamless. When regression testing becomes part of your deployment workflow rather than an afterthought, shipping daily across hundreds of services becomes the norm.

Ready to see how context-aware AI, OPA policies, and automated test intelligence can accelerate your releases while maintaining enterprise governance? Explore Harness Continuous Integration and discover how leading teams turn regression testing into their competitive advantage.

FAQ: Practical Answers for Regression Testing in CI/CD

These practical answers address timing, strategy, and operational decisions platform engineers encounter when implementing regression testing at scale.

When should regression tests run in a CI/CD pipeline?

Run targeted regression subsets on every pull request for fast feedback. Execute broader suites on the main branch merges with parallelization. Schedule comprehensive regression testing before production deployments, then use core end-to-end tests as synthetic testing during canary rollouts to catch issues under live traffic.

How do we differentiate regression testing from retesting in practice?

Retesting validates a specific bug fix — did the payment timeout issue get resolved? Regression testing ensures that the fix doesn’t break related functionality like order processing or inventory updates. Run retests first, then targeted regression suites scoped to affected services.

How much regression coverage is enough for production?

There's no universal number. Coverage requirements depend on risk tolerance, service criticality, and regulatory context. Focus on covering critical user paths and high-risk integration points rather than chasing percentage targets. Use policy-as-code to enforce minimum thresholds where compliance requires it, and supplement test coverage with AI-powered deployment verification to catch regressions that test suites miss.

Should we run full regression suites on every commit?

No. Full regression on every commit creates bottlenecks. Use change-based test selection to run only tests affected by code modifications. Reserve comprehensive suites for nightly runs or pre-release gates. This approach maintains confidence while preserving velocity across your enterprise delivery pipelines.

What's the best way to handle flaky tests without blocking releases?

Quarantine flaky tests immediately, rather than letting them block pipelines. Tag unstable tests, move them to separate jobs, and set clear SLAs for fixes. Use failure strategies like retry logic and conditional execution to handle intermittent issues while maintaining deployment flow.

How do we maintain regression test quality at scale?

Treat test code with the same rigor as application code. That means version control, code reviews, and regular cleanup of obsolete tests. Use policy-as-code to enforce coverage thresholds across teams, and leverage pipeline templates to standardize how regression suites execute across your service portfolio.

Technical

Making Deploys Safe Shouldn’t be Hard

Harness introduces AI configuration and explanation for automated deplyoment verification.

Eric Minick

March 31, 2026

Time to read

Eight years ago, we shipped Continuous Verification (CV) to solve one of the most miserable parts of a great engineer’s job: babysitting deployments.

The idea was simple but powerful. At 3:00 AM, your best engineers shouldn't be staring at dashboards waiting to see if a release went sideways. CV was designed to think like those engineers, watching your APM metrics, scanning your logs, and making the call for you. Roll forward or roll back, automatically, based on what the data actually said.

It worked. Customers loved it. Hundreds of teams stopped losing sleep over deployments.

But somewhere along the way, we noticed a new problem creeping in: setting up CV had become its own burden.

The Configuration Problem

To get value from Continuous Verification, you had to know what to look for. Which metrics matter for this service? Which log patterns indicate trouble? Which thresholds separate a blip from a real incident?

When we talk to teams trying to use Argo Rollouts and set up automatic verification with its analysis templates, we hear that they hit the same challenges.

For teams with deep observability expertise, this was fine. For everyone else—and honestly, for experienced teams onboarding new services—it added friction that shouldn't exist. We’d solved the hardest part of deployments, but we’d left engineers with a new "homework assignment" just to get started.

That’s what AI Verification & Rollback is designed to fix.

What’s New: AI That Knows What to Look For

AI Verification & Rollback builds directly on the CV foundation you already trust, but adds a layer of intelligence before the analysis even begins. Instead of requiring you to define your metrics and log queries upfront, the system queries your observability provider—via MCP server—at the moment of deployment to determine what actually matters for the service you just deployed.

What that means in practice:

No pre-configuration required: You don’t need to define metrics or log patterns before adding verification to a pipeline. The AI identifies what’s relevant based on your observability data at runtime.
Step-by-step transparency: Rather than a black-box pass/fail, you see the full reasoning—which metrics were collected, how logs were aggregated, and why the AI reached its conclusion. Every step is visible.
Plain-language summaries: When a verification fails, you don't just get a red indicator. You get an analysis that explains exactly what went wrong and which data led to that determination—the same explanation a senior engineer would walk you through.
Integrated into your existing pipelines: Adding AI V&R is as simple as dropping the step into your pipeline. It configures itself for the application being deployed.

At our user conference six months ago, we showed this running live—triggering a real deployment, watching the MCP server query Dynatrace for relevant signals, and walking through a live failure analysis that caught a bad release within minutes. The response was immediate. Engineers got it instantly, because it matched how they already think about post-deploy monitoring.

What’s Changed Since the Preview

We’ve spent the past six months hardening what we showed you. A few highlights:

Broader observability support: The preview demoed Dynatrace integration. We've since expanded MCP server support to cover additional observability and logging providers, so more teams can take advantage without waiting for their toolchain to be supported.
More pipeline step coverage: The initial build focused on post-deployment verification. We've extended the capability to support additional pipeline steps, giving you AI-driven analysis at more points in your delivery workflow.
Production hardening: What you see today is meaningfully more robust than what ran on stage in the fall, improved reliability, better handling of edge cases, and refinements to the analysis and summary output based on real usage.

Where We Are Today

We're not declaring CV legacy today. AI Verification & Rollback is not yet a full replacement for traditional Continuous Verification across all use cases and customer configurations. CV remains the right choice for many teams, and we're committed to supporting it.

Bottom line: AI V&R is ready for many teams to use. It's available now, and for teams setting up verification for the first time—or looking to reduce the operational overhead of maintaining verification configs—it's the faster, smarter path forward.

The takeaway here is simple: If you've been putting off setting up Continuous Verification because of the configuration overhead, this is the version you were waiting for.

Ready to stop babysitting your releases? Drop the AI V&R step into your next pipeline and see what it finds.

How is your team currently handling the "3:00 AM dashboard stare"—and how much time would you save if the pipeline just told you why it rolled back?

‍

Technical

Release Orchestration Bridges the AI Delivery Gap

Learn about Harness's new Release Orchestration capability for coordinated multi-team or multi-service releases.

Eric Minick

March 31, 2026

Time to read

AI has officially made writing code cheap.

Your developers are shipping more changes, across more microservices, more frequently than ever before. If you’re a developer, it feels like a golden age.

But for the Release Engineer? This isn't necessarily a celebration; it’s a scaling nightmare.

We’re currently seeing what I call the "AI delivery gap." It’s that uncomfortable space between the breakneck speed at which we can now generate code and the manual, spreadsheet-driven processes we still use to actually release it.

The reality is that while individual CI/CD pipelines might be automated, the coordination between them remains a stubbornly human bottleneck. We’ve automated the "how" of shipping code, but we’re still stuck in the Dark Ages when it comes to the "when" and "with whom."

Today, we are introducing Harness Release Orchestration alongside four other capabilities that ensure confident releases. Release Orchestration is designed to transform the release management process from a fragmented, manual effort into a standardized, visible, and scalable operation.

‍

Easily see your releases at a glance, and the status of each

The Architecture of a Modern Release: A "Pipeline of Pipelines"

Most release engineers I talk to spend about 40% of their time "chasing humans for status." You’re checking Slack threads for sign-offs, updating Confluence pages, and obsessively watching spreadsheets to ensure Team A’s service doesn't break Team B’s dependency. (And let’s be honest, it usually does anyway.)

We could call it a team sport, but it’s really a multi-team sport. Teams from multiple services and functions need to come together to deliver a big release.

If we rely on a person to coordinate, we can’t move fast enough.

Harness Release Orchestration moves beyond the single pipeline. It introduces a process-based framework that acts as your release "blueprint."

Visual Release Blueprints: Instead of hundreds of bespoke scripts, you define reusable processes composed of phases (Dev, QA, Security, Production) and activities. These activities represent the complete workflow, including manual sign-offs and automated progressive delivery.
The Shared Release Calendar: No more "When is [Service X] shipping?" questions. The shared calendar provides a single source of truth for planning and deconflicting release schedules. It gives visibility to stakeholders from Engineering all the way to Product and Marketing.
Governance at Speed: We’ve built in "paved roads." You can establish standards that allow developers to self-serve their releases without bypassing the safety checks that keep you out of the news for the wrong reasons.

Native to Continuous Delivery

Release management software isn’t an entirely new idea. It’s been tried before, but never widely adopted. The industry went wrong by building separate tools for continuous delivery and release orchestration.

With separate tools, you incur integration overhead, have multiple places to look, and experience awkwardness.

We’ve built ours alongside our CD experience, so everything is as seamless and fast as possible. Yes, this is for releases that are more complex than a simple microservice, which the app team delivers on their own. No, that doesn’t mean introducing big processes and standalone tools.

AI to the Rescue: Migration and Ingestion

Here’s the “gotcha”: the biggest barrier to adopting a new release tool is the hassle of migrating. You likely have years of proven workflows documented in SharePoint/Confluence, in early-release management tools like XL Release, or in the fading memory of that one person who isn't allowed to retire.

Harness AI now handles the heavy lifting. Our AI Process Ingestion can instantly generate a comprehensive release process from a simple natural-language prompt, existing documentation, or export from a tool.

What used to take months of manual configuration now takes seconds. Simply put, we’re removing the friction of modernization.

Leverage, Not Heroics

For the Release Engineer, the goal is leverage. You shouldn't need to perform heroics every Friday night to ensure a successful release. (Though if you enjoy the adrenaline of a 2:00 AM war room, I suppose I can’t stop you.)

Harness Release Orchestration creates a standardized release motion that scales with AI-driven output. It allows you to move from being a "release waiter" to a "release architect."

AI made writing code cheap. Harness makes releasing it safe, scalable, and sustainable.

‍

Technical

Harness Ships Five Capabilities to Power Confident Releases at AI Speed

Harness releases 5 new capabilities: Release Orchestration, Warehouse Native Feature Management, FME in Pipeline, AI Verification and Database DevOps for Snowflake

Brad Rydzewski

March 31, 2026

Time to read

Engineering teams are generating more shippable code than ever before — and today, Harness is shipping five new capabilities designed to help teams release confidently. AI coding assistants lowered the barrier to writing software, and the volume of changes moving through delivery pipelines has grown accordingly. But the release process itself hasn't kept pace.

The evidence shows up in the data. In our 2026 State of DevOps Modernization Report, we surveyed 700 engineering teams about what AI-assisted development is actually doing to their delivery. The finding stands out: while 35% of the most active AI coding users are already releasing daily or more, those same teams have the highest rate of deployments needing remediation (22%) and the longest MTTR at 7.6 hours.

This is the velocity paradox: the faster teams can write code, the more pressure accumulates at the release, where the process hasn't changed nearly as much as the tooling that feeds it.

The AI Delivery Gap

What changed is well understood. For years, the bottleneck in software delivery was writing code. Developers couldn't produce changes fast enough to stress the release process. AI coding assistants changed that. Teams are now generating more change across more services, more frequently than before — but the tools for releasing that change are largely the same.

In the past, DevSecOps vendors built entire separate products to coordinate multi-team, multi-service releases. That made sense when CD pipelines were simpler. It doesn't make sense now. At AI speed, a separate tool means another context switch, another approval flow, and another human-in-the-loop at exactly the moment you need the system to move on its own.

The tools that help developers write code faster have created a delivery gap that only widens as adoption grows.

What Harness Is Shipping

Today Harness is releasing five capabilities, all natively integrated into Continuous Delivery. Together, they cover the full arc of a modern release: coordinating changes across teams and services, verifying health in real time, managing schema changes alongside code, and progressively controlling feature exposure.

Coordinate multi-team releases without the war room

Release Orchestration replaces Slack threads, spreadsheets, and war-room calls that still coordinate most multi-team releases. Services and the teams supporting them move through shared orchestration logic with the same controls, gates, and sequence, so a release behaves like a system rather than a series of handoffs. And everything is seamlessly integrated with Harness Continuous Delivery, rather than in a separate tool.

Know when to stop — automatically

AI-Powered Verification and Rollback connects to your existing observability stack, automatically identifies which signals matter for each release, and determines in real time whether a rollout should proceed, pause, or roll back. Most teams have rollback capability in theory. In practice it's an emergency procedure, not a routine one. Ancestry.com made it routine and saw a 50% reduction in overall production outages, with deployment-related incidents dropping significantly.

Ship code and schema changes together

Database DevOps, now with Snowflake support, brings schema changes into the same pipeline as application code, so the two move together through the same controls with the same auditability. If a rollback is needed, the application and database schema can rollback together seamlessly. This matters especially for teams building AI applications on warehouse data, where schema changes are increasingly frequent and consequential.

Roll out features gradually, measure what actually happens

Improved pipeline and policy support for feature flags and experimentation enables teams to deploy safely, and release progressively to the right users even though the number of releases is increasing due to AI-generated code. They can quickly measure impact on technical and business metrics, and stop or roll back when results are off track. All of this within a familiar Harness user interface they are already using for CI/CD.

Warehouse-Native Feature Management and Experimentation lets teams test features and measure business impact directly with data warehouses like Snowflake and Redshift, without ETL pipelines or shadow infrastructure. This way they can keep PII and behavioral data inside governed environments for compliance and security.

These aren't five separate features. They're one answer to one question: can we safely keep going at AI speed?

From Deployment to Verified Outcome

Traditional CD pipelines treat deployment as the finish line. The model Harness is building around treats it as one step in a longer sequence: application and database changes move through orchestrated pipelines together, verification checks real-time signals before a rollout continues, features are exposed progressively, and experiments measure actual business outcomes against governed data.

A release isn't complete when the pipeline finishes. It's complete when the system has confirmed the change is healthy, the exposure is intentional, and the outcome is understood.

That shift from deployment to verified outcome is what Harness customers say they need most. "AI has made it much easier to generate change, but that doesn't mean organizations are automatically better at releasing it," said Marc Pearce, Head of DevOps at Intelliflo. "Capabilities like these are exactly what teams need right now. The more you can standardize and automate that release motion, the more confidently you can scale."

Release Becomes a System, Not a Scramble

The real shift here is operational. The work of coordinating a release today depends heavily on human judgment, informal communication, and organizational heroics. That worked when the volume of change was lower. As AI development accelerates, it's becoming the bottleneck.

The release process needs to become more standardized, more repeatable, and less dependent on any individual's ability to hold it together at the moment of deployment. Automation doesn't just make releases faster. It makes them more consistent, and consistency is what makes scaling safe.

For Ancestry.com, implementing Harness helped them achieve 99.9% uptime by cutting outages in half while accelerating deployment velocity threefold.

At Speedway Motors, progressive delivery and 20-second rollbacks enabled a move from biweekly releases to multiple deployments per day, with enough confidence to run five to 10 feature experiments per sprint.

AI made writing code cheap. Releasing that code safely, at scale, is still the hard part.

Harness Release Orchestration, AI-Powered Verification and Rollback, Database DevOps, Warehouse-Native Feature Management and Experimentation, and Improve Pipeline and Policy support for FME are available now. Learn more and book a demo.

Technical

How to Scale Sandbox Environments with an Internal Developer Portal

Scale sandbox environments with self-service automation and governance. Empower teams, reduce ticket-ops, and accelerate onboarding with Harness.

Bri Strozewski

March 30, 2026

Time to read

Self-service sandbox environments powered by an Internal Developer Portal cut through provisioning bottlenecks, giving developers access to secure, policy-driven test environments in minutes — not days.
Automated sandbox environment provisioning with built-in guardrails keeps governance consistent, reduces security risk, and frees platform teams from endless manual requests.
Tracking metrics like time-to-first-environment, ticket reduction, and sandbox utilization helps you prove the ROI of your platform investments and ship software faster (and safer) at scale.

Here's a scenario that probably sounds familiar: a developer needs a sandbox environment to test something. They file a ticket. Then they wait. And wait. Maybe a day goes by, maybe three. Meanwhile, your platform team is buried in provisioning requests, and somewhere, someone has already spun up an unsanctioned workaround that bypasses every governance policy you've put in place.

It's a lose-lose. Developers lose velocity, platform teams lose their sanity, and security gaps quietly multiply.

An Internal Developer Portal flips this whole dynamic. Instead of ticket queues and manual provisioning, you get a self-service sandbox environment automation with guardrails baked in. Developers get instant access to governed, policy-driven environments. Platform engineers get visibility and control — without becoming a bottleneck.

In this post, we'll walk through how to build sandbox automation that actually works, set governance that holds up at scale, accelerate developer onboarding, and measure ROI in a way that makes leadership pay attention.

Harness Internal Developer Portal brings enterprise-grade orchestration together with developer-friendly self-service — built on the Backstage framework with the security, scalability, and governance that enterprises actually need.

Why Sandbox Environments Are the Fastest Path to Safer, Faster Delivery

Let's be real: if your developers are waiting days for a test environment while your security team is blocking deployments they can't validate, something is broken. Teams end up either skipping testing altogether or delaying releases — neither of which is a good outcome. A solid sandbox environment strategy breaks this cycle by giving teams secure, isolated spaces to validate changes without compromising governance or speed.

Isolation Reduces Blast Radius Through Disposable Infrastructure

The best way to think about sandboxes? Treat them as disposable. Each sandbox environment gets the minimum privileges needed for the task at hand — whether that's validating a database migration, testing a third-party API integration, or experimenting with a new service configuration.

If something breaks, you just delete the sandbox environment and start fresh. No incident reports. No painful rollback procedures. No 2 a.m. pages to the on-call engineer. Teams that adopt this pattern tend to catch configuration errors much earlier in the development cycle.

Security-by-Default with Controlled Testing Spaces

Think of sandbox environments as your first line of defense. When you're dealing with untrusted code, unfamiliar dependencies, or configuration changes you're not 100% sure about, run them in an isolated sandbox first. Validate the behavior, confirm it's clean, and only then promote the changes to staging or production.

This pattern is incredibly effective at catching supply chain issues, configuration drift, and integration failures before they ever touch your shared systems. It shifts security left in a way that's practical, not just aspirational.

Enterprise Patterns That Balance Fidelity, Cost, and Speed

Not every test needs a full production replica. A smart sandbox environment strategy uses a tiered approach: lightweight developer sandboxes for day-to-day feature work, partial-copy environments for integration testing with realistic (but masked) data, and full-copy sandboxes reserved for final validation before release.

Add in ephemeral PR environments that spin up automatically when a pull request is opened and tear themselves down after a set time window, and you've got realistic testing without persistent infrastructure costs or manual provisioning overhead.

Automating Sandbox Environment Provisioning in Your Internal Developer Portal

When sandbox environment requests take days instead of minutes, you're not just slowing down development — you're creating a governance problem that only gets worse as you scale. Automating provisioning through your IDP eliminates this friction while keeping the controls you need.

Here's what that looks like in practice:

Golden path templates that bundle VPC isolation, namespace creation, data seeding, and data masking into standardized, repeatable deployments — so sensitive information is protected by default, not as an afterthought.
Policy as Code enforcement (using tools like OPA) for automatic approval workflows, resource quotas, and RBAC — no manual reviews required.
Fast spin-up times for ephemeral PR environments, leveraging pre-warmed infrastructure and cached images.
Automatic teardown with sensible TTLs to prevent cost drift and resource sprawl.
CI/CD integration, so sandbox environment automation triggers naturally from pull requests and deployment events.

This approach transforms your IDP from a service catalog into a genuine self-service platform — one that scales governance instead of bypassing it.

Choosing the Right Sandbox Environment Type for Your Workflows

Not all sandbox environments are created equal. Picking the wrong type wastes resources and creates either security gaps or unnecessary developer friction. Knowing which type fits each workflow helps platform teams design self-service templates that match what developers actually need.

Type	Primary Use	Isolation Level	Typical TTL	Data Strategy	Automation Pattern
Developer Sandbox	Feature development, unit testing	Kubernetes namespace with RBAC	7–14 days (auto-cleanup)	Minimal seed data, synthetic records	Template-driven, auto-refresh
Integration Sandbox	API testing, third-party validation	Dedicated VPC with security groups	2–4 hours (auto-cleanup)	Masked production subset	PR-triggered, auto-cleanup
Full Copy Sandbox	End-to-end testing, data migration	Separate AWS account or cluster	24–48 hours (manual extension)	Full production replica with masking	Scheduled refresh, manual approval
Security Detonation	Malware analysis, untrusted code execution	Air-gapped containers with monitoring	30–60 minutes (immediate destruction)	No persistent data, logging only	Event-triggered, immediate destruction

Developer sandboxes prioritize speed and productivity. Security detonation environments prioritize total isolation, even at the cost of convenience. Most teams will use a mix across their workflows.

Onboarding at Scale: Self-Service Sandbox Environments from Day One

Here's something that frustrates every engineering leader: a new hire joins your team, and it takes them three days just to get a working development environment. That's three days of lost productivity, plus a terrible first impression of your engineering culture.

With an Internal Developer Portal, you can bundle everything a new developer needs — sandbox environment provisioning, documentation, credentials, and golden path templates — into a single catalog request. Day one, they click a button, and they're up and running. No more Slack threads asking "where's my environment?"

Use Scorecards — a native feature in Harness IDP — to track the metrics that actually matter: time-to-first-PR-environment, sandbox environment mean time to recovery, and reuse rates. When self-service becomes the default, ticket provisioning drops dramatically.

Operationalize Sandbox Environments with Harness Internal Developer Portal

Scaling sandbox environments across your organization doesn't mean you need to double your platform team or accept governance trade-offs. Harness IDP extends the Backstage framework with the enterprise capabilities that self-managed Backstage doesn't offer out of the box — fine-grained RBAC, hierarchical organization, native Scorecards tied to your KPIs, access to the 200+ Backstage plugin ecosystem, and environment management that scales with your teams.

Ready to ship governed, self-service sandbox environments that actually reduce toil? Try Harness Internal Developer Portal and see how fast your teams move when the friction disappears.

Sandbox Environment FAQ for Platform Engineering Leaders

These are the questions that come up most often when platform leaders are planning sandbox environment rollouts or making the case to leadership.

How do sandbox environments improve software delivery security?

Sandbox environments create isolated failure boundaries where you can test risky changes, third-party integrations, and security patches without touching shared systems. Teams validate behavior in production-like conditions, catching issues before they reach staging or production — which means fewer incidents and faster recovery when things do go wrong.

What metrics should platform teams track to measure sandbox environment success?

Focus on time-to-first-environment for new developers, sandbox environment utilization rates, and ticket reduction percentages. Also track cost per sandbox hour and cleanup effectiveness. Developer satisfaction scores and time saved on provisioning requests are the numbers that help you demonstrate platform ROI to leadership and justify continued investment.

How do you prevent sandbox environment sprawl and cost overruns?

The biggest culprit is sandbox environments that get spun up and never torn down. Set TTL-based auto-teardown policies so environments self-destruct after a defined window — a few hours for ephemeral PR environments, a couple of weeks for longer-lived developer sandboxes. Pair that with resource quotas enforced through policy-as-code, and you've got cost controls that don't depend on anyone remembering to clean up after themselves.

Can sandbox environments work across multi-cloud or hybrid infrastructure setups?

Yes, and this is where a platform approach really pays off. Instead of building separate provisioning workflows for each cloud provider, golden path templates in your IDP abstract the infrastructure layer. Developers request a sandbox environment from a single catalog — whether it lands on AWS, Azure, GCP, or an on-prem cluster is handled by the template logic and your team's policies. Harness IDP supports multi-cloud and multi-region deployments natively, so you're not stitching together one-off scripts for each environment.

How We Build

Product Portfolio Management for New Paradigms - DevOps, AI, and Beyond - Job Task Analysis

New paradigms are a constant in technology. AI currently has lots of momentum, but how do you build a practice or portfolio around a new or emerging paradigm? Look to your DevOps Product Portfolio Management for inspiration.

Ravi Lachhman

March 27, 2026

Time to read

Taking a look back over the last ten years in enterprise technology, paradigm shifts are occurring more frequently. For example, the maturity of DevOps/Platform Engineering and Cloud Native infrastructure has occurred. The new frontier depending where you are in adoption is AI. As your adoption and maturity curve progress, operationalizing these paradigms become important. Many sophisticated firms that have to manage and even innovate paradigms that are possibly not core to their business model look to Product Portfolio Management aka PPM to manage the business aspect of the paradigm.

A core pillar to PPM is resource management or the management of skillsets across the portfolio. We looked inwards to our own Professional Service team and ran a Job Task Analysis aka JTA to analyze how Harness itself built a portfolio of implementation experts.

For organizations/firms looking not only to scale Harness but potentially other platforms or paradigms, can use the Harness Implementation Job Task Analysis as one framework to use and apply to your specific need.

Remember Your Org’s DevOps Adoption?

In 2026, it is exciting to say that DevOps is mature. Most in industry agree today to the pillars and practices of what DevOps has been trying to achieve are normal for most engineering orgs. If you turn the clock back 15 years ago, DevOps was still very much emerging as a paradigm. The trio of people, process, and technology followed as the movement became more mature.

In an early episode of ShipTalk, we talked to Nidhi Allipuram, who at the time was leading a DevOps Org at a large insurance company. In the episode we talked about how she went from 0-1 in scaling a DevOps idea to building a DevOps Org. Like any technology adoption with a new or fast moving paradigm (think of AI today), there are business considerations that typically follow the maturity curve. The process she followed was incremental starting to gather internal expertise then having to answer “is this problem even worth solving”. From there once momentum is made, scale will follow.

The episode was great timing, for myself I was becoming a new first-time manager here at Harness and Harness itself was turning into a multi-module platform. I was really curious about how she load-balanced skills on her team to continue to expand into an evolving paradigm. Her advice was solid that you do have to balance the team to grow complementary skillsets and be able to keep up with their internal customer demand. This is exactly what resource management in a PPM based discipline/org would strive for. The Job Task Analysis that was just concluded can be used in resource management when looking to scale the program/portfolio.

Harness Implementation Job Task Analysis - Hard and Soft Skills

Looking at the Implementation Job Task Analysis and how Harness itself scales our Professional Services Practice, there is a needed mixture of both hard and soft skills. The hard skills are vertical skills in the tooling/platforms/ecosystem that Harness participates in. Second and just as important are the soft skills around problem solving and requirements gathering / stakeholder management. The hard skills would answer the “how” and “where” type of questions. The soft skills will answer just as importantly the “why” and “what” type of questions. As Harness is a platform of outcomes, solving for both hard and soft skills is important.

Extrapolating this to building a portfolio or program in evolving or even mature technology domains, the soft skills bolster the initial organic question “is this problem even worth solving”. In technology there is rarely a singular big-bang type of innovation occurrence. A good amount of innovation happens incrementally and with a solid team/consensus driven approach. Getting consensus can also be a challenge and a typical approach is requirements/stakeholder management gathering inputs from many points-of-view and blending them for trade-offs.

Distilling down the findings in the Implementation Job Task Analysis, for highly technical resources stakeholder / requirements management can be challenging as there is a lot of grey area. For example in the Site Reliability Engineering aka SRE world, defining your SLIs/SLOs is one set of challenges but having to get consensus to renegotiate them is a different order of problem. As we look towards AI, the fundamentals do not go away.

Well What About AI?

AI is certainly a rapidly evolving paradigm that is rapidly changing how we use technology. In a recent Cloud Native Podcast where the topic was Generative AI in Platform Engineering. AI, especially how it is used in Platform Engineering, is still early in the adoption curve. In 2026, The Linux Foundation will have its first AI centric conference with AGNTCon + MCPCon which is telling that maturity will eventually arrive. With items that are either early or late maturity, fundamentals do not go away.

On the Cloud Native Podcast, there was a conversation around how Kubernetes was during the first KubeCon in 2015 and how AGNTCon + MCPCon is a telling similar signal in 2026. The panel in the podcast reflected on their own K8s journey and early questions around innovation vs control/operationalization came up when looking towards AI; similar concerns when looking at K8s early on.

Crucial to navigating the radio dials of innovation vs control is again the fundamentals of soft skills getting stakeholder or internal customer alignment which is by incremental consensus. In the Implementation Job Task Analysis, the people who are agents of change in software delivery need to blend both their hard and soft skills on a daily basis. These very people are available to help further your innovation goals with top notch software delivery via the Harness Platform.

Partner with Harness for New Paradigm Expertise

As you start to embrace new paradigms and the Harness Platform, we are here to help. Harness has been at the forefront of the Cloud Native and now AI-Centric software delivery worlds. Our platform continues to evolve to meet these new and similar challenges in the technology adoption curve. Harness Professional Services are the people helping your organization making these new paradigms become reality. Feel free to take a look at our Services Offerings which includes consulting/coaching to training. We are excited to share how we scale our own internal skills to help our customers/partners/and public continue on to better the software delivery craft.

Engineering Blog

Build Numbers That Actually Make Sense: Branch-Scoped Sequence IDs in Harness CI

Most CI tools use global build counters, causing confusion. Learn how Harness solves this with branch-scoped build numbers.

Abhay Ganvir

March 26, 2026

Time to read

You're tagging Docker images with build numbers.

-Build #47 is your latest production release on main. A developer pushes a hotfix to release-v2.1, that run becomes build #48.

-Another merges to develop, build #49. A week later someone asks: "What build number are we on for production?" You check the registry.

-You see #47, #52, #58, #61 on main. The numbers in between? Scattered across feature branches that may never ship. Your build numbers have stopped telling a useful story.

That's the reality when your CI platform uses a single global counter. Every run, on every branch, increments the same number. For teams using GitFlow, trunk-based development, or any branching strategy, that means gaps, confusion, and versioning that doesn't match how you actually ship.

Branch-scoped versioning has been a long-standing gap in CI tooling.
Build numbers should reflect release reality—not CI activity.
Harness CI makes that possible as a first-class capability: no scripts, no plugins, no workarounds.

TL;DR: Harness CI now supports branch-scoped build sequence IDs via <+pipeline.branchSeqId>.

Each branch gets its own counter. No gaps. No confusion.

Why Global Build Counters Break Down

Most CI platforms give you one incrementing counter per pipeline. Push to main, push to develop, push to a feature branch, same counter. So you get:

Gaps in the sequence for any given branch (e.g. main might have #1, #4, #7).
No clear answer to "what's the latest build on main?"
Semantic versioning and artifact naming that don't line up with branch reality.
Registries and artifact stores full of numbers that don't map to how you release.

This is now built directly into Harness CI as a first-class capability.

What It Feels Like in Practice

Add <+pipeline.branchSeqId> where you need the number—for example, in a Docker build-and-push step:

tags:
  - <+pipeline.branchSeqId>
  - <+codebase.branch>-<+pipeline.branchSeqId>
  - latest

Trigger runs on main, then on develop, then on a feature branch. Each branch gets its own sequence: main might be 1, 2, 3… develop 1, 2, 3… feature/x 1, 2. Your tags become meaningful: main-42, develop-15, feature-auth-3. No more guessing which number belongs to which branch.

What You Get

Per-branch counters – One sequence per pipeline + repo + branch, stored and incremented atomically.
Pipeline expression – <+pipeline.branchSeqId>. Check out Harness variables documentation.
REST API – List sequences for a pipeline, get the current value for a branch/repo, reset a branch counter, or set it to a specific value (for example, after a major release or when migrating from another CI).
Consistent identification – Repo URLs and branch names are normalized (for example, refs/heads/main → main, different URL forms to one canonical host/owner/repo). Same logical branch and repo always share the same counter.
Cleanup – When a pipeline is deleted, its branch-sequence data is removed so you don't leave orphaned counters.

Webhook triggers (push, PR, branch, release) and manual runs (with branch from codebase config) are supported. For tag-only or other runs without branch context, the expression returns null so you can handle that in your pipeline if needed.

How It Works Under the Hood

Branch and repo are taken from the trigger payload when possible (webhooks) or from the pipeline's codebase configuration (for example, manual runs). We normalize them so that the same repo and branch always map to the same logical key: branch names get refs/heads/ (or similar) stripped, and repo URLs are reduced to a canonical form (for example, github.com/org/repo). That way, whether you use https://..., git@..., or different casing, you get one counter per branch.

The counter is stored and updated with an atomic increment. Parallel runs on the same branch still get distinct, sequential numbers. The value is attached to the run's metadata and exposed through the pipeline execution context so <+pipeline.branchSeqId> resolves correctly at runtime.

Putting It to Work

Docker image tagging: use <+pipeline.branchSeqId> and optionally <+codebase.branch>-<+pipeline.branchSeqId> for clear, branch-specific tags.
Helm chart versioning: e.g. --version 1.0.<+pipeline.branchSeqId> --app-version <+codebase.commitSha> so the chart version tracks the build number and the app version tracks the commit.
Release notes or deployment labels: for example, "Release Build #<+pipeline.branchSeqId>" so production and staging each have a clear, branch-local build number.

For teams that need control or migration support, branch sequences are also manageable via API:

# List all branch sequences for a pipeline
GET /pipelines/{pipelineIdentifier}/branch-sequences

# Reset counter for a specific branch
DELETE /pipelines/{pipelineIdentifier}/branch-sequences/branch?branch=main&repoUrl=github.com/org/repo

# Set counter to a specific value (e.g., after major release)
PUT /pipelines/{pipelineIdentifier}/branch-sequences/set?branch=main&repoUrl=github.com/org/repo&sequenceId=100

All of this is gated by the same feature flag so only accounts that have adopted the feature use the APIs.

Try It (With Smart Guardrails)

Enable the feature – Turn on CI_ENABLE_BRANCH_SEQUENCE_ID (Account Settings → Feature Flags, or Reach out to the Harness team).
Use the expression – Add <+pipeline.branchSeqId> in steps, tags, or env vars.
Verify – Run the pipeline on two or three branches and confirm each branch has its own 1, 2, 3…

If branch context isn't available, the expression returns null. Design your pipeline to handle that (for example, skip tagging or use a fallback) for tag builds or edge cases.

✅ Recommendation: Try it with a non-production pipeline first. Confirm that each branch gets its own sequence and that your tags or version strings look right. Then roll it out to production pipelines with confidence.

Feature availability may vary by plan. Check with your Harness account or Harness Developer Hub for your setup.

How Other CI Platforms Handle This (Spoiler: Most Don't)

This isn't just a Harness problem we solved—it's an industry gap. Here's how major CI platforms compare:

Platform	Native Per-Branch Sequence	What you Actually Get
Harness	✅	<+pipeline.branchSeqId> — zero config, atomic counters, full API
Jenkins	~	Multibranch jobs have per-branch numbers, but isolated, each starts at 1, no coordination
GitHub Actions	✗	github.run_number is global. Workaround: third-party actions with Git tag storage (race conditions possible)
GitLab CI	✗	CI_PIPELINE_IID is project-wide, not branch-scoped. Feature requested since 2017, still not implemented
CircleCI	✗	CIRCLE_BUILD_NUM increments per job, not per pipeline (5 jobs = 5 increments per build)
Bitbucket Pipelines	✗	Repository-wide counter by design. No per-branch option
Azure DevOps	✅	counter(prefix, seed) works, but requires YAML setup per use case

Most platforms treat build numbers as an afterthought. Harness CI treats them as a first-class versioning primitive. For teams migrating from Jenkins or Azure DevOps, the model will feel familiar. For teams on GitHub Actions, GitLab, or CircleCI, this fills a gap that previously required external services or custom scripts

What's Coming

This is the first release of branch-scoped sequence IDs. The foundations are in place: per-branch counters, expression support, and APIs. We're not done.

We're listening. If you use this feature and hit rough edges—or have ideas for tag-scoped sequences, dashboard visibility, or trigger conditions—we want to hear about it. Share feedback .

Technical

AI Deployment in Production: Orchestrate LLMs, RAG, Agents

AI deployment in production in 2026: ship LLMs, RAG, and agents safely with CI/CD, semantic testing, guardrails, and progressive delivery.

Chinmay Gaikwad

March 26, 2026

Time to read

For the past few years, the narrative around Artificial Intelligence has been dominated by what I like to call the "magic box" illusion. We assumed that deploying AI simply meant passing a user’s question through an API key to a Large Language Model (LLM) and waiting for a brilliant answer.

Today, we are building systems that can reason, access private databases, utilize tools, and—hopefully—correct their own mistakes. However, the reality is that while AI code generation tools are helping us write more code than ever , we are actually getting worse at shipping it. Google's DORA research found that delivery throughput is decreasing by 1.5% and stability is worsening by 7.5%. Deploying AI is no longer a machine learning experiment; it’s one of the most complex system integration challenges in modern software engineering.

That's why integrated CI/CD is no longer optional for AI deployment—it's the foundation. As teams adopt platforms like Harness Continuous Integration and Harness Continuous Delivery, testing and release orchestration shift from isolated checkpoints to continuous safeguards that protect quality and safety at every layer of the AI stack.

What Is AI Deployment in 2026?

Most definitions of AI deployment are stuck in the "model era." They describe deployment as taking a trained model, wrapping it in an API, and integrating it into a single application to make predictions.

That description is technically accurate—but strategically wrong.

In 2026, AI deployment means:

Integrating a full AI application stack—models, prompts, data pipelines, RAG components, agents, tools, and guardrails—into your production environment so it can safely power real user workflows and business decisions.

You're not just deploying "a model." You are deploying the instructions that define the AI's behavior, the engines (LLMs and other models) that do the reasoning, the data and embeddings that feed those engines context, the RAG and orchestration code that glue everything together, the agents and tools that let AI take actions in your systems, and the guardrails and policies that keep it all safe, compliant, and affordable.

Classic "model deployment" was a single component behind a predictable API. Modern AI deployment is end‑to‑end, cross‑cutting, and deeply entangled with your existing software delivery process.

If you want a great reference for the more traditional view, IBM's overview of model deployment is a good baseline. But in this article, we're going to go beyond that to talk about the compound system you are actually shipping today.

Why AI Deployment Has Become the Bottleneck

The paradox of this moment is simple: coding has sped up, but delivery has slowed down.

AI coding assistants take mere seconds to generate the scaffolding. Platform teams spin up infrastructure on demand. Product leaders are under pressure to add "AI" to every experience. But in many organizations, the actual path from "we built it" to "it's safely in front of customers" is getting more fragile—instead of less.

There are a few reasons for this:

The AI stack is multi‑layered and non‑deterministic. Traditional CI/CD pipelines were designed for deterministic systems: if the code compiles and tests pass, you can be reasonably confident in the behavior. With LLMs and agents, the same input might result in a range of outputs, some acceptable and some dangerous. Testing no longer has a simple pass/fail shape.
Ownership is fractured. MLOps teams worry about training and serving models. Application teams bolt on AI features. Security teams scramble to backfill policies around data access and tool usage. Platform teams are left trying to orchestrate releases that touch all of the above, often without having clear control over any of them.
We've created tool silos instead of integrated delivery. We now talk about MLOps, LLMOps, AgentOps, DevOps, SecOps—as if each deserved its own stack and dashboard—while the actual releases that matter to customers cut straight across those boundaries.

The result is what many teams are feeling right now: shipping AI features feels risky, brittle, and slow, even as the pressure to "move faster" keeps rising.

To fix that, we have to start with the stack itself.

Part 1: Deconstructing the Modern AI Stack

To understand how to deploy AI, you have to stop treating it as a single entity. The modern AI application is a compound system of highly distinct, interdependent layers. If any single component in this stack fails or drifts, the entire application degrades.

1. The Instructions: Prompts as Code

A prompt is no longer just a text string typed into a chat window; it is the source code that dictates the behavior and persona of your application.

The Deployment Reality: Prompts require the same rigor as traditional code—version control, peer review, and automated testing. Because LLMs are sensitive to minute phrasing changes, updating a prompt requires running it against hundreds of baseline test cases to ensure the model doesn't experience "regression" and forget its core instructions.

2. The Engine: Large Language Models (LLMs)

The LLM is the reasoning engine. It has vast general knowledge but zero awareness of your company’s proprietary data.

The Deployment (LLMOps) Reality: Most companies consume these via APIs or host smaller models on cloud infrastructure. The deployment challenge is routing. A sophisticated pipeline will dynamically route simple tasks to faster, cheaper models and complex reasoning tasks to massive, expensive models to optimize both latency and cloud spend, which currently sees significant waste in many organizations.

3. The Fuel: Data and Vector Embeddings

An AI's output is only as reliable as the context it is given. To make an LLM useful, it needs a continuous feed of your company’s internal data.

The Deployment Reality: This requires automated data pipelines that ingest raw information, "chunk" it, and store it in a Vector Database. If the embedding model changes, the entire database must be re-indexed. This data pipeline must be continuously deployed and synced without disrupting the live application.

4. The Architecture: Retrieval-Augmented Generation (RAG)

RAG is not a model; it is a separate software architecture deployed to act as the LLM's research assistant.

The RAG Deployment Reality: When a user asks a question, the RAG code intercepts it, queries the Vector Database, and packages that data into a prompt. Deploying RAG means deploying the integration code that securely manages this retrieval and hand-off process.

5. The Doer: AI Agents

If RAG is a researcher, an AI Agent is an employee. Agents are LLMs given access to external tools. Instead of just answering a question, an agent can formulate a plan, search the web, and execute code.

The Deployment Reality: Moving from linear flows to "Agentic Workflows" introduces massive complexity. You are now deploying systems that iterate and loop. Deploying an agent requires monitoring its step-by-step reasoning traces and ensuring it doesn't get stuck in an infinite loop or misuse its tools.

Part 2: The Guardrails (DevSecOps for AI)

You cannot expose a raw LLM or an autonomous agent to the public, or even to internal employees, without armor. Because AI is non-deterministic, traditional software security falls short. Modern AI deployment requires distinct "Guardrails as Code".

Input Guardrails

Prompt Injection Defenses: Malicious users will attempt to "jailbreak" the AI. Input guardrails use separate, smaller models to intercept adversarial prompts before they reach the core LLM.
PII Scrubbing: Automated systems must redact Personally Identifiable Information (PII) to ensure sensitive data never leaves your secure environment or reaches a third-party LLM provider.

These kinds of controls are a natural fit for policy‑as‑code engines and CI/CD gates. With something like Harness Continuous Delivery & GitOps, you can enforce Open Policy Agent (OPA) rules at deployment time—ensuring that applications with missing or misconfigured input guardrails simply never make it to production.

Output Guardrails

Hallucination Detection: These cross-reference the model’s answer against the retrieved RAG documents to increase confidence by checking support/citations for key claims in your proprietary data.
Schema Enforcement: If your system expects the AI to return data in a strict format, output guardrails will validate the structure and automatically reject or re-prompt the LLM if it outputs unstructured text.

Agentic and Operational Guardrails

Blast-Radius Containment: When deploying agents that can execute actions, strict Role-Based Access Control (RBAC) must be enforced. Agents must operate on the principle of least privilege.
FinOps and Rate Limiting: AI is computationally expensive. Guardrails must enforce strict token-usage tracking and throttle actions to prevent a runaway agent from racking up thousands of dollars in cloud compute costs.

Part 3: The Interplay and the Need for Release Orchestration

Understanding the stack reveals the ultimate challenge: The Cascade Effect. In traditional software, a database error throws a clean error code. In an AI application, a bug in the data pipeline silently ruins everything downstream. This is why deployment cannot be disjointed. It requires rigorous Release Orchestration.

Fuzzy Integration Testing: Traditional CI/CD pipelines rely on exact-match assertions. Because LLMs return varying text, we now require "semantic evaluation"—often using a separate LLM acting as a judge to grade the output based on meaning and accuracy during the automated testing phase.
Progressive Rollout Strategies: Because you cannot perfectly predict AI behavior, orchestration must support Canary releases—rolling out the new model to 5% of users to monitor drift before a full launch.
Synchronizing the Moving Parts: A prompt update might require a different RAG strategy. A new embedding model demands a full database re-indexing. Release orchestration ensures that when one layer is updated, the corresponding dependencies are automatically tested and deployed in lockstep.

How to Deploy AI in Production (A Practical Pipeline)

Version prompts/configs/policies as code
Build eval suite (golden set + safety tests)
CI: semantic eval + regression thresholds
Security gates: PII redaction + prompt injection tests
CD: canary rollout for prompt/model/RAG changes
Observability: quality + safety + cost signals
Rollback rules tied to metrics
Post-deploy review and dataset refresh cadence

The Bottom Line: Orchestrate or Stall

For years, we've been obsessed with specialized silos: MLOps, LLMOps, AgentOps. But a vital realization is sweeping the enterprise: the time of siloed, specialized AI operations tools is coming to an end.

The future belongs to unified release management. The organizations that succeed will not be the ones with the smartest standalone AI models, but the ones who master the orchestration required to deploy and evolve those models, alongside everything else they ship, safely, efficiently, and continuously.

If you want a platform that brings semantic testing, progressive rollouts, and coordinated AI releases into your day-to-day workflows, Harness Continuous Integration and Harness Continuous Delivery were built for this.

‍

Key Takeaways:

AI deployment is deploying a stack, not a model.
Treat prompts, evals, policies, and configs as code.
Use semantic evaluation plus standard CI tests.
Use progressive delivery (canaries) for models/prompt/RAG changes.
Orchestrate dependencies (prompt ↔ RAG ↔ embeddings ↔ guardrails) to prevent silent regressions.

‍

AI Deployment: Frequently Asked Questions (FAQs)

What is AI deployment?
AI deployment is the process of integrating AI systems, models, prompts, data pipelines, RAG architectures, agents, tools, and guardrails, into production environments so they can safely power real applications and business workflows.

How is AI deployment different from traditional model deployment?
Traditional model deployment focuses on serving a single model behind an API. Modern AI deployment involves a multi‑layer stack: instructions, engines, context, retrieval, agents, and policies. Failures are more likely to be silent regressions or unsafe behaviors than obvious crashes, which is why you need semantic testing, guardrails, and release orchestration.

How do you deploy AI safely in production?
Safe AI deployment starts with treating prompts and configurations as code, embedding guardrails at input, output, and action levels, and using semantic evaluation and progressive rollout strategies. It also requires immutable logging and audit trails so you can trace decisions back to specific versions of your AI stack. Combining CI for semantic tests with CD for orchestrated releases is the practical path to safety.

What tools are used for AI deployment?
Teams typically use a mix of LLM providers or model‑serving platforms, vector databases, observability tools, and CI/CD systems for orchestrating releases. On top of that, they add policy engines and specialized evaluation frameworks. The critical shift is moving from isolated "AI tools" to integrated pipelines that tie everything together.

How do canary releases work for AI models and prompts?
With canary releases, you send a small portion of traffic to the new behavior, a new model, prompt, or RAG strategy, while most users continue on the old path. You observe semantic quality, safety signals, and performance. If the canary behaves well, you gradually increase its share. If it misbehaves, you automatically roll back to the previous version.

DevOps & Automation Blogs

Featured Blogs

Harness Ships Five Capabilities to Power Confident Releases at AI Speed

What Harness Is Shipping

Coordinate multi-team releases without the war room

Know when to stop — automatically

Ship code and schema changes together

Roll out features gradually, measure what actually happens

From Deployment to Verified Outcome

Release Becomes a System, Not a Scramble

When Faster Code Starts to Break the Delivery System

The Emerging “Velocity Paradox”

Why the Delivery System Is Straining

What Organizations Should Do Next

1. Standardize delivery foundations

2. Automate quality and security checks earlier

3. Build guardrails into the release process

4. Remember measurement, not just automation

The Next Phase of AI in Software Delivery

Reimagining Artifact Management for DevSecOps: Harness Artifact Registry GA

A Startup Inside Harness

Why This Matters: The Registry as a Control Point

Introducing Dependency Firewall: Blocking Risk at Ingest

The Future of Artifact Management is here

Latest Blogs

The pipeline that never reached production

How template-driven CD prevents governance drift

The subtle risk in fast CI/CD platforms

The architecture shift: separate authoring from execution

Architecture diagram

Guardrail #1: production pipelines must use templates

Example policy

Guardrail #2: governance starts in the non-prod organization

Example policy

The source of truth: Git

Governance flow

The promotion workflow

The day the model proved its value

Why this model works

The lesson for platform teams

Final takeaway

Introducing Zero Trust Architecture for Software Delivery

The Challenge: When Permissions Aren't Enough

Introducing Harness Zero Trust

How It Works: The Final Line of Defense

Why This Matters for Enterprise DevSecOps

The Takeaway

Defeating Context Rot: Mastering the Flow of AI Sessions

Context Rot Is Not a Bug — It’s a Property

Why Long Sessions Break Down

The Link Between Context Rot and Hallucination

Sessions Are Not Conversations

The Plan → Execute → Reset Discipline

Planning before execution

Stepwise execution

Resetting the session

Meta-Prompting: Forcing the Model to Think First

Checkpoints: Turning Drift into Signal

The Connection to Part 1

Why This Matters Now

Before You Scale, You Stabilize

Operationalizing Production Data Testing with Harness Database DevOps

A Pipeline-Driven Reference Workflow

Stage 1: Environment Provisioning and Data Preparation

Stage 2: Schema Application Within a Harness Pipeline

Stage 3: Automated Rollback and Undo Migration Testing

Stage 4: Comparison Against Baseline and Validation

How This Aligns with Harness Capabilities?

Strategic Outcomes for Engineering Teams

Conclusion

The Dangerous Myth: “We Have SCA, So We’re Covered”

Where Modern Supply Chains Actually Get Compromised

Artifact Integrity and Registry Blind Spots

CI/CD Pipelines as Privileged Infrastructure

Containers and Upstream Image Poisoning

Vendor and Third-Party Integration Risk

Build Systems Are the New Trust Boundary

Why Pipelines Must Be Treated as Untrusted

AI Is Expanding the Supply Chain Attack Surface

A Practical Framework for End-to-End Supply Chain Security