Harness Blog

• Self-service sandbox environments powered by an Internal Developer Portal cut through provisioning bottlenecks, giving developers access to secure, policy-driven test environments in minutes — not days.
• Automated sandbox environment provisioning with built-in guardrails keeps governance consistent, reduces security risk, and frees platform teams from endless manual requests.
• Tracking metrics like time-to-first-environment, ticket reduction, and sandbox utilization helps you prove the ROI of your platform investments and ship software faster (and safer) at scale.

Here's a scenario that probably sounds familiar: a developer needs a sandbox environment to test something. They file a ticket. Then they wait. And wait. Maybe a day goes by, maybe three. Meanwhile, your platform team is buried in provisioning requests, and somewhere, someone has already spun up an unsanctioned workaround that bypasses every governance policy you've put in place.

It's a lose-lose. Developers lose velocity, platform teams lose their sanity, and security gaps quietly multiply.

An Internal Developer Portal flips this whole dynamic. Instead of ticket queues and manual provisioning, you get a self-service sandbox environment automation with guardrails baked in. Developers get instant access to governed, policy-driven environments. Platform engineers get visibility and control — without becoming a bottleneck.

In this post, we'll walk through how to build sandbox automation that actually works, set governance that holds up at scale, accelerate developer onboarding, and measure ROI in a way that makes leadership pay attention.

Harness Internal Developer Portal brings enterprise-grade orchestration together with developer-friendly self-service — built on the Backstage framework with the security, scalability, and governance that enterprises actually need.

Why Sandbox Environments Are the Fastest Path to Safer, Faster Delivery

Let's be real: if your developers are waiting days for a test environment while your security team is blocking deployments they can't validate, something is broken. Teams end up either skipping testing altogether or delaying releases — neither of which is a good outcome. A solid sandbox environment strategy breaks this cycle by giving teams secure, isolated spaces to validate changes without compromising governance or speed.

Isolation Reduces Blast Radius Through Disposable Infrastructure

The best way to think about sandboxes? Treat them as disposable. Each sandbox environment gets the minimum privileges needed for the task at hand — whether that's validating a database migration, testing a third-party API integration, or experimenting with a new service configuration.

If something breaks, you just delete the sandbox environment and start fresh. No incident reports. No painful rollback procedures. No 2 a.m. pages to the on-call engineer. Teams that adopt this pattern tend to catch configuration errors much earlier in the development cycle.

Security-by-Default with Controlled Testing Spaces

Think of sandbox environments as your first line of defense. When you're dealing with untrusted code, unfamiliar dependencies, or configuration changes you're not 100% sure about, run them in an isolated sandbox first. Validate the behavior, confirm it's clean, and only then promote the changes to staging or production.

This pattern is incredibly effective at catching supply chain issues, configuration drift, and integration failures before they ever touch your shared systems. It shifts security left in a way that's practical, not just aspirational.

Enterprise Patterns That Balance Fidelity, Cost, and Speed

Not every test needs a full production replica. A smart sandbox environment strategy uses a tiered approach: lightweight developer sandboxes for day-to-day feature work, partial-copy environments for integration testing with realistic (but masked) data, and full-copy sandboxes reserved for final validation before release.

Add in ephemeral PR environments that spin up automatically when a pull request is opened and tear themselves down after a set time window, and you've got realistic testing without persistent infrastructure costs or manual provisioning overhead.

Automating Sandbox Environment Provisioning in Your Internal Developer Portal

When sandbox environment requests take days instead of minutes, you're not just slowing down development — you're creating a governance problem that only gets worse as you scale. Automating provisioning through your IDP eliminates this friction while keeping the controls you need.

Here's what that looks like in practice:

• Golden path templates that bundle VPC isolation, namespace creation, data seeding, and data masking into standardized, repeatable deployments — so sensitive information is protected by default, not as an afterthought.
• Policy as Code enforcement (using tools like OPA) for automatic approval workflows, resource quotas, and RBAC — no manual reviews required.
• Fast spin-up times for ephemeral PR environments, leveraging pre-warmed infrastructure and cached images.
• Automatic teardown with sensible TTLs to prevent cost drift and resource sprawl.
• CI/CD integration, so sandbox environment automation triggers naturally from pull requests and deployment events.

This approach transforms your IDP from a service catalog into a genuine self-service platform — one that scales governance instead of bypassing it.

Choosing the Right Sandbox Environment Type for Your Workflows

Not all sandbox environments are created equal. Picking the wrong type wastes resources and creates either security gaps or unnecessary developer friction. Knowing which type fits each workflow helps platform teams design self-service templates that match what developers actually need.

Developer sandboxes prioritize speed and productivity. Security detonation environments prioritize total isolation, even at the cost of convenience. Most teams will use a mix across their workflows.

Onboarding at Scale: Self-Service Sandbox Environments from Day One

Here's something that frustrates every engineering leader: a new hire joins your team, and it takes them three days just to get a working development environment. That's three days of lost productivity, plus a terrible first impression of your engineering culture.

With an Internal Developer Portal, you can bundle everything a new developer needs — sandbox environment provisioning, documentation, credentials, and golden path templates — into a single catalog request. Day one, they click a button, and they're up and running. No more Slack threads asking "where's my environment?"

Use Scorecards — a native feature in Harness IDP — to track the metrics that actually matter: time-to-first-PR-environment, sandbox environment mean time to recovery, and reuse rates. When self-service becomes the default, ticket provisioning drops dramatically.

Operationalize Sandbox Environments with Harness Internal Developer Portal

Scaling sandbox environments across your organization doesn't mean you need to double your platform team or accept governance trade-offs. Harness IDP extends the Backstage framework with the enterprise capabilities that self-managed Backstage doesn't offer out of the box — fine-grained RBAC, hierarchical organization, native Scorecards tied to your KPIs, access to the 200+ Backstage plugin ecosystem, and environment management that scales with your teams.

Ready to ship governed, self-service sandbox environments that actually reduce toil? Try Harness Internal Developer Portal and see how fast your teams move when the friction disappears.

Sandbox Environment FAQ for Platform Engineering Leaders

These are the questions that come up most often when platform leaders are planning sandbox environment rollouts or making the case to leadership.

How do sandbox environments improve software delivery security?

Sandbox environments create isolated failure boundaries where you can test risky changes, third-party integrations, and security patches without touching shared systems. Teams validate behavior in production-like conditions, catching issues before they reach staging or production — which means fewer incidents and faster recovery when things do go wrong.

What metrics should platform teams track to measure sandbox environment success?

Focus on time-to-first-environment for new developers, sandbox environment utilization rates, and ticket reduction percentages. Also track cost per sandbox hour and cleanup effectiveness. Developer satisfaction scores and time saved on provisioning requests are the numbers that help you demonstrate platform ROI to leadership and justify continued investment.

How do you prevent sandbox environment sprawl and cost overruns?

The biggest culprit is sandbox environments that get spun up and never torn down. Set TTL-based auto-teardown policies so environments self-destruct after a defined window — a few hours for ephemeral PR environments, a couple of weeks for longer-lived developer sandboxes. Pair that with resource quotas enforced through policy-as-code, and you've got cost controls that don't depend on anyone remembering to clean up after themselves.

Can sandbox environments work across multi-cloud or hybrid infrastructure setups?

Yes, and this is where a platform approach really pays off. Instead of building separate provisioning workflows for each cloud provider, golden path templates in your IDP abstract the infrastructure layer. Developers request a sandbox environment from a single catalog — whether it lands on AWS, Azure, GCP, or an on-prem cluster is handled by the template logic and your team's policies. Harness IDP supports multi-cloud and multi-region deployments natively, so you're not stitching together one-off scripts for each environment.

On March 19th, the risks of running open execution pipelines — where what code runs in your CI/CD environment is largely uncontrolled — went from theoretical to catastrophic.

A threat actor known as TeamPCP compromised the GitHub Actions supply chain at a scale we haven't seen before (tracked as CVE-2026-33634, CVSS 9.4). They compromised Trivy, the most widely used vulnerability scanner in the cloud-native ecosystem, and turned it into a credential-harvesting tool that ran inside victims' own pipelines.

Between March 19 and March 24, 2026, organizations running affected tag-based GitHub Actions references were sending their AWS tokens, SSH keys, and Kubernetes secrets directly to the attacker. SANS Institute estimates over 10,000 CI/CD workflows were directly affected. According to multiple security research firms, the downstream exposure extends to tens of thousands of repositories and hundreds of thousands of accounts.

Five ecosystems. Five days. One stolen Personal Access Token.

This is a fundamental failure of the open execution pipeline model — where what runs in your pipeline is determined by external references to public repositories, mutable version tags, and third-party code that executes with full privileges. GitHub Actions is the most prominent implementation. 

The alternative, governed execution pipelines, where what runs is controlled through policy gates, customer-owned infrastructure, scoped credentials, and immutable references, is the model we designed Harness around years ago, precisely because we saw this class of attack coming.

Part I: The Long Road to TeamPCP (2025–2026)

TeamPCP wasn't an anomaly; it was the inevitable conclusion of an eighteen-month escalation in CI/CD attack tactics.

1. The tj-actions Proof of Concept (March 2025)

CVE-2025-30066. Attackers compromised a PAT from an upstream dependency (reviewdog/action-setup) and force-pushed malicious code to every single version tag of tj-actions/changed-files. 23,000 repositories were exposed. The attack was later connected to a targeted campaign against Coinbase. CISA issued a formal advisory.

This proved that the industry's reliance on mutable tags (like @v2) was a serious structural vulnerability. According to Wiz, only 3.9% of repositories pin to immutable SHAs. The other 96% are trusting whoever owns the tag today.

2. The Shai-Hulud Worm (Sept–Nov 2025)

The first self-replicating worm in the CI/CD ecosystem. Shai-Hulud 2.0 backdoored 796 npm packages representing over 20 million weekly downloads — including packages from Zapier, PostHog, and Postman. 

It used TruffleHog to harvest 800+ credential types, registered compromised machines as self-hosted GitHub runners named SHA1HULUD for persistent C2 over github.com, and built a distributed token-sharing network where compromised machines could replace each other's expired credentials.

PostHog's candid post-mortem revealed that attackers stole their GitHub bot's PAT via a pull_request_target workflow exploit, then used it to steal npm publishing tokens from CI runner secrets. Their admission that this kind of attack "simply wasn't something we'd prepared for" reflects the industry-wide gap between application security and CI/CD security maturity. CISA issued another formal advisory.

3. The Trivy Compromise (March 19, 2026)

TeamPCP went after the security tools themselves.

They exploited a misconfigured GitHub Actions workflow to steal a PAT from Aqua Security's aqua-bot service account. Aqua detected the breach and initiated credential rotation — but reporting suggests the rotation did not fully cut off attacker access. TeamPCP appears to have retained or regained access to Trivy's release infrastructure, enabling the March 19 attack weeks after initial detection.

On March 19, they force-pushed a malicious "Cloud Stealer" to 76 of 77 version tags in trivy-action and all 7 tags in setup-trivy. Simultaneously, they published an infected Trivy binary (v0.69.4) to GitHub Releases and Docker Hub. Every pipeline referencing those tags by name started executing the attacker's code on its next run. No visible change to the release page. No notification. No diff to review.

Part II: Inside the "Cloud Stealer" Tradecraft

TeamPCP's payload was purpose-built for CI/CD runner environments:

Memory Scraping. It read /proc/*/mem to extract decrypted secrets held in RAM. GitHub's log-masking can't hide what's in process memory.

Cloud Metadata Harvesting. It queried the AWS Instance Metadata Service (IMDS) at 169.254.169.254, pivoting from "build job" to full IAM role access in the cloud.

Filesystem Sweep. It searched over 50 specific paths — .env files, .aws/credentials, .kube/config, SSH keys, GPG keys, Docker configs, database connection strings, and cryptocurrency wallet keys.

Encrypted Exfiltration. All data was bundled into tpcp.tar.gz, encrypted with AES-256 and RSA-4096, and sent to typosquatted domains like scan.aquasecurtiy[.]org (note the "tiy"). These domains returned clean verdicts from threat intelligence feeds during the attack. As a fallback, the stealer created public GitHub repos named tpcp-docs under the victim's own account.

The malicious payload executed before the legitimate Trivy scan. Pipelines appeared to work normally. CrowdStrike noted: "To an operator reviewing workflow logs, the step appears to have completed successfully."

The Five-Day Cascade

Sysdig observed that the vendor-specific typosquat domains were a deliberate deception — an analyst reviewing CI/CD logs would see traffic to what appears to be the vendor's own domain. 

It took Aqua five days to fully evict the attacker, during which TeamPCP pushed additional malicious Docker images (v0.69.5 and v0.69.6).

Part III: Why Open Execution Pipelines Break at Scale

Why did this work so well? Because GitHub Actions is the leading example of an open execution pipeline — where what code runs in your pipeline is determined by external references that anyone can modify.

This trust problem isn't new. Jenkins had a similar issue with plugins. Third-party code ran with full process privileges. But Jenkins ran inside your firewall; exfiltrating data required getting past your network perimeter. 

GitHub Actions took the same open execution approach but moved execution to cloud-hosted runners with broad internet egress, making exfiltration trivially easy. TeamPCP's Cloud Stealer just needed to make an HTTPS POST to an external domain, which runners are designed to do freely. 

Here are a few reasons why open execution pipelines break at scale:

Mutable Trust. When you use @v2, you are trusting a pointer, not a piece of code. Tags can be silently redirected by anyone with write access. TeamPCP rewrote 76 tags in a single operation. 96% of the ecosystem is exposed.

Flat Privileges. Third-party Actions run with the same permissions as your code. No sandbox. No permission isolation. This is why TeamPCP targeted security scanners — tools that by design have elevated access to your pipeline infrastructure. The attacker doesn't need to break in. The workflow invites them in.

Secret Sprawl. Secrets are typically injected into the runner's environment or process memory during job execution, where they remain accessible for the job's duration. TeamPCP's /proc/*/mem scraper didn't need any special privilege. It just needed to be running on the same machine.

Unbounded Credential Cascades. There is no architectural boundary that stops a credential stolen in one context from unlocking another. TeamPCP proved this definitively: Trivy → Checkmarx → LiteLLM → AI API keys across thousands of enterprises. One PAT, five ecosystems.

Part IV: Governed Execution Pipelines — Three Structural Walls

Harness CI/CD pipelines are built as governed execution pipelines — where what runs is controlled through customer-owned infrastructure, policy gates, scoped credentials, immutable references, and explicit trust boundaries. At its core is the Delegate — a lightweight worker process that runs inside your infrastructure (your VPC, your Kubernetes cluster), executes tasks locally, and communicates with the Harness control plane via outbound-only connections.

When we designed this architecture, we assumed the execution plane would become the primary target in the enterprise. If TeamPCP tried to attack a Harness-powered environment, they would hit three architectural walls.

Wall 1: The Airlock (Outbound-Only, Egress-Filtered Execution)

The Architecture. 

The Delegate lives inside your VPC or cluster. It communicates with our SaaS control plane via outbound-only HTTPS/WSS. No inbound ports are opened.

The Defense. 

You control the firewall. Allowlist app.harness.io and the specific endpoints your pipelines need, deny everything else. TeamPCP's exfiltration to typosquat domains would fail at the network layer — not because of a detection rule, but because the path doesn't exist. Both typosquat domains returned clean verdicts from threat intel feeds. Egress filtering by allowlist is more reliable than detection by reputation.

Wall 2: The Vault (Secret Isolation at the Source)

The Architecture. 

Rather than bulk-injecting secrets as flat environment variables at job start, Harness can resolve secrets at runtime through your secret manager — HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager, Azure Key Vault — via the Delegate, inside your network. Harness SaaS stores encrypted references and metadata, not plaintext secret values.

The Defense. 

TeamPCP's Cloud Stealer worked because in an open execution pipeline, secrets are typically injected into the runner's process memory where they remain accessible for the job's duration. In a governed execution pipeline, this exposure is structurally reduced: secrets can be resolved from your controlled vault at the point they're needed, rather than broadcast as environment variables to every step in the pipeline.

An important caveat: Vault-based resolution alone doesn't eliminate runtime exfiltration. Once a secret is resolved and passed to a step that legitimately needs it — say, an npm token during npm publish — that secret exists in the step's runtime. If malicious code is executing in that same context (for example, a tampered package.json that exfiltrates credentials during npm run test), the secret is exposed regardless of where it came from. This is why the three walls work as a system: Wall 2 reduces the surface of secret exposure, Wall 1 blocks the exfiltration path, and (as we'll see) Wall 3 limits the blast radius to the scoped environment. No single wall is sufficient on its own.

To further strengthen how pipelines use secrets, leverage ephemeral credentials — AWS STS temporary tokens, Vault dynamic secrets, or GCP short-lived service account tokens — that auto-expire after a defined window, often minutes. Even if TeamPCP’s memory scraper extracted an ephemeral credential, it likely would have expired before the attacker could pivot to the next target.

Wall 3: The Dead End (Environment-Scoped Isolation)

The Architecture. 

Harness supports environment-scoped delegates as a core architecture pattern. Your "Dev" scanner delegate runs in a different cluster, with different network boundaries and different credentials, than your "Prod" deployment delegate.

The Defense. 

The credential cascade that defined TeamPCP hits a dead end. Stolen Dev credentials cannot reach Production publishing gates or AI API keys, because those credentials live in a different vault, resolved by a different delegate, in a different network segment. If the Trivy compromise only yielded credentials scoped to a dev environment, the attack stops at phase one.

Beyond the walls, governed execution pipelines provide additional structural controls:

• No default marketplace dependency: In GitHub Actions, the primary building block is a reference to an external Action in a public repository. In Harness, the primary building blocks are native pipeline steps that don't reference external Git repos. Harness does support running GitHub Actions as steps for teams that need compatibility, but external Actions are an optional path — not the default architecture.

• Reduced tooling and attack surface. Customers can use minimal delegate images with a significantly reduced binary footprint and least-privilege Kubernetes roles to restrict available tooling. TeamPCP's kubectl get secrets --all-namespaces would require tooling and permissions that a properly hardened delegate environment wouldn't provide.

The Comparison

What TeamPCP Actually Exploited — Mapped to Harness Defenses

Part V: The Nuance — Governed Doesn't Mean Automatically Safe

Architecture is a foundation, not a guarantee. Governed execution pipelines are materially safer against this class of attack, but you can still create avoidable risk by running unvetted containers on delegates, skipping egress filtering, using the same delegate across dev and prod, granting overly broad cloud access, or exposing excessive secrets to jobs that don't need them, or using long-lived static credentials when ephemeral alternatives exist.

I am not claiming that Harness is safe and GitHub Actions is unsafe. That would be too simplistic. 

What I am claiming is that governed execution pipelines — where what runs is controlled through policy gates, customer-owned infrastructure, scoped credentials, and immutable references — are a materially safer foundation than open execution pipelines. We designed Harness as our implementation of a governed execution pipeline. But architecture is a starting point — you still have to operate it well.

Part VI: The Strategic Bottom Line — From Open to Governed

As we enter the era of Agentic AI — where AI is generating pipelines, suggesting dependencies, and submitting pull requests at machine speed — we can no longer rely on human review to catch a malicious tag in an AI-generated PR.

But there's a more fundamental shift: AI agents will become the primary actors inside CI/CD pipelines. Not just generating code — autonomously executing tasks, selecting dependencies, making deployment decisions, remediating incidents.

Now imagine an AI agent in an open execution pipeline — downloaded from a public marketplace, referenced by a mutable tag, executing with full privileges, making dynamic runtime decisions you didn't define. It has access to your secrets, your cloud credentials, and your deployment infrastructure. Unlike a static script, an agent makes decisions at runtime — fetching resources, calling APIs, modifying files.

If TeamPCP showed us what happens when a static scanner is compromised, imagine what happens when an autonomous AI agent is compromised — or simply makes a decision you didn't anticipate.

This is why governed execution pipelines aren't just a security improvement — they're an architectural prerequisite for the AI era. In a governed pipeline, even an AI agent operates within structural boundaries: it runs on infrastructure you control, accesses only scoped secrets, has restricted egress, and its actions are audited. The agent may be autonomous, but the pipeline constrains what it can reach.

The questions every engineering leader should be asking:

1. Is my pipeline open or governed? Do I control what code executes, or is it determined by external references I don't audit?
2. Where does execution happen? In infrastructure I control, or in an environment assembled from public dependencies?
3. Who controls the network boundary? My security team, or the maintainer of a third-party Action?
4. Are secrets sitting in runner memory or safely in my Vault?
5. What stops a credential cascade from crossing environment boundaries?
6. When AI agents start running autonomously in my pipelines, what structural boundaries constrain them?

What You Should Do Right Now

If you use Trivy, Checkmarx, or LiteLLM: 

• Assume compromise if you ran any of these tools between March 19–25. Rotate all credentials accessible to affected CI/CD runners. Check your GitHub org for repos named tpcp-docs — their presence indicates successful exfiltration. 
• Block scan.aquasecurtiy[.]org, checkmarx[.]zone, and models.litellm[.]cloud at the network level.
• Update to safe versions: check with the providers of each impacted package and update the scanner and actions.

If you use GitHub Actions: 

• Pin every Action to an immutable commit SHA. Today. 
  
  • Commit SHAs are not sufficient. They pin to the right commit, but do not guarantee that the right repository is selected.
• Add provenance verification: To close the gap left by SHA pinning alone, verify the Action’s source and publisher, restrict which external Actions are allowed, and prefer artifacts with verifiable provenance or attestations.
• Audit workflows for pull_request_target triggers. 
• Enforce Least Privilege on GitHub Tokens: Audit every Personal Access Token and GitHub App permission. If it’s not scoped to the specific repository and the specific task (e.g., "contents: read"), it is a liability.
• Monitor egress for unexpected destinations: Domain reputation alone is insufficient.

For the longer term: 

• Evaluate whether your CI/CD pipelines are open or governed. If production credentials flow through your pipelines, you need a governed execution pipeline where you control the infrastructure, the network boundary, the secret resolution, and the audit trail.
• Establish policies: Implement platform-wide automated governance to enforce SHAs and least-privilege token usage programmatically through systems like OPA.

The Responsibility We Share

I'm writing this as the CEO of a company that competes with GitHub in the CI/CD space. I want to be transparent about that.

But I'm also writing this as someone who has spent two decades building infrastructure software and who saw this threat model coming. When we designed Harness, the open execution pipeline model had already evolved from Jenkins plugins to GitHub Actions — each generation making it easier for third-party code to run with full privileges and, by moving execution further from the customer's network perimeter, making exfiltration easier. We deliberately chose to build governed execution pipelines instead.

The TeamPCP campaign didn't teach us anything new about the risk. What it did was make the difference between open and governed execution impossible for the rest of the industry to ignore.

Open source security tools are invaluable. The developers and companies who build them — including Aqua Security and Checkmarx — are doing essential work. The problem isn't the tools. The problem is running them inside open execution pipelines where third-party code has full privileges, secrets sit in memory, and exfiltration faces no structural barrier.

If you want to explore how the delegate architecture works in practice, we're here to show you. But more importantly, regardless of what platform you choose, please take these structural questions seriously. The next TeamPCP is already studying the credential graph.

‍

Taking a look back over the last ten years in enterprise technology, paradigm shifts are occurring more frequently. For example, the maturity of DevOps/Platform Engineering and Cloud Native infrastructure has occurred. The new frontier depending where you are in adoption is AI. As your adoption and maturity curve progress, operationalizing these paradigms become important. Many sophisticated firms that have to manage and even innovate paradigms that are possibly not core to their business model look to Product Portfolio Management aka PPM to manage the business aspect of the paradigm.

A core pillar to PPM is resource management or the management of skillsets across the portfolio. We looked inwards to our own Professional Service team and ran a Job Task Analysis aka JTA to analyze how Harness itself built a portfolio of implementation experts. 

For organizations/firms looking not only to scale Harness but potentially other platforms or paradigms, can use the Harness Implementation Job Task Analysis as one framework to use and apply to your specific need. 

Remember Your Org’s DevOps Adoption? 

In 2026, it is exciting to say that DevOps is mature. Most in industry agree today to the pillars and practices of what DevOps has been trying to achieve are normal for most engineering orgs. If you turn the clock back 15 years ago, DevOps was still very much emerging as a paradigm. The trio of people, process, and technology followed as the movement became more mature. 

In an early episode of ShipTalk, we talked to Nidhi Allipuram, who at the time was leading a DevOps Org at a large insurance company. In the episode we talked about how she went from 0-1 in scaling a DevOps idea to building a DevOps Org.  Like any technology adoption with a new or fast moving paradigm (think of AI today), there are business considerations that typically follow the maturity curve. The process she followed was incremental starting to gather internal expertise then having to answer “is this problem even worth solving”. From there once momentum is made, scale will follow. 

The episode was great timing, for myself I was becoming a new first-time manager here at Harness and Harness itself was turning into a multi-module platform. I was really curious about how she load-balanced skills on her team to continue to expand into an evolving paradigm. Her advice was solid that you do have to balance the team to grow complementary skillsets and be able to keep up with their internal customer demand. This is exactly what resource management in a PPM based discipline/org would strive for. The Job Task Analysis that was just concluded can be used in resource management when looking to scale the program/portfolio. 

Harness Implementation Job Task Analysis - Hard and Soft Skills

Looking at the Implementation Job Task Analysis and how Harness itself scales our Professional Services Practice, there is a needed mixture of both hard and soft skills. The hard skills are vertical skills in the tooling/platforms/ecosystem that Harness participates in. Second and just as important are the soft skills around problem solving and requirements gathering / stakeholder management. The hard skills would answer the “how” and “where” type of questions. The soft skills will answer just as importantly the “why” and “what” type of questions.  As Harness is a platform of outcomes, solving for both hard and soft skills is important. 

Extrapolating this to building a portfolio or program in evolving or even mature technology domains, the soft skills bolster the initial organic question “is this problem even worth solving”. In technology there is rarely a singular big-bang type of innovation occurrence. A good amount of innovation happens incrementally and with a solid team/consensus driven approach. Getting consensus can also be a challenge and a typical approach is requirements/stakeholder management gathering inputs from many points-of-view and blending them for trade-offs. 

Distilling down the findings in the Implementation Job Task Analysis, for highly technical resources stakeholder / requirements management can be challenging as there is a lot of grey area. For example in the Site Reliability Engineering aka SRE world, defining your SLIs/SLOs is one set of challenges but having to get consensus to renegotiate them is a different order of problem. As we look towards AI, the fundamentals do not go away.  

Well What About AI? 

AI is certainly a rapidly evolving paradigm that is rapidly changing how we use technology. In a recent Cloud Native Podcast where the topic was Generative AI in Platform Engineering. AI, especially how it is used in Platform Engineering, is still early in the adoption curve. In 2026, The Linux Foundation will have its first AI centric conference with AGNTCon + MCPCon which is telling that maturity will eventually arrive. With items that are either early or late maturity, fundamentals do not go away. 

On the Cloud Native Podcast, there was a conversation around how Kubernetes was during the first KubeCon in 2015 and how AGNTCon + MCPCon is a telling similar signal in 2026. The panel in the podcast reflected on their own K8s journey and early questions around innovation vs control/operationalization came up when looking towards AI; similar concerns when looking at K8s early on. 

Crucial to navigating the radio dials of innovation vs control is again the fundamentals of soft skills getting stakeholder or internal customer alignment which is by incremental consensus. In the Implementation Job Task Analysis, the people who are agents of change in software delivery need to blend both their hard and soft skills on a daily basis. These very people are available to help further your innovation goals with top notch software delivery via the Harness Platform. 

Partner with Harness for New Paradigm Expertise

As you start to embrace new paradigms and the Harness Platform, we are here to help. Harness has been at the forefront of the Cloud Native and now AI-Centric software delivery worlds. Our platform continues to evolve to meet these new and similar challenges in the technology adoption curve. Harness Professional Services are the people helping your organization making these new paradigms become reality.  Feel free to take a look at our Services Offerings which includes consulting/coaching to training. We are excited to share how we scale our own internal skills to help our customers/partners/and public continue on to better the software delivery craft.

In today's always-on digital economy, a single slow page or unexpected crash during peak traffic can cost businesses thousands or even millions of dollars in lost revenue, damaged reputation, and frustrated customers. Imagine Black Friday shoppers abandoning carts because your e-commerce site buckles under load, or a SaaS platform going down during a major product launch. This is where load testing becomes non-negotiable.

Load testing simulates real-world user traffic to ensure your applications, websites, and APIs stay fast, stable, and scalable. It's a cornerstone of performance testing that helps teams catch bottlenecks early, validate SLAs, and build resilient systems.

If you're searching for a complete load testing guide, what is load testing, or how to perform load testing, you're in the right place. This beginner-friendly introduction covers everything from the basics to best practices, with practical steps anyone can follow.

What Is Load Testing?

Load testing is a type of performance testing that evaluates how your system behaves under expected (and sometimes peak) user loads. It simulates concurrent users, requests, or transactions to measure key metrics such as Response times (average, p95, p99), Throughput (requests per second), Error rates, Resource utilization (CPU, memory, database connections), Latency and scalability.


Unlike unit or functional tests that check "does it work?", load testing answers: "How does it perform when 1,000 (or 100,000) people use it at once?"

Done early and often, load testing reduces risk across the lifecycle. It confirms capacity assumptions, reveals infrastructure limits, and proves that recent changes haven’t slowed critical paths. The result is fewer production incidents and fewer late-night fire drills.

Key terminology to anchor your approach:

• Response time: End-to-end time to complete a request.
• Latency: The network delay portion of response time.
• Throughput: Requests or transactions per second.
• Concurrency: The number of simultaneous users or sessions.
• Virtual users: Emulated users that generate traffic.
• Think time: Pauses between actions to mimic real behavior.
• Error rate: Percentage of requests that fail.
• Saturation point: Load level where performance drops sharply.
• Service-level objective (SLO): A target like p95 response time under 500 ms.

Why Load Testing Matters?

Effective load testing quantifies capacity, validates autoscaling, and uncovers issues like thread pool starvation, database contention, cache thrash, and third-party limits. With data in hand, you can tune connection pools, garbage collection, caching tiers, and CDN strategies so the app stays fast when it counts.

Skipping load testing is like launching a rocket without wind-tunnel tests, risky and expensive. Here's why it's essential:

• Prevents costly downtime: Unplanned outages average $5,600–$14,000+ per minute for enterprises, with some large companies facing $1M+ per hour.
• Improves user experience and conversions: Even 100ms of added latency can reduce sales by 1% and 1-second delays can drop conversions by 7% for businesses.
• Validates scalability and auto-scaling: Especially critical for cloud-native apps on Kubernetes or AWS.
• Saves money long-term: Catch issues in staging instead of production; reduce infrastructure over-provisioning.
• Boosts confidence for high-traffic events: Product launches, sales, seasonal peaks.

Investing in load testing upfront keeps teams focused on building, not firefighting. Many major outages (think major retailers or banking apps) trace back to untested load scenarios. Load testing helps you ship with confidence.

Types of Load Testing

No single test answers every question. Combine these to get a complete picture:

• Functional load testing: Confirms core flows (checkout, login, search) behave correctly at volume. Accuracy at scale is the priority.
• Performance load testing: Measures response times, throughput, and error rates across varying loads to verify SLOs and locate degradation points.
• Endurance (soak) testing: Runs for hours or days to catch memory leaks, connection leaks, growing logs, or scheduler drift.
• Scalability testing: Validates scale-up and scale-out plans. Confirms that larger or additional instances deliver predictable gains and that autoscaling triggers map to realistic signals.

How to Perform Load Testing?

Load testing isn’t just about throwing traffic at your system, it’s about understanding how your application behaves under real-world conditions and uncovering hidden bottlenecks before your users do.

Here's a Step-by-Step Guide to do load testing:

1. Define objectives and scope - Start by clearly defining what success looks like for your system. Identify your SLAs, expected concurrent users, peak traffic events, and the most critical user journeys like login, checkout, or key API calls.
2. Identify test scenarios -Use real production data such as analytics and logs to design realistic test scenarios. Make sure to include both common user flows (happy paths) and edge cases that could potentially break the system.
3. Set up test environment- Create a test environment that closely mirrors production in terms of infrastructure, data volume, and network conditions.
4. Create scripts - Develop scripts that simulate real user behavior using load testing tools. Incorporate think times, random delays, and varied actions to mimic how actual users interact with your system.
5. Configure load profile - Define how the load will be applied, gradually ramp up users, maintain a steady load, and then ramp down. This helps you observe how the system behaves under increasing and sustained pressure.
6. Execute the test -  Run your tests in a controlled, non-production environment first to avoid impacting real users. Monitor system performance in real-time to catch any immediate issues.
7. Analyze results - Dive into the results using logs, dashboards, and APM tools to identify performance bottlenecks. Focus on metrics like response time, error rates, and system throughput.
8. Iterate and report - Fix the issues identified during testing and re-run tests to validate improvements. Document your findings and share insights with stakeholders for better decision-making.
9. Integrate into CI/CD pipelines - Automate load testing by integrating it into your CI/CD pipelines. This ensures performance is continuously validated with every build or deployment.

Load testing is an iterative process, not a one-time activity. The more consistently you test and refine, the more resilient and reliable your system becomes over time.

Types of Load Testing Scenarios

Not all traffic patterns are the same, and your system shouldn’t be tested with a one-size-fits-all approach. Different load testing scenarios help you understand how your application behaves under various real-world conditions, from everyday usage to extreme, unpredictable events.

• Baseline testing - This scenario tests your system under normal, expected load conditions. It helps establish a performance benchmark so you can compare how the system behaves under higher stress levels later.

• Ramp-up / Steady-state testing - In this approach, the load is gradually increased until it reaches a peak and is then maintained for a period of time. This helps you observe how your system handles growth in traffic and whether it can sustain peak load consistently.

• Spike testing - Spike testing simulates sudden and extreme increases in traffic over a short period. It is useful for understanding how your system reacts to unexpected surges, such as flash sales or viral events.

• Soak / Endurance testing - This type of testing runs the system under a steady load for an extended duration. It helps uncover issues like memory leaks, resource exhaustion, or performance degradation over time.

• Combined testing (Load + Chaos) - Here, load testing is combined with chaos engineering practices, such as injecting failures like network latency or pod crashes during high traffic. This helps evaluate how resilient your system is under both stress and failure conditions.

Role of AI in load testing

Moving into 2026 and beyond, AI is shifting load testing from a manual, scheduled chore into an intelligent, autonomous process. Instead of relying on static scripts, AI agents now ingest vast streams of real-world data including recent incident reports, deployment logs, and even design changes documented in wikis to generate context-sensitive testing scenarios. This ensures that performance suites are no longer generic; they are hyper-targeted to the specific risks introduced by the latest code commits or environmental shifts, allowing teams to catch bottlenecks before they ever reach production.

The relationship between testing and infrastructure has also become a two-way street. Beyond just identifying breaking points, AI-driven analysis of load test results now provides proactive recommendations for deployment configurations. By correlating performance metrics with resource allocation, these systems can suggest the "golden path" for auto-scaling thresholds, memory limits, and container orchestration. This creates a continuous feedback loop where the load test doesn't just pass or fail it actively optimizes the production environment for peak efficiency.

Load testing for AI Agents

In the new landscape of AI agents proliferation, load testing is no longer just about hitting a server with traffic it's about managing the explosion of agentic orchestration. With organizations deploying hundreds of specialized AI agents, a single user request can trigger a "storm" of inter-agent communication, where one agent's output becomes another's prompt. Traditional load tests fail here because they can't predict these emergent behaviors or the cascading latency that occurs when multiple agents reason, call external APIs, and update shared memory simultaneously. Testing must now account for "prompt bloat" and context contamination, where excessive or conflicting data fed into these agent chains causes performance to degrade or costs to spike unexpectedly.

To survive this complexity, performance engineering in 2026 has shifted toward dynamic environment testing and automated "prompt volume" estimation. Load testers are now using tools like AI Gateways to monitor and rate-limit the massive volume of prompts moving between agents, ensuring that "reasoning loops" don't turn into infinite, resource-draining cycles. By simulating thousands of parallel agent trajectories in virtual sandboxes, teams can identify the specific point where a flurry of prompts causes an LLM's context window to "clash," leading to the 30–40% drops in accuracy often seen under heavy organizational load.

Popular Load Testing Tools



When selecting a load testing tool, teams often start with open-source options for flexibility and cost, then move to enterprise or cloud-managed solutions for scale, collaboration, and integrations.


Here are some of the most popular and widely used load testing tools in 2026:

• Open-source/free: Apache JMeter, k6 (Grafana), Gatling, Locust (Python-based).
• Enterprise: Harness, LoadRunner, NeoLoad, BlazeMeter, LoadNinja.

Choose based on scripting language, scale needs, and integration. For teams already invested in Locust or seeking to combine load testing with chaos engineering in CI/CD pipelines, platforms like Harness Resilience Testing provide seamless native support to elevate your testing strategy.

Load Testing Best Practices in 2026

As systems grow more distributed and user expectations continue to rise, load testing in 2026 is no longer optional, it’s a continuous discipline. Following the right best practices ensures that your application is not just fast, but also resilient and reliable under real-world conditions.

• Test early and often (shift-left) - Start load testing during the development phase instead of waiting until pre-release. This  will help you to catch performance issues early when they are easier and cheaper to fix.

• Use realistic data and traffic models - Base your tests on actual production data, analytics, and user behavior patterns. This ensures your test scenarios closely reflect how users interact with your system in reality.

• Match production environments - Ensure your testing environment mirrors production as closely as possible in terms of configurations, data volume, and scaling policies. This improves the accuracy and reliability of your test results.

• Focus on user journeys - Instead of only testing raw request throughput, simulate complete user workflows like login, search, or checkout.

• Monitor golden signals - Track key performance indicators such as latency, traffic, error rates, and system saturation. These “golden signals” help you quickly identify and diagnose performance issues.

• Automate wisely - Keep smoke-level checks in CI to catch obvious regressions. Reserve heavier runs for staging or pre-production where you can mirror production closely.

• Combine with chaos engineering - Introduce controlled failures during load testing, such as network delays or service disruptions. This helps evaluate how well your system performs under both stress and failure conditions.

Adopting these best practices helps you move beyond basic performance testing toward building truly resilient systems. In 2026, it’s not just about handling traffic, it’s about thriving under pressure.

Conclusion

Load testing turns unknowns into knowns and panic into process. It  isn't a "nice-to-have", it's essential for delivering fast, reliable digital experiences that customers (and your bottom line) demand. 

By following this guide, you'll identify issues early, optimize performance, and build systems that scale confidently.


Ship faster, break less, and stay resilient.

You're tagging Docker images with build numbers.

-Build #47 is your latest production release on main. A developer pushes a hotfix to release-v2.1, that run becomes build #48.

-Another merges to develop, build #49. A week later someone asks: "What build number are we on for production?" You check the registry.

-You see #47, #52, #58, #61 on main. The numbers in between? Scattered across feature branches that may never ship. Your build numbers have stopped telling a useful story.

That's the reality when your CI platform uses a single global counter. Every run, on every branch, increments the same number. For teams using GitFlow, trunk-based development, or any branching strategy, that means gaps, confusion, and versioning that doesn't match how you actually ship.

TL;DR: Harness CI now supports branch-scoped build sequence IDs via <+pipeline.branchSeqId>.

Each branch gets its own counter. No gaps. No confusion.

Why Global Build Counters Break Down

Most CI platforms give you one incrementing counter per pipeline. Push to main, push to develop, push to a feature branch, same counter. So you get:

• Gaps in the sequence for any given branch (e.g. main might have #1, #4, #7).
• No clear answer to "what's the latest build on main?"
• Semantic versioning and artifact naming that don't line up with branch reality.
• Registries and artifact stores full of numbers that don't map to how you release.

This is now built directly into Harness CI as a first-class capability.

What It Feels Like in Practice

Add <+pipeline.branchSeqId> where you need the number—for example, in a Docker build-and-push step:

tags:
  - <+pipeline.branchSeqId>
  - <+codebase.branch>-<+pipeline.branchSeqId>
  - latest

Trigger runs on main, then on develop, then on a feature branch. Each branch gets its own sequence: main might be 1, 2, 3… develop 1, 2, 3… feature/x 1, 2. Your tags become meaningful: main-42, develop-15, feature-auth-3. No more guessing which number belongs to which branch.

What You Get

• Per-branch counters – One sequence per pipeline + repo + branch, stored and incremented atomically.
• Pipeline expression – <+pipeline.branchSeqId>. Check out Harness variables documentation.
  
• REST API – List sequences for a pipeline, get the current value for a branch/repo, reset a branch counter, or set it to a specific value (for example, after a major release or when migrating from another CI).
• Consistent identification – Repo URLs and branch names are normalized (for example, refs/heads/main → main, different URL forms to one canonical host/owner/repo). Same logical branch and repo always share the same counter.
• Cleanup – When a pipeline is deleted, its branch-sequence data is removed so you don't leave orphaned counters.

Webhook triggers (push, PR, branch, release) and manual runs (with branch from codebase config) are supported. For tag-only or other runs without branch context, the expression returns null so you can handle that in your pipeline if needed.

How It Works Under the Hood

Branch and repo are taken from the trigger payload when possible (webhooks) or from the pipeline's codebase configuration (for example, manual runs). We normalize them so that the same repo and branch always map to the same logical key: branch names get refs/heads/ (or similar) stripped, and repo URLs are reduced to a canonical form (for example, github.com/org/repo). That way, whether you use https://..., git@..., or different casing, you get one counter per branch.

The counter is stored and updated with an atomic increment. Parallel runs on the same branch still get distinct, sequential numbers. The value is attached to the run's metadata and exposed through the pipeline execution context so <+pipeline.branchSeqId> resolves correctly at runtime.

Putting It to Work

• Docker image tagging: use <+pipeline.branchSeqId> and optionally <+codebase.branch>-<+pipeline.branchSeqId> for clear, branch-specific tags.
• Helm chart versioning: e.g. --version 1.0.<+pipeline.branchSeqId> --app-version <+codebase.commitSha> so the chart version tracks the build number and the app version tracks the commit.
• Release notes or deployment labels: for example, "Release Build #<+pipeline.branchSeqId>" so production and staging each have a clear, branch-local build number.

For teams that need control or migration support, branch sequences are also manageable via API:

# List all branch sequences for a pipeline
GET /pipelines/{pipelineIdentifier}/branch-sequences

# Reset counter for a specific branch
DELETE /pipelines/{pipelineIdentifier}/branch-sequences/branch?branch=main&repoUrl=github.com/org/repo

# Set counter to a specific value (e.g., after major release)
PUT /pipelines/{pipelineIdentifier}/branch-sequences/set?branch=main&repoUrl=github.com/org/repo&sequenceId=100

All of this is gated by the same feature flag so only accounts that have adopted the feature use the APIs.

Try It (With Smart Guardrails)

1. Enable the feature – Turn on CI_ENABLE_BRANCH_SEQUENCE_ID (Account Settings → Feature Flags, or Reach out to the Harness team).
2. Use the expression – Add <+pipeline.branchSeqId> in steps, tags, or env vars.
3. Verify – Run the pipeline on two or three branches and confirm each branch has its own 1, 2, 3…

If branch context isn't available, the expression returns null. Design your pipeline to handle that (for example, skip tagging or use a fallback) for tag builds or edge cases.

Feature availability may vary by plan. Check with your Harness account or https://developer.harness.io for your setup.

How Other CI Platforms Handle This (Spoiler: Most Don't)

This isn't just a Harness problem we solved—it's an industry gap. Here's how major CI platforms compare:

Most platforms treat build numbers as an afterthought. Harness CI treats them as a first-class versioning primitive. For teams migrating from Jenkins or Azure DevOps, the model will feel familiar. For teams on GitHub Actions, GitLab, or CircleCI, this fills a gap that previously required external services or custom scripts

What's Coming

This is the first release of branch-scoped sequence IDs. The foundations are in place: per-branch counters, expression support, and APIs. We're not done.

We're listening. If you use this feature and hit rough edges—or have ideas for tag-scoped sequences, dashboard visibility, or trigger conditions—we want to hear about it. Share feedback .

For the past few years, the narrative around Artificial Intelligence has been dominated by what I like to call the "magic box" illusion. We assumed that deploying AI simply meant passing a user’s question through an API key to a Large Language Model (LLM) and waiting for a brilliant answer.

Today, we are building systems that can reason, access private databases, utilize tools, and—hopefully—correct their own mistakes. However, the reality is that while AI code generation tools are helping us write more code than ever , we are actually getting worse at shipping it. Google's DORA research found that delivery throughput is decreasing by 1.5% and stability is worsening by 7.5%. Deploying AI is no longer a machine learning experiment; it’s one of the most complex system integration challenges in modern software engineering.

That's why integrated CI/CD is no longer optional for AI deployment—it's the foundation. As teams adopt platforms like Harness Continuous Integration and Harness Continuous Delivery, testing and release orchestration shift from isolated checkpoints to continuous safeguards that protect quality and safety at every layer of the AI stack.

What Is AI Deployment in 2026?

Most definitions of AI deployment are stuck in the "model era." They describe deployment as taking a trained model, wrapping it in an API, and integrating it into a single application to make predictions.

That description is technically accurate—but strategically wrong.

In 2026, AI deployment means:

Integrating a full AI application stack—models, prompts, data pipelines, RAG components, agents, tools, and guardrails—into your production environment so it can safely power real user workflows and business decisions.

You're not just deploying "a model." You are deploying the instructions that define the AI's behavior, the engines (LLMs and other models) that do the reasoning, the data and embeddings that feed those engines context, the RAG and orchestration code that glue everything together, the agents and tools that let AI take actions in your systems, and the guardrails and policies that keep it all safe, compliant, and affordable.

Classic "model deployment" was a single component behind a predictable API. Modern AI deployment is end‑to‑end, cross‑cutting, and deeply entangled with your existing software delivery process.

If you want a great reference for the more traditional view, IBM's overview of model deployment is a good baseline. But in this article, we're going to go beyond that to talk about the compound system you are actually shipping today.

Why AI Deployment Has Become the Bottleneck

The paradox of this moment is simple: coding has sped up, but delivery has slowed down.

AI coding assistants take mere seconds to generate the scaffolding. Platform teams spin up infrastructure on demand. Product leaders are under pressure to add "AI" to every experience. But in many organizations, the actual path from "we built it" to "it's safely in front of customers" is getting more fragile—instead of less.

There are a few reasons for this:

• The AI stack is multi‑layered and non‑deterministic. Traditional CI/CD pipelines were designed for deterministic systems: if the code compiles and tests pass, you can be reasonably confident in the behavior. With LLMs and agents, the same input might result in a range of outputs, some acceptable and some dangerous. Testing no longer has a simple pass/fail shape.
• Ownership is fractured. MLOps teams worry about training and serving models. Application teams bolt on AI features. Security teams scramble to backfill policies around data access and tool usage. Platform teams are left trying to orchestrate releases that touch all of the above, often without having clear control over any of them.
• We've created tool silos instead of integrated delivery. We now talk about MLOps, LLMOps, AgentOps, DevOps, SecOps—as if each deserved its own stack and dashboard—while the actual releases that matter to customers cut straight across those boundaries.

The result is what many teams are feeling right now: shipping AI features feels risky, brittle, and slow, even as the pressure to "move faster" keeps rising.

To fix that, we have to start with the stack itself.

Part 1: Deconstructing the Modern AI Stack

To understand how to deploy AI, you have to stop treating it as a single entity. The modern AI application is a compound system of highly distinct, interdependent layers. If any single component in this stack fails or drifts, the entire application degrades.

1. The Instructions: Prompts as Code

A prompt is no longer just a text string typed into a chat window; it is the source code that dictates the behavior and persona of your application.

• The Deployment Reality: Prompts require the same rigor as traditional code—version control, peer review, and automated testing. Because LLMs are sensitive to minute phrasing changes, updating a prompt requires running it against hundreds of baseline test cases to ensure the model doesn't experience "regression" and forget its core instructions.

2. The Engine: Large Language Models (LLMs)

The LLM is the reasoning engine. It has vast general knowledge but zero awareness of your company’s proprietary data.

• The Deployment (LLMOps) Reality: Most companies consume these via APIs or host smaller models on cloud infrastructure. The deployment challenge is routing. A sophisticated pipeline will dynamically route simple tasks to faster, cheaper models and complex reasoning tasks to massive, expensive models to optimize both latency and cloud spend, which currently sees significant waste in many organizations.

3. The Fuel: Data and Vector Embeddings

An AI's output is only as reliable as the context it is given. To make an LLM useful, it needs a continuous feed of your company’s internal data.

• The Deployment Reality: This requires automated data pipelines that ingest raw information, "chunk" it, and store it in a Vector Database. If the embedding model changes, the entire database must be re-indexed. This data pipeline must be continuously deployed and synced without disrupting the live application.

4. The Architecture: Retrieval-Augmented Generation (RAG)

RAG is not a model; it is a separate software architecture deployed to act as the LLM's research assistant.

• The RAG Deployment Reality: When a user asks a question, the RAG code intercepts it, queries the Vector Database, and packages that data into a prompt. Deploying RAG means deploying the integration code that securely manages this retrieval and hand-off process.

5. The Doer: AI Agents

If RAG is a researcher, an AI Agent is an employee. Agents are LLMs given access to external tools. Instead of just answering a question, an agent can formulate a plan, search the web, and execute code.

• The Deployment Reality: Moving from linear flows to "Agentic Workflows" introduces massive complexity. You are now deploying systems that iterate and loop. Deploying an agent requires monitoring its step-by-step reasoning traces and ensuring it doesn't get stuck in an infinite loop or misuse its tools.
  
  

Part 2: The Guardrails (DevSecOps for AI)

You cannot expose a raw LLM or an autonomous agent to the public, or even to internal employees, without armor. Because AI is non-deterministic, traditional software security falls short. Modern AI deployment requires distinct "Guardrails as Code".

Input Guardrails

• Prompt Injection Defenses: Malicious users will attempt to "jailbreak" the AI. Input guardrails use separate, smaller models to intercept adversarial prompts before they reach the core LLM.
• PII Scrubbing: Automated systems must redact Personally Identifiable Information (PII) to ensure sensitive data never leaves your secure environment or reaches a third-party LLM provider.

These kinds of controls are a natural fit for policy‑as‑code engines and CI/CD gates. With something like Harness Continuous Delivery & GitOps, you can enforce Open Policy Agent (OPA) rules at deployment time—ensuring that applications with missing or misconfigured input guardrails simply never make it to production.

Output Guardrails

• Hallucination Detection: These cross-reference the model’s answer against the retrieved RAG documents to increase confidence by checking support/citations for key claims in your proprietary data.
• Schema Enforcement: If your system expects the AI to return data in a strict format, output guardrails will validate the structure and automatically reject or re-prompt the LLM if it outputs unstructured text.

Agentic and Operational Guardrails

• Blast-Radius Containment: When deploying agents that can execute actions, strict Role-Based Access Control (RBAC) must be enforced. Agents must operate on the principle of least privilege.
• FinOps and Rate Limiting: AI is computationally expensive. Guardrails must enforce strict token-usage tracking and throttle actions to prevent a runaway agent from racking up thousands of dollars in cloud compute costs.

Part 3: The Interplay and the Need for Release Orchestration

Understanding the stack reveals the ultimate challenge: The Cascade Effect. In traditional software, a database error throws a clean error code. In an AI application, a bug in the data pipeline silently ruins everything downstream. This is why deployment cannot be disjointed. It requires rigorous Release Orchestration.

1. Fuzzy Integration Testing: Traditional CI/CD pipelines rely on exact-match assertions. Because LLMs return varying text, we now require "semantic evaluation"—often using a separate LLM acting as a judge to grade the output based on meaning and accuracy during the automated testing phase.
2. Progressive Rollout Strategies: Because you cannot perfectly predict AI behavior, orchestration must support Canary releases—rolling out the new model to 5% of users to monitor drift before a full launch.
3. Synchronizing the Moving Parts: A prompt update might require a different RAG strategy. A new embedding model demands a full database re-indexing. Release orchestration ensures that when one layer is updated, the corresponding dependencies are automatically tested and deployed in lockstep.

How to Deploy AI in Production (A Practical Pipeline)

1. Version prompts/configs/policies as code
   
   
2. Build eval suite (golden set + safety tests)
   
   
3. CI: semantic eval + regression thresholds
   
   
4. Security gates: PII redaction + prompt injection tests
   
   
5. CD: canary rollout for prompt/model/RAG changes
   
   
6. Observability: quality + safety + cost signals
   
   
7. Rollback rules tied to metrics
   
   
8. Post-deploy review and dataset refresh cadence

The Bottom Line: Orchestrate or Stall

For years, we've been obsessed with specialized silos: MLOps, LLMOps, AgentOps. But a vital realization is sweeping the enterprise: the time of siloed, specialized AI operations tools is coming to an end.

The future belongs to unified release management. The organizations that succeed will not be the ones with the smartest standalone AI models, but the ones who master the orchestration required to deploy and evolve those models, alongside everything else they ship, safely, efficiently, and continuously. 

If you want a platform that brings semantic testing, progressive rollouts, and coordinated AI releases into your day-to-day workflows, Harness Continuous Integration and Harness Continuous Delivery were built for this.

‍

Key Takeaways:

• AI deployment is deploying a stack, not a model.
• Treat prompts, evals, policies, and configs as code.
• Use semantic evaluation plus standard CI tests.
• Use progressive delivery (canaries) for models/prompt/RAG changes.
• Orchestrate dependencies (prompt ↔ RAG ↔ embeddings ↔ guardrails) to prevent silent regressions.

‍

AI Deployment: Frequently Asked Questions (FAQs)

What is AI deployment?
AI deployment is the process of integrating AI systems, models, prompts, data pipelines, RAG architectures, agents, tools, and guardrails, into production environments so they can safely power real applications and business workflows.

How is AI deployment different from traditional model deployment?
Traditional model deployment focuses on serving a single model behind an API. Modern AI deployment involves a multi‑layer stack: instructions, engines, context, retrieval, agents, and policies. Failures are more likely to be silent regressions or unsafe behaviors than obvious crashes, which is why you need semantic testing, guardrails, and release orchestration.

How do you deploy AI safely in production?
Safe AI deployment starts with treating prompts and configurations as code, embedding guardrails at input, output, and action levels, and using semantic evaluation and progressive rollout strategies. It also requires immutable logging and audit trails so you can trace decisions back to specific versions of your AI stack. Combining CI for semantic tests with CD for orchestrated releases is the practical path to safety.

What tools are used for AI deployment?
Teams typically use a mix of LLM providers or model‑serving platforms, vector databases, observability tools, and CI/CD systems for orchestrating releases. On top of that, they add policy engines and specialized evaluation frameworks. The critical shift is moving from isolated "AI tools" to integrated pipelines that tie everything together.

How do canary releases work for AI models and prompts?
With canary releases, you send a small portion of traffic to the new behavior, a new model, prompt, or RAG strategy, while most users continue on the old path. You observe semantic quality, safety signals, and performance. If the canary behaves well, you gradually increase its share. If it misbehaves, you automatically roll back to the previous version.

On March 24, 2026, the AI open-source ecosystem was impacted by a critical supply chain attack involving the widely used Python package LiteLLM. Attackers compromised the LiteLLM PyPI distribution pipeline and published malicious versions (notably in the 1.82.7-1.82.8 range), embedding a multi-stage payload designed to steal credentials and execute remote code.

The malicious code executed during normal library usage, making it particularly dangerous for AI applications that integrate LiteLLM into inference pipelines, RAG systems, and API gateways. The attack leveraged Python’s .pth file mechanism, a lesser-known but powerful feature that allows arbitrary code execution during interpreter initialization. The malicious package dropped a file (e.g., litellm_init.pth) into the Python environment, ensuring that the payload was executed every time the Python interpreter started, regardless of whether LiteLLM was explicitly imported. 

However, this was not a simple static backdoor. The package implemented a multi-stage execution chain, where initial execution triggered outbound communication to attacker-controlled infrastructure to retrieve further instructions dynamically.

Notably, the malware leveraged blockchain-based command-and-control (C2) and extracting payload locations from transaction metadata. This allowed attackers to update payload delivery dynamically without modifying the package itself, making detection and takedown significantly harder.

The attack also performed a cross-language execution pivot, downloading a Node.js runtime onto the host system and executing an obfuscated second-stage JavaScript payload. This technique enabled evasion of Python-focused security controls and introduced runtime behavior that would not typically be expected during dependency execution.

Once triggered, the payload attempted to exfiltrate sensitive environment variables and secrets, including API keys for major LLM providers (OpenAI, Anthropic), cloud credentials, and internal tokens.

Python .pth Files: Silent Execution Hooks at Interpreter Startup

A critical aspect of this attack is the abuse of Python’s .pth file mechanism, an often overlooked feature that can be weaponized for persistent, pre-execution code injection.

.pth files are automatically processed during Python interpreter startup via the site module. They are typically located in site-packages or user site directories and are intended to extend Python’s module search path.

However, .pth files support more than just path configuration. If a line starts with an import, it is executed immediately:

Example: Simple Demo of Hidden Code Execution

DIY: Find your site-packages path

python3 -c "import site; print(site.getsitepackages())"
# Or 
python3 -c "import site; print(site.getusersitepackages())"
#  Choose the one with write access and 
#  write a demo1.pth file to that site packages dir
echo 'import sys; print("[PTH DEMO EXECUTED]")' > /path/to/site-packages/demo1.pth

Or

Automated script to create the demo .pth file

git clone https://github.com/aspen-labs/python_pth_autoexecution.git
cd python_pth_autoexecution
python3 demo_pth.py

Now, every time Python runs, 

> python3 -c "print('Hello')"
[PTH DEMO EXECUTED]
Hello

Impact:

• Compromised PyPI versions of LiteLLM (primarily 1.82.x series) distributed malicious code to downstream users.
• Persistent execution via .pth interpreter hijacking, enabling code execution on every Python startup without intended use of the library.
• Use of decentralized C2 (blockchain) makes traditional takedown and IOC blocking less effective.
• Execution of second-stage JavaScript payloads (e.g., i.js), enabling cross-language runtime pivoting and evasion of Python-focused detection controls
• Exposure of sensitive credentials, including LLM API keys, cloud provider secrets, and environment variables.
• High risk to AI/LLM applications, especially RAG pipelines and backend services handling user prompts and responses.
• Potential lateral movement into CI/CD systems and production environments due to leaked credentials.

This attack highlights a new class of supply chain risk centered around AI infrastructure dependencies. Unlike traditional package compromises, the blast radius extends into model access, prompt data, and downstream AI workflows. It reinforces the need for continuous dependency monitoring, real-time SBOM visibility, and strict policy enforcement to prevent compromised packages from entering build and runtime environments.

How Harness Supply Chain Security Helps

Harness SCS helps you quickly detect and contain compromised dependencies like the LiteLLM package before they impact your pipelines. With real-time visibility into your SBOMs and dependency graph, you can identify affected versions, trace their usage across builds and environments, and block them using OPA policies. This ensures malicious packages never propagate through your CI/CD or AI workflows.

Detect Compromised LiteLLM Package

Harness SCS enables instant search across all repositories and artifacts to quickly identify if compromised LiteLLM versions exist in your environment. The moment such a malicious package is disclosed, you can pinpoint its presence and assess impact across your entire supply chain in seconds.

Block Compromised LiteLLM Packages

Harness AI streamlines response to incidents like the LiteLLM compromise through simple natural-language prompts. With a single prompt, you can generate OPA policies to block affected LiteLLM versions across all pipelines, preventing malicious packages from entering builds or deployments. As new compromised versions emerge, these policies can be quickly updated to maintain strong preventive controls across your SDLC. SCS customers can use this OPA policy to detect and block the affected versions

Track & Remediate Issues with Developers

Harness SCS automatically detects compromised LiteLLM versions across both production and non-production environments. Teams can track remediation, assign fixes, and monitor progress through to deployment, ensuring exposed credentials and vulnerable dependencies are addressed quickly. This end-to-end visibility helps contain the impact and prevents compromised packages from persisting in your supply chain.

Next Steps in the Face of Supply Chain Attacks

The LiteLLM compromise highlights how quickly a malicious package can expose high-value secrets when embedded deep within AI application stacks. Given its role in handling LLM requests, the impact extends beyond code to API keys, prompt data, and downstream systems, often bypassing traditional security checks.

Defending against such attacks requires more than reactive fixes. Teams need real-time visibility into dependencies, the ability to enforce policies to block compromised versions, and continuous tracking to ensure remediation is complete across all environments. Harness SCS enables teams to quickly identify where affected LiteLLM versions are used, prevent them from entering new builds, and ensure fixes are consistently rolled out.

With these controls in place, organizations can limit credential exposure, contain threats early, and secure their AI supply chain against attacks like the LiteLLM compromise.

Also, learn how Harness SCS defends against Shai-Hulud, TJ Actions, NPM 1.0, xz-utils supply chain attacks.

---

Key Takeaway: Harness AI helped deflect 95% of the platform support tickets for a major financial institution

---

These days, success is often measured by what doesn’t happen:

• Outages that don’t happen, so there are no 3 AM wake-up calls for engineers.
• Zero "emergency" rollbacks after a Friday afternoon deployment to prod.
• No configuration drift creeping into production environments.
• No "how-to" tickets clogging up the backlog of a busy platform team.

When things go right, the software delivery platform is invisible. But what happens when an organization’s delivery velocity increases multifold? Can the platform still stay out of the way?

For one of the world’s largest financial institutions, the answer was a resounding "yes" but only after they solved the Velocity Paradox.

The Success Tax: When Scale Outpaces Support

By standardizing their deployment pipelines on Harness, this organization’s engineering teams hit their stride. Deployments became predictable. Manual toil went away. Teams were shipping more code, faster, with fewer headaches.

But as throughput increased, the complexity of the SDLC's outer loop also increased. More pipelines meant more execution states to interpret; more automation meant more configuration nuance.

That’s when the tickets started coming in. The increase in tickets was not a symptom of failure; it was a success tax. The platform team found itself acting as a human search engine, gathering logs, reviewing configuration state, and translating technical findings into actionable guidance for every request. The work was necessary, but it was fundamentally reactive.

Strategy Over Hype: AI as a Core OKR

Rather than simply expanding headcount to keep pace with this increasing workload, the organization made a deliberate shift in its operating model. They didn't treat AI as an experiment. AI adoption was formalized within team OKRs and became an explicit objective from executive leadership. Harness AI was selected as the primary engine to meet that goal. The directive to the engineering organization was simple:

“Ask Harness AI first.”

Developers were encouraged to treat Harness AI as their first line of inquiry. 

The Engine Behind the Answer: Knowledge Graph

What makes Harness AI more than just a chatbot is what powers it underneath, a Knowledge Graph purpose-built for software delivery.

Unlike a generic large language model that reasons from broad training data alone, Harness AI is grounded in a living, interconnected map of your delivery environment. The Knowledge Graph continuously indexes and relates the entities that matter most in your SDLC: pipelines, services, environments, deployments, configurations, governance policies, and execution histories, and critically, the relationships between them.

When someone asks Harness AI a question, it traverses the optimal Knowledge Graph to understand the full context: what changed recently, which policy rule was triggered, how this pipeline relates to other services, and what similar failures looked like in the past, etc. This graph-powered reasoning is what enables answers that are specific, actionable, and contextually accurate and not generic.

Three things came up over and over in how developers used it:

1. Figuring out where a deploy went wrong. Not just “step 4 failed” but actually tracing through logs, execution history, recent changes, etc., to say “this is what failed and here’s why.”
2. Making sense of policy blocks. In a financial institution, governance rules are everywhere, and they’re not always obvious. The AI could explain why something got blocked and what you’d need to change to fix it.
3. Giving you the actual fix. Not just flagging the error: suggesting the specific YAML tweak or config change to unblock the build. Developers stopped waiting for someone to tell them what to do next.

For this financial institution’s engineering teams, this meant that every question asked of Harness AI was answered with awareness of their unique environment, their governance rules, and their delivery history. The AI wasn’t guessing. It knew.

The Result: 95% Tickets Deflected

The behavioral shift was both gradual and massive. Over the following months, the majority of internal support tickets were deflected. Engineers resolved a significant portion of troubleshooting questions independently, within their workflow, without waiting for platform team intervention.

Over the months that followed, tens of thousands of support tickets just didn’t happen. Engineers figured things out on their own, in the moment, without filing anything or waiting for a response.

The result was a radical rebalancing of capacity. Instead of being a help desk, the platform team redirected its efforts toward designing and building forward-looking architecture.

Innovation Without the Friction

Scaling a platform isn’t just a technical problem. It’s an operational one. You can modernize your entire SDLC and still end up drowning in tickets if the support model doesn’t scale with it.

Here's what makes Harness AI different from every other AI tool your team has probably tried: Harness sits inside your SDLC. Not bolted on from the outside, not reading a summary of what happened. It’s actually inside it. That means it picks up on context that no generic AI agent ever could. Your pipelines, your policies, your deployment history, your governance rules, how your services relate to each other, your infrastructure, and a lot more. All of it. And it uses that context to build a customized Knowledge Graph for your organization.

That's what makes Harness AI feel like part of your team rather than just another tool you have to babysit.

‍

FAQs

What does Harness AI use to answer questions?

Harness AI uses context from the software delivery environment, including pipelines, services, environments, policies, configurations, and execution history, to generate more relevant answers.

How is Harness AI different from a generic chatbot?

A generic chatbot mainly relies on broad model training data. Harness AI is grounded in an organization-specific delivery context, which helps it explain failures and suggest next steps based on how the environment is actually configured.

Why is a Knowledge Graph useful for software delivery?

A Knowledge Graph helps connect related entities across the SDLC, such as pipelines, services, environments, and policy rules. That structure makes it easier to retrieve relevant context for troubleshooting and operational guidance.

Can AI reduce internal support tickets for platform teams?

Yes. When developers can get contextual answers inside their workflow, many troubleshooting and policy questions can be resolved without filing support tickets.

‍

• Install Argo CD with Helm and pinned versions so your setup is repeatable and upgrades are predictable.
  
  
• Secure Argo CD before scaling adoption by using a stable hostname, SSO, least-privilege RBAC, and AppProjects for team guardrails.
  
  
• Run Argo CD like platform infrastructure with HA, monitoring, backups/restore tests, and a staged upgrade process to keep GitOps reliable at scale.

It’s easy to install Argo CD once. The harder problem, especially in production environments, is installing Argo CD in a way that supports multiple teams, survives upgrades, meets audit requirements, and does not become a fragile shared dependency.

This guide walks you through an enterprise-ready Argo CD install with a Helm-first approach, plus an upstream manifest option for evaluation. You’ll also get useful tips on how to secure access (SSO/RBAC), safely onboard teams, and run Argo CD every day without any problems.

If you’re standardizing GitOps across multiple clusters and teams, Harness CD can help you manage and govern Argo CD at scale.

What You’re Installing (and Why Enterprises Standardize on Argo CD)

Argo CD is a Kubernetes-native continuous delivery controller that follows GitOps principles: Git is the source of truth, and Argo CD continuously reconciles what’s running in your cluster with what’s declared in Git.

That pull-based reconciliation loop is the real shift. Instead of pipelines pushing manifests into clusters, Argo CD runs inside the cluster and pulls the desired state from Git (or Helm registries) and syncs it to the cluster. The result is an auditable deployment model where drift is visible and rollbacks are often as simple as reverting a Git commit.

For enterprise teams, Argo CD becomes a shared platform infrastructure. And that changes what “install” means. Once Argo CD is a shared control plane, availability, access control, and upgrade safety matter as much as basic deployment correctness because failures impact every team relying on GitOps.

What It Means In An Enterprise

A basic install is “pods are running.” An enterprise install is:

• Secure access (SSO + least-privilege RBAC)
  
  
• Safe multi-team usage (AppProjects guardrails, predictable onboarding)
  
  
• Stable operations (monitoring, backups, upgrades)
  
  
• Repeatability (version pinning, Git-driven configuration)

Argo CD can be installed in two ways: as a “core” (headless) install for cluster admins who don’t need the UI/API server, or as a multi-tenant install, which is common for platform teams. Multi-tenant is the default for most enterprise DevOps teams that use GitOps with a lot of teams.

Setup Prerequisites

Before you start your Argo CD install, make sure the basics are in place. You can brute-force a proof of concept with broad permissions and port-forwarding. But if you’re building a shared service, doing a bit of prep up front saves weeks of rework.

Cluster Prerequisites

• A Kubernetes cluster you can administer (or at least create namespaces and cluster-scoped resources).
  
  
• Network path to the API server from your workstation/CI environment.
  
  
• A plan for ingress and TLS (internal-only is fine, just decide early).

Workstation Tools

• kubectl configured for the target cluster
  
  
• helm (recommended approach)
  
  
• Optional: argocd CLI (useful for scripting and verification)

Platform Prerequisites to Confirm

• Ingress controller availability (NGINX, ALB Ingress Controller, Traefik, etc.), or a willingness to use a cloud LoadBalancer.
  
  
• DNS for a stable Argo CD hostname (even if internal).
  
  
• Certificate strategy for TLS (cert-manager, corporate PKI, or managed certs).

If your team is in a regulated environment, align on these early:

• Where Argo CD secrets will live (Kubernetes Secrets vs external secrets tooling)
  
  
• Audit requirements (SSO provider logs, Kubernetes audit logs, etc.)
  
  
• Network restrictions (private clusters, egress policies)

Decide Your Argo CD Installation Approach

Argo CD install choices aren’t about “works vs doesn’t work.” They’re about how you want to operate Argo CD a year from now.

Helm vs. Upstream Manifests

Helm (recommended for enterprise):

• Repeatable installs across environments (dev/stage/prod)
  
  
• Easy upgrades via version pinning
  
  
• Values-driven configuration you can store in Git

Upstream manifests:

• Fast and close to upstream defaults
  
  
• Great for evaluation or a quick validation environment
  
  
• Less structured change management unless you wrap it in GitOps

If your Argo CD instance is shared across teams, Helm usually wins because version pinning, values-driven configuration, and repeatable upgrades are easier to audit, roll back, and operate safely over time.

Single Instance vs. Multiple Instances

Enterprises often land in one of these models:

• One shared Argo CD instance per cluster (common in platform teams)
  
  
• One shared instance managing multiple clusters (central GitOps control plane)
  
  
• Multiple instances (per business unit or compliance boundary)

As a rule: start with one shared instance and use guardrails (RBAC + AppProjects) to keep teams apart. Add instances only when you really need to (for example, because of regulatory separation, disconnected environments, or blast-radius requirements).

When Argo CD is a shared dependency, high availability (HA) is important. If teams depend on Argo CD to deploy, having just one replica Argo CD server can slow things down and cause problems with pagers.

How You’ll Expose Argo CD

There are three common access patterns:

• Port-forward (setup only): safest for a first login, not an enterprise default.
  
  
• Ingress (most common): use your standard ingress + TLS termination.
  
  
• LoadBalancer service: simple in cloud environments, but can increase cost and widen exposure.

For most enterprise teams, the sweet spot is Ingress + TLS + SSO, with internal-only access unless your operating model demands external access.

Install Argo CD Using Helm (Step-by-Step)

If you’re building Argo CD as a shared service, Helm gives you the cleanest path to versioned, repeatable installs.

Step 1: Add the Helm Repo and Pin a Version

helm repo add argo https://argoproj.github.io/argo-helm
helm repo update

# Optional: list available versions so you can pin one
helm search repo argo/argo-cd --versions | head -n 10

In enterprise environments, “latest” isn’t a strategy. Pin a chart version so you can reproduce your install and upgrade intentionally.

Step 2: Create the argocd Namespace

kubectl create namespace argocd

Keeping Argo CD isolated in its own namespace simplifies RBAC, backup scope, and day-2 operations.

Step 3: Export Default Values and Make Minimal Enterprise Edits

Start by pulling the chart’s defaults:

helm show values argo/argo-cd > values.yaml

Then make the minimum changes needed to match your access model. Many tutorials demonstrate NodePort because it’s easy, but most enterprises should standardize on Ingress + TLS.

Here’s a practical starting point (adjust hostnames, ingress class, and TLS secret to match your environment):

# values.yaml (example starter)

global:
  domain: argocd.example.internal

configs:
  params:
    # Common when TLS is terminated at an ingress or load balancer.
    server.insecure: "true"

server:
  ingress:
    enabled: true
    ingressClassName: nginx
    hosts:
      - argocd.example.internal
    tls:
      - secretName: argocd-tls
        hosts:
          - argocd.example.internal

# Baseline resource requests to reduce noisy-neighbor issues.
controller:
  resources:
    requests:
      cpu: 200m
      memory: 512Mi

repoServer:
  resources:
    requests:
      cpu: 200m
      memory: 512Mi

This example focuses on access configuration and baseline resource isolation. In most enterprise environments, teams also explicitly manage RBAC policies, NetworkPolicies, and Redis high-availability decisions as part of the Argo CD platform configuration.

If your clusters can’t pull from public registries, you’ll need to mirror Argo CD and dependency images (Argo CD, Dex, Redis) into an internal registry and override chart values accordingly.

Step 4: Install (Or Upgrade) Argo CD

Use helm upgrade --install so your install and upgrade command is consistent.

helm upgrade --install argocd argo/argo-cd \  
	--namespace argocd \  
	--values values.yaml

Validate that core components are healthy:

kubectl get pods -n argocd
kubectl get svc -n argocd
kubectl get ingress -n argocd

If something is stuck, look at events:

kubectl get events -n argocd --sort-by=.lastTimestamp | tail -n 30

Step 5: Confirm the Installation Shape (What’s Running)

Most installs include these core components:

• argocd-server (UI/API)
  
  
• argocd-repo-server (fetches repos and renders manifests)
  
  
• argocd-application-controller (reconciliation)
  
  
• ApplicationSet Controller (optional but common at scale)
  
  
• Dex (if enabled for SSO integration)
  
  
• Redis (caching and coordination)

Knowing what each component does helps you troubleshoot quickly when teams start scaling usage.

Access the Argo CD UI and First Login

Your goal is to get a clean first login and then move toward enterprise access (Ingress + TLS + SSO).

Option 1: Port-Forward (Best for Initial Setup)

kubectl port-forward -n argocd svc/argocd-server 8080:443

Then open https://localhost:8080.

It’s common to see an SSL warning because Argo CD ships with a self-signed cert by default. For a quick validation, proceed. For enterprise usage, use real TLS via your ingress/load balancer.

Option 2: Ingress (Enterprise Default)

Once DNS and TLS are wired:

• Browse to https://argocd.example.internal
• Confirm you’re hitting the ingress you expect (and that TLS is correct)

If your ingress terminates TLS at the edge, running the Argo CD API server with TLS disabled behind it (for example, server.insecure: “true”) is a common pattern.

Get the Initial Admin Password

Default username is typically admin. Retrieve the password from the initial secret:

kubectl -n argocd get secret argocd-initial-admin-secret \  
-o jsonpath="{.data.password}" | base64 --decode; echo

After you’ve logged in and set a real admin strategy using SSO and RBAC, the initial admin account should be treated as a break-glass mechanism only. Disable or tightly control its use, rotate credentials, and document when and how it is allowed.

Install Argo CD Using Upstream Manifests (Fast Path for Evaluation)

If you want a quick Argo CD install for learning or validation, upstream manifests get you there fast.

Important context: the standard install.yaml manifest is designed for same-cluster deployments and includes cluster-level privileges. It’s also the non-HA install type that’s typically used for evaluation, not production. If you need a more locked-down footprint, Argo CD also provides namespace-scoped and HA manifest options in the upstream manifests.

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

Validate:

kubectl get pods -n argocd
kubectl get svc -n argocd

Then port-forward to access the UI:

kubectl port-forward -n argocd svc/argocd-server 8080:443

Use admin plus the password from argocd-initial-admin-secret as shown in the prior section.

For enterprise rollouts, treat manifest installs as a starting point. If you’re standardizing Argo CD across environments, Helm is easier to control and upgrade.

Deploy Your First Application With Argo CD

A real install isn’t “pods are running.” A real install is “we can deploy from Git safely.” This quick validation proves:

• repo access works
  
  
• sync works
  
  
• drift shows up
  
  
• rollbacks are Git-driven

Step 1: Pick a Simple Repo Layout

Keep it boring and repeatable. For example:

apps/
  guestbook/
    base/
    overlays/
      dev/
      prod/

Or, if you deploy with Helm:

apps/
  my-service/
    chart/
    values/
      dev.yaml
      prod.yaml

Step 2: Create an AppProject (The Enterprise Guardrail)

Even for a test app, start with the guardrail. AppProjects define what a team is allowed to deploy, and where.

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: team-sandbox
  namespace: argocd
spec:
  description: "Sandbox boundary for initial validation"
  sourceRepos:
    - "https://github.com/argoproj/argocd-example-apps.git"
  destinations:
    - namespace: sandbox
      server: https://kubernetes.default.svc
  namespaceResourceWhitelist:
    - group: "apps"
      kind: Deployment
    - group: ""
      kind: Service
    - group: "networking.k8s.io"
      kind: Ingress

Apply it:

kubectl apply -f appproject-sandbox.yaml

Step 3: Create an Application and Sync

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: guestbook
  namespace: argocd
spec:
  project: team-sandbox
  source:
    repoURL: https://github.com/argoproj/argocd-example-apps.git
    targetRevision: HEAD
    path: guestbook
  destination:
    server: https://kubernetes.default.svc
    namespace: sandbox
  syncPolicy:
    automated:
      selfHeal: true
      prune: false
    syncOptions:
      - CreateNamespace=true

Note: In many enterprise environments, namespace creation is restricted to platform workflows or Infrastructure as Code pipelines. If that applies to your organization, remove CreateNamespace=true and require namespaces to be provisioned separately.

Apply it:

kubectl apply -f application-guestbook.yaml

Now confirm:

• The app shows up in the UI
  
  
• It syncs successfully
  
  
• If you change something manually in-cluster, the app becomes OutOfSync
  
  
• If you revert Git, syncing takes you back to the previous state

Optional: Add Git Webhooks for Faster Sync

By default, Argo CD polls repos periodically. Many teams configure webhooks (GitHub/GitLab) so Argo CD can refresh and sync quickly when changes land. It’s not required for day one, but it improves feedback loops in active repos.

Secure Access and Multi-Team Guardrails

This is where most enterprise rollouts either earn trust or lose it. If teams don’t trust the platform, they won’t onboard their workloads.

Focus on these enterprise minimums:

• SSO first: your identity provider should be the source of truth.
  
  
• Least privilege: app teams deploy only to approved namespaces/clusters.
  
  
• Guardrails as code: AppProjects prevent accidental cross-team deploys.

Practical rollout order:

1. Establish a stable hostname (so SSO callbacks are consistent).
   
   
2. Configure SSO (OIDC or SAML) and group mapping.
   
   
3. Apply RBAC aligned to roles (platform admin, app owner, read-only).
   
   
4. Define AppProjects for team boundaries.

Break-glass access should exist, but it should be documented, auditable, and rare.

Production Hardening and Day-2 Operations

Enterprise teams don’t struggle because they can’t install Argo CD. They struggle because Argo CD becomes a shared dependency—and shared dependencies need operational maturity.

High Availability and Scaling

At scale, pressure points are predictable:

• argocd-server: UI/API and auth flows
  
  
• repo-server: Git/Helm fetches, rendering, and caching
  
  
• application-controller: reconciliation across many apps/clusters

Plan a path to HA before you onboard many teams. If HA Redis is part of your design, validate node capacity so workloads can spread across failure domains.

Monitoring and Alerting

Keep monitoring simple and useful:

• Argo CD API availability
  
  
• sync failure rate
  
  
• number of degraded apps
  
  
• reconciliation lag (are apps taking too long to converge?)

Also, decide alert ownership and escalation paths early. Platform teams typically own Argo CD availability and control-plane health, while application teams own application-level sync and runtime issues within their defined boundaries.

• Platform team owns Argo CD availability.
  
  
• App teams own app-level health within their boundary.

Backups, Restore Tests, and DR

Git is the source of truth for desired state, but you still need to recover platform configuration quickly.

Backup:

• argocd namespace ConfigMaps and Secrets
  
  
• Argo CD custom resources (Applications, AppProjects, ApplicationSets)

Then run restore tests on a schedule. The goal isn’t perfection—it’s proving you can regain GitOps control safely.

Upgrade Strategy

A safe enterprise approach:

1. Pin chart/app versions and document what’s running.
   
   
2. Stage upgrades in a non-production environment.
   
   
3. Validate core workflows after upgrade (login, repo access, sync).
   
   
4. Promote the same change through environments.

Avoid “random upgrades.” Treat Argo CD as platform infrastructure with controlled change management.

Argo CD Install on EKS: Enterprise Notes

Argo CD works well on EKS, but enterprise teams often have extra constraints: private clusters, restricted egress, and standard AWS ingress patterns.

Common installation approaches on EKS:

• Manual install (Helm/manifests): direct control; easiest if you already have platform standards.
  
  
• Terraform: repeatable infrastructure and bootstrap.
  
  
• EKS Blueprints: a structured AWS-aligned framework for adding platform components.

For access, most EKS enterprise teams standardize on an ingress backed by AWS Load Balancer Controller (ALB) or NGINX, with TLS termination at the edge.

Making Argo CD Production-Ready

An enterprise-grade Argo CD install is less about getting a UI running and more about putting the right foundations in place: a repeatable deployment method (typically Helm), a stable endpoint for access and SSO, and clear boundaries so teams can move fast without stepping on each other. If you take away one thing, make it this: treat Argo CD like shared platform infrastructure, not a one-off tool.

Start with a pinned, values-driven Helm install. Then lock in the enterprise minimums: SSO, RBAC, and AppProjects, before you onboard your second team. Finally, operationalize it with monitoring, backups, and a staged upgrade process so Argo CD stays reliable as your cluster and application footprint grows.

When you need orchestration, approvals, and progressive delivery across complex releases, pair GitOps with Harness CD. Request a demo.

Argo CD Installation: Frequently Asked Questions (FAQs)

These are quick answers to the most common questions that business teams have when they install Argo CD.

What’s the best way to install Argo CD for production?

Most enterprise teams should use Helm to install Argo CD because it lets you pin versions, keep configuration in Git, and upgrade in a predictable way. Upstream manifests are a great way to get started quickly if you’re thinking about Argo CD.

How can we safely expose Argo CD?

Use an internal hostname, end TLS at your ingress/load balancer, and make sure that SSO is required for interactive access. Do not make Argo CD public unless your business model really needs it.

What is the safest way to upgrade Argo CD?

Pin your chart/app versions, test upgrades in a non-production environment, and then move the same change to other environments. After the upgrade, check that you can log in, access the repo, and sync with a real app.

What’s the right model for multi-team access?

Use RBAC and AppProjects to set limits on a single shared instance. Only approved repos should be used by app teams to deploy to approved namespaces and clusters.

How do we back up and restore Argo CD?

Back up the argocd namespace (ConfigMaps, Secrets, and CRs) and keep app definitions in Git. Run restore tests on a schedule so recovery steps are proven, not theoretical.

Leon Adato on SRE Lessons from Spider-Verse and Surviving Tech Failures

At SREday NYC 2026, the ShipTalk podcast welcomed Leon Adato, Principal Technical Marketing Engineer at Cribl and host of the Technically Religious podcast, for a conversation about how engineers navigate failure and uncertainty in complex systems.

In the episode, ShipTalk host Dewan Ahmed, Principal Developer Advocate at Harness, spoke with Leon about finding lessons for reliability engineering in unexpected places—including movies like Spider-Man: Into the Spider-Verse.

For Leon, the world of SRE is full of moments that feel like plot twists: systems fail, vendors disappear, and tools that once seemed essential suddenly become obsolete.

The key is not avoiding those moments—but learning how to respond to them.

🎧 Listen to the Full Episode

What Spider-Verse Teaches Us About Being an SRE

Leon often draws parallels between technology and storytelling, and one of his favorite examples comes from Spider-Man: Into the Spider-Verse.

In the movie, Miles Morales struggles to control his powers because he is overwhelmed by pressure. At one point he literally gets stuck to a ceiling because he cannot relax.

Leon sees that moment as a perfect metaphor for engineers during a production incident.

When systems break and the pressure is high, engineers can become overwhelmed by the situation. That stress can make it harder to think clearly and move forward.

Just like Miles learning to trust himself, SREs often need to pause, refocus, and trust their experience to navigate a difficult outage.

When a Technology Choice Falls Apart

Leon’s talk at SREday focused on a scenario many engineers eventually face: watching a technology choice fail.

Sometimes the failure comes from a vendor implosion.
Sometimes the product simply becomes obsolete.
Sometimes the tool just doesn’t live up to its promises.

In those moments, engineers may feel like the decision reflects badly on them.

But Leon argues that these situations often produce valuable outcomes.

When a tool collapses or a platform fails, teams are forced to rethink assumptions, improve architecture, and make better decisions moving forward.

What initially feels like a disaster can become an opportunity to build stronger systems and stronger teams.

Owning the Glitch

Another theme Leon emphasized is the importance of owning failures openly.

When something breaks, engineers can respond in very different ways. Some people try to hide the issue or shift blame. Others acknowledge the problem and focus on fixing it.

Leon believes the second approach leads to healthier engineering cultures.

Reliability engineering depends on transparency. Systems fail, and the best teams treat those moments as opportunities to learn rather than something to hide.

Owning the glitch helps organizations improve their systems—and helps engineers grow in the process.

Final Thoughts

Leon Adato’s message for SREs is simple but powerful.

Technology will always change. Tools will come and go. Systems will occasionally fail.

What matters most is how engineers respond to those moments.

Staying calm during outages, learning from failed technology choices, and approaching problems with honesty and humility are what ultimately make teams stronger.

And sometimes, a good lesson in reliability engineering can even come from a superhero movie.

🎧 Listen to the Full Episode

Subscribe to the ShipTalk Podcast

Enjoy conversations like this with engineers, platform builders, and reliability leaders from across the industry.

Follow ShipTalk on your favorite podcast platform and stay tuned for more stories from the people building the systems that power modern technology. 🎙️🚀

• Treat CI/CD migration like a developer platform launch: define what “no disruption” means, baseline your metrics, and set clear cutover rules.
  
  
• Migrate the foundations before the YAML: runners, networking, caching, and artifact handling determine whether feedback stays fast and reliable.
  
  
• Roll out in waves with parallel runs: start with a representative pilot, then expand using a repeatable checklist for readiness, performance, and rollback.

Modern engineering teams run on CI/CD. It’s where pull requests get validated, artifacts get produced, and releases get promoted to production. That also makes CI/CD migration very risky because you're not just moving a "tool"; you're moving the workflow that developers use dozens or hundreds of times a day.

The good news: disruption is optional. If you plan the migration like a product launch for developers, you can change platforms while keeping shipping velocity steady, often improving reliability, security, and cost along the way.

Harness CI can help you reduce migration friction by standardizing pipeline patterns and improving build performance without asking every team to rebuild their workflows from scratch.

What a CI/CD Migration Really Includes (and What to Defer)

A CI/CD migration is more than just "moving pipelines." In reality, you're moving or re-implementing four layers that work together:

• Workflow definitions: pipelines, templates, triggers, branch rules, environments, and approvals.
  
  
• Execution layer: build agents/runners, container orchestration, machine pools, concurrency, network access.
  
  
• Integrations and dependencies: source control, artifact registries, IaC tools, notifications, ticketing, scanners, and secrets.
  
  
• Governance: RBAC, SSO, approvals, audit logs, policy enforcement, and compliance evidence.

What to defer on purpose so you don’t disrupt developers:

• A full rewrite of every edge-case pipeline “to make it perfect.”
  
  
• A complete standardization effort across every language, framework, and release process.
  
  
• A platform-wide re-architecture that turns the migration into an 18‑month program.

Aim for parity first, then iterate for standardization and optimization once the new platform is stable.

CI/CD Migration Steps (A Practical Plan)

Use this step-by-step plan to migrate safely while developers keep shipping. Start with measurable guardrails, prove parity in a pilot, then scale with wave-based cutovers.

Step 1: Define “No Disruption” for Your CI/CD Migration (and Measure It)

You can’t protect developer experience if you don’t define it.

Start by writing a one-page “rules of engagement” that answers:

• What must keep working with zero/minimal downtime (for example: production deployments, security scans, release approvals)?
  
  
• What can tolerate change (for example: non-prod deploys, nightly builds)?
  
  
• What does rollback look like if a cutover fails?
  
  
• Who owns decisions, and who is on point when a pipeline breaks?

Then baseline two sets of metrics: delivery outcomes and pipeline health.

Delivery outcomes (DORA metrics)

• Deployment frequency
  
  
• Lead time for changes
  
  
• Change failure rate
  
  
• Recovery time/time to restore service (DORA has expanded the model over time)

You can use DORA’s official guide as your shared vocabulary and measurement reference.

Pipeline health

• Median and P95 pipeline duration (by pipeline type: PR checks, mainline builds, deploys)
  
  
• Queue time and agent utilization
  
  
• Failure rate (overall and by stage)
  
  
• Flake rate (tests that fail and pass without code changes)
  
  
• Cost per run (compute + licensing + developer time)

Tip: pick a small number of “must not regress” thresholds (for example: PR checks stay under your current P95, deployment approvals still work, and failure rate doesn’t spike).

Step 2: Inventory Your Current CI/CD Reality

Most migration pain comes from what you didn’t discover up front: the secret integration, the shared library, the one pipeline that deploys five services, the hardcoded credential that “nobody owns.”

Build a pipeline catalog with the minimum fields needed to plan waves and parity:

• Repo/service name and owner (team + on-call)
  
  
• Pipeline type (PR checks, mainline build, release, deploy)
  
  
• Triggers (branch rules, tags, schedules, manual)
  
  
• Environments and approvals (dev/stage/prod, gates, checks)
  
  
• Artifact outputs (container image, package, Helm chart, etc.)
  
  
• Integrations (registry, secrets manager, scanners, Slack/Jira, cloud accounts)
  
  
• Execution details (runner type, machine size, caches, custom images)
  
  
• “Break glass” notes (special cases, manual steps, tribal knowledge)

Then do two passes:

1. Critical path first: production deploy pipelines, shared templates/libraries, monorepo builds, release trains.
   
   
2. Representative complexity: to show how complicated things can get, add a few "messy but real" pipelines early on so you can find edge cases before the big wave.

If you’re planning migration waves, the Azure Cloud Adoption Framework has a good, useful overview of "wave planning" that works well for CI/CD moves if you're planning migration waves.

Step 3: Choose a CI/CD Migration Strategy That Keeps Teams Shipping

There are three common CI/CD migration strategies. The safest choice depends on your risk tolerance, your compliance constraints, and how tightly coupled your current system is.

Parallel run (recommended for most teams)

• Run old and new pipelines side-by-side until outputs match and reliability stabilizes.
  
  
• Use the new platform to build confidence before it becomes the system of record.

Strangler pattern (migrate shared steps first)

• Migrate shared templates, artifact publishing, caching, and scanning first.
  
  
• Move full pipelines once the building blocks and standards are stable.

Big bang (use only when forced)

• Sometimes required (tool EOL, hard compliance deadlines), but it needs rehearsals, rollback drills, and heavy coverage.

If you want one crisp rule: default to waves + parallel run. Avoid turning your CI/CD migration into a cliff.

Step 4: Design the Execution Layer Before You Move YAML

Developers don’t experience “YAML,” they experience feedback time and pipeline reliability. Execution decisions will make or break disruption.

Use this checklist to design the execution layer intentionally:

Where do builds run?

• Managed cloud build infrastructure, Kubernetes-based runners, VMs, or a mix.
  
  
• Network placement for private dependencies (databases, internal package registries).
  
  
• Egress controls and allowlists.

How do you protect performance?

• Dependency caching (language/package caches)
  
  
• Docker layer caching (if you build images)
  
  
• Reusing build outputs when inputs haven’t changed
  
  
• Concurrency limits and resource sizing

How do you handle artifacts and promotion?

• Standard artifact naming and versioning
  
  
• Artifact retention rules
  
  
• Promotion rules between environments

This is also where you can win developer trust quickly: if the new system’s PR checks are noticeably faster (or at least not slower), adoption becomes easier.

Step 5: Make Identity, Secrets, and Governance “Day 1” Work

CI/CD systems are a big target because if an attacker can change your pipeline, they can change what gets deployed. The U.S. CISA and NSA have published guidance just for protecting CI/CD environments. Use it to make your migration plan and your target platform more secure.

Treat security and governance as migration requirements, not a later phase.

Lock down access with RBAC + separation of duties

• Define who can edit pipelines and templates, manage connectors and secrets, approve promotions, and override gates.
  
  
• If you have separation-of-duties requirements, document them and build them into the model.

Prefer short-lived credentials for automation

• Static credentials in pipelines are a long-term risk.
  
  
• Where possible, use OIDC-based federation or workload identity.
  
  
• AWS’s guidance is explicit about preferring temporary credentials when you can.

Centralize secrets (and plan rotation)

• When you can, use an external secrets manager.
• Minimize secret exposure by keeping secrets out of logs and environment variables whenever possible.
• Before the cutover, make sure you know what rotation ownership and cadence are.
  Don't forget proof of compliance. 

Don’t forget compliance evidence. CI/CD migration often changes approval workflows, audit logging, and evidence retention. Validate evidence captured during the pilot, not at the end of wave three.

Step 6: Build a Migration Starter Kit Developers Can Copy

To avoid disrupting developers, you need a migration path that feels familiar and removes decision fatigue.

Build a “starter kit” that includes:

• Golden-path templates for the top 5–10 pipeline patterns (PR checks, mainline build, container build, deploy to stage, deploy to prod).
  
  
• Standard integrations, configured once: registries, IaC, scanners, notifications, tickets.
  
  
• Naming conventions (pipelines, stages, environments, artifacts) so teams can read each other’s pipelines.
  
  
• Docs for common tasks, written for developers:
  
  
  
  • How to add a new service
    
    
  • How to add an integration test stage
    
    
  • How to deploy to staging
    
    
  • How to request an exception

If your platform supports it, make guardrails policy-driven instead of copy/paste. For example: require scanning steps for certain artifacts, restrict prod deploy permissions, and enforce approved base images.

Step 7: Keep Developer Workflows Familiar With a Simple Rollout Plan

Even if the new platform is “better,” developers experience migration through small moments: Where do I rerun a build? How do I find logs? How do approvals work? Who do I ping when something is blocked?

A lightweight rollout plan reduces friction more than another week of pipeline refactoring:

• Publish a “before → after” map for the top workflows: trigger a build, view logs, download artifacts, rerun a failed step, request a prod approval, and roll back a deployment.
  
  
• Create a migration FAQ that answers the uncomfortable questions: “Will my pipeline break?”, “Do I need to learn a new syntax?” “What happens to my secrets?” “What if I’m on-call during cutover?”
  
  
• Time-box behavior changes. If you’re changing branch conventions, artifact naming, or approval flows, do it later unless it’s required for parity.
  
  
• Run enablement like onboarding. A 30-minute live walkthrough plus a recorded demo is usually enough for most teams.
  
  
• Make support visible. Pin your escalation path, office hours, and known issues in the channel developers already use.

Treat developer feedback as a platform signal. If teams struggle, it’s often because the golden path isn’t obvious yet, so improve templates and docs rather than asking every team to invent their own best practices.

Step 8: Pilot for Parity, Then Roll Out in Waves

A successful pilot proves three things:

1. Parity: the new pipeline produces the same artifacts and deploys the same way.
   
   
2. Reliability: failure rates and flakiness don’t spike.
   
   
3. Developer experience: feedback time and workflow friction are acceptable (or better).

Pick a pilot that is:

• Actively developed (not a dormant repo)
  
  
• Medium complexity (not the simplest “hello world,” not the most mission-critical)
  
  
• Owned by a team willing to give feedback quickly

Prove parity with a parallel run window

• Compare artifact digests, test outcomes, deploy behavior, and approvals.
  
  
• Track top failure reasons and fix templates, not just the pilot pipeline.
  
  
• Publish a short “pilot report” so leadership and developers see proof, not promises.

Roll out in waves with a cutover checklist.

 For each wave, define a “ready to cut over” checklist:

• Success rate meets a threshold (for example: within X% of baseline)
  
  
• Performance is within bounds (for example: PR checks P95 not worse than baseline)
  
  
• Approvals, RBAC, and audit logging verified
  
  
• Rollback tested (you can revert to the old system quickly)

Run migration like a service

• A dedicated Slack channel and published escalation path
  
  
• Office hours during the first waves
  
  
• “Champions” in each org who can answer common questions

Step 9: Optimize, Decommission, and Prevent Pipeline Drift

Once most teams are migrated, the work shifts from “move” to “make it better.”

Improve speed and reliability (without churn)

• Tighten caching and reuse outputs
  
  
• Split slow tests and reduce flakiness
  
  
• Right-size runners and concurrency
  
  
• Remove redundant stages (duplicate scans, repeated builds)

Prevent drift. If teams can fork templates endlessly, you’ll end up with a new version of the old problem. Decide where standardization is required and where flexibility is allowed:

• Standardize: security gates, artifact publishing, environment promotion rules, audit logging
  
  
• Flexible: language-specific steps, unit test frameworks, and optional quality checks

Retire the old system safely before decommissioning:

• Confirm audit log retention requirements are met.
  
  
• Rotate or delete legacy credentials and service accounts.
  
  
• Remove access and document the “new normal.”

Common CI/CD Migration Pitfalls (and How to Avoid Them)

• Migrating YAML without migrating execution reality. Fix runners, caching, and networking first.
  
  
• Treating security as “phase two.” CI/CD is part of your software supply chain; harden identity and secrets early.
  
  
• Over-standardizing too soon. Move the 80% path first; handle exceptions with a time-boxed process.
  
  
• No baselines. If you didn’t measure pipeline health before, you can’t prove improvement after.
  
  
• No support model. Developers won’t “just adopt” a new platform during busy release cycles.

Keep Developers Shipping, Then Make the New System Better

A successful CI/CD migration is repeatable: define success, inventory the real system, and design execution and security before you touch every pipeline. Prove parity in a pilot, then roll out in waves with clear cutover and rollback rules so teams can keep shipping.

Once the new platform is stable, use your baselines to optimize build speed, reliability, and governance, and decommission the old system cleanly to prevent drift and orphaned credentials. If you’re looking for a pragmatic way to standardize pipelines and shorten feedback loops as you migrate, Harness CI can help.

CI/CD Migration: Frequently Asked Questions (FAQs)

These FAQs cover the practical questions teams ask during a CI/CD migration: timelines, sequencing CI vs. CD, and how to reduce risk during cutover.

How long does a CI/CD migration take?

For many teams, a safe migration happens in waves over 6–12 weeks, starting with a pilot and expanding based on readiness. The timeline depends more on integrations, governance, and execution infrastructure than on pipeline definitions.

Should we migrate CI and CD at the same time?

Not always. If your deploy workflows are complex or tightly governed, migrating CI first can reduce risk while you validate identity, artifacts, and approvals. In other cases, migrating CI and CD together can simplify end-to-end standardization, just keep the rollout wave-based.

What’s the safest way to cut over production deployments?

Use a parallel run window, validate parity (artifacts, approvals, behavior), and enforce a cutover checklist with rollback steps rehearsed. Avoid silent changes, announce the cutover, and provide a clear escalation path.

How do we handle secrets and credentials during the migration?

Start with an inventory, move toward short-lived credentials (for example, OIDC federation), and centralize secrets where possible. Rotate credentials during cutover and delete legacy service accounts once decommissioned.

How do we prove the migration improved developer productivity?

Compare pre- and post-migration baselines: PR feedback time, pipeline reliability, queue time, time-to-fix failures, plus DORA metrics where you can measure them. Share results with developers so the migration feels like an improvement, not change for change’s sake.

What should we standardize vs. keep flexible?

Standardize what protects the organization (security gates, artifact promotion rules, audit logging, prod approvals). Keep flexibility where teams need it (language tooling, test frameworks, optional quality checks), and use templates to make the right path easy.

‍

Visualizing the DNA of Distributed Attacks

We are thrilled to announce the launch of Radar, a major update to Harness Bot & Abuse Protection that fundamentally changes how security teams investigate and respond to sophisticated threats.

Sophisticated bot attacks are rarely isolated incidents. They are complex networks of connected identities, shared fingerprints, and coordinated behaviors. Traditional lists and tables pulled from disparate logs and event data fail to capture this reality, forcing SOC analysts to manually correlate relevant attack data—a process that costs hours when seconds matter in a potential breach.

Introducing Radar in Harness Bot & Abuse Protection

Radar changes the game for bot attack response. It is an intelligent threat visualization engine that instantly unmasks hidden relationships in bot attacks, turning the abstract "DNA" of an automated threat into a clear, interactive map.

It helps with

• Speed to Verdict: What used to take hours of manual cross-referencing now takes seconds of visual inspection.
• Explainable Security: Radar "white-boxes" our detection logic, giving you visual proof of why we flagged a cluster of users.
• Enterprise-Grade Depth: Built to meet the rigorous demands of leading financial institutions and E-Commerce, Radar handles the complexity of high-stakes fraud investigations with ease.

Why SecOps Teams Need Threat Visualization

SOC analysts struggle to quickly respond to advanced threats and bot attacks targeting their organization’s systems. Analysis of recent breach data shows that on average organizations take roughly six months to identify the related incident. Advanced automated bot and abuse attacks, increasingly accelerated with AI, frequently go undiscovered for this time period. The harsh reality is that data relationships and system interconnects in enterprise environments are complex to unwind as automated threats play out, leaving significant incident response gaps for threat detection.

• Inhibited response: Security teams spend days, weeks, or months manually cross-referencing siloed data, delaying incident response when speed is critical.
• Opaque Security: Detection logic in other bot mitigation offerings is a black box, leaving SecOps teams with little insight or evidence to justify why users or clusters are flagged.
• Limited Depth: Existing tools struggle to handle the complexity and scale of high-stakes fraud investigations, falling short of enterprise requirements for financial institutions and e-commerce.

Radar Key Capabilities

1. The Unified Threat Canvas

Stop investigating accounts in silos. Radar projects user connections onto an interactive graph, instantly revealing the shared threat signatures—such as IP subnets, device fingerprints, and behavioral patterns—that bind disparate accounts and activities together. You can now see the shape of the attack campaign at a glance.

‍

2. AI-Powered Context

Understanding why accounts are connected is just as important as knowing that they are connected. Radar integrates deep AI insights that analyze the cluster to provide a narrative summary of the attack. Whether it's a "distributed credential stuffing attack via a residential proxy ring" or "coordinated inventory hoarding," Harness AI explains the security context so you don't have to guess.

3. Portable Forensics

We know your investigation doesn't always end in our console. Radar features a robust export engine that packages every data point—user details, threat activities, and linkage evidence—into a comprehensive format ready for your SIEM and SOAR tools or related DFIR workflows.

‍

Get Started Today

Radar is available immediately with your existing Harness Bot & Abuse Protection subscription. There is no additional cost or setup required.

• Current Customers: Log in to your dashboard today to start exploring your threat data in a whole new dimension.
• New to the Platform? If you aren't yet protected, contact us to schedule a personalized demo of Radar in action.

Request a demo

• Flaky tests waste 16–24% of developers' time on average because of false failures, re-runs, and investigations. This costs engineering companies millions of dollars in lost productivity every year.
• Timing problems, test pollution, unstable infrastructure, and race conditions are all possible root causes. However, AI-powered detection can automatically identify and isolate flaky tests, improving signal quality.
• Harness CI and other modern CI platforms use machine learning to identify flaky tests, automatically quarantine unreliable tests, and maintain developers' trust without manual triage or pipeline interruptions.

‍

What are Flaky Tests?

Flaky tests are automated tests that pass or fail inconsistently without changes to the code. In this guide, you’ll learn why flaky tests happen, how to detect them automatically in CI pipelines, and how modern platforms prevent them from slowing teams down.

Your test went well three times yesterday. It didn't work this morning. You ran it again without changing anything, and now it works. Congratulations, you've just passed a flaky test, and now someone's day is going to be ruined.

Flaky tests are like smoke alarms that go off for no reason. Everyone looks into it the first few times. Eventually, your entire test suite stops being an early warning system and becomes background noise. Harness CI uses AI to automatically identify flaky tests and put them in quarantine, so your pipelines send you reliable signals instead of random noise.

Why Tests That Don't Work Are Expensive (Even If You Don't Pay Attention to Them)

The 30 seconds it takes to hit "retry" isn't the real cost of flaky tests. It's everything that happens after developers stop trusting the test results.

Developer Time Goes Up in Investigating Black Holes

Someone has to figure out if a test failure is a real bug or just flakiness. An industrial case study found flaky tests consuming about 2.5% of developers' productive time - 1.1% on investigation, 1.3% on repairs, and 0.1% on tooling. For a team of 50 engineers, that's the equivalent of more than one full-time engineer's worth of work... gone.

And that's the best-case scenario, where teams really look into things. The worst-case scenario is that developers think everything is flaky, stop looking into failures, and real bugs make it to production. You're paying for tests that hurt your confidence instead of helping it.

Changing Contexts Breaks the Flow State

This is what really happens when a flaky test breaks your build. You're deep into the code, working on a complicated feature. The build doesn't work. You stop, switch contexts to look into the problem, find out it's not your fault, run the pipeline again, and wait. When the green build comes back 15 minutes later, you've lost your train of thought and spent 20 minutes on Slack instead.

Studies on productivity show that it takes 15 to 25 minutes to get back to full focus after being interrupted. If you have dozens of flaky test interruptions every week across your team, you're losing a lot of productive hours.

The Hidden Multiplier Is Trust Degradation

The cultural cost is the most harmful. When tests stop working, developers find other ways to do things. They automatically run builds again. After the third retry passes, they combine PRs with red builds. They stop making new tests because "tests are flaky anyway."

This loss of trust gets worse over time. Teams that tolerate flaky tests have lower test coverage, longer feedback loops, and more problems in production. Your quality assurance system will only be useful if developers trust the test results.

Root Causes: Why Tests Fail and How to Tell When They Do

The first step in fixing tests is to figure out why they fail. You can hunt down flaky tests in a systematic way instead of playing whack-a-mole because most of them follow a pattern.

Timing and Race Conditions: The Usual Suspects

Assumptions about timing are the main reason why tests fail. Your test says that element X should be ready in 100ms. It's always ready in 80 milliseconds on your laptop. It takes 120ms on a shared CI runner that is busy. Boom, failure that happens sometimes.

You could have problems with network calls, database queries, UI rendering, or async operations if you have to "wait for something to happen." Hard-coded sleep statements are especially bad because they're either too short (flaky) or too long (slow tests that waste time even when they pass).

The fix is to use explicit waits with timeouts: wait for specific conditions (such as an element becoming visible, an API response being received, or a state being updated) rather than arbitrary time intervals. You need to find out which tests have these problems first.

Pollution in Tests and Shared State

Tests that depend on the order in which they are run or share mutable state are like ticking time bombs. Test A runs first and puts data into the database. Test B assumes that the data is there. If you run them in parallel or in the opposite order, Test B fails at random.

Global variables, singleton patterns, shared file systems, and database records that don't get cleaned up all make tests depend on each other in ways that aren't obvious. When you run your tests in parallel to speed them up, test pollution shows up in a big way.

Test Intelligence helps by looking at test dependencies and running tests that are affected in isolation, which makes them less flaky because of pollution.

Unstable Infrastructure and Environment

The test is fine, but the setting isn't always. Network problems, shared CI runners fighting for resources, external API rate limits, and database connection pool exhaustion are all environmental factors that can cause your code to fail from time to time.

This is why teams that use shared, static Jenkins clusters have more problems than teams that use ephemeral build environments. You get rid of the "noisy neighbor" problem completely when every build runs in a clean, separate space with its own resources.

Code That Isn't Deterministic and Dependencies From Outside

Tests that rely on the current time, random number generation, external APIs, or other inputs that aren't always the same will eventually fail. Anything that isn't completely under your control in your test setup could cause flakiness. For example, today's date changes, APIs go down, and random seeds give you different values.

Dependency injection and test doubles are the answer. For example, you can mock the clock, stub external APIs, and seed random generators in a way that is predictable. But first, you need to know which tests have these problems.

Detection Strategies That Really Work

You can't fix something if you can't see it. The first step is to make systems that automatically show flaky tests instead of making developers remember and report them.

Using AI to Find Flaky Tests Automatically

It doesn't work to keep track of flaky tests by hand. You need automated detection that watches test runs over time and finds patterns that show flakiness.

AI-powered test intelligence looks at past test results to find tests that pass and fail on the same code without making any changes. After just a few runs, machine learning models can find flaky behavior and flag tests for further investigation before they turn into big problems.

The most important thing is to run the same test suite on the same code several times. Newer platforms can do this automatically without any help from people.

Quarantine Systems That Keep the Pipeline Signal

You have a problem when you find a flaky test. If you turn it off, you won't be able to test it. If you leave it running, it will keep breaking builds and teaching developers to ignore failures.

The answer is automatic quarantine. Put a flaky test in quarantine so it can still run, but doesn't block the pipeline. Failures are recorded and tracked, but developers don't have to deal with random failures from tests that are known to be flaky.

This keeps the quality of the signals in your main test suite while letting platform teams see the tests that are in quarantine and need to be fixed. You're separating the noise from the signal without losing either.

Using Flaky Test Rate as an Important Metric

Along with build duration and deployment frequency, treat flaky test rate as a top operational metric. Healthy test suites keep flaky rates below 1–2%, while rates above 5% show that there are big problems.

Keep an eye on this over time to see if it changes. A sudden spike usually means that the infrastructure has changed or that new code patterns have made things less stable. Platform teams should set up alerts and SLOs for flaky test rates so they can catch problems early.

How to Fix Flaky Tests: Best Practices That Work

Finding the problem is half the battle. You can't just hide the problems anymore; you need to use systematic methods to fix them.

First, Isolate and Reproduce

You need to be able to consistently reproduce the failure before you can fix a flaky test. Run the test hundreds of times on your own computer or in CI until you see how it fails.

Tools that make it easy to run tests again and again are helpful here. Some platforms let you run a single test 50 times with a single command, making it easy to find intermittent failures. Once you can consistently reproduce the failure, it becomes easier to investigate.

Should You Fix the Test or the Code?

Not all flaky tests are bad tests. Sometimes, flakiness in your production code indicates real race conditions, timing issues, or behavior that isn't always consistent.

Think about this: Is this flakiness testing something that could happen in production, or is it just a result of how we wrote the test? The flakiness is a signal that users could see this timing problem. Make the code work. If it's just a test artifact, fix the test.

Common Problems and Their Solutions

Different flaky test types need different fixes:

• Problems with timing: Instead of hard-coded sleeps, use explicit waits. When you have external dependencies, use retry logic with exponential backoff. If you need to, increase the timeouts, but it's better to make operations go faster than to make tests wait longer.
• Resource contention: Create temporary directories, test-specific namespaces, and database schemas. After tests, clean up the resources. Don't use global shared state.
• External dependencies: Mock APIs and services. Use test doubles. Use circuit breakers and fallbacks for integration tests that require connecting to real services.
• Inputs that aren't deterministic: Seed random number generators. Mock the clock on the system. Instead of generated data, use fixed test data.

Refactor to Make Things More Certain

The end goal is to make your test suite completely deterministic. Every time, the same code gives the same test results. This means making choices about architecture:

• Dependency injection that lets you use test doubles
• Whenever possible, use pure functions that don't have side effects
• There is a clear line between business logic and I/O
• Test fixtures that make a clean, separate state

These are good software design rules that make your production code more reliable, not just for tests.

Creating a Culture That Stops Flaky Tests

Flaky tests can't be fixed by technology alone. You need team rules and practices that stop flakiness from building up in the first place.

Make Flakiness Clear and Not Okay

Teams put up with what they keep track of. Flaky tests spread when you can't see them. Make the flaky test rate a dashboard metric. During code reviews, point out tests that are flaky. When you add flaky tests, think of them as production bugs that you should avoid and fix right away.

Some teams have a "you flake it, you fix it" policy, which means that the person who wrote the flaky test is responsible for finding out what went wrong and fixing it. This makes people responsible and encourages them to write stable tests ahead of time.

Put Money Into Test Infrastructure

Flaky tests are often a sign that the test infrastructure isn't good enough. Flakiness comes from shared, overloaded CI runners. So do test environments that are too fragile and test tools that are missing.

Platform teams should give:

• Isolated build environments for each test run
• Libraries of test fixtures that are in good shape
• Clear patterns and examples for common test situations
• Local CI environments are easy to set up again

Flakiness goes down naturally when it's easier to write stable tests than flaky ones.

Keep Fast Unit Tests and Slow Integration Tests Separate

When you mix fast, predictable unit tests with slow, environment-dependent integration tests, the integration test flakiness spreads to everything else. Instead of just the integration layer, developers learn not to trust any tests.

Group test suites by how fast and stable they are. Every time you commit, run fast, stable unit tests. Run integration tests less often or on a different track. Test Intelligence will only run the integration tests that are needed based on changes to the code.

This tiered approach means that most developer feedback comes from quick, reliable tests, and full integration coverage still happens without breaking the inner loop.

How Modern CI Platforms Automatically Deal With Flakiness

When a team gets too big, manual flaky test management doesn't work anymore. Modern platforms use automation and smart technology to solve the problem.

ML-Powered Detection That Gets Better Over Time

Harness CI uses machine learning to look at test patterns from thousands of runs. The system learns which tests tend to fail, when, and how often.

This is more than just finding out if someone "passed then failed." Advanced algorithms can find patterns like "fails more often under load," "flakes in parallel but not sequential runs," or "only flakes on certain OS versions."

The longer the system runs, the better it gets at telling the difference between real problems and false alarms.

Automatic Quarantine Without Any Human Action

The system automatically quarantines when it finds a flaky test. No platform team meetings, no filing tickets by hand, and no arguing about whether this test is "flaky enough" to be quarantined.

Quarantined tests still run and report results, but they don't stop builds or count as failures. Developers can look into quarantined tests when they have time, but they aren't held up by random failures.

This keeps both coverage (tests still run) and signal quality (builds aren't randomly red).

A Lot of Analytics and Reporting

Platform teams need to see not only the status of individual tests, but also the trends of flaky tests. Dashboards on modern CI platforms show:

• Flaky test rate over time for all teams and repositories
• The most problematic tests, ranked by how much they affect the system (frequency × failure count)
• Metrics for quarantine status and time spent in quarantine
• Patterns of root causes and suggested fixes

This information helps decide which problems to fix first and shows whether the flakiness is improving or worsening over time.

Real-World Effect: What Teams Get Out of Fixing Flakiness

When teams deal with flaky tests in a planned way, the benefits spread across many areas.

Developer productivity returns: Teams say they get 10–20% more done after eliminating flaky tests. This is because they don't have to spend time on false investigations and reruns.

Restoring trust: Developers only pay attention to failures and look into them thoroughly when they trust the test results again. This finds real bugs sooner and improves the quality of production.

Faster feedback loops: PR validation runs finish faster and provide useful feedback the first time, without needing to retry or investigate failures.

Less expensive infrastructure: Teams stop running tests "just to be sure" or the whole suite because they don't trust selective execution. When the tests that Cache Intelligence and test selection are based on are reliable, they work better.

Cultural change: Getting rid of flakiness shows that the platform team cares about developers' experience. It gives other CI improvements greater credibility and moves the whole company toward better testing practices.

One engineering team reported cutting test maintenance from around 10 hours per week to about 2 hours per week by aggressively removing and refactoring flaky end-to-end tests. Another organization claimed flaky tests cost them 40 hours per week before they deleted 70% of their problematic tests. With systematic detection, quarantine, and remediation, teams see faster builds, happier developers, and fewer production incidents.

Stop Putting Up with Flaky Tests

Flaky tests don't have to happen all the time when you make software. They're a sign of not having the right tools, not following the right practices, and having too much technical debt.

To fix the problem, you need three things: automated detection to identify where the flakiness is, systematic remediation to fix the root causes quickly, and preventive practices to ensure new flakiness doesn't build up faster than you can fix old problems.

All three of these things are made smarter and more automated by modern CI platforms. AI-powered detection finds flaky patterns on its own. Quarantine systems maintain signal quality without blocking teams. Analytics reveal patterns and help set priorities for problem-solving.

Your developers shouldn't have to be detectives every time a test fails. Make flaky tests someone else's problem, like the CI platform's, so your team can spend less time fixing test infrastructure and more time adding new features.

Are you ready to get rid of flaky tests in your pipelines? Learn how Harness Continuous Integration uses AI to find flaky tests, put them in quarantine, and help fix them on their own.

Flaky Tests: Frequently Asked Questions

How many tests that are flaky are normal?

Healthy test suites keep flaky rates between 1% and 2%. You have a systemic problem that needs to be fixed right away if more than 5% of your tests are flaky. 

Should I get rid of flaky tests or try to fix them?

Not at first. Quarantine flaky tests first, so they don't stop builds but still send signals. Then look into whether they're showing real problems or just poorly written tests. If they're testing important situations, make sure they work. Think about deleting them if they are unnecessary or not worth much.

How long does it take to fix a test that keeps failing?

It can take anywhere from 15 minutes for simple timing issues to several days for more complicated race conditions or architectural problems. The average time for all the studies is 1 to 3 hours per test. This is why it's important to automate detection and prioritization: you want to fix the flaky tests that have the biggest effect first.

Can flaky tests show bugs in production?

Yes. Some flaky tests show real race conditions, timing problems, or behavior that isn't always the same, which could affect users. Don't just call a flaky test "just a bad test." Look into whether it's showing real problems with the code. Flakiness can sometimes be a signal, not just noise.

Do parallel test runs make things more flaky?

Parallel execution shows problems that sequential runs hide, like test pollution, race conditions, and resource contention. The parallelism isn't causing problems; it's just showing problems that were always there. Instead of avoiding parallelism, fix the root problems.

How do tools that use AI find flaky tests?

Machine learning models look at test results from hundreds or thousands of runs and find patterns like "passes and fails on the same code," "fails more often under certain conditions," or "failure rate correlates with infrastructure load." These systems are much better and faster at finding flaky tests than people are.

‍

• CI/CD best practices let teams deliver software faster by creating a standard way to build, test, and deploy.
• Making small, frequent changes and using fast, reliable pipelines lowers risk and keeps builds stable.
• The right platform helps your teams make CI/CD best practices part of their daily routine.

Modern software teams are under constant pressure to ship faster without breaking production. That’s why CI/CD best practices have become essential for high-performing DevOps organizations. Continuous integration and continuous delivery (CI/CD) help automate builds, testing, and deployments — but simply installing a pipeline tool isn’t enough. Without the right practices, pipelines become slow, flaky, and difficult to govern.

In this guide, we break down the most important CI/CD best practices for building fast, stable pipelines - from trunk-based development and intelligent test selection to progressive delivery and DORA metrics.

Implementing Continuous Integration and Continuous Delivery (CI/CD) has become a critical success factor. CI/CD enables teams to rapidly and reliably deliver high-quality software by automating the build, test, and deployment processes. However, simply adopting CI/CD is not enough; to truly reap the benefits, teams must follow best practices that ensure efficiency, reliability, and consistency. In this blog post, we'll explore key CI/CD best practices and how the Harness Software Delivery Platform can help you optimize your software delivery pipeline.

What Are CI/CD Best Practices?

CI/CD best practices are the habits that keep your pipelines fast, reliable, and predictable as your teams and systems grow. They guide how you commit and review code, build and test artifacts, deploy changes, and measure and improve the process. When teams follow the same best practices, there are fewer surprises in production, less time spent fixing deployments, and more time to deliver new features.

This guide covers the most important CI/CD best practices and explains how they help create a strong software delivery process.

1. Commit Early, Commit Often

Making frequent, small integrations is a simple but powerful CI/CD best practice. It helps keep your pipeline fast and your main branch stable.

• Try to merge code into the main branch often, ideally once per developer each day. This keeps merge conflicts small and helps you get feedback quickly.
• Split your work into small, reviewable pieces that can pass tests and be deployed on their own, instead of using long, risky branches.
• Run fast checks locally before pushing to avoid trivial red builds and wasted CI cycles.
• Keep branches short-lived and use trunk-based development. This keeps your history clean and makes sure the main branch is always ready to release.
• Use fast, intelligent pipelines so frequent commits do not slow anyone down. Run only the tests affected by each change.
• Use incremental builds to cut down rebuild time. This way, developers get feedback in minutes, even when they commit often.
• Use CI analytics to track build and test performance over time. This data helps you keep the 'commit early, commit often' approach working as your codebase grows.
• In trunk-based development, developers integrate into the main branch at least once per day. In GitFlow-style workflows, this principle still applies — but changes are merged into short-lived feature branches that integrate back into main quickly.
  

2. Get Back to Green Quickly

A green build is a happy build. In CI/CD, it's crucial to maintain a stable and reliable build process. If the build is failing, it should be the top priority to fix it. Failing not only hinders the delivery process but also erodes team confidence and productivity. Implement automated tests, linters, and code quality checks to catch issues early and ensure that the main branch remains in a deployable state.

This said, if tests are never failing and the build never turns red, you are probably not testing well enough or moving quickly enough. The occasional broken build is fine. The team simply needs to prioritize

Harness CI offers extensive testing capabilities, including automated unit, integration, and acceptance tests. With Harness's Test Intelligence feature, you can optimize your test execution by automatically identifying and running only the tests affected by code changes, saving time and resources.

3. Build Only Once

Building artifacts multiple times across different stages of the pipeline introduces unnecessary complexity and inconsistency. Instead, adopt the practice of building once and promoting the same artifact through the various stages of testing and deployment. This ensures that the artifact being tested and deployed is the same one that was built, reducing the risk of introducing discrepancies.

Harness simplifies artifact management with centralized artifact storage. You can store and version your build artifacts in one place, ensuring the same artifact is promoted consistently through every stage of your CI/CD pipeline. This practice is often called artifact immutability, i.e., build once, then promote the exact same artifact across staging and production to prevent environment drift.

4. Standardize Pipelines with Templates and DRY Patterns

If every team has its own one-off pipeline, CI/CD best practices will never stick. Standardization is how platform teams encode the “golden path” and keep pipelines maintainable over time. Start by identifying the common stages every service needs, such as build, unit tests, security scans, and deployment to staging and production, then capture those stages in reusable templates. Give application teams a clear extension model so they can add service-specific steps without copy-pasting entire pipelines. This DRY approach makes it easier to roll out improvements, because you change the template once instead of editing dozens of separate configurations. 

Harness pipeline templates are built for exactly this: platform engineers define the shared workflows, while product teams plug into those templates and still keep the autonomy they need.

5. Streamline Your Tests

Slow, noisy test suites can quickly ruin CI/CD best practices by making every commit a long wait. The goal is to keep quality high and make your pipeline smart about which tests run and when.

1. Organize your tests in layers. Start with fast unit tests, then run API or integration tests, and finish with a few end-to-end or UI tests. This way, developers get feedback in minutes instead of hours.
2. Run fast tests early and often. Trigger unit and key integration tests on every commit, and reserve full regression suites for scheduled or pre‑release runs.
3. Parallelize wherever you can. Split them across multiple workers or nodes so the total test time depends on your infrastructure, not just your codebase size.
4. Continuously prune and refactor. Remove duplicate tests, fix or isolate flaky ones, and keep your suites small so you get clear, useful results.
5. Use intelligent test selection rather than brute-force testing. Test impact analysis lets you run only the tests affected by a change, keeping coverage high and cutting pipeline time.
6. Keep an eye on your test metrics. Use CI analytics to track test duration, failure rates, and other trends. Use this data to improve your test suites.

Most high-performing CI/CD pipelines follow the testing pyramid:

• Many fast unit tests
  
  
• Fewer integration tests
  
  
• Minimal end-to-end/UI tests

6. Secure Your CI/CD Pipeline

Security should be part of CI/CD from the start, not added at the end. Begin by keeping secrets out of source control, limiting who can change pipelines and environments, and using SSO and multi-factor authentication for access.

Next, make security checks a main part of your pipeline, not just an extra step. Add dependency scans, container image scans, and policy-as-code steps to block non-compliant changes before they go live.

Strong audit trails are another core CI/CD best practice, so you always know who deployed what, when, and where. Harness supports these practices with environment-aware RBAC, policy-as-code, and detailed deployment history, so you can move fast without losing control.

Modern CI/CD best practices include embedding SAST, DAST, container scanning, and SBOM generation directly into pipelines to support DevSecOps and supply chain security initiatives.

7. Clean Your Environments

Consistent and reliable environments are essential for successful CI/CD. Ensure that your environments are versioned, reproducible, and disposable. Use infrastructure-as-code (IaC) practices to define and manage your environments, enabling version control and easy rollbacks. Clean up environments after each deployment to avoid configuration drift and ensure a fresh start for the next deployment.

Harness provides robust deployment and environment management capabilities. With Harness's IaCM, you can define and manage your environments using popular IaC tools like Terraform, CloudFormation, and Kubernetes manifests. Harness also supports automatic environment cleanup, keeping your environments clean and consistent.

8. Make It the Only Way to Deploy to Production

To ensure consistency and reliability, establish your CI/CD pipeline as the sole path to production deployment. Discourage manual deployments or ad-hoc changes to production environments. By enforcing deployment through the pipeline, you maintain a standardized and auditable process, reducing the risk of human error and enabling easier rollbacks if needed.

With Harness's pipeline governance features, you can enforce policies and approvals, ensuring that only authorized changes make it to production.

9. Release Progressively

Deploying an entire application all at once is no longer in vogue. We now understand that deploying little by little delivers a better user experience while minimizing risks. Consider deploying an application to a cluster using techniques like a Canary deployment. Canary deployments deploy the new version alongside the existing, sending only a small amount of traffic to the new one. Only after seeing that users are successful with the new version is the deployment completed, removing the old version. This approach exposes only a few users to the new version at first, helping minimize the risk and ensuring that rollback (disabling the new version) is easy.

Another approach to progressive delivery is to enable individual features separately from releasing the new version of the code. A feature management tool will allow you to first see that the new version of the code is stable, then experiment with each new feature, making sure they have the desired impact. This approach refines your CD significantly.

10. Monitor and Measure Your Pipeline

To keep improving your CI/CD process, you need to see how your pipeline works in real situations. Track basics like how long pipelines take, where they fail most, and how often deployments succeed or need rollbacks. Use analytics to find bottlenecks, spot slow or flaky stages, and check if your changes help. Treat this as an ongoing feedback loop: review the data, pick one thing to improve, make the change, and check the results. For a more detailed view, you can add DORA metrics, which we’ll discuss next.

11.    Measure CI/CD Health with DORA and Beyond

You can’t improve what you don’t measure, and CI/CD is no different. Start with the four DORA metrics: deployment frequency, lead time for changes, change failure rate, and mean time to recovery (MTTR). These show how fast you deliver changes, how often things go wrong, and how quickly you recover.

As you get more advanced, add other metrics like build time, test flakiness, or time waiting for approvals to find specific pipeline bottlenecks.

A key CI/CD best practice is to make these metrics visible to your team, review them often, and connect process changes to real improvements. Harness helps by showing delivery analytics from your pipelines, so you can see your metrics change as you improve.

12. Make It a Team Effort

CI/CD isn't just a tool or a process; it's part of a DevOps culture. Get everyone involved, including developers, testers, and operations, when designing and running your CI/CD pipeline. Encourage teamwork and shared ownership so everyone helps improve the process. Offer training and support to make sure everyone understands and follows best practices.

Harness supports collaboration and teamwork through features like role-based access control (RBAC) and policy-as-code. You can define granular permissions and policies to ensure that team members have the right level of access and control over the pipeline. Harness also integrates with popular collaboration tools, making it easy to share information and work together effectively.

Harness Your CI/CD Potential

While following CI/CD best practices is essential, having the right tools and platform can greatly streamline and enhance your software delivery process. The Harness Software Delivery Platform streamlines software delivery so pipelines stay fast and reliable instead of becoming another source of toil. 

Harness CI accelerates builds and tests with intelligent caching, optimized cloud builds, and features like Harness Test Intelligence to prioritize the most relevant tests and shrink feedback cycles. Out-of-the-box integrations and templates minimize custom scripting and heavy configuration, so teams can onboard quickly and focus on delivering features, not wiring tools together.

Governance and compliance are built in rather than bolted on. With granular RBAC and policy-as-code, including DevOps pipeline governance,  you can enforce approvals, security scans, and compliance checks, without blocking developers.

Putting CI/CD Best Practices into Action

CI/CD best practices help teams move from fragile, unpredictable releases to a steady, reliable delivery process. By committing early and often, keeping builds green, building once, streamlining tests, securing and cleaning environments, using the pipeline for all production deployments, releasing in stages, and tracking key metrics, you build a pipeline that supports fast change. Start with one or two practices, make them habits, and add more over time. Soon, your CI/CD pipeline will be a strength, not a bottleneck.

If you want a platform that bakes these practices into your day-to-day workflows, try Harness and see how quickly your CI/CD pipeline can evolve.

Frequently Asked Questions About CI/CD Best Practices

What are the most important CI/CD best practices to start with?

If you’re just starting out, focus on a few CI/CD best practices that give the most value: commit early and often, keep the main branch ready to deploy, run automated tests on every change, and use the pipeline as the only way to reach production. Once you have these basics, you can add progressive delivery, security checks, and advanced governance without overwhelming your team.

How do CI/CD best practices change for microservices versus a monolith?

The main principles don’t change, but the impact is bigger with microservices. You need consistent templates and standards so every service uses the same process for builds, tests, and deployments. You also need better observability and progressive delivery, since one release might involve several services rolling out together instead of just one big application.

How can we speed up our pipelines without sacrificing test quality?

Start by cutting out obvious waste: remove duplicate tests, fix or isolate flaky ones, and run fast unit tests early so developers get quick feedback. Use test impact analysis and incremental builds to avoid repeating work that hasn’t changed. The goal is to keep quality high while making the pipeline smart about which tests matter for each change.

What metrics should we track to know if our CI/CD is healthy?

Start by tracking the four DORA metrics, since they show how fast and stable your process is: deployment frequency, lead time for changes, change failure rate, and MTTR. Then add a few extra metrics that fit your team’s needs, like average build time, CI queue time, or time from merge to production. Healthy pipelines have frequent, small deployments, short lead times, low failure rates, and quick recovery when things go wrong.

How do we keep CI/CD secure while still enabling fast releases?

Make security checks part of your automated pipeline, running on every change instead of being done manually at the end. Use a secret manager, limit access to CI/CD systems, and add vulnerability scans and policy-as-code rules to your pipelines. When these controls are built into the process, developers can move quickly while the pipeline enforces security and compliance.

When is it time to adopt progressive delivery and feature flags?

If deployments start to feel risky or you delay releases 'just in case,' it’s time to try progressive delivery and feature flags. Strategies like canary and blue/green deployments let you release more often by limiting the impact of each change. Feature flags let you turn features on or off without redeploying. These approaches turn big, stressful launches into smaller, safer steps that fit well with modern CI/CD.