How Harness orchestrates LLM security scanning | Harness Blog

Large language models are effective at security review for the same reason they are effective at many other tasks: they reason rather than pattern match. In plain terms, a traditional scanner checks code against a list of known bad patterns, the way a spell checker flags a misspelled word, regardless of what the sentence means. An LLM can instead follow the program's logic: trace a piece of attacker-controlled input through several layers of application code to determine whether it is reachable. That is a genuine advantage over many traditional static application security testing (SAST) tools, which can only catch what someone has already written a rule for.

That reasoning ability comes at a cost. Point the same model at the same, unchanged repository twice, and the two scans can return different findings, even though nothing in the code changed. Give it an open ended instruction like "find the vulnerabilities in this codebase" and it will spend an unpredictable amount of time and compute doing so, and the resulting report will vary in shape and detail from run to run. For a scanner meant to run automatically on every commit, that is a real problem: a security backlog that reshuffles itself with no corresponding code change is difficult for any team to trust or act on.

None of this is a reason to avoid using an LLM for security scanning. It is a reason you cannot simply point a model at a repository and call the result a scanner. What we built in Harness Security Test Orchestration (STO) is the layer that sits around the model, the part that decides what the model is asked to do and when, and turns a powerful but unpredictable reasoning engine into a scan that produces structured, validated output you can compare over time. In practical terms, that means four things: where the scan is triggered, how its scope is controlled, how its reasoning is constrained at each step, and how its output is standardized once the scan finishes. It also means being clear about where this approach fits: LLM scan orchestration is built for deep, periodic scans, while AI SAST is the better fit for the pipelines.

What this delivers: the benchmark results

We benchmarked incremental scanning against full re-scans on three intentionally vulnerable test applications commonly used for security tool evaluation: dvpwa, govwa, and vulpy. Each repository was scanned three ways using the same model (GPT-5.5, with a 258k token context window per session): a full scan at the latest commit (HEAD), a full scan at an older base commit, and an incremental scan spanning the same base to HEAD range.

Incremental scanning significantly improves scan efficiency and accuracy by focusing the model on code changes, resulting in dramatic reductions in cost and time while retaining the context around findings across multiple scans.

On dvpwa, the full scan cost $26.20 and took 50 minutes; the incremental scan covering the same range cost $4.51 and took 16 minutes. It also surfaced more findings overall (20 versus 16): each retained finding kept the same title, description, and evidence carried over from the prior scan, not just a coincidentally similar count. govwa showed the same pattern: 83% cheaper, 72% faster, and a stable finding count (11 versus 12). On both repositories, the incremental scan caught every finding the full scan did; none were missed.

Because incremental scanning is meant for periodic deep scans rather than a complete one-off, the savings compound: modeling 10 scans as one full scan plus nine incremental scans, dvpwa and govwa both come out 75% cheaper than running 10 full scans, and that gap widens further as scan count grows.

Where this runs: deep scans, not every pipeline run

The scanner itself is currently built on top of OpenAI Codex Security with a Mythos agent plugin and is invoked as a named "skill," a pre-defined task the underlying agent knows how to run (for example, security-scan, deep-security-scan, or security-diff-scan). Harness STO controls when that invocation happens and how much of the repository it covers.

We recommend running it as a periodic deep scan. A user, or a separate pipeline, can invokes a full deep-security-scan directly. Typical reasons include a compliance audit, a pre-release review, or onboarding a new repository for the first time when there is no scan history to build on yet, or periodic deep scans of critical applications, where a longer, costlier scan is worth the deeper reasoning.

Every scan runs the same underlying four-phase workflow (threat model, finding discovery, validation, and attack path analysis). What differs is how much of that work the orchestration layer decides to redo on each run.

Incremental versus full scans: controlling scope, not just the prompt

A full scan runs all four phases from a blank slate: it builds or regenerates a threat model for the repository, ranks every file in scope by how much security relevant surface area it exposes, deep reviews the highest ranked files, validates every candidate finding using the strongest method available (for example a proof of concept, a test harness, or a debugger trace), and finally runs attack path analysis and severity scoring on whatever survives validation.

Redoing all of that on every commit is expensive and also part of why the results vary. The more code the model has to reason over from scratch, the more room there is for its conclusions to differ between runs.

An incremental scan changes what the model is asked to do, not what it is capable of doing:

• It loads the results of the previous scan as context, retrieved from the API, rather than starting from scratch.
• It revalidates every finding carried over from the prior scan against the current state of the code. A finding is never simply assumed to still be true. If the model can no longer confirm it, because the vulnerable code was fixed, refactored, or removed, that finding is dropped rather than carried forward indefinitely.
• It scopes new discovery, the most computationally expensive part of the pipeline (this is the step where the model actually reads and reasons over code, looking for new problems), to only the files that changed since the last scan, rather than re-ranking and re-reviewing the entire repository.

Validation runs against both newly discovered vulnerabilities and everything carried forward from the prior scan, and severity scoring and reporting proceed the same way regardless of which mode produced a given finding.

The point of this design is not that incremental scanning trades accuracy for speed. It reframes the task from "rediscover everything in this repository" to "confirm what was already found, and look carefully at what changed." That reframing is also what keeps the finding list stable between scans of unchanged code.

Focusing on the LLM: what each phase does for you

The single most effective way to get consistent, useful output from an LLM scanner is not a clever system prompt. We want to avoid asking one open ended question across the entire codebase at once. The workflow is instead split into four narrow phases, each with one job and a clear handoff to the next, so the model is never asked to hold in mind "understand this entire repository and find every bug in it" all at once. 

Here's what that looks like for the OpenAI Codex Security integration, one of the engines we orchestrate. Other engines we support, like Mythos Agent, are structured differently, since the reasoning approach isn't identical across backends. But the underlying principle, narrow scope beats one big open question, holds regardless of which engine is doing the reasoning.

1. Threat model. Before looking for a single issue, the model builds a picture of what actually matters in your repository: what is exposed to the outside world, what data is sensitive, and who a realistic attacker would be. It also decides upfront what counts as Critical versus Low for your specific application, so severity ratings later on are not just a guess made in the moment. The output is a readable summary of what the scan considers risky in your codebase, and why.

2. Finding discovery. The model works through your code and records every candidate issue it finds, along with the evidence supporting it. It is deliberately told not to bundle similar-looking issues together, so two separate problems that happen to look alike do not get collapsed into one and quietly lost.

3. Validation. This is where the model checks its own homework. Rather than taking a candidate finding at face value, it tries to prove it: by building a small test, tracing the code path, or reproducing the behavior, using whatever method provides the strongest evidence. Only findings that can be backed up with real evidence move forward. Anything it cannot confirm is marked as unproven rather than dropped silently or reported as fact.

4. Attack path analysis. For everything that survives validation, the model has to argue against reporting it before it can argue for it: is this actually reachable by an attacker, does it matter in the context of this application, is it in scope at all? Only what survives gets a final severity rating and makes it into your report. This is the step that keeps the final list from being cluttered with things that are technically true but not actually worth your team's time.

You do not need to write or tune any of this yourself. It is built into how the scan runs. What you get at the end is a report where every finding has already been through a check for plausibility, evidence, and relevance before it reaches you.

Normalizing the output: making scans comparable to one another

A scan is only useful over time if you can compare it to the last one. You need to know whether a given finding was fixed, whether something new just appeared, and whether your backlog is actually shrinking. That only works if the scan's output follows a consistent structure every time, even though the findings themselves vary.

A few things make that possible:

• Every finding looks the same on paper. Regardless of which run produced it, a finding always comes with the same basic information: what the problem is, where it is, how serious it is, and why. The model fills in the details, but it does not get to change the shape of the report.
• Every finding has a paper trail. Each one is tracked through discovery, validation, and final review, with a record of why it was reported, ruled out, or held for follow-up. That means a finding's status is never just a final yes or no with no explanation.
• The report format itself is checked before it ships. A separate, non AI check confirms the report has all the right sections and details before it is delivered. If something is missing, it gets fixed before you see it.
• Findings get a stable identity across scans. Once a finding is identified, it keeps the same identity from one scan to the next. That is what lets us say "this issue from last week is still open" or "this one just appeared," and it is also what makes incremental scanning's carry forward logic possible in the first place, and what lets us directly measure finding stability across scans rather than only measuring cost and speed.

Why we recommend AI SAST in the pipeline

Orchestration narrows what the model is asked to do, but it does not make the model deterministic, and a gate on every pull request depends on results you can repeat. Our LLMScanBench Q3 2026 benchmark tested this directly. It compared GPT-5.5 and Claude Opus 4.8, both run through OpenAI Codex Security's workflow, against Harness AI SAST on OWASP WebGoat, OWASP Juice Shop, and OWASP Benchmark for Java. Three results explain why LLM scans fit deep review better than every commit:

• Results vary from run to run. Across three runs on WebGoat, Opus 4.8's recall was 16.2%, 19.7%, and 35.5%. On Juice Shop it was 16.0%, 30.0%, and 18.8%. A single scan cannot tell you which pattern you are looking at. Variance was tested on Opus 4.8 only.
• Full scans are slow. On WebGoat, AI SAST finished in 87 seconds, while the fastest LLM approach took more than 30 minutes. Across the benchmark, full LLM scans took 1.6x to 37x longer.
• Full scans are expensive. Full LLM scans cost 6x to 39x more than AI SAST: $21.79 to $133.58 per scan for GPT-5.5 and Opus 4.8, against an amortized $3.40 for AI SAST.

Incremental scanning, described above, can cut that cost and time substantially. These benchmark results are for full scans on different applications, so they are not directly comparable to the incremental results earlier, but they are still useful in showing how far full LLM scans are from the speed and cost a pipeline gate needs. Incremental scanning also still depends on the model's judgment on each run, and it assumes changes are small. On vulpy, the commit range added more than 44,000 lines and the incremental scan missed 9 of 13 rule categories, which the benchmark authors believe broke the assumption that only changed code needs review.

For pull requests and commits, we recommend AI SAST, a deterministic scanner with an AI reasoning layer that validates its findings, so results are repeatable and fast enough to gate a build. On WebGoat, AI SAST reached 85.1% precision at its strictest filtering setting (confirmed risk only), with 40.3% recall, without relying on a model having seen the application before. Use LLM scan orchestration for pen testing and periodic deep scans of critical applications, where deeper reasoning is worth a longer, costlier scan. The two work together: AI SAST provides a fast, repeatable baseline on every change, while LLM scans add depth and can help identify novel vulnerabilities. Anthropic describes a similar layered approach in How Anthropic secures its AI-native software development lifecycle, where it combines agentic scans with SAST tools that post directly on pull requests.