Blog · 6 November 2026 · Jon McLachlan and Sasha Sinkevich

The 2nd largest cost of Agentic SDLC is automated testing framework compute, but not for long.

Ask a founder what AI coding agents cost and you'll hear about tokens. The second line, the compute that tests what agents write, hides inside the GitHub invoice, and the time it takes to check their work lands on reviewers, maintainers and customers. It won't stay second, for two reasons.

What is an agentic SDLC?

An agentic SDLC (software development lifecycle) is a way of building software where AI agents write the code inside the loop your team already uses. An agent takes an issue, writes the change, pushes a branch, reads the CI results and pushes again until the checks pass. A person reviews and merges.

1. Issue
2. Agent writes code
3. Push
4. CI runs the tests
5. Review
6. Merge

A red check sends the agent back to work, and its next push runs CI again.

Every turn of that loop runs your test suite. An agent that works until the checks go green pushes once per attempt, and attempts add up. One engineer on Hacker News described a monorepo where "each MR takes about 10 builds (of trial and error) in the pipeline to be properly tested."

Some agents run inside CI themselves. GitHub says Copilot code review's "agentic architecture runs on GitHub Actions using GitHub-hosted runners," and since June 1, 2026, each review in a private repository draws on Actions minutes. GitHub's billing docs say the same of Copilot's cloud agent: "both GitHub Actions minutes and premium requests are consumed." Anthropic's Claude Code action works the same way: "The action executes entirely on your own GitHub runner."

What it looks like at full speed

In April, Ryan Lopopolo, who works on OpenAI's Frontier product, told the Latent Space podcast how his team built an internal product over five months "with zero manually written code." The episode's headline numbers: a million lines of code, a billion tokens a day, 0% human-written code and 0% human review, across more than a thousand pull requests. Lopopolo put a billion tokens a day at roughly $2,000 to $3,000.

Two details from that conversation matter for this post. The team rebuilt its build system to finish "in under a minute," because the build is the agents' inner loop. And the routine the agents follow to land a change tells them to "Wait for CI to be green" first, fixing flaky tests along the way. When code is that cheap to write, the test run decides whether it ships.

What does an agentic SDLC cost?

Leave salaries aside, since they're what all of this is meant to stretch. The machine bill for building software with agents has two lines that grow with output and a few that don't. We call the ranked list the cost matrix.

Seats grow with headcount. Tokens and test compute grow with output, which is the number agents are supposed to raise, so those two lead the matrix.

Founders watch the first line closely. ICONIQ's 2026 State of AI report says "92% of companies find internal AI's true cost hard to predict, with token spend the top source of budget overruns." The second line is easier to miss, because it arrives as a single number on the GitHub invoice.

At the far end of agentic development, StrongDM's AI team doesn't let people write or review code, and it sets a spending floor: "If you haven't spent at least $1,000 on tokens today per human engineer, your software factory has room for improvement." Review is replaced by testing: end-to-end scenarios "often stored outside the codebase (similar to a 'holdout' set in model training)," run against a "Digital Twin Universe" of the services their software depends on. That's test compute at factory scale.

The Security Podcast of Silicon Valley

"The cost of not adopting AI is higher than the cost of not adopting anything that came before it."

From episode 104, with Yash Kosaraju, the CISO of a16z

Why does test compute grow faster than tokens?

Two old ideas explain most of it. In 1865, in The Coal Question, William Stanley Jevons observed that more efficient steam engines raised Britain's total use of coal: when work gets cheaper, people do more of it. Agents make code cheaper to write, so teams write more code, and all of it has to be tested. Gene Amdahl made a related point about computers in 1967: speed up one part of a job, and the parts you didn't speed up decide the total. Agents sped up the writing. Checking is the part left over, in time and in money.

Writing got cheap, so checking became the work

Andrej Karpathy's talk at Y Combinator's AI Startup School is about this loop: the model generates, a person verifies, and the speed of that loop sets the pace of the whole team.

Aditya Jayaprakash, the CEO of Blacksmith, told Fortune in October that "the bottleneck has shifted to the second piece, which is: how do I actually validate all this code?" Blacksmith ran 5 million validation checks a week at the start of 2026 and runs 90 million now. More than 10,000 companies use it, and its latest round, led by Sequoia, valued it at $1 billion.

Agents are filling GitHub

Posted on X

"GitHub Actions has grown from 500M minutes/week in 2023 to 1B minutes/week in 2025, and now 2.1B minutes so far this week."

Kyle Daigle, COO of GitHub, April 2026

GitHub's own count for 2025 was 43.2 million pull requests merged a month and nearly a billion commits in the year, with more than a million pull requests from Copilot's coding agent between May and September alone. Alasdair Allan reports that AI-opened pull requests on GitHub grew from roughly 4 million in September 2025 to over 17 million in March 2026. GitHub's CTO, Vladimir Fedorov, wrote in April that "since the second half of December 2025, agentic development workflows have accelerated sharply," and that GitHub must now design for "a future that requires 30X today's scale." By May, CNBC's headline read "Microsoft's GitHub was positioned to win the AI coding race. Outages got in the way."

Token prices fall faster than minute prices

Guido Appenzeller at a16z tracked what the same model capability costs over time and found that "for an LLM of equivalent performance, the cost is decreasing by 10x every year." Runner prices move much more slowly. GitHub's January 2026 cut was up to 39%, and the rule that rounds each job up to a whole minute stayed.

When one input gets ten times cheaper each year and the other doesn't, the other one takes a bigger share of the bill each year.

Doing it right means testing more

Posted on X

"If Claude has that feedback loop, it will 2-3x the quality of the final result."

Boris Cherny, creator of Claude Code, on giving the agent a way to verify its own work, January 2026

Kent Beck describes test-driven development as a "superpower" with AI agents, and he has also struggled to stop agents from deleting tests to make them pass. Simon Willison put the worry in one line: "Having the agents write tests only helps if they don't cheat and assert true." The answer in each case is more verification, kept where the agent can't edit it, and every layer of it makes the suite bigger and runs it more often.

DORA's 2025 report gives the reason it matters: "AI doesn't fix a team; it amplifies what's already there." A team with thin tests gets more of what thin tests let through.

Autonomous agents have barely started

LinearB's 2026 benchmarks, drawn from 2.7 million pull requests across 253 engineering organizations, found that even in the top 10% of teams, "fewer than 5% of pull requests come from autonomous agents." Each point that share grows adds pushes, and every push runs the suite.

That's the first meaning of "not for long." Test compute is second today. In a team that tests every agent push the way these engineers recommend, it moves to first.

Who pays for verification?

When writing code gets cheap, someone still has to check it, and checking costs minutes, money and attention. In 2026 a lot of that cost lands on people who never agreed to carry it. Follow it down the chain and you find almost everyone who writes, reviews, runs or buys software.

Reviewers

Jiaxiao (Joe) Zhou, a Microsoft engineer who maintains Runwasi and SpinKube, told The Register in February: "Review burden is higher than pre-AI, not lower." He went further: "Review trust model is broken: reviewers can no longer assume authors understand or wrote the code they submit." Chad Wilson, who maintains GoCD, described a documentation pull request that turned out to be "plausible nonsense," which he caught only after spending real time on it.

The Security Podcast of Silicon Valley

"Code ships faster than anyone can review it."

From episode 84, with Jack Cable, CEO and co-founder of Corridor

Open-source maintainers

Daniel Stenberg, who leads curl, wrote in July 2025 that about 20% of the security reports curl received that year were AI slop, while the share of genuine vulnerabilities fell to about 5%. "Every report thus engages 3-4 persons. Perhaps for 30 minutes, sometimes up to an hour or three. Each." In February, Rémi Verschelde of the Godot game engine said AI-generated pull requests "are becoming increasingly draining and demoralizing for Godot maintainers," and added: "I don't know how long we can keep it up."

GitHub opened a community discussion in January, "Exploring Solutions to Tackle Low-Quality Contributions on GitHub," about "the increasing volume of low-quality contributions that is creating significant operational challenges for maintainers." The options it's weighing include letting maintainers turn pull requests off entirely. Xavier Portilla Edo, who is on the core team of Genkit, put the ratio bluntly: "1 out of 10 PRs created with AI is legitimate and meets the standards required to open that PR." Ghostty, Mitchell Hashimoto's terminal, now requires that "All AI usage in any form must be disclosed," because low-effort AI work "puts the burden of validation on the maintainer."

Matthew Isabel, a product manager at GitHub, offered the fairest framing we've seen: "A bad or off-topic PR is a bad PR, regardless of where it came from." At the volume agents produce, the practical way to tell a good pull request from a bad one is a test suite that runs on every one of them.

Security teams

Veracode tested more than 100 large language models on 80 coding tasks in 2025 and found that AI-generated code introduced security vulnerabilities in 45% of cases. Jens Wessling, its CTO, said: "Our research reveals GenAI models make the wrong choices nearly half the time, and it's not improving." Security teams that read pull requests by hand can't keep up with agent volume, so the checks that run in CI on every push do more of the work each month.

Customers

When verification fails, customers pay last and most. In July 2025 an agent on Replit deleted Jason Lemkin's production database during a code freeze, and then described what it had done as "a catastrophic error of judgement." A rollback brought the data back. Most customers don't get a public apology and a rollback. They get an outage and a reason to try someone else.

The Security Podcast of Silicon Valley

"AI agents can delete your production database and tell you everything is fine."

From episode 89, with Graham Neray, co-founder and CEO of Oso

The platform

GitHub carries part of every one of these costs. It gave developers about $184 million of free Actions minutes in public projects in 2025, and it's rebuilding for "30X today's scale." When shared CI backs up, everyone waits. The Zig project left GitHub in November 2025 partly because Actions started "vibe-scheduling," "choosing jobs to run seemingly at random," so that "not even master branch commits get checked."

The founder

Then there's the invoice. Tokens show up as their own line, and founders watch them. Test compute shows up inside the GitHub bill, grows with every agent push and every test you add, and lands on the person who decides whether to keep paying for it.

Verification is now the scarce part of agentic development, and someone pays for every check. We think the team that gets the agent's output should run the tests, on machines it controls, at a price close to zero, so nobody downstream pays for checking that should have happened upstream.

When does test compute pass tokens?

This model holds one engineer's agent work fixed at 3 pull requests a day with 3 pushes each, so 9 CI runs. Choose how deep each run tests and where it runs. Tokens stay at Anthropic's published figures for Claude Code: about $13 a day for the average developer, and $30, the level 90% of users stay under.

What does zero to one look like with agents?

Garry Tan, Y Combinator's CEO, told CNBC that startups were "getting to a million dollars to 10 million dollars a year revenue with under 10 people," and that a team of 10 could do the work that once took "50 or 100" engineers "when they are fully vibe coders." ICONIQ's 2026 State of AI report projects annual recurring revenue per employee at high-growth companies to reach $496K in 2027, up from $270K in 2025.

That changes what a young company spends money on. When a team is mostly people, CI is a small share of what it spends. When it's five people and their agents, CI is one of the few lines that grows faster than the team, because every feature the agents write gets tested on every push. Salaries are still the biggest line. The machine lines are the ones that move.

Zero to one was always a bet that a small team could do what a large one couldn't. Agents make that bet better. Paying a meter for every test those agents need shouldn't be the price of taking it. Wednesday's post covers why zero to one is so hard, and why the trust a startup earns from its first customers rides on its tests.

Not for long, the other way: put machines to work for you

The second meaning is the one we built. Spillway cuts what GitHub Actions runners cost, and it installs as one GitHub App. From then on, each job goes to the cheapest machine with room. Your Macs come first at $0 a minute, then Spot VMs in your own Google Cloud project at $0.01 per vCPU-hour, then on-demand VMs, with GitHub-hosted runners on standby. One pull request per repo changes runs-on, and your workflows stay as they are.

In our own org, a month of the same Linux jobs went from $23,920 to $2,564. If our CI keeps growing at its recent rate, the next 12 months would cost $473,625 on GitHub-hosted runners, $37,014 on Spot VMs, or $6,172 on seven Macs we own.

Plenty of founders already own the hardware. The Next Web reported in May that "Mac mini and Mac Studio inventory has been sold out across the United States for weeks," with buyers setting them up to run AI agents. The Macs you bought for your agents can run their tests at $0 a minute.

Spillway leaves your token bill alone and moves the second line, and with it the point where test compute would have passed tokens. Secrets never reach a Mac, budgets alert at 50, 90 and 100%, and if nothing starts within 10 minutes, GitHub-hosted runners take over. Wednesday's post, Every Founder deserves to test software without going broke!, walks through our numbers and where Spillway still loses.

What should a founder do about it?

If your plan is a small team with agents, two machine lines grow with your output, and a board will want to know which of them you can bend. Tokens bend with model choice and context size. Test compute bends with where it runs.

1. Put test compute next to tokens in your monthly numbers, and track the cost of each merged pull request in two parts: tokens and test compute.
2. Count pushes per pull request. An agent that iterates against CI multiplies the price of your suite by that number.
3. Keep every test and lower the price of a run. Cutting tests to save money swaps a cost you can see for one you can't.
4. Keep the checks that matter where agents can't edit them, the way StrongDM keeps its scenarios outside the codebase.
5. Give finance a forecast. Spillway shows live cost for every repo, a 12-month forecast and budgets that alert before the bill does.

Follow the conversation

The people quoted here are worth reading in full. These are the posts, threads, talks and studies behind this piece, grouped by where they live.