Spillway

Blog · · Jon McLachlan and Sasha Sinkevich

Every Founder deserves to test software without going broke!

Zero to one is about as hard as work gets, and AI agents just made the first steps faster. Testing what they write still runs on a meter. This is the story of that meter: what it costs, how it got here, who pays it, and how to run every test on every push for about a tenth.

The short answer

Run every test on every push, and make each run cheap instead of rare. A startup going from zero to one has no track record yet, so each release becomes the record, and tests are how a small team earns trust it hasn't had time to bank. GitHub Actions bills its hosted runners by the minute and rounds each job up to a whole minute, so the bill climbs with every push from your team and your agents. Our own CI would have cost $23,920 a month on GitHub-hosted runners. Spillway, which we built to cut that bill, runs the same GitHub Actions jobs on Macs we own and Spot VMs in our own cloud. The same month costs $2,564, 89% less, and every pull request still runs the full suite.

Why is zero to one so hard?

Peter Thiel split the work of building into two kinds in Zero to One: "Doing what we already know how to do takes the world from 1 to n, adding more of something familiar. But every time we create something new, we go from 0 to 1." The first kind comes with a playbook. A founder signs up for the second kind, where nobody can tell you what works until you've tried it.

The people who've done it don't sugarcoat it. Sam Altman opened the first lecture of How to Start a Startup, the class he taught at Stanford in 2014, with what founders always say once they're in it: they "couldn't have imagined how hard and painful it was going to be." Dustin Moskovitz, who co-founded Facebook and Asana, gave the same class the part that rarely makes it into a pitch deck: "fear of failure on behalf of yourself and all of the people who decided to follow you."

Sam Altman and Dustin Moskovitz, lecture 1 of How to Start a Startup, Stanford, fall 2014, from YC Root Access. Altman's line about how hard it is comes near the start. Watch on YouTube

The data says the same thing in numbers. Carta followed 4,369 startups founded in 2018, and by January 2025, 2,707 of them had shut down. Of the seed rounds Carta tracked from late 2019, 32.6% led to a Series A within two years, and no group of seed rounds since the third quarter of 2021 has reached 30%. Peter Walker, Carta's head of insights, who shares this data on LinkedIn, put the timeline for venture-backed founders in two short sentences: "it's going to take a decade. Probably more."

61.9%of 4,369 startups founded in 2018 had shut down by January 2025

32.6%of seed rounds from late 2019 led to a Series A within two years

1,000days of runway is what Carta suggests a seed founder plan for

Carta data, shared by Peter Walker in February 2025, December 2025 and February 2026.

Why is it this hard? Every founder's story is different, but three pressures come up in almost every one we hear, and they arrive together.

Trust you haven't earned yet

Your first customer is betting on people they met a month ago. Your seed investor is betting on a product that mostly doesn't exist. Your first engineers are betting years of their careers. None of them can check a track record, because there isn't one. Every demo, every release and every reply to a bug report becomes the record, and the record starts at zero.

Time you can't get back

Money can be raised again. Months can't. Walker suggests seed founders plan for about 1,000 days of runway, because "the median time to Series A from Seed is now right at 2 years." A month lost to an outage, a rewrite or a stalled security review comes out of those thousand days, and nobody gives it back.

Morale that has to last

Paul Graham wrote in 2007 that when startups die, "the official cause of death is always either running out of money or a critical founder bailing," and that "the underlying cause is usually that they've become demoralized." His advice fits in two sentences: "Startups rarely die in mid keystroke. So keep typing!" Ben Horowitz named the low point the Struggle: "The Struggle is when you wonder why you started the company in the first place." Later in the same essay: "The Struggle is when food loses its taste."

The Security Podcast of Silicon Valley

"There's never a good time to start a company."

Michael Nov, co-founder and CEO of Prime Security, in episode 76. He launched his company in a crisis and kept going. On the learning curve: "the sooner you have it, the more it compounds."

Read those three pressures again with a CI bill in mind. A meter that charges you each time you check your own work presses on all of them: the money comes out of runway, the cuts it pushes you toward put trust at risk, and the month-end number is one more thing on a founder's mind at 1 a.m. Zero to one is hard enough. Paying to test your own software shouldn't be part of it.

Why does testing matter most from zero to one?

A large company can absorb a bad release, because it has years of releases that went fine. A startup's first customers have seen only a handful. Tests are how a small team turns speed into trust it can show someone. They let you ship on a Tuesday afternoon and answer the customer who asks "how do you know it works?" with a link instead of a promise. Four things ride on them.

Runway

Paul Graham's test for whether a startup is default alive is one question: "Assuming their expenses remain constant and their revenue growth is what it has been over the last several months, do they make it to profitability on the money they have left?" A CI bill that grows every month pulls against the answer.

Ours was growing by about $2,392 a month before we moved it, and at that pace the next 12 months of our Linux CI would cost $473,625 on GitHub-hosted runners. On Spillway we keep $21,356 a month, or $256,276 a year. The savings calculator on our front page turns your own bill into days of runway.

Customers

If you sell to companies, someone will ask for your SOC 2 report. Its change management criterion, CC8.1, covers whether "the entity authorizes, designs, develops or acquires, configures, documents, tests, approves, and implements changes" to its software. Your CI history is the record an auditor reads, and a test that never ran leaves nothing to read.

Large companies learn this in public. In July 2024 one faulty CrowdStrike update crashed Windows computers around the world, and CrowdStrike traced it to a bug in its quality control process. A year later, Jason Lemkin, the founder of SaaStr, watched an AI agent on Replit delete his production database during a code freeze. He wrote on X: "There is no way to enforce a code freeze in vibe coding apps like Replit. There just isn't." The data came back after a rollback. For a startup, the same lesson usually arrives smaller, as a lost pilot or a security review that stalls.

Our other company, YSecurity, runs security and compliance programs for startups. We walk through the founder's side of SOC 2 in episode 37 of our show.

Raising

ICONIQ's 2026 State of AI report projects annual recurring revenue per employee at high-growth companies to reach $496K in 2027, up from $270K in 2025, and those teams write 59% of their code with AI. That output per person is the bet behind AI coding agents, and agent output only holds up when it's tested. DORA's 2025 report found that "without robust control systems, like strong automated testing, mature version control practices, and fast feedback loops, an increase in change volume leads to instability."

Faros AI measured the same pattern in teams: with high AI adoption, they merged 98% more pull requests, and bugs per developer rose 9%. A board that funds a small, fast team is funding its test suite too.

Speed

Google researchers Ciera Jaspan and Collin Green looked for a build time short enough that developers stay focused, and found none. In Abi Noda's summary of their paper, "every improvement to build latency contributes to developers staying on task." Ian Duncan described the far end in February: "A twenty-minute feedback loop for a one-line change." Michael Nov, talking with us about security reviews that wait on manual steps, named what any slow gate costs a young company: "The outcome of this friction is delayed development, delayed deployment of features, and in some cases lost revenue."

AI made the first 70% fast. Testing is the last 30%.

Posted on X

"There's a new kind of coding I call 'vibe coding', where you fully give in to the vibes, embrace exponentials, and forget that the code even exists."

Andrej Karpathy, February 2, 2025

By November 2025, Collins Dictionary had made "vibe coding" its word of the year. In between, Y Combinator said a quarter of its winter 2025 batch had codebases almost entirely written by AI, 95% or more. Jared Friedman, a YC managing partner, made sure nobody read that as a shortcut for people who can't code: "Every one of these people is highly technical, completely capable of building their own products from scratch." Garry Tan, YC's CEO, told CNBC that a team of 10 could do the work that once took "50 or 100" engineers "when they are fully vibe coders," and that startups were "getting to a million dollars to 10 million dollars a year revenue with under 10 people."

That's real, and it's good news for anyone starting from zero. A first version is faster to reach than it was a year ago.

The same people were careful about where it stops. Tan: "The first versions of reasoning models are not good at debugging." Diana Hu, a YC general partner: "You have to have the taste and enough training to know that an LLM is spitting bad stuff or good stuff." Addy Osmani, an engineering leader at Google, called it the 70% problem: "They can get 70% of the way there surprisingly quickly, but that final 30% becomes an exercise in diminishing returns."

Developers feel that last 30% every day. In Stack Overflow's 2025 survey of more than 49,000 developers, 84% use or plan to use AI tools, and more of them distrust the accuracy of those tools (46%) than trust it (33%). Two in three named "AI solutions that are almost right, but not quite" as a frustration, and 45.2% said "debugging AI-generated code is more time-consuming." When METR timed 16 experienced open-source developers on their own projects in early 2025, they took 19% longer with AI tools than without, while believing they'd been 20% faster. METR said in February 2026 that developers are "likely" more sped up by AI tools now, and it is redesigning the study. We read the lasting lesson as the gap between feeling fast and being right, which is the gap a test measures.

"Almost right, but not quite" is what a test suite is for. Simon Willison said it about tests at the Pragmatic Summit in February: "They're free now. They're effectively free." He opens agent sessions by telling the agent how to run the tests and asking for red-green TDD, because "the chances of you getting code that works go up so much if they're writing the test first." His conclusion: "I think tests are no longer even remotely optional."

Simon Willison on the engineering practices that make coding agents work, from The Pragmatic Engineer's channel. Tests come up at 6:13. Watch on YouTube

He's right about writing them. Running them is a separate line on a separate invoice, and each test you add runs again on every push. That's the founder's bind this year. The cheapest way to make agent code safe is more tests, and the meter charges for every one of them, every time.

The Security Podcast of Silicon Valley

"AI won't save your startup. Unless it can ship changes safely."

From episode 87, our conversation with Venkat Thiruvengadam of DuploCloud about AI agents in DevOps

Why do founders ration tests?

Nobody plans to ration tests. The GitHub Actions line jumps the month your agents go live, someone moves the end-to-end suite to a nightly run to bring it back down, and the next month the test shards drop from eight to four. Each cut saves a little money and lowers your test coverage, and the bug that suite would have caught moves one step closer to a customer.

If you're the founder who made one of those cuts, you did the reasonable thing with the numbers in front of you. The trade only looks wrong once you price the bug, and the bug's price shows up months later, in an incident, a refund or a lost renewal. A test run costs a small amount you know in advance. The bug costs an amount you learn later.

The feeling underneath is harder than the bill. You raised money to build fast. The faster you build, the more you pay to check your own work, and it starts to feel like a penalty for doing the job well.

Hacker News

"We've somehow normalized a dev loop that involves git commit -m "try fix", waiting 10 minutes, and repeating."

heftykoo, in the thread on Ian Duncan's "GitHub Actions Is Slowly Killing Your Engineering Team"

Hacker News

"I'm paying for GitHub Action now and there is zero recourse (other than leaving)."

joshstrange, in the thread on GitHub postponing its self-hosted runner fee

We love GitHub, and we kept it. What we wanted gone was the meter: compute rented by the minute, rounded up, on every push.

What does GitHub Actions cost in 2026?

GitHub Actions pricing for hosted runners is per minute. The standard Linux runner with 2 cores costs $0.006 a minute, Windows costs $0.010 and macOS costs $0.062. Larger Linux runners cost $0.012 a minute with 4 cores and $0.162 with 64.

The rounding rule is one sentence in GitHub's documentation: "GitHub rounds the minutes and partial minutes each job uses up to the nearest whole minute." It applies to each job, so a workflow of ten 12-second checks pays for ten minutes to do two minutes of work.

The rounding rule, one job at a time

GitHub bills each job in whole minutes.

How long does the job run?

92%

of what you pay for is time after the job finished. It ran 5 seconds and was billed 1 minute.

80%

of what you pay for is time after the job finished. It ran 12 seconds and was billed 1 minute.

25%

of what you pay for is time after the job finished. It ran 45 seconds and was billed 1 minute.

49%

of what you pay for is time after the job finished. It ran 61 seconds and was billed 2 minutes.

21%

of what you pay for is time after the job finished. It ran 3 minutes 10 seconds and was billed 4 minutes.

How: GitHub rounds each job up to the next whole minute. Time after the job finished = billed time minus run time, as a share of billed time.

Short jobs are where the rule costs the most. On our own CI, the same 20 everyday check jobs cost 0.60¢ to 3.4¢ a run on GitHub-hosted runners and 0.04¢ to 0.29¢ a run on Spillway.

Every plan includes some compute, stated in dollars: $12 a month on GitHub Free, $18 on Pro and Team, and $300 on Enterprise Cloud. Standard runners in public repositories are free. Self-hosted runners are free too, and they don't count against your plan.

The price list changed twice in two days. On December 16, 2025, GitHub announced cuts of up to 39% to hosted runner prices from January 1, and a new fee of $0.002 a minute for self-hosted runners in private repositories from March. Developers pushed back, and the next day GitHub said it was "postponing the announced billing change for self-hosted GitHub Actions to take time to re-evaluate our approach." Its pricing page still says so, and names no new date.

How did the GitHub Actions meter get here?

Nobody built the meter to hurt anyone. GitHub priced CI in 2019 for a world where people, not agents, pushed the code, and for years that worked. This is how the world changed underneath it.

  1. Nat Friedman announces CI/CD in GitHub Actions, "free for the 40 million developers on GitHub to use with public repositories," with "simple, pay-as-you-go pricing" for private ones. It reaches everyone on November 13.

  2. GitHub becomes free for teams, with 2,000 Actions minutes a month at no cost. Friedman: "every developer on earth should be able to use GitHub for their work."

  3. Developers run 500 million Actions minutes a week, by GitHub's later count.

  4. Andrej Karpathy names vibe coding. A month later, a quarter of Y Combinator's batch has codebases almost entirely written by AI.

  5. A billion minutes a week. Developers use 11.5 billion minutes in public projects for free, which GitHub values at about $184 million. Copilot's coding agent opens more than a million pull requests between May and September.

  6. The Zig programming language leaves GitHub for Codeberg. Andrew Kelley writes that Actions "started 'vibe-scheduling'; choosing jobs to run seemingly at random."

  7. GitHub cuts hosted runner prices by up to 39% from January 1 and announces $0.002 a minute for self-hosted runners in private repositories from March 1. Its estimate: "96% of customers will see no change to their bill."

  8. After a day of pushback, GitHub postpones the self-hosted fee "to take time to re-evaluate our approach."

  9. The price cut takes effect. "Standard hosted runner usage in public repositories remains free."

  10. Kyle Daigle, GitHub's COO: 2.1 billion minutes "so far this week." GitHub says Copilot code review will draw on Actions minutes in private repositories from June 1, and its CTO says GitHub must plan for "30X today's scale."

  11. CNBC: "Microsoft's GitHub was positioned to win the AI coding race. Outages got in the way."

  12. Fortune reports that Blacksmith, which sells cheaper runners, now runs 90 million checks a week, up from 5 million at the start of the year, at a $1 billion valuation.

  13. The self-hosted fee is still postponed, with no new date.

Read top to bottom, it's a story about volume. The price of a minute went down. The number of minutes went up much faster, first because more people wrote code and then because their agents did.

What did developers say when GitHub moved the meter?

The December announcement landed hard. On Hacker News, the thread on it reached 802 points and 819 comments. A thread on Reddit's r/devops reached 729 points. In GitHub's own community discussion, the people who'd be charged spoke for themselves.

GitHub community discussion

"Those minutes are MY processing time, some of which I already pay for on other services."

JC3, in GitHub's discussion of the change

Reddit

"when you already pay them $21/user/month, it feels like a slap in the face"

u/TheEdgeOfRage, in the r/devops thread

GitHub community discussion

"This is just crazy, it will cost us more than the AWS spot instances we are using..."

cpernot-hf, in the same discussion

Hacker News

"When I'm at a startup, I'm looking for the fastest, maybe cheapest solution."

komali2, in the thread on GitHub's pricing changes

Not everyone was angry. "I honestly don't have any issue paying the self-hosted runner fee," wrote brightball on Hacker News. Another commenter, mukundesh, pointed at what GitHub already gives away: "For a public repository who will give you a free machine to run for 6 hours at a stretch for a job !!"

Both are fair. The anger came from teams who felt charged for hardware they'd already paid for. The gratitude came from people who know what free CI did for open source. Pricing work that AI agents generate is hard for every company right now. When Cursor changed its plans in June 2025, its CEO, Michael Truell, wrote: "We recognize that we didn't handle this pricing rollout well, and we're sorry." GitHub said something close to that within a day.

What is GitHub's side of the story?

It would be easy to make GitHub the villain of this story, and it would be wrong. Running CI for the world costs real money, and GitHub has carried a lot of it for free.

GitHub's own explanation of the change starts there: "We have real costs in running the Actions control plane." It adds: "In 2025, we saw developers use 11.5 billion total Actions minutes in public projects for free (~$184 million)." It promised that "GitHub Actions will remain free for public repositories," and that it was "making investments into self-hosted runners so they work at scale in customer environments." When the self-hosted fee drew fire, it said it had "missed the mark with this change by not including more of you in our planning."

Then the load arrived. GitHub's CTO, Vladimir Fedorov, wrote in April that "since the second half of December 2025, agentic development workflows have accelerated sharply," and that GitHub now has to design for "a future that requires 30X today's scale." In May, CNBC's headline read "Microsoft's GitHub was positioned to win the AI coding race. Outages got in the way."

A company that gives away $184 million of compute to open source in a year, and apologizes in public when it gets a price wrong, is trying to do right by developers. The meter is a pricing model from 2019 meeting a workload from 2026. Nat Friedman said in 2020 that "every developer on earth should be able to use GitHub for their work." We agree, and we'd add one thing: every developer should be able to test that work too, without watching a meter.

We kept every repo, every workflow and every pull request on GitHub. Spillway changes only where the jobs run.

Who has a stake in your test suite?

Everyone adopting AI and writing software has a stake in what testing costs, including people who will never see your invoice. GitHub now counts more than 180 million developers. A new one joins every second, and nearly 80% of new developers use Copilot in their first week. Each of them pushes code that someone has to check.

Founders

Every cut to the suite is a bet with your runway and your reputation, and you're the one who signs the invoice. Tests are how you show a customer, an auditor or an investor that the product does what you say. Turn your CI bill into days of runway.

Engineers

They wait on CI and debug whatever a rationed suite lets through. Each slow or skipped check costs someone's focus, and focus is what a small team runs on.

AI agents

An agent without tests is guessing. Boris Cherny, who created Claude Code, says a way for the agent to verify its own work will "2-3x the quality of the final result." Every check you cut is one less thing an agent can learn from. More in Friday's post.

Customers

They inherit every bug you ship. One faulty update grounded flights in 2024, and one agent deleted a founder's production database in 2025. Your first customers trust you with their data before you've had time to earn it.

Investors

They're funding a small team's output, and DORA's 2025 report found that AI "amplifies what's already there." A board that funds a fast team is funding its test suite, whether or not it's on the slide.

Security and compliance

Auditors read your CI history for SOC 2. Veracode tested more than 100 models on 80 coding tasks and found that AI-generated code introduced security flaws in 45% of cases. Tests that run on every push are the record that you caught them.

Finance

They need a number they can forecast. ICONIQ found that 92% of companies find internal AI's true cost hard to predict. A CI bill that grows with every agent push makes the forecast harder.

Open-source maintainers

They review what the world sends them, often unpaid. curl's Daniel Stenberg said about 20% of the security reports curl received in 2025 were AI slop, and each one "engages 3-4 persons." A flood of unchecked pull requests spends their time first. Who pays for verification.

Students and solo builders

Public repositories run standard runners for free, and GitHub Free includes $12 of compute a month, about 2,000 minutes of standard Linux. A private side project with an agent pushing all night can use that up in days.

GitHub

It runs CI for more than 180 million developers, gave about $184 million of it to public projects in 2025, and is rebuilding for 30 times today's load. A pricing model built for people is straining under agents.

How much does your test suite cost a month?

Pick how often your team and your agents push, and how many job-minutes one push adds up to across all of its jobs. A suite split into ten parallel jobs of six minutes each is 60 job-minutes.

Price your test suite

Your team and your agents, every push, every job.

Pushes a day
Job-minutes per push

$68a month

on GitHub-hosted runners, for 25 pushes a day at 15 job-minutes each.

About $7 a month on Spillway at our own ratio. $723 a year stays in the bank.

$203a month

on GitHub-hosted runners, for 25 pushes a day at 45 job-minutes each.

About $22 a month on Spillway at our own ratio. $2,170 a year stays in the bank.

$405a month

on GitHub-hosted runners, for 25 pushes a day at 90 job-minutes each.

About $43 a month on Spillway at our own ratio. $4,339 a year stays in the bank.

$810a month

on GitHub-hosted runners, for 25 pushes a day at 180 job-minutes each.

About $87 a month on Spillway at our own ratio. $8,678 a year stays in the bank.

$270a month

on GitHub-hosted runners, for 100 pushes a day at 15 job-minutes each.

About $29 a month on Spillway at our own ratio. $2,893 a year stays in the bank.

$810a month

on GitHub-hosted runners, for 100 pushes a day at 45 job-minutes each.

About $87 a month on Spillway at our own ratio. $8,678 a year stays in the bank.

$1,620a month

on GitHub-hosted runners, for 100 pushes a day at 90 job-minutes each.

About $174 a month on Spillway at our own ratio. $17,356 a year stays in the bank.

$3,240a month

on GitHub-hosted runners, for 100 pushes a day at 180 job-minutes each.

About $347 a month on Spillway at our own ratio. $34,712 a year stays in the bank.

$675a month

on GitHub-hosted runners, for 250 pushes a day at 15 job-minutes each.

About $72 a month on Spillway at our own ratio. $7,232 a year stays in the bank.

$2,025a month

on GitHub-hosted runners, for 250 pushes a day at 45 job-minutes each.

About $217 a month on Spillway at our own ratio. $21,695 a year stays in the bank.

$4,050a month

on GitHub-hosted runners, for 250 pushes a day at 90 job-minutes each.

About $434 a month on Spillway at our own ratio. $43,391 a year stays in the bank.

$8,100a month

on GitHub-hosted runners, for 250 pushes a day at 180 job-minutes each.

About $868 a month on Spillway at our own ratio. $86,781 a year stays in the bank.

$1,350a month

on GitHub-hosted runners, for 500 pushes a day at 15 job-minutes each.

About $145 a month on Spillway at our own ratio. $14,464 a year stays in the bank.

$4,050a month

on GitHub-hosted runners, for 500 pushes a day at 45 job-minutes each.

About $434 a month on Spillway at our own ratio. $43,391 a year stays in the bank.

$8,100a month

on GitHub-hosted runners, for 500 pushes a day at 90 job-minutes each.

About $868 a month on Spillway at our own ratio. $86,781 a year stays in the bank.

$16,200a month

on GitHub-hosted runners, for 500 pushes a day at 180 job-minutes each.

About $1,736 a month on Spillway at our own ratio. $173,562 a year stays in the bank.

How: pushes a day × job-minutes per push × $0.006, GitHub's standard 2-core Linux rate, × 30 days. Larger runners cost 2 to 27 times as much per minute, and rounding adds more for short jobs, so treat this as a floor. The Spillway figure uses our own ratio, $2,564 ÷ $23,920. Yours depends on your jobs.

Who stopped paying the meter?

Some teams decided that checking their own work shouldn't be billed by the minute, and changed where their tests run. None of them stopped testing.

37signals brought CI home

The Rails 8.1 announcement in September 2025 said the test suite for HEY, 37signals' email service, with "over 30,000 assertions," used to take "over 10 minutes to run in the cloud." "Now it runs locally on a Framework Desktop AMD Linux machine in just 1m 23s and an M4 Max in 2m 22s." Rails 8.1 ships with that local CI built in, and the announcement didn't hedge: "This makes getting rid of a cloud-setup for all of CI not just feasible but desirable." It followed 37signals' move off the cloud for its servers, which The Register reported was saving the company almost $2 million a year.

Some kept GitHub Actions and bought a server

"We still use GitHub Actions but our monthly bill is now flat because we pay for a server," wrote arusahni on Hacker News, in a thread about a surprise CI invoice. RunsOn, which starts runners in your own AWS account, lists "$0.0009/min on c8a.large versus $0.0060/min on GitHub-hosted Linux" for 2 vCPUs. Cyril Rohr, who built it, shared it on LinkedIn with a one-line pitch: "10x cheaper GitHub Action runners."

Some moved to a cheaper meter

Blacksmith now runs 90 million checks a week for more than 10,000 companies, on its own machines at a lower rate per minute. Justin Duke's Buttondown is one of them, mostly on four-core runners. When GitHub announced the self-hosted fee, he did the math on what it would add: "every incremental second the runner takes costs around two bucks a year." A cheaper meter is still a meter, and it still pays GitHub's fee if that fee ever comes back.

One project left GitHub altogether

Zig moved to Codeberg in November 2025. Andrew Kelley's reasons included GitHub's push into AI and Actions itself, which he said "has inexcusable bugs while being completely neglected." We don't think a founder should have to go that far. Leaving GitHub costs a startup things it needs, from the developers who already live there to the integrations its customers expect.

Every one of these teams kept its tests and changed the price of running them.

What are your options for cheaper GitHub Actions runners?

A founder who wants lower CI/CD costs has five ways to go, and every one of them keeps your code and your workflows on GitHub.

OptionHow you payWhat you get
GitHub-hosted runnersPer minute, each job rounded up. $0.006 a minute for standard Linux.Nothing to run. The meter runs on every push.
Managed runner clouds such as Blacksmith, Depot and NamespaceA lower per-minute rate on their machines. Blacksmith lists $0.004 a minute for Linux x64, with 3,000 free minutes a month.A cheaper meter, on someone else's machines.
Runners in your own AWS account, such as RunsOnA yearly license from €300 to €3,600, plus Spot compute billed by AWS.Your cloud and your bill, on AWS.
Your own build with Actions Runner ControllerYour cloud bill, plus the engineers who build and run it.Full control, and a system your team now maintains.
SpillwayCompute in your own cloud at Spot prices, plus Spillway's own fee. Macs you own run at $0 a minute.Your Macs first, then Spot VMs in your Google Cloud project, with GitHub-hosted runners on standby.

Prices from each company's own pricing page, read November 3, 2026.

How we stopped rationing tests

Our agents open pull requests around the clock, and every one runs the full suite. By November our Linux CI, priced at GitHub-hosted rates, had reached $23,920 a month and was still climbing. We didn't want to ration tests and we didn't want to leave GitHub, so we built Spillway. It sends each GitHub Actions job to the cheapest machine with room.

  1. 01Your Macs$0 a minute
  2. 02Spot VMs in your cloud, sized to the job$0.01 per vCPU-hour
  3. 03On-demand VMs$0.036 per vCPU-hour
  4. 04GitHub-hosted runners, on standby$0.165 to $0.18 per vCPU-hour

We switched our own org on October 23, 2026, and ten days later we had kept $5,380. A month of the same Linux jobs now costs $2,564 instead of $23,920, which is 89% less. In one day Spillway started 38,097 of our jobs, with a median wait of 51 seconds.

It loses in places, and you should hear that from us. A Spot VM takes about 45 seconds to boot, and 1 job in 20 waited 6 minutes 39 seconds or longer to start. Short jobs run a few seconds slower on Spillway. It runs Linux jobs today, so Windows, macOS and GPU jobs stay on GitHub-hosted runners. And if GitHub ever brings back its $0.002 fee, it would apply to every self-hosted runner, ours included. For a 2-core job on a Spot VM, compute plus that fee would still come to less than half of GitHub's $0.006 a minute.

Secrets never reach a Mac: jobs that hold them run on a single-use VM that is deleted after the job. Budgets alert at 50, 90 and 100% and never cut a job off. If nothing starts within 10 minutes, GitHub-hosted runners take over.

Spillway installs as one GitHub App. One pull request per repo changes runs-on, and a dial sends 0 to 100% of that repo's runs to Spillway. See the routing order on our front page.

Paying for testing shouldn't be part of zero to one

We've talked with founders and security leaders on more than a hundred episodes of our show, and we've built two companies of our own. This is what we believe about testing now that agents write so much of the code.

  1. A test run should cost close to nothing. The machines already exist: the Macs you bought for your agents, and spare capacity in your own cloud at Spot prices. A Mac you own runs a job for $0 a minute.
  2. Checking your work is the last line you cut. Cutting tests trades a cost you can see for one you can't, and at zero to one the one you can't see is the one that can end a company.
  3. More code means more tests. Agents raise the number of pushes, so the price of each run has to fall as the count rises.
  4. You shouldn't have to leave home to save money. Keep GitHub, your workflows and your history. Change only the machines.
  5. Know the number before the invoice does. Live cost per repo, and budgets that warn you and never cut a job off.

If you're in the middle of it right now, deciding whether to cut the suite to make the month work, keep the tests and cut the price of running them. You have enough hard decisions this year. This one has an easy answer.

What to do this week

  1. Download your last 30 days of usage. GitHub's billing settings export a usage report.
  2. Count the jobs that finish in under a minute. Each one bills as a full minute.
  3. Bring back one rationed suite. Run the end-to-end tests you moved to nightly on every pull request in one repo for a week, then read the bill.
  4. Try the savings calculator on our front page, and join the waitlist to route your first repo.

Read the whole conversation

This post stands on a lot of other people's work. These are the threads, posts and talks worth your time, grouped by where they live.

Hacker News

GitHub

Reddit

X

LinkedIn

YouTube

Podcasts

Essays

News

Questions

Why is going from zero to one so hard?

A founder has to earn trust with no track record, on a clock that only runs one way, with morale that has to last until the company works. Carta found that 61.9% of the 4,369 startups founded in 2018 it tracked had shut down by January 2025, and no quarter's seed rounds since the third quarter of 2021 have reached a 30% rate of getting to a Series A within two years.

How much does GitHub Actions cost in 2026?

GitHub-hosted runners cost $0.006 a minute for standard Linux, $0.010 for Windows and $0.062 for macOS, and GitHub rounds each job up to a whole minute. Plans include $12 to $300 of compute a month, and standard runners in public repositories are free.

Why is my GitHub Actions bill so high?

Every push runs your suite, every job rounds up to a whole minute, and larger runners cost 2 to 27 times the standard rate. AI coding agents push around the clock, so the bill grows with your output.

How do I reduce GitHub Actions costs without testing less?

Move where the jobs run and keep every test. GitHub doesn't charge for self-hosted runners, so a job on a Mac you own or a Spot VM in your own cloud costs only the compute. Spillway sends each job to the cheapest machine with room and keeps GitHub-hosted runners on standby.

Is GitHub charging for self-hosted runners in 2026?

No. GitHub announced a $0.002 per-minute fee for self-hosted runners in private repositories in December 2025 and postponed it the next day. Its pricing page still calls the change postponed and names no new date. If it returns, it would apply to every self-hosted runner in a private repository, Spillway's included.

Does vibe coding need tests?

Yes. AI tools get most of the way to working code quickly, and the rest is where bugs hide. In Stack Overflow's 2025 survey, 66% of developers named AI solutions that are almost right, but not quite, as a frustration. A test suite that runs on every push is how you find the almost.

Should a startup run end-to-end tests on every pull request?

Yes. End-to-end tests catch the bugs your customers would see first, and AI agents open pull requests around the clock. If the price stops you, lower the price of each run and keep the runs.

How much did Spillway save on its own CI?

89%. A month of our Linux CI cost $2,564 on Spillway instead of $23,920 on GitHub-hosted runners, read from our admin portal on November 2, 2026. Your number depends on your jobs.

Jon McLachlan and Sasha Sinkevich built Spillway, which runs GitHub Actions jobs on Macs you own and Spot VMs in your own cloud, for their own CI first. They co-founded YSecurity and host The Security Podcast of Silicon Valley.

We publish on Mondays, Wednesdays and Fridays. Follow the feed.

Sources

  1. Farnam Street, on Peter Thiel's Zero to One, September 2014
  2. YC Root Access, Lecture 1, How to Start a Startup (Sam Altman, Dustin Moskovitz), YouTube
  3. Peter Walker, Carta, We tracked 4,369 startups founded in 2018. Where are they?, February 11, 2025
  4. Peter Walker, Carta, Most seed startups never reach Series A, data shows, February 10, 2026
  5. Peter Walker, Carta, VC-backed founders are signing up for a decade or more, July 22, 2025
  6. Peter Walker, Carta, Seed founders need to make their capital last 1,000 days, December 12, 2025
  7. Peter Walker on LinkedIn
  8. Paul Graham, How Not to Die, August 2007
  9. Ben Horowitz, a16z, The Struggle, June 15, 2012
  10. The Security Podcast of Silicon Valley, episode 76, The Critical Security Stage Every Startup Skips, with Michael Nov of Prime Security, August 26, 2025
  11. Paul Graham, Default Alive or Default Dead?, October 2015
  12. AICPA Trust Services Criteria, CC8.1, text via Cyberday
  13. NBC News, What caused the global Microsoft outage? A humble software update, July 2024
  14. CNBC, CrowdStrike says bug in quality control process led to botched update, July 24, 2024
  15. Jason Lemkin on X, on enforcing a code freeze, July 2025
  16. Jason Lemkin on X, on the rollback, July 2025
  17. The Security Podcast of Silicon Valley, episode 37, Founder's Guide to Compliance: The Introduction (SOC2, ISO, NIST, HITRUST, PCI-DSS, FIPS, and more)
  18. SaaStr, 10 metrics from ICONIQ's 2026 State of AI report
  19. Nathen Harvey and Derek DeBellis, Google Cloud, Announcing the 2025 DORA report, September 23, 2025
  20. Faros AI, The AI Productivity Paradox Report 2025, July 23, 2025
  21. Abi Noda, Build times and developer productivity, on research by Ciera Jaspan and Collin Green at Google, July 14, 2023
  22. Ian K. Duncan, GitHub Actions Is Slowly Killing Your Engineering Team, February 5, 2026
  23. Andrej Karpathy on X, February 2, 2025
  24. CNN, 'Vibe coding' named Collins Dictionary's Word of the Year, November 6, 2025
  25. Ivan Mehta, TechCrunch, A quarter of startups in YC's current cohort have codebases that are almost entirely AI-generated, March 6, 2025
  26. Sarah Perkel, Business Insider via Yahoo Tech, 'Vibe coding' lets 10 engineers do the work of a team of 50 to 100, says CEO of Silicon Valley incubator, March 17, 2025
  27. Addy Osmani, The 70% problem: Hard truths about AI-assisted coding, December 4, 2024
  28. Stack Overflow, 2025 Developer Survey: AI
  29. METR, Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity, July 10, 2025
  30. METR, We are Changing our Developer Productivity Experiment Design, February 24, 2026
  31. Simon Willison, Fireside chat about agentic engineering at the Pragmatic Summit, March 14, 2026
  32. The Pragmatic Engineer, Simon Willison: Engineering practices that make coding agents work, YouTube, March 19, 2026
  33. The Security Podcast of Silicon Valley, episode 87, Escape the Ticket Trap: How AI Agents Are Replacing Manual DevOps, with Venkat Thiruvengadam of DuploCloud
  34. Hacker News, discussion of GitHub Actions Is Slowly Killing Your Engineering Team
  35. Hacker News, GitHub postponing the announced billing change for self-hosted GitHub Actions
  36. GitHub Docs, Actions runner pricing
  37. GitHub Docs, GitHub Actions billing: included compute, public repositories and self-hosted runners
  38. GitHub changelog, Update to GitHub Actions pricing, December 16, 2025
  39. GitHub, Pricing changes for GitHub Actions
  40. The Register, GitHub walks back plan to charge for self-hosted runners, December 17, 2025
  41. Nat Friedman, GitHub, GitHub Actions now supports CI/CD, free for public repositories, August 8, 2019
  42. The Next Web, GitHub is now free for all developer teams, April 15, 2020
  43. Kyle Daigle on X, April 2026
  44. GitHub, Octoverse: A new developer joins GitHub every second as AI leads TypeScript to #1, October 28, 2025
  45. Andrew Kelley, Zig, Migrating from GitHub to Codeberg, November 26, 2025
  46. GitHub changelog, Reduced pricing for GitHub-hosted runners usage, January 1, 2026
  47. GitHub changelog, GitHub Copilot code review will start consuming GitHub Actions minutes on June 1, 2026, April 27, 2026
  48. Vladimir Fedorov, GitHub, An update on GitHub availability, April 28, 2026
  49. CNBC, Microsoft's GitHub was positioned to win the AI coding race. Outages got in the way, May 22, 2026
  50. Beatrice Nolan, Fortune, AI code-testing startup Blacksmith hits $1 billion valuation as AI-written code surges, October 21, 2026
  51. Hacker News, Pricing Changes for GitHub Actions
  52. Reddit, r/devops, on the self-hosted runner fee
  53. GitHub community discussion #182089, on the pricing change
  54. Michael Truell, Cursor, on its June 2025 pricing changes, July 4, 2025
  55. Boris Cherny on X, January 2026
  56. Veracode, AI-Generated Code Poses Major Security Risks in Nearly Half of All Development Tasks, July 30, 2025
  57. Daniel Stenberg, Death by a thousand slops, July 14, 2025
  58. Ruby on Rails, Rails 8.1 Beta 1, September 4, 2025
  59. Dan Robinson, The Register, on 37signals' savings from leaving the cloud, October 21, 2024
  60. Hacker News, arusahni in the thread Surprise, pay $1000
  61. RunsOn on GitHub, runs-on/runs-on
  62. Cyril Rohr on LinkedIn, on RunsOn
  63. Justin Duke, on Blacksmith and GitHub's pricing change, December 17, 2025
  64. Blacksmith, Pricing
  65. RunsOn, Pricing
  66. GitHub Docs, About usage reports
  67. Sarah Gooding, Socket, GitHub Actions pricing whiplash, January 5, 2026
  68. The Security Podcast of Silicon Valley, episode 36, Feross Aboukhadijeh, Founder and CEO of Socket.dev, a startup improving security and privacy on the web
  69. Spillway admin portal, our own org, read November 2, 2026

Run every test on every push.

Spillway cuts your GitHub Actions bill by running the same CI/CD jobs on Macs you own and Spot VMs in your own cloud. Join the waitlist to route your first repo.

Savings calculator