One setting, 39 apps,
and 95 minutes a push
Every app we own was queueing behind a single build machine. Not because of a plan limit or a bill — because one checkbox had never been ticked.
The cause
A push to our shared codebase wakes every app that might be affected. With one build machine, those builds ran one after another. So a release didn't cost one build — it cost all of them, in a line. Touch a file near the top of the repo and you woke 39 apps; the last one finished an hour and a half later.
The tell was hiding in plain sight: the actual building took about three minutes. Eighty-two to eighty-eight percent of the wait was queueing — and most of that was a push waiting on its own sibling builds. No amount of making builds faster could touch it.
Nine previous attempts missed it. They inherited a note saying build concurrency would cost $3,100–6,200 a month, and none re-checked. The real rate is $0.014 per build minute — about $250 a month at our volume. The veto was wrong by a factor of twelve to twenty-five, and it was the only thing standing in front of the fix.
The fix, in the Vercel dashboard
It is one radio button, per project.
Where to find it
- Open the project, e.g.
vercel.com/forgefx/wm - Settings → Build and Deployment
- Scroll past Build Machine to On-Demand Concurrent Builds
Direct link:
vercel.com/forgefx/<project>/settings/build-and-deployment
What the panel looks like
On-Demand Concurrent Builds
Panel redrawn from the live dashboard — wording and states are exact, but this is a drawing, not a screen capture. Open the link above to see the real one.
The bottom option — “Builds are queued, maximum of one at a time” — is what all 39 projects were on. Nothing in the interface warns you what that costs across a shared codebase.
Three things that made it hard to find
- Nothing said “one machine.” There is no banner, no quota page, no error. The fleet simply behaved as though it were throttled, and it looked like slow builds.
- The account-level view still says one. Even now, with 39 builds running side by side, the team setting reports a single concurrent build. Check there and you'll conclude the fix didn't work. Only the per-project page tells the truth.
- It turned itself back off. Forty-five minutes after we enabled it, an automation still carrying older instructions switched it off across all 39. Editing a file does not reach a process that already read it.
So it now defends itself
- Every push silently re-asserts the setting, one second before the build starts.
- A check every 15 minutes repairs any drift and has to announce it loudly in its report.
- The old “turn it off” command is now a trap that refuses to run and writes down who called it.
What is still not fixed
Queueing is solved — the wait for a machine is now about one second. But the tail is not clean. Downloading the codebase before a build normally takes 58 seconds; during US business hours it sometimes collapses to a crawl and takes 21 minutes, on identical files. Same commit, machines starting seconds apart: one app finished in 53 seconds while another took 34 minutes.
Overnight it never happens — 137 consecutive clean builds. It is congestion upstream, not our repository being large: the biggest checkout of the day produced the fastest downloads. A controlled experiment is queued to settle whether size matters at all before we spend another week assuming it does.