Finding the Commit That Broke Production with Git Bisect

A test was green on Monday. By Friday it was red. Between those two points sat 140 commits from six engineers, and nobody could say which one broke it.

This is the moment most teams reach for git log and start eyeballing diffs. That approach is O(N) in the number of commits, and it gets worse the longer the regression sits undetected. Git ships a tool built for exactly this problem, and most engineers never touch it.

Why “just read the commits” falls apart

The standard move looks like this: open git log --oneline between the last known-good tag and HEAD, and start reading diffs top to bottom.

git log --oneline v2.3.0..HEAD

With 10 commits, this works fine. With 140 commits across a monorepo touched by six people, it doesn’t. You’re pattern-matching on diffs without running the code, which means you miss regressions caused by interactions between commits, not just a single obvious bad line.

The failure mode is subtle. You think a commit looks suspicious, you check it out, you run the test, it passes, and you move to the next suspect. Each check costs you a build and a test run. If the real culprit is commit 89 out of 140 and you’re scanning linearly from either end, you’re looking at dozens of manual checkouts before you land on it. Linear search on a linear history is the wrong algorithm for the job.

What actually constrains this problem

Strip it down. You have:

• A range of commits, ordered.
• A binary predicate: does this commit exhibit the bug? Yes or no.
• A monotonic property: once the bug is introduced, every commit after it (until fixed) also has the bug.

That monotonic property is the important part. It means the commit history is a sorted array with respect to the bug. Good, good, good, …, bad, bad, bad. Finding the transition point in a sorted array is a binary search problem, not a linear scan.

The real constraint isn’t CPU or memory. It’s the cost of running your test at each candidate commit. If a full build + test cycle takes 3 minutes, you want to minimize the number of times you pay that cost. Binary search over 140 commits takes about log2(140) ≈ 8 checks instead of up to 140. That’s the lever worth pulling.

Git bisect: binary search built into your version control

git bisect automates exactly this search. You tell it one commit that’s good, one that’s bad, and it checks out the midpoint for you. You test it, tell git the result, and it narrows the range - just like binary search over an array.

git bisect start
git bisect bad HEAD          # current commit is broken
git bisect good v2.3.0       # this tag was known to work

Git checks out the midpoint commit automatically. You run your test:

npm test -- --grep "checkout flow"

Then tell git the result:

git bisect good   # if the test passed here
git bisect bad    # if the test failed here

Repeat. Each answer halves the remaining search space. After roughly 8 rounds for 140 commits, git bisect prints the exact commit:

a1b2c3d is the first bad commit

You end the session and return to your original branch:

git bisect reset

Before vs after

Manual scan:   up to 140 checkouts, ~7 hours at 3 min/check
Git bisect:    ~8 checkouts,        ~24 minutes at 3 min/check

That’s the whole pitch. Same test, same commits, O(log N) instead of O(N).

Automate the test step so a human isn’t in the loop

Running git bisect good/bad by hand still costs you attention for every round. If your regression check is a single command that exits nonzero on failure, hand the whole loop to git with run:

git bisect start HEAD v2.3.0
git bisect run npm test -- --grep "checkout flow"

Git checks out each midpoint, runs the command, reads the exit code, and keeps going without you. This is the version worth putting in a Makefile or CI job because it turns a debugging session into a one-liner.

# scripts/find-regression.sh
#!/usr/bin/env bash
set -e
git bisect start HEAD "$1"
git bisect run bash -c '
  npm ci --silent &&
  npm run build --silent &&
  npm test -- --grep "checkout flow"
'
git bisect reset

Two things matter for bisect run to give a correct answer:

1. The test command must be deterministic. A flaky test poisons the search - git will happily bisect based on a false “bad” result and hand you the wrong commit.
2. The command must exit 0 for good and nonzero for bad. If your test runner exits 0 even on failure (some do, depending on config), wrap it so the exit code reflects the real result.

If a commit in the range doesn’t build at all - say, a broken dependency lockfile - you can skip it without polluting the search:

git bisect skip

Git works around skipped commits and keeps narrowing based on the ones it can actually evaluate.

What changed and what to watch for

On the incident that prompted this: 140 commits, bisect run against a single failing Jest test, 9 automated checkouts, total wall time 19 minutes including CI cold starts. The offending commit was a seemingly unrelated change to a shared date-formatting utility three services relied on. No amount of diff-reading would have flagged it quickly, because the diff itself looked correct in isolation - the bug was in how a downstream caller used the new return type.

A few things worth knowing before you rely on this in production incidents:

• Bisect assumes monotonicity. If the bug was introduced, then accidentally half-fixed, then fully reintroduced, the good/bad signal isn’t clean and bisect will give you a confusing or wrong answer. Know your history before trusting the result blindly.
• Merge commits complicate things. Bisecting through merge commits can land you on a merge itself rather than the actual offending commit inside a branch. git bisect has a --first-parent flag for cases where you only care about mainline history.
• Keep the test narrow. Bisecting against your entire test suite is slower per-round than bisecting against the one failing test. Isolate the specific assertion first, then bisect.

The bigger lesson isn’t really about git. It’s that when you’re searching an ordered space for a transition point, reach for binary search before you reach for brute force, and check whether your tools already implement it before you build your own. Git bisect existing as a first-class command should be a reminder to look for these primitives before writing a manual scan script by hand.