Use cases / Keeping main healthy
The same test keeps breaking main. Flaky, or a real bug?
The same test keeps turning main red, the team reruns it until it passes, and nobody finds out why it failed.
Jigward reproduces the failure on purpose, traces it to an export query that never sorts its rows and opens a draft pull request with runs before and after the fix. The review, the merge and any customer note stay with the team.
The film
Chapters
Read the narration
0:00 Red, rerun, green, and a standing Duty
Main is red again, on the same test. Someone hits rerun, it turns green, everyone moves on, and nobody ever learns why it failed. I'm Jigward. On Monday, Anna gave me a standing duty: check main's builds every weekday morning, and dig into repeat failures.
0:23 Reproducing the failure on purpose
Same test, third time, and nobody owns it. Okay, instead of trying again, I'll find out why it fails. On my own copy of the repo, I run it over and over, in random order. Four runs fail the same way: orders come back shuffled.
0:42 The query never sorts
There's the gap. The export query never sorts its rows, so a customer's download can come back in a different sequence. Sorting by creation time isn't enough. It only keeps whole seconds, and orders often share one, so the unique key settles ties.
1:04 A one-line fix and 50 clean runs
One line fixes it. I repeat the same random runs, many more of them, and not one fails. The full test suite still passes. The refund export has the same gap. I'll note it for the team rather than change it quietly.
1:24 A draft pull request, and the team decides
It's a draft, so nothing merges without a person. It carries the proof, and warns that customer exports will now list orders oldest first. Ethan and Anna get the cause, the fix, the proof, and what I didn't check. The review, the merge and any customer note stay with Anna.
What changes
Today. Someone presses rerun, the build turns green and the merge goes through. CI dashboards and flaky-test tools can flag a test, retry it or quarantine it, but someone still has to reproduce the failure and find its cause in the code. Chasing a flake costs an afternoon and a rerun costs a minute, so nobody owns it, and a real bug can hide behind the word flaky.
-
Before: Rerun until green, merge, and forget the failure.
With Jigward: The failure reproduced on purpose, in 4 of 30 random-order runs, with every seed kept.
-
Before: A test called flaky that nobody owns.
With Jigward: A cause in the code. The export query never sorts, so a customer's download can come back in a different order.
-
Before: A guessed fix, judged by one green rerun.
With Jigward: A one-line fix, 50 random-order runs that all pass and the whole suite green, in a draft pull request.
What Jigward does
@Jigward check the CI on main every weekday at 9:00 AM. When a test keeps failing and then passing on rerun, find out why instead of rerunning it.
Connected beforehand
- The billing repository is connected through GitHub, and the team channel is named for the billing Context.
- An admin named the CI channel for the same Context and marked it as observed, so Jigward's checks read it and never post there.
-
Notices the pattern
At the 9:00 AM wake, the Duty reads the observed CI channel. The same test failed for the third time this week, passed on rerun and has no owner, so the wake hands the investigation to a Task.
-
Reproduces it on purpose
On its own copy of the repository, Jigward runs the export tests 30 times in random order and keeps each run's seed. Four runs fail the same way, with orders shuffled.
-
Finds the cause
The export query has no ORDER BY, so the database may return rows in any order, and the CSV export writes them as they come.
-
Chooses the sort key
The schema shows creation times keep only whole seconds and are not unique, so the fix sorts by creation time and then by the unique key.
-
Fixes it and runs it again
One line adds the sort. Another 50 random-order runs all pass, and so does the whole suite. The refund export has the same gap, which Jigward records as an initiative instead of changing it quietly.
-
Opens a draft and reports
The draft pull request carries the cause, the runs before and after, and a warning that customer exports will now list orders oldest first. The report goes to the team channel.
Jigward in this case
-
Slack
The Duty is asked for with a mention in the team channel. The observed CI channel is read at each wake and never posted to, and the report goes to the team channel.
-
Duties
Jigward creates the Duty from the mention's Task. Every weekday at 9:00 AM it reads the CI channel, and everyone in the Context sees it.
-
Tasks
The wake hands the investigation to a Task, shown as given by the Duty, and acts on what that Task reports.
-
Workspace
The 80 random-order runs and the whole suite run on Jigward's own copy of the repository, and every seed stays on the Task.
-
Project knowledge
The schema the Liquibase migrations define shows creation times kept to whole seconds, which settles the sort key.
-
GitHub
The fix arrives as a draft pull request with the runs before and after. Jigward never merges it.
What comes back, and what stays with the team
What comes back
- GitHub A draft pull request with the one-line fix, the runs before and after (4 of 30 failed, then 0 of 50) and a warning that customer exports will now list orders oldest first.
- The team channel in Slack A report to the two engineers with the cause, the fix, how it was checked, what was not checked and the refund-export initiative.
- The Duty's history The wake, the Task it gave and what that Task reported.
What the team decides
- Reviewing and merging the pull request. Nothing merges without a person.
- Whether customers need a note about the new order of their exports.
- Whether to take up the refund-export initiative.
What Jigward did not check
- Production exports. No production data was read.
- Other exports and other code that relies on row order.
- Every other order-dependent failure. Fifty passing runs make one less likely, not impossible.
Meet your next engineering coworker
Bring a question
from your own project.
See how Jigward can help your team understand the system and move the next task forward.
Jigward is in closed testing. Accounts are created by hand; sending a request does not create one.