Use cases / When production breaks
38 alerts overnight. One was real.
With no night shift, the alerts channel fills with noise overnight, and the engineer on call either reads all of it at dawn or mutes it and misses the real one.
Jigward watches the channel until morning, stays silent for the noise the team already knows, traces the one payment alert that holds in the logs and sends a single direct message with the evidence. The rollback and the incident stay with the engineer on call.
The film
Chapters
Read the narration
0:00 The alerts channel nobody watches at night
Most nights, this alerts channel fills with noise that clears itself. The engineer on call reads it all at dawn, or mutes it and misses the real one. I'm Jigward. Once he confirms the task, I'll watch that channel tonight and speak up only when something is real.
0:20 An overnight Duty and the first alerts
Nobody's awake to ask me overnight, so I set a duty: every half hour until morning, I check the channel myself. Late at night, a CPU spike and a pod restart, and both cleared. The alerts say what fired, not if that's normal here.
0:39 Known noise, so nobody is woken
The team noted this last month: the nightly batch spikes that worker, and it settles in minutes. This fits, so nobody hears from me.
0:52 One alert that does not fit, traced in the logs
More batch alerts come and go; I stay quiet. Then one fits no pattern: payment events are failing, and it isn't clearing. Okay, let's check the logs. Deliveries were fine until the signing secret reloaded. After that, every signature check fails. One count rules out a blip or a dip in traffic. Payments still arrive at the usual rate, and none get through. That's worth waking someone for.
1:23 One report, and the decisions that stay human
This one is real, so I report it once, with the evidence. I don't touch config, and I don't silence anything. He's woken once all night, for the real one. With the evidence ready, the rollback takes minutes. The incident and the alert changes stay his call.
What changes
Today. Alerting tools post every alert that fires into a channel, and at night nobody reads it. The engineer on call reads the whole channel at dawn to learn whether any of it mattered, or mutes it and hopes. An alert says what fired, not whether that is normal for this system, and what the team knows about its own noise sits in old threads. So a real failure posted between routine flaps is found late, or not at all.
-
Before: Reading the whole night's alerts at dawn, or muting the channel and hoping.
With Jigward: One direct message all night, for the one alert that held.
-
Before: Batch flaps that look the same as a real failure in the channel.
With Jigward: Known noise matched to the note the team kept, and no message when it fits.
-
Before: A log hunt at 2 a.m. before anyone knows what changed.
With Jigward: When it started, the likely trigger, the counts and a request ID, ready when they are woken.
What Jigward does
I'm on call tonight. Can you watch #alerts-billing and message me only if something is real?
Connected beforehand
- An admin marked the billing alerts channel as observed, so Jigward reads it and never posts in it.
- The on-call engineer's Slack account is linked, so their direct messages reach Jigward.
- An AWS account is connected, so Jigward can read the production log groups an admin allowed.
-
Sets an overnight Duty
Once the engineer clicks Create, Jigward sets a Duty that reads the alerts channel every 30 minutes until 7:30 a.m. It reports only what holds or touches payments, and changes nothing.
-
Reads each new alert
Late at night a CPU spike and a pod restart fire on the billing worker, and both clear. The alerts say what fired, not whether that is normal here.
-
Stays silent for known noise
A note the team kept in memory last month says the nightly batch spikes that worker and it settles in minutes. These alerts fit it, so the wake ends with nothing new and nobody hears from Jigward.
-
Traces the one that does not fit
At 2:18 a.m. a payment webhook alert fits no known pattern. The wake gives a Task to search the webhook logs, which show deliveries accepted until the signing secret reloaded at 2:00, then every signature check failing.
-
Rules out a dip in traffic
One Logs Insights query counts deliveries in 15-minute steps. They still arrive at the usual rate, and none has been accepted since the reload.
-
Reports once
Jigward sends the engineer one direct message with the finding, when it started, the likely trigger, the counts, a request ID and what it did not check. It changes no configuration and silences no alert.
Jigward in this case
-
Slack
The request arrives as a direct message, the Duty reads the observed alerts channel without posting in it, and the one report goes back as a direct message.
-
Duties
A Duty wakes every 30 minutes until 7:30 a.m., and each wake that finds only known noise ends with nothing new.
-
Memory
The team's note on the nightly batch, with where it came from and when, tells known noise from a new failure.
-
Tasks
The request becomes a Task when the engineer clicks Create. The wake hands the log search to a Task, which keeps the log lines, the count and the request IDs.
-
AWS CloudWatch Logs
A log search on the webhook service finds the secret reload and the failed signature checks, and one Logs Insights query counts the deliveries.
What comes back, and what stays with the team
What comes back
- A direct message in Slack The night's one report, with the finding, when it started, the likely trigger, the counts, a request ID, what was not checked and a plain statement that nothing was changed.
- The Task The log lines, the Logs Insights count and the request IDs behind the report, for whoever opens the incident.
What the team decides
- Rolling back the signing secret, opening the incident and asking the payment provider to resend the failed events.
- Whether webhook errors should page anyone, and how to tune the noisy CPU alert. Jigward is not a paging system. It changes no configuration and silences no alert.
What Jigward did not check
- The payment provider's side, and whether it resends the failed events.
- Whether the secret reload is the root cause, beyond every check failing six seconds after it.
Meet your next engineering coworker
Bring a question
from your own project.
See how Jigward can help your team understand the system and move the next task forward.
Jigward is in closed testing. Accounts are created by hand; sending a request does not create one.