Post-Incident Review Template for Trading Automation

A post-incident review produces four things: a timeline in UTC, the conditions that had to hold simultaneously for the incident to happen, the gap between the first symptom and the first response, and actions that each carry an owner, a date and a test. Anything else in the document is context, and a review that produces no actions was a meeting.

What a review is for

The purpose is to convert an expensive event into cheaper knowledge. Something has already been paid for: an outage, a loss, a run that went wrong. The review is the only mechanism that turns that expenditure into changes, and if no change results, the money bought nothing at all.

A second purpose, less often stated, is to establish what actually happened. Immediately after an incident, several people hold confident and incompatible accounts of the sequence. The timeline is not bureaucracy; it is the artefact that replaces four remembered stories with one reconstructed one, and it usually surprises everyone including the person who built it.

What a review is not for is deciding who was careless. That question is answered by any group in about four minutes, is almost always wrong, and reliably reduces the quality of information available at the next incident. Nothing else in this article works if that boundary is not held.

When to hold one

Decide the triggers in advance so that holding a review is not itself a judgement about severity. Four criteria cover most cases, and the fourth is the one teams skip.

Unexpected loss. Funds moved in a way that was outside the written criteria, regardless of the size of the amount.
Criteria violation. The system behaved outside its acceptance criteria, even if nothing was lost, because the next occurrence may not be as kind.
Detection failure. The problem was discovered by someone other than the monitoring, or later than the window in which it could have been contained.
Second occurrence. The same symptom has now appeared twice. Repetition is evidence that the first response did not address the condition, and it is the cheapest incident to learn from because you have two samples.

Notice that severity is not one of the triggers. Small incidents are frequently better teaching material than large ones, because they happen more often and because nobody is defensive about them. A review culture built only around disasters gets very little practice.

Preserve evidence before restarting

The first action is not a fix

Restarting a process, rotating a configuration or clearing a queue can destroy the only evidence of what happened. Before anything is restarted, capture the log files, the current configuration, the wallet addresses involved and the list of signatures produced during the window. On-chain records are durable; local logs and in-memory state are not, and they are usually where the diagnosis lives.

If the incident involves a suspected key compromise, treat every affected key as compromised permanently. Keys cannot be un-leaked, and confirmed transfers cannot be reversed. Move remaining value to keys generated fresh on a machine you trust, and never enter an existing seed phrase into any tool, site or support channel as part of the response.

The evidence list is short and worth keeping as a runbook step: signatures with slots, tool logs for the window, effective configuration at the time, endpoint and provider in use, wallet addresses, and the wall-clock time of every human action including when someone first looked. That last one is almost never captured and is almost always important.

The timeline comes first

Build the timeline before anyone proposes a cause. This ordering is not stylistic. A cause proposed early reorganises everyone's reading of the evidence around it, and contradicting facts get filed as noise. Constructing the sequence first keeps the analysis honest for longer.

Use UTC everywhere, and record the slot alongside the timestamp for on-chain events. Slots give you a precise ordering that clock skew cannot disturb, and they let you correlate your events with network conditions independently of any local clock. Mixing local time zones into an incident timeline is a reliable way to produce a sequence that is subtly wrong.

Include non-events. "No alert fired" and "nobody was looking" are timeline entries, and they are frequently the entries that generate the most valuable actions. A timeline containing only things that happened will make detection look better than it was.

The review template

INCIDENT     INC-2026-014
SUMMARY      One paragraph, factual, no cause proposed
WINDOW       First symptom (UTC) to resolution (UTC), with slots
IMPACT       What was lost, in units. Funds, time, opportunity, trust
DETECTION    How it was noticed, by whom, and how long after onset

TIMELINE
  UTC        SLOT       SOURCE        EVENT
  ...        ...        chain/log/    what happened, factually
                        human

CONDITIONS   Each condition that had to hold. One line each.
             No single root cause line.

DETECTION GAP  Onset to first look, first look to understanding,
               understanding to mitigation. Three numbers.

WHAT WORKED  Genuinely. Controls that fired, limits that held.

ACTIONS
  ID  ACTION                 OWNER   DUE       TEST THAT SHOWS IT LANDED
  A1  ...                    name    date      check id or procedure

NOT DOING    Options considered and rejected, with the reason.
EVIDENCE     Paths to logs, signature list, configuration snapshot

Two sections do unusual work. WHAT WORKED prevents the review from producing actions that remove functioning controls, which happens more often than it should when a document contains only failures. NOT DOING records the rejected options, so that the same suggestion does not get re-proposed at the next incident with no memory of why it was declined.

Contributing conditions, not a root cause

The single-root-cause habit comes from simpler systems. In a stack involving your code, a signing key, an RPC provider, a router, a venue program and a market full of other participants, incidents happen when several conditions hold at once, and any one of them could have broken the chain.

Consider an illustrative example. A run fails and funds are stuck in a partially completed state. The conditions might be: the retry path rebuilt transactions instead of checking what landed; the endpoint returned rate-limit errors that were treated as generic failures; the abort threshold was written per transaction rather than cumulatively; and nobody was watching because the run was scheduled overnight. Four conditions, four possible actions, and no single one of them is the cause.

Listing conditions rather than a cause produces more actions and cheaper ones. Often the most valuable action addresses a condition that had nothing to do with the technical trigger, such as the fact that the run was unattended. That action would never have been generated by a document whose analysis stopped at the first plausible explanation.

The detection gap

Three numbers, and they are usually more actionable than anything in the cause analysis. Time from onset to first observation. Time from observation to understanding. Time from understanding to mitigation. Each has a different remedy, and conflating them produces vague actions about "better monitoring".

The three components of the detection gap, what each measures, and the class of remedy each one implies.
Interval	What it measures	Remedy class
Onset to first look	Whether anything was watching	Alerting on the observable that changed first, not on the symptom
First look to understanding	Whether the evidence was readable	Instrumentation, log quality, having the signature list to hand
Understanding to mitigation	Whether the response was available	A documented stop procedure that does not require improvisation

The middle interval is the one testing work influences most directly. If the first look happened quickly and understanding took two hours, the evidence was insufficient, and that is a defect in the system's observability rather than in the responder. Actions here are concrete: record signatures at signing time, keep errors verbatim, write logs to a file with UTC timestamps.

An operator running a Solana volume bot across several markets typically discovers that the third interval dominates, because stopping cleanly is harder than starting. A stop procedure written in advance, tested at least once, and reachable by whoever is awake is worth more than most monitoring improvements a review will propose.

Actions that cannot be closed empty

Every action needs three fields and the third is the one that changes behaviour. An owner who has agreed, out loud, to own it. A date. And a test: the specific check, procedure or observation that will demonstrate the action actually landed.

Name a person, not a team. Actions owned by teams are owned by nobody, and the review is where the ambiguity is cheapest to resolve.
Set a date that is soon. The half-life of incident urgency is short; an action due in three months will be renegotiated rather than done.
Attach a test. "Add a cumulative abort threshold" closes when someone says it is done. "Add a cumulative abort threshold, verified by regression check R-13 which fails without it" closes when the check exists and passes.
Prefer removing a condition to adding a control. Controls need maintenance and can fail silently; removing the condition entirely does not.
Cap the list. A review producing fourteen actions will complete three. Five actions that all get done beat fourteen that decorate a document.

The test requirement also creates the link back to the regression matrix. An action that produces a new check makes the incident permanently visible in the release process, long after everyone involved has moved on, which is the most durable form of institutional memory available in software.

Blameless in practice

Blameless does not mean nobody made a mistake. It means the review treats the mistake as a property of the system that allowed it, because that is the only version of the analysis that generates useful changes. If a configuration value could be entered in the wrong unit, the finding is about the interface, not about the person who typed it.

The practical test of whether a review is blameless is whether people volunteer information that makes them look bad. If they do, the process is working and you are getting the evidence. If the room is careful, the reviews will be pleasant and the incidents will keep happening, because the part everybody is avoiding is where the conditions live.

The published record for this kind of work is worth reading rather than reinventing; the postmortem culture chapter in the Google SRE book is the standard reference and its central point is exactly this one. The technique translates directly to systems where the failure mode is financial rather than availability-related.

Review anti-patterns

Common failure modes of incident reviews, how each one presents, and what to do instead.
Anti-pattern	How it looks	Instead
Cause chosen in the first five minutes	The timeline is built to support a conclusion already reached	Build the sequence before anyone proposes an explanation
Counterfactual analysis	Repeated use of "should have" and "if only"	Describe what did happen and which conditions permitted it
Action list as apology	Fourteen actions, no owners, no dates	Five actions with an owner, a date and a test each
Human error as a conclusion	The analysis stops at a person	Ask what made that action possible and reasonable at the time
Review with no evidence	Reconstructed from memory because logs were cleared	Preserve evidence before restarting, as a runbook step
Documented and unread	The report is filed and nobody sees it again	Read the previous review's actions at the start of the next one
Severity gate set too high	Only disasters get reviewed, so nobody has practice	Review the second occurrence of anything, regardless of size

Follow-through

The review is the cheap part. Follow-through is where the value is realised, and it fails by default because incident urgency decays faster than action lists get completed. The single most effective countermeasure is to open every review by reading the open actions from the previous one, in front of the people who own them.

Track completion as a number and look at it occasionally. A team completing most of its incident actions is learning from incidents. A team completing a small fraction is holding meetings after incidents, which is a different activity with a similar shape and none of the benefit.

Finally, keep the reviews together and searchable. The second occurrence of a symptom is the highest-value trigger in this whole article, and it only works if someone can find the first one. A folder of dated reviews with consistent identifiers is enough; the tooling matters much less than whether the search returns the earlier case.

Questions the desk gets asked

What is a post-incident review?

It is a structured examination of an incident that produces a timeline, a set of contributing conditions and a list of actions. Its output is findings and changes rather than a narrative, and its value is measured entirely by what happens afterwards rather than by the quality of the document.

When should you hold a review for a trading bot incident?

When funds were lost outside expectations, when the system behaved outside its written criteria, when detection took longer than the impact window, or when the same symptom has now appeared twice. Repetition is the most under-used trigger and often the most informative one.

What goes in an incident timeline?

Timestamped events in UTC from every source available: transaction slots and signatures, the tool logs, provider status records, and human actions including when someone first looked. Human entries matter because the gap between the first symptom and the first look is usually the largest number in the timeline.

Why avoid naming a single root cause?

Because incidents in systems with this many moving parts rarely have one. A single named cause invites a single fix and hides the other conditions that had to hold simultaneously, which are often cheaper to remove. Listing contributing conditions produces more actions and better ones.

Should a review assign blame?

No, and the reason is practical. Reviews that assign blame reliably receive less information next time, because people who expect to be blamed report less and later. A review that cannot see what happened cannot produce useful actions, so blamelessness is an evidence-gathering technique before it is a courtesy.

How long after an incident should the review happen?

Soon enough that memory and logs are intact, and far enough away that the people involved have slept. Within a few days is usually right, and preserving the evidence must happen immediately regardless of when the meeting is scheduled.

What makes an action item real?

An owner who agreed to own it, a date, and a test that will show it landed. Without the third element, an action can be marked complete without changing anything, and the same incident recurs with a closed action item in its history.