Reproducing a Failure: From Complaint to Repeatable Case

Reproduce a failure by collecting artefacts rather than impressions, freezing every controllable variable at the values from the incident, then varying one at a time until the fault appears. If it does not appear within a time budget set in advance, classify it as non-reproducible with a named suspected class and add the instrumentation that would catch the next occurrence.

Why reproduction comes first

Everything downstream depends on it. You cannot verify a fix for a fault you cannot trigger, you cannot write a regression test for it, and you cannot tell the difference between a fix and a coincidence. Teams that skip straight to fixing produce changes that feel like progress and cannot be evaluated, and the same defect returns three releases later wearing a different report.

There is also a diagnostic argument. The act of reducing a failure to a minimal case is itself the investigation. Every variable you remove without losing the fault is a variable that was not the cause, and by the time the case is minimal you usually know where the defect lives. The reproduction is not the preparation for the analysis; it is most of the analysis.

The third reason is social and matters more than it should. A reproducible case ends the argument about whether the problem is real. Reports without reproductions get relitigated in every meeting, and the person who filed the report ends up defending their competence instead of the finding.

Intake: what a usable report contains

Most reports arrive incomplete, and the recovery cost rises sharply with time, so intake happens immediately rather than when someone picks the case up. Nine fields, and the first three are the ones that make everything else recoverable.

Frequency drives everything that follows. A fault that happens every time is a configuration or logic defect and will fall to reduction quickly. A fault that happens occasionally is usually timing, concurrency or external state, and reduction will not work on it directly; you have to instrument and wait, which is a different project with a different budget.

Freeze everything, then vary one thing

The method is old and works: hold everything constant, change exactly one variable per attempt, and record the outcome of each attempt whether or not it reproduced. The failures to reproduce are data too, and a list of twelve attempts that did not produce the fault is a map of where the fault is not.

The discipline that breaks down in practice is the one-variable rule. Under time pressure people change three things because each individually seems unlikely to matter, and when the fault appears they cannot say which change was responsible. Every such attempt has to be redone. Two careful attempts beat six careless ones and take less total time.

Record attempts in a fixed shape: attempt number, the single variable changed, its value, the outcome, and the artefact produced. That log is what you will hand to the person who fixes the defect, and it is often more useful to them than the original report.

Order the variables by cost rather than by suspicion. Suspicion is a poor guide early on, because the variable everybody suspects is usually the one they already checked, while the cheap tests eliminate whole regions of the search space for almost nothing. Configuration values and endpoint changes take seconds; recreating a specific wallet state takes minutes; cloning a market and rebuilding historical balances takes an afternoon. Working outward from the cheap end means that when the budget expires you have at least eliminated everything that was quick to eliminate, which is a genuine result rather than a shrug.

One warning about intermittent faults during reduction. If the fault reproduces only sometimes, a single attempt that does not reproduce it eliminates nothing, and treating it as elimination is how a reduction produces a confidently wrong conclusion. For anything reported as intermittent, each attempt has to be repeated enough times that a non-occurrence carries information, and the repetition count belongs in the attempt log beside the variable that was changed.

The variable table

For automation that trades on Solana, the controllable variables are a short list, and the uncontrollable ones are the reason some faults never reproduce. Separating the two before you start is what keeps the reduction from becoming an unbounded search.

The reproduction script outline

Write the reproduction as a script rather than as instructions, because instructions drift and scripts do not. The outline below is deliberately language-neutral; the shape is what matters, and the shape is that every phase leaves an artefact.

REPRO CASE  RC-018  "swap fails after account creation"
SETUP
  1  assert build == 3f2a1c            fail fast if not
  2  load config-from-incident.json    exact effective config
  3  create fresh test wallet          record pubkey
  4  fund with exact incident amount   assert balance
  5  ensure token account state        absent, as in the incident
TRIGGER
  6  simulate the transaction          save logs + unitsConsumed
  7  send with same compute + fee      save signature immediately
  8  poll status until confirmed or expired
OBSERVE
  9  fetch the transaction record      save err, logs, balances
 10  compare against expected          record match or mismatch
TEARDOWN
 11  close created accounts            reclaim rent
 12  write attempt row to repro log    attempt, variable, outcome
EXIT
  reproduced      -> attach artefacts, hand to fix
  not reproduced  -> increment attempt, change one variable

Step seven contains the detail that matters most: save the signature before waiting for confirmation. A transaction that never confirms still has a signature, and that signature is the only way to establish afterwards whether it landed late, failed, or never reached the network at all. Tools that only record signatures for successful sends destroy the evidence for the most interesting failures.

Step eleven exists because rent-exempt deposits for token accounts are recoverable, and a reproduction loop that creates accounts without closing them will slowly consume the test wallet for no reason. It is a small amount per account and it adds up across dozens of attempts.

Reproducing against frozen state

Many trading failures are state-dependent: the fault occurred because a pool held a particular balance, an account existed or did not, or a route was priced a particular way. Live reproduction cannot recreate that, because the state has moved. The tool for this is a local validator running with copies of the relevant mainnet accounts, which freezes the world at the moment you took the copy.

The limitation is honest and important: cloning copies current state, not the state at the incident slot. Reconstructing historical state requires either an archival data provider or rebuilding the balances from the transaction history around the incident. The getTransaction response includes pre and post balances for both SOL and token accounts, which makes that reconstruction possible for a small number of accounts and tedious for a large one.

When a state-dependent fault can be reproduced against a frozen snapshot, the case becomes deterministic and the fix becomes verifiable. This is the single highest-value technique in the article, and it is underused because setting up the clone feels like more work than trying the live reproduction one more time.

When it will not reproduce

Set the budget before you start: a number of attempts or a number of hours, written down. Reproduction attempts expand indefinitely because each failure feels like evidence that the next attempt is closer, which is not how independent trials behave. When the budget expires, you switch activities rather than continuing on momentum.

The switch is not to giving up. It is to instrumentation. The question changes from "how do I make this happen again" to "what would I need to have recorded to diagnose it if it happens again", and that question is almost always answerable. Add the logging, deploy it, and wait, having converted an unsolvable problem into a slower solvable one.

This is a particular problem for anything running as a volume bot on Solana DEXs, where a single run can produce hundreds of transactions across several markets and the failing one is buried among successes. The instrumentation that makes those cases diagnosable is per-transaction rather than per-run: signature, venue, route, compute requested and consumed, and the error verbatim, recorded for every attempt including the ones that worked.

Classifying a non-reproducible fault

An unreproduced fault still gets a classification, because "could not reproduce" is not a cause and closing on it teaches nobody anything. Five classes cover almost everything in this domain, and each implies a different next action.

1. Timing or concurrency. Suspected when the fault appears under parallelism and vanishes when operations are serialised. Next action: add ordering instrumentation and try to reproduce with artificially increased concurrency.
2. External state. Suspected when the fault is tied to a specific market or account. Next action: attempt a cloned-state reproduction rather than a live one.
3. Network conditions. Suspected when failures cluster in time across otherwise unrelated operations. Next action: record slot and fee data alongside every attempt so the correlation can be tested rather than assumed.
4. Provider behaviour. Suspected when the fault follows the endpoint rather than the code. Next action: run the same case against a second provider, which is the cheapest discriminating test available.
5. Report artefact. Suspected when the evidence does not support the described behaviour at all. Next action: return to the reporter with the on-chain record, without treating this as an accusation, because misread interfaces are common and are themselves a finding about the interface.

Recording the reproduction

When the fault does reproduce, the record is worth more than the fix, because the fix will be rewritten and the record will not. Keep the exact script, the configuration file, the artefacts from a successful reproduction, and the attempt log showing what did not reproduce it.

Give the case an identifier and use it everywhere: in the defect tracker, in the commit that fixes it, in the regression matrix entry it becomes, and in any incident review that references it. Cases without identifiers get discussed by description, descriptions drift, and two people end up fixing what they believe is the same defect.

Then convert it. A reproduced fault becomes a regression test, which is the only mechanism that will notice if the defect returns after a later change. That conversion is the point at which the investigation stops being a cost and starts being an asset.

Closing a case honestly

There are three honest endings. The fault reproduced and became a case with a fix and a regression test. The fault did not reproduce within budget, was classified, and instrumentation was added with a named person watching for recurrence. Or the evidence contradicted the report, which gets recorded with the evidence attached and no verdict about the reporter.

The dishonest ending is closing as "could not reproduce" with nothing attached. It resolves a ticket and preserves the defect, and it teaches everyone who files reports that filing them accomplishes nothing. The cost of that lesson is much higher than the cost of the investigation you were trying to avoid.

Questions the desk gets asked