Regression Testing a System That Cannot Be Stopped

Build a regression matrix by listing the paths that must keep working, the previously reproduced defects that must stay fixed, and the failure behaviours that must stay defined. Run it on every release inside a written time budget, keep most of it in the free tier of simulation and a local validator, and record what it does not cover in the same document.

The problem with a system you cannot pause

Conventional regression testing assumes a staging environment that resembles production closely enough to be informative. Trading automation breaks that assumption in a specific way: the part you most need to test is the market, and the market has no staging copy. You can clone the programs and the accounts; you cannot clone the other participants.

The second complication is that the system is often running while you change it. A position may be open, wallets may be funded, and a scheduled run may be in progress. This means the rollout mechanism is part of the test design rather than an operational afterthought, and it is why the canary section below exists.

The third is that some of your dependencies change without asking you. A venue upgrades a program, a provider changes rate limits, a router adjusts how it splits orders. Nothing in your repository moved, and your system's behaviour changed anyway. A regression plan that only runs when you commit code will never see these.

What regression testing means here

The word means one thing: re-running a fixed set of checks to see whether previously working behaviour still works. The fixedness is the point. If the set changes between runs, a difference in outcome is not attributable to the change you made, and attribution is the entire product of the exercise.

That is why an ad hoc "we tested it again" does not qualify. Testing again with different attention, different inputs and different tolerance produces a feeling of coverage without the property that makes coverage useful. A matrix is a written list precisely so that the second run is genuinely the same as the first.

It also means regression tests are not exploratory tests. Exploration is valuable and belongs in a different session with a different mandate, because the two activities want opposite things: exploration wants novelty, regression wants sameness. Mixing them produces a suite that is neither repeatable nor curious.

The regression matrix

The matrix has one row per check and five columns. It fits on a page, and the fifth column is the one that keeps it honest.

Nine of the twelve rows cost nothing but time. Three require a live sample, and they share one sample rather than three, which is the single most effective way to keep a live tier affordable. Design the live tier as one small run that produces evidence for several checks at once.

Choosing what goes in it

Four categories earn a place. First, the money path: the sequence that actually moves value, because a regression there is the expensive one. Second, the failure paths, because they are exercised rarely in production and therefore rot quietly. Third, every previously reproduced defect. Fourth, the constraints you promised yourself, such as destinations and size limits.

What does not earn a place is anything whose failure you would not act on. If a check going red would produce a shrug, it is noise, and noise in a regression suite is corrosive: it trains everyone to read red as normal. Removing a useless check improves the suite more than adding a good one.

Previously reproduced defects deserve special mention because their value grows with age. A test protecting a bug fixed two years ago is protecting knowledge nobody in the room still carries, which is exactly when a regression is most likely and least likely to be recognised.

Keeping it inside a time budget

Write the budget down: the matrix runs in a stated number of minutes. Then treat that number as a constraint rather than an aspiration, because a suite that outgrows its budget stops being run, and an unrun suite is worse than no suite since it also carries the belief that coverage exists.

When a new check would exceed the budget, something has to leave. That forcing function is healthy: it makes the value of each check explicit and it prevents the slow accumulation of ceremonial tests. Most suites that became unrunnable did so one reasonable addition at a time.

Parallelism helps for the free tier and not for the live tier, since live checks share one sample by design. In practice the budget is dominated by validator startup and by waiting for confirmations, both of which are worth optimising once rather than tolerating on every run.

Running most of it without spending

Simulation covers instruction correctness, account requirements and compute consumption against real current state, at the cost of an RPC call. A local validator with cloned accounts covers everything stateful: creation, retries, restarts and reproduced defects, deterministically and for free. Between them they carry the majority of any sensible matrix.

The compute baseline in row R-01 is worth explaining. Recording the units a route consumes gives you a number that changes when the route changes, which makes it a sensitive detector of upstream behaviour shifts you were not told about. A twenty percent drift is not automatically a defect, and it is always worth reading.

Keep the free tier hermetic. If a check depends on a public endpoint being responsive, it will fail for reasons unrelated to your build, and those failures are the main source of distrust in a suite. Point the free tier at something you control, and let the live tier be the only place where the outside world can make you red.

Change classes and required coverage

That last row is a distinction people get wrong under deadline pressure. Moving between tool classes is not a change to a system, it is a different system, and the honest response is to run the original test plan rather than the regression matrix. The differences between volume bot vs bundler behaviour are a good illustration: they submit transactions differently, they fail differently, and a matrix written for one encodes assumptions about ordering and inclusion that simply do not transfer to the other.

Changes you did not make

The uncomfortable category. Programs on Solana can be upgraded by whoever holds their upgrade authority, providers change limits and pricing, and routers adjust their behaviour continuously. None of these appear in your version control, and all of them can change what your system does.

The mitigation is a scheduled run rather than a triggered one. Run the free tier of the matrix on a timer, not only on commit, and treat a red result on an unchanged build as the most interesting signal your testing produces: something outside you moved. The compute baseline and the program ID assertions are the checks most likely to catch it.

Keep a written list of external dependencies with the reason each one matters and where its status is published. When something breaks at an unlikely hour, that list is the difference between fifteen minutes of orientation and two hours of it. The RPC method reference is a reasonable anchor for the interface half of that list.

Canary runs and staged rollout

For a system that keeps running, the rollout is part of the test. A canary is a deliberately small deployment of the new build alongside the old one, sized so that a defect costs about what a rehearsal costs, and watched against a comparison rather than against a feeling.

1. Define the comparison first. Which observables you will compare between old and new, and what difference would count as a regression. Written before the canary starts.
2. Run both on the same conditions. Overlapping in time, because conditions change and a sequential comparison compares two different afternoons.
3. Keep the canary small enough to abort. Same principle as a rehearsal: the worst outcome is an amount already written off.
4. Watch the failure distribution, not just the totals. Same success rate with a different mix of errors is a change worth understanding.
5. Decide on a schedule. A canary with no decision point becomes a permanent second deployment, which doubles the surface nobody is testing.

For a system doing automated volume management across several markets, the canary is usually one market rather than a fraction of every market, because a single-market canary produces a clean comparison and a fractional one produces two half-observed systems. Pick the market where a regression would be cheapest to absorb, not the one where it would be most visible.

Flaky tests

A test that sometimes fails is telling you one of two things: the system is inconsistent, or the test is measuring something variable with a threshold that does not accommodate the variation. Both are findings, and the job is to determine which before doing anything else.

The diagnostic is to run the check many times on an unchanged build and count. A check that fails one time in twenty on a build nobody touched is describing variance, and the fix is usually a window rather than a threshold: measure over ten attempts instead of one. A check that fails in bursts is describing a real intermittent defect, and it goes to reproduction.

What must not happen is silent muting. A muted test is coverage that everybody still believes exists, and it fails at the worst possible moment: during an incident, when someone says the suite was green.

Retiring tests

Tests should be removed when the behaviour they protect is deliberately gone, and the removal should be as documented as the addition. One line is enough: the check identifier, the date, and the reason the protected behaviour no longer applies.

The reason to insist on the record is that removals cluster around deadlines, and a removal made under pressure looks identical afterwards to a removal made with judgement. A retirement log distinguishes them, and it takes ten seconds at the time.

What the matrix cannot cover

It cannot cover paths nobody added. Coverage is a statement about the list, not about the system, and the difference between them is where regressions actually live. Reading the matrix and asking what is missing is a better use of an hour than adding another row to what is already there.

It cannot cover behaviour at a size or duration you never run. A suite that exercises three transactions per check says nothing about the three-hundredth, and rate limits, account growth and fee dynamics all change shape with volume.

And it cannot tell you the release is safe. A green run means the listed paths behaved as they did before, on that build, in that environment, on that day. That is a genuinely useful thing to know, and it is not the same as safety, which is not a property any test suite can confer.

Questions the desk gets asked