Acceptance Criteria for a Trading Bot: A Working Template

An acceptance criterion for trading automation needs five fields: the claim it comes from, the observable that carries it, the threshold that separates pass from fail, the window over which the observation is taken, and the verdict rule that says what happens when the run does not produce a clean answer. Anything missing one of the five cannot be failed, and a criterion that cannot fail cannot pass either.

Why a criterion must be failable

The purpose of writing criteria before testing is to remove the observer from the verdict. If the criterion is precise, the result is a lookup. If it is loose, the result is a negotiation, and the negotiation is always won by whoever wants the software to ship. That is not a comment on anyone's integrity; it is what happens to ambiguous standards under time pressure.

Consider the sentence "the bot handles congestion well". There is no version of a test run that contradicts it. Every outcome is compatible with it, because "well" is decided after the fact by the person reading the result. Replace it with a statement about the proportion of transactions that landed within a stated number of slots during a defined window, and now the software can be wrong.

This is also the difference between a test and a demonstration. A demonstration shows the software doing something. A test states in advance what would count as the software failing, then finds out. Vendors give demonstrations, which is reasonable and expected; buyers owe themselves the other thing.

The five fields

The five fields of an acceptance criterion, what each one contributes, and the characteristic failure when it is left out.
Field	What it does	If it is missing
Source claim	Records what is being tested and who asserted it	Nobody can tell later whether the criterion mattered
Observable	Names the thing you will look at, and where you will look	Two testers measure different things and both are right
Threshold	Separates pass from fail with a number or a state	The verdict becomes an opinion
Window	Defines the sample: how many, over what period, under what conditions	A single lucky observation is presented as a result
Verdict rule	Says what to do with partial, blocked or inconclusive runs	Unfinished runs quietly get recorded as passes

The window field is the one teams add last and regret skipping first. Execution behaviour on a public blockchain varies with conditions nobody controls, so any single observation is a sample of one from a distribution. A criterion without a window invites the tester to run it until it passes, which is a real failure mode and rarely a dishonest one.

The criterion template

Copy this shape into whatever tracker you use. It is deliberately flat: eight lines, no nesting, and no field that requires prose.

CRITERION  AC-014
CLAIM      "Routes across multiple Solana venues" (product page, archived 2026-08-12)
OBSERVABLE Program IDs appearing in the logs of confirmed swap transactions
SOURCE     getTransaction for each signature produced by the run
THRESHOLD  At least two distinct venue program IDs across the run
WINDOW     One rehearsal of 20 confirmed swaps, single session
VERDICT    PASS if threshold met; FAIL if not; BLOCKED if fewer than
           20 swaps confirmed for reasons unrelated to routing
EVIDENCE   Signature list with the program IDs extracted per transaction

Two properties make this template work. First, the source line names the mechanism used to observe, not just the thing observed, which is what stops a later reader wondering whether the number came from the chain or from the tool's own dashboard. Second, the verdict line pre-commits you to what a spoiled run means, so a run that ends early cannot be retrospectively upgraded.

Notice also that the claim is archived with a date. Product pages change. A criterion whose source claim can no longer be found is a criterion nobody can adjudicate, and archiving the page you read costs one action at the time and is impossible afterwards.

Vague claims rewritten

Common claims about trading automation, why each is untestable as written, and a rewritten criterion with an observable and a threshold.
Claim as written	Why it fails	Rewritten criterion
Fast execution	No unit, no reference point, no sample	Median slots between the transaction being sent and the signature reaching confirmed status, over 50 attempts, stays at or below the stated figure
High success rate	Success undefined; denominator undefined	Of transactions actually submitted during the window, the proportion reaching confirmed status without an error is at or above the stated figure, counting each submitted signature once
Multi-DEX support	Support could mean listed in a menu	At least two distinct venue program IDs appear in the logs of confirmed swaps produced by one run
Transparent fees	Transparent is a feeling	The amount actually deducted per swap matches the published schedule within the stated tolerance, checked against balance changes on chain for the first ten swaps
Safe and secure	Unbounded claim; no observable	The tool never prompts for a seed phrase or private key at any point in setup or operation, and every action it takes is attributable to a signature you can look up
Handles errors gracefully	Graceful is not a state	Each injected fault produces a distinct operator-visible message, a bounded retry count and no transaction signed after the fault is detected
Easy to configure	Easy for whom, doing what	A new operator completes first configuration from documentation alone, without contacting support, and the effective configuration is readable back from the tool

The rewrites are longer than the claims, which is the point. Compression is what made the originals unfalsifiable. Every clause added to the right-hand column is a decision someone would otherwise have made silently during the test, and pushing those decisions in front of the run is the whole discipline.

Choosing a threshold you can defend

A threshold has to come from somewhere, and there are only three defensible sources. It can come from a baseline you measured yourself, from a limit imposed by something you depend on, or from a business constraint you can state. A number that came from none of those is a wish, and a wish makes the test unfalsifiable in the other direction: the software fails a bar nobody agreed to.

Baselines are the most useful and the least used. Before testing the tool, measure the same observable with the simplest possible method: a manual swap, a plain script, an unautomated run. That gives you a reference point, and the criterion becomes a comparison rather than an absolute. Comparisons survive changing network conditions in a way that absolute numbers do not.

Dependency limits are the hardest kind to argue with. If your RPC provider allows a certain request rate, a criterion asserting behaviour beyond it is testing the provider rather than the tool. Read what your dependencies actually promise, then write criteria that live inside those bounds. The Solana terminology reference is worth having open while doing this, because a criterion that uses "confirmed" and "finalized" loosely will be interpreted differently by every reader.

The measurement window

A window has three parts: how many observations, over what period, and under what conditions. All three matter. Fifty transactions in one quiet minute and fifty transactions spread across a busy hour are different samples of different populations, and a criterion that does not say which one it wants will get whichever was convenient.

Conditions are the part most often left implicit. If you care about behaviour under congestion, the window has to specify that the sample was taken under congestion, which means it has to say how you determined congestion was present. Otherwise a tester runs the check at a quiet moment, records a pass, and both of you believe something that was never tested.

Where the tool is an automated Solana volume bot running unattended over hours, the window is usually defined by the run itself rather than by a stopwatch, and the criterion should say so: one full session, all confirmed transactions counted, no cherry-picking of a favourable stretch. Sampling a subrange after the fact is the most common way an honest tester produces a dishonest number.

Verdict vocabulary

Five verdict states are enough, and using fewer causes trouble. PASS means the threshold was met within the window. FAIL means it was not. BLOCKED means the run could not proceed for reasons unrelated to the criterion. NOT RUN means exactly that. INCONCLUSIVE means the run completed but produced a sample too small or too contaminated to judge.

PASS requires the full window. A partial window that met the threshold is INCONCLUSIVE, not PASS.
FAIL requires the full window too, unless the threshold became unreachable earlier, which is worth noting in the evidence line.
BLOCKED always names the blocker. A blocked criterion with no named blocker is a NOT RUN with better presentation.
INCONCLUSIVE is the honest home for most first attempts and should not be treated as a failure of the tester.
NOT RUN stays visible in the report. Deleting unrun criteria is how a report reaches a hundred percent coverage of the things it happened to cover.

The single most valuable rule in this section is that a report showing four passes and six not-run entries is more useful than a report showing four passes. The second report looks better and tells the reader less, and the reader is usually the person who will be surprised later.

Negative criteria

Positive criteria say what the software must do. Negative criteria say what it must never do, and for anything holding keys or moving funds they matter more. They are also harder to test, because proving absence over an unbounded period is impossible; what you can do is bound the claim to the observed window and state that bound honestly.

Destination constraint. No transaction sends value to an address outside the configured set. Observable: the destination accounts in every confirmed transaction from the run.
Size constraint. No single transaction exceeds the configured maximum size. Observable: the amount field of each swap instruction, read from the transaction rather than from the tool's own summary.
Rate constraint. No more than the configured number of transactions are signed per interval. Observable: signature timestamps grouped by interval.
Authority constraint. The tool never changes a mint or account authority. Observable: absence of authority-changing instructions in the transaction set.
Silence constraint. No action is taken that leaves no record. Observable: every balance change in the wallet maps to a signature the tool reported.

The fifth one is the strongest test of an unfamiliar tool and the easiest to run: take the wallet's transaction history for the run period, and reconcile every entry against the tool's own log. Entries the log does not explain are the finding. This check needs no source access and no cooperation from the vendor.

Security criteria that are never negotiable

Two criteria that fail the tool outright

First: no legitimate tool needs your seed phrase or exported private key typed into a website, a chat window, a support ticket, a spreadsheet or a form. A tool that asks is asking for permanent, irreversible control of everything those keys hold. This is a FAIL and it ends the evaluation; there is no threshold to negotiate and no configuration that makes it acceptable.

Second: any operation that transfers ownership or authority is irreversible once confirmed. Criteria covering authority changes, account closures and token approvals should be written as negative criteria and tested on a wallet that holds nothing you would miss.

A softer but still important criterion concerns key custody design: where does the signing key live, who else can reach it, and what happens to it when the process stops. You can test the observable part of this by checking whether keys are readable in plain text from configuration files and whether the tool logs them, which it should never do, including at debug level.

When a claim is untestable

Some claims cannot be converted, and the correct action is to mark them untestable rather than to invent a proxy. Claims about future behaviour, about internal processes you cannot observe, or about aggregate outcomes across other people's accounts all fall in this category. Writing a weak proxy criterion is worse than writing none, because the proxy passes and the claim then appears validated.

The useful move with an untestable claim is to convert it into a question for the supplier and record the answer verbatim. That does not test anything, and it does two other useful things: it dates the assertion, and it makes the gap visible to whoever reads the report later. An evaluation that ends with three untestable claims clearly marked is a more honest document than one that ends with three green ticks.

Sign-off and what it commits you to

Sign-off is a sentence, not a signature block. It names the build, the configuration, the wallet set, the window and the date, and it says what was and was not covered. "AC-001 to AC-014 passed on build 3f2a1c with the configuration in config-rehearsal.json, over one twenty-swap session on 12 August; AC-015 to AC-018 not run" is a sign-off. "Tested and approved" is not.

What it commits you to is narrow on purpose. It does not say the tool is safe, it does not say it will behave the same next week, and it does not transfer risk to the person who wrote the criteria. It says that on that build, with that configuration, over that window, the observed results matched the expected ones. Everything a test report can honestly claim is inside that sentence.

Keep the signed report with the archived claims, the criteria and the evidence files together. When something goes wrong three months later, the first question is always what was true at the time, and the only people who can answer it are the ones who wrote it down while it was still true.

Questions the desk gets asked

What are acceptance criteria in software testing?

They are the conditions a piece of software must meet for the people paying for it to accept it as done. Each one names something observable, a threshold, and the circumstances under which the observation is taken, so that two different people running the same check reach the same verdict without discussing it.

How specific should an acceptance criterion be?

Specific enough that someone who was not in the conversation can run it. The practical test is to hand the criterion to a colleague with no context and ask what they would measure. If they ask a clarifying question, the criterion is not finished, and the answer to their question is the part that was missing.

Should acceptance criteria include performance numbers?

Yes, when the number matters, and only when you can defend where it came from. A threshold copied from a marketing page is a wish. A threshold derived from a baseline you measured, or from a limit imposed by something you depend on, is a requirement. Write the source of the number next to the number.

What is the difference between a criterion and a test case?

A criterion states what must be true. A test case states the steps that establish it, including inputs, environment and expected result. One criterion often needs several test cases, and one test case can contribute evidence to several criteria. Keeping them separate stops the criterion drifting to match whatever the test happened to do.

How many acceptance criteria does a trading bot need?

Fewer than most lists contain and more than most teams write. A workable list covers the money path, the failure path, the configuration surface and the evidence trail. If the list has fifty entries and forty of them are variations on the same observable, it will not be run twice.

Can acceptance criteria be changed after testing starts?

They can be added to freely and relaxed only with a record. Adding criteria mid-run is how discovered behaviour gets captured. Loosening a threshold because the software missed it is how a test report becomes a formality, so if a threshold moves, the old value, the new value and the reason all stay in the document.

Who writes acceptance criteria for a bought tool?

The buyer, always. The vendor writes claims; the buyer writes the criteria those claims will be judged against. If the criteria arrive from the seller, the test measures whatever the seller was already confident about, which is a demonstration rather than a test.