01
Discover
Characterise the object and observations; generate candidates, compare alternatives and search for counterexamples. Define a discriminating test before treating a promising pattern as confirmation.
Method
A useful question needs a test that can change the answer.
Discover. Test. Promote or reject. Across ORBIT, reproducibility, versioning and evidence capture surround all three stages. They keep the route from a question to a narrow claim inspectable.
Approach
Start with the claim, not the machinery
Define the object, the proposed claim and what would challenge it. The kind of evidence must fit the question.
Formal proof establishes a stated implication under explicit hypotheses. Numerical evidence describes a calculation on specified inputs, not unsearched cases. Empirical validation tests a claim against observations under a declared design. Shared software does not make those conclusions equivalent.
Scientific TDD
Discover, test, then decide what survives
Scientific test-driven development makes failure informative. A favourable discovery is a candidate, not its own confirmation.
02
Test
Freeze the candidate, experimental or inferential unit, endpoint, controls and rejection criteria. Test empirical claims on appropriate fresh evidence. For mathematics, check the argument and its hypotheses, rather than treating examples as proof.
03
Promote or reject
Accept a narrow claim, revise, reject or mark inconclusive. Publish the source and limits. A revision informed by confirmation results needs a new, clearly labelled evaluation.
Discovery material cannot retroactively become untouched confirmation material. Negative and inconclusive results remain in the record; successful execution alone does not promote a scientific claim.
Experimental design
One declared comparison, with the right controls
Decide what varies, what is matched, what is recorded and what constitutes an independent replicate.
Unit and contrast
A condition is an arm of the design. A matched block contains the prespecified episode or session chain for each condition. Declare the primary contrast and block-level analysis before confirmation: turns are nested observations, not extra independent replicates.
Matched initial conditions
Match task, initial world state and relevant nuisance factors; vary the declared factors. Realised paths may still differ stochastically. Safe-route availability must vary explicitly if it is the factor under study, rather than being described as matched.
Separate endpoints
Record proposed actions, executed effects and legitimate task completion separately. Define failure independently of the candidate warning. Useful warning and useful intervention are separate hypotheses.
A worked refund example
Start matched episodes from the same account state. One condition receives an untrusted claim of approval; its control does not. Compare unauthorised refunds proposed, refunds applied and legitimate work completed across matched blocks, with design-appropriate uncertainty.
A response test adds a declared intervention factor. Measure false warnings, missed failures, lead time and effects on legitimate work. Prespecify acceptance criteria, multiplicity and sequential-monitoring rules; a configuration count is not a power calculation.
Across all three stages
Keep the route from input to claim inspectable
Reproducibility is the surrounding practice, not a fourth research stage.
Capture source revisions, inputs, configuration, data provenance, software versions, exclusions, analyses and outputs throughout discovery, testing and claim decisions. For proofs, name the exact statement, assumptions and dependencies; for finite searches, the searched domain and unchecked remainder.
State what cannot be public and how an authorised reviewer can request it. A private record is not independently reproduced evidence.
Publication
Publish the claim and its actual basis
A public article, archived manuscript, software release and peer-reviewed paper have different statuses.
Identify the particular release, manuscript or proof record, with qualifications beside the claim. Overdog is canonical for site writing; syndicated versions link back to that origin.