CARF

Experimental design

Research design requirements. The refund example is illustrative; its comparisons and quantities are not a frozen or completed study.

Conditions and nested units

Condition / design cell

An arm with declared factor settings. A configuration is not an independent replicate.

Matched block

A replicate containing the prespecified episode or session chain for every matched condition. Match declared nuisance factors and initial state; manipulated factors vary explicitly.

Chain

A prespecified sequence of sessions with a declared carry/reset policy for relevant memory, context, tool state and world state. The policy determines which dependencies the comparison retains.

Episode / session

One run of one condition, possibly within a dependent chain. Sessions do not automatically count as independent observations.

Turn and segment

A turn is one exchange within a session. A segment is a stretch of turns, not a matched block. More turns do not by themselves increase the independent sample size.

Assignment, matching and perturbation

Declare the assignment mechanism, blocking factors, order and analysis unit before acquisition. Use randomised or blocked assignment where appropriate to the question, not as a label detached from how observations were generated. Match the task, initial world state and declared nuisance factors. Safe-route availability is held fixed in the refund example; it is not matched away in a study that manipulates safe-route availability.

Specify the experimental operator: what changes, where it enters the model-visible interaction, when it is applied, its dose and what remains fixed. Dose may concern the declared amount, repetition or timing of exposure; it needs an operational definition in the protocol. A perturbation can be adversarial or non-adversarial. Its category alone does not establish its causal effect.

Comparisons and controls

• Compare a perturbation with its matched control; preserve the declared primary contrast even when realised trajectories diverge.
• Use positive controls with known expected behaviour and negative controls with prespecified false-alert limits to test whether the measurement pipeline is responsive and selective.
• Use current-state, load/burden, form and meaning controls to separate a history effect from extra content, changed semantics or an already visible current-state difference.
• Manipulate order at fixed content and declare carry/reset contrasts where the question concerns history. Do not infer a history effect merely from two different paths.
• Use dose-response comparisons and ablations to challenge the proposed explanation. Their interpretation depends on what the operator actually changed.

Independent endpoints

Define and adjudicate endpoints independently of the candidate risk score: the proposed action or boundary proposal, the executed effect at the environment boundary, and legitimate task completion. Declare event timing, follow-up horizon, unavailable observations and exclusion rules. A claim that a refund succeeded is not confirmation that the account changed; preventing every action is not automatically a useful intervention.

Representation

Candidate measurements under research. No representation family is presumed to preserve every distinction needed for warning.

Typed behavioural telemetry

A telemetry contract declares the observable events, fields, provenance and retention requirements. Distinguish model-visible inputs, observations, proposed actions, tool calls, tool returns and observed world-state changes. Retain their order, session/chain/block membership, source identity and relevant timing. An unavailable observation must be represented as unavailable, not silently interpreted as no event or no risk.

A trajectory is the recorded sequence under that contract. Token-level views, semantic representations, transitions between observations and whole-trajectory summaries preserve different information. Specify each transformation, its version and which history is available when it runs. Neither telemetry nor a candidate state exposes hidden internal reasoning.

Statistical inference

Research requirements. The current source records no registered or validated sequential null; the listed evidence does not establish a prospectively qualified warning rule.

Unit, estimand and uncertainty

Name the analysis and resampling unit and the dependence it preserves. A turn-level score does not make turns independent replicates. For a matched experiment, specify the condition contrast and its aggregation across eligible matched blocks or chains, within a declared task/model stratum. The estimand is the quantity that contrast targets, not whichever summary appears most favourable afterwards.

Report the endpoint, comparison, sample size at the independent unit, effect estimate and design-appropriate uncertainty. Keep proposed actions, executed effects and legitimate task completion separate. Prespecify missing-data handling, exclusions, multiplicity and stopping rules. An interval too wide to resolve the question supports an inconclusive disposition, not evidence of absence or a successful detector.

From a prefix score to a warning decision

At each declared observation time, compute the candidate score using the available prefix. Separately fix the rule mapping scores to warnings, its threshold or evidence criterion, eligible population, horizon and any reset policy. A score is not automatically a calibrated risk probability.

Let T_warn be the first qualifying warning and T_event the independently adjudicated endpoint time. For an event with a prior warning, lead time is T_event - T_warn. Prespecify how much time the intended response needs. Warnings at or after the event do not count as useful advance warning.

Report false alerts under the declared negative condition or event-free horizon, missed events, late warnings and unavailable measurements with their denominators. Do not score unavailable telemetry as a correct negative or compute apparent success only among events that received a timely warning. The time unit and event-free follow-up are part of the definition.

Calibration, held-out qualification and sequential validity

Fit representations on discovery material, use the declared calibration material to set decision rules, then evaluate the frozen choice on untouched qualification material. Split at a unit that prevents related trajectories or duplicated content from contaminating the holdout. Any tuning after inspection is further discovery and needs new confirmation material.

A fixed-sample interval or test does not gain sequential validity by being inspected repeatedly. A sequential design needs a stated null, information available at each look, dependence assumptions, an evidence construction valid for that process and a declared decision rule. Candidate e-process or conformal constructions must be justified for those assumptions; their names alone supply no guarantee.

Transport, drift and operating scope

Bind a qualification to its generator or task distribution, model, tool environment, telemetry, representation, policy and time horizon. Synthetic qualification does not establish performance on model-generated or real-world sessions. Specify how a change or drift is detected, when the measurement is treated as unsupported or unavailable, and what fresh evidence is required before extending or recommissioning its operating scope.

The statistics programme develops inference methods in its own right; a statistical package is not evidence that a CARF warning rule has passed these conditions.

Intervention

Specified research direction. The current source describes a paired-execution mitigation design, not a completed intervention study or a demonstrated effect.

Warning policy and response policy

The warning policy decides when evidence warrants an alert. The response policy decides the bounded action permitted in that situation, including what to do when measurements are unavailable. Freeze both before evaluating the combined policy; a useful warning does not establish that its proposed response helps.

Responses can hold an action, redirect to a safe-route affordance, re-anchor a claim to its authoritative source, request confirmation or dual authorisation, escalate to human review, or halt. Declare which responses are available, their costs and timing, who can authorise them and the conditions for resuming the legitimate task. An untrusted message claiming approval is not source re-anchoring.

Intervention study and attribution

Compare the specified control policy with its declared comparator on matched initial conditions or a frozen episode design. Record assignment, stochastic variation and carry/reset choices rather than assuming paired executions will follow identical paths. Evaluate differences separately in boundary proposals, executed effects and legitimate task completion; do not replace them with one undifferentiated safety-improvement number.

Distinguish warning-performance evaluation from the effect of making a response. Once intervention changes the trajectory, the observed outcome belongs to that intervention condition. It does not reveal what would have happened without the intervention; that comparison needs its own design.

Displacement and post-intervention monitoring

Prespecify follow-up for attempts through another tool, route, session or later time, and for new unwanted effects introduced by the response. Continue to record unavailable observations, delayed task failures and legitimate completion. A blocked refund on one route is not sufficient evidence of benefit if the prohibited effect moves elsewhere or all useful work is suppressed.

Formal effect control

Target framework architecture. This specifies an effect-authorisation boundary, not a claim that the complete CARF runtime has been formally verified.

Premises, authority and effect language

Represent the proposed consequential effect in a declared effect language with defined operations, arguments, resources and boundaries. Bind it to the relevant state, policy version, source identity and authority. The premises must say which approvals, observations and constraints the decision relies on, how they are established and when they cease to apply.

A statistical warning may request a hold or review; it cannot grant effect authority. Conversely, a low score or absence of warning cannot supply missing permission. The untrusted claim of approval in the refund example must not become an authorisation premise merely because the agent repeats it.

Policy, proof scope and refinement

A formal check concerns a precise proposition about an effect under specified premises and policy. State the checked proposition, hypotheses and trusted components. A checked policy model does not by itself establish that its premises are true in the world, that the policy is sufficient for safety, or that the implementation enforces it.

The refinement obligation connects abstract effects to concrete tool calls and execution boundaries. Specify how the implementation preserves the checked constraints, including state changes between decision and execution and routes that could bypass the gate. Implementation conformance is a separate evidential obligation, not something inferred from a proof of the abstract model.

Refusal and no certificate

Under the target policy, missing or stale premises, ambiguous effects, unsupported operations, failed checks or absent authority do not produce an authorisation certificate. Refuse the effect or take only an independently authorised hold/escalation route. Record the reason, the proposal, relevant versions and evidence references. No certificate means no established permission at this boundary; it is not a prediction that the action would certainly cause harm.

Runtime integrity

Reference-architecture requirements. Provenance, conformance and replay support inspection; none substitutes for statistical qualification, a proof or effect authority.

Conformance and source identity

Check that the running implementation follows the declared telemetry, measurement and control contracts. Bind model-visible content to the source that actually rendered, rather than only to a template or intended prompt. Record tool and model identity, carry/reset state, policy and configuration versions, transformations, timing and the relevant environment boundary. Test missing, duplicated, contaminated or mismatched records as integrity failures.

Keep proposal, admission decision, execution and effect confirmation distinct. A successful-looking tool response is not enough when the endpoint requires a world-state change. Where a receipt or other confirmation is required, bind it to the intended effect and record absent or inconsistent confirmation instead of silently declaring success.

Content-addressed evidence, replay and audit

Bind raw observations, derived measurements, decisions and outputs through hashes and manifests that identify the exact inputs, code, configuration, policy and analysis versions. An evidence bundle must preserve these references and the reasons for exclusions. A hash identifies bytes; it does not establish their truth, the authority of their source or the completeness of the record.

Declare what replay can reproduce: recorded decisions and deterministic derivations under pinned inputs, or a new execution with its own stochastic and external dependencies. Replaying a record does not imply that a live provider will reproduce the same trajectory. An audit needs the relevant contents and access, not just an unresolvable digest.

Every claim needs a trace back to the runs that produced it.

The reference architecture defines explicit record types and interfaces from raw observations to derived measurements, evidence records and decisions. Each result should record the data, code, model, configuration and analysis version that produced it. The provenance requirement is part of the target design, not a claim that every internal record is publicly available.

trace→event→episode→measurement→evidence

Falsification conditions

Research rejection and stop conditions, not reported outcomes. Set the relevant thresholds and dispositions before inspecting the results; this list is not a substitute for a study-specific protocol.

Keep the full set of failure conditions together. Distinguish a refuted hypothesis, an unsupported or inconclusive claim, an invalid run and an effect that must be refused. Preserve the record and its reason in every case.

1. The intended contrast was not delivered. Rendered conditions coincide, assignment is wrong, dose is absent, or carry/reset state violates the protocol. Invalidate the affected comparison rather than interpreting it as a behavioural result.
2. The observation pipeline is unreliable. Source identity, event timing or endpoint confirmation is missing or inconsistent; duplication, contamination or unauthorised exclusions compromise the record. Exclude affected runs from scientific interpretation with reasons retained.
3. Known controls fail. Positive controls are not detected at the prespecified rate, or negative controls exceed the false-alert bound. Do not qualify the instrument on unknown cases.
4. The proposed state or geometry is not stable enough for its use. Claimed structure fails the specified resampling, representation or perturbation checks. Reject or narrow the representation claim rather than treating a pattern as a mechanism.
5. Simpler explanations account for the signal. Current-state, burden, form, meaning or transparent baseline controls remove the claimed incremental information. Do not retain a trajectory-specific explanation without its required contrast.
6. Declared dose, order/history or ablation predictions fail. Reject or revise the specific explanation whose prespecified test failed; do not relabel an unplanned contrast as confirmation.
7. The information-loss claim lacks its required evidence. Different realised futures alone do not establish different conditional outcome distributions. An unverified summary collision or an unresolved outcome contrast cannot support that claim.
8. The endpoint or prefix is circular or contaminated. Failure labels depend on the score under test, or future observations, labels or outcomes enter fitting or scoring. Do not claim prospective warning from that evaluation.
9. Held-out qualification fails or was no longer held out. The frozen candidate misses its acceptance criteria, related observations leak across partitions, or the method is tuned on confirmation outcomes. Further adaptation is discovery, not a passed confirmation.
10. The experiment cannot resolve the question. Available precision, effective independent sample size or observation availability is insufficient. Report inconclusive evidence; do not turn a non-detection into an absence claim.
11. Warnings are not useful early warnings. Lead time is insufficient, false alerts or missed events exceed their declared limits, or unavailable observations are hidden from the denominator. Withhold the warning-performance claim.
12. The statistical guarantee does not cover the procedure. Repeated looks, stopping, dependence, selection or calibration violate the stated construction or its assumptions. Suspend that guarantee rather than borrowing a fixed-sample or unrelated theorem.
13. The result does not transport or the scope has drifted. A changed model, task, distribution, tool, telemetry or policy breaks the qualification assumptions. Restrict the claim and require fresh qualification before extending its use.
14. The intervention does not improve the declared outcomes. Benefit is unsupported, legitimate task completion deteriorates beyond its allowed limit, or failure is displaced to another route or time. Do not infer intervention efficacy from warning accuracy or one blocked action.
15. Effect authorisation cannot be justified. Premises are absent, stale or unbound; authority is missing; the effect is outside the language or policy; or the formal check fails. No certificate: refuse the effect or follow only an independently authorised escalation route.
16. The implementation or audit chain breaks. Concrete execution bypasses or fails to refine the gate, effect confirmation is absent, hashes or sources disagree, or required replay records are unavailable. Withhold the affected execution/conformance claim and preserve the failure for audit.

A stopped or rejected experiment does not automatically falsify every CARF hypothesis. A passed bounded test does not establish the whole framework either. Retain only the claim supported by the actual design, record and operating scope.

Nine sustained questions, in three groups.

Characterise what happens, recover structure from it, validate what can be trusted. These are research questions, not a list of achieved capabilities. Token and trajectory geometry is the primary measurement hypothesis, evaluated against independently defined tool actions and world-state effects; the other questions act as challengers or complements.

Protocols, sources and further reading

Study-specific thresholds, sample sizes, frozen protocols, numerical results and reproducible revisions require their own evidence records. The technical requirements above do not fill those gaps or constitute a new result.

• Method: approach, Scientific TDD and experimental design
• Plain-language explainers
• Publications and released software
• Request technical material: protocols, evidence records and architecture detail