Experiment Mode

What Experiment Mode is

Experiment Mode answers one processing question at a time: given a specific image state, which of several candidate approaches produces the best result, and how confident should that conclusion be. It runs a defined set of candidates from the same starting point, measures and compares them, and records what happened -- including when nothing beat doing nothing at all.

What counts as a candidate

A candidate can be a reference method, an alternative method, a parameter probe of the same method, an object-specific strategy, a fallback, a no-op control, or an explicitly historical/deprecated approach kept for comparison. Treating these as one undifferentiated list would blur what a given comparison is actually testing -- a parameter probe answers a different question than a method alternative does, even when both appear in the same candidate set. When a Handbook article shows a candidate table, it uses this same vocabulary: what the candidate is, what varies about it, whether it was adapted to the image, whether it is a control or fallback, and what evidence currently supports it.

A fair comparison starts from the same input

Every candidate in one experiment begins from the same exact image state. Candidate outputs do not chain into each other -- each one is an independent branch from the same starting point, not a pipeline where one candidate's output becomes another's input. That is what makes the comparison fair: any difference between candidates comes from the method itself, not from an accumulated difference in what each one started with.

Adaptive parameters

A candidate's ontology-declared default parameters are not always what actually ran. NOVA can adapt parameters to the current image's own statistics, so the evidence that matters is the executed runtime value, not just the declared default. When adaptation itself fails, that failure is recorded rather than silently treating the fallback-to-default run as equivalent to a successfully adapted one.

Measurements before judgment

NOVA measures candidates before any visual or AI judgment happens, and uses those measurements as multi-dimensional safeguards and evidence rather than one aggregate score to maximize. Hard analytic rejection removes candidates that violate a current process-specific analytic heuristic or safeguard -- a threshold that is appropriate and revisitable for that family, not a universal physical validity rule, and one that is not yet defined evenly across every process family -- before visual comparison even starts, so a visual assessor is never asked to pick a candidate that a quantitative safeguard already ruled out.

How NOVA compares survivors

Candidates that pass the safety checks are compared using their measurements, a visual/AI assessment, and any relevant historical context. When only one candidate survives, it is selected by default rather than by comparison -- see 'What failure modes mean' below for why that distinction matters.

Human review

Optional human review uses blinded candidate labels rather than named methods, so a reviewer's judgment is not steered by knowing which candidate is which. When a person's judgment disagrees with NOVA's own assessment, that disagreement is recorded as evidence in its own right, not smoothed away in favor of one source overriding the other.

What 'winner' means

A winner is a local-run winner: the candidate selected under one specific input, candidate set, and software state. It is not automatically the universal best choice for every future image of that kind. Treating a single run's winner as a permanent rule is exactly the overgeneralization this framework exists to prevent.

Controls and 'do nothing'

A no-op control is not a weak candidate added to make the real candidates look good -- it is the reference needed to answer whether doing anything helped at all. Without a control, every candidate can leave the image worse than it started, and one of them will still be ranked first. A good control supports three honest conclusions: a treatment clearly improves on the unchanged input, treatments are roughly tied with it (suggesting the step may be unnecessary), or every treatment is worse than the control (meaning skip the process). 'Skip' is a valid winner whenever the process itself turns out to be unnecessary.

Learning over time

Repeated experiments build a historical record: win rates, adapted parameters, and evidence counts across runs. This is memory and context accumulating over time, not the underlying model being retrained -- a historical win rate describes how often something was selected across recorded runs, and is read that way rather than as a growing certainty score.

A win rate is only honest evidence when its denominator is honest. Failed, rejected, no-op-won, single-survivor, and inconclusive outcomes belong in that denominator before any strong aggregate claim is made -- a historical corpus that only remembers clean wins reads as stronger than the underlying evidence actually supports. Repeated runs on the same target are kept separate from independent-target evidence rather than pooled as if they were equally informative, and evidence is partitioned whenever the code, tool, model, evaluator, prompt, metric, or treatment identity behind it materially changes -- a win recorded under one of those no longer counts as evidence for a materially different one. For the same reason, a win recorded against an older code, tool, or model version is not carried forward as current evidence for a newer version without revalidation; historical evidence stays visible, but it is not silently treated as if it were measured against what NOVA runs today.

Evidence strength for experiments

Experiment evidence sits alongside, not above, the other evidence origins described in How NOVA Knows: source-confirmed code, execution-confirmed runs, and human validation each answer a different question than a run-local winner or a historical win fraction does. A narrow score margin between two candidates is not a statistically significant result -- generalizing it requires repeated trials, not a single comparison.

What failure modes mean

Different failures mean different things, and Experiment Mode keeps them distinguishable. A candidate that cannot execute in the current environment is an operational failure, not evidence of lower image quality. An experiment where every candidate fails should fail closed rather than fabricate a winner. A candidate that violates an analytic safeguard is excluded from selection, with the evidence retained rather than discarded. When adaptation itself fails and a candidate falls back to ontology defaults, that fallback is recorded so later learning does not treat it as an equivalent, successfully adapted trial.

Auditable claims

1. A no-op or 'do nothing' candidate is a legitimate winner, not a placeholder inserted to flatter the other candidates. NOVA already has the execution mechanism for an unchanged candidate, and a process step that a control beats is one the Handbook can honestly say should be skipped.

   source-confirmed · none

2. A single measurement is not the aesthetic answer. A denoise method can lower background noise while destroying real detail; a deconvolution method can lower measured FWHM while adding halos. Experiment Mode uses measurements as safeguards and evidence before judgment, not as one scalar objective to maximize.

   source-confirmed · none

3. A candidate that fails to run in the current environment is recorded as an operational failure, not as evidence that its method produces worse images than the candidates that did run.

   source-confirmed · none

4. When no comparison judgment is available and only one candidate survives, NOVA's current runtime selects it by default and labels that outcome an operational fallback -- never presented as an AI-selected winner, because no comparison actually happened.

   execution-confirmed · none

5. A single run's winning candidate is a conclusion under that run's specific input, candidate set, and software state -- it is local evidence, not a cross-target recommendation, until repeated independent evidence supports generalizing it.

   source-confirmed · none

6. A narrow score difference between two candidates, such as a 9 against an 8, is not equivalent to a statistically significant treatment effect. Translating a score margin into a generalized rule requires repeated trials, not one comparison.

   source-confirmed · none