Skip to main content

Statuses

Every decision response carries experiment_status: There is no “winner declared” state. Variants that lose are retired; the ones that remain keep serving while Curious generates new challengers from what the results taught, so an element keeps improving for as long as it is live.

From declaration to the first variants

The first decide or declare for an element creates the experiment with your control and rewards. Curious researches your product and audience once per organization, works out the dimensions along which this element could vary, generates a set of candidate variants through several heuristics, extracts each one’s hypothesis and assumptions, compares them pairwise, and keeps the best few. Under manual approval they wait for you; once live, the status moves to exploring. Your code does nothing during this; decisions keep returning the control.

While exploring

Each decision is a draw from a contextual bandit over the live arms, tilted toward what works for users like this one (see Assignment and stickiness). Rewards you report attribute to the decisions that preceded them within the declared windows, and the model refits as outcomes mature. Variants are generated in rounds, not all at once, so each round benefits from what the last one taught.

How a variant is concluded

A variant is concluded when one of three things happens, each recorded with its reason and visible per arm in the dashboard:
  • It lost to another arm, under a sequential test that is valid at any time, so a conclusion is never an artefact of when Curious looked.
  • It is practically equivalent to another arm: the two are confidently closer than options.minimum_effect_size (default 0.2, the smallest relative difference from the control you care about), so one of the pair is retired rather than tested forever. A control and a variant that prove equivalent always retire the variant.
  • It breached a constraint you declared.
You can also conclude a variant by hand in the dashboard. The control is never concluded by any path. When variants are concluded, Curious generates new ones in a round informed by everything learned so far.

What ends an experiment

Changing a declaration-class field (control content, rewards, format, requirements, parameters, options.holdout) concludes the experiment with reason superseded and starts a successor from the new declaration on the same element. Concluding in the dashboard ends it with reason manual. Mutable fields never end an experiment. See Declarations.