A/B experiments

An experiment splits one collection's traffic between variants. Each variant is a set of search parameters (and optionally recommendation parameters) merged over the requests of the sessions assigned to it. Reports compare the variants using the events your storefront sends.

Experiments need the experiments plan feature, which is on Pro. Without it every experiment operation answers 403. They are managed with a full key; search keys only take part through the session header.

Create

const experiment = await client.experiments.createExperiment({
  name: "Popularity first",
  collection: "products",
  traffic: 0.5,
  variants: [
    { id: "control", search_params: {} },
    { id: "popular", weight: 1, search_params: { sort_by: "popularity:desc,_text_match:desc" } },
  ],
});

An invalid configuration answers 400. A new experiment is a draft and serves nothing until startExperiment.

Lifecycle

• startExperiment sets running and records started_at. It answers 409 if the experiment is already running. Starting clears any winner detected in an earlier run.
• stopExperiment sets stopped and records stopped_at. On an experiment that is not running it changes nothing.
• updateExperiment answers 409 while the experiment runs: stop it, change it, start it again.
• deleteExperiment removes the experiment.

Starting, stopping and promoting send the experiment.started, experiment.stopped and experiment.promoted webhooks.

Enrolment: the session header

A request takes part only when it carries X-NOVA-SESSION-ID. Use one id per browser tab; the aacsearch-ui packages and the widget send it for you. Without the header a request is never enrolled and gets the untouched parameters.

Assignment is deterministic. The experiment id and the session id are hashed to a point: below traffic the session is enrolled, and the point then picks a variant by weight. The same session always sees the same variant, with no lookup and no cookie of ours.

The header enrols searchCollection and multiSearch requests. For getRecommendations, the header or the body's session_id is used. The applied variants come back in the response header:

X-Nova-Experiment: 8c1f…:popular

Its value is <experiment_id>:<variant_id>, comma-separated when several running experiments on the collection apply.

Measure

Reports join your events to assignments by session. Send the same value as session_id in every event that you send in X-NOVA-SESSION-ID, and pass query_id so clicks and purchases are attributed.

getExperimentReport counts, per variant, the events inside the experiment's running window (started_at to stopped_at):

significant_vs_control is either true or absent. It is absent while either side has fewer than 30 searches, or while the intervals overlap. Variants that a promotion removed stay in the report.

Winner detection

A nightly job (04:00 UTC) runs the same report for every running experiment that has no winner recorded yet. It records a winner only when both are true:

• The control and the candidate each have at least 30 searches.
• The candidate is significant_vs_control.

If several variants qualify, the higher conversion rate wins, and then the one with more searches. The job then sends the experiment.winner_detected webhook:

{
  "experiment_id": "8c1f…",
  "collection": "products",
  "winner_variant_id": "popular",
  "conversion_rate": 0.042,
  "reason": "Clears control's confidence interval (1840 vs 1795 searches)."
}

Detection never promotes on its own. Revenue against conversion trade-offs and seasonal effects still need a person. Subscribe to the event to prompt someone — see Webhooks for delivery setup, signature verification and retry handling.

Promote and roll back

await client.experiments.promoteExperiment({ experimentId, variant_id: "popular" });

Promotion collapses variants to the chosen one and sets traffic to 1. The experiment stays running, so every enrolled session gets the winner. The previous configuration and the report at that moment are saved as a revision. An unknown variant_id answers 400.

rollbackExperimentPromotion restores the variants and traffic saved by the latest promotion and keeps the status. It answers 409 when there is no promotion to undo, or when the configuration has changed since.

The console's Experiments screen runs the same operations; see Insights.