Insights and quality

The Insights group turns the events your storefront and widgets report into dashboards, experiments, an offline quality gate and operational alerting. All of it depends on events — search, click, add-to-cart, purchase, visit — arriving from the client with a stable X-NOVA-SESSION-ID.

Analytics

/dashboard/analytics (the analytics plan feature) is the traffic dashboard. The same numbers are available programmatically — see the analytics API. A collection filter, a from/to range (last 30 days by default, with 7 / 30 / 90-day shortcuts), an optional "compare to previous period" and a Refresh control sit above eight tabs:

A header action links straight to Search issues. Owner or admin rights are required to change retention. If the range is too large for the event store, the report says it is truncated and that session metrics cover the earliest events only.

Each tab loads its data independently: successful panels keep rendering while a failed call is listed in one error line above them, so a broken chart does not blank the screen.

Search issues

/dashboard/search-issues finds the searches that are failing. With the analytics feature, pick a collection and a 7 / 30 / 90-day window, and it shows:

• CTR, searches, clicks and the latest zero-result rate, with a low-CTR card when CTR is under 10% over at least ten searches.
• Days where the zero-result rate was high (at least five searches and 25%).
• Zero-hit queries with occurrence counts and share, each with Debug (opens the search playground), Suggest (opens synonym suggestions for that query) and Rules (opens the collection's search rules).
• Ranking opportunities — queries with at least three searches that are not in the zero-hit list, capped at the top fifteen.
• Top queries marked as zero-hit or has-results, each with its own Debug link.

The header offers Synonym suggestions for the chosen collection and a Refresh. Empty states cover "pick a collection" and "no search data yet", and the zero-hit panel links to a nohits_queries analytics rule when there is nothing to mine yet.

Analytics rules

/dashboard/analytics-rules (the analytics feature) maintains the engine's analytics rules. New opens a dialog with the rule's name, type (with a hint per type), source collection, event type, destination collection, limit (minimum 1, default 1000) and tag (default "console"). Each rule has a type — nohits_queries, popular_queries, counter or log — a source collection, an event type (search, click, conversion, visit, custom), a destination collection (required for the nohits / popular collectors), a limit and a tag. A nohits_queries rule links to Synonym suggestions for its collection; any other rule with a collection links to Search issues.

Only the common fields are exposed here; advanced parameters (expand_query, meta_fields, counter weights) live in the admin resource editor. A footer debug panel takes a user id, a rule name and a limit (1–1000, default 50) and lists the matching events (When, Type, Query, Doc, Collection) — the contract requires both a user id and a rule name, so it is not a general event browser.

Experiments

/dashboard/experiments runs A/B tests on ranking or merchandising changes. The experiments plan feature is required (Pro).

• The list shows each experiment's collection, status (draft / running / stopped, plus a promoted badge), its variants with weights and search parameters, the enrolled traffic share, and the dates.
• Creating one takes a name, a collection, the traffic percentage to enrol (1–100), a variant name (default "Variant B") and the variant's search-parameter JSON; the control and the variant are created as a draft.
• Start and Stop control traffic, Report opens the per-variant table (sessions, searches, clicks, CTR, carts, purchases, conversion, revenue) with a "beats control" flag when the difference is significant and 95% confidence intervals, and Promote collapses the experiment to the winning variant at 100% traffic while leaving it running. The report drawer has its own Refresh, an empty state when no sessions have enrolled yet, and the raw report JSON under the table.
• Successful actions show a dismissible notice ("… is running", "stopped", "promoted … to 100% traffic", "deleted").

Nothing stops an experiment automatically. Metrics only accrue for sessions that carry a matching X-NOVA-SESSION-ID on a started experiment. Deleting an experiment removes its assignments and stops it.

Campaigns

/dashboard/campaigns schedules merchandising for a window. A campaign has a name, an optional collection (blank means all), a status, a priority (higher wins when campaigns disagree), start and end timestamps, a weekly recurrence (weekday toggles, from/to hours that can wrap past midnight, and an IANA timezone) and the same rule fragments as search rules — boosts, pins, redirects, banners, sponsored placements and an exclusion filter.

Only active and scheduled campaigns run. Creating, editing, pausing, activating, ending and deleting require an owner or admin.

Quality lab

/dashboard/quality is the offline relevance gate: golden query sets, human judgments, scored runs and a regression comparison. A Live tools card links straight to the eight screens this work touches — Search issues, the search and chat playgrounds, Analytics, Experiments, Search rules, Synonym suggestions and the knowledge graph.

• Sets — a name, collection and description, with the number of queries, the number of judgments, the alert thresholds and the last run's status and nDCG@k.
• Queries and judgments — add queries one per line (with optional query_by / filter_by), or import a query,doc_id,grade CSV, or import judgments from real clicks over a chosen number of days (click = 1, add-to-cart = 2, purchase = 3). Grades run 0–3 (not relevant, marginal, relevant, perfect) and each carries a source badge. Human judgments are never overwritten by clicks. Queries and individual judgments can be deleted.
• Runs — evaluate with a chosen k (default 10), a label, an optional baseline and an optional preset. A run reports nDCG@k, MRR, P@k, R@k, MAP and hit rate, and opens a per-query result with the top hits; runs poll while in progress and can be deleted.
• Compare — pick a run and a baseline: the screen flags a regression against the drop threshold (default 0.05), reports the metric deltas and lists the queries that regressed, improved, were added or removed.
• Thresholds — "alert when mean nDCG@k is below X" and "alert when the drop versus baseline exceeds Y". Alerts are delivered to webhooks subscribed to quality.regression; leaving a field empty disables that alert.
• A 30-day snapshot card has its own collection picker and Refresh snapshot, shows CTR, searches and the latest zero-result rate, lists top and zero-hit queries with Debug links, and links on to Full analytics and Search issues.

Autopilot

/dashboard/autopilot proposes merchandising changes from real behaviour and lets an owner or admin accept or reject them. Suggestions arrive in kinds — synonym, pin, redirect, boost, curation, experiment, exclusion and banner — each with a summary, status, collection, score, timestamps, the query it came from and its evidence payload. The list filters by status (pending, accepted, rejected or all) and by collection, shows a suggestion count and the last run's draft count, and has a Refresh.

• Run now generates a fresh batch from the last 30 days of search and click events. Pins, redirects and boosts need at least 20 searches on the query; synonyms need at least three zero-result searches.
• Boost, pin and exclusion suggestions can Preview quality impact against the quality lab; when the preview regresses and a decision conflicts with it, the console asks for explicit confirmation before forcing it through. Kinds that cannot be checked say why — no golden set for the collection, no graded queries, or no ranking preview for that kind.
• Accepting is gated by owner/admin rights, and a synonym suggestion may ask you to supply the terms.

Observability

Three screens read the query log. Unless an organization raises QUERY_LOG_SAMPLE_RATE, the log is a sample — requests made with debug=true are always recorded, so the search playground is the reliable way to capture one.

• Query log (/dashboard/observability) — filter by collection, query text, session, window (1 hour to all time), zero results only and errors only; filters round-trip through the URL. The table is paged 50 rows at a time with Load more, and shows time, collection, query, found, status, latency, applied features (experiment variant, rewrite, fallback, trace) and a session id that links to that session's view. A row opens the full query with its engine parameters, top hits, stored trace (or a "no stored trace" note) and a Replay now action that takes an optional JSON overrides field and renders a diff of the recorded against the current hit list. A counterfactual replay card runs up to 50 logged queries with JSON parameter overrides and reports queries, changed, failed, mean Jaccard, rank correlation, mean found before and after, and how many queries gained or lost hits, plus a per-query table (query, collection, recorded → current found, added, removed, moved, Jaccard, identical / changed / error).
• Sessions (/dashboard/observability/sessions) — look up a session by id, or browse the session index over a 24-hour / 7-day / 30-day window with Load more paging (session, queries, zero results, errors, started, last activity, state). Opening a session writes ?sessionId= into the URL, so a session is shareable; its summary reports queries, zero results, errors and the distinct collections touched, and every query opens the same drawer. The empty states explain QUERY_LOG_SAMPLE_RATE sampling and that the widget sends the session id automatically.
• Latency (/dashboard/observability/metrics) — end-to-end and engine latency percentiles (p50 / p90 / p95 / p99 / mean / max), query count, error rate, zero-result share and a throughput table, over a 1-hour to 30-day window, optionally narrowed to one collection.

Alerts and SLOs

/dashboard/alerts combines metric alerts with service-level objectives. Owner or admin rights are needed to create rules, edit SLO targets, evaluate or delete.

Alert rules watch a metric — zero-result rate, error rate, p95 latency, searches, CTR, no-click rate or revenue — over a collection or all, with a condition of above, below or anomaly (a z-score deviation), a threshold (or a sensitivity of 1–10 for anomalies), an evaluation window (15 minutes to 24 hours) and a cooldown (15 minutes to 7 days). A fired alert can be delivered to a webhook as alert.fired; the email field is present but the console notes that no mail provider is configured. The rules table lists the name (and collection), metric, condition, window and last-fired time, with an Enabled toggle — the only way to pause a rule, disabled for non-admins — and delete. Evaluate now renders a "Last evaluation" table (rule, outcome badge, value, expected, message) and a notice of how many rules were evaluated and how many fired.

SLOs cover availability, p95 latency and zero-result rate over a target window (1–90 days, default 30), showing met / missed / no-data, the actual value, the error budget left, the burn rate and the bad requests used. The targets form takes an availability percentage (50–100), a p95 latency in milliseconds, a zero-result fraction (0–1) and the window. Availability means searches answered without a search error; the p95 objective tolerates 5% over target.

The alerts list can be filtered by type and to show resolved alerts, and each row shows a severity badge (info / warning / critical) and can be acknowledged or resolved; when a webhook delivery of an alert email failed, the row says so inline. Both rules and alerts are fetched in a window of up to 200 rows. Platform operators also see a Search cluster dependencies section on this screen.

Next

• Publishing and support
• Quality, analytics and alerts in the API reference