Ingest failures
Documents pushed to AACsearch but never appear in search — diagnose buffer queue lag, schema mismatch, validation errors, and reindex pressure.
Ingestion is decoupled from indexing: a write enters SearchIngestBuffer, a worker picks it up and applies it to the live index. A "failure" can mean a 4xx at the API edge, a 4xx at the worker, or a stuck queue. Each has a different fix.
Symptom
| What you see | Where the issue is |
|---|---|
4xx returned by sync/full, sync/delta, documents:batch | Edge validation failed — request never enqueued |
2xx returned but document does not appear after 2 minutes | Worker rejected it (schema, type) or queue lag |
| Some documents appear, others do not | Per-document validation; check the response errors[] field |
| Document count stuck after a job | Worker stalled or schema migration in progress |
Decision tree
Did the API return 2xx?
├─ no → fix the request body; see "Validation errors" below
└─ yes → check buffer queue depth
├─ depth 0 → worker has processed it; check schema match
└─ depth > 0 → wait; see "Queue lag" belowCheck 1: API response
A successful enqueue returns { "enqueued": <n>, "errors": [] }. A partial-success returns the per-document errors:
{
"enqueued": 998,
"errors": [
{
"external_id": "product-123",
"error": "validation_failed",
"message": "Field 'price' expected float, got string"
},
{
"external_id": "product-456",
"error": "missing_field",
"message": "Required field 'title' is missing"
}
]
}Fix each row according to its error code. The other 998 are already on their way to the index.
Common validation errors
| Error code | Meaning | Fix |
|---|---|---|
validation_failed | Field type does not match schema | Cast or coerce on the connector side |
missing_field | Required field absent | Add it; or mark it optional in the index schema |
payload_too_large | Single document > 1 MB | Split or trim large text fields |
batch_too_large | Batch > 1000 documents | Split into multiple batches |
unknown_field (warning, not failure) | Extra field not in schema | Either add to schema or strip on the connector side |
Check 2: buffer queue depth
If the API said enqueued: N but documents are not visible after 2 minutes, check the queue:
curl -X GET https://app.aacsearch.com/api/v1/projects/<projectId>/usage \
-H "Authorization: Bearer aa_admin_..."Response includes a bufferDepth for the org. Healthy depth is < 1000 most of the time. Sustained depth > 50,000 indicates the worker is throttled.
| Depth | Meaning |
|---|---|
| 0 | Worker idle, all caught up |
| 1–10,000 | Normal — typical lag < 30 s |
| 10,000–50,000 | Catching up after a burst — typical lag < 5 min |
| > 50,000 | Worker throttled — see "Pressure" below |
Check 3: schema mismatch
A document can pass edge validation but fail at the worker if a field type changed since the index schema was set. Worker rejections are surfaced in the dashboard:
Search → Indexes → the index → "Recent ingest errors" tab.
Common causes:
- The connector started sending a new field that has no facet/type definition.
- A field changed from
stringtostring[]upstream. - A nested object got flattened differently than the schema expects.
Update the schema (admin API → PATCH /v1/indexes/<id>) and re-push the affected documents.
Check 4: reindex in progress
During a full reindex, the live alias is swapped at the end. While reindex is running, new writes are written to both the old and new collection. Writes are not lost, but you may see a temporary lag.
Check Search → Indexes → status. If status is reindexing, wait for it to complete. See reindexing and zero-downtime for the full lifecycle.
Check 5: connector specifics
If you are using a connector (PrestaShop, Bitrix, Postgres-sync), check the connector's last sync job:
curl -X GET https://app.aacsearch.com/api/v1/projects/<projectId>/sync-jobs \
-H "Authorization: Bearer aa_admin_..."Each row has a status: pending, running, completed, failed. Failed jobs include the underlying error and a sample document that failed.
Pressure: worker throttled
If bufferDepth is climbing into the hundreds of thousands and not draining, one of these is happening:
| Cause | Fix |
|---|---|
| Plan-level concurrency cap reached | Upgrade plan or contact support to raise the per-org worker count |
| Upstream search engine slow | Check status page; contact support if our side is healthy |
| Large reindex consuming worker | Wait for reindex to finish; consider scheduling reindexes for off-peak |
| Many small batches (1 doc each) | Switch to documents:batch with 100–1000 per request |
Fix matrix
| Diagnosis | Fix |
|---|---|
| Validation 4xx at edge | Fix the request body using the per-row errors[] |
| Schema mismatch | Update index schema, re-push affected docs |
| Queue lag < 5 min | Wait |
| Sustained queue lag | Switch to bigger batches; contact support if persistent |
| Reindex in progress | Wait for swap |
| Connector job failed | Inspect job; fix the connector or the source data |
Diagnostics packet
| Field | Notes |
|---|---|
| Organization ID | required |
| Index slug | required |
| Job ID | from the sync response or sync-jobs list |
bufferDepth snapshot | required |
| Sample failed document | full payload + the error/message from the response |
| When it started | UTC timestamp |
| Connector type | if applicable (PrestaShop, Bitrix, etc.) |
Document IDs are deterministic from external_id. If you re-push the same external_id, it overwrites — no need to delete first.
Related
- Connector API lifecycle — endpoint reference
- Connector sync — connector-specific issues
- Reindexing and zero-downtime
- Auth errors — for 401/403 on ingest endpoints
Widget not loading
Widget script tag is on the page but the search UI never appears — diagnose script load failure, missing container, CSP blocks, key issues, and Shadow DOM mounting.
Connector sync
Connector marked offline, heartbeat failing, sync jobs stuck — diagnose connector tokens, network reachability, sync mode mismatch, and dashboard cache.