Yandex Market YML feeds

A Yandex Market Language (YML) feed is the <yml_catalog><shop>…<offers> XML most Russian storefronts already export. AACSearch turns every <offer> into a document and keeps the category tree, so a feed URL is all a first product search needs.

What an offer becomes

If the target collection does not exist, the job creates it with exactly these fields (and enable_nested_fields for params). An existing collection is used as is — its schema must accept the fields above.

By URL

const job = await client.ingest.createIngestJob({
  collection: "products",
  source: "yml",
  url: "https://shop.example.ru/export/yandex.yml",
});

The feed may be plain XML or gzip (.yml.gz, or a server that compresses anyway) and declare any charset in its prolog — windows-1251 feeds work. max_pages does not apply to feeds: the whole catalog is imported.

By file

A feed file goes through uploadIngestFeed — a raw body streamed to storage, up to 500 MB (and the request body limit of your Workers plan):

aacsearch ingest file ./yandex.yml.gz --collection products --wait

The console's Import → File tab accepts .yml, .xml and .gz and does the same. A feed under 2 MB can also be sent inline as content with source: "yml".

Large feeds

Feeds of hundreds of megabytes and hundreds of thousands of offers are the normal case, so a yml job never holds the feed in memory:

• The feed is streamed — gunzipped and decoded as it arrives — and scanned for complete <offer> elements; only the current offer and the feed head (<categories>) are in memory.
• Offers are imported in batches of 500, and progress is saved after every batch.
• One queue invocation handles a pass of up to 20 000 offers (or four minutes), then hands the job back to the queue. The next pass re-reads the feed, fast-forwards over the offers already imported, and continues. stats.cursor is the resume point and stats.passes the number of passes so far; both are visible while the job is processing.
• A crash or engine outage retries from the last saved cursor. With the default upsert action a re-imported batch is harmless.
• A single offer larger than 2 MB is skipped with an error entry; a job stops after 2 000 000 offers.

A URL feed is downloaded once per pass, so a very large feed on a slow host is better uploaded (uploadIngestFeed) — passes then read it from storage inside Cloudflare. An uploaded file is deleted once the job is ready; a failed job keeps it so retryIngestJob can read it again.

Following a job

let job = await client.ingest.getIngestJob({ jobId: created.id });
while (job.status === "queued" || job.status === "processing") {
  await new Promise((r) => setTimeout(r, 2000));
  job = await client.ingest.getIngestJob({ jobId: created.id });
}
console.log(job.stats); // { fetched, imported, failed, errors, cursor, passes }

stats.errors keeps the first 20 problems (an offer without an id, without a name, a duplicate id, a document the engine rejected). A job is failed only when nothing at all was imported; a feed with a few bad offers is ready with failed > 0.