Semantic, image and voice search

Next to keyword search, a collection can match by meaning. Documents carry float[] vector fields (num_dim: 1024); queries are embedded with the same multilingual model (@cf/baai/bge-m3), so a text query in any language, a photo and a product description all land in one vector space.

Every vector search shares the same options:

Search by meaning

searchSemantic (POST /collections/{collectionName}/documents/search/semantic) embeds the query text and returns the nearest documents by vector_field. Works in any language the embedding model understands — a Russian query finds an English description.

const found = await client.media.searchSemantic({
  collectionName: "products",
  q: "тёплая куртка на зиму",
  vector_field: "embedding",
  filter_by: "in_stock:true",
  k: 50,
  distance_threshold: 0.35,
});

Documents need vectors first. Fill the float[] field at import time with your own embeddings, or let the platform produce them:

• createEmbeddings — embed text yourself and write it into documents.
• uploadMedia — upload an image; its generated description is embedded into the document you point it at.
• YML feeds and connectors can carry embedding-capable fields — see YML feeds and connectors.

Search by image

searchByImage finds documents whose vector_field is closest to a given image. Pass one of media_id (an image uploaded with uploadMedia), image_url (a public HTTPS URL, fetched server-side, up to 8 MB) or image_base64 (inline bytes, with or without a data-URI prefix). The image is embedded the same way uploads are, so results are consistent with what a photo search on your storefront sees.

const similar = await client.media.searchByImage({
  collectionName: "products",
  media_id: "med_123",
  vector_field: "embedding",
  k: 30,
});

Embed text

createEmbeddings turns text into 1024-dimensional multilingual vectors — the same model embeds queries and image descriptions, so anything you embed with it is searchable by searchSemantic and searchByImage. Each input is up to about 500 tokens. Embedding calls are metered in credits.

const { embeddings } = await client.media.createEmbeddings({
  input: ["down winter jacket with a hood"],
});

The media library

uploadMedia stores an image (PNG, JPEG, WebP or GIF, up to 8 MB) and computes its embedding in the background: a vision model describes the image, and the description is embedded with the same multilingual text model. Target a document with collection, document_id and vector_field; optionally capture the generated description in a caption_field.

await client.media.uploadMedia({
  body: {
    file: imageFile, // PNG, JPEG, WebP or GIF, ≤ 8 MB
    collection: "products",
    document_id: "sku-4821",
    vector_field: "embedding",
    caption_field: "image_caption",
  },
});

For large files, upload in two steps: createMediaUpload registers the asset, then putMediaContent streams the raw bytes (up to 100 MB, Content-Length required). The embedding job starts once the bytes arrive; 409 means the asset already has content.

Voice search

createTranscription turns a short recording into text — webm/opus, mp4/m4a, ogg, wav, mp3 or flac, up to 25 MB, no WAV conversion needed; the language is detected when not given (language takes an ISO 639-1 code). searchByVoice transcribes and searches in one call: pass any search parameters (query_by, filter_by, nl_query) as a JSON string in the params field, and read the transcript from request_params.voice_query.transcribed_query in the response.

const result = await client.voice.searchByVoice({
  collectionName: "products",
  body: {
    file: audioBlob,
    language: "ru",
    params: JSON.stringify({ query_by: "title,description", filter_by: "in_stock:true" }),
  },
});

The storefront package and the widget already ship microphone buttons wired to these endpoints — use them instead of calling voice search by hand from a browser.