Languages

@weber-development/cosine-i18n is for sites in several languages. The English model is about 23 MB, the multilingual one about 120 MB. With one index for everything, every visitor downloads the large model. With one index per language, English pages use the small model, German and French pages use the multilingual one, and a visitor downloads only the model of the language they are reading.

npm i -D @weber-development/cosine-i18n

With Vite

@weber-development/cosine-vite takes the same options, so for Vite, Astro and VitePress sites this is a setting:

import cosine from "@weber-development/cosine-vite";

export default defineConfig({
  plugins: [
    cosine({
      docs: "docs",
      baseUrl: "/docs",
      languages: { en: "english", de: "multilingual", fr: "multilingual" },
    }),
  ],
});

The build writes cosine/cosine-languages.json and a folder per language (cosine/de/cosine-index.json). virtual:cosine exports languagesUrl next to indexUrl.

In the browser

<cosine-search></cosine-search>
<script type="module">
  import "@sweberdev/cosine/element";
  import { mountLanguageSearch } from "@weber-development/cosine-i18n/client";
  mountLanguageSearch(document.querySelector("cosine-search"), "/cosine/cosine-languages.json");
</script>

mountLanguageSearch reads <html lang>. It picks the exact language (pt-BR), then the base language (de for de-CH), then the default, and sets index and lang on the element. Pass a language as the third argument to override <html lang>. loadLanguageIndex and languageIndexUrl return the URL if you build the field yourself.

How a page gets its language

In this order:

1. lang: (or language:, locale:) in the front matter
2. a language suffix in the file name: intro.de.md
3. a language folder in the file id or the URL: de/intro.md, /de/intro
4. your own detect function (detectLanguage in the Vite plugin), which runs before these three rules
5. otherwise the default language: defaultLanguage, by default the first one

Only the languages you list can match, so a folder called deploy is not read as German.

Without a build tool

import { readDocs } from "@sweberdev/cosine/node";
import { buildLanguageIndexes, writeLanguageIndexes } from "@weber-development/cosine-i18n";

const build = await buildLanguageIndexes(await readDocs("docs", { baseUrl: "/docs" }), {
  languages: { en: "english", de: "multilingual" },
});
await writeLanguageIndexes(build, "public/cosine", { shardSize: 500 });

languageFiles(build) returns the files as { name, data } for your own build step.

Per language options

A language can be keyword-only (de: false), use another Hugging Face model (de: "<model id>") or bring its own embedder and synonyms: de: { embedder, synonyms: [["Rechnung", "Faktura"]] }. Shared synonyms, boost, chunking options and shardSize apply to every language.

Limits

• The presets are the two that ship with Cosine, english and multilingual. Other models work through their Hugging Face id, but they are untested.
• A visitor reading a language that has no index gets the default language's index.
• Queries cannot search across languages in one field. For that, use one multilingual index or give the field several indexes (index="/cosine/en/cosine-index.json /cosine/de/cosine-index.json"), which loads both models.