Skip to main content
Introducing packages.sweber.dev
Cosine overview

CosineLive demo, free and Pro.

Everything below runs in your browser: the index of the Cosine and Surjection docs, keyword search, and, once you load it, the embedding model that searches by meaning. There is no search server.

Your queries are not sent anywhere. Loading the model downloads about 23 MB from Hugging Face once; your browser caches it.

Insights

Search insights (Pro)What your docs are missing.

@weber-development/cosine-insights records which searches find nothing and which results nobody opens, without cookies, ids or IP addresses, and turns them into a report. The example below was made from a month of made-up searches on a docs site.

npx cosine-insights report --log data/searches.ndjson --out insights.html

Free 2.0

Bigger sites, more languages, fewer files to worry aboutCosine, free.

Cosine 2.0 makes the MIT package ready for very large docs: a smaller index format, shards, and 4-bit vectors. It also finds words in Chinese, Japanese and Thai, and ships integrations for Astro (Starlight), Docusaurus and VitePress.

Shards for large sites

Split the index into several files that the browser loads in parallel. Results rank exactly like one file, and unchanged shards keep their name so your CDN can cache them.

Build with shards

cosine build docs --out public/cosine --lexical-only --shard-size 10

Real output (13 pages, lexical build) and the manifest that lists the shards

Indexed 13 pages as 25 chunks in 0.0 s (lexical only)
  public/cosine/cosine-index.json
  public/cosine/cosine-shard-0.json
  public/cosine/cosine-shard-1.json
  public/cosine/cosine-shard-2.json

{"version":2,"model":null,"dimensions":0,"createdAt":"...","vectors":null,"shards":[{"file":"cosine-shard-0.json","chunks":10},{"file":"cosine-shard-1.json","chunks":10},{"file":"cosine-shard-2.json","chunks":5}]}

Search in Chinese, Japanese and Thai

Cosine 2 splits scripts without spaces into words with Intl.Segmenter, so a search for a short word finds longer phrases and highlights it.

Search the shard index

cosine search public/cosine 東京 --mode lexical --limit 1

Real output

1. Tokyo  (lexical)
   /tokyo
   東京 東京タワーは東京の有名な観光地です。

4-bit vectors

Store each vector value in 4 bits. The vector file is half the size, at a small cost in ranking quality.

Build with smaller vectors

cosine build docs --shard-size 2000 --vector-bits 4

Docusaurus, Astro and VitePress integrations

Add the plugin and the index is built with your site. The same package exports /astro and /vitepress.

docusaurus.config.js

plugins: [["@sweberdev/cosine/docusaurus", { model: "english" }]],

Real output comes from @sweberdev/cosine 2.0.0 from npm, using a keyword-only build. The 4-bit vectors and the integrations are shown from the docs.

Read the docs

Pro 2.1.0

Search for every docs framework and languageCosine Pro.

Pro 2.1.0 adds ready-made search for Docusaurus, VitePress and Astro Starlight, and a separate index and model for each language of your site. Visitors only download the model of the language they are reading.

One index per language

Give each language its own model, for example the small English model for English pages and the multilingual one for German. The language of a page comes from front matter, the file name or the folder.

build.ts

import { readDocs } from "@sweberdev/cosine/node";
import { buildLanguageIndexes, writeLanguageIndexes } from "@weber-development/cosine-i18n";

const docs = await readDocs("docs", { baseUrl: "/docs" });
const build = await buildLanguageIndexes(docs, {
  languages: { en: "english", de: "multilingual", fr: "multilingual" },
});
await writeLanguageIndexes(build, "public/cosine", { shardSize: 500 });

Docusaurus pluginBeta

Replaces the navbar search with Cosine and builds the index of every locale after the build, with an embedding cache so rebuilds only embed changed sections.

docusaurus.config.js

themes: [
  [
    "@weber-development/cosine-docusaurus",
    { models: { en: "english", de: "multilingual" } },
  ],
],
themeConfig: {
  navbar: {
    items: [{ type: "search", position: "right" }],
  },
},

VitePress search fieldBeta

Wrap your config and use the theme: the search field replaces the built-in search, the index is built in dev and in build, and VitePress locales map to languages.

.vitepress/config.ts

import { defineConfig } from "vitepress";
import { withCosine } from "@weber-development/cosine-vitepress";

export default withCosine(defineConfig({ title: "My docs", base: "/docs/" }), {
  model: "english",
});

Astro and StarlightBeta

A Starlight plugin swaps the Pagefind search for Cosine, follows the light and dark theme and picks the index of the page language.

astro.config.mjs

import starlight from "@astrojs/starlight";
import cosineStarlight from "@weber-development/cosine-astro/starlight";
import { defineConfig } from "astro/config";

export default defineConfig({
  integrations: [
    starlight({
      title: "Docs",
      plugins: [cosineStarlight()],
    }),
  ],
});

This demo cannot show the framework plugins: they need your own Docusaurus, VitePress or Astro site. The search field of the plugins was not yet tried in a real browser, hence Beta.

Read the docs

Next

Add it to your docsTwo commands and one tag.

Build the index with npx cosine build docs --out public/cosine, add <cosine-search index="/cosine/cosine-index.json"> to your layout, done. Cosine Pro builds the index in every Vite build and adds search insights.

Getting started