@grundlag/provider-sources (0.4.0)

Published 2026-08-22 08:53:53 +02:00 by morten-olsen in incubator/grundlag

Setup this registry in your project .npmrc file:

@grundlag:registry=

To install the package using npm, run the following command:

npm install @grundlag/provider-sources@0.4.0

or add it to the package.json file:

"@grundlag/provider-sources": "0.4.0"

For more information on the npm registry, see the documentation.

@grundlag/provider-sources

A library of source documents — articles, files, pasted text — with hybrid search across them. The storage layer behind a read-it-later service, a research notebook, or anything that needs to answer questions from documents it holds.

It holds content; it does not write it. Summaries, cleaned-up titles and metadata are fields you fill in, typically with a model's help. Nothing here generates them.

Install

npm install @grundlag/provider-sources

Register it

import { ProviderRegistry, Services } from '@grundlag/core';
import { sourcesProvider } from '@grundlag/provider-sources';

const providers = services.get(ProviderRegistry);

await providers.register(sourcesProvider, {
  // Omit `embedding` entirely for keyword-only search — a supported setup.
  embedding: {
    model: 'text-embedding-3-small',
    apiKey: process.env.OPENAI_API_KEY,
    // Any OpenAI-compatible /embeddings endpoint: Ollama, llama.cpp, vLLM.
    baseUrl: process.env.EMBEDDINGS_URL,
  },
});

It owns its data and needs no credentials, so it can always be registered.

It needs a database with vector support, which is what a creator is expected to provide — createPgliteCreator includes it, and a hosted Postgres needs pgvector installed. Without it the provider refuses to start rather than coming up half-working. Leaving embedding unset still skips embedding entirely and searches by keyword alone; the column just exists unused.

Two ways to add a source

Supply the content yourself. Nothing is fetched, and a body written this way is never replaced by the pipeline:

await createSource({
  title: 'Fastify plugin scopes',
  markdown: '# Encapsulation\n\nA decorator added inside a plugin is not visible outside it.',
  summary: 'Write this yourself; nothing generates it.',
  tags: ['fastify'],
  collectionId: webReading.id,
});

Or hand over a URL. It is fetched, the readable article is extracted and converted to markdown, then chunked and embedded:

const { source } = await ingestUrl({ url: 'https://example.com/post' });
// source.status === 'pending' — the fetch has not happened yet.

Ingestion runs in the background, because a fetch plus an embedding round-trip is far too slow to hold a call open. Wait on the sourceReady event, or poll getSource. Re-ingesting a URL already stored refreshes it in place rather than adding a duplicate, and a re-fetch whose content is unchanged does no work at all.

A URL that turns out to be a PDF or an image is stored as a file with no body: extracting their text is a different problem, and extraction: 'none' keeps that visible instead of looking like a failure. Write the body in yourself with updateSource when you have it.

Searching

const { hits } = await searchSources({ query: 'serializer compiler', limit: 5 });

for (const hit of hits) {
  hit.chunk.text; //     the passage
  hit.chunk.headingPath; // 'Setup > Validation'
  hit.source.title; //   where to cite it from
  hit.matchedOn; //      ['lexical', 'semantic'] — which signals agreed
}

Results are passages, not whole documents, each carrying the source it came from so an answer built on them can cite it.

mode selects the legs: hybrid (default) fuses keyword and vector matches by reciprocal rank, falling back to keyword alone when no embedder is configured; lexical is keyword only; semantic needs an embedder and fails without one. Scope with collectionId, tag, or sourceId.

Scores are only meaningful for ranking within one result set — bm25 and cosine distance are on incomparable scales, so ranks are fused rather than scores. Never compare a score between searches.

The surface

Action	Does
`createSource`	Stores content you already have. Never fetches.
`ingestUrl`	Adds a URL and fetches it in the background.
`updateSource`	Writes a summary, corrects a body, retags, recollects.
`getSource`	Metadata and pipeline status, without the body.
`getSourceContent`	One body, in the `format` you name.
`getSourceChunks`	The passages a source was split into.
`getSourceOriginal`	A stored file's bytes, base64-encoded.
`searchSources`	Hybrid retrieval, returning citing passages.
`deleteSource`	Removes a source and everything derived from it.
`reprocessSource`	Forces a re-fetch and re-extract.
`reembedSource`	Rebuilds vectors with the current model, text untouched.
`createCollection` / `updateCollection` / `deleteCollection` / `listCollections`	Grouping, for scoping searches.
`listTags`	Every tag in use, with counts.

Sources are also exposed as a source entity, so they can be browsed and filtered through the entity surface (and its generated search/get tools).

Events: sourceCreated, sourceReady, sourceFailed.

Things worth knowing

Retrieved passages are not addressable. Chunk ids are regenerated whenever a body changes, so there is deliberately no passage entity. Store a citation as (sourceId, startOffset, endOffset, quote), not a chunk id.
Vectors record their model. Two models are not comparable, so re-embedding is additive and search filters by model. A requested dimensions is folded into the recorded model id, because a truncated vector must not share an index with a full-length one.
The originals bucket is private. An exposed bucket is readable and writable by anyone who can reach the host, and these are whole user documents. Bytes come out through getSourceOriginal, capped at 8 MiB.
URLs are filtered before fetching. Non-http schemes and loopback or private-range addresses are refused — a server-side fetch of http://169.254.169.254/ is the classic way to read cloud credentials out of a host. This filters what the caller asked for; no DNS resolution happens, so anywhere it matters wants an egress proxy or an allowlist.
Search input goes straight to the database. websearch_to_tsquery parses a query the way a search box does, so a dangling OR or an unbalanced quote is handled rather than raised as a syntax error — no sanitising step in between.
Status is a resumption point. A source interrupted mid-pipeline is re-queued on startup, and every step skips when its output is already current.
Similarity search is exact, not approximate. pgvector fixes a dimension in the column type, so indexing would pin the library to a single embedding model for good. Scanning keeps several models possible and returns true nearest neighbours, at a cost linear in the vectors held for the model being searched — the trade to revisit past a few hundred thousand chunks.

Not in scope

Generating summaries, answering questions, and holding a conversation. This package stores documents and finds the relevant parts; what you do with them belongs to the agent layer. Retrieval returns enough to cite.

Also absent today: PDF text extraction, per-user read state, and highlights.

Dependencies

ID	Version
@grundlag/core	0.4.0
@mozilla/readability	^0.6.0
kysely	^0.29.4
linkedom	^0.18.13
openai	^6.48.0
turndown	^7.2.4
zod	^4.4.3

Development dependencies

ID	Version
@electric-sql/pglite	^0.5.5
@grundlag/config	0.4.0
@types/node	^26.1.1
@types/turndown	^5.0.6
typescript	^6.0.3
vitest	^4.1.10

Details

npm

incubator/grundlag

2026-08-22 08:53:53 +02:00

MIT

134 KiB

Assets (1)

provider-sources-0.4.0.tgz 134 KiB

Versions (2) View all

0.5.0

2026-08-22

0.4.0