@grundlag/provider-sources (0.5.0)
Installation
@grundlag:registry=npm install @grundlag/provider-sources@0.5.0"@grundlag/provider-sources": "0.5.0"About this package
@grundlag/provider-sources
A library of source documents — articles, files, pasted text — with hybrid search across them. The storage layer behind a read-it-later service, a research notebook, or anything that needs to answer questions from documents it holds.
It holds content; it does not write it. Summaries, cleaned-up titles and metadata are fields you fill in, typically with a model's help. Nothing here generates them.
Install
npm install @grundlag/provider-sources
Register it
import { ProviderRegistry, Services } from '@grundlag/core';
import { sourcesProvider } from '@grundlag/provider-sources';
const providers = services.get(ProviderRegistry);
await providers.register(sourcesProvider, {
// Omit `embedding` entirely for keyword-only search — a supported setup.
embedding: {
model: 'text-embedding-3-small',
apiKey: process.env.OPENAI_API_KEY,
// Any OpenAI-compatible /embeddings endpoint: Ollama, llama.cpp, vLLM.
baseUrl: process.env.EMBEDDINGS_URL,
},
});
It owns its data and needs no credentials, so it can always be registered.
It needs a database with vector support, which is what a creator is expected to
provide — createPgliteCreator includes it, and a hosted Postgres needs pgvector
installed. Without it the provider refuses to start rather than coming up
half-working. Leaving embedding unset still skips embedding entirely and searches
by keyword alone; the column just exists unused.
Two ways to add a source
Supply the content yourself. Nothing is fetched, and a body written this way is never replaced by the pipeline:
await createSource({
title: 'Fastify plugin scopes',
markdown: '# Encapsulation\n\nA decorator added inside a plugin is not visible outside it.',
summary: 'Write this yourself; nothing generates it.',
tags: ['fastify'],
collectionId: webReading.id,
});
Or hand over a URL. It is fetched, the readable article is extracted and converted to markdown, then chunked and embedded:
const { source } = await ingestUrl({ url: 'https://example.com/post' });
// source.status === 'pending' — the fetch has not happened yet.
Ingestion runs in the background, because a fetch plus an embedding round-trip is
far too slow to hold a call open. Wait on the sourceReady event, or poll
getSource. Re-ingesting a URL already stored refreshes it in place rather than
adding a duplicate, and a re-fetch whose content is unchanged does no work at all.
A URL that turns out to be a PDF or an image is stored as a file with no body:
extracting their text is a different problem, and extraction: 'none' keeps that
visible instead of looking like a failure. Write the body in yourself with
updateSource when you have it.
Searching
const { hits } = await searchSources({ query: 'serializer compiler', limit: 5 });
for (const hit of hits) {
hit.chunk.text; // the passage
hit.chunk.headingPath; // 'Setup > Validation'
hit.source.title; // where to cite it from
hit.matchedOn; // ['lexical', 'semantic'] — which signals agreed
}
Results are passages, not whole documents, each carrying the source it came from so an answer built on them can cite it.
mode selects the legs: hybrid (default) fuses keyword and vector matches by
reciprocal rank, falling back to keyword alone when no embedder is configured;
lexical is keyword only; semantic needs an embedder and fails without one.
Scope with collectionId, tag, or sourceId.
Scores are only meaningful for ranking within one result set — bm25 and cosine distance are on incomparable scales, so ranks are fused rather than scores. Never compare a score between searches.
The surface
| Action | Does |
|---|---|
createSource |
Stores content you already have. Never fetches. |
ingestUrl |
Adds a URL and fetches it in the background. |
updateSource |
Writes a summary, corrects a body, retags, recollects. |
getSource |
Metadata and pipeline status, without the body. |
getSourceContent |
One body, in the format you name. |
getSourceChunks |
The passages a source was split into. |
getSourceOriginal |
A stored file's bytes, base64-encoded. |
searchSources |
Hybrid retrieval, returning citing passages. |
deleteSource |
Removes a source and everything derived from it. |
reprocessSource |
Forces a re-fetch and re-extract. |
reembedSource |
Rebuilds vectors with the current model, text untouched. |
createCollection / updateCollection / deleteCollection / listCollections |
Grouping, for scoping searches. |
listTags |
Every tag in use, with counts. |
Sources are also exposed as a source entity, so they can be browsed and
filtered through the entity surface (and its generated search/get tools).
Events: sourceCreated, sourceReady, sourceFailed.
Things worth knowing
- Retrieved passages are not addressable. Chunk ids are regenerated whenever a
body changes, so there is deliberately no
passageentity. Store a citation as(sourceId, startOffset, endOffset, quote), not a chunk id. - Vectors record their model. Two models are not comparable, so re-embedding is
additive and search filters by model. A requested
dimensionsis folded into the recorded model id, because a truncated vector must not share an index with a full-length one. - The originals bucket is private. An exposed bucket is readable and writable
by anyone who can reach the host, and these are whole user documents. Bytes come
out through
getSourceOriginal, capped at 8 MiB. - URLs are filtered before fetching. Non-http schemes and loopback or
private-range addresses are refused — a server-side fetch of
http://169.254.169.254/is the classic way to read cloud credentials out of a host. This filters what the caller asked for; no DNS resolution happens, so anywhere it matters wants an egress proxy or an allowlist. - Search input goes straight to the database.
websearch_to_tsqueryparses a query the way a search box does, so a danglingORor an unbalanced quote is handled rather than raised as a syntax error — no sanitising step in between. - Status is a resumption point. A source interrupted mid-pipeline is re-queued on startup, and every step skips when its output is already current.
- Similarity search is exact, not approximate. pgvector fixes a dimension in the column type, so indexing would pin the library to a single embedding model for good. Scanning keeps several models possible and returns true nearest neighbours, at a cost linear in the vectors held for the model being searched — the trade to revisit past a few hundred thousand chunks.
Not in scope
Generating summaries, answering questions, and holding a conversation. This package stores documents and finds the relevant parts; what you do with them belongs to the agent layer. Retrieval returns enough to cite.
Also absent today: PDF text extraction, per-user read state, and highlights.
Dependencies
Dependencies
| ID | Version |
|---|---|
| @grundlag/core | 0.5.0 |
| @mozilla/readability | ^0.6.0 |
| kysely | ^0.29.4 |
| linkedom | ^0.18.13 |
| openai | ^6.48.0 |
| turndown | ^7.2.4 |
| zod | ^4.4.3 |
Development dependencies
| ID | Version |
|---|---|
| @electric-sql/pglite | ^0.5.5 |
| @grundlag/config | 0.5.0 |
| @types/node | ^26.1.1 |
| @types/turndown | ^5.0.6 |
| typescript | ^6.0.3 |
| vitest | ^4.1.10 |