editor-testbed

Evaluation bench for the news-reader focus-scoring algorithm: users add RSS sources, define personal topics ("focuses"), and an algorithm scores every (article, focus) pair — fast enough for self-hosted hardware, improving in real time from votes. This repo measures which techniques earn their compute.

Read first: HYPOTHESES.md — the experimental charter (H0–H13 + pitfalls). Every command below exists to test something in it. PROGRESS.md tracks status and findings. Both render in the dashboard alongside the results.

Setup

mise install        # pinned node/task toolchain
task install        # pnpm install (via corepack)
task check && task lint && task test

External dependency: an OpenAI-compatible LLM server (llama-swap) for the labeling oracle and server-side embeddings — configured in testbed.config.json (oracle, embeddings.models). Local embedding models run via transformers.js on CPU, no server needed.

The workflow

Everything is a CLI subcommand (pnpm tsx src/cli.ts <cmd>), incremental and idempotent — re-running only does missing work. task daily chains the whole pipeline (cron runs it at 07:00):

fetch → label → embed → eval → analyze → simulate

Command	What it does	Hypotheses
fetch	Pull feeds → extract to markdown → QA flags → MinHash dedup	corpus (P1/P2/P8)
label	LLM oracle grades every (article × focus) pair 0–3 + article quality + tags. --source <lane> for parallel oracle experiments, --reasoning false for fast mode	H0
spotcheck	Blind human grading of a stratified sample (terminal UI)	H0
agreement --a oracle --b human	Cohen's κ between any two label sources — gate: κ ≥ 0.7	H0
embed	Store article vectors for every configured embedding model, with ingest ms/article	H1/H8
eval	Score all pairs with all scorers, persist per-pair, report nDCG@20/P@10/AUC per focus × language	H1/H2/H6/H7/H8
analyze	Per-focus best scorer, ceilings, disagreement matrix, calibration (P7)	H3
analyze --fit "a,b,c"	Leave-one-focus-out hybrid weight fitting	H3
expand	Generate + store one-time LLM focus expansions	H7
quality	Cross-validated quality probe on embeddings	H11
simulate	Vote-feedback curves (5 mechanisms), overlap-downvote scenario, motivation routing. --gamma, --votes	H4/H5/H5b/H11/H12
cascade --base <scorer> --reranker nli|llm:<model> --k 5,10,20	Cheap-rank + expensive-rerank sweep. ⚠️ H9 ceiling must use a model ≠ the labeling oracle	H9/H10
snapshot --from <iso>	Frozen, checksummed JSONL corpus export	reproducibility
stats	Corpus summary	—

Scorer names for eval --scorers / analyze --fit / cascade --base: bm25, feed-prior, emb-raw:<model>, emb-centered:<model>, emb-feedcentered:<model>, hybrid:<id> (from config), plus H7 query variants ...@title / ...@expanded and bm25@title / bm25@expanded. Default = everything configured.

Adding things (all incremental)

• Articles: next fetch — only new (feed, guid) items are ingested/labeled/embedded.
• A focus: append to testbed.config.json → next label grades only the new pairs. Focus meaning is immutable once labeled — changed meaning = new id.
• An embedding model: append to embeddings.models (id, provider server/local, task prefixes, pooling, dtype, H8 text variant) → embed.
• A hybrid: append to hybrids (components + weights; scores are z-normed per focus).
• A new technique: implement the 4-field Scorer interface, one registry line in src/scorers/scorers.ts. Exhaustive labeling means no pooling bias against it.

The dashboard

task dash          # hot-reload preview
task dash:build    # static site (what CI publishes)

Report-style pages (charter → progress → corpus → labels → scoring → complementarity → feedback), built from the committed data/corpus.db — CI publishes without recomputation. After writing fresh data, checkpoint the WAL (sqlite3 data/corpus.db "PRAGMA wal_checkpoint(TRUNCATE);") or the published dashboard lags.

Data model (SQLite, data/corpus.db, committed)

articles (+ QA flags, dedup clusters) · labels (article × focus × source) · quality_labels · article_tags · focus_expansions · embeddings (per model) · embed_stats (ingest cost) · eval_runs + scores (per-pair, per run, corpus-hashed) · sim_runs · cascade_runs.

Conventions, gotchas, and agent guidance: CLAUDE.md.