LLM Intent Preservation

Does LLM-generated code "remember" the prompt that created it? This repo finds out by generating code from prompts, then asking models to reconstruct the original prompt from the code alone — and measuring how close they get.

Three experiments test different angles: direct reconstruction from code, degradation under repeated refactoring, and whether converting code to documentation helps or hurts.

Quick Start

All scripts use uv with PEP 723 inline dependencies — no virtual environment setup needed.

cp .env.example .env
# Edit .env with your OpenRouter API key

# Run the main experiment (generates results.json)
uv run experiment.py

# Run the rewrite chain experiment (reads results.json)
uv run rewrite_experiment.py

# Run the documentation experiment (reads results.json)
uv run docs_experiment.py

# Regenerate all figures for the paper
uv run generate_figures.py

# Compile the paper
pdflatex report.tex

Experiments

1. Direct Reconstruction (experiment.py)

The core experiment. For each model and prompt:

• Generate code from the prompt (no comments, no docstrings)
• Ask the same model to guess the original prompt from the code
• Score the reconstruction against the original using cosine similarity

Also tests a "telephone" translation path (Python → Rust → Go → TypeScript).

uv run experiment.py                               # all defaults
uv run experiment.py --models openai/gpt-4o-mini   # single model
uv run experiment.py --runs 8                      # more reconstruction runs
uv run experiment.py --paths code_only             # skip telephone path

Output: results.json

2. Rewrite Chain (rewrite_experiment.py)

A "telephone game" for code refactoring. Takes the generated code from experiment 1 and rewrites it 4 times in succession, reconstructing the prompt at each hop to see if intent degrades.

hop 0: original code         → reconstruct → score
hop 1: rewrite(code)         → reconstruct → score
hop 2: rewrite(rewrite(code)) → reconstruct → score
...

uv run rewrite_experiment.py                  # defaults: 4 hops, 5 concurrent
uv run rewrite_experiment.py --hops 2         # fewer hops
uv run rewrite_experiment.py --concurrency 3  # limit parallel API calls

Output: rewrite_results.json

3. Documentation Path (docs_experiment.py)

Tests whether converting code to prose documentation preserves or loses intent. Gives the model both the original prompt and generated code, asks for comprehensive documentation, then scores both the docs and a prompt reconstructed from the docs.

uv run docs_experiment.py                     # defaults: 3 models, 4 runs
uv run docs_experiment.py --runs 8            # more reconstruction runs

Output: docs_results.json

Key Findings

Condition	Mean Similarity
Code → Docs → Reconstruct	0.858
Code → Reconstruct	0.829
Code → 4x Rewrite → Reconstruct	0.819
Code → Documentation (direct)	0.782

• Code preserves intent well — ~83% cosine similarity across 4 models
• Complex tasks preserve more detail — hard tasks retain 3x more specificity than easy ones
• Refactoring barely degrades intent — at most -0.039 after 4 rewrites; some models improve
• Documentation amplifies recovery — reconstruction from docs (0.858) beats code-only (0.829), even though docs as prose scores lower (0.782)
• Model choice matters less than prompt choice — 0.056 model spread vs 0.136 prompt spread

Models

All accessed via OpenRouter:

Model	Experiments
google/gemini-2.0-flash-001	1, 2, 3
openai/gpt-4o-mini	1, 2, 3
anthropic/claude-sonnet-4	1, 2, 3
anthropic/claude-opus-4	1, 2

Prompts

9 prompts across 3 difficulty tiers:

#	Difficulty	Task
P1	Easy	Palindrome checker
P2	Easy	Two-sum
P3	Easy	Temperature converter
P4	Medium	Line-of-code counter CLI
P5	Medium	Weather API client
P6	Medium	Apache log parser
P7	Hard	Async producer-consumer pipeline
P8	Hard	Rate limiter (token bucket + sliding window)
P9	Hard	Persistent undo/redo with rope data structure

Metric

Intent Preservation Score (IPS) = cosine similarity between sentence embeddings (all-MiniLM-L6-v2) of the original prompt and the reconstructed prompt.

Each reconstruction is scored against both a short "base intent" and the full detailed prompt. The gap between these measures how much specificity (the details beyond the gist) survives the round-trip.

Paper

The writeup is in report.tex — a deliberately over-the-top academic paper. Compile with pdflatex report.tex. Figures are pre-generated in figures/; regenerate with uv run generate_figures.py.

Repo Structure

.
├── experiment.py             # Experiment 1: direct reconstruction + telephone
├── rewrite_experiment.py     # Experiment 2: rewrite chain (4 hops)
├── docs_experiment.py        # Experiment 3: documentation path
├── generate_figures.py       # Generate all paper figures from result data
├── results.json              # Experiment 1 output
├── rewrite_results.json      # Experiment 2 output
├── docs_results.json         # Experiment 3 output
├── report.tex                # LaTeX paper
├── figures/                  # Paper figures (PDF + PNG)
├── .env.example              # API key template
└── README.md