haiku.rag AGENTS.md changelog

https://raw.githubusercontent.com/ggozad/haiku.rag/HEAD/AGENTS.md

− │   ├── upgrades/           # Version migrations (v0_20_0 … v0_89_0)− - Hook points, run in configured order: `before_ingest`, `after_ingest`, `after_delete`, `before_search`, `after_search`, `lifespan`, `ingester_lifespan`. `before_ingest` refuses a document with `DocumentRejected`, and any other exception fails the ingest as `HookError`. Search hook failures propagate. `after_ingest` and `after_delete` run through `notify`, which logs a failure and never raises. A FULL rebuild passes `notify_hooks=False`.− - `QAConfig` - model, max_searches (default 5), max_executions (default 15). The budgets are per question. `max_searches` counts search *units*: searches emitted in one model response (same `RunContext.run_step`) share a unit, up to `FREE_SIBLINGS_PER_ROUND` (3) per unit, so a run searches at most `max_searches × 3` times and a sequential searcher pays one unit per search. A budget-rejected round fails its remaining siblings− `pytest` runs the whole core suite, slow and integration included. CI's fast lane is `pytest -m "not slow and not integration" --cov`. `test-windows` runs it on Windows without `--cov`, as a check that is not required. `cd evaluations && uv run pytest --cov` runs the evaluations suite and its coverage gate. Tests run under xdist (`-n auto` in addopts); pass `-n0` when debugging races or container lifecycle. Async tests are detected through pytest-asyncio's auto mode, and asyncio fixtures default− - **rapidocr 3.9.2** ships LaTeX in a docstring that beartype's import hook compiles. `pyproject.toml` carries `"ignore:invalid escape sequence:SyntaxWarning"`.− - **Maintenance** (`ingester.maintenance`, off by default): `DatabaseMaintenance` per database, built in `_apply`, run as a task only under `serve` (`_maintaining`), once per written database in `run-batch` and manifest replay, after the drain and after `_maintain_written` exits every ingester lifespan with the clients still open. A pass holds claims with `WorkerPool.pause`, a set apart from `_served`, so `resume` in a `finally` never re-serves a source that a reload unserved. `close_client` can− - **Reaping is lease-based.** Workers renew `last_heartbeat_at` every `heartbeat_interval_s`. `reap_stale` resets a claim when `COALESCE(last_heartbeat_at, claimed_at)` is older than `lease_ttl_s`, and never cancels a live coroutine. `WorkerConfig` enforces `heartbeat_interval_s <= lease_ttl_s / 3`. `claim_timeout_s` is gone.− - **`POST /sources/{id}/refresh`** runs `_sweep_once`, which skips (`refreshed: false`) while the source has pending jobs or an open breaker. Default FS `poll_interval_s` is 300.− │   │   ├── doctor.py       # doctor and projection routes− │   │   └── projection.py   # t-SNE of document centroids, similarity, Retained− │   └── lib/                # tested logic: addresses, chat, sessions, ingestion, inspect, map, doctor, proxy− - **Doctor and projection jobs** run on one daemon thread with its own event loop, one at a time across databases, each over the `Store` it opens. Progress reaches the request loop through `call_soon_threadsafe`. A projection's unit centroids are kept within one 256 MB budget (`Retained`).− - **The frontend proxies.** `/api/backend/*` and `/api/ingester/*` stream bodies and forward the browser's abort. The backend proxy refuses `reload`, and the ingester proxy adds `INGESTER_TOKEN` and forwards only the operator actions as writes (refresh a source, retry a job or a dead letter, cancel a job). The dev server allows the `127.0.0.1` origin.− - **The address is the inspector's state.** `app/(inspector)/layout.tsx` renders `InspectPage` for `/inspect/…` and `/search`, which reads its view from the path and the query (`lib/addresses.ts`), so it stays mounted while the address changes. `DocumentView`, `ChunkView`, `CorpusMap` and `SearchPanel` take that state as props. The corpus map job's controls, families and memory estimate live in `components/doctor/MapJob.tsx`, shared by the doctor and the inspector's map. An image search's image + │   ├── upgrades/           # Version migrations (v0_75_0 … v0_91_0)+ - Hook points, run in configured order: `before_ingest`, `after_convert`, `after_ingest`, `after_delete`, `before_search`, `after_search`, `lifespan`, `ingester_lifespan`. `before_ingest` refuses a document with `DocumentRejected`, and any other exception fails the ingest as `HookError`. `after_convert` (`ConvertedDocument`) runs between haiku.rag's conversion and chunking wherever it converts: client/documents.py (source, `create_document`, `update_document(content=)`) and FULL rebuild's stored+ - `QAConfig` - model, max_searches (default 5), max_executions (default 15), max_images (default None). The budgets are per question. `max_searches` counts search *units*: searches emitted in one model response (same `RunContext.run_step`) share a unit, up to `FREE_SIBLINGS_PER_ROUND` (3) per unit, so a run searches at most `max_searches × 3` times and a sequential searcher pays one unit per search. A budget-rejected round fails its remaining siblings+ `pytest` runs the whole core suite, slow and integration included. CI's fast lane is `pytest -m "not slow and not integration" --cov`. `test-windows` runs it on Windows without `--cov`, as a check that is not required. `test-py314` runs it on Python 3.14 without `--cov`. `cd evaluations && uv run pytest --cov` runs the evaluations suite and its coverage gate. Tests run under xdist (`-n auto` in addopts); pass `-n0` when debugging races or container lifecycle. Async tests are detected through pyt+ - **One OpenCV build.** `rapidocr` requires `opencv-python`, the GUI build, which writes the same `cv2/` files as our `opencv-python-headless` and needs X11 libraries. The root `pyproject.toml` excludes it (`exclude-dependencies`), and `tests/test_dependencies.py` checks the lock. A pip install still gets both. Uninstalling either build deletes the shared files: `uv sync --reinstall-package opencv-python-headless` restores `cv2`.+ - **rapidocr and google-genai** ship invalid escapes in string literals that beartype's import hook compiles. `pyproject.toml` carries `"ignore:.*invalid escape sequence:SyntaxWarning"`: Python 3.14 starts the message with the escape.+ - **Maintenance** (`ingester.maintenance`, off by default): `DatabaseMaintenance` per database, built in `_apply`, run as a task only under `serve` (`_maintaining`), once per written database in `run-batch` and manifest replay, after the drain and after `_maintain_written` exits every ingester lifespan with the clients still open. A pass holds claims with `WorkerPool.pause`, a set apart from `_served`, so `resume` in a `finally` never re-serves a source that a reload unserved. A pass does not wa+ - **Reaping is lease-based.** Workers renew `last_heartbeat_at` every `heartbeat_interval_s`. `reap_stale` resets a claim when `COALESCE(last_heartbeat_at, claimed_at)` is older than `lease_ttl_s`, and never cancels a live coroutine. `WorkerConfig` enforces `heartbeat_interval_s <= lease_ttl_s / 3`. The reaper passes the pool's in-flight job ids as `reap_stale(keep=)` under `claim_lock`, which `_claim` holds while it claims and tracks, so a process never reaps its own live claims, whatever its c+ - **`POST /sources/{id}/refresh`** runs `_sweep_once`, which answers `refreshed: false` for an open breaker, a failed discovery or a stopped poller. `serve`'s sweeps and refresh run whatever is queued: `uq_jobs_live` deduplicates a queued file, and an UNCHANGED event writes no revision, so a sweep never overwrites one a worker recorded meanwhile. Only `sweep_all` (`run-batch`) passes `skip_if_pending=True`, since a batch sweeps once and an event deduplicated against a live job would never be see+ - **`qa.max_images`** bounds the number of images in one request. `RAGCapability.prepare_model_request` applies it through `limit_retrieved_images` (`_tools.py`). That hook runs after every `wrap_model_request` and `before_model_request`, so it sees the capsule pictures whatever the registration order, and it changes the request only. Every image counts, tool returns included, and only label-plus-picture pairs carrying `RETRIEVED_IMAGE_TAG` are dropped: search pictures outside the last message f+ │   │   ├── doctor.py       # doctor and similarity routes+ │   │   └── similarity.py   # document centroids compared, near-duplicate families, Retained+ │   └── lib/                # tested logic: addresses, chat, sessions, ingestion, inspect, doctor, proxy+ - **Doctor and similarity jobs** run on one daemon thread with its own event loop, one at a time across databases, each over the `Store` it opens. Progress reaches the request loop through `call_soon_threadsafe`. A similarity job's unit centroids are kept within one 256 MB budget (`Retained`), and answer the nearest-documents lookup.+ - **The frontend proxies.** `/api/backend/*` and `/api/ingester/*` stream bodies and forward the browser's abort. The backend proxy refuses `reload`, and the ingester proxy adds `INGESTER_TOKEN` and forwards only the operator actions as writes (refresh a source, retry a job or a dead letter, cancel a job, edit or delete a document). The inspector offers document edits only on an untagged database listed by the ingester's `/sources`, and shows the ingester's answer to a save while the document st+ - **The address is the inspector's state.** `app/(inspector)/layout.tsx` renders `InspectPage` for `/inspect/…` and `/search`, which reads its view from the path and the query (`lib/addresses.ts`), so it stays mounted while the address changes. `DocumentView`, `ChunkView` and `SearchPanel` take that state as props. The similarity job's controls and families live in `components/doctor/SimilarityJob.tsx`, and the memory estimate in `components/doctor/Estimate.tsx`, shared by the doctor and the ins

https://raw.githubusercontent.com/ggozad/haiku.rag/HEAD/AGENTS.md

− │   └── exceptions.py       # ReadOnlyError, MigrationRequiredError, AmbiguousDatabaseError, UnknownDatabaseError,− │                           # AmbiguousCitationError, SourceUnavailableError, ConfigMismatchError, TagError− │   ├── datasets/           # frames, hotpotqa, mtrag, open_rag_bench, t2_ragbench− │   └── evaluators/         # judge, map, citation, retrieval, number_match, refusal, conversation, transcript− - `create_capability()` → `CitationPolicyCapability` (capabilities/policy.py). Optional. Requires every answer to declare its grounding, asks once per question, records failures in `CitationPolicyState.violations`, never asks the model to change its answer. Enforced when the question retrieved evidence or the conversation already cited something.− - `IngesterConfig` - databases (database name → `IngestedDatabase`: sources), queue (QueueConfig), workers (WorkerConfig), api (APIConfig)− **Core coverage is enforced at 100%** (`fail_under = 100`, `source = ["haiku_rag_slim"]`); the evaluations workspace enforces 85%. New code needs a test or a `# pragma: no cover - <short reason>` on one line. CI prints term-missing, so a gate failure names the line. Run coverage with `--cov` and scope the report by grepping it: a deep-dotted `--cov=haiku.rag.<module>` crashes beartype, and a file-path `--cov=<path>.py` collects nothing.− - Every critical dependency carries an upper bound, and `pydantic-ai-slim` is capped below its next minor release. pydantic-ai changes capability hook semantics in minor releases: 2.54 runs `before_model_request` inside the `wrap_model_request` handler, and `EvidenceCompactionCapability` then compacts nothing. Raise a cap only in a change that runs every suite and, when capability behaviour can move, the evals.− - **Lock order** is `_rebuild_lock → _write_lock` (tag ops, restore) and `_vacuum_lock → _write_lock` (vacuum). Rebuild holds `_rebuild_lock` for its whole run, and tag operations fail fast against it. All in-process: cross-process writers need a quiescent boundary for tag create and restore.− - **Opening a Store never writes to it.** Stored embedding settings change only via `init`/`rebuild`, and the version is a forward-only marker written by `init` and `migrate`. `Store.stored_settings` and `Store.stored_embedding` are read at open and never follow a later write. Refresh them together through `_remember_settings`.− - **Index sets** live in `index_specs()` / `ensure_indexes()` (schema.py), and every table-creating path routes through them. `ensure_indexes` matches on (column, index type) and never drops an index it did not declare. It skips FTS on an empty table (an index born over zero rows never catches up on `add`) and rebuilds an FTS index covering zero rows. The chunk write paths call it after writing. So a test needing the zero-coverage state writes via `chunks_table.add` directly, and the first chunk− - **docling and docling-serve are pinned as a pair**: `docling==2.127.0` + `docling-core==2.96.1`, and `quay.io/docling-project/docling-serve:v1.33.0` in `tests/docker/` and `examples/docker/`. The coupling is `tests/chunkers/test_chunker.py::test_local_and_serve_chunkers_produce_same_output` (slow, vcr), which the `test-slow` CI job enforces. Take the client version from the serve release's bundled library set and move both together. Re-record the serve cassettes when the image moves: that test− - **`pdf_backend` defaults to `docling_parse`**, not docling's `threaded_docling_parse`, which never returns on some documents on Linux (all threads parked in `standard_pdf_pipeline.get_batch`). docling caches its `DocumentConverter` per process, so one wedge stalls later conversions on any backend: recovery needs a new process. Reproduce in the `docling-serve:v1.32.0` image (linux/arm64, root, `faulthandler.register(SIGUSR1, all_threads=True)` plus `kill -USR1`). Before attributing a conversion− - **`create_document_from_source` metadata merges** differ by path: full ingest does `{**user_metadata, **source_metadata}`, the revision short-circuit's `_refresh_doc_metadata` does `{**doc.metadata, **user_metadata}` before `source_metadata`. `_caller_metadata` strips `source_id` and `parent_uri` from caller metadata. `_provider_metadata` strips every `_RESERVED_METADATA_KEYS` entry (`content_type`, `md5`, `source_revision`, `source_id`, `parent_uri`) from provider output. `source_id` is the i− - **Reload is remove-then-add per database.** `IngesterApp.add_database` prepares (opens the client, builds adapters, writes no queue row) then applies (reconciles, then serves). `remove_database` unserves, stops the pollers, waits for in-flight jobs, deletes live jobs and `sync_state`, then closes: each step idempotent, and a database stays `removing` until the close lands. Claims, `unserve` and API retries hold `WorkerPool.claim_lock`, and a claim reads the served set inside it, so no claim na− - **Compaction and policy hooks.** `EvidenceCompactionCapability` discovers the evidence capability through `RunContext.capabilities` and rewrites `request_context.messages` in `wrap_model_request`, never the stored history, which keeps question identities and epochs (message counts) meaningful. A question's identity is the index of its opening request, taken at the question's first `before_model_request`: pydantic-ai normalizes the history after `for_run`, and AG-UI passes the question inside `− - **Windows.** Text mode follows the host locale (cp1252), so read and write repo files with `encoding="utf-8"`. Build an expected rendered path with `pathlib`, not a literal `/`. Write a file under test through `tmp_path`: Windows refuses to reopen an open `NamedTemporaryFile`. Creating a symlink needs Administrator or Developer Mode; without it the `requires_symlinks` tests (`tests/platform.py`) skip and the symlink branch of `sources/fs.py` goes uncovered. Guard a POSIX-only call with `sys.pl− Datasets: `frames`, `hotpotqa`, `orb_text`, `orb_multimodal`, `orb_multimodal_nemotron`, `t2_finqa`, `t2_tatdqa`, `mtrag_clapnq`, `mtrag_clapnq_rewrite`, `mtrag_clapnq_live`, `mtrag_clapnq_live_uncompacted` (the four mtrag keys share one DB).− **Per-case result files**: a single-question QA run writes `<data dir>/evaluations/results/<name>.<trace id>.jsonl` (`evaluations/results.py`, `--results DIR`), one row per case: the `CaseOutcome` fields (`case_name`, `key`, `passed`, `cited`, `cited_map`, `aborted`), trace id, answer, judge reason, `judge_decided_by` / `judge_probability` / `system_one_model` (set by the gated judge, else None), attributes, task duration. A pydantic-evals `CaseLifecycle` appends rows to `<name>.<run id>.partial− └── docker-compose*.yml     # backend, ingester (docker/Dockerfile), frontend− - **The frontend proxies.** `/api/backend/*` and `/api/ingester/*` stream bodies and forward the browser's abort, and the ingester proxy adds `INGESTER_TOKEN` and forwards only the operator actions as writes (refresh a source, retry a job or a dead letter, cancel a job). The dev server allows the `127.0.0.1` origin.+ │   └── exceptions.py       # ReadOnlyError, MigrationRequiredError, UpgradeRequiredError, AmbiguousDatabaseError,+ │                           # UnknownDatabaseError, AmbiguousCitationError, SourceUnavailableError, ConfigMismatchError, TagError+ │   ├── maintenance.py      # maintain(), DatabaseMaintenance: vacuum + missing vector index per database+ ├── hooks.py                # Hook, IngestRequest, IngestEvent, DeleteEvent, SearchRequest, build_hooks, notify+ │   ├── datasets/           # enterprise_rag, frames, hotpotqa, mtrag, open_rag_bench, t2_ragbench+ │   └── evaluators/         # judge, map, citation, retrieval, number_match, refusal, answer_facts, conversation, transcript+ - `create_capability()` → `CitationPolicyCapability` (capabilities/policy.py). Optional. Requires every answer to declare its grounding, asks a plain-text answer once per question, never asks a structured answer, records failures in `CitationPolicyState.violations`, never asks the model to change its answer. Enforced when the question retrieved evidence or the conversation already cited something.+ **Hooks** (hooks.py, user docs in docs/hooks.md):+ - A package registers a factory under the `haiku.rag.hooks` entry-point group. `config.hooks` activates it by name or as `{name, options}` (`HookConfig`), and the options are the factory's keyword arguments. `HaikuRAG.__init__` builds the hooks with `build_hooks`, which loads only the entry points the config names. An unknown name raises `ValueError` at construction.+ - Hook points, run in configured order: `before_ingest`, `after_ingest`, `after_delete`, `before_search`, `after_search`, `lifespan`, `ingester_lifespan`. `before_ingest` refuses a document with `DocumentRejected`, and any other exception fails the ingest as `HookError`. Search hook failures propagate. `after_ingest` and `after_delete` run through `notify`, which logs a failure and never raises. A FULL rebuild passes `notify_hooks=False`.+ - One set of hook instances and one lifespan per client that owns its session. A borrowed client (`clients_for`) gets `_hooks = []`. The ingester opens one client per database, so each database has its own instances and lifespan.+ - `ingester_lifespan` is entered only by `IngesterApp._apply` (`serve`, `run-batch`, manifest replay, reload), through `HaikuRAG._ingester_lifespans`: after reconcile, before the database's sources reach the pool. `_Database.close_client` exits it before the client closes, so every close path takes it, after the pool has stopped serving the database. Background work goes there, never in `lifespan`, which every process enters.+ - A hook's own tables take the `hook_` prefix.+ - The ingester's reload refuses a `hooks` change (`changed_sections`), so it needs a restart.+ - How long metadata from the caller, provider, `before_ingest` and `after_ingest` lasts: docs/hooks.md, "Metadata lifetime".+     built = await rag.create_vector_index()  # False below 256 chunks and on Cloud; writes wait for the build+ - `IngesterConfig` - databases (database name → `IngestedDatabase`: sources), queue (QueueConfig), workers (WorkerConfig), api (APIConfig), maintenance (MaintenanceConfig: enabled, quiet_s, max_wait_s)+ **Database fragments:** `load_yaml_config` pops the top-level `databases_include` and merges every `*.yaml` in that directory (`_merge_fragments`): only the two database maps, a name defined twice raises, `${HAIKU_RAG_CONFIG_DIR}` is the main file's directory. It is not an `AppConfig` field, so `AppConfig.model_validate` on a dict reads no fragments. Reload goes through the loader, so it re-reads them.+ **Core coverage is enforced at 100%, branches included** (`branch = true`, `fail_under = 100`, `source = ["haiku_rag_slim"]`); the evaluations workspace enforces 85%. New code needs a test that takes every branch, or a `# pragma: no cover - <short reason>` or `# pragma: no branch - <short reason>` on one line. CI prints term-missing, so a gate failure names the line. Run coverage with `--cov` and scope the report by grepping it: a deep-dotted `--cov=haiku.rag.<module>` crashes beartype, and a fi+ - Every critical dependency carries an upper bound, and `pydantic-ai-slim` is capped below its next minor release. pydantic-ai changes capability hook semantics in minor releases: 2.54 moved `before_model_request` inside the `wrap_model_request` handler. Raise a cap only in a change that runs every suite and, when capability behaviour can move, the evals.+ - **Lock order** is `_rebuild_lock → _write_lock` (tag ops, restore, `create_vector_index`) and `_vacuum_lock → _write_lock` (vacuum). Rebuild holds `_rebuild_lock` for its whole run, and tag operations fail fast against it. All in-process: cross-process writers need a quiescent boundary for tag create and restore.+ - **Opening a Store never writes to it.** Stored embedding settings change only via `init`/`rebuild`, and the version is a forward-only marker written by `init` and `migrate`. `init` and every `migrate` also write `schema_version`, the newest upgrade step (`latest_schema_version()`), and `recorded_settings` keeps it. A newer stored `schema_version` (`newer_schema`) raises `UpgradeRequiredError` on a writable open and warns on a read-only one. `migrate` and the `tag restore` path open with `skip_+ - **Index sets** live in `index_specs()` / `ensure_indexes()` (schema.py), and every table-creating path routes through them. `ensure_indexes` matches on (column, index type) and never drops an index it did not declare. It skips FTS on an empty table (an index born over zero rows never catches up on `add`) and rebuilds an FTS index covering zero rows. The chunk write paths call it after writing. So a test needing the zero-coverage state writes via `chunks_table.add` directly, and the first chunk+ - **docling and docling-serve are pinned as a pair**: `docling==2.142.1` + `docling-core==2.104.0`, and `quay.io/docling-project/docling-serve:v1.38.2` in `tests/docker/`, `examples/docker/` and both `app/` compose files. The coupling is `tests/chunkers/test_chunker.py::test_local_and_serve_chunkers_produce_same_output` (slow, vcr), which the `test-slow` CI job enforces. Take the client version from the serve release's bundled library set and move both together. Re-record the serve cassettes whe+ - **`pdf_backend` defaults to `threaded_docling_parse`**. `docling_parse` is a deprecated alias of the same threaded implementation, not a serialized parser. docling-serve v1.38.2 names the threaded backend `docling_parse` in its API; the serve converter translates the configuration value. Keep conversion timeouts and hold the backend constant when comparing versions.+ - **`create_document_from_source` metadata merges** differ by path: full ingest does `{**user_metadata, **source_metadata}`, the revision short-circuit's `_refresh_doc_metadata` does `{**doc.metadata, **user_metadata}` before `source_metadata`. `_caller_metadata` and `_provider_metadata` strip every `_INGESTION_OWNED_KEYS` entry (`content_type`, `md5`, `source_revision`, `source_id`, `parent_uri`) from caller and provider metadata. `update_document` restores all five from the stored record. A cr+ - **Reload is remove-then-add per database.** `IngesterApp.add_database` prepares (opens the client, builds adapters, writes no queue row) then applies (reconciles, then serves). `remove_database` unserves, stops the pollers, waits for in-flight jobs, deletes live jobs and `sync_state`, then closes: each step idempotent, and a database stays `removing` until the close lands. Claims, `unserve` and API retries hold `WorkerPool.claim_lock`, and a claim reads the served set inside it, so no claim na+ - **Maintenance** (`ingester.maintenance`, off by default): `DatabaseMaintenance` per database, built in `_apply`, run as a task only under `serve` (`_maintaining`), once per written database in `run-batch` and manifest replay, after the drain and after `_maintain_written` exits every ingester lifespan with the clients still open. A pass holds claims with `WorkerPool.pause`, a set apart from `_served`, so `resume` in a `finally` never re-serves a source that a reload unserved. `close_client` can+ - **Compaction and policy hooks.** `EvidenceCompactionCapability` discovers the evidence capability through `RunContext.capabilities` and rewrites `request_context.messages` in `wrap_model_request`, never the stored history, which keeps question identities and epochs (message counts) meaningful. A question's identity is the index of its opening request, taken by `RAGCapability.open_question` at the question's first model request: pydantic-ai normalizes the history after `for_run`, and AG-UI pass+ - **Windows.** Text mode follows the host locale (cp1252), so read and write repo files with `encoding="utf-8"`. Build an expected rendered path with `pathlib`, not a literal `/`. Write a file under test through `tmp_path`: Windows refuses to reopen an open `NamedTemporaryFile`. Creating a symlink needs Administrator or Developer Mode; without it the `requires_symlinks` tests (`tests/platform.py`) skip and the symlink branch of `sources/fs.py` goes uncovered. Guard a POSIX-only call with `sys.pl+ Datasets: `enterprise_rag`, `frames`, `hotpotqa`, `orb_text`, `orb_multimodal`, `orb_multimodal_nemotron`, `t2_finqa`, `t2_tatdqa`, `mtrag_clapnq`, `mtrag_clapnq_rewrite`, `mtrag_clapnq_live`, `mtrag_clapnq_live_uncompacted` (the four mtrag keys share one DB).+ **Answer facts (`AnswerFactsJudge`, `evaluators/answer_facts.py`)**: every single-question QA run appends it beside `RefusalJudge`. A case whose metadata carries `answer_facts` (only `enterprise_rag`) gets `answer_completeness`, the fraction of facts the answer supports, from one judge call per fact with EnterpriseRAG-Bench's own validator prompt (verbatim, a "yes" on the reply's first line). That judge sees only the answer and the fact, and the `answer_equivalent` verdict never sees the facts: + **Per-case result files**: a single-question QA run writes `<data dir>/evaluations/results/<name>.<trace id>.jsonl` (`evaluations/results.py`, `--results DIR`), one row per case: the `CaseOutcome` fields (`case_name`, `key`, `passed`, `cited`, `cited_map`, `aborted`), `answer_completeness`, trace id, answer, judge reason, `judge_decided_by` / `judge_probability` / `system_one_model` (set by the gated judge, else None), attributes, task duration. A pydantic-evals `CaseLifecycle` appends rows to `+ ├── config/                 # haiku.rag.yaml.example, databases/ (fragments), mounted at /app/config+ └── docker-compose*.yml     # backend, ingester, docling-serve, frontend: published images, or (.dev) built from the checkout+ - **Published images.** `docker-publish-app.yml` pushes `haiku.rag-app-backend` and `haiku.rag-app-frontend` after the slim package publishes, tagged with the `haiku_rag_slim` version and `latest`. `docker-compose.yml` names them and `haiku.rag-slim` at `${HAIKU_RAG_VERSION:-<release>}`, a default `bump_version.py` rewrites. The backend bundles haiku.rag, so its release must match the ingester's. The backend runs as uid 1001 and mounts `DATA_DIR` at `/data` read-only.+ - **`POST /reload` swaps the database set.** `Configuration` (main.py) holds the current `Databases`, and `PinDatabases` puts it on `request.state.databases` for the whole request, a streamed chat run included. Handlers read it there, never `app.state`, and the chat agent lives on it. A replaced set closes when its last request ends, and the lifespan awaits those closes. The reload refuses any change outside `lancedb.databases` (`changed_sections`, shared with the ingester) except the `ingester`+ - **The frontend proxies.** `/api/backend/*` and `/api/ingester/*` stream bodies and forward the browser's abort. The backend proxy refuses `reload`, and the ingester proxy adds `INGESTER_TOKEN` and forwards only the operator actions as writes (refresh a source, retry a job or a dead letter, cancel a job). The dev server allows the `127.0.0.1` origin.+ - **`BASE_PATH` is build-time.** `next.config.ts` validates it (`basePathFrom`, `lib/paths.ts`) and sets Next's `basePath`, which prefixes router navigation and `<Link>`. Every other root-absolute URL (`fetch`, `<img>`, a plain `<a>`, the chat's `HttpAgent`) goes through `withBasePath`. The published frontend image serves at the root, and the Dockerfile takes `BASE_PATH` as a build argument. The dev compose passes it as the dev server's environment.

https://raw.githubusercontent.com/ggozad/haiku.rag/HEAD/AGENTS.md

− │   ├── sandbox.py          # Sandbox, SandboxResult, recovery_hint− │   │   ├── routes/         # health, sources, jobs, dlq, stats, config, database, dashboard, providers− app/                        # Conversational RAG application (see below)− - `IngesterConfig` - sources, queue (QueueConfig), workers (WorkerConfig), api (APIConfig)− serve       Run pollers + workers, and the HTTP API unless --no-api; blocks until SIGINT/SIGTERM− run-batch   One discover sweep over every source, drain the queue, exit; non-zero on dead-letter or incomplete sweep− **Tags:** `haiku-rag tag create/list/delete NAME` name database states across all five tables. `tag restore NAME` brings the live database back to a tagged state: it creates a `before-restore-*` safety tag first, requires stopped writers, and never migrates. A tag that needs no migration restores without the live state's migration check and config validation (`Store.tag_haiku_version`), so the safety tag stays reachable from an older tag. Vacuum keeps every tagged version and the files it refere− - **Poller discovery failures** are handled in three places with the same `record_failure` + `logger.exception` body: `pollers/base.py` `_sweep_once` and `_dry_run_once`, and `pollers/fs.py` `_watch_loop`. Change one, change all three.− - **Compaction and policy hooks.** `EvidenceCompactionCapability` discovers the evidence capability through `RunContext.capabilities` and rewrites `request_context.messages` in `wrap_model_request`, never the stored history, which keeps question identities and epochs (message counts) meaningful. `CitationPolicyCapability` reads the same records in `after_model_request` / `after_run`. `cite([])` records a declaration with no refs (`ungrounded`), distinct from declaring nothing (`missing`).− - **`doctor`** renders details through Rich markup, so avoid literal `[...]`. Near-duplicate detection keeps one summed centroid per document from the single vector scan, and `run_db_checks` keeps the vector matrix a local so `del vectors` bounds memory.− ## haiku.rag.app (Conversational RAG Application)− The `app/` directory contains a conversational RAG application with pydantic-ai's native AG-UI support and a CopilotKit frontend. User documentation is `docs/apps.md`.− # Frontend: http://localhost:3000− # Backend: http://localhost:8001− ├── backend/− │   └── main.py             # Starlette app with pydantic-ai AGUIAdapter− └── frontend/−     ├── app/                # Next.js app−     │   └── api/            # API routes (copilotkit, info, visualize, documents)−     ├── components/         # React components−     │   ├── Chat.tsx        # Main chat interface−     │   ├── CitationBlock.tsx   # Citation rendering with visual grounding−     │   ├── SessionManager.tsx  # Browser-stored sessions−     │   ├── DbInfo.tsx          # Database info panel−     │   └── DocumentFilter.tsx  # Document filtering for sessions−     └── biome.json          # Biome config for linting/formatting− The backend adapts a native `RAGCapability` agent with Pydantic AI's `AGUIAdapter` and emits one final state snapshot. Both compose files bind frontend and backend to 127.0.0.1: there is no authentication, and the frontend proxies every backend route.− - Backend hot reloads automatically− - Frontend: `docker compose -f docker-compose.dev.yml up -d --build frontend`− - Run `biome check` in `app/frontend/` after frontend changes (`biome check --write` to fix)+ │   ├── sandbox.py          # Sandbox, SandboxResult+ │   │   ├── routes/         # health, sources, jobs, dlq, stats, config, database, dashboard, providers, reload+ app/                        # Web application: chat, ingestion, inspector, doctor (see below)+ - `IngesterConfig` - databases (database name → `IngestedDatabase`: sources), queue (QueueConfig), workers (WorkerConfig), api (APIConfig)+ serve       Run pollers + workers, and the HTTP API unless --no-api; blocks until SIGINT/SIGTERM; SIGHUP or POST /reload re-reads the databases+ run-batch   One discover sweep over every source (--db-name NAME: one database's), drain the queue, exit; non-zero on dead-letter or incomplete sweep+ **Tags:** `haiku-rag tag create/list/delete NAME` name database states across all five tables. `tag restore NAME` brings the live database back to a tagged state: it creates a `before-restore-*` safety tag first, requires stopped writers, and never migrates. A tag that needs no migration restores without the live state's migration check and config validation (`Store.tag_haiku_version`), so the safety tag stays reachable from an older tag. Vacuum keeps every tagged version and the files it refere+ - Every critical dependency carries an upper bound, and `pydantic-ai-slim` is capped below its next minor release. pydantic-ai changes capability hook semantics in minor releases: 2.54 runs `before_model_request` inside the `wrap_model_request` handler, and `EvidenceCompactionCapability` then compacts nothing. Raise a cap only in a change that runs every suite and, when capability behaviour can move, the evals.+ - **Workers claim only the sources they serve** (`claim_next(source_ids=)`, the pool's served set). At startup `serve`, `run-batch` and manifest replay delete queued and claimed jobs of sources absent from `ingester.databases` (`cancel_unconfigured`), never `--dry-run`. That relies on one process per namespace. The batch commands refuse an empty `ingester.databases` (`NothingToIngestError`). `serve` starts on one and deletes every live job of its namespace.+ - **Reload is remove-then-add per database.** `IngesterApp.add_database` prepares (opens the client, builds adapters, writes no queue row) then applies (reconciles, then serves). `remove_database` unserves, stops the pollers, waits for in-flight jobs, deletes live jobs and `sync_state`, then closes: each step idempotent, and a database stays `removing` until the close lands. Claims, `unserve` and API retries hold `WorkerPool.claim_lock`, and a claim reads the served set inside it, so no claim na+ - **Poller discovery failures** are handled in three places with the same `record_failure` + `_record_error` + `logger.exception` body: `pollers/base.py` `_sweep_once` and `_dry_run_once`, and `pollers/fs.py` `_watch_loop`. Change one, change all three.+ - **Compaction and policy hooks.** `EvidenceCompactionCapability` discovers the evidence capability through `RunContext.capabilities` and rewrites `request_context.messages` in `wrap_model_request`, never the stored history, which keeps question identities and epochs (message counts) meaningful. A question's identity is the index of its opening request, taken at the question's first `before_model_request`: pydantic-ai normalizes the history after `for_run`, and AG-UI passes the question inside `+ - **`doctor`** renders details through Rich markup, so avoid literal `[...]`. Near-duplicate detection keeps one summed centroid per document from the single vector scan. `scan_vectors` holds the vector matrix only while it reduces it, and returns the centroids, so a caller never keeps the matrix.+ ## haiku.rag.app (Web application)+ The `app/` directory is a browser application with four pages: chat, ingestion, inspector and doctor. User documentation is `docs/apps.md`.+ # Frontend: http://localhost:3000   Backend: http://localhost:8001   Ingester: http://localhost:8765+ (cd backend && uv run pytest --cov)           # backend suite, 100% gate+ (cd frontend && pnpm test && pnpm run typecheck && pnpm run check)+ ├── backend/                # haiku_rag_app package, a uv workspace member+ │   ├── haiku_rag_app/+ │   │   ├── main.py         # create_app(config, sessions_path), Databases, /health, /api/databases+ │   │   ├── chat.py         # AG-UI stream over stored sessions, the documents listing+ │   │   ├── sessions.py     # SQLite session store, holds, the turn commit+ │   │   ├── inspect.py      # inspector routes and /api/search+ │   │   ├── images.py       # checks on images the page sends, pictures, visualize+ │   │   ├── jobs.py         # Jobs: one worker thread with its own event loop+ │   │   ├── doctor.py       # doctor and projection routes+ │   │   └── projection.py   # t-SNE of document centroids, similarity, Retained+ │   └── tests/+ ├── frontend/               # Next.js, Tailwind, shadcn/ui, @ag-ui/client+ │   ├── app/                # pages: / (chat), /ingestion, (inspector)/ for /inspect/[[...path]] and /search, /doctor/[[...db]]; api/backend, api/ingester proxies+ │   ├── components/         # chat/, ingestion/, inspect/, doctor/, ui/ (shadcn)+ │   └── lib/                # tested logic: addresses, chat, sessions, ingestion, inspect, map, doctor, proxy+ └── docker-compose*.yml     # backend, ingester (docker/Dockerfile), frontend+ - **The backend reads, the ingester writes.** `Databases` opens one read-only client covering the configured set on first use and retries a database that failed. The ingester is the only LanceDB writer, and the backend writes only its session file.+ - **One uvicorn worker.** `Sessions` holds a session for a run, a rename or a deletion in process (`admit`, synchronous, before any await), and the Dockerfile and dev compose pin `--workers 1`.+ - **The backend owns the run input.** `stream_chat` takes only the new user message from the client and builds the run from the stored messages and state, clearing tools, context and forwarded props.+ - **The commit is the durable boundary.** On run completion `chat_events` commits the turn through `Sessions.start_commit`, shielded. The session stays held until the write lands, even when the response is collected (`tie`). A run cancelled earlier stores nothing. The frontend reconciles a lost stream by run-id membership in `runs`, and keeps the pending run in `sessionStorage` across a reload.+ - **Question identities are message counts**, so a run's messages and the state it advanced are kept or dropped together.+ - **Doctor and projection jobs** run on one daemon thread with its own event loop, one at a time across databases, each over the `Store` it opens. Progress reaches the request loop through `call_soon_threadsafe`. A projection's unit centroids are kept within one 256 MB budget (`Retained`).

https://raw.githubusercontent.com/ggozad/haiku.rag/HEAD/AGENTS.md

−   with a reranker configured still takes cosine). Over-fetch (`limit * 10`) only− - `EmbeddingsConfig` / `EmbeddingModelConfig` - provider, name, vector_dim, multimodal (gates image embedding), batch_size− - `SearchConfig` - limit, max_context_chars, vector_index_metric, vector_refine_factor, vector_nprobes− queue       init | migrate the queue DB− - **Embedding drift on open.** A `vector_dim` mismatch always raises `ConfigMismatchError`. Provider/name drift at the same dimension warns read-only and raises writable. It is either a config naming another model or an interrupted rebuild, so the advice (`EMBEDDER_DRIFT_ADVICE`, shared with doctor) is `rebuild --embed-only` or rerunning the rebuild, and `rebuild --set-embedder` only for an embedder that produces the same vectors. Maintenance commands, `list`, `get`, `visualize` and CLI deletion− - **docling and docling-serve are pinned as a pair**: `docling==2.124.0` + `docling-core==2.93.0`, and `quay.io/docling-project/docling-serve:v1.32.0` in `tests/docker/` and `examples/docker/`. The coupling is `tests/chunkers/test_chunker.py::test_local_and_serve_chunkers_produce_same_output` (slow, vcr), which the `test-slow` CI job enforces. Take the client version from the serve release's bundled library set and move both together. Re-record the serve cassettes when the image moves: that test− - **Queue path** defaults to `get_default_data_dir() / "ingester.db"`, not `storage.data_dir`. `ingester.queue.dburi` points it at Postgres.− - **Queue migrations are versioned** (`ingester/queue/migrations.py`, `SCHEMA_VERSION`). `create_all` never adds a column to an existing table: a new column needs the model change, an explicit `ALTER TABLE` (+ backfill) under `current < N`, and a `SCHEMA_VERSION` bump.− - **Docker images.** Both Dockerfiles install `--extra ingester`, which does not pull `[docling]`. The full image gets docling from the `haiku.rag` package itself. The published slim image cannot run `converter: docling-local`. `[ingester]` pulls `[s3]`.− - **`sandbox.code_timeout` is a deadline**: `_check_deadline` runs before every host call (`_timed` wraps the five readers, in-code `search()` and `list_documents()` call it first, `_run_on_loop` keeps its own), because cached reads never touch the bridge. Monty's feed limit (`max_feed_duration_secs = code_timeout`, reset per `feed_run`) stops compute and never counts time suspended on the host. It raises the same `TimeoutError` type as the read deadline and leaves the heap undefined, so `_hit_t− - **Picture-only chunks** (one `#/pictures/*` ref, produced only under a multimodal embedder) are embedded from the image, and their `content` is the description or caption followed by the text docling found inside the picture (OCR'd words, footnotes), which can be empty. Search never reads `vision`, so a non-vision model still gets them, expanded to their section's text, and `Type:` then shows the section's label. Rerankers score the unexpanded chunk: multimodal ones image plus that text, text − Separate workspace package (`evaluations/`) for benchmarking. Run its tests from inside `evaluations/` (`cd evaluations && uv run pytest ...`): from the repo root, its `tests/` collides with the core `tests/`.− cd evaluations− uv sync− evaluations run <dataset>− evaluations pair <treated>.jsonl <baseline>.jsonl− evaluations download <dataset|all>           # Pre-built eval DBs from HuggingFace− evaluations upload <dataset|all>             # Upload eval DBs+   with a reranker configured still takes cosine). Over-fetch (`limit * search.rerank_pool_factor`) only+ - `EmbeddingsConfig` / `EmbeddingModelConfig` - provider, name, vector_dim, multimodal (gates image embedding), query_prefix, document_prefix (prepended to query and chunk text, ollama/openai/vllm/openrouter only), batch_size+ - `SearchConfig` - limit, max_context_chars, rerank_pool_factor, vector_index_metric, vector_refine_factor, vector_nprobes+ queue       init | migrate the queue DB; --dburi URL (or HAIKU_RAG_QUEUE_DBURI) names a server queue's owning account+ - **Embedding drift on open.** A `vector_dim` mismatch always raises `ConfigMismatchError`. Provider, name or prefix drift at the same dimension warns read-only and raises writable (`EmbedderIdentity` and `embedder_drift` in engine.py: a missing prefix key reads as empty, and an empty prefix is not recorded). It is either a config naming another model or an interrupted rebuild, so the advice (`EMBEDDER_DRIFT_ADVICE`, shared with doctor) is `rebuild --embed-only` or rerunning the rebuild, and `re+ - **docling and docling-serve are pinned as a pair**: `docling==2.127.0` + `docling-core==2.96.1`, and `quay.io/docling-project/docling-serve:v1.33.0` in `tests/docker/` and `examples/docker/`. The coupling is `tests/chunkers/test_chunker.py::test_local_and_serve_chunkers_produce_same_output` (slow, vcr), which the `test-slow` CI job enforces. Take the client version from the serve release's bundled library set and move both together. Re-record the serve cassettes when the image moves: that test+ - **One pdfium lock.** libpdfium is not thread-safe. docling's PDF backends lock their calls with `docling.utils.locks.pypdfium2_lock`, and `PDFIUM_LOCK` (`pdf_split.py`) is that lock when docling is installed. A second lock around a pdfium call races the conversion thread under `worker_count > 1`: `Failed to import pages` from slicing, `Failed to parse file` from docling, on arbitrary slices. `test_pdf_concurrency.py` pins it.+ - **Queue path** defaults to `get_default_data_dir() / "ingester.db"`, not `storage.data_dir`. `ingester.queue.dburi` points it at Postgres, where `IngesterApp` requires a non-blank `namespace`: every repository statement carries it, and `Job` never does.+ - **Queue migrations are versioned** (`ingester/queue/migrations.py`, `SCHEMA_VERSION`). `create_all` never adds a column to an existing table: a new column needs the model change, an explicit `ALTER TABLE` (+ backfill) under `current < N`, and a `SCHEMA_VERSION` bump. On SQLite `apply_migrations` issues `BEGIN` itself: sqlite3 starts a transaction before DML only, so DDL inside an `engine.begin()` block commits as it runs without one. `open_queue` migrates a SQLite file and only verifies a serv+ - **Docker images.** Both Dockerfiles install `--extra ingester`, which does not pull `[docling]`. The full image gets docling from the `haiku.rag` package itself. The published slim image cannot run `converter: docling-local`: `DoclingLocalConverter` imports docling at construction and raises `ImportError` naming the extra, so `haiku-ingester` and `add-src` exit at startup. `[ingester]` pulls `[s3]`.+ - **`sandbox.code_timeout` is a deadline**: `_check_deadline` runs before every host call (`_timed` wraps the five readers, in-code `search()` and `list_documents()` call it first, `_run_on_loop` keeps its own), because cached reads never touch the bridge. Monty's feed limit (`max_feed_duration_secs = code_timeout`, reset per `feed_run`) stops compute and never counts time suspended on the host. It raises the same `TimeoutError` type as the read deadline and leaves the heap undefined, so `_hit_t+ - **Picture-only chunks** (one `#/pictures/*` ref, produced only under a multimodal embedder) are embedded from the image, and their `content` is the description or caption followed by the text docling found inside the picture (OCR'd words, footnotes), which can be empty. Search never reads `vision`, so a non-vision model still gets them, expanded to their section's text, and `Type:` then shows the section's label. Rerankers score the unexpanded chunk: multimodal ones image plus that text, text + Separate workspace package (`evaluations/`) for benchmarking. Its README is the operator guide. Runs use the full install, docling included, so sync from the repository root. Run its tests from inside `evaluations/` (`cd evaluations && uv run pytest ...`): from the repo root, its `tests/` collides with the core `tests/`.+ uv sync --all-packages+ uv run evaluations run <dataset>+ uv run evaluations pair <treated>.jsonl <baseline>.jsonl+ uv run evaluations download <dataset|all>    # Pre-built eval DBs from HuggingFace+ uv run evaluations upload <dataset|all>      # Upload eval DBs

https://raw.githubusercontent.com/ggozad/haiku.rag/HEAD/AGENTS.md

− │                           # AmbiguousCitationError, SourceUnavailableError, ConfigMismatchError− - `EvaluationsConfig` - judge (ModelConfig | None), system_one (SystemOneConfig | None: base_url, model; key from `TYPESAFE_API_KEY`)− **Tags:** `haiku-rag tag create/list/delete NAME` name database states across all five tables. `tag restore NAME` brings the live database back to a tagged state: it creates a `before-restore-*` safety tag first, requires stopped writers, and never migrates. Vacuum retains the oldest tagged version and everything newer.− - **One version per write.** Each `create_document*` / `import_document` writes a version of `documents`, `chunks` and `document_items`. Bulk ingestion in a loop fills disk with versions. Use `import_documents([DocumentImport(...)])`, or `DocumentRepository.create([...])` / `DocumentItemRepository.create_all(...)`.− - **Index sets** live in `index_specs()` / `ensure_indexes()` (schema.py), and every table-creating path routes through them. `ensure_indexes` matches on (column, index type) and never drops an index it did not declare. It skips FTS on an empty table (an index born over zero rows never catches up on `add`) and rebuilds an FTS index covering zero rows. The chunk write paths call it after writing. So a test needing the zero-coverage state writes via `chunks_table.add` directly, and the first chunk− - **`optimize()` on an indexed table writes a version**, so an `auto_vacuum` pass touches `documents`. A test measuring version deltas sets `storage.auto_vacuum = False`.− - **Vacuum.** `Store.vacuum()` retains `storage.vacuum_retention_seconds` (default 24h). Tables with `large_binary` payloads (`documents`, `document_items`, matched by `has_payload_columns`) go through lance directly via `_run_lance_maintenance` (`compact_files(target_rows_per_fragment=…)`, `optimize_indices`, `cleanup_old_versions`, wrapped in `_wait_protected`). The other tables use `table.optimize()`. `AsyncTable.optimize` exposes no compaction options (lancedb#2325). To purge recent fragment− - **Tags and vacuum.** `optimize(cleanup_older_than=...)` hard-errors when a tagged version falls in the cleanup window, and lancedb does not expose `error_if_tagged_old_versions`. `Store.vacuum` grows the retention to keep the cutoff older than the oldest tag (`TAG_RETENTION_MARGIN`) and suppresses `OSError` only. Do not pass `error_if_tagged_old_versions=False` on the lance path: with a tag present it prunes manifests and frees no bytes.− - **`document_items` across documents** goes through the grouped accessors only (`resolve_refs_grouped`, `get_items_in_ranges`, `get_pictures_grouped`, `get_caption_picture_refs_grouped`). Each builds a per-document predicate, `(document_id = 'a' AND col IN (…)) OR (…)`: `self_ref` values collide across documents, and `self_ref IN (union)` over-fetches and can hand one document another's picture. Tests assert the query counts.− - **Embed-only rebuild recreates the chunks table**, so patching `store.chunks_table.add` on the instance is discarded. Patch `lancedb.AsyncTable.add` at class level and filter on `self.name`. Its phase 2 writes via `chunks_table.add`. A FULL rebuild refreshes source-backed documents in place, keeping the id (re-fetch with `force=True`, falling back to stored content).− - **`create_document_from_source` metadata merges** differ by path: full ingest does `{**user_metadata, **source_metadata}`, the revision short-circuit's `_refresh_doc_metadata` does `{**doc.metadata, **user_metadata}` before `source_metadata`. Strip reserved keys (`md5`, `source_revision`, `content_type`, `source_id`) from caller metadata. `source_id` is the ingesting source's id, or the stored one when the caller names none (`_keep_source_id`), so an ad-hoc re-ingest cannot detach a document. − **Gated judge (`evaluations.system_one`)**: optional, off by default, and our reference configs never set it. When set, `AnswerEquivalenceJudge` (`evaluators/answer_equivalence.py`, a `SystemOneJudge` from `evaluators/system_one.py`) asks the `/v1/systemone` endpoint first and decides alone at p ≥ 0.8 or p < 0.2. The band between, endpoint errors and `ConversationInput` cases go to the `TranscriptLLMJudge` as its fallback. `check_system_one` fails the run before any case when the endpoint does n+ │                           # AmbiguousCitationError, SourceUnavailableError, ConfigMismatchError, TagError+ │   ├── http.py             # Pooled httpx client, retry of a dropped connection+ - **An entry may carry a tag** (`DatabaseEntry`), and entries may share a location. No check refuses covering two states of one database: shared documents return once per entry and `AmbiguousCitationError` guards citation.+ - `EvaluationsConfig` - judge (ModelConfig | None), system_one (SystemOneConfig | None: provider, base_url, model; key from `TYPESAFE_API_KEY`)+ `--at TAG` is per-subcommand on the nine read commands (`search`, `ask`, `list`,+ `get`, `visualize`, `chat`, `inspect`, `info`, `mcp`) and resolves to one+ database, refusing a configured set. `resolve_scope(db, at=)` puts the tag on+ that `DatabaseRef`. Write commands have no `--at`.+ **Tags:** `haiku-rag tag create/list/delete NAME` name database states across all five tables. `tag restore NAME` brings the live database back to a tagged state: it creates a `before-restore-*` safety tag first, requires stopped writers, and never migrates. A tag that needs no migration restores without the live state's migration check and config validation (`Store.tag_haiku_version`), so the safety tag stays reachable from an older tag. Vacuum keeps every tagged version and the files it refere+ - **One version per write.** Each `create_document*` / `import_document` writes a version of `documents`, `chunks` and `document_items`. Bulk ingestion in a loop fills disk with versions. Use `import_documents([DocumentImport(...)])`, or `DocumentRepository.create([...])` / `DocumentItemRepository.create_all(...)`. `_flush_rebuild_batch` writes a batch's items with `create_all`, split whenever the whole documents gathered reach `_ITEMS_WRITE_BYTES` (16 MiB) of pictures, a threshold an add can ex+ - **An indexed `merge_insert` holds its source in lance's query memory pool**, 150 MB unless `LANCE_MEM_POOL_SIZE` is set, and cannot spill. lance takes that plan when every join column has a BTree, the target rows are under it, and there is no `when_not_matched_by_source_delete`. Fresh tables have no rows under their BTrees until vacuum or `rebuild_indexes`, so a small-table test passes either way. The `documents` merges pass `use_index(False)`. A test of a blob-row merge runs `rebuild_indexes(+ - **Index sets** live in `index_specs()` / `ensure_indexes()` (schema.py), and every table-creating path routes through them. `ensure_indexes` matches on (column, index type) and never drops an index it did not declare. It skips FTS on an empty table (an index born over zero rows never catches up on `add`) and rebuilds an FTS index covering zero rows. The chunk write paths call it after writing. So a test needing the zero-coverage state writes via `chunks_table.add` directly, and the first chunk+ - **A vacuum after writes commits compaction and index versions**, so an `auto_vacuum` pass touches `documents`. A test measuring version deltas sets `storage.auto_vacuum = False`.+ - **Vacuum.** `Store.vacuum()` retains `storage.vacuum_retention_seconds` (default 24h). Every table goes through lance directly via `_run_lance_maintenance` (`compact_files`, `optimize_indices`, `cleanup_old_versions`, wrapped in `_wait_protected`). Tables with `large_binary` payloads (`documents`, `document_items`, matched by `has_payload_columns`) compact with `target_rows_per_fragment=…`, which `AsyncTable.optimize` does not expose (lancedb#2325). To purge recent fragments, call the path wit+ - **Tags and vacuum.** Vacuum prunes with `cleanup_old_versions(error_if_tagged_old_versions=False)`, which keeps every tagged version and removes every other version past the retention with its files. That needs lance-format/lance#8708 (pylance 11+). Never call `AsyncTable.optimize`: its cleanup defaults to 7 days, hard-errors on a tagged version in the window, and lancedb does not expose `error_if_tagged_old_versions`. Vacuum suppresses `OSError` only. `test_vacuum_reclaims_versions_written_af+ - **`document_items` across documents** goes through the grouped accessors only (`resolve_refs_grouped`, `get_items_in_ranges`, `get_pictures_grouped`, `get_caption_picture_refs_grouped`, `get_all_picture_data_grouped`). Those reading by `self_ref` build a per-document predicate, `(document_id = 'a' AND col IN (…)) OR (…)`: `self_ref` values collide across documents, and `self_ref IN (union)` over-fetches and can hand one document another's picture. Tests assert the query counts.+ - **Embed-only rebuild recreates the chunks table**, so patching `store.chunks_table.add` on the instance is discarded. Patch `lancedb.AsyncTable.add` at class level and filter on `self.name`. Its phase 2 writes via `chunks_table.add`. A FULL rebuild refreshes source-backed documents in place, keeping the id (re-fetch with `force=True`, falling back to stored content). An attachment (`parent_uri` set) is never fetched itself: its URI resolves to the parent's file. Parents go first, `force` re-in+ - **`create_document_from_source` metadata merges** differ by path: full ingest does `{**user_metadata, **source_metadata}`, the revision short-circuit's `_refresh_doc_metadata` does `{**doc.metadata, **user_metadata}` before `source_metadata`. `_caller_metadata` strips `source_id` and `parent_uri` from caller metadata. `_provider_metadata` strips every `_RESERVED_METADATA_KEYS` entry (`content_type`, `md5`, `source_revision`, `source_id`, `parent_uri`) from provider output. `source_id` is the i+ **Gated judge (`evaluations.system_one`)**: optional, off by default, and our reference configs never set it. When set, `AnswerEquivalenceJudge` (`evaluators/answer_equivalence.py`, a `SystemOneJudge` from `evaluators/system_one.py`) asks the decision endpoint first and decides alone at p ≥ 0.8 or p < 0.2. The band between, endpoint errors and `ConversationInput` cases go to the `TranscriptLLMJudge` as its fallback. Unset `provider` means `/v1/systemone` (`SystemOneEndpoint`, the v1 request). A