wet AGENTS.md changelog

https://raw.githubusercontent.com/n24q02m/wet/HEAD/AGENTS.md

−   - `cli.py` -- `wet` control plane: `server start|stop|status`, `config`, `token hash`, `users path`, `docs reindex|import|reembed`, `search`, `warmup`−   - `setup.py`, `setup_tool.py` -- First-run SearXNG install + warmup logic (MCP-callable)−   - `sources/` -- Data source integrations (crawler, docs, searxng, search backends/strategies)− `wet server start`. `wet docs reindex|import|reembed` work without a server.−   - Empty embed/rerank cell -> local ONNX (fastretrieval); unkeyed chat cell -> LLM features off.− - Search backends (`SEARCH_BACKENDS` CSV, runtime fallback — thử từng backend, rơi xuống backend kế khi error/empty): `searxng` (self-host) | `tavily` | `brave` | `exa` | `kagi` (keyed, `KAGI_API_KEY`) | `openrouter` (`OPENROUTER_API_KEY`, `openrouter:web_search` server tool) | `firecrawl` (key optional — không key vẫn thử keyless, `FIRECRAWL_API_KEY`) | `duckduckgo` | `startpage` (credential-free HTML scrape, uvx-safe; datacenter IP có thể bị bot-challenge → chain tự advance).− - SearXNG: `WET_AUTO_SEARXNG` (default true), `SEARXNG_URL` (external mode)− - Browser render backends (headless leg of `extract`): `BROWSER_BACKENDS` (CSV escalation chain: `native` | `browserless` | `invisible`; empty = `native`), `BROWSERLESS_URL`/`BROWSERLESS_TOKEN` (self-host browserless), `STEALTHFOX_BINARY` (path to a patched Firefox build for the `invisible` tier — skips the engine download, seal still verified), `CAPSOLVER_API_KEY` (optional key-gated captcha tier)+   - `cli.py` -- `wet` control plane: `server start|stop|status`, `config`, `token hash`, `users path`, `docs list|reindex|import|reembed`, `search`, `warmup`+   - `setup.py`, `setup_tool.py` -- First-run browser warmup + local-model setup (MCP-callable); the SearXNG install path is retained but off by default (`WET_AUTO_SEARXNG`)+   - `sources/` -- Data source integrations (crawler, docs, search backends/strategies; the third-party searxng client lives in `sources/searxng.py`)+ `wet server start`. `wet docs list|reindex|import|reembed` work without a server.+   - Key pool (hull 0.5): every key above is seeded once into the provider's credential pool in `~/.hull/hull.db` (`WET_HULL_DB` overrides). A 401/403 disables that key and the call rotates to the next one; a 429 cools it down by `Retry-After` / `rate_limit_reset`; an exhausted pool raises `wet.keypool.ProviderUnavailable` (provider + cooldown, never a key). The legacy key is never removed from env/config.+   - Multi-account: `[models.<task>] account = "<account id>"` (default: the provider's unaccounted pool), `provider = "<identity>"` overrides the identity derived from `base_url`. Managed through hull's `CredentialStore` / `AccountRegistry`.+   - Empty embed/rerank cell -> local ONNX (fastretrieval); unkeyed chat cell -> LLM features off. A cell with no key still counts as configured when its pool holds credentials.+ - Search backends (`SEARCH_BACKENDS` CSV, runtime fallback — thử từng backend, rơi xuống backend kế khi error/empty): `tavily` | `brave` | `exa` | `kagi` (keyed, `KAGI_API_KEY`) | `openrouter` (`OPENROUTER_API_KEY`, `openrouter:web_search` server tool) | `firecrawl` (key optional — không key vẫn thử keyless, `FIRECRAWL_API_KEY`) | `keenable` (key optional — không key thì gọi endpoint public keyless, `KEENABLE_API_KEY`) | `duckduckgo` | `startpage` (credential-free HTML scrape, uvx-safe; datacent+ - Browser render backends (headless leg of `extract`): `BROWSER_BACKENDS` (CSV escalation chain: `native` | `cf-browser-rendering` | `invisible`; empty = `native`), `CF_BROWSER_RENDERING_TOKEN`. `browserless` chỉ dùng với instance managed bên thứ ba (`BROWSERLESS_URL`/`BROWSERLESS_TOKEN`); service self-host của project đã bị gỡ cùng DNS. `STEALTHFOX_BINARY` (path to a patched Firefox build for the `invisible` tier — skips the engine download, seal still verified), `CAPSOLVER_API_KEY` (optional k

https://raw.githubusercontent.com/n24q02m/wet/HEAD/AGENTS.md

− Script entry points: `wet` (`wet.cli:main`) and `wet-mcp` (legacy alias).− - Identity layer (one coherent person across the chain): `WET_IDENTITY_SEED` (int pin; default 0 = derive once and persist per namespace under `~/.wet/subs/<sub>/identity.json`), `IDENTITY_PROFILE_DIR` (persistent browser profile dir; default `~/.wet/subs/<sub>/profiles`). Requires the optional `hull-core[identity]` extra (`invisible-core`); without it strategies keep legacy behavior (warn once). First identity build may download a GeoIP mmdb (egress) and resolve locale/timezone from the egress + Script entry points (all `wet.cli:main`): `n24q02m-wet` (canonical, matches the PyPI dist), `wet` (short alias) and `wet-mcp` (legacy alias).+ - Identity layer (one coherent person across the chain): `WET_IDENTITY_SEED` (int pin; default 0 = derive once and persist per namespace under `~/.wet/subs/<sub>/identity.json`), `IDENTITY_PROFILE_DIR` (persistent browser profile dir; default `~/.wet/subs/<sub>/profiles`). Requires the optional `n24q02m-hull[identity]` extra (`invisible-core`); without it strategies keep legacy behavior (warn once). First identity build may download a GeoIP mmdb (egress) and resolve locale/timezone from the egre

https://raw.githubusercontent.com/n24q02m/wet/HEAD/AGENTS.md

−   - `server.py` -- FastMCP server (orchestrator, file lon nhat)−   - `relay_setup.py` -- `apply_config` / `load_config_from_file` env-applier used by the OAuth setup form (live setup UX = OAuth-AS browser form at `<PUBLIC_URL>/authorize`; the `ensure_config` create-session/poll path is legacy/unused in production)−   - `relay_schema.py` -- Relay form schema (2 modes: local/cloud)−   - `sync/` -- Docs sync backends: `gdrive.py` (Google Drive, OAuth Device Code, httpx) + `s3.py` (S3/R2/B2 operator mode) + `base.py`−   - `token_store.py` -- Local token storage cho OAuth (~/.wet/tokens/)−   - `setup_tool.py` -- Warmup + setup-sync logic (MCP-callable)−   - `sources/` -- Data source integrations (crawler, docs, searxng)− uv run wet                     # Chay server− mise run dev       # uv run wet− - Sync: `SYNC_ENABLED` (default true), `GOOGLE_DRIVE_CLIENT_ID` (required for sync), `SYNC_FOLDER` (default "wet"), `SYNC_INTERVAL` (default 300s)− - Sync dung Google Drive API truc tiep (httpx). OAuth Device Code flow, token luu tai `~/.wet/tokens/google_drive.json`− - HTTP auth (live self-host): credentials are configured via the OAuth-AS browser form at `<PUBLIC_URL>/authorize`; `GET /mcp` without a Bearer token returns 401 + `www-authenticate` pointing at `/.well-known/oauth-protected-resource`. The browser form is gated by `MCP_RELAY_PASSWORD` (single shared password, gate only — empty disables it; not per-user). Multi-user remote mode also requires `CREDENTIAL_SECRET` (per-sub vault key) + `MCP_DCR_SERVER_SECRET` (proof of intentional multi-user deploy)− - `MCP_RELAY_URL`: read only by the legacy `ensure_config` create-session/poll path (`relay_setup.py`), which has no production caller — the live setup UX is the OAuth-AS form above, not an ECDH relay.−       "command": "uvx", "args": ["wet-mcp"],−       "env": {−         "OPENROUTER_API_KEY": "sk-or-xxx"−       }− ## Cloudflare serverless mode− Forcing cloud embed/rerank (so the container never downloads local ONNX models):− configure the `[models.embed]` / `[models.rerank]` cells in `config.toml` (any− OpenAI-compatible endpoint; OpenRouter default) and inject the key via− `HULL_EMBED_API_KEY` / `HULL_RERANK_API_KEY` — or one `OPENROUTER_API_KEY` for− every OpenRouter-default cell.− With cloud cells keyed, `fastretrieval` is never instantiated, keeping the− container slim. Storage: set− `MCP_STORAGE_BACKEND=cf-kv` + `MCP_KV_BASE_URL=http://kv.internal`,− `DOCS_DB_BACKEND=cf-d1` + `MCP_D1_BASE_URL`/`MCP_VECTORIZE_BASE_URL`/`MCP_VECTORIZE_IDX`,− `SEARCH_BACKEND=tavily` + `TAVILY_API_KEY`, and `CREDENTIAL_SECRET`.− - **CF template (frozen for P3 replication):** `docs/cf-template.md` pins the canonical−   worker.ts/wrangler.jsonc copy contract (KEEP/DROP handler matrix, 5 footguns, the−   security rule that outbound handlers stay OFF the public `fetch`, E.1 readiness probe−   + E.2 single-user-DO/poll patterns, sync mode-gating). The 5 P3 servers copy this; do−   not re-solve E.1/E.2 per server.− - Pipeline: PSR v10 -> PyPI (uv publish) -> MCP Registry− - Cloudflare deploy remains release-triggered and pushes only its internal−   image to `registry.cloudflare.com`; no public OCI image is published.− ## Known bugs / gotchas (phat hien 2026-04-18 E2E)− 1. **Setup flow 2-phase race condition**:−    - Phase 1: user submit API keys form -> `writeConfig(SERVER_NAME, config)` -> `state=configured` (nhanh, ~5-30s)−    - Phase 2: Google Drive OAuth Device Code flow start (async, BLOCKING user hanh dong tren `google.com/device`)+   - `server.py` -- FastMCP server (orchestrator, file lon nhat); always the HTTP MCP endpoint (no stdio mode)+   - `cli.py` -- `wet` control plane: `server start|stop|status`, `config`, `token hash`, `users path`, `docs reindex|import|reembed`, `search`, `warmup`+   - `runtime.py` -- Bridge to hull-core (settings, auth mode, users store, per-sub namespace)+   - `identity.py` -- Coherent browsing identity across the strategy chain+   - `docs_import.py`, `docs_reembed.py` -- Offline docs maintenance (no running server needed)+   - `setup.py`, `setup_tool.py` -- First-run SearXNG install + warmup logic (MCP-callable)+   - `sources/` -- Data source integrations (crawler, docs, searxng, search backends/strategies)+   - `alembic/` -- Docs DB migrations+ uv run wet server start            # Chay server (background; --foreground de giu terminal)+ uv run wet server status           # pid / lock / HTTP liveness+ uv run wet server stop+ mise run dev       # uv run wet server start --foreground+ Script entry points: `wet` (`wet.cli:main`) and `wet-mcp` (legacy alias).+ `wet search` and other server-backed subcommands talk to the running server over+ streamable HTTP; with no server running they exit 2 and print a hint to run+ `wet server start`. `wet docs reindex|import|reembed` work without a server.+ ## Server, bind & auth+ - One transport: the HTTP MCP server (`wet server start`, or `python -m wet.server`).+ - Bind: `[server] host` / `port` in `~/.wet/config.toml`; `WET_HOST` / `WET_PORT` env override (container deployments).+ - Per-port lifecycle lock (`LifecycleLock("wet", port)`, lock files under `~/.config/mcp/locks`) -- one instance per port, several ports can run side by side.+ - `[server] auth` modes:+   - `no-auth` -- loopback bind only; the server refuses to start on a non-loopback host.+   - `token` -- single bearer token; store only its hash (`wet token hash <token>`).+   - `multi` -- per-user tokens from `[server] users_file` (default `~/.wet/users.toml`, see `wet users path`); data is namespaced per sub under `~/.wet/subs/<sub>/`.+ - Client config points at the HTTP endpoint (see the manual example below).+ - Docs DB backup is an external concern (e.g. rclone of `~/.wet/`); the server has no built-in sync.+ ```bash+ wet server start            # binds [server] host/port from ~/.wet/config.toml+ ```+       "type": "http",+       "url": "http://127.0.0.1:<port>/mcp"+ With `auth = "token"` or `"multi"`, add the client's bearer token header+ (`"headers": {"Authorization": "Bearer <token>"}`).+ - Pipeline (`cd.yml`): `benchmark-contract` gate -> `release` (PSR v10) -> `publish-pypi` (uv publish) -> `publish-mcp-registry` -> `sync-marketplace`+ - No container image and no Cloudflare deploy are published.+ ## Protocol & live tests+ All suites spawn the real HTTP server (`python -m wet.server`) on an ephemeral+ loopback port against a tmp instance home; there is no external E2E driver.+ - `tests/test_protocol_gate.py` -- protocol gate over the production HTTP transport, runs from the source tree (default suite).+ - `tests/test_live_protocol.py` -- `-m live` (real server + stub SearXNG).

Showing the first 40 of each.

https://raw.githubusercontent.com/n24q02m/wet/HEAD/AGENTS.md

− # AGENTS.md - wet-mcp− - `src/wet_mcp/` -- Package chinh (src layout)−   - `token_store.py` -- Local token storage cho OAuth (~/.wet-mcp/tokens/)− uv run wet-mcp                     # Chay server− mise run dev       # uv run wet-mcp− - LLM/Embed/Rerank: litellm passthrough qua `mcp_core.llm` (mcp-core[llm]). Per-task model chains, CSV `provider/model,provider/model`, order = litellm fallback:−   - `EMBEDDING_MODELS` -- chain embedding. Rong = local ONNX (fastretrieval).−   - `RERANK_MODELS` -- chain rerank. Rong = local ONNX cross-encoder.−   - `LLM_MODELS` -- chain LLM. Rong = tat feature LLM.− - Provider duoc suy ra tu prefix model. API key theo convention litellm `<PROVIDER>_API_KEY`. 7 provider servers goi y:−   | model prefix | key env var | get it at |−   |---|---|---|−   | `gemini/` | `GEMINI_API_KEY` | aistudio.google.com/apikey |−   | `vertex_express/` | `GOOGLE_VERTEX_EXPRESS_API_KEY` | cloud.google.com/vertex-ai/generative-ai/docs/start/express-mode/overview |−   | `openai/` (or bare) | `OPENAI_API_KEY` | platform.openai.com |−   | `jina_ai/` | `JINA_AI_API_KEY` | jina.ai/api-key |−   | `cohere/` | `COHERE_API_KEY` | dashboard.cohere.com |−   | `xai/` | `XAI_API_KEY` | console.x.ai |−   | `anthropic/` | `ANTHROPIC_API_KEY` | console.anthropic.com |−   For any other litellm provider (used via env passthrough), see https://docs.litellm.ai/docs/providers/<provider> for its `<PROVIDER>_API_KEY` name.−  - Custom endpoint (SSRF-guarded): `LLM_API_BASE`, `EMBEDDING_API_BASE`, `RERANK_API_BASE`− - Search backends (`SEARCH_BACKENDS` CSV, runtime fallback — thử từng backend, rơi xuống backend kế khi error/empty): `searxng` (self-host) | `tavily` | `brave` | `exa` | `kagi` (keyed, `KAGI_API_KEY`) | `firecrawl` (key optional — không key vẫn thử keyless, `FIRECRAWL_API_KEY`) | `duckduckgo` | `startpage` (credential-free HTML scrape, uvx-safe; datacenter IP có thể bị bot-challenge → chain tự advance).− - Deprecated (honored mot release voi warning): singular `EMBEDDING_MODEL`/`RERANK_MODEL` + `EMBEDDING_BACKEND`/`RERANK_BACKEND` (backend gio suy ra tu chain rong hay khong). Priority-router cu "Jina > Gemini > OpenAI > Cohere" da bo.− - Browser render backends (headless leg of `extract`): `BROWSER_BACKENDS` (CSV escalation chain: `native` | `browserless` | `cf-browser-rendering`; empty = `native`), `BROWSERLESS_URL`/`BROWSERLESS_TOKEN` (self-host browserless), `CF_BROWSER_RENDERING_TOKEN`/`CF_ACCOUNT_ID` (Cloudflare Browser Rendering), `CAPSOLVER_API_KEY` (optional key-gated captcha tier)− - Sync: `SYNC_ENABLED` (default true), `GOOGLE_DRIVE_CLIENT_ID` (required for sync), `SYNC_FOLDER` (default "wet-mcp"), `SYNC_INTERVAL` (default 300s)− - Sync dung Google Drive API truc tiep (httpx). OAuth Device Code flow, token luu tai `~/.wet-mcp/tokens/google_drive.json`−         "EMBEDDING_MODELS": "jina_ai/jina-embeddings-v5-text-small,gemini/gemini-embedding-001",−         "RERANK_MODELS": "jina_ai/jina-reranker-v3",−         "LLM_MODELS": "gemini/gemini-3-flash-preview",−         "JINA_AI_API_KEY": "jina_xxx",−         "GEMINI_API_KEY": "AIza_xxx"− | env var | value |− |---|---|− | `EMBEDDING_MODELS` | `jina_ai/jina-embeddings-v5-text-small` |− | `RERANK_MODELS` | `jina_ai/jina-reranker-v3` |− | `JINA_AI_API_KEY` | from jina.ai/api-key |− A non-empty `*_MODELS` chain whose provider key is set -> cloud backend (inferred,− not via the deprecated `*_BACKEND`). Empty chain -> local ONNX. With cloud forced,−   `fastretrieval` is never instantiated, keeping the container slim. Storage: set−    - Sau Phase 2, token save vao `~/.wet-mcp/tokens/google_drive.json`+ # AGENTS.md - wet+ - `src/wet/` -- Package chinh (src layout)+   - `token_store.py` -- Local token storage cho OAuth (~/.wet/tokens/)+ uv run wet                     # Chay server+ mise run dev       # uv run wet+ - LLM/Embed/Rerank/Chat: per-task provider cells `[models.embed|rerank|chat|jev_score]` in `~/.wet/config.toml` (each `base_url` + `api_key` + `model`, OpenAI-spec). OpenRouter is the pre-wired default for every cell.+   - Credentials: `api_key` in the cell, or `HULL_<TASK>_API_KEY` env injection (`HULL_EMBED_API_KEY` / `HULL_RERANK_API_KEY` / `HULL_CHAT_API_KEY` / `HULL_JEV_SCORE_API_KEY` — env wins over the file).+   - `OPENROUTER_API_KEY` env serves any cell that keeps the OpenRouter default `base_url` — one key can power every task.+   - Empty embed/rerank cell -> local ONNX (fastretrieval); unkeyed chat cell -> LLM features off.+ - Search backends (`SEARCH_BACKENDS` CSV, runtime fallback — thử từng backend, rơi xuống backend kế khi error/empty): `searxng` (self-host) | `tavily` | `brave` | `exa` | `kagi` (keyed, `KAGI_API_KEY`) | `openrouter` (`OPENROUTER_API_KEY`, `openrouter:web_search` server tool) | `firecrawl` (key optional — không key vẫn thử keyless, `FIRECRAWL_API_KEY`) | `duckduckgo` | `startpage` (credential-free HTML scrape, uvx-safe; datacenter IP có thể bị bot-challenge → chain tự advance).+ - Browser render backends (headless leg of `extract`): `BROWSER_BACKENDS` (CSV escalation chain: `native` | `browserless` | `invisible`; empty = `native`), `BROWSERLESS_URL`/`BROWSERLESS_TOKEN` (self-host browserless), `STEALTHFOX_BINARY` (path to a patched Firefox build for the `invisible` tier — skips the engine download, seal still verified), `CAPSOLVER_API_KEY` (optional key-gated captcha tier)+ - Identity layer (one coherent person across the chain): `WET_IDENTITY_SEED` (int pin; default 0 = derive once and persist per namespace under `~/.wet/subs/<sub>/identity.json`), `IDENTITY_PROFILE_DIR` (persistent browser profile dir; default `~/.wet/subs/<sub>/profiles`). Requires the optional `hull-core[identity]` extra (`invisible-core`); without it strategies keep legacy behavior (warn once). First identity build may download a GeoIP mmdb (egress) and resolve locale/timezone from the egress + - Sync: `SYNC_ENABLED` (default true), `GOOGLE_DRIVE_CLIENT_ID` (required for sync), `SYNC_FOLDER` (default "wet"), `SYNC_INTERVAL` (default 300s)+ - Sync dung Google Drive API truc tiep (httpx). OAuth Device Code flow, token luu tai `~/.wet/tokens/google_drive.json`+         "OPENROUTER_API_KEY": "sk-or-xxx"+ configure the `[models.embed]` / `[models.rerank]` cells in `config.toml` (any+ OpenAI-compatible endpoint; OpenRouter default) and inject the key via+ `HULL_EMBED_API_KEY` / `HULL_RERANK_API_KEY` — or one `OPENROUTER_API_KEY` for+ every OpenRouter-default cell.+ With cloud cells keyed, `fastretrieval` is never instantiated, keeping the+ container slim. Storage: set+    - Sau Phase 2, token save vao `~/.wet/tokens/google_drive.json`+ 2. **GDrive token shared voi mnemo**:+    - Neu user auth mot account Google cho wet, mnemo co the auto-detect va skip device code flow (shared account pool ong)+ t2-interaction: GDrive device-code (900s); per-sub token storage at ``~/.wet/subs/<sub>/tokens/google_drive.json``.