Configuration, costs, and operations

← Back to README

Language

Three independent settings cover the three roles a “language” can play:

The split exists because the three roles are genuinely different:

• An English speaker analyzing a Russian Telegram chat wants their wizard / saved-report metadata in English (language=en) but the LLM should produce Russian summaries (report_language=ru). No source hint needed — the chat is in Russian and the LLM detects that.
• A Russian speaker reading a Chinese Wikipedia article wants Russian reports (language=ru, report_language=ru) but the LLM doesn’t always nail “this article is in Chinese” purely from the prose (especially when the report headings are Russian). Set content_language=zh and the model trusts the hint.

# config.toml
[locale]
language = "en"             # UI: wizard, banners, settings menu.
report_language = "ru"      # LLM writes analyses in Russian.
content_language = ""       # Empty = let the AI auto-detect the source.
                            # Set to "zh"/"ru"/… as a Whisper-style override.

Per-run overrides:

# English UI, Russian report; let the LLM detect what the chat is in.
unread @somechat --language en --report-language ru

# Russian UI + Russian report, but the source is Chinese (override the
# LLM's source-language detection).
unread https://zh.wikipedia.org/wiki/物理学 --report-language ru --content-language zh

# Ask follows the same shape.
unread ask "что обсуждали?" --language en --report-language ru

Whisper transcription has its own knob ([openai] audio_language) — empty means autodetect; decoupled from all three locale axes.

Persisting preferences with unread settings

Edit your locale prefs without touching config.toml:

unread settings                                  # interactive editor (three rows under Languages)
unread settings show                             # current effective values + DB overrides
unread settings set locale.language en
unread settings set locale.report_language ru
unread settings set locale.content_language zh   # Whisper-style override; empty = auto-detect
unread settings unset locale.content_language    # drop a single override
unread settings reset                            # drop all DB overrides

Saved to storage/data.sqlite in the app_settings table. Applied on every unread invocation; explicit --language / --report-language / --content-language flags still win.

Migration note

[locale] content_language was renamed to [locale] report_language in v1.x — the old name now means a Whisper-style source-language hint (empty = auto-detect). If you had content_language = "ru" set under the old semantics (LLM writes in Russian), re-set it as report_language = "ru" and clear content_language (or leave it empty to let the AI auto-detect). The CLI flag --content-language was similarly renamed to --report-language; the new --content-language flag is the source hint.

Cost & caching

Three caches, aggressive by design:

1. Local analysis_cache

Every analysis result is hashed by preset + prompt_version + model + sorted msg_ids + options_payload + system/user-prompt hashes and stored in SQLite. Re-run the same query → zero-cost hit. Toggling --enrich, --by, the model, or any other option-payload field busts the relevant rows.

unread cache stats               # rows, disk size, saved $, breakdown + prompt-cache hit rate
unread cache ls --limit 20       # latest entries
unread cache show <hash-prefix>  # print a stored result
unread cache export -o old.jsonl --older-than 30d
unread cache purge --older-than 30d --vacuum

Truncated results are never cached. A partial summary would silently poison every future run.

2. OpenAI prompt cache (server-side)

When prompt prefix ≥ 1024 tokens and identical bytes arrive within ~5–10 minutes, OpenAI discounts repeated tokens. unread cache stats shows your hit rate per (chat, preset) at the bottom of its output. config.toml enforces temperature=0.2 and a fixed system → static_context → dynamic message order to maximize hits.

3. Enrichment dedup

Per-kind forever-cache:

• Media (voice / videonote / video / photo / doc) keyed by Telegram’s stable document_id or photo_id via media_enrichments.
• External links keyed by normalized URL hash via link_enrichments.

Forwarded 10× = fetched once.

Up-front cost guard

unread @somegroup --max-cost 0.50    # confirm if estimate exceeds
unread @somegroup --max-cost 0.50 --yes   # silently abort if over
unread @somegroup --dry-run          # estimate-and-exit, no LLM call

Estimate covers the analysis (map + reduce); enrichment cost is not included.

Spending visibility

unread stats                     # totals by preset
unread stats --by chat           # biggest spenders by chat
unread stats --by day            # spend over time
unread stats --by kind           # chat vs audio vs ask
unread cache stats               # OpenAI prompt-cache hit rate per (chat, preset) (bottom of output)

If a row says (N unpriced) next to its call count, those rows used a model not in your [pricing.chat] / [pricing.audio] table — add the entry so cost stops under-reporting. unread doctor flags missing pricing entries.

Maintenance

# Version info (use this when filing bug reports)
unread --version                               # or `unread -V`

# Health check — Telegram session, OpenAI key, ffmpeg, DB integrity, presets, disk, pricing
unread doctor

# Diagnostic bundle for GitHub issues — version, doctor, redacted config + .env
unread bug-report                              # prints to stdout
unread bug-report --out report.txt             # writes to a file

# Backup the data DB (VACUUM INTO — atomic, compact)
unread backup up                               # → storage/backups/data-YYYY-MM-DD_HHMMSS.sqlite
unread backup up mybackup.sqlite --overwrite

# Restore a backup (current DB moved aside as data-replaced-…sqlite)
unread backup restore storage/backups/data-2026-04-25_…sqlite --yes

# `unread cache` is split into three entity groups, each exposing the
# same five commands: bare `<entity>` = ls, plus purge / stats / show / export.

# Analysis cache (per-LLM-call result rows in `analysis_cache`)
unread cache ai                                       # list newest first
unread cache ai stats                                 # size, age range, by (preset, model), prompt-cache hit rate
unread cache ai show <hash-prefix>                    # print a stored result
unread cache ai purge --older-than 90d                # delete by age / preset / model / --all
unread cache ai export -o ai.jsonl                    # dump rows to JSONL or markdown

# Source caches (extracted pages / YouTube transcripts / local-file text)
unread cache sources                                  # ls all kinds
unread cache sources ls --kind website                # one kind only
unread cache sources stats                            # rows + age per kind
unread cache sources show <page_id|video_id|file_id>  # row metadata + paragraph preview
unread cache sources purge --domain zh.wikipedia.org  # filter: --url / --domain / --kind / --older-than / --all
unread cache sources export -o sources.jsonl          # JSONL inventory; pass --include-paragraphs for full text

# Telegram message cache (per-row `messages` table — synced chat history)
unread cache tg                                       # per-chat counts (text / transcripts / age)
unread cache tg stats                                 # totals + chats / messages / transcripts / age range
unread cache tg show 1234567890                       # one chat's stats
unread cache tg purge --retention 90d                 # blank old message text (renamed from `unread cleanup`)
unread cache tg purge --retention 30d --chat 1234567890
unread cache tg export -o tg.jsonl --chat 1234567890  # dump cached rows; user-facing reports = `unread dump`

# Prune old report files to reports/.trash/<ts>/
unread reports prune --older-than 30d --dry-run    # see what would move
unread reports prune --older-than 30d
unread reports prune --older-than 90d --purge       # hard delete (asks first)

# Cache hygiene
unread cache purge --older-than 30d --vacuum

cleanup preserves row metadata (ids, dates, authors, transcripts) — it only NULLs the raw text column.

config.toml

The shipped file uses the convention: only settings you override are uncommented; every knob is listed (commented-out, showing its default) so it’s discoverable. Strict mode is on — typos fail loudly with a clear “extra inputs not permitted” error and the offending key.

Most-tuned settings:

[openai]
chat_model_default = "gpt-5.6-luna"      # final / single-chunk model
filter_model_default = "gpt-5.6-luna"    # map phase + cheap rerank + self-check
# audio_language = ""                    # Whisper hint; empty = autodetect

[locale]
# language = "en"                        # UI: "en" (default) / "ru" / …
# report_language = ""                   # LLM output; empty = follow `language`
# content_language = ""                  # Whisper-style source hint; empty = AI auto-detects

[analyze]
min_msg_chars = 3                        # filter: drop messages shorter than N chars
dedupe_forwards = true                   # collapse identical forwards/memes
output_budget_tokens = 1500              # reduce / single-chunk max_tokens
confirm_cost_above_usd = 0.25            # ask before a run estimated above this
high_impact_reactions = 3                # `[high-impact]` marker threshold

[enrich]
vision_model = "gpt-4o-mini"

[ask]
rerank_enabled = true                    # default: rerank candidates before the answer
rerank_top_k = 500                       # candidate pool size before rerank
rerank_keep = 50                         # what survives rerank → flagship

[pricing.chat.<model>] / [pricing.audio] populate the cost table. Models that aren’t priced still work — they just show as “unpriced calls” in unread stats and unread doctor warns about it.

Using a model newer than your release

You don’t need an app update to switch to a newly released model:

[ai]
chat_model = "gpt-5.7-luna"              # beats every preset's `final_model` pin
filter_model = "gpt-5.7-luna"            # beats every preset's `filter_model` pin

[pricing.chat."gpt-5.7-luna"]
input = 0.20
cached_input = 0.02
output = 1.20
context_window = 1000000                 # optional; unknown models chunk at 128k

Or, on a server or Docker container: UNREAD_AI_CHAT_MODEL / UNREAD_AI_FILTER_MODEL. Model precedence for analysis is --model / --filter-model → ai.chat_model / ai.filter_model → the preset’s pin, when the provider serves it (every shipped preset pins an OpenAI model) → the provider’s own default ([openai] *_model_default on OpenAI). So naming a provider without a model is enough.

Model catalog updates

The list of models and their prices ships with each release in unread/ai/catalog.json. A running unread bot re-fetches that file from the repo’s main branch every 24 hours, and unread update refreshes it once, so new models and price changes arrive without upgrading. Other commands only read the last downloaded copy (from ~/.unread/storage/model_catalog.json); they never wait on the network for it.

[ai]
catalog_url = "https://raw.githubusercontent.com/maxbolgarin/unread/main/unread/ai/catalog.json"
catalog_refresh_hours = 24               # 0 = never fetch; "" URL also disables
live_pricing = true                      # price unknown models from OpenRouter's public list

A model that neither [pricing] nor the catalog knows is priced from OpenRouter’s public model list on its first call (one request, no key), and the price is remembered in ~/.unread/storage/live_prices.json. It is an estimate; [pricing] always wins. Env: UNREAD_AI_LIVE_PRICING.

Env: UNREAD_AI_CATALOG_URL, UNREAD_AI_CATALOG_REFRESH_HOURS. A download is applied only if it validates and is no older than the catalog your version shipped with. [pricing.chat.<model>] entries still override it.

Maintainers: the weekly Model prices workflow runs scripts/sync_model_prices.py, which compares catalog prices with OpenRouter’s public model list and opens a PR with the changes, plus a list of new models to add by hand. A row with "manual_until": "YYYY-MM-DD" (a promo where we record the list rate) is left alone until that date.

UNREAD_CONFIG_PATH=/abs/path/config.toml overrides the cwd-relative discovery.

How it works

CLI (Typer)
  ├─ Resolver (Telethon)        ──► SQLite: chats
  ├─ Backfill (incremental)     ──► SQLite: messages
  └─ Analyzer pipeline
       ├─ Filter + dedupe (+ optional --by sender filter)
       ├─ Enrich (per-kind, opt-in)
       │    ├─ voice/videonote/video → OpenAI Audio    ──► media_enrichments(kind=transcript)
       │    ├─ photo                 → OpenAI Vision    ──► media_enrichments(kind=image_description)
       │    ├─ doc                   → pypdf / python-docx
       │    └─ link                  → httpx + bs4 + LLM ──► link_enrichments
       ├─ Chunk (token-aware, soft-breaks on idle gaps)
       ├─ Map-reduce (OpenAI)   ──► analysis_cache, analysis_runs, usage_log
       ├─ Optional --self-check (cheap-model verifier)
       └─ Optional --cite-context (expand `[#msg_id](url)` to message blocks)

Three SQLite tables under storage/data.sqlite matter most:

• messages — every message you’ve synced, plus media metadata + transcripts.
• media_enrichments / link_enrichments — per-kind enrichment caches keyed by stable IDs.
• analysis_cache — keyed analyses (zero-cost re-runs).

Plus newer:

• chat_last_run_args — backs --repeat-last.
• message_embeddings — vector store for unread ask --semantic.
• usage_log — every OpenAI call, with phase= tag for cost attribution.

Reports land in reports/ (gitignored). Each cited claim is a clickable link back to the source message. With --cite-context N the report file additionally contains <details> fold blocks with N messages around every citation, so the report is self-auditable without re-opening Telegram.

Analyses larger than one context window are automatically map-reduced: filter_model summarizes chunks in parallel, final_model merges. Each map call is cached independently — adding one new message at the tail re-costs only one chunk.

unread ask is its own pipeline: keyword retrieval (or embedding cosine) → optional rerank → format → single LLM call with citations. No map-reduce; the candidate pool is bounded by --limit.

Before every network call, analyze and dump compare the local max msg_id against Telegram’s read marker — if nothing new exists, the command exits without hitting the network.

Examples / recipes

# Daily morning digest of your work folder, into Saved Messages, on a 24h cron
unread watch --interval 24h analyze --folder Work --preset digest --post-saved

# Audit a high-stakes report — citations get expanded, claims verified
unread @somegroup --preset action_items --cite-context 5 --self-check

# What did Bob say last week? In one chat, with rerank + post-answer follow-ups (default)
unread ask "what did Bob propose?" @somegroup --last-days 7

# Filter analysis to one sender (with a citable result)
unread @somegroup --by Bob --preset highlights

# Cost-bounded run, with a budget alarm
unread @somegroup --enrich-all --max-cost 0.50 --post-to me

# Re-run with the same flags as last time, but force a fresh cache
unread @somegroup --repeat-last --no-cache

# Build a semantic index over a folder, then query it
unread ask --build-index --folder Work
unread ask "open architecture questions" --folder Work --semantic

# Forum: per-topic reports for the entire forum
unread @forumchat --all-per-topic

# Dump and save every photo / voice / video / doc alongside the text
unread dump @somegroup --save-media --save-media-types photo,voice

# Analyze a long-form article — paragraph-indexed citations link back to the page
unread "https://www.paulgraham.com/greatwork.html" --preset website

# Analyze a YouTube video, force Whisper instead of captions
unread "https://youtu.be/dQw4w9WgXcQ" --youtube-source audio --post-saved

Troubleshooting

When something breaks, always start with unread doctor — it surfaces 90% of the common issues with a fix hint inline. If you’re filing a GitHub issue, run unread bug-report and paste the bundle: it gathers version, platform, doctor output, and your config with every secret masked.

For anything else: unread bug-report > report.txt and paste into a new issue at https://github.com/maxbolgarin/unread/issues.

Development

uv sync --extra dev
uv run pytest -q              # all tests (pytest-asyncio auto mode)
uv run ruff check .           # lint
uv run ruff format --check .  # format check (CI runs this)

Contributor guide — dev setup, commit conventions, presets and their cache versions, adding an AI provider — lives in CONTRIBUTING.md. The pipeline itself is sketched in How it works above.

Run unread doctor after any pull or env change — it surfaces the common breakage points (missing ffmpeg, broken Telegram session, missing pricing entries, schema drift).