Changelog

Engineering change history for Inpedia — modules, scripts, docs, and validation. Currently v0.5.0. No corpus content.

Latest

v0.5.0

Engineering changes

Added

  • Save & index — Review toolbar saves to Inbox and ingests in one step (notebookId field, persisted in browser)
  • Chat search index panel — Index sidebar: Not indexed (Inbox drafts), In search (indexed pages); index selected/all; remove draft; remove page/notebook from search
  • lib/index_manage.py — list pending/indexed entries, ingest from Inbox, delete drafts, remove Chroma chunks
  • RAG source viewer — View source on citations: original scan preview (HEIC→JPEG via sips), all region boxes gray, yellow highlight on the cited region(s); Browse all carousel (←/→) across sources; Open in Review deep link ?asset=
  • lib/asset_regions.py — regions.json beside each original on Review save (bbox + text for highlights)
  • lib/source_view.py — build source view; match highlights using user question + chunk excerpt
  • Server restart — Restart in Chat toolbar (mirrors Review)
  • GET /api/asset (Review) and POST /api/source (Chat) for source material
  • lib/ocr_patch.py — user-drawn bbox wins on Add region OCR patch
  • Tests — tests/test_index_manage.py, tests/test_source_view.py, tests/test_ocr_patch.py

Changed

  • ingest.py — chunk ids {notebook}:{asset_id}:{index}; metadata source_file, chunk_id
  • lib/rag.py — Source includes excerpt, asset_id, chunk_id, source_file
  • review/static/review.js — Focus/Map/Add modes, region resize, keyboard shortcuts skip text inputs; load asset from ?asset=
  • chat/static/ — index management UI, source dialog, actionable sources

Fixed

  • lib/store.py — remove ids from Chroma include (0.5.x API); fixes query errors in Chat
  • lib/ocr_apple.py — HEIC→JPEG before Vision; robust JSON parse on noisy stdout
  • Review M/A/F shortcuts no longer fire while typing in Notebook id or OCR text fields

Earlier releases

v0.4.1

Engineering changes

Added

  • Review ↔ Chat switcher — one-click nav between http://127.0.0.1:8766 (review) and :8765 (chat) in both local UIs

Changed

  • chat/static/ — dark header aligned with Review; shared 3-column header grid
  • lib/rag.py — resolve Chroma index path at query time (long-running chat.py picks up Knowledge Base after migration/ingest)
  • chat.py — print active search index on startup; return HTTP 503 with clear text for OllamaError

Fixed

  • lib/telemetry.py — ignore broken stderr pipe in background servers (no longer masks real errors as [Errno 32] Broken pipe)
  • lib/ollama_client.py — map dropped Ollama connections to actionable errors
  • App nav — active Review/Chat tab stays readable on hover

v0.4.0

Engineering changes

Added

  • review.py + review/static/ — local review UI at http://127.0.0.1:8766 (scan ↔ OCR text, click-to-highlight, zoom/pan, ISO Written date, save to Inbox)
  • scripts/ocr_vision.swift + lib/ocr_apple.py — Apple Vision OCR with bounding boxes and HEIC→JPEG preview via sips
  • lib/readers/ — model-agnostic reader harness (router.py, apple_vision.py); content-kind routing
  • lib/provenance.py — immutable Originals/{assetId}/ archive + manifest.json chain-of-custody stages
  • lib/corrections.py + lib/image_crop.py — glossary memory, corrections.jsonl, region JPEG crops on human correct
  • lib/notebook_meta.py — YAML frontmatter export, document_date validation, OCR date suggestion (suggest_document_date)
  • lib/ocr_refine.py — optional local Ollama refine pass using correction memory
  • review/static/date-suggest.js — client-side date prepopulation from OCR text (ranges like 9/27/17-10/17/17)
  • scripts/export_training_dataset.py — export correction crops for MLX/LoRA experiments
  • tests/test_notebook_meta.py — frontmatter, chunk metadata, date suggestion tests
  • lib/paths.py — Originals/ and Corrections/ under Knowledge Base; ensure_home creates crop folder

Changed

  • lib/chunk.py — reads YAML frontmatter; attaches document_date and asset_id to chunk metadata
  • ingest.py — persists document_date and asset_id in Chroma chunk metadata
  • requirements.txt — pillow for region crops
  • docs/CORPUS.md, docs-site/src/lib/blueprint-data.ts — OCR review path live; Inbox wired for draft exports

v0.3.0

Engineering changes

Added

  • Two roots of trust — source notes and search index in Documents/Inpedia Knowledge Base/, outside the public repo (ADR-008)
  • lib/paths.py — INPEDIA_* env configuration; ingest guard rejects corpus inside repo
  • init.py — create Knowledge Base folders in Documents
  • scripts/migrate_corpus.py — move legacy data/raw/, data/chroma/, and ~/.inpedia/ into Knowledge Base
  • scripts/check_corpus_boundary.py + GitHub Actions CI — fail if personal corpus appears in tracked files
  • docs/CORPUS.md — corpus home guide; /corpus on docs site
  • tests/fixtures/sample-corpus.md — fictional pipeline fixture (replaces in-repo data/raw/)
  • .cursor/commands/save-progress.md — end-of-session doc sync ritual
  • Blueprint — interactive /blueprint architecture map (live / next / horizon)

Changed

  • Knowledge Base location — visible Documents/Inpedia Knowledge Base/ with Source notes, Search index, Inbox (replaces hidden ~/.inpedia/)
  • ingest.py, lib/rag.py — search index in Documents/Inpedia Knowledge Base/Search index/ by default
  • ADR-001 — superseded in part by ADR-008; gitignore is safety net, not the wall

v0.2.0

Engineering changes

Added

  • Local web chat — chat.py + chat/static/ UI at http://127.0.0.1:8765 (localhost only; multi-turn conversation with collapsible sources)
  • lib/rag.py — shared retrieve + answer path for CLI and chat
  • PII-safe telemetry — lib/telemetry.py emits grep-friendly InpediaChat| lines on stderr (ingest + query timings, retrieval distance, notebook ids — never question or answer text)
  • docs/qa/LOCAL_CHAT_CONFIDENCE.md — first-use report, smell-test checklist, telemetry spec (pattern from Tasteful confidence gates)
  • scripts/stream_chat_logs.sh — helper to filter telemetry lines

Changed

  • query.py — delegates to lib/rag.py
  • ingest.py — emits ingest telemetry; suggests chat.py after ingest

Validated

  • First real corpus session — personal-essays.md (*What is Product?*, 2022-10-01): grounded answer on product-growth tradeoffs; explicit “not in excerpts” gap list; 4-source retrieval from local index

v0.1.0

Engineering changes

Added

  • Tracer bullet: ingest.py, query.py, lib/ (chunk, store, ollama client)
  • Dated Markdown chunking (## YYYY-MM-DD)
  • Local RAG with Ollama + Chroma
  • Fictional sample corpus for pipeline testing (data/raw/sample-corpus.md)
  • Public docs: PURPOSE, STACK, CONSIDERATIONS, ADR log
  • docs-site/ — Next.js site for inpedia.intrinsicvalue.llc (docs/DEPLOY.md)