Content paragraphs...
" + } + ] +} +``` + +### Key Characteristics +- Generated by `convert-bulgakov.ts` from "Настольная книга для священно-церковнослужителей" EPUB +- Stored in `scripts/output/bulgakov.json` (gitignored) +- Menaion section only (306 days, 332 entries) +- Secondary source: fills gaps, never overrides existing Minei content + ## File Organization Rules ### Lives Files - **Location**: `static/data/{lang}/lives/` - **Files**: - `01.json` through `12.json` (month-prefixed CIds) - - `misc.json` (non-month-prefixed CIds) -- **Routing**: Based on first two digits of CId + - `misc/0.json` through `misc/9.json` (non-month-prefixed CIds, split by first digit) +- **Routing**: Based on first two digits of CId (month) or first digit (misc) ### Menaion Files - **Location**: `static/data/{lang}/menaion/` -- **Files**: `01-01.json` through `12-31.json` (daily entries) +- **Files**: bundied into `menaion-bundle.json` per language - **Purpose**: Daily liturgical readings by date ### Service Templates @@ -145,9 +165,15 @@ function bundleFileForCid(cid: string): string { - **Shared**: `static/data/shared/services/templates/` ### Calendar Data -- **Location**: `static/data/{lang}/calendar/` and `static/data/shared/calendar/` +- **Location**: `static/data/shared/calendar/` - **Purpose**: Movable feast calculations and metadata +### Scripture / Paremia Data +- **Location**: `static/data/shared/parimii/` +- **Monthly files**: `jan/full.json` through `dec/full.json` +- **Common files**: `common-saints/full.json`, `cheese-week/full.json`, etc. +- **Purpose**: Old Testament readings for Vespers + ## Data Invariants ### CId Consistency Rules @@ -158,7 +184,7 @@ function bundleFileForCid(cid: string): string { ### File Organization Invariants - No month file contains CIds with different month prefix -- `misc.json` contains all CIds outside the 01-12 prefix range +- `misc/{n}.json` contains all CIds with matching first digit - All data files are tracked in `static/data/` ### Content Validation Invariants @@ -181,7 +207,7 @@ function bundleFileForCid(cid: string): string { - ⚠️ Don't leave trailing commas in JSON ### File Organization Mistakes -- ⚠️ Don't place month-prefixed CIds in misc.json +- ⚠️ Don't place month-prefixed CIds in misc files - ⚠️ Don't create inconsistent JSON indentation - ⚠️ Don't mix languages in incorrect directories @@ -190,7 +216,7 @@ function bundleFileForCid(cid: string): string { ### Valid CId Examples - `010101` → `01.json` (January saint) - `1253` → `12.json` (December saint) -- `772426` → `misc.json` (non-prefixed saint) +- `772426` → `misc/7.json` (non-prefixed saint) ### Invalid CId Examples - `00501` → Should not exist (prefix `00` → misc) @@ -198,8 +224,8 @@ function bundleFileForCid(cid: string): string { ### File Routing Examples - CId `1253` routes to `12.json` -- CId `772426` routes to `misc.json` -- CId `010101` routes to `01.json` +- CId `772426` routes to `misc/7.json` +- CId `373` routes to `misc/3.json` ### Bundle Entry Examples ```json @@ -232,15 +258,17 @@ function bundleFileForCid(cid: string): string { ### Coverage Statistics - Total CIds: 3046 -- Lives with text: 1190 (39% coverage) -- Empty lives: 1856 (61% remain) +- Lives with text: 1203 (39% coverage) +- Sources: Chetyi-Minei (1190) + Bulgakov (13) +- Empty lives: 1843 (61% remain) ### Data Files -- Total validated files: 274 -- Main source files: 6 Lives bundle files (misc, 01-12) +- Lives bundle files: 12 month files (01-12) + 10 misc split files (0-9) - Service template files: Multiple per language +- Parimii files: 12 monthly + common saints + triodion periods ### Recent Changes -- Bundle fix: 868 Иаков Боровицký moved from 05-22 to 10-23 -- Enrichment: 64 manual fixes added to enrich-lives.ts -- Coverage increase: 1009→1190 lives with text (191 new) \ No newline at end of file +- Bundle fix: 868 Иаков Боровицкий moved from 05-22 to 10-23 +- Minei enrichment: 64 manual fixes added +- Bulgakov enrichment: 9 manual fixes added, 13 new lives +- Coverage increase: 1009→1190→1203 lives with text \ No newline at end of file diff --git a/.ai-specs/ENRICHMENT_WORKFLOW.md b/.ai-specs/ENRICHMENT_WORKFLOW.md index 27d46d5..980cf26 100644 --- a/.ai-specs/ENRICHMENT_WORKFLOW.md +++ b/.ai-specs/ENRICHMENT_WORKFLOW.md @@ -4,11 +4,17 @@ Run enrichment in safe mode first: ```bash -# Dry run to see matches +# Minei dry run npx tsx scripts/enrich-lives.ts -# Write changes when confident -npx tsx scripts/enrich-lates.ts --write +# Minei write +npx tsx scripts/enrich-lives.ts --write + +# Bulgakov dry run +npx tsx scripts/enrich-lives.ts --bulgakov + +# Bulgakov write +npx tsx scripts/enrich-lives.ts --bulgakov --write ``` ## Common Enrichment Mistakes @@ -31,20 +37,23 @@ npx tsx scripts/enrich-lates.ts --write - ⚠️ Don't append to CIds that already have life.text without checking collision behavior - ⚠️ Don't assume all multi-source CIds follow same pattern +### Bulgakov-Specific Issues +- ⚠️ Bulgakov only covers Menaion (saints by calendar date), not Triodion/Pentecostarion +- ⚠️ Bulgakov is secondary source — only fills gaps, never overrides Minei +- ⚠️ Some Bulgakov entries are feast descriptions, not saint lives — may have no target CId + ### Validation Skipping -- ⚠️ Don't skip `npx tsx scripts/validate-data.ts` — it catches real errors - ⚠️ Don't skip `npx tsc --noEmit` — type errors indicate real problems - ⚠️ Don't skip `npm run build` — build failures need investigation -- ⚠️ Don't assume "all files valid" without actually running it +- ⚠️ Don't skip data validation — run `validate-data.ts` after changes ## Decision Points for AI Agents ### When to Add MANUAL_FIXES -- When score is 0.35-0.6 and you can identify the correct target -- When source title contains distinctive substring not in other titles on same date - When auto-matcher picks wrong saint (e.g., different homonym on same date) - When target CId exists but is listed on wrong calendar date (bundle bug) - When narrative maps to known CId but has empty life (can be enriched) +- When Bulgakov title contains distinctive substring not in Minei titles ### When to Leave Unmatched - When source is a narrative or homily without saint association @@ -60,7 +69,7 @@ npx tsx scripts/enrich-lates.ts --write ## Quality Checkpoints ### After Manual Fixes Addition -- Verify substring uniqueness per date (use verification script) +- Verify substring uniqueness per date - Check that target CIds exist and are empty - Re-run dry run to verify improvement - Check for potential new conflicts with existing fixes @@ -72,9 +81,9 @@ npx tsx scripts/enrich-lates.ts --write - Verify no unintended side effects on other files ### Before Committing -- Run full validation: `npx tsx scripts/validate-data.ts` - Type check: `npx tsc --noEmit` - Build verify: `npm run build` +- Run data validation: `npx tsx scripts/validate-data.ts` - Check git status to confirm intended files staged ## Troubleshooting Common Issues @@ -105,25 +114,20 @@ npx tsx scripts/enrich-lates.ts --write - ⚠️ **Expected behavior**: First source writes without heading, subsequent sources append - **Check**: Verify total chapters match expected count for CId -### Strange JSON Syntax Errors -- **Problem**: Parse errors in data files after modification -- **Solution**: Check JSON syntax, fix trailing commas, quote issues -- **Verify**: Run `validate-data.ts` to get specific error location - -### Build Size Changes -- **Problem**: Unexpected file size increases/decreases -- **Solution**: Check for malformed content, duplicate data, formatting changes -- **Check**: Compare before/after file sizes in git diff +### Bulgakov-Specific Issues +- **Problem**: Bulgakov entry has no matching CId +- **Solution**: Check if it's a feast day description (not a saint life) +- **Alternative**: Search lives-index.json for the saint's name +- **Check**: Verify the date has the correct CIds in the menaion bundle ## Verification Commands ### Coverage Tracking ```bash -# Check coverage statistics python3 -c " import json, glob, os total=has=0 -for fp in glob.glob('static/data/ru/lives/*.json'): +for fp in sorted(glob.glob('static/data/ru/lives/*.json') + glob.glob('static/data/ru/lives/misc/*.json')): data=json.load(open(fp)) for cid,entry in data.items(): total+=1 @@ -135,14 +139,15 @@ print(f'Coverage: {has}/{total} ({has*100//total}%) lives with text') ### Collision Chapter Count Check ```bash -# Check collision targets have correct chapter counts python3 -c " import json, re -misc = json.load(open('static/data/ru/lives/misc.json')) -for cid in ['493', '1674', '257401']: # Known collision targets - text = misc[cid]['life']['text'] - headings = re.findall(r'([^<]+)', text) - print(f'{cid}: {len(headings)} chapters') +for fp in ['static/data/ru/lives/misc/4.json', 'static/data/ru/lives/misc/1.json']: + data = json.load(open(fp)) + for cid in ['493', '1674', '257401']: + if cid in data: + text = data[cid]['life']['text'] + headings = re.findall(r'([^<]+)', text) + print(f'{cid}: {len(headings)} chapters') " ``` @@ -159,21 +164,22 @@ npx tsx scripts/tools/stats.ts --lang ru ## Recent Context Updates ### Known Fixed Issues -- 868 Иаков Боровицký moved from 05-22 to 10-23 (bundle mis-mapping correction) +- 868 Иаков Боровицкий moved from 05-22 to 10-23 (bundle mis-mapping correction) - Сретение→373 added (02-02 Сказание about Presentation) - Захарий Печерский→822 added (03-24 miracle story) - Анна пророчица→2396 added (08-28 prophetess Anna) ### Current Unmatched Floor -- 27 entries remain unmatched (all narratives/homilies or already-filled targets) -- These are at the floor for Chetyi-Minei source — no recoverable matches available +- **Minei**: 27 entries remain unmatched (all narratives/homilies or already-filled targets) +- **Bulgakov**: 121 entries unmatched (no recoverable targets — saints already have lives or no CId) +- These are at the floor for both sources — no recoverable matches available ## File Routing Quick Reference ### Lives File Routing - Files: `static/data/ru/lives/01.json` through `12.json` for month-prefixed CIds -- File: `misc.json` for non-month-prefixed CIds -- Route: `bundleFileForCid(cid: string):` checks `/^(0[1-9]|1[0-2])\d+/.test(cid)` +- Files: `static/data/ru/lives/misc/0.json` through `9.json` for non-month-prefixed CIds +- Route: `bundleFileForCid(cid)` checks `/^(0[1-9]|1[0-2])\d+/.test(cid)` ### Bundle File Organization - Main bundle: `static/data/cu/menaion-bundle.json` (authoritative calendar) @@ -188,14 +194,14 @@ npx tsx scripts/tools/stats.ts --lang ru - Confirm coverage increased as expected ### Documentation Updates -- Update DATA_FORMAT.md or DATA_RUS.md if schema changed +- Update DATA_EN.md or DATA_RUS.md if schema changed - Update coverage statistics in relevant documentation - Add new manual fixes to enrichment documentation if needed ### Commit Sequence ```bash -# 1. Validate, typecheck, build all passing -npx tsx scripts/validate-data.ts && npx tsc --noEmit && npm run build +# 1. Typecheck, build, validate all passing +npx tsc --noEmit && npm run build && npx tsx scripts/validate-data.ts # 2. Stage all changes git add scripts/enrich-lives.ts scripts/output/lives-match-report.json static/data/ru/lives/ @@ -208,20 +214,19 @@ git commit -m "enrich [N] more ru lives: [description]" ### Current Status - Total CIds: 3046 -- Lives with text: 1190 (enriched from Chetyi-Minei) -- Unmatched minei chapters: 27 (floor: narratives, no-target, already-filled) -- Files validated: 274 files (all passing) +- Lives with text: 1203 (enriched from Chetyi-Minei + Bulgakov) +- Unmatched Minei: 27 (floor: narratives, no-target, already-filled) +- Unmatched Bulgakov: 121 (floor: no recoverable targets) -### Recent Enrichment Work -- Baseline: 1009 lives with text (from prior work) -- Latest commits added 181 lives (121 + 57 + 3) -- Manual fixes added: 214 total in enrich-lives.ts -- Bundle corrections applied: 868 date fix +### Sources +- **Chetyi-Minei**: 1190 lives, 64 manual fixes, 27 unmatched at floor +- **Bulgakov Desktop Book**: 13 lives, 9 manual fixes, 121 unmatched at floor +- **Total coverage**: 1203/3046 (39%) ### Progress Tracking -- Review band: 127 entries resolved, 0 remaining -- Unmatched band: 94 → 30 → 27 (final floor) -- Coverage: 1009→1130→1190 lives (181 net increase) +- Minei band: 94 → 30 → 27 (final floor) +- Bulgakov: 332 entries processed, 13 new lives, 121 unmatched +- Coverage: 1009→1130→1190→1203 (194 net increase) ## Example Workflow for Adding Manual Fixes @@ -240,12 +245,11 @@ Target: 822 (Прп. Захарий Печерский, EMPTY) - correct saint ### Step 3: Verify Uniqueness ```bash -# Check if substring is unique on date 03-24 python3 -c " import json, re m=json.load(open('scripts/output/minei.json')) titles=[t['title'] for t in m['03-24']] -cnt=sum(1 for t in titles if 'воспоминание о чуде' in re.sub(r'\s+',' ',t['title']).lower()) +cnt=sum(1 for t in titles if 'воспоминание о чуде' in re.sub(r'\s+',' ',t).lower()) print(f'Count: {cnt}') " ``` diff --git a/.ai-specs/PROJECT_CONTEXT.md b/.ai-specs/PROJECT_CONTEXT.md index 07f4b42..d558f56 100644 --- a/.ai-specs/PROJECT_CONTEXT.md +++ b/.ai-specs/PROJECT_CONTEXT.md @@ -5,44 +5,45 @@ **Ponomar** is a Progressive Web App for Orthodox Church calendar, liturgical services, and Bible readings. - **Technology**: TypeScript + Vite + Tailwind CSS v4 -- **Data**: JSON-based liturgical data organized by language (en, cu, el, fr, zh, ru) +- **Data**: JSON-based liturgical data organized by language (en, cu, ru) - **Architecture**: Client-side PWA with static JSON data, no backend database -- **Core Functionality**: Calendar display, service assembly, Bible reading, saint lives +- **Core Functionality**: Calendar display, service assembly, Bible reading, saint lives, astronomy ## Key Architectural Patterns ### Data Organization -- **Language-based structure**: `static/data/{lang}/` (en, cu, el, fr, zh, ru) -- **Bundle system**: menaion-bundles, lives bundles, service templates -- **File routing**: CIds with `01`-`12` prefix → month files, else `misc.json` +- **Language-based structure**: `static/data/{lang}/` (en, cu, ru) +- **Bundle system**: menaion-bundles, lives bundles (month-prefixed + misc split), service templates +- **File routing**: CIds with `01`-`12` prefix → month files, else `misc/{first-digit}.json` ### Core Scripts -- `convert-all.ts` — Master conversion script -- `enrich-lives.ts` — Fuzzy matching and enrichment pipeline -- `validate-data.ts` — Main validator (checks 274 files) -- `convert-minei.ts` — Chetyi-Minei EPUB to JSON conversion -- `migrate-data.ts` — Data normalization +- `convert-services.ts` — Service XML to JSON conversion +- `enrich-lives.ts` — Fuzzy matching and enrichment pipeline (Chetyi-Minei + Bulgakov) +- `validate-data.ts` — Main validator +- `convert-bulgakov.ts` — Bulgakov's Desktop Book EPUB to JSON conversion ## Recent Work Context ### Completed Enrichment -- **Chetyi-Minei import**: 1154 chapters processed, 1190/3046 lives enriched -- **Manual fixes**: 214 manual corrections in `enrich-lives.ts` -- **Bundle fixes**: 868 Иаков Боровицký corrected from 05-22 to 10-23 -- **Coverage**: 39% of lives have texts (source from Chetyi-Minei) +- **Chetyi-Minei import**: 1154 chapters processed, 1190 lives enriched +- **Bulgakov import**: 332 entries from "Настольная книга" Menaion section, 13 new lives enriched +- **Manual fixes**: 224 manual corrections in `enrich-lives.ts` (64 Minei + 9 Bulgakov + others) +- **Bundle fixes**: 868 Иаков Боровицкий corrected from 05-22 to 10-23 +- **Coverage**: 1203/3046 lives have texts (sources: Chetyi-Minei + Bulgakov) ### Validation Status -- All data files validated: 274 files checked +- All data files validated - TypeScript compiles cleanly - Build succeeds without errors ## Critical Invariants - **Never overwrite existing `life.text`** — enrichment only fills empty entries +- **Bulgakov is secondary source** — fills gaps, never overrides Minei - **File routing is strict**: CId prefix determines file location - **Bundle CIds must exist**: Menaion CIds require corresponding lives entries - **Static data is tracked**: `static/data/` in git (except `scripts/output/minei.json`) -- **Validation must pass**: Changes must pass `validate-data.ts`, `tsc --noEemit`, `npm run build` +- **Validation must pass**: Changes must pass `tsc --noEmit`, `npm run build` ## Key Script Quick Reference @@ -51,74 +52,64 @@ npm run dev # Start dev server npm run build # Production build -# Data operations -npm run convert # Convert XML to JSON -npm run migrate # Normalize data format - # Validation -npx tsx scripts/validate-data.ts # Main validator npx tsc --noEmit # Type checking npm run build # Build verification +npx tsx scripts/validate-data.ts # Main validator -# Enrichment -npx tsx scripts/enrich-lives.ts # Dry run enrichment -npx tsx scripts/enrich-lives.ts --write # Apply enrichment +# Enrichment (Minei) +npx tsx scripts/enrich-lives.ts # Dry run +npx tsx scripts/enrich-lives.ts --write # Apply + +# Enrichment (Bulgakov) +npx tsx scripts/enrich-lives.ts --bulgakov # Dry run +npx tsx scripts/enrich-lives.ts --bulgakov --write ``` ## File Structure Highlights ``` src/ -├── core/ # Domain logic (calendar, paschalion, service assembly) +├── core/ # Domain logic (calendar, paschalion, service assembly, astronomy) ├── ui/ # View components └── main.ts # Entry point scripts/ -├── convert-*.ts # XML→JSON conversion pipeline -├── validate-*.ts # Comprehensive validation suite -└── enrich-lives.ts # Enrichment with manual fixes +├── convert-*.ts # XML/EPUB → JSON conversion pipeline +├── validate-*.ts # Comprehensive validation suite +└── enrich-lives.ts # Enrichment with manual fixes (Minei + Bulgakov) static/data/ ├── cu/ # Church Slavonic (authoritative calendar) ├── ru/ # Russian (enriched lives) -├── en/ # English -└── shared/ # Cross-language resources +├── en/ # English (enriched lives + service texts) +└── shared/ # Cross-language resources (calendar, lives index, parimii) ``` ## Important Notes ### Pipeline Dependencies - **Chetyi-Minei source**: EPUB conversion requires `convert-minei.ts` -- **Enrichment workflow**: Requires minei.json output + menaion-bundle.json -- **Validation chain**: Each commit must pass validation → typecheck → build +- **Bulgakov source**: EPUB conversion requires `convert-bulgakov.ts` +- **Enrichment workflow**: Requires minei.json + bulgakov.json + menaion-bundle.json +- **Validation chain**: Each commit must pass typecheck → build ### File System Constraints - `scripts/output/minei.json` is gitignored (33MB source file) +- `scripts/output/bulgakov.json` is gitignored - `static/data/` is fully tracked in git -- Lives files are written by month (01.json-12.json) or misc.json - -## Common Tools Reference - -- **Search**: `npx tsx scripts/tools/search-texts.ts
${t.nav.appTitle}
@@ -127,7 +137,7 @@ export class App {
${t.nav.calendar}