A Windows e-reader that uses a local language model to follow a book's mood and play generated ambient music. I built the Python prototype, Flutter reader, reusable Dart/Python/Flutter packages, and an iOS reader prototype.
I wanted background music that follows what I'm reading without demanding attention of its own. That meant building more than a mood classifier: I needed a reader, a way to prepare usable audio, and playback rules that would let a scene settle instead of changing the music every paragraph.
The Windows app imports books, tags paragraphs locally with qwen2.5:7b, and plays prepared MusicGen loops. The models are pretrained; my work is the app, orchestration, prompts, cue selection, audio processing, evaluation tools, and integration between them.
The reading app
Built a Flutter library and reader for ePub and plain text, with chapter navigation, paginated and scrolling layouts, reading-position storage, and appearance settings. Pagination uses real text layout and invalidates its cache when the reading style changes.
Added a plain-reading mode that skips model work, plus inspectors for mood tags, cue decisions, bed sets, and audio auditions. Page turns and scrolling remain usable when background tagging or playback fails; cold AI-enabled startup can still require preparation.
Built a confirmation-based watermark cleaner that detects repeated distributor text, including styled Unicode. On the documented 862-chapter book, it removed 450 mentions without losing a paragraph of story text.
Added optional proofreading as a cached display layer, preserving the source text and mood-cache keys. Trials found that the local 7B model still made meaning-changing edits despite code rules and a confirmation pass, so the feature stays off by default.
Preparing a book's music
Preparation runs ahead of reading. Each story arc gets a theme and a limited set of music beds: short loops that can repeat under the page. The important planning input is how long each cue will actually play, not just how often the classifier assigns a label.
Preparation: local text processing and cue replay produce a music plan. Cloud generation receives prompts and generation metadata; returned audio is checked before installation.
Import and theme. Parse the book's chapters and paragraphs, scan for storyline changes in 50–100 chapter windows, and sample excerpts to derive a musical theme for each arc.
Tag and plan. Classify paragraphs into eight moods with intensity and a short reason. Replay those tags through the same cue-selection logic used by the reader, then allocate a typical 32-loop budget in proportion to playback time, with coverage for moods that appear.
Generate. Export a job containing prompts and generation metadata rather than book passages. Generate multiple candidates per slot, with bounded retries for inaudible or excessively bright output. Kaggle jobs resume after interruption and stop starting work before the session limit.
Process and select. Resample and high-pass the audio, select a steady 24-second segment, close the seam with an equal-power crossfade, and normalize levels. Rank candidates using audibility bounds, a relative register gate, high-frequency harshness, and measured movement. Harsh takes are demoted with warnings rather than silently leaving a cue empty.
Install. Verify returned files, re-measure candidates locally, and write the kept loops, manifest, and pairwise audio distances. A recorded two-arc Kaggle regeneration returned and verified all 192 candidates, installing 32 loops per arc.
Following the reader
During reading, local tags are cached and smoothed before they reach the cue selector. SoLoud plays prepared loops rather than generating music on each page turn.
Read ahead in batches and cache tags by text, model, and prompt version, so changed prompts cannot quietly reuse old classifications. Reader position comes from the top of a page or roughly one third down a scrolling viewport.
Smooth cues with a weighted four-paragraph window, a two-paragraph confirmation rule, and a 15-second minimum dwell. A 2.5× vote for short gentle moods helps keep them from disappearing between longer tense or neutral passages.
Use SoLoud for 2.5-second crossfades, take rotation, intensity-driven gain and a 320 Hz–5 kHz low-pass sweep. Warm up over the first 16 paragraphs and use measured bed distances to decide when to lead into a larger change.
Reusable packages and iOS
Separated a reusable framework into music_book_core (pure Dart ingest, tagging, arc planning, runtime decisions, and CLI), music_book_gen (Python generation, measurement, jobs, and cloud transports), and music_book_player (Flutter/SoLoud playback). Plain JSON and WAV files define the shared workspace contract.
Built an iOS reader host with a library, reader, settings, and prepared-book pack import. Book preparation stays on a computer. The iOS app has widget tests, but it has not been built or tested on an iPhone; device audio and background behavior remain unverified.
Guarded the port with parity tests for 3,072 generation prompts and the Dart/Python mood-mapping arithmetic, plus a contract test that runs the real Python exporter on a Dart-written workspace. The framework remains separate from the Windows implementation; newer Windows-only classifier and audio-guard changes have not all been ported.
What I measured and changed
Mood bias: the original pipeline played tense music for 82% of the replayed time across two arcs of one book. Revised mood definitions brought that to 46.7%; this measures playback allocation, not classifier accuracy. The change also squeezed out some gentler moods, which led to the selector weighting work.
Audio quality: register and texture trials separated prompt changes that affected the sound from ones that did not. Listening reports exposed two different kinds of screech, leading to guards at 4 kHz and 6 kHz calibrated on a small set of judged takes. The latest regeneration passed those checks and was installed; its listening review is still pending.
Hardware and reliability: moved expensive generation from an 8 GB laptop GPU to Kaggle T4s, with resumable exports, checked imports, and a test that catches book text in exported jobs. In the recorded two-arc regeneration, the jobs completed in roughly 60 and 75 minutes while running side by side.
Verification: the documented Windows suite has 286 Flutter tests plus integration tests; the framework has 167 Dart tests and four Python suites, and the iOS host has five headless widget tests. The work also exposed rounding differences between Dart and Python, stale pagination, a ported idle-timer bug, and audio tests that passed while audition playback was silent.
In progress
Solo project