Oh Sheet: AI sheet music from any song

The pipeline

1. 1 · Ingest

   yt-dlp + FFmpeg pull and normalize audio; a cover-search fast path can swap a dense pop mix for a piano cover that transcribes cleaner, with automatic fallback to the original when the cover fails. Duration and livestream guards keep hour-long videos from monopolizing workers.

2. 2 · Transcribe

   ML pitch detection (Basic Pitch / PyTorch models) converts audio into note events with onset, duration, and velocity.

3. 3 · Arrange

   Note events become a two-hand piano arrangement: voice splitting, register mapping, and density control tuned by an evaluation suite.

4. 4 · Humanize

   Velocity and timing refinement so playback sounds performed, not quantized.

5. 5 · Engrave

   MusicXML and PDF engraving, delivered via a claim-check blob store and published straight into a TuneChat room for collaborative practice.

Engineering highlights

• Orchestration: FastAPI job manager fans out stages as Celery tasks over Redis, streaming progress to the browser over WebSockets with a typed Pydantic contract shared across services.
• Resilience: silent- failure contracts at every ingest boundary: cover-search misses, bot-checked downloads, and over-long videos degrade to clear user errors instead of hung jobs.
• Operations: deployed as a single-container Railway monolith with env-driven feature flags, worker isolation, and an eval suite gating arrangement quality in CI.
• Ecosystem: finished scores publish directly into TuneChat rooms, where the same score becomes a multiplayer practice session. Both apps share the “Warm Vinyl” design system.

Testing and evals

• An evaluation suite runs against a fixed set of reference songs to gate arrangement quality in CI -- a failing eval blocks the deploy.
• Each ingest boundary is tested for its failure modes: cover miss, bot-challenged download, over-long video, and malformed audio input.
• Pydantic contracts enforce the inter-stage data shape so type errors surface at service boundaries instead of corrupting output downstream.

Tradeoffs

I deployed Oh Sheet as a single-container Railway monolith. That made the initial deployment fast and the system easy to operate solo, with one surface for logs, restarts, and environment configuration. The cost is that workers, web, and storage all share the same process boundary. If job throughput grew significantly, splitting those concerns into dedicated services would be the right next step.

What I would improve next

• Separate worker processes from the web server to allow horizontal scaling of the transcription and arrangement stages independently.
• Add a job queue depth monitor and rate limiter at the HTTP boundary so the worker pool does not fall behind during traffic spikes.
• Expand the evaluation suite to cover more genres. Current coverage is strongest on popular Western pop and piano ballads.

Stack

• Python 3.12
• FastAPI
• Celery
• Redis
• Pydantic
• Basic Pitch
• yt-dlp
• FFmpeg
• WebSockets
• Vite
• Docker
• Railway