diff --git a/docs/plans/2026-04-04-image-pipeline-gpu-wasm-webgl-design.md b/docs/plans/2026-04-04-image-pipeline-gpu-wasm-webgl-design.md new file mode 100644 index 0000000..53baef7 --- /dev/null +++ b/docs/plans/2026-04-04-image-pipeline-gpu-wasm-webgl-design.md @@ -0,0 +1,144 @@ +# Image Pipeline GPU/WASM/WebGL Design + +## Goal + +Migrate the current CPU pixel-loop image pipeline to an incremental hybrid backend (WebGL first, optional WASM kernels, guarded WebGPU experiments) without breaking the existing worker fallback and cancellation semantics. + +## Current Branch Context + +- This branch already reduced duplicate preview work (shared in-flight preview requests, preview coalescing, opt-in histogram behavior). +- `lib/image-pipeline/worker-client.ts` is already the stable orchestration seam for worker-first execution with main-thread fallback. +- `lib/image-pipeline/preview-renderer.ts` and `lib/image-pipeline/bridge.ts` currently depend on CPU `ImageData` mutation, so backend switching must preserve existing return contracts (`ImageData` for preview, encoded `Blob` for full render). + +## Codebase Mapping Findings + +### `lib/image-pipeline/render-core.ts` + +- Contains all effect kernels as CPU loops over `Uint8ClampedArray` (`curves`, `color-adjust`, `light-adjust`, `detail-adjust`). +- `applyPipelineStep` is a strong integration seam: per-step dispatch already exists and can route by backend capability. +- Abort checks are periodic and lightweight; migration must preserve `AbortError` behavior. + +### `lib/image-pipeline/preview-renderer.ts` + +- Flow today: decode source bitmap -> draw 2D canvas -> `getImageData` -> apply step loops -> optional histogram. +- This file owns preview sizing and output shape; it should remain the public API while delegating kernel execution to a backend adapter. +- The yield loop (`requestAnimationFrame`/`setTimeout`) is CPU-friendly but should become backend-aware (GPU path should avoid unnecessary per-step yielding). + +### `lib/image-pipeline/worker-client.ts` + +- Central execution coordinator with worker transport, request dedupe, cancel fan-out, and main-thread fallback. +- Best place to inject backend telemetry and backend selection hints (for rollout and safety). +- Existing fallback logic (worker unavailable -> main thread) maps directly to backend fallback ladder and kill-switch controls. + +## Architecture Choice + +Adopt a **Backend Adapter + Capability Router** architecture: + +1. Keep existing preview/full public APIs and message formats stable. +2. Introduce an internal backend interface for step execution and pixel transport. +3. Select backend at runtime by capability + feature flag + step support matrix. +4. Preserve CPU backend as correctness source and terminal fallback. + +### Integration Seam + +- New seam: `executePipelineSteps()` in a backend-router module called by both `preview-renderer.ts` and `bridge.ts`. +- `render-core.ts` becomes the CPU backend implementation instead of the global default path. +- Worker protocol receives optional diagnostics metadata (selected backend, fallback reason), not required for correctness. + +### Fallback Ladder (authoritative order) + +1. **WebGL2 worker backend** (preferred for preview/full when supported and flag-enabled) +2. **WASM SIMD kernel backend** (for unsupported GPU steps or devices with poor GPU availability) +3. **CPU worker backend** (`render-core.ts` current path) +4. **CPU main-thread fallback** (existing `worker-client.ts` behavior) + +Each downgrade records a structured reason (`unsupported_api`, `shader_compile_failed`, `capability_mismatch`, `flag_disabled`, `runtime_error`) so rollout decisions are data-driven. + +## Context7 Guidance Buckets + +### Safe now + +- **WebGL2 shader pipeline** for color/curve/light/detail kernels with runtime feature detection. +- **OffscreenCanvas in worker** where available, with same-origin + secure-context guardrails. +- **WASM SIMD (single-threaded)** behind feature detection (`WebAssembly.validate` on SIMD module or runtime probe). +- **Worker-first execution with transferable buffers** (already used in worker client). + +### Experimental + +- **WebGPU compute path** (`navigator.gpu` / `WorkerNavigator.gpu`): gate behind an explicit experiment flag and per-browser allowlist. +- **Step fusion and mixed GPU+WASM scheduling**: only after baseline correctness/perf metrics are stable. +- **Advanced WebGL packing optimizations** (multi-pass fusion, half-float intermediates) after parity suite exists. + +### Not now + +- **WASM threads + SharedArrayBuffer requirement**: requires cross-origin isolation (COOP/COEP) and can impact app embedding/3rd-party integrations. +- **WebGPU-first default backend**: browser/feature variability still too high for this repository's reliability target. +- **Full rewrite of decode/encode stack around custom codecs**: exceeds scope; current `drawImage` + canvas/blob path remains. + +## Phased Rollout + +### Phase 0: Foundation + Instrumentation (no behavior change) + +- Add backend router with CPU-only implementation. +- Add telemetry points for preview/full latency, fallback reason, and error class. +- Add golden-image parity harness for all 4 current step types. + +### Phase 1: WebGL PoC (preview only) + +- Implement WebGL2 backend for `curves` and `color-adjust` first. +- Enable by dev flag on preview pipeline only. +- Validate output parity against CPU baseline under tolerance thresholds. + +### Phase 2: WebGL expansion + guarded production rollout + +- Add `light-adjust` and `detail-adjust` kernels. +- Enable progressive rollout (1% -> 10% -> 25% -> 50% -> 100%) with kill switch. +- Keep full render on CPU backend until preview parity and crash rate are stable. + +### Phase 3: WASM SIMD fallback kernels + +- Add WASM SIMD implementation for hot CPU kernels as middle rung between WebGL and CPU. +- Use when WebGL unavailable or disabled, still inside worker. + +### Phase 4: Full-render backend adoption + +- Route full render through same backend router. +- Keep output encode path unchanged. +- Roll out in smaller increments than preview due to export-critical path. + +### Phase 5: WebGPU experiment track + +- Optional experiment branch with strict allowlist and observability. +- No default enablement in this plan. + +## Risk Controls + +- Feature flags: + - `imagePipeline.backend.webgl.enabled` + - `imagePipeline.backend.wasm.enabled` + - `imagePipeline.backend.webgpu.experiment` + - `imagePipeline.backend.forceCpu` +- Auto-disable circuit breaker when fallback/error thresholds exceed SLO windows. +- Golden-image regression suite enforced in CI before any rollout increase. +- Keep worker protocol backward compatible until migration fully lands. + +## Metrics and Success Criteria + +- Preview latency (`p50`, `p95`) for representative step stacks and image sizes. +- Full render latency (`p50`, `p95`) for default export sizes. +- Main-thread long task count during rapid slider edits. +- Backend selection distribution (% webgl/wasm/cpu/main-thread). +- Fallback and runtime error rate per backend. +- Pixel parity drift (max absolute channel delta and histogram similarity). + +Success target for initial migration: + +- >=30% preview `p95` improvement on supported devices. +- No increase in user-visible render failures. +- <=1% forced downgrade due to runtime backend errors after rollout stabilization. + +## Out of Scope + +- New visual adjustment features. +- Server-side/offline rendering architecture changes. +- COOP/COEP rollout for SAB-threaded WASM.