From 1d2654fec16b65ffac8321750163224454d5201b Mon Sep 17 00:00:00 2001 From: Matthias Date: Sat, 4 Apr 2026 14:15:56 +0200 Subject: [PATCH] docs(plans): add image pipeline gpu wasm webgl plan --- ...026-04-04-image-pipeline-gpu-wasm-webgl.md | 286 ++++++++++++++++++ 1 file changed, 286 insertions(+) create mode 100644 docs/plans/2026-04-04-image-pipeline-gpu-wasm-webgl.md diff --git a/docs/plans/2026-04-04-image-pipeline-gpu-wasm-webgl.md b/docs/plans/2026-04-04-image-pipeline-gpu-wasm-webgl.md new file mode 100644 index 0000000..d244215 --- /dev/null +++ b/docs/plans/2026-04-04-image-pipeline-gpu-wasm-webgl.md @@ -0,0 +1,286 @@ +# Image Pipeline GPU/WASM/WebGL Implementation Plan + +> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. + +**Goal:** Incrementally migrate preview/full image processing from CPU-only loops to a backend-routed pipeline with WebGL acceleration, WASM fallback kernels, and CPU safety fallback. + +**Architecture:** Keep `renderPreviewWithWorkerFallback` and `renderFullWithWorkerFallback` as stable entry points while introducing a backend router inside `lib/image-pipeline`. The router selects `webgl -> wasm -> cpu` by capability and feature flags, and always preserves current abort behavior and worker/main-thread fallback. Rollout is phased: CPU-compatible foundation, preview PoC, guarded production rollout, then full-render adoption. + +**Tech Stack:** Next.js 16, TypeScript, Web Workers, OffscreenCanvas/WebGL2, WebAssembly SIMD, Vitest + +--- + +### Task 1: Establish backend router contract (CPU-only behavior) + +**Files:** +- Create: `lib/image-pipeline/backend/backend-types.ts` +- Create: `lib/image-pipeline/backend/backend-router.ts` +- Modify: `lib/image-pipeline/preview-renderer.ts` +- Modify: `lib/image-pipeline/bridge.ts` +- Create: `tests/image-pipeline/backend-router.test.ts` + +**Step 1: Write the failing test** + +Add tests proving router behavior with only CPU backend registered: +- preview path returns identical pixels to current `render-core` behavior +- full render path still produces valid blob output +- unknown/unsupported backend hint falls back to CPU + +**Step 2: Run test to verify it fails** + +Run: `pnpm test tests/image-pipeline/backend-router.test.ts` +Expected: FAIL because backend router modules do not exist. + +**Step 3: Write minimal implementation** + +Implement `backend-types.ts` and `backend-router.ts`, then route `preview-renderer.ts` and `bridge.ts` through router with CPU backend only. + +**Step 4: Run test to verify it passes** + +Run: `pnpm test tests/image-pipeline/backend-router.test.ts` +Expected: PASS + +**Step 5: Commit** + +Run: +- `git add lib/image-pipeline/backend/backend-types.ts lib/image-pipeline/backend/backend-router.ts lib/image-pipeline/preview-renderer.ts lib/image-pipeline/bridge.ts tests/image-pipeline/backend-router.test.ts` +- `git commit -m "refactor(image-pipeline): add backend router seam"` + +### Task 2: Add capability detection and structured fallback reasons + +**Files:** +- Create: `lib/image-pipeline/backend/capabilities.ts` +- Modify: `lib/image-pipeline/backend/backend-router.ts` +- Modify: `lib/image-pipeline/worker-client.ts` +- Create: `tests/image-pipeline/backend-capabilities.test.ts` + +**Step 1: Write the failing test** + +Add tests proving: +- capability detection reports `webgl`, `wasmSimd`, `offscreenCanvas` independently +- router emits fallback reason codes (`unsupported_api`, `flag_disabled`, `runtime_error`) +- `worker-client` can surface backend diagnostics metadata without breaking existing return types + +**Step 2: Run test to verify it fails** + +Run: `pnpm test tests/image-pipeline/backend-capabilities.test.ts tests/image-pipeline/backend-router.test.ts` +Expected: FAIL because capabilities and reason propagation are not implemented. + +**Step 3: Write minimal implementation** + +Implement capability probes and fallback reason plumbing in router and worker client diagnostics. + +**Step 4: Run tests to verify they pass** + +Run: `pnpm test tests/image-pipeline/backend-capabilities.test.ts tests/image-pipeline/backend-router.test.ts` +Expected: PASS + +**Step 5: Commit** + +Run: +- `git add lib/image-pipeline/backend/capabilities.ts lib/image-pipeline/backend/backend-router.ts lib/image-pipeline/worker-client.ts tests/image-pipeline/backend-capabilities.test.ts tests/image-pipeline/backend-router.test.ts` +- `git commit -m "feat(image-pipeline): add backend capability and fallback diagnostics"` + +### Task 3: Introduce feature flags and CPU force-kill switch + +**Files:** +- Create: `lib/image-pipeline/backend/feature-flags.ts` +- Modify: `lib/image-pipeline/backend/backend-router.ts` +- Create: `tests/image-pipeline/backend-feature-flags.test.ts` + +**Step 1: Write the failing test** + +Add tests proving: +- `forceCpu` overrides all other backend choices +- `webgl` and `wasm` can be independently enabled/disabled +- defaults preserve current CPU behavior when no explicit flags are set + +**Step 2: Run test to verify it fails** + +Run: `pnpm test tests/image-pipeline/backend-feature-flags.test.ts` +Expected: FAIL because feature flag module and router integration are missing. + +**Step 3: Write minimal implementation** + +Add typed flag readers and router wiring to honor force-cpu and backend enablement toggles. + +**Step 4: Run test to verify it passes** + +Run: `pnpm test tests/image-pipeline/backend-feature-flags.test.ts tests/image-pipeline/backend-router.test.ts` +Expected: PASS + +**Step 5: Commit** + +Run: +- `git add lib/image-pipeline/backend/feature-flags.ts lib/image-pipeline/backend/backend-router.ts tests/image-pipeline/backend-feature-flags.test.ts tests/image-pipeline/backend-router.test.ts` +- `git commit -m "feat(image-pipeline): add backend rollout flags"` + +### Task 4: WebGL PoC backend for preview (`curves`, `color-adjust`) + +**Files:** +- Create: `lib/image-pipeline/backend/webgl/webgl-backend.ts` +- Create: `lib/image-pipeline/backend/webgl/shaders/curves.frag.glsl` +- Create: `lib/image-pipeline/backend/webgl/shaders/color-adjust.frag.glsl` +- Modify: `lib/image-pipeline/backend/backend-router.ts` +- Modify: `lib/image-pipeline/preview-renderer.ts` +- Create: `tests/image-pipeline/webgl-backend-poc.test.ts` + +**Step 1: Write the failing test** + +Add tests proving: +- when WebGL is available and flagged on, preview router selects WebGL for supported steps +- unsupported steps in same pipeline force safe fallback to CPU for that request +- compile/link failure downgrades with `runtime_error` reason + +**Step 2: Run test to verify it fails** + +Run: `pnpm test tests/image-pipeline/webgl-backend-poc.test.ts` +Expected: FAIL because WebGL backend modules are not implemented. + +**Step 3: Write minimal implementation** + +Implement WebGL PoC backend for `curves` and `color-adjust`, register it in router, and ensure preview renderer receives `ImageData` output unchanged. + +**Step 4: Run tests to verify they pass** + +Run: `pnpm test tests/image-pipeline/webgl-backend-poc.test.ts tests/image-pipeline/backend-router.test.ts` +Expected: PASS + +**Step 5: Commit** + +Run: +- `git add lib/image-pipeline/backend/webgl/webgl-backend.ts lib/image-pipeline/backend/webgl/shaders/curves.frag.glsl lib/image-pipeline/backend/webgl/shaders/color-adjust.frag.glsl lib/image-pipeline/backend/backend-router.ts lib/image-pipeline/preview-renderer.ts tests/image-pipeline/webgl-backend-poc.test.ts` +- `git commit -m "feat(image-pipeline): add webgl preview backend poc"` + +### Task 5: Add pixel parity regression harness for CPU vs WebGL + +**Files:** +- Create: `tests/image-pipeline/parity/fixtures.ts` +- Create: `tests/image-pipeline/parity/cpu-webgl-parity.test.ts` +- Modify: `tests/image-pipeline/webgl-backend-poc.test.ts` + +**Step 1: Write the failing test** + +Add parity tests that validate max per-channel delta and histogram similarity for representative pipelines: +- only curves +- only color-adjust +- curves + color-adjust chain + +**Step 2: Run test to verify it fails** + +Run: `pnpm test tests/image-pipeline/parity/cpu-webgl-parity.test.ts` +Expected: FAIL because parity fixtures/tolerances are not implemented. + +**Step 3: Write minimal implementation** + +Create shared fixtures and numeric tolerance helpers; update PoC tests to enforce parity gates before backend selection assertions. + +**Step 4: Run tests to verify they pass** + +Run: `pnpm test tests/image-pipeline/parity/cpu-webgl-parity.test.ts tests/image-pipeline/webgl-backend-poc.test.ts` +Expected: PASS + +**Step 5: Commit** + +Run: +- `git add tests/image-pipeline/parity/fixtures.ts tests/image-pipeline/parity/cpu-webgl-parity.test.ts tests/image-pipeline/webgl-backend-poc.test.ts` +- `git commit -m "test(image-pipeline): add cpu webgl parity coverage"` + +### Task 6: Expand WebGL backend to `light-adjust` and `detail-adjust` + +**Files:** +- Create: `lib/image-pipeline/backend/webgl/shaders/light-adjust.frag.glsl` +- Create: `lib/image-pipeline/backend/webgl/shaders/detail-adjust.frag.glsl` +- Modify: `lib/image-pipeline/backend/webgl/webgl-backend.ts` +- Modify: `tests/image-pipeline/parity/cpu-webgl-parity.test.ts` + +**Step 1: Write the failing test** + +Add parity cases for: +- `light-adjust` +- `detail-adjust` +- mixed 4-step pipeline (`curves`, `color-adjust`, `light-adjust`, `detail-adjust`) + +**Step 2: Run test to verify it fails** + +Run: `pnpm test tests/image-pipeline/parity/cpu-webgl-parity.test.ts` +Expected: FAIL because new shaders are not implemented. + +**Step 3: Write minimal implementation** + +Implement and register light/detail shaders and uniforms in WebGL backend with same parameter normalization as CPU path. + +**Step 4: Run test to verify it passes** + +Run: `pnpm test tests/image-pipeline/parity/cpu-webgl-parity.test.ts` +Expected: PASS + +**Step 5: Commit** + +Run: +- `git add lib/image-pipeline/backend/webgl/shaders/light-adjust.frag.glsl lib/image-pipeline/backend/webgl/shaders/detail-adjust.frag.glsl lib/image-pipeline/backend/webgl/webgl-backend.ts tests/image-pipeline/parity/cpu-webgl-parity.test.ts` +- `git commit -m "feat(image-pipeline): expand webgl backend step coverage"` + +### Task 7: Add WASM SIMD fallback backend scaffold (single-threaded) + +**Files:** +- Create: `lib/image-pipeline/backend/wasm/wasm-backend.ts` +- Create: `lib/image-pipeline/backend/wasm/wasm-loader.ts` +- Modify: `lib/image-pipeline/backend/capabilities.ts` +- Modify: `lib/image-pipeline/backend/backend-router.ts` +- Create: `tests/image-pipeline/wasm-backend.test.ts` + +**Step 1: Write the failing test** + +Add tests proving: +- when WebGL is unavailable and WASM SIMD is available + enabled, router selects WASM +- WASM init failure downgrades to CPU with `runtime_error` +- no SharedArrayBuffer/thread requirement is introduced + +**Step 2: Run test to verify it fails** + +Run: `pnpm test tests/image-pipeline/wasm-backend.test.ts` +Expected: FAIL because WASM backend and loader do not exist. + +**Step 3: Write minimal implementation** + +Implement WASM backend scaffold and loader with capability probe, then wire into router between WebGL and CPU. + +**Step 4: Run tests to verify they pass** + +Run: `pnpm test tests/image-pipeline/wasm-backend.test.ts tests/image-pipeline/backend-router.test.ts` +Expected: PASS + +**Step 5: Commit** + +Run: +- `git add lib/image-pipeline/backend/wasm/wasm-backend.ts lib/image-pipeline/backend/wasm/wasm-loader.ts lib/image-pipeline/backend/capabilities.ts lib/image-pipeline/backend/backend-router.ts tests/image-pipeline/wasm-backend.test.ts` +- `git commit -m "feat(image-pipeline): add wasm simd fallback backend scaffold"` + +### Task 8: Verification, rollout guardrails, and documentation + +**Files:** +- Modify: `docs/plans/2026-04-04-image-pipeline-gpu-wasm-webgl-design.md` +- Modify: `docs/plans/2026-04-04-image-pipeline-gpu-wasm-webgl.md` +- Verify only: `tests/image-pipeline/*.test.ts`, `tests/image-pipeline/parity/*.test.ts`, `tests/worker-client.test.ts`, `tests/use-pipeline-preview.test.ts` + +**Step 1: Run targeted backend and parity tests** + +Run: `pnpm test tests/image-pipeline/backend-router.test.ts tests/image-pipeline/backend-capabilities.test.ts tests/image-pipeline/backend-feature-flags.test.ts tests/image-pipeline/webgl-backend-poc.test.ts tests/image-pipeline/parity/cpu-webgl-parity.test.ts tests/image-pipeline/wasm-backend.test.ts` +Expected: PASS + +**Step 2: Run integration tests around worker + preview hooks** + +Run: `pnpm test tests/worker-client.test.ts tests/use-pipeline-preview.test.ts` +Expected: PASS + +**Step 3: Run lint for touched pipeline files** + +Run: `pnpm lint lib/image-pipeline/backend/backend-types.ts lib/image-pipeline/backend/backend-router.ts lib/image-pipeline/backend/capabilities.ts lib/image-pipeline/backend/feature-flags.ts lib/image-pipeline/backend/webgl/webgl-backend.ts lib/image-pipeline/backend/wasm/wasm-backend.ts lib/image-pipeline/backend/wasm/wasm-loader.ts lib/image-pipeline/preview-renderer.ts lib/image-pipeline/bridge.ts lib/image-pipeline/worker-client.ts tests/image-pipeline/backend-router.test.ts tests/image-pipeline/backend-capabilities.test.ts tests/image-pipeline/backend-feature-flags.test.ts tests/image-pipeline/webgl-backend-poc.test.ts tests/image-pipeline/parity/cpu-webgl-parity.test.ts tests/image-pipeline/wasm-backend.test.ts tests/worker-client.test.ts tests/use-pipeline-preview.test.ts` +Expected: PASS + +**Step 4: Commit plan updates and verification notes** + +Run: +- `git add docs/plans/2026-04-04-image-pipeline-gpu-wasm-webgl-design.md docs/plans/2026-04-04-image-pipeline-gpu-wasm-webgl.md` +- `git commit -m "docs(plans): finalize image pipeline gpu wasm webgl rollout plan"`