feat: add website enrichment crawler

This commit is contained in:
2026-06-04 20:29:23 +02:00
parent ca42c8d5a6
commit 1f6e31c01c
25 changed files with 3539 additions and 56 deletions

View File

@@ -28,6 +28,7 @@ Copy `.env.example` to `.env.local` for local development. Keep real secrets out
- **SMTP / Stalwart:** `SMTP_HOST`, `SMTP_PORT`, `SMTP_USER`, `SMTP_PASSWORD`, `SMTP_FROM`
- **Rybbit:** `RYBBIT_API_URL`, `RYBBIT_API_KEY`, `NEXT_PUBLIC_RYBBIT_SITE_ID`
- **Auth:** `BETTER_AUTH_SECRET`
- **TASK-8 enrichment:** `TASK8_BROWSER_ASSET_URL`
Only variables prefixed with `NEXT_PUBLIC_` are intended for browser exposure. All API keys, SMTP credentials, and server-only URLs must stay server-side.
@@ -48,3 +49,25 @@ Only variables prefixed with `NEXT_PUBLIC_` are intended for browser exposure. A
## Deployment Notes
Coolify should run `pnpm install`, `pnpm build`, and `pnpm start`. The current font setup uses `next/font/google`, so production builds need outbound access to Google Fonts unless fonts are later self-hosted.
TASK-8 enrichment uses `playwright-core` with `@sparticuz/chromium-min` in Convex. Local `npx playwright install` is a browser-testing helper only and does not affect the Convex runtime bundle.
TASK-8 requires a browser binary source URL configured on Convex. The preferred
variable is:
- `TASK8_BROWSER_ASSET_URL` (for example your self-hosted or CDN Chromium bundle URL if you do not rely on package defaults).
For backward compatibility, the action also supports:
- `TASK8_CHROMIUM_EXECUTABLE_URL`
- `TASK8_CHROMIUM_EXECUTABLE`
If none are set, enrichment deployment/startup will fail with a clear configuration
error so no silent fallback is used.
If the URL is missing and no default is available in your environment, the enqueue action will throw a clear deploy/configuration error so enrichment does not silently fall back to a missing binary.
For TASK-8 deployment updates, run Convex restart/deploy after code changes:
- Local: `pnpm exec convex dev`
- Remote: `pnpm exec convex deploy`