diff --git a/CLAUDE.md b/CLAUDE.md index af3edf10..391b61db 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -31,46 +31,19 @@ Out-of-sync = test failures. **CRITICAL**: The `claude-win10` user has **READ-ONLY** access to production and test servers. -**Claude Code does NOT have:** +| Capability | Status | +| ---------------------- | ---------------------- | +| Root/sudo access | NO | +| Write permissions | NO | +| PM2 restart, systemctl | NO - User must execute | -- Root or sudo access -- Write permissions on servers -- Ability to execute PM2 restart, systemctl, or other write operations directly +**Server Operations Workflow**: Diagnose → User executes → Analyze → Fix (1-3 commands) → User executes → Verify -**Correct Workflow for Server Operations:** +**Rules**: -| Step | Actor | Action | -| ---- | ------ | --------------------------------------------------------------------------- | -| 1 | Claude | Provide **diagnostic commands** (read-only checks) for user to run | -| 2 | User | Execute commands on server, report results | -| 3 | Claude | Analyze results, provide **fix commands** (1-3 at a time) | -| 4 | User | Execute fix commands, report results | -| 5 | Claude | Provide **verification commands** to confirm success and check side effects | -| 6 | Claude | Document progress stage by stage | -| 7 | Claude | Update ALL relevant documentation when complete | - -**Example - Diagnosing PM2 Issues:** - -```bash -# Step 1: Claude provides diagnostic commands (user runs these) -pm2 list -pm2 logs flyer-crawler-api --lines 50 -systemctl status redis - -# Step 3: After user reports results, Claude provides fix commands -pm2 restart flyer-crawler-api -# Wait for user confirmation before next command - -# Step 5: Claude provides verification -pm2 list -curl -s https://flyer-crawler.projectium.com/api/health/ready | jq . -``` - -**Never Do:** - -- `ssh root@projectium.com "pm2 restart all"` (Claude cannot execute this) -- Assume commands succeeded without user confirmation -- Provide more than 3 fix commands at once (errors may cascade) +- Provide diagnostic commands first, wait for user to report results +- Maximum 3 fix commands at a time (errors may cascade) +- Always verify after fixes complete ### Communication Style @@ -105,12 +78,12 @@ Ask before assuming. Never assume: ### Key Patterns (with file locations) -| Pattern | ADR | Implementation | File | -| ------------------ | ------- | ------------------------------------------------- | ----------------------------------- | -| Error Handling | ADR-001 | `handleDbError()`, throw `NotFoundError` | `src/services/db/errors.db.ts` | -| Repository Methods | ADR-034 | `get*` (throws), `find*` (null), `list*` (array) | `src/services/db/*.db.ts` | -| API Responses | ADR-028 | `sendSuccess()`, `sendPaginated()`, `sendError()` | `src/utils/apiResponse.ts` | -| Transactions | ADR-002 | `withTransaction(async (client) => {...})` | `src/services/db/transaction.db.ts` | +| Pattern | ADR | Implementation | File | +| ------------------ | ------- | ------------------------------------------------- | ---------------------------------- | +| Error Handling | ADR-001 | `handleDbError()`, throw `NotFoundError` | `src/services/db/errors.db.ts` | +| Repository Methods | ADR-034 | `get*` (throws), `find*` (null), `list*` (array) | `src/services/db/*.db.ts` | +| API Responses | ADR-028 | `sendSuccess()`, `sendPaginated()`, `sendError()` | `src/utils/apiResponse.ts` | +| Transactions | ADR-002 | `withTransaction(async (client) => {...})` | `src/services/db/connection.db.ts` | ### Key Files Quick Access @@ -166,7 +139,7 @@ The dev container now matches production by using PM2 for process management. - `flyer-crawler-worker-dev` - Background job worker - `flyer-crawler-vite-dev` - Vite frontend dev server (port 5173) -### Log Aggregation (ADR-050) +### Log Aggregation (ADR-015) All logs flow to Bugsink via Logstash with 3-project routing: @@ -249,7 +222,7 @@ All logs flow to Bugsink via Logstash with 3-project routing: **Launch Pattern**: -``` +```text Use Task tool with subagent_type: "coder", "db-dev", "tester", etc. ``` @@ -330,8 +303,8 @@ podman cp "d:/path/file" container:/tmp/file **Quick Access**: -- **Dev**: https://localhost:8443 (`admin@localhost`/`admin`) -- **Prod**: https://bugsink.projectium.com +- **Dev**: (`admin@localhost`/`admin`) +- **Prod**: **Token Creation** (required for MCP): @@ -347,7 +320,7 @@ cd /opt/bugsink && bugsink-manage create_auth_token **See**: [docs/operations/LOGSTASH-QUICK-REF.md](docs/operations/LOGSTASH-QUICK-REF.md) -Log aggregation: PostgreSQL + PM2 + Redis + NGINX → Bugsink (ADR-050) +Log aggregation: PostgreSQL + PM2 + Redis + NGINX → Bugsink (ADR-015) --- @@ -367,84 +340,3 @@ Log aggregation: PostgreSQL + PM2 + Redis + NGINX → Bugsink (ADR-050) | **Logstash** | [LOGSTASH-QUICK-REF.md](docs/operations/LOGSTASH-QUICK-REF.md) | | **ADRs** | [docs/adr/index.md](docs/adr/index.md) | | **All Docs** | [docs/README.md](docs/README.md) | - ---- - -## Appendix: Integration Test Issues (Full Details) - -### 1. Vitest globalSetup Context Isolation - -Vitest's `globalSetup` runs in separate Node.js context. Singletons, spies, mocks do NOT share instances with test files. - -**Affected**: BullMQ worker service mocks (AI/DB failure tests) - -**Solutions**: Mark `.todo()`, create test-only API endpoints, use Redis-based mock flags - -```typescript -// DOES NOT WORK - different instances -const { flyerProcessingService } = await import('../../services/workers.server'); -flyerProcessingService._getAiProcessor()._setExtractAndValidateData(mockFn); -``` - -### 2. Cleanup Queue Deletes Before Verification - -Cleanup worker processes jobs in globalSetup context, ignoring test spies. - -**Solution**: Drain and pause queue: - -```typescript -const { cleanupQueue } = await import('../../services/queues.server'); -await cleanupQueue.drain(); -await cleanupQueue.pause(); -// ... test ... -await cleanupQueue.resume(); -``` - -### 3. Cache Stale After Direct SQL - -Direct `pool.query()` inserts bypass cache invalidation. - -**Solution**: `await cacheService.invalidateFlyers();` after inserts - -### 4. Test Filename Collisions - -Multer predictable filenames cause race conditions. - -**Solution**: Use unique suffix: `${Date.now()}-${Math.round(Math.random() * 1e9)}` - -### 5. Response Format Mismatches - -API formats change: `data.jobId` vs `data.job.id`, nested vs flat, string vs number IDs. - -**Solution**: Log response bodies, update assertions - -### 6. External Service Availability - -PM2/Redis health checks fail when unavailable. - -**Solution**: try/catch with graceful degradation or mock - -### 7. TZ Environment Variable Breaking Async Hooks - -**Problem**: When `TZ=America/Los_Angeles` (or other timezone values) is set in the environment, Node.js async_hooks module can produce `RangeError: Invalid triggerAsyncId value: NaN`. This breaks React Testing Library's `render()` function which uses async hooks internally. - -**Root Cause**: Setting `TZ` to certain timezone values interferes with Node.js's internal async tracking mechanism, causing invalid async IDs to be generated. - -**Symptoms**: - -```text -RangeError: Invalid triggerAsyncId value: NaN - ❯ process.env.NODE_ENV.queueSeveralMicrotasks node_modules/react/cjs/react.development.js:751:15 - ❯ process.env.NODE_ENV.exports.act node_modules/react/cjs/react.development.js:886:11 - ❯ node_modules/@testing-library/react/dist/act-compat.js:46:25 - ❯ renderRoot node_modules/@testing-library/react/dist/pure.js:189:26 -``` - -**Solution**: Explicitly unset `TZ` in all test scripts by adding `TZ=` (empty value) to cross-env: - -```json -"test:unit": "cross-env NODE_ENV=test TZ= tsx ..." -"test:integration": "cross-env NODE_ENV=test TZ= tsx ..." -``` - -**Context**: This issue was introduced in commit `d03900c` which added `TZ: 'America/Los_Angeles'` to PM2 ecosystem configs for consistent log timestamps in production/dev environments. Tests must explicitly override this to prevent the async hooks error. diff --git a/docs/AI-DOCUMENTATION-INDEX.md b/docs/AI-DOCUMENTATION-INDEX.md new file mode 100644 index 00000000..8ac5bbac --- /dev/null +++ b/docs/AI-DOCUMENTATION-INDEX.md @@ -0,0 +1,393 @@ +# AI Documentation Index + +Machine-optimized navigation for AI agents. Structured for vector retrieval and semantic search. + +--- + +## Quick Lookup Table + +| Task/Question | Primary Doc | Section/ADR | +| ----------------------- | --------------------------------------------------- | --------------------------------------- | +| Add new API endpoint | [CODE-PATTERNS.md](development/CODE-PATTERNS.md) | API Response Patterns, Input Validation | +| Add repository method | [CODE-PATTERNS.md](development/CODE-PATTERNS.md) | Repository Patterns (get*/find*/list\*) | +| Fix failing test | [TESTING.md](development/TESTING.md) | Known Integration Test Issues | +| Run tests correctly | [TESTING.md](development/TESTING.md) | Test Execution Environment | +| Add database column | [DATABASE-GUIDE.md](subagents/DATABASE-GUIDE.md) | Schema sync required | +| Deploy to production | [DEPLOYMENT.md](operations/DEPLOYMENT.md) | Application Deployment | +| Debug container issue | [DEBUGGING.md](development/DEBUGGING.md) | Container Issues | +| Configure environment | [ENVIRONMENT.md](getting-started/ENVIRONMENT.md) | Configuration by Environment | +| Add background job | [CODE-PATTERNS.md](development/CODE-PATTERNS.md) | Background Jobs | +| Handle errors correctly | [CODE-PATTERNS.md](development/CODE-PATTERNS.md) | Error Handling | +| Use transactions | [CODE-PATTERNS.md](development/CODE-PATTERNS.md) | Transaction Management | +| Add authentication | [AUTHENTICATION.md](architecture/AUTHENTICATION.md) | JWT Token Architecture | +| Cache data | [CODE-PATTERNS.md](development/CODE-PATTERNS.md) | Caching | +| Check PM2 status | [DEV-CONTAINER.md](development/DEV-CONTAINER.md) | PM2 Process Management | +| View logs | [DEBUGGING.md](development/DEBUGGING.md) | PM2 Log Access | +| Understand architecture | [OVERVIEW.md](architecture/OVERVIEW.md) | System Architecture Diagram | +| Check ADR for decision | [adr/index.md](adr/index.md) | ADR by category | +| Use subagent | [subagents/OVERVIEW.md](subagents/OVERVIEW.md) | Available Subagents | +| API versioning | [API-VERSIONING.md](development/API-VERSIONING.md) | Phase 2 infrastructure | + +--- + +## Documentation Tree + +``` +docs/ ++-- AI-DOCUMENTATION-INDEX.md # THIS FILE - AI navigation index ++-- README.md # Human-readable doc hub +| ++-- adr/ # Architecture Decision Records (57 ADRs) +| +-- index.md # ADR index by category +| +-- 0001-*.md # Standardized error handling +| +-- 0002-*.md # Transaction management (withTransaction) +| +-- 0003-*.md # Input validation (Zod middleware) +| +-- 0008-*.md # API versioning (/api/v1/) +| +-- 0014-*.md # Platform: Linux only (CRITICAL) +| +-- 0028-*.md # API response (sendSuccess/sendError) +| +-- 0034-*.md # Repository pattern (get*/find*/list*) +| +-- 0035-*.md # Service layer architecture +| +-- 0050-*.md # PostgreSQL observability + Logstash +| +-- 0057-*.md # Test remediation post-API versioning +| +-- adr-implementation-tracker.md # Implementation status +| ++-- architecture/ +| +-- OVERVIEW.md # System architecture, data flows, entities +| +-- DATABASE.md # Schema design, extensions, setup +| +-- AUTHENTICATION.md # OAuth, JWT, security features +| +-- WEBSOCKET_USAGE.md # Real-time communication patterns +| +-- api-versioning-infrastructure.md # Phase 2 versioning details +| ++-- development/ +| +-- CODE-PATTERNS.md # Error handling, repos, API responses +| +-- TESTING.md # Unit/integration/E2E, known issues +| +-- DEBUGGING.md # Container, DB, API, PM2 debugging +| +-- DEV-CONTAINER.md # PM2, Logstash, container services +| +-- API-VERSIONING.md # API versioning workflows +| +-- DESIGN_TOKENS.md # Neo-Brutalism design system +| +-- ERROR-LOGGING-PATHS.md # req.originalUrl pattern +| +-- test-path-migration.md # Test file reorganization +| ++-- getting-started/ +| +-- QUICKSTART.md # Quick setup instructions +| +-- INSTALL.md # Full installation guide +| +-- ENVIRONMENT.md # Environment variables reference +| ++-- operations/ +| +-- DEPLOYMENT.md # Production deployment guide +| +-- BARE-METAL-SETUP.md # Server provisioning +| +-- MONITORING.md # Bugsink, health checks +| +-- LOGSTASH-QUICK-REF.md # Log aggregation reference +| +-- LOGSTASH-TROUBLESHOOTING.md # Logstash debugging +| ++-- subagents/ +| +-- OVERVIEW.md # Subagent system introduction +| +-- CODER-GUIDE.md # Code development patterns +| +-- TESTER-GUIDE.md # Testing strategies +| +-- DATABASE-GUIDE.md # Database workflows +| +-- DEVOPS-GUIDE.md # Deployment/infrastructure +| +-- FRONTEND-GUIDE.md # UI/UX development +| +-- AI-USAGE-GUIDE.md # Gemini integration +| +-- DOCUMENTATION-GUIDE.md # Writing docs +| +-- SECURITY-DEBUG-GUIDE.md # Security and debugging +| ++-- tools/ +| +-- MCP-CONFIGURATION.md # MCP servers setup +| +-- BUGSINK-SETUP.md # Error tracking setup +| +-- VSCODE-SETUP.md # Editor configuration +| ++-- archive/ # Historical docs, session notes + +-- sessions/ # Development session logs + +-- plans/ # Feature implementation plans + +-- research/ # Investigation notes +``` + +--- + +## Problem-to-Document Mapping + +### Database Issues + +| Problem | Documents | +| -------------------- | ----------------------------------------------------------------------------------------------- | +| Schema out of sync | [DATABASE-GUIDE.md](subagents/DATABASE-GUIDE.md), [CLAUDE.md](../CLAUDE.md) schema sync section | +| Migration needed | [DATABASE.md](architecture/DATABASE.md), ADR-013, ADR-023 | +| Query performance | [DEBUGGING.md](development/DEBUGGING.md) Query Performance Issues | +| Connection errors | [DEBUGGING.md](development/DEBUGGING.md) Database Issues | +| Transaction patterns | [CODE-PATTERNS.md](development/CODE-PATTERNS.md) Transaction Management, ADR-002 | +| Repository methods | [CODE-PATTERNS.md](development/CODE-PATTERNS.md) Repository Patterns, ADR-034 | + +### Test Failures + +| Problem | Documents | +| ---------------------------- | --------------------------------------------------------------------- | +| Tests fail in container | [TESTING.md](development/TESTING.md), ADR-014 | +| Vitest globalSetup isolation | [CLAUDE.md](../CLAUDE.md) Integration Test Issues #1 | +| Cache stale after insert | [CLAUDE.md](../CLAUDE.md) Integration Test Issues #3 | +| Queue interference | [CLAUDE.md](../CLAUDE.md) Integration Test Issues #2 | +| API path mismatches | [TESTING.md](development/TESTING.md) API Versioning in Tests, ADR-057 | +| Type check failures | [DEBUGGING.md](development/DEBUGGING.md) Type Check Failures | +| TZ environment breaks async | [CLAUDE.md](../CLAUDE.md) Integration Test Issues #7 | + +### Deployment Issues + +| Problem | Documents | +| --------------------- | ------------------------------------------------------------------------------------- | +| PM2 not starting | [DEBUGGING.md](development/DEBUGGING.md) PM2 Process Issues | +| NGINX configuration | [DEPLOYMENT.md](operations/DEPLOYMENT.md) NGINX Configuration | +| SSL certificates | [DEBUGGING.md](development/DEBUGGING.md) SSL Certificate Issues | +| CI/CD failures | [DEPLOYMENT.md](operations/DEPLOYMENT.md) CI/CD Pipeline, ADR-017 | +| Container won't start | [DEBUGGING.md](development/DEBUGGING.md) Container Issues | +| Bugsink not receiving | [BUGSINK-SETUP.md](tools/BUGSINK-SETUP.md), [MONITORING.md](operations/MONITORING.md) | + +### Frontend/UI Changes + +| Problem | Documents | +| ------------------ | --------------------------------------------------------------- | +| Component patterns | [FRONTEND-GUIDE.md](subagents/FRONTEND-GUIDE.md), ADR-044 | +| Design tokens | [DESIGN_TOKENS.md](development/DESIGN_TOKENS.md), ADR-012 | +| State management | ADR-005, [OVERVIEW.md](architecture/OVERVIEW.md) Frontend Stack | +| Hot reload broken | [DEBUGGING.md](development/DEBUGGING.md) Frontend Issues | +| CORS errors | [DEBUGGING.md](development/DEBUGGING.md) API Calls Failing | + +### API Development + +| Problem | Documents | +| ---------------- | ------------------------------------------------------------------------------- | +| Response format | [CODE-PATTERNS.md](development/CODE-PATTERNS.md) API Response Patterns, ADR-028 | +| Input validation | [CODE-PATTERNS.md](development/CODE-PATTERNS.md) Input Validation, ADR-003 | +| Error handling | [CODE-PATTERNS.md](development/CODE-PATTERNS.md) Error Handling, ADR-001 | +| Rate limiting | ADR-032, [OVERVIEW.md](architecture/OVERVIEW.md) | +| API versioning | [API-VERSIONING.md](development/API-VERSIONING.md), ADR-008 | +| Authentication | [AUTHENTICATION.md](architecture/AUTHENTICATION.md), ADR-048 | + +### Background Jobs + +| Problem | Documents | +| ------------------- | ------------------------------------------------------------------------- | +| Jobs not processing | [DEBUGGING.md](development/DEBUGGING.md) Background Job Issues | +| Queue configuration | [CODE-PATTERNS.md](development/CODE-PATTERNS.md) Background Jobs, ADR-006 | +| Worker crashes | [DEBUGGING.md](development/DEBUGGING.md), ADR-053 | +| Scheduled jobs | ADR-037, [OVERVIEW.md](architecture/OVERVIEW.md) Scheduled Jobs | + +--- + +## Document Priority Matrix + +### CRITICAL (Read First) + +| Document | Purpose | Key Content | +| --------------------------------------------------------------- | ----------------------- | ----------------------------- | +| [CLAUDE.md](../CLAUDE.md) | AI agent instructions | Rules, patterns, known issues | +| [ADR-014](adr/0014-containerization-and-deployment-strategy.md) | Platform requirement | Tests MUST run in container | +| [DEV-CONTAINER.md](development/DEV-CONTAINER.md) | Development environment | PM2, Logstash, services | + +### HIGH (Core Development) + +| Document | Purpose | Key Content | +| --------------------------------------------------- | ----------------- | ---------------------------- | +| [CODE-PATTERNS.md](development/CODE-PATTERNS.md) | Code templates | Error handling, repos, APIs | +| [TESTING.md](development/TESTING.md) | Test execution | Commands, known issues | +| [DATABASE.md](architecture/DATABASE.md) | Schema reference | Setup, extensions, users | +| [ADR-034](adr/0034-repository-pattern-standards.md) | Repository naming | get*/find*/list\* | +| [ADR-028](adr/0028-api-response-standardization.md) | API responses | sendSuccess/sendError | +| [ADR-001](adr/0001-standardized-error-handling.md) | Error handling | handleDbError, NotFoundError | + +### MEDIUM (Specialized Tasks) + +| Document | Purpose | Key Content | +| --------------------------------------------------- | --------------------- | ------------------------ | +| [subagents/OVERVIEW.md](subagents/OVERVIEW.md) | Subagent selection | When to delegate | +| [DEPLOYMENT.md](operations/DEPLOYMENT.md) | Production deployment | PM2, NGINX, CI/CD | +| [DEBUGGING.md](development/DEBUGGING.md) | Troubleshooting | Common issues, solutions | +| [ENVIRONMENT.md](getting-started/ENVIRONMENT.md) | Config reference | Variables by environment | +| [AUTHENTICATION.md](architecture/AUTHENTICATION.md) | Auth patterns | OAuth, JWT, security | +| [API-VERSIONING.md](development/API-VERSIONING.md) | Versioning | /api/v1/ prefix | + +### LOW (Reference/Historical) + +| Document | Purpose | Key Content | +| -------------------- | ------------------ | ------------------------- | +| [archive/](archive/) | Historical docs | Session notes, old plans | +| ADR-013, ADR-023 | Migration strategy | Proposed, not implemented | +| ADR-024 | Feature flags | Proposed | +| ADR-025 | i18n/l10n | Proposed | + +--- + +## Cross-Reference Matrix + +| Document | References | Referenced By | +| -------------------- | ------------------------------------------------------------------------------- | ------------------------------------------------------ | +| **CLAUDE.md** | ADR-001, ADR-002, ADR-008, ADR-014, ADR-028, ADR-034, ADR-035, ADR-050, ADR-057 | All development docs | +| **ADR-008** | ADR-028 | API-VERSIONING.md, TESTING.md, ADR-057 | +| **ADR-014** | - | CLAUDE.md, TESTING.md, DEPLOYMENT.md, DEV-CONTAINER.md | +| **ADR-028** | ADR-001 | CODE-PATTERNS.md, OVERVIEW.md | +| **ADR-034** | ADR-001 | CODE-PATTERNS.md, DATABASE-GUIDE.md | +| **ADR-057** | ADR-008, ADR-028 | TESTING.md | +| **CODE-PATTERNS.md** | ADR-001, ADR-002, ADR-003, ADR-028, ADR-034, ADR-036, ADR-048 | CODER-GUIDE.md | +| **TESTING.md** | ADR-014, ADR-057, CLAUDE.md | TESTER-GUIDE.md, DEBUGGING.md | +| **DEBUGGING.md** | DEV-CONTAINER.md, TESTING.md, MONITORING.md | DEVOPS-GUIDE.md | +| **DEV-CONTAINER.md** | ADR-014, ADR-050, ecosystem.dev.config.cjs | DEBUGGING.md, CLAUDE.md | +| **OVERVIEW.md** | ADR-001 through ADR-050+ | All architecture docs | +| **DATABASE.md** | ADR-002, ADR-013, ADR-055 | DATABASE-GUIDE.md | + +--- + +## Navigation Patterns + +### Adding a Feature + +``` +1. CLAUDE.md -> Project rules, patterns +2. CODE-PATTERNS.md -> Implementation templates +3. Relevant subagent guide -> Domain-specific patterns +4. Related ADRs -> Design decisions +5. TESTING.md -> Test requirements +``` + +### Fixing a Bug + +``` +1. DEBUGGING.md -> Common issues checklist +2. TESTING.md -> Run tests in container +3. Error logs (pm2/bugsink) -> Identify root cause +4. CODE-PATTERNS.md -> Correct pattern reference +5. Related ADR -> Architectural context +``` + +### Deploying + +``` +1. DEPLOYMENT.md -> Deployment procedures +2. ENVIRONMENT.md -> Required variables +3. MONITORING.md -> Health check verification +4. LOGSTASH-QUICK-REF.md -> Log aggregation check +``` + +### Database Changes + +``` +1. DATABASE-GUIDE.md -> Schema sync requirements (CRITICAL) +2. DATABASE.md -> Schema design patterns +3. ADR-002 -> Transaction patterns +4. ADR-034 -> Repository methods +5. ADR-055 -> Normalization rules +``` + +### Subagent Selection + +| Task Type | Subagent | Guide | +| --------------------- | ------------------------- | ------------------------------------------------------------ | +| Write production code | `coder` | [CODER-GUIDE.md](subagents/CODER-GUIDE.md) | +| Database changes | `db-dev` | [DATABASE-GUIDE.md](subagents/DATABASE-GUIDE.md) | +| Create tests | `testwriter` | [TESTER-GUIDE.md](subagents/TESTER-GUIDE.md) | +| Fix failing tests | `tester` | [TESTER-GUIDE.md](subagents/TESTER-GUIDE.md) | +| Container/deployment | `devops` | [DEVOPS-GUIDE.md](subagents/DEVOPS-GUIDE.md) | +| UI components | `frontend-specialist` | [FRONTEND-GUIDE.md](subagents/FRONTEND-GUIDE.md) | +| External APIs | `integrations-specialist` | - | +| Security review | `security-engineer` | [SECURITY-DEBUG-GUIDE.md](subagents/SECURITY-DEBUG-GUIDE.md) | +| Production errors | `log-debug` | [SECURITY-DEBUG-GUIDE.md](subagents/SECURITY-DEBUG-GUIDE.md) | +| AI/Gemini issues | `ai-usage` | [AI-USAGE-GUIDE.md](subagents/AI-USAGE-GUIDE.md) | + +--- + +## Key File Quick Reference + +### Configuration + +| File | Purpose | +| -------------------------- | ---------------------------- | +| `server.ts` | Express app setup | +| `src/config/env.ts` | Environment validation (Zod) | +| `ecosystem.dev.config.cjs` | PM2 dev config | +| `ecosystem.config.cjs` | PM2 prod config | +| `vite.config.ts` | Vite build config | + +### Core Implementation + +| File | Purpose | +| ----------------------------------- | ----------------------------------- | +| `src/routes/*.routes.ts` | API route handlers | +| `src/services/db/*.db.ts` | Repository layer | +| `src/services/*.server.ts` | Server-only services | +| `src/services/queues.server.ts` | BullMQ queue definitions | +| `src/services/workers.server.ts` | BullMQ workers | +| `src/utils/apiResponse.ts` | sendSuccess/sendError/sendPaginated | +| `src/services/db/errors.db.ts` | handleDbError, NotFoundError | +| `src/services/db/transaction.db.ts` | withTransaction | + +### Database Schema + +| File | Purpose | +| ------------------------------ | ----------------------------------- | +| `sql/master_schema_rollup.sql` | Test DB, complete reference | +| `sql/initial_schema.sql` | Fresh install (identical to rollup) | +| `sql/migrations/*.sql` | Production ALTER statements | + +### Testing + +| File | Purpose | +| ---------------------------------- | ----------------------- | +| `vitest.config.ts` | Unit test config | +| `vitest.config.integration.ts` | Integration test config | +| `vitest.config.e2e.ts` | E2E test config | +| `src/tests/utils/mockFactories.ts` | Mock data factories | +| `src/tests/utils/storeHelpers.ts` | Store test helpers | + +--- + +## ADR Quick Reference + +### By Implementation Status + +**Implemented**: 001, 002, 003, 004, 006, 008, 009, 010, 016, 017, 020, 021, 028, 032, 033, 034, 035, 036, 037, 038, 040, 041, 043, 044, 045, 046, 050, 051, 052, 055, 057 + +**Partially Implemented**: 012, 014, 015, 048 + +**Proposed**: 011, 013, 022, 023, 024, 025, 029, 030, 031, 039, 047, 053, 054, 056 + +### By Category + +| Category | ADRs | +| --------------------- | ------------------------------------------- | +| Core Infrastructure | 002, 007, 020, 030 | +| Data Management | 009, 013, 019, 023, 031, 055 | +| API & Integration | 003, 008, 018, 022, 028 | +| Security | 001, 011, 016, 029, 032, 033, 048 | +| Observability | 004, 015, 050, 051, 052, 056 | +| Deployment & Ops | 006, 014, 017, 024, 037, 038, 053, 054 | +| Frontend/UI | 005, 012, 025, 026, 044 | +| Dev Workflow | 010, 021, 027, 040, 045, 047, 057 | +| Architecture Patterns | 034, 035, 036, 039, 041, 042, 043, 046, 049 | + +--- + +## Essential Commands + +```bash +# Run all tests (MUST use container) +podman exec -it flyer-crawler-dev npm test + +# Run unit tests +podman exec -it flyer-crawler-dev npm run test:unit + +# Run type check +podman exec -it flyer-crawler-dev npm run type-check + +# Run integration tests +podman exec -it flyer-crawler-dev npm run test:integration + +# PM2 status +podman exec -it flyer-crawler-dev pm2 status + +# PM2 logs +podman exec -it flyer-crawler-dev pm2 logs + +# Restart all processes +podman exec -it flyer-crawler-dev pm2 restart all +``` + +--- + +_This index is optimized for AI agent consumption. Updated: 2026-01-28_ diff --git a/docs/adr/0008-api-versioning-strategy.md b/docs/adr/0008-api-versioning-strategy.md index c82b147f..a73b6adb 100644 --- a/docs/adr/0008-api-versioning-strategy.md +++ b/docs/adr/0008-api-versioning-strategy.md @@ -316,6 +316,7 @@ app.use('/api/v1', (req, res, next) => { - [ADR-018](./0018-api-documentation-strategy.md) - API Documentation Strategy (versioned OpenAPI specs) - [ADR-028](./0028-api-response-standardization.md) - Response Standardization (envelope pattern applies to all versions) - [ADR-016](./0016-api-security-hardening.md) - Security Hardening (applies to all versions) +- [ADR-057](./0057-test-remediation-post-api-versioning.md) - Test Remediation Post-API Versioning (documents test migration) ## Implementation Checklist diff --git a/docs/adr/0010-testing-strategy-and-standards.md b/docs/adr/0010-testing-strategy-and-standards.md index c7579b95..905e1c61 100644 --- a/docs/adr/0010-testing-strategy-and-standards.md +++ b/docs/adr/0010-testing-strategy-and-standards.md @@ -363,6 +363,13 @@ The following files contain acknowledged code smell violations that are deferred - `src/tests/utils/mockFactories.ts` - Mock factories (1553 lines) - `src/tests/utils/testHelpers.ts` - Test utilities +## Related ADRs + +- [ADR-014](./0014-containerization-and-deployment-strategy.md) - Containerization (tests must run in dev container) +- [ADR-040](./0040-testing-economics-and-priorities.md) - Testing Economics and Priorities +- [ADR-045](./0045-test-data-factories-and-fixtures.md) - Test Data Factories and Fixtures +- [ADR-057](./0057-test-remediation-post-api-versioning.md) - Test Remediation Post-API Versioning + ## Future Enhancements 1. **Browser E2E Tests**: Consider adding Playwright for actual browser testing diff --git a/docs/adr/0013-database-schema-migration-strategy.md b/docs/adr/0013-database-schema-migration-strategy.md index fb29d1d5..28e40b59 100644 --- a/docs/adr/0013-database-schema-migration-strategy.md +++ b/docs/adr/0013-database-schema-migration-strategy.md @@ -2,7 +2,9 @@ **Date**: 2025-12-12 -**Status**: Proposed +**Status**: Superseded by [ADR-023](./0023-database-schema-migration-strategy.md) + +**Note**: This ADR was an early draft. ADR-023 provides a more detailed specification for the same topic. ## Context diff --git a/docs/adr/0023-database-schema-migration-strategy.md b/docs/adr/0023-database-schema-migration-strategy.md index 0d392852..0efe8350 100644 --- a/docs/adr/0023-database-schema-migration-strategy.md +++ b/docs/adr/0023-database-schema-migration-strategy.md @@ -4,6 +4,8 @@ **Status**: Proposed +**Supersedes**: [ADR-013](./0013-database-schema-migration-strategy.md) + ## Context The `README.md` indicates that the database schema is managed by manually running a large `schema.sql.txt` file. This approach is highly error-prone, makes tracking changes difficult, and is not feasible for updating a live production database without downtime or data loss. diff --git a/docs/adr/0040-testing-economics-and-priorities.md b/docs/adr/0040-testing-economics-and-priorities.md index 5f9b2209..39f8241f 100644 --- a/docs/adr/0040-testing-economics-and-priorities.md +++ b/docs/adr/0040-testing-economics-and-priorities.md @@ -195,6 +195,12 @@ Do NOT add tests: - Coverage percentages may not satisfy external audits - Requires judgment calls that may be inconsistent +## Related ADRs + +- [ADR-010](./0010-testing-strategy-and-standards.md) - Testing Strategy and Standards (this ADR extends ADR-010) +- [ADR-045](./0045-test-data-factories-and-fixtures.md) - Test Data Factories and Fixtures +- [ADR-057](./0057-test-remediation-post-api-versioning.md) - Test Remediation Post-API Versioning + ## Key Files - `docs/adr/0010-testing-strategy-and-standards.md` - Testing mechanics diff --git a/docs/adr/0050-postgresql-function-observability.md b/docs/adr/0050-postgresql-function-observability.md index d26df29b..c76ec0ff 100644 --- a/docs/adr/0050-postgresql-function-observability.md +++ b/docs/adr/0050-postgresql-function-observability.md @@ -4,7 +4,7 @@ **Status**: Accepted (Fully Implemented) -**Related**: [ADR-015](0015-application-performance-monitoring-and-error-tracking.md), [ADR-004](0004-standardized-application-wide-structured-logging.md) +**Related**: [ADR-015](0015-error-tracking-and-observability.md), [ADR-004](0004-standardized-application-wide-structured-logging.md) ## Context @@ -335,7 +335,7 @@ SELECT award_achievement('user-uuid', 'Nonexistent Badge'); ## References -- [ADR-015: Application Performance Monitoring](0015-application-performance-monitoring-and-error-tracking.md) +- [ADR-015: Error Tracking and Observability](0015-error-tracking-and-observability.md) - [ADR-004: Standardized Structured Logging](0004-standardized-application-wide-structured-logging.md) - [PostgreSQL RAISE Documentation](https://www.postgresql.org/docs/current/plpgsql-errors-and-messages.html) - [PostgreSQL Logging Configuration](https://www.postgresql.org/docs/current/runtime-config-logging.html) diff --git a/docs/adr/0054-bugsink-gitea-issue-sync.md b/docs/adr/0054-bugsink-gitea-issue-sync.md index c0a6f6ec..bdbbf67f 100644 --- a/docs/adr/0054-bugsink-gitea-issue-sync.md +++ b/docs/adr/0054-bugsink-gitea-issue-sync.md @@ -332,6 +332,6 @@ Response: ## References - [ADR-006: Background Job Processing](./0006-background-job-processing-and-task-queues.md) -- [ADR-015: Application Performance Monitoring](./0015-application-performance-monitoring-and-error-tracking.md) +- [ADR-015: Error Tracking and Observability](./0015-error-tracking-and-observability.md) - [Bugsink API Documentation](https://bugsink.com/docs/api/) - [Gitea API Documentation](https://docs.gitea.io/en-us/api-usage/) diff --git a/docs/adr/adr-implementation-tracker.md b/docs/adr/adr-implementation-tracker.md index c02b4f79..6f8fe857 100644 --- a/docs/adr/adr-implementation-tracker.md +++ b/docs/adr/adr-implementation-tracker.md @@ -15,9 +15,10 @@ This document tracks the implementation status and estimated effort for all Arch | Status | Count | | ---------------------------- | ----- | -| Accepted (Fully Implemented) | 40 | +| Accepted (Fully Implemented) | 41 | | Partially Implemented | 2 | -| Proposed (Not Started) | 14 | +| Proposed (Not Started) | 13 | +| Superseded | 1 | --- @@ -34,13 +35,13 @@ This document tracks the implementation status and estimated effort for all Arch ### Category 2: Data Management -| ADR | Title | Status | Effort | Notes | -| --------------------------------------------------------------- | ------------------------ | -------- | ------ | ------------------------------ | -| [ADR-009](./0009-caching-strategy-for-read-heavy-operations.md) | Caching Strategy | Accepted | - | Fully implemented | -| [ADR-013](./0013-database-schema-migration-strategy.md) | Schema Migrations v1 | Proposed | M | Superseded by ADR-023 | -| [ADR-019](./0019-data-backup-and-recovery-strategy.md) | Backup & Recovery | Accepted | - | Fully implemented | -| [ADR-023](./0023-database-schema-migration-strategy.md) | Schema Migrations v2 | Proposed | L | Requires tooling setup | -| [ADR-031](./0031-data-retention-and-privacy-compliance.md) | Data Retention & Privacy | Proposed | XL | Legal/compliance review needed | +| ADR | Title | Status | Effort | Notes | +| --------------------------------------------------------------- | ------------------------ | ---------- | ------ | ------------------------------ | +| [ADR-009](./0009-caching-strategy-for-read-heavy-operations.md) | Caching Strategy | Accepted | - | Fully implemented | +| [ADR-013](./0013-database-schema-migration-strategy.md) | Schema Migrations v1 | Superseded | - | Superseded by ADR-023 | +| [ADR-019](./0019-data-backup-and-recovery-strategy.md) | Backup & Recovery | Accepted | - | Fully implemented | +| [ADR-023](./0023-database-schema-migration-strategy.md) | Schema Migrations v2 | Proposed | L | Requires tooling setup | +| [ADR-031](./0031-data-retention-and-privacy-compliance.md) | Data Retention & Privacy | Proposed | XL | Legal/compliance review needed | ### Category 3: API & Integration @@ -108,6 +109,7 @@ This document tracks the implementation status and estimated effort for all Arch | [ADR-040](./0040-testing-economics-and-priorities.md) | Testing Economics | Accepted | - | Fully implemented | | [ADR-045](./0045-test-data-factories-and-fixtures.md) | Test Data Factories | Accepted | - | Fully implemented | | [ADR-047](./0047-project-file-and-folder-organization.md) | Project Organization | Proposed | XL | Major reorganization | +| [ADR-057](./0057-test-remediation-post-api-versioning.md) | Test Remediation | Accepted | - | Fully implemented | ### Category 9: Architecture Patterns @@ -148,6 +150,8 @@ These ADRs are proposed or partially implemented, ordered by suggested implement | Date | ADR | Change | | ---------- | ------- | ----------------------------------------------------------------------------------- | +| 2026-01-28 | ADR-057 | Created - Test remediation documentation for ADR-008 Phase 2 migration | +| 2026-01-28 | ADR-013 | Marked as Superseded by ADR-023 | | 2026-01-27 | ADR-008 | Test path migration complete - 23 files, ~70 paths updated, 274->345 tests passing | | 2026-01-27 | ADR-008 | Phase 2 Complete - Version router factory, deprecation headers, 82 versioning tests | | 2026-01-26 | ADR-015 | Completed - Added Sentry user context in AuthProvider, now fully implemented | diff --git a/docs/adr/index.md b/docs/adr/index.md index 52448bcf..fd7f618b 100644 --- a/docs/adr/index.md +++ b/docs/adr/index.md @@ -2,6 +2,8 @@ This directory contains a log of the architectural decisions made for the Flyer Crawler project. +**[Implementation Tracker](./adr-implementation-tracker.md)**: Track implementation status and effort estimates for all ADRs. + ## 1. Foundational / Core Infrastructure **[ADR-002](./0002-standardized-transaction-management.md)**: Standardized Transaction Management and Unit of Work Pattern (Accepted) @@ -12,7 +14,7 @@ This directory contains a log of the architectural decisions made for the Flyer ## 2. Data Management **[ADR-009](./0009-caching-strategy-for-read-heavy-operations.md)**: Caching Strategy for Read-Heavy Operations (Accepted) -**[ADR-013](./0013-database-schema-migration-strategy.md)**: Database Schema Migration Strategy (Proposed) +**[ADR-013](./0013-database-schema-migration-strategy.md)**: Database Schema Migration Strategy (Superseded by ADR-023) **[ADR-019](./0019-data-backup-and-recovery-strategy.md)**: Data Backup and Recovery Strategy (Accepted) **[ADR-023](./0023-database-schema-migration-strategy.md)**: Database Schema Migration Strategy (Proposed) **[ADR-031](./0031-data-retention-and-privacy-compliance.md)**: Data Retention and Privacy Compliance (Proposed) @@ -20,9 +22,9 @@ This directory contains a log of the architectural decisions made for the Flyer ## 3. API & Integration **[ADR-003](./0003-standardized-input-validation-using-middleware.md)**: Standardized Input Validation using Middleware (Accepted) -**[ADR-008](./0008-api-versioning-strategy.md)**: API Versioning Strategy (Accepted - Phase 1 Complete) +**[ADR-008](./0008-api-versioning-strategy.md)**: API Versioning Strategy (Accepted - Phase 2 Complete) **[ADR-018](./0018-api-documentation-strategy.md)**: API Documentation Strategy (Accepted) -**[ADR-022](./0022-real-time-notification-system.md)**: Real-time Notification System (Proposed) +**[ADR-022](./0022-real-time-notification-system.md)**: Real-time Notification System (Accepted) **[ADR-028](./0028-api-response-standardization.md)**: API Response Standardization and Envelope Pattern (Implemented) ## 4. Security & Compliance @@ -33,12 +35,12 @@ This directory contains a log of the architectural decisions made for the Flyer **[ADR-029](./0029-secret-rotation-and-key-management.md)**: Secret Rotation and Key Management Strategy (Proposed) **[ADR-032](./0032-rate-limiting-strategy.md)**: Rate Limiting Strategy (Accepted) **[ADR-033](./0033-file-upload-and-storage-strategy.md)**: File Upload and Storage Strategy (Accepted) -**[ADR-048](./0048-authentication-strategy.md)**: Authentication Strategy (Partially Implemented) +**[ADR-048](./0048-authentication-strategy.md)**: Authentication Strategy (Accepted) ## 5. Observability & Monitoring **[ADR-004](./0004-standardized-application-wide-structured-logging.md)**: Standardized Application-Wide Structured Logging (Accepted) -**[ADR-015](./0015-error-tracking-and-observability.md)**: Error Tracking and Observability (Partial) +**[ADR-015](./0015-error-tracking-and-observability.md)**: Error Tracking and Observability (Accepted) **[ADR-050](./0050-postgresql-function-observability.md)**: PostgreSQL Function Observability (Accepted) **[ADR-051](./0051-asynchronous-context-propagation.md)**: Asynchronous Context Propagation (Accepted) **[ADR-052](./0052-granular-debug-logging-strategy.md)**: Granular Debug Logging Strategy (Accepted) @@ -52,7 +54,7 @@ This directory contains a log of the architectural decisions made for the Flyer **[ADR-024](./0024-feature-flagging-strategy.md)**: Feature Flagging Strategy (Proposed) **[ADR-037](./0037-scheduled-jobs-and-cron-pattern.md)**: Scheduled Jobs and Cron Pattern (Accepted) **[ADR-038](./0038-graceful-shutdown-pattern.md)**: Graceful Shutdown Pattern (Accepted) -**[ADR-053](./0053-worker-health-checks-and-monitoring.md)**: Worker Health Checks and Monitoring (Proposed) +**[ADR-053](./0053-worker-health-checks.md)**: Worker Health Checks and Stalled Job Monitoring (Accepted) **[ADR-054](./0054-bugsink-gitea-issue-sync.md)**: Bugsink to Gitea Issue Synchronization (Proposed) ## 7. Frontend / User Interface diff --git a/docs/architecture/DATABASE.md b/docs/architecture/DATABASE.md index 539759bf..af89d5b2 100644 --- a/docs/architecture/DATABASE.md +++ b/docs/architecture/DATABASE.md @@ -1,10 +1,168 @@ -# Database Setup +# Database Architecture -Flyer Crawler uses PostgreSQL with several extensions for full-text search, geographic data, and UUID generation. +**Version**: 0.12.20 +**Last Updated**: 2026-01-28 + +Flyer Crawler uses PostgreSQL 16 with PostGIS for geographic data, pg_trgm for fuzzy text search, and uuid-ossp for UUID generation. The database contains 65 tables organized into logical domains. + +## Table of Contents + +1. [Schema Overview](#schema-overview) +2. [Database Setup](#database-setup) +3. [Schema Reference](#schema-reference) +4. [Related Documentation](#related-documentation) --- -## Required Extensions +## Schema Overview + +The database is organized into the following domains: + +### Core Infrastructure (6 tables) + +| Table | Purpose | Primary Key | +| ----------------------- | ----------------------------------------- | ----------------- | +| `users` | Authentication credentials and login data | `user_id` (UUID) | +| `profiles` | Public user data, preferences, points | `user_id` (UUID) | +| `addresses` | Normalized address storage with geocoding | `address_id` | +| `activity_log` | User activity audit trail | `activity_log_id` | +| `password_reset_tokens` | Temporary tokens for password reset | `token_id` | +| `schema_info` | Schema deployment metadata | `environment` | + +### Stores and Locations (4 tables) + +| Table | Purpose | Primary Key | +| ------------------------ | --------------------------------------- | ------------------- | +| `stores` | Grocery store chains (Safeway, Kroger) | `store_id` | +| `store_locations` | Physical store locations with addresses | `store_location_id` | +| `favorite_stores` | User store favorites | `user_id, store_id` | +| `store_receipt_patterns` | Receipt text patterns for store ID | `pattern_id` | + +### Flyers and Items (7 tables) + +| Table | Purpose | Primary Key | +| ----------------------- | -------------------------------------- | ------------------------ | +| `flyers` | Uploaded flyer metadata and status | `flyer_id` | +| `flyer_items` | Individual deals extracted from flyers | `flyer_item_id` | +| `flyer_locations` | Flyer-to-location associations | `flyer_location_id` | +| `categories` | Item categorization (Produce, Dairy) | `category_id` | +| `master_grocery_items` | Canonical grocery item dictionary | `master_grocery_item_id` | +| `master_item_aliases` | Alternative names for master items | `alias_id` | +| `unmatched_flyer_items` | Items pending master item matching | `unmatched_item_id` | + +### Products and Brands (2 tables) + +| Table | Purpose | Primary Key | +| ---------- | ---------------------------------------------- | ------------ | +| `brands` | Brand names (Coca-Cola, Kraft) | `brand_id` | +| `products` | Specific products (master item + brand + size) | `product_id` | + +### Price Tracking (3 tables) + +| Table | Purpose | Primary Key | +| ----------------------- | ---------------------------------- | ------------------ | +| `item_price_history` | Historical prices for master items | `price_history_id` | +| `user_submitted_prices` | User-contributed price reports | `submission_id` | +| `suggested_corrections` | Suggested edits to flyer items | `correction_id` | + +### User Features (8 tables) + +| Table | Purpose | Primary Key | +| -------------------- | ------------------------------------ | --------------------------- | +| `user_watched_items` | Items user wants to track prices for | `user_watched_item_id` | +| `user_alerts` | Price alert thresholds | `alert_id` | +| `notifications` | User notifications | `notification_id` | +| `user_item_aliases` | User-defined item name aliases | `alias_id` | +| `user_follows` | User-to-user follow relationships | `follower_id, following_id` | +| `user_reactions` | Reactions to content (likes, etc.) | `reaction_id` | +| `budgets` | User-defined spending budgets | `budget_id` | +| `search_queries` | Search history for analytics | `query_id` | + +### Shopping Lists (4 tables) + +| Table | Purpose | Primary Key | +| ----------------------- | ------------------------ | ------------------------- | +| `shopping_lists` | User shopping lists | `shopping_list_id` | +| `shopping_list_items` | Items on shopping lists | `shopping_list_item_id` | +| `shared_shopping_lists` | Shopping list sharing | `shared_shopping_list_id` | +| `shopping_trips` | Completed shopping trips | `trip_id` | +| `shopping_trip_items` | Items purchased on trips | `trip_item_id` | + +### Recipes (11 tables) + +| Table | Purpose | Primary Key | +| --------------------------------- | -------------------------------- | ------------------------- | +| `recipes` | User recipes with metadata | `recipe_id` | +| `recipe_ingredients` | Recipe ingredient list | `recipe_ingredient_id` | +| `recipe_ingredient_substitutions` | Ingredient alternatives | `substitution_id` | +| `tags` | Recipe tags (vegan, quick, etc.) | `tag_id` | +| `recipe_tags` | Recipe-to-tag associations | `recipe_id, tag_id` | +| `appliances` | Kitchen appliances | `appliance_id` | +| `recipe_appliances` | Appliances needed for recipes | `recipe_id, appliance_id` | +| `recipe_ratings` | User ratings for recipes | `rating_id` | +| `recipe_comments` | User comments on recipes | `comment_id` | +| `favorite_recipes` | User recipe favorites | `user_id, recipe_id` | +| `recipe_collections` | User recipe collections | `collection_id` | + +### Meal Planning (3 tables) + +| Table | Purpose | Primary Key | +| ------------------- | -------------------------- | ----------------- | +| `menu_plans` | Weekly/monthly meal plans | `menu_plan_id` | +| `shared_menu_plans` | Menu plan sharing | `share_id` | +| `planned_meals` | Individual meals in a plan | `planned_meal_id` | + +### Pantry and Inventory (4 tables) + +| Table | Purpose | Primary Key | +| -------------------- | ------------------------------------ | ----------------- | +| `pantry_items` | User pantry inventory | `pantry_item_id` | +| `pantry_locations` | Storage locations (fridge, freezer) | `location_id` | +| `expiry_date_ranges` | Reference shelf life data | `expiry_range_id` | +| `expiry_alerts` | User expiry notification preferences | `expiry_alert_id` | +| `expiry_alert_log` | Sent expiry notifications | `alert_log_id` | + +### Receipts (4 tables) + +| Table | Purpose | Primary Key | +| ------------------------ | ----------------------------- | ----------------- | +| `receipts` | Scanned receipt metadata | `receipt_id` | +| `receipt_items` | Items parsed from receipts | `receipt_item_id` | +| `receipt_processing_log` | OCR/AI processing audit trail | `log_id` | + +### UPC Scanning (2 tables) + +| Table | Purpose | Primary Key | +| ---------------------- | ------------------------------- | ----------- | +| `upc_scan_history` | User barcode scan history | `scan_id` | +| `upc_external_lookups` | External UPC API response cache | `lookup_id` | + +### Gamification (2 tables) + +| Table | Purpose | Primary Key | +| ------------------- | ---------------------------- | ------------------------- | +| `achievements` | Defined achievements | `achievement_id` | +| `user_achievements` | Achievements earned by users | `user_id, achievement_id` | + +### User Preferences (3 tables) + +| Table | Purpose | Primary Key | +| --------------------------- | ---------------------------- | ------------------------- | +| `dietary_restrictions` | Defined dietary restrictions | `restriction_id` | +| `user_dietary_restrictions` | User dietary preferences | `user_id, restriction_id` | +| `user_appliances` | Appliances user owns | `user_id, appliance_id` | + +### Reference Data (1 table) + +| Table | Purpose | Primary Key | +| ------------------ | ----------------------- | --------------- | +| `unit_conversions` | Unit conversion factors | `conversion_id` | + +--- + +## Database Setup + +### Required Extensions | Extension | Purpose | | ----------- | ------------------------------------------- | @@ -14,7 +172,7 @@ Flyer Crawler uses PostgreSQL with several extensions for full-text search, geog --- -## Database Users +### Database Users This project uses **environment-specific database users** to isolate production and test environments: diff --git a/docs/architecture/OVERVIEW.md b/docs/architecture/OVERVIEW.md index 4681640e..df91fc06 100644 --- a/docs/architecture/OVERVIEW.md +++ b/docs/architecture/OVERVIEW.md @@ -1,7 +1,7 @@ # Flyer Crawler - System Architecture Overview -**Version**: 0.12.5 -**Last Updated**: 2026-01-22 +**Version**: 0.12.20 +**Last Updated**: 2026-01-28 **Platform**: Linux (Production and Development) --- @@ -41,7 +41,7 @@ ## System Architecture Diagram -``` +```text +-----------------------------------------------------------------------------------+ | CLIENT LAYER | +-----------------------------------------------------------------------------------+ @@ -153,10 +153,10 @@ | Component | Technology | Version | Purpose | | ---------------------- | ---------- | -------- | -------------------------------- | | **Runtime** | Node.js | 22.x LTS | Server-side JavaScript runtime | -| **Language** | TypeScript | 5.9.x | Type-safe JavaScript superset | -| **Web Framework** | Express.js | 5.1.x | HTTP server and routing | -| **Frontend Framework** | React | 19.2.x | UI component library | -| **Build Tool** | Vite | 7.2.x | Frontend bundling and dev server | +| **Language** | TypeScript | 5.9.3 | Type-safe JavaScript superset | +| **Web Framework** | Express.js | 5.1.0 | HTTP server and routing | +| **Frontend Framework** | React | 19.2.0 | UI component library | +| **Build Tool** | Vite | 7.2.4 | Frontend bundling and dev server | ### Data Storage @@ -176,23 +176,23 @@ | **OAuth** | Google, GitHub | Social authentication | | **Email** | Nodemailer (SMTP) | Transactional emails | -### Background Processing +### Background Processing Stack | Component | Technology | Version | Purpose | | ------------------- | ---------- | ------- | --------------------------------- | -| **Job Queues** | BullMQ | 5.65.x | Reliable async job processing | +| **Job Queues** | BullMQ | 5.65.1 | Reliable async job processing | | **Process Manager** | PM2 | Latest | Process management and clustering | -| **Scheduler** | node-cron | 4.2.x | Scheduled tasks | +| **Scheduler** | node-cron | 4.2.1 | Scheduled tasks | ### Frontend Stack | Component | Technology | Version | Purpose | | -------------------- | -------------- | ------- | ---------------------------------------- | -| **State Management** | TanStack Query | 5.90.x | Server state caching and synchronization | -| **Routing** | React Router | 7.9.x | Client-side routing | -| **Styling** | Tailwind CSS | 4.1.x | Utility-first CSS framework | -| **Icons** | Lucide React | 0.555.x | Icon components | -| **Charts** | Recharts | 3.4.x | Data visualization | +| **State Management** | TanStack Query | 5.90.12 | Server state caching and synchronization | +| **Routing** | React Router | 7.9.6 | Client-side routing | +| **Styling** | Tailwind CSS | 4.1.17 | Utility-first CSS framework | +| **Icons** | Lucide React | 0.555.0 | Icon components | +| **Charts** | Recharts | 3.4.1 | Data visualization | ### Observability and Quality @@ -221,7 +221,7 @@ The frontend is a single-page application (SPA) built with React 19 and Vite. **Directory Structure**: -``` +```text src/ +-- components/ # Reusable UI components +-- contexts/ # React context providers @@ -244,17 +244,30 @@ The backend is a RESTful API server built with Express.js 5. - Structured logging with Pino - Standardized error handling (ADR-001) -**API Route Modules**: -| Route | Purpose | -|-------|---------| -| `/api/auth` | Authentication (login, register, OAuth) | -| `/api/users` | User profile management | -| `/api/flyers` | Flyer CRUD and processing | -| `/api/recipes` | Recipe management | -| `/api/deals` | Best prices and deal discovery | -| `/api/stores` | Store management | -| `/api/admin` | Administrative functions | -| `/api/health` | Health checks and monitoring | +**API Route Modules** (all versioned under `/api/v1/*`): + +| Route | Purpose | +| ------------------------- | ----------------------------------------------- | +| `/api/v1/auth` | Authentication (login, register, OAuth) | +| `/api/v1/health` | Health checks and monitoring | +| `/api/v1/system` | System administration (PM2 status, server info) | +| `/api/v1/users` | User profile management | +| `/api/v1/ai` | AI-powered features and flyer processing | +| `/api/v1/admin` | Administrative functions | +| `/api/v1/budgets` | Budget management and spending analysis | +| `/api/v1/achievements` | Gamification and achievement system | +| `/api/v1/flyers` | Flyer CRUD and processing | +| `/api/v1/recipes` | Recipe management and recommendations | +| `/api/v1/personalization` | Master items and user preferences | +| `/api/v1/price-history` | Price tracking and trend analysis | +| `/api/v1/stats` | Public statistics and analytics | +| `/api/v1/upc` | UPC barcode scanning and product lookup | +| `/api/v1/inventory` | Inventory and expiry tracking | +| `/api/v1/receipts` | Receipt scanning and purchase history | +| `/api/v1/deals` | Best prices and deal discovery | +| `/api/v1/reactions` | Social features (reactions, sharing) | +| `/api/v1/stores` | Store management and location services | +| `/api/v1/categories` | Category browsing and product categorization | ### Database (PostgreSQL/PostGIS) @@ -331,7 +344,7 @@ BullMQ workers handle asynchronous processing tasks. PM2 manages both the API se ### Flyer Processing Pipeline -``` +```text +-------------+ +----------------+ +------------------+ +---------------+ | User | | Express | | BullMQ | | PostgreSQL | | Upload +---->+ Route +---->+ Queue +---->+ Storage | @@ -395,7 +408,7 @@ BullMQ workers handle asynchronous processing tasks. PM2 manages both the API se The application follows a strict layered architecture as defined in ADR-035. -``` +```text +-----------------------------------------------------------------------+ | ROUTES LAYER | | Responsibilities: | @@ -458,7 +471,7 @@ The application follows a strict layered architecture as defined in ADR-035. ### Entity Relationship Overview -``` +```text +------------------+ +------------------+ +------------------+ | users | | profiles | | addresses | |------------------| |------------------| |------------------| @@ -537,7 +550,7 @@ The application follows a strict layered architecture as defined in ADR-035. ### JWT Token Architecture -``` +```text +-------------------+ +-------------------+ +-------------------+ | Login Request | | Server | | Database | | (email/pass) +---->+ Validates +---->+ Verify User | @@ -576,7 +589,7 @@ The application follows a strict layered architecture as defined in ADR-035. ### Protected Route Flow -``` +```text +-------------------+ +-------------------+ +-------------------+ | API Request | | requireAuth | | JWT Strategy | | + Bearer Token +---->+ Middleware +---->+ Validate | @@ -603,7 +616,7 @@ The application follows a strict layered architecture as defined in ADR-035. ### Worker Architecture -``` +```text +-------------------+ +-------------------+ +-------------------+ | API Server | | Redis | | Worker Process | | (Queue Producer)| | (Job Storage) | | (Consumer) | @@ -635,7 +648,7 @@ The application follows a strict layered architecture as defined in ADR-035. Jobs use exponential backoff for retries: -``` +```text Attempt 1: Immediate Attempt 2: Initial delay (e.g., 5 seconds) Attempt 3: 2x delay (e.g., 10 seconds) @@ -658,7 +671,7 @@ Attempt 4: 4x delay (e.g., 20 seconds) ### Environment Overview -``` +```text +-----------------------------------------------------------------------------------+ | DEVELOPMENT | +-----------------------------------------------------------------------------------+ @@ -710,7 +723,7 @@ Attempt 4: 4x delay (e.g., 20 seconds) ### Deployment Pipeline (ADR-017) -``` +```text +------------+ +------------+ +------------+ +------------+ | Push to | | Gitea | | Build & | | Deploy | | main +---->+ Actions +---->+ Test +---->+ to Prod | @@ -839,22 +852,55 @@ The system architecture is governed by Architecture Decision Records (ADRs). Key | File | Purpose | | ----------------------------------------------- | --------------------------------------- | | `src/services/flyerProcessingService.server.ts` | Flyer processing pipeline orchestration | +| `src/services/flyerAiProcessor.server.ts` | AI extraction for flyers | | `src/services/aiService.server.ts` | Google Gemini AI integration | | `src/services/cacheService.server.ts` | Redis caching abstraction | | `src/services/emailService.server.ts` | Email sending | | `src/services/queues.server.ts` | BullMQ queue definitions | +| `src/services/queueService.server.ts` | Queue management and scheduling | | `src/services/workers.server.ts` | BullMQ worker definitions | +| `src/services/websocketService.server.ts` | Real-time WebSocket notifications | +| `src/services/receiptService.server.ts` | Receipt scanning and OCR | +| `src/services/upcService.server.ts` | UPC barcode lookup | +| `src/services/expiryService.server.ts` | Pantry expiry tracking | +| `src/services/geocodingService.server.ts` | Address geocoding | +| `src/services/analyticsService.server.ts` | Analytics and reporting | +| `src/services/monitoringService.server.ts` | Health monitoring | +| `src/services/barcodeService.server.ts` | Barcode detection | +| `src/services/logger.server.ts` | Structured logging (Pino) | +| `src/services/redis.server.ts` | Redis connection management | +| `src/services/sentry.server.ts` | Error tracking (Sentry/Bugsink) | ### Database Files -| File | Purpose | -| ---------------------------------- | -------------------------------------------- | -| `src/services/db/connection.db.ts` | Database pool and transaction management | -| `src/services/db/errors.db.ts` | Database error types | -| `src/services/db/user.db.ts` | User repository | -| `src/services/db/flyer.db.ts` | Flyer repository | -| `sql/master_schema_rollup.sql` | Complete database schema (for test DB setup) | -| `sql/initial_schema.sql` | Fresh installation schema | +| File | Purpose | +| --------------------------------------- | -------------------------------------------- | +| `src/services/db/connection.db.ts` | Database pool and transaction management | +| `src/services/db/errors.db.ts` | Database error types | +| `src/services/db/index.db.ts` | Repository exports | +| `src/services/db/user.db.ts` | User repository | +| `src/services/db/flyer.db.ts` | Flyer repository | +| `src/services/db/store.db.ts` | Store repository | +| `src/services/db/storeLocation.db.ts` | Store location repository | +| `src/services/db/recipe.db.ts` | Recipe repository | +| `src/services/db/category.db.ts` | Category repository | +| `src/services/db/personalization.db.ts` | Master items and personalization | +| `src/services/db/shopping.db.ts` | Shopping lists repository | +| `src/services/db/deals.db.ts` | Deals and best prices repository | +| `src/services/db/price.db.ts` | Price history repository | +| `src/services/db/receipt.db.ts` | Receipt repository | +| `src/services/db/upc.db.ts` | UPC scan history repository | +| `src/services/db/expiry.db.ts` | Expiry tracking repository | +| `src/services/db/gamification.db.ts` | Achievements repository | +| `src/services/db/budget.db.ts` | Budget repository | +| `src/services/db/reaction.db.ts` | User reactions repository | +| `src/services/db/notification.db.ts` | Notifications repository | +| `src/services/db/address.db.ts` | Address repository | +| `src/services/db/admin.db.ts` | Admin operations repository | +| `src/services/db/conversion.db.ts` | Unit conversion repository | +| `src/services/db/flyerLocation.db.ts` | Flyer locations repository | +| `sql/master_schema_rollup.sql` | Complete database schema (for test DB setup) | +| `sql/initial_schema.sql` | Fresh installation schema | ### Type Definitions diff --git a/docs/development/CODE-PATTERNS.md b/docs/development/CODE-PATTERNS.md index 048278cb..b2f6e0b9 100644 --- a/docs/development/CODE-PATTERNS.md +++ b/docs/development/CODE-PATTERNS.md @@ -2,6 +2,21 @@ Common code patterns extracted from Architecture Decision Records (ADRs). Use these as templates when writing new code. +## Quick Reference + +| Pattern | Key Function/Class | Import From | +| ------------------ | ------------------------------------------------- | ---------------------------------- | +| Error Handling | `handleDbError()`, `NotFoundError` | `src/services/db/errors.db.ts` | +| Repository Methods | `get*`, `find*`, `list*` | `src/services/db/*.db.ts` | +| API Responses | `sendSuccess()`, `sendPaginated()`, `sendError()` | `src/utils/apiResponse.ts` | +| Transactions | `withTransaction()` | `src/services/db/connection.db.ts` | +| Validation | `validateRequest()` | `src/middleware/validation.ts` | +| Authentication | `authenticateJWT` | `src/middleware/auth.ts` | +| Caching | `cacheService` | `src/services/cache.server.ts` | +| Background Jobs | Queue classes | `src/services/queues.server.ts` | + +--- + ## Table of Contents - [Error Handling](#error-handling) @@ -17,7 +32,7 @@ Common code patterns extracted from Architecture Decision Records (ADRs). Use th ## Error Handling -**ADR**: [ADR-001](../adr/0001-standardized-error-handling-for-database-operations.md) +**ADR**: [ADR-001](../adr/0001-standardized-error-handling.md) ### Repository Layer Error Handling @@ -78,7 +93,7 @@ throw new DatabaseError('Failed to insert flyer', originalError); ## Repository Patterns -**ADR**: [ADR-034](../adr/0034-repository-layer-method-naming-conventions.md) +**ADR**: [ADR-034](../adr/0034-repository-pattern-standards.md) ### Method Naming Conventions @@ -155,16 +170,17 @@ export async function listActiveFlyers(client?: PoolClient): Promise { ## API Response Patterns -**ADR**: [ADR-028](../adr/0028-consistent-api-response-format.md) +**ADR**: [ADR-028](../adr/0028-api-response-standardization.md) ### Success Response ```typescript import { sendSuccess } from '../utils/apiResponse'; -app.post('/api/flyers', async (req, res) => { +app.post('/api/v1/flyers', async (req, res) => { const flyer = await flyerService.createFlyer(req.body); - return sendSuccess(res, flyer, 'Flyer created successfully', 201); + // sendSuccess(res, data, statusCode?, meta?) + return sendSuccess(res, flyer, 201); }); ``` @@ -173,30 +189,32 @@ app.post('/api/flyers', async (req, res) => { ```typescript import { sendPaginated } from '../utils/apiResponse'; -app.get('/api/flyers', async (req, res) => { - const { page = 1, pageSize = 20 } = req.query; - const { items, total } = await flyerService.listFlyers(page, pageSize); +app.get('/api/v1/flyers', async (req, res) => { + const page = parseInt(req.query.page as string) || 1; + const limit = parseInt(req.query.limit as string) || 20; + const { items, total } = await flyerService.listFlyers(page, limit); - return sendPaginated(res, { - items, - total, - page: parseInt(page), - pageSize: parseInt(pageSize), - }); + // sendPaginated(res, data[], { page, limit, total }, meta?) + return sendPaginated(res, items, { page, limit, total }); }); ``` ### Error Response ```typescript -import { sendError } from '../utils/apiResponse'; +import { sendError, sendSuccess, ErrorCode } from '../utils/apiResponse'; -app.get('/api/flyers/:id', async (req, res) => { +app.get('/api/v1/flyers/:id', async (req, res) => { try { const flyer = await flyerDb.getFlyerById(parseInt(req.params.id)); return sendSuccess(res, flyer); } catch (error) { - return sendError(res, error); // Automatically maps error to correct status + // sendError(res, code, message, statusCode?, details?, meta?) + if (error instanceof NotFoundError) { + return sendError(res, ErrorCode.NOT_FOUND, error.message, 404); + } + req.log.error({ error }, `Error in ${req.originalUrl.split('?')[0]}:`); + return sendError(res, ErrorCode.INTERNAL_ERROR, 'An error occurred', 500); } }); ``` @@ -205,12 +223,12 @@ app.get('/api/flyers/:id', async (req, res) => { ## Transaction Management -**ADR**: [ADR-002](../adr/0002-transaction-management-pattern.md) +**ADR**: [ADR-002](../adr/0002-standardized-transaction-management.md) ### Basic Transaction ```typescript -import { withTransaction } from '../services/db/transaction.db'; +import { withTransaction } from '../services/db/connection.db'; export async function createFlyerWithItems( flyerData: FlyerInput, @@ -262,7 +280,7 @@ export async function bulkImportFlyers(flyersData: FlyerInput[]): Promise; import { validateRequest } from '../middleware/validation'; import { createFlyerSchema } from '../schemas/flyer.schemas'; -app.post('/api/flyers', validateRequest(createFlyerSchema), async (req, res) => { +app.post('/api/v1/flyers', validateRequest(createFlyerSchema), async (req, res) => { // req.body is now type-safe and validated const flyer = await flyerService.createFlyer(req.body); - return sendSuccess(res, flyer, 'Flyer created successfully', 201); + return sendSuccess(res, flyer, 201); }); ``` @@ -331,7 +349,7 @@ export async function processFlyer(data: unknown): Promise { import { authenticateJWT } from '../middleware/auth'; app.get( - '/api/profile', + '/api/v1/profile', authenticateJWT, // Middleware adds req.user async (req, res) => { // req.user is guaranteed to exist @@ -347,7 +365,7 @@ app.get( import { optionalAuth } from '../middleware/auth'; app.get( - '/api/flyers', + '/api/v1/flyers', optionalAuth, // req.user may or may not exist async (req, res) => { const flyers = req.user @@ -374,7 +392,7 @@ export function generateToken(user: User): string { ## Caching -**ADR**: [ADR-029](../adr/0029-redis-caching-strategy.md) +**ADR**: [ADR-009](../adr/0009-caching-strategy-for-read-heavy-operations.md) ### Cache Pattern @@ -414,7 +432,7 @@ export async function updateFlyer(id: number, data: UpdateFlyerInput): Promise{deal.store_name} @@ -113,15 +127,26 @@ Located throughout `src/` directory alongside source files with `.test.ts` or `. npm run test:unit ``` -### Integration Tests (5 test files) +### Integration Tests (28 test files) -Located in `src/tests/integration/`: +Located in `src/tests/integration/`. Key test files include: -- `admin.integration.test.ts` -- `flyer.integration.test.ts` -- `price.integration.test.ts` -- `public.routes.integration.test.ts` -- `receipt.integration.test.ts` +| Test File | Domain | +| -------------------------------------- | -------------------------- | +| `admin.integration.test.ts` | Admin dashboard operations | +| `auth.integration.test.ts` | Authentication flows | +| `budget.integration.test.ts` | Budget management | +| `flyer.integration.test.ts` | Flyer CRUD operations | +| `flyer-processing.integration.test.ts` | AI flyer processing | +| `gamification.integration.test.ts` | Achievements and points | +| `inventory.integration.test.ts` | Inventory management | +| `notification.integration.test.ts` | User notifications | +| `receipt.integration.test.ts` | Receipt processing | +| `recipe.integration.test.ts` | Recipe management | +| `shopping-list.integration.test.ts` | Shopping list operations | +| `user.integration.test.ts` | User profile operations | + +See `src/tests/integration/` for the complete list. Requires PostgreSQL and Redis services running. @@ -129,13 +154,23 @@ Requires PostgreSQL and Redis services running. npm run test:integration ``` -### E2E Tests (3 test files) +### E2E Tests (11 test files) -Located in `src/tests/e2e/`: +Located in `src/tests/e2e/`. Full user journey tests: -- `deals-journey.e2e.test.ts` -- `budget-journey.e2e.test.ts` -- `receipt-journey.e2e.test.ts` +| Test File | Journey | +| --------------------------------- | ----------------------------- | +| `admin-authorization.e2e.test.ts` | Admin access control | +| `admin-dashboard.e2e.test.ts` | Admin dashboard flows | +| `auth.e2e.test.ts` | Login/logout/registration | +| `budget-journey.e2e.test.ts` | Budget tracking workflow | +| `deals-journey.e2e.test.ts` | Finding and saving deals | +| `error-reporting.e2e.test.ts` | Error handling verification | +| `flyer-upload.e2e.test.ts` | Flyer upload and processing | +| `inventory-journey.e2e.test.ts` | Pantry management | +| `receipt-journey.e2e.test.ts` | Receipt scanning and tracking | +| `upc-journey.e2e.test.ts` | UPC barcode scanning | +| `user-journey.e2e.test.ts` | User profile management | Requires all services (PostgreSQL, Redis, BullMQ workers) running. @@ -157,20 +192,18 @@ Located in `src/tests/utils/storeHelpers.ts`: ```typescript // Create a store with a location in one call -const store = await createStoreWithLocation({ - storeName: 'Test Store', - address: { - address_line_1: '123 Main St', - city: 'Toronto', - province_state: 'ON', - postal_code: 'M1M 1M1', - }, - pool, - log, +const store = await createStoreWithLocation(pool, { + name: 'Test Store', + address: '123 Main St', + city: 'Toronto', + province: 'ON', + postalCode: 'M1M 1M1', }); +// Returns: { storeId, addressId, storeLocationId } + // Cleanup stores and their locations -await cleanupStoreLocations([storeId1, storeId2], pool, log); +await cleanupStoreLocation(pool, store); ``` ### Mock Factories diff --git a/docs/getting-started/ENVIRONMENT.md b/docs/getting-started/ENVIRONMENT.md index d5d2a4f4..ec3606bd 100644 --- a/docs/getting-started/ENVIRONMENT.md +++ b/docs/getting-started/ENVIRONMENT.md @@ -2,134 +2,259 @@ Complete guide to environment variables used in Flyer Crawler. +--- + +## Quick Reference + +### Minimum Required Variables (Development) + +| Variable | Example | Purpose | +| ---------------- | ------------------------ | -------------------- | +| `DB_HOST` | `localhost` | PostgreSQL host | +| `DB_USER` | `postgres` | PostgreSQL username | +| `DB_PASSWORD` | `postgres` | PostgreSQL password | +| `DB_NAME` | `flyer_crawler_dev` | Database name | +| `REDIS_URL` | `redis://localhost:6379` | Redis connection URL | +| `JWT_SECRET` | (32+ character string) | JWT signing key | +| `GEMINI_API_KEY` | `AIzaSy...` | Google Gemini API | + +### Source of Truth + +The Zod schema at `src/config/env.ts` is the authoritative source for all environment variables. If a variable is not in this file, it is not used by the application. + +--- + ## Configuration by Environment ### Production -**Location**: Gitea CI/CD secrets injected during deployment -**Path**: `/var/www/flyer-crawler.projectium.com/` -**Note**: No `.env` file exists - all variables come from CI/CD +| Aspect | Details | +| -------- | ------------------------------------------ | +| Location | Gitea CI/CD secrets injected at deployment | +| Path | `/var/www/flyer-crawler.projectium.com/` | +| File | No `.env` file - all from CI/CD secrets | ### Test -**Location**: Gitea CI/CD secrets + `.env.test` file -**Path**: `/var/www/flyer-crawler-test.projectium.com/` -**Note**: `.env.test` overrides for test-specific values +| Aspect | Details | +| -------- | --------------------------------------------- | +| Location | Gitea CI/CD secrets + `.env.test` overrides | +| Path | `/var/www/flyer-crawler-test.projectium.com/` | +| File | `.env.test` for test-specific values | ### Development Container -**Location**: `.env.local` file in project root -**Note**: Overrides default DSNs in `compose.dev.yml` +| Aspect | Details | +| -------- | --------------------------------------- | +| Location | `.env.local` file in project root | +| Priority | Overrides defaults in `compose.dev.yml` | +| File | `.env.local` (gitignored) | -## Required Variables +--- -### Database +## Complete Variable Reference -| Variable | Description | Example | -| ------------------ | ---------------------------- | ------------------------------------------ | -| `DB_HOST` | PostgreSQL host | `localhost` (dev), `projectium.com` (prod) | -| `DB_PORT` | PostgreSQL port | `5432` | -| `DB_USER_PROD` | Production database user | `flyer_crawler_prod` | -| `DB_PASSWORD_PROD` | Production database password | (secret) | -| `DB_DATABASE_PROD` | Production database name | `flyer-crawler-prod` | -| `DB_USER_TEST` | Test database user | `flyer_crawler_test` | -| `DB_PASSWORD_TEST` | Test database password | (secret) | -| `DB_DATABASE_TEST` | Test database name | `flyer-crawler-test` | -| `DB_USER` | Dev database user | `postgres` | -| `DB_PASSWORD` | Dev database password | `postgres` | -| `DB_NAME` | Dev database name | `flyer_crawler_dev` | +### Database Configuration -**Note**: Production and test use separate `_PROD` and `_TEST` suffixed variables. Development uses unsuffixed variables. +| Variable | Required | Default | Description | +| ------------- | -------- | ------- | ----------------- | +| `DB_HOST` | Yes | - | PostgreSQL host | +| `DB_PORT` | No | `5432` | PostgreSQL port | +| `DB_USER` | Yes | - | Database username | +| `DB_PASSWORD` | Yes | - | Database password | +| `DB_NAME` | Yes | - | Database name | -### Redis +**Environment-Specific Variables** (Gitea Secrets): -| Variable | Description | Example | -| --------------------- | ------------------------- | ------------------------------ | -| `REDIS_URL` | Redis connection URL | `redis://localhost:6379` (dev) | -| `REDIS_PASSWORD_PROD` | Production Redis password | (secret) | -| `REDIS_PASSWORD_TEST` | Test Redis password | (secret) | +| Variable | Environment | Description | +| ------------------ | ----------- | ------------------------ | +| `DB_USER_PROD` | Production | Production database user | +| `DB_PASSWORD_PROD` | Production | Production database pass | +| `DB_DATABASE_PROD` | Production | Production database name | +| `DB_USER_TEST` | Test | Test database user | +| `DB_PASSWORD_TEST` | Test | Test database password | +| `DB_DATABASE_TEST` | Test | Test database name | + +### Redis Configuration + +| Variable | Required | Default | Description | +| ---------------- | -------- | ------- | ------------------------- | +| `REDIS_URL` | Yes | - | Redis connection URL | +| `REDIS_PASSWORD` | No | - | Redis password (optional) | + +**URL Format**: `redis://[user:password@]host:port` + +**Examples**: + +```bash +# Development (no auth) +REDIS_URL=redis://localhost:6379 + +# Production (with auth) +REDIS_URL=redis://:${REDIS_PASSWORD_PROD}@localhost:6379 +``` ### Authentication -| Variable | Description | Example | -| ---------------------- | -------------------------- | -------------------------------- | -| `JWT_SECRET` | JWT token signing key | (minimum 32 characters) | -| `SESSION_SECRET` | Session encryption key | (minimum 32 characters) | -| `GOOGLE_CLIENT_ID` | Google OAuth client ID | `xxx.apps.googleusercontent.com` | -| `GOOGLE_CLIENT_SECRET` | Google OAuth client secret | (secret) | -| `GH_CLIENT_ID` | GitHub OAuth client ID | `xxx` | -| `GH_CLIENT_SECRET` | GitHub OAuth client secret | (secret) | +| Variable | Required | Min Length | Description | +| ---------------------- | -------- | ---------- | ----------------------- | +| `JWT_SECRET` | Yes | 32 chars | JWT token signing key | +| `JWT_SECRET_PREVIOUS` | No | - | Previous key (rotation) | +| `GOOGLE_CLIENT_ID` | No | - | Google OAuth client ID | +| `GOOGLE_CLIENT_SECRET` | No | - | Google OAuth secret | +| `GITHUB_CLIENT_ID` | No | - | GitHub OAuth client ID | +| `GITHUB_CLIENT_SECRET` | No | - | GitHub OAuth secret | + +**Generate Secure Secret**: + +```bash +node -e "console.log(require('crypto').randomBytes(32).toString('hex'))" +``` ### AI Services -| Variable | Description | Example | -| -------------------------------- | ---------------------------- | ----------- | -| `VITE_GOOGLE_GENAI_API_KEY` | Google Gemini API key (prod) | `AIzaSy...` | -| `VITE_GOOGLE_GENAI_API_KEY_TEST` | Google Gemini API key (test) | `AIzaSy...` | -| `GOOGLE_MAPS_API_KEY` | Google Maps Geocoding API | `AIzaSy...` | +| Variable | Required | Description | +| ---------------------------- | -------- | -------------------------------- | +| `GEMINI_API_KEY` | Yes\* | Google Gemini API key | +| `GEMINI_RPM` | No | Rate limit (default: 5) | +| `AI_PRICE_QUALITY_THRESHOLD` | No | Quality threshold (default: 0.5) | -### Application +\*Required for flyer processing. Application works without it but cannot extract flyer data. -| Variable | Description | Example | -| -------------- | ------------------------ | ----------------------------------- | -| `NODE_ENV` | Environment mode | `development`, `test`, `production` | -| `PORT` | Backend server port | `3001` | -| `FRONTEND_URL` | Frontend application URL | `http://localhost:5173` (dev) | +**Get API Key**: [Google AI Studio](https://aistudio.google.com/app/apikey) -### Error Tracking +### Google Services -| Variable | Description | Example | -| ---------------------- | -------------------------------- | --------------------------- | -| `SENTRY_DSN` | Sentry DSN (production) | `https://xxx@sentry.io/xxx` | -| `VITE_SENTRY_DSN` | Frontend Sentry DSN (production) | `https://xxx@sentry.io/xxx` | -| `SENTRY_DSN_TEST` | Sentry DSN (test) | `https://xxx@sentry.io/xxx` | -| `VITE_SENTRY_DSN_TEST` | Frontend Sentry DSN (test) | `https://xxx@sentry.io/xxx` | -| `SENTRY_AUTH_TOKEN` | Sentry API token for releases | (secret) | +| Variable | Required | Description | +| ---------------------- | -------- | -------------------------------- | +| `GOOGLE_MAPS_API_KEY` | No | Google Maps Geocoding API | +| `GOOGLE_CLIENT_ID` | No | OAuth (see Authentication above) | +| `GOOGLE_CLIENT_SECRET` | No | OAuth (see Authentication above) | -## Optional Variables +### UPC Lookup APIs -| Variable | Description | Default | -| ------------------- | ----------------------- | ----------------- | -| `LOG_LEVEL` | Logging verbosity | `info` | -| `REDIS_TTL` | Cache TTL in seconds | `3600` | -| `MAX_UPLOAD_SIZE` | Max file upload size | `10mb` | -| `RATE_LIMIT_WINDOW` | Rate limit window (ms) | `900000` (15 min) | -| `RATE_LIMIT_MAX` | Max requests per window | `100` | +| Variable | Required | Description | +| ------------------------ | -------- | ---------------------- | +| `UPC_ITEM_DB_API_KEY` | No | UPC Item DB API key | +| `BARCODE_LOOKUP_API_KEY` | No | Barcode Lookup API key | + +### Application Settings + +| Variable | Required | Default | Description | +| -------------- | -------- | ------------- | ------------------------ | +| `NODE_ENV` | No | `development` | Environment mode | +| `PORT` | No | `3001` | Backend server port | +| `FRONTEND_URL` | No | - | Frontend URL (CORS) | +| `BASE_URL` | No | - | API base URL | +| `STORAGE_PATH` | No | (see below) | Flyer image storage path | + +**NODE_ENV Values**: `development`, `test`, `staging`, `production` + +**Default STORAGE_PATH**: `/var/www/flyer-crawler.projectium.com/flyer-images` + +### Email/SMTP Configuration + +| Variable | Required | Default | Description | +| ----------------- | -------- | ------- | ----------------------- | +| `SMTP_HOST` | No | - | SMTP server hostname | +| `SMTP_PORT` | No | `587` | SMTP server port | +| `SMTP_USER` | No | - | SMTP username | +| `SMTP_PASS` | No | - | SMTP password | +| `SMTP_SECURE` | No | `false` | Use TLS | +| `SMTP_FROM_EMAIL` | No | - | From address for emails | + +**Note**: Email functionality degrades gracefully if not configured. + +### Worker Configuration + +| Variable | Default | Description | +| ------------------------------------- | ------- | ---------------------------- | +| `WORKER_CONCURRENCY` | `1` | Main worker concurrency | +| `WORKER_LOCK_DURATION` | `30000` | Lock duration (ms) | +| `EMAIL_WORKER_CONCURRENCY` | `10` | Email worker concurrency | +| `ANALYTICS_WORKER_CONCURRENCY` | `1` | Analytics worker concurrency | +| `CLEANUP_WORKER_CONCURRENCY` | `10` | Cleanup worker concurrency | +| `WEEKLY_ANALYTICS_WORKER_CONCURRENCY` | `1` | Weekly analytics concurrency | + +### Error Tracking (Bugsink/Sentry) + +| Variable | Required | Default | Description | +| --------------------- | -------- | -------- | ------------------------------- | +| `SENTRY_DSN` | No | - | Backend Sentry DSN | +| `SENTRY_ENABLED` | No | `true` | Enable error tracking | +| `SENTRY_ENVIRONMENT` | No | NODE_ENV | Environment name for errors | +| `SENTRY_DEBUG` | No | `false` | Enable Sentry SDK debug logging | +| `VITE_SENTRY_DSN` | No | - | Frontend Sentry DSN | +| `VITE_SENTRY_ENABLED` | No | `true` | Enable frontend error tracking | +| `VITE_SENTRY_DEBUG` | No | `false` | Frontend SDK debug logging | + +**DSN Format**: `http://[key]@[host]:[port]/[project_id]` + +**Dev Container DSNs**: + +```bash +# Backend (internal) +SENTRY_DSN=http://@localhost:8000/1 + +# Frontend (via nginx proxy) +VITE_SENTRY_DSN=https://@localhost/bugsink-api/2 +``` + +--- ## Configuration Files | File | Purpose | | ------------------------------------- | ------------------------------------------- | | `src/config/env.ts` | Zod schema validation - **source of truth** | -| `ecosystem.config.cjs` | PM2 process manager config | +| `ecosystem.config.cjs` | PM2 process manager (production) | +| `ecosystem.dev.config.cjs` | PM2 process manager (development) | | `.gitea/workflows/deploy-to-prod.yml` | Production deployment workflow | | `.gitea/workflows/deploy-to-test.yml` | Test deployment workflow | | `.env.example` | Template with all variables | | `.env.local` | Dev container overrides (not in git) | | `.env.test` | Test environment overrides (not in git) | +--- + ## Adding New Variables -### 1. Update Zod Schema +### Checklist + +1. [ ] **Update Zod Schema** - Edit `src/config/env.ts` +2. [ ] **Add to Gitea Secrets** - For prod/test environments +3. [ ] **Update Deployment Workflows** - `.gitea/workflows/*.yml` +4. [ ] **Update PM2 Config** - `ecosystem.config.cjs` +5. [ ] **Update .env.example** - Template for developers +6. [ ] **Update this document** - Add to appropriate section + +### Step-by-Step + +#### 1. Update Zod Schema Edit `src/config/env.ts`: ```typescript const envSchema = z.object({ // ... existing variables ... - NEW_VARIABLE: z.string().min(1), + newSection: z.object({ + newVariable: z.string().min(1, 'NEW_VARIABLE is required'), + }), }); + +// In loadEnvVars(): +newSection: { + newVariable: process.env.NEW_VARIABLE, +}, ``` -### 2. Add to Gitea Secrets - -For prod/test environments: +#### 2. Add to Gitea Secrets 1. Go to Gitea repository Settings > Secrets -2. Add `NEW_VARIABLE` with value +2. Add `NEW_VARIABLE` with production value 3. Add `NEW_VARIABLE_TEST` if test needs different value -### 3. Update Deployment Workflows +#### 3. Update Deployment Workflows Edit `.gitea/workflows/deploy-to-prod.yml`: @@ -145,7 +270,7 @@ env: NEW_VARIABLE: ${{ secrets.NEW_VARIABLE_TEST }} ``` -### 4. Update PM2 Config +#### 4. Update PM2 Config Edit `ecosystem.config.cjs`: @@ -161,31 +286,36 @@ module.exports = { }; ``` -### 5. Update Documentation - -- Add to `.env.example` -- Update this document -- Document in relevant feature docs +--- ## Security Best Practices -### Secrets Management +### Do -- **NEVER** commit secrets to git - Use Gitea Secrets for prod/test - Use `.env.local` for dev (gitignored) +- Generate secrets with cryptographic randomness - Rotate secrets regularly +- Use environment-specific database users + +### Do Not + +- Commit secrets to git +- Use short or predictable secrets +- Share secrets across environments +- Log sensitive values ### Secret Generation ```bash -# Generate secure random secrets +# Generate secure random secrets (64 hex characters) node -e "console.log(require('crypto').randomBytes(32).toString('hex'))" + +# Example output: +# a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4e5f6a1b2 ``` -### Database Users - -Each environment has its own PostgreSQL user: +### Database Users by Environment | Environment | User | Database | | ----------- | -------------------- | -------------------- | @@ -193,44 +323,61 @@ Each environment has its own PostgreSQL user: | Test | `flyer_crawler_test` | `flyer-crawler-test` | | Development | `postgres` | `flyer_crawler_dev` | -**Setup Commands** (as postgres superuser): - -```sql --- Production -CREATE DATABASE "flyer-crawler-prod"; -CREATE USER flyer_crawler_prod WITH PASSWORD 'secure-password'; -ALTER DATABASE "flyer-crawler-prod" OWNER TO flyer_crawler_prod; -\c "flyer-crawler-prod" -ALTER SCHEMA public OWNER TO flyer_crawler_prod; -GRANT CREATE, USAGE ON SCHEMA public TO flyer_crawler_prod; -CREATE EXTENSION IF NOT EXISTS "uuid-ossp"; -CREATE EXTENSION IF NOT EXISTS postgis; -CREATE EXTENSION IF NOT EXISTS pg_trgm; - --- Test (similar commands with _test suffix) -``` +--- ## Validation -Environment variables are validated at startup via `src/config/env.ts`. If validation fails: +Environment variables are validated at startup via `src/config/env.ts`. -1. Check the error message for missing/invalid variables -2. Verify `.env.local` (dev) or Gitea Secrets (prod/test) -3. Ensure values match schema requirements (min length, format, etc.) +### Startup Validation + +If validation fails, you will see: + +```text +╔════════════════════════════════════════════════════════════════╗ +║ CONFIGURATION ERROR - APPLICATION STARTUP ║ +╚════════════════════════════════════════════════════════════════╝ + +The following environment variables are missing or invalid: + + - database.host: DB_HOST is required + - auth.jwtSecret: JWT_SECRET must be at least 32 characters + +Please check your .env file or environment configuration. +``` + +### Debugging Configuration + +```bash +# Check what variables are set (dev container) +podman exec flyer-crawler-dev env | grep -E "^(DB_|REDIS_|JWT_|SENTRY_)" + +# Test database connection +podman exec flyer-crawler-postgres psql -U postgres -d flyer_crawler_dev -c "SELECT 1;" + +# Test Redis connection +podman exec flyer-crawler-redis redis-cli ping +``` + +--- ## Troubleshooting ### Variable Not Found -``` +```text Error: Missing required environment variable: JWT_SECRET ``` -**Solution**: Add the variable to your environment configuration. +**Solutions**: + +1. Check `.env.local` exists and has the variable +2. Verify variable name matches schema exactly +3. Restart the application after changes ### Invalid Value -``` +```text Error: JWT_SECRET must be at least 32 characters ``` @@ -240,32 +387,36 @@ Error: JWT_SECRET must be at least 32 characters Check `NODE_ENV` is set correctly: -- `development` - Local dev container -- `test` - CI/CD test server -- `production` - Production server +| Value | Purpose | +| ------------- | ---------------------- | +| `development` | Local dev container | +| `test` | CI/CD test server | +| `staging` | Pre-production testing | +| `production` | Production server | ### Database Connection Issues -Verify database credentials: - ```bash # Development podman exec flyer-crawler-postgres psql -U postgres -d flyer_crawler_dev -c "SELECT 1;" -# Production (via SSH) -ssh root@projectium.com "psql -U flyer_crawler_prod -d flyer-crawler-prod -c 'SELECT 1;'" +# If connection fails, check: +# 1. Container is running: podman ps +# 2. DB_HOST matches container network +# 3. DB_PASSWORD is correct ``` -## Reference +--- -- **Validation Schema**: [src/config/env.ts](../../src/config/env.ts) -- **Template**: [.env.example](../../.env.example) -- **Deployment Workflows**: [.gitea/workflows/](../../.gitea/workflows/) -- **PM2 Config**: [ecosystem.config.cjs](../../ecosystem.config.cjs) - -## See Also +## Related Documentation - [QUICKSTART.md](QUICKSTART.md) - Quick setup guide - [INSTALL.md](INSTALL.md) - Detailed installation +- [DEV-CONTAINER.md](../development/DEV-CONTAINER.md) - Dev container setup - [DEPLOYMENT.md](../operations/DEPLOYMENT.md) - Production deployment - [AUTHENTICATION.md](../architecture/AUTHENTICATION.md) - OAuth setup +- [ADR-007](../adr/0007-configuration-and-secrets-management.md) - Configuration decisions + +--- + +Last updated: January 2026 diff --git a/docs/getting-started/INSTALL.md b/docs/getting-started/INSTALL.md index e4090aeb..c4a1fc5f 100644 --- a/docs/getting-started/INSTALL.md +++ b/docs/getting-started/INSTALL.md @@ -1,203 +1,453 @@ # Installation Guide -This guide covers setting up a local development environment for Flyer Crawler. +Complete setup instructions for the Flyer Crawler local development environment. + +--- + +## Quick Reference + +| Setup Method | Best For | Time | Document Section | +| ----------------- | --------------------------- | ------ | --------------------------------------------------- | +| Quick Start | Already have Postgres/Redis | 5 min | [Quick Start](#quick-start) | +| Dev Container | Full production-like setup | 15 min | [Dev Container](#development-container-recommended) | +| Manual Containers | Learning the components | 20 min | [Podman Setup](#podman-setup-manual) | + +--- ## Prerequisites -- Node.js 20.x or later -- Access to a PostgreSQL database (local or remote) -- Redis instance (for session management) -- Google Gemini API key -- Google Maps API key (for geocoding) +### Required Software + +| Software | Minimum Version | Purpose | Download | +| -------------- | --------------- | -------------------- | ----------------------------------------------- | +| Node.js | 20.x | Runtime | [nodejs.org](https://nodejs.org/) | +| Podman Desktop | 4.x | Container management | [podman-desktop.io](https://podman-desktop.io/) | +| Git | 2.x | Version control | [git-scm.com](https://git-scm.com/) | + +### Windows-Specific Requirements + +| Requirement | Purpose | Setup Command | +| ----------- | ------------------------------ | ---------------------------------- | +| WSL 2 | Linux compatibility for Podman | `wsl --install` (admin PowerShell) | + +### Verify Installation + +```bash +# Check all prerequisites +node --version # Expected: v20.x or higher +podman --version # Expected: podman version 4.x or higher +git --version # Expected: git version 2.x or higher +wsl --list -v # Expected: Shows WSL 2 distro +``` + +--- ## Quick Start -If you already have PostgreSQL and Redis configured: +If you already have PostgreSQL and Redis configured externally: ```bash -# Install dependencies +# 1. Clone the repository +git clone https://gitea.projectium.com/flyer-crawler/flyer-crawler.git +cd flyer-crawler + +# 2. Install dependencies npm install -# Run in development mode +# 3. Create .env.local (see Environment section below) + +# 4. Run in development mode npm run dev ``` +**Access Points**: + +- Frontend: `http://localhost:5173` +- Backend API: `http://localhost:3001` + --- -## Development Environment with Podman (Recommended for Windows) +## Development Container (Recommended) -This approach uses Podman with an Ubuntu container for a consistent development environment. +The dev container provides a complete, production-like environment. + +### What's Included + +| Service | Purpose | Port | +| ---------- | ------------------------ | ---------- | +| Node.js | API server, worker, Vite | 3001, 5173 | +| PostgreSQL | Database with PostGIS | 5432 | +| Redis | Cache and job queues | 6379 | +| NGINX | HTTPS reverse proxy | 443 | +| Bugsink | Error tracking | 8443 | +| Logstash | Log aggregation | - | +| PM2 | Process management | - | + +### Setup Steps + +#### Step 1: Initialize Podman + +```bash +# Windows: Start Podman Desktop, or from terminal: +podman machine init +podman machine start +``` + +#### Step 2: Start Dev Container + +```bash +# Start all services +podman-compose -f compose.dev.yml up -d + +# View logs (optional) +podman-compose -f compose.dev.yml logs -f +``` + +**Expected Output**: + +```text +[+] Running 3/3 + - Container flyer-crawler-postgres Started + - Container flyer-crawler-redis Started + - Container flyer-crawler-dev Started +``` + +#### Step 3: Verify Services + +```bash +# Check containers are running +podman ps + +# Check PM2 processes +podman exec -it flyer-crawler-dev pm2 status +``` + +**Expected PM2 Status**: + +```text ++---------------------------+--------+-------+ +| name | status | cpu | ++---------------------------+--------+-------+ +| flyer-crawler-api-dev | online | 0% | +| flyer-crawler-worker-dev | online | 0% | +| flyer-crawler-vite-dev | online | 0% | ++---------------------------+--------+-------+ +``` + +#### Step 4: Access Application + +| Service | URL | Notes | +| ----------- | ------------------------ | ---------------------------- | +| Frontend | `https://localhost` | NGINX proxies to Vite | +| Backend API | `http://localhost:3001` | Express server | +| Bugsink | `https://localhost:8443` | Login: admin@localhost/admin | + +### SSL Certificate Setup (Optional but Recommended) + +To eliminate browser security warnings: + +**Windows**: + +1. Double-click `certs/mkcert-ca.crt` +2. Click "Install Certificate..." +3. Select "Local Machine" > Next +4. Select "Place all certificates in the following store" +5. Browse > Select "Trusted Root Certification Authorities" > OK +6. Click Next > Finish +7. Restart browser + +**Other Platforms**: See [`certs/README.md`](../../certs/README.md) + +### Managing the Dev Container + +| Action | Command | +| --------- | ------------------------------------------- | +| Start | `podman-compose -f compose.dev.yml up -d` | +| Stop | `podman-compose -f compose.dev.yml down` | +| View logs | `podman-compose -f compose.dev.yml logs -f` | +| Restart | `podman-compose -f compose.dev.yml restart` | +| Rebuild | `podman-compose -f compose.dev.yml build` | + +--- + +## Podman Setup (Manual) + +For understanding the individual components or custom configurations. ### Step 1: Install Prerequisites on Windows -1. **Install WSL 2**: Podman on Windows relies on the Windows Subsystem for Linux. +```powershell +# Run in administrator PowerShell +wsl --install +``` - ```powershell - wsl --install - ``` +Restart computer after WSL installation. - Run this in an administrator PowerShell. +### Step 2: Initialize Podman -2. **Install Podman Desktop**: Download and install [Podman Desktop for Windows](https://podman-desktop.io/). +1. Launch **Podman Desktop** +2. Follow the setup wizard to initialize Podman machine +3. Start the Podman machine -### Step 2: Set Up Podman - -1. **Initialize Podman**: Launch Podman Desktop. It will automatically set up its WSL 2 machine. -2. **Start Podman**: Ensure the Podman machine is running from the Podman Desktop interface. - -### Step 3: Set Up the Ubuntu Container - -1. **Pull Ubuntu Image**: - - ```bash - podman pull ubuntu:latest - ``` - -2. **Create a Podman Volume** (persists node_modules between container restarts): - - ```bash - podman volume create node_modules_cache - ``` - -3. **Run the Ubuntu Container**: - - Open a terminal in your project's root directory and run: - - ```bash - podman run -it -p 3001:3001 -p 5173:5173 --name flyer-dev \ - -v "$(pwd):/app" \ - -v "node_modules_cache:/app/node_modules" \ - ubuntu:latest - ``` - - | Flag | Purpose | - | ------------------------------------------- | ------------------------------------------------ | - | `-p 3001:3001` | Forwards the backend server port | - | `-p 5173:5173` | Forwards the Vite frontend server port | - | `--name flyer-dev` | Names the container for easy reference | - | `-v "...:/app"` | Mounts your project directory into the container | - | `-v "node_modules_cache:/app/node_modules"` | Mounts the named volume for node_modules | - -### Step 4: Configure the Ubuntu Environment - -You are now inside the Ubuntu container's shell. - -1. **Update Package Lists**: - - ```bash - apt-get update - ``` - -2. **Install Dependencies**: - - ```bash - apt-get install -y curl git - curl -sL https://deb.nodesource.com/setup_20.x | bash - - apt-get install -y nodejs - ``` - -3. **Navigate to Project Directory**: - - ```bash - cd /app - ``` - -4. **Install Project Dependencies**: - - ```bash - npm install - ``` - -### Step 5: Run the Development Server +Or from terminal: ```bash +podman machine init +podman machine start +``` + +### Step 3: Create Podman Network + +```bash +podman network create flyer-crawler-net +``` + +### Step 4: Create PostgreSQL Container + +```bash +podman run -d \ + --name flyer-crawler-postgres \ + --network flyer-crawler-net \ + -e POSTGRES_USER=postgres \ + -e POSTGRES_PASSWORD=postgres \ + -e POSTGRES_DB=flyer_crawler_dev \ + -p 5432:5432 \ + -v flyer-crawler-pgdata:/var/lib/postgresql/data \ + docker.io/postgis/postgis:15-3.3 +``` + +### Step 5: Create Redis Container + +```bash +podman run -d \ + --name flyer-crawler-redis \ + --network flyer-crawler-net \ + -p 6379:6379 \ + -v flyer-crawler-redis:/data \ + docker.io/library/redis:alpine +``` + +### Step 6: Initialize Database + +```bash +# Wait for PostgreSQL to be ready +podman exec flyer-crawler-postgres pg_isready -U postgres + +# Install required extensions +podman exec flyer-crawler-postgres psql -U postgres -d flyer_crawler_dev -c " + CREATE EXTENSION IF NOT EXISTS postgis; + CREATE EXTENSION IF NOT EXISTS pg_trgm; + CREATE EXTENSION IF NOT EXISTS \"uuid-ossp\"; +" + +# Apply schema +podman exec -i flyer-crawler-postgres psql -U postgres -d flyer_crawler_dev < sql/master_schema_rollup.sql +``` + +### Step 7: Create Node.js Container + +```bash +# Create volume for node_modules +podman volume create node_modules_cache + +# Run Ubuntu container with project mounted +podman run -it \ + --name flyer-dev \ + --network flyer-crawler-net \ + -p 3001:3001 \ + -p 5173:5173 \ + -v "$(pwd):/app" \ + -v "node_modules_cache:/app/node_modules" \ + ubuntu:latest +``` + +### Step 8: Configure Container Environment + +Inside the container: + +```bash +# Update and install dependencies +apt-get update +apt-get install -y curl git + +# Install Node.js 20 +curl -sL https://deb.nodesource.com/setup_20.x | bash - +apt-get install -y nodejs + +# Navigate to project and install +cd /app +npm install + +# Start development server npm run dev ``` -### Step 6: Access the Application +### Container Management Commands -- **Frontend**: http://localhost:5173 -- **Backend API**: http://localhost:3001 - -### Dev Container with HTTPS (Full Stack) - -When using the full dev container stack with NGINX (via `compose.dev.yml`), access the application over HTTPS: - -- **Frontend**: https://localhost or https://127.0.0.1 -- **Backend API**: http://localhost:3001 - -**SSL Certificate Notes:** - -- The dev container uses self-signed certificates generated by mkcert -- Both `localhost` and `127.0.0.1` are valid hostnames (certificate includes both as SANs) -- If images fail to load with SSL errors, see [FLYER-URL-CONFIGURATION.md](../FLYER-URL-CONFIGURATION.md#ssl-certificate-configuration-dev-container) - -**Eliminate SSL Warnings (Recommended):** - -To avoid browser security warnings for self-signed certificates, install the mkcert CA certificate on your system. The CA certificate is located at `certs/mkcert-ca.crt` in the project root. - -See [`certs/README.md`](../../certs/README.md) for platform-specific installation instructions (Windows, macOS, Linux, Firefox). - -After installation: - -- Your browser will trust all mkcert certificates without warnings -- Both `https://localhost/` and `https://127.0.0.1/` will work without SSL errors -- Flyer images will load without `ERR_CERT_AUTHORITY_INVALID` errors - -### Managing the Container - -| Action | Command | -| --------------------- | -------------------------------- | -| Stop the container | Press `Ctrl+C`, then type `exit` | -| Restart the container | `podman start -a -i flyer-dev` | -| Remove the container | `podman rm flyer-dev` | +| Action | Command | +| -------------- | ------------------------------ | +| Stop container | Press `Ctrl+C`, then `exit` | +| Restart | `podman start -a -i flyer-dev` | +| Remove | `podman rm flyer-dev` | +| List running | `podman ps` | +| List all | `podman ps -a` | --- -## Environment Variables +## Environment Configuration -This project is configured to run in a CI/CD environment and does not use `.env` files. All configuration must be provided as environment variables. +### Create .env.local -For local development, you can export these in your shell or use your IDE's environment configuration: +Create `.env.local` in the project root with your configuration: -| Variable | Description | -| --------------------------- | ------------------------------------- | -| `DB_HOST` | PostgreSQL server hostname | -| `DB_USER` | PostgreSQL username | -| `DB_PASSWORD` | PostgreSQL password | -| `DB_DATABASE_PROD` | Production database name | -| `JWT_SECRET` | Secret string for signing auth tokens | -| `VITE_GOOGLE_GENAI_API_KEY` | Google Gemini API key | -| `GOOGLE_MAPS_API_KEY` | Google Maps Geocoding API key | -| `REDIS_PASSWORD_PROD` | Production Redis password | -| `REDIS_PASSWORD_TEST` | Test Redis password | +```bash +# Database (adjust host based on your setup) +DB_HOST=localhost # Use 'postgres' if inside dev container +DB_PORT=5432 +DB_USER=postgres +DB_PASSWORD=postgres +DB_NAME=flyer_crawler_dev + +# Redis (adjust host based on your setup) +REDIS_URL=redis://localhost:6379 # Use 'redis://redis:6379' inside container + +# Application +NODE_ENV=development +PORT=3001 +FRONTEND_URL=http://localhost:5173 + +# Authentication (generate secure values) +JWT_SECRET=your-secret-at-least-32-characters-long + +# AI Services +GEMINI_API_KEY=your-google-gemini-api-key +GOOGLE_MAPS_API_KEY=your-google-maps-api-key # Optional +``` + +**Generate Secure Secrets**: + +```bash +node -e "console.log(require('crypto').randomBytes(32).toString('hex'))" +``` + +### Environment Differences + +| Variable | Host Development | Inside Dev Container | +| ----------- | ------------------------ | -------------------- | +| `DB_HOST` | `localhost` | `postgres` | +| `REDIS_URL` | `redis://localhost:6379` | `redis://redis:6379` | + +See [ENVIRONMENT.md](ENVIRONMENT.md) for complete variable reference. --- ## Seeding Development Data -To create initial test accounts (`admin@example.com` and `user@example.com`) and sample data: +Create test accounts and sample data: ```bash npm run seed ``` -The seed script performs the following actions: +### What the Seed Script Does -1. Rebuilds the database schema from `sql/master_schema_rollup.sql` -2. Creates test user accounts (admin and regular user) -3. Copies test flyer images from `src/tests/assets/` to `public/flyer-images/` -4. Creates a sample flyer with items linked to the test images -5. Seeds watched items and a shopping list for the test user +1. Rebuilds database schema from `sql/master_schema_rollup.sql` +2. Creates test user accounts: + - `admin@example.com` (admin user) + - `user@example.com` (regular user) +3. Copies test flyer images to `public/flyer-images/` +4. Creates sample flyer with items +5. Seeds watched items and shopping list -**Test Images**: The seed script copies `test-flyer-image.jpg` and `test-flyer-icon.png` to the `public/flyer-images/` directory, which is served by NGINX at `/flyer-images/`. +### Test Images -After running, you may need to restart your IDE's TypeScript server to pick up any generated types. +The seed script copies these files from `src/tests/assets/`: + +- `test-flyer-image.jpg` +- `test-flyer-icon.png` + +Images are served by NGINX at `/flyer-images/`. + +--- + +## Verification Checklist + +After installation, verify everything works: + +- [ ] **Containers running**: `podman ps` shows postgres and redis +- [ ] **Database accessible**: `podman exec flyer-crawler-postgres psql -U postgres -c "SELECT 1;"` +- [ ] **Frontend loads**: Open `http://localhost:5173` (or `https://localhost` for dev container) +- [ ] **API responds**: `curl http://localhost:3001/health` +- [ ] **Tests pass**: `npm run test:unit` (or in container: `podman exec -it flyer-crawler-dev npm run test:unit`) +- [ ] **Type check passes**: `npm run type-check` + +--- + +## Troubleshooting + +### Podman Machine Won't Start + +```bash +# Reset Podman machine +podman machine rm +podman machine init +podman machine start +``` + +### Port Already in Use + +```bash +# Find process using port +netstat -ano | findstr :5432 + +# Option: Use different port +podman run -d --name flyer-crawler-postgres -p 5433:5432 ... +# Then set DB_PORT=5433 in .env.local +``` + +### Database Extensions Missing + +```bash +podman exec flyer-crawler-postgres psql -U postgres -d flyer_crawler_dev -c " + CREATE EXTENSION IF NOT EXISTS postgis; + CREATE EXTENSION IF NOT EXISTS pg_trgm; + CREATE EXTENSION IF NOT EXISTS \"uuid-ossp\"; +" +``` + +### Permission Denied on Windows Paths + +Use `MSYS_NO_PATHCONV=1` prefix: + +```bash +MSYS_NO_PATHCONV=1 podman exec flyer-crawler-dev /path/to/script.sh +``` + +### Tests Fail with Timezone Errors + +Tests must run in the dev container, not on Windows host: + +```bash +# CORRECT +podman exec -it flyer-crawler-dev npm test + +# INCORRECT (may fail with TZ errors) +npm test +``` --- ## Next Steps -- [Database Setup](DATABASE.md) - Set up PostgreSQL with required extensions -- [Authentication Setup](AUTHENTICATION.md) - Configure OAuth providers -- [Deployment Guide](DEPLOYMENT.md) - Deploy to production +| Goal | Document | +| --------------------- | ------------------------------------------------------ | +| Quick setup guide | [QUICKSTART.md](QUICKSTART.md) | +| Environment variables | [ENVIRONMENT.md](ENVIRONMENT.md) | +| Database schema | [DATABASE.md](../architecture/DATABASE.md) | +| Authentication setup | [AUTHENTICATION.md](../architecture/AUTHENTICATION.md) | +| Dev container details | [DEV-CONTAINER.md](../development/DEV-CONTAINER.md) | +| Deployment | [DEPLOYMENT.md](../operations/DEPLOYMENT.md) | + +--- + +Last updated: January 2026 diff --git a/docs/getting-started/QUICKSTART.md b/docs/getting-started/QUICKSTART.md index 0e957c53..69674266 100644 --- a/docs/getting-started/QUICKSTART.md +++ b/docs/getting-started/QUICKSTART.md @@ -2,13 +2,38 @@ Get Flyer Crawler running in 5 minutes. -## Prerequisites +--- -- **Windows 10/11** with WSL 2 -- **Podman Desktop** installed -- **Node.js 20+** installed +## Prerequisites Checklist -## 1. Start Containers (1 minute) +Before starting, verify you have: + +- [ ] **Windows 10/11** with WSL 2 enabled +- [ ] **Podman Desktop** installed ([download](https://podman-desktop.io/)) +- [ ] **Node.js 20+** installed +- [ ] **Git** for cloning the repository + +**Verify Prerequisites**: + +```bash +# Check Podman +podman --version +# Expected: podman version 4.x or higher + +# Check Node.js +node --version +# Expected: v20.x or higher + +# Check WSL +wsl --list --verbose +# Expected: Shows WSL 2 distro +``` + +--- + +## Quick Setup (5 Steps) + +### Step 1: Start Containers (1 minute) ```bash # Start PostgreSQL and Redis @@ -27,11 +52,18 @@ podman run -d --name flyer-crawler-redis \ docker.io/library/redis:alpine ``` -## 2. Initialize Database (2 minutes) +**Expected Output**: + +```text +# Container IDs displayed, no errors +``` + +### Step 2: Initialize Database (2 minutes) ```bash # Wait for PostgreSQL to be ready podman exec flyer-crawler-postgres pg_isready -U postgres +# Expected: localhost:5432 - accepting connections # Install extensions podman exec flyer-crawler-postgres psql -U postgres -d flyer_crawler_dev \ @@ -41,7 +73,17 @@ podman exec flyer-crawler-postgres psql -U postgres -d flyer_crawler_dev \ podman exec -i flyer-crawler-postgres psql -U postgres -d flyer_crawler_dev < sql/master_schema_rollup.sql ``` -## 3. Configure Environment (1 minute) +**Expected Output**: + +```text +CREATE EXTENSION +CREATE EXTENSION +CREATE EXTENSION +CREATE TABLE +... (many tables created) +``` + +### Step 3: Configure Environment (1 minute) Create `.env.local` in the project root: @@ -61,16 +103,22 @@ NODE_ENV=development PORT=3001 FRONTEND_URL=http://localhost:5173 -# Secrets (generate your own) +# Secrets (generate your own - see command below) JWT_SECRET=your-dev-jwt-secret-at-least-32-chars-long SESSION_SECRET=your-dev-session-secret-at-least-32-chars-long # AI Services (get your own keys) -VITE_GOOGLE_GENAI_API_KEY=your-google-genai-api-key +GEMINI_API_KEY=your-google-gemini-api-key GOOGLE_MAPS_API_KEY=your-google-maps-api-key ``` -## 4. Install & Run (1 minute) +**Generate Secure Secrets**: + +```bash +node -e "console.log(require('crypto').randomBytes(32).toString('hex'))" +``` + +### Step 4: Install and Run (1 minute) ```bash # Install dependencies (first time only) @@ -80,35 +128,61 @@ npm install npm run dev ``` -## 5. Access Application +**Expected Output**: -- **Frontend**: http://localhost:5173 -- **Backend API**: http://localhost:3001 -- **Health Check**: http://localhost:3001/health +```text +> flyer-crawler@x.x.x dev +> concurrently ... -### Dev Container (HTTPS) +[API] Server listening on port 3001 +[Vite] VITE ready at http://localhost:5173 +``` -When using the full dev container with NGINX, access via HTTPS: +### Step 5: Verify Installation -- **Frontend**: https://localhost or https://127.0.0.1 -- **Backend API**: http://localhost:3001 -- **Bugsink**: `https://localhost:8443` (error tracking) +| Check | URL/Command | Expected Result | +| ----------- | ------------------------------ | ----------------------------------- | +| Frontend | `http://localhost:5173` | Flyer Crawler app loads | +| Backend API | `http://localhost:3001/health` | `{ "status": "ok", ... }` | +| Database | `podman exec ... psql -c ...` | `SELECT version()` returns Postgres | +| Containers | `podman ps` | Shows postgres and redis running | -**Note:** The dev container accepts both `localhost` and `127.0.0.1` for HTTPS connections. The self-signed certificate is valid for both hostnames. +--- -**SSL Certificate Warnings:** To eliminate browser security warnings for self-signed certificates, install the mkcert CA certificate. See [`certs/README.md`](../../certs/README.md) for platform-specific installation instructions. This is optional but recommended for a better development experience. +## Full Dev Container (Recommended) -### Dev Container Architecture +For a production-like environment with NGINX, Bugsink error tracking, and PM2 process management: -The dev container uses PM2 for process management, matching production (ADR-014): +### Starting the Dev Container -| Process | Description | Port | -| -------------------------- | ------------------------ | ---- | -| `flyer-crawler-api-dev` | API server (tsx watch) | 3001 | -| `flyer-crawler-worker-dev` | Background job worker | - | -| `flyer-crawler-vite-dev` | Vite frontend dev server | 5173 | +```bash +# Start all services +podman-compose -f compose.dev.yml up -d -**PM2 Commands** (run inside container): +# View logs +podman-compose -f compose.dev.yml logs -f +``` + +### Access Points + +| Service | URL | Notes | +| ----------- | ------------------------ | ---------------------------- | +| Frontend | `https://localhost` | NGINX proxy to Vite | +| Backend API | `http://localhost:3001` | Express server | +| Bugsink | `https://localhost:8443` | Error tracking (admin/admin) | +| PostgreSQL | `localhost:5432` | Database | +| Redis | `localhost:6379` | Cache | + +**SSL Certificate Setup (Recommended)**: + +To eliminate browser security warnings, install the mkcert CA certificate: + +```bash +# Windows: Double-click certs/mkcert-ca.crt and install to Trusted Root CAs +# See certs/README.md for detailed instructions per platform +``` + +### PM2 Commands ```bash # View process status @@ -124,63 +198,152 @@ podman exec -it flyer-crawler-dev pm2 restart all podman exec -it flyer-crawler-dev pm2 restart flyer-crawler-api-dev ``` -## Verify Installation +### Dev Container Processes + +| Process | Description | Port | +| -------------------------- | ------------------------ | ---- | +| `flyer-crawler-api-dev` | API server (tsx watch) | 3001 | +| `flyer-crawler-worker-dev` | Background job worker | - | +| `flyer-crawler-vite-dev` | Vite frontend dev server | 5173 | + +--- + +## Verification Commands + +Run these to confirm everything is working: ```bash # Check containers are running podman ps +# Expected: flyer-crawler-postgres and flyer-crawler-redis both running # Test database connection podman exec flyer-crawler-postgres psql -U postgres -d flyer_crawler_dev -c "SELECT version();" +# Expected: PostgreSQL 15.x with PostGIS # Run tests (in dev container) podman exec -it flyer-crawler-dev npm run test:unit +# Expected: All tests pass + +# Run type check +podman exec -it flyer-crawler-dev npm run type-check +# Expected: No type errors ``` -## Common Issues +--- + +## Common Issues and Solutions ### "Unable to connect to Podman socket" +**Cause**: Podman machine not running + +**Solution**: + ```bash podman machine start ``` ### "Connection refused" to PostgreSQL -Wait a few seconds for PostgreSQL to initialize: +**Cause**: PostgreSQL still initializing + +**Solution**: ```bash +# Wait for PostgreSQL to be ready podman exec flyer-crawler-postgres pg_isready -U postgres +# Retry after "accepting connections" message ``` ### Port 5432 or 6379 already in use -Stop conflicting services or change port mappings: +**Cause**: Another service using the port + +**Solution**: ```bash -# Use different host port +# Option 1: Stop conflicting service +# Option 2: Use different host port podman run -d --name flyer-crawler-postgres -p 5433:5432 ... +# Then update DB_PORT=5433 in .env.local ``` -Then update `DB_PORT=5433` in `.env.local`. +### "JWT_SECRET must be at least 32 characters" + +**Cause**: Secret too short in .env.local + +**Solution**: Generate a longer secret: + +```bash +node -e "console.log(require('crypto').randomBytes(32).toString('hex'))" +``` + +### Tests fail with "TZ environment variable" errors + +**Cause**: Timezone setting interfering with Node.js async hooks + +**Solution**: Tests must run in dev container (not Windows host): + +```bash +# CORRECT - run in container +podman exec -it flyer-crawler-dev npm test + +# INCORRECT - do not run on Windows host +npm test +``` + +--- ## Next Steps -- **Read the docs**: [docs/README.md](../README.md) -- **Understand the architecture**: [docs/architecture/DATABASE.md](../architecture/DATABASE.md) -- **Learn testing**: [docs/development/TESTING.md](../development/TESTING.md) -- **Explore ADRs**: [docs/adr/index.md](../adr/index.md) -- **Contributing**: [CONTRIBUTING.md](../../CONTRIBUTING.md) +| Goal | Document | +| ----------------------- | ----------------------------------------------------- | +| Understand the codebase | [Architecture Overview](../architecture/OVERVIEW.md) | +| Configure environment | [Environment Variables](ENVIRONMENT.md) | +| Set up MCP tools | [MCP Configuration](../tools/MCP-CONFIGURATION.md) | +| Learn testing | [Testing Guide](../development/TESTING.md) | +| Understand DB schema | [Database Documentation](../architecture/DATABASE.md) | +| Read ADRs | [ADR Index](../adr/index.md) | +| Full installation guide | [Installation Guide](INSTALL.md) | -## Development Workflow +--- + +## Daily Development Workflow ```bash -# Daily workflow +# 1. Start containers podman start flyer-crawler-postgres flyer-crawler-redis + +# 2. Start dev server npm run dev -# ... make changes ... + +# 3. Make changes and test npm test + +# 4. Type check before commit +npm run type-check + +# 5. Commit changes git commit ``` -For detailed setup instructions, see [INSTALL.md](INSTALL.md). +**For dev container users**: + +```bash +# 1. Start dev container +podman-compose -f compose.dev.yml up -d + +# 2. View logs +podman exec -it flyer-crawler-dev pm2 logs + +# 3. Run tests +podman exec -it flyer-crawler-dev npm test + +# 4. Stop when done +podman-compose -f compose.dev.yml down +``` + +--- + +Last updated: January 2026 diff --git a/docs/operations/BARE-METAL-SETUP.md b/docs/operations/BARE-METAL-SETUP.md index 39f0f9a6..1af508df 100644 --- a/docs/operations/BARE-METAL-SETUP.md +++ b/docs/operations/BARE-METAL-SETUP.md @@ -2,8 +2,54 @@ This guide covers the manual installation of Flyer Crawler and its dependencies on a bare-metal Ubuntu server (e.g., a colocation server). This is the definitive reference for setting up a production environment without containers. +**Last verified**: 2026-01-28 + **Target Environment**: Ubuntu 22.04 LTS (or newer) +**Related documentation**: + +- [ADR-014: Containerization and Deployment Strategy](../adr/0014-containerization-and-deployment-strategy.md) +- [ADR-015: Error Tracking and Observability](../adr/0015-error-tracking-and-observability.md) +- [ADR-050: PostgreSQL Function Observability](../adr/0050-postgresql-function-observability.md) +- [Deployment Guide](DEPLOYMENT.md) +- [Monitoring Guide](MONITORING.md) + +--- + +## Quick Reference + +### Installation Time Estimates + +| Component | Estimated Time | Notes | +| ----------- | --------------- | ----------------------------- | +| PostgreSQL | 10-15 minutes | Including PostGIS extensions | +| Redis | 5 minutes | Quick install | +| Node.js | 5 minutes | Via NodeSource repository | +| Application | 15-20 minutes | Clone, install, build | +| PM2 | 5 minutes | Global install + config | +| NGINX | 10-15 minutes | Including SSL via Certbot | +| Bugsink | 20-30 minutes | Python venv, systemd services | +| Logstash | 15-20 minutes | Including pipeline config | +| **Total** | **~90 minutes** | For complete fresh install | + +### Post-Installation Verification + +After completing setup, verify all services: + +```bash +# Check all services are running +systemctl status postgresql nginx redis-server gunicorn-bugsink snappea logstash + +# Verify application health +curl -s https://flyer-crawler.projectium.com/api/health/ready | jq . + +# Check PM2 processes +pm2 list + +# Verify Bugsink is accessible +curl -s https://bugsink.projectium.com/accounts/login/ | head -5 +``` + --- ## Server Access Model diff --git a/docs/operations/DEPLOYMENT.md b/docs/operations/DEPLOYMENT.md index 82f092e6..c1a084d9 100644 --- a/docs/operations/DEPLOYMENT.md +++ b/docs/operations/DEPLOYMENT.md @@ -2,6 +2,41 @@ This guide covers deploying Flyer Crawler to a production server. +**Last verified**: 2026-01-28 + +**Related documentation**: + +- [ADR-014: Containerization and Deployment Strategy](../adr/0014-containerization-and-deployment-strategy.md) +- [ADR-015: Error Tracking and Observability](../adr/0015-error-tracking-and-observability.md) +- [Bare-Metal Setup Guide](BARE-METAL-SETUP.md) +- [Monitoring Guide](MONITORING.md) + +--- + +## Quick Reference + +### Command Reference Table + +| Task | Command | +| -------------------- | ----------------------------------------------------------------------- | +| Deploy to production | Gitea Actions workflow (manual trigger) | +| Deploy to test | Automatic on push to `main` | +| Check PM2 status | `pm2 list` | +| View logs | `pm2 logs flyer-crawler-api --lines 100` | +| Restart all | `pm2 restart all` | +| Check NGINX | `sudo nginx -t && sudo systemctl status nginx` | +| Check health | `curl -s https://flyer-crawler.projectium.com/api/health/ready \| jq .` | + +### Deployment URLs + +| Environment | URL | API Port | +| ------------- | ------------------------------------------- | -------- | +| Production | `https://flyer-crawler.projectium.com` | 3001 | +| Test | `https://flyer-crawler-test.projectium.com` | 3002 | +| Dev Container | `https://localhost` | 3001 | + +--- + ## Server Access Model **Important**: Claude Code (and AI tools) have **READ-ONLY** access to production/test servers. The deployment workflow is: @@ -24,12 +59,24 @@ When troubleshooting deployment issues: ## Prerequisites -- Ubuntu server (22.04 LTS recommended) -- PostgreSQL 14+ with PostGIS extension -- Redis -- Node.js 20.x -- NGINX (reverse proxy) -- PM2 (process manager) +| Component | Version | Purpose | +| ---------- | --------- | ------------------------------- | +| Ubuntu | 22.04 LTS | Operating system | +| PostgreSQL | 14+ | Database with PostGIS extension | +| Redis | 6+ | Caching and job queues | +| Node.js | 20.x LTS | Application runtime | +| NGINX | 1.18+ | Reverse proxy and static files | +| PM2 | Latest | Process manager | + +**Verify prerequisites**: + +```bash +node --version # Should be v20.x.x +psql --version # Should be 14+ +redis-cli ping # Should return PONG +nginx -v # Should be 1.18+ +pm2 --version # Any recent version +``` ## Dev Container Parity (ADR-014) @@ -210,7 +257,7 @@ types { **Option 2**: Edit `/etc/nginx/mime.types` globally: -``` +```text # Change this line: application/javascript js; @@ -341,9 +388,78 @@ The Sentry SDK v10+ enforces HTTPS-only DSNs by default. Since Bugsink runs loca --- +## Deployment Troubleshooting + +### Decision Tree: Deployment Issues + +```text +Deployment failed? + | + +-- Build step failed? + | | + | +-- TypeScript errors --> Fix type issues, run `npm run type-check` + | +-- Missing dependencies --> Run `npm ci` + | +-- Out of memory --> Increase Node heap size + | + +-- Tests failed? + | | + | +-- Database connection --> Check DB_HOST, credentials + | +-- Redis connection --> Check REDIS_URL + | +-- Test isolation --> Check for race conditions + | + +-- SSH/Deploy failed? + | + +-- Permission denied --> Check SSH keys in Gitea secrets + +-- Host unreachable --> Check firewall, VPN + +-- PM2 error --> Check PM2 logs on server +``` + +### Common Deployment Issues + +| Symptom | Diagnosis | Solution | +| ------------------------------------ | ----------------------- | ------------------------------------------------ | +| "Connection refused" on health check | API not started | Check `pm2 logs flyer-crawler-api` | +| 502 Bad Gateway | NGINX cannot reach API | Verify API port (3001), restart PM2 | +| CSS/JS not loading | Build artifacts missing | Re-run `npm run build`, check NGINX static paths | +| Database migrations failed | Schema mismatch | Run migrations manually, check DB connectivity | +| "ENOSPC" error | Disk full | Clear old logs: `pm2 flush`, clean npm cache | +| SSL certificate error | Cert expired/missing | Run `certbot renew`, check NGINX config | + +### Post-Deployment Verification Checklist + +After every deployment, verify: + +- [ ] Health check passes: `curl -s https://flyer-crawler.projectium.com/api/health/ready` +- [ ] PM2 processes running: `pm2 list` shows `online` status +- [ ] No recent errors: Check Bugsink for new issues +- [ ] Frontend loads: Browser shows login page +- [ ] API responds: `curl https://flyer-crawler.projectium.com/api/health/ping` + +### Rollback Procedure + +If deployment causes issues: + +```bash +# 1. Check current release +cd /var/www/flyer-crawler.projectium.com +git log --oneline -5 + +# 2. Revert to previous commit +git checkout HEAD~1 + +# 3. Rebuild and restart +npm ci && npm run build +pm2 restart all + +# 4. Verify health +curl -s http://localhost:3001/api/health/ready | jq . +``` + +--- + ## Related Documentation -- [Database Setup](DATABASE.md) - PostgreSQL and PostGIS configuration -- [Authentication Setup](AUTHENTICATION.md) - OAuth provider configuration -- [Installation Guide](INSTALL.md) - Local development setup -- [Bare-Metal Server Setup](docs/BARE-METAL-SETUP.md) - Manual server installation guide +- [Database Setup](../architecture/DATABASE.md) - PostgreSQL and PostGIS configuration +- [Monitoring Guide](MONITORING.md) - Health checks and error tracking +- [Logstash Quick Reference](LOGSTASH-QUICK-REF.md) - Log aggregation +- [Bare-Metal Server Setup](BARE-METAL-SETUP.md) - Manual server installation guide diff --git a/docs/operations/LOGSTASH-QUICK-REF.md b/docs/operations/LOGSTASH-QUICK-REF.md index 3df284f4..83ba109c 100644 --- a/docs/operations/LOGSTASH-QUICK-REF.md +++ b/docs/operations/LOGSTASH-QUICK-REF.md @@ -2,10 +2,47 @@ Aggregates logs from PostgreSQL, PM2, Redis, NGINX; forwards errors to Bugsink. +**Last verified**: 2026-01-28 + +**Related documentation**: + +- [ADR-050: PostgreSQL Function Observability](../adr/0050-postgresql-function-observability.md) +- [ADR-015: Error Tracking and Observability](../adr/0015-error-tracking-and-observability.md) +- [Monitoring Guide](MONITORING.md) +- [Logstash Troubleshooting Runbook](LOGSTASH-TROUBLESHOOTING.md) + +--- + +## Quick Reference + +### Bugsink Project Routing + +| Source Type | Environment | Bugsink Project | Project ID | +| -------------- | ----------- | -------------------- | ---------- | +| PM2 API/Worker | Dev | Backend API (Dev) | 1 | +| PostgreSQL | Dev | Backend API (Dev) | 1 | +| Frontend JS | Dev | Frontend (Dev) | 2 | +| Redis/NGINX | Dev | Infrastructure (Dev) | 4 | +| PM2 API/Worker | Production | Backend API (Prod) | 1 | +| PostgreSQL | Production | Backend API (Prod) | 1 | +| PM2 API/Worker | Test | Backend API (Test) | 3 | + +### Key DSN Keys (Dev Container) + +| Project | DSN Key | +| -------------------- | ---------------------------------- | +| Backend API (Dev) | `cea01396c56246adb5878fa5ee6b1d22` | +| Frontend (Dev) | `d92663cb73cf4145b677b84029e4b762` | +| Infrastructure (Dev) | `14e8791da3d347fa98073261b596cab9` | + +--- + ## Configuration **Primary config**: `/etc/logstash/conf.d/bugsink.conf` +**Dev container config**: `docker/logstash/bugsink.conf` + ### Related Files | Path | Purpose | @@ -89,6 +126,34 @@ MSYS_NO_PATHCONV=1 podman exec flyer-crawler-dev ls -la /var/log/redis/ ## Troubleshooting +### Decision Tree: Logs Not Appearing in Bugsink + +```text +Errors not showing in Bugsink? + | + +-- Logstash running? + | | + | +-- No --> systemctl start logstash + | +-- Yes --> Check pipeline stats + | | + | +-- Events in = 0? + | | | + | | +-- Log files exist? --> ls /var/log/pm2/*.log + | | +-- Permissions OK? --> groups logstash + | | + | +-- Events filtered = high? + | | | + | | +-- Grok failures --> Check log format matches pattern + | | + | +-- Events out but no Bugsink? + | | + | +-- 403 error --> Wrong DSN key + | +-- 500 error --> Invalid event format (check sentry_level) + | +-- Connection refused --> Bugsink not running +``` + +### Common Issues Table + | Issue | Check | Solution | | --------------------- | ---------------- | ---------------------------------------------------------------------------------------------- | | No Bugsink errors | Logstash running | `systemctl status logstash` | @@ -103,6 +168,25 @@ MSYS_NO_PATHCONV=1 podman exec flyer-crawler-dev ls -la /var/log/redis/ | High disk usage | Log rotation | Verify `/etc/logrotate.d/logstash` configured | | varchar(7) error | Level validation | Add Ruby filter to validate/normalize `sentry_level` before output | +### Expected Output Examples + +**Successful Logstash pipeline stats**: + +```json +{ + "in": 1523, + "out": 1520, + "filtered": 1520, + "queue_push_duration_in_millis": 45 +} +``` + +**Healthy Bugsink HTTP response**: + +```json +{ "id": "a1b2c3d4e5f6..." } +``` + ## Related Documentation - **Dev Container Guide**: [DEV-CONTAINER.md](../development/DEV-CONTAINER.md) - PM2 and log aggregation in dev diff --git a/docs/operations/LOGSTASH-TROUBLESHOOTING.md b/docs/operations/LOGSTASH-TROUBLESHOOTING.md index 93219231..e4cdd4c0 100644 --- a/docs/operations/LOGSTASH-TROUBLESHOOTING.md +++ b/docs/operations/LOGSTASH-TROUBLESHOOTING.md @@ -2,6 +2,16 @@ This runbook provides step-by-step diagnostics and solutions for common Logstash issues in the PostgreSQL observability pipeline (ADR-050). +**Last verified**: 2026-01-28 + +**Related documentation**: + +- [ADR-050: PostgreSQL Function Observability](../adr/0050-postgresql-function-observability.md) +- [Logstash Quick Reference](LOGSTASH-QUICK-REF.md) +- [Monitoring Guide](MONITORING.md) + +--- + ## Quick Reference | Symptom | Most Likely Cause | Quick Check | diff --git a/docs/operations/MONITORING.md b/docs/operations/MONITORING.md index 791369a6..729f3a14 100644 --- a/docs/operations/MONITORING.md +++ b/docs/operations/MONITORING.md @@ -2,6 +2,72 @@ This guide covers all aspects of monitoring the Flyer Crawler application across development, test, and production environments. +**Last verified**: 2026-01-28 + +**Related documentation**: + +- [ADR-015: Error Tracking and Observability](../adr/0015-error-tracking-and-observability.md) +- [ADR-020: Health Checks](../adr/0020-health-checks-and-liveness-readiness-probes.md) +- [ADR-050: PostgreSQL Function Observability](../adr/0050-postgresql-function-observability.md) +- [Logstash Quick Reference](LOGSTASH-QUICK-REF.md) +- [Deployment Guide](DEPLOYMENT.md) + +--- + +## Quick Reference + +### Monitoring URLs + +| Service | Production URL | Dev Container URL | +| ------------ | ------------------------------------------------------- | ---------------------------------------- | +| Health Check | `https://flyer-crawler.projectium.com/api/health/ready` | `http://localhost:3001/api/health/ready` | +| Bugsink | `https://bugsink.projectium.com` | `https://localhost:8443` | +| Bull Board | `https://flyer-crawler.projectium.com/api/admin/jobs` | `http://localhost:3001/api/admin/jobs` | + +### Quick Diagnostic Commands + +```bash +# Check all services at once (production) +curl -s https://flyer-crawler.projectium.com/api/health/ready | jq '.data.services' + +# Dev container health check +podman exec flyer-crawler-dev curl -s http://localhost:3001/api/health/ready | jq . + +# PM2 process overview +pm2 list + +# Recent errors in Bugsink (via MCP) +# mcp__bugsink__list_issues --project_id 1 --status unresolved +``` + +### Monitoring Decision Tree + +```text +Application seems slow or unresponsive? + | + +-- Check health endpoint first + | | + | +-- Returns unhealthy? + | | | + | | +-- Database unhealthy --> Check DB pool, connections + | | +-- Redis unhealthy --> Check Redis memory, connection + | | +-- Storage unhealthy --> Check disk space, permissions + | | + | +-- Returns healthy but slow? + | | + | +-- Check PM2 memory/CPU usage + | +-- Check database slow query log + | +-- Check Redis queue depth + | + +-- Health endpoint not responding? + | + +-- Check PM2 status --> Process crashed? + +-- Check NGINX --> 502 errors? + +-- Check network --> Firewall/DNS issues? +``` + +--- + ## Table of Contents 1. [Health Checks](#health-checks) @@ -294,7 +360,7 @@ The command outputs a 40-character hex token. **Error Anatomy**: -``` +```text TypeError: Cannot read properties of undefined (reading 'map') ├── Exception Type: TypeError ├── Message: Cannot read properties of undefined (reading 'map') @@ -357,7 +423,7 @@ Logstash aggregates logs from multiple sources and forwards errors to Bugsink (A ### Architecture -``` +```text Log Sources Logstash Outputs ┌──────────────┐ ┌─────────────┐ ┌─────────────┐ │ PostgreSQL │──────────────│ │───────────│ Bugsink │ @@ -520,7 +586,7 @@ pm2 stop flyer-crawler-api **Healthy Process**: -``` +```text ┌─────────────────────┬────┬─────────┬─────────┬───────┬────────┬─────────┬──────────┐ │ Name │ id │ mode │ status │ cpu │ mem │ uptime │ restarts │ ├─────────────────────┼────┼─────────┼─────────┼───────┼────────┼─────────┼──────────┤ @@ -833,7 +899,7 @@ Configure alerts in your monitoring tool (UptimeRobot, Datadog, etc.): 2. Review during business hours 3. Create Gitea issue for tracking -### Quick Diagnostic Commands +### On-Call Diagnostic Commands > **Note**: User executes these commands on the server. Claude Code provides commands but cannot run them directly. diff --git a/docs/subagents/AI-USAGE-GUIDE.md b/docs/subagents/AI-USAGE-GUIDE.md index 730b2766..61c6bed4 100644 --- a/docs/subagents/AI-USAGE-GUIDE.md +++ b/docs/subagents/AI-USAGE-GUIDE.md @@ -2,6 +2,17 @@ The **ai-usage** subagent specializes in LLM APIs (Gemini, Claude), prompt engineering, and AI-powered features in the Flyer Crawler project. +## Quick Reference + +| Aspect | Details | +| ------------------ | ----------------------------------------------------------------------------------- | +| **Primary Use** | Gemini API integration, prompt engineering, AI extraction | +| **Key Files** | `src/services/aiService.server.ts`, `src/services/flyerProcessingService.server.ts` | +| **Key ADRs** | ADR-041 (AI Integration), ADR-046 (Image Processing) | +| **API Key Env** | `VITE_GOOGLE_GENAI_API_KEY` (prod), `VITE_GOOGLE_GENAI_API_KEY_TEST` (test) | +| **Error Handling** | Rate limits (429), JSON parse errors, timeout handling | +| **Delegate To** | `coder` (implementation), `testwriter` (tests), `integrations-specialist` | + ## When to Use Use the **ai-usage** subagent when you need to: @@ -295,6 +306,9 @@ const fixtureResponse = await fs.readFile('fixtures/gemini-response.json'); ## Related Documentation - [OVERVIEW.md](./OVERVIEW.md) - Subagent system overview +- [CODER-GUIDE.md](./CODER-GUIDE.md) - For implementing AI features +- [TESTER-GUIDE.md](./TESTER-GUIDE.md) - Testing AI features +- [INTEGRATIONS-GUIDE.md](./INTEGRATIONS-GUIDE.md) - External API patterns - [../adr/0041-ai-gemini-integration-architecture.md](../adr/0041-ai-gemini-integration-architecture.md) - AI integration ADR - [../adr/0046-image-processing-pipeline.md](../adr/0046-image-processing-pipeline.md) - Image processing -- [CODER-GUIDE.md](./CODER-GUIDE.md) - For implementing AI features +- [../getting-started/ENVIRONMENT.md](../getting-started/ENVIRONMENT.md) - Environment configuration diff --git a/docs/subagents/CODER-GUIDE.md b/docs/subagents/CODER-GUIDE.md index 87606143..b492b4e0 100644 --- a/docs/subagents/CODER-GUIDE.md +++ b/docs/subagents/CODER-GUIDE.md @@ -2,6 +2,17 @@ The **coder** subagent is your primary tool for writing and modifying production Node.js/TypeScript code in the Flyer Crawler project. This guide explains how to work effectively with the coder subagent. +## Quick Reference + +| Aspect | Details | +| ---------------- | ------------------------------------------------------------------------ | +| **Primary Use** | Write/modify production TypeScript code | +| **Key Files** | `src/routes/*.routes.ts`, `src/services/**/*.ts`, `src/components/*.tsx` | +| **Key ADRs** | ADR-034 (Repository), ADR-035 (Services), ADR-028 (API Response) | +| **Test Command** | `podman exec -it flyer-crawler-dev npm run test:unit` | +| **Type Check** | `podman exec -it flyer-crawler-dev npm run type-check` | +| **Delegate To** | `db-dev` (database), `frontend-specialist` (UI), `testwriter` (tests) | + ## When to Use the Coder Subagent Use the coder subagent when you need to: @@ -307,6 +318,8 @@ error classes for all database operations" - [OVERVIEW.md](./OVERVIEW.md) - Subagent system overview - [TESTER-GUIDE.md](./TESTER-GUIDE.md) - Testing strategies +- [DATABASE-GUIDE.md](./DATABASE-GUIDE.md) - Database development workflows - [../adr/0034-repository-pattern-standards.md](../adr/0034-repository-pattern-standards.md) - Repository patterns - [../adr/0035-service-layer-architecture.md](../adr/0035-service-layer-architecture.md) - Service layer architecture - [../adr/0028-api-response-standardization.md](../adr/0028-api-response-standardization.md) - API response patterns +- [../development/CODE-PATTERNS.md](../development/CODE-PATTERNS.md) - Code patterns reference diff --git a/docs/subagents/DATABASE-GUIDE.md b/docs/subagents/DATABASE-GUIDE.md index 8b652ed6..36363ce4 100644 --- a/docs/subagents/DATABASE-GUIDE.md +++ b/docs/subagents/DATABASE-GUIDE.md @@ -5,6 +5,17 @@ This guide covers two database-focused subagents: - **db-dev**: Database development - schemas, queries, migrations, optimization - **db-admin**: Database administration - PostgreSQL/Redis admin, security, backups +## Quick Reference + +| Aspect | db-dev | db-admin | +| ---------------- | -------------------------------------------- | ------------------------------------------ | +| **Primary Use** | Schemas, queries, migrations | Performance tuning, backups, security | +| **Key Files** | `src/services/db/*.db.ts`, `sql/migrations/` | `postgresql.conf`, `pg_hba.conf` | +| **Key ADRs** | ADR-034 (Repository), ADR-002 (Transactions) | ADR-019 (Backups), ADR-050 (Observability) | +| **Test Command** | `podman exec -it flyer-crawler-dev npm test` | N/A | +| **MCP Tool** | `mcp__devdb__query` | SSH to production | +| **Delegate To** | `coder` (service layer), `db-admin` (perf) | `devops` (infrastructure) | + ## Understanding the Difference | Aspect | db-dev | db-admin | @@ -412,8 +423,9 @@ This is useful for: - [OVERVIEW.md](./OVERVIEW.md) - Subagent system overview - [CODER-GUIDE.md](./CODER-GUIDE.md) - Working with the coder subagent +- [DEVOPS-GUIDE.md](./DEVOPS-GUIDE.md) - DevOps and deployment workflows - [../adr/0034-repository-pattern-standards.md](../adr/0034-repository-pattern-standards.md) - Repository patterns - [../adr/0002-standardized-transaction-management.md](../adr/0002-standardized-transaction-management.md) - Transaction management - [../adr/0019-data-backup-and-recovery-strategy.md](../adr/0019-data-backup-and-recovery-strategy.md) - Backup strategy - [../adr/0050-postgresql-function-observability.md](../adr/0050-postgresql-function-observability.md) - Database observability -- [../BARE-METAL-SETUP.md](../BARE-METAL-SETUP.md) - Production database setup +- [../operations/BARE-METAL-SETUP.md](../operations/BARE-METAL-SETUP.md) - Production database setup diff --git a/docs/subagents/DEVOPS-GUIDE.md b/docs/subagents/DEVOPS-GUIDE.md index 7d7fb6a7..c16efb12 100644 --- a/docs/subagents/DEVOPS-GUIDE.md +++ b/docs/subagents/DEVOPS-GUIDE.md @@ -6,6 +6,17 @@ This guide covers DevOps-related subagents for deployment, infrastructure, and o - **infra-architect**: Resource optimization, capacity planning - **bg-worker**: Background jobs, PM2 workers, BullMQ queues +## Quick Reference + +| Aspect | devops | infra-architect | bg-worker | +| ---------------- | ------------------------------------------ | --------------------------- | ------------------------------- | +| **Primary Use** | Containers, CI/CD, deployments | Resource optimization | BullMQ queues, PM2 workers | +| **Key Files** | `compose.dev.yml`, `.gitea/workflows/` | `ecosystem.config.cjs` | `src/services/queues.server.ts` | +| **Key ADRs** | ADR-014 (Containers), ADR-017 (CI/CD) | N/A | ADR-006 (Background Jobs) | +| **Commands** | `podman-compose`, `pm2` | `pm2 monit`, system metrics | Redis CLI, `pm2 logs` | +| **MCP Tools** | `mcp__podman__*` | N/A | N/A | +| **Access Model** | Read-only on production (provide commands) | Same | Same | + --- ## CRITICAL: Server Access Model @@ -543,8 +554,13 @@ podman exec -it flyer-crawler-dev npm test ## Related Documentation - [OVERVIEW.md](./OVERVIEW.md) - Subagent system overview -- [../BARE-METAL-SETUP.md](../BARE-METAL-SETUP.md) - Production setup guide +- [DATABASE-GUIDE.md](./DATABASE-GUIDE.md) - Database administration +- [SECURITY-DEBUG-GUIDE.md](./SECURITY-DEBUG-GUIDE.md) - Production debugging +- [../operations/BARE-METAL-SETUP.md](../operations/BARE-METAL-SETUP.md) - Production setup guide +- [../operations/DEPLOYMENT.md](../operations/DEPLOYMENT.md) - Deployment guide +- [../operations/MONITORING.md](../operations/MONITORING.md) - Monitoring guide +- [../development/DEV-CONTAINER.md](../development/DEV-CONTAINER.md) - Dev container guide - [../adr/0014-containerization-and-deployment-strategy.md](../adr/0014-containerization-and-deployment-strategy.md) - Containerization ADR - [../adr/0006-background-job-processing-and-task-queues.md](../adr/0006-background-job-processing-and-task-queues.md) - Background jobs ADR - [../adr/0017-ci-cd-and-branching-strategy.md](../adr/0017-ci-cd-and-branching-strategy.md) - CI/CD strategy -- [../adr/0053-worker-health-checks.md](../adr/0053-worker-health-checks.md) - Worker health checks +- [../adr/0053-worker-health-checks-and-monitoring.md](../adr/0053-worker-health-checks-and-monitoring.md) - Worker health checks diff --git a/docs/subagents/DOCUMENTATION-GUIDE.md b/docs/subagents/DOCUMENTATION-GUIDE.md index 235ba52d..006802ac 100644 --- a/docs/subagents/DOCUMENTATION-GUIDE.md +++ b/docs/subagents/DOCUMENTATION-GUIDE.md @@ -7,6 +7,15 @@ This guide covers documentation-focused subagents: - **planner**: Feature breakdown, roadmaps, scope management - **product-owner**: Requirements, user stories, backlog prioritization +## Quick Reference + +| Aspect | documenter | describer-for-ai | planner | product-owner | +| --------------- | -------------------- | ------------------------ | --------------------- | ---------------------- | +| **Primary Use** | User docs, API specs | ADRs, technical specs | Feature breakdown | User stories, backlog | +| **Key Files** | `docs/`, API docs | `docs/adr/`, `CLAUDE.md` | `docs/plans/` | Issue tracker | +| **Output** | Markdown guides | ADRs, context docs | Task lists, roadmaps | User stories, criteria | +| **Delegate To** | `coder` (implement) | `documenter` (user docs) | `coder` (build tasks) | `planner` (breakdown) | + ## The documenter Subagent ### When to Use @@ -437,6 +446,8 @@ Include dates on documentation that may become stale: ## Related Documentation - [OVERVIEW.md](./OVERVIEW.md) - Subagent system overview +- [CODER-GUIDE.md](./CODER-GUIDE.md) - For implementing documented features - [../adr/index.md](../adr/index.md) - ADR index -- [../TESTING.md](../TESTING.md) - Testing guide +- [../development/TESTING.md](../development/TESTING.md) - Testing guide +- [../development/CODE-PATTERNS.md](../development/CODE-PATTERNS.md) - Code patterns reference - [../../CLAUDE.md](../../CLAUDE.md) - AI instructions diff --git a/docs/subagents/FRONTEND-GUIDE.md b/docs/subagents/FRONTEND-GUIDE.md index 4f2d90c3..dbfb3d46 100644 --- a/docs/subagents/FRONTEND-GUIDE.md +++ b/docs/subagents/FRONTEND-GUIDE.md @@ -5,6 +5,17 @@ This guide covers frontend-focused subagents: - **frontend-specialist**: UI components, Neo-Brutalism, Core Web Vitals, accessibility - **uiux-designer**: UI/UX decisions, component design, user experience +## Quick Reference + +| Aspect | frontend-specialist | uiux-designer | +| ----------------- | ---------------------------------------------- | -------------------------------------- | +| **Primary Use** | React components, performance, accessibility | Design decisions, user flows | +| **Key Files** | `src/components/`, `src/features/` | Design specs, mockups | +| **Key ADRs** | ADR-012 (Design System), ADR-044 (Feature Org) | ADR-012 (Design System) | +| **Design System** | Neo-Brutalism (bold borders, high contrast) | Same | +| **State Mgmt** | TanStack Query (server), Zustand (client) | N/A | +| **Delegate To** | `coder` (backend), `tester` (test coverage) | `frontend-specialist` (implementation) | + ## The frontend-specialist Subagent ### When to Use @@ -406,7 +417,8 @@ const handleSelect = useCallback((id: string) => { - [OVERVIEW.md](./OVERVIEW.md) - Subagent system overview - [CODER-GUIDE.md](./CODER-GUIDE.md) - For implementing features -- [../DESIGN_TOKENS.md](../DESIGN_TOKENS.md) - Design token reference +- [TESTER-GUIDE.md](./TESTER-GUIDE.md) - Component testing patterns +- [../development/DESIGN_TOKENS.md](../development/DESIGN_TOKENS.md) - Design token reference - [../adr/0012-frontend-component-library-and-design-system.md](../adr/0012-frontend-component-library-and-design-system.md) - Design system ADR - [../adr/0005-frontend-state-management-and-server-cache-strategy.md](../adr/0005-frontend-state-management-and-server-cache-strategy.md) - State management ADR - [../adr/0044-frontend-feature-organization.md](../adr/0044-frontend-feature-organization.md) - Feature organization diff --git a/docs/subagents/INTEGRATIONS-GUIDE.md b/docs/subagents/INTEGRATIONS-GUIDE.md new file mode 100644 index 00000000..73ecde9c --- /dev/null +++ b/docs/subagents/INTEGRATIONS-GUIDE.md @@ -0,0 +1,396 @@ +# Integrations Subagent Guide + +The **integrations-specialist** subagent handles third-party services, webhooks, and external API integrations in the Flyer Crawler project. + +## Quick Reference + +| Aspect | Details | +| --------------- | --------------------------------------------------------------------------- | +| **Primary Use** | External APIs, webhooks, OAuth, third-party services | +| **Key Files** | `src/services/external/`, `src/routes/webhooks.routes.ts` | +| **Key ADRs** | ADR-041 (AI Integration), ADR-016 (API Security), ADR-048 (Auth) | +| **MCP Tools** | `mcp__gitea-projectium__*`, `mcp__bugsink__*` | +| **Security** | API key storage, webhook signatures, OAuth state param | +| **Delegate To** | `coder` (implementation), `security-engineer` (review), `ai-usage` (Gemini) | + +## When to Use + +Use the **integrations-specialist** subagent when you need to: + +- Integrate with external APIs (OAuth, REST, GraphQL) +- Implement webhook handlers +- Configure third-party services +- Debug external service connectivity +- Handle API authentication flows +- Manage external service rate limits + +## What integrations-specialist Knows + +The integrations-specialist subagent understands: + +- OAuth 2.0 flows (authorization code, client credentials) +- REST API integration patterns +- Webhook security (signature verification) +- External service error handling +- Rate limiting and retry strategies +- API key management + +## Current Integrations + +| Service | Purpose | Integration Type | Key Files | +| ------------- | ---------------------- | ---------------- | ---------------------------------- | +| Google Gemini | AI flyer extraction | REST API | `src/services/aiService.server.ts` | +| Bugsink | Error tracking | REST API | MCP: `mcp__bugsink__*` | +| Gitea | Repository and CI/CD | REST API | MCP: `mcp__gitea-projectium__*` | +| Redis | Caching and job queues | Native client | `src/services/redis.server.ts` | +| PostgreSQL | Primary database | Native client | `src/services/db/pool.db.ts` | + +## Example Requests + +### Adding External API Integration + +``` +"Use integrations-specialist to integrate with the Store API +to automatically fetch store location data. Include proper +error handling, rate limiting, and caching." +``` + +### OAuth Implementation + +``` +"Use integrations-specialist to implement Google OAuth for +user authentication. Include token refresh handling and +session management." +``` + +### Webhook Handler + +``` +"Use integrations-specialist to create a webhook handler for +receiving store inventory updates. Include signature verification +and idempotency handling." +``` + +### Debugging External Service Issues + +``` +"Use integrations-specialist to debug why the Gemini API calls +are intermittently failing with timeout errors. Check connection +pooling, retry logic, and error handling." +``` + +## Integration Patterns + +### REST API Client Pattern + +```typescript +// src/services/external/storeApi.server.ts +import { env } from '@/config/env'; +import { log } from '@/services/logger.server'; + +interface StoreApiConfig { + baseUrl: string; + apiKey: string; + timeout: number; +} + +class StoreApiClient { + private config: StoreApiConfig; + + constructor(config: StoreApiConfig) { + this.config = config; + } + + async getStoreLocations(storeId: string): Promise { + const url = `${this.config.baseUrl}/stores/${storeId}/locations`; + + try { + const response = await fetch(url, { + headers: { + Authorization: `Bearer ${this.config.apiKey}`, + 'Content-Type': 'application/json', + }, + signal: AbortSignal.timeout(this.config.timeout), + }); + + if (!response.ok) { + throw new ExternalApiError(`Store API error: ${response.status}`, response.status); + } + + return response.json(); + } catch (error) { + log.error({ error, storeId }, 'Failed to fetch store locations'); + throw error; + } + } +} + +export const storeApiClient = new StoreApiClient({ + baseUrl: env.STORE_API_BASE_URL, + apiKey: env.STORE_API_KEY, + timeout: 10000, +}); +``` + +### Webhook Handler Pattern + +```typescript +// src/routes/webhooks.routes.ts +import { Router } from 'express'; +import crypto from 'crypto'; +import { env } from '@/config/env'; + +const router = Router(); + +function verifyWebhookSignature(payload: string, signature: string, secret: string): boolean { + const expected = crypto.createHmac('sha256', secret).update(payload).digest('hex'); + return crypto.timingSafeEqual(Buffer.from(signature), Buffer.from(`sha256=${expected}`)); +} + +router.post('/store-updates', async (req, res, next) => { + try { + const signature = req.headers['x-webhook-signature'] as string; + const payload = JSON.stringify(req.body); + + if (!verifyWebhookSignature(payload, signature, env.WEBHOOK_SECRET)) { + return res.status(401).json({ error: 'Invalid signature' }); + } + + // Process webhook with idempotency check + const eventId = req.headers['x-event-id'] as string; + const alreadyProcessed = await checkIdempotencyKey(eventId); + + if (alreadyProcessed) { + return res.status(200).json({ status: 'already_processed' }); + } + + await processStoreUpdate(req.body); + await markEventProcessed(eventId); + + res.status(200).json({ status: 'processed' }); + } catch (error) { + next(error); + } +}); +``` + +### OAuth Flow Pattern + +```typescript +// src/services/oauth/googleOAuth.server.ts +import { OAuth2Client } from 'google-auth-library'; +import { env } from '@/config/env'; + +const oauth2Client = new OAuth2Client( + env.GOOGLE_CLIENT_ID, + env.GOOGLE_CLIENT_SECRET, + env.GOOGLE_REDIRECT_URI, +); + +export function getAuthorizationUrl(): string { + return oauth2Client.generateAuthUrl({ + access_type: 'offline', + scope: ['email', 'profile'], + prompt: 'consent', + }); +} + +export async function exchangeCodeForTokens(code: string) { + const { tokens } = await oauth2Client.getToken(code); + return tokens; +} + +export async function refreshAccessToken(refreshToken: string) { + oauth2Client.setCredentials({ refresh_token: refreshToken }); + const { credentials } = await oauth2Client.refreshAccessToken(); + return credentials; +} +``` + +## Error Handling for External Services + +### Custom Error Classes + +```typescript +// src/services/external/errors.ts +export class ExternalApiError extends Error { + constructor( + message: string, + public statusCode: number, + public retryable: boolean = false, + ) { + super(message); + this.name = 'ExternalApiError'; + } +} + +export class RateLimitError extends ExternalApiError { + constructor( + message: string, + public retryAfter: number, + ) { + super(message, 429, true); + this.name = 'RateLimitError'; + } +} +``` + +### Retry with Exponential Backoff + +```typescript +async function fetchWithRetry( + fn: () => Promise, + options: { maxRetries: number; baseDelay: number }, +): Promise { + let lastError: Error; + + for (let attempt = 0; attempt <= options.maxRetries; attempt++) { + try { + return await fn(); + } catch (error) { + lastError = error as Error; + + if (error instanceof ExternalApiError && !error.retryable) { + throw error; + } + + if (attempt < options.maxRetries) { + const delay = options.baseDelay * Math.pow(2, attempt); + await new Promise((resolve) => setTimeout(resolve, delay)); + } + } + } + + throw lastError!; +} +``` + +## Rate Limiting Strategies + +### Token Bucket Pattern + +```typescript +class RateLimiter { + private tokens: number; + private lastRefill: number; + private readonly maxTokens: number; + private readonly refillRate: number; // tokens per second + + constructor(maxTokens: number, refillRate: number) { + this.maxTokens = maxTokens; + this.tokens = maxTokens; + this.refillRate = refillRate; + this.lastRefill = Date.now(); + } + + async acquire(): Promise { + this.refill(); + + if (this.tokens < 1) { + const waitTime = ((1 - this.tokens) / this.refillRate) * 1000; + await new Promise((resolve) => setTimeout(resolve, waitTime)); + this.refill(); + } + + this.tokens -= 1; + } + + private refill(): void { + const now = Date.now(); + const elapsed = (now - this.lastRefill) / 1000; + this.tokens = Math.min(this.maxTokens, this.tokens + elapsed * this.refillRate); + this.lastRefill = now; + } +} +``` + +## Testing Integrations + +### Mocking External Services + +```typescript +// src/tests/mocks/storeApi.mock.ts +import { vi } from 'vitest'; + +export const mockStoreApiClient = { + getStoreLocations: vi.fn(), +}; + +vi.mock('@/services/external/storeApi.server', () => ({ + storeApiClient: mockStoreApiClient, +})); +``` + +### Integration Test with Real Service + +```typescript +// src/tests/integration/storeApi.integration.test.ts +describe('Store API Integration', () => { + it.skipIf(!env.STORE_API_KEY)('fetches real store locations', async () => { + const locations = await storeApiClient.getStoreLocations('test-store'); + expect(locations).toBeInstanceOf(Array); + }); +}); +``` + +## MCP Tools for Integrations + +### Gitea Integration + +``` +// List repositories +mcp__gitea-projectium__list_my_repos() + +// Create issue +mcp__gitea-projectium__create_issue({ + owner: "projectium", + repo: "flyer-crawler", + title: "Issue title", + body: "Issue description" +}) +``` + +### Bugsink Integration + +``` +// List projects +mcp__bugsink__list_projects() + +// Get issue details +mcp__bugsink__get_issue({ issue_id: "..." }) + +// Get stacktrace +mcp__bugsink__get_stacktrace({ event_id: "..." }) +``` + +## Security Considerations + +### API Key Storage + +- Never commit API keys to version control +- Use environment variables via `src/config/env.ts` +- Rotate keys periodically +- Use separate keys for dev/test/prod + +### Webhook Security + +- Always verify webhook signatures +- Use HTTPS for webhook endpoints +- Implement idempotency +- Log webhook events for audit + +### OAuth Security + +- Use state parameter to prevent CSRF +- Store tokens securely (encrypted at rest) +- Implement token refresh before expiration +- Validate token scopes + +## Related Documentation + +- [OVERVIEW.md](./OVERVIEW.md) - Subagent system overview +- [SECURITY-DEBUG-GUIDE.md](./SECURITY-DEBUG-GUIDE.md) - Security patterns +- [AI-USAGE-GUIDE.md](./AI-USAGE-GUIDE.md) - Gemini API integration +- [../adr/0041-ai-gemini-integration-architecture.md](../adr/0041-ai-gemini-integration-architecture.md) - AI integration ADR +- [../adr/0016-api-security-hardening.md](../adr/0016-api-security-hardening.md) - API security +- [../adr/0048-authentication-strategy.md](../adr/0048-authentication-strategy.md) - Authentication diff --git a/docs/subagents/OVERVIEW.md b/docs/subagents/OVERVIEW.md index f93d4384..017c6c6e 100644 --- a/docs/subagents/OVERVIEW.md +++ b/docs/subagents/OVERVIEW.md @@ -89,6 +89,47 @@ Or: Claude will automatically invoke the appropriate subagent with the relevant context. +## Quick Reference Decision Tree + +Use this flowchart to quickly identify the right subagent: + +``` +What do you need to do? +| ++-- Write/modify code? ----------------> Is it database-related? +| | +| +-- Yes -> db-dev +| +-- No --> Is it frontend? +| | +| +-- Yes -> frontend-specialist +| +-- No --> Is it AI/Gemini? +| | +| +-- Yes -> ai-usage +| +-- No --> coder +| ++-- Test something? -------------------> Write new tests? -> testwriter +| Find bugs/vulnerabilities? -> tester +| Review existing code? -> code-reviewer +| ++-- Debug an issue? -------------------> Production error? -> log-debug +| Database slow? -> db-admin +| External API failing? -> integrations-specialist +| AI extraction failing? -> ai-usage +| ++-- Infrastructure/Deployment? --------> Container/CI/CD? -> devops +| Resource optimization? -> infra-architect +| Background jobs? -> bg-worker +| ++-- Documentation? --------------------> User-facing docs? -> documenter +| ADRs/Technical specs? -> describer-for-ai +| Feature planning? -> planner +| User stories? -> product-owner +| ++-- Security? -------------------------> security-engineer +| ++-- Design/UX? ------------------------> uiux-designer +``` + ## Subagent Selection Guide ### Which Subagent Should I Use? @@ -183,12 +224,26 @@ Subagents can pass information back to the main conversation and to each other t ## Related Documentation -- [CODER-GUIDE.md](./CODER-GUIDE.md) - Working with the coder subagent -- [TESTER-GUIDE.md](./TESTER-GUIDE.md) - Testing strategies and patterns -- [DATABASE-GUIDE.md](./DATABASE-GUIDE.md) - Database development workflows -- [DEVOPS-GUIDE.md](./DEVOPS-GUIDE.md) - DevOps and deployment workflows +### Subagent Guides + +| Guide | Subagents Covered | +| ---------------------------------------------------- | ----------------------------------------------------- | +| [CODER-GUIDE.md](./CODER-GUIDE.md) | coder | +| [TESTER-GUIDE.md](./TESTER-GUIDE.md) | tester, testwriter | +| [DATABASE-GUIDE.md](./DATABASE-GUIDE.md) | db-dev, db-admin | +| [DEVOPS-GUIDE.md](./DEVOPS-GUIDE.md) | devops, infra-architect, bg-worker | +| [FRONTEND-GUIDE.md](./FRONTEND-GUIDE.md) | frontend-specialist, uiux-designer | +| [SECURITY-DEBUG-GUIDE.md](./SECURITY-DEBUG-GUIDE.md) | security-engineer, log-debug, code-reviewer | +| [AI-USAGE-GUIDE.md](./AI-USAGE-GUIDE.md) | ai-usage | +| [INTEGRATIONS-GUIDE.md](./INTEGRATIONS-GUIDE.md) | integrations-specialist, tools-integration-specialist | +| [DOCUMENTATION-GUIDE.md](./DOCUMENTATION-GUIDE.md) | documenter, describer-for-ai, planner, product-owner | + +### Project Documentation + - [../adr/index.md](../adr/index.md) - Architecture Decision Records -- [../TESTING.md](../TESTING.md) - Testing guide +- [../development/TESTING.md](../development/TESTING.md) - Testing guide +- [../development/CODE-PATTERNS.md](../development/CODE-PATTERNS.md) - Code patterns reference +- [../architecture/OVERVIEW.md](../architecture/OVERVIEW.md) - System architecture ## Troubleshooting diff --git a/docs/subagents/SECURITY-DEBUG-GUIDE.md b/docs/subagents/SECURITY-DEBUG-GUIDE.md index 26f45f21..386393aa 100644 --- a/docs/subagents/SECURITY-DEBUG-GUIDE.md +++ b/docs/subagents/SECURITY-DEBUG-GUIDE.md @@ -6,6 +6,16 @@ This guide covers security and debugging-focused subagents: - **log-debug**: Production errors, observability, Bugsink/Sentry analysis - **code-reviewer**: Code quality, security review, best practices +## Quick Reference + +| Aspect | security-engineer | log-debug | code-reviewer | +| --------------- | ---------------------------------- | ---------------------------------------- | --------------------------- | +| **Primary Use** | Security audits, OWASP | Production debugging | Code quality review | +| **Key ADRs** | ADR-016 (Security), ADR-032 (Rate) | ADR-050 (Observability) | ADR-034, ADR-035 (Patterns) | +| **MCP Tools** | N/A | `mcp__bugsink__*`, `mcp__localerrors__*` | N/A | +| **Key Checks** | Auth, input validation, CORS | Logs, stacktraces, error patterns | Patterns, tests, security | +| **Delegate To** | `coder` (fix issues) | `devops` (infra), `coder` (fixes) | `coder`, `testwriter` | + ## The security-engineer Subagent ### When to Use @@ -432,8 +442,10 @@ tail -f /var/log/postgresql/postgresql-$(date +%Y-%m-%d).log | grep "duration:" - [OVERVIEW.md](./OVERVIEW.md) - Subagent system overview - [DEVOPS-GUIDE.md](./DEVOPS-GUIDE.md) - Infrastructure debugging +- [TESTER-GUIDE.md](./TESTER-GUIDE.md) - Security testing - [../adr/0016-api-security-hardening.md](../adr/0016-api-security-hardening.md) - Security ADR - [../adr/0032-rate-limiting-strategy.md](../adr/0032-rate-limiting-strategy.md) - Rate limiting -- [../adr/0015-application-performance-monitoring-and-error-tracking.md](../adr/0015-application-performance-monitoring-and-error-tracking.md) - Monitoring ADR +- [../adr/0015-error-tracking-and-observability.md](../adr/0015-error-tracking-and-observability.md) - Monitoring ADR - [../adr/0050-postgresql-function-observability.md](../adr/0050-postgresql-function-observability.md) - Database observability -- [../BARE-METAL-SETUP.md](../BARE-METAL-SETUP.md) - Production setup +- [../operations/BARE-METAL-SETUP.md](../operations/BARE-METAL-SETUP.md) - Production setup +- [../tools/BUGSINK-SETUP.md](../tools/BUGSINK-SETUP.md) - Bugsink configuration diff --git a/docs/subagents/TESTER-GUIDE.md b/docs/subagents/TESTER-GUIDE.md index 2119d224..e7242bea 100644 --- a/docs/subagents/TESTER-GUIDE.md +++ b/docs/subagents/TESTER-GUIDE.md @@ -5,6 +5,17 @@ This guide covers two related but distinct subagents for testing in the Flyer Cr - **tester**: Adversarial testing to find edge cases, race conditions, and vulnerabilities - **testwriter**: Creating comprehensive test suites for features and fixes +## Quick Reference + +| Aspect | tester | testwriter | +| ---------------- | -------------------------------------------- | ------------------------------------------ | +| **Primary Use** | Find bugs, security issues, edge cases | Create test suites, improve coverage | +| **Key Files** | N/A (analysis-focused) | `*.test.ts`, `src/tests/utils/` | +| **Key ADRs** | ADR-010 (Testing), ADR-040 (Test Economics) | ADR-010 (Testing), ADR-045 (Test Fixtures) | +| **Test Command** | `podman exec -it flyer-crawler-dev npm test` | Same | +| **Test Stack** | Vitest, Supertest, Testing Library | Same | +| **Delegate To** | `testwriter` (write tests for findings) | `coder` (fix failing tests) | + ## Understanding the Difference | Aspect | tester | testwriter | @@ -399,6 +410,7 @@ A typical workflow for thorough testing: - [OVERVIEW.md](./OVERVIEW.md) - Subagent system overview - [CODER-GUIDE.md](./CODER-GUIDE.md) - Working with the coder subagent -- [../TESTING.md](../TESTING.md) - Testing guide +- [SECURITY-DEBUG-GUIDE.md](./SECURITY-DEBUG-GUIDE.md) - Security testing and code review +- [../development/TESTING.md](../development/TESTING.md) - Testing guide - [../adr/0010-testing-strategy-and-standards.md](../adr/0010-testing-strategy-and-standards.md) - Testing ADR - [../adr/0040-testing-economics-and-priorities.md](../adr/0040-testing-economics-and-priorities.md) - Testing priorities diff --git a/src/features/store/StoreCard.test.tsx b/src/features/store/StoreCard.test.tsx index fad1d1a1..5ae80a50 100644 --- a/src/features/store/StoreCard.test.tsx +++ b/src/features/store/StoreCard.test.tsx @@ -185,10 +185,7 @@ describe('StoreCard', () => { it('should show "No location data" when locations is undefined', () => { renderWithProviders( - , + , ); expect(screen.getByText('No location data')).toBeInTheDocument(); diff --git a/src/hooks/useEventBus.test.ts b/src/hooks/useEventBus.test.ts index 5389f302..d15a136d 100644 --- a/src/hooks/useEventBus.test.ts +++ b/src/hooks/useEventBus.test.ts @@ -14,7 +14,7 @@ vi.mock('../services/eventBus', () => ({ import { eventBus } from '../services/eventBus'; -const mockEventBus = eventBus as { +const mockEventBus = eventBus as unknown as { on: Mock; off: Mock; dispatch: Mock; @@ -206,7 +206,7 @@ describe('useEventBus', () => { name: string; } - const callback = vi.fn<[TestData?], void>(); + const callback = vi.fn(); renderHook(() => useEventBus('typed-event', callback)); @@ -217,7 +217,7 @@ describe('useEventBus', () => { }); it('should handle callback with optional parameter', () => { - const callback = vi.fn<[string?], void>(); + const callback = vi.fn(); renderHook(() => useEventBus('optional-event', callback)); diff --git a/src/hooks/useOnboardingTour.test.ts b/src/hooks/useOnboardingTour.test.ts index b427aa0c..acda894c 100644 --- a/src/hooks/useOnboardingTour.test.ts +++ b/src/hooks/useOnboardingTour.test.ts @@ -17,6 +17,7 @@ vi.mock('driver.js', () => ({ DriveStep: vi.fn(), })); +// @ts-expect-error - driver.js types are not fully compatible, but the mock works import { driver } from 'driver.js'; const mockDriver = driver as Mock;