Compare commits

..

2 Commits

Author SHA1 Message Date
Gitea Actions
de3f0cf26e ci: Bump version to 0.12.21 [skip ci] 2026-01-29 05:37:59 +05:00
45ac4fccf5 comprehensive documentation review + test fixes
Some checks failed
Deploy to Test Environment / deploy-to-test (push) Failing after 2m15s
2026-01-28 16:35:38 -08:00
40 changed files with 2605 additions and 604 deletions

150
CLAUDE.md
View File

@@ -31,46 +31,19 @@ Out-of-sync = test failures.
**CRITICAL**: The `claude-win10` user has **READ-ONLY** access to production and test servers. **CRITICAL**: The `claude-win10` user has **READ-ONLY** access to production and test servers.
**Claude Code does NOT have:** | Capability | Status |
| ---------------------- | ---------------------- |
| Root/sudo access | NO |
| Write permissions | NO |
| PM2 restart, systemctl | NO - User must execute |
- Root or sudo access **Server Operations Workflow**: Diagnose → User executes → Analyze → Fix (1-3 commands) → User executes → Verify
- Write permissions on servers
- Ability to execute PM2 restart, systemctl, or other write operations directly
**Correct Workflow for Server Operations:** **Rules**:
| Step | Actor | Action | - Provide diagnostic commands first, wait for user to report results
| ---- | ------ | --------------------------------------------------------------------------- | - Maximum 3 fix commands at a time (errors may cascade)
| 1 | Claude | Provide **diagnostic commands** (read-only checks) for user to run | - Always verify after fixes complete
| 2 | User | Execute commands on server, report results |
| 3 | Claude | Analyze results, provide **fix commands** (1-3 at a time) |
| 4 | User | Execute fix commands, report results |
| 5 | Claude | Provide **verification commands** to confirm success and check side effects |
| 6 | Claude | Document progress stage by stage |
| 7 | Claude | Update ALL relevant documentation when complete |
**Example - Diagnosing PM2 Issues:**
```bash
# Step 1: Claude provides diagnostic commands (user runs these)
pm2 list
pm2 logs flyer-crawler-api --lines 50
systemctl status redis
# Step 3: After user reports results, Claude provides fix commands
pm2 restart flyer-crawler-api
# Wait for user confirmation before next command
# Step 5: Claude provides verification
pm2 list
curl -s https://flyer-crawler.projectium.com/api/health/ready | jq .
```
**Never Do:**
- `ssh root@projectium.com "pm2 restart all"` (Claude cannot execute this)
- Assume commands succeeded without user confirmation
- Provide more than 3 fix commands at once (errors may cascade)
### Communication Style ### Communication Style
@@ -105,12 +78,12 @@ Ask before assuming. Never assume:
### Key Patterns (with file locations) ### Key Patterns (with file locations)
| Pattern | ADR | Implementation | File | | Pattern | ADR | Implementation | File |
| ------------------ | ------- | ------------------------------------------------- | ----------------------------------- | | ------------------ | ------- | ------------------------------------------------- | ---------------------------------- |
| Error Handling | ADR-001 | `handleDbError()`, throw `NotFoundError` | `src/services/db/errors.db.ts` | | Error Handling | ADR-001 | `handleDbError()`, throw `NotFoundError` | `src/services/db/errors.db.ts` |
| Repository Methods | ADR-034 | `get*` (throws), `find*` (null), `list*` (array) | `src/services/db/*.db.ts` | | Repository Methods | ADR-034 | `get*` (throws), `find*` (null), `list*` (array) | `src/services/db/*.db.ts` |
| API Responses | ADR-028 | `sendSuccess()`, `sendPaginated()`, `sendError()` | `src/utils/apiResponse.ts` | | API Responses | ADR-028 | `sendSuccess()`, `sendPaginated()`, `sendError()` | `src/utils/apiResponse.ts` |
| Transactions | ADR-002 | `withTransaction(async (client) => {...})` | `src/services/db/transaction.db.ts` | | Transactions | ADR-002 | `withTransaction(async (client) => {...})` | `src/services/db/connection.db.ts` |
### Key Files Quick Access ### Key Files Quick Access
@@ -166,7 +139,7 @@ The dev container now matches production by using PM2 for process management.
- `flyer-crawler-worker-dev` - Background job worker - `flyer-crawler-worker-dev` - Background job worker
- `flyer-crawler-vite-dev` - Vite frontend dev server (port 5173) - `flyer-crawler-vite-dev` - Vite frontend dev server (port 5173)
### Log Aggregation (ADR-050) ### Log Aggregation (ADR-015)
All logs flow to Bugsink via Logstash with 3-project routing: All logs flow to Bugsink via Logstash with 3-project routing:
@@ -249,7 +222,7 @@ All logs flow to Bugsink via Logstash with 3-project routing:
**Launch Pattern**: **Launch Pattern**:
``` ```text
Use Task tool with subagent_type: "coder", "db-dev", "tester", etc. Use Task tool with subagent_type: "coder", "db-dev", "tester", etc.
``` ```
@@ -330,8 +303,8 @@ podman cp "d:/path/file" container:/tmp/file
**Quick Access**: **Quick Access**:
- **Dev**: https://localhost:8443 (`admin@localhost`/`admin`) - **Dev**: <https://localhost:8443> (`admin@localhost`/`admin`)
- **Prod**: https://bugsink.projectium.com - **Prod**: <https://bugsink.projectium.com>
**Token Creation** (required for MCP): **Token Creation** (required for MCP):
@@ -347,7 +320,7 @@ cd /opt/bugsink && bugsink-manage create_auth_token
**See**: [docs/operations/LOGSTASH-QUICK-REF.md](docs/operations/LOGSTASH-QUICK-REF.md) **See**: [docs/operations/LOGSTASH-QUICK-REF.md](docs/operations/LOGSTASH-QUICK-REF.md)
Log aggregation: PostgreSQL + PM2 + Redis + NGINX → Bugsink (ADR-050) Log aggregation: PostgreSQL + PM2 + Redis + NGINX → Bugsink (ADR-015)
--- ---
@@ -367,84 +340,3 @@ Log aggregation: PostgreSQL + PM2 + Redis + NGINX → Bugsink (ADR-050)
| **Logstash** | [LOGSTASH-QUICK-REF.md](docs/operations/LOGSTASH-QUICK-REF.md) | | **Logstash** | [LOGSTASH-QUICK-REF.md](docs/operations/LOGSTASH-QUICK-REF.md) |
| **ADRs** | [docs/adr/index.md](docs/adr/index.md) | | **ADRs** | [docs/adr/index.md](docs/adr/index.md) |
| **All Docs** | [docs/README.md](docs/README.md) | | **All Docs** | [docs/README.md](docs/README.md) |
---
## Appendix: Integration Test Issues (Full Details)
### 1. Vitest globalSetup Context Isolation
Vitest's `globalSetup` runs in separate Node.js context. Singletons, spies, mocks do NOT share instances with test files.
**Affected**: BullMQ worker service mocks (AI/DB failure tests)
**Solutions**: Mark `.todo()`, create test-only API endpoints, use Redis-based mock flags
```typescript
// DOES NOT WORK - different instances
const { flyerProcessingService } = await import('../../services/workers.server');
flyerProcessingService._getAiProcessor()._setExtractAndValidateData(mockFn);
```
### 2. Cleanup Queue Deletes Before Verification
Cleanup worker processes jobs in globalSetup context, ignoring test spies.
**Solution**: Drain and pause queue:
```typescript
const { cleanupQueue } = await import('../../services/queues.server');
await cleanupQueue.drain();
await cleanupQueue.pause();
// ... test ...
await cleanupQueue.resume();
```
### 3. Cache Stale After Direct SQL
Direct `pool.query()` inserts bypass cache invalidation.
**Solution**: `await cacheService.invalidateFlyers();` after inserts
### 4. Test Filename Collisions
Multer predictable filenames cause race conditions.
**Solution**: Use unique suffix: `${Date.now()}-${Math.round(Math.random() * 1e9)}`
### 5. Response Format Mismatches
API formats change: `data.jobId` vs `data.job.id`, nested vs flat, string vs number IDs.
**Solution**: Log response bodies, update assertions
### 6. External Service Availability
PM2/Redis health checks fail when unavailable.
**Solution**: try/catch with graceful degradation or mock
### 7. TZ Environment Variable Breaking Async Hooks
**Problem**: When `TZ=America/Los_Angeles` (or other timezone values) is set in the environment, Node.js async_hooks module can produce `RangeError: Invalid triggerAsyncId value: NaN`. This breaks React Testing Library's `render()` function which uses async hooks internally.
**Root Cause**: Setting `TZ` to certain timezone values interferes with Node.js's internal async tracking mechanism, causing invalid async IDs to be generated.
**Symptoms**:
```text
RangeError: Invalid triggerAsyncId value: NaN
process.env.NODE_ENV.queueSeveralMicrotasks node_modules/react/cjs/react.development.js:751:15
process.env.NODE_ENV.exports.act node_modules/react/cjs/react.development.js:886:11
node_modules/@testing-library/react/dist/act-compat.js:46:25
renderRoot node_modules/@testing-library/react/dist/pure.js:189:26
```
**Solution**: Explicitly unset `TZ` in all test scripts by adding `TZ=` (empty value) to cross-env:
```json
"test:unit": "cross-env NODE_ENV=test TZ= tsx ..."
"test:integration": "cross-env NODE_ENV=test TZ= tsx ..."
```
**Context**: This issue was introduced in commit `d03900c` which added `TZ: 'America/Los_Angeles'` to PM2 ecosystem configs for consistent log timestamps in production/dev environments. Tests must explicitly override this to prevent the async hooks error.

View File

@@ -0,0 +1,393 @@
# AI Documentation Index
Machine-optimized navigation for AI agents. Structured for vector retrieval and semantic search.
---
## Quick Lookup Table
| Task/Question | Primary Doc | Section/ADR |
| ----------------------- | --------------------------------------------------- | --------------------------------------- |
| Add new API endpoint | [CODE-PATTERNS.md](development/CODE-PATTERNS.md) | API Response Patterns, Input Validation |
| Add repository method | [CODE-PATTERNS.md](development/CODE-PATTERNS.md) | Repository Patterns (get*/find*/list\*) |
| Fix failing test | [TESTING.md](development/TESTING.md) | Known Integration Test Issues |
| Run tests correctly | [TESTING.md](development/TESTING.md) | Test Execution Environment |
| Add database column | [DATABASE-GUIDE.md](subagents/DATABASE-GUIDE.md) | Schema sync required |
| Deploy to production | [DEPLOYMENT.md](operations/DEPLOYMENT.md) | Application Deployment |
| Debug container issue | [DEBUGGING.md](development/DEBUGGING.md) | Container Issues |
| Configure environment | [ENVIRONMENT.md](getting-started/ENVIRONMENT.md) | Configuration by Environment |
| Add background job | [CODE-PATTERNS.md](development/CODE-PATTERNS.md) | Background Jobs |
| Handle errors correctly | [CODE-PATTERNS.md](development/CODE-PATTERNS.md) | Error Handling |
| Use transactions | [CODE-PATTERNS.md](development/CODE-PATTERNS.md) | Transaction Management |
| Add authentication | [AUTHENTICATION.md](architecture/AUTHENTICATION.md) | JWT Token Architecture |
| Cache data | [CODE-PATTERNS.md](development/CODE-PATTERNS.md) | Caching |
| Check PM2 status | [DEV-CONTAINER.md](development/DEV-CONTAINER.md) | PM2 Process Management |
| View logs | [DEBUGGING.md](development/DEBUGGING.md) | PM2 Log Access |
| Understand architecture | [OVERVIEW.md](architecture/OVERVIEW.md) | System Architecture Diagram |
| Check ADR for decision | [adr/index.md](adr/index.md) | ADR by category |
| Use subagent | [subagents/OVERVIEW.md](subagents/OVERVIEW.md) | Available Subagents |
| API versioning | [API-VERSIONING.md](development/API-VERSIONING.md) | Phase 2 infrastructure |
---
## Documentation Tree
```
docs/
+-- AI-DOCUMENTATION-INDEX.md # THIS FILE - AI navigation index
+-- README.md # Human-readable doc hub
|
+-- adr/ # Architecture Decision Records (57 ADRs)
| +-- index.md # ADR index by category
| +-- 0001-*.md # Standardized error handling
| +-- 0002-*.md # Transaction management (withTransaction)
| +-- 0003-*.md # Input validation (Zod middleware)
| +-- 0008-*.md # API versioning (/api/v1/)
| +-- 0014-*.md # Platform: Linux only (CRITICAL)
| +-- 0028-*.md # API response (sendSuccess/sendError)
| +-- 0034-*.md # Repository pattern (get*/find*/list*)
| +-- 0035-*.md # Service layer architecture
| +-- 0050-*.md # PostgreSQL observability + Logstash
| +-- 0057-*.md # Test remediation post-API versioning
| +-- adr-implementation-tracker.md # Implementation status
|
+-- architecture/
| +-- OVERVIEW.md # System architecture, data flows, entities
| +-- DATABASE.md # Schema design, extensions, setup
| +-- AUTHENTICATION.md # OAuth, JWT, security features
| +-- WEBSOCKET_USAGE.md # Real-time communication patterns
| +-- api-versioning-infrastructure.md # Phase 2 versioning details
|
+-- development/
| +-- CODE-PATTERNS.md # Error handling, repos, API responses
| +-- TESTING.md # Unit/integration/E2E, known issues
| +-- DEBUGGING.md # Container, DB, API, PM2 debugging
| +-- DEV-CONTAINER.md # PM2, Logstash, container services
| +-- API-VERSIONING.md # API versioning workflows
| +-- DESIGN_TOKENS.md # Neo-Brutalism design system
| +-- ERROR-LOGGING-PATHS.md # req.originalUrl pattern
| +-- test-path-migration.md # Test file reorganization
|
+-- getting-started/
| +-- QUICKSTART.md # Quick setup instructions
| +-- INSTALL.md # Full installation guide
| +-- ENVIRONMENT.md # Environment variables reference
|
+-- operations/
| +-- DEPLOYMENT.md # Production deployment guide
| +-- BARE-METAL-SETUP.md # Server provisioning
| +-- MONITORING.md # Bugsink, health checks
| +-- LOGSTASH-QUICK-REF.md # Log aggregation reference
| +-- LOGSTASH-TROUBLESHOOTING.md # Logstash debugging
|
+-- subagents/
| +-- OVERVIEW.md # Subagent system introduction
| +-- CODER-GUIDE.md # Code development patterns
| +-- TESTER-GUIDE.md # Testing strategies
| +-- DATABASE-GUIDE.md # Database workflows
| +-- DEVOPS-GUIDE.md # Deployment/infrastructure
| +-- FRONTEND-GUIDE.md # UI/UX development
| +-- AI-USAGE-GUIDE.md # Gemini integration
| +-- DOCUMENTATION-GUIDE.md # Writing docs
| +-- SECURITY-DEBUG-GUIDE.md # Security and debugging
|
+-- tools/
| +-- MCP-CONFIGURATION.md # MCP servers setup
| +-- BUGSINK-SETUP.md # Error tracking setup
| +-- VSCODE-SETUP.md # Editor configuration
|
+-- archive/ # Historical docs, session notes
+-- sessions/ # Development session logs
+-- plans/ # Feature implementation plans
+-- research/ # Investigation notes
```
---
## Problem-to-Document Mapping
### Database Issues
| Problem | Documents |
| -------------------- | ----------------------------------------------------------------------------------------------- |
| Schema out of sync | [DATABASE-GUIDE.md](subagents/DATABASE-GUIDE.md), [CLAUDE.md](../CLAUDE.md) schema sync section |
| Migration needed | [DATABASE.md](architecture/DATABASE.md), ADR-013, ADR-023 |
| Query performance | [DEBUGGING.md](development/DEBUGGING.md) Query Performance Issues |
| Connection errors | [DEBUGGING.md](development/DEBUGGING.md) Database Issues |
| Transaction patterns | [CODE-PATTERNS.md](development/CODE-PATTERNS.md) Transaction Management, ADR-002 |
| Repository methods | [CODE-PATTERNS.md](development/CODE-PATTERNS.md) Repository Patterns, ADR-034 |
### Test Failures
| Problem | Documents |
| ---------------------------- | --------------------------------------------------------------------- |
| Tests fail in container | [TESTING.md](development/TESTING.md), ADR-014 |
| Vitest globalSetup isolation | [CLAUDE.md](../CLAUDE.md) Integration Test Issues #1 |
| Cache stale after insert | [CLAUDE.md](../CLAUDE.md) Integration Test Issues #3 |
| Queue interference | [CLAUDE.md](../CLAUDE.md) Integration Test Issues #2 |
| API path mismatches | [TESTING.md](development/TESTING.md) API Versioning in Tests, ADR-057 |
| Type check failures | [DEBUGGING.md](development/DEBUGGING.md) Type Check Failures |
| TZ environment breaks async | [CLAUDE.md](../CLAUDE.md) Integration Test Issues #7 |
### Deployment Issues
| Problem | Documents |
| --------------------- | ------------------------------------------------------------------------------------- |
| PM2 not starting | [DEBUGGING.md](development/DEBUGGING.md) PM2 Process Issues |
| NGINX configuration | [DEPLOYMENT.md](operations/DEPLOYMENT.md) NGINX Configuration |
| SSL certificates | [DEBUGGING.md](development/DEBUGGING.md) SSL Certificate Issues |
| CI/CD failures | [DEPLOYMENT.md](operations/DEPLOYMENT.md) CI/CD Pipeline, ADR-017 |
| Container won't start | [DEBUGGING.md](development/DEBUGGING.md) Container Issues |
| Bugsink not receiving | [BUGSINK-SETUP.md](tools/BUGSINK-SETUP.md), [MONITORING.md](operations/MONITORING.md) |
### Frontend/UI Changes
| Problem | Documents |
| ------------------ | --------------------------------------------------------------- |
| Component patterns | [FRONTEND-GUIDE.md](subagents/FRONTEND-GUIDE.md), ADR-044 |
| Design tokens | [DESIGN_TOKENS.md](development/DESIGN_TOKENS.md), ADR-012 |
| State management | ADR-005, [OVERVIEW.md](architecture/OVERVIEW.md) Frontend Stack |
| Hot reload broken | [DEBUGGING.md](development/DEBUGGING.md) Frontend Issues |
| CORS errors | [DEBUGGING.md](development/DEBUGGING.md) API Calls Failing |
### API Development
| Problem | Documents |
| ---------------- | ------------------------------------------------------------------------------- |
| Response format | [CODE-PATTERNS.md](development/CODE-PATTERNS.md) API Response Patterns, ADR-028 |
| Input validation | [CODE-PATTERNS.md](development/CODE-PATTERNS.md) Input Validation, ADR-003 |
| Error handling | [CODE-PATTERNS.md](development/CODE-PATTERNS.md) Error Handling, ADR-001 |
| Rate limiting | ADR-032, [OVERVIEW.md](architecture/OVERVIEW.md) |
| API versioning | [API-VERSIONING.md](development/API-VERSIONING.md), ADR-008 |
| Authentication | [AUTHENTICATION.md](architecture/AUTHENTICATION.md), ADR-048 |
### Background Jobs
| Problem | Documents |
| ------------------- | ------------------------------------------------------------------------- |
| Jobs not processing | [DEBUGGING.md](development/DEBUGGING.md) Background Job Issues |
| Queue configuration | [CODE-PATTERNS.md](development/CODE-PATTERNS.md) Background Jobs, ADR-006 |
| Worker crashes | [DEBUGGING.md](development/DEBUGGING.md), ADR-053 |
| Scheduled jobs | ADR-037, [OVERVIEW.md](architecture/OVERVIEW.md) Scheduled Jobs |
---
## Document Priority Matrix
### CRITICAL (Read First)
| Document | Purpose | Key Content |
| --------------------------------------------------------------- | ----------------------- | ----------------------------- |
| [CLAUDE.md](../CLAUDE.md) | AI agent instructions | Rules, patterns, known issues |
| [ADR-014](adr/0014-containerization-and-deployment-strategy.md) | Platform requirement | Tests MUST run in container |
| [DEV-CONTAINER.md](development/DEV-CONTAINER.md) | Development environment | PM2, Logstash, services |
### HIGH (Core Development)
| Document | Purpose | Key Content |
| --------------------------------------------------- | ----------------- | ---------------------------- |
| [CODE-PATTERNS.md](development/CODE-PATTERNS.md) | Code templates | Error handling, repos, APIs |
| [TESTING.md](development/TESTING.md) | Test execution | Commands, known issues |
| [DATABASE.md](architecture/DATABASE.md) | Schema reference | Setup, extensions, users |
| [ADR-034](adr/0034-repository-pattern-standards.md) | Repository naming | get*/find*/list\* |
| [ADR-028](adr/0028-api-response-standardization.md) | API responses | sendSuccess/sendError |
| [ADR-001](adr/0001-standardized-error-handling.md) | Error handling | handleDbError, NotFoundError |
### MEDIUM (Specialized Tasks)
| Document | Purpose | Key Content |
| --------------------------------------------------- | --------------------- | ------------------------ |
| [subagents/OVERVIEW.md](subagents/OVERVIEW.md) | Subagent selection | When to delegate |
| [DEPLOYMENT.md](operations/DEPLOYMENT.md) | Production deployment | PM2, NGINX, CI/CD |
| [DEBUGGING.md](development/DEBUGGING.md) | Troubleshooting | Common issues, solutions |
| [ENVIRONMENT.md](getting-started/ENVIRONMENT.md) | Config reference | Variables by environment |
| [AUTHENTICATION.md](architecture/AUTHENTICATION.md) | Auth patterns | OAuth, JWT, security |
| [API-VERSIONING.md](development/API-VERSIONING.md) | Versioning | /api/v1/ prefix |
### LOW (Reference/Historical)
| Document | Purpose | Key Content |
| -------------------- | ------------------ | ------------------------- |
| [archive/](archive/) | Historical docs | Session notes, old plans |
| ADR-013, ADR-023 | Migration strategy | Proposed, not implemented |
| ADR-024 | Feature flags | Proposed |
| ADR-025 | i18n/l10n | Proposed |
---
## Cross-Reference Matrix
| Document | References | Referenced By |
| -------------------- | ------------------------------------------------------------------------------- | ------------------------------------------------------ |
| **CLAUDE.md** | ADR-001, ADR-002, ADR-008, ADR-014, ADR-028, ADR-034, ADR-035, ADR-050, ADR-057 | All development docs |
| **ADR-008** | ADR-028 | API-VERSIONING.md, TESTING.md, ADR-057 |
| **ADR-014** | - | CLAUDE.md, TESTING.md, DEPLOYMENT.md, DEV-CONTAINER.md |
| **ADR-028** | ADR-001 | CODE-PATTERNS.md, OVERVIEW.md |
| **ADR-034** | ADR-001 | CODE-PATTERNS.md, DATABASE-GUIDE.md |
| **ADR-057** | ADR-008, ADR-028 | TESTING.md |
| **CODE-PATTERNS.md** | ADR-001, ADR-002, ADR-003, ADR-028, ADR-034, ADR-036, ADR-048 | CODER-GUIDE.md |
| **TESTING.md** | ADR-014, ADR-057, CLAUDE.md | TESTER-GUIDE.md, DEBUGGING.md |
| **DEBUGGING.md** | DEV-CONTAINER.md, TESTING.md, MONITORING.md | DEVOPS-GUIDE.md |
| **DEV-CONTAINER.md** | ADR-014, ADR-050, ecosystem.dev.config.cjs | DEBUGGING.md, CLAUDE.md |
| **OVERVIEW.md** | ADR-001 through ADR-050+ | All architecture docs |
| **DATABASE.md** | ADR-002, ADR-013, ADR-055 | DATABASE-GUIDE.md |
---
## Navigation Patterns
### Adding a Feature
```
1. CLAUDE.md -> Project rules, patterns
2. CODE-PATTERNS.md -> Implementation templates
3. Relevant subagent guide -> Domain-specific patterns
4. Related ADRs -> Design decisions
5. TESTING.md -> Test requirements
```
### Fixing a Bug
```
1. DEBUGGING.md -> Common issues checklist
2. TESTING.md -> Run tests in container
3. Error logs (pm2/bugsink) -> Identify root cause
4. CODE-PATTERNS.md -> Correct pattern reference
5. Related ADR -> Architectural context
```
### Deploying
```
1. DEPLOYMENT.md -> Deployment procedures
2. ENVIRONMENT.md -> Required variables
3. MONITORING.md -> Health check verification
4. LOGSTASH-QUICK-REF.md -> Log aggregation check
```
### Database Changes
```
1. DATABASE-GUIDE.md -> Schema sync requirements (CRITICAL)
2. DATABASE.md -> Schema design patterns
3. ADR-002 -> Transaction patterns
4. ADR-034 -> Repository methods
5. ADR-055 -> Normalization rules
```
### Subagent Selection
| Task Type | Subagent | Guide |
| --------------------- | ------------------------- | ------------------------------------------------------------ |
| Write production code | `coder` | [CODER-GUIDE.md](subagents/CODER-GUIDE.md) |
| Database changes | `db-dev` | [DATABASE-GUIDE.md](subagents/DATABASE-GUIDE.md) |
| Create tests | `testwriter` | [TESTER-GUIDE.md](subagents/TESTER-GUIDE.md) |
| Fix failing tests | `tester` | [TESTER-GUIDE.md](subagents/TESTER-GUIDE.md) |
| Container/deployment | `devops` | [DEVOPS-GUIDE.md](subagents/DEVOPS-GUIDE.md) |
| UI components | `frontend-specialist` | [FRONTEND-GUIDE.md](subagents/FRONTEND-GUIDE.md) |
| External APIs | `integrations-specialist` | - |
| Security review | `security-engineer` | [SECURITY-DEBUG-GUIDE.md](subagents/SECURITY-DEBUG-GUIDE.md) |
| Production errors | `log-debug` | [SECURITY-DEBUG-GUIDE.md](subagents/SECURITY-DEBUG-GUIDE.md) |
| AI/Gemini issues | `ai-usage` | [AI-USAGE-GUIDE.md](subagents/AI-USAGE-GUIDE.md) |
---
## Key File Quick Reference
### Configuration
| File | Purpose |
| -------------------------- | ---------------------------- |
| `server.ts` | Express app setup |
| `src/config/env.ts` | Environment validation (Zod) |
| `ecosystem.dev.config.cjs` | PM2 dev config |
| `ecosystem.config.cjs` | PM2 prod config |
| `vite.config.ts` | Vite build config |
### Core Implementation
| File | Purpose |
| ----------------------------------- | ----------------------------------- |
| `src/routes/*.routes.ts` | API route handlers |
| `src/services/db/*.db.ts` | Repository layer |
| `src/services/*.server.ts` | Server-only services |
| `src/services/queues.server.ts` | BullMQ queue definitions |
| `src/services/workers.server.ts` | BullMQ workers |
| `src/utils/apiResponse.ts` | sendSuccess/sendError/sendPaginated |
| `src/services/db/errors.db.ts` | handleDbError, NotFoundError |
| `src/services/db/transaction.db.ts` | withTransaction |
### Database Schema
| File | Purpose |
| ------------------------------ | ----------------------------------- |
| `sql/master_schema_rollup.sql` | Test DB, complete reference |
| `sql/initial_schema.sql` | Fresh install (identical to rollup) |
| `sql/migrations/*.sql` | Production ALTER statements |
### Testing
| File | Purpose |
| ---------------------------------- | ----------------------- |
| `vitest.config.ts` | Unit test config |
| `vitest.config.integration.ts` | Integration test config |
| `vitest.config.e2e.ts` | E2E test config |
| `src/tests/utils/mockFactories.ts` | Mock data factories |
| `src/tests/utils/storeHelpers.ts` | Store test helpers |
---
## ADR Quick Reference
### By Implementation Status
**Implemented**: 001, 002, 003, 004, 006, 008, 009, 010, 016, 017, 020, 021, 028, 032, 033, 034, 035, 036, 037, 038, 040, 041, 043, 044, 045, 046, 050, 051, 052, 055, 057
**Partially Implemented**: 012, 014, 015, 048
**Proposed**: 011, 013, 022, 023, 024, 025, 029, 030, 031, 039, 047, 053, 054, 056
### By Category
| Category | ADRs |
| --------------------- | ------------------------------------------- |
| Core Infrastructure | 002, 007, 020, 030 |
| Data Management | 009, 013, 019, 023, 031, 055 |
| API & Integration | 003, 008, 018, 022, 028 |
| Security | 001, 011, 016, 029, 032, 033, 048 |
| Observability | 004, 015, 050, 051, 052, 056 |
| Deployment & Ops | 006, 014, 017, 024, 037, 038, 053, 054 |
| Frontend/UI | 005, 012, 025, 026, 044 |
| Dev Workflow | 010, 021, 027, 040, 045, 047, 057 |
| Architecture Patterns | 034, 035, 036, 039, 041, 042, 043, 046, 049 |
---
## Essential Commands
```bash
# Run all tests (MUST use container)
podman exec -it flyer-crawler-dev npm test
# Run unit tests
podman exec -it flyer-crawler-dev npm run test:unit
# Run type check
podman exec -it flyer-crawler-dev npm run type-check
# Run integration tests
podman exec -it flyer-crawler-dev npm run test:integration
# PM2 status
podman exec -it flyer-crawler-dev pm2 status
# PM2 logs
podman exec -it flyer-crawler-dev pm2 logs
# Restart all processes
podman exec -it flyer-crawler-dev pm2 restart all
```
---
_This index is optimized for AI agent consumption. Updated: 2026-01-28_

View File

@@ -316,6 +316,7 @@ app.use('/api/v1', (req, res, next) => {
- [ADR-018](./0018-api-documentation-strategy.md) - API Documentation Strategy (versioned OpenAPI specs) - [ADR-018](./0018-api-documentation-strategy.md) - API Documentation Strategy (versioned OpenAPI specs)
- [ADR-028](./0028-api-response-standardization.md) - Response Standardization (envelope pattern applies to all versions) - [ADR-028](./0028-api-response-standardization.md) - Response Standardization (envelope pattern applies to all versions)
- [ADR-016](./0016-api-security-hardening.md) - Security Hardening (applies to all versions) - [ADR-016](./0016-api-security-hardening.md) - Security Hardening (applies to all versions)
- [ADR-057](./0057-test-remediation-post-api-versioning.md) - Test Remediation Post-API Versioning (documents test migration)
## Implementation Checklist ## Implementation Checklist

View File

@@ -363,6 +363,13 @@ The following files contain acknowledged code smell violations that are deferred
- `src/tests/utils/mockFactories.ts` - Mock factories (1553 lines) - `src/tests/utils/mockFactories.ts` - Mock factories (1553 lines)
- `src/tests/utils/testHelpers.ts` - Test utilities - `src/tests/utils/testHelpers.ts` - Test utilities
## Related ADRs
- [ADR-014](./0014-containerization-and-deployment-strategy.md) - Containerization (tests must run in dev container)
- [ADR-040](./0040-testing-economics-and-priorities.md) - Testing Economics and Priorities
- [ADR-045](./0045-test-data-factories-and-fixtures.md) - Test Data Factories and Fixtures
- [ADR-057](./0057-test-remediation-post-api-versioning.md) - Test Remediation Post-API Versioning
## Future Enhancements ## Future Enhancements
1. **Browser E2E Tests**: Consider adding Playwright for actual browser testing 1. **Browser E2E Tests**: Consider adding Playwright for actual browser testing

View File

@@ -2,7 +2,9 @@
**Date**: 2025-12-12 **Date**: 2025-12-12
**Status**: Proposed **Status**: Superseded by [ADR-023](./0023-database-schema-migration-strategy.md)
**Note**: This ADR was an early draft. ADR-023 provides a more detailed specification for the same topic.
## Context ## Context

View File

@@ -4,6 +4,8 @@
**Status**: Proposed **Status**: Proposed
**Supersedes**: [ADR-013](./0013-database-schema-migration-strategy.md)
## Context ## Context
The `README.md` indicates that the database schema is managed by manually running a large `schema.sql.txt` file. This approach is highly error-prone, makes tracking changes difficult, and is not feasible for updating a live production database without downtime or data loss. The `README.md` indicates that the database schema is managed by manually running a large `schema.sql.txt` file. This approach is highly error-prone, makes tracking changes difficult, and is not feasible for updating a live production database without downtime or data loss.

View File

@@ -195,6 +195,12 @@ Do NOT add tests:
- Coverage percentages may not satisfy external audits - Coverage percentages may not satisfy external audits
- Requires judgment calls that may be inconsistent - Requires judgment calls that may be inconsistent
## Related ADRs
- [ADR-010](./0010-testing-strategy-and-standards.md) - Testing Strategy and Standards (this ADR extends ADR-010)
- [ADR-045](./0045-test-data-factories-and-fixtures.md) - Test Data Factories and Fixtures
- [ADR-057](./0057-test-remediation-post-api-versioning.md) - Test Remediation Post-API Versioning
## Key Files ## Key Files
- `docs/adr/0010-testing-strategy-and-standards.md` - Testing mechanics - `docs/adr/0010-testing-strategy-and-standards.md` - Testing mechanics

View File

@@ -4,7 +4,7 @@
**Status**: Accepted (Fully Implemented) **Status**: Accepted (Fully Implemented)
**Related**: [ADR-015](0015-application-performance-monitoring-and-error-tracking.md), [ADR-004](0004-standardized-application-wide-structured-logging.md) **Related**: [ADR-015](0015-error-tracking-and-observability.md), [ADR-004](0004-standardized-application-wide-structured-logging.md)
## Context ## Context
@@ -335,7 +335,7 @@ SELECT award_achievement('user-uuid', 'Nonexistent Badge');
## References ## References
- [ADR-015: Application Performance Monitoring](0015-application-performance-monitoring-and-error-tracking.md) - [ADR-015: Error Tracking and Observability](0015-error-tracking-and-observability.md)
- [ADR-004: Standardized Structured Logging](0004-standardized-application-wide-structured-logging.md) - [ADR-004: Standardized Structured Logging](0004-standardized-application-wide-structured-logging.md)
- [PostgreSQL RAISE Documentation](https://www.postgresql.org/docs/current/plpgsql-errors-and-messages.html) - [PostgreSQL RAISE Documentation](https://www.postgresql.org/docs/current/plpgsql-errors-and-messages.html)
- [PostgreSQL Logging Configuration](https://www.postgresql.org/docs/current/runtime-config-logging.html) - [PostgreSQL Logging Configuration](https://www.postgresql.org/docs/current/runtime-config-logging.html)

View File

@@ -332,6 +332,6 @@ Response:
## References ## References
- [ADR-006: Background Job Processing](./0006-background-job-processing-and-task-queues.md) - [ADR-006: Background Job Processing](./0006-background-job-processing-and-task-queues.md)
- [ADR-015: Application Performance Monitoring](./0015-application-performance-monitoring-and-error-tracking.md) - [ADR-015: Error Tracking and Observability](./0015-error-tracking-and-observability.md)
- [Bugsink API Documentation](https://bugsink.com/docs/api/) - [Bugsink API Documentation](https://bugsink.com/docs/api/)
- [Gitea API Documentation](https://docs.gitea.io/en-us/api-usage/) - [Gitea API Documentation](https://docs.gitea.io/en-us/api-usage/)

View File

@@ -15,9 +15,10 @@ This document tracks the implementation status and estimated effort for all Arch
| Status | Count | | Status | Count |
| ---------------------------- | ----- | | ---------------------------- | ----- |
| Accepted (Fully Implemented) | 40 | | Accepted (Fully Implemented) | 41 |
| Partially Implemented | 2 | | Partially Implemented | 2 |
| Proposed (Not Started) | 14 | | Proposed (Not Started) | 13 |
| Superseded | 1 |
--- ---
@@ -34,13 +35,13 @@ This document tracks the implementation status and estimated effort for all Arch
### Category 2: Data Management ### Category 2: Data Management
| ADR | Title | Status | Effort | Notes | | ADR | Title | Status | Effort | Notes |
| --------------------------------------------------------------- | ------------------------ | -------- | ------ | ------------------------------ | | --------------------------------------------------------------- | ------------------------ | ---------- | ------ | ------------------------------ |
| [ADR-009](./0009-caching-strategy-for-read-heavy-operations.md) | Caching Strategy | Accepted | - | Fully implemented | | [ADR-009](./0009-caching-strategy-for-read-heavy-operations.md) | Caching Strategy | Accepted | - | Fully implemented |
| [ADR-013](./0013-database-schema-migration-strategy.md) | Schema Migrations v1 | Proposed | M | Superseded by ADR-023 | | [ADR-013](./0013-database-schema-migration-strategy.md) | Schema Migrations v1 | Superseded | - | Superseded by ADR-023 |
| [ADR-019](./0019-data-backup-and-recovery-strategy.md) | Backup & Recovery | Accepted | - | Fully implemented | | [ADR-019](./0019-data-backup-and-recovery-strategy.md) | Backup & Recovery | Accepted | - | Fully implemented |
| [ADR-023](./0023-database-schema-migration-strategy.md) | Schema Migrations v2 | Proposed | L | Requires tooling setup | | [ADR-023](./0023-database-schema-migration-strategy.md) | Schema Migrations v2 | Proposed | L | Requires tooling setup |
| [ADR-031](./0031-data-retention-and-privacy-compliance.md) | Data Retention & Privacy | Proposed | XL | Legal/compliance review needed | | [ADR-031](./0031-data-retention-and-privacy-compliance.md) | Data Retention & Privacy | Proposed | XL | Legal/compliance review needed |
### Category 3: API & Integration ### Category 3: API & Integration
@@ -108,6 +109,7 @@ This document tracks the implementation status and estimated effort for all Arch
| [ADR-040](./0040-testing-economics-and-priorities.md) | Testing Economics | Accepted | - | Fully implemented | | [ADR-040](./0040-testing-economics-and-priorities.md) | Testing Economics | Accepted | - | Fully implemented |
| [ADR-045](./0045-test-data-factories-and-fixtures.md) | Test Data Factories | Accepted | - | Fully implemented | | [ADR-045](./0045-test-data-factories-and-fixtures.md) | Test Data Factories | Accepted | - | Fully implemented |
| [ADR-047](./0047-project-file-and-folder-organization.md) | Project Organization | Proposed | XL | Major reorganization | | [ADR-047](./0047-project-file-and-folder-organization.md) | Project Organization | Proposed | XL | Major reorganization |
| [ADR-057](./0057-test-remediation-post-api-versioning.md) | Test Remediation | Accepted | - | Fully implemented |
### Category 9: Architecture Patterns ### Category 9: Architecture Patterns
@@ -148,6 +150,8 @@ These ADRs are proposed or partially implemented, ordered by suggested implement
| Date | ADR | Change | | Date | ADR | Change |
| ---------- | ------- | ----------------------------------------------------------------------------------- | | ---------- | ------- | ----------------------------------------------------------------------------------- |
| 2026-01-28 | ADR-057 | Created - Test remediation documentation for ADR-008 Phase 2 migration |
| 2026-01-28 | ADR-013 | Marked as Superseded by ADR-023 |
| 2026-01-27 | ADR-008 | Test path migration complete - 23 files, ~70 paths updated, 274->345 tests passing | | 2026-01-27 | ADR-008 | Test path migration complete - 23 files, ~70 paths updated, 274->345 tests passing |
| 2026-01-27 | ADR-008 | Phase 2 Complete - Version router factory, deprecation headers, 82 versioning tests | | 2026-01-27 | ADR-008 | Phase 2 Complete - Version router factory, deprecation headers, 82 versioning tests |
| 2026-01-26 | ADR-015 | Completed - Added Sentry user context in AuthProvider, now fully implemented | | 2026-01-26 | ADR-015 | Completed - Added Sentry user context in AuthProvider, now fully implemented |

View File

@@ -2,6 +2,8 @@
This directory contains a log of the architectural decisions made for the Flyer Crawler project. This directory contains a log of the architectural decisions made for the Flyer Crawler project.
**[Implementation Tracker](./adr-implementation-tracker.md)**: Track implementation status and effort estimates for all ADRs.
## 1. Foundational / Core Infrastructure ## 1. Foundational / Core Infrastructure
**[ADR-002](./0002-standardized-transaction-management.md)**: Standardized Transaction Management and Unit of Work Pattern (Accepted) **[ADR-002](./0002-standardized-transaction-management.md)**: Standardized Transaction Management and Unit of Work Pattern (Accepted)
@@ -12,7 +14,7 @@ This directory contains a log of the architectural decisions made for the Flyer
## 2. Data Management ## 2. Data Management
**[ADR-009](./0009-caching-strategy-for-read-heavy-operations.md)**: Caching Strategy for Read-Heavy Operations (Accepted) **[ADR-009](./0009-caching-strategy-for-read-heavy-operations.md)**: Caching Strategy for Read-Heavy Operations (Accepted)
**[ADR-013](./0013-database-schema-migration-strategy.md)**: Database Schema Migration Strategy (Proposed) **[ADR-013](./0013-database-schema-migration-strategy.md)**: Database Schema Migration Strategy (Superseded by ADR-023)
**[ADR-019](./0019-data-backup-and-recovery-strategy.md)**: Data Backup and Recovery Strategy (Accepted) **[ADR-019](./0019-data-backup-and-recovery-strategy.md)**: Data Backup and Recovery Strategy (Accepted)
**[ADR-023](./0023-database-schema-migration-strategy.md)**: Database Schema Migration Strategy (Proposed) **[ADR-023](./0023-database-schema-migration-strategy.md)**: Database Schema Migration Strategy (Proposed)
**[ADR-031](./0031-data-retention-and-privacy-compliance.md)**: Data Retention and Privacy Compliance (Proposed) **[ADR-031](./0031-data-retention-and-privacy-compliance.md)**: Data Retention and Privacy Compliance (Proposed)
@@ -20,9 +22,9 @@ This directory contains a log of the architectural decisions made for the Flyer
## 3. API & Integration ## 3. API & Integration
**[ADR-003](./0003-standardized-input-validation-using-middleware.md)**: Standardized Input Validation using Middleware (Accepted) **[ADR-003](./0003-standardized-input-validation-using-middleware.md)**: Standardized Input Validation using Middleware (Accepted)
**[ADR-008](./0008-api-versioning-strategy.md)**: API Versioning Strategy (Accepted - Phase 1 Complete) **[ADR-008](./0008-api-versioning-strategy.md)**: API Versioning Strategy (Accepted - Phase 2 Complete)
**[ADR-018](./0018-api-documentation-strategy.md)**: API Documentation Strategy (Accepted) **[ADR-018](./0018-api-documentation-strategy.md)**: API Documentation Strategy (Accepted)
**[ADR-022](./0022-real-time-notification-system.md)**: Real-time Notification System (Proposed) **[ADR-022](./0022-real-time-notification-system.md)**: Real-time Notification System (Accepted)
**[ADR-028](./0028-api-response-standardization.md)**: API Response Standardization and Envelope Pattern (Implemented) **[ADR-028](./0028-api-response-standardization.md)**: API Response Standardization and Envelope Pattern (Implemented)
## 4. Security & Compliance ## 4. Security & Compliance
@@ -33,12 +35,12 @@ This directory contains a log of the architectural decisions made for the Flyer
**[ADR-029](./0029-secret-rotation-and-key-management.md)**: Secret Rotation and Key Management Strategy (Proposed) **[ADR-029](./0029-secret-rotation-and-key-management.md)**: Secret Rotation and Key Management Strategy (Proposed)
**[ADR-032](./0032-rate-limiting-strategy.md)**: Rate Limiting Strategy (Accepted) **[ADR-032](./0032-rate-limiting-strategy.md)**: Rate Limiting Strategy (Accepted)
**[ADR-033](./0033-file-upload-and-storage-strategy.md)**: File Upload and Storage Strategy (Accepted) **[ADR-033](./0033-file-upload-and-storage-strategy.md)**: File Upload and Storage Strategy (Accepted)
**[ADR-048](./0048-authentication-strategy.md)**: Authentication Strategy (Partially Implemented) **[ADR-048](./0048-authentication-strategy.md)**: Authentication Strategy (Accepted)
## 5. Observability & Monitoring ## 5. Observability & Monitoring
**[ADR-004](./0004-standardized-application-wide-structured-logging.md)**: Standardized Application-Wide Structured Logging (Accepted) **[ADR-004](./0004-standardized-application-wide-structured-logging.md)**: Standardized Application-Wide Structured Logging (Accepted)
**[ADR-015](./0015-error-tracking-and-observability.md)**: Error Tracking and Observability (Partial) **[ADR-015](./0015-error-tracking-and-observability.md)**: Error Tracking and Observability (Accepted)
**[ADR-050](./0050-postgresql-function-observability.md)**: PostgreSQL Function Observability (Accepted) **[ADR-050](./0050-postgresql-function-observability.md)**: PostgreSQL Function Observability (Accepted)
**[ADR-051](./0051-asynchronous-context-propagation.md)**: Asynchronous Context Propagation (Accepted) **[ADR-051](./0051-asynchronous-context-propagation.md)**: Asynchronous Context Propagation (Accepted)
**[ADR-052](./0052-granular-debug-logging-strategy.md)**: Granular Debug Logging Strategy (Accepted) **[ADR-052](./0052-granular-debug-logging-strategy.md)**: Granular Debug Logging Strategy (Accepted)
@@ -52,7 +54,7 @@ This directory contains a log of the architectural decisions made for the Flyer
**[ADR-024](./0024-feature-flagging-strategy.md)**: Feature Flagging Strategy (Proposed) **[ADR-024](./0024-feature-flagging-strategy.md)**: Feature Flagging Strategy (Proposed)
**[ADR-037](./0037-scheduled-jobs-and-cron-pattern.md)**: Scheduled Jobs and Cron Pattern (Accepted) **[ADR-037](./0037-scheduled-jobs-and-cron-pattern.md)**: Scheduled Jobs and Cron Pattern (Accepted)
**[ADR-038](./0038-graceful-shutdown-pattern.md)**: Graceful Shutdown Pattern (Accepted) **[ADR-038](./0038-graceful-shutdown-pattern.md)**: Graceful Shutdown Pattern (Accepted)
**[ADR-053](./0053-worker-health-checks-and-monitoring.md)**: Worker Health Checks and Monitoring (Proposed) **[ADR-053](./0053-worker-health-checks.md)**: Worker Health Checks and Stalled Job Monitoring (Accepted)
**[ADR-054](./0054-bugsink-gitea-issue-sync.md)**: Bugsink to Gitea Issue Synchronization (Proposed) **[ADR-054](./0054-bugsink-gitea-issue-sync.md)**: Bugsink to Gitea Issue Synchronization (Proposed)
## 7. Frontend / User Interface ## 7. Frontend / User Interface

View File

@@ -1,10 +1,168 @@
# Database Setup # Database Architecture
Flyer Crawler uses PostgreSQL with several extensions for full-text search, geographic data, and UUID generation. **Version**: 0.12.20
**Last Updated**: 2026-01-28
Flyer Crawler uses PostgreSQL 16 with PostGIS for geographic data, pg_trgm for fuzzy text search, and uuid-ossp for UUID generation. The database contains 65 tables organized into logical domains.
## Table of Contents
1. [Schema Overview](#schema-overview)
2. [Database Setup](#database-setup)
3. [Schema Reference](#schema-reference)
4. [Related Documentation](#related-documentation)
--- ---
## Required Extensions ## Schema Overview
The database is organized into the following domains:
### Core Infrastructure (6 tables)
| Table | Purpose | Primary Key |
| ----------------------- | ----------------------------------------- | ----------------- |
| `users` | Authentication credentials and login data | `user_id` (UUID) |
| `profiles` | Public user data, preferences, points | `user_id` (UUID) |
| `addresses` | Normalized address storage with geocoding | `address_id` |
| `activity_log` | User activity audit trail | `activity_log_id` |
| `password_reset_tokens` | Temporary tokens for password reset | `token_id` |
| `schema_info` | Schema deployment metadata | `environment` |
### Stores and Locations (4 tables)
| Table | Purpose | Primary Key |
| ------------------------ | --------------------------------------- | ------------------- |
| `stores` | Grocery store chains (Safeway, Kroger) | `store_id` |
| `store_locations` | Physical store locations with addresses | `store_location_id` |
| `favorite_stores` | User store favorites | `user_id, store_id` |
| `store_receipt_patterns` | Receipt text patterns for store ID | `pattern_id` |
### Flyers and Items (7 tables)
| Table | Purpose | Primary Key |
| ----------------------- | -------------------------------------- | ------------------------ |
| `flyers` | Uploaded flyer metadata and status | `flyer_id` |
| `flyer_items` | Individual deals extracted from flyers | `flyer_item_id` |
| `flyer_locations` | Flyer-to-location associations | `flyer_location_id` |
| `categories` | Item categorization (Produce, Dairy) | `category_id` |
| `master_grocery_items` | Canonical grocery item dictionary | `master_grocery_item_id` |
| `master_item_aliases` | Alternative names for master items | `alias_id` |
| `unmatched_flyer_items` | Items pending master item matching | `unmatched_item_id` |
### Products and Brands (2 tables)
| Table | Purpose | Primary Key |
| ---------- | ---------------------------------------------- | ------------ |
| `brands` | Brand names (Coca-Cola, Kraft) | `brand_id` |
| `products` | Specific products (master item + brand + size) | `product_id` |
### Price Tracking (3 tables)
| Table | Purpose | Primary Key |
| ----------------------- | ---------------------------------- | ------------------ |
| `item_price_history` | Historical prices for master items | `price_history_id` |
| `user_submitted_prices` | User-contributed price reports | `submission_id` |
| `suggested_corrections` | Suggested edits to flyer items | `correction_id` |
### User Features (8 tables)
| Table | Purpose | Primary Key |
| -------------------- | ------------------------------------ | --------------------------- |
| `user_watched_items` | Items user wants to track prices for | `user_watched_item_id` |
| `user_alerts` | Price alert thresholds | `alert_id` |
| `notifications` | User notifications | `notification_id` |
| `user_item_aliases` | User-defined item name aliases | `alias_id` |
| `user_follows` | User-to-user follow relationships | `follower_id, following_id` |
| `user_reactions` | Reactions to content (likes, etc.) | `reaction_id` |
| `budgets` | User-defined spending budgets | `budget_id` |
| `search_queries` | Search history for analytics | `query_id` |
### Shopping Lists (4 tables)
| Table | Purpose | Primary Key |
| ----------------------- | ------------------------ | ------------------------- |
| `shopping_lists` | User shopping lists | `shopping_list_id` |
| `shopping_list_items` | Items on shopping lists | `shopping_list_item_id` |
| `shared_shopping_lists` | Shopping list sharing | `shared_shopping_list_id` |
| `shopping_trips` | Completed shopping trips | `trip_id` |
| `shopping_trip_items` | Items purchased on trips | `trip_item_id` |
### Recipes (11 tables)
| Table | Purpose | Primary Key |
| --------------------------------- | -------------------------------- | ------------------------- |
| `recipes` | User recipes with metadata | `recipe_id` |
| `recipe_ingredients` | Recipe ingredient list | `recipe_ingredient_id` |
| `recipe_ingredient_substitutions` | Ingredient alternatives | `substitution_id` |
| `tags` | Recipe tags (vegan, quick, etc.) | `tag_id` |
| `recipe_tags` | Recipe-to-tag associations | `recipe_id, tag_id` |
| `appliances` | Kitchen appliances | `appliance_id` |
| `recipe_appliances` | Appliances needed for recipes | `recipe_id, appliance_id` |
| `recipe_ratings` | User ratings for recipes | `rating_id` |
| `recipe_comments` | User comments on recipes | `comment_id` |
| `favorite_recipes` | User recipe favorites | `user_id, recipe_id` |
| `recipe_collections` | User recipe collections | `collection_id` |
### Meal Planning (3 tables)
| Table | Purpose | Primary Key |
| ------------------- | -------------------------- | ----------------- |
| `menu_plans` | Weekly/monthly meal plans | `menu_plan_id` |
| `shared_menu_plans` | Menu plan sharing | `share_id` |
| `planned_meals` | Individual meals in a plan | `planned_meal_id` |
### Pantry and Inventory (4 tables)
| Table | Purpose | Primary Key |
| -------------------- | ------------------------------------ | ----------------- |
| `pantry_items` | User pantry inventory | `pantry_item_id` |
| `pantry_locations` | Storage locations (fridge, freezer) | `location_id` |
| `expiry_date_ranges` | Reference shelf life data | `expiry_range_id` |
| `expiry_alerts` | User expiry notification preferences | `expiry_alert_id` |
| `expiry_alert_log` | Sent expiry notifications | `alert_log_id` |
### Receipts (4 tables)
| Table | Purpose | Primary Key |
| ------------------------ | ----------------------------- | ----------------- |
| `receipts` | Scanned receipt metadata | `receipt_id` |
| `receipt_items` | Items parsed from receipts | `receipt_item_id` |
| `receipt_processing_log` | OCR/AI processing audit trail | `log_id` |
### UPC Scanning (2 tables)
| Table | Purpose | Primary Key |
| ---------------------- | ------------------------------- | ----------- |
| `upc_scan_history` | User barcode scan history | `scan_id` |
| `upc_external_lookups` | External UPC API response cache | `lookup_id` |
### Gamification (2 tables)
| Table | Purpose | Primary Key |
| ------------------- | ---------------------------- | ------------------------- |
| `achievements` | Defined achievements | `achievement_id` |
| `user_achievements` | Achievements earned by users | `user_id, achievement_id` |
### User Preferences (3 tables)
| Table | Purpose | Primary Key |
| --------------------------- | ---------------------------- | ------------------------- |
| `dietary_restrictions` | Defined dietary restrictions | `restriction_id` |
| `user_dietary_restrictions` | User dietary preferences | `user_id, restriction_id` |
| `user_appliances` | Appliances user owns | `user_id, appliance_id` |
### Reference Data (1 table)
| Table | Purpose | Primary Key |
| ------------------ | ----------------------- | --------------- |
| `unit_conversions` | Unit conversion factors | `conversion_id` |
---
## Database Setup
### Required Extensions
| Extension | Purpose | | Extension | Purpose |
| ----------- | ------------------------------------------- | | ----------- | ------------------------------------------- |
@@ -14,7 +172,7 @@ Flyer Crawler uses PostgreSQL with several extensions for full-text search, geog
--- ---
## Database Users ### Database Users
This project uses **environment-specific database users** to isolate production and test environments: This project uses **environment-specific database users** to isolate production and test environments:

View File

@@ -1,7 +1,7 @@
# Flyer Crawler - System Architecture Overview # Flyer Crawler - System Architecture Overview
**Version**: 0.12.5 **Version**: 0.12.20
**Last Updated**: 2026-01-22 **Last Updated**: 2026-01-28
**Platform**: Linux (Production and Development) **Platform**: Linux (Production and Development)
--- ---
@@ -41,7 +41,7 @@
## System Architecture Diagram ## System Architecture Diagram
``` ```text
+-----------------------------------------------------------------------------------+ +-----------------------------------------------------------------------------------+
| CLIENT LAYER | | CLIENT LAYER |
+-----------------------------------------------------------------------------------+ +-----------------------------------------------------------------------------------+
@@ -153,10 +153,10 @@
| Component | Technology | Version | Purpose | | Component | Technology | Version | Purpose |
| ---------------------- | ---------- | -------- | -------------------------------- | | ---------------------- | ---------- | -------- | -------------------------------- |
| **Runtime** | Node.js | 22.x LTS | Server-side JavaScript runtime | | **Runtime** | Node.js | 22.x LTS | Server-side JavaScript runtime |
| **Language** | TypeScript | 5.9.x | Type-safe JavaScript superset | | **Language** | TypeScript | 5.9.3 | Type-safe JavaScript superset |
| **Web Framework** | Express.js | 5.1.x | HTTP server and routing | | **Web Framework** | Express.js | 5.1.0 | HTTP server and routing |
| **Frontend Framework** | React | 19.2.x | UI component library | | **Frontend Framework** | React | 19.2.0 | UI component library |
| **Build Tool** | Vite | 7.2.x | Frontend bundling and dev server | | **Build Tool** | Vite | 7.2.4 | Frontend bundling and dev server |
### Data Storage ### Data Storage
@@ -176,23 +176,23 @@
| **OAuth** | Google, GitHub | Social authentication | | **OAuth** | Google, GitHub | Social authentication |
| **Email** | Nodemailer (SMTP) | Transactional emails | | **Email** | Nodemailer (SMTP) | Transactional emails |
### Background Processing ### Background Processing Stack
| Component | Technology | Version | Purpose | | Component | Technology | Version | Purpose |
| ------------------- | ---------- | ------- | --------------------------------- | | ------------------- | ---------- | ------- | --------------------------------- |
| **Job Queues** | BullMQ | 5.65.x | Reliable async job processing | | **Job Queues** | BullMQ | 5.65.1 | Reliable async job processing |
| **Process Manager** | PM2 | Latest | Process management and clustering | | **Process Manager** | PM2 | Latest | Process management and clustering |
| **Scheduler** | node-cron | 4.2.x | Scheduled tasks | | **Scheduler** | node-cron | 4.2.1 | Scheduled tasks |
### Frontend Stack ### Frontend Stack
| Component | Technology | Version | Purpose | | Component | Technology | Version | Purpose |
| -------------------- | -------------- | ------- | ---------------------------------------- | | -------------------- | -------------- | ------- | ---------------------------------------- |
| **State Management** | TanStack Query | 5.90.x | Server state caching and synchronization | | **State Management** | TanStack Query | 5.90.12 | Server state caching and synchronization |
| **Routing** | React Router | 7.9.x | Client-side routing | | **Routing** | React Router | 7.9.6 | Client-side routing |
| **Styling** | Tailwind CSS | 4.1.x | Utility-first CSS framework | | **Styling** | Tailwind CSS | 4.1.17 | Utility-first CSS framework |
| **Icons** | Lucide React | 0.555.x | Icon components | | **Icons** | Lucide React | 0.555.0 | Icon components |
| **Charts** | Recharts | 3.4.x | Data visualization | | **Charts** | Recharts | 3.4.1 | Data visualization |
### Observability and Quality ### Observability and Quality
@@ -221,7 +221,7 @@ The frontend is a single-page application (SPA) built with React 19 and Vite.
**Directory Structure**: **Directory Structure**:
``` ```text
src/ src/
+-- components/ # Reusable UI components +-- components/ # Reusable UI components
+-- contexts/ # React context providers +-- contexts/ # React context providers
@@ -244,17 +244,30 @@ The backend is a RESTful API server built with Express.js 5.
- Structured logging with Pino - Structured logging with Pino
- Standardized error handling (ADR-001) - Standardized error handling (ADR-001)
**API Route Modules**: **API Route Modules** (all versioned under `/api/v1/*`):
| Route | Purpose |
|-------|---------| | Route | Purpose |
| `/api/auth` | Authentication (login, register, OAuth) | | ------------------------- | ----------------------------------------------- |
| `/api/users` | User profile management | | `/api/v1/auth` | Authentication (login, register, OAuth) |
| `/api/flyers` | Flyer CRUD and processing | | `/api/v1/health` | Health checks and monitoring |
| `/api/recipes` | Recipe management | | `/api/v1/system` | System administration (PM2 status, server info) |
| `/api/deals` | Best prices and deal discovery | | `/api/v1/users` | User profile management |
| `/api/stores` | Store management | | `/api/v1/ai` | AI-powered features and flyer processing |
| `/api/admin` | Administrative functions | | `/api/v1/admin` | Administrative functions |
| `/api/health` | Health checks and monitoring | | `/api/v1/budgets` | Budget management and spending analysis |
| `/api/v1/achievements` | Gamification and achievement system |
| `/api/v1/flyers` | Flyer CRUD and processing |
| `/api/v1/recipes` | Recipe management and recommendations |
| `/api/v1/personalization` | Master items and user preferences |
| `/api/v1/price-history` | Price tracking and trend analysis |
| `/api/v1/stats` | Public statistics and analytics |
| `/api/v1/upc` | UPC barcode scanning and product lookup |
| `/api/v1/inventory` | Inventory and expiry tracking |
| `/api/v1/receipts` | Receipt scanning and purchase history |
| `/api/v1/deals` | Best prices and deal discovery |
| `/api/v1/reactions` | Social features (reactions, sharing) |
| `/api/v1/stores` | Store management and location services |
| `/api/v1/categories` | Category browsing and product categorization |
### Database (PostgreSQL/PostGIS) ### Database (PostgreSQL/PostGIS)
@@ -331,7 +344,7 @@ BullMQ workers handle asynchronous processing tasks. PM2 manages both the API se
### Flyer Processing Pipeline ### Flyer Processing Pipeline
``` ```text
+-------------+ +----------------+ +------------------+ +---------------+ +-------------+ +----------------+ +------------------+ +---------------+
| User | | Express | | BullMQ | | PostgreSQL | | User | | Express | | BullMQ | | PostgreSQL |
| Upload +---->+ Route +---->+ Queue +---->+ Storage | | Upload +---->+ Route +---->+ Queue +---->+ Storage |
@@ -395,7 +408,7 @@ BullMQ workers handle asynchronous processing tasks. PM2 manages both the API se
The application follows a strict layered architecture as defined in ADR-035. The application follows a strict layered architecture as defined in ADR-035.
``` ```text
+-----------------------------------------------------------------------+ +-----------------------------------------------------------------------+
| ROUTES LAYER | | ROUTES LAYER |
| Responsibilities: | | Responsibilities: |
@@ -458,7 +471,7 @@ The application follows a strict layered architecture as defined in ADR-035.
### Entity Relationship Overview ### Entity Relationship Overview
``` ```text
+------------------+ +------------------+ +------------------+ +------------------+ +------------------+ +------------------+
| users | | profiles | | addresses | | users | | profiles | | addresses |
|------------------| |------------------| |------------------| |------------------| |------------------| |------------------|
@@ -537,7 +550,7 @@ The application follows a strict layered architecture as defined in ADR-035.
### JWT Token Architecture ### JWT Token Architecture
``` ```text
+-------------------+ +-------------------+ +-------------------+ +-------------------+ +-------------------+ +-------------------+
| Login Request | | Server | | Database | | Login Request | | Server | | Database |
| (email/pass) +---->+ Validates +---->+ Verify User | | (email/pass) +---->+ Validates +---->+ Verify User |
@@ -576,7 +589,7 @@ The application follows a strict layered architecture as defined in ADR-035.
### Protected Route Flow ### Protected Route Flow
``` ```text
+-------------------+ +-------------------+ +-------------------+ +-------------------+ +-------------------+ +-------------------+
| API Request | | requireAuth | | JWT Strategy | | API Request | | requireAuth | | JWT Strategy |
| + Bearer Token +---->+ Middleware +---->+ Validate | | + Bearer Token +---->+ Middleware +---->+ Validate |
@@ -603,7 +616,7 @@ The application follows a strict layered architecture as defined in ADR-035.
### Worker Architecture ### Worker Architecture
``` ```text
+-------------------+ +-------------------+ +-------------------+ +-------------------+ +-------------------+ +-------------------+
| API Server | | Redis | | Worker Process | | API Server | | Redis | | Worker Process |
| (Queue Producer)| | (Job Storage) | | (Consumer) | | (Queue Producer)| | (Job Storage) | | (Consumer) |
@@ -635,7 +648,7 @@ The application follows a strict layered architecture as defined in ADR-035.
Jobs use exponential backoff for retries: Jobs use exponential backoff for retries:
``` ```text
Attempt 1: Immediate Attempt 1: Immediate
Attempt 2: Initial delay (e.g., 5 seconds) Attempt 2: Initial delay (e.g., 5 seconds)
Attempt 3: 2x delay (e.g., 10 seconds) Attempt 3: 2x delay (e.g., 10 seconds)
@@ -658,7 +671,7 @@ Attempt 4: 4x delay (e.g., 20 seconds)
### Environment Overview ### Environment Overview
``` ```text
+-----------------------------------------------------------------------------------+ +-----------------------------------------------------------------------------------+
| DEVELOPMENT | | DEVELOPMENT |
+-----------------------------------------------------------------------------------+ +-----------------------------------------------------------------------------------+
@@ -710,7 +723,7 @@ Attempt 4: 4x delay (e.g., 20 seconds)
### Deployment Pipeline (ADR-017) ### Deployment Pipeline (ADR-017)
``` ```text
+------------+ +------------+ +------------+ +------------+ +------------+ +------------+ +------------+ +------------+
| Push to | | Gitea | | Build & | | Deploy | | Push to | | Gitea | | Build & | | Deploy |
| main +---->+ Actions +---->+ Test +---->+ to Prod | | main +---->+ Actions +---->+ Test +---->+ to Prod |
@@ -839,22 +852,55 @@ The system architecture is governed by Architecture Decision Records (ADRs). Key
| File | Purpose | | File | Purpose |
| ----------------------------------------------- | --------------------------------------- | | ----------------------------------------------- | --------------------------------------- |
| `src/services/flyerProcessingService.server.ts` | Flyer processing pipeline orchestration | | `src/services/flyerProcessingService.server.ts` | Flyer processing pipeline orchestration |
| `src/services/flyerAiProcessor.server.ts` | AI extraction for flyers |
| `src/services/aiService.server.ts` | Google Gemini AI integration | | `src/services/aiService.server.ts` | Google Gemini AI integration |
| `src/services/cacheService.server.ts` | Redis caching abstraction | | `src/services/cacheService.server.ts` | Redis caching abstraction |
| `src/services/emailService.server.ts` | Email sending | | `src/services/emailService.server.ts` | Email sending |
| `src/services/queues.server.ts` | BullMQ queue definitions | | `src/services/queues.server.ts` | BullMQ queue definitions |
| `src/services/queueService.server.ts` | Queue management and scheduling |
| `src/services/workers.server.ts` | BullMQ worker definitions | | `src/services/workers.server.ts` | BullMQ worker definitions |
| `src/services/websocketService.server.ts` | Real-time WebSocket notifications |
| `src/services/receiptService.server.ts` | Receipt scanning and OCR |
| `src/services/upcService.server.ts` | UPC barcode lookup |
| `src/services/expiryService.server.ts` | Pantry expiry tracking |
| `src/services/geocodingService.server.ts` | Address geocoding |
| `src/services/analyticsService.server.ts` | Analytics and reporting |
| `src/services/monitoringService.server.ts` | Health monitoring |
| `src/services/barcodeService.server.ts` | Barcode detection |
| `src/services/logger.server.ts` | Structured logging (Pino) |
| `src/services/redis.server.ts` | Redis connection management |
| `src/services/sentry.server.ts` | Error tracking (Sentry/Bugsink) |
### Database Files ### Database Files
| File | Purpose | | File | Purpose |
| ---------------------------------- | -------------------------------------------- | | --------------------------------------- | -------------------------------------------- |
| `src/services/db/connection.db.ts` | Database pool and transaction management | | `src/services/db/connection.db.ts` | Database pool and transaction management |
| `src/services/db/errors.db.ts` | Database error types | | `src/services/db/errors.db.ts` | Database error types |
| `src/services/db/user.db.ts` | User repository | | `src/services/db/index.db.ts` | Repository exports |
| `src/services/db/flyer.db.ts` | Flyer repository | | `src/services/db/user.db.ts` | User repository |
| `sql/master_schema_rollup.sql` | Complete database schema (for test DB setup) | | `src/services/db/flyer.db.ts` | Flyer repository |
| `sql/initial_schema.sql` | Fresh installation schema | | `src/services/db/store.db.ts` | Store repository |
| `src/services/db/storeLocation.db.ts` | Store location repository |
| `src/services/db/recipe.db.ts` | Recipe repository |
| `src/services/db/category.db.ts` | Category repository |
| `src/services/db/personalization.db.ts` | Master items and personalization |
| `src/services/db/shopping.db.ts` | Shopping lists repository |
| `src/services/db/deals.db.ts` | Deals and best prices repository |
| `src/services/db/price.db.ts` | Price history repository |
| `src/services/db/receipt.db.ts` | Receipt repository |
| `src/services/db/upc.db.ts` | UPC scan history repository |
| `src/services/db/expiry.db.ts` | Expiry tracking repository |
| `src/services/db/gamification.db.ts` | Achievements repository |
| `src/services/db/budget.db.ts` | Budget repository |
| `src/services/db/reaction.db.ts` | User reactions repository |
| `src/services/db/notification.db.ts` | Notifications repository |
| `src/services/db/address.db.ts` | Address repository |
| `src/services/db/admin.db.ts` | Admin operations repository |
| `src/services/db/conversion.db.ts` | Unit conversion repository |
| `src/services/db/flyerLocation.db.ts` | Flyer locations repository |
| `sql/master_schema_rollup.sql` | Complete database schema (for test DB setup) |
| `sql/initial_schema.sql` | Fresh installation schema |
### Type Definitions ### Type Definitions

View File

@@ -2,6 +2,21 @@
Common code patterns extracted from Architecture Decision Records (ADRs). Use these as templates when writing new code. Common code patterns extracted from Architecture Decision Records (ADRs). Use these as templates when writing new code.
## Quick Reference
| Pattern | Key Function/Class | Import From |
| ------------------ | ------------------------------------------------- | ---------------------------------- |
| Error Handling | `handleDbError()`, `NotFoundError` | `src/services/db/errors.db.ts` |
| Repository Methods | `get*`, `find*`, `list*` | `src/services/db/*.db.ts` |
| API Responses | `sendSuccess()`, `sendPaginated()`, `sendError()` | `src/utils/apiResponse.ts` |
| Transactions | `withTransaction()` | `src/services/db/connection.db.ts` |
| Validation | `validateRequest()` | `src/middleware/validation.ts` |
| Authentication | `authenticateJWT` | `src/middleware/auth.ts` |
| Caching | `cacheService` | `src/services/cache.server.ts` |
| Background Jobs | Queue classes | `src/services/queues.server.ts` |
---
## Table of Contents ## Table of Contents
- [Error Handling](#error-handling) - [Error Handling](#error-handling)
@@ -17,7 +32,7 @@ Common code patterns extracted from Architecture Decision Records (ADRs). Use th
## Error Handling ## Error Handling
**ADR**: [ADR-001](../adr/0001-standardized-error-handling-for-database-operations.md) **ADR**: [ADR-001](../adr/0001-standardized-error-handling.md)
### Repository Layer Error Handling ### Repository Layer Error Handling
@@ -78,7 +93,7 @@ throw new DatabaseError('Failed to insert flyer', originalError);
## Repository Patterns ## Repository Patterns
**ADR**: [ADR-034](../adr/0034-repository-layer-method-naming-conventions.md) **ADR**: [ADR-034](../adr/0034-repository-pattern-standards.md)
### Method Naming Conventions ### Method Naming Conventions
@@ -155,16 +170,17 @@ export async function listActiveFlyers(client?: PoolClient): Promise<Flyer[]> {
## API Response Patterns ## API Response Patterns
**ADR**: [ADR-028](../adr/0028-consistent-api-response-format.md) **ADR**: [ADR-028](../adr/0028-api-response-standardization.md)
### Success Response ### Success Response
```typescript ```typescript
import { sendSuccess } from '../utils/apiResponse'; import { sendSuccess } from '../utils/apiResponse';
app.post('/api/flyers', async (req, res) => { app.post('/api/v1/flyers', async (req, res) => {
const flyer = await flyerService.createFlyer(req.body); const flyer = await flyerService.createFlyer(req.body);
return sendSuccess(res, flyer, 'Flyer created successfully', 201); // sendSuccess(res, data, statusCode?, meta?)
return sendSuccess(res, flyer, 201);
}); });
``` ```
@@ -173,30 +189,32 @@ app.post('/api/flyers', async (req, res) => {
```typescript ```typescript
import { sendPaginated } from '../utils/apiResponse'; import { sendPaginated } from '../utils/apiResponse';
app.get('/api/flyers', async (req, res) => { app.get('/api/v1/flyers', async (req, res) => {
const { page = 1, pageSize = 20 } = req.query; const page = parseInt(req.query.page as string) || 1;
const { items, total } = await flyerService.listFlyers(page, pageSize); const limit = parseInt(req.query.limit as string) || 20;
const { items, total } = await flyerService.listFlyers(page, limit);
return sendPaginated(res, { // sendPaginated(res, data[], { page, limit, total }, meta?)
items, return sendPaginated(res, items, { page, limit, total });
total,
page: parseInt(page),
pageSize: parseInt(pageSize),
});
}); });
``` ```
### Error Response ### Error Response
```typescript ```typescript
import { sendError } from '../utils/apiResponse'; import { sendError, sendSuccess, ErrorCode } from '../utils/apiResponse';
app.get('/api/flyers/:id', async (req, res) => { app.get('/api/v1/flyers/:id', async (req, res) => {
try { try {
const flyer = await flyerDb.getFlyerById(parseInt(req.params.id)); const flyer = await flyerDb.getFlyerById(parseInt(req.params.id));
return sendSuccess(res, flyer); return sendSuccess(res, flyer);
} catch (error) { } catch (error) {
return sendError(res, error); // Automatically maps error to correct status // sendError(res, code, message, statusCode?, details?, meta?)
if (error instanceof NotFoundError) {
return sendError(res, ErrorCode.NOT_FOUND, error.message, 404);
}
req.log.error({ error }, `Error in ${req.originalUrl.split('?')[0]}:`);
return sendError(res, ErrorCode.INTERNAL_ERROR, 'An error occurred', 500);
} }
}); });
``` ```
@@ -205,12 +223,12 @@ app.get('/api/flyers/:id', async (req, res) => {
## Transaction Management ## Transaction Management
**ADR**: [ADR-002](../adr/0002-transaction-management-pattern.md) **ADR**: [ADR-002](../adr/0002-standardized-transaction-management.md)
### Basic Transaction ### Basic Transaction
```typescript ```typescript
import { withTransaction } from '../services/db/transaction.db'; import { withTransaction } from '../services/db/connection.db';
export async function createFlyerWithItems( export async function createFlyerWithItems(
flyerData: FlyerInput, flyerData: FlyerInput,
@@ -262,7 +280,7 @@ export async function bulkImportFlyers(flyersData: FlyerInput[]): Promise<Import
## Input Validation ## Input Validation
**ADR**: [ADR-003](../adr/0003-input-validation-framework.md) **ADR**: [ADR-003](../adr/0003-standardized-input-validation-using-middleware.md)
### Zod Schema Definition ### Zod Schema Definition
@@ -298,10 +316,10 @@ export type CreateFlyerInput = z.infer<typeof createFlyerSchema>;
import { validateRequest } from '../middleware/validation'; import { validateRequest } from '../middleware/validation';
import { createFlyerSchema } from '../schemas/flyer.schemas'; import { createFlyerSchema } from '../schemas/flyer.schemas';
app.post('/api/flyers', validateRequest(createFlyerSchema), async (req, res) => { app.post('/api/v1/flyers', validateRequest(createFlyerSchema), async (req, res) => {
// req.body is now type-safe and validated // req.body is now type-safe and validated
const flyer = await flyerService.createFlyer(req.body); const flyer = await flyerService.createFlyer(req.body);
return sendSuccess(res, flyer, 'Flyer created successfully', 201); return sendSuccess(res, flyer, 201);
}); });
``` ```
@@ -331,7 +349,7 @@ export async function processFlyer(data: unknown): Promise<Flyer> {
import { authenticateJWT } from '../middleware/auth'; import { authenticateJWT } from '../middleware/auth';
app.get( app.get(
'/api/profile', '/api/v1/profile',
authenticateJWT, // Middleware adds req.user authenticateJWT, // Middleware adds req.user
async (req, res) => { async (req, res) => {
// req.user is guaranteed to exist // req.user is guaranteed to exist
@@ -347,7 +365,7 @@ app.get(
import { optionalAuth } from '../middleware/auth'; import { optionalAuth } from '../middleware/auth';
app.get( app.get(
'/api/flyers', '/api/v1/flyers',
optionalAuth, // req.user may or may not exist optionalAuth, // req.user may or may not exist
async (req, res) => { async (req, res) => {
const flyers = req.user const flyers = req.user
@@ -374,7 +392,7 @@ export function generateToken(user: User): string {
## Caching ## Caching
**ADR**: [ADR-029](../adr/0029-redis-caching-strategy.md) **ADR**: [ADR-009](../adr/0009-caching-strategy-for-read-heavy-operations.md)
### Cache Pattern ### Cache Pattern
@@ -414,7 +432,7 @@ export async function updateFlyer(id: number, data: UpdateFlyerInput): Promise<F
## Background Jobs ## Background Jobs
**ADR**: [ADR-036](../adr/0036-background-job-processing-architecture.md) **ADR**: [ADR-006](../adr/0006-background-job-processing-and-task-queues.md)
### Queue Job ### Queue Job

View File

@@ -229,7 +229,7 @@ SELECT * FROM flyers WHERE store_id = 1;
- Add missing indexes - Add missing indexes
- Optimize WHERE clauses - Optimize WHERE clauses
- Use connection pooling - Use connection pooling
- See [ADR-034](../adr/0034-repository-layer-method-naming-conventions.md) - See [ADR-034](../adr/0034-repository-pattern-standards.md)
--- ---
@@ -237,7 +237,7 @@ SELECT * FROM flyers WHERE store_id = 1;
### Tests Pass on Windows, Fail in Container ### Tests Pass on Windows, Fail in Container
**Cause**: Platform-specific behavior (ADR-014) **Cause**: Platform-specific behavior ([ADR-014](../adr/0014-containerization-and-deployment-strategy.md))
**Rule**: Container results are authoritative. Windows results are unreliable. **Rule**: Container results are authoritative. Windows results are unreliable.

View File

@@ -93,7 +93,7 @@ When the container starts (`scripts/dev-entrypoint.sh`):
PM2 manages three processes in the dev container: PM2 manages three processes in the dev container:
``` ```text
+--------------------+ +------------------------+ +--------------------+ +--------------------+ +------------------------+ +--------------------+
| flyer-crawler- | | flyer-crawler- | | flyer-crawler- | | flyer-crawler- | | flyer-crawler- | | flyer-crawler- |
| api-dev | | worker-dev | | vite-dev | | api-dev | | worker-dev | | vite-dev |
@@ -404,5 +404,5 @@ podman exec -it flyer-crawler-dev pm2 restart flyer-crawler-api-dev
- [DEBUGGING.md](DEBUGGING.md) - Debugging strategies - [DEBUGGING.md](DEBUGGING.md) - Debugging strategies
- [LOGSTASH-QUICK-REF.md](../operations/LOGSTASH-QUICK-REF.md) - Logstash quick reference - [LOGSTASH-QUICK-REF.md](../operations/LOGSTASH-QUICK-REF.md) - Logstash quick reference
- [DEV-CONTAINER-BUGSINK.md](../DEV-CONTAINER-BUGSINK.md) - Bugsink setup in dev container - [DEV-CONTAINER-BUGSINK.md](../DEV-CONTAINER-BUGSINK.md) - Bugsink setup in dev container
- [ADR-014](../adr/0014-linux-only-platform.md) - Linux-only platform decision - [ADR-014](../adr/0014-containerization-and-deployment-strategy.md) - Containerization and deployment strategy
- [ADR-050](../adr/0050-postgresql-function-observability.md) - PostgreSQL function observability - [ADR-050](../adr/0050-postgresql-function-observability.md) - PostgreSQL function observability (includes log aggregation)

View File

@@ -1,5 +1,19 @@
# Testing Guide # Testing Guide
## Quick Reference
| Command | Purpose |
| ------------------------------------------------------------ | ---------------------------- |
| `podman exec -it flyer-crawler-dev npm test` | Run all tests |
| `podman exec -it flyer-crawler-dev npm run test:unit` | Unit tests (~2900) |
| `podman exec -it flyer-crawler-dev npm run test:integration` | Integration tests (28 files) |
| `podman exec -it flyer-crawler-dev npm run test:e2e` | E2E tests (11 files) |
| `podman exec -it flyer-crawler-dev npm run type-check` | TypeScript check |
**Critical**: Always run tests in the dev container. Windows results are unreliable.
---
## Overview ## Overview
This project has comprehensive test coverage including unit tests, integration tests, and E2E tests. All tests must be run in the **Linux dev container environment** for reliable results. This project has comprehensive test coverage including unit tests, integration tests, and E2E tests. All tests must be run in the **Linux dev container environment** for reliable results.
@@ -76,7 +90,7 @@ To verify type-check is working correctly:
Example error output: Example error output:
``` ```text
src/pages/MyDealsPage.tsx:68:31 - error TS2339: Property 'store_name' does not exist on type 'WatchedItemDeal'. src/pages/MyDealsPage.tsx:68:31 - error TS2339: Property 'store_name' does not exist on type 'WatchedItemDeal'.
68 <span>{deal.store_name}</span> 68 <span>{deal.store_name}</span>
@@ -113,15 +127,26 @@ Located throughout `src/` directory alongside source files with `.test.ts` or `.
npm run test:unit npm run test:unit
``` ```
### Integration Tests (5 test files) ### Integration Tests (28 test files)
Located in `src/tests/integration/`: Located in `src/tests/integration/`. Key test files include:
- `admin.integration.test.ts` | Test File | Domain |
- `flyer.integration.test.ts` | -------------------------------------- | -------------------------- |
- `price.integration.test.ts` | `admin.integration.test.ts` | Admin dashboard operations |
- `public.routes.integration.test.ts` | `auth.integration.test.ts` | Authentication flows |
- `receipt.integration.test.ts` | `budget.integration.test.ts` | Budget management |
| `flyer.integration.test.ts` | Flyer CRUD operations |
| `flyer-processing.integration.test.ts` | AI flyer processing |
| `gamification.integration.test.ts` | Achievements and points |
| `inventory.integration.test.ts` | Inventory management |
| `notification.integration.test.ts` | User notifications |
| `receipt.integration.test.ts` | Receipt processing |
| `recipe.integration.test.ts` | Recipe management |
| `shopping-list.integration.test.ts` | Shopping list operations |
| `user.integration.test.ts` | User profile operations |
See `src/tests/integration/` for the complete list.
Requires PostgreSQL and Redis services running. Requires PostgreSQL and Redis services running.
@@ -129,13 +154,23 @@ Requires PostgreSQL and Redis services running.
npm run test:integration npm run test:integration
``` ```
### E2E Tests (3 test files) ### E2E Tests (11 test files)
Located in `src/tests/e2e/`: Located in `src/tests/e2e/`. Full user journey tests:
- `deals-journey.e2e.test.ts` | Test File | Journey |
- `budget-journey.e2e.test.ts` | --------------------------------- | ----------------------------- |
- `receipt-journey.e2e.test.ts` | `admin-authorization.e2e.test.ts` | Admin access control |
| `admin-dashboard.e2e.test.ts` | Admin dashboard flows |
| `auth.e2e.test.ts` | Login/logout/registration |
| `budget-journey.e2e.test.ts` | Budget tracking workflow |
| `deals-journey.e2e.test.ts` | Finding and saving deals |
| `error-reporting.e2e.test.ts` | Error handling verification |
| `flyer-upload.e2e.test.ts` | Flyer upload and processing |
| `inventory-journey.e2e.test.ts` | Pantry management |
| `receipt-journey.e2e.test.ts` | Receipt scanning and tracking |
| `upc-journey.e2e.test.ts` | UPC barcode scanning |
| `user-journey.e2e.test.ts` | User profile management |
Requires all services (PostgreSQL, Redis, BullMQ workers) running. Requires all services (PostgreSQL, Redis, BullMQ workers) running.
@@ -157,20 +192,18 @@ Located in `src/tests/utils/storeHelpers.ts`:
```typescript ```typescript
// Create a store with a location in one call // Create a store with a location in one call
const store = await createStoreWithLocation({ const store = await createStoreWithLocation(pool, {
storeName: 'Test Store', name: 'Test Store',
address: { address: '123 Main St',
address_line_1: '123 Main St', city: 'Toronto',
city: 'Toronto', province: 'ON',
province_state: 'ON', postalCode: 'M1M 1M1',
postal_code: 'M1M 1M1',
},
pool,
log,
}); });
// Returns: { storeId, addressId, storeLocationId }
// Cleanup stores and their locations // Cleanup stores and their locations
await cleanupStoreLocations([storeId1, storeId2], pool, log); await cleanupStoreLocation(pool, store);
``` ```
### Mock Factories ### Mock Factories

View File

@@ -2,134 +2,259 @@
Complete guide to environment variables used in Flyer Crawler. Complete guide to environment variables used in Flyer Crawler.
---
## Quick Reference
### Minimum Required Variables (Development)
| Variable | Example | Purpose |
| ---------------- | ------------------------ | -------------------- |
| `DB_HOST` | `localhost` | PostgreSQL host |
| `DB_USER` | `postgres` | PostgreSQL username |
| `DB_PASSWORD` | `postgres` | PostgreSQL password |
| `DB_NAME` | `flyer_crawler_dev` | Database name |
| `REDIS_URL` | `redis://localhost:6379` | Redis connection URL |
| `JWT_SECRET` | (32+ character string) | JWT signing key |
| `GEMINI_API_KEY` | `AIzaSy...` | Google Gemini API |
### Source of Truth
The Zod schema at `src/config/env.ts` is the authoritative source for all environment variables. If a variable is not in this file, it is not used by the application.
---
## Configuration by Environment ## Configuration by Environment
### Production ### Production
**Location**: Gitea CI/CD secrets injected during deployment | Aspect | Details |
**Path**: `/var/www/flyer-crawler.projectium.com/` | -------- | ------------------------------------------ |
**Note**: No `.env` file exists - all variables come from CI/CD | Location | Gitea CI/CD secrets injected at deployment |
| Path | `/var/www/flyer-crawler.projectium.com/` |
| File | No `.env` file - all from CI/CD secrets |
### Test ### Test
**Location**: Gitea CI/CD secrets + `.env.test` file | Aspect | Details |
**Path**: `/var/www/flyer-crawler-test.projectium.com/` | -------- | --------------------------------------------- |
**Note**: `.env.test` overrides for test-specific values | Location | Gitea CI/CD secrets + `.env.test` overrides |
| Path | `/var/www/flyer-crawler-test.projectium.com/` |
| File | `.env.test` for test-specific values |
### Development Container ### Development Container
**Location**: `.env.local` file in project root | Aspect | Details |
**Note**: Overrides default DSNs in `compose.dev.yml` | -------- | --------------------------------------- |
| Location | `.env.local` file in project root |
| Priority | Overrides defaults in `compose.dev.yml` |
| File | `.env.local` (gitignored) |
## Required Variables ---
### Database ## Complete Variable Reference
| Variable | Description | Example | ### Database Configuration
| ------------------ | ---------------------------- | ------------------------------------------ |
| `DB_HOST` | PostgreSQL host | `localhost` (dev), `projectium.com` (prod) |
| `DB_PORT` | PostgreSQL port | `5432` |
| `DB_USER_PROD` | Production database user | `flyer_crawler_prod` |
| `DB_PASSWORD_PROD` | Production database password | (secret) |
| `DB_DATABASE_PROD` | Production database name | `flyer-crawler-prod` |
| `DB_USER_TEST` | Test database user | `flyer_crawler_test` |
| `DB_PASSWORD_TEST` | Test database password | (secret) |
| `DB_DATABASE_TEST` | Test database name | `flyer-crawler-test` |
| `DB_USER` | Dev database user | `postgres` |
| `DB_PASSWORD` | Dev database password | `postgres` |
| `DB_NAME` | Dev database name | `flyer_crawler_dev` |
**Note**: Production and test use separate `_PROD` and `_TEST` suffixed variables. Development uses unsuffixed variables. | Variable | Required | Default | Description |
| ------------- | -------- | ------- | ----------------- |
| `DB_HOST` | Yes | - | PostgreSQL host |
| `DB_PORT` | No | `5432` | PostgreSQL port |
| `DB_USER` | Yes | - | Database username |
| `DB_PASSWORD` | Yes | - | Database password |
| `DB_NAME` | Yes | - | Database name |
### Redis **Environment-Specific Variables** (Gitea Secrets):
| Variable | Description | Example | | Variable | Environment | Description |
| --------------------- | ------------------------- | ------------------------------ | | ------------------ | ----------- | ------------------------ |
| `REDIS_URL` | Redis connection URL | `redis://localhost:6379` (dev) | | `DB_USER_PROD` | Production | Production database user |
| `REDIS_PASSWORD_PROD` | Production Redis password | (secret) | | `DB_PASSWORD_PROD` | Production | Production database pass |
| `REDIS_PASSWORD_TEST` | Test Redis password | (secret) | | `DB_DATABASE_PROD` | Production | Production database name |
| `DB_USER_TEST` | Test | Test database user |
| `DB_PASSWORD_TEST` | Test | Test database password |
| `DB_DATABASE_TEST` | Test | Test database name |
### Redis Configuration
| Variable | Required | Default | Description |
| ---------------- | -------- | ------- | ------------------------- |
| `REDIS_URL` | Yes | - | Redis connection URL |
| `REDIS_PASSWORD` | No | - | Redis password (optional) |
**URL Format**: `redis://[user:password@]host:port`
**Examples**:
```bash
# Development (no auth)
REDIS_URL=redis://localhost:6379
# Production (with auth)
REDIS_URL=redis://:${REDIS_PASSWORD_PROD}@localhost:6379
```
### Authentication ### Authentication
| Variable | Description | Example | | Variable | Required | Min Length | Description |
| ---------------------- | -------------------------- | -------------------------------- | | ---------------------- | -------- | ---------- | ----------------------- |
| `JWT_SECRET` | JWT token signing key | (minimum 32 characters) | | `JWT_SECRET` | Yes | 32 chars | JWT token signing key |
| `SESSION_SECRET` | Session encryption key | (minimum 32 characters) | | `JWT_SECRET_PREVIOUS` | No | - | Previous key (rotation) |
| `GOOGLE_CLIENT_ID` | Google OAuth client ID | `xxx.apps.googleusercontent.com` | | `GOOGLE_CLIENT_ID` | No | - | Google OAuth client ID |
| `GOOGLE_CLIENT_SECRET` | Google OAuth client secret | (secret) | | `GOOGLE_CLIENT_SECRET` | No | - | Google OAuth secret |
| `GH_CLIENT_ID` | GitHub OAuth client ID | `xxx` | | `GITHUB_CLIENT_ID` | No | - | GitHub OAuth client ID |
| `GH_CLIENT_SECRET` | GitHub OAuth client secret | (secret) | | `GITHUB_CLIENT_SECRET` | No | - | GitHub OAuth secret |
**Generate Secure Secret**:
```bash
node -e "console.log(require('crypto').randomBytes(32).toString('hex'))"
```
### AI Services ### AI Services
| Variable | Description | Example | | Variable | Required | Description |
| -------------------------------- | ---------------------------- | ----------- | | ---------------------------- | -------- | -------------------------------- |
| `VITE_GOOGLE_GENAI_API_KEY` | Google Gemini API key (prod) | `AIzaSy...` | | `GEMINI_API_KEY` | Yes\* | Google Gemini API key |
| `VITE_GOOGLE_GENAI_API_KEY_TEST` | Google Gemini API key (test) | `AIzaSy...` | | `GEMINI_RPM` | No | Rate limit (default: 5) |
| `GOOGLE_MAPS_API_KEY` | Google Maps Geocoding API | `AIzaSy...` | | `AI_PRICE_QUALITY_THRESHOLD` | No | Quality threshold (default: 0.5) |
### Application \*Required for flyer processing. Application works without it but cannot extract flyer data.
| Variable | Description | Example | **Get API Key**: [Google AI Studio](https://aistudio.google.com/app/apikey)
| -------------- | ------------------------ | ----------------------------------- |
| `NODE_ENV` | Environment mode | `development`, `test`, `production` |
| `PORT` | Backend server port | `3001` |
| `FRONTEND_URL` | Frontend application URL | `http://localhost:5173` (dev) |
### Error Tracking ### Google Services
| Variable | Description | Example | | Variable | Required | Description |
| ---------------------- | -------------------------------- | --------------------------- | | ---------------------- | -------- | -------------------------------- |
| `SENTRY_DSN` | Sentry DSN (production) | `https://xxx@sentry.io/xxx` | | `GOOGLE_MAPS_API_KEY` | No | Google Maps Geocoding API |
| `VITE_SENTRY_DSN` | Frontend Sentry DSN (production) | `https://xxx@sentry.io/xxx` | | `GOOGLE_CLIENT_ID` | No | OAuth (see Authentication above) |
| `SENTRY_DSN_TEST` | Sentry DSN (test) | `https://xxx@sentry.io/xxx` | | `GOOGLE_CLIENT_SECRET` | No | OAuth (see Authentication above) |
| `VITE_SENTRY_DSN_TEST` | Frontend Sentry DSN (test) | `https://xxx@sentry.io/xxx` |
| `SENTRY_AUTH_TOKEN` | Sentry API token for releases | (secret) |
## Optional Variables ### UPC Lookup APIs
| Variable | Description | Default | | Variable | Required | Description |
| ------------------- | ----------------------- | ----------------- | | ------------------------ | -------- | ---------------------- |
| `LOG_LEVEL` | Logging verbosity | `info` | | `UPC_ITEM_DB_API_KEY` | No | UPC Item DB API key |
| `REDIS_TTL` | Cache TTL in seconds | `3600` | | `BARCODE_LOOKUP_API_KEY` | No | Barcode Lookup API key |
| `MAX_UPLOAD_SIZE` | Max file upload size | `10mb` |
| `RATE_LIMIT_WINDOW` | Rate limit window (ms) | `900000` (15 min) | ### Application Settings
| `RATE_LIMIT_MAX` | Max requests per window | `100` |
| Variable | Required | Default | Description |
| -------------- | -------- | ------------- | ------------------------ |
| `NODE_ENV` | No | `development` | Environment mode |
| `PORT` | No | `3001` | Backend server port |
| `FRONTEND_URL` | No | - | Frontend URL (CORS) |
| `BASE_URL` | No | - | API base URL |
| `STORAGE_PATH` | No | (see below) | Flyer image storage path |
**NODE_ENV Values**: `development`, `test`, `staging`, `production`
**Default STORAGE_PATH**: `/var/www/flyer-crawler.projectium.com/flyer-images`
### Email/SMTP Configuration
| Variable | Required | Default | Description |
| ----------------- | -------- | ------- | ----------------------- |
| `SMTP_HOST` | No | - | SMTP server hostname |
| `SMTP_PORT` | No | `587` | SMTP server port |
| `SMTP_USER` | No | - | SMTP username |
| `SMTP_PASS` | No | - | SMTP password |
| `SMTP_SECURE` | No | `false` | Use TLS |
| `SMTP_FROM_EMAIL` | No | - | From address for emails |
**Note**: Email functionality degrades gracefully if not configured.
### Worker Configuration
| Variable | Default | Description |
| ------------------------------------- | ------- | ---------------------------- |
| `WORKER_CONCURRENCY` | `1` | Main worker concurrency |
| `WORKER_LOCK_DURATION` | `30000` | Lock duration (ms) |
| `EMAIL_WORKER_CONCURRENCY` | `10` | Email worker concurrency |
| `ANALYTICS_WORKER_CONCURRENCY` | `1` | Analytics worker concurrency |
| `CLEANUP_WORKER_CONCURRENCY` | `10` | Cleanup worker concurrency |
| `WEEKLY_ANALYTICS_WORKER_CONCURRENCY` | `1` | Weekly analytics concurrency |
### Error Tracking (Bugsink/Sentry)
| Variable | Required | Default | Description |
| --------------------- | -------- | -------- | ------------------------------- |
| `SENTRY_DSN` | No | - | Backend Sentry DSN |
| `SENTRY_ENABLED` | No | `true` | Enable error tracking |
| `SENTRY_ENVIRONMENT` | No | NODE_ENV | Environment name for errors |
| `SENTRY_DEBUG` | No | `false` | Enable Sentry SDK debug logging |
| `VITE_SENTRY_DSN` | No | - | Frontend Sentry DSN |
| `VITE_SENTRY_ENABLED` | No | `true` | Enable frontend error tracking |
| `VITE_SENTRY_DEBUG` | No | `false` | Frontend SDK debug logging |
**DSN Format**: `http://[key]@[host]:[port]/[project_id]`
**Dev Container DSNs**:
```bash
# Backend (internal)
SENTRY_DSN=http://<key>@localhost:8000/1
# Frontend (via nginx proxy)
VITE_SENTRY_DSN=https://<key>@localhost/bugsink-api/2
```
---
## Configuration Files ## Configuration Files
| File | Purpose | | File | Purpose |
| ------------------------------------- | ------------------------------------------- | | ------------------------------------- | ------------------------------------------- |
| `src/config/env.ts` | Zod schema validation - **source of truth** | | `src/config/env.ts` | Zod schema validation - **source of truth** |
| `ecosystem.config.cjs` | PM2 process manager config | | `ecosystem.config.cjs` | PM2 process manager (production) |
| `ecosystem.dev.config.cjs` | PM2 process manager (development) |
| `.gitea/workflows/deploy-to-prod.yml` | Production deployment workflow | | `.gitea/workflows/deploy-to-prod.yml` | Production deployment workflow |
| `.gitea/workflows/deploy-to-test.yml` | Test deployment workflow | | `.gitea/workflows/deploy-to-test.yml` | Test deployment workflow |
| `.env.example` | Template with all variables | | `.env.example` | Template with all variables |
| `.env.local` | Dev container overrides (not in git) | | `.env.local` | Dev container overrides (not in git) |
| `.env.test` | Test environment overrides (not in git) | | `.env.test` | Test environment overrides (not in git) |
---
## Adding New Variables ## Adding New Variables
### 1. Update Zod Schema ### Checklist
1. [ ] **Update Zod Schema** - Edit `src/config/env.ts`
2. [ ] **Add to Gitea Secrets** - For prod/test environments
3. [ ] **Update Deployment Workflows** - `.gitea/workflows/*.yml`
4. [ ] **Update PM2 Config** - `ecosystem.config.cjs`
5. [ ] **Update .env.example** - Template for developers
6. [ ] **Update this document** - Add to appropriate section
### Step-by-Step
#### 1. Update Zod Schema
Edit `src/config/env.ts`: Edit `src/config/env.ts`:
```typescript ```typescript
const envSchema = z.object({ const envSchema = z.object({
// ... existing variables ... // ... existing variables ...
NEW_VARIABLE: z.string().min(1), newSection: z.object({
newVariable: z.string().min(1, 'NEW_VARIABLE is required'),
}),
}); });
// In loadEnvVars():
newSection: {
newVariable: process.env.NEW_VARIABLE,
},
``` ```
### 2. Add to Gitea Secrets #### 2. Add to Gitea Secrets
For prod/test environments:
1. Go to Gitea repository Settings > Secrets 1. Go to Gitea repository Settings > Secrets
2. Add `NEW_VARIABLE` with value 2. Add `NEW_VARIABLE` with production value
3. Add `NEW_VARIABLE_TEST` if test needs different value 3. Add `NEW_VARIABLE_TEST` if test needs different value
### 3. Update Deployment Workflows #### 3. Update Deployment Workflows
Edit `.gitea/workflows/deploy-to-prod.yml`: Edit `.gitea/workflows/deploy-to-prod.yml`:
@@ -145,7 +270,7 @@ env:
NEW_VARIABLE: ${{ secrets.NEW_VARIABLE_TEST }} NEW_VARIABLE: ${{ secrets.NEW_VARIABLE_TEST }}
``` ```
### 4. Update PM2 Config #### 4. Update PM2 Config
Edit `ecosystem.config.cjs`: Edit `ecosystem.config.cjs`:
@@ -161,31 +286,36 @@ module.exports = {
}; };
``` ```
### 5. Update Documentation ---
- Add to `.env.example`
- Update this document
- Document in relevant feature docs
## Security Best Practices ## Security Best Practices
### Secrets Management ### Do
- **NEVER** commit secrets to git
- Use Gitea Secrets for prod/test - Use Gitea Secrets for prod/test
- Use `.env.local` for dev (gitignored) - Use `.env.local` for dev (gitignored)
- Generate secrets with cryptographic randomness
- Rotate secrets regularly - Rotate secrets regularly
- Use environment-specific database users
### Do Not
- Commit secrets to git
- Use short or predictable secrets
- Share secrets across environments
- Log sensitive values
### Secret Generation ### Secret Generation
```bash ```bash
# Generate secure random secrets # Generate secure random secrets (64 hex characters)
node -e "console.log(require('crypto').randomBytes(32).toString('hex'))" node -e "console.log(require('crypto').randomBytes(32).toString('hex'))"
# Example output:
# a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4e5f6a1b2
``` ```
### Database Users ### Database Users by Environment
Each environment has its own PostgreSQL user:
| Environment | User | Database | | Environment | User | Database |
| ----------- | -------------------- | -------------------- | | ----------- | -------------------- | -------------------- |
@@ -193,44 +323,61 @@ Each environment has its own PostgreSQL user:
| Test | `flyer_crawler_test` | `flyer-crawler-test` | | Test | `flyer_crawler_test` | `flyer-crawler-test` |
| Development | `postgres` | `flyer_crawler_dev` | | Development | `postgres` | `flyer_crawler_dev` |
**Setup Commands** (as postgres superuser): ---
```sql
-- Production
CREATE DATABASE "flyer-crawler-prod";
CREATE USER flyer_crawler_prod WITH PASSWORD 'secure-password';
ALTER DATABASE "flyer-crawler-prod" OWNER TO flyer_crawler_prod;
\c "flyer-crawler-prod"
ALTER SCHEMA public OWNER TO flyer_crawler_prod;
GRANT CREATE, USAGE ON SCHEMA public TO flyer_crawler_prod;
CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
CREATE EXTENSION IF NOT EXISTS postgis;
CREATE EXTENSION IF NOT EXISTS pg_trgm;
-- Test (similar commands with _test suffix)
```
## Validation ## Validation
Environment variables are validated at startup via `src/config/env.ts`. If validation fails: Environment variables are validated at startup via `src/config/env.ts`.
1. Check the error message for missing/invalid variables ### Startup Validation
2. Verify `.env.local` (dev) or Gitea Secrets (prod/test)
3. Ensure values match schema requirements (min length, format, etc.) If validation fails, you will see:
```text
╔════════════════════════════════════════════════════════════════╗
║ CONFIGURATION ERROR - APPLICATION STARTUP ║
╚════════════════════════════════════════════════════════════════╝
The following environment variables are missing or invalid:
- database.host: DB_HOST is required
- auth.jwtSecret: JWT_SECRET must be at least 32 characters
Please check your .env file or environment configuration.
```
### Debugging Configuration
```bash
# Check what variables are set (dev container)
podman exec flyer-crawler-dev env | grep -E "^(DB_|REDIS_|JWT_|SENTRY_)"
# Test database connection
podman exec flyer-crawler-postgres psql -U postgres -d flyer_crawler_dev -c "SELECT 1;"
# Test Redis connection
podman exec flyer-crawler-redis redis-cli ping
```
---
## Troubleshooting ## Troubleshooting
### Variable Not Found ### Variable Not Found
``` ```text
Error: Missing required environment variable: JWT_SECRET Error: Missing required environment variable: JWT_SECRET
``` ```
**Solution**: Add the variable to your environment configuration. **Solutions**:
1. Check `.env.local` exists and has the variable
2. Verify variable name matches schema exactly
3. Restart the application after changes
### Invalid Value ### Invalid Value
``` ```text
Error: JWT_SECRET must be at least 32 characters Error: JWT_SECRET must be at least 32 characters
``` ```
@@ -240,32 +387,36 @@ Error: JWT_SECRET must be at least 32 characters
Check `NODE_ENV` is set correctly: Check `NODE_ENV` is set correctly:
- `development` - Local dev container | Value | Purpose |
- `test` - CI/CD test server | ------------- | ---------------------- |
- `production` - Production server | `development` | Local dev container |
| `test` | CI/CD test server |
| `staging` | Pre-production testing |
| `production` | Production server |
### Database Connection Issues ### Database Connection Issues
Verify database credentials:
```bash ```bash
# Development # Development
podman exec flyer-crawler-postgres psql -U postgres -d flyer_crawler_dev -c "SELECT 1;" podman exec flyer-crawler-postgres psql -U postgres -d flyer_crawler_dev -c "SELECT 1;"
# Production (via SSH) # If connection fails, check:
ssh root@projectium.com "psql -U flyer_crawler_prod -d flyer-crawler-prod -c 'SELECT 1;'" # 1. Container is running: podman ps
# 2. DB_HOST matches container network
# 3. DB_PASSWORD is correct
``` ```
## Reference ---
- **Validation Schema**: [src/config/env.ts](../../src/config/env.ts) ## Related Documentation
- **Template**: [.env.example](../../.env.example)
- **Deployment Workflows**: [.gitea/workflows/](../../.gitea/workflows/)
- **PM2 Config**: [ecosystem.config.cjs](../../ecosystem.config.cjs)
## See Also
- [QUICKSTART.md](QUICKSTART.md) - Quick setup guide - [QUICKSTART.md](QUICKSTART.md) - Quick setup guide
- [INSTALL.md](INSTALL.md) - Detailed installation - [INSTALL.md](INSTALL.md) - Detailed installation
- [DEV-CONTAINER.md](../development/DEV-CONTAINER.md) - Dev container setup
- [DEPLOYMENT.md](../operations/DEPLOYMENT.md) - Production deployment - [DEPLOYMENT.md](../operations/DEPLOYMENT.md) - Production deployment
- [AUTHENTICATION.md](../architecture/AUTHENTICATION.md) - OAuth setup - [AUTHENTICATION.md](../architecture/AUTHENTICATION.md) - OAuth setup
- [ADR-007](../adr/0007-configuration-and-secrets-management.md) - Configuration decisions
---
Last updated: January 2026

View File

@@ -1,203 +1,453 @@
# Installation Guide # Installation Guide
This guide covers setting up a local development environment for Flyer Crawler. Complete setup instructions for the Flyer Crawler local development environment.
---
## Quick Reference
| Setup Method | Best For | Time | Document Section |
| ----------------- | --------------------------- | ------ | --------------------------------------------------- |
| Quick Start | Already have Postgres/Redis | 5 min | [Quick Start](#quick-start) |
| Dev Container | Full production-like setup | 15 min | [Dev Container](#development-container-recommended) |
| Manual Containers | Learning the components | 20 min | [Podman Setup](#podman-setup-manual) |
---
## Prerequisites ## Prerequisites
- Node.js 20.x or later ### Required Software
- Access to a PostgreSQL database (local or remote)
- Redis instance (for session management) | Software | Minimum Version | Purpose | Download |
- Google Gemini API key | -------------- | --------------- | -------------------- | ----------------------------------------------- |
- Google Maps API key (for geocoding) | Node.js | 20.x | Runtime | [nodejs.org](https://nodejs.org/) |
| Podman Desktop | 4.x | Container management | [podman-desktop.io](https://podman-desktop.io/) |
| Git | 2.x | Version control | [git-scm.com](https://git-scm.com/) |
### Windows-Specific Requirements
| Requirement | Purpose | Setup Command |
| ----------- | ------------------------------ | ---------------------------------- |
| WSL 2 | Linux compatibility for Podman | `wsl --install` (admin PowerShell) |
### Verify Installation
```bash
# Check all prerequisites
node --version # Expected: v20.x or higher
podman --version # Expected: podman version 4.x or higher
git --version # Expected: git version 2.x or higher
wsl --list -v # Expected: Shows WSL 2 distro
```
---
## Quick Start ## Quick Start
If you already have PostgreSQL and Redis configured: If you already have PostgreSQL and Redis configured externally:
```bash ```bash
# Install dependencies # 1. Clone the repository
git clone https://gitea.projectium.com/flyer-crawler/flyer-crawler.git
cd flyer-crawler
# 2. Install dependencies
npm install npm install
# Run in development mode # 3. Create .env.local (see Environment section below)
# 4. Run in development mode
npm run dev npm run dev
``` ```
**Access Points**:
- Frontend: `http://localhost:5173`
- Backend API: `http://localhost:3001`
--- ---
## Development Environment with Podman (Recommended for Windows) ## Development Container (Recommended)
This approach uses Podman with an Ubuntu container for a consistent development environment. The dev container provides a complete, production-like environment.
### What's Included
| Service | Purpose | Port |
| ---------- | ------------------------ | ---------- |
| Node.js | API server, worker, Vite | 3001, 5173 |
| PostgreSQL | Database with PostGIS | 5432 |
| Redis | Cache and job queues | 6379 |
| NGINX | HTTPS reverse proxy | 443 |
| Bugsink | Error tracking | 8443 |
| Logstash | Log aggregation | - |
| PM2 | Process management | - |
### Setup Steps
#### Step 1: Initialize Podman
```bash
# Windows: Start Podman Desktop, or from terminal:
podman machine init
podman machine start
```
#### Step 2: Start Dev Container
```bash
# Start all services
podman-compose -f compose.dev.yml up -d
# View logs (optional)
podman-compose -f compose.dev.yml logs -f
```
**Expected Output**:
```text
[+] Running 3/3
- Container flyer-crawler-postgres Started
- Container flyer-crawler-redis Started
- Container flyer-crawler-dev Started
```
#### Step 3: Verify Services
```bash
# Check containers are running
podman ps
# Check PM2 processes
podman exec -it flyer-crawler-dev pm2 status
```
**Expected PM2 Status**:
```text
+---------------------------+--------+-------+
| name | status | cpu |
+---------------------------+--------+-------+
| flyer-crawler-api-dev | online | 0% |
| flyer-crawler-worker-dev | online | 0% |
| flyer-crawler-vite-dev | online | 0% |
+---------------------------+--------+-------+
```
#### Step 4: Access Application
| Service | URL | Notes |
| ----------- | ------------------------ | ---------------------------- |
| Frontend | `https://localhost` | NGINX proxies to Vite |
| Backend API | `http://localhost:3001` | Express server |
| Bugsink | `https://localhost:8443` | Login: admin@localhost/admin |
### SSL Certificate Setup (Optional but Recommended)
To eliminate browser security warnings:
**Windows**:
1. Double-click `certs/mkcert-ca.crt`
2. Click "Install Certificate..."
3. Select "Local Machine" > Next
4. Select "Place all certificates in the following store"
5. Browse > Select "Trusted Root Certification Authorities" > OK
6. Click Next > Finish
7. Restart browser
**Other Platforms**: See [`certs/README.md`](../../certs/README.md)
### Managing the Dev Container
| Action | Command |
| --------- | ------------------------------------------- |
| Start | `podman-compose -f compose.dev.yml up -d` |
| Stop | `podman-compose -f compose.dev.yml down` |
| View logs | `podman-compose -f compose.dev.yml logs -f` |
| Restart | `podman-compose -f compose.dev.yml restart` |
| Rebuild | `podman-compose -f compose.dev.yml build` |
---
## Podman Setup (Manual)
For understanding the individual components or custom configurations.
### Step 1: Install Prerequisites on Windows ### Step 1: Install Prerequisites on Windows
1. **Install WSL 2**: Podman on Windows relies on the Windows Subsystem for Linux. ```powershell
# Run in administrator PowerShell
wsl --install
```
```powershell Restart computer after WSL installation.
wsl --install
```
Run this in an administrator PowerShell. ### Step 2: Initialize Podman
2. **Install Podman Desktop**: Download and install [Podman Desktop for Windows](https://podman-desktop.io/). 1. Launch **Podman Desktop**
2. Follow the setup wizard to initialize Podman machine
3. Start the Podman machine
### Step 2: Set Up Podman Or from terminal:
1. **Initialize Podman**: Launch Podman Desktop. It will automatically set up its WSL 2 machine.
2. **Start Podman**: Ensure the Podman machine is running from the Podman Desktop interface.
### Step 3: Set Up the Ubuntu Container
1. **Pull Ubuntu Image**:
```bash
podman pull ubuntu:latest
```
2. **Create a Podman Volume** (persists node_modules between container restarts):
```bash
podman volume create node_modules_cache
```
3. **Run the Ubuntu Container**:
Open a terminal in your project's root directory and run:
```bash
podman run -it -p 3001:3001 -p 5173:5173 --name flyer-dev \
-v "$(pwd):/app" \
-v "node_modules_cache:/app/node_modules" \
ubuntu:latest
```
| Flag | Purpose |
| ------------------------------------------- | ------------------------------------------------ |
| `-p 3001:3001` | Forwards the backend server port |
| `-p 5173:5173` | Forwards the Vite frontend server port |
| `--name flyer-dev` | Names the container for easy reference |
| `-v "...:/app"` | Mounts your project directory into the container |
| `-v "node_modules_cache:/app/node_modules"` | Mounts the named volume for node_modules |
### Step 4: Configure the Ubuntu Environment
You are now inside the Ubuntu container's shell.
1. **Update Package Lists**:
```bash
apt-get update
```
2. **Install Dependencies**:
```bash
apt-get install -y curl git
curl -sL https://deb.nodesource.com/setup_20.x | bash -
apt-get install -y nodejs
```
3. **Navigate to Project Directory**:
```bash
cd /app
```
4. **Install Project Dependencies**:
```bash
npm install
```
### Step 5: Run the Development Server
```bash ```bash
podman machine init
podman machine start
```
### Step 3: Create Podman Network
```bash
podman network create flyer-crawler-net
```
### Step 4: Create PostgreSQL Container
```bash
podman run -d \
--name flyer-crawler-postgres \
--network flyer-crawler-net \
-e POSTGRES_USER=postgres \
-e POSTGRES_PASSWORD=postgres \
-e POSTGRES_DB=flyer_crawler_dev \
-p 5432:5432 \
-v flyer-crawler-pgdata:/var/lib/postgresql/data \
docker.io/postgis/postgis:15-3.3
```
### Step 5: Create Redis Container
```bash
podman run -d \
--name flyer-crawler-redis \
--network flyer-crawler-net \
-p 6379:6379 \
-v flyer-crawler-redis:/data \
docker.io/library/redis:alpine
```
### Step 6: Initialize Database
```bash
# Wait for PostgreSQL to be ready
podman exec flyer-crawler-postgres pg_isready -U postgres
# Install required extensions
podman exec flyer-crawler-postgres psql -U postgres -d flyer_crawler_dev -c "
CREATE EXTENSION IF NOT EXISTS postgis;
CREATE EXTENSION IF NOT EXISTS pg_trgm;
CREATE EXTENSION IF NOT EXISTS \"uuid-ossp\";
"
# Apply schema
podman exec -i flyer-crawler-postgres psql -U postgres -d flyer_crawler_dev < sql/master_schema_rollup.sql
```
### Step 7: Create Node.js Container
```bash
# Create volume for node_modules
podman volume create node_modules_cache
# Run Ubuntu container with project mounted
podman run -it \
--name flyer-dev \
--network flyer-crawler-net \
-p 3001:3001 \
-p 5173:5173 \
-v "$(pwd):/app" \
-v "node_modules_cache:/app/node_modules" \
ubuntu:latest
```
### Step 8: Configure Container Environment
Inside the container:
```bash
# Update and install dependencies
apt-get update
apt-get install -y curl git
# Install Node.js 20
curl -sL https://deb.nodesource.com/setup_20.x | bash -
apt-get install -y nodejs
# Navigate to project and install
cd /app
npm install
# Start development server
npm run dev npm run dev
``` ```
### Step 6: Access the Application ### Container Management Commands
- **Frontend**: http://localhost:5173 | Action | Command |
- **Backend API**: http://localhost:3001 | -------------- | ------------------------------ |
| Stop container | Press `Ctrl+C`, then `exit` |
### Dev Container with HTTPS (Full Stack) | Restart | `podman start -a -i flyer-dev` |
| Remove | `podman rm flyer-dev` |
When using the full dev container stack with NGINX (via `compose.dev.yml`), access the application over HTTPS: | List running | `podman ps` |
| List all | `podman ps -a` |
- **Frontend**: https://localhost or https://127.0.0.1
- **Backend API**: http://localhost:3001
**SSL Certificate Notes:**
- The dev container uses self-signed certificates generated by mkcert
- Both `localhost` and `127.0.0.1` are valid hostnames (certificate includes both as SANs)
- If images fail to load with SSL errors, see [FLYER-URL-CONFIGURATION.md](../FLYER-URL-CONFIGURATION.md#ssl-certificate-configuration-dev-container)
**Eliminate SSL Warnings (Recommended):**
To avoid browser security warnings for self-signed certificates, install the mkcert CA certificate on your system. The CA certificate is located at `certs/mkcert-ca.crt` in the project root.
See [`certs/README.md`](../../certs/README.md) for platform-specific installation instructions (Windows, macOS, Linux, Firefox).
After installation:
- Your browser will trust all mkcert certificates without warnings
- Both `https://localhost/` and `https://127.0.0.1/` will work without SSL errors
- Flyer images will load without `ERR_CERT_AUTHORITY_INVALID` errors
### Managing the Container
| Action | Command |
| --------------------- | -------------------------------- |
| Stop the container | Press `Ctrl+C`, then type `exit` |
| Restart the container | `podman start -a -i flyer-dev` |
| Remove the container | `podman rm flyer-dev` |
--- ---
## Environment Variables ## Environment Configuration
This project is configured to run in a CI/CD environment and does not use `.env` files. All configuration must be provided as environment variables. ### Create .env.local
For local development, you can export these in your shell or use your IDE's environment configuration: Create `.env.local` in the project root with your configuration:
| Variable | Description | ```bash
| --------------------------- | ------------------------------------- | # Database (adjust host based on your setup)
| `DB_HOST` | PostgreSQL server hostname | DB_HOST=localhost # Use 'postgres' if inside dev container
| `DB_USER` | PostgreSQL username | DB_PORT=5432
| `DB_PASSWORD` | PostgreSQL password | DB_USER=postgres
| `DB_DATABASE_PROD` | Production database name | DB_PASSWORD=postgres
| `JWT_SECRET` | Secret string for signing auth tokens | DB_NAME=flyer_crawler_dev
| `VITE_GOOGLE_GENAI_API_KEY` | Google Gemini API key |
| `GOOGLE_MAPS_API_KEY` | Google Maps Geocoding API key | # Redis (adjust host based on your setup)
| `REDIS_PASSWORD_PROD` | Production Redis password | REDIS_URL=redis://localhost:6379 # Use 'redis://redis:6379' inside container
| `REDIS_PASSWORD_TEST` | Test Redis password |
# Application
NODE_ENV=development
PORT=3001
FRONTEND_URL=http://localhost:5173
# Authentication (generate secure values)
JWT_SECRET=your-secret-at-least-32-characters-long
# AI Services
GEMINI_API_KEY=your-google-gemini-api-key
GOOGLE_MAPS_API_KEY=your-google-maps-api-key # Optional
```
**Generate Secure Secrets**:
```bash
node -e "console.log(require('crypto').randomBytes(32).toString('hex'))"
```
### Environment Differences
| Variable | Host Development | Inside Dev Container |
| ----------- | ------------------------ | -------------------- |
| `DB_HOST` | `localhost` | `postgres` |
| `REDIS_URL` | `redis://localhost:6379` | `redis://redis:6379` |
See [ENVIRONMENT.md](ENVIRONMENT.md) for complete variable reference.
--- ---
## Seeding Development Data ## Seeding Development Data
To create initial test accounts (`admin@example.com` and `user@example.com`) and sample data: Create test accounts and sample data:
```bash ```bash
npm run seed npm run seed
``` ```
The seed script performs the following actions: ### What the Seed Script Does
1. Rebuilds the database schema from `sql/master_schema_rollup.sql` 1. Rebuilds database schema from `sql/master_schema_rollup.sql`
2. Creates test user accounts (admin and regular user) 2. Creates test user accounts:
3. Copies test flyer images from `src/tests/assets/` to `public/flyer-images/` - `admin@example.com` (admin user)
4. Creates a sample flyer with items linked to the test images - `user@example.com` (regular user)
5. Seeds watched items and a shopping list for the test user 3. Copies test flyer images to `public/flyer-images/`
4. Creates sample flyer with items
5. Seeds watched items and shopping list
**Test Images**: The seed script copies `test-flyer-image.jpg` and `test-flyer-icon.png` to the `public/flyer-images/` directory, which is served by NGINX at `/flyer-images/`. ### Test Images
After running, you may need to restart your IDE's TypeScript server to pick up any generated types. The seed script copies these files from `src/tests/assets/`:
- `test-flyer-image.jpg`
- `test-flyer-icon.png`
Images are served by NGINX at `/flyer-images/`.
---
## Verification Checklist
After installation, verify everything works:
- [ ] **Containers running**: `podman ps` shows postgres and redis
- [ ] **Database accessible**: `podman exec flyer-crawler-postgres psql -U postgres -c "SELECT 1;"`
- [ ] **Frontend loads**: Open `http://localhost:5173` (or `https://localhost` for dev container)
- [ ] **API responds**: `curl http://localhost:3001/health`
- [ ] **Tests pass**: `npm run test:unit` (or in container: `podman exec -it flyer-crawler-dev npm run test:unit`)
- [ ] **Type check passes**: `npm run type-check`
---
## Troubleshooting
### Podman Machine Won't Start
```bash
# Reset Podman machine
podman machine rm
podman machine init
podman machine start
```
### Port Already in Use
```bash
# Find process using port
netstat -ano | findstr :5432
# Option: Use different port
podman run -d --name flyer-crawler-postgres -p 5433:5432 ...
# Then set DB_PORT=5433 in .env.local
```
### Database Extensions Missing
```bash
podman exec flyer-crawler-postgres psql -U postgres -d flyer_crawler_dev -c "
CREATE EXTENSION IF NOT EXISTS postgis;
CREATE EXTENSION IF NOT EXISTS pg_trgm;
CREATE EXTENSION IF NOT EXISTS \"uuid-ossp\";
"
```
### Permission Denied on Windows Paths
Use `MSYS_NO_PATHCONV=1` prefix:
```bash
MSYS_NO_PATHCONV=1 podman exec flyer-crawler-dev /path/to/script.sh
```
### Tests Fail with Timezone Errors
Tests must run in the dev container, not on Windows host:
```bash
# CORRECT
podman exec -it flyer-crawler-dev npm test
# INCORRECT (may fail with TZ errors)
npm test
```
--- ---
## Next Steps ## Next Steps
- [Database Setup](DATABASE.md) - Set up PostgreSQL with required extensions | Goal | Document |
- [Authentication Setup](AUTHENTICATION.md) - Configure OAuth providers | --------------------- | ------------------------------------------------------ |
- [Deployment Guide](DEPLOYMENT.md) - Deploy to production | Quick setup guide | [QUICKSTART.md](QUICKSTART.md) |
| Environment variables | [ENVIRONMENT.md](ENVIRONMENT.md) |
| Database schema | [DATABASE.md](../architecture/DATABASE.md) |
| Authentication setup | [AUTHENTICATION.md](../architecture/AUTHENTICATION.md) |
| Dev container details | [DEV-CONTAINER.md](../development/DEV-CONTAINER.md) |
| Deployment | [DEPLOYMENT.md](../operations/DEPLOYMENT.md) |
---
Last updated: January 2026

View File

@@ -2,13 +2,38 @@
Get Flyer Crawler running in 5 minutes. Get Flyer Crawler running in 5 minutes.
## Prerequisites ---
- **Windows 10/11** with WSL 2 ## Prerequisites Checklist
- **Podman Desktop** installed
- **Node.js 20+** installed
## 1. Start Containers (1 minute) Before starting, verify you have:
- [ ] **Windows 10/11** with WSL 2 enabled
- [ ] **Podman Desktop** installed ([download](https://podman-desktop.io/))
- [ ] **Node.js 20+** installed
- [ ] **Git** for cloning the repository
**Verify Prerequisites**:
```bash
# Check Podman
podman --version
# Expected: podman version 4.x or higher
# Check Node.js
node --version
# Expected: v20.x or higher
# Check WSL
wsl --list --verbose
# Expected: Shows WSL 2 distro
```
---
## Quick Setup (5 Steps)
### Step 1: Start Containers (1 minute)
```bash ```bash
# Start PostgreSQL and Redis # Start PostgreSQL and Redis
@@ -27,11 +52,18 @@ podman run -d --name flyer-crawler-redis \
docker.io/library/redis:alpine docker.io/library/redis:alpine
``` ```
## 2. Initialize Database (2 minutes) **Expected Output**:
```text
# Container IDs displayed, no errors
```
### Step 2: Initialize Database (2 minutes)
```bash ```bash
# Wait for PostgreSQL to be ready # Wait for PostgreSQL to be ready
podman exec flyer-crawler-postgres pg_isready -U postgres podman exec flyer-crawler-postgres pg_isready -U postgres
# Expected: localhost:5432 - accepting connections
# Install extensions # Install extensions
podman exec flyer-crawler-postgres psql -U postgres -d flyer_crawler_dev \ podman exec flyer-crawler-postgres psql -U postgres -d flyer_crawler_dev \
@@ -41,7 +73,17 @@ podman exec flyer-crawler-postgres psql -U postgres -d flyer_crawler_dev \
podman exec -i flyer-crawler-postgres psql -U postgres -d flyer_crawler_dev < sql/master_schema_rollup.sql podman exec -i flyer-crawler-postgres psql -U postgres -d flyer_crawler_dev < sql/master_schema_rollup.sql
``` ```
## 3. Configure Environment (1 minute) **Expected Output**:
```text
CREATE EXTENSION
CREATE EXTENSION
CREATE EXTENSION
CREATE TABLE
... (many tables created)
```
### Step 3: Configure Environment (1 minute)
Create `.env.local` in the project root: Create `.env.local` in the project root:
@@ -61,16 +103,22 @@ NODE_ENV=development
PORT=3001 PORT=3001
FRONTEND_URL=http://localhost:5173 FRONTEND_URL=http://localhost:5173
# Secrets (generate your own) # Secrets (generate your own - see command below)
JWT_SECRET=your-dev-jwt-secret-at-least-32-chars-long JWT_SECRET=your-dev-jwt-secret-at-least-32-chars-long
SESSION_SECRET=your-dev-session-secret-at-least-32-chars-long SESSION_SECRET=your-dev-session-secret-at-least-32-chars-long
# AI Services (get your own keys) # AI Services (get your own keys)
VITE_GOOGLE_GENAI_API_KEY=your-google-genai-api-key GEMINI_API_KEY=your-google-gemini-api-key
GOOGLE_MAPS_API_KEY=your-google-maps-api-key GOOGLE_MAPS_API_KEY=your-google-maps-api-key
``` ```
## 4. Install & Run (1 minute) **Generate Secure Secrets**:
```bash
node -e "console.log(require('crypto').randomBytes(32).toString('hex'))"
```
### Step 4: Install and Run (1 minute)
```bash ```bash
# Install dependencies (first time only) # Install dependencies (first time only)
@@ -80,35 +128,61 @@ npm install
npm run dev npm run dev
``` ```
## 5. Access Application **Expected Output**:
- **Frontend**: http://localhost:5173 ```text
- **Backend API**: http://localhost:3001 > flyer-crawler@x.x.x dev
- **Health Check**: http://localhost:3001/health > concurrently ...
### Dev Container (HTTPS) [API] Server listening on port 3001
[Vite] VITE ready at http://localhost:5173
```
When using the full dev container with NGINX, access via HTTPS: ### Step 5: Verify Installation
- **Frontend**: https://localhost or https://127.0.0.1 | Check | URL/Command | Expected Result |
- **Backend API**: http://localhost:3001 | ----------- | ------------------------------ | ----------------------------------- |
- **Bugsink**: `https://localhost:8443` (error tracking) | Frontend | `http://localhost:5173` | Flyer Crawler app loads |
| Backend API | `http://localhost:3001/health` | `{ "status": "ok", ... }` |
| Database | `podman exec ... psql -c ...` | `SELECT version()` returns Postgres |
| Containers | `podman ps` | Shows postgres and redis running |
**Note:** The dev container accepts both `localhost` and `127.0.0.1` for HTTPS connections. The self-signed certificate is valid for both hostnames. ---
**SSL Certificate Warnings:** To eliminate browser security warnings for self-signed certificates, install the mkcert CA certificate. See [`certs/README.md`](../../certs/README.md) for platform-specific installation instructions. This is optional but recommended for a better development experience. ## Full Dev Container (Recommended)
### Dev Container Architecture For a production-like environment with NGINX, Bugsink error tracking, and PM2 process management:
The dev container uses PM2 for process management, matching production (ADR-014): ### Starting the Dev Container
| Process | Description | Port | ```bash
| -------------------------- | ------------------------ | ---- | # Start all services
| `flyer-crawler-api-dev` | API server (tsx watch) | 3001 | podman-compose -f compose.dev.yml up -d
| `flyer-crawler-worker-dev` | Background job worker | - |
| `flyer-crawler-vite-dev` | Vite frontend dev server | 5173 |
**PM2 Commands** (run inside container): # View logs
podman-compose -f compose.dev.yml logs -f
```
### Access Points
| Service | URL | Notes |
| ----------- | ------------------------ | ---------------------------- |
| Frontend | `https://localhost` | NGINX proxy to Vite |
| Backend API | `http://localhost:3001` | Express server |
| Bugsink | `https://localhost:8443` | Error tracking (admin/admin) |
| PostgreSQL | `localhost:5432` | Database |
| Redis | `localhost:6379` | Cache |
**SSL Certificate Setup (Recommended)**:
To eliminate browser security warnings, install the mkcert CA certificate:
```bash
# Windows: Double-click certs/mkcert-ca.crt and install to Trusted Root CAs
# See certs/README.md for detailed instructions per platform
```
### PM2 Commands
```bash ```bash
# View process status # View process status
@@ -124,63 +198,152 @@ podman exec -it flyer-crawler-dev pm2 restart all
podman exec -it flyer-crawler-dev pm2 restart flyer-crawler-api-dev podman exec -it flyer-crawler-dev pm2 restart flyer-crawler-api-dev
``` ```
## Verify Installation ### Dev Container Processes
| Process | Description | Port |
| -------------------------- | ------------------------ | ---- |
| `flyer-crawler-api-dev` | API server (tsx watch) | 3001 |
| `flyer-crawler-worker-dev` | Background job worker | - |
| `flyer-crawler-vite-dev` | Vite frontend dev server | 5173 |
---
## Verification Commands
Run these to confirm everything is working:
```bash ```bash
# Check containers are running # Check containers are running
podman ps podman ps
# Expected: flyer-crawler-postgres and flyer-crawler-redis both running
# Test database connection # Test database connection
podman exec flyer-crawler-postgres psql -U postgres -d flyer_crawler_dev -c "SELECT version();" podman exec flyer-crawler-postgres psql -U postgres -d flyer_crawler_dev -c "SELECT version();"
# Expected: PostgreSQL 15.x with PostGIS
# Run tests (in dev container) # Run tests (in dev container)
podman exec -it flyer-crawler-dev npm run test:unit podman exec -it flyer-crawler-dev npm run test:unit
# Expected: All tests pass
# Run type check
podman exec -it flyer-crawler-dev npm run type-check
# Expected: No type errors
``` ```
## Common Issues ---
## Common Issues and Solutions
### "Unable to connect to Podman socket" ### "Unable to connect to Podman socket"
**Cause**: Podman machine not running
**Solution**:
```bash ```bash
podman machine start podman machine start
``` ```
### "Connection refused" to PostgreSQL ### "Connection refused" to PostgreSQL
Wait a few seconds for PostgreSQL to initialize: **Cause**: PostgreSQL still initializing
**Solution**:
```bash ```bash
# Wait for PostgreSQL to be ready
podman exec flyer-crawler-postgres pg_isready -U postgres podman exec flyer-crawler-postgres pg_isready -U postgres
# Retry after "accepting connections" message
``` ```
### Port 5432 or 6379 already in use ### Port 5432 or 6379 already in use
Stop conflicting services or change port mappings: **Cause**: Another service using the port
**Solution**:
```bash ```bash
# Use different host port # Option 1: Stop conflicting service
# Option 2: Use different host port
podman run -d --name flyer-crawler-postgres -p 5433:5432 ... podman run -d --name flyer-crawler-postgres -p 5433:5432 ...
# Then update DB_PORT=5433 in .env.local
``` ```
Then update `DB_PORT=5433` in `.env.local`. ### "JWT_SECRET must be at least 32 characters"
**Cause**: Secret too short in .env.local
**Solution**: Generate a longer secret:
```bash
node -e "console.log(require('crypto').randomBytes(32).toString('hex'))"
```
### Tests fail with "TZ environment variable" errors
**Cause**: Timezone setting interfering with Node.js async hooks
**Solution**: Tests must run in dev container (not Windows host):
```bash
# CORRECT - run in container
podman exec -it flyer-crawler-dev npm test
# INCORRECT - do not run on Windows host
npm test
```
---
## Next Steps ## Next Steps
- **Read the docs**: [docs/README.md](../README.md) | Goal | Document |
- **Understand the architecture**: [docs/architecture/DATABASE.md](../architecture/DATABASE.md) | ----------------------- | ----------------------------------------------------- |
- **Learn testing**: [docs/development/TESTING.md](../development/TESTING.md) | Understand the codebase | [Architecture Overview](../architecture/OVERVIEW.md) |
- **Explore ADRs**: [docs/adr/index.md](../adr/index.md) | Configure environment | [Environment Variables](ENVIRONMENT.md) |
- **Contributing**: [CONTRIBUTING.md](../../CONTRIBUTING.md) | Set up MCP tools | [MCP Configuration](../tools/MCP-CONFIGURATION.md) |
| Learn testing | [Testing Guide](../development/TESTING.md) |
| Understand DB schema | [Database Documentation](../architecture/DATABASE.md) |
| Read ADRs | [ADR Index](../adr/index.md) |
| Full installation guide | [Installation Guide](INSTALL.md) |
## Development Workflow ---
## Daily Development Workflow
```bash ```bash
# Daily workflow # 1. Start containers
podman start flyer-crawler-postgres flyer-crawler-redis podman start flyer-crawler-postgres flyer-crawler-redis
# 2. Start dev server
npm run dev npm run dev
# ... make changes ...
# 3. Make changes and test
npm test npm test
# 4. Type check before commit
npm run type-check
# 5. Commit changes
git commit git commit
``` ```
For detailed setup instructions, see [INSTALL.md](INSTALL.md). **For dev container users**:
```bash
# 1. Start dev container
podman-compose -f compose.dev.yml up -d
# 2. View logs
podman exec -it flyer-crawler-dev pm2 logs
# 3. Run tests
podman exec -it flyer-crawler-dev npm test
# 4. Stop when done
podman-compose -f compose.dev.yml down
```
---
Last updated: January 2026

View File

@@ -2,8 +2,54 @@
This guide covers the manual installation of Flyer Crawler and its dependencies on a bare-metal Ubuntu server (e.g., a colocation server). This is the definitive reference for setting up a production environment without containers. This guide covers the manual installation of Flyer Crawler and its dependencies on a bare-metal Ubuntu server (e.g., a colocation server). This is the definitive reference for setting up a production environment without containers.
**Last verified**: 2026-01-28
**Target Environment**: Ubuntu 22.04 LTS (or newer) **Target Environment**: Ubuntu 22.04 LTS (or newer)
**Related documentation**:
- [ADR-014: Containerization and Deployment Strategy](../adr/0014-containerization-and-deployment-strategy.md)
- [ADR-015: Error Tracking and Observability](../adr/0015-error-tracking-and-observability.md)
- [ADR-050: PostgreSQL Function Observability](../adr/0050-postgresql-function-observability.md)
- [Deployment Guide](DEPLOYMENT.md)
- [Monitoring Guide](MONITORING.md)
---
## Quick Reference
### Installation Time Estimates
| Component | Estimated Time | Notes |
| ----------- | --------------- | ----------------------------- |
| PostgreSQL | 10-15 minutes | Including PostGIS extensions |
| Redis | 5 minutes | Quick install |
| Node.js | 5 minutes | Via NodeSource repository |
| Application | 15-20 minutes | Clone, install, build |
| PM2 | 5 minutes | Global install + config |
| NGINX | 10-15 minutes | Including SSL via Certbot |
| Bugsink | 20-30 minutes | Python venv, systemd services |
| Logstash | 15-20 minutes | Including pipeline config |
| **Total** | **~90 minutes** | For complete fresh install |
### Post-Installation Verification
After completing setup, verify all services:
```bash
# Check all services are running
systemctl status postgresql nginx redis-server gunicorn-bugsink snappea logstash
# Verify application health
curl -s https://flyer-crawler.projectium.com/api/health/ready | jq .
# Check PM2 processes
pm2 list
# Verify Bugsink is accessible
curl -s https://bugsink.projectium.com/accounts/login/ | head -5
```
--- ---
## Server Access Model ## Server Access Model

View File

@@ -2,6 +2,41 @@
This guide covers deploying Flyer Crawler to a production server. This guide covers deploying Flyer Crawler to a production server.
**Last verified**: 2026-01-28
**Related documentation**:
- [ADR-014: Containerization and Deployment Strategy](../adr/0014-containerization-and-deployment-strategy.md)
- [ADR-015: Error Tracking and Observability](../adr/0015-error-tracking-and-observability.md)
- [Bare-Metal Setup Guide](BARE-METAL-SETUP.md)
- [Monitoring Guide](MONITORING.md)
---
## Quick Reference
### Command Reference Table
| Task | Command |
| -------------------- | ----------------------------------------------------------------------- |
| Deploy to production | Gitea Actions workflow (manual trigger) |
| Deploy to test | Automatic on push to `main` |
| Check PM2 status | `pm2 list` |
| View logs | `pm2 logs flyer-crawler-api --lines 100` |
| Restart all | `pm2 restart all` |
| Check NGINX | `sudo nginx -t && sudo systemctl status nginx` |
| Check health | `curl -s https://flyer-crawler.projectium.com/api/health/ready \| jq .` |
### Deployment URLs
| Environment | URL | API Port |
| ------------- | ------------------------------------------- | -------- |
| Production | `https://flyer-crawler.projectium.com` | 3001 |
| Test | `https://flyer-crawler-test.projectium.com` | 3002 |
| Dev Container | `https://localhost` | 3001 |
---
## Server Access Model ## Server Access Model
**Important**: Claude Code (and AI tools) have **READ-ONLY** access to production/test servers. The deployment workflow is: **Important**: Claude Code (and AI tools) have **READ-ONLY** access to production/test servers. The deployment workflow is:
@@ -24,12 +59,24 @@ When troubleshooting deployment issues:
## Prerequisites ## Prerequisites
- Ubuntu server (22.04 LTS recommended) | Component | Version | Purpose |
- PostgreSQL 14+ with PostGIS extension | ---------- | --------- | ------------------------------- |
- Redis | Ubuntu | 22.04 LTS | Operating system |
- Node.js 20.x | PostgreSQL | 14+ | Database with PostGIS extension |
- NGINX (reverse proxy) | Redis | 6+ | Caching and job queues |
- PM2 (process manager) | Node.js | 20.x LTS | Application runtime |
| NGINX | 1.18+ | Reverse proxy and static files |
| PM2 | Latest | Process manager |
**Verify prerequisites**:
```bash
node --version # Should be v20.x.x
psql --version # Should be 14+
redis-cli ping # Should return PONG
nginx -v # Should be 1.18+
pm2 --version # Any recent version
```
## Dev Container Parity (ADR-014) ## Dev Container Parity (ADR-014)
@@ -210,7 +257,7 @@ types {
**Option 2**: Edit `/etc/nginx/mime.types` globally: **Option 2**: Edit `/etc/nginx/mime.types` globally:
``` ```text
# Change this line: # Change this line:
application/javascript js; application/javascript js;
@@ -341,9 +388,78 @@ The Sentry SDK v10+ enforces HTTPS-only DSNs by default. Since Bugsink runs loca
--- ---
## Deployment Troubleshooting
### Decision Tree: Deployment Issues
```text
Deployment failed?
|
+-- Build step failed?
| |
| +-- TypeScript errors --> Fix type issues, run `npm run type-check`
| +-- Missing dependencies --> Run `npm ci`
| +-- Out of memory --> Increase Node heap size
|
+-- Tests failed?
| |
| +-- Database connection --> Check DB_HOST, credentials
| +-- Redis connection --> Check REDIS_URL
| +-- Test isolation --> Check for race conditions
|
+-- SSH/Deploy failed?
|
+-- Permission denied --> Check SSH keys in Gitea secrets
+-- Host unreachable --> Check firewall, VPN
+-- PM2 error --> Check PM2 logs on server
```
### Common Deployment Issues
| Symptom | Diagnosis | Solution |
| ------------------------------------ | ----------------------- | ------------------------------------------------ |
| "Connection refused" on health check | API not started | Check `pm2 logs flyer-crawler-api` |
| 502 Bad Gateway | NGINX cannot reach API | Verify API port (3001), restart PM2 |
| CSS/JS not loading | Build artifacts missing | Re-run `npm run build`, check NGINX static paths |
| Database migrations failed | Schema mismatch | Run migrations manually, check DB connectivity |
| "ENOSPC" error | Disk full | Clear old logs: `pm2 flush`, clean npm cache |
| SSL certificate error | Cert expired/missing | Run `certbot renew`, check NGINX config |
### Post-Deployment Verification Checklist
After every deployment, verify:
- [ ] Health check passes: `curl -s https://flyer-crawler.projectium.com/api/health/ready`
- [ ] PM2 processes running: `pm2 list` shows `online` status
- [ ] No recent errors: Check Bugsink for new issues
- [ ] Frontend loads: Browser shows login page
- [ ] API responds: `curl https://flyer-crawler.projectium.com/api/health/ping`
### Rollback Procedure
If deployment causes issues:
```bash
# 1. Check current release
cd /var/www/flyer-crawler.projectium.com
git log --oneline -5
# 2. Revert to previous commit
git checkout HEAD~1
# 3. Rebuild and restart
npm ci && npm run build
pm2 restart all
# 4. Verify health
curl -s http://localhost:3001/api/health/ready | jq .
```
---
## Related Documentation ## Related Documentation
- [Database Setup](DATABASE.md) - PostgreSQL and PostGIS configuration - [Database Setup](../architecture/DATABASE.md) - PostgreSQL and PostGIS configuration
- [Authentication Setup](AUTHENTICATION.md) - OAuth provider configuration - [Monitoring Guide](MONITORING.md) - Health checks and error tracking
- [Installation Guide](INSTALL.md) - Local development setup - [Logstash Quick Reference](LOGSTASH-QUICK-REF.md) - Log aggregation
- [Bare-Metal Server Setup](docs/BARE-METAL-SETUP.md) - Manual server installation guide - [Bare-Metal Server Setup](BARE-METAL-SETUP.md) - Manual server installation guide

View File

@@ -2,10 +2,47 @@
Aggregates logs from PostgreSQL, PM2, Redis, NGINX; forwards errors to Bugsink. Aggregates logs from PostgreSQL, PM2, Redis, NGINX; forwards errors to Bugsink.
**Last verified**: 2026-01-28
**Related documentation**:
- [ADR-050: PostgreSQL Function Observability](../adr/0050-postgresql-function-observability.md)
- [ADR-015: Error Tracking and Observability](../adr/0015-error-tracking-and-observability.md)
- [Monitoring Guide](MONITORING.md)
- [Logstash Troubleshooting Runbook](LOGSTASH-TROUBLESHOOTING.md)
---
## Quick Reference
### Bugsink Project Routing
| Source Type | Environment | Bugsink Project | Project ID |
| -------------- | ----------- | -------------------- | ---------- |
| PM2 API/Worker | Dev | Backend API (Dev) | 1 |
| PostgreSQL | Dev | Backend API (Dev) | 1 |
| Frontend JS | Dev | Frontend (Dev) | 2 |
| Redis/NGINX | Dev | Infrastructure (Dev) | 4 |
| PM2 API/Worker | Production | Backend API (Prod) | 1 |
| PostgreSQL | Production | Backend API (Prod) | 1 |
| PM2 API/Worker | Test | Backend API (Test) | 3 |
### Key DSN Keys (Dev Container)
| Project | DSN Key |
| -------------------- | ---------------------------------- |
| Backend API (Dev) | `cea01396c56246adb5878fa5ee6b1d22` |
| Frontend (Dev) | `d92663cb73cf4145b677b84029e4b762` |
| Infrastructure (Dev) | `14e8791da3d347fa98073261b596cab9` |
---
## Configuration ## Configuration
**Primary config**: `/etc/logstash/conf.d/bugsink.conf` **Primary config**: `/etc/logstash/conf.d/bugsink.conf`
**Dev container config**: `docker/logstash/bugsink.conf`
### Related Files ### Related Files
| Path | Purpose | | Path | Purpose |
@@ -89,6 +126,34 @@ MSYS_NO_PATHCONV=1 podman exec flyer-crawler-dev ls -la /var/log/redis/
## Troubleshooting ## Troubleshooting
### Decision Tree: Logs Not Appearing in Bugsink
```text
Errors not showing in Bugsink?
|
+-- Logstash running?
| |
| +-- No --> systemctl start logstash
| +-- Yes --> Check pipeline stats
| |
| +-- Events in = 0?
| | |
| | +-- Log files exist? --> ls /var/log/pm2/*.log
| | +-- Permissions OK? --> groups logstash
| |
| +-- Events filtered = high?
| | |
| | +-- Grok failures --> Check log format matches pattern
| |
| +-- Events out but no Bugsink?
| |
| +-- 403 error --> Wrong DSN key
| +-- 500 error --> Invalid event format (check sentry_level)
| +-- Connection refused --> Bugsink not running
```
### Common Issues Table
| Issue | Check | Solution | | Issue | Check | Solution |
| --------------------- | ---------------- | ---------------------------------------------------------------------------------------------- | | --------------------- | ---------------- | ---------------------------------------------------------------------------------------------- |
| No Bugsink errors | Logstash running | `systemctl status logstash` | | No Bugsink errors | Logstash running | `systemctl status logstash` |
@@ -103,6 +168,25 @@ MSYS_NO_PATHCONV=1 podman exec flyer-crawler-dev ls -la /var/log/redis/
| High disk usage | Log rotation | Verify `/etc/logrotate.d/logstash` configured | | High disk usage | Log rotation | Verify `/etc/logrotate.d/logstash` configured |
| varchar(7) error | Level validation | Add Ruby filter to validate/normalize `sentry_level` before output | | varchar(7) error | Level validation | Add Ruby filter to validate/normalize `sentry_level` before output |
### Expected Output Examples
**Successful Logstash pipeline stats**:
```json
{
"in": 1523,
"out": 1520,
"filtered": 1520,
"queue_push_duration_in_millis": 45
}
```
**Healthy Bugsink HTTP response**:
```json
{ "id": "a1b2c3d4e5f6..." }
```
## Related Documentation ## Related Documentation
- **Dev Container Guide**: [DEV-CONTAINER.md](../development/DEV-CONTAINER.md) - PM2 and log aggregation in dev - **Dev Container Guide**: [DEV-CONTAINER.md](../development/DEV-CONTAINER.md) - PM2 and log aggregation in dev

View File

@@ -2,6 +2,16 @@
This runbook provides step-by-step diagnostics and solutions for common Logstash issues in the PostgreSQL observability pipeline (ADR-050). This runbook provides step-by-step diagnostics and solutions for common Logstash issues in the PostgreSQL observability pipeline (ADR-050).
**Last verified**: 2026-01-28
**Related documentation**:
- [ADR-050: PostgreSQL Function Observability](../adr/0050-postgresql-function-observability.md)
- [Logstash Quick Reference](LOGSTASH-QUICK-REF.md)
- [Monitoring Guide](MONITORING.md)
---
## Quick Reference ## Quick Reference
| Symptom | Most Likely Cause | Quick Check | | Symptom | Most Likely Cause | Quick Check |

View File

@@ -2,6 +2,72 @@
This guide covers all aspects of monitoring the Flyer Crawler application across development, test, and production environments. This guide covers all aspects of monitoring the Flyer Crawler application across development, test, and production environments.
**Last verified**: 2026-01-28
**Related documentation**:
- [ADR-015: Error Tracking and Observability](../adr/0015-error-tracking-and-observability.md)
- [ADR-020: Health Checks](../adr/0020-health-checks-and-liveness-readiness-probes.md)
- [ADR-050: PostgreSQL Function Observability](../adr/0050-postgresql-function-observability.md)
- [Logstash Quick Reference](LOGSTASH-QUICK-REF.md)
- [Deployment Guide](DEPLOYMENT.md)
---
## Quick Reference
### Monitoring URLs
| Service | Production URL | Dev Container URL |
| ------------ | ------------------------------------------------------- | ---------------------------------------- |
| Health Check | `https://flyer-crawler.projectium.com/api/health/ready` | `http://localhost:3001/api/health/ready` |
| Bugsink | `https://bugsink.projectium.com` | `https://localhost:8443` |
| Bull Board | `https://flyer-crawler.projectium.com/api/admin/jobs` | `http://localhost:3001/api/admin/jobs` |
### Quick Diagnostic Commands
```bash
# Check all services at once (production)
curl -s https://flyer-crawler.projectium.com/api/health/ready | jq '.data.services'
# Dev container health check
podman exec flyer-crawler-dev curl -s http://localhost:3001/api/health/ready | jq .
# PM2 process overview
pm2 list
# Recent errors in Bugsink (via MCP)
# mcp__bugsink__list_issues --project_id 1 --status unresolved
```
### Monitoring Decision Tree
```text
Application seems slow or unresponsive?
|
+-- Check health endpoint first
| |
| +-- Returns unhealthy?
| | |
| | +-- Database unhealthy --> Check DB pool, connections
| | +-- Redis unhealthy --> Check Redis memory, connection
| | +-- Storage unhealthy --> Check disk space, permissions
| |
| +-- Returns healthy but slow?
| |
| +-- Check PM2 memory/CPU usage
| +-- Check database slow query log
| +-- Check Redis queue depth
|
+-- Health endpoint not responding?
|
+-- Check PM2 status --> Process crashed?
+-- Check NGINX --> 502 errors?
+-- Check network --> Firewall/DNS issues?
```
---
## Table of Contents ## Table of Contents
1. [Health Checks](#health-checks) 1. [Health Checks](#health-checks)
@@ -294,7 +360,7 @@ The command outputs a 40-character hex token.
**Error Anatomy**: **Error Anatomy**:
``` ```text
TypeError: Cannot read properties of undefined (reading 'map') TypeError: Cannot read properties of undefined (reading 'map')
├── Exception Type: TypeError ├── Exception Type: TypeError
├── Message: Cannot read properties of undefined (reading 'map') ├── Message: Cannot read properties of undefined (reading 'map')
@@ -357,7 +423,7 @@ Logstash aggregates logs from multiple sources and forwards errors to Bugsink (A
### Architecture ### Architecture
``` ```text
Log Sources Logstash Outputs Log Sources Logstash Outputs
┌──────────────┐ ┌─────────────┐ ┌─────────────┐ ┌──────────────┐ ┌─────────────┐ ┌─────────────┐
│ PostgreSQL │──────────────│ │───────────│ Bugsink │ │ PostgreSQL │──────────────│ │───────────│ Bugsink │
@@ -520,7 +586,7 @@ pm2 stop flyer-crawler-api
**Healthy Process**: **Healthy Process**:
``` ```text
┌─────────────────────┬────┬─────────┬─────────┬───────┬────────┬─────────┬──────────┐ ┌─────────────────────┬────┬─────────┬─────────┬───────┬────────┬─────────┬──────────┐
│ Name │ id │ mode │ status │ cpu │ mem │ uptime │ restarts │ │ Name │ id │ mode │ status │ cpu │ mem │ uptime │ restarts │
├─────────────────────┼────┼─────────┼─────────┼───────┼────────┼─────────┼──────────┤ ├─────────────────────┼────┼─────────┼─────────┼───────┼────────┼─────────┼──────────┤
@@ -833,7 +899,7 @@ Configure alerts in your monitoring tool (UptimeRobot, Datadog, etc.):
2. Review during business hours 2. Review during business hours
3. Create Gitea issue for tracking 3. Create Gitea issue for tracking
### Quick Diagnostic Commands ### On-Call Diagnostic Commands
> **Note**: User executes these commands on the server. Claude Code provides commands but cannot run them directly. > **Note**: User executes these commands on the server. Claude Code provides commands but cannot run them directly.

View File

@@ -2,6 +2,17 @@
The **ai-usage** subagent specializes in LLM APIs (Gemini, Claude), prompt engineering, and AI-powered features in the Flyer Crawler project. The **ai-usage** subagent specializes in LLM APIs (Gemini, Claude), prompt engineering, and AI-powered features in the Flyer Crawler project.
## Quick Reference
| Aspect | Details |
| ------------------ | ----------------------------------------------------------------------------------- |
| **Primary Use** | Gemini API integration, prompt engineering, AI extraction |
| **Key Files** | `src/services/aiService.server.ts`, `src/services/flyerProcessingService.server.ts` |
| **Key ADRs** | ADR-041 (AI Integration), ADR-046 (Image Processing) |
| **API Key Env** | `VITE_GOOGLE_GENAI_API_KEY` (prod), `VITE_GOOGLE_GENAI_API_KEY_TEST` (test) |
| **Error Handling** | Rate limits (429), JSON parse errors, timeout handling |
| **Delegate To** | `coder` (implementation), `testwriter` (tests), `integrations-specialist` |
## When to Use ## When to Use
Use the **ai-usage** subagent when you need to: Use the **ai-usage** subagent when you need to:
@@ -295,6 +306,9 @@ const fixtureResponse = await fs.readFile('fixtures/gemini-response.json');
## Related Documentation ## Related Documentation
- [OVERVIEW.md](./OVERVIEW.md) - Subagent system overview - [OVERVIEW.md](./OVERVIEW.md) - Subagent system overview
- [CODER-GUIDE.md](./CODER-GUIDE.md) - For implementing AI features
- [TESTER-GUIDE.md](./TESTER-GUIDE.md) - Testing AI features
- [INTEGRATIONS-GUIDE.md](./INTEGRATIONS-GUIDE.md) - External API patterns
- [../adr/0041-ai-gemini-integration-architecture.md](../adr/0041-ai-gemini-integration-architecture.md) - AI integration ADR - [../adr/0041-ai-gemini-integration-architecture.md](../adr/0041-ai-gemini-integration-architecture.md) - AI integration ADR
- [../adr/0046-image-processing-pipeline.md](../adr/0046-image-processing-pipeline.md) - Image processing - [../adr/0046-image-processing-pipeline.md](../adr/0046-image-processing-pipeline.md) - Image processing
- [CODER-GUIDE.md](./CODER-GUIDE.md) - For implementing AI features - [../getting-started/ENVIRONMENT.md](../getting-started/ENVIRONMENT.md) - Environment configuration

View File

@@ -2,6 +2,17 @@
The **coder** subagent is your primary tool for writing and modifying production Node.js/TypeScript code in the Flyer Crawler project. This guide explains how to work effectively with the coder subagent. The **coder** subagent is your primary tool for writing and modifying production Node.js/TypeScript code in the Flyer Crawler project. This guide explains how to work effectively with the coder subagent.
## Quick Reference
| Aspect | Details |
| ---------------- | ------------------------------------------------------------------------ |
| **Primary Use** | Write/modify production TypeScript code |
| **Key Files** | `src/routes/*.routes.ts`, `src/services/**/*.ts`, `src/components/*.tsx` |
| **Key ADRs** | ADR-034 (Repository), ADR-035 (Services), ADR-028 (API Response) |
| **Test Command** | `podman exec -it flyer-crawler-dev npm run test:unit` |
| **Type Check** | `podman exec -it flyer-crawler-dev npm run type-check` |
| **Delegate To** | `db-dev` (database), `frontend-specialist` (UI), `testwriter` (tests) |
## When to Use the Coder Subagent ## When to Use the Coder Subagent
Use the coder subagent when you need to: Use the coder subagent when you need to:
@@ -307,6 +318,8 @@ error classes for all database operations"
- [OVERVIEW.md](./OVERVIEW.md) - Subagent system overview - [OVERVIEW.md](./OVERVIEW.md) - Subagent system overview
- [TESTER-GUIDE.md](./TESTER-GUIDE.md) - Testing strategies - [TESTER-GUIDE.md](./TESTER-GUIDE.md) - Testing strategies
- [DATABASE-GUIDE.md](./DATABASE-GUIDE.md) - Database development workflows
- [../adr/0034-repository-pattern-standards.md](../adr/0034-repository-pattern-standards.md) - Repository patterns - [../adr/0034-repository-pattern-standards.md](../adr/0034-repository-pattern-standards.md) - Repository patterns
- [../adr/0035-service-layer-architecture.md](../adr/0035-service-layer-architecture.md) - Service layer architecture - [../adr/0035-service-layer-architecture.md](../adr/0035-service-layer-architecture.md) - Service layer architecture
- [../adr/0028-api-response-standardization.md](../adr/0028-api-response-standardization.md) - API response patterns - [../adr/0028-api-response-standardization.md](../adr/0028-api-response-standardization.md) - API response patterns
- [../development/CODE-PATTERNS.md](../development/CODE-PATTERNS.md) - Code patterns reference

View File

@@ -5,6 +5,17 @@ This guide covers two database-focused subagents:
- **db-dev**: Database development - schemas, queries, migrations, optimization - **db-dev**: Database development - schemas, queries, migrations, optimization
- **db-admin**: Database administration - PostgreSQL/Redis admin, security, backups - **db-admin**: Database administration - PostgreSQL/Redis admin, security, backups
## Quick Reference
| Aspect | db-dev | db-admin |
| ---------------- | -------------------------------------------- | ------------------------------------------ |
| **Primary Use** | Schemas, queries, migrations | Performance tuning, backups, security |
| **Key Files** | `src/services/db/*.db.ts`, `sql/migrations/` | `postgresql.conf`, `pg_hba.conf` |
| **Key ADRs** | ADR-034 (Repository), ADR-002 (Transactions) | ADR-019 (Backups), ADR-050 (Observability) |
| **Test Command** | `podman exec -it flyer-crawler-dev npm test` | N/A |
| **MCP Tool** | `mcp__devdb__query` | SSH to production |
| **Delegate To** | `coder` (service layer), `db-admin` (perf) | `devops` (infrastructure) |
## Understanding the Difference ## Understanding the Difference
| Aspect | db-dev | db-admin | | Aspect | db-dev | db-admin |
@@ -412,8 +423,9 @@ This is useful for:
- [OVERVIEW.md](./OVERVIEW.md) - Subagent system overview - [OVERVIEW.md](./OVERVIEW.md) - Subagent system overview
- [CODER-GUIDE.md](./CODER-GUIDE.md) - Working with the coder subagent - [CODER-GUIDE.md](./CODER-GUIDE.md) - Working with the coder subagent
- [DEVOPS-GUIDE.md](./DEVOPS-GUIDE.md) - DevOps and deployment workflows
- [../adr/0034-repository-pattern-standards.md](../adr/0034-repository-pattern-standards.md) - Repository patterns - [../adr/0034-repository-pattern-standards.md](../adr/0034-repository-pattern-standards.md) - Repository patterns
- [../adr/0002-standardized-transaction-management.md](../adr/0002-standardized-transaction-management.md) - Transaction management - [../adr/0002-standardized-transaction-management.md](../adr/0002-standardized-transaction-management.md) - Transaction management
- [../adr/0019-data-backup-and-recovery-strategy.md](../adr/0019-data-backup-and-recovery-strategy.md) - Backup strategy - [../adr/0019-data-backup-and-recovery-strategy.md](../adr/0019-data-backup-and-recovery-strategy.md) - Backup strategy
- [../adr/0050-postgresql-function-observability.md](../adr/0050-postgresql-function-observability.md) - Database observability - [../adr/0050-postgresql-function-observability.md](../adr/0050-postgresql-function-observability.md) - Database observability
- [../BARE-METAL-SETUP.md](../BARE-METAL-SETUP.md) - Production database setup - [../operations/BARE-METAL-SETUP.md](../operations/BARE-METAL-SETUP.md) - Production database setup

View File

@@ -6,6 +6,17 @@ This guide covers DevOps-related subagents for deployment, infrastructure, and o
- **infra-architect**: Resource optimization, capacity planning - **infra-architect**: Resource optimization, capacity planning
- **bg-worker**: Background jobs, PM2 workers, BullMQ queues - **bg-worker**: Background jobs, PM2 workers, BullMQ queues
## Quick Reference
| Aspect | devops | infra-architect | bg-worker |
| ---------------- | ------------------------------------------ | --------------------------- | ------------------------------- |
| **Primary Use** | Containers, CI/CD, deployments | Resource optimization | BullMQ queues, PM2 workers |
| **Key Files** | `compose.dev.yml`, `.gitea/workflows/` | `ecosystem.config.cjs` | `src/services/queues.server.ts` |
| **Key ADRs** | ADR-014 (Containers), ADR-017 (CI/CD) | N/A | ADR-006 (Background Jobs) |
| **Commands** | `podman-compose`, `pm2` | `pm2 monit`, system metrics | Redis CLI, `pm2 logs` |
| **MCP Tools** | `mcp__podman__*` | N/A | N/A |
| **Access Model** | Read-only on production (provide commands) | Same | Same |
--- ---
## CRITICAL: Server Access Model ## CRITICAL: Server Access Model
@@ -543,8 +554,13 @@ podman exec -it flyer-crawler-dev npm test
## Related Documentation ## Related Documentation
- [OVERVIEW.md](./OVERVIEW.md) - Subagent system overview - [OVERVIEW.md](./OVERVIEW.md) - Subagent system overview
- [../BARE-METAL-SETUP.md](../BARE-METAL-SETUP.md) - Production setup guide - [DATABASE-GUIDE.md](./DATABASE-GUIDE.md) - Database administration
- [SECURITY-DEBUG-GUIDE.md](./SECURITY-DEBUG-GUIDE.md) - Production debugging
- [../operations/BARE-METAL-SETUP.md](../operations/BARE-METAL-SETUP.md) - Production setup guide
- [../operations/DEPLOYMENT.md](../operations/DEPLOYMENT.md) - Deployment guide
- [../operations/MONITORING.md](../operations/MONITORING.md) - Monitoring guide
- [../development/DEV-CONTAINER.md](../development/DEV-CONTAINER.md) - Dev container guide
- [../adr/0014-containerization-and-deployment-strategy.md](../adr/0014-containerization-and-deployment-strategy.md) - Containerization ADR - [../adr/0014-containerization-and-deployment-strategy.md](../adr/0014-containerization-and-deployment-strategy.md) - Containerization ADR
- [../adr/0006-background-job-processing-and-task-queues.md](../adr/0006-background-job-processing-and-task-queues.md) - Background jobs ADR - [../adr/0006-background-job-processing-and-task-queues.md](../adr/0006-background-job-processing-and-task-queues.md) - Background jobs ADR
- [../adr/0017-ci-cd-and-branching-strategy.md](../adr/0017-ci-cd-and-branching-strategy.md) - CI/CD strategy - [../adr/0017-ci-cd-and-branching-strategy.md](../adr/0017-ci-cd-and-branching-strategy.md) - CI/CD strategy
- [../adr/0053-worker-health-checks.md](../adr/0053-worker-health-checks.md) - Worker health checks - [../adr/0053-worker-health-checks-and-monitoring.md](../adr/0053-worker-health-checks-and-monitoring.md) - Worker health checks

View File

@@ -7,6 +7,15 @@ This guide covers documentation-focused subagents:
- **planner**: Feature breakdown, roadmaps, scope management - **planner**: Feature breakdown, roadmaps, scope management
- **product-owner**: Requirements, user stories, backlog prioritization - **product-owner**: Requirements, user stories, backlog prioritization
## Quick Reference
| Aspect | documenter | describer-for-ai | planner | product-owner |
| --------------- | -------------------- | ------------------------ | --------------------- | ---------------------- |
| **Primary Use** | User docs, API specs | ADRs, technical specs | Feature breakdown | User stories, backlog |
| **Key Files** | `docs/`, API docs | `docs/adr/`, `CLAUDE.md` | `docs/plans/` | Issue tracker |
| **Output** | Markdown guides | ADRs, context docs | Task lists, roadmaps | User stories, criteria |
| **Delegate To** | `coder` (implement) | `documenter` (user docs) | `coder` (build tasks) | `planner` (breakdown) |
## The documenter Subagent ## The documenter Subagent
### When to Use ### When to Use
@@ -437,6 +446,8 @@ Include dates on documentation that may become stale:
## Related Documentation ## Related Documentation
- [OVERVIEW.md](./OVERVIEW.md) - Subagent system overview - [OVERVIEW.md](./OVERVIEW.md) - Subagent system overview
- [CODER-GUIDE.md](./CODER-GUIDE.md) - For implementing documented features
- [../adr/index.md](../adr/index.md) - ADR index - [../adr/index.md](../adr/index.md) - ADR index
- [../TESTING.md](../TESTING.md) - Testing guide - [../development/TESTING.md](../development/TESTING.md) - Testing guide
- [../development/CODE-PATTERNS.md](../development/CODE-PATTERNS.md) - Code patterns reference
- [../../CLAUDE.md](../../CLAUDE.md) - AI instructions - [../../CLAUDE.md](../../CLAUDE.md) - AI instructions

View File

@@ -5,6 +5,17 @@ This guide covers frontend-focused subagents:
- **frontend-specialist**: UI components, Neo-Brutalism, Core Web Vitals, accessibility - **frontend-specialist**: UI components, Neo-Brutalism, Core Web Vitals, accessibility
- **uiux-designer**: UI/UX decisions, component design, user experience - **uiux-designer**: UI/UX decisions, component design, user experience
## Quick Reference
| Aspect | frontend-specialist | uiux-designer |
| ----------------- | ---------------------------------------------- | -------------------------------------- |
| **Primary Use** | React components, performance, accessibility | Design decisions, user flows |
| **Key Files** | `src/components/`, `src/features/` | Design specs, mockups |
| **Key ADRs** | ADR-012 (Design System), ADR-044 (Feature Org) | ADR-012 (Design System) |
| **Design System** | Neo-Brutalism (bold borders, high contrast) | Same |
| **State Mgmt** | TanStack Query (server), Zustand (client) | N/A |
| **Delegate To** | `coder` (backend), `tester` (test coverage) | `frontend-specialist` (implementation) |
## The frontend-specialist Subagent ## The frontend-specialist Subagent
### When to Use ### When to Use
@@ -406,7 +417,8 @@ const handleSelect = useCallback((id: string) => {
- [OVERVIEW.md](./OVERVIEW.md) - Subagent system overview - [OVERVIEW.md](./OVERVIEW.md) - Subagent system overview
- [CODER-GUIDE.md](./CODER-GUIDE.md) - For implementing features - [CODER-GUIDE.md](./CODER-GUIDE.md) - For implementing features
- [../DESIGN_TOKENS.md](../DESIGN_TOKENS.md) - Design token reference - [TESTER-GUIDE.md](./TESTER-GUIDE.md) - Component testing patterns
- [../development/DESIGN_TOKENS.md](../development/DESIGN_TOKENS.md) - Design token reference
- [../adr/0012-frontend-component-library-and-design-system.md](../adr/0012-frontend-component-library-and-design-system.md) - Design system ADR - [../adr/0012-frontend-component-library-and-design-system.md](../adr/0012-frontend-component-library-and-design-system.md) - Design system ADR
- [../adr/0005-frontend-state-management-and-server-cache-strategy.md](../adr/0005-frontend-state-management-and-server-cache-strategy.md) - State management ADR - [../adr/0005-frontend-state-management-and-server-cache-strategy.md](../adr/0005-frontend-state-management-and-server-cache-strategy.md) - State management ADR
- [../adr/0044-frontend-feature-organization.md](../adr/0044-frontend-feature-organization.md) - Feature organization - [../adr/0044-frontend-feature-organization.md](../adr/0044-frontend-feature-organization.md) - Feature organization

View File

@@ -0,0 +1,396 @@
# Integrations Subagent Guide
The **integrations-specialist** subagent handles third-party services, webhooks, and external API integrations in the Flyer Crawler project.
## Quick Reference
| Aspect | Details |
| --------------- | --------------------------------------------------------------------------- |
| **Primary Use** | External APIs, webhooks, OAuth, third-party services |
| **Key Files** | `src/services/external/`, `src/routes/webhooks.routes.ts` |
| **Key ADRs** | ADR-041 (AI Integration), ADR-016 (API Security), ADR-048 (Auth) |
| **MCP Tools** | `mcp__gitea-projectium__*`, `mcp__bugsink__*` |
| **Security** | API key storage, webhook signatures, OAuth state param |
| **Delegate To** | `coder` (implementation), `security-engineer` (review), `ai-usage` (Gemini) |
## When to Use
Use the **integrations-specialist** subagent when you need to:
- Integrate with external APIs (OAuth, REST, GraphQL)
- Implement webhook handlers
- Configure third-party services
- Debug external service connectivity
- Handle API authentication flows
- Manage external service rate limits
## What integrations-specialist Knows
The integrations-specialist subagent understands:
- OAuth 2.0 flows (authorization code, client credentials)
- REST API integration patterns
- Webhook security (signature verification)
- External service error handling
- Rate limiting and retry strategies
- API key management
## Current Integrations
| Service | Purpose | Integration Type | Key Files |
| ------------- | ---------------------- | ---------------- | ---------------------------------- |
| Google Gemini | AI flyer extraction | REST API | `src/services/aiService.server.ts` |
| Bugsink | Error tracking | REST API | MCP: `mcp__bugsink__*` |
| Gitea | Repository and CI/CD | REST API | MCP: `mcp__gitea-projectium__*` |
| Redis | Caching and job queues | Native client | `src/services/redis.server.ts` |
| PostgreSQL | Primary database | Native client | `src/services/db/pool.db.ts` |
## Example Requests
### Adding External API Integration
```
"Use integrations-specialist to integrate with the Store API
to automatically fetch store location data. Include proper
error handling, rate limiting, and caching."
```
### OAuth Implementation
```
"Use integrations-specialist to implement Google OAuth for
user authentication. Include token refresh handling and
session management."
```
### Webhook Handler
```
"Use integrations-specialist to create a webhook handler for
receiving store inventory updates. Include signature verification
and idempotency handling."
```
### Debugging External Service Issues
```
"Use integrations-specialist to debug why the Gemini API calls
are intermittently failing with timeout errors. Check connection
pooling, retry logic, and error handling."
```
## Integration Patterns
### REST API Client Pattern
```typescript
// src/services/external/storeApi.server.ts
import { env } from '@/config/env';
import { log } from '@/services/logger.server';
interface StoreApiConfig {
baseUrl: string;
apiKey: string;
timeout: number;
}
class StoreApiClient {
private config: StoreApiConfig;
constructor(config: StoreApiConfig) {
this.config = config;
}
async getStoreLocations(storeId: string): Promise<StoreLocation[]> {
const url = `${this.config.baseUrl}/stores/${storeId}/locations`;
try {
const response = await fetch(url, {
headers: {
Authorization: `Bearer ${this.config.apiKey}`,
'Content-Type': 'application/json',
},
signal: AbortSignal.timeout(this.config.timeout),
});
if (!response.ok) {
throw new ExternalApiError(`Store API error: ${response.status}`, response.status);
}
return response.json();
} catch (error) {
log.error({ error, storeId }, 'Failed to fetch store locations');
throw error;
}
}
}
export const storeApiClient = new StoreApiClient({
baseUrl: env.STORE_API_BASE_URL,
apiKey: env.STORE_API_KEY,
timeout: 10000,
});
```
### Webhook Handler Pattern
```typescript
// src/routes/webhooks.routes.ts
import { Router } from 'express';
import crypto from 'crypto';
import { env } from '@/config/env';
const router = Router();
function verifyWebhookSignature(payload: string, signature: string, secret: string): boolean {
const expected = crypto.createHmac('sha256', secret).update(payload).digest('hex');
return crypto.timingSafeEqual(Buffer.from(signature), Buffer.from(`sha256=${expected}`));
}
router.post('/store-updates', async (req, res, next) => {
try {
const signature = req.headers['x-webhook-signature'] as string;
const payload = JSON.stringify(req.body);
if (!verifyWebhookSignature(payload, signature, env.WEBHOOK_SECRET)) {
return res.status(401).json({ error: 'Invalid signature' });
}
// Process webhook with idempotency check
const eventId = req.headers['x-event-id'] as string;
const alreadyProcessed = await checkIdempotencyKey(eventId);
if (alreadyProcessed) {
return res.status(200).json({ status: 'already_processed' });
}
await processStoreUpdate(req.body);
await markEventProcessed(eventId);
res.status(200).json({ status: 'processed' });
} catch (error) {
next(error);
}
});
```
### OAuth Flow Pattern
```typescript
// src/services/oauth/googleOAuth.server.ts
import { OAuth2Client } from 'google-auth-library';
import { env } from '@/config/env';
const oauth2Client = new OAuth2Client(
env.GOOGLE_CLIENT_ID,
env.GOOGLE_CLIENT_SECRET,
env.GOOGLE_REDIRECT_URI,
);
export function getAuthorizationUrl(): string {
return oauth2Client.generateAuthUrl({
access_type: 'offline',
scope: ['email', 'profile'],
prompt: 'consent',
});
}
export async function exchangeCodeForTokens(code: string) {
const { tokens } = await oauth2Client.getToken(code);
return tokens;
}
export async function refreshAccessToken(refreshToken: string) {
oauth2Client.setCredentials({ refresh_token: refreshToken });
const { credentials } = await oauth2Client.refreshAccessToken();
return credentials;
}
```
## Error Handling for External Services
### Custom Error Classes
```typescript
// src/services/external/errors.ts
export class ExternalApiError extends Error {
constructor(
message: string,
public statusCode: number,
public retryable: boolean = false,
) {
super(message);
this.name = 'ExternalApiError';
}
}
export class RateLimitError extends ExternalApiError {
constructor(
message: string,
public retryAfter: number,
) {
super(message, 429, true);
this.name = 'RateLimitError';
}
}
```
### Retry with Exponential Backoff
```typescript
async function fetchWithRetry<T>(
fn: () => Promise<T>,
options: { maxRetries: number; baseDelay: number },
): Promise<T> {
let lastError: Error;
for (let attempt = 0; attempt <= options.maxRetries; attempt++) {
try {
return await fn();
} catch (error) {
lastError = error as Error;
if (error instanceof ExternalApiError && !error.retryable) {
throw error;
}
if (attempt < options.maxRetries) {
const delay = options.baseDelay * Math.pow(2, attempt);
await new Promise((resolve) => setTimeout(resolve, delay));
}
}
}
throw lastError!;
}
```
## Rate Limiting Strategies
### Token Bucket Pattern
```typescript
class RateLimiter {
private tokens: number;
private lastRefill: number;
private readonly maxTokens: number;
private readonly refillRate: number; // tokens per second
constructor(maxTokens: number, refillRate: number) {
this.maxTokens = maxTokens;
this.tokens = maxTokens;
this.refillRate = refillRate;
this.lastRefill = Date.now();
}
async acquire(): Promise<void> {
this.refill();
if (this.tokens < 1) {
const waitTime = ((1 - this.tokens) / this.refillRate) * 1000;
await new Promise((resolve) => setTimeout(resolve, waitTime));
this.refill();
}
this.tokens -= 1;
}
private refill(): void {
const now = Date.now();
const elapsed = (now - this.lastRefill) / 1000;
this.tokens = Math.min(this.maxTokens, this.tokens + elapsed * this.refillRate);
this.lastRefill = now;
}
}
```
## Testing Integrations
### Mocking External Services
```typescript
// src/tests/mocks/storeApi.mock.ts
import { vi } from 'vitest';
export const mockStoreApiClient = {
getStoreLocations: vi.fn(),
};
vi.mock('@/services/external/storeApi.server', () => ({
storeApiClient: mockStoreApiClient,
}));
```
### Integration Test with Real Service
```typescript
// src/tests/integration/storeApi.integration.test.ts
describe('Store API Integration', () => {
it.skipIf(!env.STORE_API_KEY)('fetches real store locations', async () => {
const locations = await storeApiClient.getStoreLocations('test-store');
expect(locations).toBeInstanceOf(Array);
});
});
```
## MCP Tools for Integrations
### Gitea Integration
```
// List repositories
mcp__gitea-projectium__list_my_repos()
// Create issue
mcp__gitea-projectium__create_issue({
owner: "projectium",
repo: "flyer-crawler",
title: "Issue title",
body: "Issue description"
})
```
### Bugsink Integration
```
// List projects
mcp__bugsink__list_projects()
// Get issue details
mcp__bugsink__get_issue({ issue_id: "..." })
// Get stacktrace
mcp__bugsink__get_stacktrace({ event_id: "..." })
```
## Security Considerations
### API Key Storage
- Never commit API keys to version control
- Use environment variables via `src/config/env.ts`
- Rotate keys periodically
- Use separate keys for dev/test/prod
### Webhook Security
- Always verify webhook signatures
- Use HTTPS for webhook endpoints
- Implement idempotency
- Log webhook events for audit
### OAuth Security
- Use state parameter to prevent CSRF
- Store tokens securely (encrypted at rest)
- Implement token refresh before expiration
- Validate token scopes
## Related Documentation
- [OVERVIEW.md](./OVERVIEW.md) - Subagent system overview
- [SECURITY-DEBUG-GUIDE.md](./SECURITY-DEBUG-GUIDE.md) - Security patterns
- [AI-USAGE-GUIDE.md](./AI-USAGE-GUIDE.md) - Gemini API integration
- [../adr/0041-ai-gemini-integration-architecture.md](../adr/0041-ai-gemini-integration-architecture.md) - AI integration ADR
- [../adr/0016-api-security-hardening.md](../adr/0016-api-security-hardening.md) - API security
- [../adr/0048-authentication-strategy.md](../adr/0048-authentication-strategy.md) - Authentication

View File

@@ -89,6 +89,47 @@ Or:
Claude will automatically invoke the appropriate subagent with the relevant context. Claude will automatically invoke the appropriate subagent with the relevant context.
## Quick Reference Decision Tree
Use this flowchart to quickly identify the right subagent:
```
What do you need to do?
|
+-- Write/modify code? ----------------> Is it database-related?
| |
| +-- Yes -> db-dev
| +-- No --> Is it frontend?
| |
| +-- Yes -> frontend-specialist
| +-- No --> Is it AI/Gemini?
| |
| +-- Yes -> ai-usage
| +-- No --> coder
|
+-- Test something? -------------------> Write new tests? -> testwriter
| Find bugs/vulnerabilities? -> tester
| Review existing code? -> code-reviewer
|
+-- Debug an issue? -------------------> Production error? -> log-debug
| Database slow? -> db-admin
| External API failing? -> integrations-specialist
| AI extraction failing? -> ai-usage
|
+-- Infrastructure/Deployment? --------> Container/CI/CD? -> devops
| Resource optimization? -> infra-architect
| Background jobs? -> bg-worker
|
+-- Documentation? --------------------> User-facing docs? -> documenter
| ADRs/Technical specs? -> describer-for-ai
| Feature planning? -> planner
| User stories? -> product-owner
|
+-- Security? -------------------------> security-engineer
|
+-- Design/UX? ------------------------> uiux-designer
```
## Subagent Selection Guide ## Subagent Selection Guide
### Which Subagent Should I Use? ### Which Subagent Should I Use?
@@ -183,12 +224,26 @@ Subagents can pass information back to the main conversation and to each other t
## Related Documentation ## Related Documentation
- [CODER-GUIDE.md](./CODER-GUIDE.md) - Working with the coder subagent ### Subagent Guides
- [TESTER-GUIDE.md](./TESTER-GUIDE.md) - Testing strategies and patterns
- [DATABASE-GUIDE.md](./DATABASE-GUIDE.md) - Database development workflows | Guide | Subagents Covered |
- [DEVOPS-GUIDE.md](./DEVOPS-GUIDE.md) - DevOps and deployment workflows | ---------------------------------------------------- | ----------------------------------------------------- |
| [CODER-GUIDE.md](./CODER-GUIDE.md) | coder |
| [TESTER-GUIDE.md](./TESTER-GUIDE.md) | tester, testwriter |
| [DATABASE-GUIDE.md](./DATABASE-GUIDE.md) | db-dev, db-admin |
| [DEVOPS-GUIDE.md](./DEVOPS-GUIDE.md) | devops, infra-architect, bg-worker |
| [FRONTEND-GUIDE.md](./FRONTEND-GUIDE.md) | frontend-specialist, uiux-designer |
| [SECURITY-DEBUG-GUIDE.md](./SECURITY-DEBUG-GUIDE.md) | security-engineer, log-debug, code-reviewer |
| [AI-USAGE-GUIDE.md](./AI-USAGE-GUIDE.md) | ai-usage |
| [INTEGRATIONS-GUIDE.md](./INTEGRATIONS-GUIDE.md) | integrations-specialist, tools-integration-specialist |
| [DOCUMENTATION-GUIDE.md](./DOCUMENTATION-GUIDE.md) | documenter, describer-for-ai, planner, product-owner |
### Project Documentation
- [../adr/index.md](../adr/index.md) - Architecture Decision Records - [../adr/index.md](../adr/index.md) - Architecture Decision Records
- [../TESTING.md](../TESTING.md) - Testing guide - [../development/TESTING.md](../development/TESTING.md) - Testing guide
- [../development/CODE-PATTERNS.md](../development/CODE-PATTERNS.md) - Code patterns reference
- [../architecture/OVERVIEW.md](../architecture/OVERVIEW.md) - System architecture
## Troubleshooting ## Troubleshooting

View File

@@ -6,6 +6,16 @@ This guide covers security and debugging-focused subagents:
- **log-debug**: Production errors, observability, Bugsink/Sentry analysis - **log-debug**: Production errors, observability, Bugsink/Sentry analysis
- **code-reviewer**: Code quality, security review, best practices - **code-reviewer**: Code quality, security review, best practices
## Quick Reference
| Aspect | security-engineer | log-debug | code-reviewer |
| --------------- | ---------------------------------- | ---------------------------------------- | --------------------------- |
| **Primary Use** | Security audits, OWASP | Production debugging | Code quality review |
| **Key ADRs** | ADR-016 (Security), ADR-032 (Rate) | ADR-050 (Observability) | ADR-034, ADR-035 (Patterns) |
| **MCP Tools** | N/A | `mcp__bugsink__*`, `mcp__localerrors__*` | N/A |
| **Key Checks** | Auth, input validation, CORS | Logs, stacktraces, error patterns | Patterns, tests, security |
| **Delegate To** | `coder` (fix issues) | `devops` (infra), `coder` (fixes) | `coder`, `testwriter` |
## The security-engineer Subagent ## The security-engineer Subagent
### When to Use ### When to Use
@@ -432,8 +442,10 @@ tail -f /var/log/postgresql/postgresql-$(date +%Y-%m-%d).log | grep "duration:"
- [OVERVIEW.md](./OVERVIEW.md) - Subagent system overview - [OVERVIEW.md](./OVERVIEW.md) - Subagent system overview
- [DEVOPS-GUIDE.md](./DEVOPS-GUIDE.md) - Infrastructure debugging - [DEVOPS-GUIDE.md](./DEVOPS-GUIDE.md) - Infrastructure debugging
- [TESTER-GUIDE.md](./TESTER-GUIDE.md) - Security testing
- [../adr/0016-api-security-hardening.md](../adr/0016-api-security-hardening.md) - Security ADR - [../adr/0016-api-security-hardening.md](../adr/0016-api-security-hardening.md) - Security ADR
- [../adr/0032-rate-limiting-strategy.md](../adr/0032-rate-limiting-strategy.md) - Rate limiting - [../adr/0032-rate-limiting-strategy.md](../adr/0032-rate-limiting-strategy.md) - Rate limiting
- [../adr/0015-application-performance-monitoring-and-error-tracking.md](../adr/0015-application-performance-monitoring-and-error-tracking.md) - Monitoring ADR - [../adr/0015-error-tracking-and-observability.md](../adr/0015-error-tracking-and-observability.md) - Monitoring ADR
- [../adr/0050-postgresql-function-observability.md](../adr/0050-postgresql-function-observability.md) - Database observability - [../adr/0050-postgresql-function-observability.md](../adr/0050-postgresql-function-observability.md) - Database observability
- [../BARE-METAL-SETUP.md](../BARE-METAL-SETUP.md) - Production setup - [../operations/BARE-METAL-SETUP.md](../operations/BARE-METAL-SETUP.md) - Production setup
- [../tools/BUGSINK-SETUP.md](../tools/BUGSINK-SETUP.md) - Bugsink configuration

View File

@@ -5,6 +5,17 @@ This guide covers two related but distinct subagents for testing in the Flyer Cr
- **tester**: Adversarial testing to find edge cases, race conditions, and vulnerabilities - **tester**: Adversarial testing to find edge cases, race conditions, and vulnerabilities
- **testwriter**: Creating comprehensive test suites for features and fixes - **testwriter**: Creating comprehensive test suites for features and fixes
## Quick Reference
| Aspect | tester | testwriter |
| ---------------- | -------------------------------------------- | ------------------------------------------ |
| **Primary Use** | Find bugs, security issues, edge cases | Create test suites, improve coverage |
| **Key Files** | N/A (analysis-focused) | `*.test.ts`, `src/tests/utils/` |
| **Key ADRs** | ADR-010 (Testing), ADR-040 (Test Economics) | ADR-010 (Testing), ADR-045 (Test Fixtures) |
| **Test Command** | `podman exec -it flyer-crawler-dev npm test` | Same |
| **Test Stack** | Vitest, Supertest, Testing Library | Same |
| **Delegate To** | `testwriter` (write tests for findings) | `coder` (fix failing tests) |
## Understanding the Difference ## Understanding the Difference
| Aspect | tester | testwriter | | Aspect | tester | testwriter |
@@ -399,6 +410,7 @@ A typical workflow for thorough testing:
- [OVERVIEW.md](./OVERVIEW.md) - Subagent system overview - [OVERVIEW.md](./OVERVIEW.md) - Subagent system overview
- [CODER-GUIDE.md](./CODER-GUIDE.md) - Working with the coder subagent - [CODER-GUIDE.md](./CODER-GUIDE.md) - Working with the coder subagent
- [../TESTING.md](../TESTING.md) - Testing guide - [SECURITY-DEBUG-GUIDE.md](./SECURITY-DEBUG-GUIDE.md) - Security testing and code review
- [../development/TESTING.md](../development/TESTING.md) - Testing guide
- [../adr/0010-testing-strategy-and-standards.md](../adr/0010-testing-strategy-and-standards.md) - Testing ADR - [../adr/0010-testing-strategy-and-standards.md](../adr/0010-testing-strategy-and-standards.md) - Testing ADR
- [../adr/0040-testing-economics-and-priorities.md](../adr/0040-testing-economics-and-priorities.md) - Testing priorities - [../adr/0040-testing-economics-and-priorities.md](../adr/0040-testing-economics-and-priorities.md) - Testing priorities

4
package-lock.json generated
View File

@@ -1,12 +1,12 @@
{ {
"name": "flyer-crawler", "name": "flyer-crawler",
"version": "0.12.20", "version": "0.12.21",
"lockfileVersion": 3, "lockfileVersion": 3,
"requires": true, "requires": true,
"packages": { "packages": {
"": { "": {
"name": "flyer-crawler", "name": "flyer-crawler",
"version": "0.12.20", "version": "0.12.21",
"dependencies": { "dependencies": {
"@bull-board/api": "^6.14.2", "@bull-board/api": "^6.14.2",
"@bull-board/express": "^6.14.2", "@bull-board/express": "^6.14.2",

View File

@@ -1,7 +1,7 @@
{ {
"name": "flyer-crawler", "name": "flyer-crawler",
"private": true, "private": true,
"version": "0.12.20", "version": "0.12.21",
"type": "module", "type": "module",
"scripts": { "scripts": {
"dev": "concurrently \"npm:start:dev\" \"vite\"", "dev": "concurrently \"npm:start:dev\" \"vite\"",

View File

@@ -185,10 +185,7 @@ describe('StoreCard', () => {
it('should show "No location data" when locations is undefined', () => { it('should show "No location data" when locations is undefined', () => {
renderWithProviders( renderWithProviders(
<StoreCard <StoreCard store={mockStoreUndefinedLocations as any} showLocations={true} />,
store={mockStoreUndefinedLocations as typeof mockStoreWithLogo}
showLocations={true}
/>,
); );
expect(screen.getByText('No location data')).toBeInTheDocument(); expect(screen.getByText('No location data')).toBeInTheDocument();

View File

@@ -14,7 +14,7 @@ vi.mock('../services/eventBus', () => ({
import { eventBus } from '../services/eventBus'; import { eventBus } from '../services/eventBus';
const mockEventBus = eventBus as { const mockEventBus = eventBus as unknown as {
on: Mock; on: Mock;
off: Mock; off: Mock;
dispatch: Mock; dispatch: Mock;
@@ -206,7 +206,7 @@ describe('useEventBus', () => {
name: string; name: string;
} }
const callback = vi.fn<[TestData?], void>(); const callback = vi.fn();
renderHook(() => useEventBus<TestData>('typed-event', callback)); renderHook(() => useEventBus<TestData>('typed-event', callback));
@@ -217,7 +217,7 @@ describe('useEventBus', () => {
}); });
it('should handle callback with optional parameter', () => { it('should handle callback with optional parameter', () => {
const callback = vi.fn<[string?], void>(); const callback = vi.fn();
renderHook(() => useEventBus<string>('optional-event', callback)); renderHook(() => useEventBus<string>('optional-event', callback));

View File

@@ -17,6 +17,7 @@ vi.mock('driver.js', () => ({
DriveStep: vi.fn(), DriveStep: vi.fn(),
})); }));
// @ts-expect-error - driver.js types are not fully compatible, but the mock works
import { driver } from 'driver.js'; import { driver } from 'driver.js';
const mockDriver = driver as Mock; const mockDriver = driver as Mock;