Compare commits

...

2 Commits

Author SHA1 Message Date
Gitea Actions
de3f0cf26e ci: Bump version to 0.12.21 [skip ci] 2026-01-29 05:37:59 +05:00
45ac4fccf5 comprehensive documentation review + test fixes
Some checks failed
Deploy to Test Environment / deploy-to-test (push) Failing after 2m15s
2026-01-28 16:35:38 -08:00
40 changed files with 2605 additions and 604 deletions

150
CLAUDE.md
View File

@@ -31,46 +31,19 @@ Out-of-sync = test failures.
**CRITICAL**: The `claude-win10` user has **READ-ONLY** access to production and test servers.
**Claude Code does NOT have:**
| Capability | Status |
| ---------------------- | ---------------------- |
| Root/sudo access | NO |
| Write permissions | NO |
| PM2 restart, systemctl | NO - User must execute |
- Root or sudo access
- Write permissions on servers
- Ability to execute PM2 restart, systemctl, or other write operations directly
**Server Operations Workflow**: Diagnose → User executes → Analyze → Fix (1-3 commands) → User executes → Verify
**Correct Workflow for Server Operations:**
**Rules**:
| Step | Actor | Action |
| ---- | ------ | --------------------------------------------------------------------------- |
| 1 | Claude | Provide **diagnostic commands** (read-only checks) for user to run |
| 2 | User | Execute commands on server, report results |
| 3 | Claude | Analyze results, provide **fix commands** (1-3 at a time) |
| 4 | User | Execute fix commands, report results |
| 5 | Claude | Provide **verification commands** to confirm success and check side effects |
| 6 | Claude | Document progress stage by stage |
| 7 | Claude | Update ALL relevant documentation when complete |
**Example - Diagnosing PM2 Issues:**
```bash
# Step 1: Claude provides diagnostic commands (user runs these)
pm2 list
pm2 logs flyer-crawler-api --lines 50
systemctl status redis
# Step 3: After user reports results, Claude provides fix commands
pm2 restart flyer-crawler-api
# Wait for user confirmation before next command
# Step 5: Claude provides verification
pm2 list
curl -s https://flyer-crawler.projectium.com/api/health/ready | jq .
```
**Never Do:**
- `ssh root@projectium.com "pm2 restart all"` (Claude cannot execute this)
- Assume commands succeeded without user confirmation
- Provide more than 3 fix commands at once (errors may cascade)
- Provide diagnostic commands first, wait for user to report results
- Maximum 3 fix commands at a time (errors may cascade)
- Always verify after fixes complete
### Communication Style
@@ -105,12 +78,12 @@ Ask before assuming. Never assume:
### Key Patterns (with file locations)
| Pattern | ADR | Implementation | File |
| ------------------ | ------- | ------------------------------------------------- | ----------------------------------- |
| Error Handling | ADR-001 | `handleDbError()`, throw `NotFoundError` | `src/services/db/errors.db.ts` |
| Repository Methods | ADR-034 | `get*` (throws), `find*` (null), `list*` (array) | `src/services/db/*.db.ts` |
| API Responses | ADR-028 | `sendSuccess()`, `sendPaginated()`, `sendError()` | `src/utils/apiResponse.ts` |
| Transactions | ADR-002 | `withTransaction(async (client) => {...})` | `src/services/db/transaction.db.ts` |
| Pattern | ADR | Implementation | File |
| ------------------ | ------- | ------------------------------------------------- | ---------------------------------- |
| Error Handling | ADR-001 | `handleDbError()`, throw `NotFoundError` | `src/services/db/errors.db.ts` |
| Repository Methods | ADR-034 | `get*` (throws), `find*` (null), `list*` (array) | `src/services/db/*.db.ts` |
| API Responses | ADR-028 | `sendSuccess()`, `sendPaginated()`, `sendError()` | `src/utils/apiResponse.ts` |
| Transactions | ADR-002 | `withTransaction(async (client) => {...})` | `src/services/db/connection.db.ts` |
### Key Files Quick Access
@@ -166,7 +139,7 @@ The dev container now matches production by using PM2 for process management.
- `flyer-crawler-worker-dev` - Background job worker
- `flyer-crawler-vite-dev` - Vite frontend dev server (port 5173)
### Log Aggregation (ADR-050)
### Log Aggregation (ADR-015)
All logs flow to Bugsink via Logstash with 3-project routing:
@@ -249,7 +222,7 @@ All logs flow to Bugsink via Logstash with 3-project routing:
**Launch Pattern**:
```
```text
Use Task tool with subagent_type: "coder", "db-dev", "tester", etc.
```
@@ -330,8 +303,8 @@ podman cp "d:/path/file" container:/tmp/file
**Quick Access**:
- **Dev**: https://localhost:8443 (`admin@localhost`/`admin`)
- **Prod**: https://bugsink.projectium.com
- **Dev**: <https://localhost:8443> (`admin@localhost`/`admin`)
- **Prod**: <https://bugsink.projectium.com>
**Token Creation** (required for MCP):
@@ -347,7 +320,7 @@ cd /opt/bugsink && bugsink-manage create_auth_token
**See**: [docs/operations/LOGSTASH-QUICK-REF.md](docs/operations/LOGSTASH-QUICK-REF.md)
Log aggregation: PostgreSQL + PM2 + Redis + NGINX → Bugsink (ADR-050)
Log aggregation: PostgreSQL + PM2 + Redis + NGINX → Bugsink (ADR-015)
---
@@ -367,84 +340,3 @@ Log aggregation: PostgreSQL + PM2 + Redis + NGINX → Bugsink (ADR-050)
| **Logstash** | [LOGSTASH-QUICK-REF.md](docs/operations/LOGSTASH-QUICK-REF.md) |
| **ADRs** | [docs/adr/index.md](docs/adr/index.md) |
| **All Docs** | [docs/README.md](docs/README.md) |
---
## Appendix: Integration Test Issues (Full Details)
### 1. Vitest globalSetup Context Isolation
Vitest's `globalSetup` runs in separate Node.js context. Singletons, spies, mocks do NOT share instances with test files.
**Affected**: BullMQ worker service mocks (AI/DB failure tests)
**Solutions**: Mark `.todo()`, create test-only API endpoints, use Redis-based mock flags
```typescript
// DOES NOT WORK - different instances
const { flyerProcessingService } = await import('../../services/workers.server');
flyerProcessingService._getAiProcessor()._setExtractAndValidateData(mockFn);
```
### 2. Cleanup Queue Deletes Before Verification
Cleanup worker processes jobs in globalSetup context, ignoring test spies.
**Solution**: Drain and pause queue:
```typescript
const { cleanupQueue } = await import('../../services/queues.server');
await cleanupQueue.drain();
await cleanupQueue.pause();
// ... test ...
await cleanupQueue.resume();
```
### 3. Cache Stale After Direct SQL
Direct `pool.query()` inserts bypass cache invalidation.
**Solution**: `await cacheService.invalidateFlyers();` after inserts
### 4. Test Filename Collisions
Multer predictable filenames cause race conditions.
**Solution**: Use unique suffix: `${Date.now()}-${Math.round(Math.random() * 1e9)}`
### 5. Response Format Mismatches
API formats change: `data.jobId` vs `data.job.id`, nested vs flat, string vs number IDs.
**Solution**: Log response bodies, update assertions
### 6. External Service Availability
PM2/Redis health checks fail when unavailable.
**Solution**: try/catch with graceful degradation or mock
### 7. TZ Environment Variable Breaking Async Hooks
**Problem**: When `TZ=America/Los_Angeles` (or other timezone values) is set in the environment, Node.js async_hooks module can produce `RangeError: Invalid triggerAsyncId value: NaN`. This breaks React Testing Library's `render()` function which uses async hooks internally.
**Root Cause**: Setting `TZ` to certain timezone values interferes with Node.js's internal async tracking mechanism, causing invalid async IDs to be generated.
**Symptoms**:
```text
RangeError: Invalid triggerAsyncId value: NaN
process.env.NODE_ENV.queueSeveralMicrotasks node_modules/react/cjs/react.development.js:751:15
process.env.NODE_ENV.exports.act node_modules/react/cjs/react.development.js:886:11
node_modules/@testing-library/react/dist/act-compat.js:46:25
renderRoot node_modules/@testing-library/react/dist/pure.js:189:26
```
**Solution**: Explicitly unset `TZ` in all test scripts by adding `TZ=` (empty value) to cross-env:
```json
"test:unit": "cross-env NODE_ENV=test TZ= tsx ..."
"test:integration": "cross-env NODE_ENV=test TZ= tsx ..."
```
**Context**: This issue was introduced in commit `d03900c` which added `TZ: 'America/Los_Angeles'` to PM2 ecosystem configs for consistent log timestamps in production/dev environments. Tests must explicitly override this to prevent the async hooks error.

View File

@@ -0,0 +1,393 @@
# AI Documentation Index
Machine-optimized navigation for AI agents. Structured for vector retrieval and semantic search.
---
## Quick Lookup Table
| Task/Question | Primary Doc | Section/ADR |
| ----------------------- | --------------------------------------------------- | --------------------------------------- |
| Add new API endpoint | [CODE-PATTERNS.md](development/CODE-PATTERNS.md) | API Response Patterns, Input Validation |
| Add repository method | [CODE-PATTERNS.md](development/CODE-PATTERNS.md) | Repository Patterns (get*/find*/list\*) |
| Fix failing test | [TESTING.md](development/TESTING.md) | Known Integration Test Issues |
| Run tests correctly | [TESTING.md](development/TESTING.md) | Test Execution Environment |
| Add database column | [DATABASE-GUIDE.md](subagents/DATABASE-GUIDE.md) | Schema sync required |
| Deploy to production | [DEPLOYMENT.md](operations/DEPLOYMENT.md) | Application Deployment |
| Debug container issue | [DEBUGGING.md](development/DEBUGGING.md) | Container Issues |
| Configure environment | [ENVIRONMENT.md](getting-started/ENVIRONMENT.md) | Configuration by Environment |
| Add background job | [CODE-PATTERNS.md](development/CODE-PATTERNS.md) | Background Jobs |
| Handle errors correctly | [CODE-PATTERNS.md](development/CODE-PATTERNS.md) | Error Handling |
| Use transactions | [CODE-PATTERNS.md](development/CODE-PATTERNS.md) | Transaction Management |
| Add authentication | [AUTHENTICATION.md](architecture/AUTHENTICATION.md) | JWT Token Architecture |
| Cache data | [CODE-PATTERNS.md](development/CODE-PATTERNS.md) | Caching |
| Check PM2 status | [DEV-CONTAINER.md](development/DEV-CONTAINER.md) | PM2 Process Management |
| View logs | [DEBUGGING.md](development/DEBUGGING.md) | PM2 Log Access |
| Understand architecture | [OVERVIEW.md](architecture/OVERVIEW.md) | System Architecture Diagram |
| Check ADR for decision | [adr/index.md](adr/index.md) | ADR by category |
| Use subagent | [subagents/OVERVIEW.md](subagents/OVERVIEW.md) | Available Subagents |
| API versioning | [API-VERSIONING.md](development/API-VERSIONING.md) | Phase 2 infrastructure |
---
## Documentation Tree
```
docs/
+-- AI-DOCUMENTATION-INDEX.md # THIS FILE - AI navigation index
+-- README.md # Human-readable doc hub
|
+-- adr/ # Architecture Decision Records (57 ADRs)
| +-- index.md # ADR index by category
| +-- 0001-*.md # Standardized error handling
| +-- 0002-*.md # Transaction management (withTransaction)
| +-- 0003-*.md # Input validation (Zod middleware)
| +-- 0008-*.md # API versioning (/api/v1/)
| +-- 0014-*.md # Platform: Linux only (CRITICAL)
| +-- 0028-*.md # API response (sendSuccess/sendError)
| +-- 0034-*.md # Repository pattern (get*/find*/list*)
| +-- 0035-*.md # Service layer architecture
| +-- 0050-*.md # PostgreSQL observability + Logstash
| +-- 0057-*.md # Test remediation post-API versioning
| +-- adr-implementation-tracker.md # Implementation status
|
+-- architecture/
| +-- OVERVIEW.md # System architecture, data flows, entities
| +-- DATABASE.md # Schema design, extensions, setup
| +-- AUTHENTICATION.md # OAuth, JWT, security features
| +-- WEBSOCKET_USAGE.md # Real-time communication patterns
| +-- api-versioning-infrastructure.md # Phase 2 versioning details
|
+-- development/
| +-- CODE-PATTERNS.md # Error handling, repos, API responses
| +-- TESTING.md # Unit/integration/E2E, known issues
| +-- DEBUGGING.md # Container, DB, API, PM2 debugging
| +-- DEV-CONTAINER.md # PM2, Logstash, container services
| +-- API-VERSIONING.md # API versioning workflows
| +-- DESIGN_TOKENS.md # Neo-Brutalism design system
| +-- ERROR-LOGGING-PATHS.md # req.originalUrl pattern
| +-- test-path-migration.md # Test file reorganization
|
+-- getting-started/
| +-- QUICKSTART.md # Quick setup instructions
| +-- INSTALL.md # Full installation guide
| +-- ENVIRONMENT.md # Environment variables reference
|
+-- operations/
| +-- DEPLOYMENT.md # Production deployment guide
| +-- BARE-METAL-SETUP.md # Server provisioning
| +-- MONITORING.md # Bugsink, health checks
| +-- LOGSTASH-QUICK-REF.md # Log aggregation reference
| +-- LOGSTASH-TROUBLESHOOTING.md # Logstash debugging
|
+-- subagents/
| +-- OVERVIEW.md # Subagent system introduction
| +-- CODER-GUIDE.md # Code development patterns
| +-- TESTER-GUIDE.md # Testing strategies
| +-- DATABASE-GUIDE.md # Database workflows
| +-- DEVOPS-GUIDE.md # Deployment/infrastructure
| +-- FRONTEND-GUIDE.md # UI/UX development
| +-- AI-USAGE-GUIDE.md # Gemini integration
| +-- DOCUMENTATION-GUIDE.md # Writing docs
| +-- SECURITY-DEBUG-GUIDE.md # Security and debugging
|
+-- tools/
| +-- MCP-CONFIGURATION.md # MCP servers setup
| +-- BUGSINK-SETUP.md # Error tracking setup
| +-- VSCODE-SETUP.md # Editor configuration
|
+-- archive/ # Historical docs, session notes
+-- sessions/ # Development session logs
+-- plans/ # Feature implementation plans
+-- research/ # Investigation notes
```
---
## Problem-to-Document Mapping
### Database Issues
| Problem | Documents |
| -------------------- | ----------------------------------------------------------------------------------------------- |
| Schema out of sync | [DATABASE-GUIDE.md](subagents/DATABASE-GUIDE.md), [CLAUDE.md](../CLAUDE.md) schema sync section |
| Migration needed | [DATABASE.md](architecture/DATABASE.md), ADR-013, ADR-023 |
| Query performance | [DEBUGGING.md](development/DEBUGGING.md) Query Performance Issues |
| Connection errors | [DEBUGGING.md](development/DEBUGGING.md) Database Issues |
| Transaction patterns | [CODE-PATTERNS.md](development/CODE-PATTERNS.md) Transaction Management, ADR-002 |
| Repository methods | [CODE-PATTERNS.md](development/CODE-PATTERNS.md) Repository Patterns, ADR-034 |
### Test Failures
| Problem | Documents |
| ---------------------------- | --------------------------------------------------------------------- |
| Tests fail in container | [TESTING.md](development/TESTING.md), ADR-014 |
| Vitest globalSetup isolation | [CLAUDE.md](../CLAUDE.md) Integration Test Issues #1 |
| Cache stale after insert | [CLAUDE.md](../CLAUDE.md) Integration Test Issues #3 |
| Queue interference | [CLAUDE.md](../CLAUDE.md) Integration Test Issues #2 |
| API path mismatches | [TESTING.md](development/TESTING.md) API Versioning in Tests, ADR-057 |
| Type check failures | [DEBUGGING.md](development/DEBUGGING.md) Type Check Failures |
| TZ environment breaks async | [CLAUDE.md](../CLAUDE.md) Integration Test Issues #7 |
### Deployment Issues
| Problem | Documents |
| --------------------- | ------------------------------------------------------------------------------------- |
| PM2 not starting | [DEBUGGING.md](development/DEBUGGING.md) PM2 Process Issues |
| NGINX configuration | [DEPLOYMENT.md](operations/DEPLOYMENT.md) NGINX Configuration |
| SSL certificates | [DEBUGGING.md](development/DEBUGGING.md) SSL Certificate Issues |
| CI/CD failures | [DEPLOYMENT.md](operations/DEPLOYMENT.md) CI/CD Pipeline, ADR-017 |
| Container won't start | [DEBUGGING.md](development/DEBUGGING.md) Container Issues |
| Bugsink not receiving | [BUGSINK-SETUP.md](tools/BUGSINK-SETUP.md), [MONITORING.md](operations/MONITORING.md) |
### Frontend/UI Changes
| Problem | Documents |
| ------------------ | --------------------------------------------------------------- |
| Component patterns | [FRONTEND-GUIDE.md](subagents/FRONTEND-GUIDE.md), ADR-044 |
| Design tokens | [DESIGN_TOKENS.md](development/DESIGN_TOKENS.md), ADR-012 |
| State management | ADR-005, [OVERVIEW.md](architecture/OVERVIEW.md) Frontend Stack |
| Hot reload broken | [DEBUGGING.md](development/DEBUGGING.md) Frontend Issues |
| CORS errors | [DEBUGGING.md](development/DEBUGGING.md) API Calls Failing |
### API Development
| Problem | Documents |
| ---------------- | ------------------------------------------------------------------------------- |
| Response format | [CODE-PATTERNS.md](development/CODE-PATTERNS.md) API Response Patterns, ADR-028 |
| Input validation | [CODE-PATTERNS.md](development/CODE-PATTERNS.md) Input Validation, ADR-003 |
| Error handling | [CODE-PATTERNS.md](development/CODE-PATTERNS.md) Error Handling, ADR-001 |
| Rate limiting | ADR-032, [OVERVIEW.md](architecture/OVERVIEW.md) |
| API versioning | [API-VERSIONING.md](development/API-VERSIONING.md), ADR-008 |
| Authentication | [AUTHENTICATION.md](architecture/AUTHENTICATION.md), ADR-048 |
### Background Jobs
| Problem | Documents |
| ------------------- | ------------------------------------------------------------------------- |
| Jobs not processing | [DEBUGGING.md](development/DEBUGGING.md) Background Job Issues |
| Queue configuration | [CODE-PATTERNS.md](development/CODE-PATTERNS.md) Background Jobs, ADR-006 |
| Worker crashes | [DEBUGGING.md](development/DEBUGGING.md), ADR-053 |
| Scheduled jobs | ADR-037, [OVERVIEW.md](architecture/OVERVIEW.md) Scheduled Jobs |
---
## Document Priority Matrix
### CRITICAL (Read First)
| Document | Purpose | Key Content |
| --------------------------------------------------------------- | ----------------------- | ----------------------------- |
| [CLAUDE.md](../CLAUDE.md) | AI agent instructions | Rules, patterns, known issues |
| [ADR-014](adr/0014-containerization-and-deployment-strategy.md) | Platform requirement | Tests MUST run in container |
| [DEV-CONTAINER.md](development/DEV-CONTAINER.md) | Development environment | PM2, Logstash, services |
### HIGH (Core Development)
| Document | Purpose | Key Content |
| --------------------------------------------------- | ----------------- | ---------------------------- |
| [CODE-PATTERNS.md](development/CODE-PATTERNS.md) | Code templates | Error handling, repos, APIs |
| [TESTING.md](development/TESTING.md) | Test execution | Commands, known issues |
| [DATABASE.md](architecture/DATABASE.md) | Schema reference | Setup, extensions, users |
| [ADR-034](adr/0034-repository-pattern-standards.md) | Repository naming | get*/find*/list\* |
| [ADR-028](adr/0028-api-response-standardization.md) | API responses | sendSuccess/sendError |
| [ADR-001](adr/0001-standardized-error-handling.md) | Error handling | handleDbError, NotFoundError |
### MEDIUM (Specialized Tasks)
| Document | Purpose | Key Content |
| --------------------------------------------------- | --------------------- | ------------------------ |
| [subagents/OVERVIEW.md](subagents/OVERVIEW.md) | Subagent selection | When to delegate |
| [DEPLOYMENT.md](operations/DEPLOYMENT.md) | Production deployment | PM2, NGINX, CI/CD |
| [DEBUGGING.md](development/DEBUGGING.md) | Troubleshooting | Common issues, solutions |
| [ENVIRONMENT.md](getting-started/ENVIRONMENT.md) | Config reference | Variables by environment |
| [AUTHENTICATION.md](architecture/AUTHENTICATION.md) | Auth patterns | OAuth, JWT, security |
| [API-VERSIONING.md](development/API-VERSIONING.md) | Versioning | /api/v1/ prefix |
### LOW (Reference/Historical)
| Document | Purpose | Key Content |
| -------------------- | ------------------ | ------------------------- |
| [archive/](archive/) | Historical docs | Session notes, old plans |
| ADR-013, ADR-023 | Migration strategy | Proposed, not implemented |
| ADR-024 | Feature flags | Proposed |
| ADR-025 | i18n/l10n | Proposed |
---
## Cross-Reference Matrix
| Document | References | Referenced By |
| -------------------- | ------------------------------------------------------------------------------- | ------------------------------------------------------ |
| **CLAUDE.md** | ADR-001, ADR-002, ADR-008, ADR-014, ADR-028, ADR-034, ADR-035, ADR-050, ADR-057 | All development docs |
| **ADR-008** | ADR-028 | API-VERSIONING.md, TESTING.md, ADR-057 |
| **ADR-014** | - | CLAUDE.md, TESTING.md, DEPLOYMENT.md, DEV-CONTAINER.md |
| **ADR-028** | ADR-001 | CODE-PATTERNS.md, OVERVIEW.md |
| **ADR-034** | ADR-001 | CODE-PATTERNS.md, DATABASE-GUIDE.md |
| **ADR-057** | ADR-008, ADR-028 | TESTING.md |
| **CODE-PATTERNS.md** | ADR-001, ADR-002, ADR-003, ADR-028, ADR-034, ADR-036, ADR-048 | CODER-GUIDE.md |
| **TESTING.md** | ADR-014, ADR-057, CLAUDE.md | TESTER-GUIDE.md, DEBUGGING.md |
| **DEBUGGING.md** | DEV-CONTAINER.md, TESTING.md, MONITORING.md | DEVOPS-GUIDE.md |
| **DEV-CONTAINER.md** | ADR-014, ADR-050, ecosystem.dev.config.cjs | DEBUGGING.md, CLAUDE.md |
| **OVERVIEW.md** | ADR-001 through ADR-050+ | All architecture docs |
| **DATABASE.md** | ADR-002, ADR-013, ADR-055 | DATABASE-GUIDE.md |
---
## Navigation Patterns
### Adding a Feature
```
1. CLAUDE.md -> Project rules, patterns
2. CODE-PATTERNS.md -> Implementation templates
3. Relevant subagent guide -> Domain-specific patterns
4. Related ADRs -> Design decisions
5. TESTING.md -> Test requirements
```
### Fixing a Bug
```
1. DEBUGGING.md -> Common issues checklist
2. TESTING.md -> Run tests in container
3. Error logs (pm2/bugsink) -> Identify root cause
4. CODE-PATTERNS.md -> Correct pattern reference
5. Related ADR -> Architectural context
```
### Deploying
```
1. DEPLOYMENT.md -> Deployment procedures
2. ENVIRONMENT.md -> Required variables
3. MONITORING.md -> Health check verification
4. LOGSTASH-QUICK-REF.md -> Log aggregation check
```
### Database Changes
```
1. DATABASE-GUIDE.md -> Schema sync requirements (CRITICAL)
2. DATABASE.md -> Schema design patterns
3. ADR-002 -> Transaction patterns
4. ADR-034 -> Repository methods
5. ADR-055 -> Normalization rules
```
### Subagent Selection
| Task Type | Subagent | Guide |
| --------------------- | ------------------------- | ------------------------------------------------------------ |
| Write production code | `coder` | [CODER-GUIDE.md](subagents/CODER-GUIDE.md) |
| Database changes | `db-dev` | [DATABASE-GUIDE.md](subagents/DATABASE-GUIDE.md) |
| Create tests | `testwriter` | [TESTER-GUIDE.md](subagents/TESTER-GUIDE.md) |
| Fix failing tests | `tester` | [TESTER-GUIDE.md](subagents/TESTER-GUIDE.md) |
| Container/deployment | `devops` | [DEVOPS-GUIDE.md](subagents/DEVOPS-GUIDE.md) |
| UI components | `frontend-specialist` | [FRONTEND-GUIDE.md](subagents/FRONTEND-GUIDE.md) |
| External APIs | `integrations-specialist` | - |
| Security review | `security-engineer` | [SECURITY-DEBUG-GUIDE.md](subagents/SECURITY-DEBUG-GUIDE.md) |
| Production errors | `log-debug` | [SECURITY-DEBUG-GUIDE.md](subagents/SECURITY-DEBUG-GUIDE.md) |
| AI/Gemini issues | `ai-usage` | [AI-USAGE-GUIDE.md](subagents/AI-USAGE-GUIDE.md) |
---
## Key File Quick Reference
### Configuration
| File | Purpose |
| -------------------------- | ---------------------------- |
| `server.ts` | Express app setup |
| `src/config/env.ts` | Environment validation (Zod) |
| `ecosystem.dev.config.cjs` | PM2 dev config |
| `ecosystem.config.cjs` | PM2 prod config |
| `vite.config.ts` | Vite build config |
### Core Implementation
| File | Purpose |
| ----------------------------------- | ----------------------------------- |
| `src/routes/*.routes.ts` | API route handlers |
| `src/services/db/*.db.ts` | Repository layer |
| `src/services/*.server.ts` | Server-only services |
| `src/services/queues.server.ts` | BullMQ queue definitions |
| `src/services/workers.server.ts` | BullMQ workers |
| `src/utils/apiResponse.ts` | sendSuccess/sendError/sendPaginated |
| `src/services/db/errors.db.ts` | handleDbError, NotFoundError |
| `src/services/db/transaction.db.ts` | withTransaction |
### Database Schema
| File | Purpose |
| ------------------------------ | ----------------------------------- |
| `sql/master_schema_rollup.sql` | Test DB, complete reference |
| `sql/initial_schema.sql` | Fresh install (identical to rollup) |
| `sql/migrations/*.sql` | Production ALTER statements |
### Testing
| File | Purpose |
| ---------------------------------- | ----------------------- |
| `vitest.config.ts` | Unit test config |
| `vitest.config.integration.ts` | Integration test config |
| `vitest.config.e2e.ts` | E2E test config |
| `src/tests/utils/mockFactories.ts` | Mock data factories |
| `src/tests/utils/storeHelpers.ts` | Store test helpers |
---
## ADR Quick Reference
### By Implementation Status
**Implemented**: 001, 002, 003, 004, 006, 008, 009, 010, 016, 017, 020, 021, 028, 032, 033, 034, 035, 036, 037, 038, 040, 041, 043, 044, 045, 046, 050, 051, 052, 055, 057
**Partially Implemented**: 012, 014, 015, 048
**Proposed**: 011, 013, 022, 023, 024, 025, 029, 030, 031, 039, 047, 053, 054, 056
### By Category
| Category | ADRs |
| --------------------- | ------------------------------------------- |
| Core Infrastructure | 002, 007, 020, 030 |
| Data Management | 009, 013, 019, 023, 031, 055 |
| API & Integration | 003, 008, 018, 022, 028 |
| Security | 001, 011, 016, 029, 032, 033, 048 |
| Observability | 004, 015, 050, 051, 052, 056 |
| Deployment & Ops | 006, 014, 017, 024, 037, 038, 053, 054 |
| Frontend/UI | 005, 012, 025, 026, 044 |
| Dev Workflow | 010, 021, 027, 040, 045, 047, 057 |
| Architecture Patterns | 034, 035, 036, 039, 041, 042, 043, 046, 049 |
---
## Essential Commands
```bash
# Run all tests (MUST use container)
podman exec -it flyer-crawler-dev npm test
# Run unit tests
podman exec -it flyer-crawler-dev npm run test:unit
# Run type check
podman exec -it flyer-crawler-dev npm run type-check
# Run integration tests
podman exec -it flyer-crawler-dev npm run test:integration
# PM2 status
podman exec -it flyer-crawler-dev pm2 status
# PM2 logs
podman exec -it flyer-crawler-dev pm2 logs
# Restart all processes
podman exec -it flyer-crawler-dev pm2 restart all
```
---
_This index is optimized for AI agent consumption. Updated: 2026-01-28_

View File

@@ -316,6 +316,7 @@ app.use('/api/v1', (req, res, next) => {
- [ADR-018](./0018-api-documentation-strategy.md) - API Documentation Strategy (versioned OpenAPI specs)
- [ADR-028](./0028-api-response-standardization.md) - Response Standardization (envelope pattern applies to all versions)
- [ADR-016](./0016-api-security-hardening.md) - Security Hardening (applies to all versions)
- [ADR-057](./0057-test-remediation-post-api-versioning.md) - Test Remediation Post-API Versioning (documents test migration)
## Implementation Checklist

View File

@@ -363,6 +363,13 @@ The following files contain acknowledged code smell violations that are deferred
- `src/tests/utils/mockFactories.ts` - Mock factories (1553 lines)
- `src/tests/utils/testHelpers.ts` - Test utilities
## Related ADRs
- [ADR-014](./0014-containerization-and-deployment-strategy.md) - Containerization (tests must run in dev container)
- [ADR-040](./0040-testing-economics-and-priorities.md) - Testing Economics and Priorities
- [ADR-045](./0045-test-data-factories-and-fixtures.md) - Test Data Factories and Fixtures
- [ADR-057](./0057-test-remediation-post-api-versioning.md) - Test Remediation Post-API Versioning
## Future Enhancements
1. **Browser E2E Tests**: Consider adding Playwright for actual browser testing

View File

@@ -2,7 +2,9 @@
**Date**: 2025-12-12
**Status**: Proposed
**Status**: Superseded by [ADR-023](./0023-database-schema-migration-strategy.md)
**Note**: This ADR was an early draft. ADR-023 provides a more detailed specification for the same topic.
## Context

View File

@@ -4,6 +4,8 @@
**Status**: Proposed
**Supersedes**: [ADR-013](./0013-database-schema-migration-strategy.md)
## Context
The `README.md` indicates that the database schema is managed by manually running a large `schema.sql.txt` file. This approach is highly error-prone, makes tracking changes difficult, and is not feasible for updating a live production database without downtime or data loss.

View File

@@ -195,6 +195,12 @@ Do NOT add tests:
- Coverage percentages may not satisfy external audits
- Requires judgment calls that may be inconsistent
## Related ADRs
- [ADR-010](./0010-testing-strategy-and-standards.md) - Testing Strategy and Standards (this ADR extends ADR-010)
- [ADR-045](./0045-test-data-factories-and-fixtures.md) - Test Data Factories and Fixtures
- [ADR-057](./0057-test-remediation-post-api-versioning.md) - Test Remediation Post-API Versioning
## Key Files
- `docs/adr/0010-testing-strategy-and-standards.md` - Testing mechanics

View File

@@ -4,7 +4,7 @@
**Status**: Accepted (Fully Implemented)
**Related**: [ADR-015](0015-application-performance-monitoring-and-error-tracking.md), [ADR-004](0004-standardized-application-wide-structured-logging.md)
**Related**: [ADR-015](0015-error-tracking-and-observability.md), [ADR-004](0004-standardized-application-wide-structured-logging.md)
## Context
@@ -335,7 +335,7 @@ SELECT award_achievement('user-uuid', 'Nonexistent Badge');
## References
- [ADR-015: Application Performance Monitoring](0015-application-performance-monitoring-and-error-tracking.md)
- [ADR-015: Error Tracking and Observability](0015-error-tracking-and-observability.md)
- [ADR-004: Standardized Structured Logging](0004-standardized-application-wide-structured-logging.md)
- [PostgreSQL RAISE Documentation](https://www.postgresql.org/docs/current/plpgsql-errors-and-messages.html)
- [PostgreSQL Logging Configuration](https://www.postgresql.org/docs/current/runtime-config-logging.html)

View File

@@ -332,6 +332,6 @@ Response:
## References
- [ADR-006: Background Job Processing](./0006-background-job-processing-and-task-queues.md)
- [ADR-015: Application Performance Monitoring](./0015-application-performance-monitoring-and-error-tracking.md)
- [ADR-015: Error Tracking and Observability](./0015-error-tracking-and-observability.md)
- [Bugsink API Documentation](https://bugsink.com/docs/api/)
- [Gitea API Documentation](https://docs.gitea.io/en-us/api-usage/)

View File

@@ -15,9 +15,10 @@ This document tracks the implementation status and estimated effort for all Arch
| Status | Count |
| ---------------------------- | ----- |
| Accepted (Fully Implemented) | 40 |
| Accepted (Fully Implemented) | 41 |
| Partially Implemented | 2 |
| Proposed (Not Started) | 14 |
| Proposed (Not Started) | 13 |
| Superseded | 1 |
---
@@ -34,13 +35,13 @@ This document tracks the implementation status and estimated effort for all Arch
### Category 2: Data Management
| ADR | Title | Status | Effort | Notes |
| --------------------------------------------------------------- | ------------------------ | -------- | ------ | ------------------------------ |
| [ADR-009](./0009-caching-strategy-for-read-heavy-operations.md) | Caching Strategy | Accepted | - | Fully implemented |
| [ADR-013](./0013-database-schema-migration-strategy.md) | Schema Migrations v1 | Proposed | M | Superseded by ADR-023 |
| [ADR-019](./0019-data-backup-and-recovery-strategy.md) | Backup & Recovery | Accepted | - | Fully implemented |
| [ADR-023](./0023-database-schema-migration-strategy.md) | Schema Migrations v2 | Proposed | L | Requires tooling setup |
| [ADR-031](./0031-data-retention-and-privacy-compliance.md) | Data Retention & Privacy | Proposed | XL | Legal/compliance review needed |
| ADR | Title | Status | Effort | Notes |
| --------------------------------------------------------------- | ------------------------ | ---------- | ------ | ------------------------------ |
| [ADR-009](./0009-caching-strategy-for-read-heavy-operations.md) | Caching Strategy | Accepted | - | Fully implemented |
| [ADR-013](./0013-database-schema-migration-strategy.md) | Schema Migrations v1 | Superseded | - | Superseded by ADR-023 |
| [ADR-019](./0019-data-backup-and-recovery-strategy.md) | Backup & Recovery | Accepted | - | Fully implemented |
| [ADR-023](./0023-database-schema-migration-strategy.md) | Schema Migrations v2 | Proposed | L | Requires tooling setup |
| [ADR-031](./0031-data-retention-and-privacy-compliance.md) | Data Retention & Privacy | Proposed | XL | Legal/compliance review needed |
### Category 3: API & Integration
@@ -108,6 +109,7 @@ This document tracks the implementation status and estimated effort for all Arch
| [ADR-040](./0040-testing-economics-and-priorities.md) | Testing Economics | Accepted | - | Fully implemented |
| [ADR-045](./0045-test-data-factories-and-fixtures.md) | Test Data Factories | Accepted | - | Fully implemented |
| [ADR-047](./0047-project-file-and-folder-organization.md) | Project Organization | Proposed | XL | Major reorganization |
| [ADR-057](./0057-test-remediation-post-api-versioning.md) | Test Remediation | Accepted | - | Fully implemented |
### Category 9: Architecture Patterns
@@ -148,6 +150,8 @@ These ADRs are proposed or partially implemented, ordered by suggested implement
| Date | ADR | Change |
| ---------- | ------- | ----------------------------------------------------------------------------------- |
| 2026-01-28 | ADR-057 | Created - Test remediation documentation for ADR-008 Phase 2 migration |
| 2026-01-28 | ADR-013 | Marked as Superseded by ADR-023 |
| 2026-01-27 | ADR-008 | Test path migration complete - 23 files, ~70 paths updated, 274->345 tests passing |
| 2026-01-27 | ADR-008 | Phase 2 Complete - Version router factory, deprecation headers, 82 versioning tests |
| 2026-01-26 | ADR-015 | Completed - Added Sentry user context in AuthProvider, now fully implemented |

View File

@@ -2,6 +2,8 @@
This directory contains a log of the architectural decisions made for the Flyer Crawler project.
**[Implementation Tracker](./adr-implementation-tracker.md)**: Track implementation status and effort estimates for all ADRs.
## 1. Foundational / Core Infrastructure
**[ADR-002](./0002-standardized-transaction-management.md)**: Standardized Transaction Management and Unit of Work Pattern (Accepted)
@@ -12,7 +14,7 @@ This directory contains a log of the architectural decisions made for the Flyer
## 2. Data Management
**[ADR-009](./0009-caching-strategy-for-read-heavy-operations.md)**: Caching Strategy for Read-Heavy Operations (Accepted)
**[ADR-013](./0013-database-schema-migration-strategy.md)**: Database Schema Migration Strategy (Proposed)
**[ADR-013](./0013-database-schema-migration-strategy.md)**: Database Schema Migration Strategy (Superseded by ADR-023)
**[ADR-019](./0019-data-backup-and-recovery-strategy.md)**: Data Backup and Recovery Strategy (Accepted)
**[ADR-023](./0023-database-schema-migration-strategy.md)**: Database Schema Migration Strategy (Proposed)
**[ADR-031](./0031-data-retention-and-privacy-compliance.md)**: Data Retention and Privacy Compliance (Proposed)
@@ -20,9 +22,9 @@ This directory contains a log of the architectural decisions made for the Flyer
## 3. API & Integration
**[ADR-003](./0003-standardized-input-validation-using-middleware.md)**: Standardized Input Validation using Middleware (Accepted)
**[ADR-008](./0008-api-versioning-strategy.md)**: API Versioning Strategy (Accepted - Phase 1 Complete)
**[ADR-008](./0008-api-versioning-strategy.md)**: API Versioning Strategy (Accepted - Phase 2 Complete)
**[ADR-018](./0018-api-documentation-strategy.md)**: API Documentation Strategy (Accepted)
**[ADR-022](./0022-real-time-notification-system.md)**: Real-time Notification System (Proposed)
**[ADR-022](./0022-real-time-notification-system.md)**: Real-time Notification System (Accepted)
**[ADR-028](./0028-api-response-standardization.md)**: API Response Standardization and Envelope Pattern (Implemented)
## 4. Security & Compliance
@@ -33,12 +35,12 @@ This directory contains a log of the architectural decisions made for the Flyer
**[ADR-029](./0029-secret-rotation-and-key-management.md)**: Secret Rotation and Key Management Strategy (Proposed)
**[ADR-032](./0032-rate-limiting-strategy.md)**: Rate Limiting Strategy (Accepted)
**[ADR-033](./0033-file-upload-and-storage-strategy.md)**: File Upload and Storage Strategy (Accepted)
**[ADR-048](./0048-authentication-strategy.md)**: Authentication Strategy (Partially Implemented)
**[ADR-048](./0048-authentication-strategy.md)**: Authentication Strategy (Accepted)
## 5. Observability & Monitoring
**[ADR-004](./0004-standardized-application-wide-structured-logging.md)**: Standardized Application-Wide Structured Logging (Accepted)
**[ADR-015](./0015-error-tracking-and-observability.md)**: Error Tracking and Observability (Partial)
**[ADR-015](./0015-error-tracking-and-observability.md)**: Error Tracking and Observability (Accepted)
**[ADR-050](./0050-postgresql-function-observability.md)**: PostgreSQL Function Observability (Accepted)
**[ADR-051](./0051-asynchronous-context-propagation.md)**: Asynchronous Context Propagation (Accepted)
**[ADR-052](./0052-granular-debug-logging-strategy.md)**: Granular Debug Logging Strategy (Accepted)
@@ -52,7 +54,7 @@ This directory contains a log of the architectural decisions made for the Flyer
**[ADR-024](./0024-feature-flagging-strategy.md)**: Feature Flagging Strategy (Proposed)
**[ADR-037](./0037-scheduled-jobs-and-cron-pattern.md)**: Scheduled Jobs and Cron Pattern (Accepted)
**[ADR-038](./0038-graceful-shutdown-pattern.md)**: Graceful Shutdown Pattern (Accepted)
**[ADR-053](./0053-worker-health-checks-and-monitoring.md)**: Worker Health Checks and Monitoring (Proposed)
**[ADR-053](./0053-worker-health-checks.md)**: Worker Health Checks and Stalled Job Monitoring (Accepted)
**[ADR-054](./0054-bugsink-gitea-issue-sync.md)**: Bugsink to Gitea Issue Synchronization (Proposed)
## 7. Frontend / User Interface

View File

@@ -1,10 +1,168 @@
# Database Setup
# Database Architecture
Flyer Crawler uses PostgreSQL with several extensions for full-text search, geographic data, and UUID generation.
**Version**: 0.12.20
**Last Updated**: 2026-01-28
Flyer Crawler uses PostgreSQL 16 with PostGIS for geographic data, pg_trgm for fuzzy text search, and uuid-ossp for UUID generation. The database contains 65 tables organized into logical domains.
## Table of Contents
1. [Schema Overview](#schema-overview)
2. [Database Setup](#database-setup)
3. [Schema Reference](#schema-reference)
4. [Related Documentation](#related-documentation)
---
## Required Extensions
## Schema Overview
The database is organized into the following domains:
### Core Infrastructure (6 tables)
| Table | Purpose | Primary Key |
| ----------------------- | ----------------------------------------- | ----------------- |
| `users` | Authentication credentials and login data | `user_id` (UUID) |
| `profiles` | Public user data, preferences, points | `user_id` (UUID) |
| `addresses` | Normalized address storage with geocoding | `address_id` |
| `activity_log` | User activity audit trail | `activity_log_id` |
| `password_reset_tokens` | Temporary tokens for password reset | `token_id` |
| `schema_info` | Schema deployment metadata | `environment` |
### Stores and Locations (4 tables)
| Table | Purpose | Primary Key |
| ------------------------ | --------------------------------------- | ------------------- |
| `stores` | Grocery store chains (Safeway, Kroger) | `store_id` |
| `store_locations` | Physical store locations with addresses | `store_location_id` |
| `favorite_stores` | User store favorites | `user_id, store_id` |
| `store_receipt_patterns` | Receipt text patterns for store ID | `pattern_id` |
### Flyers and Items (7 tables)
| Table | Purpose | Primary Key |
| ----------------------- | -------------------------------------- | ------------------------ |
| `flyers` | Uploaded flyer metadata and status | `flyer_id` |
| `flyer_items` | Individual deals extracted from flyers | `flyer_item_id` |
| `flyer_locations` | Flyer-to-location associations | `flyer_location_id` |
| `categories` | Item categorization (Produce, Dairy) | `category_id` |
| `master_grocery_items` | Canonical grocery item dictionary | `master_grocery_item_id` |
| `master_item_aliases` | Alternative names for master items | `alias_id` |
| `unmatched_flyer_items` | Items pending master item matching | `unmatched_item_id` |
### Products and Brands (2 tables)
| Table | Purpose | Primary Key |
| ---------- | ---------------------------------------------- | ------------ |
| `brands` | Brand names (Coca-Cola, Kraft) | `brand_id` |
| `products` | Specific products (master item + brand + size) | `product_id` |
### Price Tracking (3 tables)
| Table | Purpose | Primary Key |
| ----------------------- | ---------------------------------- | ------------------ |
| `item_price_history` | Historical prices for master items | `price_history_id` |
| `user_submitted_prices` | User-contributed price reports | `submission_id` |
| `suggested_corrections` | Suggested edits to flyer items | `correction_id` |
### User Features (8 tables)
| Table | Purpose | Primary Key |
| -------------------- | ------------------------------------ | --------------------------- |
| `user_watched_items` | Items user wants to track prices for | `user_watched_item_id` |
| `user_alerts` | Price alert thresholds | `alert_id` |
| `notifications` | User notifications | `notification_id` |
| `user_item_aliases` | User-defined item name aliases | `alias_id` |
| `user_follows` | User-to-user follow relationships | `follower_id, following_id` |
| `user_reactions` | Reactions to content (likes, etc.) | `reaction_id` |
| `budgets` | User-defined spending budgets | `budget_id` |
| `search_queries` | Search history for analytics | `query_id` |
### Shopping Lists (4 tables)
| Table | Purpose | Primary Key |
| ----------------------- | ------------------------ | ------------------------- |
| `shopping_lists` | User shopping lists | `shopping_list_id` |
| `shopping_list_items` | Items on shopping lists | `shopping_list_item_id` |
| `shared_shopping_lists` | Shopping list sharing | `shared_shopping_list_id` |
| `shopping_trips` | Completed shopping trips | `trip_id` |
| `shopping_trip_items` | Items purchased on trips | `trip_item_id` |
### Recipes (11 tables)
| Table | Purpose | Primary Key |
| --------------------------------- | -------------------------------- | ------------------------- |
| `recipes` | User recipes with metadata | `recipe_id` |
| `recipe_ingredients` | Recipe ingredient list | `recipe_ingredient_id` |
| `recipe_ingredient_substitutions` | Ingredient alternatives | `substitution_id` |
| `tags` | Recipe tags (vegan, quick, etc.) | `tag_id` |
| `recipe_tags` | Recipe-to-tag associations | `recipe_id, tag_id` |
| `appliances` | Kitchen appliances | `appliance_id` |
| `recipe_appliances` | Appliances needed for recipes | `recipe_id, appliance_id` |
| `recipe_ratings` | User ratings for recipes | `rating_id` |
| `recipe_comments` | User comments on recipes | `comment_id` |
| `favorite_recipes` | User recipe favorites | `user_id, recipe_id` |
| `recipe_collections` | User recipe collections | `collection_id` |
### Meal Planning (3 tables)
| Table | Purpose | Primary Key |
| ------------------- | -------------------------- | ----------------- |
| `menu_plans` | Weekly/monthly meal plans | `menu_plan_id` |
| `shared_menu_plans` | Menu plan sharing | `share_id` |
| `planned_meals` | Individual meals in a plan | `planned_meal_id` |
### Pantry and Inventory (4 tables)
| Table | Purpose | Primary Key |
| -------------------- | ------------------------------------ | ----------------- |
| `pantry_items` | User pantry inventory | `pantry_item_id` |
| `pantry_locations` | Storage locations (fridge, freezer) | `location_id` |
| `expiry_date_ranges` | Reference shelf life data | `expiry_range_id` |
| `expiry_alerts` | User expiry notification preferences | `expiry_alert_id` |
| `expiry_alert_log` | Sent expiry notifications | `alert_log_id` |
### Receipts (4 tables)
| Table | Purpose | Primary Key |
| ------------------------ | ----------------------------- | ----------------- |
| `receipts` | Scanned receipt metadata | `receipt_id` |
| `receipt_items` | Items parsed from receipts | `receipt_item_id` |
| `receipt_processing_log` | OCR/AI processing audit trail | `log_id` |
### UPC Scanning (2 tables)
| Table | Purpose | Primary Key |
| ---------------------- | ------------------------------- | ----------- |
| `upc_scan_history` | User barcode scan history | `scan_id` |
| `upc_external_lookups` | External UPC API response cache | `lookup_id` |
### Gamification (2 tables)
| Table | Purpose | Primary Key |
| ------------------- | ---------------------------- | ------------------------- |
| `achievements` | Defined achievements | `achievement_id` |
| `user_achievements` | Achievements earned by users | `user_id, achievement_id` |
### User Preferences (3 tables)
| Table | Purpose | Primary Key |
| --------------------------- | ---------------------------- | ------------------------- |
| `dietary_restrictions` | Defined dietary restrictions | `restriction_id` |
| `user_dietary_restrictions` | User dietary preferences | `user_id, restriction_id` |
| `user_appliances` | Appliances user owns | `user_id, appliance_id` |
### Reference Data (1 table)
| Table | Purpose | Primary Key |
| ------------------ | ----------------------- | --------------- |
| `unit_conversions` | Unit conversion factors | `conversion_id` |
---
## Database Setup
### Required Extensions
| Extension | Purpose |
| ----------- | ------------------------------------------- |
@@ -14,7 +172,7 @@ Flyer Crawler uses PostgreSQL with several extensions for full-text search, geog
---
## Database Users
### Database Users
This project uses **environment-specific database users** to isolate production and test environments:

View File

@@ -1,7 +1,7 @@
# Flyer Crawler - System Architecture Overview
**Version**: 0.12.5
**Last Updated**: 2026-01-22
**Version**: 0.12.20
**Last Updated**: 2026-01-28
**Platform**: Linux (Production and Development)
---
@@ -41,7 +41,7 @@
## System Architecture Diagram
```
```text
+-----------------------------------------------------------------------------------+
| CLIENT LAYER |
+-----------------------------------------------------------------------------------+
@@ -153,10 +153,10 @@
| Component | Technology | Version | Purpose |
| ---------------------- | ---------- | -------- | -------------------------------- |
| **Runtime** | Node.js | 22.x LTS | Server-side JavaScript runtime |
| **Language** | TypeScript | 5.9.x | Type-safe JavaScript superset |
| **Web Framework** | Express.js | 5.1.x | HTTP server and routing |
| **Frontend Framework** | React | 19.2.x | UI component library |
| **Build Tool** | Vite | 7.2.x | Frontend bundling and dev server |
| **Language** | TypeScript | 5.9.3 | Type-safe JavaScript superset |
| **Web Framework** | Express.js | 5.1.0 | HTTP server and routing |
| **Frontend Framework** | React | 19.2.0 | UI component library |
| **Build Tool** | Vite | 7.2.4 | Frontend bundling and dev server |
### Data Storage
@@ -176,23 +176,23 @@
| **OAuth** | Google, GitHub | Social authentication |
| **Email** | Nodemailer (SMTP) | Transactional emails |
### Background Processing
### Background Processing Stack
| Component | Technology | Version | Purpose |
| ------------------- | ---------- | ------- | --------------------------------- |
| **Job Queues** | BullMQ | 5.65.x | Reliable async job processing |
| **Job Queues** | BullMQ | 5.65.1 | Reliable async job processing |
| **Process Manager** | PM2 | Latest | Process management and clustering |
| **Scheduler** | node-cron | 4.2.x | Scheduled tasks |
| **Scheduler** | node-cron | 4.2.1 | Scheduled tasks |
### Frontend Stack
| Component | Technology | Version | Purpose |
| -------------------- | -------------- | ------- | ---------------------------------------- |
| **State Management** | TanStack Query | 5.90.x | Server state caching and synchronization |
| **Routing** | React Router | 7.9.x | Client-side routing |
| **Styling** | Tailwind CSS | 4.1.x | Utility-first CSS framework |
| **Icons** | Lucide React | 0.555.x | Icon components |
| **Charts** | Recharts | 3.4.x | Data visualization |
| **State Management** | TanStack Query | 5.90.12 | Server state caching and synchronization |
| **Routing** | React Router | 7.9.6 | Client-side routing |
| **Styling** | Tailwind CSS | 4.1.17 | Utility-first CSS framework |
| **Icons** | Lucide React | 0.555.0 | Icon components |
| **Charts** | Recharts | 3.4.1 | Data visualization |
### Observability and Quality
@@ -221,7 +221,7 @@ The frontend is a single-page application (SPA) built with React 19 and Vite.
**Directory Structure**:
```
```text
src/
+-- components/ # Reusable UI components
+-- contexts/ # React context providers
@@ -244,17 +244,30 @@ The backend is a RESTful API server built with Express.js 5.
- Structured logging with Pino
- Standardized error handling (ADR-001)
**API Route Modules**:
| Route | Purpose |
|-------|---------|
| `/api/auth` | Authentication (login, register, OAuth) |
| `/api/users` | User profile management |
| `/api/flyers` | Flyer CRUD and processing |
| `/api/recipes` | Recipe management |
| `/api/deals` | Best prices and deal discovery |
| `/api/stores` | Store management |
| `/api/admin` | Administrative functions |
| `/api/health` | Health checks and monitoring |
**API Route Modules** (all versioned under `/api/v1/*`):
| Route | Purpose |
| ------------------------- | ----------------------------------------------- |
| `/api/v1/auth` | Authentication (login, register, OAuth) |
| `/api/v1/health` | Health checks and monitoring |
| `/api/v1/system` | System administration (PM2 status, server info) |
| `/api/v1/users` | User profile management |
| `/api/v1/ai` | AI-powered features and flyer processing |
| `/api/v1/admin` | Administrative functions |
| `/api/v1/budgets` | Budget management and spending analysis |
| `/api/v1/achievements` | Gamification and achievement system |
| `/api/v1/flyers` | Flyer CRUD and processing |
| `/api/v1/recipes` | Recipe management and recommendations |
| `/api/v1/personalization` | Master items and user preferences |
| `/api/v1/price-history` | Price tracking and trend analysis |
| `/api/v1/stats` | Public statistics and analytics |
| `/api/v1/upc` | UPC barcode scanning and product lookup |
| `/api/v1/inventory` | Inventory and expiry tracking |
| `/api/v1/receipts` | Receipt scanning and purchase history |
| `/api/v1/deals` | Best prices and deal discovery |
| `/api/v1/reactions` | Social features (reactions, sharing) |
| `/api/v1/stores` | Store management and location services |
| `/api/v1/categories` | Category browsing and product categorization |
### Database (PostgreSQL/PostGIS)
@@ -331,7 +344,7 @@ BullMQ workers handle asynchronous processing tasks. PM2 manages both the API se
### Flyer Processing Pipeline
```
```text
+-------------+ +----------------+ +------------------+ +---------------+
| User | | Express | | BullMQ | | PostgreSQL |
| Upload +---->+ Route +---->+ Queue +---->+ Storage |
@@ -395,7 +408,7 @@ BullMQ workers handle asynchronous processing tasks. PM2 manages both the API se
The application follows a strict layered architecture as defined in ADR-035.
```
```text
+-----------------------------------------------------------------------+
| ROUTES LAYER |
| Responsibilities: |
@@ -458,7 +471,7 @@ The application follows a strict layered architecture as defined in ADR-035.
### Entity Relationship Overview
```
```text
+------------------+ +------------------+ +------------------+
| users | | profiles | | addresses |
|------------------| |------------------| |------------------|
@@ -537,7 +550,7 @@ The application follows a strict layered architecture as defined in ADR-035.
### JWT Token Architecture
```
```text
+-------------------+ +-------------------+ +-------------------+
| Login Request | | Server | | Database |
| (email/pass) +---->+ Validates +---->+ Verify User |
@@ -576,7 +589,7 @@ The application follows a strict layered architecture as defined in ADR-035.
### Protected Route Flow
```
```text
+-------------------+ +-------------------+ +-------------------+
| API Request | | requireAuth | | JWT Strategy |
| + Bearer Token +---->+ Middleware +---->+ Validate |
@@ -603,7 +616,7 @@ The application follows a strict layered architecture as defined in ADR-035.
### Worker Architecture
```
```text
+-------------------+ +-------------------+ +-------------------+
| API Server | | Redis | | Worker Process |
| (Queue Producer)| | (Job Storage) | | (Consumer) |
@@ -635,7 +648,7 @@ The application follows a strict layered architecture as defined in ADR-035.
Jobs use exponential backoff for retries:
```
```text
Attempt 1: Immediate
Attempt 2: Initial delay (e.g., 5 seconds)
Attempt 3: 2x delay (e.g., 10 seconds)
@@ -658,7 +671,7 @@ Attempt 4: 4x delay (e.g., 20 seconds)
### Environment Overview
```
```text
+-----------------------------------------------------------------------------------+
| DEVELOPMENT |
+-----------------------------------------------------------------------------------+
@@ -710,7 +723,7 @@ Attempt 4: 4x delay (e.g., 20 seconds)
### Deployment Pipeline (ADR-017)
```
```text
+------------+ +------------+ +------------+ +------------+
| Push to | | Gitea | | Build & | | Deploy |
| main +---->+ Actions +---->+ Test +---->+ to Prod |
@@ -839,22 +852,55 @@ The system architecture is governed by Architecture Decision Records (ADRs). Key
| File | Purpose |
| ----------------------------------------------- | --------------------------------------- |
| `src/services/flyerProcessingService.server.ts` | Flyer processing pipeline orchestration |
| `src/services/flyerAiProcessor.server.ts` | AI extraction for flyers |
| `src/services/aiService.server.ts` | Google Gemini AI integration |
| `src/services/cacheService.server.ts` | Redis caching abstraction |
| `src/services/emailService.server.ts` | Email sending |
| `src/services/queues.server.ts` | BullMQ queue definitions |
| `src/services/queueService.server.ts` | Queue management and scheduling |
| `src/services/workers.server.ts` | BullMQ worker definitions |
| `src/services/websocketService.server.ts` | Real-time WebSocket notifications |
| `src/services/receiptService.server.ts` | Receipt scanning and OCR |
| `src/services/upcService.server.ts` | UPC barcode lookup |
| `src/services/expiryService.server.ts` | Pantry expiry tracking |
| `src/services/geocodingService.server.ts` | Address geocoding |
| `src/services/analyticsService.server.ts` | Analytics and reporting |
| `src/services/monitoringService.server.ts` | Health monitoring |
| `src/services/barcodeService.server.ts` | Barcode detection |
| `src/services/logger.server.ts` | Structured logging (Pino) |
| `src/services/redis.server.ts` | Redis connection management |
| `src/services/sentry.server.ts` | Error tracking (Sentry/Bugsink) |
### Database Files
| File | Purpose |
| ---------------------------------- | -------------------------------------------- |
| `src/services/db/connection.db.ts` | Database pool and transaction management |
| `src/services/db/errors.db.ts` | Database error types |
| `src/services/db/user.db.ts` | User repository |
| `src/services/db/flyer.db.ts` | Flyer repository |
| `sql/master_schema_rollup.sql` | Complete database schema (for test DB setup) |
| `sql/initial_schema.sql` | Fresh installation schema |
| File | Purpose |
| --------------------------------------- | -------------------------------------------- |
| `src/services/db/connection.db.ts` | Database pool and transaction management |
| `src/services/db/errors.db.ts` | Database error types |
| `src/services/db/index.db.ts` | Repository exports |
| `src/services/db/user.db.ts` | User repository |
| `src/services/db/flyer.db.ts` | Flyer repository |
| `src/services/db/store.db.ts` | Store repository |
| `src/services/db/storeLocation.db.ts` | Store location repository |
| `src/services/db/recipe.db.ts` | Recipe repository |
| `src/services/db/category.db.ts` | Category repository |
| `src/services/db/personalization.db.ts` | Master items and personalization |
| `src/services/db/shopping.db.ts` | Shopping lists repository |
| `src/services/db/deals.db.ts` | Deals and best prices repository |
| `src/services/db/price.db.ts` | Price history repository |
| `src/services/db/receipt.db.ts` | Receipt repository |
| `src/services/db/upc.db.ts` | UPC scan history repository |
| `src/services/db/expiry.db.ts` | Expiry tracking repository |
| `src/services/db/gamification.db.ts` | Achievements repository |
| `src/services/db/budget.db.ts` | Budget repository |
| `src/services/db/reaction.db.ts` | User reactions repository |
| `src/services/db/notification.db.ts` | Notifications repository |
| `src/services/db/address.db.ts` | Address repository |
| `src/services/db/admin.db.ts` | Admin operations repository |
| `src/services/db/conversion.db.ts` | Unit conversion repository |
| `src/services/db/flyerLocation.db.ts` | Flyer locations repository |
| `sql/master_schema_rollup.sql` | Complete database schema (for test DB setup) |
| `sql/initial_schema.sql` | Fresh installation schema |
### Type Definitions

View File

@@ -2,6 +2,21 @@
Common code patterns extracted from Architecture Decision Records (ADRs). Use these as templates when writing new code.
## Quick Reference
| Pattern | Key Function/Class | Import From |
| ------------------ | ------------------------------------------------- | ---------------------------------- |
| Error Handling | `handleDbError()`, `NotFoundError` | `src/services/db/errors.db.ts` |
| Repository Methods | `get*`, `find*`, `list*` | `src/services/db/*.db.ts` |
| API Responses | `sendSuccess()`, `sendPaginated()`, `sendError()` | `src/utils/apiResponse.ts` |
| Transactions | `withTransaction()` | `src/services/db/connection.db.ts` |
| Validation | `validateRequest()` | `src/middleware/validation.ts` |
| Authentication | `authenticateJWT` | `src/middleware/auth.ts` |
| Caching | `cacheService` | `src/services/cache.server.ts` |
| Background Jobs | Queue classes | `src/services/queues.server.ts` |
---
## Table of Contents
- [Error Handling](#error-handling)
@@ -17,7 +32,7 @@ Common code patterns extracted from Architecture Decision Records (ADRs). Use th
## Error Handling
**ADR**: [ADR-001](../adr/0001-standardized-error-handling-for-database-operations.md)
**ADR**: [ADR-001](../adr/0001-standardized-error-handling.md)
### Repository Layer Error Handling
@@ -78,7 +93,7 @@ throw new DatabaseError('Failed to insert flyer', originalError);
## Repository Patterns
**ADR**: [ADR-034](../adr/0034-repository-layer-method-naming-conventions.md)
**ADR**: [ADR-034](../adr/0034-repository-pattern-standards.md)
### Method Naming Conventions
@@ -155,16 +170,17 @@ export async function listActiveFlyers(client?: PoolClient): Promise<Flyer[]> {
## API Response Patterns
**ADR**: [ADR-028](../adr/0028-consistent-api-response-format.md)
**ADR**: [ADR-028](../adr/0028-api-response-standardization.md)
### Success Response
```typescript
import { sendSuccess } from '../utils/apiResponse';
app.post('/api/flyers', async (req, res) => {
app.post('/api/v1/flyers', async (req, res) => {
const flyer = await flyerService.createFlyer(req.body);
return sendSuccess(res, flyer, 'Flyer created successfully', 201);
// sendSuccess(res, data, statusCode?, meta?)
return sendSuccess(res, flyer, 201);
});
```
@@ -173,30 +189,32 @@ app.post('/api/flyers', async (req, res) => {
```typescript
import { sendPaginated } from '../utils/apiResponse';
app.get('/api/flyers', async (req, res) => {
const { page = 1, pageSize = 20 } = req.query;
const { items, total } = await flyerService.listFlyers(page, pageSize);
app.get('/api/v1/flyers', async (req, res) => {
const page = parseInt(req.query.page as string) || 1;
const limit = parseInt(req.query.limit as string) || 20;
const { items, total } = await flyerService.listFlyers(page, limit);
return sendPaginated(res, {
items,
total,
page: parseInt(page),
pageSize: parseInt(pageSize),
});
// sendPaginated(res, data[], { page, limit, total }, meta?)
return sendPaginated(res, items, { page, limit, total });
});
```
### Error Response
```typescript
import { sendError } from '../utils/apiResponse';
import { sendError, sendSuccess, ErrorCode } from '../utils/apiResponse';
app.get('/api/flyers/:id', async (req, res) => {
app.get('/api/v1/flyers/:id', async (req, res) => {
try {
const flyer = await flyerDb.getFlyerById(parseInt(req.params.id));
return sendSuccess(res, flyer);
} catch (error) {
return sendError(res, error); // Automatically maps error to correct status
// sendError(res, code, message, statusCode?, details?, meta?)
if (error instanceof NotFoundError) {
return sendError(res, ErrorCode.NOT_FOUND, error.message, 404);
}
req.log.error({ error }, `Error in ${req.originalUrl.split('?')[0]}:`);
return sendError(res, ErrorCode.INTERNAL_ERROR, 'An error occurred', 500);
}
});
```
@@ -205,12 +223,12 @@ app.get('/api/flyers/:id', async (req, res) => {
## Transaction Management
**ADR**: [ADR-002](../adr/0002-transaction-management-pattern.md)
**ADR**: [ADR-002](../adr/0002-standardized-transaction-management.md)
### Basic Transaction
```typescript
import { withTransaction } from '../services/db/transaction.db';
import { withTransaction } from '../services/db/connection.db';
export async function createFlyerWithItems(
flyerData: FlyerInput,
@@ -262,7 +280,7 @@ export async function bulkImportFlyers(flyersData: FlyerInput[]): Promise<Import
## Input Validation
**ADR**: [ADR-003](../adr/0003-input-validation-framework.md)
**ADR**: [ADR-003](../adr/0003-standardized-input-validation-using-middleware.md)
### Zod Schema Definition
@@ -298,10 +316,10 @@ export type CreateFlyerInput = z.infer<typeof createFlyerSchema>;
import { validateRequest } from '../middleware/validation';
import { createFlyerSchema } from '../schemas/flyer.schemas';
app.post('/api/flyers', validateRequest(createFlyerSchema), async (req, res) => {
app.post('/api/v1/flyers', validateRequest(createFlyerSchema), async (req, res) => {
// req.body is now type-safe and validated
const flyer = await flyerService.createFlyer(req.body);
return sendSuccess(res, flyer, 'Flyer created successfully', 201);
return sendSuccess(res, flyer, 201);
});
```
@@ -331,7 +349,7 @@ export async function processFlyer(data: unknown): Promise<Flyer> {
import { authenticateJWT } from '../middleware/auth';
app.get(
'/api/profile',
'/api/v1/profile',
authenticateJWT, // Middleware adds req.user
async (req, res) => {
// req.user is guaranteed to exist
@@ -347,7 +365,7 @@ app.get(
import { optionalAuth } from '../middleware/auth';
app.get(
'/api/flyers',
'/api/v1/flyers',
optionalAuth, // req.user may or may not exist
async (req, res) => {
const flyers = req.user
@@ -374,7 +392,7 @@ export function generateToken(user: User): string {
## Caching
**ADR**: [ADR-029](../adr/0029-redis-caching-strategy.md)
**ADR**: [ADR-009](../adr/0009-caching-strategy-for-read-heavy-operations.md)
### Cache Pattern
@@ -414,7 +432,7 @@ export async function updateFlyer(id: number, data: UpdateFlyerInput): Promise<F
## Background Jobs
**ADR**: [ADR-036](../adr/0036-background-job-processing-architecture.md)
**ADR**: [ADR-006](../adr/0006-background-job-processing-and-task-queues.md)
### Queue Job

View File

@@ -229,7 +229,7 @@ SELECT * FROM flyers WHERE store_id = 1;
- Add missing indexes
- Optimize WHERE clauses
- Use connection pooling
- See [ADR-034](../adr/0034-repository-layer-method-naming-conventions.md)
- See [ADR-034](../adr/0034-repository-pattern-standards.md)
---
@@ -237,7 +237,7 @@ SELECT * FROM flyers WHERE store_id = 1;
### Tests Pass on Windows, Fail in Container
**Cause**: Platform-specific behavior (ADR-014)
**Cause**: Platform-specific behavior ([ADR-014](../adr/0014-containerization-and-deployment-strategy.md))
**Rule**: Container results are authoritative. Windows results are unreliable.

View File

@@ -93,7 +93,7 @@ When the container starts (`scripts/dev-entrypoint.sh`):
PM2 manages three processes in the dev container:
```
```text
+--------------------+ +------------------------+ +--------------------+
| flyer-crawler- | | flyer-crawler- | | flyer-crawler- |
| api-dev | | worker-dev | | vite-dev |
@@ -404,5 +404,5 @@ podman exec -it flyer-crawler-dev pm2 restart flyer-crawler-api-dev
- [DEBUGGING.md](DEBUGGING.md) - Debugging strategies
- [LOGSTASH-QUICK-REF.md](../operations/LOGSTASH-QUICK-REF.md) - Logstash quick reference
- [DEV-CONTAINER-BUGSINK.md](../DEV-CONTAINER-BUGSINK.md) - Bugsink setup in dev container
- [ADR-014](../adr/0014-linux-only-platform.md) - Linux-only platform decision
- [ADR-050](../adr/0050-postgresql-function-observability.md) - PostgreSQL function observability
- [ADR-014](../adr/0014-containerization-and-deployment-strategy.md) - Containerization and deployment strategy
- [ADR-050](../adr/0050-postgresql-function-observability.md) - PostgreSQL function observability (includes log aggregation)

View File

@@ -1,5 +1,19 @@
# Testing Guide
## Quick Reference
| Command | Purpose |
| ------------------------------------------------------------ | ---------------------------- |
| `podman exec -it flyer-crawler-dev npm test` | Run all tests |
| `podman exec -it flyer-crawler-dev npm run test:unit` | Unit tests (~2900) |
| `podman exec -it flyer-crawler-dev npm run test:integration` | Integration tests (28 files) |
| `podman exec -it flyer-crawler-dev npm run test:e2e` | E2E tests (11 files) |
| `podman exec -it flyer-crawler-dev npm run type-check` | TypeScript check |
**Critical**: Always run tests in the dev container. Windows results are unreliable.
---
## Overview
This project has comprehensive test coverage including unit tests, integration tests, and E2E tests. All tests must be run in the **Linux dev container environment** for reliable results.
@@ -76,7 +90,7 @@ To verify type-check is working correctly:
Example error output:
```
```text
src/pages/MyDealsPage.tsx:68:31 - error TS2339: Property 'store_name' does not exist on type 'WatchedItemDeal'.
68 <span>{deal.store_name}</span>
@@ -113,15 +127,26 @@ Located throughout `src/` directory alongside source files with `.test.ts` or `.
npm run test:unit
```
### Integration Tests (5 test files)
### Integration Tests (28 test files)
Located in `src/tests/integration/`:
Located in `src/tests/integration/`. Key test files include:
- `admin.integration.test.ts`
- `flyer.integration.test.ts`
- `price.integration.test.ts`
- `public.routes.integration.test.ts`
- `receipt.integration.test.ts`
| Test File | Domain |
| -------------------------------------- | -------------------------- |
| `admin.integration.test.ts` | Admin dashboard operations |
| `auth.integration.test.ts` | Authentication flows |
| `budget.integration.test.ts` | Budget management |
| `flyer.integration.test.ts` | Flyer CRUD operations |
| `flyer-processing.integration.test.ts` | AI flyer processing |
| `gamification.integration.test.ts` | Achievements and points |
| `inventory.integration.test.ts` | Inventory management |
| `notification.integration.test.ts` | User notifications |
| `receipt.integration.test.ts` | Receipt processing |
| `recipe.integration.test.ts` | Recipe management |
| `shopping-list.integration.test.ts` | Shopping list operations |
| `user.integration.test.ts` | User profile operations |
See `src/tests/integration/` for the complete list.
Requires PostgreSQL and Redis services running.
@@ -129,13 +154,23 @@ Requires PostgreSQL and Redis services running.
npm run test:integration
```
### E2E Tests (3 test files)
### E2E Tests (11 test files)
Located in `src/tests/e2e/`:
Located in `src/tests/e2e/`. Full user journey tests:
- `deals-journey.e2e.test.ts`
- `budget-journey.e2e.test.ts`
- `receipt-journey.e2e.test.ts`
| Test File | Journey |
| --------------------------------- | ----------------------------- |
| `admin-authorization.e2e.test.ts` | Admin access control |
| `admin-dashboard.e2e.test.ts` | Admin dashboard flows |
| `auth.e2e.test.ts` | Login/logout/registration |
| `budget-journey.e2e.test.ts` | Budget tracking workflow |
| `deals-journey.e2e.test.ts` | Finding and saving deals |
| `error-reporting.e2e.test.ts` | Error handling verification |
| `flyer-upload.e2e.test.ts` | Flyer upload and processing |
| `inventory-journey.e2e.test.ts` | Pantry management |
| `receipt-journey.e2e.test.ts` | Receipt scanning and tracking |
| `upc-journey.e2e.test.ts` | UPC barcode scanning |
| `user-journey.e2e.test.ts` | User profile management |
Requires all services (PostgreSQL, Redis, BullMQ workers) running.
@@ -157,20 +192,18 @@ Located in `src/tests/utils/storeHelpers.ts`:
```typescript
// Create a store with a location in one call
const store = await createStoreWithLocation({
storeName: 'Test Store',
address: {
address_line_1: '123 Main St',
city: 'Toronto',
province_state: 'ON',
postal_code: 'M1M 1M1',
},
pool,
log,
const store = await createStoreWithLocation(pool, {
name: 'Test Store',
address: '123 Main St',
city: 'Toronto',
province: 'ON',
postalCode: 'M1M 1M1',
});
// Returns: { storeId, addressId, storeLocationId }
// Cleanup stores and their locations
await cleanupStoreLocations([storeId1, storeId2], pool, log);
await cleanupStoreLocation(pool, store);
```
### Mock Factories

View File

@@ -2,134 +2,259 @@
Complete guide to environment variables used in Flyer Crawler.
---
## Quick Reference
### Minimum Required Variables (Development)
| Variable | Example | Purpose |
| ---------------- | ------------------------ | -------------------- |
| `DB_HOST` | `localhost` | PostgreSQL host |
| `DB_USER` | `postgres` | PostgreSQL username |
| `DB_PASSWORD` | `postgres` | PostgreSQL password |
| `DB_NAME` | `flyer_crawler_dev` | Database name |
| `REDIS_URL` | `redis://localhost:6379` | Redis connection URL |
| `JWT_SECRET` | (32+ character string) | JWT signing key |
| `GEMINI_API_KEY` | `AIzaSy...` | Google Gemini API |
### Source of Truth
The Zod schema at `src/config/env.ts` is the authoritative source for all environment variables. If a variable is not in this file, it is not used by the application.
---
## Configuration by Environment
### Production
**Location**: Gitea CI/CD secrets injected during deployment
**Path**: `/var/www/flyer-crawler.projectium.com/`
**Note**: No `.env` file exists - all variables come from CI/CD
| Aspect | Details |
| -------- | ------------------------------------------ |
| Location | Gitea CI/CD secrets injected at deployment |
| Path | `/var/www/flyer-crawler.projectium.com/` |
| File | No `.env` file - all from CI/CD secrets |
### Test
**Location**: Gitea CI/CD secrets + `.env.test` file
**Path**: `/var/www/flyer-crawler-test.projectium.com/`
**Note**: `.env.test` overrides for test-specific values
| Aspect | Details |
| -------- | --------------------------------------------- |
| Location | Gitea CI/CD secrets + `.env.test` overrides |
| Path | `/var/www/flyer-crawler-test.projectium.com/` |
| File | `.env.test` for test-specific values |
### Development Container
**Location**: `.env.local` file in project root
**Note**: Overrides default DSNs in `compose.dev.yml`
| Aspect | Details |
| -------- | --------------------------------------- |
| Location | `.env.local` file in project root |
| Priority | Overrides defaults in `compose.dev.yml` |
| File | `.env.local` (gitignored) |
## Required Variables
---
### Database
## Complete Variable Reference
| Variable | Description | Example |
| ------------------ | ---------------------------- | ------------------------------------------ |
| `DB_HOST` | PostgreSQL host | `localhost` (dev), `projectium.com` (prod) |
| `DB_PORT` | PostgreSQL port | `5432` |
| `DB_USER_PROD` | Production database user | `flyer_crawler_prod` |
| `DB_PASSWORD_PROD` | Production database password | (secret) |
| `DB_DATABASE_PROD` | Production database name | `flyer-crawler-prod` |
| `DB_USER_TEST` | Test database user | `flyer_crawler_test` |
| `DB_PASSWORD_TEST` | Test database password | (secret) |
| `DB_DATABASE_TEST` | Test database name | `flyer-crawler-test` |
| `DB_USER` | Dev database user | `postgres` |
| `DB_PASSWORD` | Dev database password | `postgres` |
| `DB_NAME` | Dev database name | `flyer_crawler_dev` |
### Database Configuration
**Note**: Production and test use separate `_PROD` and `_TEST` suffixed variables. Development uses unsuffixed variables.
| Variable | Required | Default | Description |
| ------------- | -------- | ------- | ----------------- |
| `DB_HOST` | Yes | - | PostgreSQL host |
| `DB_PORT` | No | `5432` | PostgreSQL port |
| `DB_USER` | Yes | - | Database username |
| `DB_PASSWORD` | Yes | - | Database password |
| `DB_NAME` | Yes | - | Database name |
### Redis
**Environment-Specific Variables** (Gitea Secrets):
| Variable | Description | Example |
| --------------------- | ------------------------- | ------------------------------ |
| `REDIS_URL` | Redis connection URL | `redis://localhost:6379` (dev) |
| `REDIS_PASSWORD_PROD` | Production Redis password | (secret) |
| `REDIS_PASSWORD_TEST` | Test Redis password | (secret) |
| Variable | Environment | Description |
| ------------------ | ----------- | ------------------------ |
| `DB_USER_PROD` | Production | Production database user |
| `DB_PASSWORD_PROD` | Production | Production database pass |
| `DB_DATABASE_PROD` | Production | Production database name |
| `DB_USER_TEST` | Test | Test database user |
| `DB_PASSWORD_TEST` | Test | Test database password |
| `DB_DATABASE_TEST` | Test | Test database name |
### Redis Configuration
| Variable | Required | Default | Description |
| ---------------- | -------- | ------- | ------------------------- |
| `REDIS_URL` | Yes | - | Redis connection URL |
| `REDIS_PASSWORD` | No | - | Redis password (optional) |
**URL Format**: `redis://[user:password@]host:port`
**Examples**:
```bash
# Development (no auth)
REDIS_URL=redis://localhost:6379
# Production (with auth)
REDIS_URL=redis://:${REDIS_PASSWORD_PROD}@localhost:6379
```
### Authentication
| Variable | Description | Example |
| ---------------------- | -------------------------- | -------------------------------- |
| `JWT_SECRET` | JWT token signing key | (minimum 32 characters) |
| `SESSION_SECRET` | Session encryption key | (minimum 32 characters) |
| `GOOGLE_CLIENT_ID` | Google OAuth client ID | `xxx.apps.googleusercontent.com` |
| `GOOGLE_CLIENT_SECRET` | Google OAuth client secret | (secret) |
| `GH_CLIENT_ID` | GitHub OAuth client ID | `xxx` |
| `GH_CLIENT_SECRET` | GitHub OAuth client secret | (secret) |
| Variable | Required | Min Length | Description |
| ---------------------- | -------- | ---------- | ----------------------- |
| `JWT_SECRET` | Yes | 32 chars | JWT token signing key |
| `JWT_SECRET_PREVIOUS` | No | - | Previous key (rotation) |
| `GOOGLE_CLIENT_ID` | No | - | Google OAuth client ID |
| `GOOGLE_CLIENT_SECRET` | No | - | Google OAuth secret |
| `GITHUB_CLIENT_ID` | No | - | GitHub OAuth client ID |
| `GITHUB_CLIENT_SECRET` | No | - | GitHub OAuth secret |
**Generate Secure Secret**:
```bash
node -e "console.log(require('crypto').randomBytes(32).toString('hex'))"
```
### AI Services
| Variable | Description | Example |
| -------------------------------- | ---------------------------- | ----------- |
| `VITE_GOOGLE_GENAI_API_KEY` | Google Gemini API key (prod) | `AIzaSy...` |
| `VITE_GOOGLE_GENAI_API_KEY_TEST` | Google Gemini API key (test) | `AIzaSy...` |
| `GOOGLE_MAPS_API_KEY` | Google Maps Geocoding API | `AIzaSy...` |
| Variable | Required | Description |
| ---------------------------- | -------- | -------------------------------- |
| `GEMINI_API_KEY` | Yes\* | Google Gemini API key |
| `GEMINI_RPM` | No | Rate limit (default: 5) |
| `AI_PRICE_QUALITY_THRESHOLD` | No | Quality threshold (default: 0.5) |
### Application
\*Required for flyer processing. Application works without it but cannot extract flyer data.
| Variable | Description | Example |
| -------------- | ------------------------ | ----------------------------------- |
| `NODE_ENV` | Environment mode | `development`, `test`, `production` |
| `PORT` | Backend server port | `3001` |
| `FRONTEND_URL` | Frontend application URL | `http://localhost:5173` (dev) |
**Get API Key**: [Google AI Studio](https://aistudio.google.com/app/apikey)
### Error Tracking
### Google Services
| Variable | Description | Example |
| ---------------------- | -------------------------------- | --------------------------- |
| `SENTRY_DSN` | Sentry DSN (production) | `https://xxx@sentry.io/xxx` |
| `VITE_SENTRY_DSN` | Frontend Sentry DSN (production) | `https://xxx@sentry.io/xxx` |
| `SENTRY_DSN_TEST` | Sentry DSN (test) | `https://xxx@sentry.io/xxx` |
| `VITE_SENTRY_DSN_TEST` | Frontend Sentry DSN (test) | `https://xxx@sentry.io/xxx` |
| `SENTRY_AUTH_TOKEN` | Sentry API token for releases | (secret) |
| Variable | Required | Description |
| ---------------------- | -------- | -------------------------------- |
| `GOOGLE_MAPS_API_KEY` | No | Google Maps Geocoding API |
| `GOOGLE_CLIENT_ID` | No | OAuth (see Authentication above) |
| `GOOGLE_CLIENT_SECRET` | No | OAuth (see Authentication above) |
## Optional Variables
### UPC Lookup APIs
| Variable | Description | Default |
| ------------------- | ----------------------- | ----------------- |
| `LOG_LEVEL` | Logging verbosity | `info` |
| `REDIS_TTL` | Cache TTL in seconds | `3600` |
| `MAX_UPLOAD_SIZE` | Max file upload size | `10mb` |
| `RATE_LIMIT_WINDOW` | Rate limit window (ms) | `900000` (15 min) |
| `RATE_LIMIT_MAX` | Max requests per window | `100` |
| Variable | Required | Description |
| ------------------------ | -------- | ---------------------- |
| `UPC_ITEM_DB_API_KEY` | No | UPC Item DB API key |
| `BARCODE_LOOKUP_API_KEY` | No | Barcode Lookup API key |
### Application Settings
| Variable | Required | Default | Description |
| -------------- | -------- | ------------- | ------------------------ |
| `NODE_ENV` | No | `development` | Environment mode |
| `PORT` | No | `3001` | Backend server port |
| `FRONTEND_URL` | No | - | Frontend URL (CORS) |
| `BASE_URL` | No | - | API base URL |
| `STORAGE_PATH` | No | (see below) | Flyer image storage path |
**NODE_ENV Values**: `development`, `test`, `staging`, `production`
**Default STORAGE_PATH**: `/var/www/flyer-crawler.projectium.com/flyer-images`
### Email/SMTP Configuration
| Variable | Required | Default | Description |
| ----------------- | -------- | ------- | ----------------------- |
| `SMTP_HOST` | No | - | SMTP server hostname |
| `SMTP_PORT` | No | `587` | SMTP server port |
| `SMTP_USER` | No | - | SMTP username |
| `SMTP_PASS` | No | - | SMTP password |
| `SMTP_SECURE` | No | `false` | Use TLS |
| `SMTP_FROM_EMAIL` | No | - | From address for emails |
**Note**: Email functionality degrades gracefully if not configured.
### Worker Configuration
| Variable | Default | Description |
| ------------------------------------- | ------- | ---------------------------- |
| `WORKER_CONCURRENCY` | `1` | Main worker concurrency |
| `WORKER_LOCK_DURATION` | `30000` | Lock duration (ms) |
| `EMAIL_WORKER_CONCURRENCY` | `10` | Email worker concurrency |
| `ANALYTICS_WORKER_CONCURRENCY` | `1` | Analytics worker concurrency |
| `CLEANUP_WORKER_CONCURRENCY` | `10` | Cleanup worker concurrency |
| `WEEKLY_ANALYTICS_WORKER_CONCURRENCY` | `1` | Weekly analytics concurrency |
### Error Tracking (Bugsink/Sentry)
| Variable | Required | Default | Description |
| --------------------- | -------- | -------- | ------------------------------- |
| `SENTRY_DSN` | No | - | Backend Sentry DSN |
| `SENTRY_ENABLED` | No | `true` | Enable error tracking |
| `SENTRY_ENVIRONMENT` | No | NODE_ENV | Environment name for errors |
| `SENTRY_DEBUG` | No | `false` | Enable Sentry SDK debug logging |
| `VITE_SENTRY_DSN` | No | - | Frontend Sentry DSN |
| `VITE_SENTRY_ENABLED` | No | `true` | Enable frontend error tracking |
| `VITE_SENTRY_DEBUG` | No | `false` | Frontend SDK debug logging |
**DSN Format**: `http://[key]@[host]:[port]/[project_id]`
**Dev Container DSNs**:
```bash
# Backend (internal)
SENTRY_DSN=http://<key>@localhost:8000/1
# Frontend (via nginx proxy)
VITE_SENTRY_DSN=https://<key>@localhost/bugsink-api/2
```
---
## Configuration Files
| File | Purpose |
| ------------------------------------- | ------------------------------------------- |
| `src/config/env.ts` | Zod schema validation - **source of truth** |
| `ecosystem.config.cjs` | PM2 process manager config |
| `ecosystem.config.cjs` | PM2 process manager (production) |
| `ecosystem.dev.config.cjs` | PM2 process manager (development) |
| `.gitea/workflows/deploy-to-prod.yml` | Production deployment workflow |
| `.gitea/workflows/deploy-to-test.yml` | Test deployment workflow |
| `.env.example` | Template with all variables |
| `.env.local` | Dev container overrides (not in git) |
| `.env.test` | Test environment overrides (not in git) |
---
## Adding New Variables
### 1. Update Zod Schema
### Checklist
1. [ ] **Update Zod Schema** - Edit `src/config/env.ts`
2. [ ] **Add to Gitea Secrets** - For prod/test environments
3. [ ] **Update Deployment Workflows** - `.gitea/workflows/*.yml`
4. [ ] **Update PM2 Config** - `ecosystem.config.cjs`
5. [ ] **Update .env.example** - Template for developers
6. [ ] **Update this document** - Add to appropriate section
### Step-by-Step
#### 1. Update Zod Schema
Edit `src/config/env.ts`:
```typescript
const envSchema = z.object({
// ... existing variables ...
NEW_VARIABLE: z.string().min(1),
newSection: z.object({
newVariable: z.string().min(1, 'NEW_VARIABLE is required'),
}),
});
// In loadEnvVars():
newSection: {
newVariable: process.env.NEW_VARIABLE,
},
```
### 2. Add to Gitea Secrets
For prod/test environments:
#### 2. Add to Gitea Secrets
1. Go to Gitea repository Settings > Secrets
2. Add `NEW_VARIABLE` with value
2. Add `NEW_VARIABLE` with production value
3. Add `NEW_VARIABLE_TEST` if test needs different value
### 3. Update Deployment Workflows
#### 3. Update Deployment Workflows
Edit `.gitea/workflows/deploy-to-prod.yml`:
@@ -145,7 +270,7 @@ env:
NEW_VARIABLE: ${{ secrets.NEW_VARIABLE_TEST }}
```
### 4. Update PM2 Config
#### 4. Update PM2 Config
Edit `ecosystem.config.cjs`:
@@ -161,31 +286,36 @@ module.exports = {
};
```
### 5. Update Documentation
- Add to `.env.example`
- Update this document
- Document in relevant feature docs
---
## Security Best Practices
### Secrets Management
### Do
- **NEVER** commit secrets to git
- Use Gitea Secrets for prod/test
- Use `.env.local` for dev (gitignored)
- Generate secrets with cryptographic randomness
- Rotate secrets regularly
- Use environment-specific database users
### Do Not
- Commit secrets to git
- Use short or predictable secrets
- Share secrets across environments
- Log sensitive values
### Secret Generation
```bash
# Generate secure random secrets
# Generate secure random secrets (64 hex characters)
node -e "console.log(require('crypto').randomBytes(32).toString('hex'))"
# Example output:
# a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4e5f6a1b2
```
### Database Users
Each environment has its own PostgreSQL user:
### Database Users by Environment
| Environment | User | Database |
| ----------- | -------------------- | -------------------- |
@@ -193,44 +323,61 @@ Each environment has its own PostgreSQL user:
| Test | `flyer_crawler_test` | `flyer-crawler-test` |
| Development | `postgres` | `flyer_crawler_dev` |
**Setup Commands** (as postgres superuser):
```sql
-- Production
CREATE DATABASE "flyer-crawler-prod";
CREATE USER flyer_crawler_prod WITH PASSWORD 'secure-password';
ALTER DATABASE "flyer-crawler-prod" OWNER TO flyer_crawler_prod;
\c "flyer-crawler-prod"
ALTER SCHEMA public OWNER TO flyer_crawler_prod;
GRANT CREATE, USAGE ON SCHEMA public TO flyer_crawler_prod;
CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
CREATE EXTENSION IF NOT EXISTS postgis;
CREATE EXTENSION IF NOT EXISTS pg_trgm;
-- Test (similar commands with _test suffix)
```
---
## Validation
Environment variables are validated at startup via `src/config/env.ts`. If validation fails:
Environment variables are validated at startup via `src/config/env.ts`.
1. Check the error message for missing/invalid variables
2. Verify `.env.local` (dev) or Gitea Secrets (prod/test)
3. Ensure values match schema requirements (min length, format, etc.)
### Startup Validation
If validation fails, you will see:
```text
╔════════════════════════════════════════════════════════════════╗
║ CONFIGURATION ERROR - APPLICATION STARTUP ║
╚════════════════════════════════════════════════════════════════╝
The following environment variables are missing or invalid:
- database.host: DB_HOST is required
- auth.jwtSecret: JWT_SECRET must be at least 32 characters
Please check your .env file or environment configuration.
```
### Debugging Configuration
```bash
# Check what variables are set (dev container)
podman exec flyer-crawler-dev env | grep -E "^(DB_|REDIS_|JWT_|SENTRY_)"
# Test database connection
podman exec flyer-crawler-postgres psql -U postgres -d flyer_crawler_dev -c "SELECT 1;"
# Test Redis connection
podman exec flyer-crawler-redis redis-cli ping
```
---
## Troubleshooting
### Variable Not Found
```
```text
Error: Missing required environment variable: JWT_SECRET
```
**Solution**: Add the variable to your environment configuration.
**Solutions**:
1. Check `.env.local` exists and has the variable
2. Verify variable name matches schema exactly
3. Restart the application after changes
### Invalid Value
```
```text
Error: JWT_SECRET must be at least 32 characters
```
@@ -240,32 +387,36 @@ Error: JWT_SECRET must be at least 32 characters
Check `NODE_ENV` is set correctly:
- `development` - Local dev container
- `test` - CI/CD test server
- `production` - Production server
| Value | Purpose |
| ------------- | ---------------------- |
| `development` | Local dev container |
| `test` | CI/CD test server |
| `staging` | Pre-production testing |
| `production` | Production server |
### Database Connection Issues
Verify database credentials:
```bash
# Development
podman exec flyer-crawler-postgres psql -U postgres -d flyer_crawler_dev -c "SELECT 1;"
# Production (via SSH)
ssh root@projectium.com "psql -U flyer_crawler_prod -d flyer-crawler-prod -c 'SELECT 1;'"
# If connection fails, check:
# 1. Container is running: podman ps
# 2. DB_HOST matches container network
# 3. DB_PASSWORD is correct
```
## Reference
---
- **Validation Schema**: [src/config/env.ts](../../src/config/env.ts)
- **Template**: [.env.example](../../.env.example)
- **Deployment Workflows**: [.gitea/workflows/](../../.gitea/workflows/)
- **PM2 Config**: [ecosystem.config.cjs](../../ecosystem.config.cjs)
## See Also
## Related Documentation
- [QUICKSTART.md](QUICKSTART.md) - Quick setup guide
- [INSTALL.md](INSTALL.md) - Detailed installation
- [DEV-CONTAINER.md](../development/DEV-CONTAINER.md) - Dev container setup
- [DEPLOYMENT.md](../operations/DEPLOYMENT.md) - Production deployment
- [AUTHENTICATION.md](../architecture/AUTHENTICATION.md) - OAuth setup
- [ADR-007](../adr/0007-configuration-and-secrets-management.md) - Configuration decisions
---
Last updated: January 2026

View File

@@ -1,203 +1,453 @@
# Installation Guide
This guide covers setting up a local development environment for Flyer Crawler.
Complete setup instructions for the Flyer Crawler local development environment.
---
## Quick Reference
| Setup Method | Best For | Time | Document Section |
| ----------------- | --------------------------- | ------ | --------------------------------------------------- |
| Quick Start | Already have Postgres/Redis | 5 min | [Quick Start](#quick-start) |
| Dev Container | Full production-like setup | 15 min | [Dev Container](#development-container-recommended) |
| Manual Containers | Learning the components | 20 min | [Podman Setup](#podman-setup-manual) |
---
## Prerequisites
- Node.js 20.x or later
- Access to a PostgreSQL database (local or remote)
- Redis instance (for session management)
- Google Gemini API key
- Google Maps API key (for geocoding)
### Required Software
| Software | Minimum Version | Purpose | Download |
| -------------- | --------------- | -------------------- | ----------------------------------------------- |
| Node.js | 20.x | Runtime | [nodejs.org](https://nodejs.org/) |
| Podman Desktop | 4.x | Container management | [podman-desktop.io](https://podman-desktop.io/) |
| Git | 2.x | Version control | [git-scm.com](https://git-scm.com/) |
### Windows-Specific Requirements
| Requirement | Purpose | Setup Command |
| ----------- | ------------------------------ | ---------------------------------- |
| WSL 2 | Linux compatibility for Podman | `wsl --install` (admin PowerShell) |
### Verify Installation
```bash
# Check all prerequisites
node --version # Expected: v20.x or higher
podman --version # Expected: podman version 4.x or higher
git --version # Expected: git version 2.x or higher
wsl --list -v # Expected: Shows WSL 2 distro
```
---
## Quick Start
If you already have PostgreSQL and Redis configured:
If you already have PostgreSQL and Redis configured externally:
```bash
# Install dependencies
# 1. Clone the repository
git clone https://gitea.projectium.com/flyer-crawler/flyer-crawler.git
cd flyer-crawler
# 2. Install dependencies
npm install
# Run in development mode
# 3. Create .env.local (see Environment section below)
# 4. Run in development mode
npm run dev
```
**Access Points**:
- Frontend: `http://localhost:5173`
- Backend API: `http://localhost:3001`
---
## Development Environment with Podman (Recommended for Windows)
## Development Container (Recommended)
This approach uses Podman with an Ubuntu container for a consistent development environment.
The dev container provides a complete, production-like environment.
### What's Included
| Service | Purpose | Port |
| ---------- | ------------------------ | ---------- |
| Node.js | API server, worker, Vite | 3001, 5173 |
| PostgreSQL | Database with PostGIS | 5432 |
| Redis | Cache and job queues | 6379 |
| NGINX | HTTPS reverse proxy | 443 |
| Bugsink | Error tracking | 8443 |
| Logstash | Log aggregation | - |
| PM2 | Process management | - |
### Setup Steps
#### Step 1: Initialize Podman
```bash
# Windows: Start Podman Desktop, or from terminal:
podman machine init
podman machine start
```
#### Step 2: Start Dev Container
```bash
# Start all services
podman-compose -f compose.dev.yml up -d
# View logs (optional)
podman-compose -f compose.dev.yml logs -f
```
**Expected Output**:
```text
[+] Running 3/3
- Container flyer-crawler-postgres Started
- Container flyer-crawler-redis Started
- Container flyer-crawler-dev Started
```
#### Step 3: Verify Services
```bash
# Check containers are running
podman ps
# Check PM2 processes
podman exec -it flyer-crawler-dev pm2 status
```
**Expected PM2 Status**:
```text
+---------------------------+--------+-------+
| name | status | cpu |
+---------------------------+--------+-------+
| flyer-crawler-api-dev | online | 0% |
| flyer-crawler-worker-dev | online | 0% |
| flyer-crawler-vite-dev | online | 0% |
+---------------------------+--------+-------+
```
#### Step 4: Access Application
| Service | URL | Notes |
| ----------- | ------------------------ | ---------------------------- |
| Frontend | `https://localhost` | NGINX proxies to Vite |
| Backend API | `http://localhost:3001` | Express server |
| Bugsink | `https://localhost:8443` | Login: admin@localhost/admin |
### SSL Certificate Setup (Optional but Recommended)
To eliminate browser security warnings:
**Windows**:
1. Double-click `certs/mkcert-ca.crt`
2. Click "Install Certificate..."
3. Select "Local Machine" > Next
4. Select "Place all certificates in the following store"
5. Browse > Select "Trusted Root Certification Authorities" > OK
6. Click Next > Finish
7. Restart browser
**Other Platforms**: See [`certs/README.md`](../../certs/README.md)
### Managing the Dev Container
| Action | Command |
| --------- | ------------------------------------------- |
| Start | `podman-compose -f compose.dev.yml up -d` |
| Stop | `podman-compose -f compose.dev.yml down` |
| View logs | `podman-compose -f compose.dev.yml logs -f` |
| Restart | `podman-compose -f compose.dev.yml restart` |
| Rebuild | `podman-compose -f compose.dev.yml build` |
---
## Podman Setup (Manual)
For understanding the individual components or custom configurations.
### Step 1: Install Prerequisites on Windows
1. **Install WSL 2**: Podman on Windows relies on the Windows Subsystem for Linux.
```powershell
# Run in administrator PowerShell
wsl --install
```
```powershell
wsl --install
```
Restart computer after WSL installation.
Run this in an administrator PowerShell.
### Step 2: Initialize Podman
2. **Install Podman Desktop**: Download and install [Podman Desktop for Windows](https://podman-desktop.io/).
1. Launch **Podman Desktop**
2. Follow the setup wizard to initialize Podman machine
3. Start the Podman machine
### Step 2: Set Up Podman
1. **Initialize Podman**: Launch Podman Desktop. It will automatically set up its WSL 2 machine.
2. **Start Podman**: Ensure the Podman machine is running from the Podman Desktop interface.
### Step 3: Set Up the Ubuntu Container
1. **Pull Ubuntu Image**:
```bash
podman pull ubuntu:latest
```
2. **Create a Podman Volume** (persists node_modules between container restarts):
```bash
podman volume create node_modules_cache
```
3. **Run the Ubuntu Container**:
Open a terminal in your project's root directory and run:
```bash
podman run -it -p 3001:3001 -p 5173:5173 --name flyer-dev \
-v "$(pwd):/app" \
-v "node_modules_cache:/app/node_modules" \
ubuntu:latest
```
| Flag | Purpose |
| ------------------------------------------- | ------------------------------------------------ |
| `-p 3001:3001` | Forwards the backend server port |
| `-p 5173:5173` | Forwards the Vite frontend server port |
| `--name flyer-dev` | Names the container for easy reference |
| `-v "...:/app"` | Mounts your project directory into the container |
| `-v "node_modules_cache:/app/node_modules"` | Mounts the named volume for node_modules |
### Step 4: Configure the Ubuntu Environment
You are now inside the Ubuntu container's shell.
1. **Update Package Lists**:
```bash
apt-get update
```
2. **Install Dependencies**:
```bash
apt-get install -y curl git
curl -sL https://deb.nodesource.com/setup_20.x | bash -
apt-get install -y nodejs
```
3. **Navigate to Project Directory**:
```bash
cd /app
```
4. **Install Project Dependencies**:
```bash
npm install
```
### Step 5: Run the Development Server
Or from terminal:
```bash
podman machine init
podman machine start
```
### Step 3: Create Podman Network
```bash
podman network create flyer-crawler-net
```
### Step 4: Create PostgreSQL Container
```bash
podman run -d \
--name flyer-crawler-postgres \
--network flyer-crawler-net \
-e POSTGRES_USER=postgres \
-e POSTGRES_PASSWORD=postgres \
-e POSTGRES_DB=flyer_crawler_dev \
-p 5432:5432 \
-v flyer-crawler-pgdata:/var/lib/postgresql/data \
docker.io/postgis/postgis:15-3.3
```
### Step 5: Create Redis Container
```bash
podman run -d \
--name flyer-crawler-redis \
--network flyer-crawler-net \
-p 6379:6379 \
-v flyer-crawler-redis:/data \
docker.io/library/redis:alpine
```
### Step 6: Initialize Database
```bash
# Wait for PostgreSQL to be ready
podman exec flyer-crawler-postgres pg_isready -U postgres
# Install required extensions
podman exec flyer-crawler-postgres psql -U postgres -d flyer_crawler_dev -c "
CREATE EXTENSION IF NOT EXISTS postgis;
CREATE EXTENSION IF NOT EXISTS pg_trgm;
CREATE EXTENSION IF NOT EXISTS \"uuid-ossp\";
"
# Apply schema
podman exec -i flyer-crawler-postgres psql -U postgres -d flyer_crawler_dev < sql/master_schema_rollup.sql
```
### Step 7: Create Node.js Container
```bash
# Create volume for node_modules
podman volume create node_modules_cache
# Run Ubuntu container with project mounted
podman run -it \
--name flyer-dev \
--network flyer-crawler-net \
-p 3001:3001 \
-p 5173:5173 \
-v "$(pwd):/app" \
-v "node_modules_cache:/app/node_modules" \
ubuntu:latest
```
### Step 8: Configure Container Environment
Inside the container:
```bash
# Update and install dependencies
apt-get update
apt-get install -y curl git
# Install Node.js 20
curl -sL https://deb.nodesource.com/setup_20.x | bash -
apt-get install -y nodejs
# Navigate to project and install
cd /app
npm install
# Start development server
npm run dev
```
### Step 6: Access the Application
### Container Management Commands
- **Frontend**: http://localhost:5173
- **Backend API**: http://localhost:3001
### Dev Container with HTTPS (Full Stack)
When using the full dev container stack with NGINX (via `compose.dev.yml`), access the application over HTTPS:
- **Frontend**: https://localhost or https://127.0.0.1
- **Backend API**: http://localhost:3001
**SSL Certificate Notes:**
- The dev container uses self-signed certificates generated by mkcert
- Both `localhost` and `127.0.0.1` are valid hostnames (certificate includes both as SANs)
- If images fail to load with SSL errors, see [FLYER-URL-CONFIGURATION.md](../FLYER-URL-CONFIGURATION.md#ssl-certificate-configuration-dev-container)
**Eliminate SSL Warnings (Recommended):**
To avoid browser security warnings for self-signed certificates, install the mkcert CA certificate on your system. The CA certificate is located at `certs/mkcert-ca.crt` in the project root.
See [`certs/README.md`](../../certs/README.md) for platform-specific installation instructions (Windows, macOS, Linux, Firefox).
After installation:
- Your browser will trust all mkcert certificates without warnings
- Both `https://localhost/` and `https://127.0.0.1/` will work without SSL errors
- Flyer images will load without `ERR_CERT_AUTHORITY_INVALID` errors
### Managing the Container
| Action | Command |
| --------------------- | -------------------------------- |
| Stop the container | Press `Ctrl+C`, then type `exit` |
| Restart the container | `podman start -a -i flyer-dev` |
| Remove the container | `podman rm flyer-dev` |
| Action | Command |
| -------------- | ------------------------------ |
| Stop container | Press `Ctrl+C`, then `exit` |
| Restart | `podman start -a -i flyer-dev` |
| Remove | `podman rm flyer-dev` |
| List running | `podman ps` |
| List all | `podman ps -a` |
---
## Environment Variables
## Environment Configuration
This project is configured to run in a CI/CD environment and does not use `.env` files. All configuration must be provided as environment variables.
### Create .env.local
For local development, you can export these in your shell or use your IDE's environment configuration:
Create `.env.local` in the project root with your configuration:
| Variable | Description |
| --------------------------- | ------------------------------------- |
| `DB_HOST` | PostgreSQL server hostname |
| `DB_USER` | PostgreSQL username |
| `DB_PASSWORD` | PostgreSQL password |
| `DB_DATABASE_PROD` | Production database name |
| `JWT_SECRET` | Secret string for signing auth tokens |
| `VITE_GOOGLE_GENAI_API_KEY` | Google Gemini API key |
| `GOOGLE_MAPS_API_KEY` | Google Maps Geocoding API key |
| `REDIS_PASSWORD_PROD` | Production Redis password |
| `REDIS_PASSWORD_TEST` | Test Redis password |
```bash
# Database (adjust host based on your setup)
DB_HOST=localhost # Use 'postgres' if inside dev container
DB_PORT=5432
DB_USER=postgres
DB_PASSWORD=postgres
DB_NAME=flyer_crawler_dev
# Redis (adjust host based on your setup)
REDIS_URL=redis://localhost:6379 # Use 'redis://redis:6379' inside container
# Application
NODE_ENV=development
PORT=3001
FRONTEND_URL=http://localhost:5173
# Authentication (generate secure values)
JWT_SECRET=your-secret-at-least-32-characters-long
# AI Services
GEMINI_API_KEY=your-google-gemini-api-key
GOOGLE_MAPS_API_KEY=your-google-maps-api-key # Optional
```
**Generate Secure Secrets**:
```bash
node -e "console.log(require('crypto').randomBytes(32).toString('hex'))"
```
### Environment Differences
| Variable | Host Development | Inside Dev Container |
| ----------- | ------------------------ | -------------------- |
| `DB_HOST` | `localhost` | `postgres` |
| `REDIS_URL` | `redis://localhost:6379` | `redis://redis:6379` |
See [ENVIRONMENT.md](ENVIRONMENT.md) for complete variable reference.
---
## Seeding Development Data
To create initial test accounts (`admin@example.com` and `user@example.com`) and sample data:
Create test accounts and sample data:
```bash
npm run seed
```
The seed script performs the following actions:
### What the Seed Script Does
1. Rebuilds the database schema from `sql/master_schema_rollup.sql`
2. Creates test user accounts (admin and regular user)
3. Copies test flyer images from `src/tests/assets/` to `public/flyer-images/`
4. Creates a sample flyer with items linked to the test images
5. Seeds watched items and a shopping list for the test user
1. Rebuilds database schema from `sql/master_schema_rollup.sql`
2. Creates test user accounts:
- `admin@example.com` (admin user)
- `user@example.com` (regular user)
3. Copies test flyer images to `public/flyer-images/`
4. Creates sample flyer with items
5. Seeds watched items and shopping list
**Test Images**: The seed script copies `test-flyer-image.jpg` and `test-flyer-icon.png` to the `public/flyer-images/` directory, which is served by NGINX at `/flyer-images/`.
### Test Images
After running, you may need to restart your IDE's TypeScript server to pick up any generated types.
The seed script copies these files from `src/tests/assets/`:
- `test-flyer-image.jpg`
- `test-flyer-icon.png`
Images are served by NGINX at `/flyer-images/`.
---
## Verification Checklist
After installation, verify everything works:
- [ ] **Containers running**: `podman ps` shows postgres and redis
- [ ] **Database accessible**: `podman exec flyer-crawler-postgres psql -U postgres -c "SELECT 1;"`
- [ ] **Frontend loads**: Open `http://localhost:5173` (or `https://localhost` for dev container)
- [ ] **API responds**: `curl http://localhost:3001/health`
- [ ] **Tests pass**: `npm run test:unit` (or in container: `podman exec -it flyer-crawler-dev npm run test:unit`)
- [ ] **Type check passes**: `npm run type-check`
---
## Troubleshooting
### Podman Machine Won't Start
```bash
# Reset Podman machine
podman machine rm
podman machine init
podman machine start
```
### Port Already in Use
```bash
# Find process using port
netstat -ano | findstr :5432
# Option: Use different port
podman run -d --name flyer-crawler-postgres -p 5433:5432 ...
# Then set DB_PORT=5433 in .env.local
```
### Database Extensions Missing
```bash
podman exec flyer-crawler-postgres psql -U postgres -d flyer_crawler_dev -c "
CREATE EXTENSION IF NOT EXISTS postgis;
CREATE EXTENSION IF NOT EXISTS pg_trgm;
CREATE EXTENSION IF NOT EXISTS \"uuid-ossp\";
"
```
### Permission Denied on Windows Paths
Use `MSYS_NO_PATHCONV=1` prefix:
```bash
MSYS_NO_PATHCONV=1 podman exec flyer-crawler-dev /path/to/script.sh
```
### Tests Fail with Timezone Errors
Tests must run in the dev container, not on Windows host:
```bash
# CORRECT
podman exec -it flyer-crawler-dev npm test
# INCORRECT (may fail with TZ errors)
npm test
```
---
## Next Steps
- [Database Setup](DATABASE.md) - Set up PostgreSQL with required extensions
- [Authentication Setup](AUTHENTICATION.md) - Configure OAuth providers
- [Deployment Guide](DEPLOYMENT.md) - Deploy to production
| Goal | Document |
| --------------------- | ------------------------------------------------------ |
| Quick setup guide | [QUICKSTART.md](QUICKSTART.md) |
| Environment variables | [ENVIRONMENT.md](ENVIRONMENT.md) |
| Database schema | [DATABASE.md](../architecture/DATABASE.md) |
| Authentication setup | [AUTHENTICATION.md](../architecture/AUTHENTICATION.md) |
| Dev container details | [DEV-CONTAINER.md](../development/DEV-CONTAINER.md) |
| Deployment | [DEPLOYMENT.md](../operations/DEPLOYMENT.md) |
---
Last updated: January 2026

View File

@@ -2,13 +2,38 @@
Get Flyer Crawler running in 5 minutes.
## Prerequisites
---
- **Windows 10/11** with WSL 2
- **Podman Desktop** installed
- **Node.js 20+** installed
## Prerequisites Checklist
## 1. Start Containers (1 minute)
Before starting, verify you have:
- [ ] **Windows 10/11** with WSL 2 enabled
- [ ] **Podman Desktop** installed ([download](https://podman-desktop.io/))
- [ ] **Node.js 20+** installed
- [ ] **Git** for cloning the repository
**Verify Prerequisites**:
```bash
# Check Podman
podman --version
# Expected: podman version 4.x or higher
# Check Node.js
node --version
# Expected: v20.x or higher
# Check WSL
wsl --list --verbose
# Expected: Shows WSL 2 distro
```
---
## Quick Setup (5 Steps)
### Step 1: Start Containers (1 minute)
```bash
# Start PostgreSQL and Redis
@@ -27,11 +52,18 @@ podman run -d --name flyer-crawler-redis \
docker.io/library/redis:alpine
```
## 2. Initialize Database (2 minutes)
**Expected Output**:
```text
# Container IDs displayed, no errors
```
### Step 2: Initialize Database (2 minutes)
```bash
# Wait for PostgreSQL to be ready
podman exec flyer-crawler-postgres pg_isready -U postgres
# Expected: localhost:5432 - accepting connections
# Install extensions
podman exec flyer-crawler-postgres psql -U postgres -d flyer_crawler_dev \
@@ -41,7 +73,17 @@ podman exec flyer-crawler-postgres psql -U postgres -d flyer_crawler_dev \
podman exec -i flyer-crawler-postgres psql -U postgres -d flyer_crawler_dev < sql/master_schema_rollup.sql
```
## 3. Configure Environment (1 minute)
**Expected Output**:
```text
CREATE EXTENSION
CREATE EXTENSION
CREATE EXTENSION
CREATE TABLE
... (many tables created)
```
### Step 3: Configure Environment (1 minute)
Create `.env.local` in the project root:
@@ -61,16 +103,22 @@ NODE_ENV=development
PORT=3001
FRONTEND_URL=http://localhost:5173
# Secrets (generate your own)
# Secrets (generate your own - see command below)
JWT_SECRET=your-dev-jwt-secret-at-least-32-chars-long
SESSION_SECRET=your-dev-session-secret-at-least-32-chars-long
# AI Services (get your own keys)
VITE_GOOGLE_GENAI_API_KEY=your-google-genai-api-key
GEMINI_API_KEY=your-google-gemini-api-key
GOOGLE_MAPS_API_KEY=your-google-maps-api-key
```
## 4. Install & Run (1 minute)
**Generate Secure Secrets**:
```bash
node -e "console.log(require('crypto').randomBytes(32).toString('hex'))"
```
### Step 4: Install and Run (1 minute)
```bash
# Install dependencies (first time only)
@@ -80,35 +128,61 @@ npm install
npm run dev
```
## 5. Access Application
**Expected Output**:
- **Frontend**: http://localhost:5173
- **Backend API**: http://localhost:3001
- **Health Check**: http://localhost:3001/health
```text
> flyer-crawler@x.x.x dev
> concurrently ...
### Dev Container (HTTPS)
[API] Server listening on port 3001
[Vite] VITE ready at http://localhost:5173
```
When using the full dev container with NGINX, access via HTTPS:
### Step 5: Verify Installation
- **Frontend**: https://localhost or https://127.0.0.1
- **Backend API**: http://localhost:3001
- **Bugsink**: `https://localhost:8443` (error tracking)
| Check | URL/Command | Expected Result |
| ----------- | ------------------------------ | ----------------------------------- |
| Frontend | `http://localhost:5173` | Flyer Crawler app loads |
| Backend API | `http://localhost:3001/health` | `{ "status": "ok", ... }` |
| Database | `podman exec ... psql -c ...` | `SELECT version()` returns Postgres |
| Containers | `podman ps` | Shows postgres and redis running |
**Note:** The dev container accepts both `localhost` and `127.0.0.1` for HTTPS connections. The self-signed certificate is valid for both hostnames.
---
**SSL Certificate Warnings:** To eliminate browser security warnings for self-signed certificates, install the mkcert CA certificate. See [`certs/README.md`](../../certs/README.md) for platform-specific installation instructions. This is optional but recommended for a better development experience.
## Full Dev Container (Recommended)
### Dev Container Architecture
For a production-like environment with NGINX, Bugsink error tracking, and PM2 process management:
The dev container uses PM2 for process management, matching production (ADR-014):
### Starting the Dev Container
| Process | Description | Port |
| -------------------------- | ------------------------ | ---- |
| `flyer-crawler-api-dev` | API server (tsx watch) | 3001 |
| `flyer-crawler-worker-dev` | Background job worker | - |
| `flyer-crawler-vite-dev` | Vite frontend dev server | 5173 |
```bash
# Start all services
podman-compose -f compose.dev.yml up -d
**PM2 Commands** (run inside container):
# View logs
podman-compose -f compose.dev.yml logs -f
```
### Access Points
| Service | URL | Notes |
| ----------- | ------------------------ | ---------------------------- |
| Frontend | `https://localhost` | NGINX proxy to Vite |
| Backend API | `http://localhost:3001` | Express server |
| Bugsink | `https://localhost:8443` | Error tracking (admin/admin) |
| PostgreSQL | `localhost:5432` | Database |
| Redis | `localhost:6379` | Cache |
**SSL Certificate Setup (Recommended)**:
To eliminate browser security warnings, install the mkcert CA certificate:
```bash
# Windows: Double-click certs/mkcert-ca.crt and install to Trusted Root CAs
# See certs/README.md for detailed instructions per platform
```
### PM2 Commands
```bash
# View process status
@@ -124,63 +198,152 @@ podman exec -it flyer-crawler-dev pm2 restart all
podman exec -it flyer-crawler-dev pm2 restart flyer-crawler-api-dev
```
## Verify Installation
### Dev Container Processes
| Process | Description | Port |
| -------------------------- | ------------------------ | ---- |
| `flyer-crawler-api-dev` | API server (tsx watch) | 3001 |
| `flyer-crawler-worker-dev` | Background job worker | - |
| `flyer-crawler-vite-dev` | Vite frontend dev server | 5173 |
---
## Verification Commands
Run these to confirm everything is working:
```bash
# Check containers are running
podman ps
# Expected: flyer-crawler-postgres and flyer-crawler-redis both running
# Test database connection
podman exec flyer-crawler-postgres psql -U postgres -d flyer_crawler_dev -c "SELECT version();"
# Expected: PostgreSQL 15.x with PostGIS
# Run tests (in dev container)
podman exec -it flyer-crawler-dev npm run test:unit
# Expected: All tests pass
# Run type check
podman exec -it flyer-crawler-dev npm run type-check
# Expected: No type errors
```
## Common Issues
---
## Common Issues and Solutions
### "Unable to connect to Podman socket"
**Cause**: Podman machine not running
**Solution**:
```bash
podman machine start
```
### "Connection refused" to PostgreSQL
Wait a few seconds for PostgreSQL to initialize:
**Cause**: PostgreSQL still initializing
**Solution**:
```bash
# Wait for PostgreSQL to be ready
podman exec flyer-crawler-postgres pg_isready -U postgres
# Retry after "accepting connections" message
```
### Port 5432 or 6379 already in use
Stop conflicting services or change port mappings:
**Cause**: Another service using the port
**Solution**:
```bash
# Use different host port
# Option 1: Stop conflicting service
# Option 2: Use different host port
podman run -d --name flyer-crawler-postgres -p 5433:5432 ...
# Then update DB_PORT=5433 in .env.local
```
Then update `DB_PORT=5433` in `.env.local`.
### "JWT_SECRET must be at least 32 characters"
**Cause**: Secret too short in .env.local
**Solution**: Generate a longer secret:
```bash
node -e "console.log(require('crypto').randomBytes(32).toString('hex'))"
```
### Tests fail with "TZ environment variable" errors
**Cause**: Timezone setting interfering with Node.js async hooks
**Solution**: Tests must run in dev container (not Windows host):
```bash
# CORRECT - run in container
podman exec -it flyer-crawler-dev npm test
# INCORRECT - do not run on Windows host
npm test
```
---
## Next Steps
- **Read the docs**: [docs/README.md](../README.md)
- **Understand the architecture**: [docs/architecture/DATABASE.md](../architecture/DATABASE.md)
- **Learn testing**: [docs/development/TESTING.md](../development/TESTING.md)
- **Explore ADRs**: [docs/adr/index.md](../adr/index.md)
- **Contributing**: [CONTRIBUTING.md](../../CONTRIBUTING.md)
| Goal | Document |
| ----------------------- | ----------------------------------------------------- |
| Understand the codebase | [Architecture Overview](../architecture/OVERVIEW.md) |
| Configure environment | [Environment Variables](ENVIRONMENT.md) |
| Set up MCP tools | [MCP Configuration](../tools/MCP-CONFIGURATION.md) |
| Learn testing | [Testing Guide](../development/TESTING.md) |
| Understand DB schema | [Database Documentation](../architecture/DATABASE.md) |
| Read ADRs | [ADR Index](../adr/index.md) |
| Full installation guide | [Installation Guide](INSTALL.md) |
## Development Workflow
---
## Daily Development Workflow
```bash
# Daily workflow
# 1. Start containers
podman start flyer-crawler-postgres flyer-crawler-redis
# 2. Start dev server
npm run dev
# ... make changes ...
# 3. Make changes and test
npm test
# 4. Type check before commit
npm run type-check
# 5. Commit changes
git commit
```
For detailed setup instructions, see [INSTALL.md](INSTALL.md).
**For dev container users**:
```bash
# 1. Start dev container
podman-compose -f compose.dev.yml up -d
# 2. View logs
podman exec -it flyer-crawler-dev pm2 logs
# 3. Run tests
podman exec -it flyer-crawler-dev npm test
# 4. Stop when done
podman-compose -f compose.dev.yml down
```
---
Last updated: January 2026

View File

@@ -2,8 +2,54 @@
This guide covers the manual installation of Flyer Crawler and its dependencies on a bare-metal Ubuntu server (e.g., a colocation server). This is the definitive reference for setting up a production environment without containers.
**Last verified**: 2026-01-28
**Target Environment**: Ubuntu 22.04 LTS (or newer)
**Related documentation**:
- [ADR-014: Containerization and Deployment Strategy](../adr/0014-containerization-and-deployment-strategy.md)
- [ADR-015: Error Tracking and Observability](../adr/0015-error-tracking-and-observability.md)
- [ADR-050: PostgreSQL Function Observability](../adr/0050-postgresql-function-observability.md)
- [Deployment Guide](DEPLOYMENT.md)
- [Monitoring Guide](MONITORING.md)
---
## Quick Reference
### Installation Time Estimates
| Component | Estimated Time | Notes |
| ----------- | --------------- | ----------------------------- |
| PostgreSQL | 10-15 minutes | Including PostGIS extensions |
| Redis | 5 minutes | Quick install |
| Node.js | 5 minutes | Via NodeSource repository |
| Application | 15-20 minutes | Clone, install, build |
| PM2 | 5 minutes | Global install + config |
| NGINX | 10-15 minutes | Including SSL via Certbot |
| Bugsink | 20-30 minutes | Python venv, systemd services |
| Logstash | 15-20 minutes | Including pipeline config |
| **Total** | **~90 minutes** | For complete fresh install |
### Post-Installation Verification
After completing setup, verify all services:
```bash
# Check all services are running
systemctl status postgresql nginx redis-server gunicorn-bugsink snappea logstash
# Verify application health
curl -s https://flyer-crawler.projectium.com/api/health/ready | jq .
# Check PM2 processes
pm2 list
# Verify Bugsink is accessible
curl -s https://bugsink.projectium.com/accounts/login/ | head -5
```
---
## Server Access Model

View File

@@ -2,6 +2,41 @@
This guide covers deploying Flyer Crawler to a production server.
**Last verified**: 2026-01-28
**Related documentation**:
- [ADR-014: Containerization and Deployment Strategy](../adr/0014-containerization-and-deployment-strategy.md)
- [ADR-015: Error Tracking and Observability](../adr/0015-error-tracking-and-observability.md)
- [Bare-Metal Setup Guide](BARE-METAL-SETUP.md)
- [Monitoring Guide](MONITORING.md)
---
## Quick Reference
### Command Reference Table
| Task | Command |
| -------------------- | ----------------------------------------------------------------------- |
| Deploy to production | Gitea Actions workflow (manual trigger) |
| Deploy to test | Automatic on push to `main` |
| Check PM2 status | `pm2 list` |
| View logs | `pm2 logs flyer-crawler-api --lines 100` |
| Restart all | `pm2 restart all` |
| Check NGINX | `sudo nginx -t && sudo systemctl status nginx` |
| Check health | `curl -s https://flyer-crawler.projectium.com/api/health/ready \| jq .` |
### Deployment URLs
| Environment | URL | API Port |
| ------------- | ------------------------------------------- | -------- |
| Production | `https://flyer-crawler.projectium.com` | 3001 |
| Test | `https://flyer-crawler-test.projectium.com` | 3002 |
| Dev Container | `https://localhost` | 3001 |
---
## Server Access Model
**Important**: Claude Code (and AI tools) have **READ-ONLY** access to production/test servers. The deployment workflow is:
@@ -24,12 +59,24 @@ When troubleshooting deployment issues:
## Prerequisites
- Ubuntu server (22.04 LTS recommended)
- PostgreSQL 14+ with PostGIS extension
- Redis
- Node.js 20.x
- NGINX (reverse proxy)
- PM2 (process manager)
| Component | Version | Purpose |
| ---------- | --------- | ------------------------------- |
| Ubuntu | 22.04 LTS | Operating system |
| PostgreSQL | 14+ | Database with PostGIS extension |
| Redis | 6+ | Caching and job queues |
| Node.js | 20.x LTS | Application runtime |
| NGINX | 1.18+ | Reverse proxy and static files |
| PM2 | Latest | Process manager |
**Verify prerequisites**:
```bash
node --version # Should be v20.x.x
psql --version # Should be 14+
redis-cli ping # Should return PONG
nginx -v # Should be 1.18+
pm2 --version # Any recent version
```
## Dev Container Parity (ADR-014)
@@ -210,7 +257,7 @@ types {
**Option 2**: Edit `/etc/nginx/mime.types` globally:
```
```text
# Change this line:
application/javascript js;
@@ -341,9 +388,78 @@ The Sentry SDK v10+ enforces HTTPS-only DSNs by default. Since Bugsink runs loca
---
## Deployment Troubleshooting
### Decision Tree: Deployment Issues
```text
Deployment failed?
|
+-- Build step failed?
| |
| +-- TypeScript errors --> Fix type issues, run `npm run type-check`
| +-- Missing dependencies --> Run `npm ci`
| +-- Out of memory --> Increase Node heap size
|
+-- Tests failed?
| |
| +-- Database connection --> Check DB_HOST, credentials
| +-- Redis connection --> Check REDIS_URL
| +-- Test isolation --> Check for race conditions
|
+-- SSH/Deploy failed?
|
+-- Permission denied --> Check SSH keys in Gitea secrets
+-- Host unreachable --> Check firewall, VPN
+-- PM2 error --> Check PM2 logs on server
```
### Common Deployment Issues
| Symptom | Diagnosis | Solution |
| ------------------------------------ | ----------------------- | ------------------------------------------------ |
| "Connection refused" on health check | API not started | Check `pm2 logs flyer-crawler-api` |
| 502 Bad Gateway | NGINX cannot reach API | Verify API port (3001), restart PM2 |
| CSS/JS not loading | Build artifacts missing | Re-run `npm run build`, check NGINX static paths |
| Database migrations failed | Schema mismatch | Run migrations manually, check DB connectivity |
| "ENOSPC" error | Disk full | Clear old logs: `pm2 flush`, clean npm cache |
| SSL certificate error | Cert expired/missing | Run `certbot renew`, check NGINX config |
### Post-Deployment Verification Checklist
After every deployment, verify:
- [ ] Health check passes: `curl -s https://flyer-crawler.projectium.com/api/health/ready`
- [ ] PM2 processes running: `pm2 list` shows `online` status
- [ ] No recent errors: Check Bugsink for new issues
- [ ] Frontend loads: Browser shows login page
- [ ] API responds: `curl https://flyer-crawler.projectium.com/api/health/ping`
### Rollback Procedure
If deployment causes issues:
```bash
# 1. Check current release
cd /var/www/flyer-crawler.projectium.com
git log --oneline -5
# 2. Revert to previous commit
git checkout HEAD~1
# 3. Rebuild and restart
npm ci && npm run build
pm2 restart all
# 4. Verify health
curl -s http://localhost:3001/api/health/ready | jq .
```
---
## Related Documentation
- [Database Setup](DATABASE.md) - PostgreSQL and PostGIS configuration
- [Authentication Setup](AUTHENTICATION.md) - OAuth provider configuration
- [Installation Guide](INSTALL.md) - Local development setup
- [Bare-Metal Server Setup](docs/BARE-METAL-SETUP.md) - Manual server installation guide
- [Database Setup](../architecture/DATABASE.md) - PostgreSQL and PostGIS configuration
- [Monitoring Guide](MONITORING.md) - Health checks and error tracking
- [Logstash Quick Reference](LOGSTASH-QUICK-REF.md) - Log aggregation
- [Bare-Metal Server Setup](BARE-METAL-SETUP.md) - Manual server installation guide

View File

@@ -2,10 +2,47 @@
Aggregates logs from PostgreSQL, PM2, Redis, NGINX; forwards errors to Bugsink.
**Last verified**: 2026-01-28
**Related documentation**:
- [ADR-050: PostgreSQL Function Observability](../adr/0050-postgresql-function-observability.md)
- [ADR-015: Error Tracking and Observability](../adr/0015-error-tracking-and-observability.md)
- [Monitoring Guide](MONITORING.md)
- [Logstash Troubleshooting Runbook](LOGSTASH-TROUBLESHOOTING.md)
---
## Quick Reference
### Bugsink Project Routing
| Source Type | Environment | Bugsink Project | Project ID |
| -------------- | ----------- | -------------------- | ---------- |
| PM2 API/Worker | Dev | Backend API (Dev) | 1 |
| PostgreSQL | Dev | Backend API (Dev) | 1 |
| Frontend JS | Dev | Frontend (Dev) | 2 |
| Redis/NGINX | Dev | Infrastructure (Dev) | 4 |
| PM2 API/Worker | Production | Backend API (Prod) | 1 |
| PostgreSQL | Production | Backend API (Prod) | 1 |
| PM2 API/Worker | Test | Backend API (Test) | 3 |
### Key DSN Keys (Dev Container)
| Project | DSN Key |
| -------------------- | ---------------------------------- |
| Backend API (Dev) | `cea01396c56246adb5878fa5ee6b1d22` |
| Frontend (Dev) | `d92663cb73cf4145b677b84029e4b762` |
| Infrastructure (Dev) | `14e8791da3d347fa98073261b596cab9` |
---
## Configuration
**Primary config**: `/etc/logstash/conf.d/bugsink.conf`
**Dev container config**: `docker/logstash/bugsink.conf`
### Related Files
| Path | Purpose |
@@ -89,6 +126,34 @@ MSYS_NO_PATHCONV=1 podman exec flyer-crawler-dev ls -la /var/log/redis/
## Troubleshooting
### Decision Tree: Logs Not Appearing in Bugsink
```text
Errors not showing in Bugsink?
|
+-- Logstash running?
| |
| +-- No --> systemctl start logstash
| +-- Yes --> Check pipeline stats
| |
| +-- Events in = 0?
| | |
| | +-- Log files exist? --> ls /var/log/pm2/*.log
| | +-- Permissions OK? --> groups logstash
| |
| +-- Events filtered = high?
| | |
| | +-- Grok failures --> Check log format matches pattern
| |
| +-- Events out but no Bugsink?
| |
| +-- 403 error --> Wrong DSN key
| +-- 500 error --> Invalid event format (check sentry_level)
| +-- Connection refused --> Bugsink not running
```
### Common Issues Table
| Issue | Check | Solution |
| --------------------- | ---------------- | ---------------------------------------------------------------------------------------------- |
| No Bugsink errors | Logstash running | `systemctl status logstash` |
@@ -103,6 +168,25 @@ MSYS_NO_PATHCONV=1 podman exec flyer-crawler-dev ls -la /var/log/redis/
| High disk usage | Log rotation | Verify `/etc/logrotate.d/logstash` configured |
| varchar(7) error | Level validation | Add Ruby filter to validate/normalize `sentry_level` before output |
### Expected Output Examples
**Successful Logstash pipeline stats**:
```json
{
"in": 1523,
"out": 1520,
"filtered": 1520,
"queue_push_duration_in_millis": 45
}
```
**Healthy Bugsink HTTP response**:
```json
{ "id": "a1b2c3d4e5f6..." }
```
## Related Documentation
- **Dev Container Guide**: [DEV-CONTAINER.md](../development/DEV-CONTAINER.md) - PM2 and log aggregation in dev

View File

@@ -2,6 +2,16 @@
This runbook provides step-by-step diagnostics and solutions for common Logstash issues in the PostgreSQL observability pipeline (ADR-050).
**Last verified**: 2026-01-28
**Related documentation**:
- [ADR-050: PostgreSQL Function Observability](../adr/0050-postgresql-function-observability.md)
- [Logstash Quick Reference](LOGSTASH-QUICK-REF.md)
- [Monitoring Guide](MONITORING.md)
---
## Quick Reference
| Symptom | Most Likely Cause | Quick Check |

View File

@@ -2,6 +2,72 @@
This guide covers all aspects of monitoring the Flyer Crawler application across development, test, and production environments.
**Last verified**: 2026-01-28
**Related documentation**:
- [ADR-015: Error Tracking and Observability](../adr/0015-error-tracking-and-observability.md)
- [ADR-020: Health Checks](../adr/0020-health-checks-and-liveness-readiness-probes.md)
- [ADR-050: PostgreSQL Function Observability](../adr/0050-postgresql-function-observability.md)
- [Logstash Quick Reference](LOGSTASH-QUICK-REF.md)
- [Deployment Guide](DEPLOYMENT.md)
---
## Quick Reference
### Monitoring URLs
| Service | Production URL | Dev Container URL |
| ------------ | ------------------------------------------------------- | ---------------------------------------- |
| Health Check | `https://flyer-crawler.projectium.com/api/health/ready` | `http://localhost:3001/api/health/ready` |
| Bugsink | `https://bugsink.projectium.com` | `https://localhost:8443` |
| Bull Board | `https://flyer-crawler.projectium.com/api/admin/jobs` | `http://localhost:3001/api/admin/jobs` |
### Quick Diagnostic Commands
```bash
# Check all services at once (production)
curl -s https://flyer-crawler.projectium.com/api/health/ready | jq '.data.services'
# Dev container health check
podman exec flyer-crawler-dev curl -s http://localhost:3001/api/health/ready | jq .
# PM2 process overview
pm2 list
# Recent errors in Bugsink (via MCP)
# mcp__bugsink__list_issues --project_id 1 --status unresolved
```
### Monitoring Decision Tree
```text
Application seems slow or unresponsive?
|
+-- Check health endpoint first
| |
| +-- Returns unhealthy?
| | |
| | +-- Database unhealthy --> Check DB pool, connections
| | +-- Redis unhealthy --> Check Redis memory, connection
| | +-- Storage unhealthy --> Check disk space, permissions
| |
| +-- Returns healthy but slow?
| |
| +-- Check PM2 memory/CPU usage
| +-- Check database slow query log
| +-- Check Redis queue depth
|
+-- Health endpoint not responding?
|
+-- Check PM2 status --> Process crashed?
+-- Check NGINX --> 502 errors?
+-- Check network --> Firewall/DNS issues?
```
---
## Table of Contents
1. [Health Checks](#health-checks)
@@ -294,7 +360,7 @@ The command outputs a 40-character hex token.
**Error Anatomy**:
```
```text
TypeError: Cannot read properties of undefined (reading 'map')
├── Exception Type: TypeError
├── Message: Cannot read properties of undefined (reading 'map')
@@ -357,7 +423,7 @@ Logstash aggregates logs from multiple sources and forwards errors to Bugsink (A
### Architecture
```
```text
Log Sources Logstash Outputs
┌──────────────┐ ┌─────────────┐ ┌─────────────┐
│ PostgreSQL │──────────────│ │───────────│ Bugsink │
@@ -520,7 +586,7 @@ pm2 stop flyer-crawler-api
**Healthy Process**:
```
```text
┌─────────────────────┬────┬─────────┬─────────┬───────┬────────┬─────────┬──────────┐
│ Name │ id │ mode │ status │ cpu │ mem │ uptime │ restarts │
├─────────────────────┼────┼─────────┼─────────┼───────┼────────┼─────────┼──────────┤
@@ -833,7 +899,7 @@ Configure alerts in your monitoring tool (UptimeRobot, Datadog, etc.):
2. Review during business hours
3. Create Gitea issue for tracking
### Quick Diagnostic Commands
### On-Call Diagnostic Commands
> **Note**: User executes these commands on the server. Claude Code provides commands but cannot run them directly.

View File

@@ -2,6 +2,17 @@
The **ai-usage** subagent specializes in LLM APIs (Gemini, Claude), prompt engineering, and AI-powered features in the Flyer Crawler project.
## Quick Reference
| Aspect | Details |
| ------------------ | ----------------------------------------------------------------------------------- |
| **Primary Use** | Gemini API integration, prompt engineering, AI extraction |
| **Key Files** | `src/services/aiService.server.ts`, `src/services/flyerProcessingService.server.ts` |
| **Key ADRs** | ADR-041 (AI Integration), ADR-046 (Image Processing) |
| **API Key Env** | `VITE_GOOGLE_GENAI_API_KEY` (prod), `VITE_GOOGLE_GENAI_API_KEY_TEST` (test) |
| **Error Handling** | Rate limits (429), JSON parse errors, timeout handling |
| **Delegate To** | `coder` (implementation), `testwriter` (tests), `integrations-specialist` |
## When to Use
Use the **ai-usage** subagent when you need to:
@@ -295,6 +306,9 @@ const fixtureResponse = await fs.readFile('fixtures/gemini-response.json');
## Related Documentation
- [OVERVIEW.md](./OVERVIEW.md) - Subagent system overview
- [CODER-GUIDE.md](./CODER-GUIDE.md) - For implementing AI features
- [TESTER-GUIDE.md](./TESTER-GUIDE.md) - Testing AI features
- [INTEGRATIONS-GUIDE.md](./INTEGRATIONS-GUIDE.md) - External API patterns
- [../adr/0041-ai-gemini-integration-architecture.md](../adr/0041-ai-gemini-integration-architecture.md) - AI integration ADR
- [../adr/0046-image-processing-pipeline.md](../adr/0046-image-processing-pipeline.md) - Image processing
- [CODER-GUIDE.md](./CODER-GUIDE.md) - For implementing AI features
- [../getting-started/ENVIRONMENT.md](../getting-started/ENVIRONMENT.md) - Environment configuration

View File

@@ -2,6 +2,17 @@
The **coder** subagent is your primary tool for writing and modifying production Node.js/TypeScript code in the Flyer Crawler project. This guide explains how to work effectively with the coder subagent.
## Quick Reference
| Aspect | Details |
| ---------------- | ------------------------------------------------------------------------ |
| **Primary Use** | Write/modify production TypeScript code |
| **Key Files** | `src/routes/*.routes.ts`, `src/services/**/*.ts`, `src/components/*.tsx` |
| **Key ADRs** | ADR-034 (Repository), ADR-035 (Services), ADR-028 (API Response) |
| **Test Command** | `podman exec -it flyer-crawler-dev npm run test:unit` |
| **Type Check** | `podman exec -it flyer-crawler-dev npm run type-check` |
| **Delegate To** | `db-dev` (database), `frontend-specialist` (UI), `testwriter` (tests) |
## When to Use the Coder Subagent
Use the coder subagent when you need to:
@@ -307,6 +318,8 @@ error classes for all database operations"
- [OVERVIEW.md](./OVERVIEW.md) - Subagent system overview
- [TESTER-GUIDE.md](./TESTER-GUIDE.md) - Testing strategies
- [DATABASE-GUIDE.md](./DATABASE-GUIDE.md) - Database development workflows
- [../adr/0034-repository-pattern-standards.md](../adr/0034-repository-pattern-standards.md) - Repository patterns
- [../adr/0035-service-layer-architecture.md](../adr/0035-service-layer-architecture.md) - Service layer architecture
- [../adr/0028-api-response-standardization.md](../adr/0028-api-response-standardization.md) - API response patterns
- [../development/CODE-PATTERNS.md](../development/CODE-PATTERNS.md) - Code patterns reference

View File

@@ -5,6 +5,17 @@ This guide covers two database-focused subagents:
- **db-dev**: Database development - schemas, queries, migrations, optimization
- **db-admin**: Database administration - PostgreSQL/Redis admin, security, backups
## Quick Reference
| Aspect | db-dev | db-admin |
| ---------------- | -------------------------------------------- | ------------------------------------------ |
| **Primary Use** | Schemas, queries, migrations | Performance tuning, backups, security |
| **Key Files** | `src/services/db/*.db.ts`, `sql/migrations/` | `postgresql.conf`, `pg_hba.conf` |
| **Key ADRs** | ADR-034 (Repository), ADR-002 (Transactions) | ADR-019 (Backups), ADR-050 (Observability) |
| **Test Command** | `podman exec -it flyer-crawler-dev npm test` | N/A |
| **MCP Tool** | `mcp__devdb__query` | SSH to production |
| **Delegate To** | `coder` (service layer), `db-admin` (perf) | `devops` (infrastructure) |
## Understanding the Difference
| Aspect | db-dev | db-admin |
@@ -412,8 +423,9 @@ This is useful for:
- [OVERVIEW.md](./OVERVIEW.md) - Subagent system overview
- [CODER-GUIDE.md](./CODER-GUIDE.md) - Working with the coder subagent
- [DEVOPS-GUIDE.md](./DEVOPS-GUIDE.md) - DevOps and deployment workflows
- [../adr/0034-repository-pattern-standards.md](../adr/0034-repository-pattern-standards.md) - Repository patterns
- [../adr/0002-standardized-transaction-management.md](../adr/0002-standardized-transaction-management.md) - Transaction management
- [../adr/0019-data-backup-and-recovery-strategy.md](../adr/0019-data-backup-and-recovery-strategy.md) - Backup strategy
- [../adr/0050-postgresql-function-observability.md](../adr/0050-postgresql-function-observability.md) - Database observability
- [../BARE-METAL-SETUP.md](../BARE-METAL-SETUP.md) - Production database setup
- [../operations/BARE-METAL-SETUP.md](../operations/BARE-METAL-SETUP.md) - Production database setup

View File

@@ -6,6 +6,17 @@ This guide covers DevOps-related subagents for deployment, infrastructure, and o
- **infra-architect**: Resource optimization, capacity planning
- **bg-worker**: Background jobs, PM2 workers, BullMQ queues
## Quick Reference
| Aspect | devops | infra-architect | bg-worker |
| ---------------- | ------------------------------------------ | --------------------------- | ------------------------------- |
| **Primary Use** | Containers, CI/CD, deployments | Resource optimization | BullMQ queues, PM2 workers |
| **Key Files** | `compose.dev.yml`, `.gitea/workflows/` | `ecosystem.config.cjs` | `src/services/queues.server.ts` |
| **Key ADRs** | ADR-014 (Containers), ADR-017 (CI/CD) | N/A | ADR-006 (Background Jobs) |
| **Commands** | `podman-compose`, `pm2` | `pm2 monit`, system metrics | Redis CLI, `pm2 logs` |
| **MCP Tools** | `mcp__podman__*` | N/A | N/A |
| **Access Model** | Read-only on production (provide commands) | Same | Same |
---
## CRITICAL: Server Access Model
@@ -543,8 +554,13 @@ podman exec -it flyer-crawler-dev npm test
## Related Documentation
- [OVERVIEW.md](./OVERVIEW.md) - Subagent system overview
- [../BARE-METAL-SETUP.md](../BARE-METAL-SETUP.md) - Production setup guide
- [DATABASE-GUIDE.md](./DATABASE-GUIDE.md) - Database administration
- [SECURITY-DEBUG-GUIDE.md](./SECURITY-DEBUG-GUIDE.md) - Production debugging
- [../operations/BARE-METAL-SETUP.md](../operations/BARE-METAL-SETUP.md) - Production setup guide
- [../operations/DEPLOYMENT.md](../operations/DEPLOYMENT.md) - Deployment guide
- [../operations/MONITORING.md](../operations/MONITORING.md) - Monitoring guide
- [../development/DEV-CONTAINER.md](../development/DEV-CONTAINER.md) - Dev container guide
- [../adr/0014-containerization-and-deployment-strategy.md](../adr/0014-containerization-and-deployment-strategy.md) - Containerization ADR
- [../adr/0006-background-job-processing-and-task-queues.md](../adr/0006-background-job-processing-and-task-queues.md) - Background jobs ADR
- [../adr/0017-ci-cd-and-branching-strategy.md](../adr/0017-ci-cd-and-branching-strategy.md) - CI/CD strategy
- [../adr/0053-worker-health-checks.md](../adr/0053-worker-health-checks.md) - Worker health checks
- [../adr/0053-worker-health-checks-and-monitoring.md](../adr/0053-worker-health-checks-and-monitoring.md) - Worker health checks

View File

@@ -7,6 +7,15 @@ This guide covers documentation-focused subagents:
- **planner**: Feature breakdown, roadmaps, scope management
- **product-owner**: Requirements, user stories, backlog prioritization
## Quick Reference
| Aspect | documenter | describer-for-ai | planner | product-owner |
| --------------- | -------------------- | ------------------------ | --------------------- | ---------------------- |
| **Primary Use** | User docs, API specs | ADRs, technical specs | Feature breakdown | User stories, backlog |
| **Key Files** | `docs/`, API docs | `docs/adr/`, `CLAUDE.md` | `docs/plans/` | Issue tracker |
| **Output** | Markdown guides | ADRs, context docs | Task lists, roadmaps | User stories, criteria |
| **Delegate To** | `coder` (implement) | `documenter` (user docs) | `coder` (build tasks) | `planner` (breakdown) |
## The documenter Subagent
### When to Use
@@ -437,6 +446,8 @@ Include dates on documentation that may become stale:
## Related Documentation
- [OVERVIEW.md](./OVERVIEW.md) - Subagent system overview
- [CODER-GUIDE.md](./CODER-GUIDE.md) - For implementing documented features
- [../adr/index.md](../adr/index.md) - ADR index
- [../TESTING.md](../TESTING.md) - Testing guide
- [../development/TESTING.md](../development/TESTING.md) - Testing guide
- [../development/CODE-PATTERNS.md](../development/CODE-PATTERNS.md) - Code patterns reference
- [../../CLAUDE.md](../../CLAUDE.md) - AI instructions

View File

@@ -5,6 +5,17 @@ This guide covers frontend-focused subagents:
- **frontend-specialist**: UI components, Neo-Brutalism, Core Web Vitals, accessibility
- **uiux-designer**: UI/UX decisions, component design, user experience
## Quick Reference
| Aspect | frontend-specialist | uiux-designer |
| ----------------- | ---------------------------------------------- | -------------------------------------- |
| **Primary Use** | React components, performance, accessibility | Design decisions, user flows |
| **Key Files** | `src/components/`, `src/features/` | Design specs, mockups |
| **Key ADRs** | ADR-012 (Design System), ADR-044 (Feature Org) | ADR-012 (Design System) |
| **Design System** | Neo-Brutalism (bold borders, high contrast) | Same |
| **State Mgmt** | TanStack Query (server), Zustand (client) | N/A |
| **Delegate To** | `coder` (backend), `tester` (test coverage) | `frontend-specialist` (implementation) |
## The frontend-specialist Subagent
### When to Use
@@ -406,7 +417,8 @@ const handleSelect = useCallback((id: string) => {
- [OVERVIEW.md](./OVERVIEW.md) - Subagent system overview
- [CODER-GUIDE.md](./CODER-GUIDE.md) - For implementing features
- [../DESIGN_TOKENS.md](../DESIGN_TOKENS.md) - Design token reference
- [TESTER-GUIDE.md](./TESTER-GUIDE.md) - Component testing patterns
- [../development/DESIGN_TOKENS.md](../development/DESIGN_TOKENS.md) - Design token reference
- [../adr/0012-frontend-component-library-and-design-system.md](../adr/0012-frontend-component-library-and-design-system.md) - Design system ADR
- [../adr/0005-frontend-state-management-and-server-cache-strategy.md](../adr/0005-frontend-state-management-and-server-cache-strategy.md) - State management ADR
- [../adr/0044-frontend-feature-organization.md](../adr/0044-frontend-feature-organization.md) - Feature organization

View File

@@ -0,0 +1,396 @@
# Integrations Subagent Guide
The **integrations-specialist** subagent handles third-party services, webhooks, and external API integrations in the Flyer Crawler project.
## Quick Reference
| Aspect | Details |
| --------------- | --------------------------------------------------------------------------- |
| **Primary Use** | External APIs, webhooks, OAuth, third-party services |
| **Key Files** | `src/services/external/`, `src/routes/webhooks.routes.ts` |
| **Key ADRs** | ADR-041 (AI Integration), ADR-016 (API Security), ADR-048 (Auth) |
| **MCP Tools** | `mcp__gitea-projectium__*`, `mcp__bugsink__*` |
| **Security** | API key storage, webhook signatures, OAuth state param |
| **Delegate To** | `coder` (implementation), `security-engineer` (review), `ai-usage` (Gemini) |
## When to Use
Use the **integrations-specialist** subagent when you need to:
- Integrate with external APIs (OAuth, REST, GraphQL)
- Implement webhook handlers
- Configure third-party services
- Debug external service connectivity
- Handle API authentication flows
- Manage external service rate limits
## What integrations-specialist Knows
The integrations-specialist subagent understands:
- OAuth 2.0 flows (authorization code, client credentials)
- REST API integration patterns
- Webhook security (signature verification)
- External service error handling
- Rate limiting and retry strategies
- API key management
## Current Integrations
| Service | Purpose | Integration Type | Key Files |
| ------------- | ---------------------- | ---------------- | ---------------------------------- |
| Google Gemini | AI flyer extraction | REST API | `src/services/aiService.server.ts` |
| Bugsink | Error tracking | REST API | MCP: `mcp__bugsink__*` |
| Gitea | Repository and CI/CD | REST API | MCP: `mcp__gitea-projectium__*` |
| Redis | Caching and job queues | Native client | `src/services/redis.server.ts` |
| PostgreSQL | Primary database | Native client | `src/services/db/pool.db.ts` |
## Example Requests
### Adding External API Integration
```
"Use integrations-specialist to integrate with the Store API
to automatically fetch store location data. Include proper
error handling, rate limiting, and caching."
```
### OAuth Implementation
```
"Use integrations-specialist to implement Google OAuth for
user authentication. Include token refresh handling and
session management."
```
### Webhook Handler
```
"Use integrations-specialist to create a webhook handler for
receiving store inventory updates. Include signature verification
and idempotency handling."
```
### Debugging External Service Issues
```
"Use integrations-specialist to debug why the Gemini API calls
are intermittently failing with timeout errors. Check connection
pooling, retry logic, and error handling."
```
## Integration Patterns
### REST API Client Pattern
```typescript
// src/services/external/storeApi.server.ts
import { env } from '@/config/env';
import { log } from '@/services/logger.server';
interface StoreApiConfig {
baseUrl: string;
apiKey: string;
timeout: number;
}
class StoreApiClient {
private config: StoreApiConfig;
constructor(config: StoreApiConfig) {
this.config = config;
}
async getStoreLocations(storeId: string): Promise<StoreLocation[]> {
const url = `${this.config.baseUrl}/stores/${storeId}/locations`;
try {
const response = await fetch(url, {
headers: {
Authorization: `Bearer ${this.config.apiKey}`,
'Content-Type': 'application/json',
},
signal: AbortSignal.timeout(this.config.timeout),
});
if (!response.ok) {
throw new ExternalApiError(`Store API error: ${response.status}`, response.status);
}
return response.json();
} catch (error) {
log.error({ error, storeId }, 'Failed to fetch store locations');
throw error;
}
}
}
export const storeApiClient = new StoreApiClient({
baseUrl: env.STORE_API_BASE_URL,
apiKey: env.STORE_API_KEY,
timeout: 10000,
});
```
### Webhook Handler Pattern
```typescript
// src/routes/webhooks.routes.ts
import { Router } from 'express';
import crypto from 'crypto';
import { env } from '@/config/env';
const router = Router();
function verifyWebhookSignature(payload: string, signature: string, secret: string): boolean {
const expected = crypto.createHmac('sha256', secret).update(payload).digest('hex');
return crypto.timingSafeEqual(Buffer.from(signature), Buffer.from(`sha256=${expected}`));
}
router.post('/store-updates', async (req, res, next) => {
try {
const signature = req.headers['x-webhook-signature'] as string;
const payload = JSON.stringify(req.body);
if (!verifyWebhookSignature(payload, signature, env.WEBHOOK_SECRET)) {
return res.status(401).json({ error: 'Invalid signature' });
}
// Process webhook with idempotency check
const eventId = req.headers['x-event-id'] as string;
const alreadyProcessed = await checkIdempotencyKey(eventId);
if (alreadyProcessed) {
return res.status(200).json({ status: 'already_processed' });
}
await processStoreUpdate(req.body);
await markEventProcessed(eventId);
res.status(200).json({ status: 'processed' });
} catch (error) {
next(error);
}
});
```
### OAuth Flow Pattern
```typescript
// src/services/oauth/googleOAuth.server.ts
import { OAuth2Client } from 'google-auth-library';
import { env } from '@/config/env';
const oauth2Client = new OAuth2Client(
env.GOOGLE_CLIENT_ID,
env.GOOGLE_CLIENT_SECRET,
env.GOOGLE_REDIRECT_URI,
);
export function getAuthorizationUrl(): string {
return oauth2Client.generateAuthUrl({
access_type: 'offline',
scope: ['email', 'profile'],
prompt: 'consent',
});
}
export async function exchangeCodeForTokens(code: string) {
const { tokens } = await oauth2Client.getToken(code);
return tokens;
}
export async function refreshAccessToken(refreshToken: string) {
oauth2Client.setCredentials({ refresh_token: refreshToken });
const { credentials } = await oauth2Client.refreshAccessToken();
return credentials;
}
```
## Error Handling for External Services
### Custom Error Classes
```typescript
// src/services/external/errors.ts
export class ExternalApiError extends Error {
constructor(
message: string,
public statusCode: number,
public retryable: boolean = false,
) {
super(message);
this.name = 'ExternalApiError';
}
}
export class RateLimitError extends ExternalApiError {
constructor(
message: string,
public retryAfter: number,
) {
super(message, 429, true);
this.name = 'RateLimitError';
}
}
```
### Retry with Exponential Backoff
```typescript
async function fetchWithRetry<T>(
fn: () => Promise<T>,
options: { maxRetries: number; baseDelay: number },
): Promise<T> {
let lastError: Error;
for (let attempt = 0; attempt <= options.maxRetries; attempt++) {
try {
return await fn();
} catch (error) {
lastError = error as Error;
if (error instanceof ExternalApiError && !error.retryable) {
throw error;
}
if (attempt < options.maxRetries) {
const delay = options.baseDelay * Math.pow(2, attempt);
await new Promise((resolve) => setTimeout(resolve, delay));
}
}
}
throw lastError!;
}
```
## Rate Limiting Strategies
### Token Bucket Pattern
```typescript
class RateLimiter {
private tokens: number;
private lastRefill: number;
private readonly maxTokens: number;
private readonly refillRate: number; // tokens per second
constructor(maxTokens: number, refillRate: number) {
this.maxTokens = maxTokens;
this.tokens = maxTokens;
this.refillRate = refillRate;
this.lastRefill = Date.now();
}
async acquire(): Promise<void> {
this.refill();
if (this.tokens < 1) {
const waitTime = ((1 - this.tokens) / this.refillRate) * 1000;
await new Promise((resolve) => setTimeout(resolve, waitTime));
this.refill();
}
this.tokens -= 1;
}
private refill(): void {
const now = Date.now();
const elapsed = (now - this.lastRefill) / 1000;
this.tokens = Math.min(this.maxTokens, this.tokens + elapsed * this.refillRate);
this.lastRefill = now;
}
}
```
## Testing Integrations
### Mocking External Services
```typescript
// src/tests/mocks/storeApi.mock.ts
import { vi } from 'vitest';
export const mockStoreApiClient = {
getStoreLocations: vi.fn(),
};
vi.mock('@/services/external/storeApi.server', () => ({
storeApiClient: mockStoreApiClient,
}));
```
### Integration Test with Real Service
```typescript
// src/tests/integration/storeApi.integration.test.ts
describe('Store API Integration', () => {
it.skipIf(!env.STORE_API_KEY)('fetches real store locations', async () => {
const locations = await storeApiClient.getStoreLocations('test-store');
expect(locations).toBeInstanceOf(Array);
});
});
```
## MCP Tools for Integrations
### Gitea Integration
```
// List repositories
mcp__gitea-projectium__list_my_repos()
// Create issue
mcp__gitea-projectium__create_issue({
owner: "projectium",
repo: "flyer-crawler",
title: "Issue title",
body: "Issue description"
})
```
### Bugsink Integration
```
// List projects
mcp__bugsink__list_projects()
// Get issue details
mcp__bugsink__get_issue({ issue_id: "..." })
// Get stacktrace
mcp__bugsink__get_stacktrace({ event_id: "..." })
```
## Security Considerations
### API Key Storage
- Never commit API keys to version control
- Use environment variables via `src/config/env.ts`
- Rotate keys periodically
- Use separate keys for dev/test/prod
### Webhook Security
- Always verify webhook signatures
- Use HTTPS for webhook endpoints
- Implement idempotency
- Log webhook events for audit
### OAuth Security
- Use state parameter to prevent CSRF
- Store tokens securely (encrypted at rest)
- Implement token refresh before expiration
- Validate token scopes
## Related Documentation
- [OVERVIEW.md](./OVERVIEW.md) - Subagent system overview
- [SECURITY-DEBUG-GUIDE.md](./SECURITY-DEBUG-GUIDE.md) - Security patterns
- [AI-USAGE-GUIDE.md](./AI-USAGE-GUIDE.md) - Gemini API integration
- [../adr/0041-ai-gemini-integration-architecture.md](../adr/0041-ai-gemini-integration-architecture.md) - AI integration ADR
- [../adr/0016-api-security-hardening.md](../adr/0016-api-security-hardening.md) - API security
- [../adr/0048-authentication-strategy.md](../adr/0048-authentication-strategy.md) - Authentication

View File

@@ -89,6 +89,47 @@ Or:
Claude will automatically invoke the appropriate subagent with the relevant context.
## Quick Reference Decision Tree
Use this flowchart to quickly identify the right subagent:
```
What do you need to do?
|
+-- Write/modify code? ----------------> Is it database-related?
| |
| +-- Yes -> db-dev
| +-- No --> Is it frontend?
| |
| +-- Yes -> frontend-specialist
| +-- No --> Is it AI/Gemini?
| |
| +-- Yes -> ai-usage
| +-- No --> coder
|
+-- Test something? -------------------> Write new tests? -> testwriter
| Find bugs/vulnerabilities? -> tester
| Review existing code? -> code-reviewer
|
+-- Debug an issue? -------------------> Production error? -> log-debug
| Database slow? -> db-admin
| External API failing? -> integrations-specialist
| AI extraction failing? -> ai-usage
|
+-- Infrastructure/Deployment? --------> Container/CI/CD? -> devops
| Resource optimization? -> infra-architect
| Background jobs? -> bg-worker
|
+-- Documentation? --------------------> User-facing docs? -> documenter
| ADRs/Technical specs? -> describer-for-ai
| Feature planning? -> planner
| User stories? -> product-owner
|
+-- Security? -------------------------> security-engineer
|
+-- Design/UX? ------------------------> uiux-designer
```
## Subagent Selection Guide
### Which Subagent Should I Use?
@@ -183,12 +224,26 @@ Subagents can pass information back to the main conversation and to each other t
## Related Documentation
- [CODER-GUIDE.md](./CODER-GUIDE.md) - Working with the coder subagent
- [TESTER-GUIDE.md](./TESTER-GUIDE.md) - Testing strategies and patterns
- [DATABASE-GUIDE.md](./DATABASE-GUIDE.md) - Database development workflows
- [DEVOPS-GUIDE.md](./DEVOPS-GUIDE.md) - DevOps and deployment workflows
### Subagent Guides
| Guide | Subagents Covered |
| ---------------------------------------------------- | ----------------------------------------------------- |
| [CODER-GUIDE.md](./CODER-GUIDE.md) | coder |
| [TESTER-GUIDE.md](./TESTER-GUIDE.md) | tester, testwriter |
| [DATABASE-GUIDE.md](./DATABASE-GUIDE.md) | db-dev, db-admin |
| [DEVOPS-GUIDE.md](./DEVOPS-GUIDE.md) | devops, infra-architect, bg-worker |
| [FRONTEND-GUIDE.md](./FRONTEND-GUIDE.md) | frontend-specialist, uiux-designer |
| [SECURITY-DEBUG-GUIDE.md](./SECURITY-DEBUG-GUIDE.md) | security-engineer, log-debug, code-reviewer |
| [AI-USAGE-GUIDE.md](./AI-USAGE-GUIDE.md) | ai-usage |
| [INTEGRATIONS-GUIDE.md](./INTEGRATIONS-GUIDE.md) | integrations-specialist, tools-integration-specialist |
| [DOCUMENTATION-GUIDE.md](./DOCUMENTATION-GUIDE.md) | documenter, describer-for-ai, planner, product-owner |
### Project Documentation
- [../adr/index.md](../adr/index.md) - Architecture Decision Records
- [../TESTING.md](../TESTING.md) - Testing guide
- [../development/TESTING.md](../development/TESTING.md) - Testing guide
- [../development/CODE-PATTERNS.md](../development/CODE-PATTERNS.md) - Code patterns reference
- [../architecture/OVERVIEW.md](../architecture/OVERVIEW.md) - System architecture
## Troubleshooting

View File

@@ -6,6 +6,16 @@ This guide covers security and debugging-focused subagents:
- **log-debug**: Production errors, observability, Bugsink/Sentry analysis
- **code-reviewer**: Code quality, security review, best practices
## Quick Reference
| Aspect | security-engineer | log-debug | code-reviewer |
| --------------- | ---------------------------------- | ---------------------------------------- | --------------------------- |
| **Primary Use** | Security audits, OWASP | Production debugging | Code quality review |
| **Key ADRs** | ADR-016 (Security), ADR-032 (Rate) | ADR-050 (Observability) | ADR-034, ADR-035 (Patterns) |
| **MCP Tools** | N/A | `mcp__bugsink__*`, `mcp__localerrors__*` | N/A |
| **Key Checks** | Auth, input validation, CORS | Logs, stacktraces, error patterns | Patterns, tests, security |
| **Delegate To** | `coder` (fix issues) | `devops` (infra), `coder` (fixes) | `coder`, `testwriter` |
## The security-engineer Subagent
### When to Use
@@ -432,8 +442,10 @@ tail -f /var/log/postgresql/postgresql-$(date +%Y-%m-%d).log | grep "duration:"
- [OVERVIEW.md](./OVERVIEW.md) - Subagent system overview
- [DEVOPS-GUIDE.md](./DEVOPS-GUIDE.md) - Infrastructure debugging
- [TESTER-GUIDE.md](./TESTER-GUIDE.md) - Security testing
- [../adr/0016-api-security-hardening.md](../adr/0016-api-security-hardening.md) - Security ADR
- [../adr/0032-rate-limiting-strategy.md](../adr/0032-rate-limiting-strategy.md) - Rate limiting
- [../adr/0015-application-performance-monitoring-and-error-tracking.md](../adr/0015-application-performance-monitoring-and-error-tracking.md) - Monitoring ADR
- [../adr/0015-error-tracking-and-observability.md](../adr/0015-error-tracking-and-observability.md) - Monitoring ADR
- [../adr/0050-postgresql-function-observability.md](../adr/0050-postgresql-function-observability.md) - Database observability
- [../BARE-METAL-SETUP.md](../BARE-METAL-SETUP.md) - Production setup
- [../operations/BARE-METAL-SETUP.md](../operations/BARE-METAL-SETUP.md) - Production setup
- [../tools/BUGSINK-SETUP.md](../tools/BUGSINK-SETUP.md) - Bugsink configuration

View File

@@ -5,6 +5,17 @@ This guide covers two related but distinct subagents for testing in the Flyer Cr
- **tester**: Adversarial testing to find edge cases, race conditions, and vulnerabilities
- **testwriter**: Creating comprehensive test suites for features and fixes
## Quick Reference
| Aspect | tester | testwriter |
| ---------------- | -------------------------------------------- | ------------------------------------------ |
| **Primary Use** | Find bugs, security issues, edge cases | Create test suites, improve coverage |
| **Key Files** | N/A (analysis-focused) | `*.test.ts`, `src/tests/utils/` |
| **Key ADRs** | ADR-010 (Testing), ADR-040 (Test Economics) | ADR-010 (Testing), ADR-045 (Test Fixtures) |
| **Test Command** | `podman exec -it flyer-crawler-dev npm test` | Same |
| **Test Stack** | Vitest, Supertest, Testing Library | Same |
| **Delegate To** | `testwriter` (write tests for findings) | `coder` (fix failing tests) |
## Understanding the Difference
| Aspect | tester | testwriter |
@@ -399,6 +410,7 @@ A typical workflow for thorough testing:
- [OVERVIEW.md](./OVERVIEW.md) - Subagent system overview
- [CODER-GUIDE.md](./CODER-GUIDE.md) - Working with the coder subagent
- [../TESTING.md](../TESTING.md) - Testing guide
- [SECURITY-DEBUG-GUIDE.md](./SECURITY-DEBUG-GUIDE.md) - Security testing and code review
- [../development/TESTING.md](../development/TESTING.md) - Testing guide
- [../adr/0010-testing-strategy-and-standards.md](../adr/0010-testing-strategy-and-standards.md) - Testing ADR
- [../adr/0040-testing-economics-and-priorities.md](../adr/0040-testing-economics-and-priorities.md) - Testing priorities

4
package-lock.json generated
View File

@@ -1,12 +1,12 @@
{
"name": "flyer-crawler",
"version": "0.12.20",
"version": "0.12.21",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "flyer-crawler",
"version": "0.12.20",
"version": "0.12.21",
"dependencies": {
"@bull-board/api": "^6.14.2",
"@bull-board/express": "^6.14.2",

View File

@@ -1,7 +1,7 @@
{
"name": "flyer-crawler",
"private": true,
"version": "0.12.20",
"version": "0.12.21",
"type": "module",
"scripts": {
"dev": "concurrently \"npm:start:dev\" \"vite\"",

View File

@@ -185,10 +185,7 @@ describe('StoreCard', () => {
it('should show "No location data" when locations is undefined', () => {
renderWithProviders(
<StoreCard
store={mockStoreUndefinedLocations as typeof mockStoreWithLogo}
showLocations={true}
/>,
<StoreCard store={mockStoreUndefinedLocations as any} showLocations={true} />,
);
expect(screen.getByText('No location data')).toBeInTheDocument();

View File

@@ -14,7 +14,7 @@ vi.mock('../services/eventBus', () => ({
import { eventBus } from '../services/eventBus';
const mockEventBus = eventBus as {
const mockEventBus = eventBus as unknown as {
on: Mock;
off: Mock;
dispatch: Mock;
@@ -206,7 +206,7 @@ describe('useEventBus', () => {
name: string;
}
const callback = vi.fn<[TestData?], void>();
const callback = vi.fn();
renderHook(() => useEventBus<TestData>('typed-event', callback));
@@ -217,7 +217,7 @@ describe('useEventBus', () => {
});
it('should handle callback with optional parameter', () => {
const callback = vi.fn<[string?], void>();
const callback = vi.fn();
renderHook(() => useEventBus<string>('optional-event', callback));

View File

@@ -17,6 +17,7 @@ vi.mock('driver.js', () => ({
DriveStep: vi.fn(),
}));
// @ts-expect-error - driver.js types are not fully compatible, but the mock works
import { driver } from 'driver.js';
const mockDriver = driver as Mock;