Compare commits
2 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
de3f0cf26e | ||
| 45ac4fccf5 |
150
CLAUDE.md
150
CLAUDE.md
@@ -31,46 +31,19 @@ Out-of-sync = test failures.
|
||||
|
||||
**CRITICAL**: The `claude-win10` user has **READ-ONLY** access to production and test servers.
|
||||
|
||||
**Claude Code does NOT have:**
|
||||
| Capability | Status |
|
||||
| ---------------------- | ---------------------- |
|
||||
| Root/sudo access | NO |
|
||||
| Write permissions | NO |
|
||||
| PM2 restart, systemctl | NO - User must execute |
|
||||
|
||||
- Root or sudo access
|
||||
- Write permissions on servers
|
||||
- Ability to execute PM2 restart, systemctl, or other write operations directly
|
||||
**Server Operations Workflow**: Diagnose → User executes → Analyze → Fix (1-3 commands) → User executes → Verify
|
||||
|
||||
**Correct Workflow for Server Operations:**
|
||||
**Rules**:
|
||||
|
||||
| Step | Actor | Action |
|
||||
| ---- | ------ | --------------------------------------------------------------------------- |
|
||||
| 1 | Claude | Provide **diagnostic commands** (read-only checks) for user to run |
|
||||
| 2 | User | Execute commands on server, report results |
|
||||
| 3 | Claude | Analyze results, provide **fix commands** (1-3 at a time) |
|
||||
| 4 | User | Execute fix commands, report results |
|
||||
| 5 | Claude | Provide **verification commands** to confirm success and check side effects |
|
||||
| 6 | Claude | Document progress stage by stage |
|
||||
| 7 | Claude | Update ALL relevant documentation when complete |
|
||||
|
||||
**Example - Diagnosing PM2 Issues:**
|
||||
|
||||
```bash
|
||||
# Step 1: Claude provides diagnostic commands (user runs these)
|
||||
pm2 list
|
||||
pm2 logs flyer-crawler-api --lines 50
|
||||
systemctl status redis
|
||||
|
||||
# Step 3: After user reports results, Claude provides fix commands
|
||||
pm2 restart flyer-crawler-api
|
||||
# Wait for user confirmation before next command
|
||||
|
||||
# Step 5: Claude provides verification
|
||||
pm2 list
|
||||
curl -s https://flyer-crawler.projectium.com/api/health/ready | jq .
|
||||
```
|
||||
|
||||
**Never Do:**
|
||||
|
||||
- `ssh root@projectium.com "pm2 restart all"` (Claude cannot execute this)
|
||||
- Assume commands succeeded without user confirmation
|
||||
- Provide more than 3 fix commands at once (errors may cascade)
|
||||
- Provide diagnostic commands first, wait for user to report results
|
||||
- Maximum 3 fix commands at a time (errors may cascade)
|
||||
- Always verify after fixes complete
|
||||
|
||||
### Communication Style
|
||||
|
||||
@@ -105,12 +78,12 @@ Ask before assuming. Never assume:
|
||||
|
||||
### Key Patterns (with file locations)
|
||||
|
||||
| Pattern | ADR | Implementation | File |
|
||||
| ------------------ | ------- | ------------------------------------------------- | ----------------------------------- |
|
||||
| Error Handling | ADR-001 | `handleDbError()`, throw `NotFoundError` | `src/services/db/errors.db.ts` |
|
||||
| Repository Methods | ADR-034 | `get*` (throws), `find*` (null), `list*` (array) | `src/services/db/*.db.ts` |
|
||||
| API Responses | ADR-028 | `sendSuccess()`, `sendPaginated()`, `sendError()` | `src/utils/apiResponse.ts` |
|
||||
| Transactions | ADR-002 | `withTransaction(async (client) => {...})` | `src/services/db/transaction.db.ts` |
|
||||
| Pattern | ADR | Implementation | File |
|
||||
| ------------------ | ------- | ------------------------------------------------- | ---------------------------------- |
|
||||
| Error Handling | ADR-001 | `handleDbError()`, throw `NotFoundError` | `src/services/db/errors.db.ts` |
|
||||
| Repository Methods | ADR-034 | `get*` (throws), `find*` (null), `list*` (array) | `src/services/db/*.db.ts` |
|
||||
| API Responses | ADR-028 | `sendSuccess()`, `sendPaginated()`, `sendError()` | `src/utils/apiResponse.ts` |
|
||||
| Transactions | ADR-002 | `withTransaction(async (client) => {...})` | `src/services/db/connection.db.ts` |
|
||||
|
||||
### Key Files Quick Access
|
||||
|
||||
@@ -166,7 +139,7 @@ The dev container now matches production by using PM2 for process management.
|
||||
- `flyer-crawler-worker-dev` - Background job worker
|
||||
- `flyer-crawler-vite-dev` - Vite frontend dev server (port 5173)
|
||||
|
||||
### Log Aggregation (ADR-050)
|
||||
### Log Aggregation (ADR-015)
|
||||
|
||||
All logs flow to Bugsink via Logstash with 3-project routing:
|
||||
|
||||
@@ -249,7 +222,7 @@ All logs flow to Bugsink via Logstash with 3-project routing:
|
||||
|
||||
**Launch Pattern**:
|
||||
|
||||
```
|
||||
```text
|
||||
Use Task tool with subagent_type: "coder", "db-dev", "tester", etc.
|
||||
```
|
||||
|
||||
@@ -330,8 +303,8 @@ podman cp "d:/path/file" container:/tmp/file
|
||||
|
||||
**Quick Access**:
|
||||
|
||||
- **Dev**: https://localhost:8443 (`admin@localhost`/`admin`)
|
||||
- **Prod**: https://bugsink.projectium.com
|
||||
- **Dev**: <https://localhost:8443> (`admin@localhost`/`admin`)
|
||||
- **Prod**: <https://bugsink.projectium.com>
|
||||
|
||||
**Token Creation** (required for MCP):
|
||||
|
||||
@@ -347,7 +320,7 @@ cd /opt/bugsink && bugsink-manage create_auth_token
|
||||
|
||||
**See**: [docs/operations/LOGSTASH-QUICK-REF.md](docs/operations/LOGSTASH-QUICK-REF.md)
|
||||
|
||||
Log aggregation: PostgreSQL + PM2 + Redis + NGINX → Bugsink (ADR-050)
|
||||
Log aggregation: PostgreSQL + PM2 + Redis + NGINX → Bugsink (ADR-015)
|
||||
|
||||
---
|
||||
|
||||
@@ -367,84 +340,3 @@ Log aggregation: PostgreSQL + PM2 + Redis + NGINX → Bugsink (ADR-050)
|
||||
| **Logstash** | [LOGSTASH-QUICK-REF.md](docs/operations/LOGSTASH-QUICK-REF.md) |
|
||||
| **ADRs** | [docs/adr/index.md](docs/adr/index.md) |
|
||||
| **All Docs** | [docs/README.md](docs/README.md) |
|
||||
|
||||
---
|
||||
|
||||
## Appendix: Integration Test Issues (Full Details)
|
||||
|
||||
### 1. Vitest globalSetup Context Isolation
|
||||
|
||||
Vitest's `globalSetup` runs in separate Node.js context. Singletons, spies, mocks do NOT share instances with test files.
|
||||
|
||||
**Affected**: BullMQ worker service mocks (AI/DB failure tests)
|
||||
|
||||
**Solutions**: Mark `.todo()`, create test-only API endpoints, use Redis-based mock flags
|
||||
|
||||
```typescript
|
||||
// DOES NOT WORK - different instances
|
||||
const { flyerProcessingService } = await import('../../services/workers.server');
|
||||
flyerProcessingService._getAiProcessor()._setExtractAndValidateData(mockFn);
|
||||
```
|
||||
|
||||
### 2. Cleanup Queue Deletes Before Verification
|
||||
|
||||
Cleanup worker processes jobs in globalSetup context, ignoring test spies.
|
||||
|
||||
**Solution**: Drain and pause queue:
|
||||
|
||||
```typescript
|
||||
const { cleanupQueue } = await import('../../services/queues.server');
|
||||
await cleanupQueue.drain();
|
||||
await cleanupQueue.pause();
|
||||
// ... test ...
|
||||
await cleanupQueue.resume();
|
||||
```
|
||||
|
||||
### 3. Cache Stale After Direct SQL
|
||||
|
||||
Direct `pool.query()` inserts bypass cache invalidation.
|
||||
|
||||
**Solution**: `await cacheService.invalidateFlyers();` after inserts
|
||||
|
||||
### 4. Test Filename Collisions
|
||||
|
||||
Multer predictable filenames cause race conditions.
|
||||
|
||||
**Solution**: Use unique suffix: `${Date.now()}-${Math.round(Math.random() * 1e9)}`
|
||||
|
||||
### 5. Response Format Mismatches
|
||||
|
||||
API formats change: `data.jobId` vs `data.job.id`, nested vs flat, string vs number IDs.
|
||||
|
||||
**Solution**: Log response bodies, update assertions
|
||||
|
||||
### 6. External Service Availability
|
||||
|
||||
PM2/Redis health checks fail when unavailable.
|
||||
|
||||
**Solution**: try/catch with graceful degradation or mock
|
||||
|
||||
### 7. TZ Environment Variable Breaking Async Hooks
|
||||
|
||||
**Problem**: When `TZ=America/Los_Angeles` (or other timezone values) is set in the environment, Node.js async_hooks module can produce `RangeError: Invalid triggerAsyncId value: NaN`. This breaks React Testing Library's `render()` function which uses async hooks internally.
|
||||
|
||||
**Root Cause**: Setting `TZ` to certain timezone values interferes with Node.js's internal async tracking mechanism, causing invalid async IDs to be generated.
|
||||
|
||||
**Symptoms**:
|
||||
|
||||
```text
|
||||
RangeError: Invalid triggerAsyncId value: NaN
|
||||
❯ process.env.NODE_ENV.queueSeveralMicrotasks node_modules/react/cjs/react.development.js:751:15
|
||||
❯ process.env.NODE_ENV.exports.act node_modules/react/cjs/react.development.js:886:11
|
||||
❯ node_modules/@testing-library/react/dist/act-compat.js:46:25
|
||||
❯ renderRoot node_modules/@testing-library/react/dist/pure.js:189:26
|
||||
```
|
||||
|
||||
**Solution**: Explicitly unset `TZ` in all test scripts by adding `TZ=` (empty value) to cross-env:
|
||||
|
||||
```json
|
||||
"test:unit": "cross-env NODE_ENV=test TZ= tsx ..."
|
||||
"test:integration": "cross-env NODE_ENV=test TZ= tsx ..."
|
||||
```
|
||||
|
||||
**Context**: This issue was introduced in commit `d03900c` which added `TZ: 'America/Los_Angeles'` to PM2 ecosystem configs for consistent log timestamps in production/dev environments. Tests must explicitly override this to prevent the async hooks error.
|
||||
|
||||
393
docs/AI-DOCUMENTATION-INDEX.md
Normal file
393
docs/AI-DOCUMENTATION-INDEX.md
Normal file
@@ -0,0 +1,393 @@
|
||||
# AI Documentation Index
|
||||
|
||||
Machine-optimized navigation for AI agents. Structured for vector retrieval and semantic search.
|
||||
|
||||
---
|
||||
|
||||
## Quick Lookup Table
|
||||
|
||||
| Task/Question | Primary Doc | Section/ADR |
|
||||
| ----------------------- | --------------------------------------------------- | --------------------------------------- |
|
||||
| Add new API endpoint | [CODE-PATTERNS.md](development/CODE-PATTERNS.md) | API Response Patterns, Input Validation |
|
||||
| Add repository method | [CODE-PATTERNS.md](development/CODE-PATTERNS.md) | Repository Patterns (get*/find*/list\*) |
|
||||
| Fix failing test | [TESTING.md](development/TESTING.md) | Known Integration Test Issues |
|
||||
| Run tests correctly | [TESTING.md](development/TESTING.md) | Test Execution Environment |
|
||||
| Add database column | [DATABASE-GUIDE.md](subagents/DATABASE-GUIDE.md) | Schema sync required |
|
||||
| Deploy to production | [DEPLOYMENT.md](operations/DEPLOYMENT.md) | Application Deployment |
|
||||
| Debug container issue | [DEBUGGING.md](development/DEBUGGING.md) | Container Issues |
|
||||
| Configure environment | [ENVIRONMENT.md](getting-started/ENVIRONMENT.md) | Configuration by Environment |
|
||||
| Add background job | [CODE-PATTERNS.md](development/CODE-PATTERNS.md) | Background Jobs |
|
||||
| Handle errors correctly | [CODE-PATTERNS.md](development/CODE-PATTERNS.md) | Error Handling |
|
||||
| Use transactions | [CODE-PATTERNS.md](development/CODE-PATTERNS.md) | Transaction Management |
|
||||
| Add authentication | [AUTHENTICATION.md](architecture/AUTHENTICATION.md) | JWT Token Architecture |
|
||||
| Cache data | [CODE-PATTERNS.md](development/CODE-PATTERNS.md) | Caching |
|
||||
| Check PM2 status | [DEV-CONTAINER.md](development/DEV-CONTAINER.md) | PM2 Process Management |
|
||||
| View logs | [DEBUGGING.md](development/DEBUGGING.md) | PM2 Log Access |
|
||||
| Understand architecture | [OVERVIEW.md](architecture/OVERVIEW.md) | System Architecture Diagram |
|
||||
| Check ADR for decision | [adr/index.md](adr/index.md) | ADR by category |
|
||||
| Use subagent | [subagents/OVERVIEW.md](subagents/OVERVIEW.md) | Available Subagents |
|
||||
| API versioning | [API-VERSIONING.md](development/API-VERSIONING.md) | Phase 2 infrastructure |
|
||||
|
||||
---
|
||||
|
||||
## Documentation Tree
|
||||
|
||||
```
|
||||
docs/
|
||||
+-- AI-DOCUMENTATION-INDEX.md # THIS FILE - AI navigation index
|
||||
+-- README.md # Human-readable doc hub
|
||||
|
|
||||
+-- adr/ # Architecture Decision Records (57 ADRs)
|
||||
| +-- index.md # ADR index by category
|
||||
| +-- 0001-*.md # Standardized error handling
|
||||
| +-- 0002-*.md # Transaction management (withTransaction)
|
||||
| +-- 0003-*.md # Input validation (Zod middleware)
|
||||
| +-- 0008-*.md # API versioning (/api/v1/)
|
||||
| +-- 0014-*.md # Platform: Linux only (CRITICAL)
|
||||
| +-- 0028-*.md # API response (sendSuccess/sendError)
|
||||
| +-- 0034-*.md # Repository pattern (get*/find*/list*)
|
||||
| +-- 0035-*.md # Service layer architecture
|
||||
| +-- 0050-*.md # PostgreSQL observability + Logstash
|
||||
| +-- 0057-*.md # Test remediation post-API versioning
|
||||
| +-- adr-implementation-tracker.md # Implementation status
|
||||
|
|
||||
+-- architecture/
|
||||
| +-- OVERVIEW.md # System architecture, data flows, entities
|
||||
| +-- DATABASE.md # Schema design, extensions, setup
|
||||
| +-- AUTHENTICATION.md # OAuth, JWT, security features
|
||||
| +-- WEBSOCKET_USAGE.md # Real-time communication patterns
|
||||
| +-- api-versioning-infrastructure.md # Phase 2 versioning details
|
||||
|
|
||||
+-- development/
|
||||
| +-- CODE-PATTERNS.md # Error handling, repos, API responses
|
||||
| +-- TESTING.md # Unit/integration/E2E, known issues
|
||||
| +-- DEBUGGING.md # Container, DB, API, PM2 debugging
|
||||
| +-- DEV-CONTAINER.md # PM2, Logstash, container services
|
||||
| +-- API-VERSIONING.md # API versioning workflows
|
||||
| +-- DESIGN_TOKENS.md # Neo-Brutalism design system
|
||||
| +-- ERROR-LOGGING-PATHS.md # req.originalUrl pattern
|
||||
| +-- test-path-migration.md # Test file reorganization
|
||||
|
|
||||
+-- getting-started/
|
||||
| +-- QUICKSTART.md # Quick setup instructions
|
||||
| +-- INSTALL.md # Full installation guide
|
||||
| +-- ENVIRONMENT.md # Environment variables reference
|
||||
|
|
||||
+-- operations/
|
||||
| +-- DEPLOYMENT.md # Production deployment guide
|
||||
| +-- BARE-METAL-SETUP.md # Server provisioning
|
||||
| +-- MONITORING.md # Bugsink, health checks
|
||||
| +-- LOGSTASH-QUICK-REF.md # Log aggregation reference
|
||||
| +-- LOGSTASH-TROUBLESHOOTING.md # Logstash debugging
|
||||
|
|
||||
+-- subagents/
|
||||
| +-- OVERVIEW.md # Subagent system introduction
|
||||
| +-- CODER-GUIDE.md # Code development patterns
|
||||
| +-- TESTER-GUIDE.md # Testing strategies
|
||||
| +-- DATABASE-GUIDE.md # Database workflows
|
||||
| +-- DEVOPS-GUIDE.md # Deployment/infrastructure
|
||||
| +-- FRONTEND-GUIDE.md # UI/UX development
|
||||
| +-- AI-USAGE-GUIDE.md # Gemini integration
|
||||
| +-- DOCUMENTATION-GUIDE.md # Writing docs
|
||||
| +-- SECURITY-DEBUG-GUIDE.md # Security and debugging
|
||||
|
|
||||
+-- tools/
|
||||
| +-- MCP-CONFIGURATION.md # MCP servers setup
|
||||
| +-- BUGSINK-SETUP.md # Error tracking setup
|
||||
| +-- VSCODE-SETUP.md # Editor configuration
|
||||
|
|
||||
+-- archive/ # Historical docs, session notes
|
||||
+-- sessions/ # Development session logs
|
||||
+-- plans/ # Feature implementation plans
|
||||
+-- research/ # Investigation notes
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Problem-to-Document Mapping
|
||||
|
||||
### Database Issues
|
||||
|
||||
| Problem | Documents |
|
||||
| -------------------- | ----------------------------------------------------------------------------------------------- |
|
||||
| Schema out of sync | [DATABASE-GUIDE.md](subagents/DATABASE-GUIDE.md), [CLAUDE.md](../CLAUDE.md) schema sync section |
|
||||
| Migration needed | [DATABASE.md](architecture/DATABASE.md), ADR-013, ADR-023 |
|
||||
| Query performance | [DEBUGGING.md](development/DEBUGGING.md) Query Performance Issues |
|
||||
| Connection errors | [DEBUGGING.md](development/DEBUGGING.md) Database Issues |
|
||||
| Transaction patterns | [CODE-PATTERNS.md](development/CODE-PATTERNS.md) Transaction Management, ADR-002 |
|
||||
| Repository methods | [CODE-PATTERNS.md](development/CODE-PATTERNS.md) Repository Patterns, ADR-034 |
|
||||
|
||||
### Test Failures
|
||||
|
||||
| Problem | Documents |
|
||||
| ---------------------------- | --------------------------------------------------------------------- |
|
||||
| Tests fail in container | [TESTING.md](development/TESTING.md), ADR-014 |
|
||||
| Vitest globalSetup isolation | [CLAUDE.md](../CLAUDE.md) Integration Test Issues #1 |
|
||||
| Cache stale after insert | [CLAUDE.md](../CLAUDE.md) Integration Test Issues #3 |
|
||||
| Queue interference | [CLAUDE.md](../CLAUDE.md) Integration Test Issues #2 |
|
||||
| API path mismatches | [TESTING.md](development/TESTING.md) API Versioning in Tests, ADR-057 |
|
||||
| Type check failures | [DEBUGGING.md](development/DEBUGGING.md) Type Check Failures |
|
||||
| TZ environment breaks async | [CLAUDE.md](../CLAUDE.md) Integration Test Issues #7 |
|
||||
|
||||
### Deployment Issues
|
||||
|
||||
| Problem | Documents |
|
||||
| --------------------- | ------------------------------------------------------------------------------------- |
|
||||
| PM2 not starting | [DEBUGGING.md](development/DEBUGGING.md) PM2 Process Issues |
|
||||
| NGINX configuration | [DEPLOYMENT.md](operations/DEPLOYMENT.md) NGINX Configuration |
|
||||
| SSL certificates | [DEBUGGING.md](development/DEBUGGING.md) SSL Certificate Issues |
|
||||
| CI/CD failures | [DEPLOYMENT.md](operations/DEPLOYMENT.md) CI/CD Pipeline, ADR-017 |
|
||||
| Container won't start | [DEBUGGING.md](development/DEBUGGING.md) Container Issues |
|
||||
| Bugsink not receiving | [BUGSINK-SETUP.md](tools/BUGSINK-SETUP.md), [MONITORING.md](operations/MONITORING.md) |
|
||||
|
||||
### Frontend/UI Changes
|
||||
|
||||
| Problem | Documents |
|
||||
| ------------------ | --------------------------------------------------------------- |
|
||||
| Component patterns | [FRONTEND-GUIDE.md](subagents/FRONTEND-GUIDE.md), ADR-044 |
|
||||
| Design tokens | [DESIGN_TOKENS.md](development/DESIGN_TOKENS.md), ADR-012 |
|
||||
| State management | ADR-005, [OVERVIEW.md](architecture/OVERVIEW.md) Frontend Stack |
|
||||
| Hot reload broken | [DEBUGGING.md](development/DEBUGGING.md) Frontend Issues |
|
||||
| CORS errors | [DEBUGGING.md](development/DEBUGGING.md) API Calls Failing |
|
||||
|
||||
### API Development
|
||||
|
||||
| Problem | Documents |
|
||||
| ---------------- | ------------------------------------------------------------------------------- |
|
||||
| Response format | [CODE-PATTERNS.md](development/CODE-PATTERNS.md) API Response Patterns, ADR-028 |
|
||||
| Input validation | [CODE-PATTERNS.md](development/CODE-PATTERNS.md) Input Validation, ADR-003 |
|
||||
| Error handling | [CODE-PATTERNS.md](development/CODE-PATTERNS.md) Error Handling, ADR-001 |
|
||||
| Rate limiting | ADR-032, [OVERVIEW.md](architecture/OVERVIEW.md) |
|
||||
| API versioning | [API-VERSIONING.md](development/API-VERSIONING.md), ADR-008 |
|
||||
| Authentication | [AUTHENTICATION.md](architecture/AUTHENTICATION.md), ADR-048 |
|
||||
|
||||
### Background Jobs
|
||||
|
||||
| Problem | Documents |
|
||||
| ------------------- | ------------------------------------------------------------------------- |
|
||||
| Jobs not processing | [DEBUGGING.md](development/DEBUGGING.md) Background Job Issues |
|
||||
| Queue configuration | [CODE-PATTERNS.md](development/CODE-PATTERNS.md) Background Jobs, ADR-006 |
|
||||
| Worker crashes | [DEBUGGING.md](development/DEBUGGING.md), ADR-053 |
|
||||
| Scheduled jobs | ADR-037, [OVERVIEW.md](architecture/OVERVIEW.md) Scheduled Jobs |
|
||||
|
||||
---
|
||||
|
||||
## Document Priority Matrix
|
||||
|
||||
### CRITICAL (Read First)
|
||||
|
||||
| Document | Purpose | Key Content |
|
||||
| --------------------------------------------------------------- | ----------------------- | ----------------------------- |
|
||||
| [CLAUDE.md](../CLAUDE.md) | AI agent instructions | Rules, patterns, known issues |
|
||||
| [ADR-014](adr/0014-containerization-and-deployment-strategy.md) | Platform requirement | Tests MUST run in container |
|
||||
| [DEV-CONTAINER.md](development/DEV-CONTAINER.md) | Development environment | PM2, Logstash, services |
|
||||
|
||||
### HIGH (Core Development)
|
||||
|
||||
| Document | Purpose | Key Content |
|
||||
| --------------------------------------------------- | ----------------- | ---------------------------- |
|
||||
| [CODE-PATTERNS.md](development/CODE-PATTERNS.md) | Code templates | Error handling, repos, APIs |
|
||||
| [TESTING.md](development/TESTING.md) | Test execution | Commands, known issues |
|
||||
| [DATABASE.md](architecture/DATABASE.md) | Schema reference | Setup, extensions, users |
|
||||
| [ADR-034](adr/0034-repository-pattern-standards.md) | Repository naming | get*/find*/list\* |
|
||||
| [ADR-028](adr/0028-api-response-standardization.md) | API responses | sendSuccess/sendError |
|
||||
| [ADR-001](adr/0001-standardized-error-handling.md) | Error handling | handleDbError, NotFoundError |
|
||||
|
||||
### MEDIUM (Specialized Tasks)
|
||||
|
||||
| Document | Purpose | Key Content |
|
||||
| --------------------------------------------------- | --------------------- | ------------------------ |
|
||||
| [subagents/OVERVIEW.md](subagents/OVERVIEW.md) | Subagent selection | When to delegate |
|
||||
| [DEPLOYMENT.md](operations/DEPLOYMENT.md) | Production deployment | PM2, NGINX, CI/CD |
|
||||
| [DEBUGGING.md](development/DEBUGGING.md) | Troubleshooting | Common issues, solutions |
|
||||
| [ENVIRONMENT.md](getting-started/ENVIRONMENT.md) | Config reference | Variables by environment |
|
||||
| [AUTHENTICATION.md](architecture/AUTHENTICATION.md) | Auth patterns | OAuth, JWT, security |
|
||||
| [API-VERSIONING.md](development/API-VERSIONING.md) | Versioning | /api/v1/ prefix |
|
||||
|
||||
### LOW (Reference/Historical)
|
||||
|
||||
| Document | Purpose | Key Content |
|
||||
| -------------------- | ------------------ | ------------------------- |
|
||||
| [archive/](archive/) | Historical docs | Session notes, old plans |
|
||||
| ADR-013, ADR-023 | Migration strategy | Proposed, not implemented |
|
||||
| ADR-024 | Feature flags | Proposed |
|
||||
| ADR-025 | i18n/l10n | Proposed |
|
||||
|
||||
---
|
||||
|
||||
## Cross-Reference Matrix
|
||||
|
||||
| Document | References | Referenced By |
|
||||
| -------------------- | ------------------------------------------------------------------------------- | ------------------------------------------------------ |
|
||||
| **CLAUDE.md** | ADR-001, ADR-002, ADR-008, ADR-014, ADR-028, ADR-034, ADR-035, ADR-050, ADR-057 | All development docs |
|
||||
| **ADR-008** | ADR-028 | API-VERSIONING.md, TESTING.md, ADR-057 |
|
||||
| **ADR-014** | - | CLAUDE.md, TESTING.md, DEPLOYMENT.md, DEV-CONTAINER.md |
|
||||
| **ADR-028** | ADR-001 | CODE-PATTERNS.md, OVERVIEW.md |
|
||||
| **ADR-034** | ADR-001 | CODE-PATTERNS.md, DATABASE-GUIDE.md |
|
||||
| **ADR-057** | ADR-008, ADR-028 | TESTING.md |
|
||||
| **CODE-PATTERNS.md** | ADR-001, ADR-002, ADR-003, ADR-028, ADR-034, ADR-036, ADR-048 | CODER-GUIDE.md |
|
||||
| **TESTING.md** | ADR-014, ADR-057, CLAUDE.md | TESTER-GUIDE.md, DEBUGGING.md |
|
||||
| **DEBUGGING.md** | DEV-CONTAINER.md, TESTING.md, MONITORING.md | DEVOPS-GUIDE.md |
|
||||
| **DEV-CONTAINER.md** | ADR-014, ADR-050, ecosystem.dev.config.cjs | DEBUGGING.md, CLAUDE.md |
|
||||
| **OVERVIEW.md** | ADR-001 through ADR-050+ | All architecture docs |
|
||||
| **DATABASE.md** | ADR-002, ADR-013, ADR-055 | DATABASE-GUIDE.md |
|
||||
|
||||
---
|
||||
|
||||
## Navigation Patterns
|
||||
|
||||
### Adding a Feature
|
||||
|
||||
```
|
||||
1. CLAUDE.md -> Project rules, patterns
|
||||
2. CODE-PATTERNS.md -> Implementation templates
|
||||
3. Relevant subagent guide -> Domain-specific patterns
|
||||
4. Related ADRs -> Design decisions
|
||||
5. TESTING.md -> Test requirements
|
||||
```
|
||||
|
||||
### Fixing a Bug
|
||||
|
||||
```
|
||||
1. DEBUGGING.md -> Common issues checklist
|
||||
2. TESTING.md -> Run tests in container
|
||||
3. Error logs (pm2/bugsink) -> Identify root cause
|
||||
4. CODE-PATTERNS.md -> Correct pattern reference
|
||||
5. Related ADR -> Architectural context
|
||||
```
|
||||
|
||||
### Deploying
|
||||
|
||||
```
|
||||
1. DEPLOYMENT.md -> Deployment procedures
|
||||
2. ENVIRONMENT.md -> Required variables
|
||||
3. MONITORING.md -> Health check verification
|
||||
4. LOGSTASH-QUICK-REF.md -> Log aggregation check
|
||||
```
|
||||
|
||||
### Database Changes
|
||||
|
||||
```
|
||||
1. DATABASE-GUIDE.md -> Schema sync requirements (CRITICAL)
|
||||
2. DATABASE.md -> Schema design patterns
|
||||
3. ADR-002 -> Transaction patterns
|
||||
4. ADR-034 -> Repository methods
|
||||
5. ADR-055 -> Normalization rules
|
||||
```
|
||||
|
||||
### Subagent Selection
|
||||
|
||||
| Task Type | Subagent | Guide |
|
||||
| --------------------- | ------------------------- | ------------------------------------------------------------ |
|
||||
| Write production code | `coder` | [CODER-GUIDE.md](subagents/CODER-GUIDE.md) |
|
||||
| Database changes | `db-dev` | [DATABASE-GUIDE.md](subagents/DATABASE-GUIDE.md) |
|
||||
| Create tests | `testwriter` | [TESTER-GUIDE.md](subagents/TESTER-GUIDE.md) |
|
||||
| Fix failing tests | `tester` | [TESTER-GUIDE.md](subagents/TESTER-GUIDE.md) |
|
||||
| Container/deployment | `devops` | [DEVOPS-GUIDE.md](subagents/DEVOPS-GUIDE.md) |
|
||||
| UI components | `frontend-specialist` | [FRONTEND-GUIDE.md](subagents/FRONTEND-GUIDE.md) |
|
||||
| External APIs | `integrations-specialist` | - |
|
||||
| Security review | `security-engineer` | [SECURITY-DEBUG-GUIDE.md](subagents/SECURITY-DEBUG-GUIDE.md) |
|
||||
| Production errors | `log-debug` | [SECURITY-DEBUG-GUIDE.md](subagents/SECURITY-DEBUG-GUIDE.md) |
|
||||
| AI/Gemini issues | `ai-usage` | [AI-USAGE-GUIDE.md](subagents/AI-USAGE-GUIDE.md) |
|
||||
|
||||
---
|
||||
|
||||
## Key File Quick Reference
|
||||
|
||||
### Configuration
|
||||
|
||||
| File | Purpose |
|
||||
| -------------------------- | ---------------------------- |
|
||||
| `server.ts` | Express app setup |
|
||||
| `src/config/env.ts` | Environment validation (Zod) |
|
||||
| `ecosystem.dev.config.cjs` | PM2 dev config |
|
||||
| `ecosystem.config.cjs` | PM2 prod config |
|
||||
| `vite.config.ts` | Vite build config |
|
||||
|
||||
### Core Implementation
|
||||
|
||||
| File | Purpose |
|
||||
| ----------------------------------- | ----------------------------------- |
|
||||
| `src/routes/*.routes.ts` | API route handlers |
|
||||
| `src/services/db/*.db.ts` | Repository layer |
|
||||
| `src/services/*.server.ts` | Server-only services |
|
||||
| `src/services/queues.server.ts` | BullMQ queue definitions |
|
||||
| `src/services/workers.server.ts` | BullMQ workers |
|
||||
| `src/utils/apiResponse.ts` | sendSuccess/sendError/sendPaginated |
|
||||
| `src/services/db/errors.db.ts` | handleDbError, NotFoundError |
|
||||
| `src/services/db/transaction.db.ts` | withTransaction |
|
||||
|
||||
### Database Schema
|
||||
|
||||
| File | Purpose |
|
||||
| ------------------------------ | ----------------------------------- |
|
||||
| `sql/master_schema_rollup.sql` | Test DB, complete reference |
|
||||
| `sql/initial_schema.sql` | Fresh install (identical to rollup) |
|
||||
| `sql/migrations/*.sql` | Production ALTER statements |
|
||||
|
||||
### Testing
|
||||
|
||||
| File | Purpose |
|
||||
| ---------------------------------- | ----------------------- |
|
||||
| `vitest.config.ts` | Unit test config |
|
||||
| `vitest.config.integration.ts` | Integration test config |
|
||||
| `vitest.config.e2e.ts` | E2E test config |
|
||||
| `src/tests/utils/mockFactories.ts` | Mock data factories |
|
||||
| `src/tests/utils/storeHelpers.ts` | Store test helpers |
|
||||
|
||||
---
|
||||
|
||||
## ADR Quick Reference
|
||||
|
||||
### By Implementation Status
|
||||
|
||||
**Implemented**: 001, 002, 003, 004, 006, 008, 009, 010, 016, 017, 020, 021, 028, 032, 033, 034, 035, 036, 037, 038, 040, 041, 043, 044, 045, 046, 050, 051, 052, 055, 057
|
||||
|
||||
**Partially Implemented**: 012, 014, 015, 048
|
||||
|
||||
**Proposed**: 011, 013, 022, 023, 024, 025, 029, 030, 031, 039, 047, 053, 054, 056
|
||||
|
||||
### By Category
|
||||
|
||||
| Category | ADRs |
|
||||
| --------------------- | ------------------------------------------- |
|
||||
| Core Infrastructure | 002, 007, 020, 030 |
|
||||
| Data Management | 009, 013, 019, 023, 031, 055 |
|
||||
| API & Integration | 003, 008, 018, 022, 028 |
|
||||
| Security | 001, 011, 016, 029, 032, 033, 048 |
|
||||
| Observability | 004, 015, 050, 051, 052, 056 |
|
||||
| Deployment & Ops | 006, 014, 017, 024, 037, 038, 053, 054 |
|
||||
| Frontend/UI | 005, 012, 025, 026, 044 |
|
||||
| Dev Workflow | 010, 021, 027, 040, 045, 047, 057 |
|
||||
| Architecture Patterns | 034, 035, 036, 039, 041, 042, 043, 046, 049 |
|
||||
|
||||
---
|
||||
|
||||
## Essential Commands
|
||||
|
||||
```bash
|
||||
# Run all tests (MUST use container)
|
||||
podman exec -it flyer-crawler-dev npm test
|
||||
|
||||
# Run unit tests
|
||||
podman exec -it flyer-crawler-dev npm run test:unit
|
||||
|
||||
# Run type check
|
||||
podman exec -it flyer-crawler-dev npm run type-check
|
||||
|
||||
# Run integration tests
|
||||
podman exec -it flyer-crawler-dev npm run test:integration
|
||||
|
||||
# PM2 status
|
||||
podman exec -it flyer-crawler-dev pm2 status
|
||||
|
||||
# PM2 logs
|
||||
podman exec -it flyer-crawler-dev pm2 logs
|
||||
|
||||
# Restart all processes
|
||||
podman exec -it flyer-crawler-dev pm2 restart all
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
_This index is optimized for AI agent consumption. Updated: 2026-01-28_
|
||||
@@ -316,6 +316,7 @@ app.use('/api/v1', (req, res, next) => {
|
||||
- [ADR-018](./0018-api-documentation-strategy.md) - API Documentation Strategy (versioned OpenAPI specs)
|
||||
- [ADR-028](./0028-api-response-standardization.md) - Response Standardization (envelope pattern applies to all versions)
|
||||
- [ADR-016](./0016-api-security-hardening.md) - Security Hardening (applies to all versions)
|
||||
- [ADR-057](./0057-test-remediation-post-api-versioning.md) - Test Remediation Post-API Versioning (documents test migration)
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
|
||||
@@ -363,6 +363,13 @@ The following files contain acknowledged code smell violations that are deferred
|
||||
- `src/tests/utils/mockFactories.ts` - Mock factories (1553 lines)
|
||||
- `src/tests/utils/testHelpers.ts` - Test utilities
|
||||
|
||||
## Related ADRs
|
||||
|
||||
- [ADR-014](./0014-containerization-and-deployment-strategy.md) - Containerization (tests must run in dev container)
|
||||
- [ADR-040](./0040-testing-economics-and-priorities.md) - Testing Economics and Priorities
|
||||
- [ADR-045](./0045-test-data-factories-and-fixtures.md) - Test Data Factories and Fixtures
|
||||
- [ADR-057](./0057-test-remediation-post-api-versioning.md) - Test Remediation Post-API Versioning
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
1. **Browser E2E Tests**: Consider adding Playwright for actual browser testing
|
||||
|
||||
@@ -2,7 +2,9 @@
|
||||
|
||||
**Date**: 2025-12-12
|
||||
|
||||
**Status**: Proposed
|
||||
**Status**: Superseded by [ADR-023](./0023-database-schema-migration-strategy.md)
|
||||
|
||||
**Note**: This ADR was an early draft. ADR-023 provides a more detailed specification for the same topic.
|
||||
|
||||
## Context
|
||||
|
||||
|
||||
@@ -4,6 +4,8 @@
|
||||
|
||||
**Status**: Proposed
|
||||
|
||||
**Supersedes**: [ADR-013](./0013-database-schema-migration-strategy.md)
|
||||
|
||||
## Context
|
||||
|
||||
The `README.md` indicates that the database schema is managed by manually running a large `schema.sql.txt` file. This approach is highly error-prone, makes tracking changes difficult, and is not feasible for updating a live production database without downtime or data loss.
|
||||
|
||||
@@ -195,6 +195,12 @@ Do NOT add tests:
|
||||
- Coverage percentages may not satisfy external audits
|
||||
- Requires judgment calls that may be inconsistent
|
||||
|
||||
## Related ADRs
|
||||
|
||||
- [ADR-010](./0010-testing-strategy-and-standards.md) - Testing Strategy and Standards (this ADR extends ADR-010)
|
||||
- [ADR-045](./0045-test-data-factories-and-fixtures.md) - Test Data Factories and Fixtures
|
||||
- [ADR-057](./0057-test-remediation-post-api-versioning.md) - Test Remediation Post-API Versioning
|
||||
|
||||
## Key Files
|
||||
|
||||
- `docs/adr/0010-testing-strategy-and-standards.md` - Testing mechanics
|
||||
|
||||
@@ -4,7 +4,7 @@
|
||||
|
||||
**Status**: Accepted (Fully Implemented)
|
||||
|
||||
**Related**: [ADR-015](0015-application-performance-monitoring-and-error-tracking.md), [ADR-004](0004-standardized-application-wide-structured-logging.md)
|
||||
**Related**: [ADR-015](0015-error-tracking-and-observability.md), [ADR-004](0004-standardized-application-wide-structured-logging.md)
|
||||
|
||||
## Context
|
||||
|
||||
@@ -335,7 +335,7 @@ SELECT award_achievement('user-uuid', 'Nonexistent Badge');
|
||||
|
||||
## References
|
||||
|
||||
- [ADR-015: Application Performance Monitoring](0015-application-performance-monitoring-and-error-tracking.md)
|
||||
- [ADR-015: Error Tracking and Observability](0015-error-tracking-and-observability.md)
|
||||
- [ADR-004: Standardized Structured Logging](0004-standardized-application-wide-structured-logging.md)
|
||||
- [PostgreSQL RAISE Documentation](https://www.postgresql.org/docs/current/plpgsql-errors-and-messages.html)
|
||||
- [PostgreSQL Logging Configuration](https://www.postgresql.org/docs/current/runtime-config-logging.html)
|
||||
|
||||
@@ -332,6 +332,6 @@ Response:
|
||||
## References
|
||||
|
||||
- [ADR-006: Background Job Processing](./0006-background-job-processing-and-task-queues.md)
|
||||
- [ADR-015: Application Performance Monitoring](./0015-application-performance-monitoring-and-error-tracking.md)
|
||||
- [ADR-015: Error Tracking and Observability](./0015-error-tracking-and-observability.md)
|
||||
- [Bugsink API Documentation](https://bugsink.com/docs/api/)
|
||||
- [Gitea API Documentation](https://docs.gitea.io/en-us/api-usage/)
|
||||
|
||||
@@ -15,9 +15,10 @@ This document tracks the implementation status and estimated effort for all Arch
|
||||
|
||||
| Status | Count |
|
||||
| ---------------------------- | ----- |
|
||||
| Accepted (Fully Implemented) | 40 |
|
||||
| Accepted (Fully Implemented) | 41 |
|
||||
| Partially Implemented | 2 |
|
||||
| Proposed (Not Started) | 14 |
|
||||
| Proposed (Not Started) | 13 |
|
||||
| Superseded | 1 |
|
||||
|
||||
---
|
||||
|
||||
@@ -34,13 +35,13 @@ This document tracks the implementation status and estimated effort for all Arch
|
||||
|
||||
### Category 2: Data Management
|
||||
|
||||
| ADR | Title | Status | Effort | Notes |
|
||||
| --------------------------------------------------------------- | ------------------------ | -------- | ------ | ------------------------------ |
|
||||
| [ADR-009](./0009-caching-strategy-for-read-heavy-operations.md) | Caching Strategy | Accepted | - | Fully implemented |
|
||||
| [ADR-013](./0013-database-schema-migration-strategy.md) | Schema Migrations v1 | Proposed | M | Superseded by ADR-023 |
|
||||
| [ADR-019](./0019-data-backup-and-recovery-strategy.md) | Backup & Recovery | Accepted | - | Fully implemented |
|
||||
| [ADR-023](./0023-database-schema-migration-strategy.md) | Schema Migrations v2 | Proposed | L | Requires tooling setup |
|
||||
| [ADR-031](./0031-data-retention-and-privacy-compliance.md) | Data Retention & Privacy | Proposed | XL | Legal/compliance review needed |
|
||||
| ADR | Title | Status | Effort | Notes |
|
||||
| --------------------------------------------------------------- | ------------------------ | ---------- | ------ | ------------------------------ |
|
||||
| [ADR-009](./0009-caching-strategy-for-read-heavy-operations.md) | Caching Strategy | Accepted | - | Fully implemented |
|
||||
| [ADR-013](./0013-database-schema-migration-strategy.md) | Schema Migrations v1 | Superseded | - | Superseded by ADR-023 |
|
||||
| [ADR-019](./0019-data-backup-and-recovery-strategy.md) | Backup & Recovery | Accepted | - | Fully implemented |
|
||||
| [ADR-023](./0023-database-schema-migration-strategy.md) | Schema Migrations v2 | Proposed | L | Requires tooling setup |
|
||||
| [ADR-031](./0031-data-retention-and-privacy-compliance.md) | Data Retention & Privacy | Proposed | XL | Legal/compliance review needed |
|
||||
|
||||
### Category 3: API & Integration
|
||||
|
||||
@@ -108,6 +109,7 @@ This document tracks the implementation status and estimated effort for all Arch
|
||||
| [ADR-040](./0040-testing-economics-and-priorities.md) | Testing Economics | Accepted | - | Fully implemented |
|
||||
| [ADR-045](./0045-test-data-factories-and-fixtures.md) | Test Data Factories | Accepted | - | Fully implemented |
|
||||
| [ADR-047](./0047-project-file-and-folder-organization.md) | Project Organization | Proposed | XL | Major reorganization |
|
||||
| [ADR-057](./0057-test-remediation-post-api-versioning.md) | Test Remediation | Accepted | - | Fully implemented |
|
||||
|
||||
### Category 9: Architecture Patterns
|
||||
|
||||
@@ -148,6 +150,8 @@ These ADRs are proposed or partially implemented, ordered by suggested implement
|
||||
|
||||
| Date | ADR | Change |
|
||||
| ---------- | ------- | ----------------------------------------------------------------------------------- |
|
||||
| 2026-01-28 | ADR-057 | Created - Test remediation documentation for ADR-008 Phase 2 migration |
|
||||
| 2026-01-28 | ADR-013 | Marked as Superseded by ADR-023 |
|
||||
| 2026-01-27 | ADR-008 | Test path migration complete - 23 files, ~70 paths updated, 274->345 tests passing |
|
||||
| 2026-01-27 | ADR-008 | Phase 2 Complete - Version router factory, deprecation headers, 82 versioning tests |
|
||||
| 2026-01-26 | ADR-015 | Completed - Added Sentry user context in AuthProvider, now fully implemented |
|
||||
|
||||
@@ -2,6 +2,8 @@
|
||||
|
||||
This directory contains a log of the architectural decisions made for the Flyer Crawler project.
|
||||
|
||||
**[Implementation Tracker](./adr-implementation-tracker.md)**: Track implementation status and effort estimates for all ADRs.
|
||||
|
||||
## 1. Foundational / Core Infrastructure
|
||||
|
||||
**[ADR-002](./0002-standardized-transaction-management.md)**: Standardized Transaction Management and Unit of Work Pattern (Accepted)
|
||||
@@ -12,7 +14,7 @@ This directory contains a log of the architectural decisions made for the Flyer
|
||||
## 2. Data Management
|
||||
|
||||
**[ADR-009](./0009-caching-strategy-for-read-heavy-operations.md)**: Caching Strategy for Read-Heavy Operations (Accepted)
|
||||
**[ADR-013](./0013-database-schema-migration-strategy.md)**: Database Schema Migration Strategy (Proposed)
|
||||
**[ADR-013](./0013-database-schema-migration-strategy.md)**: Database Schema Migration Strategy (Superseded by ADR-023)
|
||||
**[ADR-019](./0019-data-backup-and-recovery-strategy.md)**: Data Backup and Recovery Strategy (Accepted)
|
||||
**[ADR-023](./0023-database-schema-migration-strategy.md)**: Database Schema Migration Strategy (Proposed)
|
||||
**[ADR-031](./0031-data-retention-and-privacy-compliance.md)**: Data Retention and Privacy Compliance (Proposed)
|
||||
@@ -20,9 +22,9 @@ This directory contains a log of the architectural decisions made for the Flyer
|
||||
## 3. API & Integration
|
||||
|
||||
**[ADR-003](./0003-standardized-input-validation-using-middleware.md)**: Standardized Input Validation using Middleware (Accepted)
|
||||
**[ADR-008](./0008-api-versioning-strategy.md)**: API Versioning Strategy (Accepted - Phase 1 Complete)
|
||||
**[ADR-008](./0008-api-versioning-strategy.md)**: API Versioning Strategy (Accepted - Phase 2 Complete)
|
||||
**[ADR-018](./0018-api-documentation-strategy.md)**: API Documentation Strategy (Accepted)
|
||||
**[ADR-022](./0022-real-time-notification-system.md)**: Real-time Notification System (Proposed)
|
||||
**[ADR-022](./0022-real-time-notification-system.md)**: Real-time Notification System (Accepted)
|
||||
**[ADR-028](./0028-api-response-standardization.md)**: API Response Standardization and Envelope Pattern (Implemented)
|
||||
|
||||
## 4. Security & Compliance
|
||||
@@ -33,12 +35,12 @@ This directory contains a log of the architectural decisions made for the Flyer
|
||||
**[ADR-029](./0029-secret-rotation-and-key-management.md)**: Secret Rotation and Key Management Strategy (Proposed)
|
||||
**[ADR-032](./0032-rate-limiting-strategy.md)**: Rate Limiting Strategy (Accepted)
|
||||
**[ADR-033](./0033-file-upload-and-storage-strategy.md)**: File Upload and Storage Strategy (Accepted)
|
||||
**[ADR-048](./0048-authentication-strategy.md)**: Authentication Strategy (Partially Implemented)
|
||||
**[ADR-048](./0048-authentication-strategy.md)**: Authentication Strategy (Accepted)
|
||||
|
||||
## 5. Observability & Monitoring
|
||||
|
||||
**[ADR-004](./0004-standardized-application-wide-structured-logging.md)**: Standardized Application-Wide Structured Logging (Accepted)
|
||||
**[ADR-015](./0015-error-tracking-and-observability.md)**: Error Tracking and Observability (Partial)
|
||||
**[ADR-015](./0015-error-tracking-and-observability.md)**: Error Tracking and Observability (Accepted)
|
||||
**[ADR-050](./0050-postgresql-function-observability.md)**: PostgreSQL Function Observability (Accepted)
|
||||
**[ADR-051](./0051-asynchronous-context-propagation.md)**: Asynchronous Context Propagation (Accepted)
|
||||
**[ADR-052](./0052-granular-debug-logging-strategy.md)**: Granular Debug Logging Strategy (Accepted)
|
||||
@@ -52,7 +54,7 @@ This directory contains a log of the architectural decisions made for the Flyer
|
||||
**[ADR-024](./0024-feature-flagging-strategy.md)**: Feature Flagging Strategy (Proposed)
|
||||
**[ADR-037](./0037-scheduled-jobs-and-cron-pattern.md)**: Scheduled Jobs and Cron Pattern (Accepted)
|
||||
**[ADR-038](./0038-graceful-shutdown-pattern.md)**: Graceful Shutdown Pattern (Accepted)
|
||||
**[ADR-053](./0053-worker-health-checks-and-monitoring.md)**: Worker Health Checks and Monitoring (Proposed)
|
||||
**[ADR-053](./0053-worker-health-checks.md)**: Worker Health Checks and Stalled Job Monitoring (Accepted)
|
||||
**[ADR-054](./0054-bugsink-gitea-issue-sync.md)**: Bugsink to Gitea Issue Synchronization (Proposed)
|
||||
|
||||
## 7. Frontend / User Interface
|
||||
|
||||
@@ -1,10 +1,168 @@
|
||||
# Database Setup
|
||||
# Database Architecture
|
||||
|
||||
Flyer Crawler uses PostgreSQL with several extensions for full-text search, geographic data, and UUID generation.
|
||||
**Version**: 0.12.20
|
||||
**Last Updated**: 2026-01-28
|
||||
|
||||
Flyer Crawler uses PostgreSQL 16 with PostGIS for geographic data, pg_trgm for fuzzy text search, and uuid-ossp for UUID generation. The database contains 65 tables organized into logical domains.
|
||||
|
||||
## Table of Contents
|
||||
|
||||
1. [Schema Overview](#schema-overview)
|
||||
2. [Database Setup](#database-setup)
|
||||
3. [Schema Reference](#schema-reference)
|
||||
4. [Related Documentation](#related-documentation)
|
||||
|
||||
---
|
||||
|
||||
## Required Extensions
|
||||
## Schema Overview
|
||||
|
||||
The database is organized into the following domains:
|
||||
|
||||
### Core Infrastructure (6 tables)
|
||||
|
||||
| Table | Purpose | Primary Key |
|
||||
| ----------------------- | ----------------------------------------- | ----------------- |
|
||||
| `users` | Authentication credentials and login data | `user_id` (UUID) |
|
||||
| `profiles` | Public user data, preferences, points | `user_id` (UUID) |
|
||||
| `addresses` | Normalized address storage with geocoding | `address_id` |
|
||||
| `activity_log` | User activity audit trail | `activity_log_id` |
|
||||
| `password_reset_tokens` | Temporary tokens for password reset | `token_id` |
|
||||
| `schema_info` | Schema deployment metadata | `environment` |
|
||||
|
||||
### Stores and Locations (4 tables)
|
||||
|
||||
| Table | Purpose | Primary Key |
|
||||
| ------------------------ | --------------------------------------- | ------------------- |
|
||||
| `stores` | Grocery store chains (Safeway, Kroger) | `store_id` |
|
||||
| `store_locations` | Physical store locations with addresses | `store_location_id` |
|
||||
| `favorite_stores` | User store favorites | `user_id, store_id` |
|
||||
| `store_receipt_patterns` | Receipt text patterns for store ID | `pattern_id` |
|
||||
|
||||
### Flyers and Items (7 tables)
|
||||
|
||||
| Table | Purpose | Primary Key |
|
||||
| ----------------------- | -------------------------------------- | ------------------------ |
|
||||
| `flyers` | Uploaded flyer metadata and status | `flyer_id` |
|
||||
| `flyer_items` | Individual deals extracted from flyers | `flyer_item_id` |
|
||||
| `flyer_locations` | Flyer-to-location associations | `flyer_location_id` |
|
||||
| `categories` | Item categorization (Produce, Dairy) | `category_id` |
|
||||
| `master_grocery_items` | Canonical grocery item dictionary | `master_grocery_item_id` |
|
||||
| `master_item_aliases` | Alternative names for master items | `alias_id` |
|
||||
| `unmatched_flyer_items` | Items pending master item matching | `unmatched_item_id` |
|
||||
|
||||
### Products and Brands (2 tables)
|
||||
|
||||
| Table | Purpose | Primary Key |
|
||||
| ---------- | ---------------------------------------------- | ------------ |
|
||||
| `brands` | Brand names (Coca-Cola, Kraft) | `brand_id` |
|
||||
| `products` | Specific products (master item + brand + size) | `product_id` |
|
||||
|
||||
### Price Tracking (3 tables)
|
||||
|
||||
| Table | Purpose | Primary Key |
|
||||
| ----------------------- | ---------------------------------- | ------------------ |
|
||||
| `item_price_history` | Historical prices for master items | `price_history_id` |
|
||||
| `user_submitted_prices` | User-contributed price reports | `submission_id` |
|
||||
| `suggested_corrections` | Suggested edits to flyer items | `correction_id` |
|
||||
|
||||
### User Features (8 tables)
|
||||
|
||||
| Table | Purpose | Primary Key |
|
||||
| -------------------- | ------------------------------------ | --------------------------- |
|
||||
| `user_watched_items` | Items user wants to track prices for | `user_watched_item_id` |
|
||||
| `user_alerts` | Price alert thresholds | `alert_id` |
|
||||
| `notifications` | User notifications | `notification_id` |
|
||||
| `user_item_aliases` | User-defined item name aliases | `alias_id` |
|
||||
| `user_follows` | User-to-user follow relationships | `follower_id, following_id` |
|
||||
| `user_reactions` | Reactions to content (likes, etc.) | `reaction_id` |
|
||||
| `budgets` | User-defined spending budgets | `budget_id` |
|
||||
| `search_queries` | Search history for analytics | `query_id` |
|
||||
|
||||
### Shopping Lists (4 tables)
|
||||
|
||||
| Table | Purpose | Primary Key |
|
||||
| ----------------------- | ------------------------ | ------------------------- |
|
||||
| `shopping_lists` | User shopping lists | `shopping_list_id` |
|
||||
| `shopping_list_items` | Items on shopping lists | `shopping_list_item_id` |
|
||||
| `shared_shopping_lists` | Shopping list sharing | `shared_shopping_list_id` |
|
||||
| `shopping_trips` | Completed shopping trips | `trip_id` |
|
||||
| `shopping_trip_items` | Items purchased on trips | `trip_item_id` |
|
||||
|
||||
### Recipes (11 tables)
|
||||
|
||||
| Table | Purpose | Primary Key |
|
||||
| --------------------------------- | -------------------------------- | ------------------------- |
|
||||
| `recipes` | User recipes with metadata | `recipe_id` |
|
||||
| `recipe_ingredients` | Recipe ingredient list | `recipe_ingredient_id` |
|
||||
| `recipe_ingredient_substitutions` | Ingredient alternatives | `substitution_id` |
|
||||
| `tags` | Recipe tags (vegan, quick, etc.) | `tag_id` |
|
||||
| `recipe_tags` | Recipe-to-tag associations | `recipe_id, tag_id` |
|
||||
| `appliances` | Kitchen appliances | `appliance_id` |
|
||||
| `recipe_appliances` | Appliances needed for recipes | `recipe_id, appliance_id` |
|
||||
| `recipe_ratings` | User ratings for recipes | `rating_id` |
|
||||
| `recipe_comments` | User comments on recipes | `comment_id` |
|
||||
| `favorite_recipes` | User recipe favorites | `user_id, recipe_id` |
|
||||
| `recipe_collections` | User recipe collections | `collection_id` |
|
||||
|
||||
### Meal Planning (3 tables)
|
||||
|
||||
| Table | Purpose | Primary Key |
|
||||
| ------------------- | -------------------------- | ----------------- |
|
||||
| `menu_plans` | Weekly/monthly meal plans | `menu_plan_id` |
|
||||
| `shared_menu_plans` | Menu plan sharing | `share_id` |
|
||||
| `planned_meals` | Individual meals in a plan | `planned_meal_id` |
|
||||
|
||||
### Pantry and Inventory (4 tables)
|
||||
|
||||
| Table | Purpose | Primary Key |
|
||||
| -------------------- | ------------------------------------ | ----------------- |
|
||||
| `pantry_items` | User pantry inventory | `pantry_item_id` |
|
||||
| `pantry_locations` | Storage locations (fridge, freezer) | `location_id` |
|
||||
| `expiry_date_ranges` | Reference shelf life data | `expiry_range_id` |
|
||||
| `expiry_alerts` | User expiry notification preferences | `expiry_alert_id` |
|
||||
| `expiry_alert_log` | Sent expiry notifications | `alert_log_id` |
|
||||
|
||||
### Receipts (4 tables)
|
||||
|
||||
| Table | Purpose | Primary Key |
|
||||
| ------------------------ | ----------------------------- | ----------------- |
|
||||
| `receipts` | Scanned receipt metadata | `receipt_id` |
|
||||
| `receipt_items` | Items parsed from receipts | `receipt_item_id` |
|
||||
| `receipt_processing_log` | OCR/AI processing audit trail | `log_id` |
|
||||
|
||||
### UPC Scanning (2 tables)
|
||||
|
||||
| Table | Purpose | Primary Key |
|
||||
| ---------------------- | ------------------------------- | ----------- |
|
||||
| `upc_scan_history` | User barcode scan history | `scan_id` |
|
||||
| `upc_external_lookups` | External UPC API response cache | `lookup_id` |
|
||||
|
||||
### Gamification (2 tables)
|
||||
|
||||
| Table | Purpose | Primary Key |
|
||||
| ------------------- | ---------------------------- | ------------------------- |
|
||||
| `achievements` | Defined achievements | `achievement_id` |
|
||||
| `user_achievements` | Achievements earned by users | `user_id, achievement_id` |
|
||||
|
||||
### User Preferences (3 tables)
|
||||
|
||||
| Table | Purpose | Primary Key |
|
||||
| --------------------------- | ---------------------------- | ------------------------- |
|
||||
| `dietary_restrictions` | Defined dietary restrictions | `restriction_id` |
|
||||
| `user_dietary_restrictions` | User dietary preferences | `user_id, restriction_id` |
|
||||
| `user_appliances` | Appliances user owns | `user_id, appliance_id` |
|
||||
|
||||
### Reference Data (1 table)
|
||||
|
||||
| Table | Purpose | Primary Key |
|
||||
| ------------------ | ----------------------- | --------------- |
|
||||
| `unit_conversions` | Unit conversion factors | `conversion_id` |
|
||||
|
||||
---
|
||||
|
||||
## Database Setup
|
||||
|
||||
### Required Extensions
|
||||
|
||||
| Extension | Purpose |
|
||||
| ----------- | ------------------------------------------- |
|
||||
@@ -14,7 +172,7 @@ Flyer Crawler uses PostgreSQL with several extensions for full-text search, geog
|
||||
|
||||
---
|
||||
|
||||
## Database Users
|
||||
### Database Users
|
||||
|
||||
This project uses **environment-specific database users** to isolate production and test environments:
|
||||
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
# Flyer Crawler - System Architecture Overview
|
||||
|
||||
**Version**: 0.12.5
|
||||
**Last Updated**: 2026-01-22
|
||||
**Version**: 0.12.20
|
||||
**Last Updated**: 2026-01-28
|
||||
**Platform**: Linux (Production and Development)
|
||||
|
||||
---
|
||||
@@ -41,7 +41,7 @@
|
||||
|
||||
## System Architecture Diagram
|
||||
|
||||
```
|
||||
```text
|
||||
+-----------------------------------------------------------------------------------+
|
||||
| CLIENT LAYER |
|
||||
+-----------------------------------------------------------------------------------+
|
||||
@@ -153,10 +153,10 @@
|
||||
| Component | Technology | Version | Purpose |
|
||||
| ---------------------- | ---------- | -------- | -------------------------------- |
|
||||
| **Runtime** | Node.js | 22.x LTS | Server-side JavaScript runtime |
|
||||
| **Language** | TypeScript | 5.9.x | Type-safe JavaScript superset |
|
||||
| **Web Framework** | Express.js | 5.1.x | HTTP server and routing |
|
||||
| **Frontend Framework** | React | 19.2.x | UI component library |
|
||||
| **Build Tool** | Vite | 7.2.x | Frontend bundling and dev server |
|
||||
| **Language** | TypeScript | 5.9.3 | Type-safe JavaScript superset |
|
||||
| **Web Framework** | Express.js | 5.1.0 | HTTP server and routing |
|
||||
| **Frontend Framework** | React | 19.2.0 | UI component library |
|
||||
| **Build Tool** | Vite | 7.2.4 | Frontend bundling and dev server |
|
||||
|
||||
### Data Storage
|
||||
|
||||
@@ -176,23 +176,23 @@
|
||||
| **OAuth** | Google, GitHub | Social authentication |
|
||||
| **Email** | Nodemailer (SMTP) | Transactional emails |
|
||||
|
||||
### Background Processing
|
||||
### Background Processing Stack
|
||||
|
||||
| Component | Technology | Version | Purpose |
|
||||
| ------------------- | ---------- | ------- | --------------------------------- |
|
||||
| **Job Queues** | BullMQ | 5.65.x | Reliable async job processing |
|
||||
| **Job Queues** | BullMQ | 5.65.1 | Reliable async job processing |
|
||||
| **Process Manager** | PM2 | Latest | Process management and clustering |
|
||||
| **Scheduler** | node-cron | 4.2.x | Scheduled tasks |
|
||||
| **Scheduler** | node-cron | 4.2.1 | Scheduled tasks |
|
||||
|
||||
### Frontend Stack
|
||||
|
||||
| Component | Technology | Version | Purpose |
|
||||
| -------------------- | -------------- | ------- | ---------------------------------------- |
|
||||
| **State Management** | TanStack Query | 5.90.x | Server state caching and synchronization |
|
||||
| **Routing** | React Router | 7.9.x | Client-side routing |
|
||||
| **Styling** | Tailwind CSS | 4.1.x | Utility-first CSS framework |
|
||||
| **Icons** | Lucide React | 0.555.x | Icon components |
|
||||
| **Charts** | Recharts | 3.4.x | Data visualization |
|
||||
| **State Management** | TanStack Query | 5.90.12 | Server state caching and synchronization |
|
||||
| **Routing** | React Router | 7.9.6 | Client-side routing |
|
||||
| **Styling** | Tailwind CSS | 4.1.17 | Utility-first CSS framework |
|
||||
| **Icons** | Lucide React | 0.555.0 | Icon components |
|
||||
| **Charts** | Recharts | 3.4.1 | Data visualization |
|
||||
|
||||
### Observability and Quality
|
||||
|
||||
@@ -221,7 +221,7 @@ The frontend is a single-page application (SPA) built with React 19 and Vite.
|
||||
|
||||
**Directory Structure**:
|
||||
|
||||
```
|
||||
```text
|
||||
src/
|
||||
+-- components/ # Reusable UI components
|
||||
+-- contexts/ # React context providers
|
||||
@@ -244,17 +244,30 @@ The backend is a RESTful API server built with Express.js 5.
|
||||
- Structured logging with Pino
|
||||
- Standardized error handling (ADR-001)
|
||||
|
||||
**API Route Modules**:
|
||||
| Route | Purpose |
|
||||
|-------|---------|
|
||||
| `/api/auth` | Authentication (login, register, OAuth) |
|
||||
| `/api/users` | User profile management |
|
||||
| `/api/flyers` | Flyer CRUD and processing |
|
||||
| `/api/recipes` | Recipe management |
|
||||
| `/api/deals` | Best prices and deal discovery |
|
||||
| `/api/stores` | Store management |
|
||||
| `/api/admin` | Administrative functions |
|
||||
| `/api/health` | Health checks and monitoring |
|
||||
**API Route Modules** (all versioned under `/api/v1/*`):
|
||||
|
||||
| Route | Purpose |
|
||||
| ------------------------- | ----------------------------------------------- |
|
||||
| `/api/v1/auth` | Authentication (login, register, OAuth) |
|
||||
| `/api/v1/health` | Health checks and monitoring |
|
||||
| `/api/v1/system` | System administration (PM2 status, server info) |
|
||||
| `/api/v1/users` | User profile management |
|
||||
| `/api/v1/ai` | AI-powered features and flyer processing |
|
||||
| `/api/v1/admin` | Administrative functions |
|
||||
| `/api/v1/budgets` | Budget management and spending analysis |
|
||||
| `/api/v1/achievements` | Gamification and achievement system |
|
||||
| `/api/v1/flyers` | Flyer CRUD and processing |
|
||||
| `/api/v1/recipes` | Recipe management and recommendations |
|
||||
| `/api/v1/personalization` | Master items and user preferences |
|
||||
| `/api/v1/price-history` | Price tracking and trend analysis |
|
||||
| `/api/v1/stats` | Public statistics and analytics |
|
||||
| `/api/v1/upc` | UPC barcode scanning and product lookup |
|
||||
| `/api/v1/inventory` | Inventory and expiry tracking |
|
||||
| `/api/v1/receipts` | Receipt scanning and purchase history |
|
||||
| `/api/v1/deals` | Best prices and deal discovery |
|
||||
| `/api/v1/reactions` | Social features (reactions, sharing) |
|
||||
| `/api/v1/stores` | Store management and location services |
|
||||
| `/api/v1/categories` | Category browsing and product categorization |
|
||||
|
||||
### Database (PostgreSQL/PostGIS)
|
||||
|
||||
@@ -331,7 +344,7 @@ BullMQ workers handle asynchronous processing tasks. PM2 manages both the API se
|
||||
|
||||
### Flyer Processing Pipeline
|
||||
|
||||
```
|
||||
```text
|
||||
+-------------+ +----------------+ +------------------+ +---------------+
|
||||
| User | | Express | | BullMQ | | PostgreSQL |
|
||||
| Upload +---->+ Route +---->+ Queue +---->+ Storage |
|
||||
@@ -395,7 +408,7 @@ BullMQ workers handle asynchronous processing tasks. PM2 manages both the API se
|
||||
|
||||
The application follows a strict layered architecture as defined in ADR-035.
|
||||
|
||||
```
|
||||
```text
|
||||
+-----------------------------------------------------------------------+
|
||||
| ROUTES LAYER |
|
||||
| Responsibilities: |
|
||||
@@ -458,7 +471,7 @@ The application follows a strict layered architecture as defined in ADR-035.
|
||||
|
||||
### Entity Relationship Overview
|
||||
|
||||
```
|
||||
```text
|
||||
+------------------+ +------------------+ +------------------+
|
||||
| users | | profiles | | addresses |
|
||||
|------------------| |------------------| |------------------|
|
||||
@@ -537,7 +550,7 @@ The application follows a strict layered architecture as defined in ADR-035.
|
||||
|
||||
### JWT Token Architecture
|
||||
|
||||
```
|
||||
```text
|
||||
+-------------------+ +-------------------+ +-------------------+
|
||||
| Login Request | | Server | | Database |
|
||||
| (email/pass) +---->+ Validates +---->+ Verify User |
|
||||
@@ -576,7 +589,7 @@ The application follows a strict layered architecture as defined in ADR-035.
|
||||
|
||||
### Protected Route Flow
|
||||
|
||||
```
|
||||
```text
|
||||
+-------------------+ +-------------------+ +-------------------+
|
||||
| API Request | | requireAuth | | JWT Strategy |
|
||||
| + Bearer Token +---->+ Middleware +---->+ Validate |
|
||||
@@ -603,7 +616,7 @@ The application follows a strict layered architecture as defined in ADR-035.
|
||||
|
||||
### Worker Architecture
|
||||
|
||||
```
|
||||
```text
|
||||
+-------------------+ +-------------------+ +-------------------+
|
||||
| API Server | | Redis | | Worker Process |
|
||||
| (Queue Producer)| | (Job Storage) | | (Consumer) |
|
||||
@@ -635,7 +648,7 @@ The application follows a strict layered architecture as defined in ADR-035.
|
||||
|
||||
Jobs use exponential backoff for retries:
|
||||
|
||||
```
|
||||
```text
|
||||
Attempt 1: Immediate
|
||||
Attempt 2: Initial delay (e.g., 5 seconds)
|
||||
Attempt 3: 2x delay (e.g., 10 seconds)
|
||||
@@ -658,7 +671,7 @@ Attempt 4: 4x delay (e.g., 20 seconds)
|
||||
|
||||
### Environment Overview
|
||||
|
||||
```
|
||||
```text
|
||||
+-----------------------------------------------------------------------------------+
|
||||
| DEVELOPMENT |
|
||||
+-----------------------------------------------------------------------------------+
|
||||
@@ -710,7 +723,7 @@ Attempt 4: 4x delay (e.g., 20 seconds)
|
||||
|
||||
### Deployment Pipeline (ADR-017)
|
||||
|
||||
```
|
||||
```text
|
||||
+------------+ +------------+ +------------+ +------------+
|
||||
| Push to | | Gitea | | Build & | | Deploy |
|
||||
| main +---->+ Actions +---->+ Test +---->+ to Prod |
|
||||
@@ -839,22 +852,55 @@ The system architecture is governed by Architecture Decision Records (ADRs). Key
|
||||
| File | Purpose |
|
||||
| ----------------------------------------------- | --------------------------------------- |
|
||||
| `src/services/flyerProcessingService.server.ts` | Flyer processing pipeline orchestration |
|
||||
| `src/services/flyerAiProcessor.server.ts` | AI extraction for flyers |
|
||||
| `src/services/aiService.server.ts` | Google Gemini AI integration |
|
||||
| `src/services/cacheService.server.ts` | Redis caching abstraction |
|
||||
| `src/services/emailService.server.ts` | Email sending |
|
||||
| `src/services/queues.server.ts` | BullMQ queue definitions |
|
||||
| `src/services/queueService.server.ts` | Queue management and scheduling |
|
||||
| `src/services/workers.server.ts` | BullMQ worker definitions |
|
||||
| `src/services/websocketService.server.ts` | Real-time WebSocket notifications |
|
||||
| `src/services/receiptService.server.ts` | Receipt scanning and OCR |
|
||||
| `src/services/upcService.server.ts` | UPC barcode lookup |
|
||||
| `src/services/expiryService.server.ts` | Pantry expiry tracking |
|
||||
| `src/services/geocodingService.server.ts` | Address geocoding |
|
||||
| `src/services/analyticsService.server.ts` | Analytics and reporting |
|
||||
| `src/services/monitoringService.server.ts` | Health monitoring |
|
||||
| `src/services/barcodeService.server.ts` | Barcode detection |
|
||||
| `src/services/logger.server.ts` | Structured logging (Pino) |
|
||||
| `src/services/redis.server.ts` | Redis connection management |
|
||||
| `src/services/sentry.server.ts` | Error tracking (Sentry/Bugsink) |
|
||||
|
||||
### Database Files
|
||||
|
||||
| File | Purpose |
|
||||
| ---------------------------------- | -------------------------------------------- |
|
||||
| `src/services/db/connection.db.ts` | Database pool and transaction management |
|
||||
| `src/services/db/errors.db.ts` | Database error types |
|
||||
| `src/services/db/user.db.ts` | User repository |
|
||||
| `src/services/db/flyer.db.ts` | Flyer repository |
|
||||
| `sql/master_schema_rollup.sql` | Complete database schema (for test DB setup) |
|
||||
| `sql/initial_schema.sql` | Fresh installation schema |
|
||||
| File | Purpose |
|
||||
| --------------------------------------- | -------------------------------------------- |
|
||||
| `src/services/db/connection.db.ts` | Database pool and transaction management |
|
||||
| `src/services/db/errors.db.ts` | Database error types |
|
||||
| `src/services/db/index.db.ts` | Repository exports |
|
||||
| `src/services/db/user.db.ts` | User repository |
|
||||
| `src/services/db/flyer.db.ts` | Flyer repository |
|
||||
| `src/services/db/store.db.ts` | Store repository |
|
||||
| `src/services/db/storeLocation.db.ts` | Store location repository |
|
||||
| `src/services/db/recipe.db.ts` | Recipe repository |
|
||||
| `src/services/db/category.db.ts` | Category repository |
|
||||
| `src/services/db/personalization.db.ts` | Master items and personalization |
|
||||
| `src/services/db/shopping.db.ts` | Shopping lists repository |
|
||||
| `src/services/db/deals.db.ts` | Deals and best prices repository |
|
||||
| `src/services/db/price.db.ts` | Price history repository |
|
||||
| `src/services/db/receipt.db.ts` | Receipt repository |
|
||||
| `src/services/db/upc.db.ts` | UPC scan history repository |
|
||||
| `src/services/db/expiry.db.ts` | Expiry tracking repository |
|
||||
| `src/services/db/gamification.db.ts` | Achievements repository |
|
||||
| `src/services/db/budget.db.ts` | Budget repository |
|
||||
| `src/services/db/reaction.db.ts` | User reactions repository |
|
||||
| `src/services/db/notification.db.ts` | Notifications repository |
|
||||
| `src/services/db/address.db.ts` | Address repository |
|
||||
| `src/services/db/admin.db.ts` | Admin operations repository |
|
||||
| `src/services/db/conversion.db.ts` | Unit conversion repository |
|
||||
| `src/services/db/flyerLocation.db.ts` | Flyer locations repository |
|
||||
| `sql/master_schema_rollup.sql` | Complete database schema (for test DB setup) |
|
||||
| `sql/initial_schema.sql` | Fresh installation schema |
|
||||
|
||||
### Type Definitions
|
||||
|
||||
|
||||
@@ -2,6 +2,21 @@
|
||||
|
||||
Common code patterns extracted from Architecture Decision Records (ADRs). Use these as templates when writing new code.
|
||||
|
||||
## Quick Reference
|
||||
|
||||
| Pattern | Key Function/Class | Import From |
|
||||
| ------------------ | ------------------------------------------------- | ---------------------------------- |
|
||||
| Error Handling | `handleDbError()`, `NotFoundError` | `src/services/db/errors.db.ts` |
|
||||
| Repository Methods | `get*`, `find*`, `list*` | `src/services/db/*.db.ts` |
|
||||
| API Responses | `sendSuccess()`, `sendPaginated()`, `sendError()` | `src/utils/apiResponse.ts` |
|
||||
| Transactions | `withTransaction()` | `src/services/db/connection.db.ts` |
|
||||
| Validation | `validateRequest()` | `src/middleware/validation.ts` |
|
||||
| Authentication | `authenticateJWT` | `src/middleware/auth.ts` |
|
||||
| Caching | `cacheService` | `src/services/cache.server.ts` |
|
||||
| Background Jobs | Queue classes | `src/services/queues.server.ts` |
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
|
||||
- [Error Handling](#error-handling)
|
||||
@@ -17,7 +32,7 @@ Common code patterns extracted from Architecture Decision Records (ADRs). Use th
|
||||
|
||||
## Error Handling
|
||||
|
||||
**ADR**: [ADR-001](../adr/0001-standardized-error-handling-for-database-operations.md)
|
||||
**ADR**: [ADR-001](../adr/0001-standardized-error-handling.md)
|
||||
|
||||
### Repository Layer Error Handling
|
||||
|
||||
@@ -78,7 +93,7 @@ throw new DatabaseError('Failed to insert flyer', originalError);
|
||||
|
||||
## Repository Patterns
|
||||
|
||||
**ADR**: [ADR-034](../adr/0034-repository-layer-method-naming-conventions.md)
|
||||
**ADR**: [ADR-034](../adr/0034-repository-pattern-standards.md)
|
||||
|
||||
### Method Naming Conventions
|
||||
|
||||
@@ -155,16 +170,17 @@ export async function listActiveFlyers(client?: PoolClient): Promise<Flyer[]> {
|
||||
|
||||
## API Response Patterns
|
||||
|
||||
**ADR**: [ADR-028](../adr/0028-consistent-api-response-format.md)
|
||||
**ADR**: [ADR-028](../adr/0028-api-response-standardization.md)
|
||||
|
||||
### Success Response
|
||||
|
||||
```typescript
|
||||
import { sendSuccess } from '../utils/apiResponse';
|
||||
|
||||
app.post('/api/flyers', async (req, res) => {
|
||||
app.post('/api/v1/flyers', async (req, res) => {
|
||||
const flyer = await flyerService.createFlyer(req.body);
|
||||
return sendSuccess(res, flyer, 'Flyer created successfully', 201);
|
||||
// sendSuccess(res, data, statusCode?, meta?)
|
||||
return sendSuccess(res, flyer, 201);
|
||||
});
|
||||
```
|
||||
|
||||
@@ -173,30 +189,32 @@ app.post('/api/flyers', async (req, res) => {
|
||||
```typescript
|
||||
import { sendPaginated } from '../utils/apiResponse';
|
||||
|
||||
app.get('/api/flyers', async (req, res) => {
|
||||
const { page = 1, pageSize = 20 } = req.query;
|
||||
const { items, total } = await flyerService.listFlyers(page, pageSize);
|
||||
app.get('/api/v1/flyers', async (req, res) => {
|
||||
const page = parseInt(req.query.page as string) || 1;
|
||||
const limit = parseInt(req.query.limit as string) || 20;
|
||||
const { items, total } = await flyerService.listFlyers(page, limit);
|
||||
|
||||
return sendPaginated(res, {
|
||||
items,
|
||||
total,
|
||||
page: parseInt(page),
|
||||
pageSize: parseInt(pageSize),
|
||||
});
|
||||
// sendPaginated(res, data[], { page, limit, total }, meta?)
|
||||
return sendPaginated(res, items, { page, limit, total });
|
||||
});
|
||||
```
|
||||
|
||||
### Error Response
|
||||
|
||||
```typescript
|
||||
import { sendError } from '../utils/apiResponse';
|
||||
import { sendError, sendSuccess, ErrorCode } from '../utils/apiResponse';
|
||||
|
||||
app.get('/api/flyers/:id', async (req, res) => {
|
||||
app.get('/api/v1/flyers/:id', async (req, res) => {
|
||||
try {
|
||||
const flyer = await flyerDb.getFlyerById(parseInt(req.params.id));
|
||||
return sendSuccess(res, flyer);
|
||||
} catch (error) {
|
||||
return sendError(res, error); // Automatically maps error to correct status
|
||||
// sendError(res, code, message, statusCode?, details?, meta?)
|
||||
if (error instanceof NotFoundError) {
|
||||
return sendError(res, ErrorCode.NOT_FOUND, error.message, 404);
|
||||
}
|
||||
req.log.error({ error }, `Error in ${req.originalUrl.split('?')[0]}:`);
|
||||
return sendError(res, ErrorCode.INTERNAL_ERROR, 'An error occurred', 500);
|
||||
}
|
||||
});
|
||||
```
|
||||
@@ -205,12 +223,12 @@ app.get('/api/flyers/:id', async (req, res) => {
|
||||
|
||||
## Transaction Management
|
||||
|
||||
**ADR**: [ADR-002](../adr/0002-transaction-management-pattern.md)
|
||||
**ADR**: [ADR-002](../adr/0002-standardized-transaction-management.md)
|
||||
|
||||
### Basic Transaction
|
||||
|
||||
```typescript
|
||||
import { withTransaction } from '../services/db/transaction.db';
|
||||
import { withTransaction } from '../services/db/connection.db';
|
||||
|
||||
export async function createFlyerWithItems(
|
||||
flyerData: FlyerInput,
|
||||
@@ -262,7 +280,7 @@ export async function bulkImportFlyers(flyersData: FlyerInput[]): Promise<Import
|
||||
|
||||
## Input Validation
|
||||
|
||||
**ADR**: [ADR-003](../adr/0003-input-validation-framework.md)
|
||||
**ADR**: [ADR-003](../adr/0003-standardized-input-validation-using-middleware.md)
|
||||
|
||||
### Zod Schema Definition
|
||||
|
||||
@@ -298,10 +316,10 @@ export type CreateFlyerInput = z.infer<typeof createFlyerSchema>;
|
||||
import { validateRequest } from '../middleware/validation';
|
||||
import { createFlyerSchema } from '../schemas/flyer.schemas';
|
||||
|
||||
app.post('/api/flyers', validateRequest(createFlyerSchema), async (req, res) => {
|
||||
app.post('/api/v1/flyers', validateRequest(createFlyerSchema), async (req, res) => {
|
||||
// req.body is now type-safe and validated
|
||||
const flyer = await flyerService.createFlyer(req.body);
|
||||
return sendSuccess(res, flyer, 'Flyer created successfully', 201);
|
||||
return sendSuccess(res, flyer, 201);
|
||||
});
|
||||
```
|
||||
|
||||
@@ -331,7 +349,7 @@ export async function processFlyer(data: unknown): Promise<Flyer> {
|
||||
import { authenticateJWT } from '../middleware/auth';
|
||||
|
||||
app.get(
|
||||
'/api/profile',
|
||||
'/api/v1/profile',
|
||||
authenticateJWT, // Middleware adds req.user
|
||||
async (req, res) => {
|
||||
// req.user is guaranteed to exist
|
||||
@@ -347,7 +365,7 @@ app.get(
|
||||
import { optionalAuth } from '../middleware/auth';
|
||||
|
||||
app.get(
|
||||
'/api/flyers',
|
||||
'/api/v1/flyers',
|
||||
optionalAuth, // req.user may or may not exist
|
||||
async (req, res) => {
|
||||
const flyers = req.user
|
||||
@@ -374,7 +392,7 @@ export function generateToken(user: User): string {
|
||||
|
||||
## Caching
|
||||
|
||||
**ADR**: [ADR-029](../adr/0029-redis-caching-strategy.md)
|
||||
**ADR**: [ADR-009](../adr/0009-caching-strategy-for-read-heavy-operations.md)
|
||||
|
||||
### Cache Pattern
|
||||
|
||||
@@ -414,7 +432,7 @@ export async function updateFlyer(id: number, data: UpdateFlyerInput): Promise<F
|
||||
|
||||
## Background Jobs
|
||||
|
||||
**ADR**: [ADR-036](../adr/0036-background-job-processing-architecture.md)
|
||||
**ADR**: [ADR-006](../adr/0006-background-job-processing-and-task-queues.md)
|
||||
|
||||
### Queue Job
|
||||
|
||||
|
||||
@@ -229,7 +229,7 @@ SELECT * FROM flyers WHERE store_id = 1;
|
||||
- Add missing indexes
|
||||
- Optimize WHERE clauses
|
||||
- Use connection pooling
|
||||
- See [ADR-034](../adr/0034-repository-layer-method-naming-conventions.md)
|
||||
- See [ADR-034](../adr/0034-repository-pattern-standards.md)
|
||||
|
||||
---
|
||||
|
||||
@@ -237,7 +237,7 @@ SELECT * FROM flyers WHERE store_id = 1;
|
||||
|
||||
### Tests Pass on Windows, Fail in Container
|
||||
|
||||
**Cause**: Platform-specific behavior (ADR-014)
|
||||
**Cause**: Platform-specific behavior ([ADR-014](../adr/0014-containerization-and-deployment-strategy.md))
|
||||
|
||||
**Rule**: Container results are authoritative. Windows results are unreliable.
|
||||
|
||||
|
||||
@@ -93,7 +93,7 @@ When the container starts (`scripts/dev-entrypoint.sh`):
|
||||
|
||||
PM2 manages three processes in the dev container:
|
||||
|
||||
```
|
||||
```text
|
||||
+--------------------+ +------------------------+ +--------------------+
|
||||
| flyer-crawler- | | flyer-crawler- | | flyer-crawler- |
|
||||
| api-dev | | worker-dev | | vite-dev |
|
||||
@@ -404,5 +404,5 @@ podman exec -it flyer-crawler-dev pm2 restart flyer-crawler-api-dev
|
||||
- [DEBUGGING.md](DEBUGGING.md) - Debugging strategies
|
||||
- [LOGSTASH-QUICK-REF.md](../operations/LOGSTASH-QUICK-REF.md) - Logstash quick reference
|
||||
- [DEV-CONTAINER-BUGSINK.md](../DEV-CONTAINER-BUGSINK.md) - Bugsink setup in dev container
|
||||
- [ADR-014](../adr/0014-linux-only-platform.md) - Linux-only platform decision
|
||||
- [ADR-050](../adr/0050-postgresql-function-observability.md) - PostgreSQL function observability
|
||||
- [ADR-014](../adr/0014-containerization-and-deployment-strategy.md) - Containerization and deployment strategy
|
||||
- [ADR-050](../adr/0050-postgresql-function-observability.md) - PostgreSQL function observability (includes log aggregation)
|
||||
|
||||
@@ -1,5 +1,19 @@
|
||||
# Testing Guide
|
||||
|
||||
## Quick Reference
|
||||
|
||||
| Command | Purpose |
|
||||
| ------------------------------------------------------------ | ---------------------------- |
|
||||
| `podman exec -it flyer-crawler-dev npm test` | Run all tests |
|
||||
| `podman exec -it flyer-crawler-dev npm run test:unit` | Unit tests (~2900) |
|
||||
| `podman exec -it flyer-crawler-dev npm run test:integration` | Integration tests (28 files) |
|
||||
| `podman exec -it flyer-crawler-dev npm run test:e2e` | E2E tests (11 files) |
|
||||
| `podman exec -it flyer-crawler-dev npm run type-check` | TypeScript check |
|
||||
|
||||
**Critical**: Always run tests in the dev container. Windows results are unreliable.
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
This project has comprehensive test coverage including unit tests, integration tests, and E2E tests. All tests must be run in the **Linux dev container environment** for reliable results.
|
||||
@@ -76,7 +90,7 @@ To verify type-check is working correctly:
|
||||
|
||||
Example error output:
|
||||
|
||||
```
|
||||
```text
|
||||
src/pages/MyDealsPage.tsx:68:31 - error TS2339: Property 'store_name' does not exist on type 'WatchedItemDeal'.
|
||||
|
||||
68 <span>{deal.store_name}</span>
|
||||
@@ -113,15 +127,26 @@ Located throughout `src/` directory alongside source files with `.test.ts` or `.
|
||||
npm run test:unit
|
||||
```
|
||||
|
||||
### Integration Tests (5 test files)
|
||||
### Integration Tests (28 test files)
|
||||
|
||||
Located in `src/tests/integration/`:
|
||||
Located in `src/tests/integration/`. Key test files include:
|
||||
|
||||
- `admin.integration.test.ts`
|
||||
- `flyer.integration.test.ts`
|
||||
- `price.integration.test.ts`
|
||||
- `public.routes.integration.test.ts`
|
||||
- `receipt.integration.test.ts`
|
||||
| Test File | Domain |
|
||||
| -------------------------------------- | -------------------------- |
|
||||
| `admin.integration.test.ts` | Admin dashboard operations |
|
||||
| `auth.integration.test.ts` | Authentication flows |
|
||||
| `budget.integration.test.ts` | Budget management |
|
||||
| `flyer.integration.test.ts` | Flyer CRUD operations |
|
||||
| `flyer-processing.integration.test.ts` | AI flyer processing |
|
||||
| `gamification.integration.test.ts` | Achievements and points |
|
||||
| `inventory.integration.test.ts` | Inventory management |
|
||||
| `notification.integration.test.ts` | User notifications |
|
||||
| `receipt.integration.test.ts` | Receipt processing |
|
||||
| `recipe.integration.test.ts` | Recipe management |
|
||||
| `shopping-list.integration.test.ts` | Shopping list operations |
|
||||
| `user.integration.test.ts` | User profile operations |
|
||||
|
||||
See `src/tests/integration/` for the complete list.
|
||||
|
||||
Requires PostgreSQL and Redis services running.
|
||||
|
||||
@@ -129,13 +154,23 @@ Requires PostgreSQL and Redis services running.
|
||||
npm run test:integration
|
||||
```
|
||||
|
||||
### E2E Tests (3 test files)
|
||||
### E2E Tests (11 test files)
|
||||
|
||||
Located in `src/tests/e2e/`:
|
||||
Located in `src/tests/e2e/`. Full user journey tests:
|
||||
|
||||
- `deals-journey.e2e.test.ts`
|
||||
- `budget-journey.e2e.test.ts`
|
||||
- `receipt-journey.e2e.test.ts`
|
||||
| Test File | Journey |
|
||||
| --------------------------------- | ----------------------------- |
|
||||
| `admin-authorization.e2e.test.ts` | Admin access control |
|
||||
| `admin-dashboard.e2e.test.ts` | Admin dashboard flows |
|
||||
| `auth.e2e.test.ts` | Login/logout/registration |
|
||||
| `budget-journey.e2e.test.ts` | Budget tracking workflow |
|
||||
| `deals-journey.e2e.test.ts` | Finding and saving deals |
|
||||
| `error-reporting.e2e.test.ts` | Error handling verification |
|
||||
| `flyer-upload.e2e.test.ts` | Flyer upload and processing |
|
||||
| `inventory-journey.e2e.test.ts` | Pantry management |
|
||||
| `receipt-journey.e2e.test.ts` | Receipt scanning and tracking |
|
||||
| `upc-journey.e2e.test.ts` | UPC barcode scanning |
|
||||
| `user-journey.e2e.test.ts` | User profile management |
|
||||
|
||||
Requires all services (PostgreSQL, Redis, BullMQ workers) running.
|
||||
|
||||
@@ -157,20 +192,18 @@ Located in `src/tests/utils/storeHelpers.ts`:
|
||||
|
||||
```typescript
|
||||
// Create a store with a location in one call
|
||||
const store = await createStoreWithLocation({
|
||||
storeName: 'Test Store',
|
||||
address: {
|
||||
address_line_1: '123 Main St',
|
||||
city: 'Toronto',
|
||||
province_state: 'ON',
|
||||
postal_code: 'M1M 1M1',
|
||||
},
|
||||
pool,
|
||||
log,
|
||||
const store = await createStoreWithLocation(pool, {
|
||||
name: 'Test Store',
|
||||
address: '123 Main St',
|
||||
city: 'Toronto',
|
||||
province: 'ON',
|
||||
postalCode: 'M1M 1M1',
|
||||
});
|
||||
|
||||
// Returns: { storeId, addressId, storeLocationId }
|
||||
|
||||
// Cleanup stores and their locations
|
||||
await cleanupStoreLocations([storeId1, storeId2], pool, log);
|
||||
await cleanupStoreLocation(pool, store);
|
||||
```
|
||||
|
||||
### Mock Factories
|
||||
|
||||
@@ -2,134 +2,259 @@
|
||||
|
||||
Complete guide to environment variables used in Flyer Crawler.
|
||||
|
||||
---
|
||||
|
||||
## Quick Reference
|
||||
|
||||
### Minimum Required Variables (Development)
|
||||
|
||||
| Variable | Example | Purpose |
|
||||
| ---------------- | ------------------------ | -------------------- |
|
||||
| `DB_HOST` | `localhost` | PostgreSQL host |
|
||||
| `DB_USER` | `postgres` | PostgreSQL username |
|
||||
| `DB_PASSWORD` | `postgres` | PostgreSQL password |
|
||||
| `DB_NAME` | `flyer_crawler_dev` | Database name |
|
||||
| `REDIS_URL` | `redis://localhost:6379` | Redis connection URL |
|
||||
| `JWT_SECRET` | (32+ character string) | JWT signing key |
|
||||
| `GEMINI_API_KEY` | `AIzaSy...` | Google Gemini API |
|
||||
|
||||
### Source of Truth
|
||||
|
||||
The Zod schema at `src/config/env.ts` is the authoritative source for all environment variables. If a variable is not in this file, it is not used by the application.
|
||||
|
||||
---
|
||||
|
||||
## Configuration by Environment
|
||||
|
||||
### Production
|
||||
|
||||
**Location**: Gitea CI/CD secrets injected during deployment
|
||||
**Path**: `/var/www/flyer-crawler.projectium.com/`
|
||||
**Note**: No `.env` file exists - all variables come from CI/CD
|
||||
| Aspect | Details |
|
||||
| -------- | ------------------------------------------ |
|
||||
| Location | Gitea CI/CD secrets injected at deployment |
|
||||
| Path | `/var/www/flyer-crawler.projectium.com/` |
|
||||
| File | No `.env` file - all from CI/CD secrets |
|
||||
|
||||
### Test
|
||||
|
||||
**Location**: Gitea CI/CD secrets + `.env.test` file
|
||||
**Path**: `/var/www/flyer-crawler-test.projectium.com/`
|
||||
**Note**: `.env.test` overrides for test-specific values
|
||||
| Aspect | Details |
|
||||
| -------- | --------------------------------------------- |
|
||||
| Location | Gitea CI/CD secrets + `.env.test` overrides |
|
||||
| Path | `/var/www/flyer-crawler-test.projectium.com/` |
|
||||
| File | `.env.test` for test-specific values |
|
||||
|
||||
### Development Container
|
||||
|
||||
**Location**: `.env.local` file in project root
|
||||
**Note**: Overrides default DSNs in `compose.dev.yml`
|
||||
| Aspect | Details |
|
||||
| -------- | --------------------------------------- |
|
||||
| Location | `.env.local` file in project root |
|
||||
| Priority | Overrides defaults in `compose.dev.yml` |
|
||||
| File | `.env.local` (gitignored) |
|
||||
|
||||
## Required Variables
|
||||
---
|
||||
|
||||
### Database
|
||||
## Complete Variable Reference
|
||||
|
||||
| Variable | Description | Example |
|
||||
| ------------------ | ---------------------------- | ------------------------------------------ |
|
||||
| `DB_HOST` | PostgreSQL host | `localhost` (dev), `projectium.com` (prod) |
|
||||
| `DB_PORT` | PostgreSQL port | `5432` |
|
||||
| `DB_USER_PROD` | Production database user | `flyer_crawler_prod` |
|
||||
| `DB_PASSWORD_PROD` | Production database password | (secret) |
|
||||
| `DB_DATABASE_PROD` | Production database name | `flyer-crawler-prod` |
|
||||
| `DB_USER_TEST` | Test database user | `flyer_crawler_test` |
|
||||
| `DB_PASSWORD_TEST` | Test database password | (secret) |
|
||||
| `DB_DATABASE_TEST` | Test database name | `flyer-crawler-test` |
|
||||
| `DB_USER` | Dev database user | `postgres` |
|
||||
| `DB_PASSWORD` | Dev database password | `postgres` |
|
||||
| `DB_NAME` | Dev database name | `flyer_crawler_dev` |
|
||||
### Database Configuration
|
||||
|
||||
**Note**: Production and test use separate `_PROD` and `_TEST` suffixed variables. Development uses unsuffixed variables.
|
||||
| Variable | Required | Default | Description |
|
||||
| ------------- | -------- | ------- | ----------------- |
|
||||
| `DB_HOST` | Yes | - | PostgreSQL host |
|
||||
| `DB_PORT` | No | `5432` | PostgreSQL port |
|
||||
| `DB_USER` | Yes | - | Database username |
|
||||
| `DB_PASSWORD` | Yes | - | Database password |
|
||||
| `DB_NAME` | Yes | - | Database name |
|
||||
|
||||
### Redis
|
||||
**Environment-Specific Variables** (Gitea Secrets):
|
||||
|
||||
| Variable | Description | Example |
|
||||
| --------------------- | ------------------------- | ------------------------------ |
|
||||
| `REDIS_URL` | Redis connection URL | `redis://localhost:6379` (dev) |
|
||||
| `REDIS_PASSWORD_PROD` | Production Redis password | (secret) |
|
||||
| `REDIS_PASSWORD_TEST` | Test Redis password | (secret) |
|
||||
| Variable | Environment | Description |
|
||||
| ------------------ | ----------- | ------------------------ |
|
||||
| `DB_USER_PROD` | Production | Production database user |
|
||||
| `DB_PASSWORD_PROD` | Production | Production database pass |
|
||||
| `DB_DATABASE_PROD` | Production | Production database name |
|
||||
| `DB_USER_TEST` | Test | Test database user |
|
||||
| `DB_PASSWORD_TEST` | Test | Test database password |
|
||||
| `DB_DATABASE_TEST` | Test | Test database name |
|
||||
|
||||
### Redis Configuration
|
||||
|
||||
| Variable | Required | Default | Description |
|
||||
| ---------------- | -------- | ------- | ------------------------- |
|
||||
| `REDIS_URL` | Yes | - | Redis connection URL |
|
||||
| `REDIS_PASSWORD` | No | - | Redis password (optional) |
|
||||
|
||||
**URL Format**: `redis://[user:password@]host:port`
|
||||
|
||||
**Examples**:
|
||||
|
||||
```bash
|
||||
# Development (no auth)
|
||||
REDIS_URL=redis://localhost:6379
|
||||
|
||||
# Production (with auth)
|
||||
REDIS_URL=redis://:${REDIS_PASSWORD_PROD}@localhost:6379
|
||||
```
|
||||
|
||||
### Authentication
|
||||
|
||||
| Variable | Description | Example |
|
||||
| ---------------------- | -------------------------- | -------------------------------- |
|
||||
| `JWT_SECRET` | JWT token signing key | (minimum 32 characters) |
|
||||
| `SESSION_SECRET` | Session encryption key | (minimum 32 characters) |
|
||||
| `GOOGLE_CLIENT_ID` | Google OAuth client ID | `xxx.apps.googleusercontent.com` |
|
||||
| `GOOGLE_CLIENT_SECRET` | Google OAuth client secret | (secret) |
|
||||
| `GH_CLIENT_ID` | GitHub OAuth client ID | `xxx` |
|
||||
| `GH_CLIENT_SECRET` | GitHub OAuth client secret | (secret) |
|
||||
| Variable | Required | Min Length | Description |
|
||||
| ---------------------- | -------- | ---------- | ----------------------- |
|
||||
| `JWT_SECRET` | Yes | 32 chars | JWT token signing key |
|
||||
| `JWT_SECRET_PREVIOUS` | No | - | Previous key (rotation) |
|
||||
| `GOOGLE_CLIENT_ID` | No | - | Google OAuth client ID |
|
||||
| `GOOGLE_CLIENT_SECRET` | No | - | Google OAuth secret |
|
||||
| `GITHUB_CLIENT_ID` | No | - | GitHub OAuth client ID |
|
||||
| `GITHUB_CLIENT_SECRET` | No | - | GitHub OAuth secret |
|
||||
|
||||
**Generate Secure Secret**:
|
||||
|
||||
```bash
|
||||
node -e "console.log(require('crypto').randomBytes(32).toString('hex'))"
|
||||
```
|
||||
|
||||
### AI Services
|
||||
|
||||
| Variable | Description | Example |
|
||||
| -------------------------------- | ---------------------------- | ----------- |
|
||||
| `VITE_GOOGLE_GENAI_API_KEY` | Google Gemini API key (prod) | `AIzaSy...` |
|
||||
| `VITE_GOOGLE_GENAI_API_KEY_TEST` | Google Gemini API key (test) | `AIzaSy...` |
|
||||
| `GOOGLE_MAPS_API_KEY` | Google Maps Geocoding API | `AIzaSy...` |
|
||||
| Variable | Required | Description |
|
||||
| ---------------------------- | -------- | -------------------------------- |
|
||||
| `GEMINI_API_KEY` | Yes\* | Google Gemini API key |
|
||||
| `GEMINI_RPM` | No | Rate limit (default: 5) |
|
||||
| `AI_PRICE_QUALITY_THRESHOLD` | No | Quality threshold (default: 0.5) |
|
||||
|
||||
### Application
|
||||
\*Required for flyer processing. Application works without it but cannot extract flyer data.
|
||||
|
||||
| Variable | Description | Example |
|
||||
| -------------- | ------------------------ | ----------------------------------- |
|
||||
| `NODE_ENV` | Environment mode | `development`, `test`, `production` |
|
||||
| `PORT` | Backend server port | `3001` |
|
||||
| `FRONTEND_URL` | Frontend application URL | `http://localhost:5173` (dev) |
|
||||
**Get API Key**: [Google AI Studio](https://aistudio.google.com/app/apikey)
|
||||
|
||||
### Error Tracking
|
||||
### Google Services
|
||||
|
||||
| Variable | Description | Example |
|
||||
| ---------------------- | -------------------------------- | --------------------------- |
|
||||
| `SENTRY_DSN` | Sentry DSN (production) | `https://xxx@sentry.io/xxx` |
|
||||
| `VITE_SENTRY_DSN` | Frontend Sentry DSN (production) | `https://xxx@sentry.io/xxx` |
|
||||
| `SENTRY_DSN_TEST` | Sentry DSN (test) | `https://xxx@sentry.io/xxx` |
|
||||
| `VITE_SENTRY_DSN_TEST` | Frontend Sentry DSN (test) | `https://xxx@sentry.io/xxx` |
|
||||
| `SENTRY_AUTH_TOKEN` | Sentry API token for releases | (secret) |
|
||||
| Variable | Required | Description |
|
||||
| ---------------------- | -------- | -------------------------------- |
|
||||
| `GOOGLE_MAPS_API_KEY` | No | Google Maps Geocoding API |
|
||||
| `GOOGLE_CLIENT_ID` | No | OAuth (see Authentication above) |
|
||||
| `GOOGLE_CLIENT_SECRET` | No | OAuth (see Authentication above) |
|
||||
|
||||
## Optional Variables
|
||||
### UPC Lookup APIs
|
||||
|
||||
| Variable | Description | Default |
|
||||
| ------------------- | ----------------------- | ----------------- |
|
||||
| `LOG_LEVEL` | Logging verbosity | `info` |
|
||||
| `REDIS_TTL` | Cache TTL in seconds | `3600` |
|
||||
| `MAX_UPLOAD_SIZE` | Max file upload size | `10mb` |
|
||||
| `RATE_LIMIT_WINDOW` | Rate limit window (ms) | `900000` (15 min) |
|
||||
| `RATE_LIMIT_MAX` | Max requests per window | `100` |
|
||||
| Variable | Required | Description |
|
||||
| ------------------------ | -------- | ---------------------- |
|
||||
| `UPC_ITEM_DB_API_KEY` | No | UPC Item DB API key |
|
||||
| `BARCODE_LOOKUP_API_KEY` | No | Barcode Lookup API key |
|
||||
|
||||
### Application Settings
|
||||
|
||||
| Variable | Required | Default | Description |
|
||||
| -------------- | -------- | ------------- | ------------------------ |
|
||||
| `NODE_ENV` | No | `development` | Environment mode |
|
||||
| `PORT` | No | `3001` | Backend server port |
|
||||
| `FRONTEND_URL` | No | - | Frontend URL (CORS) |
|
||||
| `BASE_URL` | No | - | API base URL |
|
||||
| `STORAGE_PATH` | No | (see below) | Flyer image storage path |
|
||||
|
||||
**NODE_ENV Values**: `development`, `test`, `staging`, `production`
|
||||
|
||||
**Default STORAGE_PATH**: `/var/www/flyer-crawler.projectium.com/flyer-images`
|
||||
|
||||
### Email/SMTP Configuration
|
||||
|
||||
| Variable | Required | Default | Description |
|
||||
| ----------------- | -------- | ------- | ----------------------- |
|
||||
| `SMTP_HOST` | No | - | SMTP server hostname |
|
||||
| `SMTP_PORT` | No | `587` | SMTP server port |
|
||||
| `SMTP_USER` | No | - | SMTP username |
|
||||
| `SMTP_PASS` | No | - | SMTP password |
|
||||
| `SMTP_SECURE` | No | `false` | Use TLS |
|
||||
| `SMTP_FROM_EMAIL` | No | - | From address for emails |
|
||||
|
||||
**Note**: Email functionality degrades gracefully if not configured.
|
||||
|
||||
### Worker Configuration
|
||||
|
||||
| Variable | Default | Description |
|
||||
| ------------------------------------- | ------- | ---------------------------- |
|
||||
| `WORKER_CONCURRENCY` | `1` | Main worker concurrency |
|
||||
| `WORKER_LOCK_DURATION` | `30000` | Lock duration (ms) |
|
||||
| `EMAIL_WORKER_CONCURRENCY` | `10` | Email worker concurrency |
|
||||
| `ANALYTICS_WORKER_CONCURRENCY` | `1` | Analytics worker concurrency |
|
||||
| `CLEANUP_WORKER_CONCURRENCY` | `10` | Cleanup worker concurrency |
|
||||
| `WEEKLY_ANALYTICS_WORKER_CONCURRENCY` | `1` | Weekly analytics concurrency |
|
||||
|
||||
### Error Tracking (Bugsink/Sentry)
|
||||
|
||||
| Variable | Required | Default | Description |
|
||||
| --------------------- | -------- | -------- | ------------------------------- |
|
||||
| `SENTRY_DSN` | No | - | Backend Sentry DSN |
|
||||
| `SENTRY_ENABLED` | No | `true` | Enable error tracking |
|
||||
| `SENTRY_ENVIRONMENT` | No | NODE_ENV | Environment name for errors |
|
||||
| `SENTRY_DEBUG` | No | `false` | Enable Sentry SDK debug logging |
|
||||
| `VITE_SENTRY_DSN` | No | - | Frontend Sentry DSN |
|
||||
| `VITE_SENTRY_ENABLED` | No | `true` | Enable frontend error tracking |
|
||||
| `VITE_SENTRY_DEBUG` | No | `false` | Frontend SDK debug logging |
|
||||
|
||||
**DSN Format**: `http://[key]@[host]:[port]/[project_id]`
|
||||
|
||||
**Dev Container DSNs**:
|
||||
|
||||
```bash
|
||||
# Backend (internal)
|
||||
SENTRY_DSN=http://<key>@localhost:8000/1
|
||||
|
||||
# Frontend (via nginx proxy)
|
||||
VITE_SENTRY_DSN=https://<key>@localhost/bugsink-api/2
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Configuration Files
|
||||
|
||||
| File | Purpose |
|
||||
| ------------------------------------- | ------------------------------------------- |
|
||||
| `src/config/env.ts` | Zod schema validation - **source of truth** |
|
||||
| `ecosystem.config.cjs` | PM2 process manager config |
|
||||
| `ecosystem.config.cjs` | PM2 process manager (production) |
|
||||
| `ecosystem.dev.config.cjs` | PM2 process manager (development) |
|
||||
| `.gitea/workflows/deploy-to-prod.yml` | Production deployment workflow |
|
||||
| `.gitea/workflows/deploy-to-test.yml` | Test deployment workflow |
|
||||
| `.env.example` | Template with all variables |
|
||||
| `.env.local` | Dev container overrides (not in git) |
|
||||
| `.env.test` | Test environment overrides (not in git) |
|
||||
|
||||
---
|
||||
|
||||
## Adding New Variables
|
||||
|
||||
### 1. Update Zod Schema
|
||||
### Checklist
|
||||
|
||||
1. [ ] **Update Zod Schema** - Edit `src/config/env.ts`
|
||||
2. [ ] **Add to Gitea Secrets** - For prod/test environments
|
||||
3. [ ] **Update Deployment Workflows** - `.gitea/workflows/*.yml`
|
||||
4. [ ] **Update PM2 Config** - `ecosystem.config.cjs`
|
||||
5. [ ] **Update .env.example** - Template for developers
|
||||
6. [ ] **Update this document** - Add to appropriate section
|
||||
|
||||
### Step-by-Step
|
||||
|
||||
#### 1. Update Zod Schema
|
||||
|
||||
Edit `src/config/env.ts`:
|
||||
|
||||
```typescript
|
||||
const envSchema = z.object({
|
||||
// ... existing variables ...
|
||||
NEW_VARIABLE: z.string().min(1),
|
||||
newSection: z.object({
|
||||
newVariable: z.string().min(1, 'NEW_VARIABLE is required'),
|
||||
}),
|
||||
});
|
||||
|
||||
// In loadEnvVars():
|
||||
newSection: {
|
||||
newVariable: process.env.NEW_VARIABLE,
|
||||
},
|
||||
```
|
||||
|
||||
### 2. Add to Gitea Secrets
|
||||
|
||||
For prod/test environments:
|
||||
#### 2. Add to Gitea Secrets
|
||||
|
||||
1. Go to Gitea repository Settings > Secrets
|
||||
2. Add `NEW_VARIABLE` with value
|
||||
2. Add `NEW_VARIABLE` with production value
|
||||
3. Add `NEW_VARIABLE_TEST` if test needs different value
|
||||
|
||||
### 3. Update Deployment Workflows
|
||||
#### 3. Update Deployment Workflows
|
||||
|
||||
Edit `.gitea/workflows/deploy-to-prod.yml`:
|
||||
|
||||
@@ -145,7 +270,7 @@ env:
|
||||
NEW_VARIABLE: ${{ secrets.NEW_VARIABLE_TEST }}
|
||||
```
|
||||
|
||||
### 4. Update PM2 Config
|
||||
#### 4. Update PM2 Config
|
||||
|
||||
Edit `ecosystem.config.cjs`:
|
||||
|
||||
@@ -161,31 +286,36 @@ module.exports = {
|
||||
};
|
||||
```
|
||||
|
||||
### 5. Update Documentation
|
||||
|
||||
- Add to `.env.example`
|
||||
- Update this document
|
||||
- Document in relevant feature docs
|
||||
---
|
||||
|
||||
## Security Best Practices
|
||||
|
||||
### Secrets Management
|
||||
### Do
|
||||
|
||||
- **NEVER** commit secrets to git
|
||||
- Use Gitea Secrets for prod/test
|
||||
- Use `.env.local` for dev (gitignored)
|
||||
- Generate secrets with cryptographic randomness
|
||||
- Rotate secrets regularly
|
||||
- Use environment-specific database users
|
||||
|
||||
### Do Not
|
||||
|
||||
- Commit secrets to git
|
||||
- Use short or predictable secrets
|
||||
- Share secrets across environments
|
||||
- Log sensitive values
|
||||
|
||||
### Secret Generation
|
||||
|
||||
```bash
|
||||
# Generate secure random secrets
|
||||
# Generate secure random secrets (64 hex characters)
|
||||
node -e "console.log(require('crypto').randomBytes(32).toString('hex'))"
|
||||
|
||||
# Example output:
|
||||
# a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4e5f6a1b2
|
||||
```
|
||||
|
||||
### Database Users
|
||||
|
||||
Each environment has its own PostgreSQL user:
|
||||
### Database Users by Environment
|
||||
|
||||
| Environment | User | Database |
|
||||
| ----------- | -------------------- | -------------------- |
|
||||
@@ -193,44 +323,61 @@ Each environment has its own PostgreSQL user:
|
||||
| Test | `flyer_crawler_test` | `flyer-crawler-test` |
|
||||
| Development | `postgres` | `flyer_crawler_dev` |
|
||||
|
||||
**Setup Commands** (as postgres superuser):
|
||||
|
||||
```sql
|
||||
-- Production
|
||||
CREATE DATABASE "flyer-crawler-prod";
|
||||
CREATE USER flyer_crawler_prod WITH PASSWORD 'secure-password';
|
||||
ALTER DATABASE "flyer-crawler-prod" OWNER TO flyer_crawler_prod;
|
||||
\c "flyer-crawler-prod"
|
||||
ALTER SCHEMA public OWNER TO flyer_crawler_prod;
|
||||
GRANT CREATE, USAGE ON SCHEMA public TO flyer_crawler_prod;
|
||||
CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
|
||||
CREATE EXTENSION IF NOT EXISTS postgis;
|
||||
CREATE EXTENSION IF NOT EXISTS pg_trgm;
|
||||
|
||||
-- Test (similar commands with _test suffix)
|
||||
```
|
||||
---
|
||||
|
||||
## Validation
|
||||
|
||||
Environment variables are validated at startup via `src/config/env.ts`. If validation fails:
|
||||
Environment variables are validated at startup via `src/config/env.ts`.
|
||||
|
||||
1. Check the error message for missing/invalid variables
|
||||
2. Verify `.env.local` (dev) or Gitea Secrets (prod/test)
|
||||
3. Ensure values match schema requirements (min length, format, etc.)
|
||||
### Startup Validation
|
||||
|
||||
If validation fails, you will see:
|
||||
|
||||
```text
|
||||
╔════════════════════════════════════════════════════════════════╗
|
||||
║ CONFIGURATION ERROR - APPLICATION STARTUP ║
|
||||
╚════════════════════════════════════════════════════════════════╝
|
||||
|
||||
The following environment variables are missing or invalid:
|
||||
|
||||
- database.host: DB_HOST is required
|
||||
- auth.jwtSecret: JWT_SECRET must be at least 32 characters
|
||||
|
||||
Please check your .env file or environment configuration.
|
||||
```
|
||||
|
||||
### Debugging Configuration
|
||||
|
||||
```bash
|
||||
# Check what variables are set (dev container)
|
||||
podman exec flyer-crawler-dev env | grep -E "^(DB_|REDIS_|JWT_|SENTRY_)"
|
||||
|
||||
# Test database connection
|
||||
podman exec flyer-crawler-postgres psql -U postgres -d flyer_crawler_dev -c "SELECT 1;"
|
||||
|
||||
# Test Redis connection
|
||||
podman exec flyer-crawler-redis redis-cli ping
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Variable Not Found
|
||||
|
||||
```
|
||||
```text
|
||||
Error: Missing required environment variable: JWT_SECRET
|
||||
```
|
||||
|
||||
**Solution**: Add the variable to your environment configuration.
|
||||
**Solutions**:
|
||||
|
||||
1. Check `.env.local` exists and has the variable
|
||||
2. Verify variable name matches schema exactly
|
||||
3. Restart the application after changes
|
||||
|
||||
### Invalid Value
|
||||
|
||||
```
|
||||
```text
|
||||
Error: JWT_SECRET must be at least 32 characters
|
||||
```
|
||||
|
||||
@@ -240,32 +387,36 @@ Error: JWT_SECRET must be at least 32 characters
|
||||
|
||||
Check `NODE_ENV` is set correctly:
|
||||
|
||||
- `development` - Local dev container
|
||||
- `test` - CI/CD test server
|
||||
- `production` - Production server
|
||||
| Value | Purpose |
|
||||
| ------------- | ---------------------- |
|
||||
| `development` | Local dev container |
|
||||
| `test` | CI/CD test server |
|
||||
| `staging` | Pre-production testing |
|
||||
| `production` | Production server |
|
||||
|
||||
### Database Connection Issues
|
||||
|
||||
Verify database credentials:
|
||||
|
||||
```bash
|
||||
# Development
|
||||
podman exec flyer-crawler-postgres psql -U postgres -d flyer_crawler_dev -c "SELECT 1;"
|
||||
|
||||
# Production (via SSH)
|
||||
ssh root@projectium.com "psql -U flyer_crawler_prod -d flyer-crawler-prod -c 'SELECT 1;'"
|
||||
# If connection fails, check:
|
||||
# 1. Container is running: podman ps
|
||||
# 2. DB_HOST matches container network
|
||||
# 3. DB_PASSWORD is correct
|
||||
```
|
||||
|
||||
## Reference
|
||||
---
|
||||
|
||||
- **Validation Schema**: [src/config/env.ts](../../src/config/env.ts)
|
||||
- **Template**: [.env.example](../../.env.example)
|
||||
- **Deployment Workflows**: [.gitea/workflows/](../../.gitea/workflows/)
|
||||
- **PM2 Config**: [ecosystem.config.cjs](../../ecosystem.config.cjs)
|
||||
|
||||
## See Also
|
||||
## Related Documentation
|
||||
|
||||
- [QUICKSTART.md](QUICKSTART.md) - Quick setup guide
|
||||
- [INSTALL.md](INSTALL.md) - Detailed installation
|
||||
- [DEV-CONTAINER.md](../development/DEV-CONTAINER.md) - Dev container setup
|
||||
- [DEPLOYMENT.md](../operations/DEPLOYMENT.md) - Production deployment
|
||||
- [AUTHENTICATION.md](../architecture/AUTHENTICATION.md) - OAuth setup
|
||||
- [ADR-007](../adr/0007-configuration-and-secrets-management.md) - Configuration decisions
|
||||
|
||||
---
|
||||
|
||||
Last updated: January 2026
|
||||
|
||||
@@ -1,203 +1,453 @@
|
||||
# Installation Guide
|
||||
|
||||
This guide covers setting up a local development environment for Flyer Crawler.
|
||||
Complete setup instructions for the Flyer Crawler local development environment.
|
||||
|
||||
---
|
||||
|
||||
## Quick Reference
|
||||
|
||||
| Setup Method | Best For | Time | Document Section |
|
||||
| ----------------- | --------------------------- | ------ | --------------------------------------------------- |
|
||||
| Quick Start | Already have Postgres/Redis | 5 min | [Quick Start](#quick-start) |
|
||||
| Dev Container | Full production-like setup | 15 min | [Dev Container](#development-container-recommended) |
|
||||
| Manual Containers | Learning the components | 20 min | [Podman Setup](#podman-setup-manual) |
|
||||
|
||||
---
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Node.js 20.x or later
|
||||
- Access to a PostgreSQL database (local or remote)
|
||||
- Redis instance (for session management)
|
||||
- Google Gemini API key
|
||||
- Google Maps API key (for geocoding)
|
||||
### Required Software
|
||||
|
||||
| Software | Minimum Version | Purpose | Download |
|
||||
| -------------- | --------------- | -------------------- | ----------------------------------------------- |
|
||||
| Node.js | 20.x | Runtime | [nodejs.org](https://nodejs.org/) |
|
||||
| Podman Desktop | 4.x | Container management | [podman-desktop.io](https://podman-desktop.io/) |
|
||||
| Git | 2.x | Version control | [git-scm.com](https://git-scm.com/) |
|
||||
|
||||
### Windows-Specific Requirements
|
||||
|
||||
| Requirement | Purpose | Setup Command |
|
||||
| ----------- | ------------------------------ | ---------------------------------- |
|
||||
| WSL 2 | Linux compatibility for Podman | `wsl --install` (admin PowerShell) |
|
||||
|
||||
### Verify Installation
|
||||
|
||||
```bash
|
||||
# Check all prerequisites
|
||||
node --version # Expected: v20.x or higher
|
||||
podman --version # Expected: podman version 4.x or higher
|
||||
git --version # Expected: git version 2.x or higher
|
||||
wsl --list -v # Expected: Shows WSL 2 distro
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Quick Start
|
||||
|
||||
If you already have PostgreSQL and Redis configured:
|
||||
If you already have PostgreSQL and Redis configured externally:
|
||||
|
||||
```bash
|
||||
# Install dependencies
|
||||
# 1. Clone the repository
|
||||
git clone https://gitea.projectium.com/flyer-crawler/flyer-crawler.git
|
||||
cd flyer-crawler
|
||||
|
||||
# 2. Install dependencies
|
||||
npm install
|
||||
|
||||
# Run in development mode
|
||||
# 3. Create .env.local (see Environment section below)
|
||||
|
||||
# 4. Run in development mode
|
||||
npm run dev
|
||||
```
|
||||
|
||||
**Access Points**:
|
||||
|
||||
- Frontend: `http://localhost:5173`
|
||||
- Backend API: `http://localhost:3001`
|
||||
|
||||
---
|
||||
|
||||
## Development Environment with Podman (Recommended for Windows)
|
||||
## Development Container (Recommended)
|
||||
|
||||
This approach uses Podman with an Ubuntu container for a consistent development environment.
|
||||
The dev container provides a complete, production-like environment.
|
||||
|
||||
### What's Included
|
||||
|
||||
| Service | Purpose | Port |
|
||||
| ---------- | ------------------------ | ---------- |
|
||||
| Node.js | API server, worker, Vite | 3001, 5173 |
|
||||
| PostgreSQL | Database with PostGIS | 5432 |
|
||||
| Redis | Cache and job queues | 6379 |
|
||||
| NGINX | HTTPS reverse proxy | 443 |
|
||||
| Bugsink | Error tracking | 8443 |
|
||||
| Logstash | Log aggregation | - |
|
||||
| PM2 | Process management | - |
|
||||
|
||||
### Setup Steps
|
||||
|
||||
#### Step 1: Initialize Podman
|
||||
|
||||
```bash
|
||||
# Windows: Start Podman Desktop, or from terminal:
|
||||
podman machine init
|
||||
podman machine start
|
||||
```
|
||||
|
||||
#### Step 2: Start Dev Container
|
||||
|
||||
```bash
|
||||
# Start all services
|
||||
podman-compose -f compose.dev.yml up -d
|
||||
|
||||
# View logs (optional)
|
||||
podman-compose -f compose.dev.yml logs -f
|
||||
```
|
||||
|
||||
**Expected Output**:
|
||||
|
||||
```text
|
||||
[+] Running 3/3
|
||||
- Container flyer-crawler-postgres Started
|
||||
- Container flyer-crawler-redis Started
|
||||
- Container flyer-crawler-dev Started
|
||||
```
|
||||
|
||||
#### Step 3: Verify Services
|
||||
|
||||
```bash
|
||||
# Check containers are running
|
||||
podman ps
|
||||
|
||||
# Check PM2 processes
|
||||
podman exec -it flyer-crawler-dev pm2 status
|
||||
```
|
||||
|
||||
**Expected PM2 Status**:
|
||||
|
||||
```text
|
||||
+---------------------------+--------+-------+
|
||||
| name | status | cpu |
|
||||
+---------------------------+--------+-------+
|
||||
| flyer-crawler-api-dev | online | 0% |
|
||||
| flyer-crawler-worker-dev | online | 0% |
|
||||
| flyer-crawler-vite-dev | online | 0% |
|
||||
+---------------------------+--------+-------+
|
||||
```
|
||||
|
||||
#### Step 4: Access Application
|
||||
|
||||
| Service | URL | Notes |
|
||||
| ----------- | ------------------------ | ---------------------------- |
|
||||
| Frontend | `https://localhost` | NGINX proxies to Vite |
|
||||
| Backend API | `http://localhost:3001` | Express server |
|
||||
| Bugsink | `https://localhost:8443` | Login: admin@localhost/admin |
|
||||
|
||||
### SSL Certificate Setup (Optional but Recommended)
|
||||
|
||||
To eliminate browser security warnings:
|
||||
|
||||
**Windows**:
|
||||
|
||||
1. Double-click `certs/mkcert-ca.crt`
|
||||
2. Click "Install Certificate..."
|
||||
3. Select "Local Machine" > Next
|
||||
4. Select "Place all certificates in the following store"
|
||||
5. Browse > Select "Trusted Root Certification Authorities" > OK
|
||||
6. Click Next > Finish
|
||||
7. Restart browser
|
||||
|
||||
**Other Platforms**: See [`certs/README.md`](../../certs/README.md)
|
||||
|
||||
### Managing the Dev Container
|
||||
|
||||
| Action | Command |
|
||||
| --------- | ------------------------------------------- |
|
||||
| Start | `podman-compose -f compose.dev.yml up -d` |
|
||||
| Stop | `podman-compose -f compose.dev.yml down` |
|
||||
| View logs | `podman-compose -f compose.dev.yml logs -f` |
|
||||
| Restart | `podman-compose -f compose.dev.yml restart` |
|
||||
| Rebuild | `podman-compose -f compose.dev.yml build` |
|
||||
|
||||
---
|
||||
|
||||
## Podman Setup (Manual)
|
||||
|
||||
For understanding the individual components or custom configurations.
|
||||
|
||||
### Step 1: Install Prerequisites on Windows
|
||||
|
||||
1. **Install WSL 2**: Podman on Windows relies on the Windows Subsystem for Linux.
|
||||
```powershell
|
||||
# Run in administrator PowerShell
|
||||
wsl --install
|
||||
```
|
||||
|
||||
```powershell
|
||||
wsl --install
|
||||
```
|
||||
Restart computer after WSL installation.
|
||||
|
||||
Run this in an administrator PowerShell.
|
||||
### Step 2: Initialize Podman
|
||||
|
||||
2. **Install Podman Desktop**: Download and install [Podman Desktop for Windows](https://podman-desktop.io/).
|
||||
1. Launch **Podman Desktop**
|
||||
2. Follow the setup wizard to initialize Podman machine
|
||||
3. Start the Podman machine
|
||||
|
||||
### Step 2: Set Up Podman
|
||||
|
||||
1. **Initialize Podman**: Launch Podman Desktop. It will automatically set up its WSL 2 machine.
|
||||
2. **Start Podman**: Ensure the Podman machine is running from the Podman Desktop interface.
|
||||
|
||||
### Step 3: Set Up the Ubuntu Container
|
||||
|
||||
1. **Pull Ubuntu Image**:
|
||||
|
||||
```bash
|
||||
podman pull ubuntu:latest
|
||||
```
|
||||
|
||||
2. **Create a Podman Volume** (persists node_modules between container restarts):
|
||||
|
||||
```bash
|
||||
podman volume create node_modules_cache
|
||||
```
|
||||
|
||||
3. **Run the Ubuntu Container**:
|
||||
|
||||
Open a terminal in your project's root directory and run:
|
||||
|
||||
```bash
|
||||
podman run -it -p 3001:3001 -p 5173:5173 --name flyer-dev \
|
||||
-v "$(pwd):/app" \
|
||||
-v "node_modules_cache:/app/node_modules" \
|
||||
ubuntu:latest
|
||||
```
|
||||
|
||||
| Flag | Purpose |
|
||||
| ------------------------------------------- | ------------------------------------------------ |
|
||||
| `-p 3001:3001` | Forwards the backend server port |
|
||||
| `-p 5173:5173` | Forwards the Vite frontend server port |
|
||||
| `--name flyer-dev` | Names the container for easy reference |
|
||||
| `-v "...:/app"` | Mounts your project directory into the container |
|
||||
| `-v "node_modules_cache:/app/node_modules"` | Mounts the named volume for node_modules |
|
||||
|
||||
### Step 4: Configure the Ubuntu Environment
|
||||
|
||||
You are now inside the Ubuntu container's shell.
|
||||
|
||||
1. **Update Package Lists**:
|
||||
|
||||
```bash
|
||||
apt-get update
|
||||
```
|
||||
|
||||
2. **Install Dependencies**:
|
||||
|
||||
```bash
|
||||
apt-get install -y curl git
|
||||
curl -sL https://deb.nodesource.com/setup_20.x | bash -
|
||||
apt-get install -y nodejs
|
||||
```
|
||||
|
||||
3. **Navigate to Project Directory**:
|
||||
|
||||
```bash
|
||||
cd /app
|
||||
```
|
||||
|
||||
4. **Install Project Dependencies**:
|
||||
|
||||
```bash
|
||||
npm install
|
||||
```
|
||||
|
||||
### Step 5: Run the Development Server
|
||||
Or from terminal:
|
||||
|
||||
```bash
|
||||
podman machine init
|
||||
podman machine start
|
||||
```
|
||||
|
||||
### Step 3: Create Podman Network
|
||||
|
||||
```bash
|
||||
podman network create flyer-crawler-net
|
||||
```
|
||||
|
||||
### Step 4: Create PostgreSQL Container
|
||||
|
||||
```bash
|
||||
podman run -d \
|
||||
--name flyer-crawler-postgres \
|
||||
--network flyer-crawler-net \
|
||||
-e POSTGRES_USER=postgres \
|
||||
-e POSTGRES_PASSWORD=postgres \
|
||||
-e POSTGRES_DB=flyer_crawler_dev \
|
||||
-p 5432:5432 \
|
||||
-v flyer-crawler-pgdata:/var/lib/postgresql/data \
|
||||
docker.io/postgis/postgis:15-3.3
|
||||
```
|
||||
|
||||
### Step 5: Create Redis Container
|
||||
|
||||
```bash
|
||||
podman run -d \
|
||||
--name flyer-crawler-redis \
|
||||
--network flyer-crawler-net \
|
||||
-p 6379:6379 \
|
||||
-v flyer-crawler-redis:/data \
|
||||
docker.io/library/redis:alpine
|
||||
```
|
||||
|
||||
### Step 6: Initialize Database
|
||||
|
||||
```bash
|
||||
# Wait for PostgreSQL to be ready
|
||||
podman exec flyer-crawler-postgres pg_isready -U postgres
|
||||
|
||||
# Install required extensions
|
||||
podman exec flyer-crawler-postgres psql -U postgres -d flyer_crawler_dev -c "
|
||||
CREATE EXTENSION IF NOT EXISTS postgis;
|
||||
CREATE EXTENSION IF NOT EXISTS pg_trgm;
|
||||
CREATE EXTENSION IF NOT EXISTS \"uuid-ossp\";
|
||||
"
|
||||
|
||||
# Apply schema
|
||||
podman exec -i flyer-crawler-postgres psql -U postgres -d flyer_crawler_dev < sql/master_schema_rollup.sql
|
||||
```
|
||||
|
||||
### Step 7: Create Node.js Container
|
||||
|
||||
```bash
|
||||
# Create volume for node_modules
|
||||
podman volume create node_modules_cache
|
||||
|
||||
# Run Ubuntu container with project mounted
|
||||
podman run -it \
|
||||
--name flyer-dev \
|
||||
--network flyer-crawler-net \
|
||||
-p 3001:3001 \
|
||||
-p 5173:5173 \
|
||||
-v "$(pwd):/app" \
|
||||
-v "node_modules_cache:/app/node_modules" \
|
||||
ubuntu:latest
|
||||
```
|
||||
|
||||
### Step 8: Configure Container Environment
|
||||
|
||||
Inside the container:
|
||||
|
||||
```bash
|
||||
# Update and install dependencies
|
||||
apt-get update
|
||||
apt-get install -y curl git
|
||||
|
||||
# Install Node.js 20
|
||||
curl -sL https://deb.nodesource.com/setup_20.x | bash -
|
||||
apt-get install -y nodejs
|
||||
|
||||
# Navigate to project and install
|
||||
cd /app
|
||||
npm install
|
||||
|
||||
# Start development server
|
||||
npm run dev
|
||||
```
|
||||
|
||||
### Step 6: Access the Application
|
||||
### Container Management Commands
|
||||
|
||||
- **Frontend**: http://localhost:5173
|
||||
- **Backend API**: http://localhost:3001
|
||||
|
||||
### Dev Container with HTTPS (Full Stack)
|
||||
|
||||
When using the full dev container stack with NGINX (via `compose.dev.yml`), access the application over HTTPS:
|
||||
|
||||
- **Frontend**: https://localhost or https://127.0.0.1
|
||||
- **Backend API**: http://localhost:3001
|
||||
|
||||
**SSL Certificate Notes:**
|
||||
|
||||
- The dev container uses self-signed certificates generated by mkcert
|
||||
- Both `localhost` and `127.0.0.1` are valid hostnames (certificate includes both as SANs)
|
||||
- If images fail to load with SSL errors, see [FLYER-URL-CONFIGURATION.md](../FLYER-URL-CONFIGURATION.md#ssl-certificate-configuration-dev-container)
|
||||
|
||||
**Eliminate SSL Warnings (Recommended):**
|
||||
|
||||
To avoid browser security warnings for self-signed certificates, install the mkcert CA certificate on your system. The CA certificate is located at `certs/mkcert-ca.crt` in the project root.
|
||||
|
||||
See [`certs/README.md`](../../certs/README.md) for platform-specific installation instructions (Windows, macOS, Linux, Firefox).
|
||||
|
||||
After installation:
|
||||
|
||||
- Your browser will trust all mkcert certificates without warnings
|
||||
- Both `https://localhost/` and `https://127.0.0.1/` will work without SSL errors
|
||||
- Flyer images will load without `ERR_CERT_AUTHORITY_INVALID` errors
|
||||
|
||||
### Managing the Container
|
||||
|
||||
| Action | Command |
|
||||
| --------------------- | -------------------------------- |
|
||||
| Stop the container | Press `Ctrl+C`, then type `exit` |
|
||||
| Restart the container | `podman start -a -i flyer-dev` |
|
||||
| Remove the container | `podman rm flyer-dev` |
|
||||
| Action | Command |
|
||||
| -------------- | ------------------------------ |
|
||||
| Stop container | Press `Ctrl+C`, then `exit` |
|
||||
| Restart | `podman start -a -i flyer-dev` |
|
||||
| Remove | `podman rm flyer-dev` |
|
||||
| List running | `podman ps` |
|
||||
| List all | `podman ps -a` |
|
||||
|
||||
---
|
||||
|
||||
## Environment Variables
|
||||
## Environment Configuration
|
||||
|
||||
This project is configured to run in a CI/CD environment and does not use `.env` files. All configuration must be provided as environment variables.
|
||||
### Create .env.local
|
||||
|
||||
For local development, you can export these in your shell or use your IDE's environment configuration:
|
||||
Create `.env.local` in the project root with your configuration:
|
||||
|
||||
| Variable | Description |
|
||||
| --------------------------- | ------------------------------------- |
|
||||
| `DB_HOST` | PostgreSQL server hostname |
|
||||
| `DB_USER` | PostgreSQL username |
|
||||
| `DB_PASSWORD` | PostgreSQL password |
|
||||
| `DB_DATABASE_PROD` | Production database name |
|
||||
| `JWT_SECRET` | Secret string for signing auth tokens |
|
||||
| `VITE_GOOGLE_GENAI_API_KEY` | Google Gemini API key |
|
||||
| `GOOGLE_MAPS_API_KEY` | Google Maps Geocoding API key |
|
||||
| `REDIS_PASSWORD_PROD` | Production Redis password |
|
||||
| `REDIS_PASSWORD_TEST` | Test Redis password |
|
||||
```bash
|
||||
# Database (adjust host based on your setup)
|
||||
DB_HOST=localhost # Use 'postgres' if inside dev container
|
||||
DB_PORT=5432
|
||||
DB_USER=postgres
|
||||
DB_PASSWORD=postgres
|
||||
DB_NAME=flyer_crawler_dev
|
||||
|
||||
# Redis (adjust host based on your setup)
|
||||
REDIS_URL=redis://localhost:6379 # Use 'redis://redis:6379' inside container
|
||||
|
||||
# Application
|
||||
NODE_ENV=development
|
||||
PORT=3001
|
||||
FRONTEND_URL=http://localhost:5173
|
||||
|
||||
# Authentication (generate secure values)
|
||||
JWT_SECRET=your-secret-at-least-32-characters-long
|
||||
|
||||
# AI Services
|
||||
GEMINI_API_KEY=your-google-gemini-api-key
|
||||
GOOGLE_MAPS_API_KEY=your-google-maps-api-key # Optional
|
||||
```
|
||||
|
||||
**Generate Secure Secrets**:
|
||||
|
||||
```bash
|
||||
node -e "console.log(require('crypto').randomBytes(32).toString('hex'))"
|
||||
```
|
||||
|
||||
### Environment Differences
|
||||
|
||||
| Variable | Host Development | Inside Dev Container |
|
||||
| ----------- | ------------------------ | -------------------- |
|
||||
| `DB_HOST` | `localhost` | `postgres` |
|
||||
| `REDIS_URL` | `redis://localhost:6379` | `redis://redis:6379` |
|
||||
|
||||
See [ENVIRONMENT.md](ENVIRONMENT.md) for complete variable reference.
|
||||
|
||||
---
|
||||
|
||||
## Seeding Development Data
|
||||
|
||||
To create initial test accounts (`admin@example.com` and `user@example.com`) and sample data:
|
||||
Create test accounts and sample data:
|
||||
|
||||
```bash
|
||||
npm run seed
|
||||
```
|
||||
|
||||
The seed script performs the following actions:
|
||||
### What the Seed Script Does
|
||||
|
||||
1. Rebuilds the database schema from `sql/master_schema_rollup.sql`
|
||||
2. Creates test user accounts (admin and regular user)
|
||||
3. Copies test flyer images from `src/tests/assets/` to `public/flyer-images/`
|
||||
4. Creates a sample flyer with items linked to the test images
|
||||
5. Seeds watched items and a shopping list for the test user
|
||||
1. Rebuilds database schema from `sql/master_schema_rollup.sql`
|
||||
2. Creates test user accounts:
|
||||
- `admin@example.com` (admin user)
|
||||
- `user@example.com` (regular user)
|
||||
3. Copies test flyer images to `public/flyer-images/`
|
||||
4. Creates sample flyer with items
|
||||
5. Seeds watched items and shopping list
|
||||
|
||||
**Test Images**: The seed script copies `test-flyer-image.jpg` and `test-flyer-icon.png` to the `public/flyer-images/` directory, which is served by NGINX at `/flyer-images/`.
|
||||
### Test Images
|
||||
|
||||
After running, you may need to restart your IDE's TypeScript server to pick up any generated types.
|
||||
The seed script copies these files from `src/tests/assets/`:
|
||||
|
||||
- `test-flyer-image.jpg`
|
||||
- `test-flyer-icon.png`
|
||||
|
||||
Images are served by NGINX at `/flyer-images/`.
|
||||
|
||||
---
|
||||
|
||||
## Verification Checklist
|
||||
|
||||
After installation, verify everything works:
|
||||
|
||||
- [ ] **Containers running**: `podman ps` shows postgres and redis
|
||||
- [ ] **Database accessible**: `podman exec flyer-crawler-postgres psql -U postgres -c "SELECT 1;"`
|
||||
- [ ] **Frontend loads**: Open `http://localhost:5173` (or `https://localhost` for dev container)
|
||||
- [ ] **API responds**: `curl http://localhost:3001/health`
|
||||
- [ ] **Tests pass**: `npm run test:unit` (or in container: `podman exec -it flyer-crawler-dev npm run test:unit`)
|
||||
- [ ] **Type check passes**: `npm run type-check`
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Podman Machine Won't Start
|
||||
|
||||
```bash
|
||||
# Reset Podman machine
|
||||
podman machine rm
|
||||
podman machine init
|
||||
podman machine start
|
||||
```
|
||||
|
||||
### Port Already in Use
|
||||
|
||||
```bash
|
||||
# Find process using port
|
||||
netstat -ano | findstr :5432
|
||||
|
||||
# Option: Use different port
|
||||
podman run -d --name flyer-crawler-postgres -p 5433:5432 ...
|
||||
# Then set DB_PORT=5433 in .env.local
|
||||
```
|
||||
|
||||
### Database Extensions Missing
|
||||
|
||||
```bash
|
||||
podman exec flyer-crawler-postgres psql -U postgres -d flyer_crawler_dev -c "
|
||||
CREATE EXTENSION IF NOT EXISTS postgis;
|
||||
CREATE EXTENSION IF NOT EXISTS pg_trgm;
|
||||
CREATE EXTENSION IF NOT EXISTS \"uuid-ossp\";
|
||||
"
|
||||
```
|
||||
|
||||
### Permission Denied on Windows Paths
|
||||
|
||||
Use `MSYS_NO_PATHCONV=1` prefix:
|
||||
|
||||
```bash
|
||||
MSYS_NO_PATHCONV=1 podman exec flyer-crawler-dev /path/to/script.sh
|
||||
```
|
||||
|
||||
### Tests Fail with Timezone Errors
|
||||
|
||||
Tests must run in the dev container, not on Windows host:
|
||||
|
||||
```bash
|
||||
# CORRECT
|
||||
podman exec -it flyer-crawler-dev npm test
|
||||
|
||||
# INCORRECT (may fail with TZ errors)
|
||||
npm test
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Next Steps
|
||||
|
||||
- [Database Setup](DATABASE.md) - Set up PostgreSQL with required extensions
|
||||
- [Authentication Setup](AUTHENTICATION.md) - Configure OAuth providers
|
||||
- [Deployment Guide](DEPLOYMENT.md) - Deploy to production
|
||||
| Goal | Document |
|
||||
| --------------------- | ------------------------------------------------------ |
|
||||
| Quick setup guide | [QUICKSTART.md](QUICKSTART.md) |
|
||||
| Environment variables | [ENVIRONMENT.md](ENVIRONMENT.md) |
|
||||
| Database schema | [DATABASE.md](../architecture/DATABASE.md) |
|
||||
| Authentication setup | [AUTHENTICATION.md](../architecture/AUTHENTICATION.md) |
|
||||
| Dev container details | [DEV-CONTAINER.md](../development/DEV-CONTAINER.md) |
|
||||
| Deployment | [DEPLOYMENT.md](../operations/DEPLOYMENT.md) |
|
||||
|
||||
---
|
||||
|
||||
Last updated: January 2026
|
||||
|
||||
@@ -2,13 +2,38 @@
|
||||
|
||||
Get Flyer Crawler running in 5 minutes.
|
||||
|
||||
## Prerequisites
|
||||
---
|
||||
|
||||
- **Windows 10/11** with WSL 2
|
||||
- **Podman Desktop** installed
|
||||
- **Node.js 20+** installed
|
||||
## Prerequisites Checklist
|
||||
|
||||
## 1. Start Containers (1 minute)
|
||||
Before starting, verify you have:
|
||||
|
||||
- [ ] **Windows 10/11** with WSL 2 enabled
|
||||
- [ ] **Podman Desktop** installed ([download](https://podman-desktop.io/))
|
||||
- [ ] **Node.js 20+** installed
|
||||
- [ ] **Git** for cloning the repository
|
||||
|
||||
**Verify Prerequisites**:
|
||||
|
||||
```bash
|
||||
# Check Podman
|
||||
podman --version
|
||||
# Expected: podman version 4.x or higher
|
||||
|
||||
# Check Node.js
|
||||
node --version
|
||||
# Expected: v20.x or higher
|
||||
|
||||
# Check WSL
|
||||
wsl --list --verbose
|
||||
# Expected: Shows WSL 2 distro
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Quick Setup (5 Steps)
|
||||
|
||||
### Step 1: Start Containers (1 minute)
|
||||
|
||||
```bash
|
||||
# Start PostgreSQL and Redis
|
||||
@@ -27,11 +52,18 @@ podman run -d --name flyer-crawler-redis \
|
||||
docker.io/library/redis:alpine
|
||||
```
|
||||
|
||||
## 2. Initialize Database (2 minutes)
|
||||
**Expected Output**:
|
||||
|
||||
```text
|
||||
# Container IDs displayed, no errors
|
||||
```
|
||||
|
||||
### Step 2: Initialize Database (2 minutes)
|
||||
|
||||
```bash
|
||||
# Wait for PostgreSQL to be ready
|
||||
podman exec flyer-crawler-postgres pg_isready -U postgres
|
||||
# Expected: localhost:5432 - accepting connections
|
||||
|
||||
# Install extensions
|
||||
podman exec flyer-crawler-postgres psql -U postgres -d flyer_crawler_dev \
|
||||
@@ -41,7 +73,17 @@ podman exec flyer-crawler-postgres psql -U postgres -d flyer_crawler_dev \
|
||||
podman exec -i flyer-crawler-postgres psql -U postgres -d flyer_crawler_dev < sql/master_schema_rollup.sql
|
||||
```
|
||||
|
||||
## 3. Configure Environment (1 minute)
|
||||
**Expected Output**:
|
||||
|
||||
```text
|
||||
CREATE EXTENSION
|
||||
CREATE EXTENSION
|
||||
CREATE EXTENSION
|
||||
CREATE TABLE
|
||||
... (many tables created)
|
||||
```
|
||||
|
||||
### Step 3: Configure Environment (1 minute)
|
||||
|
||||
Create `.env.local` in the project root:
|
||||
|
||||
@@ -61,16 +103,22 @@ NODE_ENV=development
|
||||
PORT=3001
|
||||
FRONTEND_URL=http://localhost:5173
|
||||
|
||||
# Secrets (generate your own)
|
||||
# Secrets (generate your own - see command below)
|
||||
JWT_SECRET=your-dev-jwt-secret-at-least-32-chars-long
|
||||
SESSION_SECRET=your-dev-session-secret-at-least-32-chars-long
|
||||
|
||||
# AI Services (get your own keys)
|
||||
VITE_GOOGLE_GENAI_API_KEY=your-google-genai-api-key
|
||||
GEMINI_API_KEY=your-google-gemini-api-key
|
||||
GOOGLE_MAPS_API_KEY=your-google-maps-api-key
|
||||
```
|
||||
|
||||
## 4. Install & Run (1 minute)
|
||||
**Generate Secure Secrets**:
|
||||
|
||||
```bash
|
||||
node -e "console.log(require('crypto').randomBytes(32).toString('hex'))"
|
||||
```
|
||||
|
||||
### Step 4: Install and Run (1 minute)
|
||||
|
||||
```bash
|
||||
# Install dependencies (first time only)
|
||||
@@ -80,35 +128,61 @@ npm install
|
||||
npm run dev
|
||||
```
|
||||
|
||||
## 5. Access Application
|
||||
**Expected Output**:
|
||||
|
||||
- **Frontend**: http://localhost:5173
|
||||
- **Backend API**: http://localhost:3001
|
||||
- **Health Check**: http://localhost:3001/health
|
||||
```text
|
||||
> flyer-crawler@x.x.x dev
|
||||
> concurrently ...
|
||||
|
||||
### Dev Container (HTTPS)
|
||||
[API] Server listening on port 3001
|
||||
[Vite] VITE ready at http://localhost:5173
|
||||
```
|
||||
|
||||
When using the full dev container with NGINX, access via HTTPS:
|
||||
### Step 5: Verify Installation
|
||||
|
||||
- **Frontend**: https://localhost or https://127.0.0.1
|
||||
- **Backend API**: http://localhost:3001
|
||||
- **Bugsink**: `https://localhost:8443` (error tracking)
|
||||
| Check | URL/Command | Expected Result |
|
||||
| ----------- | ------------------------------ | ----------------------------------- |
|
||||
| Frontend | `http://localhost:5173` | Flyer Crawler app loads |
|
||||
| Backend API | `http://localhost:3001/health` | `{ "status": "ok", ... }` |
|
||||
| Database | `podman exec ... psql -c ...` | `SELECT version()` returns Postgres |
|
||||
| Containers | `podman ps` | Shows postgres and redis running |
|
||||
|
||||
**Note:** The dev container accepts both `localhost` and `127.0.0.1` for HTTPS connections. The self-signed certificate is valid for both hostnames.
|
||||
---
|
||||
|
||||
**SSL Certificate Warnings:** To eliminate browser security warnings for self-signed certificates, install the mkcert CA certificate. See [`certs/README.md`](../../certs/README.md) for platform-specific installation instructions. This is optional but recommended for a better development experience.
|
||||
## Full Dev Container (Recommended)
|
||||
|
||||
### Dev Container Architecture
|
||||
For a production-like environment with NGINX, Bugsink error tracking, and PM2 process management:
|
||||
|
||||
The dev container uses PM2 for process management, matching production (ADR-014):
|
||||
### Starting the Dev Container
|
||||
|
||||
| Process | Description | Port |
|
||||
| -------------------------- | ------------------------ | ---- |
|
||||
| `flyer-crawler-api-dev` | API server (tsx watch) | 3001 |
|
||||
| `flyer-crawler-worker-dev` | Background job worker | - |
|
||||
| `flyer-crawler-vite-dev` | Vite frontend dev server | 5173 |
|
||||
```bash
|
||||
# Start all services
|
||||
podman-compose -f compose.dev.yml up -d
|
||||
|
||||
**PM2 Commands** (run inside container):
|
||||
# View logs
|
||||
podman-compose -f compose.dev.yml logs -f
|
||||
```
|
||||
|
||||
### Access Points
|
||||
|
||||
| Service | URL | Notes |
|
||||
| ----------- | ------------------------ | ---------------------------- |
|
||||
| Frontend | `https://localhost` | NGINX proxy to Vite |
|
||||
| Backend API | `http://localhost:3001` | Express server |
|
||||
| Bugsink | `https://localhost:8443` | Error tracking (admin/admin) |
|
||||
| PostgreSQL | `localhost:5432` | Database |
|
||||
| Redis | `localhost:6379` | Cache |
|
||||
|
||||
**SSL Certificate Setup (Recommended)**:
|
||||
|
||||
To eliminate browser security warnings, install the mkcert CA certificate:
|
||||
|
||||
```bash
|
||||
# Windows: Double-click certs/mkcert-ca.crt and install to Trusted Root CAs
|
||||
# See certs/README.md for detailed instructions per platform
|
||||
```
|
||||
|
||||
### PM2 Commands
|
||||
|
||||
```bash
|
||||
# View process status
|
||||
@@ -124,63 +198,152 @@ podman exec -it flyer-crawler-dev pm2 restart all
|
||||
podman exec -it flyer-crawler-dev pm2 restart flyer-crawler-api-dev
|
||||
```
|
||||
|
||||
## Verify Installation
|
||||
### Dev Container Processes
|
||||
|
||||
| Process | Description | Port |
|
||||
| -------------------------- | ------------------------ | ---- |
|
||||
| `flyer-crawler-api-dev` | API server (tsx watch) | 3001 |
|
||||
| `flyer-crawler-worker-dev` | Background job worker | - |
|
||||
| `flyer-crawler-vite-dev` | Vite frontend dev server | 5173 |
|
||||
|
||||
---
|
||||
|
||||
## Verification Commands
|
||||
|
||||
Run these to confirm everything is working:
|
||||
|
||||
```bash
|
||||
# Check containers are running
|
||||
podman ps
|
||||
# Expected: flyer-crawler-postgres and flyer-crawler-redis both running
|
||||
|
||||
# Test database connection
|
||||
podman exec flyer-crawler-postgres psql -U postgres -d flyer_crawler_dev -c "SELECT version();"
|
||||
# Expected: PostgreSQL 15.x with PostGIS
|
||||
|
||||
# Run tests (in dev container)
|
||||
podman exec -it flyer-crawler-dev npm run test:unit
|
||||
# Expected: All tests pass
|
||||
|
||||
# Run type check
|
||||
podman exec -it flyer-crawler-dev npm run type-check
|
||||
# Expected: No type errors
|
||||
```
|
||||
|
||||
## Common Issues
|
||||
---
|
||||
|
||||
## Common Issues and Solutions
|
||||
|
||||
### "Unable to connect to Podman socket"
|
||||
|
||||
**Cause**: Podman machine not running
|
||||
|
||||
**Solution**:
|
||||
|
||||
```bash
|
||||
podman machine start
|
||||
```
|
||||
|
||||
### "Connection refused" to PostgreSQL
|
||||
|
||||
Wait a few seconds for PostgreSQL to initialize:
|
||||
**Cause**: PostgreSQL still initializing
|
||||
|
||||
**Solution**:
|
||||
|
||||
```bash
|
||||
# Wait for PostgreSQL to be ready
|
||||
podman exec flyer-crawler-postgres pg_isready -U postgres
|
||||
# Retry after "accepting connections" message
|
||||
```
|
||||
|
||||
### Port 5432 or 6379 already in use
|
||||
|
||||
Stop conflicting services or change port mappings:
|
||||
**Cause**: Another service using the port
|
||||
|
||||
**Solution**:
|
||||
|
||||
```bash
|
||||
# Use different host port
|
||||
# Option 1: Stop conflicting service
|
||||
# Option 2: Use different host port
|
||||
podman run -d --name flyer-crawler-postgres -p 5433:5432 ...
|
||||
# Then update DB_PORT=5433 in .env.local
|
||||
```
|
||||
|
||||
Then update `DB_PORT=5433` in `.env.local`.
|
||||
### "JWT_SECRET must be at least 32 characters"
|
||||
|
||||
**Cause**: Secret too short in .env.local
|
||||
|
||||
**Solution**: Generate a longer secret:
|
||||
|
||||
```bash
|
||||
node -e "console.log(require('crypto').randomBytes(32).toString('hex'))"
|
||||
```
|
||||
|
||||
### Tests fail with "TZ environment variable" errors
|
||||
|
||||
**Cause**: Timezone setting interfering with Node.js async hooks
|
||||
|
||||
**Solution**: Tests must run in dev container (not Windows host):
|
||||
|
||||
```bash
|
||||
# CORRECT - run in container
|
||||
podman exec -it flyer-crawler-dev npm test
|
||||
|
||||
# INCORRECT - do not run on Windows host
|
||||
npm test
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Next Steps
|
||||
|
||||
- **Read the docs**: [docs/README.md](../README.md)
|
||||
- **Understand the architecture**: [docs/architecture/DATABASE.md](../architecture/DATABASE.md)
|
||||
- **Learn testing**: [docs/development/TESTING.md](../development/TESTING.md)
|
||||
- **Explore ADRs**: [docs/adr/index.md](../adr/index.md)
|
||||
- **Contributing**: [CONTRIBUTING.md](../../CONTRIBUTING.md)
|
||||
| Goal | Document |
|
||||
| ----------------------- | ----------------------------------------------------- |
|
||||
| Understand the codebase | [Architecture Overview](../architecture/OVERVIEW.md) |
|
||||
| Configure environment | [Environment Variables](ENVIRONMENT.md) |
|
||||
| Set up MCP tools | [MCP Configuration](../tools/MCP-CONFIGURATION.md) |
|
||||
| Learn testing | [Testing Guide](../development/TESTING.md) |
|
||||
| Understand DB schema | [Database Documentation](../architecture/DATABASE.md) |
|
||||
| Read ADRs | [ADR Index](../adr/index.md) |
|
||||
| Full installation guide | [Installation Guide](INSTALL.md) |
|
||||
|
||||
## Development Workflow
|
||||
---
|
||||
|
||||
## Daily Development Workflow
|
||||
|
||||
```bash
|
||||
# Daily workflow
|
||||
# 1. Start containers
|
||||
podman start flyer-crawler-postgres flyer-crawler-redis
|
||||
|
||||
# 2. Start dev server
|
||||
npm run dev
|
||||
# ... make changes ...
|
||||
|
||||
# 3. Make changes and test
|
||||
npm test
|
||||
|
||||
# 4. Type check before commit
|
||||
npm run type-check
|
||||
|
||||
# 5. Commit changes
|
||||
git commit
|
||||
```
|
||||
|
||||
For detailed setup instructions, see [INSTALL.md](INSTALL.md).
|
||||
**For dev container users**:
|
||||
|
||||
```bash
|
||||
# 1. Start dev container
|
||||
podman-compose -f compose.dev.yml up -d
|
||||
|
||||
# 2. View logs
|
||||
podman exec -it flyer-crawler-dev pm2 logs
|
||||
|
||||
# 3. Run tests
|
||||
podman exec -it flyer-crawler-dev npm test
|
||||
|
||||
# 4. Stop when done
|
||||
podman-compose -f compose.dev.yml down
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
Last updated: January 2026
|
||||
|
||||
@@ -2,8 +2,54 @@
|
||||
|
||||
This guide covers the manual installation of Flyer Crawler and its dependencies on a bare-metal Ubuntu server (e.g., a colocation server). This is the definitive reference for setting up a production environment without containers.
|
||||
|
||||
**Last verified**: 2026-01-28
|
||||
|
||||
**Target Environment**: Ubuntu 22.04 LTS (or newer)
|
||||
|
||||
**Related documentation**:
|
||||
|
||||
- [ADR-014: Containerization and Deployment Strategy](../adr/0014-containerization-and-deployment-strategy.md)
|
||||
- [ADR-015: Error Tracking and Observability](../adr/0015-error-tracking-and-observability.md)
|
||||
- [ADR-050: PostgreSQL Function Observability](../adr/0050-postgresql-function-observability.md)
|
||||
- [Deployment Guide](DEPLOYMENT.md)
|
||||
- [Monitoring Guide](MONITORING.md)
|
||||
|
||||
---
|
||||
|
||||
## Quick Reference
|
||||
|
||||
### Installation Time Estimates
|
||||
|
||||
| Component | Estimated Time | Notes |
|
||||
| ----------- | --------------- | ----------------------------- |
|
||||
| PostgreSQL | 10-15 minutes | Including PostGIS extensions |
|
||||
| Redis | 5 minutes | Quick install |
|
||||
| Node.js | 5 minutes | Via NodeSource repository |
|
||||
| Application | 15-20 minutes | Clone, install, build |
|
||||
| PM2 | 5 minutes | Global install + config |
|
||||
| NGINX | 10-15 minutes | Including SSL via Certbot |
|
||||
| Bugsink | 20-30 minutes | Python venv, systemd services |
|
||||
| Logstash | 15-20 minutes | Including pipeline config |
|
||||
| **Total** | **~90 minutes** | For complete fresh install |
|
||||
|
||||
### Post-Installation Verification
|
||||
|
||||
After completing setup, verify all services:
|
||||
|
||||
```bash
|
||||
# Check all services are running
|
||||
systemctl status postgresql nginx redis-server gunicorn-bugsink snappea logstash
|
||||
|
||||
# Verify application health
|
||||
curl -s https://flyer-crawler.projectium.com/api/health/ready | jq .
|
||||
|
||||
# Check PM2 processes
|
||||
pm2 list
|
||||
|
||||
# Verify Bugsink is accessible
|
||||
curl -s https://bugsink.projectium.com/accounts/login/ | head -5
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Server Access Model
|
||||
|
||||
@@ -2,6 +2,41 @@
|
||||
|
||||
This guide covers deploying Flyer Crawler to a production server.
|
||||
|
||||
**Last verified**: 2026-01-28
|
||||
|
||||
**Related documentation**:
|
||||
|
||||
- [ADR-014: Containerization and Deployment Strategy](../adr/0014-containerization-and-deployment-strategy.md)
|
||||
- [ADR-015: Error Tracking and Observability](../adr/0015-error-tracking-and-observability.md)
|
||||
- [Bare-Metal Setup Guide](BARE-METAL-SETUP.md)
|
||||
- [Monitoring Guide](MONITORING.md)
|
||||
|
||||
---
|
||||
|
||||
## Quick Reference
|
||||
|
||||
### Command Reference Table
|
||||
|
||||
| Task | Command |
|
||||
| -------------------- | ----------------------------------------------------------------------- |
|
||||
| Deploy to production | Gitea Actions workflow (manual trigger) |
|
||||
| Deploy to test | Automatic on push to `main` |
|
||||
| Check PM2 status | `pm2 list` |
|
||||
| View logs | `pm2 logs flyer-crawler-api --lines 100` |
|
||||
| Restart all | `pm2 restart all` |
|
||||
| Check NGINX | `sudo nginx -t && sudo systemctl status nginx` |
|
||||
| Check health | `curl -s https://flyer-crawler.projectium.com/api/health/ready \| jq .` |
|
||||
|
||||
### Deployment URLs
|
||||
|
||||
| Environment | URL | API Port |
|
||||
| ------------- | ------------------------------------------- | -------- |
|
||||
| Production | `https://flyer-crawler.projectium.com` | 3001 |
|
||||
| Test | `https://flyer-crawler-test.projectium.com` | 3002 |
|
||||
| Dev Container | `https://localhost` | 3001 |
|
||||
|
||||
---
|
||||
|
||||
## Server Access Model
|
||||
|
||||
**Important**: Claude Code (and AI tools) have **READ-ONLY** access to production/test servers. The deployment workflow is:
|
||||
@@ -24,12 +59,24 @@ When troubleshooting deployment issues:
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Ubuntu server (22.04 LTS recommended)
|
||||
- PostgreSQL 14+ with PostGIS extension
|
||||
- Redis
|
||||
- Node.js 20.x
|
||||
- NGINX (reverse proxy)
|
||||
- PM2 (process manager)
|
||||
| Component | Version | Purpose |
|
||||
| ---------- | --------- | ------------------------------- |
|
||||
| Ubuntu | 22.04 LTS | Operating system |
|
||||
| PostgreSQL | 14+ | Database with PostGIS extension |
|
||||
| Redis | 6+ | Caching and job queues |
|
||||
| Node.js | 20.x LTS | Application runtime |
|
||||
| NGINX | 1.18+ | Reverse proxy and static files |
|
||||
| PM2 | Latest | Process manager |
|
||||
|
||||
**Verify prerequisites**:
|
||||
|
||||
```bash
|
||||
node --version # Should be v20.x.x
|
||||
psql --version # Should be 14+
|
||||
redis-cli ping # Should return PONG
|
||||
nginx -v # Should be 1.18+
|
||||
pm2 --version # Any recent version
|
||||
```
|
||||
|
||||
## Dev Container Parity (ADR-014)
|
||||
|
||||
@@ -210,7 +257,7 @@ types {
|
||||
|
||||
**Option 2**: Edit `/etc/nginx/mime.types` globally:
|
||||
|
||||
```
|
||||
```text
|
||||
# Change this line:
|
||||
application/javascript js;
|
||||
|
||||
@@ -341,9 +388,78 @@ The Sentry SDK v10+ enforces HTTPS-only DSNs by default. Since Bugsink runs loca
|
||||
|
||||
---
|
||||
|
||||
## Deployment Troubleshooting
|
||||
|
||||
### Decision Tree: Deployment Issues
|
||||
|
||||
```text
|
||||
Deployment failed?
|
||||
|
|
||||
+-- Build step failed?
|
||||
| |
|
||||
| +-- TypeScript errors --> Fix type issues, run `npm run type-check`
|
||||
| +-- Missing dependencies --> Run `npm ci`
|
||||
| +-- Out of memory --> Increase Node heap size
|
||||
|
|
||||
+-- Tests failed?
|
||||
| |
|
||||
| +-- Database connection --> Check DB_HOST, credentials
|
||||
| +-- Redis connection --> Check REDIS_URL
|
||||
| +-- Test isolation --> Check for race conditions
|
||||
|
|
||||
+-- SSH/Deploy failed?
|
||||
|
|
||||
+-- Permission denied --> Check SSH keys in Gitea secrets
|
||||
+-- Host unreachable --> Check firewall, VPN
|
||||
+-- PM2 error --> Check PM2 logs on server
|
||||
```
|
||||
|
||||
### Common Deployment Issues
|
||||
|
||||
| Symptom | Diagnosis | Solution |
|
||||
| ------------------------------------ | ----------------------- | ------------------------------------------------ |
|
||||
| "Connection refused" on health check | API not started | Check `pm2 logs flyer-crawler-api` |
|
||||
| 502 Bad Gateway | NGINX cannot reach API | Verify API port (3001), restart PM2 |
|
||||
| CSS/JS not loading | Build artifacts missing | Re-run `npm run build`, check NGINX static paths |
|
||||
| Database migrations failed | Schema mismatch | Run migrations manually, check DB connectivity |
|
||||
| "ENOSPC" error | Disk full | Clear old logs: `pm2 flush`, clean npm cache |
|
||||
| SSL certificate error | Cert expired/missing | Run `certbot renew`, check NGINX config |
|
||||
|
||||
### Post-Deployment Verification Checklist
|
||||
|
||||
After every deployment, verify:
|
||||
|
||||
- [ ] Health check passes: `curl -s https://flyer-crawler.projectium.com/api/health/ready`
|
||||
- [ ] PM2 processes running: `pm2 list` shows `online` status
|
||||
- [ ] No recent errors: Check Bugsink for new issues
|
||||
- [ ] Frontend loads: Browser shows login page
|
||||
- [ ] API responds: `curl https://flyer-crawler.projectium.com/api/health/ping`
|
||||
|
||||
### Rollback Procedure
|
||||
|
||||
If deployment causes issues:
|
||||
|
||||
```bash
|
||||
# 1. Check current release
|
||||
cd /var/www/flyer-crawler.projectium.com
|
||||
git log --oneline -5
|
||||
|
||||
# 2. Revert to previous commit
|
||||
git checkout HEAD~1
|
||||
|
||||
# 3. Rebuild and restart
|
||||
npm ci && npm run build
|
||||
pm2 restart all
|
||||
|
||||
# 4. Verify health
|
||||
curl -s http://localhost:3001/api/health/ready | jq .
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [Database Setup](DATABASE.md) - PostgreSQL and PostGIS configuration
|
||||
- [Authentication Setup](AUTHENTICATION.md) - OAuth provider configuration
|
||||
- [Installation Guide](INSTALL.md) - Local development setup
|
||||
- [Bare-Metal Server Setup](docs/BARE-METAL-SETUP.md) - Manual server installation guide
|
||||
- [Database Setup](../architecture/DATABASE.md) - PostgreSQL and PostGIS configuration
|
||||
- [Monitoring Guide](MONITORING.md) - Health checks and error tracking
|
||||
- [Logstash Quick Reference](LOGSTASH-QUICK-REF.md) - Log aggregation
|
||||
- [Bare-Metal Server Setup](BARE-METAL-SETUP.md) - Manual server installation guide
|
||||
|
||||
@@ -2,10 +2,47 @@
|
||||
|
||||
Aggregates logs from PostgreSQL, PM2, Redis, NGINX; forwards errors to Bugsink.
|
||||
|
||||
**Last verified**: 2026-01-28
|
||||
|
||||
**Related documentation**:
|
||||
|
||||
- [ADR-050: PostgreSQL Function Observability](../adr/0050-postgresql-function-observability.md)
|
||||
- [ADR-015: Error Tracking and Observability](../adr/0015-error-tracking-and-observability.md)
|
||||
- [Monitoring Guide](MONITORING.md)
|
||||
- [Logstash Troubleshooting Runbook](LOGSTASH-TROUBLESHOOTING.md)
|
||||
|
||||
---
|
||||
|
||||
## Quick Reference
|
||||
|
||||
### Bugsink Project Routing
|
||||
|
||||
| Source Type | Environment | Bugsink Project | Project ID |
|
||||
| -------------- | ----------- | -------------------- | ---------- |
|
||||
| PM2 API/Worker | Dev | Backend API (Dev) | 1 |
|
||||
| PostgreSQL | Dev | Backend API (Dev) | 1 |
|
||||
| Frontend JS | Dev | Frontend (Dev) | 2 |
|
||||
| Redis/NGINX | Dev | Infrastructure (Dev) | 4 |
|
||||
| PM2 API/Worker | Production | Backend API (Prod) | 1 |
|
||||
| PostgreSQL | Production | Backend API (Prod) | 1 |
|
||||
| PM2 API/Worker | Test | Backend API (Test) | 3 |
|
||||
|
||||
### Key DSN Keys (Dev Container)
|
||||
|
||||
| Project | DSN Key |
|
||||
| -------------------- | ---------------------------------- |
|
||||
| Backend API (Dev) | `cea01396c56246adb5878fa5ee6b1d22` |
|
||||
| Frontend (Dev) | `d92663cb73cf4145b677b84029e4b762` |
|
||||
| Infrastructure (Dev) | `14e8791da3d347fa98073261b596cab9` |
|
||||
|
||||
---
|
||||
|
||||
## Configuration
|
||||
|
||||
**Primary config**: `/etc/logstash/conf.d/bugsink.conf`
|
||||
|
||||
**Dev container config**: `docker/logstash/bugsink.conf`
|
||||
|
||||
### Related Files
|
||||
|
||||
| Path | Purpose |
|
||||
@@ -89,6 +126,34 @@ MSYS_NO_PATHCONV=1 podman exec flyer-crawler-dev ls -la /var/log/redis/
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Decision Tree: Logs Not Appearing in Bugsink
|
||||
|
||||
```text
|
||||
Errors not showing in Bugsink?
|
||||
|
|
||||
+-- Logstash running?
|
||||
| |
|
||||
| +-- No --> systemctl start logstash
|
||||
| +-- Yes --> Check pipeline stats
|
||||
| |
|
||||
| +-- Events in = 0?
|
||||
| | |
|
||||
| | +-- Log files exist? --> ls /var/log/pm2/*.log
|
||||
| | +-- Permissions OK? --> groups logstash
|
||||
| |
|
||||
| +-- Events filtered = high?
|
||||
| | |
|
||||
| | +-- Grok failures --> Check log format matches pattern
|
||||
| |
|
||||
| +-- Events out but no Bugsink?
|
||||
| |
|
||||
| +-- 403 error --> Wrong DSN key
|
||||
| +-- 500 error --> Invalid event format (check sentry_level)
|
||||
| +-- Connection refused --> Bugsink not running
|
||||
```
|
||||
|
||||
### Common Issues Table
|
||||
|
||||
| Issue | Check | Solution |
|
||||
| --------------------- | ---------------- | ---------------------------------------------------------------------------------------------- |
|
||||
| No Bugsink errors | Logstash running | `systemctl status logstash` |
|
||||
@@ -103,6 +168,25 @@ MSYS_NO_PATHCONV=1 podman exec flyer-crawler-dev ls -la /var/log/redis/
|
||||
| High disk usage | Log rotation | Verify `/etc/logrotate.d/logstash` configured |
|
||||
| varchar(7) error | Level validation | Add Ruby filter to validate/normalize `sentry_level` before output |
|
||||
|
||||
### Expected Output Examples
|
||||
|
||||
**Successful Logstash pipeline stats**:
|
||||
|
||||
```json
|
||||
{
|
||||
"in": 1523,
|
||||
"out": 1520,
|
||||
"filtered": 1520,
|
||||
"queue_push_duration_in_millis": 45
|
||||
}
|
||||
```
|
||||
|
||||
**Healthy Bugsink HTTP response**:
|
||||
|
||||
```json
|
||||
{ "id": "a1b2c3d4e5f6..." }
|
||||
```
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- **Dev Container Guide**: [DEV-CONTAINER.md](../development/DEV-CONTAINER.md) - PM2 and log aggregation in dev
|
||||
|
||||
@@ -2,6 +2,16 @@
|
||||
|
||||
This runbook provides step-by-step diagnostics and solutions for common Logstash issues in the PostgreSQL observability pipeline (ADR-050).
|
||||
|
||||
**Last verified**: 2026-01-28
|
||||
|
||||
**Related documentation**:
|
||||
|
||||
- [ADR-050: PostgreSQL Function Observability](../adr/0050-postgresql-function-observability.md)
|
||||
- [Logstash Quick Reference](LOGSTASH-QUICK-REF.md)
|
||||
- [Monitoring Guide](MONITORING.md)
|
||||
|
||||
---
|
||||
|
||||
## Quick Reference
|
||||
|
||||
| Symptom | Most Likely Cause | Quick Check |
|
||||
|
||||
@@ -2,6 +2,72 @@
|
||||
|
||||
This guide covers all aspects of monitoring the Flyer Crawler application across development, test, and production environments.
|
||||
|
||||
**Last verified**: 2026-01-28
|
||||
|
||||
**Related documentation**:
|
||||
|
||||
- [ADR-015: Error Tracking and Observability](../adr/0015-error-tracking-and-observability.md)
|
||||
- [ADR-020: Health Checks](../adr/0020-health-checks-and-liveness-readiness-probes.md)
|
||||
- [ADR-050: PostgreSQL Function Observability](../adr/0050-postgresql-function-observability.md)
|
||||
- [Logstash Quick Reference](LOGSTASH-QUICK-REF.md)
|
||||
- [Deployment Guide](DEPLOYMENT.md)
|
||||
|
||||
---
|
||||
|
||||
## Quick Reference
|
||||
|
||||
### Monitoring URLs
|
||||
|
||||
| Service | Production URL | Dev Container URL |
|
||||
| ------------ | ------------------------------------------------------- | ---------------------------------------- |
|
||||
| Health Check | `https://flyer-crawler.projectium.com/api/health/ready` | `http://localhost:3001/api/health/ready` |
|
||||
| Bugsink | `https://bugsink.projectium.com` | `https://localhost:8443` |
|
||||
| Bull Board | `https://flyer-crawler.projectium.com/api/admin/jobs` | `http://localhost:3001/api/admin/jobs` |
|
||||
|
||||
### Quick Diagnostic Commands
|
||||
|
||||
```bash
|
||||
# Check all services at once (production)
|
||||
curl -s https://flyer-crawler.projectium.com/api/health/ready | jq '.data.services'
|
||||
|
||||
# Dev container health check
|
||||
podman exec flyer-crawler-dev curl -s http://localhost:3001/api/health/ready | jq .
|
||||
|
||||
# PM2 process overview
|
||||
pm2 list
|
||||
|
||||
# Recent errors in Bugsink (via MCP)
|
||||
# mcp__bugsink__list_issues --project_id 1 --status unresolved
|
||||
```
|
||||
|
||||
### Monitoring Decision Tree
|
||||
|
||||
```text
|
||||
Application seems slow or unresponsive?
|
||||
|
|
||||
+-- Check health endpoint first
|
||||
| |
|
||||
| +-- Returns unhealthy?
|
||||
| | |
|
||||
| | +-- Database unhealthy --> Check DB pool, connections
|
||||
| | +-- Redis unhealthy --> Check Redis memory, connection
|
||||
| | +-- Storage unhealthy --> Check disk space, permissions
|
||||
| |
|
||||
| +-- Returns healthy but slow?
|
||||
| |
|
||||
| +-- Check PM2 memory/CPU usage
|
||||
| +-- Check database slow query log
|
||||
| +-- Check Redis queue depth
|
||||
|
|
||||
+-- Health endpoint not responding?
|
||||
|
|
||||
+-- Check PM2 status --> Process crashed?
|
||||
+-- Check NGINX --> 502 errors?
|
||||
+-- Check network --> Firewall/DNS issues?
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
|
||||
1. [Health Checks](#health-checks)
|
||||
@@ -294,7 +360,7 @@ The command outputs a 40-character hex token.
|
||||
|
||||
**Error Anatomy**:
|
||||
|
||||
```
|
||||
```text
|
||||
TypeError: Cannot read properties of undefined (reading 'map')
|
||||
├── Exception Type: TypeError
|
||||
├── Message: Cannot read properties of undefined (reading 'map')
|
||||
@@ -357,7 +423,7 @@ Logstash aggregates logs from multiple sources and forwards errors to Bugsink (A
|
||||
|
||||
### Architecture
|
||||
|
||||
```
|
||||
```text
|
||||
Log Sources Logstash Outputs
|
||||
┌──────────────┐ ┌─────────────┐ ┌─────────────┐
|
||||
│ PostgreSQL │──────────────│ │───────────│ Bugsink │
|
||||
@@ -520,7 +586,7 @@ pm2 stop flyer-crawler-api
|
||||
|
||||
**Healthy Process**:
|
||||
|
||||
```
|
||||
```text
|
||||
┌─────────────────────┬────┬─────────┬─────────┬───────┬────────┬─────────┬──────────┐
|
||||
│ Name │ id │ mode │ status │ cpu │ mem │ uptime │ restarts │
|
||||
├─────────────────────┼────┼─────────┼─────────┼───────┼────────┼─────────┼──────────┤
|
||||
@@ -833,7 +899,7 @@ Configure alerts in your monitoring tool (UptimeRobot, Datadog, etc.):
|
||||
2. Review during business hours
|
||||
3. Create Gitea issue for tracking
|
||||
|
||||
### Quick Diagnostic Commands
|
||||
### On-Call Diagnostic Commands
|
||||
|
||||
> **Note**: User executes these commands on the server. Claude Code provides commands but cannot run them directly.
|
||||
|
||||
|
||||
@@ -2,6 +2,17 @@
|
||||
|
||||
The **ai-usage** subagent specializes in LLM APIs (Gemini, Claude), prompt engineering, and AI-powered features in the Flyer Crawler project.
|
||||
|
||||
## Quick Reference
|
||||
|
||||
| Aspect | Details |
|
||||
| ------------------ | ----------------------------------------------------------------------------------- |
|
||||
| **Primary Use** | Gemini API integration, prompt engineering, AI extraction |
|
||||
| **Key Files** | `src/services/aiService.server.ts`, `src/services/flyerProcessingService.server.ts` |
|
||||
| **Key ADRs** | ADR-041 (AI Integration), ADR-046 (Image Processing) |
|
||||
| **API Key Env** | `VITE_GOOGLE_GENAI_API_KEY` (prod), `VITE_GOOGLE_GENAI_API_KEY_TEST` (test) |
|
||||
| **Error Handling** | Rate limits (429), JSON parse errors, timeout handling |
|
||||
| **Delegate To** | `coder` (implementation), `testwriter` (tests), `integrations-specialist` |
|
||||
|
||||
## When to Use
|
||||
|
||||
Use the **ai-usage** subagent when you need to:
|
||||
@@ -295,6 +306,9 @@ const fixtureResponse = await fs.readFile('fixtures/gemini-response.json');
|
||||
## Related Documentation
|
||||
|
||||
- [OVERVIEW.md](./OVERVIEW.md) - Subagent system overview
|
||||
- [CODER-GUIDE.md](./CODER-GUIDE.md) - For implementing AI features
|
||||
- [TESTER-GUIDE.md](./TESTER-GUIDE.md) - Testing AI features
|
||||
- [INTEGRATIONS-GUIDE.md](./INTEGRATIONS-GUIDE.md) - External API patterns
|
||||
- [../adr/0041-ai-gemini-integration-architecture.md](../adr/0041-ai-gemini-integration-architecture.md) - AI integration ADR
|
||||
- [../adr/0046-image-processing-pipeline.md](../adr/0046-image-processing-pipeline.md) - Image processing
|
||||
- [CODER-GUIDE.md](./CODER-GUIDE.md) - For implementing AI features
|
||||
- [../getting-started/ENVIRONMENT.md](../getting-started/ENVIRONMENT.md) - Environment configuration
|
||||
|
||||
@@ -2,6 +2,17 @@
|
||||
|
||||
The **coder** subagent is your primary tool for writing and modifying production Node.js/TypeScript code in the Flyer Crawler project. This guide explains how to work effectively with the coder subagent.
|
||||
|
||||
## Quick Reference
|
||||
|
||||
| Aspect | Details |
|
||||
| ---------------- | ------------------------------------------------------------------------ |
|
||||
| **Primary Use** | Write/modify production TypeScript code |
|
||||
| **Key Files** | `src/routes/*.routes.ts`, `src/services/**/*.ts`, `src/components/*.tsx` |
|
||||
| **Key ADRs** | ADR-034 (Repository), ADR-035 (Services), ADR-028 (API Response) |
|
||||
| **Test Command** | `podman exec -it flyer-crawler-dev npm run test:unit` |
|
||||
| **Type Check** | `podman exec -it flyer-crawler-dev npm run type-check` |
|
||||
| **Delegate To** | `db-dev` (database), `frontend-specialist` (UI), `testwriter` (tests) |
|
||||
|
||||
## When to Use the Coder Subagent
|
||||
|
||||
Use the coder subagent when you need to:
|
||||
@@ -307,6 +318,8 @@ error classes for all database operations"
|
||||
|
||||
- [OVERVIEW.md](./OVERVIEW.md) - Subagent system overview
|
||||
- [TESTER-GUIDE.md](./TESTER-GUIDE.md) - Testing strategies
|
||||
- [DATABASE-GUIDE.md](./DATABASE-GUIDE.md) - Database development workflows
|
||||
- [../adr/0034-repository-pattern-standards.md](../adr/0034-repository-pattern-standards.md) - Repository patterns
|
||||
- [../adr/0035-service-layer-architecture.md](../adr/0035-service-layer-architecture.md) - Service layer architecture
|
||||
- [../adr/0028-api-response-standardization.md](../adr/0028-api-response-standardization.md) - API response patterns
|
||||
- [../development/CODE-PATTERNS.md](../development/CODE-PATTERNS.md) - Code patterns reference
|
||||
|
||||
@@ -5,6 +5,17 @@ This guide covers two database-focused subagents:
|
||||
- **db-dev**: Database development - schemas, queries, migrations, optimization
|
||||
- **db-admin**: Database administration - PostgreSQL/Redis admin, security, backups
|
||||
|
||||
## Quick Reference
|
||||
|
||||
| Aspect | db-dev | db-admin |
|
||||
| ---------------- | -------------------------------------------- | ------------------------------------------ |
|
||||
| **Primary Use** | Schemas, queries, migrations | Performance tuning, backups, security |
|
||||
| **Key Files** | `src/services/db/*.db.ts`, `sql/migrations/` | `postgresql.conf`, `pg_hba.conf` |
|
||||
| **Key ADRs** | ADR-034 (Repository), ADR-002 (Transactions) | ADR-019 (Backups), ADR-050 (Observability) |
|
||||
| **Test Command** | `podman exec -it flyer-crawler-dev npm test` | N/A |
|
||||
| **MCP Tool** | `mcp__devdb__query` | SSH to production |
|
||||
| **Delegate To** | `coder` (service layer), `db-admin` (perf) | `devops` (infrastructure) |
|
||||
|
||||
## Understanding the Difference
|
||||
|
||||
| Aspect | db-dev | db-admin |
|
||||
@@ -412,8 +423,9 @@ This is useful for:
|
||||
|
||||
- [OVERVIEW.md](./OVERVIEW.md) - Subagent system overview
|
||||
- [CODER-GUIDE.md](./CODER-GUIDE.md) - Working with the coder subagent
|
||||
- [DEVOPS-GUIDE.md](./DEVOPS-GUIDE.md) - DevOps and deployment workflows
|
||||
- [../adr/0034-repository-pattern-standards.md](../adr/0034-repository-pattern-standards.md) - Repository patterns
|
||||
- [../adr/0002-standardized-transaction-management.md](../adr/0002-standardized-transaction-management.md) - Transaction management
|
||||
- [../adr/0019-data-backup-and-recovery-strategy.md](../adr/0019-data-backup-and-recovery-strategy.md) - Backup strategy
|
||||
- [../adr/0050-postgresql-function-observability.md](../adr/0050-postgresql-function-observability.md) - Database observability
|
||||
- [../BARE-METAL-SETUP.md](../BARE-METAL-SETUP.md) - Production database setup
|
||||
- [../operations/BARE-METAL-SETUP.md](../operations/BARE-METAL-SETUP.md) - Production database setup
|
||||
|
||||
@@ -6,6 +6,17 @@ This guide covers DevOps-related subagents for deployment, infrastructure, and o
|
||||
- **infra-architect**: Resource optimization, capacity planning
|
||||
- **bg-worker**: Background jobs, PM2 workers, BullMQ queues
|
||||
|
||||
## Quick Reference
|
||||
|
||||
| Aspect | devops | infra-architect | bg-worker |
|
||||
| ---------------- | ------------------------------------------ | --------------------------- | ------------------------------- |
|
||||
| **Primary Use** | Containers, CI/CD, deployments | Resource optimization | BullMQ queues, PM2 workers |
|
||||
| **Key Files** | `compose.dev.yml`, `.gitea/workflows/` | `ecosystem.config.cjs` | `src/services/queues.server.ts` |
|
||||
| **Key ADRs** | ADR-014 (Containers), ADR-017 (CI/CD) | N/A | ADR-006 (Background Jobs) |
|
||||
| **Commands** | `podman-compose`, `pm2` | `pm2 monit`, system metrics | Redis CLI, `pm2 logs` |
|
||||
| **MCP Tools** | `mcp__podman__*` | N/A | N/A |
|
||||
| **Access Model** | Read-only on production (provide commands) | Same | Same |
|
||||
|
||||
---
|
||||
|
||||
## CRITICAL: Server Access Model
|
||||
@@ -543,8 +554,13 @@ podman exec -it flyer-crawler-dev npm test
|
||||
## Related Documentation
|
||||
|
||||
- [OVERVIEW.md](./OVERVIEW.md) - Subagent system overview
|
||||
- [../BARE-METAL-SETUP.md](../BARE-METAL-SETUP.md) - Production setup guide
|
||||
- [DATABASE-GUIDE.md](./DATABASE-GUIDE.md) - Database administration
|
||||
- [SECURITY-DEBUG-GUIDE.md](./SECURITY-DEBUG-GUIDE.md) - Production debugging
|
||||
- [../operations/BARE-METAL-SETUP.md](../operations/BARE-METAL-SETUP.md) - Production setup guide
|
||||
- [../operations/DEPLOYMENT.md](../operations/DEPLOYMENT.md) - Deployment guide
|
||||
- [../operations/MONITORING.md](../operations/MONITORING.md) - Monitoring guide
|
||||
- [../development/DEV-CONTAINER.md](../development/DEV-CONTAINER.md) - Dev container guide
|
||||
- [../adr/0014-containerization-and-deployment-strategy.md](../adr/0014-containerization-and-deployment-strategy.md) - Containerization ADR
|
||||
- [../adr/0006-background-job-processing-and-task-queues.md](../adr/0006-background-job-processing-and-task-queues.md) - Background jobs ADR
|
||||
- [../adr/0017-ci-cd-and-branching-strategy.md](../adr/0017-ci-cd-and-branching-strategy.md) - CI/CD strategy
|
||||
- [../adr/0053-worker-health-checks.md](../adr/0053-worker-health-checks.md) - Worker health checks
|
||||
- [../adr/0053-worker-health-checks-and-monitoring.md](../adr/0053-worker-health-checks-and-monitoring.md) - Worker health checks
|
||||
|
||||
@@ -7,6 +7,15 @@ This guide covers documentation-focused subagents:
|
||||
- **planner**: Feature breakdown, roadmaps, scope management
|
||||
- **product-owner**: Requirements, user stories, backlog prioritization
|
||||
|
||||
## Quick Reference
|
||||
|
||||
| Aspect | documenter | describer-for-ai | planner | product-owner |
|
||||
| --------------- | -------------------- | ------------------------ | --------------------- | ---------------------- |
|
||||
| **Primary Use** | User docs, API specs | ADRs, technical specs | Feature breakdown | User stories, backlog |
|
||||
| **Key Files** | `docs/`, API docs | `docs/adr/`, `CLAUDE.md` | `docs/plans/` | Issue tracker |
|
||||
| **Output** | Markdown guides | ADRs, context docs | Task lists, roadmaps | User stories, criteria |
|
||||
| **Delegate To** | `coder` (implement) | `documenter` (user docs) | `coder` (build tasks) | `planner` (breakdown) |
|
||||
|
||||
## The documenter Subagent
|
||||
|
||||
### When to Use
|
||||
@@ -437,6 +446,8 @@ Include dates on documentation that may become stale:
|
||||
## Related Documentation
|
||||
|
||||
- [OVERVIEW.md](./OVERVIEW.md) - Subagent system overview
|
||||
- [CODER-GUIDE.md](./CODER-GUIDE.md) - For implementing documented features
|
||||
- [../adr/index.md](../adr/index.md) - ADR index
|
||||
- [../TESTING.md](../TESTING.md) - Testing guide
|
||||
- [../development/TESTING.md](../development/TESTING.md) - Testing guide
|
||||
- [../development/CODE-PATTERNS.md](../development/CODE-PATTERNS.md) - Code patterns reference
|
||||
- [../../CLAUDE.md](../../CLAUDE.md) - AI instructions
|
||||
|
||||
@@ -5,6 +5,17 @@ This guide covers frontend-focused subagents:
|
||||
- **frontend-specialist**: UI components, Neo-Brutalism, Core Web Vitals, accessibility
|
||||
- **uiux-designer**: UI/UX decisions, component design, user experience
|
||||
|
||||
## Quick Reference
|
||||
|
||||
| Aspect | frontend-specialist | uiux-designer |
|
||||
| ----------------- | ---------------------------------------------- | -------------------------------------- |
|
||||
| **Primary Use** | React components, performance, accessibility | Design decisions, user flows |
|
||||
| **Key Files** | `src/components/`, `src/features/` | Design specs, mockups |
|
||||
| **Key ADRs** | ADR-012 (Design System), ADR-044 (Feature Org) | ADR-012 (Design System) |
|
||||
| **Design System** | Neo-Brutalism (bold borders, high contrast) | Same |
|
||||
| **State Mgmt** | TanStack Query (server), Zustand (client) | N/A |
|
||||
| **Delegate To** | `coder` (backend), `tester` (test coverage) | `frontend-specialist` (implementation) |
|
||||
|
||||
## The frontend-specialist Subagent
|
||||
|
||||
### When to Use
|
||||
@@ -406,7 +417,8 @@ const handleSelect = useCallback((id: string) => {
|
||||
|
||||
- [OVERVIEW.md](./OVERVIEW.md) - Subagent system overview
|
||||
- [CODER-GUIDE.md](./CODER-GUIDE.md) - For implementing features
|
||||
- [../DESIGN_TOKENS.md](../DESIGN_TOKENS.md) - Design token reference
|
||||
- [TESTER-GUIDE.md](./TESTER-GUIDE.md) - Component testing patterns
|
||||
- [../development/DESIGN_TOKENS.md](../development/DESIGN_TOKENS.md) - Design token reference
|
||||
- [../adr/0012-frontend-component-library-and-design-system.md](../adr/0012-frontend-component-library-and-design-system.md) - Design system ADR
|
||||
- [../adr/0005-frontend-state-management-and-server-cache-strategy.md](../adr/0005-frontend-state-management-and-server-cache-strategy.md) - State management ADR
|
||||
- [../adr/0044-frontend-feature-organization.md](../adr/0044-frontend-feature-organization.md) - Feature organization
|
||||
|
||||
396
docs/subagents/INTEGRATIONS-GUIDE.md
Normal file
396
docs/subagents/INTEGRATIONS-GUIDE.md
Normal file
@@ -0,0 +1,396 @@
|
||||
# Integrations Subagent Guide
|
||||
|
||||
The **integrations-specialist** subagent handles third-party services, webhooks, and external API integrations in the Flyer Crawler project.
|
||||
|
||||
## Quick Reference
|
||||
|
||||
| Aspect | Details |
|
||||
| --------------- | --------------------------------------------------------------------------- |
|
||||
| **Primary Use** | External APIs, webhooks, OAuth, third-party services |
|
||||
| **Key Files** | `src/services/external/`, `src/routes/webhooks.routes.ts` |
|
||||
| **Key ADRs** | ADR-041 (AI Integration), ADR-016 (API Security), ADR-048 (Auth) |
|
||||
| **MCP Tools** | `mcp__gitea-projectium__*`, `mcp__bugsink__*` |
|
||||
| **Security** | API key storage, webhook signatures, OAuth state param |
|
||||
| **Delegate To** | `coder` (implementation), `security-engineer` (review), `ai-usage` (Gemini) |
|
||||
|
||||
## When to Use
|
||||
|
||||
Use the **integrations-specialist** subagent when you need to:
|
||||
|
||||
- Integrate with external APIs (OAuth, REST, GraphQL)
|
||||
- Implement webhook handlers
|
||||
- Configure third-party services
|
||||
- Debug external service connectivity
|
||||
- Handle API authentication flows
|
||||
- Manage external service rate limits
|
||||
|
||||
## What integrations-specialist Knows
|
||||
|
||||
The integrations-specialist subagent understands:
|
||||
|
||||
- OAuth 2.0 flows (authorization code, client credentials)
|
||||
- REST API integration patterns
|
||||
- Webhook security (signature verification)
|
||||
- External service error handling
|
||||
- Rate limiting and retry strategies
|
||||
- API key management
|
||||
|
||||
## Current Integrations
|
||||
|
||||
| Service | Purpose | Integration Type | Key Files |
|
||||
| ------------- | ---------------------- | ---------------- | ---------------------------------- |
|
||||
| Google Gemini | AI flyer extraction | REST API | `src/services/aiService.server.ts` |
|
||||
| Bugsink | Error tracking | REST API | MCP: `mcp__bugsink__*` |
|
||||
| Gitea | Repository and CI/CD | REST API | MCP: `mcp__gitea-projectium__*` |
|
||||
| Redis | Caching and job queues | Native client | `src/services/redis.server.ts` |
|
||||
| PostgreSQL | Primary database | Native client | `src/services/db/pool.db.ts` |
|
||||
|
||||
## Example Requests
|
||||
|
||||
### Adding External API Integration
|
||||
|
||||
```
|
||||
"Use integrations-specialist to integrate with the Store API
|
||||
to automatically fetch store location data. Include proper
|
||||
error handling, rate limiting, and caching."
|
||||
```
|
||||
|
||||
### OAuth Implementation
|
||||
|
||||
```
|
||||
"Use integrations-specialist to implement Google OAuth for
|
||||
user authentication. Include token refresh handling and
|
||||
session management."
|
||||
```
|
||||
|
||||
### Webhook Handler
|
||||
|
||||
```
|
||||
"Use integrations-specialist to create a webhook handler for
|
||||
receiving store inventory updates. Include signature verification
|
||||
and idempotency handling."
|
||||
```
|
||||
|
||||
### Debugging External Service Issues
|
||||
|
||||
```
|
||||
"Use integrations-specialist to debug why the Gemini API calls
|
||||
are intermittently failing with timeout errors. Check connection
|
||||
pooling, retry logic, and error handling."
|
||||
```
|
||||
|
||||
## Integration Patterns
|
||||
|
||||
### REST API Client Pattern
|
||||
|
||||
```typescript
|
||||
// src/services/external/storeApi.server.ts
|
||||
import { env } from '@/config/env';
|
||||
import { log } from '@/services/logger.server';
|
||||
|
||||
interface StoreApiConfig {
|
||||
baseUrl: string;
|
||||
apiKey: string;
|
||||
timeout: number;
|
||||
}
|
||||
|
||||
class StoreApiClient {
|
||||
private config: StoreApiConfig;
|
||||
|
||||
constructor(config: StoreApiConfig) {
|
||||
this.config = config;
|
||||
}
|
||||
|
||||
async getStoreLocations(storeId: string): Promise<StoreLocation[]> {
|
||||
const url = `${this.config.baseUrl}/stores/${storeId}/locations`;
|
||||
|
||||
try {
|
||||
const response = await fetch(url, {
|
||||
headers: {
|
||||
Authorization: `Bearer ${this.config.apiKey}`,
|
||||
'Content-Type': 'application/json',
|
||||
},
|
||||
signal: AbortSignal.timeout(this.config.timeout),
|
||||
});
|
||||
|
||||
if (!response.ok) {
|
||||
throw new ExternalApiError(`Store API error: ${response.status}`, response.status);
|
||||
}
|
||||
|
||||
return response.json();
|
||||
} catch (error) {
|
||||
log.error({ error, storeId }, 'Failed to fetch store locations');
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
export const storeApiClient = new StoreApiClient({
|
||||
baseUrl: env.STORE_API_BASE_URL,
|
||||
apiKey: env.STORE_API_KEY,
|
||||
timeout: 10000,
|
||||
});
|
||||
```
|
||||
|
||||
### Webhook Handler Pattern
|
||||
|
||||
```typescript
|
||||
// src/routes/webhooks.routes.ts
|
||||
import { Router } from 'express';
|
||||
import crypto from 'crypto';
|
||||
import { env } from '@/config/env';
|
||||
|
||||
const router = Router();
|
||||
|
||||
function verifyWebhookSignature(payload: string, signature: string, secret: string): boolean {
|
||||
const expected = crypto.createHmac('sha256', secret).update(payload).digest('hex');
|
||||
return crypto.timingSafeEqual(Buffer.from(signature), Buffer.from(`sha256=${expected}`));
|
||||
}
|
||||
|
||||
router.post('/store-updates', async (req, res, next) => {
|
||||
try {
|
||||
const signature = req.headers['x-webhook-signature'] as string;
|
||||
const payload = JSON.stringify(req.body);
|
||||
|
||||
if (!verifyWebhookSignature(payload, signature, env.WEBHOOK_SECRET)) {
|
||||
return res.status(401).json({ error: 'Invalid signature' });
|
||||
}
|
||||
|
||||
// Process webhook with idempotency check
|
||||
const eventId = req.headers['x-event-id'] as string;
|
||||
const alreadyProcessed = await checkIdempotencyKey(eventId);
|
||||
|
||||
if (alreadyProcessed) {
|
||||
return res.status(200).json({ status: 'already_processed' });
|
||||
}
|
||||
|
||||
await processStoreUpdate(req.body);
|
||||
await markEventProcessed(eventId);
|
||||
|
||||
res.status(200).json({ status: 'processed' });
|
||||
} catch (error) {
|
||||
next(error);
|
||||
}
|
||||
});
|
||||
```
|
||||
|
||||
### OAuth Flow Pattern
|
||||
|
||||
```typescript
|
||||
// src/services/oauth/googleOAuth.server.ts
|
||||
import { OAuth2Client } from 'google-auth-library';
|
||||
import { env } from '@/config/env';
|
||||
|
||||
const oauth2Client = new OAuth2Client(
|
||||
env.GOOGLE_CLIENT_ID,
|
||||
env.GOOGLE_CLIENT_SECRET,
|
||||
env.GOOGLE_REDIRECT_URI,
|
||||
);
|
||||
|
||||
export function getAuthorizationUrl(): string {
|
||||
return oauth2Client.generateAuthUrl({
|
||||
access_type: 'offline',
|
||||
scope: ['email', 'profile'],
|
||||
prompt: 'consent',
|
||||
});
|
||||
}
|
||||
|
||||
export async function exchangeCodeForTokens(code: string) {
|
||||
const { tokens } = await oauth2Client.getToken(code);
|
||||
return tokens;
|
||||
}
|
||||
|
||||
export async function refreshAccessToken(refreshToken: string) {
|
||||
oauth2Client.setCredentials({ refresh_token: refreshToken });
|
||||
const { credentials } = await oauth2Client.refreshAccessToken();
|
||||
return credentials;
|
||||
}
|
||||
```
|
||||
|
||||
## Error Handling for External Services
|
||||
|
||||
### Custom Error Classes
|
||||
|
||||
```typescript
|
||||
// src/services/external/errors.ts
|
||||
export class ExternalApiError extends Error {
|
||||
constructor(
|
||||
message: string,
|
||||
public statusCode: number,
|
||||
public retryable: boolean = false,
|
||||
) {
|
||||
super(message);
|
||||
this.name = 'ExternalApiError';
|
||||
}
|
||||
}
|
||||
|
||||
export class RateLimitError extends ExternalApiError {
|
||||
constructor(
|
||||
message: string,
|
||||
public retryAfter: number,
|
||||
) {
|
||||
super(message, 429, true);
|
||||
this.name = 'RateLimitError';
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Retry with Exponential Backoff
|
||||
|
||||
```typescript
|
||||
async function fetchWithRetry<T>(
|
||||
fn: () => Promise<T>,
|
||||
options: { maxRetries: number; baseDelay: number },
|
||||
): Promise<T> {
|
||||
let lastError: Error;
|
||||
|
||||
for (let attempt = 0; attempt <= options.maxRetries; attempt++) {
|
||||
try {
|
||||
return await fn();
|
||||
} catch (error) {
|
||||
lastError = error as Error;
|
||||
|
||||
if (error instanceof ExternalApiError && !error.retryable) {
|
||||
throw error;
|
||||
}
|
||||
|
||||
if (attempt < options.maxRetries) {
|
||||
const delay = options.baseDelay * Math.pow(2, attempt);
|
||||
await new Promise((resolve) => setTimeout(resolve, delay));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
throw lastError!;
|
||||
}
|
||||
```
|
||||
|
||||
## Rate Limiting Strategies
|
||||
|
||||
### Token Bucket Pattern
|
||||
|
||||
```typescript
|
||||
class RateLimiter {
|
||||
private tokens: number;
|
||||
private lastRefill: number;
|
||||
private readonly maxTokens: number;
|
||||
private readonly refillRate: number; // tokens per second
|
||||
|
||||
constructor(maxTokens: number, refillRate: number) {
|
||||
this.maxTokens = maxTokens;
|
||||
this.tokens = maxTokens;
|
||||
this.refillRate = refillRate;
|
||||
this.lastRefill = Date.now();
|
||||
}
|
||||
|
||||
async acquire(): Promise<void> {
|
||||
this.refill();
|
||||
|
||||
if (this.tokens < 1) {
|
||||
const waitTime = ((1 - this.tokens) / this.refillRate) * 1000;
|
||||
await new Promise((resolve) => setTimeout(resolve, waitTime));
|
||||
this.refill();
|
||||
}
|
||||
|
||||
this.tokens -= 1;
|
||||
}
|
||||
|
||||
private refill(): void {
|
||||
const now = Date.now();
|
||||
const elapsed = (now - this.lastRefill) / 1000;
|
||||
this.tokens = Math.min(this.maxTokens, this.tokens + elapsed * this.refillRate);
|
||||
this.lastRefill = now;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Testing Integrations
|
||||
|
||||
### Mocking External Services
|
||||
|
||||
```typescript
|
||||
// src/tests/mocks/storeApi.mock.ts
|
||||
import { vi } from 'vitest';
|
||||
|
||||
export const mockStoreApiClient = {
|
||||
getStoreLocations: vi.fn(),
|
||||
};
|
||||
|
||||
vi.mock('@/services/external/storeApi.server', () => ({
|
||||
storeApiClient: mockStoreApiClient,
|
||||
}));
|
||||
```
|
||||
|
||||
### Integration Test with Real Service
|
||||
|
||||
```typescript
|
||||
// src/tests/integration/storeApi.integration.test.ts
|
||||
describe('Store API Integration', () => {
|
||||
it.skipIf(!env.STORE_API_KEY)('fetches real store locations', async () => {
|
||||
const locations = await storeApiClient.getStoreLocations('test-store');
|
||||
expect(locations).toBeInstanceOf(Array);
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
## MCP Tools for Integrations
|
||||
|
||||
### Gitea Integration
|
||||
|
||||
```
|
||||
// List repositories
|
||||
mcp__gitea-projectium__list_my_repos()
|
||||
|
||||
// Create issue
|
||||
mcp__gitea-projectium__create_issue({
|
||||
owner: "projectium",
|
||||
repo: "flyer-crawler",
|
||||
title: "Issue title",
|
||||
body: "Issue description"
|
||||
})
|
||||
```
|
||||
|
||||
### Bugsink Integration
|
||||
|
||||
```
|
||||
// List projects
|
||||
mcp__bugsink__list_projects()
|
||||
|
||||
// Get issue details
|
||||
mcp__bugsink__get_issue({ issue_id: "..." })
|
||||
|
||||
// Get stacktrace
|
||||
mcp__bugsink__get_stacktrace({ event_id: "..." })
|
||||
```
|
||||
|
||||
## Security Considerations
|
||||
|
||||
### API Key Storage
|
||||
|
||||
- Never commit API keys to version control
|
||||
- Use environment variables via `src/config/env.ts`
|
||||
- Rotate keys periodically
|
||||
- Use separate keys for dev/test/prod
|
||||
|
||||
### Webhook Security
|
||||
|
||||
- Always verify webhook signatures
|
||||
- Use HTTPS for webhook endpoints
|
||||
- Implement idempotency
|
||||
- Log webhook events for audit
|
||||
|
||||
### OAuth Security
|
||||
|
||||
- Use state parameter to prevent CSRF
|
||||
- Store tokens securely (encrypted at rest)
|
||||
- Implement token refresh before expiration
|
||||
- Validate token scopes
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [OVERVIEW.md](./OVERVIEW.md) - Subagent system overview
|
||||
- [SECURITY-DEBUG-GUIDE.md](./SECURITY-DEBUG-GUIDE.md) - Security patterns
|
||||
- [AI-USAGE-GUIDE.md](./AI-USAGE-GUIDE.md) - Gemini API integration
|
||||
- [../adr/0041-ai-gemini-integration-architecture.md](../adr/0041-ai-gemini-integration-architecture.md) - AI integration ADR
|
||||
- [../adr/0016-api-security-hardening.md](../adr/0016-api-security-hardening.md) - API security
|
||||
- [../adr/0048-authentication-strategy.md](../adr/0048-authentication-strategy.md) - Authentication
|
||||
@@ -89,6 +89,47 @@ Or:
|
||||
|
||||
Claude will automatically invoke the appropriate subagent with the relevant context.
|
||||
|
||||
## Quick Reference Decision Tree
|
||||
|
||||
Use this flowchart to quickly identify the right subagent:
|
||||
|
||||
```
|
||||
What do you need to do?
|
||||
|
|
||||
+-- Write/modify code? ----------------> Is it database-related?
|
||||
| |
|
||||
| +-- Yes -> db-dev
|
||||
| +-- No --> Is it frontend?
|
||||
| |
|
||||
| +-- Yes -> frontend-specialist
|
||||
| +-- No --> Is it AI/Gemini?
|
||||
| |
|
||||
| +-- Yes -> ai-usage
|
||||
| +-- No --> coder
|
||||
|
|
||||
+-- Test something? -------------------> Write new tests? -> testwriter
|
||||
| Find bugs/vulnerabilities? -> tester
|
||||
| Review existing code? -> code-reviewer
|
||||
|
|
||||
+-- Debug an issue? -------------------> Production error? -> log-debug
|
||||
| Database slow? -> db-admin
|
||||
| External API failing? -> integrations-specialist
|
||||
| AI extraction failing? -> ai-usage
|
||||
|
|
||||
+-- Infrastructure/Deployment? --------> Container/CI/CD? -> devops
|
||||
| Resource optimization? -> infra-architect
|
||||
| Background jobs? -> bg-worker
|
||||
|
|
||||
+-- Documentation? --------------------> User-facing docs? -> documenter
|
||||
| ADRs/Technical specs? -> describer-for-ai
|
||||
| Feature planning? -> planner
|
||||
| User stories? -> product-owner
|
||||
|
|
||||
+-- Security? -------------------------> security-engineer
|
||||
|
|
||||
+-- Design/UX? ------------------------> uiux-designer
|
||||
```
|
||||
|
||||
## Subagent Selection Guide
|
||||
|
||||
### Which Subagent Should I Use?
|
||||
@@ -183,12 +224,26 @@ Subagents can pass information back to the main conversation and to each other t
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [CODER-GUIDE.md](./CODER-GUIDE.md) - Working with the coder subagent
|
||||
- [TESTER-GUIDE.md](./TESTER-GUIDE.md) - Testing strategies and patterns
|
||||
- [DATABASE-GUIDE.md](./DATABASE-GUIDE.md) - Database development workflows
|
||||
- [DEVOPS-GUIDE.md](./DEVOPS-GUIDE.md) - DevOps and deployment workflows
|
||||
### Subagent Guides
|
||||
|
||||
| Guide | Subagents Covered |
|
||||
| ---------------------------------------------------- | ----------------------------------------------------- |
|
||||
| [CODER-GUIDE.md](./CODER-GUIDE.md) | coder |
|
||||
| [TESTER-GUIDE.md](./TESTER-GUIDE.md) | tester, testwriter |
|
||||
| [DATABASE-GUIDE.md](./DATABASE-GUIDE.md) | db-dev, db-admin |
|
||||
| [DEVOPS-GUIDE.md](./DEVOPS-GUIDE.md) | devops, infra-architect, bg-worker |
|
||||
| [FRONTEND-GUIDE.md](./FRONTEND-GUIDE.md) | frontend-specialist, uiux-designer |
|
||||
| [SECURITY-DEBUG-GUIDE.md](./SECURITY-DEBUG-GUIDE.md) | security-engineer, log-debug, code-reviewer |
|
||||
| [AI-USAGE-GUIDE.md](./AI-USAGE-GUIDE.md) | ai-usage |
|
||||
| [INTEGRATIONS-GUIDE.md](./INTEGRATIONS-GUIDE.md) | integrations-specialist, tools-integration-specialist |
|
||||
| [DOCUMENTATION-GUIDE.md](./DOCUMENTATION-GUIDE.md) | documenter, describer-for-ai, planner, product-owner |
|
||||
|
||||
### Project Documentation
|
||||
|
||||
- [../adr/index.md](../adr/index.md) - Architecture Decision Records
|
||||
- [../TESTING.md](../TESTING.md) - Testing guide
|
||||
- [../development/TESTING.md](../development/TESTING.md) - Testing guide
|
||||
- [../development/CODE-PATTERNS.md](../development/CODE-PATTERNS.md) - Code patterns reference
|
||||
- [../architecture/OVERVIEW.md](../architecture/OVERVIEW.md) - System architecture
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
|
||||
@@ -6,6 +6,16 @@ This guide covers security and debugging-focused subagents:
|
||||
- **log-debug**: Production errors, observability, Bugsink/Sentry analysis
|
||||
- **code-reviewer**: Code quality, security review, best practices
|
||||
|
||||
## Quick Reference
|
||||
|
||||
| Aspect | security-engineer | log-debug | code-reviewer |
|
||||
| --------------- | ---------------------------------- | ---------------------------------------- | --------------------------- |
|
||||
| **Primary Use** | Security audits, OWASP | Production debugging | Code quality review |
|
||||
| **Key ADRs** | ADR-016 (Security), ADR-032 (Rate) | ADR-050 (Observability) | ADR-034, ADR-035 (Patterns) |
|
||||
| **MCP Tools** | N/A | `mcp__bugsink__*`, `mcp__localerrors__*` | N/A |
|
||||
| **Key Checks** | Auth, input validation, CORS | Logs, stacktraces, error patterns | Patterns, tests, security |
|
||||
| **Delegate To** | `coder` (fix issues) | `devops` (infra), `coder` (fixes) | `coder`, `testwriter` |
|
||||
|
||||
## The security-engineer Subagent
|
||||
|
||||
### When to Use
|
||||
@@ -432,8 +442,10 @@ tail -f /var/log/postgresql/postgresql-$(date +%Y-%m-%d).log | grep "duration:"
|
||||
|
||||
- [OVERVIEW.md](./OVERVIEW.md) - Subagent system overview
|
||||
- [DEVOPS-GUIDE.md](./DEVOPS-GUIDE.md) - Infrastructure debugging
|
||||
- [TESTER-GUIDE.md](./TESTER-GUIDE.md) - Security testing
|
||||
- [../adr/0016-api-security-hardening.md](../adr/0016-api-security-hardening.md) - Security ADR
|
||||
- [../adr/0032-rate-limiting-strategy.md](../adr/0032-rate-limiting-strategy.md) - Rate limiting
|
||||
- [../adr/0015-application-performance-monitoring-and-error-tracking.md](../adr/0015-application-performance-monitoring-and-error-tracking.md) - Monitoring ADR
|
||||
- [../adr/0015-error-tracking-and-observability.md](../adr/0015-error-tracking-and-observability.md) - Monitoring ADR
|
||||
- [../adr/0050-postgresql-function-observability.md](../adr/0050-postgresql-function-observability.md) - Database observability
|
||||
- [../BARE-METAL-SETUP.md](../BARE-METAL-SETUP.md) - Production setup
|
||||
- [../operations/BARE-METAL-SETUP.md](../operations/BARE-METAL-SETUP.md) - Production setup
|
||||
- [../tools/BUGSINK-SETUP.md](../tools/BUGSINK-SETUP.md) - Bugsink configuration
|
||||
|
||||
@@ -5,6 +5,17 @@ This guide covers two related but distinct subagents for testing in the Flyer Cr
|
||||
- **tester**: Adversarial testing to find edge cases, race conditions, and vulnerabilities
|
||||
- **testwriter**: Creating comprehensive test suites for features and fixes
|
||||
|
||||
## Quick Reference
|
||||
|
||||
| Aspect | tester | testwriter |
|
||||
| ---------------- | -------------------------------------------- | ------------------------------------------ |
|
||||
| **Primary Use** | Find bugs, security issues, edge cases | Create test suites, improve coverage |
|
||||
| **Key Files** | N/A (analysis-focused) | `*.test.ts`, `src/tests/utils/` |
|
||||
| **Key ADRs** | ADR-010 (Testing), ADR-040 (Test Economics) | ADR-010 (Testing), ADR-045 (Test Fixtures) |
|
||||
| **Test Command** | `podman exec -it flyer-crawler-dev npm test` | Same |
|
||||
| **Test Stack** | Vitest, Supertest, Testing Library | Same |
|
||||
| **Delegate To** | `testwriter` (write tests for findings) | `coder` (fix failing tests) |
|
||||
|
||||
## Understanding the Difference
|
||||
|
||||
| Aspect | tester | testwriter |
|
||||
@@ -399,6 +410,7 @@ A typical workflow for thorough testing:
|
||||
|
||||
- [OVERVIEW.md](./OVERVIEW.md) - Subagent system overview
|
||||
- [CODER-GUIDE.md](./CODER-GUIDE.md) - Working with the coder subagent
|
||||
- [../TESTING.md](../TESTING.md) - Testing guide
|
||||
- [SECURITY-DEBUG-GUIDE.md](./SECURITY-DEBUG-GUIDE.md) - Security testing and code review
|
||||
- [../development/TESTING.md](../development/TESTING.md) - Testing guide
|
||||
- [../adr/0010-testing-strategy-and-standards.md](../adr/0010-testing-strategy-and-standards.md) - Testing ADR
|
||||
- [../adr/0040-testing-economics-and-priorities.md](../adr/0040-testing-economics-and-priorities.md) - Testing priorities
|
||||
|
||||
4
package-lock.json
generated
4
package-lock.json
generated
@@ -1,12 +1,12 @@
|
||||
{
|
||||
"name": "flyer-crawler",
|
||||
"version": "0.12.20",
|
||||
"version": "0.12.21",
|
||||
"lockfileVersion": 3,
|
||||
"requires": true,
|
||||
"packages": {
|
||||
"": {
|
||||
"name": "flyer-crawler",
|
||||
"version": "0.12.20",
|
||||
"version": "0.12.21",
|
||||
"dependencies": {
|
||||
"@bull-board/api": "^6.14.2",
|
||||
"@bull-board/express": "^6.14.2",
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"name": "flyer-crawler",
|
||||
"private": true,
|
||||
"version": "0.12.20",
|
||||
"version": "0.12.21",
|
||||
"type": "module",
|
||||
"scripts": {
|
||||
"dev": "concurrently \"npm:start:dev\" \"vite\"",
|
||||
|
||||
@@ -185,10 +185,7 @@ describe('StoreCard', () => {
|
||||
|
||||
it('should show "No location data" when locations is undefined', () => {
|
||||
renderWithProviders(
|
||||
<StoreCard
|
||||
store={mockStoreUndefinedLocations as typeof mockStoreWithLogo}
|
||||
showLocations={true}
|
||||
/>,
|
||||
<StoreCard store={mockStoreUndefinedLocations as any} showLocations={true} />,
|
||||
);
|
||||
|
||||
expect(screen.getByText('No location data')).toBeInTheDocument();
|
||||
|
||||
@@ -14,7 +14,7 @@ vi.mock('../services/eventBus', () => ({
|
||||
|
||||
import { eventBus } from '../services/eventBus';
|
||||
|
||||
const mockEventBus = eventBus as {
|
||||
const mockEventBus = eventBus as unknown as {
|
||||
on: Mock;
|
||||
off: Mock;
|
||||
dispatch: Mock;
|
||||
@@ -206,7 +206,7 @@ describe('useEventBus', () => {
|
||||
name: string;
|
||||
}
|
||||
|
||||
const callback = vi.fn<[TestData?], void>();
|
||||
const callback = vi.fn();
|
||||
|
||||
renderHook(() => useEventBus<TestData>('typed-event', callback));
|
||||
|
||||
@@ -217,7 +217,7 @@ describe('useEventBus', () => {
|
||||
});
|
||||
|
||||
it('should handle callback with optional parameter', () => {
|
||||
const callback = vi.fn<[string?], void>();
|
||||
const callback = vi.fn();
|
||||
|
||||
renderHook(() => useEventBus<string>('optional-event', callback));
|
||||
|
||||
|
||||
@@ -17,6 +17,7 @@ vi.mock('driver.js', () => ({
|
||||
DriveStep: vi.fn(),
|
||||
}));
|
||||
|
||||
// @ts-expect-error - driver.js types are not fully compatible, but the mock works
|
||||
import { driver } from 'driver.js';
|
||||
|
||||
const mockDriver = driver as Mock;
|
||||
|
||||
Reference in New Issue
Block a user