flyer-crawler.projectium.com/CLAUDE.md

Claude Code Project Instructions

CRITICAL RULES (READ FIRST)

Platform: Linux Only (ADR-014)

ALL tests MUST run in dev container - Windows results are unreliable.

Test Result	Container	Windows	Status
Pass	Fail	=	BROKEN (must fix)
Fail	Pass	=	PASSING (acceptable)

# Always test in container
podman exec -it flyer-crawler-dev npm test
podman exec -it flyer-crawler-dev npm run type-check

Database Schema Sync

CRITICAL: Keep these files synchronized:

• sql/master_schema_rollup.sql (test DB, complete reference)
• sql/initial_schema.sql (fresh install, identical to rollup)
• sql/migrations/*.sql (production ALTER TABLE statements)

Out-of-sync = test failures.

Server Access: READ-ONLY (Production/Test Servers)

CRITICAL: The claude-win10 user has READ-ONLY access to production and test servers.

Capability	Status
Root/sudo access	NO
Write permissions	NO
PM2 restart, systemctl	NO - User must execute

Server Operations Workflow: Diagnose → User executes → Analyze → Fix (1-3 commands) → User executes → Verify

Rules:

• Provide diagnostic commands first, wait for user to report results
• Maximum 3 fix commands at a time (errors may cascade)
• Always verify after fixes complete

PM2 Process Isolation (Production/Test Servers)

CRITICAL: Production and test environments share the same PM2 daemon on the server.

Environment	Processes	Config File
Production	flyer-crawler-api, flyer-crawler-worker, flyer-crawler-analytics-worker	ecosystem.config.cjs
Test	flyer-crawler-api-test, flyer-crawler-worker-test, flyer-crawler-analytics-worker-test	ecosystem-test.config.cjs
Development	flyer-crawler-api-dev, flyer-crawler-worker-dev, flyer-crawler-vite-dev	ecosystem.dev.config.cjs

Deployment Scripts MUST:

• ✅ Filter PM2 commands by exact process names or name patterns (e.g., endsWith('-test'))
• ❌ NEVER use pm2 stop all, pm2 delete all, or pm2 restart all
• ❌ NEVER delete/stop processes based solely on status without name filtering
• ✅ Always verify process names match the target environment before any operation

Examples:

# ✅ CORRECT - Production cleanup (filter by name)
pm2 stop flyer-crawler-api flyer-crawler-worker flyer-crawler-analytics-worker

# ✅ CORRECT - Test cleanup (filter by name pattern)
# Only delete test processes that are errored/stopped
list.forEach(p => {
  if ((p.pm2_env.status === 'errored' || p.pm2_env.status === 'stopped') &&
      p.name && p.name.endsWith('-test')) {
    exec('pm2 delete ' + p.pm2_env.pm_id);
  }
});

# ❌ WRONG - Affects all environments
pm2 stop all
pm2 delete all

# ❌ WRONG - No name filtering (could delete test processes during prod deploy)
if (p.pm2_env.status === 'errored') {
  exec('pm2 delete ' + p.pm2_env.pm_id);
}

Communication Style

Ask before assuming. Never assume:

• Steps completed / User knowledge / External services configured / Secrets created

---

Session Startup Checklist

1. Memory: mcp__memory__read_graph - Recall project context, credentials, known issues
2. Git: git log --oneline -10 - Recent changes
3. Containers: mcp__podman__container_list - Running state
4. PM2 Status: podman exec flyer-crawler-dev pm2 status - Process health (API, Worker, Vite)

---

Quick Reference

Essential Commands

Command	Description
podman exec -it flyer-crawler-dev npm test	Run all tests
podman exec -it flyer-crawler-dev npm run test:unit	Unit tests only
podman exec -it flyer-crawler-dev npm run type-check	TypeScript check
podman exec -it flyer-crawler-dev npm run test:integration	Integration tests
podman exec -it flyer-crawler-dev pm2 status	PM2 process status
podman exec -it flyer-crawler-dev pm2 logs	View all PM2 logs
podman exec -it flyer-crawler-dev pm2 restart all	Restart all processes

Key Patterns (with file locations)

Pattern	ADR	Implementation	File
Error Handling	ADR-001	handleDbError(), throw NotFoundError	src/services/db/errors.db.ts
Repository Methods	ADR-034	get* (throws), find* (null), list* (array)	src/services/db/*.db.ts
API Responses	ADR-028	sendSuccess(), sendPaginated(), sendError()	src/utils/apiResponse.ts
Transactions	ADR-002	withTransaction(async (client) => {...})	src/services/db/connection.db.ts
Feature Flags	ADR-024	isFeatureEnabled(), useFeatureFlag()	src/services/featureFlags.server.ts

Key Files Quick Access

Purpose	File
Express app	server.ts
Environment	src/config/env.ts
Routes	src/routes/*.routes.ts
Repositories	src/services/db/*.db.ts
Workers	src/services/workers.server.ts
Queues	src/services/queues.server.ts
Feature Flags	src/services/featureFlags.server.ts
PM2 Config (Dev)	ecosystem.dev.config.cjs
PM2 Config (Prod)	ecosystem.config.cjs

---

Application Overview

Flyer Crawler - AI-powered grocery deal extraction and analysis platform.

Data Flow: Upload → AI extraction (Gemini) → PostgreSQL → Cache (Redis) → API → React display

Architecture (ADR-035):

Routes → Services → Repositories → Database
         ↓
    External APIs (*.server.ts)

Key Entities: Flyers, FlyerItems, Stores, StoreLocations, Users, Watchlists, ShoppingLists, Recipes, Achievements

Full Architecture: See docs/architecture/OVERVIEW.md

---

Dev Container Architecture (ADR-014)

The dev container now matches production by using PM2 for process management.

Process Management

Component	Production	Dev Container
API Server	PM2 cluster mode	PM2 fork mode + tsx watch
Worker	PM2 process	PM2 process + tsx watch
Frontend	Static files via NGINX	PM2 + Vite dev server
Logs	PM2 logs -> Logstash	PM2 logs -> Logstash

PM2 Processes in Dev Container:

• flyer-crawler-api-dev - API server (port 3001)
• flyer-crawler-worker-dev - Background job worker
• flyer-crawler-vite-dev - Vite frontend dev server (port 5173)

Log Aggregation (ADR-015)

All logs flow to Bugsink via Logstash with 3-project routing:

Source	Log Location	Bugsink Project
Backend (Pino)	/var/log/pm2/api-*.log	Backend API (1)
Worker (Pino)	/var/log/pm2/worker-*.log	Backend API (1)
PostgreSQL	/var/log/postgresql/*.log	Backend API (1)
Vite	/var/log/pm2/vite-*.log	Infrastructure (4)
Redis	/var/log/redis/redis-server.log	Infrastructure (4)
NGINX	/var/log/nginx/*.log	Infrastructure (4)
Frontend (Sentry)	Browser -> nginx proxy	Frontend (2)

Bugsink Projects (Dev Container):

• Project 1: Backend API (Dev) - Application errors
• Project 2: Frontend (Dev) - Browser errors via nginx proxy
• Project 4: Infrastructure (Dev) - Redis, NGINX, Vite errors

Key Files:

• ecosystem.dev.config.cjs - PM2 development configuration
• scripts/dev-entrypoint.sh - Container startup script
• docker/logstash/bugsink.conf - Logstash pipeline configuration
• docker/nginx/dev.conf - NGINX config with Bugsink API proxy

Full Dev Container Guide: See docs/development/DEV-CONTAINER.md

---

Common Workflows

Adding a New API Endpoint

1. Add route in src/routes/{domain}.routes.ts
2. Use validateRequest(schema) middleware for input validation
3. Call service layer (never access DB directly from routes)
4. Return via sendSuccess() or sendPaginated()
5. Add tests in *.routes.test.ts

Example Pattern: See docs/development/CODE-PATTERNS.md

Adding a New Database Operation

1. Add method to src/services/db/{domain}.db.ts
2. Follow naming: get* (throws), find* (returns null), list* (array)
3. Use handleDbError() for error handling
4. Accept optional PoolClient for transaction support
5. Add unit test

Adding a Background Job

1. Define queue in src/services/queues.server.ts
2. Add worker in src/services/workers.server.ts
3. Call queue.add() from service layer

---

Subagent Delegation Guide

When to Delegate: Complex work, specialized expertise, multi-domain tasks

Decision Matrix

Task Type	Subagent	Key Docs
Write production code	coder	CODER-GUIDE.md
Database changes	db-dev	DATABASE-GUIDE.md
Create tests	testwriter	TESTER-GUIDE.md
Fix failing tests	tester	TESTER-GUIDE.md
Container/deployment	devops	DEVOPS-GUIDE.md
UI components	frontend-specialist	FRONTEND-GUIDE.md
External APIs	integrations-specialist	INTEGRATIONS-GUIDE.md
Security review	security-engineer	SECURITY-DEBUG-GUIDE.md
Production errors	log-debug	SECURITY-DEBUG-GUIDE.md
AI/Gemini issues	ai-usage	AI-USAGE-GUIDE.md
Planning features	planner	DOCUMENTATION-GUIDE.md

All Subagents: See docs/subagents/OVERVIEW.md

Launch Pattern:

Use Task tool with subagent_type: "coder", "db-dev", "tester", etc.

---

Known Issues & Gotchas

Integration Test Issues (Summary)

Common issues with solutions:

1. Vitest globalSetup context isolation - Mocks/spies don't share instances → Mark .todo() or use Redis-based flags
2. Cleanup queue interference - Worker processes jobs during tests → cleanupQueue.drain() and .pause()
3. Cache staleness - Direct SQL bypasses cache → cacheService.invalidateFlyers() after inserts
4. Filename collisions - Multer predictable names → Use ${Date.now()}-${Math.round(Math.random() * 1e9)}
5. Response format mismatches - API format changes → Log response bodies, update assertions
6. External service failures - PM2/Redis unavailable → try/catch with graceful degradation
7. TZ environment variable breaks async hooks - TZ=America/Los_Angeles causes RangeError: Invalid triggerAsyncId value: NaN → Tests now explicitly set TZ= (empty) in package.json scripts

Full Details: See test issues section at end of this document or docs/development/TESTING.md

Git Bash Path Conversion (Windows)

Git Bash auto-converts Unix paths, breaking container commands.

Solutions:

# Use sh -c with single quotes
podman exec container sh -c '/usr/local/bin/script.sh'

# Use MSYS_NO_PATHCONV=1
MSYS_NO_PATHCONV=1 podman exec container /path/to/script

# Use Windows paths for host files
podman cp "d:/path/file" container:/tmp/file

---

Configuration & Environment

Environment Variables

See: docs/getting-started/ENVIRONMENT.md for complete reference.

Quick Overview:

• Production: Gitea CI/CD secrets only (no .env file)
• Test: Gitea secrets + .env.test overrides
• Dev: .env.local file (overrides compose.dev.yml)

Key Variables: DB_HOST, DB_USER, DB_PASSWORD, JWT_SECRET, VITE_GOOGLE_GENAI_API_KEY, REDIS_URL

Adding Variables: Update src/config/env.ts, Gitea Secrets, workflows, ecosystem.config.cjs, .env.example

MCP Servers

See: docs/tools/MCP-CONFIGURATION.md for setup.

Quick Overview:

Server	Purpose	Config
gitea-projectium/torbonium	Gitea API	Global settings.json
podman	Container management	Global settings.json
memory	Knowledge graph	Global settings.json
redis	Cache access	Global settings.json
bugsink	Prod error tracking	Global settings.json
localerrors	Dev Bugsink	Project .mcp.json
devdb	Dev PostgreSQL	Project .mcp.json

Note: Localhost servers use project .mcp.json due to Windows/loader issues.

Bugsink Error Tracking

See: docs/tools/BUGSINK-SETUP.md for setup.

Quick Access:

• Dev: https://localhost:8443 (admin@localhost/admin)
• Prod: https://bugsink.projectium.com

Token Creation (required for MCP):

# Dev container
MSYS_NO_PATHCONV=1 podman exec -e DATABASE_URL=postgresql://bugsink:bugsink_dev_password@postgres:5432/bugsink -e SECRET_KEY=dev-bugsink-secret-key-minimum-50-characters-for-security flyer-crawler-dev sh -c 'cd /opt/bugsink/conf && DJANGO_SETTINGS_MODULE=bugsink_conf PYTHONPATH=/opt/bugsink/conf:/opt/bugsink/lib/python3.10/site-packages /opt/bugsink/bin/python -m django create_auth_token'

# Production (user executes on server)
cd /opt/bugsink && bugsink-manage create_auth_token

Logstash

See: docs/operations/LOGSTASH-QUICK-REF.md

Log aggregation: PostgreSQL + PM2 + Redis + NGINX → Bugsink (ADR-015)

---

Documentation Quick Links

Topic	Document
Getting Started	QUICKSTART.md
Dev Container	DEV-CONTAINER.md
Architecture	OVERVIEW.md
Code Patterns	CODE-PATTERNS.md
Testing	TESTING.md
Debugging	DEBUGGING.md
Database	DATABASE.md
Deployment	DEPLOYMENT.md
Monitoring	MONITORING.md
Logstash	LOGSTASH-QUICK-REF.md
ADRs	docs/adr/index.md
All Docs	docs/README.md