torbo/flyer-crawler.projectium.com
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
209d7ceba1a3cde54ef8e3afebf548cdd9d8d289
flyer-crawler.projectium.com/CLAUDE.md
Gitea Actions 645c1784b7 style: auto-format code via Prettier [skip ci]
2026-02-20 03:54:16 +05:00

22 KiB
Raw Blame History

Claude Code Project Instructions

CRITICAL RULES (READ FIRST)

Platform: Linux Only (ADR-014)

ALL tests MUST run in dev container - Windows results are unreliable.

Test Result Container Windows Status
Pass Fail = BROKEN (must fix)
Fail Pass = PASSING (acceptable) 
# Always test in container
podman exec -it flyer-crawler-dev npm test
podman exec -it flyer-crawler-dev npm run type-check

Database Schema Sync

CRITICAL: Keep these files synchronized:

  • sql/master_schema_rollup.sql (test DB, complete reference)
  • sql/initial_schema.sql (fresh install, identical to rollup)
  • sql/migrations/*.sql (production ALTER TABLE statements)

Out-of-sync = test failures.

Server Access: READ-ONLY (Production/Test Servers)

CRITICAL: The claude-win10 user has READ-ONLY access to production and test servers.

Capability Status
Root/sudo access NO
Write permissions NO
PM2 restart, systemctl NO - User must execute

Server Operations Workflow: Diagnose → User executes → Analyze → Fix (1-3 commands) → User executes → Verify

Rules:

  • Provide diagnostic commands first, wait for user to report results
  • Maximum 3 fix commands at a time (errors may cascade)
  • Always verify after fixes complete

PM2 Namespace Isolation (Production/Test Servers)

CRITICAL: Production and test environments share the same PM2 daemon on the server.

Flyer-crawler uses PM2 namespaces to isolate test and production processes:

Namespace Purpose Config File
flyer-crawler-prod Production environment ecosystem.config.cjs
flyer-crawler-test Test environment ecosystem-test.config.cjs
flyer-crawler-dev Development container ecosystem.dev.config.cjs

This prevents pm2 save race conditions during simultaneous deployments. See ADR-063 for details.

See also: PM2 Process Isolation Incidents for past incidents and response procedures.

Environment Processes Namespace
Production flyer-crawler-api, flyer-crawler-worker, flyer-crawler-analytics-worker flyer-crawler-prod
Test flyer-crawler-api-test, flyer-crawler-worker-test, flyer-crawler-analytics-worker-test flyer-crawler-test
Development flyer-crawler-api-dev, flyer-crawler-worker-dev, flyer-crawler-vite-dev flyer-crawler-dev

Deployment Scripts MUST:

  • ✅ Use --namespace flag for all PM2 commands to scope to correct environment
  • ✅ Filter PM2 commands by exact process names or name patterns (e.g., endsWith('-test'))
  • ❌ NEVER use pm2 stop all, pm2 delete all, or pm2 restart all without namespace
  • ❌ NEVER delete/stop processes based solely on status without name filtering
  • ✅ Always verify process names match the target environment before any operation

Examples:

# ✅ CORRECT - Production commands with namespace
pm2 start ecosystem.config.cjs --namespace flyer-crawler-prod
pm2 stop flyer-crawler-api flyer-crawler-worker --namespace flyer-crawler-prod
pm2 restart all --namespace flyer-crawler-prod && pm2 save --namespace flyer-crawler-prod
pm2 logs --namespace flyer-crawler-prod

# ✅ CORRECT - Test commands with namespace
pm2 start ecosystem-test.config.cjs --namespace flyer-crawler-test
pm2 status --namespace flyer-crawler-test
pm2 delete all --namespace flyer-crawler-test && pm2 save --namespace flyer-crawler-test

# ✅ CORRECT - Dev container commands with namespace
pm2 start ecosystem.dev.config.cjs --namespace flyer-crawler-dev
pm2 logs --namespace flyer-crawler-dev

# ✅ CORRECT - Test cleanup (filter by namespace + name pattern)
# Only delete test processes that are errored/stopped
list.forEach(p => {
  if ((p.pm2_env.status === 'errored' || p.pm2_env.status === 'stopped') &&
      p.name && p.name.endsWith('-test') &&
      p.pm2_env.namespace === 'flyer-crawler-test') {
    exec('pm2 delete ' + p.pm2_env.pm_id + ' --namespace flyer-crawler-test');
  }
});
exec('pm2 save --namespace flyer-crawler-test');

# ❌ WRONG - Missing namespace (affects all environments)
pm2 stop all
pm2 delete all
pm2 restart all

# ❌ WRONG - No name/namespace filtering (could delete test processes during prod deploy)
if (p.pm2_env.status === 'errored') {
  exec('pm2 delete ' + p.pm2_env.pm_id);
}

PM2 Save Requirement (CRITICAL)

CRITICAL: Every pm2 start, pm2 restart, pm2 stop, or pm2 delete command MUST be immediately followed by pm2 save with the same namespace.

Without pm2 save, processes become ephemeral and will disappear on:

  • PM2 daemon restarts
  • Server reboots
  • Internal PM2 reconciliation events

Pattern:

# ✅ CORRECT - Save after every state change (with namespace)
pm2 start ecosystem.config.cjs --namespace flyer-crawler-prod && pm2 save --namespace flyer-crawler-prod
pm2 restart my-app --namespace flyer-crawler-prod && pm2 save --namespace flyer-crawler-prod
pm2 stop my-app --namespace flyer-crawler-test && pm2 save --namespace flyer-crawler-test
pm2 delete my-app --namespace flyer-crawler-test && pm2 save --namespace flyer-crawler-test

# ❌ WRONG - Missing save (processes become ephemeral)
pm2 start ecosystem.config.cjs --namespace flyer-crawler-prod
pm2 restart my-app --namespace flyer-crawler-prod

# ❌ WRONG - Missing namespace (affects wrong environment)
pm2 start ecosystem.config.cjs && pm2 save

In Cleanup Scripts:

// ✅ CORRECT - Save after cleanup loop completes (with namespace)
const namespace = 'flyer-crawler-test';
targetProcesses.forEach((p) => {
  exec(`pm2 delete ${p.pm2_env.pm_id} --namespace ${namespace}`);
});
exec(`pm2 save --namespace ${namespace}`); // Persist all deletions

// ❌ WRONG - Missing save and namespace
targetProcesses.forEach((p) => {
  exec('pm2 delete ' + p.pm2_env.pm_id);
});

Why This Matters:

PM2 maintains an in-memory process list. The pm2 save command writes this list to ~/.pm2/dump.pm2, which PM2 uses to resurrect processes after daemon restarts. Without it, your carefully managed process state is lost. Using namespaces ensures that pm2 save in one environment does not affect another.

See Also:

Communication Style

Ask before assuming. Never assume:

  • Steps completed / User knowledge / External services configured / Secrets created

Session Startup Checklist

  1. Memory: mcp__memory__read_graph - Recall project context, credentials, known issues
  2. Git: git log --oneline -10 - Recent changes
  3. Containers: mcp__podman__container_list - Running state
  4. PM2 Status: podman exec flyer-crawler-dev pm2 status --namespace flyer-crawler-dev - Process health (API, Worker, Vite)

Quick Reference

Essential Commands

Command Description
podman exec -it flyer-crawler-dev npm test Run all tests
podman exec -it flyer-crawler-dev npm run test:unit Unit tests only
podman exec -it flyer-crawler-dev npm run type-check TypeScript check
podman exec -it flyer-crawler-dev npm run test:integration Integration tests
podman exec -it flyer-crawler-dev pm2 status --namespace flyer-crawler-dev PM2 process status (dev)
podman exec -it flyer-crawler-dev pm2 logs --namespace flyer-crawler-dev View PM2 logs (dev)
podman exec -it flyer-crawler-dev pm2 restart all --namespace flyer-crawler-dev Restart all (dev)
pm2 status --namespace flyer-crawler-prod PM2 status (production)
pm2 status --namespace flyer-crawler-test PM2 status (test)

Key Patterns (with file locations)

Pattern ADR Implementation File
Error Handling ADR-001 handleDbError(), throw NotFoundError src/services/db/errors.db.ts
Repository Methods ADR-034 get* (throws), find* (null), list* (array) src/services/db/*.db.ts
API Responses ADR-028 sendSuccess(), sendPaginated(), sendError() src/utils/apiResponse.ts
Transactions ADR-002 withTransaction(async (client) => {...}) src/services/db/connection.db.ts
Feature Flags ADR-024 isFeatureEnabled(), useFeatureFlag() src/services/featureFlags.server.ts

Key Files Quick Access

Purpose File
Express app server.ts
Environment src/config/env.ts
Routes src/routes/*.routes.ts
Repositories src/services/db/*.db.ts
Workers src/services/workers.server.ts
Queues src/services/queues.server.ts
Feature Flags src/services/featureFlags.server.ts
PM2 Config (Dev) ecosystem.dev.config.cjs
PM2 Config (Prod) ecosystem.config.cjs

Application Overview

Flyer Crawler - AI-powered grocery deal extraction and analysis platform.

Data Flow: Upload → AI extraction (Gemini) → PostgreSQL → Cache (Redis) → API → React display

Architecture (ADR-035):

Routes → Services → Repositories → Database
         ↓
    External APIs (*.server.ts)

Key Entities: Flyers, FlyerItems, Stores, StoreLocations, Users, Watchlists, ShoppingLists, Recipes, Achievements

Full Architecture: See docs/architecture/OVERVIEW.md

Dev Container Architecture (ADR-014)

The dev container now matches production by using PM2 for process management.

Process Management

Component Production Dev Container
API Server PM2 cluster mode PM2 fork mode + tsx watch
Worker PM2 process PM2 process + tsx watch
Frontend Static files via NGINX PM2 + Vite dev server
Logs PM2 logs -> Logstash PM2 logs -> Logstash

PM2 Processes in Dev Container:

  • flyer-crawler-api-dev - API server (port 3001)
  • flyer-crawler-worker-dev - Background job worker
  • flyer-crawler-vite-dev - Vite frontend dev server (port 5173)

Log Aggregation (ADR-015)

All logs flow to Bugsink via Logstash with 3-project routing:

Source Log Location Bugsink Project
Backend (Pino) /var/log/pm2/api-*.log Backend API (1)
Worker (Pino) /var/log/pm2/worker-*.log Backend API (1)
PostgreSQL /var/log/postgresql/*.log Backend API (1)
Vite /var/log/pm2/vite-*.log Infrastructure (4)
Redis /var/log/redis/redis-server.log Infrastructure (4)
NGINX /var/log/nginx/*.log Infrastructure (4)
Frontend (Sentry) Browser -> nginx proxy Frontend (2)

Bugsink Projects (Dev Container):

  • Project 1: Backend API (Dev) - Application errors
  • Project 2: Frontend (Dev) - Browser errors via nginx proxy
  • Project 4: Infrastructure (Dev) - Redis, NGINX, Vite errors

Key Files:

  • ecosystem.dev.config.cjs - PM2 development configuration
  • scripts/dev-entrypoint.sh - Container startup script
  • docker/logstash/bugsink.conf - Logstash pipeline configuration
  • docker/nginx/dev.conf - NGINX config with Bugsink API proxy

Full Dev Container Guide: See docs/development/DEV-CONTAINER.md

Common Workflows

Adding a New API Endpoint

  1. Add route in src/routes/{domain}.routes.ts
  2. Use validateRequest(schema) middleware for input validation
  3. Call service layer (never access DB directly from routes)
  4. Return via sendSuccess() or sendPaginated()
  5. Add tests in *.routes.test.ts

Example Pattern: See docs/development/CODE-PATTERNS.md

Adding a New Database Operation

  1. Add method to src/services/db/{domain}.db.ts
  2. Follow naming: get* (throws), find* (returns null), list* (array)
  3. Use handleDbError() for error handling
  4. Accept optional PoolClient for transaction support
  5. Add unit test

Adding a Background Job

  1. Define queue in src/services/queues.server.ts
  2. Add worker in src/services/workers.server.ts
  3. Call queue.add() from service layer

Subagent Delegation Guide

When to Delegate: Complex work, specialized expertise, multi-domain tasks

Decision Matrix

Task Type Subagent Key Docs
Write production code coder CODER-GUIDE.md
Database changes db-dev DATABASE-GUIDE.md
Create tests testwriter TESTER-GUIDE.md
Fix failing tests tester TESTER-GUIDE.md
Container/deployment devops DEVOPS-GUIDE.md
UI components frontend-specialist FRONTEND-GUIDE.md
External APIs integrations-specialist INTEGRATIONS-GUIDE.md
Security review security-engineer SECURITY-DEBUG-GUIDE.md
Production errors log-debug SECURITY-DEBUG-GUIDE.md
AI/Gemini issues ai-usage AI-USAGE-GUIDE.md
Planning features planner DOCUMENTATION-GUIDE.md

All Subagents: See docs/subagents/OVERVIEW.md

Launch Pattern:

Use Task tool with subagent_type: "coder", "db-dev", "tester", etc.

Known Issues & Gotchas

Integration Test Issues (Summary)

Common issues with solutions:

  1. Vitest globalSetup context isolation - Mocks/spies don't share instances → Mark .todo() or use Redis-based flags
  2. Cleanup queue interference - Worker processes jobs during tests → cleanupQueue.drain() and .pause()
  3. Cache staleness - Direct SQL bypasses cache → cacheService.invalidateFlyers() after inserts
  4. Filename collisions - Multer predictable names → Use ${Date.now()}-${Math.round(Math.random() * 1e9)}
  5. Response format mismatches - API format changes → Log response bodies, update assertions
  6. External service failures - PM2/Redis unavailable → try/catch with graceful degradation
  7. TZ environment variable breaks async hooks - TZ=America/Los_Angeles causes RangeError: Invalid triggerAsyncId value: NaN → Tests now explicitly set TZ= (empty) in package.json scripts

Full Details: See test issues section at end of this document or docs/development/TESTING.md

PM2 Process Isolation Incidents

CRITICAL: PM2 process cleanup scripts can affect all PM2 processes if not properly filtered.

Incident: 2026-02-17 Production Deployment (v0.15.0)

  • Impact: ALL PM2 processes on production server were killed
  • Affected: stock-alert.projectium.com and all other PM2-managed applications
  • Root Cause: Under investigation (see incident report)
  • Status: Safeguards added to prevent recurrence

Prevention Measures (implemented):

  1. Name-based filtering (exact match or pattern-based)
  2. Pre-cleanup process list logging
  3. Process count validation (abort if filtering all processes)
  4. Explicit name verification in logs
  5. Post-cleanup verification
  6. Workflow version hash logging

If PM2 Incident Occurs:

  • DO NOT attempt another deployment immediately
  • Follow the PM2 Incident Response Runbook
  • Manually restore affected processes
  • Investigate workflow execution logs before next deployment

Related Documentation:

Git Bash Path Conversion (Windows)

Git Bash auto-converts Unix paths, breaking container commands.

Solutions:

# Use sh -c with single quotes
podman exec container sh -c '/usr/local/bin/script.sh'

# Use MSYS_NO_PATHCONV=1
MSYS_NO_PATHCONV=1 podman exec container /path/to/script

# Use Windows paths for host files
podman cp "d:/path/file" container:/tmp/file

Configuration & Environment

Environment Variables

See: docs/getting-started/ENVIRONMENT.md for complete reference.

Quick Overview:

  • Production: Gitea CI/CD secrets only (no .env file)
  • Test: Gitea secrets + .env.test overrides
  • Dev: .env.local file (overrides compose.dev.yml)

Key Variables: DB_HOST, DB_USER, DB_PASSWORD, JWT_SECRET, VITE_GOOGLE_GENAI_API_KEY, REDIS_URL

Adding Variables: Update src/config/env.ts, Gitea Secrets, workflows, ecosystem.config.cjs, .env.example

MCP Servers

See: docs/tools/MCP-CONFIGURATION.md for setup.

Quick Overview:

Server Purpose Config
gitea-projectium/torbonium Gitea API Global settings.json
podman Container management Global settings.json
memory Knowledge graph Global settings.json
redis Cache access Global settings.json
bugsink Prod error tracking Global settings.json
localerrors Dev Bugsink Project .mcp.json
devdb Dev PostgreSQL Project .mcp.json

Note: Localhost servers use project .mcp.json due to Windows/loader issues.

Bugsink Error Tracking

See: docs/tools/BUGSINK-SETUP.md for setup.

Quick Access:

Token Creation (required for MCP):

# Dev container
MSYS_NO_PATHCONV=1 podman exec -e DATABASE_URL=postgresql://bugsink:bugsink_dev_password@postgres:5432/bugsink -e SECRET_KEY=dev-bugsink-secret-key-minimum-50-characters-for-security flyer-crawler-dev sh -c 'cd /opt/bugsink/conf && DJANGO_SETTINGS_MODULE=bugsink_conf PYTHONPATH=/opt/bugsink/conf:/opt/bugsink/lib/python3.10/site-packages /opt/bugsink/bin/python -m django create_auth_token'

# Production (user executes on server)
cd /opt/bugsink && bugsink-manage create_auth_token

Logstash

See: docs/operations/LOGSTASH-QUICK-REF.md

Log aggregation: PostgreSQL + PM2 + Redis + NGINX → Bugsink (ADR-015)

Documentation Quick Links

Topic Document
Getting Started QUICKSTART.md
Dev Container DEV-CONTAINER.md
Architecture OVERVIEW.md
Code Patterns CODE-PATTERNS.md
Testing TESTING.md
Debugging DEBUGGING.md
Database DATABASE.md
Deployment DEPLOYMENT.md
Monitoring MONITORING.md
Logstash LOGSTASH-QUICK-REF.md
ADRs docs/adr/index.md
All Docs docs/README.md
Reference in New Issue View Git Blame Copy Permalink