flyer-crawler.projectium.com/CLAUDE.md

# Claude Code Project Instructions

## CRITICAL RULES (READ FIRST)

### Platform: Linux Only (ADR-014)

**ALL tests MUST run in dev container** - Windows results are unreliable.

| Test Result | Container | Windows | Status                   |
| ----------- | --------- | ------- | ------------------------ |
| Pass        | Fail      | =       | **BROKEN** (must fix)    |
| Fail        | Pass      | =       | **PASSING** (acceptable) |

```bash
# Always test in container
podman exec -it flyer-crawler-dev npm test
podman exec -it flyer-crawler-dev npm run type-check
```

### Database Schema Sync

**CRITICAL**: Keep these files synchronized:

- `sql/master_schema_rollup.sql` (test DB, complete reference)
- `sql/initial_schema.sql` (fresh install, identical to rollup)
- `sql/migrations/*.sql` (production ALTER TABLE statements)

Out-of-sync = test failures.

### Server Access: READ-ONLY (Production/Test Servers)

**CRITICAL**: The `claude-win10` user has **READ-ONLY** access to production and test servers.

| Capability             | Status                 |
| ---------------------- | ---------------------- |
| Root/sudo access       | NO                     |
| Write permissions      | NO                     |
| PM2 restart, systemctl | NO - User must execute |

**Server Operations Workflow**: Diagnose → User executes → Analyze → Fix (1-3 commands) → User executes → Verify

**Rules**:

- Provide diagnostic commands first, wait for user to report results
- Maximum 3 fix commands at a time (errors may cascade)
- Always verify after fixes complete

### Communication Style

Ask before assuming. Never assume:

- Steps completed / User knowledge / External services configured / Secrets created

---

## Session Startup Checklist

1. **Memory**: `mcp__memory__read_graph` - Recall project context, credentials, known issues
2. **Git**: `git log --oneline -10` - Recent changes
3. **Containers**: `mcp__podman__container_list` - Running state
4. **PM2 Status**: `podman exec flyer-crawler-dev pm2 status` - Process health (API, Worker, Vite)

---

## Quick Reference

### Essential Commands

| Command                                                      | Description           |
| ------------------------------------------------------------ | --------------------- |
| `podman exec -it flyer-crawler-dev npm test`                 | Run all tests         |
| `podman exec -it flyer-crawler-dev npm run test:unit`        | Unit tests only       |
| `podman exec -it flyer-crawler-dev npm run type-check`       | TypeScript check      |
| `podman exec -it flyer-crawler-dev npm run test:integration` | Integration tests     |
| `podman exec -it flyer-crawler-dev pm2 status`               | PM2 process status    |
| `podman exec -it flyer-crawler-dev pm2 logs`                 | View all PM2 logs     |
| `podman exec -it flyer-crawler-dev pm2 restart all`          | Restart all processes |

### Key Patterns (with file locations)

| Pattern            | ADR     | Implementation                                    | File                                  |
| ------------------ | ------- | ------------------------------------------------- | ------------------------------------- |
| Error Handling     | ADR-001 | `handleDbError()`, throw `NotFoundError`          | `src/services/db/errors.db.ts`        |
| Repository Methods | ADR-034 | `get*` (throws), `find*` (null), `list*` (array)  | `src/services/db/*.db.ts`             |
| API Responses      | ADR-028 | `sendSuccess()`, `sendPaginated()`, `sendError()` | `src/utils/apiResponse.ts`            |
| Transactions       | ADR-002 | `withTransaction(async (client) => {...})`        | `src/services/db/connection.db.ts`    |
| Feature Flags      | ADR-024 | `isFeatureEnabled()`, `useFeatureFlag()`          | `src/services/featureFlags.server.ts` |

### Key Files Quick Access

| Purpose           | File                                  |
| ----------------- | ------------------------------------- |
| Express app       | `server.ts`                           |
| Environment       | `src/config/env.ts`                   |
| Routes            | `src/routes/*.routes.ts`              |
| Repositories      | `src/services/db/*.db.ts`             |
| Workers           | `src/services/workers.server.ts`      |
| Queues            | `src/services/queues.server.ts`       |
| Feature Flags     | `src/services/featureFlags.server.ts` |
| PM2 Config (Dev)  | `ecosystem.dev.config.cjs`            |
| PM2 Config (Prod) | `ecosystem.config.cjs`                |

---

## Application Overview

**Flyer Crawler** - AI-powered grocery deal extraction and analysis platform.

**Data Flow**: Upload → AI extraction (Gemini) → PostgreSQL → Cache (Redis) → API → React display

**Architecture** (ADR-035):

```text
Routes → Services → Repositories → Database
         ↓
    External APIs (*.server.ts)
```

**Key Entities**: Flyers, FlyerItems, Stores, StoreLocations, Users, Watchlists, ShoppingLists, Recipes, Achievements

**Full Architecture**: See [docs/architecture/OVERVIEW.md](docs/architecture/OVERVIEW.md)

---

## Dev Container Architecture (ADR-014)

The dev container now matches production by using PM2 for process management.

### Process Management

| Component  | Production             | Dev Container             |
| ---------- | ---------------------- | ------------------------- |
| API Server | PM2 cluster mode       | PM2 fork mode + tsx watch |
| Worker     | PM2 process            | PM2 process + tsx watch   |
| Frontend   | Static files via NGINX | PM2 + Vite dev server     |
| Logs       | PM2 logs -> Logstash   | PM2 logs -> Logstash      |

**PM2 Processes in Dev Container**:

- `flyer-crawler-api-dev` - API server (port 3001)
- `flyer-crawler-worker-dev` - Background job worker
- `flyer-crawler-vite-dev` - Vite frontend dev server (port 5173)

### Log Aggregation (ADR-015)

All logs flow to Bugsink via Logstash with 3-project routing:

| Source            | Log Location                      | Bugsink Project    |
| ----------------- | --------------------------------- | ------------------ |
| Backend (Pino)    | `/var/log/pm2/api-*.log`          | Backend API (1)    |
| Worker (Pino)     | `/var/log/pm2/worker-*.log`       | Backend API (1)    |
| PostgreSQL        | `/var/log/postgresql/*.log`       | Backend API (1)    |
| Vite              | `/var/log/pm2/vite-*.log`         | Infrastructure (4) |
| Redis             | `/var/log/redis/redis-server.log` | Infrastructure (4) |
| NGINX             | `/var/log/nginx/*.log`            | Infrastructure (4) |
| Frontend (Sentry) | Browser -> nginx proxy            | Frontend (2)       |

**Bugsink Projects (Dev Container)**:

- Project 1: Backend API (Dev) - Application errors
- Project 2: Frontend (Dev) - Browser errors via nginx proxy
- Project 4: Infrastructure (Dev) - Redis, NGINX, Vite errors

**Key Files**:

- `ecosystem.dev.config.cjs` - PM2 development configuration
- `scripts/dev-entrypoint.sh` - Container startup script
- `docker/logstash/bugsink.conf` - Logstash pipeline configuration
- `docker/nginx/dev.conf` - NGINX config with Bugsink API proxy

**Full Dev Container Guide**: See [docs/development/DEV-CONTAINER.md](docs/development/DEV-CONTAINER.md)

---

## Common Workflows

### Adding a New API Endpoint

1. Add route in `src/routes/{domain}.routes.ts`
2. Use `validateRequest(schema)` middleware for input validation
3. Call service layer (never access DB directly from routes)
4. Return via `sendSuccess()` or `sendPaginated()`
5. Add tests in `*.routes.test.ts`

**Example Pattern**: See [docs/development/CODE-PATTERNS.md](docs/development/CODE-PATTERNS.md)

### Adding a New Database Operation

1. Add method to `src/services/db/{domain}.db.ts`
2. Follow naming: `get*` (throws), `find*` (returns null), `list*` (array)
3. Use `handleDbError()` for error handling
4. Accept optional `PoolClient` for transaction support
5. Add unit test

### Adding a Background Job

1. Define queue in `src/services/queues.server.ts`
2. Add worker in `src/services/workers.server.ts`
3. Call `queue.add()` from service layer

---

## Subagent Delegation Guide

**When to Delegate**: Complex work, specialized expertise, multi-domain tasks

### Decision Matrix

| Task Type             | Subagent                | Key Docs                                                          |
| --------------------- | ----------------------- | ----------------------------------------------------------------- |
| Write production code | coder                   | [CODER-GUIDE.md](docs/subagents/CODER-GUIDE.md)                   |
| Database changes      | db-dev                  | [DATABASE-GUIDE.md](docs/subagents/DATABASE-GUIDE.md)             |
| Create tests          | testwriter              | [TESTER-GUIDE.md](docs/subagents/TESTER-GUIDE.md)                 |
| Fix failing tests     | tester                  | [TESTER-GUIDE.md](docs/subagents/TESTER-GUIDE.md)                 |
| Container/deployment  | devops                  | [DEVOPS-GUIDE.md](docs/subagents/DEVOPS-GUIDE.md)                 |
| UI components         | frontend-specialist     | [FRONTEND-GUIDE.md](docs/subagents/FRONTEND-GUIDE.md)             |
| External APIs         | integrations-specialist | [INTEGRATIONS-GUIDE.md](docs/subagents/INTEGRATIONS-GUIDE.md)     |
| Security review       | security-engineer       | [SECURITY-DEBUG-GUIDE.md](docs/subagents/SECURITY-DEBUG-GUIDE.md) |
| Production errors     | log-debug               | [SECURITY-DEBUG-GUIDE.md](docs/subagents/SECURITY-DEBUG-GUIDE.md) |
| AI/Gemini issues      | ai-usage                | [AI-USAGE-GUIDE.md](docs/subagents/AI-USAGE-GUIDE.md)             |
| Planning features     | planner                 | [DOCUMENTATION-GUIDE.md](docs/subagents/DOCUMENTATION-GUIDE.md)   |

**All Subagents**: See [docs/subagents/OVERVIEW.md](docs/subagents/OVERVIEW.md)

**Launch Pattern**:

```text
Use Task tool with subagent_type: "coder", "db-dev", "tester", etc.
```

---

## Known Issues & Gotchas

### Integration Test Issues (Summary)

Common issues with solutions:

1. **Vitest globalSetup context isolation** - Mocks/spies don't share instances → Mark `.todo()` or use Redis-based flags
2. **Cleanup queue interference** - Worker processes jobs during tests → `cleanupQueue.drain()` and `.pause()`
3. **Cache staleness** - Direct SQL bypasses cache → `cacheService.invalidateFlyers()` after inserts
4. **Filename collisions** - Multer predictable names → Use `${Date.now()}-${Math.round(Math.random() * 1e9)}`
5. **Response format mismatches** - API format changes → Log response bodies, update assertions
6. **External service failures** - PM2/Redis unavailable → try/catch with graceful degradation
7. **TZ environment variable breaks async hooks** - `TZ=America/Los_Angeles` causes `RangeError: Invalid triggerAsyncId value: NaN` → Tests now explicitly set `TZ=` (empty) in package.json scripts

**Full Details**: See test issues section at end of this document or [docs/development/TESTING.md](docs/development/TESTING.md)

### Git Bash Path Conversion (Windows)

Git Bash auto-converts Unix paths, breaking container commands.

**Solutions**:

```bash
# Use sh -c with single quotes
podman exec container sh -c '/usr/local/bin/script.sh'

# Use MSYS_NO_PATHCONV=1
MSYS_NO_PATHCONV=1 podman exec container /path/to/script

# Use Windows paths for host files
podman cp "d:/path/file" container:/tmp/file
```

---

## Configuration & Environment

### Environment Variables

**See**: [docs/getting-started/ENVIRONMENT.md](docs/getting-started/ENVIRONMENT.md) for complete reference.

**Quick Overview**:

- **Production**: Gitea CI/CD secrets only (no `.env` file)
- **Test**: Gitea secrets + `.env.test` overrides
- **Dev**: `.env.local` file (overrides `compose.dev.yml`)

**Key Variables**: `DB_HOST`, `DB_USER`, `DB_PASSWORD`, `JWT_SECRET`, `VITE_GOOGLE_GENAI_API_KEY`, `REDIS_URL`

**Adding Variables**: Update `src/config/env.ts`, Gitea Secrets, workflows, `ecosystem.config.cjs`, `.env.example`

### MCP Servers

**See**: [docs/tools/MCP-CONFIGURATION.md](docs/tools/MCP-CONFIGURATION.md) for setup.

**Quick Overview**:

| Server                     | Purpose              | Config                 |
| -------------------------- | -------------------- | ---------------------- |
| gitea-projectium/torbonium | Gitea API            | Global `settings.json` |
| podman                     | Container management | Global `settings.json` |
| memory                     | Knowledge graph      | Global `settings.json` |
| redis                      | Cache access         | Global `settings.json` |
| bugsink                    | Prod error tracking  | Global `settings.json` |
| localerrors                | Dev Bugsink          | Project `.mcp.json`    |
| devdb                      | Dev PostgreSQL       | Project `.mcp.json`    |

**Note**: Localhost servers use project `.mcp.json` due to Windows/loader issues.

### Bugsink Error Tracking

**See**: [docs/tools/BUGSINK-SETUP.md](docs/tools/BUGSINK-SETUP.md) for setup.

**Quick Access**:

- **Dev**: <https://localhost:8443> (`admin@localhost`/`admin`)
- **Prod**: <https://bugsink.projectium.com>

**Token Creation** (required for MCP):

```bash
# Dev container
MSYS_NO_PATHCONV=1 podman exec -e DATABASE_URL=postgresql://bugsink:bugsink_dev_password@postgres:5432/bugsink -e SECRET_KEY=dev-bugsink-secret-key-minimum-50-characters-for-security flyer-crawler-dev sh -c 'cd /opt/bugsink/conf && DJANGO_SETTINGS_MODULE=bugsink_conf PYTHONPATH=/opt/bugsink/conf:/opt/bugsink/lib/python3.10/site-packages /opt/bugsink/bin/python -m django create_auth_token'

# Production (user executes on server)
cd /opt/bugsink && bugsink-manage create_auth_token
```

### Logstash

**See**: [docs/operations/LOGSTASH-QUICK-REF.md](docs/operations/LOGSTASH-QUICK-REF.md)

Log aggregation: PostgreSQL + PM2 + Redis + NGINX → Bugsink (ADR-015)

---

## Documentation Quick Links

| Topic               | Document                                                       |
| ------------------- | -------------------------------------------------------------- |
| **Getting Started** | [QUICKSTART.md](docs/getting-started/QUICKSTART.md)            |
| **Dev Container**   | [DEV-CONTAINER.md](docs/development/DEV-CONTAINER.md)          |
| **Architecture**    | [OVERVIEW.md](docs/architecture/OVERVIEW.md)                   |
| **Code Patterns**   | [CODE-PATTERNS.md](docs/development/CODE-PATTERNS.md)          |
| **Testing**         | [TESTING.md](docs/development/TESTING.md)                      |
| **Debugging**       | [DEBUGGING.md](docs/development/DEBUGGING.md)                  |
| **Database**        | [DATABASE.md](docs/architecture/DATABASE.md)                   |
| **Deployment**      | [DEPLOYMENT.md](docs/operations/DEPLOYMENT.md)                 |
| **Monitoring**      | [MONITORING.md](docs/operations/MONITORING.md)                 |
| **Logstash**        | [LOGSTASH-QUICK-REF.md](docs/operations/LOGSTASH-QUICK-REF.md) |
| **ADRs**            | [docs/adr/index.md](docs/adr/index.md)                         |
| **All Docs**        | [docs/README.md](docs/README.md)                               |