doc updates and test fixin

2026-01-22 11:17:06 -08:00
parent 9f7b821760
commit fac98f4c54
56 changed files with 11967 additions and 357 deletions
--- a/docs/development/DEBUGGING.md
+++ b/docs/development/DEBUGGING.md
@@ -0,0 +1,668 @@
+# Debugging Guide
+
+Common debugging strategies and troubleshooting patterns for Flyer Crawler.
+
+## Table of Contents
+
+- [Quick Debugging Checklist](#quick-debugging-checklist)
+- [Container Issues](#container-issues)
+- [Database Issues](#database-issues)
+- [Test Failures](#test-failures)
+- [API Errors](#api-errors)
+- [Authentication Problems](#authentication-problems)
+- [Background Job Issues](#background-job-issues)
+- [Frontend Issues](#frontend-issues)
+- [Performance Problems](#performance-problems)
+- [Debugging Tools](#debugging-tools)
+
+---
+
+## Quick Debugging Checklist
+
+When something breaks, check these first:
+
+1. **Are containers running?**
+
+   ```bash
+   podman ps
+   ```
+
+2. **Is the database accessible?**
+
+   ```bash
+   podman exec flyer-crawler-postgres pg_isready -U postgres
+   ```
+
+3. **Are environment variables set?**
+
+   ```bash
+   # Check .env.local exists
+   cat .env.local
+   ```
+
+4. **Are there recent errors in logs?**
+
+   ```bash
+   # Application logs
+   podman logs -f flyer-crawler-dev
+
+   # PM2 logs (production)
+   pm2 logs flyer-crawler-api
+   ```
+
+5. **Is Redis accessible?**
+   ```bash
+   podman exec flyer-crawler-redis redis-cli ping
+   ```
+
+---
+
+## Container Issues
+
+### Container Won't Start
+
+**Symptom**: `podman start` fails or container exits immediately
+
+**Debug**:
+
+```bash
+# Check container status
+podman ps -a
+
+# View container logs
+podman logs flyer-crawler-postgres
+podman logs flyer-crawler-redis
+podman logs flyer-crawler-dev
+
+# Inspect container
+podman inspect flyer-crawler-dev
+```
+
+**Common Causes**:
+
+- Port already in use
+- Insufficient resources
+- Configuration error
+
+**Solutions**:
+
+```bash
+# Check port usage
+netstat -an | findstr "5432"
+netstat -an | findstr "6379"
+
+# Remove and recreate container
+podman stop flyer-crawler-postgres
+podman rm flyer-crawler-postgres
+# ... recreate with podman run ...
+```
+
+### "Unable to connect to Podman socket"
+
+**Symptom**: `Error: unable to connect to Podman socket`
+
+**Solution**:
+
+```bash
+# Start Podman machine
+podman machine start
+
+# Verify it's running
+podman machine list
+```
+
+### Port Already in Use
+
+**Symptom**: `Error: port 5432 is already allocated`
+
+**Solutions**:
+
+**Option 1**: Stop conflicting service
+
+```bash
+# Find process using port
+netstat -ano | findstr "5432"
+
+# Stop the service or kill process
+```
+
+**Option 2**: Use different port
+
+```bash
+# Run container on different host port
+podman run -d --name flyer-crawler-postgres -p 5433:5432 ...
+
+# Update .env.local
+DB_PORT=5433
+```
+
+---
+
+## Database Issues
+
+### Connection Refused
+
+**Symptom**: `Error: connect ECONNREFUSED 127.0.0.1:5432`
+
+**Debug**:
+
+```bash
+# 1. Check if PostgreSQL container is running
+podman ps | grep postgres
+
+# 2. Check if PostgreSQL is ready
+podman exec flyer-crawler-postgres pg_isready -U postgres
+
+# 3. Test connection
+podman exec flyer-crawler-postgres psql -U postgres -d flyer_crawler_dev -c "SELECT 1;"
+```
+
+**Common Causes**:
+
+- Container not running
+- PostgreSQL still initializing
+- Wrong credentials in `.env.local`
+
+**Solutions**:
+
+```bash
+# Start container
+podman start flyer-crawler-postgres
+
+# Wait for initialization (check logs)
+podman logs -f flyer-crawler-postgres
+
+# Verify credentials match .env.local
+cat .env.local | grep DB_
+```
+
+### Schema Out of Sync
+
+**Symptom**: Tests fail with missing column or table errors
+
+**Cause**: `master_schema_rollup.sql` not in sync with migrations
+
+**Solution**:
+
+```bash
+# Reset database with current schema
+podman exec -i flyer-crawler-postgres psql -U postgres -d flyer_crawler_dev < sql/drop_tables.sql
+podman exec -i flyer-crawler-postgres psql -U postgres -d flyer_crawler_dev < sql/master_schema_rollup.sql
+
+# Verify schema
+podman exec flyer-crawler-postgres psql -U postgres -d flyer_crawler_dev -c "\dt"
+```
+
+### Query Performance Issues
+
+**Debug**:
+
+```sql
+-- Enable query logging
+ALTER DATABASE flyer_crawler_dev SET log_statement = 'all';
+
+-- Check slow queries
+SELECT query, mean_exec_time, calls
+FROM pg_stat_statements
+WHERE mean_exec_time > 100
+ORDER BY mean_exec_time DESC
+LIMIT 10;
+
+-- Analyze query plan
+EXPLAIN ANALYZE
+SELECT * FROM flyers WHERE store_id = 1;
+```
+
+**Solutions**:
+
+- Add missing indexes
+- Optimize WHERE clauses
+- Use connection pooling
+- See [ADR-034](../adr/0034-repository-layer-method-naming-conventions.md)
+
+---
+
+## Test Failures
+
+### Tests Pass on Windows, Fail in Container
+
+**Cause**: Platform-specific behavior (ADR-014)
+
+**Rule**: Container results are authoritative. Windows results are unreliable.
+
+**Solution**:
+
+```bash
+# Always run tests in container
+podman exec -it flyer-crawler-dev npm test
+
+# For specific test
+podman exec -it flyer-crawler-dev npm test -- --run src/path/to/test.test.ts
+```
+
+### Integration Tests Fail
+
+**Common Issues**:
+
+**1. Vitest globalSetup Context Isolation**
+
+**Symptom**: Mocks or spies don't work in integration tests
+
+**Cause**: `globalSetup` runs in separate Node.js context
+
+**Solutions**:
+
+- Mark test as `.todo()` and document limitation
+- Create test-only API endpoints
+- Use Redis-based mock flags
+
+See [CLAUDE.md#integration-test-issues](../../CLAUDE.md#integration-test-issues) for details.
+
+**2. Cache Stale After Direct SQL**
+
+**Symptom**: Test reads stale data after direct database insert
+
+**Cause**: Cache not invalidated
+
+**Solution**:
+
+```typescript
+// After direct SQL insert
+await cacheService.invalidateFlyers();
+```
+
+**3. Queue Interference**
+
+**Symptom**: Cleanup worker processes test data before assertions
+
+**Solution**:
+
+```typescript
+const { cleanupQueue } = await import('../../services/queues.server');
+await cleanupQueue.drain();
+await cleanupQueue.pause();
+// ... test ...
+await cleanupQueue.resume();
+```
+
+### Type Check Failures
+
+**Symptom**: `npm run type-check` fails
+
+**Debug**:
+
+```bash
+# Run type check in container
+podman exec -it flyer-crawler-dev npm run type-check
+
+# Check specific file
+podman exec -it flyer-crawler-dev npx tsc --noEmit src/path/to/file.ts
+```
+
+**Common Causes**:
+
+- Missing type definitions
+- Incorrect imports
+- Type mismatch in function calls
+
+---
+
+## API Errors
+
+### 404 Not Found
+
+**Debug**:
+
+```bash
+# Check route registration
+grep -r "router.get" src/routes/
+
+# Check route path matches request
+# Verify middleware order
+```
+
+**Common Causes**:
+
+- Route not registered in `server.ts`
+- Typo in route path
+- Middleware blocking request
+
+### 500 Internal Server Error
+
+**Debug**:
+
+```bash
+# Check application logs
+podman logs -f flyer-crawler-dev
+
+# Check Bugsink for errors
+# Visit: http://localhost:8443 (dev) or https://bugsink.projectium.com (prod)
+```
+
+**Common Causes**:
+
+- Unhandled exception
+- Database error
+- Missing environment variable
+
+**Solution Pattern**:
+
+```typescript
+// Always wrap route handlers
+app.get('/api/endpoint', async (req, res) => {
+  try {
+    const result = await service.doSomething();
+    return sendSuccess(res, result);
+  } catch (error) {
+    return sendError(res, error); // Handles error types automatically
+  }
+});
+```
+
+### 401 Unauthorized
+
+**Debug**:
+
+```bash
+# Check JWT token in request
+# Verify token is valid and not expired
+
+# Test token decoding
+node -e "console.log(require('jsonwebtoken').decode('YOUR_TOKEN_HERE'))"
+```
+
+**Common Causes**:
+
+- Token expired
+- Invalid token format
+- Missing Authorization header
+- Wrong JWT_SECRET
+
+---
+
+## Authentication Problems
+
+### OAuth Not Working
+
+**Debug**:
+
+```bash
+# 1. Verify OAuth credentials
+cat .env.local | grep GOOGLE_CLIENT
+
+# 2. Check OAuth routes are registered
+grep -r "passport.authenticate" src/routes/
+
+# 3. Verify redirect URI matches Google Console
+# Should be: http://localhost:3001/api/auth/google/callback
+```
+
+**Common Issues**:
+
+- Redirect URI mismatch in Google Console
+- OAuth not enabled (commented out in config)
+- Wrong client ID/secret
+
+See [AUTHENTICATION.md](../architecture/AUTHENTICATION.md) for setup.
+
+### JWT Token Invalid
+
+**Debug**:
+
+```typescript
+// Decode token to inspect
+import jwt from 'jsonwebtoken';
+
+const decoded = jwt.decode(token);
+console.log('Token payload:', decoded);
+console.log('Expired:', decoded.exp < Date.now() / 1000);
+```
+
+**Solutions**:
+
+- Regenerate token
+- Check JWT_SECRET matches between environments
+- Verify token hasn't expired
+
+---
+
+## Background Job Issues
+
+### Jobs Not Processing
+
+**Debug**:
+
+```bash
+# Check if worker is running
+pm2 list
+
+# Check worker logs
+pm2 logs flyer-crawler-worker
+
+# Check Redis connection
+podman exec flyer-crawler-redis redis-cli ping
+
+# Check queue status
+node -e "
+const { flyerProcessingQueue } = require('./dist/services/queues.server.js');
+flyerProcessingQueue.getJobCounts().then(console.log);
+"
+```
+
+**Common Causes**:
+
+- Worker not running
+- Redis connection lost
+- Queue paused
+- Job stuck in failed state
+
+**Solutions**:
+
+```bash
+# Restart worker
+pm2 restart flyer-crawler-worker
+
+# Clear failed jobs
+node -e "
+const { flyerProcessingQueue } = require('./dist/services/queues.server.js');
+flyerProcessingQueue.clean(0, 1000, 'failed');
+"
+```
+
+### Jobs Failing
+
+**Debug**:
+
+```bash
+# Check failed jobs
+node -e "
+const { flyerProcessingQueue } = require('./dist/services/queues.server.js');
+flyerProcessingQueue.getFailed().then(jobs => {
+  jobs.forEach(job => console.log(job.failedReason));
+});
+"
+
+# Check worker logs for stack traces
+pm2 logs flyer-crawler-worker --lines 100
+```
+
+**Common Causes**:
+
+- Gemini API errors
+- Database errors
+- Invalid job data
+
+---
+
+## Frontend Issues
+
+### Hot Reload Not Working
+
+**Debug**:
+
+```bash
+# Check Vite is running
+curl http://localhost:5173
+
+# Check for port conflicts
+netstat -an | findstr "5173"
+```
+
+**Solution**:
+
+```bash
+# Restart dev server
+npm run dev
+```
+
+### API Calls Failing (CORS)
+
+**Symptom**: `CORS policy: No 'Access-Control-Allow-Origin' header`
+
+**Debug**:
+
+```typescript
+// Check CORS configuration in server.ts
+import cors from 'cors';
+
+app.use(
+  cors({
+    origin: env.FRONTEND_URL, // Should match http://localhost:5173 in dev
+    credentials: true,
+  }),
+);
+```
+
+**Solution**: Verify `FRONTEND_URL` in `.env.local` matches the frontend URL
+
+---
+
+## Performance Problems
+
+### Slow API Responses
+
+**Debug**:
+
+```typescript
+// Add timing logs
+const start = Date.now();
+const result = await slowOperation();
+console.log(`Operation took ${Date.now() - start}ms`);
+```
+
+**Common Causes**:
+
+- N+1 query problem
+- Missing database indexes
+- Large payload size
+- No caching
+
+**Solutions**:
+
+- Use JOINs instead of multiple queries
+- Add indexes: `CREATE INDEX idx_name ON table(column);`
+- Implement pagination
+- Add Redis caching
+
+### High Memory Usage
+
+**Debug**:
+
+```bash
+# Check PM2 memory usage
+pm2 monit
+
+# Check container memory
+podman stats flyer-crawler-dev
+```
+
+**Common Causes**:
+
+- Memory leak
+- Large in-memory cache
+- Unbounded array growth
+
+---
+
+## Debugging Tools
+
+### VS Code Debugger
+
+**Launch Configuration** (`.vscode/launch.json`):
+
+```json
+{
+  "version": "0.2.0",
+  "configurations": [
+    {
+      "type": "node",
+      "request": "launch",
+      "name": "Debug Tests",
+      "program": "${workspaceFolder}/node_modules/vitest/vitest.mjs",
+      "args": ["--run", "${file}"],
+      "console": "integratedTerminal",
+      "internalConsoleOptions": "neverOpen"
+    }
+  ]
+}
+```
+
+### Logging
+
+```typescript
+import { logger } from './utils/logger';
+
+// Structured logging
+logger.info('Processing flyer', { flyerId, userId });
+logger.error('Failed to process', { error, context });
+logger.debug('Cache hit', { key, ttl });
+```
+
+### Database Query Logging
+
+```typescript
+// In development, log all queries
+if (env.NODE_ENV === 'development') {
+  pool.on('connect', () => {
+    console.log('Database connected');
+  });
+
+  // Log slow queries
+  const originalQuery = pool.query.bind(pool);
+  pool.query = async (...args) => {
+    const start = Date.now();
+    const result = await originalQuery(...args);
+    const duration = Date.now() - start;
+    if (duration > 100) {
+      console.log(`Slow query (${duration}ms):`, args[0]);
+    }
+    return result;
+  };
+}
+```
+
+### Redis Debugging
+
+```bash
+# Monitor Redis commands
+podman exec -it flyer-crawler-redis redis-cli monitor
+
+# Check keys
+podman exec flyer-crawler-redis redis-cli keys "*"
+
+# Get key value
+podman exec flyer-crawler-redis redis-cli get "flyer:123"
+
+# Check cache stats
+podman exec flyer-crawler-redis redis-cli info stats
+```
+
+---
+
+## See Also
+
+- [TESTING.md](TESTING.md) - Testing strategies
+- [CODE-PATTERNS.md](CODE-PATTERNS.md) - Common patterns
+- [MONITORING.md](../operations/MONITORING.md) - Production monitoring
+- [Bugsink Setup](../tools/BUGSINK-SETUP.md) - Error tracking
+- [DevOps Guide](../subagents/DEVOPS-GUIDE.md) - Container debugging