flyer-crawler.projectium.com/docs/development/DEBUGGING.md

# Debugging Guide

Common debugging strategies and troubleshooting patterns for Flyer Crawler.

## Table of Contents

- [Quick Debugging Checklist](#quick-debugging-checklist)
- [Container Issues](#container-issues)
- [Database Issues](#database-issues)
- [Test Failures](#test-failures)
- [API Errors](#api-errors)
- [Authentication Problems](#authentication-problems)
- [Background Job Issues](#background-job-issues)
- [SSL Certificate Issues](#ssl-certificate-issues)
- [Frontend Issues](#frontend-issues)
- [Performance Problems](#performance-problems)
- [Debugging Tools](#debugging-tools)

---

## Quick Debugging Checklist

When something breaks, check these first:

1. **Are containers running?**

   ```bash
   podman ps
   ```

2. **Is the database accessible?**

   ```bash
   podman exec flyer-crawler-postgres pg_isready -U postgres
   ```

3. **Are environment variables set?**

   ```bash
   # Check .env.local exists
   cat .env.local
   ```

4. **Are there recent errors in logs?**

   ```bash
   # Application logs
   podman logs -f flyer-crawler-dev

   # PM2 logs (production)
   pm2 logs flyer-crawler-api
   ```

5. **Is Redis accessible?**
   ```bash
   podman exec flyer-crawler-redis redis-cli ping
   ```

---

## Container Issues

### Container Won't Start

**Symptom**: `podman start` fails or container exits immediately

**Debug**:

```bash
# Check container status
podman ps -a

# View container logs
podman logs flyer-crawler-postgres
podman logs flyer-crawler-redis
podman logs flyer-crawler-dev

# Inspect container
podman inspect flyer-crawler-dev
```

**Common Causes**:

- Port already in use
- Insufficient resources
- Configuration error

**Solutions**:

```bash
# Check port usage
netstat -an | findstr "5432"
netstat -an | findstr "6379"

# Remove and recreate container
podman stop flyer-crawler-postgres
podman rm flyer-crawler-postgres
# ... recreate with podman run ...
```

### "Unable to connect to Podman socket"

**Symptom**: `Error: unable to connect to Podman socket`

**Solution**:

```bash
# Start Podman machine
podman machine start

# Verify it's running
podman machine list
```

### Port Already in Use

**Symptom**: `Error: port 5432 is already allocated`

**Solutions**:

**Option 1**: Stop conflicting service

```bash
# Find process using port
netstat -ano | findstr "5432"

# Stop the service or kill process
```

**Option 2**: Use different port

```bash
# Run container on different host port
podman run -d --name flyer-crawler-postgres -p 5433:5432 ...

# Update .env.local
DB_PORT=5433
```

---

## Database Issues

### Connection Refused

**Symptom**: `Error: connect ECONNREFUSED 127.0.0.1:5432`

**Debug**:

```bash
# 1. Check if PostgreSQL container is running
podman ps | grep postgres

# 2. Check if PostgreSQL is ready
podman exec flyer-crawler-postgres pg_isready -U postgres

# 3. Test connection
podman exec flyer-crawler-postgres psql -U postgres -d flyer_crawler_dev -c "SELECT 1;"
```

**Common Causes**:

- Container not running
- PostgreSQL still initializing
- Wrong credentials in `.env.local`

**Solutions**:

```bash
# Start container
podman start flyer-crawler-postgres

# Wait for initialization (check logs)
podman logs -f flyer-crawler-postgres

# Verify credentials match .env.local
cat .env.local | grep DB_
```

### Schema Out of Sync

**Symptom**: Tests fail with missing column or table errors

**Cause**: `master_schema_rollup.sql` not in sync with migrations

**Solution**:

```bash
# Reset database with current schema
podman exec -i flyer-crawler-postgres psql -U postgres -d flyer_crawler_dev < sql/drop_tables.sql
podman exec -i flyer-crawler-postgres psql -U postgres -d flyer_crawler_dev < sql/master_schema_rollup.sql

# Verify schema
podman exec flyer-crawler-postgres psql -U postgres -d flyer_crawler_dev -c "\dt"
```

### Query Performance Issues

**Debug**:

```sql
-- Enable query logging
ALTER DATABASE flyer_crawler_dev SET log_statement = 'all';

-- Check slow queries
SELECT query, mean_exec_time, calls
FROM pg_stat_statements
WHERE mean_exec_time > 100
ORDER BY mean_exec_time DESC
LIMIT 10;

-- Analyze query plan
EXPLAIN ANALYZE
SELECT * FROM flyers WHERE store_id = 1;
```

**Solutions**:

- Add missing indexes
- Optimize WHERE clauses
- Use connection pooling
- See [ADR-034](../adr/0034-repository-layer-method-naming-conventions.md)

---

## Test Failures

### Tests Pass on Windows, Fail in Container

**Cause**: Platform-specific behavior (ADR-014)

**Rule**: Container results are authoritative. Windows results are unreliable.

**Solution**:

```bash
# Always run tests in container
podman exec -it flyer-crawler-dev npm test

# For specific test
podman exec -it flyer-crawler-dev npm test -- --run src/path/to/test.test.ts
```

### Integration Tests Fail

**Common Issues**:

**1. Vitest globalSetup Context Isolation**

**Symptom**: Mocks or spies don't work in integration tests

**Cause**: `globalSetup` runs in separate Node.js context

**Solutions**:

- Mark test as `.todo()` and document limitation
- Create test-only API endpoints
- Use Redis-based mock flags

See [CLAUDE.md#integration-test-issues](../../CLAUDE.md#integration-test-issues) for details.

**2. Cache Stale After Direct SQL**

**Symptom**: Test reads stale data after direct database insert

**Cause**: Cache not invalidated

**Solution**:

```typescript
// After direct SQL insert
await cacheService.invalidateFlyers();
```

**3. Queue Interference**

**Symptom**: Cleanup worker processes test data before assertions

**Solution**:

```typescript
const { cleanupQueue } = await import('../../services/queues.server');
await cleanupQueue.drain();
await cleanupQueue.pause();
// ... test ...
await cleanupQueue.resume();
```

### Type Check Failures

**Symptom**: `npm run type-check` fails

**Debug**:

```bash
# Run type check in container
podman exec -it flyer-crawler-dev npm run type-check

# Check specific file
podman exec -it flyer-crawler-dev npx tsc --noEmit src/path/to/file.ts
```

**Common Causes**:

- Missing type definitions
- Incorrect imports
- Type mismatch in function calls

---

## API Errors

### 404 Not Found

**Debug**:

```bash
# Check route registration
grep -r "router.get" src/routes/

# Check route path matches request
# Verify middleware order
```

**Common Causes**:

- Route not registered in `server.ts`
- Typo in route path
- Middleware blocking request

### 500 Internal Server Error

**Debug**:

```bash
# Check application logs
podman logs -f flyer-crawler-dev

# Check Bugsink for errors
# Visit: http://localhost:8443 (dev) or https://bugsink.projectium.com (prod)
```

**Common Causes**:

- Unhandled exception
- Database error
- Missing environment variable

**Solution Pattern**:

```typescript
// Always wrap route handlers
app.get('/api/endpoint', async (req, res) => {
  try {
    const result = await service.doSomething();
    return sendSuccess(res, result);
  } catch (error) {
    return sendError(res, error); // Handles error types automatically
  }
});
```

### 401 Unauthorized

**Debug**:

```bash
# Check JWT token in request
# Verify token is valid and not expired

# Test token decoding
node -e "console.log(require('jsonwebtoken').decode('YOUR_TOKEN_HERE'))"
```

**Common Causes**:

- Token expired
- Invalid token format
- Missing Authorization header
- Wrong JWT_SECRET

---

## Authentication Problems

### OAuth Not Working

**Debug**:

```bash
# 1. Verify OAuth credentials
cat .env.local | grep GOOGLE_CLIENT

# 2. Check OAuth routes are registered
grep -r "passport.authenticate" src/routes/

# 3. Verify redirect URI matches Google Console
# Should be: http://localhost:3001/api/auth/google/callback
```

**Common Issues**:

- Redirect URI mismatch in Google Console
- OAuth not enabled (commented out in config)
- Wrong client ID/secret

See [AUTHENTICATION.md](../architecture/AUTHENTICATION.md) for setup.

### JWT Token Invalid

**Debug**:

```typescript
// Decode token to inspect
import jwt from 'jsonwebtoken';

const decoded = jwt.decode(token);
console.log('Token payload:', decoded);
console.log('Expired:', decoded.exp < Date.now() / 1000);
```

**Solutions**:

- Regenerate token
- Check JWT_SECRET matches between environments
- Verify token hasn't expired

---

## Background Job Issues

### Jobs Not Processing

**Debug**:

```bash
# Check if worker is running
pm2 list

# Check worker logs
pm2 logs flyer-crawler-worker

# Check Redis connection
podman exec flyer-crawler-redis redis-cli ping

# Check queue status
node -e "
const { flyerProcessingQueue } = require('./dist/services/queues.server.js');
flyerProcessingQueue.getJobCounts().then(console.log);
"
```

**Common Causes**:

- Worker not running
- Redis connection lost
- Queue paused
- Job stuck in failed state

**Solutions**:

```bash
# Restart worker
pm2 restart flyer-crawler-worker

# Clear failed jobs
node -e "
const { flyerProcessingQueue } = require('./dist/services/queues.server.js');
flyerProcessingQueue.clean(0, 1000, 'failed');
"
```

### Jobs Failing

**Debug**:

```bash
# Check failed jobs
node -e "
const { flyerProcessingQueue } = require('./dist/services/queues.server.js');
flyerProcessingQueue.getFailed().then(jobs => {
  jobs.forEach(job => console.log(job.failedReason));
});
"

# Check worker logs for stack traces
pm2 logs flyer-crawler-worker --lines 100
```

**Common Causes**:

- Gemini API errors
- Database errors
- Invalid job data

---

## SSL Certificate Issues

### Images Not Loading (ERR_CERT_AUTHORITY_INVALID)

**Symptom**: Flyer images fail to load with `ERR_CERT_AUTHORITY_INVALID` in browser console

**Cause**: Mixed hostname origins - user accesses via `localhost` but images use `127.0.0.1` (or vice versa)

**Debug**:

```bash
# Check which hostname images are using
podman exec flyer-crawler-dev psql -U postgres -d flyer_crawler_dev \
  -c "SELECT image_url FROM flyers LIMIT 1;"

# Verify certificate includes both hostnames
podman exec flyer-crawler-dev openssl x509 -in /app/certs/localhost.crt -text -noout | grep -A1 "Subject Alternative Name"

# Check NGINX accepts both hostnames
podman exec flyer-crawler-dev grep "server_name" /etc/nginx/sites-available/default
```

**Solution**: The dev container is configured to handle both hostnames:

1. Certificate includes SANs for `localhost`, `127.0.0.1`, and `::1`
2. NGINX `server_name` directive includes both `localhost` and `127.0.0.1`

If you still see errors:

```bash
# Rebuild container to regenerate certificate
podman-compose down
podman-compose build --no-cache flyer-crawler-dev
podman-compose up -d
```

See [FLYER-URL-CONFIGURATION.md](../FLYER-URL-CONFIGURATION.md#ssl-certificate-configuration-dev-container) for full details.

### Self-Signed Certificate Not Trusted

**Symptom**: Browser shows security warning for `https://localhost`

**Temporary Workaround**: Accept the warning by clicking "Advanced" > "Proceed to localhost"

**Permanent Fix (Recommended)**: Install the mkcert CA certificate to eliminate all SSL warnings.

The CA certificate is located at `certs/mkcert-ca.crt` in the project root. See [`certs/README.md`](../../certs/README.md) for platform-specific installation instructions (Windows, macOS, Linux, Firefox).

After installation:

- Your browser will trust all mkcert certificates without warnings
- Both `https://localhost/` and `https://127.0.0.1/` will work without SSL errors
- Flyer images will load without `ERR_CERT_AUTHORITY_INVALID` errors

### NGINX SSL Configuration Test

**Debug**:

```bash
# Test NGINX configuration
podman exec flyer-crawler-dev nginx -t

# Check if NGINX is listening on 443
podman exec flyer-crawler-dev netstat -tlnp | grep 443

# Verify certificate files exist
podman exec flyer-crawler-dev ls -la /app/certs/
```

---

## Frontend Issues

### Hot Reload Not Working

**Debug**:

```bash
# Check Vite is running
curl http://localhost:5173

# Check for port conflicts
netstat -an | findstr "5173"
```

**Solution**:

```bash
# Restart dev server
npm run dev
```

### API Calls Failing (CORS)

**Symptom**: `CORS policy: No 'Access-Control-Allow-Origin' header`

**Debug**:

```typescript
// Check CORS configuration in server.ts
import cors from 'cors';

app.use(
  cors({
    origin: env.FRONTEND_URL, // Should match http://localhost:5173 in dev
    credentials: true,
  }),
);
```

**Solution**: Verify `FRONTEND_URL` in `.env.local` matches the frontend URL

---

## Performance Problems

### Slow API Responses

**Debug**:

```typescript
// Add timing logs
const start = Date.now();
const result = await slowOperation();
console.log(`Operation took ${Date.now() - start}ms`);
```

**Common Causes**:

- N+1 query problem
- Missing database indexes
- Large payload size
- No caching

**Solutions**:

- Use JOINs instead of multiple queries
- Add indexes: `CREATE INDEX idx_name ON table(column);`
- Implement pagination
- Add Redis caching

### High Memory Usage

**Debug**:

```bash
# Check PM2 memory usage
pm2 monit

# Check container memory
podman stats flyer-crawler-dev
```

**Common Causes**:

- Memory leak
- Large in-memory cache
- Unbounded array growth

---

## Debugging Tools

### VS Code Debugger

**Launch Configuration** (`.vscode/launch.json`):

```json
{
  "version": "0.2.0",
  "configurations": [
    {
      "type": "node",
      "request": "launch",
      "name": "Debug Tests",
      "program": "${workspaceFolder}/node_modules/vitest/vitest.mjs",
      "args": ["--run", "${file}"],
      "console": "integratedTerminal",
      "internalConsoleOptions": "neverOpen"
    }
  ]
}
```

### Logging

```typescript
import { logger } from './utils/logger';

// Structured logging
logger.info('Processing flyer', { flyerId, userId });
logger.error('Failed to process', { error, context });
logger.debug('Cache hit', { key, ttl });
```

### Database Query Logging

```typescript
// In development, log all queries
if (env.NODE_ENV === 'development') {
  pool.on('connect', () => {
    console.log('Database connected');
  });

  // Log slow queries
  const originalQuery = pool.query.bind(pool);
  pool.query = async (...args) => {
    const start = Date.now();
    const result = await originalQuery(...args);
    const duration = Date.now() - start;
    if (duration > 100) {
      console.log(`Slow query (${duration}ms):`, args[0]);
    }
    return result;
  };
}
```

### Redis Debugging

```bash
# Monitor Redis commands
podman exec -it flyer-crawler-redis redis-cli monitor

# Check keys
podman exec flyer-crawler-redis redis-cli keys "*"

# Get key value
podman exec flyer-crawler-redis redis-cli get "flyer:123"

# Check cache stats
podman exec flyer-crawler-redis redis-cli info stats
```

---

## See Also

- [TESTING.md](TESTING.md) - Testing strategies
- [CODE-PATTERNS.md](CODE-PATTERNS.md) - Common patterns
- [MONITORING.md](../operations/MONITORING.md) - Production monitoring
- [Bugsink Setup](../tools/BUGSINK-SETUP.md) - Error tracking
- [DevOps Guide](../subagents/DEVOPS-GUIDE.md) - Container debugging