torbo/flyer-crawler.projectium.com

Fork 0

Files

Torben Sorensen 8a38befb1c fix cert and image display issues

2026-01-22 12:46:28 -08:00

15 KiB

Raw Blame History

Debugging Guide

Common debugging strategies and troubleshooting patterns for Flyer Crawler.

Quick Debugging Checklist
Container Issues
Database Issues
Test Failures
API Errors
Authentication Problems
Background Job Issues
SSL Certificate Issues
Frontend Issues
Performance Problems
Debugging Tools

Quick Debugging Checklist

When something breaks, check these first:

Are containers running?
```
podman ps
```

Is the database accessible?

podman exec flyer-crawler-postgres pg_isready -U postgres

Are environment variables set?

# Check .env.local exists
cat .env.local

Are there recent errors in logs?

# Application logs
podman logs -f flyer-crawler-dev

# PM2 logs (production)
pm2 logs flyer-crawler-api

Is Redis accessible?

podman exec flyer-crawler-redis redis-cli ping

Container Issues

Container Won't Start

Symptom: podman start fails or container exits immediately

Debug:

# Check container status
podman ps -a

# View container logs
podman logs flyer-crawler-postgres
podman logs flyer-crawler-redis
podman logs flyer-crawler-dev

# Inspect container
podman inspect flyer-crawler-dev

Common Causes:

Port already in use
Insufficient resources
Configuration error

Solutions:

# Check port usage
netstat -an | findstr "5432"
netstat -an | findstr "6379"

# Remove and recreate container
podman stop flyer-crawler-postgres
podman rm flyer-crawler-postgres
# ... recreate with podman run ...

"Unable to connect to Podman socket"

Symptom: Error: unable to connect to Podman socket

Solution:

# Start Podman machine
podman machine start

# Verify it's running
podman machine list

Port Already in Use

Symptom: Error: port 5432 is already allocated

Solutions:

Option 1: Stop conflicting service

# Find process using port
netstat -ano | findstr "5432"

# Stop the service or kill process

Option 2: Use different port

# Run container on different host port
podman run -d --name flyer-crawler-postgres -p 5433:5432 ...

# Update .env.local
DB_PORT=5433

Database Issues

Connection Refused

Symptom: Error: connect ECONNREFUSED 127.0.0.1:5432

Debug:

# 1. Check if PostgreSQL container is running
podman ps | grep postgres

# 2. Check if PostgreSQL is ready
podman exec flyer-crawler-postgres pg_isready -U postgres

# 3. Test connection
podman exec flyer-crawler-postgres psql -U postgres -d flyer_crawler_dev -c "SELECT 1;"

Common Causes:

Container not running
PostgreSQL still initializing
Wrong credentials in .env.local

Solutions:

# Start container
podman start flyer-crawler-postgres

# Wait for initialization (check logs)
podman logs -f flyer-crawler-postgres

# Verify credentials match .env.local
cat .env.local | grep DB_

Schema Out of Sync

Symptom: Tests fail with missing column or table errors

Cause: master_schema_rollup.sql not in sync with migrations

Solution:

# Reset database with current schema
podman exec -i flyer-crawler-postgres psql -U postgres -d flyer_crawler_dev < sql/drop_tables.sql
podman exec -i flyer-crawler-postgres psql -U postgres -d flyer_crawler_dev < sql/master_schema_rollup.sql

# Verify schema
podman exec flyer-crawler-postgres psql -U postgres -d flyer_crawler_dev -c "\dt"

Query Performance Issues

Debug:

-- Enable query logging
ALTER DATABASE flyer_crawler_dev SET log_statement = 'all';

-- Check slow queries
SELECT query, mean_exec_time, calls
FROM pg_stat_statements
WHERE mean_exec_time > 100
ORDER BY mean_exec_time DESC
LIMIT 10;

-- Analyze query plan
EXPLAIN ANALYZE
SELECT * FROM flyers WHERE store_id = 1;

Solutions:

Add missing indexes
Optimize WHERE clauses
Use connection pooling
See ADR-034

Test Failures

Tests Pass on Windows, Fail in Container

Cause: Platform-specific behavior (ADR-014)

Rule: Container results are authoritative. Windows results are unreliable.

Solution:

# Always run tests in container
podman exec -it flyer-crawler-dev npm test

# For specific test
podman exec -it flyer-crawler-dev npm test -- --run src/path/to/test.test.ts

Integration Tests Fail

Common Issues:

1. Vitest globalSetup Context Isolation

Symptom: Mocks or spies don't work in integration tests

Cause: globalSetup runs in separate Node.js context

Solutions:

Mark test as .todo() and document limitation
Create test-only API endpoints
Use Redis-based mock flags

See CLAUDE.md#integration-test-issues for details.

2. Cache Stale After Direct SQL

Symptom: Test reads stale data after direct database insert

Cause: Cache not invalidated

Solution:

// After direct SQL insert
await cacheService.invalidateFlyers();

3. Queue Interference

Symptom: Cleanup worker processes test data before assertions

Solution:

const { cleanupQueue } = await import('../../services/queues.server');
await cleanupQueue.drain();
await cleanupQueue.pause();
// ... test ...
await cleanupQueue.resume();

Type Check Failures

Symptom: npm run type-check fails

Debug:

# Run type check in container
podman exec -it flyer-crawler-dev npm run type-check

# Check specific file
podman exec -it flyer-crawler-dev npx tsc --noEmit src/path/to/file.ts

Common Causes:

Missing type definitions
Incorrect imports
Type mismatch in function calls

API Errors

404 Not Found

Debug:

# Check route registration
grep -r "router.get" src/routes/

# Check route path matches request
# Verify middleware order

Common Causes:

Route not registered in server.ts
Typo in route path
Middleware blocking request

500 Internal Server Error

Debug:

# Check application logs
podman logs -f flyer-crawler-dev

# Check Bugsink for errors
# Visit: http://localhost:8443 (dev) or https://bugsink.projectium.com (prod)

Common Causes:

Unhandled exception
Database error
Missing environment variable

Solution Pattern:

// Always wrap route handlers
app.get('/api/endpoint', async (req, res) => {
  try {
    const result = await service.doSomething();
    return sendSuccess(res, result);
  } catch (error) {
    return sendError(res, error); // Handles error types automatically
  }
});

401 Unauthorized

Debug:

# Check JWT token in request
# Verify token is valid and not expired

# Test token decoding
node -e "console.log(require('jsonwebtoken').decode('YOUR_TOKEN_HERE'))"

Common Causes:

Token expired
Invalid token format
Missing Authorization header
Wrong JWT_SECRET

Authentication Problems

OAuth Not Working

Debug:

# 1. Verify OAuth credentials
cat .env.local | grep GOOGLE_CLIENT

# 2. Check OAuth routes are registered
grep -r "passport.authenticate" src/routes/

# 3. Verify redirect URI matches Google Console
# Should be: http://localhost:3001/api/auth/google/callback

Common Issues:

Redirect URI mismatch in Google Console
OAuth not enabled (commented out in config)
Wrong client ID/secret

See AUTHENTICATION.md for setup.

JWT Token Invalid

Debug:

// Decode token to inspect
import jwt from 'jsonwebtoken';

const decoded = jwt.decode(token);
console.log('Token payload:', decoded);
console.log('Expired:', decoded.exp < Date.now() / 1000);

Solutions:

Regenerate token
Check JWT_SECRET matches between environments
Verify token hasn't expired

Background Job Issues

Jobs Not Processing

Debug:

# Check if worker is running
pm2 list

# Check worker logs
pm2 logs flyer-crawler-worker

# Check Redis connection
podman exec flyer-crawler-redis redis-cli ping

# Check queue status
node -e "
const { flyerProcessingQueue } = require('./dist/services/queues.server.js');
flyerProcessingQueue.getJobCounts().then(console.log);
"

Common Causes:

Worker not running
Redis connection lost
Queue paused
Job stuck in failed state

Solutions:

# Restart worker
pm2 restart flyer-crawler-worker

# Clear failed jobs
node -e "
const { flyerProcessingQueue } = require('./dist/services/queues.server.js');
flyerProcessingQueue.clean(0, 1000, 'failed');
"

Jobs Failing

Debug:

# Check failed jobs
node -e "
const { flyerProcessingQueue } = require('./dist/services/queues.server.js');
flyerProcessingQueue.getFailed().then(jobs => {
  jobs.forEach(job => console.log(job.failedReason));
});
"

# Check worker logs for stack traces
pm2 logs flyer-crawler-worker --lines 100

Common Causes:

Gemini API errors
Database errors
Invalid job data

SSL Certificate Issues

Images Not Loading (ERR_CERT_AUTHORITY_INVALID)

Symptom: Flyer images fail to load with ERR_CERT_AUTHORITY_INVALID in browser console

Cause: Mixed hostname origins - user accesses via localhost but images use 127.0.0.1 (or vice versa)

Debug:

# Check which hostname images are using
podman exec flyer-crawler-dev psql -U postgres -d flyer_crawler_dev \
  -c "SELECT image_url FROM flyers LIMIT 1;"

# Verify certificate includes both hostnames
podman exec flyer-crawler-dev openssl x509 -in /app/certs/localhost.crt -text -noout | grep -A1 "Subject Alternative Name"

# Check NGINX accepts both hostnames
podman exec flyer-crawler-dev grep "server_name" /etc/nginx/sites-available/default

Solution: The dev container is configured to handle both hostnames:

Certificate includes SANs for localhost, 127.0.0.1, and ::1
NGINX server_name directive includes both localhost and 127.0.0.1

If you still see errors:

# Rebuild container to regenerate certificate
podman-compose down
podman-compose build --no-cache flyer-crawler-dev
podman-compose up -d

See FLYER-URL-CONFIGURATION.md for full details.

Self-Signed Certificate Not Trusted

Symptom: Browser shows security warning for https://localhost

Temporary Workaround: Accept the warning by clicking "Advanced" > "Proceed to localhost"

Permanent Fix (Recommended): Install the mkcert CA certificate to eliminate all SSL warnings.

The CA certificate is located at certs/mkcert-ca.crt in the project root. See certs/README.md for platform-specific installation instructions (Windows, macOS, Linux, Firefox).

After installation:

Your browser will trust all mkcert certificates without warnings
Both https://localhost/ and https://127.0.0.1/ will work without SSL errors
Flyer images will load without ERR_CERT_AUTHORITY_INVALID errors

NGINX SSL Configuration Test

Debug:

# Test NGINX configuration
podman exec flyer-crawler-dev nginx -t

# Check if NGINX is listening on 443
podman exec flyer-crawler-dev netstat -tlnp | grep 443

# Verify certificate files exist
podman exec flyer-crawler-dev ls -la /app/certs/

Frontend Issues

Hot Reload Not Working

Debug:

# Check Vite is running
curl http://localhost:5173

# Check for port conflicts
netstat -an | findstr "5173"

Solution:

# Restart dev server
npm run dev

API Calls Failing (CORS)

Symptom: CORS policy: No 'Access-Control-Allow-Origin' header

Debug:

// Check CORS configuration in server.ts
import cors from 'cors';

app.use(
  cors({
    origin: env.FRONTEND_URL, // Should match http://localhost:5173 in dev
    credentials: true,
  }),
);

Solution: Verify FRONTEND_URL in .env.local matches the frontend URL

Performance Problems

Slow API Responses

Debug:

// Add timing logs
const start = Date.now();
const result = await slowOperation();
console.log(`Operation took ${Date.now() - start}ms`);

Common Causes:

N+1 query problem
Missing database indexes
Large payload size
No caching

Solutions:

Use JOINs instead of multiple queries
Add indexes: CREATE INDEX idx_name ON table(column);
Implement pagination
Add Redis caching

High Memory Usage

Debug:

# Check PM2 memory usage
pm2 monit

# Check container memory
podman stats flyer-crawler-dev

Common Causes:

Memory leak
Large in-memory cache
Unbounded array growth

Debugging Tools

VS Code Debugger

Launch Configuration (.vscode/launch.json):

{
  "version": "0.2.0",
  "configurations": [
    {
      "type": "node",
      "request": "launch",
      "name": "Debug Tests",
      "program": "${workspaceFolder}/node_modules/vitest/vitest.mjs",
      "args": ["--run", "${file}"],
      "console": "integratedTerminal",
      "internalConsoleOptions": "neverOpen"
    }
  ]
}

Logging

import { logger } from './utils/logger';

// Structured logging
logger.info('Processing flyer', { flyerId, userId });
logger.error('Failed to process', { error, context });
logger.debug('Cache hit', { key, ttl });

Database Query Logging

// In development, log all queries
if (env.NODE_ENV === 'development') {
  pool.on('connect', () => {
    console.log('Database connected');
  });

  // Log slow queries
  const originalQuery = pool.query.bind(pool);
  pool.query = async (...args) => {
    const start = Date.now();
    const result = await originalQuery(...args);
    const duration = Date.now() - start;
    if (duration > 100) {
      console.log(`Slow query (${duration}ms):`, args[0]);
    }
    return result;
  };
}

Redis Debugging

# Monitor Redis commands
podman exec -it flyer-crawler-redis redis-cli monitor

# Check keys
podman exec flyer-crawler-redis redis-cli keys "*"

# Get key value
podman exec flyer-crawler-redis redis-cli get "flyer:123"

# Check cache stats
podman exec flyer-crawler-redis redis-cli info stats

15 KiB Raw Blame History

Debugging Guide

Table of Contents

Quick Debugging Checklist

Container Issues

Container Won't Start

"Unable to connect to Podman socket"

Port Already in Use

Database Issues

Connection Refused

Schema Out of Sync

Query Performance Issues

Test Failures

Tests Pass on Windows, Fail in Container

Integration Tests Fail

Type Check Failures

API Errors

404 Not Found

500 Internal Server Error

401 Unauthorized

Authentication Problems

OAuth Not Working

JWT Token Invalid

Background Job Issues

Jobs Not Processing

Jobs Failing

SSL Certificate Issues

Images Not Loading (ERR_CERT_AUTHORITY_INVALID)

Self-Signed Certificate Not Trusted

NGINX SSL Configuration Test

Frontend Issues

Hot Reload Not Working

API Calls Failing (CORS)

Performance Problems

Slow API Responses

High Memory Usage

Debugging Tools

VS Code Debugger

Logging

Database Query Logging

Redis Debugging

See Also

15 KiB

Raw Blame History