flyer-crawler.projectium.com/READMEv2.md

Flyer Crawler - Development Environment Setup

Quick start guide for getting the development environment running with Podman containers.

Prerequisites

• Windows with WSL 2: Install WSL 2 by running wsl --install in an administrator PowerShell
• Podman Desktop: Download and install Podman Desktop for Windows
• Node.js 20+: Required for running the application

Quick Start - Container Environment

1. Initialize Podman

# Start Podman machine (do this once after installing Podman Desktop)
podman machine init
podman machine start

2. Start Required Services

Start PostgreSQL (with PostGIS) and Redis containers:

# Navigate to project directory
cd D:\gitea\flyer-crawler.projectium.com\flyer-crawler.projectium.com

# Start PostgreSQL with PostGIS
podman run -d \
  --name flyer-crawler-postgres \
  -e POSTGRES_USER=postgres \
  -e POSTGRES_PASSWORD=postgres \
  -e POSTGRES_DB=flyer_crawler_dev \
  -p 5432:5432 \
  docker.io/postgis/postgis:15-3.3

# Start Redis
podman run -d \
  --name flyer-crawler-redis \
  -e REDIS_PASSWORD="" \
  -p 6379:6379 \
  docker.io/library/redis:alpine

3. Wait for PostgreSQL to Initialize

# Wait a few seconds, then check if PostgreSQL is ready
podman exec flyer-crawler-postgres pg_isready -U postgres
# Should output: /var/run/postgresql:5432 - accepting connections

4. Install Required PostgreSQL Extensions

podman exec flyer-crawler-postgres psql -U postgres -d flyer_crawler_dev -c "CREATE EXTENSION IF NOT EXISTS postgis; CREATE EXTENSION IF NOT EXISTS pg_trgm; CREATE EXTENSION IF NOT EXISTS \"uuid-ossp\";"

5. Apply Database Schema

# Apply the complete schema with URL constraints enabled
podman exec -i flyer-crawler-postgres psql -U postgres -d flyer_crawler_dev < sql/master_schema_rollup.sql

6. Verify URL Constraints Are Enabled

podman exec flyer-crawler-postgres psql -U postgres -d flyer_crawler_dev -c "\d public.flyers" | grep -E "(image_url|icon_url|Check)"

You should see:

 image_url     | text                     |           | not null |
 icon_url      | text                     |           | not null |
Check constraints:
    "flyers_icon_url_check" CHECK (icon_url ~* '^https?://.*'::text)
    "flyers_image_url_check" CHECK (image_url ~* '^https?://.*'::text)

7. Set Environment Variables and Start Application

# Set required environment variables
$env:NODE_ENV="development"
$env:DB_HOST="localhost"
$env:DB_USER="postgres"
$env:DB_PASSWORD="postgres"
$env:DB_NAME="flyer_crawler_dev"
$env:REDIS_URL="redis://localhost:6379"
$env:PORT="3001"
$env:FRONTEND_URL="http://localhost:5173"

# Install dependencies (first time only)
npm install

# Start the development server (runs both backend and frontend)
npm run dev

The application will be available at:

• Frontend: http://localhost:5173
• Backend API: http://localhost:3001

Managing Containers

View Running Containers

podman ps

Stop Containers

podman stop flyer-crawler-postgres flyer-crawler-redis

Start Containers (After They've Been Created)

podman start flyer-crawler-postgres flyer-crawler-redis

Remove Containers (Clean Slate)

podman stop flyer-crawler-postgres flyer-crawler-redis
podman rm flyer-crawler-postgres flyer-crawler-redis

View Container Logs

podman logs flyer-crawler-postgres
podman logs flyer-crawler-redis

Database Management

Connect to PostgreSQL

podman exec -it flyer-crawler-postgres psql -U postgres -d flyer_crawler_dev

Reset Database Schema

# Drop all tables
podman exec -i flyer-crawler-postgres psql -U postgres -d flyer_crawler_dev < sql/drop_tables.sql

# Reapply schema
podman exec -i flyer-crawler-postgres psql -U postgres -d flyer_crawler_dev < sql/master_schema_rollup.sql

Seed Development Data

npm run db:reset:dev

Running Tests

Unit Tests

npm run test:unit

Integration Tests

IMPORTANT: Integration tests require the PostgreSQL and Redis containers to be running.

# Make sure containers are running
podman ps

# Run integration tests
npm run test:integration

Troubleshooting

Podman Machine Issues

If you get "unable to connect to Podman socket" errors:

podman machine start

PostgreSQL Connection Refused

Make sure PostgreSQL is ready:

podman exec flyer-crawler-postgres pg_isready -U postgres

Port Already in Use

If ports 5432 or 6379 are already in use, you can either:

1. Stop the conflicting service
2. Change the port mapping when creating containers (e.g., -p 5433:5432)

URL Validation Errors

The database now enforces URL constraints. All image_url and icon_url fields must:

• Start with http:// or https://
• Match the regex pattern: ^https?://.*

Make sure the FRONTEND_URL environment variable is set correctly to avoid URL validation errors.

ADR Implementation Status

This development environment implements:

• ADR-0002: Transaction Management ✅

  • All database operations use the withTransaction pattern
  • Automatic rollback on errors
  • No connection pool leaks

• ADR-0003: Input Validation ✅

  • Zod schemas for URL validation
  • Database constraints enabled
  • Validation at API boundaries

Development Workflow

1. Start Containers (once per development session)

   podman start flyer-crawler-postgres flyer-crawler-redis
   

2. Start Application

   npm run dev
   

3. Make Changes to code (auto-reloads via tsx watch)

4. Run Tests before committing

   npm run test:unit
   npm run test:integration
   

5. Stop Application (Ctrl+C)

6. Stop Containers (optional, or leave running)

   podman stop flyer-crawler-postgres flyer-crawler-redis
   

PM2 Worker Setup (Production-like)

To test with PM2 workers locally:

# Install PM2 globally (once)
npm install -g pm2

# Start the worker
pm2 start npm --name "flyer-crawler-worker" -- run worker:prod

# View logs
pm2 logs flyer-crawler-worker

# Stop worker
pm2 stop flyer-crawler-worker
pm2 delete flyer-crawler-worker

Next Steps

After getting the environment running:

1. Review docs/adr/ for architectural decisions
2. Check sql/master_schema_rollup.sql for database schema
3. Explore src/routes/ for API endpoints
4. Review src/types.ts for TypeScript type definitions

Common Environment Variables

Create these environment variables for development:

# Database
$env:DB_HOST="localhost"
$env:DB_USER="postgres"
$env:DB_PASSWORD="postgres"
$env:DB_NAME="flyer_crawler_dev"
$env:DB_PORT="5432"

# Redis
$env:REDIS_URL="redis://localhost:6379"

# Application
$env:NODE_ENV="development"
$env:PORT="3001"
$env:FRONTEND_URL="http://localhost:5173"

# Authentication (generate your own secrets)
$env:JWT_SECRET="your-dev-jwt-secret-change-this"
$env:SESSION_SECRET="your-dev-session-secret-change-this"

# AI Services (get your own API keys)
$env:VITE_GOOGLE_GENAI_API_KEY="your-google-genai-api-key"
$env:GOOGLE_MAPS_API_KEY="your-google-maps-api-key"

Resources

• Podman Desktop Documentation
• PostGIS Documentation
• Original README.md for production setup