flyer-crawler.projectium.com/READMEv2.md

# Flyer Crawler - Development Environment Setup

Quick start guide for getting the development environment running with Podman containers.

## Prerequisites

- **Windows with WSL 2**: Install WSL 2 by running `wsl --install` in an administrator PowerShell
- **Podman Desktop**: Download and install [Podman Desktop for Windows](https://podman-desktop.io/)
- **Node.js 20+**: Required for running the application

## Quick Start - Container Environment

### 1. Initialize Podman

```powershell
# Start Podman machine (do this once after installing Podman Desktop)
podman machine init
podman machine start
```

### 2. Start Required Services

Start PostgreSQL (with PostGIS) and Redis containers:

```powershell
# Navigate to project directory
cd D:\gitea\flyer-crawler.projectium.com\flyer-crawler.projectium.com

# Start PostgreSQL with PostGIS
podman run -d \
  --name flyer-crawler-postgres \
  -e POSTGRES_USER=postgres \
  -e POSTGRES_PASSWORD=postgres \
  -e POSTGRES_DB=flyer_crawler_dev \
  -p 5432:5432 \
  docker.io/postgis/postgis:15-3.3

# Start Redis
podman run -d \
  --name flyer-crawler-redis \
  -e REDIS_PASSWORD="" \
  -p 6379:6379 \
  docker.io/library/redis:alpine
```

### 3. Wait for PostgreSQL to Initialize

```powershell
# Wait a few seconds, then check if PostgreSQL is ready
podman exec flyer-crawler-postgres pg_isready -U postgres
# Should output: /var/run/postgresql:5432 - accepting connections
```

### 4. Install Required PostgreSQL Extensions

```powershell
podman exec flyer-crawler-postgres psql -U postgres -d flyer_crawler_dev -c "CREATE EXTENSION IF NOT EXISTS postgis; CREATE EXTENSION IF NOT EXISTS pg_trgm; CREATE EXTENSION IF NOT EXISTS \"uuid-ossp\";"
```

### 5. Apply Database Schema

```powershell
# Apply the complete schema with URL constraints enabled
podman exec -i flyer-crawler-postgres psql -U postgres -d flyer_crawler_dev < sql/master_schema_rollup.sql
```

### 6. Verify URL Constraints Are Enabled

```powershell
podman exec flyer-crawler-postgres psql -U postgres -d flyer_crawler_dev -c "\d public.flyers" | grep -E "(image_url|icon_url|Check)"
```

You should see:
```
 image_url     | text                     |           | not null |
 icon_url      | text                     |           | not null |
Check constraints:
    "flyers_icon_url_check" CHECK (icon_url ~* '^https?://.*'::text)
    "flyers_image_url_check" CHECK (image_url ~* '^https?://.*'::text)
```

### 7. Set Environment Variables and Start Application

```powershell
# Set required environment variables
$env:NODE_ENV="development"
$env:DB_HOST="localhost"
$env:DB_USER="postgres"
$env:DB_PASSWORD="postgres"
$env:DB_NAME="flyer_crawler_dev"
$env:REDIS_URL="redis://localhost:6379"
$env:PORT="3001"
$env:FRONTEND_URL="http://localhost:5173"

# Install dependencies (first time only)
npm install

# Start the development server (runs both backend and frontend)
npm run dev
```

The application will be available at:
- **Frontend**: http://localhost:5173
- **Backend API**: http://localhost:3001

## Managing Containers

### View Running Containers
```powershell
podman ps
```

### Stop Containers
```powershell
podman stop flyer-crawler-postgres flyer-crawler-redis
```

### Start Containers (After They've Been Created)
```powershell
podman start flyer-crawler-postgres flyer-crawler-redis
```

### Remove Containers (Clean Slate)
```powershell
podman stop flyer-crawler-postgres flyer-crawler-redis
podman rm flyer-crawler-postgres flyer-crawler-redis
```

### View Container Logs
```powershell
podman logs flyer-crawler-postgres
podman logs flyer-crawler-redis
```

## Database Management

### Connect to PostgreSQL
```powershell
podman exec -it flyer-crawler-postgres psql -U postgres -d flyer_crawler_dev
```

### Reset Database Schema
```powershell
# Drop all tables
podman exec -i flyer-crawler-postgres psql -U postgres -d flyer_crawler_dev < sql/drop_tables.sql

# Reapply schema
podman exec -i flyer-crawler-postgres psql -U postgres -d flyer_crawler_dev < sql/master_schema_rollup.sql
```

### Seed Development Data
```powershell
npm run db:reset:dev
```

## Running Tests

### Unit Tests
```powershell
npm run test:unit
```

### Integration Tests

**IMPORTANT**: Integration tests require the PostgreSQL and Redis containers to be running.

```powershell
# Make sure containers are running
podman ps

# Run integration tests
npm run test:integration
```

## Troubleshooting

### Podman Machine Issues
If you get "unable to connect to Podman socket" errors:
```powershell
podman machine start
```

### PostgreSQL Connection Refused
Make sure PostgreSQL is ready:
```powershell
podman exec flyer-crawler-postgres pg_isready -U postgres
```

### Port Already in Use
If ports 5432 or 6379 are already in use, you can either:
1. Stop the conflicting service
2. Change the port mapping when creating containers (e.g., `-p 5433:5432`)

### URL Validation Errors
The database now enforces URL constraints. All `image_url` and `icon_url` fields must:
- Start with `http://` or `https://`
- Match the regex pattern: `^https?://.*`

Make sure the `FRONTEND_URL` environment variable is set correctly to avoid URL validation errors.

## ADR Implementation Status

This development environment implements:

- **ADR-0002**: Transaction Management ✅
  - All database operations use the `withTransaction` pattern
  - Automatic rollback on errors
  - No connection pool leaks

- **ADR-0003**: Input Validation ✅
  - Zod schemas for URL validation
  - Database constraints enabled
  - Validation at API boundaries

## Development Workflow

1. **Start Containers** (once per development session)
   ```powershell
   podman start flyer-crawler-postgres flyer-crawler-redis
   ```

2. **Start Application**
   ```powershell
   npm run dev
   ```

3. **Make Changes** to code (auto-reloads via `tsx watch`)

4. **Run Tests** before committing
   ```powershell
   npm run test:unit
   npm run test:integration
   ```

5. **Stop Application** (Ctrl+C)

6. **Stop Containers** (optional, or leave running)
   ```powershell
   podman stop flyer-crawler-postgres flyer-crawler-redis
   ```

## PM2 Worker Setup (Production-like)

To test with PM2 workers locally:

```powershell
# Install PM2 globally (once)
npm install -g pm2

# Start the worker
pm2 start npm --name "flyer-crawler-worker" -- run worker:prod

# View logs
pm2 logs flyer-crawler-worker

# Stop worker
pm2 stop flyer-crawler-worker
pm2 delete flyer-crawler-worker
```

## Next Steps

After getting the environment running:

1. Review [docs/adr/](docs/adr/) for architectural decisions
2. Check [sql/master_schema_rollup.sql](sql/master_schema_rollup.sql) for database schema
3. Explore [src/routes/](src/routes/) for API endpoints
4. Review [src/types.ts](src/types.ts) for TypeScript type definitions

## Common Environment Variables

Create these environment variables for development:

```powershell
# Database
$env:DB_HOST="localhost"
$env:DB_USER="postgres"
$env:DB_PASSWORD="postgres"
$env:DB_NAME="flyer_crawler_dev"
$env:DB_PORT="5432"

# Redis
$env:REDIS_URL="redis://localhost:6379"

# Application
$env:NODE_ENV="development"
$env:PORT="3001"
$env:FRONTEND_URL="http://localhost:5173"

# Authentication (generate your own secrets)
$env:JWT_SECRET="your-dev-jwt-secret-change-this"
$env:SESSION_SECRET="your-dev-session-secret-change-this"

# AI Services (get your own API keys)
$env:VITE_GOOGLE_GENAI_API_KEY="your-google-genai-api-key"
$env:GOOGLE_MAPS_API_KEY="your-google-maps-api-key"
```

## Resources

- [Podman Desktop Documentation](https://podman-desktop.io/docs)
- [PostGIS Documentation](https://postgis.net/documentation/)
- [Original README.md](README.md) for production setup