torbo/flyer-crawler.projectium.com
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

comprehensive documentation review + test fixes
Some checks failed
Deploy to Test Environment / deploy-to-test (push) Failing after 2m15s
Details

Browse Source
This commit is contained in:
Torben Sorensen
2026-01-28 16:35:38 -08:00
parent b6c3ca9abe
commit 45ac4fccf5
38 changed files with 2602 additions and 601 deletions
Download Patch File Download Diff File Expand all files Collapse all files

395
docs/getting-started/ENVIRONMENT.md
View File

@@ -2,134 +2,259 @@
Complete guide to environment variables used in Flyer Crawler.
---
## Quick Reference
### Minimum Required Variables (Development)
| Variable | Example | Purpose |
| ---------------- | ------------------------ | -------------------- |
| `DB_HOST` | `localhost` | PostgreSQL host |
| `DB_USER` | `postgres` | PostgreSQL username |
| `DB_PASSWORD` | `postgres` | PostgreSQL password |
| `DB_NAME` | `flyer_crawler_dev` | Database name |
| `REDIS_URL` | `redis://localhost:6379` | Redis connection URL |
| `JWT_SECRET` | (32+ character string) | JWT signing key |
| `GEMINI_API_KEY` | `AIzaSy...` | Google Gemini API |
### Source of Truth
The Zod schema at `src/config/env.ts` is the authoritative source for all environment variables. If a variable is not in this file, it is not used by the application.
---
## Configuration by Environment
### Production
**Location**: Gitea CI/CD secrets injected during deployment
**Path**: `/var/www/flyer-crawler.projectium.com/`
**Note**: No `.env` file exists - all variables come from CI/CD
| Aspect | Details |
| -------- | ------------------------------------------ |
| Location | Gitea CI/CD secrets injected at deployment |
| Path | `/var/www/flyer-crawler.projectium.com/` |
| File | No `.env` file - all from CI/CD secrets |
### Test
**Location**: Gitea CI/CD secrets + `.env.test` file
**Path**: `/var/www/flyer-crawler-test.projectium.com/`
**Note**: `.env.test` overrides for test-specific values
| Aspect | Details |
| -------- | --------------------------------------------- |
| Location | Gitea CI/CD secrets + `.env.test` overrides |
| Path | `/var/www/flyer-crawler-test.projectium.com/` |
| File | `.env.test` for test-specific values |
### Development Container
**Location**: `.env.local` file in project root
**Note**: Overrides default DSNs in `compose.dev.yml`
| Aspect | Details |
| -------- | --------------------------------------- |
| Location | `.env.local` file in project root |
| Priority | Overrides defaults in `compose.dev.yml` |
| File | `.env.local` (gitignored) |
## Required Variables
---
### Database
## Complete Variable Reference
| Variable | Description | Example |
| ------------------ | ---------------------------- | ------------------------------------------ |
| `DB_HOST` | PostgreSQL host | `localhost` (dev), `projectium.com` (prod) |
| `DB_PORT` | PostgreSQL port | `5432` |
| `DB_USER_PROD` | Production database user | `flyer_crawler_prod` |
| `DB_PASSWORD_PROD` | Production database password | (secret) |
| `DB_DATABASE_PROD` | Production database name | `flyer-crawler-prod` |
| `DB_USER_TEST` | Test database user | `flyer_crawler_test` |
| `DB_PASSWORD_TEST` | Test database password | (secret) |
| `DB_DATABASE_TEST` | Test database name | `flyer-crawler-test` |
| `DB_USER` | Dev database user | `postgres` |
| `DB_PASSWORD` | Dev database password | `postgres` |
| `DB_NAME` | Dev database name | `flyer_crawler_dev` |
### Database Configuration
**Note**: Production and test use separate `_PROD` and `_TEST` suffixed variables. Development uses unsuffixed variables.
| Variable | Required | Default | Description |
| ------------- | -------- | ------- | ----------------- |
| `DB_HOST` | Yes | - | PostgreSQL host |
| `DB_PORT` | No | `5432` | PostgreSQL port |
| `DB_USER` | Yes | - | Database username |
| `DB_PASSWORD` | Yes | - | Database password |
| `DB_NAME` | Yes | - | Database name |
### Redis
**Environment-Specific Variables** (Gitea Secrets):
| Variable | Description | Example |
| --------------------- | ------------------------- | ------------------------------ |
| `REDIS_URL` | Redis connection URL | `redis://localhost:6379` (dev) |
| `REDIS_PASSWORD_PROD` | Production Redis password | (secret) |
| `REDIS_PASSWORD_TEST` | Test Redis password | (secret) |
| Variable | Environment | Description |
| ------------------ | ----------- | ------------------------ |
| `DB_USER_PROD` | Production | Production database user |
| `DB_PASSWORD_PROD` | Production | Production database pass |
| `DB_DATABASE_PROD` | Production | Production database name |
| `DB_USER_TEST` | Test | Test database user |
| `DB_PASSWORD_TEST` | Test | Test database password |
| `DB_DATABASE_TEST` | Test | Test database name |
### Redis Configuration
| Variable | Required | Default | Description |
| ---------------- | -------- | ------- | ------------------------- |
| `REDIS_URL` | Yes | - | Redis connection URL |
| `REDIS_PASSWORD` | No | - | Redis password (optional) |
**URL Format**: `redis://[user:password@]host:port`
**Examples**:
```bash
# Development (no auth)
REDIS_URL=redis://localhost:6379
# Production (with auth)
REDIS_URL=redis://:${REDIS_PASSWORD_PROD}@localhost:6379
```
### Authentication
| Variable | Description | Example |
| ---------------------- | -------------------------- | -------------------------------- |
| `JWT_SECRET` | JWT token signing key | (minimum 32 characters) |
| `SESSION_SECRET` | Session encryption key | (minimum 32 characters) |
| `GOOGLE_CLIENT_ID` | Google OAuth client ID | `xxx.apps.googleusercontent.com` |
| `GOOGLE_CLIENT_SECRET` | Google OAuth client secret | (secret) |
| `GH_CLIENT_ID` | GitHub OAuth client ID | `xxx` |
| `GH_CLIENT_SECRET` | GitHub OAuth client secret | (secret) |
| Variable | Required | Min Length | Description |
| ---------------------- | -------- | ---------- | ----------------------- |
| `JWT_SECRET` | Yes | 32 chars | JWT token signing key |
| `JWT_SECRET_PREVIOUS` | No | - | Previous key (rotation) |
| `GOOGLE_CLIENT_ID` | No | - | Google OAuth client ID |
| `GOOGLE_CLIENT_SECRET` | No | - | Google OAuth secret |
| `GITHUB_CLIENT_ID` | No | - | GitHub OAuth client ID |
| `GITHUB_CLIENT_SECRET` | No | - | GitHub OAuth secret |
**Generate Secure Secret**:
```bash
node -e "console.log(require('crypto').randomBytes(32).toString('hex'))"
```
### AI Services
| Variable | Description | Example |
| -------------------------------- | ---------------------------- | ----------- |
| `VITE_GOOGLE_GENAI_API_KEY` | Google Gemini API key (prod) | `AIzaSy...` |
| `VITE_GOOGLE_GENAI_API_KEY_TEST` | Google Gemini API key (test) | `AIzaSy...` |
| `GOOGLE_MAPS_API_KEY` | Google Maps Geocoding API | `AIzaSy...` |
| Variable | Required | Description |
| ---------------------------- | -------- | -------------------------------- |
| `GEMINI_API_KEY` | Yes\* | Google Gemini API key |
| `GEMINI_RPM` | No | Rate limit (default: 5) |
| `AI_PRICE_QUALITY_THRESHOLD` | No | Quality threshold (default: 0.5) |
### Application
\*Required for flyer processing. Application works without it but cannot extract flyer data.
| Variable | Description | Example |
| -------------- | ------------------------ | ----------------------------------- |
| `NODE_ENV` | Environment mode | `development`, `test`, `production` |
| `PORT` | Backend server port | `3001` |
| `FRONTEND_URL` | Frontend application URL | `http://localhost:5173` (dev) |
**Get API Key**: [Google AI Studio](https://aistudio.google.com/app/apikey)
### Error Tracking
### Google Services
| Variable | Description | Example |
| ---------------------- | -------------------------------- | --------------------------- |
| `SENTRY_DSN` | Sentry DSN (production) | `https://xxx@sentry.io/xxx` |
| `VITE_SENTRY_DSN` | Frontend Sentry DSN (production) | `https://xxx@sentry.io/xxx` |
| `SENTRY_DSN_TEST` | Sentry DSN (test) | `https://xxx@sentry.io/xxx` |
| `VITE_SENTRY_DSN_TEST` | Frontend Sentry DSN (test) | `https://xxx@sentry.io/xxx` |
| `SENTRY_AUTH_TOKEN` | Sentry API token for releases | (secret) |
| Variable | Required | Description |
| ---------------------- | -------- | -------------------------------- |
| `GOOGLE_MAPS_API_KEY` | No | Google Maps Geocoding API |
| `GOOGLE_CLIENT_ID` | No | OAuth (see Authentication above) |
| `GOOGLE_CLIENT_SECRET` | No | OAuth (see Authentication above) |
## Optional Variables
### UPC Lookup APIs
| Variable | Description | Default |
| ------------------- | ----------------------- | ----------------- |
| `LOG_LEVEL` | Logging verbosity | `info` |
| `REDIS_TTL` | Cache TTL in seconds | `3600` |
| `MAX_UPLOAD_SIZE` | Max file upload size | `10mb` |
| `RATE_LIMIT_WINDOW` | Rate limit window (ms) | `900000` (15 min) |
| `RATE_LIMIT_MAX` | Max requests per window | `100` |
| Variable | Required | Description |
| ------------------------ | -------- | ---------------------- |
| `UPC_ITEM_DB_API_KEY` | No | UPC Item DB API key |
| `BARCODE_LOOKUP_API_KEY` | No | Barcode Lookup API key |
### Application Settings
| Variable | Required | Default | Description |
| -------------- | -------- | ------------- | ------------------------ |
| `NODE_ENV` | No | `development` | Environment mode |
| `PORT` | No | `3001` | Backend server port |
| `FRONTEND_URL` | No | - | Frontend URL (CORS) |
| `BASE_URL` | No | - | API base URL |
| `STORAGE_PATH` | No | (see below) | Flyer image storage path |
**NODE_ENV Values**: `development`, `test`, `staging`, `production`
**Default STORAGE_PATH**: `/var/www/flyer-crawler.projectium.com/flyer-images`
### Email/SMTP Configuration
| Variable | Required | Default | Description |
| ----------------- | -------- | ------- | ----------------------- |
| `SMTP_HOST` | No | - | SMTP server hostname |
| `SMTP_PORT` | No | `587` | SMTP server port |
| `SMTP_USER` | No | - | SMTP username |
| `SMTP_PASS` | No | - | SMTP password |
| `SMTP_SECURE` | No | `false` | Use TLS |
| `SMTP_FROM_EMAIL` | No | - | From address for emails |
**Note**: Email functionality degrades gracefully if not configured.
### Worker Configuration
| Variable | Default | Description |
| ------------------------------------- | ------- | ---------------------------- |
| `WORKER_CONCURRENCY` | `1` | Main worker concurrency |
| `WORKER_LOCK_DURATION` | `30000` | Lock duration (ms) |
| `EMAIL_WORKER_CONCURRENCY` | `10` | Email worker concurrency |
| `ANALYTICS_WORKER_CONCURRENCY` | `1` | Analytics worker concurrency |
| `CLEANUP_WORKER_CONCURRENCY` | `10` | Cleanup worker concurrency |
| `WEEKLY_ANALYTICS_WORKER_CONCURRENCY` | `1` | Weekly analytics concurrency |
### Error Tracking (Bugsink/Sentry)
| Variable | Required | Default | Description |
| --------------------- | -------- | -------- | ------------------------------- |
| `SENTRY_DSN` | No | - | Backend Sentry DSN |
| `SENTRY_ENABLED` | No | `true` | Enable error tracking |
| `SENTRY_ENVIRONMENT` | No | NODE_ENV | Environment name for errors |
| `SENTRY_DEBUG` | No | `false` | Enable Sentry SDK debug logging |
| `VITE_SENTRY_DSN` | No | - | Frontend Sentry DSN |
| `VITE_SENTRY_ENABLED` | No | `true` | Enable frontend error tracking |
| `VITE_SENTRY_DEBUG` | No | `false` | Frontend SDK debug logging |
**DSN Format**: `http://[key]@[host]:[port]/[project_id]`
**Dev Container DSNs**:
```bash
# Backend (internal)
SENTRY_DSN=http://<key>@localhost:8000/1
# Frontend (via nginx proxy)
VITE_SENTRY_DSN=https://<key>@localhost/bugsink-api/2
```
---
## Configuration Files
| File | Purpose |
| ------------------------------------- | ------------------------------------------- |
| `src/config/env.ts` | Zod schema validation - **source of truth** |
| `ecosystem.config.cjs` | PM2 process manager config |
| `ecosystem.config.cjs` | PM2 process manager (production) |
| `ecosystem.dev.config.cjs` | PM2 process manager (development) |
| `.gitea/workflows/deploy-to-prod.yml` | Production deployment workflow |
| `.gitea/workflows/deploy-to-test.yml` | Test deployment workflow |
| `.env.example` | Template with all variables |
| `.env.local` | Dev container overrides (not in git) |
| `.env.test` | Test environment overrides (not in git) |
---
## Adding New Variables
### 1. Update Zod Schema
### Checklist
1. [ ] **Update Zod Schema** - Edit `src/config/env.ts`
2. [ ] **Add to Gitea Secrets** - For prod/test environments
3. [ ] **Update Deployment Workflows** - `.gitea/workflows/*.yml`
4. [ ] **Update PM2 Config** - `ecosystem.config.cjs`
5. [ ] **Update .env.example** - Template for developers
6. [ ] **Update this document** - Add to appropriate section
### Step-by-Step
#### 1. Update Zod Schema
Edit `src/config/env.ts`:
```typescript
const envSchema = z.object({
// ... existing variables ...
NEW_VARIABLE: z.string().min(1),
newSection: z.object({
newVariable: z.string().min(1, 'NEW_VARIABLE is required'),
}),
});
// In loadEnvVars():
newSection: {
newVariable: process.env.NEW_VARIABLE,
},
```
### 2. Add to Gitea Secrets
For prod/test environments:
#### 2. Add to Gitea Secrets
1. Go to Gitea repository Settings > Secrets
2. Add `NEW_VARIABLE` with value
2. Add `NEW_VARIABLE` with production value
3. Add `NEW_VARIABLE_TEST` if test needs different value
### 3. Update Deployment Workflows
#### 3. Update Deployment Workflows
Edit `.gitea/workflows/deploy-to-prod.yml`:
@@ -145,7 +270,7 @@ env:
NEW_VARIABLE: ${{ secrets.NEW_VARIABLE_TEST }}
```
### 4. Update PM2 Config
#### 4. Update PM2 Config
Edit `ecosystem.config.cjs`:
@@ -161,31 +286,36 @@ module.exports = {
};
```
### 5. Update Documentation
- Add to `.env.example`
- Update this document
- Document in relevant feature docs
---
## Security Best Practices
### Secrets Management
### Do
- **NEVER** commit secrets to git
- Use Gitea Secrets for prod/test
- Use `.env.local` for dev (gitignored)
- Generate secrets with cryptographic randomness
- Rotate secrets regularly
- Use environment-specific database users
### Do Not
- Commit secrets to git
- Use short or predictable secrets
- Share secrets across environments
- Log sensitive values
### Secret Generation
```bash
# Generate secure random secrets
# Generate secure random secrets (64 hex characters)
node -e "console.log(require('crypto').randomBytes(32).toString('hex'))"
# Example output:
# a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4e5f6a1b2
```
### Database Users
Each environment has its own PostgreSQL user:
### Database Users by Environment
| Environment | User | Database |
| ----------- | -------------------- | -------------------- |
@@ -193,44 +323,61 @@ Each environment has its own PostgreSQL user:
| Test | `flyer_crawler_test` | `flyer-crawler-test` |
| Development | `postgres` | `flyer_crawler_dev` |
**Setup Commands** (as postgres superuser):
```sql
-- Production
CREATE DATABASE "flyer-crawler-prod";
CREATE USER flyer_crawler_prod WITH PASSWORD 'secure-password';
ALTER DATABASE "flyer-crawler-prod" OWNER TO flyer_crawler_prod;
\c "flyer-crawler-prod"
ALTER SCHEMA public OWNER TO flyer_crawler_prod;
GRANT CREATE, USAGE ON SCHEMA public TO flyer_crawler_prod;
CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
CREATE EXTENSION IF NOT EXISTS postgis;
CREATE EXTENSION IF NOT EXISTS pg_trgm;
-- Test (similar commands with _test suffix)
```
---
## Validation
Environment variables are validated at startup via `src/config/env.ts`. If validation fails:
Environment variables are validated at startup via `src/config/env.ts`.
1. Check the error message for missing/invalid variables
2. Verify `.env.local` (dev) or Gitea Secrets (prod/test)
3. Ensure values match schema requirements (min length, format, etc.)
### Startup Validation
If validation fails, you will see:
```text
╔════════════════════════════════════════════════════════════════╗
║ CONFIGURATION ERROR - APPLICATION STARTUP ║
╚════════════════════════════════════════════════════════════════╝
The following environment variables are missing or invalid:
- database.host: DB_HOST is required
- auth.jwtSecret: JWT_SECRET must be at least 32 characters
Please check your .env file or environment configuration.
```
### Debugging Configuration
```bash
# Check what variables are set (dev container)
podman exec flyer-crawler-dev env | grep -E "^(DB_|REDIS_|JWT_|SENTRY_)"
# Test database connection
podman exec flyer-crawler-postgres psql -U postgres -d flyer_crawler_dev -c "SELECT 1;"
# Test Redis connection
podman exec flyer-crawler-redis redis-cli ping
```
---
## Troubleshooting
### Variable Not Found
```
```text
Error: Missing required environment variable: JWT_SECRET
```
**Solution**: Add the variable to your environment configuration.
**Solutions**:
1. Check `.env.local` exists and has the variable
2. Verify variable name matches schema exactly
3. Restart the application after changes
### Invalid Value
```
```text
Error: JWT_SECRET must be at least 32 characters
```
@@ -240,32 +387,36 @@ Error: JWT_SECRET must be at least 32 characters
Check `NODE_ENV` is set correctly:
- `development` - Local dev container
- `test` - CI/CD test server
- `production` - Production server
| Value | Purpose |
| ------------- | ---------------------- |
| `development` | Local dev container |
| `test` | CI/CD test server |
| `staging` | Pre-production testing |
| `production` | Production server |
### Database Connection Issues
Verify database credentials:
```bash
# Development
podman exec flyer-crawler-postgres psql -U postgres -d flyer_crawler_dev -c "SELECT 1;"
# Production (via SSH)
ssh root@projectium.com "psql -U flyer_crawler_prod -d flyer-crawler-prod -c 'SELECT 1;'"
# If connection fails, check:
# 1. Container is running: podman ps
# 2. DB_HOST matches container network
# 3. DB_PASSWORD is correct
```
## Reference
---
- **Validation Schema**: [src/config/env.ts](../../src/config/env.ts)
- **Template**: [.env.example](../../.env.example)
- **Deployment Workflows**: [.gitea/workflows/](../../.gitea/workflows/)
- **PM2 Config**: [ecosystem.config.cjs](../../ecosystem.config.cjs)
## See Also
## Related Documentation
- [QUICKSTART.md](QUICKSTART.md) - Quick setup guide
- [INSTALL.md](INSTALL.md) - Detailed installation
- [DEV-CONTAINER.md](../development/DEV-CONTAINER.md) - Dev container setup
- [DEPLOYMENT.md](../operations/DEPLOYMENT.md) - Production deployment
- [AUTHENTICATION.md](../architecture/AUTHENTICATION.md) - OAuth setup
- [ADR-007](../adr/0007-configuration-and-secrets-management.md) - Configuration decisions
---
Last updated: January 2026

546
docs/getting-started/INSTALL.md
View File

@@ -1,203 +1,453 @@
# Installation Guide
This guide covers setting up a local development environment for Flyer Crawler.
Complete setup instructions for the Flyer Crawler local development environment.
---
## Quick Reference
| Setup Method | Best For | Time | Document Section |
| ----------------- | --------------------------- | ------ | --------------------------------------------------- |
| Quick Start | Already have Postgres/Redis | 5 min | [Quick Start](#quick-start) |
| Dev Container | Full production-like setup | 15 min | [Dev Container](#development-container-recommended) |
| Manual Containers | Learning the components | 20 min | [Podman Setup](#podman-setup-manual) |
---
## Prerequisites
- Node.js 20.x or later
- Access to a PostgreSQL database (local or remote)
- Redis instance (for session management)
- Google Gemini API key
- Google Maps API key (for geocoding)
### Required Software
| Software | Minimum Version | Purpose | Download |
| -------------- | --------------- | -------------------- | ----------------------------------------------- |
| Node.js | 20.x | Runtime | [nodejs.org](https://nodejs.org/) |
| Podman Desktop | 4.x | Container management | [podman-desktop.io](https://podman-desktop.io/) |
| Git | 2.x | Version control | [git-scm.com](https://git-scm.com/) |
### Windows-Specific Requirements
| Requirement | Purpose | Setup Command |
| ----------- | ------------------------------ | ---------------------------------- |
| WSL 2 | Linux compatibility for Podman | `wsl --install` (admin PowerShell) |
### Verify Installation
```bash
# Check all prerequisites
node --version # Expected: v20.x or higher
podman --version # Expected: podman version 4.x or higher
git --version # Expected: git version 2.x or higher
wsl --list -v # Expected: Shows WSL 2 distro
```
---
## Quick Start
If you already have PostgreSQL and Redis configured:
If you already have PostgreSQL and Redis configured externally:
```bash
# Install dependencies
# 1. Clone the repository
git clone https://gitea.projectium.com/flyer-crawler/flyer-crawler.git
cd flyer-crawler
# 2. Install dependencies
npm install
# Run in development mode
# 3. Create .env.local (see Environment section below)
# 4. Run in development mode
npm run dev
```
**Access Points**:
- Frontend: `http://localhost:5173`
- Backend API: `http://localhost:3001`
---
## Development Environment with Podman (Recommended for Windows)
## Development Container (Recommended)
This approach uses Podman with an Ubuntu container for a consistent development environment.
The dev container provides a complete, production-like environment.
### What's Included
| Service | Purpose | Port |
| ---------- | ------------------------ | ---------- |
| Node.js | API server, worker, Vite | 3001, 5173 |
| PostgreSQL | Database with PostGIS | 5432 |
| Redis | Cache and job queues | 6379 |
| NGINX | HTTPS reverse proxy | 443 |
| Bugsink | Error tracking | 8443 |
| Logstash | Log aggregation | - |
| PM2 | Process management | - |
### Setup Steps
#### Step 1: Initialize Podman
```bash
# Windows: Start Podman Desktop, or from terminal:
podman machine init
podman machine start
```
#### Step 2: Start Dev Container
```bash
# Start all services
podman-compose -f compose.dev.yml up -d
# View logs (optional)
podman-compose -f compose.dev.yml logs -f
```
**Expected Output**:
```text
[+] Running 3/3
- Container flyer-crawler-postgres Started
- Container flyer-crawler-redis Started
- Container flyer-crawler-dev Started
```
#### Step 3: Verify Services
```bash
# Check containers are running
podman ps
# Check PM2 processes
podman exec -it flyer-crawler-dev pm2 status
```
**Expected PM2 Status**:
```text
+---------------------------+--------+-------+
| name | status | cpu |
+---------------------------+--------+-------+
| flyer-crawler-api-dev | online | 0% |
| flyer-crawler-worker-dev | online | 0% |
| flyer-crawler-vite-dev | online | 0% |
+---------------------------+--------+-------+
```
#### Step 4: Access Application
| Service | URL | Notes |
| ----------- | ------------------------ | ---------------------------- |
| Frontend | `https://localhost` | NGINX proxies to Vite |
| Backend API | `http://localhost:3001` | Express server |
| Bugsink | `https://localhost:8443` | Login: admin@localhost/admin |
### SSL Certificate Setup (Optional but Recommended)
To eliminate browser security warnings:
**Windows**:
1. Double-click `certs/mkcert-ca.crt`
2. Click "Install Certificate..."
3. Select "Local Machine" > Next
4. Select "Place all certificates in the following store"
5. Browse > Select "Trusted Root Certification Authorities" > OK
6. Click Next > Finish
7. Restart browser
**Other Platforms**: See [`certs/README.md`](../../certs/README.md)
### Managing the Dev Container
| Action | Command |
| --------- | ------------------------------------------- |
| Start | `podman-compose -f compose.dev.yml up -d` |
| Stop | `podman-compose -f compose.dev.yml down` |
| View logs | `podman-compose -f compose.dev.yml logs -f` |
| Restart | `podman-compose -f compose.dev.yml restart` |
| Rebuild | `podman-compose -f compose.dev.yml build` |
---
## Podman Setup (Manual)
For understanding the individual components or custom configurations.
### Step 1: Install Prerequisites on Windows
1. **Install WSL 2**: Podman on Windows relies on the Windows Subsystem for Linux.
```powershell
# Run in administrator PowerShell
wsl --install
```
```powershell
wsl --install
```
Restart computer after WSL installation.
Run this in an administrator PowerShell.
### Step 2: Initialize Podman
2. **Install Podman Desktop**: Download and install [Podman Desktop for Windows](https://podman-desktop.io/).
1. Launch **Podman Desktop**
2. Follow the setup wizard to initialize Podman machine
3. Start the Podman machine
### Step 2: Set Up Podman
1. **Initialize Podman**: Launch Podman Desktop. It will automatically set up its WSL 2 machine.
2. **Start Podman**: Ensure the Podman machine is running from the Podman Desktop interface.
### Step 3: Set Up the Ubuntu Container
1. **Pull Ubuntu Image**:
```bash
podman pull ubuntu:latest
```
2. **Create a Podman Volume** (persists node_modules between container restarts):
```bash
podman volume create node_modules_cache
```
3. **Run the Ubuntu Container**:
Open a terminal in your project's root directory and run:
```bash
podman run -it -p 3001:3001 -p 5173:5173 --name flyer-dev \
-v "$(pwd):/app" \
-v "node_modules_cache:/app/node_modules" \
ubuntu:latest
```
| Flag | Purpose |
| ------------------------------------------- | ------------------------------------------------ |
| `-p 3001:3001` | Forwards the backend server port |
| `-p 5173:5173` | Forwards the Vite frontend server port |
| `--name flyer-dev` | Names the container for easy reference |
| `-v "...:/app"` | Mounts your project directory into the container |
| `-v "node_modules_cache:/app/node_modules"` | Mounts the named volume for node_modules |
### Step 4: Configure the Ubuntu Environment
You are now inside the Ubuntu container's shell.
1. **Update Package Lists**:
```bash
apt-get update
```
2. **Install Dependencies**:
```bash
apt-get install -y curl git
curl -sL https://deb.nodesource.com/setup_20.x | bash -
apt-get install -y nodejs
```
3. **Navigate to Project Directory**:
```bash
cd /app
```
4. **Install Project Dependencies**:
```bash
npm install
```
### Step 5: Run the Development Server
Or from terminal:
```bash
podman machine init
podman machine start
```
### Step 3: Create Podman Network
```bash
podman network create flyer-crawler-net
```
### Step 4: Create PostgreSQL Container
```bash
podman run -d \
--name flyer-crawler-postgres \
--network flyer-crawler-net \
-e POSTGRES_USER=postgres \
-e POSTGRES_PASSWORD=postgres \
-e POSTGRES_DB=flyer_crawler_dev \
-p 5432:5432 \
-v flyer-crawler-pgdata:/var/lib/postgresql/data \
docker.io/postgis/postgis:15-3.3
```
### Step 5: Create Redis Container
```bash
podman run -d \
--name flyer-crawler-redis \
--network flyer-crawler-net \
-p 6379:6379 \
-v flyer-crawler-redis:/data \
docker.io/library/redis:alpine
```
### Step 6: Initialize Database
```bash
# Wait for PostgreSQL to be ready
podman exec flyer-crawler-postgres pg_isready -U postgres
# Install required extensions
podman exec flyer-crawler-postgres psql -U postgres -d flyer_crawler_dev -c "
CREATE EXTENSION IF NOT EXISTS postgis;
CREATE EXTENSION IF NOT EXISTS pg_trgm;
CREATE EXTENSION IF NOT EXISTS \"uuid-ossp\";
"
# Apply schema
podman exec -i flyer-crawler-postgres psql -U postgres -d flyer_crawler_dev < sql/master_schema_rollup.sql
```
### Step 7: Create Node.js Container
```bash
# Create volume for node_modules
podman volume create node_modules_cache
# Run Ubuntu container with project mounted
podman run -it \
--name flyer-dev \
--network flyer-crawler-net \
-p 3001:3001 \
-p 5173:5173 \
-v "$(pwd):/app" \
-v "node_modules_cache:/app/node_modules" \
ubuntu:latest
```
### Step 8: Configure Container Environment
Inside the container:
```bash
# Update and install dependencies
apt-get update
apt-get install -y curl git
# Install Node.js 20
curl -sL https://deb.nodesource.com/setup_20.x | bash -
apt-get install -y nodejs
# Navigate to project and install
cd /app
npm install
# Start development server
npm run dev
```
### Step 6: Access the Application
### Container Management Commands
- **Frontend**: http://localhost:5173
- **Backend API**: http://localhost:3001
### Dev Container with HTTPS (Full Stack)
When using the full dev container stack with NGINX (via `compose.dev.yml`), access the application over HTTPS:
- **Frontend**: https://localhost or https://127.0.0.1
- **Backend API**: http://localhost:3001
**SSL Certificate Notes:**
- The dev container uses self-signed certificates generated by mkcert
- Both `localhost` and `127.0.0.1` are valid hostnames (certificate includes both as SANs)
- If images fail to load with SSL errors, see [FLYER-URL-CONFIGURATION.md](../FLYER-URL-CONFIGURATION.md#ssl-certificate-configuration-dev-container)
**Eliminate SSL Warnings (Recommended):**
To avoid browser security warnings for self-signed certificates, install the mkcert CA certificate on your system. The CA certificate is located at `certs/mkcert-ca.crt` in the project root.
See [`certs/README.md`](../../certs/README.md) for platform-specific installation instructions (Windows, macOS, Linux, Firefox).
After installation:
- Your browser will trust all mkcert certificates without warnings
- Both `https://localhost/` and `https://127.0.0.1/` will work without SSL errors
- Flyer images will load without `ERR_CERT_AUTHORITY_INVALID` errors
### Managing the Container
| Action | Command |
| --------------------- | -------------------------------- |
| Stop the container | Press `Ctrl+C`, then type `exit` |
| Restart the container | `podman start -a -i flyer-dev` |
| Remove the container | `podman rm flyer-dev` |
| Action | Command |
| -------------- | ------------------------------ |
| Stop container | Press `Ctrl+C`, then `exit` |
| Restart | `podman start -a -i flyer-dev` |
| Remove | `podman rm flyer-dev` |
| List running | `podman ps` |
| List all | `podman ps -a` |
---
## Environment Variables
## Environment Configuration
This project is configured to run in a CI/CD environment and does not use `.env` files. All configuration must be provided as environment variables.
### Create .env.local
For local development, you can export these in your shell or use your IDE's environment configuration:
Create `.env.local` in the project root with your configuration:
| Variable | Description |
| --------------------------- | ------------------------------------- |
| `DB_HOST` | PostgreSQL server hostname |
| `DB_USER` | PostgreSQL username |
| `DB_PASSWORD` | PostgreSQL password |
| `DB_DATABASE_PROD` | Production database name |
| `JWT_SECRET` | Secret string for signing auth tokens |
| `VITE_GOOGLE_GENAI_API_KEY` | Google Gemini API key |
| `GOOGLE_MAPS_API_KEY` | Google Maps Geocoding API key |
| `REDIS_PASSWORD_PROD` | Production Redis password |
| `REDIS_PASSWORD_TEST` | Test Redis password |
```bash
# Database (adjust host based on your setup)
DB_HOST=localhost # Use 'postgres' if inside dev container
DB_PORT=5432
DB_USER=postgres
DB_PASSWORD=postgres
DB_NAME=flyer_crawler_dev
# Redis (adjust host based on your setup)
REDIS_URL=redis://localhost:6379 # Use 'redis://redis:6379' inside container
# Application
NODE_ENV=development
PORT=3001
FRONTEND_URL=http://localhost:5173
# Authentication (generate secure values)
JWT_SECRET=your-secret-at-least-32-characters-long
# AI Services
GEMINI_API_KEY=your-google-gemini-api-key
GOOGLE_MAPS_API_KEY=your-google-maps-api-key # Optional
```
**Generate Secure Secrets**:
```bash
node -e "console.log(require('crypto').randomBytes(32).toString('hex'))"
```
### Environment Differences
| Variable | Host Development | Inside Dev Container |
| ----------- | ------------------------ | -------------------- |
| `DB_HOST` | `localhost` | `postgres` |
| `REDIS_URL` | `redis://localhost:6379` | `redis://redis:6379` |
See [ENVIRONMENT.md](ENVIRONMENT.md) for complete variable reference.
---
## Seeding Development Data
To create initial test accounts (`admin@example.com` and `user@example.com`) and sample data:
Create test accounts and sample data:
```bash
npm run seed
```
The seed script performs the following actions:
### What the Seed Script Does
1. Rebuilds the database schema from `sql/master_schema_rollup.sql`
2. Creates test user accounts (admin and regular user)
3. Copies test flyer images from `src/tests/assets/` to `public/flyer-images/`
4. Creates a sample flyer with items linked to the test images
5. Seeds watched items and a shopping list for the test user
1. Rebuilds database schema from `sql/master_schema_rollup.sql`
2. Creates test user accounts:
- `admin@example.com` (admin user)
- `user@example.com` (regular user)
3. Copies test flyer images to `public/flyer-images/`
4. Creates sample flyer with items
5. Seeds watched items and shopping list
**Test Images**: The seed script copies `test-flyer-image.jpg` and `test-flyer-icon.png` to the `public/flyer-images/` directory, which is served by NGINX at `/flyer-images/`.
### Test Images
After running, you may need to restart your IDE's TypeScript server to pick up any generated types.
The seed script copies these files from `src/tests/assets/`:
- `test-flyer-image.jpg`
- `test-flyer-icon.png`
Images are served by NGINX at `/flyer-images/`.
---
## Verification Checklist
After installation, verify everything works:
- [ ] **Containers running**: `podman ps` shows postgres and redis
- [ ] **Database accessible**: `podman exec flyer-crawler-postgres psql -U postgres -c "SELECT 1;"`
- [ ] **Frontend loads**: Open `http://localhost:5173` (or `https://localhost` for dev container)
- [ ] **API responds**: `curl http://localhost:3001/health`
- [ ] **Tests pass**: `npm run test:unit` (or in container: `podman exec -it flyer-crawler-dev npm run test:unit`)
- [ ] **Type check passes**: `npm run type-check`
---
## Troubleshooting
### Podman Machine Won't Start
```bash
# Reset Podman machine
podman machine rm
podman machine init
podman machine start
```
### Port Already in Use
```bash
# Find process using port
netstat -ano | findstr :5432
# Option: Use different port
podman run -d --name flyer-crawler-postgres -p 5433:5432 ...
# Then set DB_PORT=5433 in .env.local
```
### Database Extensions Missing
```bash
podman exec flyer-crawler-postgres psql -U postgres -d flyer_crawler_dev -c "
CREATE EXTENSION IF NOT EXISTS postgis;
CREATE EXTENSION IF NOT EXISTS pg_trgm;
CREATE EXTENSION IF NOT EXISTS \"uuid-ossp\";
"
```
### Permission Denied on Windows Paths
Use `MSYS_NO_PATHCONV=1` prefix:
```bash
MSYS_NO_PATHCONV=1 podman exec flyer-crawler-dev /path/to/script.sh
```
### Tests Fail with Timezone Errors
Tests must run in the dev container, not on Windows host:
```bash
# CORRECT
podman exec -it flyer-crawler-dev npm test
# INCORRECT (may fail with TZ errors)
npm test
```
---
## Next Steps
- [Database Setup](DATABASE.md) - Set up PostgreSQL with required extensions
- [Authentication Setup](AUTHENTICATION.md) - Configure OAuth providers
- [Deployment Guide](DEPLOYMENT.md) - Deploy to production
| Goal | Document |
| --------------------- | ------------------------------------------------------ |
| Quick setup guide | [QUICKSTART.md](QUICKSTART.md) |
| Environment variables | [ENVIRONMENT.md](ENVIRONMENT.md) |
| Database schema | [DATABASE.md](../architecture/DATABASE.md) |
| Authentication setup | [AUTHENTICATION.md](../architecture/AUTHENTICATION.md) |
| Dev container details | [DEV-CONTAINER.md](../development/DEV-CONTAINER.md) |
| Deployment | [DEPLOYMENT.md](../operations/DEPLOYMENT.md) |
---
Last updated: January 2026

251
docs/getting-started/QUICKSTART.md
View File

@@ -2,13 +2,38 @@
Get Flyer Crawler running in 5 minutes.
## Prerequisites
---
- **Windows 10/11** with WSL 2
- **Podman Desktop** installed
- **Node.js 20+** installed
## Prerequisites Checklist
## 1. Start Containers (1 minute)
Before starting, verify you have:
- [ ] **Windows 10/11** with WSL 2 enabled
- [ ] **Podman Desktop** installed ([download](https://podman-desktop.io/))
- [ ] **Node.js 20+** installed
- [ ] **Git** for cloning the repository
**Verify Prerequisites**:
```bash
# Check Podman
podman --version
# Expected: podman version 4.x or higher
# Check Node.js
node --version
# Expected: v20.x or higher
# Check WSL
wsl --list --verbose
# Expected: Shows WSL 2 distro
```
---
## Quick Setup (5 Steps)
### Step 1: Start Containers (1 minute)
```bash
# Start PostgreSQL and Redis
@@ -27,11 +52,18 @@ podman run -d --name flyer-crawler-redis \
docker.io/library/redis:alpine
```
## 2. Initialize Database (2 minutes)
**Expected Output**:
```text
# Container IDs displayed, no errors
```
### Step 2: Initialize Database (2 minutes)
```bash
# Wait for PostgreSQL to be ready
podman exec flyer-crawler-postgres pg_isready -U postgres
# Expected: localhost:5432 - accepting connections
# Install extensions
podman exec flyer-crawler-postgres psql -U postgres -d flyer_crawler_dev \
@@ -41,7 +73,17 @@ podman exec flyer-crawler-postgres psql -U postgres -d flyer_crawler_dev \
podman exec -i flyer-crawler-postgres psql -U postgres -d flyer_crawler_dev < sql/master_schema_rollup.sql
```
## 3. Configure Environment (1 minute)
**Expected Output**:
```text
CREATE EXTENSION
CREATE EXTENSION
CREATE EXTENSION
CREATE TABLE
... (many tables created)
```
### Step 3: Configure Environment (1 minute)
Create `.env.local` in the project root:
@@ -61,16 +103,22 @@ NODE_ENV=development
PORT=3001
FRONTEND_URL=http://localhost:5173
# Secrets (generate your own)
# Secrets (generate your own - see command below)
JWT_SECRET=your-dev-jwt-secret-at-least-32-chars-long
SESSION_SECRET=your-dev-session-secret-at-least-32-chars-long
# AI Services (get your own keys)
VITE_GOOGLE_GENAI_API_KEY=your-google-genai-api-key
GEMINI_API_KEY=your-google-gemini-api-key
GOOGLE_MAPS_API_KEY=your-google-maps-api-key
```
## 4. Install & Run (1 minute)
**Generate Secure Secrets**:
```bash
node -e "console.log(require('crypto').randomBytes(32).toString('hex'))"
```
### Step 4: Install and Run (1 minute)
```bash
# Install dependencies (first time only)
@@ -80,35 +128,61 @@ npm install
npm run dev
```
## 5. Access Application
**Expected Output**:
- **Frontend**: http://localhost:5173
- **Backend API**: http://localhost:3001
- **Health Check**: http://localhost:3001/health
```text
> flyer-crawler@x.x.x dev
> concurrently ...
### Dev Container (HTTPS)
[API] Server listening on port 3001
[Vite] VITE ready at http://localhost:5173
```
When using the full dev container with NGINX, access via HTTPS:
### Step 5: Verify Installation
- **Frontend**: https://localhost or https://127.0.0.1
- **Backend API**: http://localhost:3001
- **Bugsink**: `https://localhost:8443` (error tracking)
| Check | URL/Command | Expected Result |
| ----------- | ------------------------------ | ----------------------------------- |
| Frontend | `http://localhost:5173` | Flyer Crawler app loads |
| Backend API | `http://localhost:3001/health` | `{ "status": "ok", ... }` |
| Database | `podman exec ... psql -c ...` | `SELECT version()` returns Postgres |
| Containers | `podman ps` | Shows postgres and redis running |
**Note:** The dev container accepts both `localhost` and `127.0.0.1` for HTTPS connections. The self-signed certificate is valid for both hostnames.
---
**SSL Certificate Warnings:** To eliminate browser security warnings for self-signed certificates, install the mkcert CA certificate. See [`certs/README.md`](../../certs/README.md) for platform-specific installation instructions. This is optional but recommended for a better development experience.
## Full Dev Container (Recommended)
### Dev Container Architecture
For a production-like environment with NGINX, Bugsink error tracking, and PM2 process management:
The dev container uses PM2 for process management, matching production (ADR-014):
### Starting the Dev Container
| Process | Description | Port |
| -------------------------- | ------------------------ | ---- |
| `flyer-crawler-api-dev` | API server (tsx watch) | 3001 |
| `flyer-crawler-worker-dev` | Background job worker | - |
| `flyer-crawler-vite-dev` | Vite frontend dev server | 5173 |
```bash
# Start all services
podman-compose -f compose.dev.yml up -d
**PM2 Commands** (run inside container):
# View logs
podman-compose -f compose.dev.yml logs -f
```
### Access Points
| Service | URL | Notes |
| ----------- | ------------------------ | ---------------------------- |
| Frontend | `https://localhost` | NGINX proxy to Vite |
| Backend API | `http://localhost:3001` | Express server |
| Bugsink | `https://localhost:8443` | Error tracking (admin/admin) |
| PostgreSQL | `localhost:5432` | Database |
| Redis | `localhost:6379` | Cache |
**SSL Certificate Setup (Recommended)**:
To eliminate browser security warnings, install the mkcert CA certificate:
```bash
# Windows: Double-click certs/mkcert-ca.crt and install to Trusted Root CAs
# See certs/README.md for detailed instructions per platform
```
### PM2 Commands
```bash
# View process status
@@ -124,63 +198,152 @@ podman exec -it flyer-crawler-dev pm2 restart all
podman exec -it flyer-crawler-dev pm2 restart flyer-crawler-api-dev
```
## Verify Installation
### Dev Container Processes
| Process | Description | Port |
| -------------------------- | ------------------------ | ---- |
| `flyer-crawler-api-dev` | API server (tsx watch) | 3001 |
| `flyer-crawler-worker-dev` | Background job worker | - |
| `flyer-crawler-vite-dev` | Vite frontend dev server | 5173 |
---
## Verification Commands
Run these to confirm everything is working:
```bash
# Check containers are running
podman ps
# Expected: flyer-crawler-postgres and flyer-crawler-redis both running
# Test database connection
podman exec flyer-crawler-postgres psql -U postgres -d flyer_crawler_dev -c "SELECT version();"
# Expected: PostgreSQL 15.x with PostGIS
# Run tests (in dev container)
podman exec -it flyer-crawler-dev npm run test:unit
# Expected: All tests pass
# Run type check
podman exec -it flyer-crawler-dev npm run type-check
# Expected: No type errors
```
## Common Issues
---
## Common Issues and Solutions
### "Unable to connect to Podman socket"
**Cause**: Podman machine not running
**Solution**:
```bash
podman machine start
```
### "Connection refused" to PostgreSQL
Wait a few seconds for PostgreSQL to initialize:
**Cause**: PostgreSQL still initializing
**Solution**:
```bash
# Wait for PostgreSQL to be ready
podman exec flyer-crawler-postgres pg_isready -U postgres
# Retry after "accepting connections" message
```
### Port 5432 or 6379 already in use
Stop conflicting services or change port mappings:
**Cause**: Another service using the port
**Solution**:
```bash
# Use different host port
# Option 1: Stop conflicting service
# Option 2: Use different host port
podman run -d --name flyer-crawler-postgres -p 5433:5432 ...
# Then update DB_PORT=5433 in .env.local
```
Then update `DB_PORT=5433` in `.env.local`.
### "JWT_SECRET must be at least 32 characters"
**Cause**: Secret too short in .env.local
**Solution**: Generate a longer secret:
```bash
node -e "console.log(require('crypto').randomBytes(32).toString('hex'))"
```
### Tests fail with "TZ environment variable" errors
**Cause**: Timezone setting interfering with Node.js async hooks
**Solution**: Tests must run in dev container (not Windows host):
```bash
# CORRECT - run in container
podman exec -it flyer-crawler-dev npm test
# INCORRECT - do not run on Windows host
npm test
```
---
## Next Steps
- **Read the docs**: [docs/README.md](../README.md)
- **Understand the architecture**: [docs/architecture/DATABASE.md](../architecture/DATABASE.md)
- **Learn testing**: [docs/development/TESTING.md](../development/TESTING.md)
- **Explore ADRs**: [docs/adr/index.md](../adr/index.md)
- **Contributing**: [CONTRIBUTING.md](../../CONTRIBUTING.md)
| Goal | Document |
| ----------------------- | ----------------------------------------------------- |
| Understand the codebase | [Architecture Overview](../architecture/OVERVIEW.md) |
| Configure environment | [Environment Variables](ENVIRONMENT.md) |
| Set up MCP tools | [MCP Configuration](../tools/MCP-CONFIGURATION.md) |
| Learn testing | [Testing Guide](../development/TESTING.md) |
| Understand DB schema | [Database Documentation](../architecture/DATABASE.md) |
| Read ADRs | [ADR Index](../adr/index.md) |
| Full installation guide | [Installation Guide](INSTALL.md) |
## Development Workflow
---
## Daily Development Workflow
```bash
# Daily workflow
# 1. Start containers
podman start flyer-crawler-postgres flyer-crawler-redis
# 2. Start dev server
npm run dev
# ... make changes ...
# 3. Make changes and test
npm test
# 4. Type check before commit
npm run type-check
# 5. Commit changes
git commit
```
For detailed setup instructions, see [INSTALL.md](INSTALL.md).
**For dev container users**:
```bash
# 1. Start dev container
podman-compose -f compose.dev.yml up -d
# 2. View logs
podman exec -it flyer-crawler-dev pm2 logs
# 3. Run tests
podman exec -it flyer-crawler-dev npm test
# 4. Stop when done
podman-compose -f compose.dev.yml down
```
---
Last updated: January 2026