flyer-crawler.projectium.com/docs/SUBAGENT-DEVOPS-REFERENCE.md

DevOps Subagent Reference

Critical Rule: Server Access is READ-ONLY

Claude Code has READ-ONLY access to production/test servers. The claude-win10 user cannot execute write operations directly.

When working with production/test servers:

1. Provide commands for the user to execute (do not attempt SSH)
2. Wait for user to report command output
3. Provide fix commands 1-3 at a time (errors may cascade)
4. Verify success with read-only commands after user executes fixes
5. Document findings in relevant documentation

Commands in this reference are for the user to run on the server, not for Claude to execute.

---

Critical Rule: Git Bash Path Conversion

Git Bash on Windows auto-converts Unix paths, breaking container commands.

Solution	Example
sh -c with single quotes	podman exec container sh -c '/usr/local/bin/script.sh'
Double slashes	podman exec container //usr//local//bin//script.sh
MSYS_NO_PATHCONV=1	MSYS_NO_PATHCONV=1 podman exec ...
Windows paths for host files	podman cp "d:/path/file" container:/tmp/file

---

Container Commands (Podman)

Dev Container Operations

# List running containers
podman ps

# Container logs
podman logs flyer-crawler-dev
podman logs -f flyer-crawler-dev  # Follow

# Execute in container
podman exec -it flyer-crawler-dev bash
podman exec -it flyer-crawler-dev npm run test:unit

# Restart container
podman restart flyer-crawler-dev

# Container resource usage
podman stats flyer-crawler-dev

Test Execution (from Windows)

# Unit tests - pipe output for AI processing
podman exec -it flyer-crawler-dev npm run test:unit 2>&1 | tee test-results.txt

# Integration tests
podman exec -it flyer-crawler-dev npm run test:integration

# Type check (CRITICAL before commit)
podman exec -it flyer-crawler-dev npm run type-check

# Specific test file
podman exec -it flyer-crawler-dev npm test -- --run src/hooks/useAuth.test.tsx

Database Operations (from Windows)

# Reset dev database
podman exec -it flyer-crawler-dev npm run db:reset:dev

# Access PostgreSQL
podman exec -it flyer-crawler-dev psql -U postgres -d flyer_crawler_dev

# Run SQL file (use MSYS_NO_PATHCONV to avoid path conversion)
MSYS_NO_PATHCONV=1 podman exec -it flyer-crawler-dev psql -U postgres -d flyer_crawler_dev -f /app/sql/master_schema_rollup.sql

---

PM2 Commands

Production Server

Note

: These commands are for the user to execute on the server. Claude Code provides commands but cannot run them directly. See Server Access is READ-ONLY above.

# List all apps
pm2 list

# App status
pm2 show flyer-crawler-api

# Logs
pm2 logs flyer-crawler-api
pm2 logs --lines 100

# Restart apps
pm2 restart flyer-crawler-api
pm2 reload flyer-crawler-api  # Zero-downtime

# Stop/Start
pm2 stop flyer-crawler-api
pm2 start flyer-crawler-api

# Delete and reload from config
pm2 delete all
pm2 start ecosystem.config.cjs

PM2 Config: ecosystem.config.cjs

App	Purpose	Memory	Mode
flyer-crawler-api	Express server	500M	Cluster
flyer-crawler-worker	BullMQ worker	1G	Fork
flyer-crawler-analytics-worker	Analytics worker	1G	Fork

Test variants: *-test suffix

---

CI/CD Workflows

Location: .gitea/workflows/

Workflow	Trigger	Purpose
deploy-to-test.yml	Push to main	Auto-deploy to test env
deploy-to-prod.yml	Manual	Deploy to production
manual-db-backup.yml	Manual	Database backup
manual-db-reset-test.yml	Manual	Reset test database
manual-db-reset-prod.yml	Manual	Reset prod database
manual-db-restore.yml	Manual	Restore database
manual-deploy-major.yml	Manual	Major version release
manual-redis-flush-prod.yml	Manual	Flush Redis cache

Deploy to Test Pipeline Steps

1. Checkout code
2. Setup Node.js 20
3. npm ci
4. Bump patch version (creates git tag)
5. npm run type-check
6. npm run test:unit
7. Check schema hash against deployed DB
8. npm run build
9. Copy files to /var/www/flyer-crawler-test.projectium.com/
10. pm2 reload ecosystem-test.config.cjs

Deploy to Production Pipeline Steps

1. Verify confirmation phrase ("deploy-to-prod")
2. Checkout main branch
3. npm ci
4. Bump minor version (creates git tag)
5. Check schema hash against prod DB
6. npm run build (with Sentry source maps)
7. Copy files to /var/www/flyer-crawler.projectium.com/
8. pm2 reload ecosystem.config.cjs

---

Deployment Paths

Environment	Path	Domain
Production	/var/www/flyer-crawler.projectium.com/	flyer-crawler.projectium.com
Test	/var/www/flyer-crawler-test.projectium.com/	flyer-crawler-test.projectium.com
Dev Container	/app/	localhost:3000

---

Environment Configuration

Files

File	Purpose
ecosystem.config.cjs	PM2 production config
ecosystem-test.config.cjs	PM2 test config
src/config/env.ts	Zod schema for env vars
.env.example	Template for env vars

Required Secrets (Gitea CI/CD)

Category	Secrets
Database	DB_HOST, DB_USER_PROD, DB_PASSWORD_PROD, DB_DATABASE_PROD
Test DB	DB_USER_TEST, DB_PASSWORD_TEST, DB_DATABASE_TEST
Redis	REDIS_PASSWORD_PROD, REDIS_PASSWORD_TEST
Auth	JWT_SECRET, GOOGLE_CLIENT_ID, GOOGLE_CLIENT_SECRET, GH_CLIENT_ID, GH_CLIENT_SECRET
AI	VITE_GOOGLE_GENAI_API_KEY, VITE_GOOGLE_GENAI_API_KEY_TEST
Monitoring	SENTRY_DSN, SENTRY_DSN_TEST, SENTRY_AUTH_TOKEN
Maps	GOOGLE_MAPS_API_KEY

Adding New Secret

1. Add to Gitea Settings > Secrets
2. Update workflow YAML: SENTRY_DSN: ${{ secrets.SENTRY_DSN }}
3. Update ecosystem.config.cjs
4. Update src/config/env.ts Zod schema
5. Update .env.example

---

Redis Commands

Dev Container

# Access Redis CLI
podman exec -it flyer-crawler-dev redis-cli

# Common commands
KEYS *
FLUSHALL
INFO

Production

Note

: User executes these commands on the server.

# Access Redis CLI
redis-cli -a $REDIS_PASSWORD

# Flush cache (use with caution)
# Or use manual-redis-flush-prod.yml workflow

---

Health Checks

Endpoints

Endpoint	Purpose
GET /api/health	Basic health check
GET /api/health/detailed	Full system status (DB, Redis, queues)

Manual Health Check

# From Windows
curl http://localhost:3000/api/health

# Or via Podman
podman exec -it flyer-crawler-dev curl http://localhost:3000/api/health

---

Log Locations

Production Server

# PM2 logs
~/.pm2/logs/

# NGINX logs
/var/log/nginx/access.log
/var/log/nginx/error.log

# Application logs (via PM2)
pm2 logs flyer-crawler-api --lines 200

Dev Container

# View container logs
podman logs flyer-crawler-dev

# Follow logs
podman logs -f flyer-crawler-dev

---

Backup/Restore

Database Backup (Manual Workflow)

Trigger manual-db-backup.yml from Gitea Actions UI.

Manual Backup

Note

: User executes these commands on the server.

# Backup
PGPASSWORD=$DB_PASSWORD pg_dump -h $DB_HOST -U $DB_USER $DB_NAME > backup_$(date +%Y%m%d).sql

# Restore
PGPASSWORD=$DB_PASSWORD psql -h $DB_HOST -U $DB_USER $DB_NAME < backup.sql

---

Bugsink (Error Tracking)

Dev Container Token Generation

MSYS_NO_PATHCONV=1 podman exec -e DATABASE_URL=postgresql://bugsink:bugsink_dev_password@postgres:5432/bugsink -e SECRET_KEY=dev-bugsink-secret-key-minimum-50-characters-for-security flyer-crawler-dev sh -c 'cd /opt/bugsink/conf && DJANGO_SETTINGS_MODULE=bugsink_conf PYTHONPATH=/opt/bugsink/conf:/opt/bugsink/lib/python3.10/site-packages /opt/bugsink/bin/python -m django create_auth_token'

Production Token Generation

Note

: User executes this command on the server.

cd /opt/bugsink && bugsink-manage create_auth_token

---

Common Troubleshooting

Container Won't Start

# Check logs
podman logs flyer-crawler-dev

# Inspect container
podman inspect flyer-crawler-dev

# Remove and recreate
podman rm -f flyer-crawler-dev
# Then recreate with docker-compose or podman run

Database Connection Issues

# Test connection inside container
podman exec -it flyer-crawler-dev psql -U postgres -d flyer_crawler_dev -c "SELECT 1"

# Check if PostgreSQL is running
podman exec -it flyer-crawler-dev pg_isready

PM2 App Keeps Restarting

# Check logs
pm2 logs flyer-crawler-api --err --lines 100

# Check for memory issues
pm2 monit

# View app details
pm2 show flyer-crawler-api

Redis Connection Issues

# Test Redis inside container
podman exec -it flyer-crawler-dev redis-cli ping

# Check Redis logs
podman logs flyer-crawler-redis