# PM2 Crash Debugging Guide

## Overview

This guide helps diagnose PM2 daemon crashes and identify which project is causing the issue.

## Common Symptoms

1. **PM2 processes disappear** between deployments
2. **`ENOENT: no such file or directory, uv_cwd`** errors in PM2 logs
3. **Processes require `pm2 resurrect`** after deployments
4. **PM2 daemon restarts** unexpectedly

## Root Cause

PM2 processes crash when their working directory (CWD) is deleted or modified while they're running. This typically happens when:

1. **rsync --delete** removes/recreates directories while processes are active
2. **npm install** modifies node_modules while processes are using them
3. **Deployments** don't stop processes before file operations

## Debugging Tools

### 1. PM2 Diagnostics Workflow

Run the comprehensive diagnostics workflow:

```bash
# In Gitea Actions UI:
# 1. Go to Actions → "PM2 Diagnostics"
# 2. Click "Run workflow"
# 3. Choose monitoring duration (default: 60s)
```

This workflow captures:

- Current PM2 state
- Working directory validation
- PM2 daemon logs
- All PM2-managed projects
- Crash patterns
- Deployment script analysis

### 2. PM2 Crash Analysis Script

Run the crash analysis script on the server:

```bash
# SSH to server
ssh gitea-runner@projectium.com

# Run analysis
cd /var/www/flyer-crawler.projectium.com
bash scripts/analyze-pm2-crashes.sh

# Or save to file
bash scripts/analyze-pm2-crashes.sh > pm2-crash-report.txt
```

### 3. Manual PM2 Inspection

Quick manual checks:

```bash
# Current PM2 state
pm2 list

# Detailed JSON state
pm2 jlist | jq '.'

# Check for missing CWDs
pm2 jlist | jq -r '.[] | "\(.name): \(.pm2_env.pm_cwd)"' | while read line; do
    PROC=$(echo "$line" | cut -d: -f1)
    CWD=$(echo "$line" | cut -d: -f2- | xargs)
    [ -d "$CWD" ] && echo "✅ $PROC" || echo "❌ $PROC (CWD missing: $CWD)"
done

# View PM2 daemon log
tail -100 ~/.pm2/pm2.log

# Search for ENOENT errors
grep -i "ENOENT\|uv_cwd" ~/.pm2/pm2.log
```

## Identifying the Problematic Project

### Check Which Projects Share PM2 Daemon

```bash
pm2 list

# Group by project
pm2 jlist | jq -r '.[] | .name' | grep -oE "^[a-z-]+" | sort -u
```

**Projects on projectium.com:**

- `flyer-crawler` (production, test)
- `stock-alert` (production, test)
- Others?

### Check Deployment Timing

1. Review PM2 daemon restart times:

   ```bash
   grep "New PM2 Daemon started" ~/.pm2/pm2.log
   ```

2. Compare with deployment times in Gitea Actions

3. Identify which deployment triggered the crash

### Check Deployment Scripts

For each project, check if deployment stops PM2 before rsync:

```bash
# Flyer-crawler
cat /var/www/flyer-crawler.projectium.com/.gitea/workflows/deploy-to-prod.yml | grep -B5 -A5 "rsync.*--delete"

# Stock-alert
cat /var/www/stock-alert.projectium.com/.gitea/workflows/deploy-to-prod.yml | grep -B5 -A5 "rsync.*--delete"
```

**Look for:**

- ❌ `rsync --delete` **before** `pm2 stop`
- ✅ `pm2 stop` **before** `rsync --delete`

## Common Culprits

### 1. Flyer-Crawler Deployments

**Before Fix:**

```yaml
# ❌ BAD - Deploys files while processes running
- name: Deploy Application
  run: |
    rsync --delete ./ /var/www/...
    pm2 restart ...
```

**After Fix:**

```yaml
# ✅ GOOD - Stops processes first
- name: Deploy Application
  run: |
    pm2 stop flyer-crawler-api flyer-crawler-worker
    rsync --delete ./ /var/www/...
    pm2 startOrReload ...
```

### 2. Stock-Alert Deployments

Check if stock-alert follows the same pattern. If it deploys without stopping PM2, it could crash the shared PM2 daemon.

### 3. Cross-Project Interference

If multiple projects share PM2:

- One project's deployment can crash another project's processes
- The crashed project's processes lose their CWD
- PM2 daemon may restart, clearing all processes

## Solutions

### Immediate Fix (Manual)

```bash
# Restore processes from dump file
pm2 resurrect

# Verify all processes are running
pm2 list
```

### Permanent Fix

1. **Update deployment workflows** to stop PM2 before file operations
2. **Isolate PM2 daemons** by user or namespace
3. **Monitor deployments** to ensure proper sequencing

## Deployment Workflow Template

**Correct sequence:**

```yaml
- name: Deploy Application
  run: |
    # 1. STOP PROCESSES FIRST
    pm2 stop my-api my-worker

    # 2. THEN deploy files
    rsync -avz --delete ./ /var/www/my-app/

    # 3. Install dependencies (safe, no processes running)
    cd /var/www/my-app
    npm install --omit=dev

    # 4. Clean up errored processes
    pm2 delete my-api my-worker || true

    # 5. START processes
    pm2 startOrReload ecosystem.config.cjs
    pm2 save
```

## Monitoring & Prevention

### Enable Verbose Logging

Enhanced deployment logging (already implemented in flyer-crawler):

```yaml
- name: Deploy Application
  run: |
    set -x  # Command tracing
    echo "Step 1: Stopping PM2..."
    pm2 stop ...
    pm2 list  # Verify stopped

    echo "Step 2: Deploying files..."
    rsync --delete ...

    echo "Step 3: Starting PM2..."
    pm2 start ...
    pm2 list  # Verify started
```

### Regular Health Checks

```bash
# Add to cron or monitoring system
*/5 * * * * pm2 jlist | jq -r '.[] | select(.pm2_env.status != "online") | "ALERT: \(.name) is \(.pm2_env.status)"'
```

## Troubleshooting Decision Tree

```
PM2 processes missing?
├─ YES → Run `pm2 resurrect`
│        └─ Check PM2 daemon log for ENOENT errors
│           ├─ ENOENT found → Working directory deleted during deployment
│           │                 └─ Fix: Add `pm2 stop` before rsync
│           └─ No ENOENT → Check other error patterns
│
└─ NO → Processes running but unstable?
         └─ Check restart counts: `pm2 jlist | jq '.[].pm2_env.restart_time'`
            └─ High restarts → Application-level issue (not PM2 crash)
```

## Related Documentation

- [PM2 Process Isolation Requirements](../../CLAUDE.md#pm2-process-isolation-productiontest-servers)
- [PM2 Incident Response Runbook](./PM2-INCIDENT-RESPONSE.md)
- [Incident Report 2026-02-17](./INCIDENT-2026-02-17-PM2-PROCESS-KILL.md)

## Quick Reference Commands

```bash
# Diagnose
pm2 list                                    # Current state
pm2 jlist | jq '.'                          # Detailed JSON
tail -100 ~/.pm2/pm2.log                    # Recent logs
grep ENOENT ~/.pm2/pm2.log                  # Find crashes

# Fix
pm2 resurrect                               # Restore from dump
pm2 restart all                             # Restart everything
pm2 save                                    # Save current state

# Analyze
bash scripts/analyze-pm2-crashes.sh         # Run analysis script
pm2 jlist | jq -r '.[].pm2_env.pm_cwd'     # Check working dirs
```
