# Incident Report: PM2 Process Kill During v0.15.0 Deployment

**Date**: 2026-02-17
**Severity**: Critical
**Status**: Mitigated - Safeguards Implemented
**Affected Systems**: All PM2-managed applications on projectium.com server

---

## Resolution Summary

**Safeguards implemented on 2026-02-17** to prevent recurrence:

1. Workflow metadata logging (audit trail)
2. Pre-cleanup PM2 state logging (forensics)
3. Process count validation with SAFETY ABORT (automatic prevention)
4. Explicit name verification (visibility)
5. Post-cleanup verification (environment isolation check)

**Documentation created**:

- [PM2 Incident Response Runbook](PM2-INCIDENT-RESPONSE.md)
- [PM2 Safeguards Session Summary](../archive/sessions/PM2_SAFEGUARDS_SESSION_2026-02-17.md)
- CLAUDE.md updated with [PM2 Process Isolation Incidents section](../../CLAUDE.md#pm2-process-isolation-incidents)

---

## Summary

During v0.15.0 production deployment, ALL PM2 processes on the server were terminated, not just flyer-crawler processes. This caused unplanned downtime for other applications including stock-alert.

## Timeline

| Time (Approx)         | Event                                                            |
| --------------------- | ---------------------------------------------------------------- |
| 2026-02-17 ~07:40 UTC | v0.15.0 production deployment triggered via `deploy-to-prod.yml` |
| Unknown               | All PM2 processes killed (flyer-crawler AND other apps)          |
| Unknown               | Incident discovered - stock-alert down                           |
| 2026-02-17            | Investigation initiated                                          |
| 2026-02-17            | Defense-in-depth safeguards implemented in all workflows         |
| 2026-02-17            | Incident response runbook created                                |
| 2026-02-17            | Status changed to Mitigated                                      |

## Impact

- **Affected Applications**: All PM2-managed processes on projectium.com
  - flyer-crawler-api, flyer-crawler-worker, flyer-crawler-analytics-worker (expected)
  - stock-alert (NOT expected - collateral damage)
  - Potentially other unidentified applications
- **Downtime Duration**: TBD
- **User Impact**: Service unavailability for all affected applications

---

## Investigation Findings

### Deployment Workflow Analysis

All deployment workflows were reviewed for PM2 process isolation:

| Workflow                  | PM2 Isolation  | Implementation                                                                                    |
| ------------------------- | -------------- | ------------------------------------------------------------------------------------------------- |
| `deploy-to-prod.yml`      | Whitelist      | `prodProcesses = ['flyer-crawler-api', 'flyer-crawler-worker', 'flyer-crawler-analytics-worker']` |
| `deploy-to-test.yml`      | Pattern        | `p.name.endsWith('-test')`                                                                        |
| `manual-deploy-major.yml` | Whitelist      | Same as deploy-to-prod                                                                            |
| `manual-db-restore.yml`   | Explicit names | `pm2 stop flyer-crawler-api flyer-crawler-worker flyer-crawler-analytics-worker`                  |

### Fix Commit Already In Place

The PM2 process isolation fix was implemented in commit `b6a62a0` (2026-02-13):

```
commit b6a62a036f39ac895271402a61e5cc4227369de7
Author: Torben Sorensen <torben.sorensen@gmail.com>
Date:   Fri Feb 13 10:19:28 2026 -0800

    be specific about pm2 processes

Files modified:
 .gitea/workflows/deploy-to-prod.yml
 .gitea/workflows/deploy-to-test.yml
 .gitea/workflows/manual-db-restore.yml
 .gitea/workflows/manual-deploy-major.yml
 CLAUDE.md
```

### v0.15.0 Release Contains Fix

Confirmed: v0.15.0 (commit `93ad624`, 2026-02-18) includes the fix commit:

```
93ad624 ci: Bump version to 0.15.0 for production release [skip ci]
...
b6a62a0 be specific about pm2 processes  <-- Fix commit included
```

### Current Workflow PM2 Commands

**Production Deploy (`deploy-to-prod.yml` line 170)**:

```javascript
const prodProcesses = [
  'flyer-crawler-api',
  'flyer-crawler-worker',
  'flyer-crawler-analytics-worker',
];
list.forEach((p) => {
  if (
    (p.pm2_env.status === 'errored' || p.pm2_env.status === 'stopped') &&
    prodProcesses.includes(p.name)
  ) {
    exec('pm2 delete ' + p.pm2_env.pm_id);
  }
});
```

**Test Deploy (`deploy-to-test.yml` line 100)**:

```javascript
list.forEach((p) => {
  if (p.name && p.name.endsWith('-test')) {
    exec('pm2 delete ' + p.pm2_env.pm_id);
  }
});
```

Both implementations have proper name filtering and should NOT affect non-flyer-crawler processes.

---

## Discrepancy Analysis

### Key Mystery

**If the fixes are in place, why did ALL processes get killed?**

### Possible Explanations

#### 1. Workflow Version Mismatch (HIGH PROBABILITY)

**Hypothesis**: Gitea runner cached an older version of the workflow file.

- Gitea Actions may cache workflow definitions
- The runner might have executed an older version without the fix
- Need to verify: What version of `deploy-to-prod.yml` actually executed?

**Investigation Required**:

- Check Gitea workflow execution logs for actual script content
- Verify runner workflow caching behavior
- Compare executed workflow vs repository version

#### 2. Concurrent Workflow Execution (MEDIUM PROBABILITY)

**Hypothesis**: Another workflow ran simultaneously with destructive PM2 commands.

Workflows with potential issues:

- `manual-db-reset-prod.yml` - Does NOT restart PM2 (schema reset only)
- `manual-redis-flush-prod.yml` - Does NOT touch PM2
- Test deployment concurrent with prod deployment

**Investigation Required**:

- Check Gitea Actions history for concurrent workflow runs
- Review timestamps of all workflow executions on 2026-02-17

#### 3. Manual SSH Command (MEDIUM PROBABILITY)

**Hypothesis**: Someone SSH'd to the server and ran `pm2 stop all` or `pm2 delete all` manually.

**Investigation Required**:

- Check server shell history (if available)
- Review any maintenance windows or manual interventions
- Ask team members about manual actions

#### 4. PM2 Internal Issue (LOW PROBABILITY)

**Hypothesis**: PM2 daemon crash or corruption caused all processes to stop.

**Investigation Required**:

- Check PM2 daemon logs on server
- Look for OOM killer events in system logs
- Check disk space issues during deployment

#### 5. Script Execution Error (LOW PROBABILITY)

**Hypothesis**: JavaScript parsing error caused the filtering logic to be bypassed.

**Investigation Required**:

- Review workflow execution logs for JavaScript errors
- Test the inline Node.js scripts locally
- Check for shell escaping issues

---

## Documentation/Code Gaps Identified

### CLAUDE.md Documentation

The PM2 isolation rules are documented in `CLAUDE.md`, but:

- Documentation uses `pm2 restart all` in the Quick Reference table (for dev container - acceptable)
- Multiple docs still reference `pm2 restart all` without environment context
- No incident response runbook for PM2 issues

### Workflow Gaps

1. **No Workflow Audit Trail**: No logging of which exact workflow version executed
2. **No Pre-deployment Verification**: Workflows don't log PM2 state before modifications
3. **No Cross-Application Impact Assessment**: No mechanism to detect/warn about other apps

---

## Next Steps for Root Cause Analysis

### Immediate (Priority 1)

1. [ ] Retrieve Gitea Actions execution logs for v0.15.0 deployment
2. [ ] Extract actual executed workflow content from logs
3. [ ] Check for concurrent workflow executions on 2026-02-17
4. [ ] Review server PM2 daemon logs around incident time

### Short-term (Priority 2)

5. [ ] Implement pre-deployment PM2 state logging in workflows
6. [ ] Add workflow version hash logging for audit trail
7. [ ] Create incident response runbook for PM2/deployment issues

### Long-term (Priority 3)

8. [ ] Evaluate PM2 namespacing for complete process isolation
9. [ ] Consider separate PM2 daemon per application
10. [ ] Implement deployment monitoring/alerting

---

## Related Documentation

- [CLAUDE.md - PM2 Process Isolation](../../../CLAUDE.md) (Critical Rules section)
- [ADR-014: Containerization and Deployment Strategy](../adr/0014-containerization-and-deployment-strategy.md)
- [Deployment Guide](./DEPLOYMENT.md)
- Workflow files in `.gitea/workflows/`

---

## Appendix: Commit Timeline

```
93ad624 ci: Bump version to 0.15.0 for production release [skip ci]  <-- v0.15.0 release
7dd4f21 ci: Bump version to 0.14.4 [skip ci]
174b637 even more typescript fixes
4f80baf ci: Bump version to 0.14.3 [skip ci]
8450b5e Generate TSOA Spec and Routes
e4d830a ci: Bump version to 0.14.2 [skip ci]
b6a62a0 be specific about pm2 processes                               <-- PM2 fix commit
2d2cd52 Massive Dependency Modernization Project
```

---

## Revision History

| Date       | Author             | Change                  |
| ---------- | ------------------ | ----------------------- |
| 2026-02-17 | Investigation Team | Initial incident report |
