pm2 save fix + test work

docs/adr/0014-containerization-and-deployment-strategy.md

@@ -249,26 +249,39 @@ module.exports = {
 
 ### PM2 Commands Reference
 
+**CRITICAL**: Every `pm2 start`, `pm2 restart`, `pm2 stop`, or `pm2 delete` command MUST be immediately followed by `pm2 save`. Without this, processes become ephemeral and will disappear on PM2 daemon restarts, server reboots, or internal reconciliation events.
+
 ```bash
-# Start/reload with environment
+# ✅ CORRECT - Start/reload with environment and save
+pm2 startOrReload ecosystem.config.cjs --env production --update-env && pm2 save
+
+# ✅ CORRECT - Restart and save
+pm2 restart flyer-crawler-api && pm2 save
+
+# ✅ CORRECT - Stop and save
+pm2 stop flyer-crawler-api && pm2 save
+
+# ✅ CORRECT - Delete and save
+pm2 delete flyer-crawler-api && pm2 save
+
+# ❌ WRONG - Missing save (processes become ephemeral)
 pm2 startOrReload ecosystem.config.cjs --env production --update-env
 
-# Save process list for startup
-pm2 save
-
-# View logs
+# View logs (read-only operation, no save needed)
 pm2 logs flyer-crawler-api --lines 50
 
-# Monitor processes
+# Monitor processes (read-only operation, no save needed)
 pm2 monit
 
-# List all processes
+# List all processes (read-only operation, no save needed)
 pm2 list
 
-# Describe process details
+# Describe process details (read-only operation, no save needed)
 pm2 describe flyer-crawler-api
 ```
 
+**Why This Matters**: PM2 maintains an in-memory process list. The `pm2 save` command writes this list to `~/.pm2/dump.pm2`, which PM2 uses to resurrect processes after daemon restarts. Without it, your carefully managed process state is lost.
+
 ### Resource Limits
 
 | Process          | Memory Limit | Restart Delay            | Kill Timeout |

docs/adr/0061-pm2-process-isolation-safeguards.md

@@ -115,6 +115,31 @@ echo "=== END POST-CLEANUP VERIFICATION ==="
 
 **Purpose**: Immediately identifies cross-environment contamination.
 
+#### Layer 6: PM2 Process List Persistence
+
+**CRITICAL**: Save the PM2 process list after every state-changing operation:
+
+```bash
+# After any pm2 start/stop/restart/delete operation
+pm2 save
+
+# Example: After cleanup loop completes
+targetProcesses.forEach(p => {
+  exec('pm2 delete ' + p.pm2_env.pm_id);
+});
+exec('pm2 save');  // Persist all deletions
+```
+
+**Purpose**: Ensures PM2 process state persists across daemon restarts, server reboots, and internal reconciliation events.
+
+**Why This Matters**: PM2 maintains an in-memory process list. Without `pm2 save`, processes become ephemeral:
+
+- Daemon restart → All unsaved processes disappear
+- Server reboot → Process list reverts to last saved state
+- PM2 internal reconciliation → Unsaved processes may be lost
+
+**Pattern**: Every `pm2 start`, `pm2 restart`, `pm2 stop`, or `pm2 delete` MUST be followed by `pm2 save`.
+
 ## Consequences
 
 ### Positive

docs/operations/DEPLOYMENT.md

@@ -17,15 +17,15 @@ This guide covers deploying Flyer Crawler to a production server.
 
 ### Command Reference Table
 
-| Task                 | Command                                                                 |
-| -------------------- | ----------------------------------------------------------------------- |
-| Deploy to production | Gitea Actions workflow (manual trigger)                                 |
-| Deploy to test       | Automatic on push to `main`                                             |
-| Check PM2 status     | `pm2 list`                                                              |
-| View logs            | `pm2 logs flyer-crawler-api --lines 100`                                |
-| Restart all          | `pm2 restart all`                                                       |
-| Check NGINX          | `sudo nginx -t && sudo systemctl status nginx`                          |
-| Check health         | `curl -s https://flyer-crawler.projectium.com/api/health/ready \| jq .` |
+| Task                 | Command                                                                                         |
+| -------------------- | ----------------------------------------------------------------------------------------------- |
+| Deploy to production | Gitea Actions workflow (manual trigger)                                                         |
+| Deploy to test       | Automatic on push to `main`                                                                     |
+| Check PM2 status     | `pm2 list`                                                                                      |
+| View logs            | `pm2 logs flyer-crawler-api --lines 100`                                                        |
+| Restart all          | `pm2 restart flyer-crawler-api flyer-crawler-worker flyer-crawler-analytics-worker && pm2 save` |
+| Check NGINX          | `sudo nginx -t && sudo systemctl status nginx`                                                  |
+| Check health         | `curl -s https://flyer-crawler.projectium.com/api/health/ready \| jq .`                         |
 
 ### Deployment URLs
 
@@ -274,7 +274,40 @@ sudo systemctl reload nginx
 
 ---
 
-## PM2 Log Management
+## PM2 Process Management
+
+### Critical: Always Save After State Changes
+
+**CRITICAL**: Every `pm2 start`, `pm2 restart`, `pm2 stop`, or `pm2 delete` command MUST be immediately followed by `pm2 save`.
+
+Without `pm2 save`, processes become ephemeral and will disappear on:
+
+- PM2 daemon restarts
+- Server reboots
+- Internal PM2 reconciliation events
+
+**Correct Pattern:**
+
+```bash
+# ✅ CORRECT - Always save after state changes
+pm2 restart flyer-crawler-api && pm2 save
+pm2 stop flyer-crawler-worker && pm2 save
+pm2 delete flyer-crawler-analytics-worker && pm2 save
+pm2 startOrReload ecosystem.config.cjs --env production --update-env && pm2 save
+
+# ❌ WRONG - Missing save (processes become ephemeral)
+pm2 restart flyer-crawler-api
+pm2 stop flyer-crawler-worker
+
+# ✅ Read-only operations don't need save
+pm2 list
+pm2 logs flyer-crawler-api
+pm2 monit
+```
+
+**Why This Matters**: PM2 maintains an in-memory process list. The `pm2 save` command writes this list to `~/.pm2/dump.pm2`, which PM2 uses to resurrect processes after daemon restarts. Without it, your carefully managed process state is lost.
+
+### PM2 Log Management
 
 Install and configure pm2-logrotate to manage log files:
 

docs/operations/PM2-INCIDENT-RESPONSE.md

@@ -42,25 +42,35 @@
 
 ### Critical Commands
 
+**IMPORTANT**: Every `pm2 start`, `pm2 restart`, `pm2 stop`, or `pm2 delete` command MUST be immediately followed by `pm2 save` to persist changes.
+
 ```bash
-# Check PM2 status
+# Check PM2 status (read-only, no save needed)
 pm2 list
 
-# Check specific process
+# Check specific process (read-only, no save needed)
 pm2 show flyer-crawler-api
 
-# View recent logs
+# View recent logs (read-only, no save needed)
 pm2 logs --lines 50
 
-# Restart specific processes (SAFE)
-pm2 restart flyer-crawler-api flyer-crawler-worker flyer-crawler-analytics-worker
+# ✅ Restart specific processes (SAFE - includes save)
+pm2 restart flyer-crawler-api flyer-crawler-worker flyer-crawler-analytics-worker && pm2 save
 
-# DO NOT USE (affects ALL apps)
+# ✅ Start processes from config (SAFE - includes save)
+pm2 startOrReload /var/www/flyer-crawler.projectium.com/ecosystem.config.cjs --update-env && pm2 save
+
+# ❌ DO NOT USE (affects ALL apps)
 # pm2 restart all    <-- DANGEROUS
 # pm2 stop all       <-- DANGEROUS
 # pm2 delete all     <-- DANGEROUS
+
+# ❌ DO NOT FORGET pm2 save after state changes
+# pm2 restart flyer-crawler-api  <-- WRONG: Missing save, process becomes ephemeral
 ```
 
+**Why `pm2 save` Matters**: Without it, processes become ephemeral and disappear on daemon restarts, server reboots, or internal PM2 reconciliation events.
+
 ### Severity Classification
 
 | Severity          | Criteria                                      | Response Time       | Example                                         |