# Logstash Quick Reference (ADR-050)

Aggregates logs from PostgreSQL, PM2, Redis, NGINX; forwards errors to Bugsink.

**Last verified**: 2026-01-28

**Related documentation**:

- [ADR-050: PostgreSQL Function Observability](../adr/0050-postgresql-function-observability.md)
- [ADR-015: Error Tracking and Observability](../adr/0015-error-tracking-and-observability.md)
- [Monitoring Guide](MONITORING.md)
- [Logstash Troubleshooting Runbook](LOGSTASH-TROUBLESHOOTING.md)

---

## Quick Reference

### Bugsink Project Routing

| Source Type    | Environment | Bugsink Project      | Project ID |
| -------------- | ----------- | -------------------- | ---------- |
| PM2 API/Worker | Dev         | Backend API (Dev)    | 1          |
| PostgreSQL     | Dev         | Backend API (Dev)    | 1          |
| Frontend JS    | Dev         | Frontend (Dev)       | 2          |
| Redis/NGINX    | Dev         | Infrastructure (Dev) | 4          |
| PM2 API/Worker | Production  | Backend API (Prod)   | 1          |
| PostgreSQL     | Production  | Backend API (Prod)   | 1          |
| PM2 API/Worker | Test        | Backend API (Test)   | 3          |

### Key DSN Keys (Dev Container)

| Project              | DSN Key                            |
| -------------------- | ---------------------------------- |
| Backend API (Dev)    | `cea01396c56246adb5878fa5ee6b1d22` |
| Frontend (Dev)       | `d92663cb73cf4145b677b84029e4b762` |
| Infrastructure (Dev) | `14e8791da3d347fa98073261b596cab9` |

---

## Configuration

**Primary config**: `/etc/logstash/conf.d/bugsink.conf`

**Dev container config**: `docker/logstash/bugsink.conf`

### Related Files

| Path                                                | Purpose                   |
| --------------------------------------------------- | ------------------------- |
| `/etc/postgresql/14/main/conf.d/observability.conf` | PostgreSQL logging config |
| `/var/log/postgresql/*.log`                         | PostgreSQL logs           |
| `/home/gitea-runner/.pm2/logs/*.log`                | PM2 worker logs           |
| `/var/log/redis/redis-server.log`                   | Redis logs                |
| `/var/log/nginx/access.log`                         | NGINX access logs         |
| `/var/log/nginx/error.log`                          | NGINX error logs          |
| `/var/log/logstash/*.log`                           | Logstash file outputs     |
| `/var/lib/logstash/sincedb_*`                       | Position tracking files   |

## Features

- **Multi-source aggregation**: PostgreSQL, PM2 workers, Redis, NGINX
- **Environment routing**: Auto-detects prod/test, routes to correct Bugsink project
- **JSON parsing**: Extracts `fn_log()` from PostgreSQL, Pino JSON from PM2
- **Sentry format**: Transforms to `event_id`, `timestamp`, `level`, `message`, `extra`
- **Error filtering**: Only forwards WARNING/ERROR to Bugsink
- **Operational storage**: Non-error logs saved to `/var/log/logstash/`
- **Request monitoring**: NGINX requests categorized by status, slow request detection

## Commands

### Production (Bare Metal)

```bash
# Status and control
systemctl status logstash
systemctl restart logstash

# Test configuration
/usr/share/logstash/bin/logstash --config.test_and_exit -f /etc/logstash/conf.d/bugsink.conf

# View logs
journalctl -u logstash -f

# Check stats (events processed, failures)
curl -XGET 'localhost:9600/_node/stats/pipelines?pretty' | jq '.pipelines.main.plugins.filters'

# Monitor sources
tail -f /var/log/postgresql/postgresql-$(date +%Y-%m-%d).log
tail -f /var/log/logstash/pm2-workers-$(date +%Y-%m-%d).log
tail -f /var/log/logstash/redis-operational-$(date +%Y-%m-%d).log
tail -f /var/log/logstash/nginx-access-$(date +%Y-%m-%d).log

# Check disk usage
du -sh /var/log/logstash/
```

### Dev Container

```bash
# View Logstash logs (inside container)
MSYS_NO_PATHCONV=1 podman exec flyer-crawler-dev cat /var/log/logstash/logstash.log

# Check PM2 API logs are being processed (ADR-014)
MSYS_NO_PATHCONV=1 podman exec flyer-crawler-dev cat /var/log/logstash/pm2-api-$(date +%Y-%m-%d).log

# Check PM2 Worker logs are being processed
MSYS_NO_PATHCONV=1 podman exec flyer-crawler-dev cat /var/log/logstash/pm2-worker-$(date +%Y-%m-%d).log

# Check Redis logs are being processed
MSYS_NO_PATHCONV=1 podman exec flyer-crawler-dev cat /var/log/logstash/redis-operational-$(date +%Y-%m-%d).log

# View raw PM2 logs
MSYS_NO_PATHCONV=1 podman exec flyer-crawler-dev cat /var/log/pm2/api-out.log
MSYS_NO_PATHCONV=1 podman exec flyer-crawler-dev cat /var/log/pm2/worker-out.log

# View raw Redis logs
MSYS_NO_PATHCONV=1 podman exec flyer-crawler-redis cat /var/log/redis/redis-server.log

# Check Logstash stats
podman exec flyer-crawler-dev curl -s localhost:9600/_node/stats/pipelines?pretty

# Verify shared volume mounts
MSYS_NO_PATHCONV=1 podman exec flyer-crawler-dev ls -la /var/log/pm2/
MSYS_NO_PATHCONV=1 podman exec flyer-crawler-dev ls -la /var/log/redis/
```

## Troubleshooting

### Decision Tree: Logs Not Appearing in Bugsink

```text
Errors not showing in Bugsink?
    |
    +-- Logstash running?
    |       |
    |       +-- No --> systemctl start logstash
    |       +-- Yes --> Check pipeline stats
    |               |
    |               +-- Events in = 0?
    |               |       |
    |               |       +-- Log files exist? --> ls /var/log/pm2/*.log
    |               |       +-- Permissions OK? --> groups logstash
    |               |
    |               +-- Events filtered = high?
    |               |       |
    |               |       +-- Grok failures --> Check log format matches pattern
    |               |
    |               +-- Events out but no Bugsink?
    |                       |
    |                       +-- 403 error --> Wrong DSN key
    |                       +-- 500 error --> Invalid event format (check sentry_level)
    |                       +-- Connection refused --> Bugsink not running
```

### Common Issues Table

| Issue                 | Check            | Solution                                                                                       |
| --------------------- | ---------------- | ---------------------------------------------------------------------------------------------- |
| No Bugsink errors     | Logstash running | `systemctl status logstash`                                                                    |
| Config syntax error   | Test config      | `/usr/share/logstash/bin/logstash --config.test_and_exit -f /etc/logstash/conf.d/bugsink.conf` |
| Grok pattern failures | Stats endpoint   | `curl localhost:9600/_node/stats/pipelines?pretty \| jq '.pipelines.main.plugins.filters'`     |
| Wrong Bugsink project | Env detection    | Check tags in logs match expected environment                                                  |
| Permission denied     | Logstash groups  | `groups logstash` should include `postgres`, `adm`                                             |
| PM2 not captured      | File paths       | Dev: `ls /var/log/pm2/`; Prod: `ls /home/gitea-runner/.pm2/logs/`                              |
| PM2 logs empty        | PM2 running      | Dev: `podman exec flyer-crawler-dev pm2 status`; Prod: `pm2 status`                            |
| NGINX logs missing    | Output directory | `ls -lh /var/log/logstash/nginx-access-*.log`                                                  |
| Redis logs missing    | Shared volume    | Dev: Check `redis_logs` volume mounted; Prod: Check `/var/log/redis/redis-server.log` exists   |
| High disk usage       | Log rotation     | Verify `/etc/logrotate.d/logstash` configured                                                  |
| varchar(7) error      | Level validation | Add Ruby filter to validate/normalize `sentry_level` before output                             |

### Expected Output Examples

**Successful Logstash pipeline stats**:

```json
{
  "in": 1523,
  "out": 1520,
  "filtered": 1520,
  "queue_push_duration_in_millis": 45
}
```

**Healthy Bugsink HTTP response**:

```json
{ "id": "a1b2c3d4e5f6..." }
```

## Related Documentation

- **Dev Container Guide**: [DEV-CONTAINER.md](../development/DEV-CONTAINER.md) - PM2 and log aggregation in dev
- **Full setup**: [BARE-METAL-SETUP.md](BARE-METAL-SETUP.md) - PostgreSQL Function Observability section
- **Architecture**: [adr/0050-postgresql-function-observability.md](../adr/0050-postgresql-function-observability.md)
- **Troubleshooting details**: [LOGSTASH-TROUBLESHOOTING.md](LOGSTASH-TROUBLESHOOTING.md)