# Logstash Quick Reference (ADR-050)

Aggregates logs from PostgreSQL, PM2, Redis, NGINX; forwards errors to Bugsink.

## Configuration

**Primary config**: `/etc/logstash/conf.d/bugsink.conf`

### Related Files

| Path                                                | Purpose                   |
| --------------------------------------------------- | ------------------------- |
| `/etc/postgresql/14/main/conf.d/observability.conf` | PostgreSQL logging config |
| `/var/log/postgresql/*.log`                         | PostgreSQL logs           |
| `/home/gitea-runner/.pm2/logs/*.log`                | PM2 worker logs           |
| `/var/log/redis/redis-server.log`                   | Redis logs                |
| `/var/log/nginx/access.log`                         | NGINX access logs         |
| `/var/log/nginx/error.log`                          | NGINX error logs          |
| `/var/log/logstash/*.log`                           | Logstash file outputs     |
| `/var/lib/logstash/sincedb_*`                       | Position tracking files   |

## Features

- **Multi-source aggregation**: PostgreSQL, PM2 workers, Redis, NGINX
- **Environment routing**: Auto-detects prod/test, routes to correct Bugsink project
- **JSON parsing**: Extracts `fn_log()` from PostgreSQL, Pino JSON from PM2
- **Sentry format**: Transforms to `event_id`, `timestamp`, `level`, `message`, `extra`
- **Error filtering**: Only forwards WARNING/ERROR to Bugsink
- **Operational storage**: Non-error logs saved to `/var/log/logstash/`
- **Request monitoring**: NGINX requests categorized by status, slow request detection

## Commands

```bash
# Status and control
systemctl status logstash
systemctl restart logstash

# Test configuration
/usr/share/logstash/bin/logstash --config.test_and_exit -f /etc/logstash/conf.d/bugsink.conf

# View logs
journalctl -u logstash -f

# Check stats (events processed, failures)
curl -XGET 'localhost:9600/_node/stats/pipelines?pretty' | jq '.pipelines.main.plugins.filters'

# Monitor sources
tail -f /var/log/postgresql/postgresql-$(date +%Y-%m-%d).log
tail -f /var/log/logstash/pm2-workers-$(date +%Y-%m-%d).log
tail -f /var/log/logstash/redis-operational-$(date +%Y-%m-%d).log
tail -f /var/log/logstash/nginx-access-$(date +%Y-%m-%d).log

# Check disk usage
du -sh /var/log/logstash/
```

## Troubleshooting

| Issue                 | Check            | Solution                                                                                       |
| --------------------- | ---------------- | ---------------------------------------------------------------------------------------------- |
| No Bugsink errors     | Logstash running | `systemctl status logstash`                                                                    |
| Config syntax error   | Test config      | `/usr/share/logstash/bin/logstash --config.test_and_exit -f /etc/logstash/conf.d/bugsink.conf` |
| Grok pattern failures | Stats endpoint   | `curl localhost:9600/_node/stats/pipelines?pretty \| jq '.pipelines.main.plugins.filters'`     |
| Wrong Bugsink project | Env detection    | Check tags in logs match expected environment                                                  |
| Permission denied     | Logstash groups  | `groups logstash` should include `postgres`, `adm`                                             |
| PM2 not captured      | File paths       | `ls /home/gitea-runner/.pm2/logs/flyer-crawler-worker-*.log`                                   |
| NGINX logs missing    | Output directory | `ls -lh /var/log/logstash/nginx-access-*.log`                                                  |
| High disk usage       | Log rotation     | Verify `/etc/logrotate.d/logstash` configured                                                  |

## Related Documentation

- **Full setup**: [BARE-METAL-SETUP.md](BARE-METAL-SETUP.md) - PostgreSQL Function Observability section
- **Architecture**: [adr/0050-postgresql-function-observability.md](adr/0050-postgresql-function-observability.md)
- **Troubleshooting details**: [LOGSTASH-TROUBLESHOOTING.md](LOGSTASH-TROUBLESHOOTING.md)