Logstash Quick Reference (ADR-050)

Aggregates logs from PostgreSQL, PM2, Redis, NGINX; forwards errors to Bugsink.

Last verified: 2026-01-28

Related documentation:

Quick Reference

Bugsink Project Routing

Source Type	Environment	Bugsink Project	Project ID
PM2 API/Worker	Dev	Backend API (Dev)	1
PostgreSQL	Dev	Backend API (Dev)	1
Frontend JS	Dev	Frontend (Dev)	2
Redis/NGINX	Dev	Infrastructure (Dev)	4
PM2 API/Worker	Production	Backend API (Prod)	1
PostgreSQL	Production	Backend API (Prod)	1
PM2 API/Worker	Test	Backend API (Test)	3

Key DSN Keys (Dev Container)

Project	DSN Key
Backend API (Dev)	`cea01396c56246adb5878fa5ee6b1d22`
Frontend (Dev)	`d92663cb73cf4145b677b84029e4b762`
Infrastructure (Dev)	`14e8791da3d347fa98073261b596cab9`

Configuration

Primary config: /etc/logstash/conf.d/bugsink.conf

Dev container config: docker/logstash/bugsink.conf

Path	Purpose
`/etc/postgresql/14/main/conf.d/observability.conf`	PostgreSQL logging config
`/var/log/postgresql/*.log`	PostgreSQL logs
`/home/gitea-runner/.pm2/logs/*.log`	PM2 worker logs
`/var/log/redis/redis-server.log`	Redis logs
`/var/log/nginx/access.log`	NGINX access logs
`/var/log/nginx/error.log`	NGINX error logs
`/var/log/logstash/*.log`	Logstash file outputs
`/var/lib/logstash/sincedb_*`	Position tracking files

Features

Multi-source aggregation: PostgreSQL, PM2 workers, Redis, NGINX
Environment routing: Auto-detects prod/test, routes to correct Bugsink project
JSON parsing: Extracts fn_log() from PostgreSQL, Pino JSON from PM2
Sentry format: Transforms to event_id, timestamp, level, message, extra
Error filtering: Only forwards WARNING/ERROR to Bugsink
Operational storage: Non-error logs saved to /var/log/logstash/
Request monitoring: NGINX requests categorized by status, slow request detection

Commands

Production (Bare Metal)

# Status and control
systemctl status logstash
systemctl restart logstash

# Test configuration
/usr/share/logstash/bin/logstash --config.test_and_exit -f /etc/logstash/conf.d/bugsink.conf

# View logs
journalctl -u logstash -f

# Check stats (events processed, failures)
curl -XGET 'localhost:9600/_node/stats/pipelines?pretty' | jq '.pipelines.main.plugins.filters'

# Monitor sources
tail -f /var/log/postgresql/postgresql-$(date +%Y-%m-%d).log
tail -f /var/log/logstash/pm2-workers-$(date +%Y-%m-%d).log
tail -f /var/log/logstash/redis-operational-$(date +%Y-%m-%d).log
tail -f /var/log/logstash/nginx-access-$(date +%Y-%m-%d).log

# Check disk usage
du -sh /var/log/logstash/

Dev Container

# View Logstash logs (inside container)
MSYS_NO_PATHCONV=1 podman exec flyer-crawler-dev cat /var/log/logstash/logstash.log

# Check PM2 API logs are being processed (ADR-014)
MSYS_NO_PATHCONV=1 podman exec flyer-crawler-dev cat /var/log/logstash/pm2-api-$(date +%Y-%m-%d).log

# Check PM2 Worker logs are being processed
MSYS_NO_PATHCONV=1 podman exec flyer-crawler-dev cat /var/log/logstash/pm2-worker-$(date +%Y-%m-%d).log

# Check Redis logs are being processed
MSYS_NO_PATHCONV=1 podman exec flyer-crawler-dev cat /var/log/logstash/redis-operational-$(date +%Y-%m-%d).log

# View raw PM2 logs
MSYS_NO_PATHCONV=1 podman exec flyer-crawler-dev cat /var/log/pm2/api-out.log
MSYS_NO_PATHCONV=1 podman exec flyer-crawler-dev cat /var/log/pm2/worker-out.log

# View raw Redis logs
MSYS_NO_PATHCONV=1 podman exec flyer-crawler-redis cat /var/log/redis/redis-server.log

# Check Logstash stats
podman exec flyer-crawler-dev curl -s localhost:9600/_node/stats/pipelines?pretty

# Verify shared volume mounts
MSYS_NO_PATHCONV=1 podman exec flyer-crawler-dev ls -la /var/log/pm2/
MSYS_NO_PATHCONV=1 podman exec flyer-crawler-dev ls -la /var/log/redis/

Troubleshooting

Decision Tree: Logs Not Appearing in Bugsink

Errors not showing in Bugsink?
    |
    +-- Logstash running?
    |       |
    |       +-- No --> systemctl start logstash
    |       +-- Yes --> Check pipeline stats
    |               |
    |               +-- Events in = 0?
    |               |       |
    |               |       +-- Log files exist? --> ls /var/log/pm2/*.log
    |               |       +-- Permissions OK? --> groups logstash
    |               |
    |               +-- Events filtered = high?
    |               |       |
    |               |       +-- Grok failures --> Check log format matches pattern
    |               |
    |               +-- Events out but no Bugsink?
    |                       |
    |                       +-- 403 error --> Wrong DSN key
    |                       +-- 500 error --> Invalid event format (check sentry_level)
    |                       +-- Connection refused --> Bugsink not running

Common Issues Table

Issue	Check	Solution
No Bugsink errors	Logstash running	`systemctl status logstash`
Config syntax error	Test config	`/usr/share/logstash/bin/logstash --config.test_and_exit -f /etc/logstash/conf.d/bugsink.conf`
Grok pattern failures	Stats endpoint	`curl localhost:9600/_node/stats/pipelines?pretty \| jq '.pipelines.main.plugins.filters'`
Wrong Bugsink project	Env detection	Check tags in logs match expected environment
Permission denied	Logstash groups	`groups logstash` should include `postgres`, `adm`
PM2 not captured	File paths	Dev: `ls /var/log/pm2/`; Prod: `ls /home/gitea-runner/.pm2/logs/`
PM2 logs empty	PM2 running	Dev: `podman exec flyer-crawler-dev pm2 status`; Prod: `pm2 status`
NGINX logs missing	Output directory	`ls -lh /var/log/logstash/nginx-access-*.log`
Redis logs missing	Shared volume	Dev: Check `redis_logs` volume mounted; Prod: Check `/var/log/redis/redis-server.log` exists
High disk usage	Log rotation	Verify `/etc/logrotate.d/logstash` configured
varchar(7) error	Level validation	Add Ruby filter to validate/normalize `sentry_level` before output

Expected Output Examples

Successful Logstash pipeline stats:

{
  "in": 1523,
  "out": 1520,
  "filtered": 1520,
  "queue_push_duration_in_millis": 45
}

Healthy Bugsink HTTP response:

{ "id": "a1b2c3d4e5f6..." }

Dev Container Guide: DEV-CONTAINER.md - PM2 and log aggregation in dev
Full setup: BARE-METAL-SETUP.md - PostgreSQL Function Observability section
Architecture: adr/0050-postgresql-function-observability.md
Troubleshooting details: LOGSTASH-TROUBLESHOOTING.md

8.3 KiB Raw Blame History