flyer-crawler.projectium.com/docs/operations/LOGSTASH-QUICK-REF.md

Logstash Quick Reference (ADR-050)

Aggregates logs from PostgreSQL, PM2, Redis, NGINX; forwards errors to Bugsink.

Configuration

Primary config: /etc/logstash/conf.d/bugsink.conf

Related Files

Path	Purpose
/etc/postgresql/14/main/conf.d/observability.conf	PostgreSQL logging config
/var/log/postgresql/*.log	PostgreSQL logs
/home/gitea-runner/.pm2/logs/*.log	PM2 worker logs
/var/log/redis/redis-server.log	Redis logs
/var/log/nginx/access.log	NGINX access logs
/var/log/nginx/error.log	NGINX error logs
/var/log/logstash/*.log	Logstash file outputs
/var/lib/logstash/sincedb_*	Position tracking files

Features

• Multi-source aggregation: PostgreSQL, PM2 workers, Redis, NGINX
• Environment routing: Auto-detects prod/test, routes to correct Bugsink project
• JSON parsing: Extracts fn_log() from PostgreSQL, Pino JSON from PM2
• Sentry format: Transforms to event_id, timestamp, level, message, extra
• Error filtering: Only forwards WARNING/ERROR to Bugsink
• Operational storage: Non-error logs saved to /var/log/logstash/
• Request monitoring: NGINX requests categorized by status, slow request detection

Commands

Production (Bare Metal)

# Status and control
systemctl status logstash
systemctl restart logstash

# Test configuration
/usr/share/logstash/bin/logstash --config.test_and_exit -f /etc/logstash/conf.d/bugsink.conf

# View logs
journalctl -u logstash -f

# Check stats (events processed, failures)
curl -XGET 'localhost:9600/_node/stats/pipelines?pretty' | jq '.pipelines.main.plugins.filters'

# Monitor sources
tail -f /var/log/postgresql/postgresql-$(date +%Y-%m-%d).log
tail -f /var/log/logstash/pm2-workers-$(date +%Y-%m-%d).log
tail -f /var/log/logstash/redis-operational-$(date +%Y-%m-%d).log
tail -f /var/log/logstash/nginx-access-$(date +%Y-%m-%d).log

# Check disk usage
du -sh /var/log/logstash/

Dev Container

# View Logstash logs (inside container)
MSYS_NO_PATHCONV=1 podman exec flyer-crawler-dev cat /var/log/logstash/logstash.log

# Check PM2 API logs are being processed (ADR-014)
MSYS_NO_PATHCONV=1 podman exec flyer-crawler-dev cat /var/log/logstash/pm2-api-$(date +%Y-%m-%d).log

# Check PM2 Worker logs are being processed
MSYS_NO_PATHCONV=1 podman exec flyer-crawler-dev cat /var/log/logstash/pm2-worker-$(date +%Y-%m-%d).log

# Check Redis logs are being processed
MSYS_NO_PATHCONV=1 podman exec flyer-crawler-dev cat /var/log/logstash/redis-operational-$(date +%Y-%m-%d).log

# View raw PM2 logs
MSYS_NO_PATHCONV=1 podman exec flyer-crawler-dev cat /var/log/pm2/api-out.log
MSYS_NO_PATHCONV=1 podman exec flyer-crawler-dev cat /var/log/pm2/worker-out.log

# View raw Redis logs
MSYS_NO_PATHCONV=1 podman exec flyer-crawler-redis cat /var/log/redis/redis-server.log

# Check Logstash stats
podman exec flyer-crawler-dev curl -s localhost:9600/_node/stats/pipelines?pretty

# Verify shared volume mounts
MSYS_NO_PATHCONV=1 podman exec flyer-crawler-dev ls -la /var/log/pm2/
MSYS_NO_PATHCONV=1 podman exec flyer-crawler-dev ls -la /var/log/redis/

Troubleshooting

Issue	Check	Solution
No Bugsink errors	Logstash running	systemctl status logstash
Config syntax error	Test config	/usr/share/logstash/bin/logstash --config.test_and_exit -f /etc/logstash/conf.d/bugsink.conf
Grok pattern failures	Stats endpoint	curl localhost:9600/_node/stats/pipelines?pretty | jq '.pipelines.main.plugins.filters'
Wrong Bugsink project	Env detection	Check tags in logs match expected environment
Permission denied	Logstash groups	groups logstash should include postgres, adm
PM2 not captured	File paths	Dev: ls /var/log/pm2/; Prod: ls /home/gitea-runner/.pm2/logs/
PM2 logs empty	PM2 running	Dev: podman exec flyer-crawler-dev pm2 status; Prod: pm2 status
NGINX logs missing	Output directory	ls -lh /var/log/logstash/nginx-access-*.log
Redis logs missing	Shared volume	Dev: Check redis_logs volume mounted; Prod: Check /var/log/redis/redis-server.log exists
High disk usage	Log rotation	Verify /etc/logrotate.d/logstash configured
varchar(7) error	Level validation	Add Ruby filter to validate/normalize sentry_level before output

Related Documentation

• Dev Container Guide: DEV-CONTAINER.md - PM2 and log aggregation in dev
• Full setup: BARE-METAL-SETUP.md - PostgreSQL Function Observability section
• Architecture: adr/0050-postgresql-function-observability.md
• Troubleshooting details: LOGSTASH-TROUBLESHOOTING.md