flyer-crawler.projectium.com/docs/operations/LOGSTASH-QUICK-REF.md

Logstash Quick Reference (ADR-050)

Aggregates logs from PostgreSQL, PM2, Redis, NGINX; forwards errors to Bugsink.

Configuration

Primary config: /etc/logstash/conf.d/bugsink.conf

Related Files

Path	Purpose
/etc/postgresql/14/main/conf.d/observability.conf	PostgreSQL logging config
/var/log/postgresql/*.log	PostgreSQL logs
/home/gitea-runner/.pm2/logs/*.log	PM2 worker logs
/var/log/redis/redis-server.log	Redis logs
/var/log/nginx/access.log	NGINX access logs
/var/log/nginx/error.log	NGINX error logs
/var/log/logstash/*.log	Logstash file outputs
/var/lib/logstash/sincedb_*	Position tracking files

Features

• Multi-source aggregation: PostgreSQL, PM2 workers, Redis, NGINX
• Environment routing: Auto-detects prod/test, routes to correct Bugsink project
• JSON parsing: Extracts fn_log() from PostgreSQL, Pino JSON from PM2
• Sentry format: Transforms to event_id, timestamp, level, message, extra
• Error filtering: Only forwards WARNING/ERROR to Bugsink
• Operational storage: Non-error logs saved to /var/log/logstash/
• Request monitoring: NGINX requests categorized by status, slow request detection

Commands

# Status and control
systemctl status logstash
systemctl restart logstash

# Test configuration
/usr/share/logstash/bin/logstash --config.test_and_exit -f /etc/logstash/conf.d/bugsink.conf

# View logs
journalctl -u logstash -f

# Check stats (events processed, failures)
curl -XGET 'localhost:9600/_node/stats/pipelines?pretty' | jq '.pipelines.main.plugins.filters'

# Monitor sources
tail -f /var/log/postgresql/postgresql-$(date +%Y-%m-%d).log
tail -f /var/log/logstash/pm2-workers-$(date +%Y-%m-%d).log
tail -f /var/log/logstash/redis-operational-$(date +%Y-%m-%d).log
tail -f /var/log/logstash/nginx-access-$(date +%Y-%m-%d).log

# Check disk usage
du -sh /var/log/logstash/

Troubleshooting

Issue	Check	Solution
No Bugsink errors	Logstash running	systemctl status logstash
Config syntax error	Test config	/usr/share/logstash/bin/logstash --config.test_and_exit -f /etc/logstash/conf.d/bugsink.conf
Grok pattern failures	Stats endpoint	curl localhost:9600/_node/stats/pipelines?pretty | jq '.pipelines.main.plugins.filters'
Wrong Bugsink project	Env detection	Check tags in logs match expected environment
Permission denied	Logstash groups	groups logstash should include postgres, adm
PM2 not captured	File paths	ls /home/gitea-runner/.pm2/logs/flyer-crawler-worker-*.log
NGINX logs missing	Output directory	ls -lh /var/log/logstash/nginx-access-*.log
High disk usage	Log rotation	Verify /etc/logrotate.d/logstash configured

Related Documentation

• Full setup: BARE-METAL-SETUP.md - PostgreSQL Function Observability section
• Architecture: adr/0050-postgresql-function-observability.md
• Troubleshooting details: LOGSTASH-TROUBLESHOOTING.md