Torben Sorensen
d4543cf4b9
All checks were successful
Deploy to Test Environment / deploy-to-test (push) Successful in 19m12s
112 lines
5.7 KiB
Markdown
112 lines
5.7 KiB
Markdown
# Logstash Quick Reference (ADR-050)
|
|
|
|
Aggregates logs from PostgreSQL, PM2, Redis, NGINX; forwards errors to Bugsink.
|
|
|
|
## Configuration
|
|
|
|
**Primary config**: `/etc/logstash/conf.d/bugsink.conf`
|
|
|
|
### Related Files
|
|
|
|
| Path | Purpose |
|
|
| --------------------------------------------------- | ------------------------- |
|
|
| `/etc/postgresql/14/main/conf.d/observability.conf` | PostgreSQL logging config |
|
|
| `/var/log/postgresql/*.log` | PostgreSQL logs |
|
|
| `/home/gitea-runner/.pm2/logs/*.log` | PM2 worker logs |
|
|
| `/var/log/redis/redis-server.log` | Redis logs |
|
|
| `/var/log/nginx/access.log` | NGINX access logs |
|
|
| `/var/log/nginx/error.log` | NGINX error logs |
|
|
| `/var/log/logstash/*.log` | Logstash file outputs |
|
|
| `/var/lib/logstash/sincedb_*` | Position tracking files |
|
|
|
|
## Features
|
|
|
|
- **Multi-source aggregation**: PostgreSQL, PM2 workers, Redis, NGINX
|
|
- **Environment routing**: Auto-detects prod/test, routes to correct Bugsink project
|
|
- **JSON parsing**: Extracts `fn_log()` from PostgreSQL, Pino JSON from PM2
|
|
- **Sentry format**: Transforms to `event_id`, `timestamp`, `level`, `message`, `extra`
|
|
- **Error filtering**: Only forwards WARNING/ERROR to Bugsink
|
|
- **Operational storage**: Non-error logs saved to `/var/log/logstash/`
|
|
- **Request monitoring**: NGINX requests categorized by status, slow request detection
|
|
|
|
## Commands
|
|
|
|
### Production (Bare Metal)
|
|
|
|
```bash
|
|
# Status and control
|
|
systemctl status logstash
|
|
systemctl restart logstash
|
|
|
|
# Test configuration
|
|
/usr/share/logstash/bin/logstash --config.test_and_exit -f /etc/logstash/conf.d/bugsink.conf
|
|
|
|
# View logs
|
|
journalctl -u logstash -f
|
|
|
|
# Check stats (events processed, failures)
|
|
curl -XGET 'localhost:9600/_node/stats/pipelines?pretty' | jq '.pipelines.main.plugins.filters'
|
|
|
|
# Monitor sources
|
|
tail -f /var/log/postgresql/postgresql-$(date +%Y-%m-%d).log
|
|
tail -f /var/log/logstash/pm2-workers-$(date +%Y-%m-%d).log
|
|
tail -f /var/log/logstash/redis-operational-$(date +%Y-%m-%d).log
|
|
tail -f /var/log/logstash/nginx-access-$(date +%Y-%m-%d).log
|
|
|
|
# Check disk usage
|
|
du -sh /var/log/logstash/
|
|
```
|
|
|
|
### Dev Container
|
|
|
|
```bash
|
|
# View Logstash logs (inside container)
|
|
MSYS_NO_PATHCONV=1 podman exec flyer-crawler-dev cat /var/log/logstash/logstash.log
|
|
|
|
# Check PM2 API logs are being processed (ADR-014)
|
|
MSYS_NO_PATHCONV=1 podman exec flyer-crawler-dev cat /var/log/logstash/pm2-api-$(date +%Y-%m-%d).log
|
|
|
|
# Check PM2 Worker logs are being processed
|
|
MSYS_NO_PATHCONV=1 podman exec flyer-crawler-dev cat /var/log/logstash/pm2-worker-$(date +%Y-%m-%d).log
|
|
|
|
# Check Redis logs are being processed
|
|
MSYS_NO_PATHCONV=1 podman exec flyer-crawler-dev cat /var/log/logstash/redis-operational-$(date +%Y-%m-%d).log
|
|
|
|
# View raw PM2 logs
|
|
MSYS_NO_PATHCONV=1 podman exec flyer-crawler-dev cat /var/log/pm2/api-out.log
|
|
MSYS_NO_PATHCONV=1 podman exec flyer-crawler-dev cat /var/log/pm2/worker-out.log
|
|
|
|
# View raw Redis logs
|
|
MSYS_NO_PATHCONV=1 podman exec flyer-crawler-redis cat /var/log/redis/redis-server.log
|
|
|
|
# Check Logstash stats
|
|
podman exec flyer-crawler-dev curl -s localhost:9600/_node/stats/pipelines?pretty
|
|
|
|
# Verify shared volume mounts
|
|
MSYS_NO_PATHCONV=1 podman exec flyer-crawler-dev ls -la /var/log/pm2/
|
|
MSYS_NO_PATHCONV=1 podman exec flyer-crawler-dev ls -la /var/log/redis/
|
|
```
|
|
|
|
## Troubleshooting
|
|
|
|
| Issue | Check | Solution |
|
|
| --------------------- | ---------------- | ---------------------------------------------------------------------------------------------- |
|
|
| No Bugsink errors | Logstash running | `systemctl status logstash` |
|
|
| Config syntax error | Test config | `/usr/share/logstash/bin/logstash --config.test_and_exit -f /etc/logstash/conf.d/bugsink.conf` |
|
|
| Grok pattern failures | Stats endpoint | `curl localhost:9600/_node/stats/pipelines?pretty \| jq '.pipelines.main.plugins.filters'` |
|
|
| Wrong Bugsink project | Env detection | Check tags in logs match expected environment |
|
|
| Permission denied | Logstash groups | `groups logstash` should include `postgres`, `adm` |
|
|
| PM2 not captured | File paths | Dev: `ls /var/log/pm2/`; Prod: `ls /home/gitea-runner/.pm2/logs/` |
|
|
| PM2 logs empty | PM2 running | Dev: `podman exec flyer-crawler-dev pm2 status`; Prod: `pm2 status` |
|
|
| NGINX logs missing | Output directory | `ls -lh /var/log/logstash/nginx-access-*.log` |
|
|
| Redis logs missing | Shared volume | Dev: Check `redis_logs` volume mounted; Prod: Check `/var/log/redis/redis-server.log` exists |
|
|
| High disk usage | Log rotation | Verify `/etc/logrotate.d/logstash` configured |
|
|
| varchar(7) error | Level validation | Add Ruby filter to validate/normalize `sentry_level` before output |
|
|
|
|
## Related Documentation
|
|
|
|
- **Dev Container Guide**: [DEV-CONTAINER.md](../development/DEV-CONTAINER.md) - PM2 and log aggregation in dev
|
|
- **Full setup**: [BARE-METAL-SETUP.md](BARE-METAL-SETUP.md) - PostgreSQL Function Observability section
|
|
- **Architecture**: [adr/0050-postgresql-function-observability.md](../adr/0050-postgresql-function-observability.md)
|
|
- **Troubleshooting details**: [LOGSTASH-TROUBLESHOOTING.md](LOGSTASH-TROUBLESHOOTING.md)
|