torbo/flyer-crawler.projectium.com
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Bugsink Fixes
Some checks failed
Deploy to Test Environment / deploy-to-test (push) Has been cancelled
Details

Browse Source
This commit is contained in:
Torben Sorensen
2026-01-22 21:47:16 -08:00
parent dd067183ed
commit 38b35f87aa
9 changed files with 425 additions and 102 deletions
Download Patch File Download Diff File Expand all files Collapse all files

242
docs/tools/BUGSINK-SETUP.md
View File

@@ -49,18 +49,24 @@ Bugsink is a lightweight, self-hosted error tracking platform that is fully comp
| Web UI | `https://localhost:8443` (nginx proxy) |
| Internal URL | `http://localhost:8000` (direct) |
| Credentials | `admin@localhost` / `admin` |
| Backend Project | Project ID 1 - `flyer-crawler-dev-backend` |
| Frontend Project | Project ID 2 - `flyer-crawler-dev-frontend` |
| Backend Project | Project ID 1 - `Backend API (Dev)` |
| Frontend Project | Project ID 2 - `Frontend (Dev)` |
| Infra Project | Project ID 4 - `Infrastructure (Dev)` |
| Backend DSN | `http://<key>@localhost:8000/1` |
| Frontend DSN | `http://<key>@localhost:8000/2` |
| Frontend DSN | `https://<key>@localhost/bugsink-api/2` (via nginx proxy) |
| Infra DSN | `http://<key>@localhost:8000/4` (Logstash only) |
| Database | `postgresql://bugsink:bugsink_dev_password@postgres:5432/bugsink` |
**Important:** The Frontend DSN uses an nginx proxy (`/bugsink-api/`) because the browser cannot reach `localhost:8000` directly (container-internal port). See [Frontend Nginx Proxy](#frontend-nginx-proxy) for details.
**Configuration Files:**
| File | Purpose |
| ----------------- | ----------------------------------------------------------------- |
| `compose.dev.yml` | Initial DSNs using `127.0.0.1:8000` (container startup) |
| `.env.local` | **OVERRIDES** compose.dev.yml with `localhost:8000` (app runtime) |
| File | Purpose |
| ------------------------------ | ------------------------------------------------------- |
| `compose.dev.yml` | Initial DSNs using `127.0.0.1:8000` (container startup) |
| `.env.local` | **OVERRIDES** compose.dev.yml (app runtime) |
| `docker/nginx/dev.conf` | Nginx proxy for Bugsink API (frontend error reporting) |
| `docker/logstash/bugsink.conf` | Log routing to Backend/Infrastructure projects |
**Note:** `.env.local` takes precedence over `compose.dev.yml` environment variables.
@@ -360,75 +366,127 @@ const config = {
---
## Frontend Nginx Proxy
The frontend Sentry SDK runs in the browser, which cannot directly reach `localhost:8000` (the Bugsink container-internal port). To solve this, we use an nginx proxy.
### How It Works
```text
Browser --HTTPS--> https://localhost/bugsink-api/2/store/
|
v (nginx proxy)
http://localhost:8000/api/2/store/
|
v
Bugsink (internal)
```
### Nginx Configuration
Location: `docker/nginx/dev.conf`
```nginx
# Proxy Bugsink Sentry API for frontend error reporting
location /bugsink-api/ {
proxy_pass http://localhost:8000/api/;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
# Allow large error payloads with stack traces
client_max_body_size 10M;
# Timeouts for error reporting
proxy_connect_timeout 10s;
proxy_send_timeout 30s;
proxy_read_timeout 30s;
}
```
### Frontend DSN Format
```bash
# .env.local
# Uses nginx proxy path instead of direct port
VITE_SENTRY_DSN=https://<key>@localhost/bugsink-api/2
```
### Testing Frontend Error Reporting
1. Open browser console at `https://localhost`
2. Trigger a test error:
```javascript
throw new Error('Test frontend error from browser');
```
3. Check Bugsink Frontend (Dev) project for the error
4. Verify browser console shows Sentry SDK activity (if VITE_SENTRY_DEBUG=true)
---
## Logstash Integration
Logstash aggregates logs from multiple sources and forwards error patterns to Bugsink.
**Note:** See [ADR-015](../adr/0015-application-performance-monitoring-and-error-tracking.md) for the full architecture.
### 3-Project Architecture
Logstash routes errors to different Bugsink projects based on log source:
| Project | ID | Receives |
| -------------------- | --- | --------------------------------------------- |
| Backend API (Dev) | 1 | Pino app errors, PostgreSQL errors |
| Frontend (Dev) | 2 | Browser errors (via Sentry SDK, not Logstash) |
| Infrastructure (Dev) | 4 | Redis warnings, NGINX errors, Vite errors |
### Log Sources
| Source | Log Path | Error Detection |
| ---------- | ---------------------- | ------------------------- |
| Pino (app) | `/app/logs/*.log` | level >= 50 (error/fatal) |
| Redis | `/var/log/redis/*.log` | WARNING/ERROR log levels |
| PostgreSQL | (future) | ERROR/FATAL log levels |
| Source | Log Path | Project Destination | Error Detection |
| ---------- | --------------------------- | ------------------- | ------------------------- |
| PM2 API | `/var/log/pm2/api-*.log` | Backend (1) | level >= 50 (error/fatal) |
| PM2 Worker | `/var/log/pm2/worker-*.log` | Backend (1) | level >= 50 (error/fatal) |
| PM2 Vite | `/var/log/pm2/vite-*.log` | Infrastructure (4) | error keyword patterns |
| PostgreSQL | `/var/log/postgresql/*.log` | Backend (1) | ERROR/FATAL log levels |
| Redis | `/var/log/redis/*.log` | Infrastructure (4) | WARNING level (`#`) |
| NGINX | `/var/log/nginx/error.log` | Infrastructure (4) | error/crit/alert/emerg |
### Pipeline Configuration
**Location:** `/etc/logstash/conf.d/bugsink.conf`
**Location:** `/etc/logstash/conf.d/bugsink.conf` (or `docker/logstash/bugsink.conf` in project)
```conf
# === INPUTS ===
input {
file {
path => "/app/logs/*.log"
codec => json
type => "pino"
tags => ["app"]
}
The configuration:
file {
path => "/var/log/redis/*.log"
type => "redis"
tags => ["redis"]
}
1. **Inputs**: Reads from PM2 logs, PostgreSQL logs, Redis logs, NGINX logs
2. **Filters**: Detects errors and assigns tags based on log type
3. **Outputs**: Routes to appropriate Bugsink project based on log source
**Key Routing Logic:**
```ruby
# Infrastructure logs -> Project 4
if "error" in [tags] and ([type] == "redis" or [type] == "nginx_error" or [type] == "pm2_vite") {
http { url => "http://localhost:8000/api/4/store/" ... }
}
# === FILTERS ===
filter {
if [type] == "pino" and [level] >= 50 {
mutate { add_tag => ["error"] }
}
if [type] == "redis" {
grok {
match => { "message" => "%{POSINT:pid}:%{WORD:role} %{MONTHDAY} %{MONTH} %{TIME} %{WORD:loglevel} %{GREEDYDATA:redis_message}" }
}
if [loglevel] in ["WARNING", "ERROR"] {
mutate { add_tag => ["error"] }
}
}
}
# === OUTPUT ===
output {
if "error" in [tags] {
http {
url => "http://localhost:8000/api/store/"
http_method => "post"
format => "json"
}
}
# Backend logs -> Project 1
else if "error" in [tags] and ([type] in ["pm2_api", "pm2_worker", "pino", "postgres"]) {
http { url => "http://localhost:8000/api/1/store/" ... }
}
```
### Benefits
1. **Secondary Capture Path**: Catches errors before SDK initialization
2. **Log-Based Errors**: Captures errors that don't throw exceptions
3. **Infrastructure Monitoring**: Redis connection issues, slow commands
4. **Historical Analysis**: Process existing log files
1. **Separation of Concerns**: Application errors separate from infrastructure issues
2. **Secondary Capture Path**: Catches errors before SDK initialization
3. **Log-Based Errors**: Captures errors that don't throw exceptions
4. **Infrastructure Monitoring**: Redis, NGINX, build tooling issues
5. **Historical Analysis**: Process existing log files
---
@@ -743,6 +801,76 @@ podman exec flyer-crawler-dev psql -U postgres -h postgres -c "\l" | grep bugsin
ssh root@projectium.com "cd /opt/bugsink && bugsink-manage check"
```
### CSRF Verification Failed
**Symptoms:** "CSRF verification failed. Request aborted." error when performing actions in Bugsink UI (resolving issues, changing settings, etc.)
**Root Cause:**
Django 4.0+ requires `CSRF_TRUSTED_ORIGINS` to be explicitly configured for HTTPS POST requests. The error occurs because:
1. Bugsink is accessed via `https://localhost:8443` (nginx HTTPS proxy)
2. Django's CSRF protection validates the `Origin` header against `CSRF_TRUSTED_ORIGINS`
3. Without explicit configuration, Django rejects POST requests from HTTPS origins
**Why localhost vs 127.0.0.1 Matters:**
- `localhost` and `127.0.0.1` are treated as DIFFERENT origins by browsers
- If you access Bugsink via `https://localhost:8443`, Django must trust `https://localhost:8443`
- If you access via `https://127.0.0.1:8443`, Django must trust `https://127.0.0.1:8443`
- The fix includes BOTH to allow either access pattern
**Configuration (Already Applied):**
The Bugsink Django configuration in `Dockerfile.dev` includes:
```python
# CSRF Trusted Origins (Django 4.0+ requires full origin for HTTPS POST requests)
CSRF_TRUSTED_ORIGINS = [
"https://localhost:8443",
"https://127.0.0.1:8443",
"http://localhost:8000",
"http://127.0.0.1:8000",
]
# HTTPS proxy support (nginx reverse proxy on port 8443)
SECURE_PROXY_SSL_HEADER = ("HTTP_X_FORWARDED_PROTO", "https")
```
**Verification:**
```bash
# Verify CSRF_TRUSTED_ORIGINS is configured
podman exec flyer-crawler-dev sh -c 'cat /opt/bugsink/conf/bugsink_conf.py | grep -A 6 CSRF_TRUSTED'
# Expected output:
# CSRF_TRUSTED_ORIGINS = [
# "https://localhost:8443",
# "https://127.0.0.1:8443",
# "http://localhost:8000",
# "http://127.0.0.1:8000",
# ]
```
**If Issue Persists After Fix:**
1. **Rebuild the container image** (configuration is baked into the image):
```bash
podman-compose -f compose.dev.yml down
podman build -f Dockerfile.dev -t localhost/flyer-crawler-dev:latest .
podman-compose -f compose.dev.yml up -d
```
2. **Clear browser cookies** for localhost:8443
3. **Check nginx X-Forwarded-Proto header** - the nginx config must set this header for Django to recognize HTTPS:
```bash
podman exec flyer-crawler-dev cat /etc/nginx/sites-available/bugsink | grep X-Forwarded-Proto
# Should show: proxy_set_header X-Forwarded-Proto $scheme;
```
---
## Related Documentation