flyer-crawler.projectium.com/docs/adr/0015-application-performance-monitoring-and-error-tracking.md

ADR-015: Application Performance Monitoring (APM) and Error Tracking

Date: 2025-12-12

Status: Accepted

Updated: 2026-01-11

Context

While ADR-004 established structured logging with Pino, the application lacks a high-level, aggregated view of its health, performance, and errors. It's difficult to spot trends, identify slow API endpoints, or be proactively notified of new types of errors.

Key requirements:

1. Self-hosted: No external SaaS dependencies for error tracking
2. Sentry SDK compatible: Leverage mature, well-documented SDKs
3. Lightweight: Minimal resource overhead in the dev container
4. Production-ready: Same architecture works on bare-metal production servers
5. AI-accessible: MCP server integration for Claude Code and other AI tools

Decision

We will implement a self-hosted error tracking stack using Bugsink as the Sentry-compatible backend, with the following components:

1. Error Tracking Backend: Bugsink

Bugsink is a lightweight, self-hosted Sentry alternative that:

• Runs as a single process (no Kafka, Redis, ClickHouse required)
• Is fully compatible with Sentry SDKs
• Supports ARM64 and AMD64 architectures
• Can use SQLite (dev) or PostgreSQL (production)

Deployment:

• Dev container: Installed as a systemd service inside the container
• Production: Runs as a systemd service on bare-metal, listening on localhost only
• Database: Uses PostgreSQL with a dedicated bugsink user and bugsink database (same PostgreSQL instance as the main application)

2. Backend Integration: @sentry/node

The Express backend will integrate @sentry/node SDK to:

• Capture unhandled exceptions before PM2/process manager restarts
• Report errors with full stack traces and context
• Integrate with Pino logger for breadcrumbs
• Track transaction performance (optional)

3. Frontend Integration: @sentry/react

The React frontend will integrate @sentry/react SDK to:

• Wrap the app in a Sentry Error Boundary
• Capture unhandled JavaScript errors
• Report errors with component stack traces
• Track user session context

4. Log Aggregation: Logstash

Logstash parses application and infrastructure logs, forwarding error patterns to Bugsink:

• Installation: Installed inside the dev container (and on bare-metal prod servers)
• Inputs:
  • Pino JSON logs from the Node.js application
  • Redis logs (connection errors, memory warnings, slow commands)
  • PostgreSQL function logs (future - see Implementation Steps)
• Filter: Identifies error-level logs (5xx responses, unhandled exceptions, Redis errors)
• Output: Sends to Bugsink via Sentry-compatible HTTP API

This provides a secondary error capture path for:

• Errors that occur before Sentry SDK initialization
• Log-based errors that don't throw exceptions
• Redis connection/performance issues
• Database function errors and slow queries
• Historical error analysis from log files

5. MCP Server Integration: sentry-selfhosted-mcp

For AI tool integration (Claude Code, Cursor, etc.), we use the open-source sentry-selfhosted-mcp server:

• No code changes required: Configurable via environment variables
• Capabilities: List projects, get issues, view events, update status, add comments
• Configuration:
  • SENTRY_URL: Points to Bugsink instance
  • SENTRY_AUTH_TOKEN: API token from Bugsink
  • SENTRY_ORG_SLUG: Organization identifier

Architecture

┌─────────────────────────────────────────────────────────────────────────┐
│                      Dev Container / Production Server                   │
├─────────────────────────────────────────────────────────────────────────┤
│                                                                          │
│  ┌──────────────────┐        ┌──────────────────┐                       │
│  │    Frontend      │        │     Backend      │                       │
│  │    (React)       │        │    (Express)     │                       │
│  │  @sentry/react   │        │   @sentry/node   │                       │
│  └────────┬─────────┘        └────────┬─────────┘                       │
│           │                           │                                  │
│           │    Sentry SDK Protocol    │                                  │
│           └───────────┬───────────────┘                                  │
│                       │                                                  │
│                       ▼                                                  │
│           ┌──────────────────────┐                                       │
│           │      Bugsink         │                                       │
│           │   (localhost:8000)   │◄──────────────────┐                   │
│           │                      │                   │                   │
│           │  PostgreSQL backend  │                   │                   │
│           └──────────────────────┘                   │                   │
│                                                      │                   │
│           ┌──────────────────────┐                   │                   │
│           │      Logstash        │───────────────────┘                   │
│           │   (Log Aggregator)   │   Sentry Output                       │
│           │                      │                                       │
│           │  Inputs:             │                                       │
│           │  - Pino app logs     │                                       │
│           │  - Redis logs        │                                       │
│           │  - PostgreSQL (future)                                       │
│           └──────────────────────┘                                       │
│                   ▲   ▲   ▲                                              │
│                   │   │   │                                              │
│       ┌───────────┘   │   └───────────┐                                  │
│       │               │               │                                  │
│  ┌────┴─────┐   ┌─────┴────┐   ┌──────┴─────┐                           │
│  │  Pino    │   │  Redis   │   │ PostgreSQL │                           │
│  │  Logs    │   │  Logs    │   │ Logs (TBD) │                           │
│  └──────────┘   └──────────┘   └────────────┘                           │
│                                                                          │
│           ┌──────────────────────┐                                       │
│           │     PostgreSQL       │                                       │
│           │  ┌────────────────┐  │                                       │
│           │  │ flyer_crawler  │  │  (main app database)                  │
│           │  ├────────────────┤  │                                       │
│           │  │   bugsink      │  │  (error tracking database)            │
│           │  └────────────────┘  │                                       │
│           └──────────────────────┘                                       │
│                                                                          │
└─────────────────────────────────────────────────────────────────────────┘

External (Developer Machine):
┌──────────────────────────────────────┐
│  Claude Code / Cursor / VS Code      │
│  ┌────────────────────────────────┐  │
│  │  sentry-selfhosted-mcp         │  │
│  │  (MCP Server)                  │  │
│  │                                │  │
│  │  SENTRY_URL=http://localhost:8000
│  │  SENTRY_AUTH_TOKEN=...         │  │
│  │  SENTRY_ORG_SLUG=...           │  │
│  └────────────────────────────────┘  │
└──────────────────────────────────────┘

Configuration

Environment Variables

Variable	Description	Default (Dev)
BUGSINK_DSN	Sentry-compatible DSN for SDKs	Set after project creation
BUGSINK_ENABLED	Enable/disable error reporting	true
BUGSINK_BASE_URL	Bugsink web UI URL (internal)	http://localhost:8000

PostgreSQL Setup

-- Create dedicated Bugsink database and user
CREATE USER bugsink WITH PASSWORD 'bugsink_dev_password';
CREATE DATABASE bugsink OWNER bugsink;
GRANT ALL PRIVILEGES ON DATABASE bugsink TO bugsink;

Bugsink Configuration

# Environment variables for Bugsink service
SECRET_KEY=<random-50-char-string>
DATABASE_URL=postgresql://bugsink:bugsink_dev_password@localhost:5432/bugsink
BASE_URL=http://localhost:8000
PORT=8000

Logstash Pipeline

# /etc/logstash/conf.d/bugsink.conf

# === INPUTS ===
input {
  # Pino application logs
  file {
    path => "/app/logs/*.log"
    codec => json
    type => "pino"
    tags => ["app"]
  }

  # Redis logs
  file {
    path => "/var/log/redis/*.log"
    type => "redis"
    tags => ["redis"]
  }

  # PostgreSQL logs (for function logging - future)
  # file {
  #   path => "/var/log/postgresql/*.log"
  #   type => "postgres"
  #   tags => ["postgres"]
  # }
}

# === FILTERS ===
filter {
  # Pino error detection (level 50 = error, 60 = fatal)
  if [type] == "pino" and [level] >= 50 {
    mutate { add_tag => ["error"] }
  }

  # Redis error detection
  if [type] == "redis" {
    grok {
      match => { "message" => "%{POSINT:pid}:%{WORD:role} %{MONTHDAY} %{MONTH} %{TIME} %{WORD:loglevel} %{GREEDYDATA:redis_message}" }
    }
    if [loglevel] in ["WARNING", "ERROR"] {
      mutate { add_tag => ["error"] }
    }
  }

  # PostgreSQL function error detection (future)
  # if [type] == "postgres" {
  #   # Parse PostgreSQL log format and detect ERROR/FATAL levels
  # }
}

# === OUTPUT ===
output {
  if "error" in [tags] {
    http {
      url => "http://localhost:8000/api/store/"
      http_method => "post"
      format => "json"
      # Sentry envelope format
    }
  }
}

Implementation Steps

1. Update Dockerfile.dev:

   • Install Bugsink (pip package or binary)
   • Install Logstash (Elastic APT repository)
   • Add systemd service files for both

2. PostgreSQL initialization:

   • Add Bugsink user/database creation to sql/00-init-extensions.sql

3. Backend SDK integration:

   • Install @sentry/node
   • Initialize in server.ts before Express app
   • Configure error handler middleware integration

4. Frontend SDK integration:

   • Install @sentry/react
   • Wrap App component with Sentry.ErrorBoundary
   • Configure in src/index.tsx

5. Environment configuration:

   • Add Bugsink variables to src/config/env.ts
   • Update .env.example and compose.dev.yml

6. Logstash configuration:

   • Create pipeline config for Pino → Bugsink
   • Configure Pino to write to log file in addition to stdout
   • Configure Redis log monitoring (connection errors, slow commands)

7. MCP server documentation:

   • Document sentry-selfhosted-mcp setup in CLAUDE.md

8. PostgreSQL function logging (future):

   • Configure PostgreSQL to log function execution errors
   • Add Logstash input for PostgreSQL logs
   • Define filter rules for function-level error detection
   • Note: Ask for implementation details when this step is reached

Consequences

Positive

• Full observability: Aggregated view of errors, trends, and performance
• Self-hosted: No external SaaS dependencies or subscription costs
• SDK compatibility: Leverages mature Sentry SDKs with excellent documentation
• AI integration: MCP server enables Claude Code to query and analyze errors
• Unified architecture: Same setup works in dev container and production
• Lightweight: Bugsink runs in a single process, unlike full Sentry (16GB+ RAM)

Negative

• Additional services: Bugsink and Logstash add complexity to the container
• PostgreSQL overhead: Additional database for error tracking
• Initial setup: Requires configuration of multiple components
• Logstash learning curve: Pipeline configuration requires Logstash knowledge

Alternatives Considered

1. Full Sentry self-hosted: Rejected due to complexity (Kafka, Redis, ClickHouse, 16GB+ RAM minimum)
2. GlitchTip: Considered, but Bugsink is lighter weight and easier to deploy
3. Sentry SaaS: Rejected due to self-hosted requirement
4. Custom error aggregation: Rejected in favor of proven Sentry SDK ecosystem

References

• Bugsink Documentation
• Bugsink Docker Install
• @sentry/node Documentation
• @sentry/react Documentation
• sentry-selfhosted-mcp
• Logstash Reference