flyer-crawler.projectium.com/docs/adr/ADR-029-error-tracking-with-bugsink.md

# ADR-029: Error Tracking and Observability with Bugsink

**Date**: 2026-02-10

**Status**: Accepted

**Source**: Imported from flyer-crawler project (ADR-015)

**Related**: [ADR-027](ADR-027-application-wide-structured-logging.md), [ADR-028](ADR-028-client-side-structured-logging.md), [ADR-030](ADR-030-postgresql-function-observability.md), [ADR-032](ADR-032-application-performance-monitoring.md)

## Context

While ADR-027 established structured logging with Pino, the application lacks a high-level, aggregated view of its health and errors. It is difficult to spot trends, identify recurring issues, or be proactively notified of new types of errors.

Key requirements:

1. **Self-hosted**: No external SaaS dependencies for error tracking
2. **Sentry SDK compatible**: Leverage mature, well-documented SDKs
3. **Lightweight**: Minimal resource overhead in the dev container
4. **Production-ready**: Same architecture works on bare-metal production servers
5. **AI-accessible**: MCP server integration for Claude Code and other AI tools

**Note**: Application Performance Monitoring (APM) and distributed tracing are covered separately in [ADR-032](ADR-032-application-performance-monitoring.md).

## Decision

We implement a self-hosted error tracking stack using **Bugsink** as the Sentry-compatible backend, with the following components:

### 1. Error Tracking Backend: Bugsink

**Bugsink** is a lightweight, self-hosted Sentry alternative that:

- Runs as a single process (no Kafka, Redis, ClickHouse required)
- Is fully compatible with Sentry SDKs
- Supports ARM64 and AMD64 architectures
- Can use SQLite (dev) or PostgreSQL (production)

**Deployment**:

- **Dev container**: Installed as a systemd service inside the container
- **Production**: Runs as a systemd service on bare-metal, listening on localhost only
- **Database**: Uses PostgreSQL with a dedicated `bugsink` user and `bugsink` database (same PostgreSQL instance as the main application)

### 2. Backend Integration: @sentry/node

The Express backend integrates `@sentry/node` SDK to:

- Capture unhandled exceptions before PM2/process manager restarts
- Report errors with full stack traces and context
- Integrate with Pino logger for breadcrumbs
- Filter errors by severity (only 5xx errors sent by default)

### 3. Frontend Integration: @sentry/react

The React frontend integrates `@sentry/react` SDK to:

- Wrap the app in an Error Boundary for graceful error handling
- Capture unhandled JavaScript errors
- Report errors with component stack traces
- Filter out browser extension errors
- **Frontend Error Correlation**: The global API client intercepts 4xx/5xx responses and can attach the `x-request-id` header to Sentry scope for correlation with backend logs

### 4. Log Aggregation: Logstash

**Logstash** parses application and infrastructure logs, forwarding error patterns to Bugsink:

- **Installation**: Installed inside the dev container (and on bare-metal prod servers)
- **Inputs**:
  - Pino JSON logs from the Node.js application (PM2 managed)
  - Redis logs (connection errors, memory warnings, slow commands)
  - PostgreSQL function logs (via `fn_log()` - see ADR-030)
  - NGINX access/error logs
- **Filter**: Identifies error-level logs (5xx responses, unhandled exceptions, Redis errors)
- **Output**: Sends to Bugsink via Sentry-compatible HTTP API

This provides a secondary error capture path for:

- Errors that occur before Sentry SDK initialization
- Log-based errors that do not throw exceptions
- Redis connection/performance issues
- Database function errors and slow queries
- Historical error analysis from log files

### 5. MCP Server Integration: bugsink-mcp

For AI tool integration (Claude Code, Cursor, etc.), we use the open-source [bugsink-mcp](https://github.com/j-shelfwood/bugsink-mcp) server:

- **No code changes required**: Configurable via environment variables
- **Capabilities**: List projects, get issues, view events, get stacktraces, manage releases
- **Configuration**:
  - `BUGSINK_URL`: Points to Bugsink instance (`http://localhost:8000` for dev, `https://bugsink.example.com` for prod)
  - `BUGSINK_API_TOKEN`: API token from Bugsink (created via Django management command)
  - `BUGSINK_ORG_SLUG`: Organization identifier (usually "sentry")

## Architecture

```text
+---------------------------------------------------------------------------+
|                      Dev Container / Production Server                     |
+---------------------------------------------------------------------------+
|                                                                            |
|  +------------------+        +------------------+                          |
|  |    Frontend      |        |     Backend      |                          |
|  |    (React)       |        |    (Express)     |                          |
|  |  @sentry/react   |        |   @sentry/node   |                          |
|  +--------+---------+        +--------+---------+                          |
|           |                           |                                    |
|           |    Sentry SDK Protocol    |                                    |
|           +-----------+---------------+                                    |
|                       |                                                    |
|                       v                                                    |
|           +----------------------+                                         |
|           |      Bugsink         |                                         |
|           |   (localhost:8000)   |<------------------+                     |
|           |                      |                   |                     |
|           |  PostgreSQL backend  |                   |                     |
|           +----------------------+                   |                     |
|                                                      |                     |
|           +----------------------+                   |                     |
|           |      Logstash        |-------------------+                     |
|           |   (Log Aggregator)   |   Sentry Output                         |
|           |                      |                                         |
|           |  Inputs:             |                                         |
|           |  - PM2/Pino logs     |                                         |
|           |  - Redis logs        |                                         |
|           |  - PostgreSQL logs   |                                         |
|           |  - NGINX logs        |                                         |
|           +----------------------+                                         |
|                   ^   ^   ^   ^                                            |
|                   |   |   |   |                                            |
|       +-----------+   |   |   +-----------+                                |
|       |               |   |               |                                |
|  +----+-----+   +-----+----+   +-----+----+   +-----+----+                 |
|  |  PM2     |   |  Redis   |   | PostgreSQL|   |  NGINX  |                 |
|  |  Logs    |   |  Logs    |   |   Logs    |   |  Logs   |                 |
|  +----------+   +----------+   +-----------+   +---------+                 |
|                                                                            |
|           +----------------------+                                         |
|           |     PostgreSQL       |                                         |
|           |  +----------------+  |                                         |
|           |  | app_database   |  |  (main app database)                    |
|           |  +----------------+  |                                         |
|           |  |   bugsink      |  |  (error tracking database)              |
|           |  +----------------+  |                                         |
|           +----------------------+                                         |
|                                                                            |
+---------------------------------------------------------------------------+

External (Developer Machine):
+--------------------------------------+
|  Claude Code / Cursor / VS Code      |
|  +--------------------------------+  |
|  |  bugsink-mcp                   |  |
|  |  (MCP Server)                  |  |
|  |                                |  |
|  |  BUGSINK_URL=http://localhost:8000
|  |  BUGSINK_API_TOKEN=...         |  |
|  |  BUGSINK_ORG_SLUG=...          |  |
|  +--------------------------------+  |
+--------------------------------------+
```

## Implementation Details

### Environment Variables

| Variable             | Description                      | Default (Dev)              |
| -------------------- | -------------------------------- | -------------------------- |
| `SENTRY_DSN`         | Sentry-compatible DSN (backend)  | Set after project creation |
| `VITE_SENTRY_DSN`    | Sentry-compatible DSN (frontend) | Set after project creation |
| `SENTRY_ENVIRONMENT` | Environment name                 | `development`              |
| `SENTRY_DEBUG`       | Enable debug logging             | `false`                    |
| `SENTRY_ENABLED`     | Enable/disable error reporting   | `true`                     |

### PostgreSQL Setup

```sql
-- Create dedicated Bugsink database and user
CREATE USER bugsink WITH PASSWORD 'bugsink_dev_password';
CREATE DATABASE bugsink OWNER bugsink;
GRANT ALL PRIVILEGES ON DATABASE bugsink TO bugsink;
```

### Bugsink Configuration

```bash
# Environment variables for Bugsink service
SECRET_KEY=<random-50-char-string>
DATABASE_URL=postgresql://bugsink:bugsink_dev_password@localhost:5432/bugsink
BASE_URL=http://localhost:8000
PORT=8000
```

### Backend Sentry Integration

Located in `src/services/sentry.server.ts`:

```typescript
import * as Sentry from '@sentry/node';
import { config } from '../config/env';

export function initSentry() {
  if (!config.sentry.enabled || !config.sentry.dsn) {
    return;
  }

  Sentry.init({
    dsn: config.sentry.dsn,
    environment: config.sentry.environment || config.server.nodeEnv,
    debug: config.sentry.debug,

    // Performance monitoring - disabled by default (see ADR-032)
    tracesSampleRate: 0,

    // Filter out 4xx errors - only report server errors
    beforeSend(event) {
      const statusCode = event.contexts?.response?.status_code;
      if (statusCode && statusCode >= 400 && statusCode < 500) {
        return null;
      }
      return event;
    },
  });
}

// Set user context after authentication
export function setUserContext(user: { id: string; email: string; name?: string }) {
  Sentry.setUser({
    id: user.id,
    email: user.email,
    username: user.name,
  });
}

// Clear user context on logout
export function clearUserContext() {
  Sentry.setUser(null);
}
```

### Frontend Sentry Integration

Located in `src/services/sentry.client.ts`:

```typescript
import * as Sentry from '@sentry/react';
import { config } from '../config';

export function initSentry() {
  if (!config.sentry.enabled || !config.sentry.dsn) {
    return;
  }

  Sentry.init({
    dsn: config.sentry.dsn,
    environment: config.sentry.environment,

    // Performance monitoring - disabled by default (see ADR-032)
    tracesSampleRate: 0,

    // Filter out browser extension errors
    beforeSend(event) {
      // Ignore errors from browser extensions
      if (
        event.exception?.values?.[0]?.stacktrace?.frames?.some((frame) =>
          frame.filename?.includes('extension://'),
        )
      ) {
        return null;
      }
      return event;
    },
  });
}

// Set user context after login
export function setUserContext(user: { id: string; email: string; name?: string }) {
  Sentry.setUser({
    id: user.id,
    email: user.email,
    username: user.name,
  });
}

// Clear user context on logout
export function clearUserContext() {
  Sentry.setUser(null);
}
```

### Error Boundary Component

Located in `src/components/ErrorBoundary.tsx`:

```typescript
import * as Sentry from '@sentry/react';
import { Component, ErrorInfo, ReactNode } from 'react';

interface Props {
  children: ReactNode;
  fallback?: ReactNode;
}

interface State {
  hasError: boolean;
}

export class ErrorBoundary extends Component<Props, State> {
  constructor(props: Props) {
    super(props);
    this.state = { hasError: false };
  }

  static getDerivedStateFromError(): State {
    return { hasError: true };
  }

  componentDidCatch(error: Error, errorInfo: ErrorInfo) {
    Sentry.withScope((scope) => {
      scope.setExtras({ componentStack: errorInfo.componentStack });
      Sentry.captureException(error);
    });
  }

  render() {
    if (this.state.hasError) {
      return this.props.fallback || (
        <div className="error-boundary">
          <h1>Something went wrong</h1>
          <p>Please refresh the page or contact support.</p>
        </div>
      );
    }

    return this.props.children;
  }
}
```

### Logstash Pipeline Configuration

Key routing for log sources:

| Source          | Bugsink Project |
| --------------- | --------------- |
| Backend (Pino)  | Backend API     |
| Worker (Pino)   | Backend API     |
| PostgreSQL logs | Backend API     |
| Vite logs       | Infrastructure  |
| Redis logs      | Infrastructure  |
| NGINX logs      | Infrastructure  |
| Frontend errors | Frontend        |

## Consequences

### Positive

- **Full observability**: Aggregated view of errors and trends
- **Self-hosted**: No external SaaS dependencies or subscription costs
- **SDK compatibility**: Leverages mature Sentry SDKs with excellent documentation
- **AI integration**: MCP server enables Claude Code to query and analyze errors
- **Unified architecture**: Same setup works in dev container and production
- **Lightweight**: Bugsink runs in a single process, unlike full Sentry (16GB+ RAM)
- **Error correlation**: Request IDs allow correlation between frontend errors and backend logs

### Negative

- **Additional services**: Bugsink and Logstash add complexity to the container
- **PostgreSQL overhead**: Additional database for error tracking
- **Initial setup**: Requires configuration of multiple components
- **Logstash learning curve**: Pipeline configuration requires Logstash knowledge

## Alternatives Considered

1. **Full Sentry self-hosted**: Rejected due to complexity (Kafka, Redis, ClickHouse, 16GB+ RAM minimum)
2. **GlitchTip**: Considered, but Bugsink is lighter weight and easier to deploy
3. **Sentry SaaS**: Rejected due to self-hosted requirement
4. **Custom error aggregation**: Rejected in favor of proven Sentry SDK ecosystem

## References

- [Bugsink Documentation](https://www.bugsink.com/docs/)
- [Bugsink Docker Install](https://www.bugsink.com/docs/docker-install/)
- [@sentry/node Documentation](https://docs.sentry.io/platforms/javascript/guides/node/)
- [@sentry/react Documentation](https://docs.sentry.io/platforms/javascript/guides/react/)
- [bugsink-mcp](https://github.com/j-shelfwood/bugsink-mcp)
- [Logstash Reference](https://www.elastic.co/guide/en/logstash/current/index.html)
- [ADR-030: PostgreSQL Function Observability](ADR-030-postgresql-function-observability.md)
- [ADR-032: Application Performance Monitoring](ADR-032-application-performance-monitoring.md)