# ADR-032: Application Performance Monitoring (APM)

**Date**: 2026-02-10

**Status**: Proposed

**Source**: Imported from flyer-crawler project (ADR-056)

**Related**: [ADR-029](ADR-029-error-tracking-with-bugsink.md) (Error Tracking with Bugsink)

## Context

Application Performance Monitoring (APM) provides visibility into application behavior through:

- **Distributed Tracing**: Track requests across services, queues, and database calls
- **Performance Metrics**: Response times, throughput, error rates
- **Resource Monitoring**: Memory usage, CPU, database connections
- **Transaction Analysis**: Identify slow endpoints and bottlenecks

While ADR-029 covers error tracking and observability, APM is a distinct concern focused on performance rather than errors. The Sentry SDK supports APM through its tracing features, but this capability is currently **intentionally disabled** in our application.

### Current State

The Sentry SDK is installed and configured for error tracking (see ADR-029), but APM features are disabled:

```typescript
// src/services/sentry.client.ts
Sentry.init({
  dsn: config.sentry.dsn,
  environment: config.sentry.environment,
  // Performance monitoring - disabled for now to keep it simple
  tracesSampleRate: 0,
  // ...
});
```

```typescript
// src/services/sentry.server.ts
Sentry.init({
  dsn: config.sentry.dsn,
  environment: config.sentry.environment || config.server.nodeEnv,
  // Performance monitoring - disabled for now to keep it simple
  tracesSampleRate: 0,
  // ...
});
```

### Why APM is Currently Disabled

1. **Complexity**: APM adds overhead and complexity to debugging
2. **Bugsink Limitations**: Bugsink's APM support is less mature than its error tracking
3. **Resource Overhead**: Tracing adds memory and CPU overhead
4. **Focus**: Error tracking provides more immediate value for our current scale
5. **Cost**: High sample rates can significantly increase storage requirements

## Decision

We propose a **staged approach** to APM implementation:

### Phase 1: Selective Backend Tracing (Low Priority)

Enable tracing for specific high-value operations:

```typescript
// Enable tracing for specific transactions only
Sentry.init({
  dsn: config.sentry.dsn,
  tracesSampleRate: 0, // Keep default at 0

  // Trace only specific high-value transactions
  tracesSampler: (samplingContext) => {
    const transactionName = samplingContext.transactionContext?.name;

    // Always trace long-running jobs
    if (transactionName?.includes('job-processing')) {
      return 0.1; // 10% sample rate
    }

    // Always trace AI/external API calls
    if (transactionName?.includes('external-api')) {
      return 0.5; // 50% sample rate
    }

    // Trace slow endpoints (determined by custom logic)
    if (samplingContext.parentSampled) {
      return 0.1; // 10% for child transactions
    }

    return 0; // Don't trace other transactions
  },
});
```

### Phase 2: Custom Performance Metrics

Add custom metrics without full tracing overhead:

```typescript
// Custom metric for slow database queries
import { metrics } from '@sentry/node';

// In repository methods
const startTime = performance.now();
const result = await pool.query(sql, params);
const duration = performance.now() - startTime;

metrics.distribution('db.query.duration', duration, {
  tags: { query_type: 'select', table: 'users' },
});

if (duration > 1000) {
  logger.warn({ duration, sql }, 'Slow query detected');
}
```

### Phase 3: Full APM Integration (Future)

When/if full APM is needed:

```typescript
Sentry.init({
  dsn: config.sentry.dsn,
  tracesSampleRate: 0.1, // 10% of transactions
  profilesSampleRate: 0.1, // 10% of traced transactions get profiled

  integrations: [
    // Database tracing
    Sentry.postgresIntegration(),
    // Redis tracing
    Sentry.redisIntegration(),
    // BullMQ job tracing (custom integration)
  ],
});
```

## Implementation Steps

### To Enable Basic APM

1. **Update Sentry Configuration**:
   - Set `tracesSampleRate` > 0 in `src/services/sentry.server.ts`
   - Set `tracesSampleRate` > 0 in `src/services/sentry.client.ts`
   - Add environment variable `SENTRY_TRACES_SAMPLE_RATE` (default: 0)

2. **Add Instrumentation**:
   - Enable automatic Express instrumentation
   - Add manual spans for BullMQ job processing
   - Add database query instrumentation

3. **Frontend Tracing**:
   - Add Browser Tracing integration
   - Configure page load and navigation tracing

4. **Environment Variables**:

   ```bash
   SENTRY_TRACES_SAMPLE_RATE=0.1  # 10% sampling
   SENTRY_PROFILES_SAMPLE_RATE=0  # Profiling disabled
   ```

5. **Bugsink Configuration**:
   - Verify Bugsink supports performance data ingestion
   - Configure retention policies for performance data

### Configuration Changes Required

```typescript
// src/config/env.ts - Add new config
sentry: {
  dsn: env.SENTRY_DSN,
  environment: env.SENTRY_ENVIRONMENT,
  debug: env.SENTRY_DEBUG === 'true',
  tracesSampleRate: parseFloat(env.SENTRY_TRACES_SAMPLE_RATE || '0'),
  profilesSampleRate: parseFloat(env.SENTRY_PROFILES_SAMPLE_RATE || '0'),
},
```

```typescript
// src/services/sentry.server.ts - Updated init
Sentry.init({
  dsn: config.sentry.dsn,
  environment: config.sentry.environment,
  tracesSampleRate: config.sentry.tracesSampleRate,
  profilesSampleRate: config.sentry.profilesSampleRate,
  // ... rest of config
});
```

## Trade-offs

### Enabling APM

**Benefits**:

- Identify performance bottlenecks
- Track distributed transactions across services
- Profile slow endpoints
- Monitor resource utilization trends

**Costs**:

- Increased memory usage (~5-15% overhead)
- Additional CPU for trace processing
- Increased storage in Bugsink/Sentry
- More complex debugging (noise in traces)
- Potential latency from tracing overhead

### Keeping APM Disabled

**Benefits**:

- Simpler operation and debugging
- Lower resource overhead
- Focused on error tracking (higher priority)
- No additional storage costs

**Costs**:

- No automated performance insights
- Manual profiling required for bottleneck detection
- Limited visibility into slow transactions

## Alternatives Considered

1. **OpenTelemetry**: More vendor-neutral, but adds another dependency and complexity
2. **Prometheus + Grafana**: Good for metrics, but doesn't provide distributed tracing
3. **Jaeger/Zipkin**: Purpose-built for tracing, but requires additional infrastructure
4. **New Relic/Datadog SaaS**: Full-featured but conflicts with self-hosted requirement

## Current Recommendation

**Keep APM disabled** (`tracesSampleRate: 0`) until:

1. Specific performance issues are identified that require tracing
2. Bugsink's APM support is verified and tested
3. Infrastructure can support the additional overhead
4. There is a clear business need for performance visibility

When enabling APM becomes necessary, start with Phase 1 (selective tracing) to minimize overhead while gaining targeted insights.

## Consequences

### Positive (When Implemented)

- Automated identification of slow endpoints
- Distributed trace visualization across async operations
- Correlation between errors and performance issues
- Proactive alerting on performance degradation

### Negative

- Additional infrastructure complexity
- Storage overhead for trace data
- Potential performance impact from tracing itself
- Learning curve for trace analysis

## References

- [Sentry Performance Monitoring](https://docs.sentry.io/product/performance/)
- [@sentry/node Performance](https://docs.sentry.io/platforms/javascript/guides/node/performance/)
- [@sentry/react Performance](https://docs.sentry.io/platforms/javascript/guides/react/performance/)
- [OpenTelemetry](https://opentelemetry.io/) (alternative approach)
- [ADR-029: Error Tracking with Bugsink](ADR-029-error-tracking-with-bugsink.md)
