torbo/flyer-crawler.projectium.com
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
4f06698dfd5e00a80c1c8dfd42bfac49a2371b3c
flyer-crawler.projectium.com/docs/adr/0054-bugsink-gitea-issue-sync.md

13 KiB
Raw Blame History

ADR-054: Bugsink to Gitea Issue Synchronization

Date: 2026-01-17

Status: Proposed

Context

The application uses Bugsink (Sentry-compatible self-hosted error tracking) to capture runtime errors across 6 projects:

Project Type Environment
flyer-crawler-backend Backend Production
flyer-crawler-backend-test Backend Test/Staging
flyer-crawler-frontend Frontend Production
flyer-crawler-frontend-test Frontend Test/Staging
flyer-crawler-infrastructure Infrastructure Production
flyer-crawler-test-infrastructure Infrastructure Test/Staging

Currently, errors remain in Bugsink until manually reviewed. There is no automated workflow to:

  1. Create trackable tickets for errors
  2. Assign errors to developers
  3. Track resolution progress
  4. Prevent errors from being forgotten

Decision

Implement an automated background worker that synchronizes unresolved Bugsink issues to Gitea as trackable tickets. The sync worker will:

  1. Run only on the test/staging server (not production, not dev container)
  2. Poll all 6 Bugsink projects for unresolved issues
  3. Create Gitea issues with full error context
  4. Mark synced issues as resolved in Bugsink (to prevent re-polling)
  5. Track sync state in Redis to ensure idempotency

Why Test/Staging Only?

  • The sync worker is a background service that needs API tokens for both Bugsink and Gitea
  • Running on test/staging provides a single sync point without duplicating infrastructure
  • All 6 Bugsink projects (including production) are synced from this one worker
  • Production server stays focused on serving users, not running sync jobs

Architecture

Component Overview

┌─────────────────────────────────────────────────────────────────────┐
│                     TEST/STAGING SERVER                              │
│                                                                      │
│  ┌──────────────────┐    ┌──────────────────┐    ┌───────────────┐ │
│  │  BullMQ Queue    │───▶│  Sync Worker     │───▶│  Redis DB 15  │ │
│  │  bugsink-sync    │    │  (15min repeat)  │    │  Sync State   │ │
│  └──────────────────┘    └────────┬─────────┘    └───────────────┘ │
│                                   │                                  │
└───────────────────────────────────┼──────────────────────────────────┘
                                    │
                    ┌───────────────┴───────────────┐
                    ▼                               ▼
           ┌──────────────┐                ┌──────────────┐
           │   Bugsink    │                │    Gitea     │
           │ (6 projects) │                │  (1 repo)    │
           └──────────────┘                └──────────────┘

Queue Configuration

Setting Value Rationale
Queue Name bugsink-sync Follows existing naming pattern
Repeat Interval 15 minutes Balances responsiveness with API rate limits
Retry Attempts 3 Standard retry policy
Backoff Exponential (30s base) Handles temporary API failures
Concurrency 1 Serial processing prevents race conditions

Redis Database Allocation

Database Usage Owner
0 BullMQ (Production) Existing queues
1 BullMQ (Test) Existing queues
2-14 Reserved Future use
15 Bugsink Sync State This feature

Redis Key Schema

bugsink:synced:{bugsink_issue_id}
  └─ Value: JSON {
       gitea_issue_number: number,
       synced_at: ISO timestamp,
       project: string,
       title: string
     }

Gitea Labels

The following labels have been created in torbo/flyer-crawler.projectium.com:

Label ID Color Purpose
bug:frontend 8 #e11d48 (Red) Frontend JavaScript/React errors
bug:backend 9 #ea580c (Orange) Backend Node.js/API errors
bug:infrastructure 10 #7c3aed (Purple) Infrastructure errors (Redis, PM2)
env:production 11 #dc2626 (Dark Red) Production environment
env:test 12 #2563eb (Blue) Test/staging environment
env:development 13 #6b7280 (Gray) Development environment
source:bugsink 14 #10b981 (Green) Auto-synced from Bugsink

Label Mapping

Bugsink Project Bug Label Env Label
flyer-crawler-backend bug:backend env:production
flyer-crawler-backend-test bug:backend env:test
flyer-crawler-frontend bug:frontend env:production
flyer-crawler-frontend-test bug:frontend env:test
flyer-crawler-infrastructure bug:infrastructure env:production
flyer-crawler-test-infrastructure bug:infrastructure env:test

All synced issues also receive the source:bugsink label.

Implementation Details

New Files

File Purpose
src/services/bugsinkSync.server.ts Core synchronization logic
src/services/bugsinkClient.server.ts HTTP client for Bugsink API
src/services/giteaClient.server.ts HTTP client for Gitea API
src/types/bugsink.ts TypeScript interfaces for Bugsink responses
src/routes/admin/bugsink-sync.ts Admin endpoints for manual trigger

Modified Files

File Changes
src/services/queues.server.ts Add bugsinkSyncQueue definition
src/services/workers.server.ts Add sync worker implementation
src/config/env.ts Add bugsink sync configuration schema
.env.example Document new environment variables
.gitea/workflows/deploy-to-test.yml Pass sync-related secrets

Environment Variables

# Bugsink Configuration
BUGSINK_URL=https://bugsink.projectium.com
BUGSINK_API_TOKEN=77deaa5e...  # Created via Django management command (see BUGSINK-SYNC.md)

# Gitea Configuration
GITEA_URL=https://gitea.projectium.com
GITEA_API_TOKEN=...           # Personal access token with repo scope
GITEA_OWNER=torbo
GITEA_REPO=flyer-crawler.projectium.com

# Sync Control
BUGSINK_SYNC_ENABLED=false    # Set true only in test environment
BUGSINK_SYNC_INTERVAL=15      # Minutes between sync runs

Gitea Issue Template

## Error Details

| Field        | Value           |
| ------------ | --------------- |
| **Type**     | {error_type}    |
| **Message**  | {error_message} |
| **Platform** | {platform}      |
| **Level**    | {level}         |

## Occurrence Statistics

- **First Seen**: {first_seen}
- **Last Seen**: {last_seen}
- **Total Occurrences**: {count}

## Request Context

- **URL**: {request_url}
- **Additional Context**: {context}

## Stacktrace

<details>
<summary>Click to expand</summary>

{stacktrace}

</details>

---

**Bugsink Issue**: {bugsink_url}
**Project**: {project_slug}
**Trace ID**: {trace_id}

Sync Workflow

1. Worker triggered (every 15 min or manual)
2. For each of 6 Bugsink projects:
   a. List issues with status='unresolved'
   b. For each issue:
      i.   Check Redis for existing sync record
      ii.  If already synced → skip
      iii. Fetch issue details + stacktrace
      iv.  Create Gitea issue with labels
      v.   Store sync record in Redis
      vi.  Mark issue as 'resolved' in Bugsink
3. Log summary (synced: N, skipped: N, failed: N)

Idempotency Guarantees

  1. Redis check before creation: Prevents duplicate Gitea issues
  2. Atomic Redis write after Gitea create: Ensures state consistency
  3. Query only unresolved issues: Resolved issues won't appear in polls
  4. No TTL on Redis keys: Permanent sync history

Consequences

Positive

  1. Visibility: All application errors become trackable tickets
  2. Accountability: Errors can be assigned to developers
  3. History: Complete audit trail of when errors were discovered and resolved
  4. Integration: Errors appear alongside feature work in Gitea
  5. Automation: No manual error triage required

Negative

  1. API Dependencies: Requires both Bugsink and Gitea APIs to be available
  2. Token Management: Additional secrets to manage in CI/CD
  3. Potential Noise: High-frequency errors could create many tickets (mitigated by Bugsink's issue grouping)
  4. Single Point: Sync only runs on test server (if test server is down, no sync occurs)

Risks & Mitigations

Risk Mitigation
Bugsink API rate limits 15-minute polling interval
Gitea API rate limits Sequential processing with delays
Redis connection issues Reuse existing connection patterns
Duplicate issues Redis tracking + idempotent checks
Missing stacktrace Graceful degradation (create issue without trace)

Admin Interface

Manual Sync Endpoint

POST /api/admin/bugsink/sync
Authorization: Bearer {admin_jwt}

Response:
{
  "success": true,
  "data": {
    "synced": 3,
    "skipped": 12,
    "failed": 0,
    "duration_ms": 2340
  }
}

Sync Status Endpoint

GET /api/admin/bugsink/sync/status
Authorization: Bearer {admin_jwt}

Response:
{
  "success": true,
  "data": {
    "enabled": true,
    "last_run": "2026-01-17T10:30:00Z",
    "next_run": "2026-01-17T10:45:00Z",
    "total_synced": 47,
    "projects": [
      { "slug": "flyer-crawler-backend", "synced_count": 12 },
      ...
    ]
  }
}

Implementation Phases

Phase 1: Core Infrastructure

  • Add environment variables to env.ts schema
  • Create BugsinkClient service (HTTP client)
  • Create GiteaClient service (HTTP client)
  • Add Redis db 15 connection for sync tracking

Phase 2: Sync Logic

  • Create BugsinkSyncService with sync logic
  • Add bugsink-sync queue to queues.server.ts
  • Add sync worker to workers.server.ts
  • Create TypeScript types for API responses

Phase 3: Integration

  • Add admin endpoints for manual sync trigger
  • Update deploy-to-test.yml with new secrets
  • Add secrets to Gitea repository settings
  • Test end-to-end in staging environment

Phase 4: Documentation

  • Update CLAUDE.md with sync information
  • Create operational runbook for sync issues

Future Enhancements

  1. Bi-directional sync: Update Bugsink when Gitea issue is closed
  2. Smart deduplication: Detect similar errors across projects
  3. Priority mapping: High occurrence count → high priority label
  4. Slack/Discord notifications: Alert on new critical errors
  5. Metrics dashboard: Track error trends over time

References

Reference in New Issue View Git Blame Copy Permalink