torbo/flyer-crawler.projectium.com
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
f6f4415aeb6d3d5af3351155f819ee61ab76140b
flyer-crawler.projectium.com/docs/adr/ADR-033-bugsink-gitea-issue-sync.md
Torben Sorensen 4d323a51ca
All checks were successful
Deploy to Test Environment / deploy-to-test (push) Successful in 49m39s
Details
fix tour / whats new collision
2026-02-12 04:29:43 -08:00

341 lines
12 KiB
Markdown
Raw Blame History

# ADR-033: Bugsink to Gitea Issue Synchronization
**Date**: 2026-02-10
**Status**: Proposed
**Source**: Imported from flyer-crawler project (ADR-054)
**Related**: [ADR-029](ADR-029-error-tracking-with-bugsink.md), [ADR-012](ADR-012-bullmq-background-job-processing.md)
## Context
The application uses Bugsink (Sentry-compatible self-hosted error tracking) to capture runtime errors across multiple projects:
| Project Type | Environment | Description |
| -------------- | ------------ | ---------------------------------------- |
| Backend | Production | Main API server errors |
| Backend | Test/Staging | Pre-production API errors |
| Frontend | Production | Client-side JavaScript errors |
| Frontend | Test/Staging | Pre-production frontend errors |
| Infrastructure | Production | Infrastructure-level errors (Redis, PM2) |
| Infrastructure | Test/Staging | Pre-production infrastructure errors |
Currently, errors remain in Bugsink until manually reviewed. There is no automated workflow to:
1. Create trackable tickets for errors
2. Assign errors to developers
3. Track resolution progress
4. Prevent errors from being forgotten
## Decision
Implement an automated background worker that synchronizes unresolved Bugsink issues to Gitea as trackable tickets. The sync worker will:
1. **Run only on the test/staging server** (not production, not dev container)
2. **Poll all Bugsink projects** for unresolved issues
3. **Create Gitea issues** with full error context
4. **Mark synced issues as resolved** in Bugsink (to prevent re-polling)
5. **Track sync state in Redis** to ensure idempotency
### Why Test/Staging Only?
- The sync worker is a background service that needs API tokens for both Bugsink and Gitea
- Running on test/staging provides a single sync point without duplicating infrastructure
- All Bugsink projects (including production) are synced from this one worker
- Production server stays focused on serving users, not running sync jobs
## Architecture
### Component Overview
```
+-----------------------------------------------------------------------+
| TEST/STAGING SERVER |
| |
| +------------------+ +------------------+ +-------------------+ |
| | BullMQ Queue |--->| Sync Worker |--->| Redis DB 15 | |
| | bugsink-sync | | (15min repeat) | | Sync State | |
| +------------------+ +--------+---------+ +-------------------+ |
| | |
+-----------------------------------+------------------------------------+
|
+---------------+---------------+
v v
+------------------+ +------------------+
| Bugsink | | Gitea |
| (all projects) | | (1 repo) |
+------------------+ +------------------+
```
### Queue Configuration
| Setting | Value | Rationale |
| --------------- | ---------------------- | -------------------------------------------- |
| Queue Name | `bugsink-sync` | Follows existing naming pattern |
| Repeat Interval | 15 minutes | Balances responsiveness with API rate limits |
| Retry Attempts | 3 | Standard retry policy |
| Backoff | Exponential (30s base) | Handles temporary API failures |
| Concurrency | 1 | Serial processing prevents race conditions |
### Redis Database Allocation
| Database | Usage | Owner |
| -------- | ------------------- | --------------- |
| 0 | BullMQ (Production) | Existing queues |
| 1 | BullMQ (Test) | Existing queues |
| 2-14 | Reserved | Future use |
| 15 | Bugsink Sync State | This feature |
### Redis Key Schema
```
bugsink:synced:{bugsink_issue_id}
+-- Value: JSON {
gitea_issue_number: number,
synced_at: ISO timestamp,
project: string,
title: string
}
```
### Gitea Labels
The following labels should be created in the repository:
| Label | Color | Purpose |
| -------------------- | ------------------ | ---------------------------------- |
| `bug:frontend` | #e11d48 (Red) | Frontend JavaScript/React errors |
| `bug:backend` | #ea580c (Orange) | Backend Node.js/API errors |
| `bug:infrastructure` | #7c3aed (Purple) | Infrastructure errors (Redis, PM2) |
| `env:production` | #dc2626 (Dark Red) | Production environment |
| `env:test` | #2563eb (Blue) | Test/staging environment |
| `env:development` | #6b7280 (Gray) | Development environment |
| `source:bugsink` | #10b981 (Green) | Auto-synced from Bugsink |
### Label Mapping
| Bugsink Project Type | Bug Label | Env Label |
| --------------------- | ------------------ | -------------- |
| backend (prod) | bug:backend | env:production |
| backend (test) | bug:backend | env:test |
| frontend (prod) | bug:frontend | env:production |
| frontend (test) | bug:frontend | env:test |
| infrastructure (prod) | bug:infrastructure | env:production |
| infrastructure (test) | bug:infrastructure | env:test |
All synced issues also receive the `source:bugsink` label.
## Implementation Details
### New Files
| File | Purpose |
| -------------------------------------- | ------------------------------------------- |
| `src/services/bugsinkSync.server.ts` | Core synchronization logic |
| `src/services/bugsinkClient.server.ts` | HTTP client for Bugsink API |
| `src/services/giteaClient.server.ts` | HTTP client for Gitea API |
| `src/types/bugsink.ts` | TypeScript interfaces for Bugsink responses |
| `src/routes/admin/bugsink-sync.ts` | Admin endpoints for manual trigger |
### Modified Files
| File | Changes |
| -------------------------------- | ------------------------------------- |
| `src/services/queues.server.ts` | Add `bugsinkSyncQueue` definition |
| `src/services/workers.server.ts` | Add sync worker implementation |
| `src/config/env.ts` | Add bugsink sync configuration schema |
| `.env.example` | Document new environment variables |
### Environment Variables
```bash
# Bugsink Configuration
BUGSINK_URL=https://bugsink.example.com
BUGSINK_API_TOKEN=... # Created via Django management command
# Gitea Configuration
GITEA_URL=https://gitea.example.com
GITEA_API_TOKEN=... # Personal access token with repo scope
GITEA_OWNER=org-name
GITEA_REPO=project-repo
# Sync Control
BUGSINK_SYNC_ENABLED=false # Set true only in test environment
BUGSINK_SYNC_INTERVAL=15 # Minutes between sync runs
```
### Gitea Issue Template
```markdown
## Error Details
| Field | Value |
| ------------ | --------------- |
| **Type** | {error_type} |
| **Message** | {error_message} |
| **Platform** | {platform} |
| **Level** | {level} |
## Occurrence Statistics
- **First Seen**: {first_seen}
- **Last Seen**: {last_seen}
- **Total Occurrences**: {count}
## Request Context
- **URL**: {request_url}
- **Additional Context**: {context}
## Stacktrace
<details>
<summary>Click to expand</summary>
{stacktrace}
</details>
---
**Bugsink Issue**: {bugsink_url}
**Project**: {project_slug}
**Trace ID**: {trace_id}
```
### Sync Workflow
```
1. Worker triggered (every 15 min or manual)
2. For each Bugsink project:
a. List issues with status='unresolved'
b. For each issue:
i. Check Redis for existing sync record
ii. If already synced -> skip
iii. Fetch issue details + stacktrace
iv. Create Gitea issue with labels
v. Store sync record in Redis
vi. Mark issue as 'resolved' in Bugsink
3. Log summary (synced: N, skipped: N, failed: N)
```
### Idempotency Guarantees
1. **Redis check before creation**: Prevents duplicate Gitea issues
2. **Atomic Redis write after Gitea create**: Ensures state consistency
3. **Query only unresolved issues**: Resolved issues won't appear in polls
4. **No TTL on Redis keys**: Permanent sync history
## Admin Interface
### Manual Sync Endpoint
```
POST /api/admin/bugsink/sync
Authorization: Bearer {admin_jwt}
Response:
{
"success": true,
"data": {
"synced": 3,
"skipped": 12,
"failed": 0,
"duration_ms": 2340
}
}
```
### Sync Status Endpoint
```
GET /api/admin/bugsink/sync/status
Authorization: Bearer {admin_jwt}
Response:
{
"success": true,
"data": {
"enabled": true,
"last_run": "2026-01-17T10:30:00Z",
"next_run": "2026-01-17T10:45:00Z",
"total_synced": 47,
"projects": [
{ "slug": "backend-prod", "synced_count": 12 },
...
]
}
}
```
## Implementation Phases
### Phase 1: Core Infrastructure
- Add environment variables to `env.ts` schema
- Create `BugsinkClient` service (HTTP client)
- Create `GiteaClient` service (HTTP client)
- Add Redis db 15 connection for sync tracking
### Phase 2: Sync Logic
- Create `BugsinkSyncService` with sync logic
- Add `bugsink-sync` queue to `queues.server.ts`
- Add sync worker to `workers.server.ts`
- Create TypeScript types for API responses
### Phase 3: Integration
- Add admin endpoints for manual sync trigger
- Update CI/CD with new secrets
- Add secrets to repository settings
- Test end-to-end in staging environment
### Phase 4: Documentation
- Update CLAUDE.md with sync information
- Create operational runbook for sync issues
## Consequences
### Positive
1. **Visibility**: All application errors become trackable tickets
2. **Accountability**: Errors can be assigned to developers
3. **History**: Complete audit trail of when errors were discovered and resolved
4. **Integration**: Errors appear alongside feature work in Gitea
5. **Automation**: No manual error triage required
### Negative
1. **API Dependencies**: Requires both Bugsink and Gitea APIs to be available
2. **Token Management**: Additional secrets to manage in CI/CD
3. **Potential Noise**: High-frequency errors could create many tickets (mitigated by Bugsink's issue grouping)
4. **Single Point**: Sync only runs on test server (if test server is down, no sync occurs)
### Risks and Mitigations
| Risk | Mitigation |
| ----------------------- | ------------------------------------------------- |
| Bugsink API rate limits | 15-minute polling interval |
| Gitea API rate limits | Sequential processing with delays |
| Redis connection issues | Reuse existing connection patterns |
| Duplicate issues | Redis tracking + idempotent checks |
| Missing stacktrace | Graceful degradation (create issue without trace) |
## Future Enhancements
1. **Bi-directional sync**: Update Bugsink when Gitea issue is closed
2. **Smart deduplication**: Detect similar errors across projects
3. **Priority mapping**: High occurrence count -> high priority label
4. **Slack/Discord notifications**: Alert on new critical errors
5. **Metrics dashboard**: Track error trends over time
## References
- [ADR-012: BullMQ Background Job Processing](ADR-012-bullmq-background-job-processing.md)
- [ADR-029: Error Tracking with Bugsink](ADR-029-error-tracking-with-bugsink.md)
- [Bugsink API Documentation](https://bugsink.com/docs/api/)
- [Gitea API Documentation](https://docs.gitea.io/en-us/api-usage/)
Reference in New Issue View Git Blame Copy Permalink