comprehensive documentation review + test fixes

2026-01-28 16:35:38 -08:00
parent b6c3ca9abe
commit 45ac4fccf5
38 changed files with 2602 additions and 601 deletions
--- a/docs/subagents/INTEGRATIONS-GUIDE.md
+++ b/docs/subagents/INTEGRATIONS-GUIDE.md
@@ -0,0 +1,396 @@
+# Integrations Subagent Guide
+
+The **integrations-specialist** subagent handles third-party services, webhooks, and external API integrations in the Flyer Crawler project.
+
+## Quick Reference
+
+| Aspect          | Details                                                                     |
+| --------------- | --------------------------------------------------------------------------- |
+| **Primary Use** | External APIs, webhooks, OAuth, third-party services                        |
+| **Key Files**   | `src/services/external/`, `src/routes/webhooks.routes.ts`                   |
+| **Key ADRs**    | ADR-041 (AI Integration), ADR-016 (API Security), ADR-048 (Auth)            |
+| **MCP Tools**   | `mcp__gitea-projectium__*`, `mcp__bugsink__*`                               |
+| **Security**    | API key storage, webhook signatures, OAuth state param                      |
+| **Delegate To** | `coder` (implementation), `security-engineer` (review), `ai-usage` (Gemini) |
+
+## When to Use
+
+Use the **integrations-specialist** subagent when you need to:
+
+- Integrate with external APIs (OAuth, REST, GraphQL)
+- Implement webhook handlers
+- Configure third-party services
+- Debug external service connectivity
+- Handle API authentication flows
+- Manage external service rate limits
+
+## What integrations-specialist Knows
+
+The integrations-specialist subagent understands:
+
+- OAuth 2.0 flows (authorization code, client credentials)
+- REST API integration patterns
+- Webhook security (signature verification)
+- External service error handling
+- Rate limiting and retry strategies
+- API key management
+
+## Current Integrations
+
+| Service       | Purpose                | Integration Type | Key Files                          |
+| ------------- | ---------------------- | ---------------- | ---------------------------------- |
+| Google Gemini | AI flyer extraction    | REST API         | `src/services/aiService.server.ts` |
+| Bugsink       | Error tracking         | REST API         | MCP: `mcp__bugsink__*`             |
+| Gitea         | Repository and CI/CD   | REST API         | MCP: `mcp__gitea-projectium__*`    |
+| Redis         | Caching and job queues | Native client    | `src/services/redis.server.ts`     |
+| PostgreSQL    | Primary database       | Native client    | `src/services/db/pool.db.ts`       |
+
+## Example Requests
+
+### Adding External API Integration
+
+```
+"Use integrations-specialist to integrate with the Store API
+to automatically fetch store location data. Include proper
+error handling, rate limiting, and caching."
+```
+
+### OAuth Implementation
+
+```
+"Use integrations-specialist to implement Google OAuth for
+user authentication. Include token refresh handling and
+session management."
+```
+
+### Webhook Handler
+
+```
+"Use integrations-specialist to create a webhook handler for
+receiving store inventory updates. Include signature verification
+and idempotency handling."
+```
+
+### Debugging External Service Issues
+
+```
+"Use integrations-specialist to debug why the Gemini API calls
+are intermittently failing with timeout errors. Check connection
+pooling, retry logic, and error handling."
+```
+
+## Integration Patterns
+
+### REST API Client Pattern
+
+```typescript
+// src/services/external/storeApi.server.ts
+import { env } from '@/config/env';
+import { log } from '@/services/logger.server';
+
+interface StoreApiConfig {
+  baseUrl: string;
+  apiKey: string;
+  timeout: number;
+}
+
+class StoreApiClient {
+  private config: StoreApiConfig;
+
+  constructor(config: StoreApiConfig) {
+    this.config = config;
+  }
+
+  async getStoreLocations(storeId: string): Promise<StoreLocation[]> {
+    const url = `${this.config.baseUrl}/stores/${storeId}/locations`;
+
+    try {
+      const response = await fetch(url, {
+        headers: {
+          Authorization: `Bearer ${this.config.apiKey}`,
+          'Content-Type': 'application/json',
+        },
+        signal: AbortSignal.timeout(this.config.timeout),
+      });
+
+      if (!response.ok) {
+        throw new ExternalApiError(`Store API error: ${response.status}`, response.status);
+      }
+
+      return response.json();
+    } catch (error) {
+      log.error({ error, storeId }, 'Failed to fetch store locations');
+      throw error;
+    }
+  }
+}
+
+export const storeApiClient = new StoreApiClient({
+  baseUrl: env.STORE_API_BASE_URL,
+  apiKey: env.STORE_API_KEY,
+  timeout: 10000,
+});
+```
+
+### Webhook Handler Pattern
+
+```typescript
+// src/routes/webhooks.routes.ts
+import { Router } from 'express';
+import crypto from 'crypto';
+import { env } from '@/config/env';
+
+const router = Router();
+
+function verifyWebhookSignature(payload: string, signature: string, secret: string): boolean {
+  const expected = crypto.createHmac('sha256', secret).update(payload).digest('hex');
+  return crypto.timingSafeEqual(Buffer.from(signature), Buffer.from(`sha256=${expected}`));
+}
+
+router.post('/store-updates', async (req, res, next) => {
+  try {
+    const signature = req.headers['x-webhook-signature'] as string;
+    const payload = JSON.stringify(req.body);
+
+    if (!verifyWebhookSignature(payload, signature, env.WEBHOOK_SECRET)) {
+      return res.status(401).json({ error: 'Invalid signature' });
+    }
+
+    // Process webhook with idempotency check
+    const eventId = req.headers['x-event-id'] as string;
+    const alreadyProcessed = await checkIdempotencyKey(eventId);
+
+    if (alreadyProcessed) {
+      return res.status(200).json({ status: 'already_processed' });
+    }
+
+    await processStoreUpdate(req.body);
+    await markEventProcessed(eventId);
+
+    res.status(200).json({ status: 'processed' });
+  } catch (error) {
+    next(error);
+  }
+});
+```
+
+### OAuth Flow Pattern
+
+```typescript
+// src/services/oauth/googleOAuth.server.ts
+import { OAuth2Client } from 'google-auth-library';
+import { env } from '@/config/env';
+
+const oauth2Client = new OAuth2Client(
+  env.GOOGLE_CLIENT_ID,
+  env.GOOGLE_CLIENT_SECRET,
+  env.GOOGLE_REDIRECT_URI,
+);
+
+export function getAuthorizationUrl(): string {
+  return oauth2Client.generateAuthUrl({
+    access_type: 'offline',
+    scope: ['email', 'profile'],
+    prompt: 'consent',
+  });
+}
+
+export async function exchangeCodeForTokens(code: string) {
+  const { tokens } = await oauth2Client.getToken(code);
+  return tokens;
+}
+
+export async function refreshAccessToken(refreshToken: string) {
+  oauth2Client.setCredentials({ refresh_token: refreshToken });
+  const { credentials } = await oauth2Client.refreshAccessToken();
+  return credentials;
+}
+```
+
+## Error Handling for External Services
+
+### Custom Error Classes
+
+```typescript
+// src/services/external/errors.ts
+export class ExternalApiError extends Error {
+  constructor(
+    message: string,
+    public statusCode: number,
+    public retryable: boolean = false,
+  ) {
+    super(message);
+    this.name = 'ExternalApiError';
+  }
+}
+
+export class RateLimitError extends ExternalApiError {
+  constructor(
+    message: string,
+    public retryAfter: number,
+  ) {
+    super(message, 429, true);
+    this.name = 'RateLimitError';
+  }
+}
+```
+
+### Retry with Exponential Backoff
+
+```typescript
+async function fetchWithRetry<T>(
+  fn: () => Promise<T>,
+  options: { maxRetries: number; baseDelay: number },
+): Promise<T> {
+  let lastError: Error;
+
+  for (let attempt = 0; attempt <= options.maxRetries; attempt++) {
+    try {
+      return await fn();
+    } catch (error) {
+      lastError = error as Error;
+
+      if (error instanceof ExternalApiError && !error.retryable) {
+        throw error;
+      }
+
+      if (attempt < options.maxRetries) {
+        const delay = options.baseDelay * Math.pow(2, attempt);
+        await new Promise((resolve) => setTimeout(resolve, delay));
+      }
+    }
+  }
+
+  throw lastError!;
+}
+```
+
+## Rate Limiting Strategies
+
+### Token Bucket Pattern
+
+```typescript
+class RateLimiter {
+  private tokens: number;
+  private lastRefill: number;
+  private readonly maxTokens: number;
+  private readonly refillRate: number; // tokens per second
+
+  constructor(maxTokens: number, refillRate: number) {
+    this.maxTokens = maxTokens;
+    this.tokens = maxTokens;
+    this.refillRate = refillRate;
+    this.lastRefill = Date.now();
+  }
+
+  async acquire(): Promise<void> {
+    this.refill();
+
+    if (this.tokens < 1) {
+      const waitTime = ((1 - this.tokens) / this.refillRate) * 1000;
+      await new Promise((resolve) => setTimeout(resolve, waitTime));
+      this.refill();
+    }
+
+    this.tokens -= 1;
+  }
+
+  private refill(): void {
+    const now = Date.now();
+    const elapsed = (now - this.lastRefill) / 1000;
+    this.tokens = Math.min(this.maxTokens, this.tokens + elapsed * this.refillRate);
+    this.lastRefill = now;
+  }
+}
+```
+
+## Testing Integrations
+
+### Mocking External Services
+
+```typescript
+// src/tests/mocks/storeApi.mock.ts
+import { vi } from 'vitest';
+
+export const mockStoreApiClient = {
+  getStoreLocations: vi.fn(),
+};
+
+vi.mock('@/services/external/storeApi.server', () => ({
+  storeApiClient: mockStoreApiClient,
+}));
+```
+
+### Integration Test with Real Service
+
+```typescript
+// src/tests/integration/storeApi.integration.test.ts
+describe('Store API Integration', () => {
+  it.skipIf(!env.STORE_API_KEY)('fetches real store locations', async () => {
+    const locations = await storeApiClient.getStoreLocations('test-store');
+    expect(locations).toBeInstanceOf(Array);
+  });
+});
+```
+
+## MCP Tools for Integrations
+
+### Gitea Integration
+
+```
+// List repositories
+mcp__gitea-projectium__list_my_repos()
+
+// Create issue
+mcp__gitea-projectium__create_issue({
+  owner: "projectium",
+  repo: "flyer-crawler",
+  title: "Issue title",
+  body: "Issue description"
+})
+```
+
+### Bugsink Integration
+
+```
+// List projects
+mcp__bugsink__list_projects()
+
+// Get issue details
+mcp__bugsink__get_issue({ issue_id: "..." })
+
+// Get stacktrace
+mcp__bugsink__get_stacktrace({ event_id: "..." })
+```
+
+## Security Considerations
+
+### API Key Storage
+
+- Never commit API keys to version control
+- Use environment variables via `src/config/env.ts`
+- Rotate keys periodically
+- Use separate keys for dev/test/prod
+
+### Webhook Security
+
+- Always verify webhook signatures
+- Use HTTPS for webhook endpoints
+- Implement idempotency
+- Log webhook events for audit
+
+### OAuth Security
+
+- Use state parameter to prevent CSRF
+- Store tokens securely (encrypted at rest)
+- Implement token refresh before expiration
+- Validate token scopes
+
+## Related Documentation
+
+- [OVERVIEW.md](./OVERVIEW.md) - Subagent system overview
+- [SECURITY-DEBUG-GUIDE.md](./SECURITY-DEBUG-GUIDE.md) - Security patterns
+- [AI-USAGE-GUIDE.md](./AI-USAGE-GUIDE.md) - Gemini API integration
+- [../adr/0041-ai-gemini-integration-architecture.md](../adr/0041-ai-gemini-integration-architecture.md) - AI integration ADR
+- [../adr/0016-api-security-hardening.md](../adr/0016-api-security-hardening.md) - API security
+- [../adr/0048-authentication-strategy.md](../adr/0048-authentication-strategy.md) - Authentication