torbo/flyer-crawler.projectium.com
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

all the new shiny things
All checks were successful
Deploy to Test Environment / deploy-to-test (push) Successful in 15m54s
Details

Browse Source
This commit is contained in:
Torben Sorensen
2026-01-11 02:02:33 -08:00
parent 9815eb3686
commit 4e22213cd1
37 changed files with 5636 additions and 375 deletions
Download Patch File Download Diff File Expand all files Collapse all files

313
docs/adr/0015-application-performance-monitoring-and-error-tracking.md
View File

@@ -2,17 +2,320 @@
**Date**: 2025-12-12
**Status**: Proposed
**Status**: Accepted
**Updated**: 2026-01-11
## Context
While `ADR-004` established structured logging, the application lacks a high-level, aggregated view of its health, performance, and errors. It's difficult to spot trends, identify slow API endpoints, or be proactively notified of new types of errors.
While `ADR-004` established structured logging with Pino, the application lacks a high-level, aggregated view of its health, performance, and errors. It's difficult to spot trends, identify slow API endpoints, or be proactively notified of new types of errors.
Key requirements:
1. **Self-hosted**: No external SaaS dependencies for error tracking
2. **Sentry SDK compatible**: Leverage mature, well-documented SDKs
3. **Lightweight**: Minimal resource overhead in the dev container
4. **Production-ready**: Same architecture works on bare-metal production servers
5. **AI-accessible**: MCP server integration for Claude Code and other AI tools
## Decision
We will integrate a dedicated Application Performance Monitoring (APM) and error tracking service like **Sentry**, **Datadog**, or **New Relic**. This will define how the service is integrated to automatically capture and report unhandled exceptions, performance data (e.g., transaction traces, database query times), and release health.
We will implement a self-hosted error tracking stack using **Bugsink** as the Sentry-compatible backend, with the following components:
### 1. Error Tracking Backend: Bugsink
**Bugsink** is a lightweight, self-hosted Sentry alternative that:
- Runs as a single process (no Kafka, Redis, ClickHouse required)
- Is fully compatible with Sentry SDKs
- Supports ARM64 and AMD64 architectures
- Can use SQLite (dev) or PostgreSQL (production)
**Deployment**:
- **Dev container**: Installed as a systemd service inside the container
- **Production**: Runs as a systemd service on bare-metal, listening on localhost only
- **Database**: Uses PostgreSQL with a dedicated `bugsink` user and `bugsink` database (same PostgreSQL instance as the main application)
### 2. Backend Integration: @sentry/node
The Express backend will integrate `@sentry/node` SDK to:
- Capture unhandled exceptions before PM2/process manager restarts
- Report errors with full stack traces and context
- Integrate with Pino logger for breadcrumbs
- Track transaction performance (optional)
### 3. Frontend Integration: @sentry/react
The React frontend will integrate `@sentry/react` SDK to:
- Wrap the app in a Sentry Error Boundary
- Capture unhandled JavaScript errors
- Report errors with component stack traces
- Track user session context
### 4. Log Aggregation: Logstash
**Logstash** parses application and infrastructure logs, forwarding error patterns to Bugsink:
- **Installation**: Installed inside the dev container (and on bare-metal prod servers)
- **Inputs**:
- Pino JSON logs from the Node.js application
- Redis logs (connection errors, memory warnings, slow commands)
- PostgreSQL function logs (future - see Implementation Steps)
- **Filter**: Identifies error-level logs (5xx responses, unhandled exceptions, Redis errors)
- **Output**: Sends to Bugsink via Sentry-compatible HTTP API
This provides a secondary error capture path for:
- Errors that occur before Sentry SDK initialization
- Log-based errors that don't throw exceptions
- Redis connection/performance issues
- Database function errors and slow queries
- Historical error analysis from log files
### 5. MCP Server Integration: sentry-selfhosted-mcp
For AI tool integration (Claude Code, Cursor, etc.), we use the open-source [sentry-selfhosted-mcp](https://github.com/ddfourtwo/sentry-selfhosted-mcp) server:
- **No code changes required**: Configurable via environment variables
- **Capabilities**: List projects, get issues, view events, update status, add comments
- **Configuration**:
- `SENTRY_URL`: Points to Bugsink instance
- `SENTRY_AUTH_TOKEN`: API token from Bugsink
- `SENTRY_ORG_SLUG`: Organization identifier
## Architecture
```text
┌─────────────────────────────────────────────────────────────────────────┐
│ Dev Container / Production Server │
├─────────────────────────────────────────────────────────────────────────┤
│ │
│ ┌──────────────────┐ ┌──────────────────┐ │
│ │ Frontend │ │ Backend │ │
│ │ (React) │ │ (Express) │ │
│ │ @sentry/react │ │ @sentry/node │ │
│ └────────┬─────────┘ └────────┬─────────┘ │
│ │ │ │
│ │ Sentry SDK Protocol │ │
│ └───────────┬───────────────┘ │
│ │ │
│ ▼ │
│ ┌──────────────────────┐ │
│ │ Bugsink │ │
│ │ (localhost:8000) │◄──────────────────┐ │
│ │ │ │ │
│ │ PostgreSQL backend │ │ │
│ └──────────────────────┘ │ │
│ │ │
│ ┌──────────────────────┐ │ │
│ │ Logstash │───────────────────┘ │
│ │ (Log Aggregator) │ Sentry Output │
│ │ │ │
│ │ Inputs: │ │
│ │ - Pino app logs │ │
│ │ - Redis logs │ │
│ │ - PostgreSQL (future) │
│ └──────────────────────┘ │
│ ▲ ▲ ▲ │
│ │ │ │ │
│ ┌───────────┘ │ └───────────┐ │
│ │ │ │ │
│ ┌────┴─────┐ ┌─────┴────┐ ┌──────┴─────┐ │
│ │ Pino │ │ Redis │ │ PostgreSQL │ │
│ │ Logs │ │ Logs │ │ Logs (TBD) │ │
│ └──────────┘ └──────────┘ └────────────┘ │
│ │
│ ┌──────────────────────┐ │
│ │ PostgreSQL │ │
│ │ ┌────────────────┐ │ │
│ │ │ flyer_crawler │ │ (main app database) │
│ │ ├────────────────┤ │ │
│ │ │ bugsink │ │ (error tracking database) │
│ │ └────────────────┘ │ │
│ └──────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────┘
External (Developer Machine):
┌──────────────────────────────────────┐
│ Claude Code / Cursor / VS Code │
│ ┌────────────────────────────────┐ │
│ │ sentry-selfhosted-mcp │ │
│ │ (MCP Server) │ │
│ │ │ │
│ │ SENTRY_URL=http://localhost:8000
│ │ SENTRY_AUTH_TOKEN=... │ │
│ │ SENTRY_ORG_SLUG=... │ │
│ └────────────────────────────────┘ │
└──────────────────────────────────────┘
```
## Configuration
### Environment Variables
| Variable | Description | Default (Dev) |
| ------------------ | ------------------------------ | -------------------------- |
| `BUGSINK_DSN` | Sentry-compatible DSN for SDKs | Set after project creation |
| `BUGSINK_ENABLED` | Enable/disable error reporting | `true` |
| `BUGSINK_BASE_URL` | Bugsink web UI URL (internal) | `http://localhost:8000` |
### PostgreSQL Setup
```sql
-- Create dedicated Bugsink database and user
CREATE USER bugsink WITH PASSWORD 'bugsink_dev_password';
CREATE DATABASE bugsink OWNER bugsink;
GRANT ALL PRIVILEGES ON DATABASE bugsink TO bugsink;
```
### Bugsink Configuration
```bash
# Environment variables for Bugsink service
SECRET_KEY=<random-50-char-string>
DATABASE_URL=postgresql://bugsink:bugsink_dev_password@localhost:5432/bugsink
BASE_URL=http://localhost:8000
PORT=8000
```
### Logstash Pipeline
```conf
# /etc/logstash/conf.d/bugsink.conf
# === INPUTS ===
input {
# Pino application logs
file {
path => "/app/logs/*.log"
codec => json
type => "pino"
tags => ["app"]
}
# Redis logs
file {
path => "/var/log/redis/*.log"
type => "redis"
tags => ["redis"]
}
# PostgreSQL logs (for function logging - future)
# file {
# path => "/var/log/postgresql/*.log"
# type => "postgres"
# tags => ["postgres"]
# }
}
# === FILTERS ===
filter {
# Pino error detection (level 50 = error, 60 = fatal)
if [type] == "pino" and [level] >= 50 {
mutate { add_tag => ["error"] }
}
# Redis error detection
if [type] == "redis" {
grok {
match => { "message" => "%{POSINT:pid}:%{WORD:role} %{MONTHDAY} %{MONTH} %{TIME} %{WORD:loglevel} %{GREEDYDATA:redis_message}" }
}
if [loglevel] in ["WARNING", "ERROR"] {
mutate { add_tag => ["error"] }
}
}
# PostgreSQL function error detection (future)
# if [type] == "postgres" {
# # Parse PostgreSQL log format and detect ERROR/FATAL levels
# }
}
# === OUTPUT ===
output {
if "error" in [tags] {
http {
url => "http://localhost:8000/api/store/"
http_method => "post"
format => "json"
# Sentry envelope format
}
}
}
```
## Implementation Steps
1. **Update Dockerfile.dev**:
- Install Bugsink (pip package or binary)
- Install Logstash (Elastic APT repository)
- Add systemd service files for both
2. **PostgreSQL initialization**:
- Add Bugsink user/database creation to `sql/00-init-extensions.sql`
3. **Backend SDK integration**:
- Install `@sentry/node`
- Initialize in `server.ts` before Express app
- Configure error handler middleware integration
4. **Frontend SDK integration**:
- Install `@sentry/react`
- Wrap `App` component with `Sentry.ErrorBoundary`
- Configure in `src/index.tsx`
5. **Environment configuration**:
- Add Bugsink variables to `src/config/env.ts`
- Update `.env.example` and `compose.dev.yml`
6. **Logstash configuration**:
- Create pipeline config for Pino → Bugsink
- Configure Pino to write to log file in addition to stdout
- Configure Redis log monitoring (connection errors, slow commands)
7. **MCP server documentation**:
- Document `sentry-selfhosted-mcp` setup in CLAUDE.md
8. **PostgreSQL function logging** (future):
- Configure PostgreSQL to log function execution errors
- Add Logstash input for PostgreSQL logs
- Define filter rules for function-level error detection
- _Note: Ask for implementation details when this step is reached_
## Consequences
**Positive**: Provides critical observability into the application's real-world behavior. Enables proactive identification and resolution of performance bottlenecks and errors. Improves overall application reliability and user experience.
**Negative**: Introduces a new third-party dependency and potential subscription costs. Requires initial setup and configuration of the APM/error tracking agent.
### Positive
- **Full observability**: Aggregated view of errors, trends, and performance
- **Self-hosted**: No external SaaS dependencies or subscription costs
- **SDK compatibility**: Leverages mature Sentry SDKs with excellent documentation
- **AI integration**: MCP server enables Claude Code to query and analyze errors
- **Unified architecture**: Same setup works in dev container and production
- **Lightweight**: Bugsink runs in a single process, unlike full Sentry (16GB+ RAM)
### Negative
- **Additional services**: Bugsink and Logstash add complexity to the container
- **PostgreSQL overhead**: Additional database for error tracking
- **Initial setup**: Requires configuration of multiple components
- **Logstash learning curve**: Pipeline configuration requires Logstash knowledge
## Alternatives Considered
1. **Full Sentry self-hosted**: Rejected due to complexity (Kafka, Redis, ClickHouse, 16GB+ RAM minimum)
2. **GlitchTip**: Considered, but Bugsink is lighter weight and easier to deploy
3. **Sentry SaaS**: Rejected due to self-hosted requirement
4. **Custom error aggregation**: Rejected in favor of proven Sentry SDK ecosystem
## References
- [Bugsink Documentation](https://www.bugsink.com/docs/)
- [Bugsink Docker Install](https://www.bugsink.com/docs/docker-install/)
- [@sentry/node Documentation](https://docs.sentry.io/platforms/javascript/guides/node/)
- [@sentry/react Documentation](https://docs.sentry.io/platforms/javascript/guides/react/)
- [sentry-selfhosted-mcp](https://github.com/ddfourtwo/sentry-selfhosted-mcp)
- [Logstash Reference](https://www.elastic.co/guide/en/logstash/current/index.html)

256
docs/adr/0018-api-documentation-strategy.md
View File

@@ -2,17 +2,265 @@
**Date**: 2025-12-12
**Status**: Proposed
**Status**: Accepted
**Implemented**: 2026-01-11
## Context
As the API grows, it becomes increasingly difficult for frontend developers and other consumers to understand its endpoints, request formats, and response structures. There is no single source of truth for API documentation.
Key requirements:
1. **Developer Experience**: Developers need interactive documentation to explore and test API endpoints.
2. **Code-Documentation Sync**: Documentation should stay in sync with the actual code to prevent drift.
3. **Low Maintenance Overhead**: The documentation approach should be "fast and lite" - minimal additional work for developers.
4. **Security**: Documentation should not expose sensitive information in production environments.
## Decision
We will adopt **OpenAPI (Swagger)** for API documentation. We will use tools (e.g., JSDoc annotations with `swagger-jsdoc`) to generate an `openapi.json` specification directly from the route handler source code. This specification will be served via a UI like Swagger UI for interactive exploration.
We will adopt **OpenAPI 3.0 (Swagger)** for API documentation using the following approach:
1. **JSDoc Annotations**: Use `swagger-jsdoc` to generate OpenAPI specs from JSDoc comments in route files.
2. **Swagger UI**: Use `swagger-ui-express` to serve interactive documentation at `/docs/api-docs`.
3. **Environment Restriction**: Only expose the Swagger UI in development and test environments, not production.
4. **Incremental Adoption**: Start with key public routes and progressively add annotations to all endpoints.
### Tooling Selection
| Tool | Purpose |
| -------------------- | ---------------------------------------------- |
| `swagger-jsdoc` | Generates OpenAPI 3.0 spec from JSDoc comments |
| `swagger-ui-express` | Serves interactive Swagger UI |
**Why JSDoc over separate schema files?**
- Documentation lives with the code, reducing drift
- No separate files to maintain
- Developers see documentation when editing routes
- Lower learning curve for the team
## Implementation Details
### OpenAPI Configuration
Located in `src/config/swagger.ts`:
```typescript
import swaggerJsdoc from 'swagger-jsdoc';
const options: swaggerJsdoc.Options = {
definition: {
openapi: '3.0.0',
info: {
title: 'Flyer Crawler API',
version: '1.0.0',
description: 'API for the Flyer Crawler application',
contact: {
name: 'API Support',
},
},
servers: [
{
url: '/api',
description: 'API server',
},
],
components: {
securitySchemes: {
bearerAuth: {
type: 'http',
scheme: 'bearer',
bearerFormat: 'JWT',
},
},
},
},
apis: ['./src/routes/*.ts'],
};
export const swaggerSpec = swaggerJsdoc(options);
```
### JSDoc Annotation Pattern
Each route handler should include OpenAPI annotations using the `@openapi` tag:
```typescript
/**
* @openapi
* /health/ping:
* get:
* summary: Simple ping endpoint
* description: Returns a pong response to verify server is responsive
* tags:
* - Health
* responses:
* 200:
* description: Server is responsive
* content:
* application/json:
* schema:
* type: object
* properties:
* success:
* type: boolean
* example: true
* data:
* type: object
* properties:
* message:
* type: string
* example: pong
*/
router.get('/ping', validateRequest(emptySchema), (_req: Request, res: Response) => {
return sendSuccess(res, { message: 'pong' });
});
```
### Route Documentation Priority
Document routes in this order of priority:
1. **Health Routes** - `/api/health/*` (public, critical for operations)
2. **Auth Routes** - `/api/auth/*` (public, essential for integration)
3. **Gamification Routes** - `/api/achievements/*` (simple, good example)
4. **Flyer Routes** - `/api/flyers/*` (core functionality)
5. **User Routes** - `/api/users/*` (common CRUD patterns)
6. **Remaining Routes** - Budget, Recipe, Admin, etc.
### Swagger UI Setup
In `server.ts`, add the Swagger UI middleware (development/test only):
```typescript
import swaggerUi from 'swagger-ui-express';
import { swaggerSpec } from './src/config/swagger';
// Only serve Swagger UI in non-production environments
if (process.env.NODE_ENV !== 'production') {
app.use('/docs/api-docs', swaggerUi.serve, swaggerUi.setup(swaggerSpec));
// Optionally expose raw JSON spec for tooling
app.get('/docs/api-docs.json', (_req, res) => {
res.setHeader('Content-Type', 'application/json');
res.send(swaggerSpec);
});
}
```
### Response Schema Standardization
All API responses follow the standardized format from [ADR-028](./0028-api-response-standardization.md):
```typescript
// Success response
{
"success": true,
"data": { ... }
}
// Error response
{
"success": false,
"error": {
"code": "ERROR_CODE",
"message": "Human-readable message"
}
}
```
Define reusable schema components for these patterns:
```typescript
/**
* @openapi
* components:
* schemas:
* SuccessResponse:
* type: object
* properties:
* success:
* type: boolean
* example: true
* data:
* type: object
* ErrorResponse:
* type: object
* properties:
* success:
* type: boolean
* example: false
* error:
* type: object
* properties:
* code:
* type: string
* message:
* type: string
*/
```
### Security Considerations
1. **Production Disabled**: Swagger UI is not available in production to prevent information disclosure.
2. **No Sensitive Data**: Never include actual secrets, tokens, or PII in example values.
3. **Authentication Documented**: Clearly document which endpoints require authentication.
## API Route Tags
Organize endpoints using consistent tags:
| Tag | Description | Routes |
| ------------ | ---------------------------------- | --------------------- |
| Health | Server health and readiness checks | `/api/health/*` |
| Auth | Authentication and authorization | `/api/auth/*` |
| Users | User profile management | `/api/users/*` |
| Flyers | Flyer uploads and retrieval | `/api/flyers/*` |
| Achievements | Gamification and leaderboards | `/api/achievements/*` |
| Budgets | Budget tracking | `/api/budgets/*` |
| Recipes | Recipe management | `/api/recipes/*` |
| Admin | Administrative operations | `/api/admin/*` |
| System | System status and monitoring | `/api/system/*` |
## Testing
Verify API documentation is correct by:
1. **Manual Review**: Navigate to `/docs/api-docs` and test each endpoint.
2. **Spec Validation**: Use OpenAPI validators to check the generated spec.
3. **Integration Tests**: Existing integration tests serve as implicit documentation verification.
## Consequences
- **Positive**: Creates a single source of truth for API documentation that stays in sync with the code. Enables auto-generation of client SDKs and simplifies testing.
- **Negative**: Requires developers to maintain JSDoc annotations on all routes. Adds a build step and new dependencies to the project.
### Positive
- **Single Source of Truth**: Documentation lives with the code and stays in sync.
- **Interactive Exploration**: Developers can try endpoints directly from the UI.
- **SDK Generation**: OpenAPI spec enables automatic client SDK generation.
- **Onboarding**: New developers can quickly understand the API surface.
- **Low Overhead**: JSDoc annotations are minimal additions to existing code.
### Negative
- **Maintenance Required**: Developers must update annotations when routes change.
- **Build Dependency**: Adds `swagger-jsdoc` and `swagger-ui-express` packages.
- **Initial Investment**: Existing routes need annotations added incrementally.
### Mitigation
- Include documentation checks in code review process.
- Start with high-priority routes and expand coverage over time.
- Use TypeScript types to reduce documentation duplication where possible.
## Key Files
- `src/config/swagger.ts` - OpenAPI configuration
- `src/routes/*.ts` - Route files with JSDoc annotations
- `server.ts` - Swagger UI middleware setup
## Related ADRs
- [ADR-003](./0003-standardized-input-validation-using-middleware.md) - Input Validation (Zod schemas)
- [ADR-028](./0028-api-response-standardization.md) - Response Standardization
- [ADR-016](./0016-api-security-hardening.md) - Security Hardening

299
docs/adr/0049-gamification-and-achievement-system.md Normal file
View File

@@ -0,0 +1,299 @@
# ADR-049: Gamification and Achievement System
**Date**: 2026-01-11
**Status**: Accepted
**Implemented**: 2026-01-11
## Context
The application implements a gamification system to encourage user engagement through achievements and points. Users earn achievements for completing specific actions within the platform, and these achievements contribute to a points-based leaderboard.
Key requirements:
1. **User Engagement**: Reward users for meaningful actions (uploads, recipes, sharing).
2. **Progress Tracking**: Show users their accomplishments and progress.
3. **Social Competition**: Leaderboard to compare users by points.
4. **Idempotent Awards**: Achievements should only be awarded once per user.
5. **Transactional Safety**: Achievement awards must be atomic with the triggering action.
## Decision
We will implement a database-driven gamification system with:
1. **Database Functions**: Core logic in PostgreSQL for atomicity and idempotency.
2. **Database Triggers**: Automatic achievement awards on specific events.
3. **Application-Level Awards**: Explicit calls from service layer when triggers aren't suitable.
4. **Points Aggregation**: Stored in user profile for efficient leaderboard queries.
### Design Principles
- **Single Award**: Each achievement can only be earned once per user (enforced by unique constraint).
- **Atomic Operations**: Achievement awards happen within the same transaction as the triggering action.
- **Silent Failure**: If an achievement doesn't exist, the award function returns silently (no error).
- **Points Sync**: Points are updated on the profile immediately when an achievement is awarded.
## Implementation Details
### Database Schema
```sql
-- Achievements master table
CREATE TABLE public.achievements (
achievement_id BIGSERIAL PRIMARY KEY,
name TEXT UNIQUE NOT NULL,
description TEXT NOT NULL,
icon TEXT NOT NULL,
points_value INTEGER NOT NULL DEFAULT 0,
created_at TIMESTAMPTZ DEFAULT NOW()
);
-- User achievements (junction table)
CREATE TABLE public.user_achievements (
user_id UUID REFERENCES public.users(user_id) ON DELETE CASCADE,
achievement_id BIGINT REFERENCES public.achievements(achievement_id) ON DELETE CASCADE,
achieved_at TIMESTAMPTZ DEFAULT NOW(),
PRIMARY KEY (user_id, achievement_id)
);
-- Points stored on profile for efficient leaderboard
ALTER TABLE public.profiles ADD COLUMN points INTEGER DEFAULT 0;
```
### Award Achievement Function
Located in `sql/Initial_triggers_and_functions.sql`:
```sql
CREATE OR REPLACE FUNCTION public.award_achievement(p_user_id UUID, p_achievement_name TEXT)
RETURNS void
LANGUAGE plpgsql
SECURITY DEFINER
AS $$
DECLARE
v_achievement_id BIGINT;
v_points_value INTEGER;
BEGIN
-- Find the achievement by name to get its ID and point value.
SELECT achievement_id, points_value INTO v_achievement_id, v_points_value
FROM public.achievements WHERE name = p_achievement_name;
-- If the achievement doesn't exist, do nothing.
IF v_achievement_id IS NULL THEN
RETURN;
END IF;
-- Insert the achievement for the user.
-- ON CONFLICT DO NOTHING ensures idempotency.
INSERT INTO public.user_achievements (user_id, achievement_id)
VALUES (p_user_id, v_achievement_id)
ON CONFLICT (user_id, achievement_id) DO NOTHING;
-- If the insert was successful (user didn't have it), update their points.
IF FOUND THEN
UPDATE public.profiles SET points = points + v_points_value WHERE user_id = p_user_id;
END IF;
END;
$$;
```
### Current Achievements
| Name | Description | Icon | Points |
| -------------------- | ----------------------------------------------------------- | ------------ | ------ |
| Welcome Aboard | Join the community by creating your account. | user-check | 5 |
| First Recipe | Create your very first recipe. | chef-hat | 10 |
| Recipe Sharer | Share a recipe with another user for the first time. | share-2 | 15 |
| List Sharer | Share a shopping list with another user for the first time. | list | 20 |
| First Favorite | Mark a recipe as one of your favorites. | heart | 5 |
| First Fork | Make a personal copy of a public recipe. | git-fork | 10 |
| First Budget Created | Create your first budget to track spending. | piggy-bank | 15 |
| First-Upload | Upload your first flyer. | upload-cloud | 25 |
### Achievement Triggers
#### User Registration (Database Trigger)
Awards "Welcome Aboard" when a new user is created:
```sql
-- In handle_new_user() function
PERFORM public.award_achievement(new.user_id, 'Welcome Aboard');
```
#### Flyer Upload (Database Trigger + Application Code)
Awards "First-Upload" when a flyer is inserted with an `uploaded_by` value:
```sql
-- In log_new_flyer() trigger function
IF NEW.uploaded_by IS NOT NULL THEN
PERFORM public.award_achievement(NEW.uploaded_by, 'First-Upload');
END IF;
```
Additionally, the `FlyerPersistenceService.saveFlyer()` method explicitly awards the achievement within the transaction:
```typescript
// In src/services/flyerPersistenceService.server.ts
if (userId) {
const gamificationRepo = new GamificationRepository(client);
await gamificationRepo.awardAchievement(userId, 'First-Upload', logger);
}
```
### Repository Layer
Located in `src/services/db/gamification.db.ts`:
```typescript
export class GamificationRepository {
private db: Pick<Pool | PoolClient, 'query'>;
constructor(db: Pick<Pool | PoolClient, 'query'> = getPool()) {
this.db = db;
}
async getUserAchievements(
userId: string,
logger: Logger,
): Promise<(UserAchievement & Achievement)[]> {
const query = `
SELECT ua.user_id, ua.achievement_id, ua.achieved_at,
a.name, a.description, a.icon, a.points_value, a.created_at
FROM public.user_achievements ua
JOIN public.achievements a ON ua.achievement_id = a.achievement_id
WHERE ua.user_id = $1
ORDER BY ua.achieved_at DESC;
`;
const res = await this.db.query(query, [userId]);
return res.rows;
}
async awardAchievement(userId: string, achievementName: string, logger: Logger): Promise<void> {
await this.db.query('SELECT public.award_achievement($1, $2)', [userId, achievementName]);
}
async getLeaderboard(limit: number, logger: Logger): Promise<LeaderboardUser[]> {
const query = `
SELECT user_id, full_name, avatar_url, points,
RANK() OVER (ORDER BY points DESC) as rank
FROM public.profiles
ORDER BY points DESC, full_name ASC
LIMIT $1;
`;
const res = await this.db.query(query, [limit]);
return res.rows;
}
}
```
### API Endpoints
| Method | Endpoint | Description |
| ------ | ------------------------------- | ------------------------------- |
| GET | `/api/achievements` | List all available achievements |
| GET | `/api/achievements/me` | Get current user's achievements |
| GET | `/api/achievements/leaderboard` | Get top users by points |
## Testing Considerations
### Critical Testing Requirements
When testing gamification features, be aware of the following:
1. **Database Seed Data**: Achievement definitions must exist in the database before tests run. The `award_achievement()` function silently returns if the achievement name doesn't exist.
2. **Transactional Context**: When awarding achievements from within a transaction:
- The achievement is visible within the transaction immediately
- External queries won't see the achievement until the transaction commits
- Tests should wait for job completion before asserting achievement state
3. **Vitest Global Setup Context**: The integration test global setup runs in a separate Node.js context. Achievement verification must use direct database queries, not mocked services.
4. **Achievement Idempotency**: Calling `award_achievement()` multiple times for the same user/achievement combination is safe and expected. Only the first call actually inserts.
### Example Integration Test Pattern
```typescript
it('should award the "First Upload" achievement after flyer processing', async () => {
// 1. Create user (awards "Welcome Aboard" via database trigger)
const { user: testUser, token } = await createAndLoginUser({...});
// 2. Upload flyer (triggers async job)
const uploadResponse = await request
.post('/api/flyers/upload')
.set('Authorization', `Bearer ${token}`)
.attach('flyerFile', testImagePath);
expect(uploadResponse.status).toBe(202);
// 3. Wait for job to complete
await poll(async () => {
const status = await request.get(`/api/flyers/job/${jobId}/status`);
return status.body.data.status === 'completed';
}, { timeout: 15000 });
// 4. Wait for achievements to be visible (transaction committed)
await vi.waitUntil(async () => {
const achievements = await db.gamificationRepo.getUserAchievements(
testUser.user.user_id,
logger
);
return achievements.length >= 2; // Welcome Aboard + First-Upload
}, { timeout: 15000, interval: 500 });
// 5. Assert specific achievements
const userAchievements = await db.gamificationRepo.getUserAchievements(
testUser.user.user_id,
logger
);
expect(userAchievements.find(a => a.name === 'Welcome Aboard')).toBeDefined();
expect(userAchievements.find(a => a.name === 'First-Upload')).toBeDefined();
});
```
### Common Test Pitfalls
1. **Missing Seed Data**: If tests fail with "achievement not found", ensure the test database has the achievements table populated.
2. **Race Conditions**: Achievement awards in async jobs may not be visible immediately. Always poll or use `vi.waitUntil()`.
3. **Wrong User ID**: Verify the user ID passed to `awardAchievement()` matches the user created in the test.
4. **Transaction Isolation**: When querying within a test, use the same database connection if checking mid-transaction state.
## Consequences
### Positive
- **Engagement**: Users have clear goals and rewards for platform activity.
- **Scalability**: Points stored on profile enable O(1) leaderboard sorting.
- **Reliability**: Database-level idempotency prevents duplicate awards.
- **Flexibility**: New achievements can be added via SQL without code changes.
### Negative
- **Complexity**: Multiple award paths (triggers + application code) require careful coordination.
- **Testing**: Async nature of some awards complicates integration testing.
- **Coupling**: Achievement names are strings; typos fail silently.
### Mitigation
- Use constants for achievement names in application code.
- Document all award trigger points clearly.
- Test each achievement path independently.
## Key Files
- `sql/initial_data.sql` - Achievement definitions (seed data)
- `sql/Initial_triggers_and_functions.sql` - `award_achievement()` function and triggers
- `src/services/db/gamification.db.ts` - Repository layer
- `src/routes/achievements.routes.ts` - API endpoints
- `src/services/flyerPersistenceService.server.ts` - First-Upload award (application code)
## Related ADRs
- [ADR-002](./0002-standardized-transaction-management.md) - Transaction Management
- [ADR-034](./0034-repository-pattern-standards.md) - Repository Pattern
- [ADR-006](./0006-background-job-processing-and-task-queues.md) - Background Jobs (flyer processing)

341
docs/adr/0050-postgresql-function-observability.md Normal file
View File

@@ -0,0 +1,341 @@
# ADR-050: PostgreSQL Function Observability
**Date**: 2026-01-11
**Status**: Proposed
**Related**: [ADR-015](0015-application-performance-monitoring-and-error-tracking.md), [ADR-004](0004-standardized-application-wide-structured-logging.md)
## Context
The application uses 30+ PostgreSQL functions and 11+ triggers for business logic, including:
- Recipe recommendations and search
- Shopping list generation from menu plans
- Price history tracking
- Achievement awards
- Activity logging
- User profile creation
**Current Problem**: These database functions can fail silently in several ways:
1. **`ON CONFLICT DO NOTHING`** - Swallows constraint violations without notification
2. **`IF NOT FOUND THEN RETURN;`** - Silently exits when data is missing
3. **Trigger functions returning `NULL`** - No indication of partial failures
4. **No logging inside functions** - No visibility into function execution
When these silent failures occur:
- The application layer receives no error (function "succeeds" but does nothing)
- No logs are generated for debugging
- Issues are only discovered when users report missing data
- Root cause analysis is extremely difficult
**Example of Silent Failure**:
```sql
-- This function silently does nothing if achievement doesn't exist
CREATE OR REPLACE FUNCTION public.award_achievement(p_user_id UUID, p_achievement_name TEXT)
RETURNS void AS $$
BEGIN
SELECT achievement_id INTO v_achievement_id FROM achievements WHERE name = p_achievement_name;
IF v_achievement_id IS NULL THEN
RETURN; -- Silent failure - no log, no error
END IF;
-- ...
END;
$$;
```
ADR-015 established Logstash + Bugsink for error tracking, with PostgreSQL log integration marked as "future". This ADR defines the implementation.
## Decision
We will implement a standardized PostgreSQL function observability strategy with three tiers of logging severity:
### 1. Function Logging Helper
Create a reusable logging function that outputs structured JSON to PostgreSQL logs:
```sql
-- Function to emit structured log messages from PL/pgSQL
CREATE OR REPLACE FUNCTION public.fn_log(
p_level TEXT, -- 'DEBUG', 'INFO', 'NOTICE', 'WARNING', 'ERROR'
p_function_name TEXT, -- The calling function name
p_message TEXT, -- Human-readable message
p_context JSONB DEFAULT NULL -- Additional context (user_id, params, etc.)
)
RETURNS void
LANGUAGE plpgsql
AS $$
DECLARE
log_line TEXT;
BEGIN
-- Build structured JSON log line
log_line := jsonb_build_object(
'timestamp', now(),
'level', p_level,
'source', 'postgresql',
'function', p_function_name,
'message', p_message,
'context', COALESCE(p_context, '{}'::jsonb)
)::text;
-- Use appropriate RAISE level
CASE p_level
WHEN 'DEBUG' THEN RAISE DEBUG '%', log_line;
WHEN 'INFO' THEN RAISE INFO '%', log_line;
WHEN 'NOTICE' THEN RAISE NOTICE '%', log_line;
WHEN 'WARNING' THEN RAISE WARNING '%', log_line;
WHEN 'ERROR' THEN RAISE LOG '%', log_line; -- Use LOG for errors to ensure capture
ELSE RAISE NOTICE '%', log_line;
END CASE;
END;
$$;
```
### 2. Logging Tiers
#### Tier 1: Critical Functions (Always Log)
Functions where silent failure causes data corruption or user-facing issues:
| Function | Log Events |
| ---------------------------------- | --------------------------------------- |
| `handle_new_user()` | User creation, profile creation, errors |
| `award_achievement()` | Achievement not found, already awarded |
| `approve_correction()` | Correction not found, permission denied |
| `complete_shopping_list()` | List not found, permission denied |
| `add_menu_plan_to_shopping_list()` | Permission denied, items added |
| `fork_recipe()` | Original not found, fork created |
**Pattern**:
```sql
CREATE OR REPLACE FUNCTION public.award_achievement(p_user_id UUID, p_achievement_name TEXT)
RETURNS void AS $$
DECLARE
v_achievement_id BIGINT;
v_points_value INTEGER;
v_context JSONB;
BEGIN
v_context := jsonb_build_object('user_id', p_user_id, 'achievement_name', p_achievement_name);
SELECT achievement_id, points_value INTO v_achievement_id, v_points_value
FROM public.achievements WHERE name = p_achievement_name;
IF v_achievement_id IS NULL THEN
-- Log the issue instead of silent return
PERFORM fn_log('WARNING', 'award_achievement',
'Achievement not found: ' || p_achievement_name, v_context);
RETURN;
END IF;
INSERT INTO public.user_achievements (user_id, achievement_id)
VALUES (p_user_id, v_achievement_id)
ON CONFLICT (user_id, achievement_id) DO NOTHING;
IF FOUND THEN
UPDATE public.profiles SET points = points + v_points_value WHERE user_id = p_user_id;
PERFORM fn_log('INFO', 'award_achievement',
'Achievement awarded: ' || p_achievement_name, v_context);
END IF;
END;
$$;
```
#### Tier 2: Business Logic Functions (Log on Anomalies)
Functions where unexpected conditions should be logged but aren't critical:
| Function | Log Events |
| -------------------------------------- | ---------------------------------- |
| `suggest_master_item_for_flyer_item()` | No match found (below threshold) |
| `recommend_recipes_for_user()` | No recommendations generated |
| `find_recipes_from_pantry()` | Empty pantry, no recipes found |
| `get_best_sale_prices_for_user()` | No watched items, no current sales |
**Pattern**: Log when results are unexpectedly empty or inputs are invalid.
#### Tier 3: Triggers (Log Errors Only)
Triggers should be fast, so only log when something goes wrong:
| Trigger Function | Log Events |
| --------------------------------------------- | ------------------------- |
| `update_price_history_on_flyer_item_insert()` | Failed to update history |
| `update_recipe_rating_aggregates()` | Rating calculation failed |
| `log_new_recipe()` | Profile lookup failed |
| `log_new_flyer()` | Store lookup failed |
### 3. PostgreSQL Configuration
Enable logging in `postgresql.conf`:
```ini
# Log all function notices and above
log_min_messages = notice
# Include function name in log prefix
log_line_prefix = '%t [%p] %u@%d '
# Log to file for Logstash pickup
logging_collector = on
log_directory = '/var/log/postgresql'
log_filename = 'postgresql-%Y-%m-%d.log'
log_rotation_age = 1d
log_rotation_size = 100MB
# Capture slow queries from functions
log_min_duration_statement = 1000 # Log queries over 1 second
```
### 4. Logstash Integration
Update the Logstash pipeline (extends ADR-015 configuration):
```conf
# PostgreSQL function log input
input {
file {
path => "/var/log/postgresql/*.log"
type => "postgres"
tags => ["postgres"]
start_position => "beginning"
sincedb_path => "/var/lib/logstash/sincedb_postgres"
}
}
filter {
if [type] == "postgres" {
# Extract timestamp and process ID from PostgreSQL log prefix
grok {
match => { "message" => "%{TIMESTAMP_ISO8601:pg_timestamp} \[%{POSINT:pg_pid}\] %{USER:pg_user}@%{WORD:pg_database} %{GREEDYDATA:pg_message}" }
}
# Check if this is a structured JSON log from fn_log()
if [pg_message] =~ /^\{.*"source":"postgresql".*\}$/ {
json {
source => "pg_message"
target => "fn_log"
}
# Mark as error if level is WARNING or ERROR
if [fn_log][level] in ["WARNING", "ERROR"] {
mutate { add_tag => ["error", "db_function"] }
}
}
# Also catch native PostgreSQL errors
if [pg_message] =~ /^ERROR:/ or [pg_message] =~ /^FATAL:/ {
mutate { add_tag => ["error", "postgres_native"] }
}
}
}
output {
if "error" in [tags] and "postgres" in [tags] {
http {
url => "http://localhost:8000/api/store/"
http_method => "post"
format => "json"
}
}
}
```
### 5. Dual-File Update Requirement
**IMPORTANT**: All SQL function changes must be applied to BOTH files:
1. `sql/Initial_triggers_and_functions.sql` - Used for incremental updates
2. `sql/master_schema_rollup.sql` - Used for fresh database setup
Both files must remain in sync for triggers and functions.
## Implementation Steps
1. **Create `fn_log()` helper function**:
- Add to both `Initial_triggers_and_functions.sql` and `master_schema_rollup.sql`
- Test with `SELECT fn_log('INFO', 'test', 'Test message', '{"key": "value"}'::jsonb);`
2. **Update Tier 1 critical functions** (highest priority):
- `award_achievement()` - Log missing achievements, duplicate awards
- `handle_new_user()` - Log user creation success/failure
- `approve_correction()` - Log not found, permission denied
- `complete_shopping_list()` - Log permission checks
- `add_menu_plan_to_shopping_list()` - Log permission checks, items added
- `fork_recipe()` - Log original not found
3. **Update Tier 2 business logic functions**:
- Add anomaly logging to suggestion/recommendation functions
- Log empty result sets with context
4. **Update Tier 3 trigger functions**:
- Add error-only logging to critical triggers
- Wrap complex trigger logic in exception handlers
5. **Configure PostgreSQL logging**:
- Update `postgresql.conf` in dev container
- Update production PostgreSQL configuration
- Verify logs appear in expected location
6. **Update Logstash pipeline**:
- Add PostgreSQL input to `bugsink.conf`
- Add filter rules for structured JSON extraction
- Test end-to-end: function log → Logstash → Bugsink
7. **Verify in Bugsink**:
- Confirm database function errors appear as issues
- Verify context (user_id, function name, params) is captured
## Consequences
### Positive
- **Visibility**: Silent failures become visible in error tracking
- **Debugging**: Function execution context captured for root cause analysis
- **Proactive detection**: Anomalies logged before users report issues
- **Unified monitoring**: Database errors appear alongside application errors in Bugsink
- **Structured logs**: JSON format enables filtering and aggregation
### Negative
- **Performance overhead**: Logging adds latency to function execution
- **Log volume**: Tier 1/2 functions may generate significant log volume
- **Maintenance**: Two SQL files must be kept in sync
- **PostgreSQL configuration**: Requires access to `postgresql.conf`
### Mitigations
- **Performance**: Only log meaningful events, not every function call
- **Log volume**: Use appropriate log levels; Logstash filters reduce noise
- **Sync**: Add CI check to verify SQL files match for function definitions
- **Configuration**: Document PostgreSQL settings in deployment runbook
## Examples
### Before (Silent Failure)
```sql
-- User thinks achievement was awarded, but it silently failed
SELECT award_achievement('user-uuid', 'Nonexistent Badge');
-- Returns: void (no error, no log)
-- Result: User never gets achievement, nobody knows why
```
### After (Observable Failure)
```sql
SELECT award_achievement('user-uuid', 'Nonexistent Badge');
-- Returns: void
-- PostgreSQL log: {"timestamp":"2026-01-11T10:30:00Z","level":"WARNING","source":"postgresql","function":"award_achievement","message":"Achievement not found: Nonexistent Badge","context":{"user_id":"user-uuid","achievement_name":"Nonexistent Badge"}}
-- Bugsink: New issue created with full context
```
## References
- [ADR-015: Application Performance Monitoring](0015-application-performance-monitoring-and-error-tracking.md)
- [ADR-004: Standardized Structured Logging](0004-standardized-application-wide-structured-logging.md)
- [PostgreSQL RAISE Documentation](https://www.postgresql.org/docs/current/plpgsql-errors-and-messages.html)
- [PostgreSQL Logging Configuration](https://www.postgresql.org/docs/current/runtime-config-logging.html)

65
docs/adr/adr-implementation-tracker.md
View File

@@ -15,7 +15,7 @@ This document tracks the implementation status and estimated effort for all Arch
| Status | Count |
| ---------------------------- | ----- |
| Accepted (Fully Implemented) | 28 |
| Accepted (Fully Implemented) | 30 |
| Partially Implemented | 2 |
| Proposed (Not Started) | 16 |
@@ -48,7 +48,7 @@ This document tracks the implementation status and estimated effort for all Arch
| ------------------------------------------------------------------- | ------------------------ | ----------- | ------ | ------------------------------------- |
| [ADR-003](./0003-standardized-input-validation-using-middleware.md) | Input Validation | Accepted | - | Fully implemented |
| [ADR-008](./0008-api-versioning-strategy.md) | API Versioning | Proposed | L | Major URL/routing changes |
| [ADR-018](./0018-api-documentation-strategy.md) | API Documentation | Proposed | M | OpenAPI/Swagger setup |
| [ADR-018](./0018-api-documentation-strategy.md) | API Documentation | Accepted | - | OpenAPI/Swagger implemented |
| [ADR-022](./0022-real-time-notification-system.md) | Real-time Notifications | Proposed | XL | WebSocket infrastructure |
| [ADR-028](./0028-api-response-standardization.md) | Response Standardization | Implemented | L | Completed (routes, middleware, tests) |
@@ -65,10 +65,11 @@ This document tracks the implementation status and estimated effort for all Arch
### Category 5: Observability & Monitoring
| ADR | Title | Status | Effort | Notes |
| -------------------------------------------------------------------------- | -------------------- | -------- | ------ | ----------------------- |
| [ADR-004](./0004-standardized-application-wide-structured-logging.md) | Structured Logging | Accepted | - | Fully implemented |
| [ADR-015](./0015-application-performance-monitoring-and-error-tracking.md) | APM & Error Tracking | Proposed | M | Third-party integration |
| ADR | Title | Status | Effort | Notes |
| -------------------------------------------------------------------------- | --------------------------- | -------- | ------ | --------------------------------- |
| [ADR-004](./0004-standardized-application-wide-structured-logging.md) | Structured Logging | Accepted | - | Fully implemented |
| [ADR-015](./0015-application-performance-monitoring-and-error-tracking.md) | APM & Error Tracking | Proposed | M | Third-party integration |
| [ADR-050](./0050-postgresql-function-observability.md) | PostgreSQL Fn Observability | Proposed | M | Depends on ADR-015 implementation |
### Category 6: Deployment & Operations
@@ -113,6 +114,7 @@ This document tracks the implementation status and estimated effort for all Arch
| [ADR-042](./0042-email-and-notification-architecture.md) | Email & Notifications | Accepted | - | Fully implemented |
| [ADR-043](./0043-express-middleware-pipeline.md) | Middleware Pipeline | Accepted | - | Fully implemented |
| [ADR-046](./0046-image-processing-pipeline.md) | Image Processing | Accepted | - | Fully implemented |
| [ADR-049](./0049-gamification-and-achievement-system.md) | Gamification System | Accepted | - | Fully implemented |
---
@@ -120,35 +122,38 @@ This document tracks the implementation status and estimated effort for all Arch
These ADRs are proposed but not yet implemented, ordered by suggested implementation priority:
| Priority | ADR | Title | Effort | Rationale |
| -------- | ------- | ------------------------ | ------ | ----------------------------------------------------- |
| 1 | ADR-018 | API Documentation | M | Improves developer experience, enables SDK generation |
| 2 | ADR-015 | APM & Error Tracking | M | Production visibility, debugging |
| 3 | ADR-024 | Feature Flags | M | Safer deployments, A/B testing |
| 4 | ADR-023 | Schema Migrations v2 | L | Database evolution support |
| 5 | ADR-029 | Secret Rotation | L | Security improvement |
| 6 | ADR-008 | API Versioning | L | Future API evolution |
| 7 | ADR-030 | Circuit Breaker | L | Resilience improvement |
| 8 | ADR-022 | Real-time Notifications | XL | Major feature enhancement |
| 9 | ADR-011 | Authorization & RBAC | XL | Advanced permission system |
| 10 | ADR-025 | i18n & l10n | XL | Multi-language support |
| 11 | ADR-031 | Data Retention & Privacy | XL | Compliance requirements |
| Priority | ADR | Title | Effort | Rationale |
| -------- | ------- | --------------------------- | ------ | ------------------------------------------------- |
| 1 | ADR-015 | APM & Error Tracking | M | Production visibility, debugging |
| 1b | ADR-050 | PostgreSQL Fn Observability | M | Database function visibility (depends on ADR-015) |
| 2 | ADR-024 | Feature Flags | M | Safer deployments, A/B testing |
| 3 | ADR-023 | Schema Migrations v2 | L | Database evolution support |
| 4 | ADR-029 | Secret Rotation | L | Security improvement |
| 5 | ADR-008 | API Versioning | L | Future API evolution |
| 6 | ADR-030 | Circuit Breaker | L | Resilience improvement |
| 7 | ADR-022 | Real-time Notifications | XL | Major feature enhancement |
| 8 | ADR-011 | Authorization & RBAC | XL | Advanced permission system |
| 9 | ADR-025 | i18n & l10n | XL | Multi-language support |
| 10 | ADR-031 | Data Retention & Privacy | XL | Compliance requirements |
---
## Recent Implementation History
| Date | ADR | Change |
| ---------- | ------- | --------------------------------------------------------------------------------------------- |
| 2026-01-09 | ADR-047 | Created - Documents target project file/folder organization with migration plan |
| 2026-01-09 | ADR-041 | Created - Documents AI/Gemini integration with model fallback and rate limiting |
| 2026-01-09 | ADR-042 | Created - Documents email and notification architecture with BullMQ queuing |
| 2026-01-09 | ADR-043 | Created - Documents Express middleware pipeline ordering and patterns |
| 2026-01-09 | ADR-044 | Created - Documents frontend feature-based folder organization |
| 2026-01-09 | ADR-045 | Created - Documents test data factory pattern for mock generation |
| 2026-01-09 | ADR-046 | Created - Documents image processing pipeline with Sharp and EXIF stripping |
| 2026-01-09 | ADR-026 | Fully implemented - all client-side components, hooks, and services now use structured logger |
| 2026-01-09 | ADR-028 | Fully implemented - all routes, middleware, and tests updated |
| Date | ADR | Change |
| ---------- | ------- | ---------------------------------------------------------------------- |
| 2026-01-11 | ADR-050 | Created - PostgreSQL function observability with fn_log() and Logstash |
| 2026-01-11 | ADR-018 | Implemented - OpenAPI/Swagger documentation at /docs/api-docs |
| 2026-01-11 | ADR-049 | Created - Gamification system, achievements, and testing requirements |
| 2026-01-09 | ADR-047 | Created - Project file/folder organization with migration plan |
| 2026-01-09 | ADR-041 | Created - AI/Gemini integration with model fallback and rate limiting |
| 2026-01-09 | ADR-042 | Created - Email and notification architecture with BullMQ queuing |
| 2026-01-09 | ADR-043 | Created - Express middleware pipeline ordering and patterns |
| 2026-01-09 | ADR-044 | Created - Frontend feature-based folder organization |
| 2026-01-09 | ADR-045 | Created - Test data factory pattern for mock generation |
| 2026-01-09 | ADR-046 | Created - Image processing pipeline with Sharp and EXIF stripping |
| 2026-01-09 | ADR-026 | Fully implemented - client-side structured logger |
| 2026-01-09 | ADR-028 | Fully implemented - all routes, middleware, and tests updated |
---