Compare commits

..

102 Commits

Author SHA1 Message Date
a997d1d0b0 ranstack query fixes
All checks were successful
Deploy to Test Environment / deploy-to-test (push) Successful in 13m21s
2026-01-10 19:03:40 -08:00
cf5f77c58e Adopt TanStack Query fixes 2026-01-10 19:02:42 -08:00
Gitea Actions
cf0f5bb820 ci: Bump version to 0.9.85 [skip ci] 2026-01-11 06:44:28 +05:00
503e7084da Adopt TanStack Query fixes
All checks were successful
Deploy to Test Environment / deploy-to-test (push) Successful in 14m41s
2026-01-10 17:42:45 -08:00
Gitea Actions
d8aa19ac40 ci: Bump version to 0.9.84 [skip ci] 2026-01-10 23:45:42 +05:00
dcd9452b8c Adopt TanStack Query
All checks were successful
Deploy to Test Environment / deploy-to-test (push) Successful in 13m46s
2026-01-10 10:45:10 -08:00
Gitea Actions
6d468544e2 ci: Bump version to 0.9.83 [skip ci] 2026-01-10 23:14:18 +05:00
2913c7aa09 tanstack
Some checks failed
Deploy to Test Environment / deploy-to-test (push) Failing after 1m1s
2026-01-10 03:20:40 -08:00
Gitea Actions
77f9cb6081 ci: Bump version to 0.9.82 [skip ci] 2026-01-10 12:17:24 +05:00
2f1d73ca12 fix(tests): access wrapped API response data correctly
Some checks failed
Deploy to Test Environment / deploy-to-test (push) Failing after 3h0m5s
Tests were accessing response.body directly instead of response.body.data,
causing failures since sendSuccess() wraps responses in { success, data }.
2026-01-09 23:16:30 -08:00
Gitea Actions
402e2617ca ci: Bump version to 0.9.81 [skip ci] 2026-01-10 11:40:07 +05:00
e14c19c112 linting docs + some fixes go claude and gemini
All checks were successful
Deploy to Test Environment / deploy-to-test (push) Successful in 16m0s
2026-01-09 22:38:57 -08:00
Gitea Actions
ea46f66c7a ci: Bump version to 0.9.80 [skip ci] 2026-01-10 11:00:30 +05:00
a42ee5a461 unit tests - wheeee! Claude is the mvp
All checks were successful
Deploy to Test Environment / deploy-to-test (push) Successful in 15m11s
2026-01-09 21:59:09 -08:00
Gitea Actions
71710c8316 ci: Bump version to 0.9.79 [skip ci] 2026-01-10 09:32:36 +05:00
1480a73ab0 more compliance
Some checks failed
Deploy to Test Environment / deploy-to-test (push) Failing after 58s
2026-01-09 20:30:52 -08:00
Gitea Actions
b3efa3c756 ci: Bump version to 0.9.78 [skip ci] 2026-01-10 08:01:56 +05:00
fb8fd57bb6 huge linting fixes
All checks were successful
Deploy to Test Environment / deploy-to-test (push) Successful in 15m3s
2026-01-09 19:01:05 -08:00
Gitea Actions
0a90d9d590 ci: Bump version to 0.9.77 [skip ci] 2026-01-10 07:54:20 +05:00
6ab473f5f0 huge linting fixes
Some checks failed
Deploy to Test Environment / deploy-to-test (push) Failing after 58s
2026-01-09 18:50:04 -08:00
Gitea Actions
c46efe1474 ci: Bump version to 0.9.76 [skip ci] 2026-01-10 06:59:56 +05:00
25d6b76f6d ADR-026: Client-Side Logging + linting fixes
Some checks failed
Deploy to Test Environment / deploy-to-test (push) Has been cancelled
2026-01-09 17:58:21 -08:00
Gitea Actions
9ffcc9d65d ci: Bump version to 0.9.75 [skip ci] 2026-01-10 03:25:25 +05:00
1285702210 adr-028 fixes for tests
All checks were successful
Deploy to Test Environment / deploy-to-test (push) Successful in 15m38s
2026-01-09 14:24:20 -08:00
Gitea Actions
d38b751b40 ci: Bump version to 0.9.74 [skip ci] 2026-01-10 03:14:12 +05:00
e122d55ced adr-028 fixes for tests
Some checks failed
Deploy to Test Environment / deploy-to-test (push) Failing after 1m1s
2026-01-09 14:12:48 -08:00
Gitea Actions
af9992f773 ci: Bump version to 0.9.73 [skip ci] 2026-01-10 01:54:56 +05:00
3912139273 adr-028 and int tests
All checks were successful
Deploy to Test Environment / deploy-to-test (push) Successful in 16m24s
2026-01-09 12:47:41 -08:00
b5f7f5e4d1 adr-0028 and int test fixes 2026-01-09 12:35:55 -08:00
Gitea Actions
5173059621 ci: Bump version to 0.9.72 [skip ci] 2026-01-10 00:46:09 +05:00
ebceb0e2e3 just work
All checks were successful
Deploy to Test Environment / deploy-to-test (push) Successful in 20m51s
2026-01-09 11:45:03 -08:00
e75054b1ab ADR work, dockerfile work, integration test fixes 2026-01-09 11:45:00 -08:00
Gitea Actions
639313485a ci: Bump version to 0.9.71 [skip ci] 2026-01-09 19:00:01 +05:00
4a04e478c4 integration test fixes - claude for the win? try 4 - i have a good feeling
Some checks failed
Deploy to Test Environment / deploy-to-test (push) Failing after 16m58s
2026-01-09 05:56:19 -08:00
Gitea Actions
1814469eb4 ci: Bump version to 0.9.70 [skip ci] 2026-01-09 18:19:13 +05:00
b777430ff7 integration test fixes - claude for the win? try 4 - i have a good feeling
Some checks failed
Deploy to Test Environment / deploy-to-test (push) Has been cancelled
2026-01-09 05:18:19 -08:00
Gitea Actions
23830c0d4e ci: Bump version to 0.9.69 [skip ci] 2026-01-09 17:24:00 +05:00
ef42fee982 integration test fixes - claude for the win? try 3
All checks were successful
Deploy to Test Environment / deploy-to-test (push) Successful in 32m3s
2026-01-09 04:23:23 -08:00
Gitea Actions
65cb54500c ci: Bump version to 0.9.68 [skip ci] 2026-01-09 16:42:51 +05:00
664ad291be integration test fixes - claude for the win? try 3
All checks were successful
Deploy to Test Environment / deploy-to-test (push) Successful in 30m3s
2026-01-09 03:41:57 -08:00
Gitea Actions
ff912b9055 ci: Bump version to 0.9.67 [skip ci] 2026-01-09 15:32:50 +05:00
ec32027bd4 integration test fixes - claude for the win?
All checks were successful
Deploy to Test Environment / deploy-to-test (push) Successful in 33m43s
2026-01-09 02:32:16 -08:00
Gitea Actions
59f773639b ci: Bump version to 0.9.66 [skip ci] 2026-01-09 15:27:50 +05:00
dd2be5eecf integration test fixes - claude for the win?
Some checks failed
Deploy to Test Environment / deploy-to-test (push) Failing after 1m0s
2026-01-09 02:27:14 -08:00
Gitea Actions
a94bfbd3e9 ci: Bump version to 0.9.65 [skip ci] 2026-01-09 14:43:36 +05:00
338bbc9440 integration test fixes - claude for the win?
All checks were successful
Deploy to Test Environment / deploy-to-test (push) Successful in 20m4s
2026-01-09 01:42:51 -08:00
Gitea Actions
60aad04642 ci: Bump version to 0.9.64 [skip ci] 2026-01-09 13:57:52 +05:00
7f2aff9a24 unit test fixes
All checks were successful
Deploy to Test Environment / deploy-to-test (push) Successful in 23m39s
2026-01-09 00:57:12 -08:00
Gitea Actions
689320e7d2 ci: Bump version to 0.9.63 [skip ci] 2026-01-09 13:19:09 +05:00
e457bbf046 more req work
All checks were successful
Deploy to Test Environment / deploy-to-test (push) Successful in 26m51s
2026-01-09 00:18:09 -08:00
68cdbb6066 progress enforcing adr-0005 2026-01-09 00:18:09 -08:00
Gitea Actions
cea6be7145 ci: Bump version to 0.9.62 [skip ci] 2026-01-09 11:31:00 +05:00
74a5ca6331 claude 1 - fixes : -/
All checks were successful
Deploy to Test Environment / deploy-to-test (push) Successful in 24m33s
2026-01-08 22:30:21 -08:00
Gitea Actions
62470e7661 ci: Bump version to 0.9.61 [skip ci] 2026-01-09 10:50:57 +05:00
2b517683fd progress enforcing adr-0005
Some checks failed
Deploy to Test Environment / deploy-to-test (push) Failing after 1m2s
2026-01-08 21:50:21 -08:00
Gitea Actions
5d06d1ba09 ci: Bump version to 0.9.60 [skip ci] 2026-01-09 10:41:14 +05:00
46c1e56b14 progress enforcing adr-0005
Some checks failed
Deploy to Test Environment / deploy-to-test (push) Failing after 46s
2026-01-08 21:40:20 -08:00
Gitea Actions
78a9b80010 ci: Bump version to 0.9.59 [skip ci] 2026-01-08 20:48:22 +05:00
d356d9dfb6 claude 1
Some checks failed
Deploy to Test Environment / deploy-to-test (push) Failing after 43s
2026-01-08 07:47:29 -08:00
Gitea Actions
ab63f83f50 ci: Bump version to 0.9.58 [skip ci] 2026-01-08 05:23:21 +05:00
b546a55eaf fix the dang integration tests
All checks were successful
Deploy to Test Environment / deploy-to-test (push) Successful in 32m3s
2026-01-07 16:22:48 -08:00
Gitea Actions
dfa53a93dd ci: Bump version to 0.9.57 [skip ci] 2026-01-08 04:39:12 +05:00
f30464cd0e fix the dang integration tests
All checks were successful
Deploy to Test Environment / deploy-to-test (push) Successful in 19m1s
2026-01-07 15:38:14 -08:00
Gitea Actions
2d2fa3c2c8 ci: Bump version to 0.9.56 [skip ci] 2026-01-08 00:40:29 +05:00
58cb391f4b fix the dang integration tests
All checks were successful
Deploy to Test Environment / deploy-to-test (push) Successful in 21m36s
2026-01-07 11:39:35 -08:00
Gitea Actions
0ebe2f0806 ci: Bump version to 0.9.55 [skip ci] 2026-01-07 14:43:38 +05:00
7867abc5bc fix the dang integration tests
All checks were successful
Deploy to Test Environment / deploy-to-test (push) Successful in 24m23s
2026-01-07 01:42:43 -08:00
Gitea Actions
cc4c8e2839 ci: Bump version to 0.9.54 [skip ci] 2026-01-07 10:49:08 +05:00
33ee2eeac9 switch to instantiating the pm2 worker in the testing threads
All checks were successful
Deploy to Test Environment / deploy-to-test (push) Successful in 26m44s
2026-01-06 21:48:35 -08:00
Gitea Actions
e0b13f26fb ci: Bump version to 0.9.53 [skip ci] 2026-01-07 09:57:37 +05:00
eee7f36756 switch to instantiating the pm2 worker in the testing threads
All checks were successful
Deploy to Test Environment / deploy-to-test (push) Successful in 30m53s
2026-01-06 20:56:39 -08:00
Gitea Actions
622c919733 ci: Bump version to 0.9.52 [skip ci] 2026-01-07 08:26:14 +05:00
c7f6b6369a fix the dang integration tests
All checks were successful
Deploy to Test Environment / deploy-to-test (push) Successful in 30m27s
2026-01-06 19:25:25 -08:00
Gitea Actions
879d956003 ci: Bump version to 0.9.51 [skip ci] 2026-01-07 07:11:22 +05:00
27eaac7ea8 fix the dang integration tests
All checks were successful
Deploy to Test Environment / deploy-to-test (push) Successful in 27m15s
2026-01-06 18:10:47 -08:00
Gitea Actions
93618c57e5 ci: Bump version to 0.9.50 [skip ci] 2026-01-07 06:41:16 +05:00
7f043ef704 fix the dang integration tests
All checks were successful
Deploy to Test Environment / deploy-to-test (push) Successful in 29m45s
2026-01-06 17:40:20 -08:00
Gitea Actions
62e35deddc ci: Bump version to 0.9.49 [skip ci] 2026-01-07 02:54:13 +05:00
59f6f43d03 fix the dang integration tests
All checks were successful
Deploy to Test Environment / deploy-to-test (push) Successful in 32m36s
2026-01-06 13:53:00 -08:00
Gitea Actions
e675c1a73c ci: Bump version to 0.9.48 [skip ci] 2026-01-07 01:35:26 +05:00
3c19084a0a fix the dang integration tests
All checks were successful
Deploy to Test Environment / deploy-to-test (push) Successful in 30m17s
2026-01-06 12:34:18 -08:00
Gitea Actions
e2049c6b9f ci: Bump version to 0.9.47 [skip ci] 2026-01-06 23:34:29 +05:00
a3839c2f0d debugging the flyer integration issue
All checks were successful
Deploy to Test Environment / deploy-to-test (push) Successful in 33m13s
2026-01-06 10:33:51 -08:00
Gitea Actions
c1df3d7b1b ci: Bump version to 0.9.46 [skip ci] 2026-01-06 22:39:47 +05:00
94782f030d debugging the flyer integration issue
All checks were successful
Deploy to Test Environment / deploy-to-test (push) Successful in 25m42s
2026-01-06 09:38:14 -08:00
Gitea Actions
1c25b79251 ci: Bump version to 0.9.45 [skip ci] 2026-01-06 14:34:44 +05:00
0b0fa8294d debugging the flyer integration issue
All checks were successful
Deploy to Test Environment / deploy-to-test (push) Successful in 27m54s
2026-01-06 01:33:48 -08:00
Gitea Actions
f49f3a75fb ci: Bump version to 0.9.44 [skip ci] 2026-01-06 13:41:43 +05:00
8f14044ae6 debugging the flyer integration issue
All checks were successful
Deploy to Test Environment / deploy-to-test (push) Successful in 26m27s
2026-01-06 00:41:03 -08:00
Gitea Actions
55e1e425f4 ci: Bump version to 0.9.43 [skip ci] 2026-01-06 12:56:47 +05:00
68b16ad2e8 fix the dang integration tests
All checks were successful
Deploy to Test Environment / deploy-to-test (push) Successful in 25m2s
2026-01-05 23:53:54 -08:00
Gitea Actions
6a28934692 ci: Bump version to 0.9.42 [skip ci] 2026-01-06 12:25:08 +05:00
78c4a5fee6 fix the dang integration tests
Some checks failed
Deploy to Test Environment / deploy-to-test (push) Has been cancelled
2026-01-05 23:20:56 -08:00
Gitea Actions
1ce5f481a8 ci: Bump version to 0.9.41 [skip ci] 2026-01-06 11:39:28 +05:00
Gitea Actions
e0120d38fd ci: Bump version to 0.9.39 [skip ci] 2026-01-06 11:39:27 +05:00
6b2079ef2c fix the dang integration tests
All checks were successful
Deploy to Test Environment / deploy-to-test (push) Successful in 32m44s
2026-01-05 22:38:21 -08:00
Gitea Actions
0478e176d5 ci: Bump version to 0.9.38 [skip ci] 2026-01-06 10:23:22 +05:00
47f7f97cd9 fuck database contraints - seems buggy
All checks were successful
Deploy to Test Environment / deploy-to-test (push) Successful in 32m10s
2026-01-05 21:16:08 -08:00
Gitea Actions
b0719d1e39 ci: Bump version to 0.9.37 [skip ci] 2026-01-06 10:11:19 +05:00
0039ac3752 fuck database contraints - seems buggy
Some checks failed
Deploy to Test Environment / deploy-to-test (push) Failing after 37s
2026-01-05 21:08:16 -08:00
Gitea Actions
3c8316f4f7 ci: Bump version to 0.9.36 [skip ci] 2026-01-06 09:03:20 +05:00
2564df1c64 get rid of localhost in tests - not a qualified URL - we'll see
All checks were successful
Deploy to Test Environment / deploy-to-test (push) Successful in 33m19s
2026-01-05 20:02:44 -08:00
372 changed files with 32852 additions and 7359 deletions

16
.claude/hooks.json Normal file
View File

@@ -0,0 +1,16 @@
{
"$schema": "https://claude.ai/schemas/hooks.json",
"hooks": {
"PreToolUse": [
{
"matcher": "Bash",
"hooks": [
{
"type": "command",
"command": "node -e \"const cmd = process.argv[1] || ''; const isTest = /\\b(npm\\s+(run\\s+)?test|vitest|jest)\\b/i.test(cmd); const isWindows = process.platform === 'win32'; const inContainer = process.env.REMOTE_CONTAINERS === 'true' || process.env.DEVCONTAINER === 'true'; if (isTest && isWindows && !inContainer) { console.error('BLOCKED: Tests must run on Linux. Use Dev Container (Reopen in Container) or WSL.'); process.exit(1); }\" -- \"$CLAUDE_TOOL_INPUT\""
}
]
}
]
}
}

View File

@@ -0,0 +1,93 @@
{
"permissions": {
"allow": [
"Bash(npm test:*)",
"Bash(podman --version:*)",
"Bash(podman ps:*)",
"Bash(podman machine start:*)",
"Bash(podman compose:*)",
"Bash(podman pull:*)",
"Bash(podman images:*)",
"Bash(podman stop:*)",
"Bash(echo:*)",
"Bash(podman rm:*)",
"Bash(podman run:*)",
"Bash(podman start:*)",
"Bash(podman exec:*)",
"Bash(cat:*)",
"Bash(PGPASSWORD=postgres psql:*)",
"Bash(npm search:*)",
"Bash(npx:*)",
"Bash(curl:*)",
"Bash(powershell:*)",
"Bash(cmd.exe:*)",
"Bash(npm run test:integration:*)",
"Bash(grep:*)",
"Bash(done)",
"Bash(podman info:*)",
"Bash(podman machine:*)",
"Bash(podman system connection:*)",
"Bash(podman inspect:*)",
"Bash(python -m json.tool:*)",
"Bash(claude mcp status)",
"Bash(powershell.exe -Command \"claude mcp status\")",
"Bash(powershell.exe -Command \"claude mcp\")",
"Bash(powershell.exe -Command \"claude mcp list\")",
"Bash(powershell.exe -Command \"claude --version\")",
"Bash(powershell.exe -Command \"claude config\")",
"Bash(powershell.exe -Command \"claude mcp get gitea-projectium\")",
"Bash(powershell.exe -Command \"claude mcp add --help\")",
"Bash(powershell.exe -Command \"claude mcp add -t stdio -s user filesystem -- D:\\\\nodejs\\\\npx.cmd -y @modelcontextprotocol/server-filesystem D:\\\\gitea\\\\flyer-crawler.projectium.com\\\\flyer-crawler.projectium.com\")",
"Bash(powershell.exe -Command \"claude mcp add -t stdio -s user fetch -- D:\\\\nodejs\\\\npx.cmd -y @modelcontextprotocol/server-fetch\")",
"Bash(powershell.exe -Command \"echo ''List files in src/hooks using filesystem MCP'' | claude --print\")",
"Bash(powershell.exe -Command \"echo ''List all podman containers'' | claude --print\")",
"Bash(powershell.exe -Command \"echo ''List my repositories on gitea.projectium.com using gitea-projectium MCP'' | claude --print\")",
"Bash(powershell.exe -Command \"echo ''List my repositories on gitea.projectium.com using gitea-projectium MCP'' | claude --print --allowedTools ''mcp__gitea-projectium__*''\")",
"Bash(powershell.exe -Command \"echo ''Fetch the homepage of https://gitea.projectium.com and summarize it'' | claude --print --allowedTools ''mcp__fetch__*''\")",
"Bash(dir \"C:\\\\Users\\\\games3\\\\.claude\")",
"Bash(dir:*)",
"Bash(D:nodejsnpx.cmd -y @modelcontextprotocol/server-fetch --help)",
"Bash(cmd /c \"dir /o-d C:\\\\Users\\\\games3\\\\.claude\\\\debug 2>nul | head -10\")",
"mcp__memory__read_graph",
"mcp__memory__create_entities",
"mcp__memory__search_nodes",
"mcp__memory__delete_entities",
"mcp__sequential-thinking__sequentialthinking",
"mcp__filesystem__list_directory",
"mcp__filesystem__read_multiple_files",
"mcp__filesystem__directory_tree",
"mcp__filesystem__read_text_file",
"Bash(wc:*)",
"Bash(npm install:*)",
"Bash(git grep:*)",
"Bash(findstr:*)",
"Bash(git add:*)",
"mcp__filesystem__write_file",
"mcp__podman__container_list",
"Bash(podman cp:*)",
"mcp__podman__container_inspect",
"mcp__podman__network_list",
"Bash(podman network connect:*)",
"Bash(npm run build:*)",
"Bash(set NODE_ENV=test)",
"Bash(podman-compose:*)",
"Bash(timeout 60 podman machine start:*)",
"Bash(podman build:*)",
"Bash(podman network rm:*)",
"Bash(npm run lint)",
"Bash(npm run typecheck:*)",
"Bash(npm run type-check:*)",
"Bash(npm run test:unit:*)",
"mcp__filesystem__move_file",
"Bash(git checkout:*)",
"Bash(podman image inspect:*)",
"Bash(node -e:*)",
"Bash(xargs -I {} sh -c 'if ! grep -q \"\"vi.mock.*apiClient\"\" \"\"{}\"\"; then echo \"\"{}\"\"; fi')",
"Bash(MSYS_NO_PATHCONV=1 podman exec:*)",
"Bash(docker ps:*)",
"Bash(find:*)",
"Bash(\"/c/Users/games3/.local/bin/uvx.exe\" markitdown-mcp --help)",
"Bash(git stash:*)"
]
}
}

View File

@@ -1,18 +1,96 @@
{
// ============================================================================
// VS CODE DEV CONTAINER CONFIGURATION
// ============================================================================
// This file configures VS Code's Dev Containers extension to provide a
// consistent, fully-configured development environment.
//
// Features:
// - Automatic PostgreSQL + Redis startup with healthchecks
// - Automatic npm install
// - Automatic database schema initialization and seeding
// - Pre-configured VS Code extensions (ESLint, Prettier)
// - Podman support for Windows users
//
// Usage:
// 1. Install the "Dev Containers" extension in VS Code
// 2. Open this project folder
// 3. Click "Reopen in Container" when prompted (or use Command Palette)
// 4. Wait for container build and initialization
// 5. Development server starts automatically
// ============================================================================
"name": "Flyer Crawler Dev (Ubuntu 22.04)",
// Use Docker Compose for multi-container setup
"dockerComposeFile": ["../compose.dev.yml"],
"service": "app",
"workspaceFolder": "/app",
// VS Code customizations
"customizations": {
"vscode": {
"extensions": ["dbaeumer.vscode-eslint", "esbenp.prettier-vscode"]
"extensions": [
// Code quality
"dbaeumer.vscode-eslint",
"esbenp.prettier-vscode",
// TypeScript
"ms-vscode.vscode-typescript-next",
// Database
"mtxr.sqltools",
"mtxr.sqltools-driver-pg",
// Utilities
"eamodio.gitlens",
"streetsidesoftware.code-spell-checker"
],
"settings": {
"editor.formatOnSave": true,
"editor.defaultFormatter": "esbenp.prettier-vscode",
"typescript.preferences.importModuleSpecifier": "relative"
}
}
},
// Run as root (required for npm global installs)
"remoteUser": "root",
// Automatically install dependencies when the container is created.
// This runs inside the container, populating the isolated node_modules volume.
"postCreateCommand": "npm install",
// ============================================================================
// Lifecycle Commands
// ============================================================================
// initializeCommand: Runs on the HOST before the container is created.
// Starts Podman machine on Windows (no-op if already running or using Docker).
"initializeCommand": "powershell -Command \"podman machine start; exit 0\"",
// postCreateCommand: Runs ONCE when the container is first created.
// This is where we do full initialization: npm install + database setup.
"postCreateCommand": "chmod +x scripts/docker-init.sh && ./scripts/docker-init.sh",
// postAttachCommand: Runs EVERY TIME VS Code attaches to the container.
// Starts the development server automatically.
"postAttachCommand": "npm run dev:container",
// Try to start podman machine, but exit with success (0) even if it's already running
"initializeCommand": "powershell -Command \"podman machine start; exit 0\""
// ============================================================================
// Port Forwarding
// ============================================================================
// Automatically forward these ports from the container to the host
"forwardPorts": [3000, 3001],
// Labels for forwarded ports in VS Code's Ports panel
"portsAttributes": {
"3000": {
"label": "Frontend (Vite)",
"onAutoForward": "notify"
},
"3001": {
"label": "Backend API",
"onAutoForward": "notify"
}
},
// ============================================================================
// Features
// ============================================================================
// Additional dev container features (optional)
"features": {}
}

77
.env.example Normal file
View File

@@ -0,0 +1,77 @@
# .env.example
# ============================================================================
# ENVIRONMENT VARIABLES TEMPLATE
# ============================================================================
# Copy this file to .env and fill in your values.
# For local development with Docker/Podman, these defaults should work out of the box.
#
# IMPORTANT: Never commit .env files with real credentials to version control!
# ============================================================================
# ===================
# Database Configuration
# ===================
# PostgreSQL connection settings
# For container development, use the service name "postgres"
DB_HOST=postgres
DB_PORT=5432
DB_USER=postgres
DB_PASSWORD=postgres
DB_NAME=flyer_crawler_dev
# ===================
# Redis Configuration
# ===================
# Redis URL for caching and job queues
# For container development, use the service name "redis"
REDIS_URL=redis://redis:6379
# Optional: Redis password (leave empty if not required)
REDIS_PASSWORD=
# ===================
# Application Settings
# ===================
NODE_ENV=development
# Frontend URL for CORS and email links
FRONTEND_URL=http://localhost:3000
# ===================
# Authentication
# ===================
# REQUIRED: Secret key for signing JWT tokens (generate a random 64+ character string)
JWT_SECRET=your-super-secret-jwt-key-change-this-in-production
# ===================
# AI/ML Services
# ===================
# REQUIRED: Google Gemini API key for flyer OCR processing
GEMINI_API_KEY=your-gemini-api-key
# ===================
# External APIs
# ===================
# Optional: Google Maps API key for geocoding store addresses
GOOGLE_MAPS_API_KEY=
# ===================
# Email Configuration (Optional)
# ===================
# SMTP settings for sending emails (deal notifications, password reset)
SMTP_HOST=
SMTP_PORT=587
SMTP_SECURE=false
SMTP_USER=
SMTP_PASS=
SMTP_FROM_EMAIL=noreply@example.com
# ===================
# Worker Configuration (Optional)
# ===================
# Concurrency settings for background job workers
WORKER_CONCURRENCY=1
EMAIL_WORKER_CONCURRENCY=10
ANALYTICS_WORKER_CONCURRENCY=1
CLEANUP_WORKER_CONCURRENCY=10
# Worker lock duration in milliseconds (default: 2 minutes)
WORKER_LOCK_DURATION=120000

6
.env.test Normal file
View File

@@ -0,0 +1,6 @@
DB_HOST=10.89.0.4
DB_USER=flyer
DB_PASSWORD=flyer
DB_NAME=flyer_crawler_test
REDIS_URL=redis://redis:6379
NODE_ENV=test

View File

@@ -117,7 +117,8 @@ jobs:
DB_USER: ${{ secrets.DB_USER }}
DB_PASSWORD: ${{ secrets.DB_PASSWORD }}
DB_NAME: ${{ secrets.DB_DATABASE_PROD }}
REDIS_URL: 'redis://localhost:6379'
# Explicitly use database 0 for production (test uses database 1)
REDIS_URL: 'redis://localhost:6379/0'
REDIS_PASSWORD: ${{ secrets.REDIS_PASSWORD_PROD }}
FRONTEND_URL: 'https://flyer-crawler.projectium.com'
JWT_SECRET: ${{ secrets.JWT_SECRET }}

View File

@@ -96,6 +96,24 @@ jobs:
# It prevents the accumulation of duplicate processes from previous test runs.
node -e "const exec = require('child_process').execSync; try { const list = JSON.parse(exec('pm2 jlist').toString()); list.forEach(p => { if (p.name && p.name.endsWith('-test')) { console.log('Deleting test process: ' + p.name + ' (' + p.pm2_env.pm_id + ')'); try { exec('pm2 delete ' + p.pm2_env.pm_id); } catch(e) { console.error('Failed to delete ' + p.pm2_env.pm_id, e.message); } } }); console.log('✅ Test process cleanup complete.'); } catch (e) { if (e.stdout.toString().includes('No process found')) { console.log('No PM2 processes running, cleanup not needed.'); } else { console.error('Error cleaning up test processes:', e.message); } }" || true
- name: Flush Redis Test Database Before Tests
# CRITICAL: Clear Redis database 1 (test database) to remove stale BullMQ jobs.
# This prevents old jobs with outdated error messages from polluting test results.
# NOTE: We use database 1 for tests to isolate from production (database 0).
env:
REDIS_PASSWORD: ${{ secrets.REDIS_PASSWORD_TEST }}
run: |
echo "--- Flushing Redis database 1 (test database) to remove stale jobs ---"
if [ -z "$REDIS_PASSWORD" ]; then
echo "⚠️ REDIS_PASSWORD_TEST not set, attempting flush without password..."
redis-cli -n 1 FLUSHDB || echo "Redis flush failed (no password)"
else
redis-cli -a "$REDIS_PASSWORD" -n 1 FLUSHDB 2>/dev/null && echo "✅ Redis database 1 (test) flushed successfully." || echo "⚠️ Redis flush failed"
fi
# Verify the flush worked by checking key count on database 1
KEY_COUNT=$(redis-cli -a "$REDIS_PASSWORD" -n 1 DBSIZE 2>/dev/null | grep -oE '[0-9]+' || echo "unknown")
echo "Redis database 1 key count after flush: $KEY_COUNT"
- name: Run All Tests and Generate Merged Coverage Report
# This single step runs both unit and integration tests, then merges their
# coverage data into a single report. It combines the environment variables
@@ -109,14 +127,23 @@ jobs:
DB_NAME: 'flyer-crawler-test' # Explicitly set for tests
# --- Redis credentials for the test suite ---
REDIS_URL: 'redis://localhost:6379'
# CRITICAL: Use Redis database 1 to isolate tests from production (which uses db 0).
# This prevents the production worker from picking up test jobs.
REDIS_URL: 'redis://localhost:6379/1'
REDIS_PASSWORD: ${{ secrets.REDIS_PASSWORD_TEST }}
# --- Integration test specific variables ---
FRONTEND_URL: 'http://example.com'
FRONTEND_URL: 'https://example.com'
VITE_API_BASE_URL: 'http://localhost:3001/api'
GEMINI_API_KEY: ${{ secrets.VITE_GOOGLE_GENAI_API_KEY }}
# --- Storage path for flyer images ---
# CRITICAL: Use an absolute path in the test runner's working directory for file storage.
# This ensures tests can read processed files to verify their contents (e.g., EXIF stripping).
# Without this, multer and flyerProcessingService default to /var/www/.../flyer-images.
# NOTE: We use ${{ github.workspace }} which resolves to the checkout directory.
STORAGE_PATH: '${{ github.workspace }}/flyer-images'
# --- JWT Secret for Passport authentication in tests ---
JWT_SECRET: ${{ secrets.JWT_SECRET }}
@@ -384,12 +411,12 @@ jobs:
DB_PASSWORD: ${{ secrets.DB_PASSWORD }}
DB_NAME: ${{ secrets.DB_DATABASE_TEST }}
# Redis Credentials
REDIS_URL: 'redis://localhost:6379'
# Redis Credentials (use database 1 to isolate from production)
REDIS_URL: 'redis://localhost:6379/1'
REDIS_PASSWORD: ${{ secrets.REDIS_PASSWORD_TEST }}
# Application Secrets
FRONTEND_URL: 'http://example.com'
FRONTEND_URL: 'https://example.com'
JWT_SECRET: ${{ secrets.JWT_SECRET }}
GEMINI_API_KEY: ${{ secrets.VITE_GOOGLE_GENAI_API_KEY_TEST }}
GOOGLE_MAPS_API_KEY: ${{ secrets.GOOGLE_MAPS_API_KEY }}

View File

@@ -116,7 +116,8 @@ jobs:
DB_USER: ${{ secrets.DB_USER }}
DB_PASSWORD: ${{ secrets.DB_PASSWORD }}
DB_NAME: ${{ secrets.DB_DATABASE_PROD }}
REDIS_URL: 'redis://localhost:6379'
# Explicitly use database 0 for production (test uses database 1)
REDIS_URL: 'redis://localhost:6379/0'
REDIS_PASSWORD: ${{ secrets.REDIS_PASSWORD_PROD }}
FRONTEND_URL: 'https://flyer-crawler.projectium.com'
JWT_SECRET: ${{ secrets.JWT_SECRET }}

View File

@@ -0,0 +1,167 @@
# .gitea/workflows/manual-redis-flush-prod.yml
#
# DANGER: This workflow is DESTRUCTIVE and intended for manual execution only.
# It will completely FLUSH the PRODUCTION Redis database (db 0).
# This will clear all BullMQ queues, sessions, caches, and any other Redis data.
#
name: Manual - Flush Production Redis
on:
workflow_dispatch:
inputs:
confirmation:
description: 'DANGER: This will FLUSH production Redis. Type "flush-production-redis" to confirm.'
required: true
default: 'do-not-run'
flush_type:
description: 'What to flush?'
required: true
type: choice
options:
- 'queues-only'
- 'entire-database'
default: 'queues-only'
jobs:
flush-redis:
runs-on: projectium.com # This job runs on your self-hosted Gitea runner.
env:
REDIS_PASSWORD: ${{ secrets.REDIS_PASSWORD_PROD }}
steps:
- name: Checkout Code
uses: actions/checkout@v3
- name: Setup Node.js
uses: actions/setup-node@v3
with:
node-version: '20'
cache: 'npm'
cache-dependency-path: '**/package-lock.json'
- name: Install Dependencies
run: npm ci
- name: Validate Secrets
run: |
if [ -z "$REDIS_PASSWORD" ]; then
echo "ERROR: REDIS_PASSWORD_PROD secret is not set in Gitea repository settings."
exit 1
fi
echo "✅ Redis password secret is present."
- name: Verify Confirmation Phrase
run: |
if [ "${{ gitea.event.inputs.confirmation }}" != "flush-production-redis" ]; then
echo "ERROR: Confirmation phrase did not match. Aborting Redis flush."
exit 1
fi
echo "✅ Confirmation accepted. Proceeding with Redis flush."
- name: Show Current Redis State
run: |
echo "--- Current Redis Database 0 (Production) State ---"
redis-cli -a "$REDIS_PASSWORD" -n 0 INFO keyspace 2>/dev/null || echo "Could not get keyspace info"
echo ""
echo "--- Key Count ---"
KEY_COUNT=$(redis-cli -a "$REDIS_PASSWORD" -n 0 DBSIZE 2>/dev/null | grep -oE '[0-9]+' || echo "unknown")
echo "Production Redis (db 0) key count: $KEY_COUNT"
echo ""
echo "--- BullMQ Queue Keys ---"
redis-cli -a "$REDIS_PASSWORD" -n 0 KEYS "bull:*" 2>/dev/null | head -20 || echo "No BullMQ keys found"
- name: 🚨 FINAL WARNING & PAUSE 🚨
run: |
echo "*********************************************************************"
echo "WARNING: YOU ARE ABOUT TO FLUSH PRODUCTION REDIS DATA."
echo "Flush type: ${{ gitea.event.inputs.flush_type }}"
echo ""
if [ "${{ gitea.event.inputs.flush_type }}" = "entire-database" ]; then
echo "This will DELETE ALL Redis data including sessions, caches, and queues!"
else
echo "This will DELETE ALL BullMQ queue data (pending jobs, failed jobs, etc.)"
fi
echo ""
echo "This action is IRREVERSIBLE. Press Ctrl+C in the runner terminal NOW to cancel."
echo "Sleeping for 10 seconds..."
echo "*********************************************************************"
sleep 10
- name: Flush BullMQ Queues Only
if: ${{ gitea.event.inputs.flush_type == 'queues-only' }}
env:
REDIS_URL: 'redis://localhost:6379/0'
run: |
echo "--- Obliterating BullMQ queues using Node.js ---"
node -e "
const { Queue } = require('bullmq');
const IORedis = require('ioredis');
const connection = new IORedis(process.env.REDIS_URL, {
maxRetriesPerRequest: null,
password: process.env.REDIS_PASSWORD,
});
const queueNames = [
'flyer-processing',
'email-sending',
'analytics-reporting',
'weekly-analytics-reporting',
'file-cleanup',
'token-cleanup'
];
(async () => {
for (const name of queueNames) {
try {
const queue = new Queue(name, { connection });
const counts = await queue.getJobCounts();
console.log('Queue \"' + name + '\" before obliterate:', JSON.stringify(counts));
await queue.obliterate({ force: true });
console.log('✅ Obliterated queue: ' + name);
await queue.close();
} catch (err) {
console.error('⚠️ Failed to obliterate queue ' + name + ':', err.message);
}
}
await connection.quit();
console.log('✅ All BullMQ queues obliterated.');
})();
"
- name: Flush Entire Redis Database
if: ${{ gitea.event.inputs.flush_type == 'entire-database' }}
run: |
echo "--- Flushing entire Redis database 0 (production) ---"
redis-cli -a "$REDIS_PASSWORD" -n 0 FLUSHDB 2>/dev/null && echo "✅ Redis database 0 flushed successfully." || echo "❌ Redis flush failed"
- name: Verify Flush Results
run: |
echo "--- Redis Database 0 (Production) State After Flush ---"
KEY_COUNT=$(redis-cli -a "$REDIS_PASSWORD" -n 0 DBSIZE 2>/dev/null | grep -oE '[0-9]+' || echo "unknown")
echo "Production Redis (db 0) key count after flush: $KEY_COUNT"
echo ""
echo "--- Remaining BullMQ Queue Keys ---"
BULL_KEYS=$(redis-cli -a "$REDIS_PASSWORD" -n 0 KEYS "bull:*" 2>/dev/null | wc -l || echo "0")
echo "BullMQ key count: $BULL_KEYS"
if [ "${{ gitea.event.inputs.flush_type }}" = "queues-only" ] && [ "$BULL_KEYS" -gt 0 ]; then
echo "⚠️ Warning: Some BullMQ keys may still exist. This can happen if new jobs were added during the flush."
fi
- name: Summary
run: |
echo ""
echo "=========================================="
echo "PRODUCTION REDIS FLUSH COMPLETE"
echo "=========================================="
echo "Flush type: ${{ gitea.event.inputs.flush_type }}"
echo "Timestamp: $(date -u '+%Y-%m-%d %H:%M:%S UTC')"
echo ""
echo "NOTE: If you flushed queues, any pending jobs (flyer processing,"
echo "emails, analytics, etc.) have been permanently deleted."
echo ""
echo "The production workers will automatically start processing"
echo "new jobs as they are added to the queues."
echo "=========================================="

15
.gitignore vendored
View File

@@ -11,6 +11,18 @@ node_modules
dist
dist-ssr
*.local
.env
*.tsbuildinfo
# Test coverage
coverage
.nyc_output
.coverage
# Test artifacts - flyer-images/ is a runtime directory
# Test fixtures are stored in src/tests/assets/ instead
flyer-images/
test-output.txt
# Editor directories and files
.vscode/*
@@ -22,3 +34,6 @@ dist-ssr
*.njsproj
*.sln
*.sw?
Thumbs.db
.claude
nul

1
.husky/pre-commit Normal file
View File

@@ -0,0 +1 @@
npx lint-staged

4
.lintstagedrc.json Normal file
View File

@@ -0,0 +1,4 @@
{
"*.{js,jsx,ts,tsx}": ["eslint --fix", "prettier --write"],
"*.{json,md,css,html,yml,yaml}": ["prettier --write"]
}

41
.prettierignore Normal file
View File

@@ -0,0 +1,41 @@
# Dependencies
node_modules/
# Build output
dist/
build/
.cache/
# Coverage reports
coverage/
.coverage/
# IDE and editor configs
.idea/
.vscode/
*.swp
*.swo
# Logs
*.log
logs/
# Environment files (may contain secrets)
.env*
!.env.example
# Lock files (managed by package managers)
package-lock.json
pnpm-lock.yaml
yarn.lock
# Generated files
*.min.js
*.min.css
# Git directory
.git/
.gitea/
# Test artifacts
__snapshots__/

110
AUTHENTICATION.md Normal file
View File

@@ -0,0 +1,110 @@
# Authentication Setup
Flyer Crawler supports OAuth authentication via Google and GitHub. This guide walks through configuring both providers.
---
## Google OAuth
### Step 1: Create OAuth Credentials
1. Go to the [Google Cloud Console](https://console.cloud.google.com/)
2. Create a new project (or select an existing one)
3. Navigate to **APIs & Services > Credentials**
4. Click **Create Credentials > OAuth client ID**
5. Select **Web application** as the application type
### Step 2: Configure Authorized Redirect URIs
Add the callback URL where Google will redirect users after authentication:
| Environment | Redirect URI |
| ----------- | -------------------------------------------------- |
| Development | `http://localhost:3001/api/auth/google/callback` |
| Production | `https://your-domain.com/api/auth/google/callback` |
### Step 3: Save Credentials
After clicking **Create**, you'll receive:
- **Client ID**
- **Client Secret**
Store these securely as environment variables:
- `GOOGLE_CLIENT_ID`
- `GOOGLE_CLIENT_SECRET`
---
## GitHub OAuth
### Step 1: Create OAuth App
1. Go to your [GitHub Developer Settings](https://github.com/settings/developers)
2. Navigate to **OAuth Apps**
3. Click **New OAuth App**
### Step 2: Fill in Application Details
| Field | Value |
| -------------------------- | ---------------------------------------------------- |
| Application name | Flyer Crawler (or your preferred name) |
| Homepage URL | `http://localhost:5173` (dev) or your production URL |
| Authorization callback URL | `http://localhost:3001/api/auth/github/callback` |
### Step 3: Save GitHub Credentials
After clicking **Register application**, you'll receive:
- **Client ID**
- **Client Secret**
Store these securely as environment variables:
- `GITHUB_CLIENT_ID`
- `GITHUB_CLIENT_SECRET`
---
## Environment Variables Summary
| Variable | Description |
| ---------------------- | ---------------------------------------- |
| `GOOGLE_CLIENT_ID` | Google OAuth client ID |
| `GOOGLE_CLIENT_SECRET` | Google OAuth client secret |
| `GITHUB_CLIENT_ID` | GitHub OAuth client ID |
| `GITHUB_CLIENT_SECRET` | GitHub OAuth client secret |
| `JWT_SECRET` | Secret for signing authentication tokens |
---
## Production Considerations
When deploying to production:
1. **Update redirect URIs** in both Google Cloud Console and GitHub OAuth settings to use your production domain
2. **Use HTTPS** for all callback URLs in production
3. **Store secrets securely** using your CI/CD platform's secrets management (e.g., Gitea repository secrets)
---
## Troubleshooting
### "redirect_uri_mismatch" Error
The callback URL in your OAuth provider settings doesn't match what the application is sending. Verify:
- The URL is exactly correct (no trailing slashes, correct port)
- You're using the right environment (dev vs production URLs)
### "invalid_client" Error
The Client ID or Client Secret is incorrect. Double-check your environment variables.
---
## Related Documentation
- [Installation Guide](INSTALL.md) - Local development setup
- [Deployment Guide](DEPLOYMENT.md) - Production deployment

71
CLAUDE.md Normal file
View File

@@ -0,0 +1,71 @@
# Claude Code Project Instructions
## Platform Requirement: Linux Only
**CRITICAL**: This application is designed to run **exclusively on Linux**. See [ADR-014](docs/adr/0014-containerization-and-deployment-strategy.md) for full details.
### Test Execution Rules
1. **ALL tests MUST be executed on Linux** - either in the Dev Container or on a Linux host
2. **NEVER run tests directly on Windows** - test results from Windows are unreliable
3. **Always use the Dev Container for testing** when developing on Windows
### How to Run Tests Correctly
```bash
# If on Windows, first open VS Code and "Reopen in Container"
# Then run tests inside the container:
npm test # Run all unit tests
npm run test:unit # Run unit tests only
npm run test:integration # Run integration tests (requires DB/Redis)
```
### Running Tests via Podman (from Windows host)
The command to run unit tests in the Linux container via podman:
```bash
podman exec -it flyer-crawler-dev npm run test:unit
```
The command to run integration tests in the Linux container via podman:
```bash
podman exec -it flyer-crawler-dev npm run test:integration
```
For running specific test files:
```bash
podman exec -it flyer-crawler-dev npm test -- --run src/hooks/useAuth.test.tsx
```
### Why Linux Only?
- Path separators: Code uses POSIX-style paths (`/`) which may break on Windows
- Shell scripts in `scripts/` directory are Linux-only
- External dependencies like `pdftocairo` assume Linux installation paths
- Unix-style file permissions are assumed throughout
### Test Result Interpretation
- Tests that **pass on Windows but fail on Linux** = **BROKEN tests** (must be fixed)
- Tests that **fail on Windows but pass on Linux** = **PASSING tests** (acceptable)
## Development Workflow
1. Open project in VS Code
2. Use "Reopen in Container" (Dev Containers extension required)
3. Wait for container initialization to complete
4. Run `npm test` to verify environment is working
5. Make changes and run tests inside the container
## Quick Reference
| Command | Description |
| -------------------------- | ---------------------------- |
| `npm test` | Run all unit tests |
| `npm run test:unit` | Run unit tests only |
| `npm run test:integration` | Run integration tests |
| `npm run dev:container` | Start dev server (container) |
| `npm run build` | Build for production |

188
DATABASE.md Normal file
View File

@@ -0,0 +1,188 @@
# Database Setup
Flyer Crawler uses PostgreSQL with several extensions for full-text search, geographic data, and UUID generation.
---
## Required Extensions
| Extension | Purpose |
| ----------- | ------------------------------------------- |
| `postgis` | Geographic/spatial data for store locations |
| `pg_trgm` | Trigram matching for fuzzy text search |
| `uuid-ossp` | UUID generation for primary keys |
---
## Production Database Setup
### Step 1: Install PostgreSQL
```bash
sudo apt update
sudo apt install postgresql postgresql-contrib
```
### Step 2: Create Database and User
Switch to the postgres system user:
```bash
sudo -u postgres psql
```
Run the following SQL commands (replace `'a_very_strong_password'` with a secure password):
```sql
-- Create a new role for your application
CREATE ROLE flyer_crawler_user WITH LOGIN PASSWORD 'a_very_strong_password';
-- Create the production database
CREATE DATABASE "flyer-crawler-prod" WITH OWNER = flyer_crawler_user;
-- Connect to the new database
\c "flyer-crawler-prod"
-- Install required extensions (must be done as superuser)
CREATE EXTENSION IF NOT EXISTS postgis;
CREATE EXTENSION IF NOT EXISTS pg_trgm;
CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
-- Exit
\q
```
### Step 3: Apply the Schema
Navigate to your project directory and run:
```bash
psql -U flyer_crawler_user -d "flyer-crawler-prod" -f sql/master_schema_rollup.sql
```
This creates all tables, functions, triggers, and seeds essential data (categories, master items).
### Step 4: Seed the Admin Account
Set the required environment variables and run the seed script:
```bash
export DB_USER=flyer_crawler_user
export DB_PASSWORD=your_password
export DB_NAME="flyer-crawler-prod"
export DB_HOST=localhost
npx tsx src/db/seed_admin_account.ts
```
---
## Test Database Setup
The test database is used by CI/CD pipelines and local test runs.
### Step 1: Create the Test Database
```bash
sudo -u postgres psql
```
```sql
-- Create the test database
CREATE DATABASE "flyer-crawler-test" WITH OWNER = flyer_crawler_user;
-- Connect to the test database
\c "flyer-crawler-test"
-- Install required extensions
CREATE EXTENSION IF NOT EXISTS postgis;
CREATE EXTENSION IF NOT EXISTS pg_trgm;
CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
-- Grant schema ownership (required for test runner to reset schema)
ALTER SCHEMA public OWNER TO flyer_crawler_user;
-- Exit
\q
```
### Step 2: Configure CI/CD Secrets
Ensure these secrets are set in your Gitea repository settings:
| Secret | Description |
| ------------- | ------------------------------------------ |
| `DB_HOST` | Database hostname (e.g., `localhost`) |
| `DB_PORT` | Database port (e.g., `5432`) |
| `DB_USER` | Database user (e.g., `flyer_crawler_user`) |
| `DB_PASSWORD` | Database password |
---
## How the Test Pipeline Works
The CI pipeline uses a permanent test database that gets reset on each test run:
1. **Setup**: The vitest global setup script connects to `flyer-crawler-test`
2. **Schema Reset**: Executes `sql/drop_tables.sql` (`DROP SCHEMA public CASCADE`)
3. **Schema Application**: Runs `sql/master_schema_rollup.sql` to build a fresh schema
4. **Test Execution**: Tests run against the clean database
This approach is faster than creating/destroying databases and doesn't require sudo access.
---
## Connecting to Production Database
```bash
psql -h localhost -U flyer_crawler_user -d "flyer-crawler-prod" -W
```
---
## Checking PostGIS Version
```sql
SELECT version();
SELECT PostGIS_Full_Version();
```
Example output:
```
PostgreSQL 14.19 (Ubuntu 14.19-0ubuntu0.22.04.1)
POSTGIS="3.2.0 c3e3cc0" GEOS="3.10.2-CAPI-1.16.0" PROJ="8.2.1"
```
---
## Schema Files
| File | Purpose |
| ------------------------------ | --------------------------------------------------------- |
| `sql/master_schema_rollup.sql` | Complete schema with all tables, functions, and seed data |
| `sql/drop_tables.sql` | Drops entire schema (used by test runner) |
| `sql/schema.sql.txt` | Legacy schema file (reference only) |
---
## Backup and Restore
### Create a Backup
```bash
pg_dump -U flyer_crawler_user -d "flyer-crawler-prod" -F c -f backup.dump
```
### Restore from Backup
```bash
pg_restore -U flyer_crawler_user -d "flyer-crawler-prod" -c backup.dump
```
---
## Related Documentation
- [Installation Guide](INSTALL.md) - Local development setup
- [Deployment Guide](DEPLOYMENT.md) - Production deployment

211
DEPLOYMENT.md Normal file
View File

@@ -0,0 +1,211 @@
# Deployment Guide
This guide covers deploying Flyer Crawler to a production server.
## Prerequisites
- Ubuntu server (22.04 LTS recommended)
- PostgreSQL 14+ with PostGIS extension
- Redis
- Node.js 20.x
- NGINX (reverse proxy)
- PM2 (process manager)
---
## Server Setup
### Install Node.js
```bash
curl -sL https://deb.nodesource.com/setup_20.x | sudo bash -
sudo apt-get install -y nodejs
```
### Install PM2
```bash
sudo npm install -g pm2
```
---
## Application Deployment
### Clone and Install
```bash
git clone <repository-url>
cd flyer-crawler.projectium.com
npm install
```
### Build for Production
```bash
npm run build
```
### Start with PM2
```bash
npm run start:prod
```
This starts three PM2 processes:
- `flyer-crawler-api` - Main API server
- `flyer-crawler-worker` - Background job worker
- `flyer-crawler-analytics-worker` - Analytics processing worker
---
## Environment Variables (Gitea Secrets)
For deployments using Gitea CI/CD workflows, configure these as **repository secrets**:
| Secret | Description |
| --------------------------- | ------------------------------------------- |
| `DB_HOST` | PostgreSQL server hostname |
| `DB_USER` | PostgreSQL username |
| `DB_PASSWORD` | PostgreSQL password |
| `DB_DATABASE_PROD` | Production database name |
| `REDIS_PASSWORD_PROD` | Production Redis password |
| `REDIS_PASSWORD_TEST` | Test Redis password |
| `JWT_SECRET` | Long, random string for signing auth tokens |
| `VITE_GOOGLE_GENAI_API_KEY` | Google Gemini API key |
| `GOOGLE_MAPS_API_KEY` | Google Maps Geocoding API key |
---
## NGINX Configuration
### Reverse Proxy Setup
Create a site configuration at `/etc/nginx/sites-available/flyer-crawler.projectium.com`:
```nginx
server {
listen 80;
server_name flyer-crawler.projectium.com;
location / {
proxy_pass http://localhost:5173;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}
location /api {
proxy_pass http://localhost:3001;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}
}
```
Enable the site:
```bash
sudo ln -s /etc/nginx/sites-available/flyer-crawler.projectium.com /etc/nginx/sites-enabled/
sudo nginx -t
sudo systemctl reload nginx
```
### MIME Types Fix for .mjs Files
If JavaScript modules (`.mjs` files) aren't loading correctly, add the proper MIME type.
**Option 1**: Edit the site configuration file directly:
```nginx
# Add inside the server block
types {
application/javascript js mjs;
}
```
**Option 2**: Edit `/etc/nginx/mime.types` globally:
```
# Change this line:
application/javascript js;
# To:
application/javascript js mjs;
```
After changes:
```bash
sudo nginx -t
sudo systemctl reload nginx
```
---
## PM2 Log Management
Install and configure pm2-logrotate to manage log files:
```bash
pm2 install pm2-logrotate
pm2 set pm2-logrotate:max_size 10M
pm2 set pm2-logrotate:retain 14
pm2 set pm2-logrotate:compress false
pm2 set pm2-logrotate:dateFormat YYYY-MM-DD_HH-mm-ss
```
---
## Rate Limiting
The application respects the Gemini AI service's rate limits. You can adjust the `GEMINI_RPM` (requests per minute) environment variable in production as needed without changing the code.
---
## CI/CD Pipeline
The project includes Gitea workflows at `.gitea/workflows/deploy.yml` that:
1. Run tests against a test database
2. Build the application
3. Deploy to production on successful builds
The workflow automatically:
- Sets up the test database schema before tests
- Tears down test data after tests complete
- Deploys to the production server
---
## Monitoring
### Check PM2 Status
```bash
pm2 status
pm2 logs
pm2 logs flyer-crawler-api --lines 100
```
### Restart Services
```bash
pm2 restart all
pm2 restart flyer-crawler-api
```
---
## Related Documentation
- [Database Setup](DATABASE.md) - PostgreSQL and PostGIS configuration
- [Authentication Setup](AUTHENTICATION.md) - OAuth provider configuration
- [Installation Guide](INSTALL.md) - Local development setup

View File

@@ -1,31 +1,60 @@
# Use Ubuntu 22.04 (LTS) as the base image to match production
# Dockerfile.dev
# ============================================================================
# DEVELOPMENT DOCKERFILE
# ============================================================================
# This Dockerfile creates a development environment that matches production
# as closely as possible while providing the tools needed for development.
#
# Base: Ubuntu 22.04 (LTS) - matches production server
# Node: v20.x (LTS) - matches production
# Includes: PostgreSQL client, Redis CLI, build tools
# ============================================================================
FROM ubuntu:22.04
# Set environment variables to non-interactive to avoid prompts during installation
ENV DEBIAN_FRONTEND=noninteractive
# Update package lists and install essential tools
# - curl: for downloading Node.js setup script
# ============================================================================
# Install System Dependencies
# ============================================================================
# - curl: for downloading Node.js setup script and health checks
# - git: for version control operations
# - build-essential: for compiling native Node.js modules (node-gyp)
# - python3: required by some Node.js build tools
# - postgresql-client: for psql CLI (database initialization)
# - redis-tools: for redis-cli (health checks)
RUN apt-get update && apt-get install -y \
curl \
git \
build-essential \
python3 \
postgresql-client \
redis-tools \
&& rm -rf /var/lib/apt/lists/*
# Install Node.js 20.x (LTS) from NodeSource
# ============================================================================
# Install Node.js 20.x (LTS)
# ============================================================================
RUN curl -fsSL https://deb.nodesource.com/setup_20.x | bash - \
&& apt-get install -y nodejs
# Set the working directory inside the container
# ============================================================================
# Set Working Directory
# ============================================================================
WORKDIR /app
# Set default environment variables for development
# ============================================================================
# Environment Configuration
# ============================================================================
# Default environment variables for development
ENV NODE_ENV=development
# Increase Node.js memory limit for large builds
ENV NODE_OPTIONS='--max-old-space-size=8192'
# Default command keeps the container running so you can attach to it
CMD ["bash"]
# ============================================================================
# Default Command
# ============================================================================
# Keep container running so VS Code can attach.
# Actual commands (npm run dev, etc.) are run via devcontainer.json.
CMD ["bash"]

167
INSTALL.md Normal file
View File

@@ -0,0 +1,167 @@
# Installation Guide
This guide covers setting up a local development environment for Flyer Crawler.
## Prerequisites
- Node.js 20.x or later
- Access to a PostgreSQL database (local or remote)
- Redis instance (for session management)
- Google Gemini API key
- Google Maps API key (for geocoding)
## Quick Start
If you already have PostgreSQL and Redis configured:
```bash
# Install dependencies
npm install
# Run in development mode
npm run dev
```
---
## Development Environment with Podman (Recommended for Windows)
This approach uses Podman with an Ubuntu container for a consistent development environment.
### Step 1: Install Prerequisites on Windows
1. **Install WSL 2**: Podman on Windows relies on the Windows Subsystem for Linux.
```powershell
wsl --install
```
Run this in an administrator PowerShell.
2. **Install Podman Desktop**: Download and install [Podman Desktop for Windows](https://podman-desktop.io/).
### Step 2: Set Up Podman
1. **Initialize Podman**: Launch Podman Desktop. It will automatically set up its WSL 2 machine.
2. **Start Podman**: Ensure the Podman machine is running from the Podman Desktop interface.
### Step 3: Set Up the Ubuntu Container
1. **Pull Ubuntu Image**:
```bash
podman pull ubuntu:latest
```
2. **Create a Podman Volume** (persists node_modules between container restarts):
```bash
podman volume create node_modules_cache
```
3. **Run the Ubuntu Container**:
Open a terminal in your project's root directory and run:
```bash
podman run -it -p 3001:3001 -p 5173:5173 --name flyer-dev \
-v "$(pwd):/app" \
-v "node_modules_cache:/app/node_modules" \
ubuntu:latest
```
| Flag | Purpose |
| ------------------------------------------- | ------------------------------------------------ |
| `-p 3001:3001` | Forwards the backend server port |
| `-p 5173:5173` | Forwards the Vite frontend server port |
| `--name flyer-dev` | Names the container for easy reference |
| `-v "...:/app"` | Mounts your project directory into the container |
| `-v "node_modules_cache:/app/node_modules"` | Mounts the named volume for node_modules |
### Step 4: Configure the Ubuntu Environment
You are now inside the Ubuntu container's shell.
1. **Update Package Lists**:
```bash
apt-get update
```
2. **Install Dependencies**:
```bash
apt-get install -y curl git
curl -sL https://deb.nodesource.com/setup_20.x | bash -
apt-get install -y nodejs
```
3. **Navigate to Project Directory**:
```bash
cd /app
```
4. **Install Project Dependencies**:
```bash
npm install
```
### Step 5: Run the Development Server
```bash
npm run dev
```
### Step 6: Access the Application
- **Frontend**: http://localhost:5173
- **Backend API**: http://localhost:3001
### Managing the Container
| Action | Command |
| --------------------- | -------------------------------- |
| Stop the container | Press `Ctrl+C`, then type `exit` |
| Restart the container | `podman start -a -i flyer-dev` |
| Remove the container | `podman rm flyer-dev` |
---
## Environment Variables
This project is configured to run in a CI/CD environment and does not use `.env` files. All configuration must be provided as environment variables.
For local development, you can export these in your shell or use your IDE's environment configuration:
| Variable | Description |
| --------------------------- | ------------------------------------- |
| `DB_HOST` | PostgreSQL server hostname |
| `DB_USER` | PostgreSQL username |
| `DB_PASSWORD` | PostgreSQL password |
| `DB_DATABASE_PROD` | Production database name |
| `JWT_SECRET` | Secret string for signing auth tokens |
| `VITE_GOOGLE_GENAI_API_KEY` | Google Gemini API key |
| `GOOGLE_MAPS_API_KEY` | Google Maps Geocoding API key |
| `REDIS_PASSWORD_PROD` | Production Redis password |
| `REDIS_PASSWORD_TEST` | Test Redis password |
---
## Seeding Development Users
To create initial test accounts (`admin@example.com` and `user@example.com`):
```bash
npm run seed
```
After running, you may need to restart your IDE's TypeScript server to pick up any generated types.
---
## Next Steps
- [Database Setup](DATABASE.md) - Set up PostgreSQL with required extensions
- [Authentication Setup](AUTHENTICATION.md) - Configure OAuth providers
- [Deployment Guide](DEPLOYMENT.md) - Deploy to production

451
README.md
View File

@@ -1,424 +1,91 @@
# Flyer Crawler - Grocery AI Analyzer
Flyer Crawler is a web application that uses the Google Gemini AI to extract, analyze, and manage data from grocery store flyers. Users can upload flyer images or PDFs, and the application will automatically identify items, prices, and sale dates, storing the structured data in a PostgreSQL database for historical analysis, price tracking, and personalized deal alerts.
Flyer Crawler is a web application that uses Google Gemini AI to extract, analyze, and manage data from grocery store flyers. Users can upload flyer images or PDFs, and the application automatically identifies items, prices, and sale dates, storing structured data in a PostgreSQL database for historical analysis, price tracking, and personalized deal alerts.
We are working on an app to help people save money, by finding good deals that are only advertized in store flyers/ads. So, the primary purpose of the site is to make uploading flyers as easy as possible and as accurate as possible, and to store peoples needs, so sales can be matched to needs.
**Our mission**: Help people save money by finding good deals that are only advertised in store flyers. The app makes uploading flyers as easy and accurate as possible, and matches sales to users' needs.
---
## Features
- **AI-Powered Data Extraction**: Upload PNG, JPG, or PDF flyers to automatically extract store names, sale dates, and a detailed list of items with prices and quantities.
- **Bulk Import**: Process multiple flyers at once with a summary report of successes, skips (duplicates), and errors.
- **Database Integration**: All extracted data is saved to a PostgreSQL database, enabling long-term persistence and analysis.
- **Personalized Watchlist**: Authenticated users can create a "watchlist" of specific grocery items they want to track.
- **Active Deal Alerts**: The app highlights current sales on your watched items from all valid flyers in the database.
- **Price History Charts**: Visualize the price trends of your watched items over time.
- **Shopping List Management**: Users can create multiple shopping lists, add items from flyers or their watchlist, and track purchased items.
- **User Authentication & Management**: Secure user sign-up, login, and profile management, including a secure account deletion process.
- **Dynamic UI**: A responsive interface with dark mode and a choice between metric/imperial unit systems.
- **AI-Powered Data Extraction**: Upload PNG, JPG, or PDF flyers to automatically extract store names, sale dates, and detailed item lists with prices and quantities
- **Bulk Import**: Process multiple flyers at once with summary reports of successes, skips (duplicates), and errors
- **Personalized Watchlist**: Create a watchlist of specific grocery items you want to track
- **Active Deal Alerts**: See current sales on your watched items from all valid flyers
- **Price History Charts**: Visualize price trends of watched items over time
- **Shopping List Management**: Create multiple shopping lists, add items from flyers or your watchlist, and track purchased items
- **User Authentication**: Secure sign-up, login, profile management, and account deletion
- **Dynamic UI**: Responsive interface with dark mode and metric/imperial unit systems
---
## Tech Stack
- **Frontend**: React, TypeScript, Tailwind CSS
- **AI**: Google Gemini API (`@google/genai`)
- **Backend**: Node.js with Express
- **Database**: PostgreSQL
- **Authentication**: Passport.js
- **UI Components**: Recharts for charts
| Layer | Technology |
| -------------- | ----------------------------------- |
| Frontend | React, TypeScript, Tailwind CSS |
| AI | Google Gemini API (`@google/genai`) |
| Backend | Node.js, Express |
| Database | PostgreSQL with PostGIS |
| Authentication | Passport.js (Google, GitHub OAuth) |
| Charts | Recharts |
---
## Required Secrets & Configuration
This project is configured to run in a CI/CD environment and does not use `.env` files. All configuration and secrets must be provided as environment variables. For deployments using the included Gitea workflows, these must be configured as **repository secrets** in your Gitea instance.
- **`DB_HOST`, `DB_USER`, `DB_PASSWORD`**: Credentials for your PostgreSQL server. The port is assumed to be `5432`.
- **`DB_DATABASE_PROD`**: The name of your production database.
- **`REDIS_PASSWORD_PROD`**: The password for your production Redis instance.
- **`REDIS_PASSWORD_TEST`**: The password for your test Redis instance.
- **`JWT_SECRET`**: A long, random, and secret string for signing authentication tokens.
- **`VITE_GOOGLE_GENAI_API_KEY`**: Your Google Gemini API key.
- **`GOOGLE_MAPS_API_KEY`**: Your Google Maps Geocoding API key.
## Setup and Installation
### Step 1: Set Up PostgreSQL Database
1. **Set up a PostgreSQL database instance.**
2. **Run the Database Schema**:
- Connect to your database using a tool like `psql` or DBeaver.
- Open `sql/schema.sql.txt`, copy its entire contents, and execute it against your database.
- This will create all necessary tables, functions, and relationships.
### Step 2: Install Dependencies and Run the Application
1. **Install Dependencies**:
```bash
npm install
```
2. **Run the Application**:
```bash
npm run start:prod
```
### Step 3: Seed Development Users (Optional)
To create the initial `admin@example.com` and `user@example.com` accounts, you can run the seed script:
## Quick Start
```bash
npm run seed
# Install dependencies
npm install
# Run in development mode
npm run dev
```
After running, you may need to restart your IDE's TypeScript server to pick up the changes.
## NGINX mime types issue
sudo nano /etc/nginx/mime.types
change
application/javascript js;
TO
application/javascript js mjs;
RESTART NGINX
sudo nginx -t
sudo systemctl reload nginx
actually the proper change was to do this in the /etc/nginx/sites-available/flyer-crawler.projectium.com file
## for OAuth
1. Get Google OAuth Credentials
This is a crucial step that you must do outside the codebase:
Go to the Google Cloud Console.
Create a new project (or select an existing one).
In the navigation menu, go to APIs & Services > Credentials.
Click Create Credentials > OAuth client ID.
Select Web application as the application type.
Under Authorized redirect URIs, click ADD URI and enter the URL where Google will redirect users back to your server. For local development, this will be: http://localhost:3001/api/auth/google/callback.
Click Create. You will be given a Client ID and a Client Secret.
2. Get GitHub OAuth Credentials
You'll need to obtain a Client ID and Client Secret from GitHub:
Go to your GitHub profile settings.
Navigate to Developer settings > OAuth Apps.
Click New OAuth App.
Fill in the required fields:
Application name: A descriptive name for your app (e.g., "Flyer Crawler").
Homepage URL: The base URL of your application (e.g., http://localhost:5173 for local development).
Authorization callback URL: This is where GitHub will redirect users after they authorize your app. For local development, this will be: <http://localhost:3001/api/auth/github/callback>.
Click Register application.
You will be given a Client ID and a Client Secret.
## connect to postgres on projectium.com
psql -h localhost -U flyer_crawler_user -d "flyer-crawler-prod" -W
## postgis
flyer-crawler-prod=> SELECT version();
version
See [INSTALL.md](INSTALL.md) for detailed setup instructions.
---
PostgreSQL 14.19 (Ubuntu 14.19-0ubuntu0.22.04.1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 11.4.0-1ubuntu1~22.04.2) 11.4.0, 64-bit
(1 row)
## Documentation
flyer-crawler-prod=> SELECT PostGIS_Full_Version();
postgis_full_version
| Document | Description |
| -------------------------------------- | ---------------------------------------- |
| [INSTALL.md](INSTALL.md) | Local development setup with Podman |
| [DATABASE.md](DATABASE.md) | PostgreSQL setup, schema, and extensions |
| [AUTHENTICATION.md](AUTHENTICATION.md) | OAuth configuration (Google, GitHub) |
| [DEPLOYMENT.md](DEPLOYMENT.md) | Production server setup, NGINX, PM2 |
---
POSTGIS="3.2.0 c3e3cc0" [EXTENSION] PGSQL="140" GEOS="3.10.2-CAPI-1.16.0" PROJ="8.2.1" LIBXML="2.9.12" LIBJSON="0.15" LIBPROTOBUF="1.3.3" WAGYU="0.5.0 (Internal)"
(1 row)
## Environment Variables
## production postgres setup
This project uses environment variables for configuration (no `.env` files). Key variables:
Part 1: Production Database Setup
This database will be the live, persistent storage for your application.
| Variable | Description |
| ----------------------------------- | -------------------------------- |
| `DB_HOST`, `DB_USER`, `DB_PASSWORD` | PostgreSQL credentials |
| `DB_DATABASE_PROD` | Production database name |
| `JWT_SECRET` | Authentication token signing key |
| `VITE_GOOGLE_GENAI_API_KEY` | Google Gemini API key |
| `GOOGLE_MAPS_API_KEY` | Google Maps Geocoding API key |
| `REDIS_PASSWORD_PROD` | Redis password |
Step 1: Install PostgreSQL (if not already installed)
First, ensure PostgreSQL is installed on your server.
See [INSTALL.md](INSTALL.md) for the complete list.
bash
sudo apt update
sudo apt install postgresql postgresql-contrib
Step 2: Create the Production Database and User
It's best practice to create a dedicated, non-superuser role for your application to connect with.
---
Switch to the postgres system user to get superuser access to the database.
## Scripts
bash
sudo -u postgres psql
Inside the psql shell, run the following SQL commands. Remember to replace 'a_very_strong_password' with a secure password that you will manage with a secrets tool or in your .env file.
| Command | Description |
| -------------------- | -------------------------------- |
| `npm run dev` | Start development server |
| `npm run build` | Build for production |
| `npm run start:prod` | Start production server with PM2 |
| `npm run test` | Run test suite |
| `npm run seed` | Seed development user accounts |
sql
-- Create a new role (user) for your application
CREATE ROLE flyer_crawler_user WITH LOGIN PASSWORD 'a_very_strong_password';
---
-- Create the production database and assign ownership to the new user
CREATE DATABASE "flyer-crawler-prod" WITH OWNER = flyer_crawler_user;
## License
-- Connect to the new database to install extensions within it.
\c "flyer-crawler-prod"
-- Install the required extensions as a superuser. This only needs to be done once.
CREATE EXTENSION IF NOT EXISTS postgis;
CREATE EXTENSION IF NOT EXISTS pg_trgm;
CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
-- Exit the psql shell
Step 3: Apply the Master Schema
Now, you'll populate your new database with all the tables, functions, and initial data. Your master_schema_rollup.sql file is perfect for this.
Navigate to your project's root directory on the server.
Run the following command to execute the master schema script against your new production database. You will be prompted for the password you created in the previous step.
bash
psql -U flyer_crawler_user -d "flyer-crawler-prod" -f sql/master_schema_rollup.sql
This single command creates all tables, extensions (pg_trgm, postgis), functions, and triggers, and seeds essential data like categories and master items.
Step 4: Seed the Admin Account (If Needed)
Your application has a separate script to create the initial admin user. To run it, you must first set the required environment variables in your shell session.
bash
# Set variables for the current session
export DB_USER=flyer_crawler_user DB_PASSWORD=your_password DB_NAME="flyer-crawler-prod" ...
# Run the seeding script
npx tsx src/db/seed_admin_account.ts
Your production database is now ready!
Part 2: Test Database Setup (for CI/CD)
Your Gitea workflow (deploy.yml) already automates the creation and teardown of the test database during the pipeline run. The steps below are for understanding what the workflow does and for manual setup if you ever need to run tests outside the CI pipeline.
The process your CI pipeline follows is:
Setup (sql/test_setup.sql):
As the postgres superuser, it runs sql/test_setup.sql.
This creates a temporary role named test_runner.
It creates a separate database named "flyer-crawler-test" owned by test_runner.
Schema Application (src/tests/setup/global-setup.ts):
The test runner (vitest) executes the global-setup.ts file.
This script connects to the "flyer-crawler-test" database using the temporary credentials.
It then runs the same sql/master_schema_rollup.sql file, ensuring your test database has the exact same structure as production.
Test Execution:
Your tests run against this clean, isolated "flyer-crawler-test" database.
Teardown (sql/test_teardown.sql):
After tests complete (whether they pass or fail), the if: always() step in your workflow ensures that sql/test_teardown.sql is executed.
This script terminates any lingering connections to the test database, drops the "flyer-crawler-test" database completely, and drops the test_runner role.
Part 3: Test Database Setup (for CI/CD and Local Testing)
Your Gitea workflow and local test runner rely on a permanent test database. This database needs to be created once on your server. The test runner will automatically reset the schema inside it before every test run.
Step 1: Create the Test Database
On your server, switch to the postgres system user to get superuser access.
bash
sudo -u postgres psql
Inside the psql shell, create a new database. We will assign ownership to the same flyer_crawler_user that your application uses. This user needs to be the owner to have permission to drop and recreate the schema during testing.
sql
-- Create the test database and assign ownership to your existing application user
CREATE DATABASE "flyer-crawler-test" WITH OWNER = flyer_crawler_user;
-- Connect to the newly created test database
\c "flyer-crawler-test"
-- Install the required extensions as a superuser. This only needs to be done once.
CREATE EXTENSION IF NOT EXISTS postgis;
CREATE EXTENSION IF NOT EXISTS pg_trgm;
CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
-- Connect to the newly created test database
\c "flyer-crawler-test"
-- Grant ownership of the public schema within this database to your application user.
-- This is CRITICAL for allowing the test runner to drop and recreate the schema.
ALTER SCHEMA public OWNER TO flyer_crawler_user;
-- Exit the psql shell
\q
Step 2: Configure Gitea Secrets for Testing
Your CI pipeline needs to know how to connect to this test database. Ensure the following secrets are set in your Gitea repository settings:
DB_HOST: The hostname of your database server (e.g., localhost).
DB_PORT: The port for your database (e.g., 5432).
DB_USER: The user for the database (e.g., flyer_crawler_user).
DB_PASSWORD: The password for the database user.
The workflow file (.gitea/workflows/deploy.yml) is configured to use these secrets and will automatically connect to the "flyer-crawler-test" database when it runs the npm test command.
How the Test Workflow Works
The CI pipeline no longer uses sudo or creates/destroys the database on each run. Instead, the process is now:
Setup: The vitest global setup script (src/tests/setup/global-setup.ts) connects to the permanent "flyer-crawler-test" database.
Schema Reset: It executes sql/drop_tables.sql (which runs DROP SCHEMA public CASCADE) to completely wipe all tables, functions, and triggers.
Schema Application: It then immediately executes sql/master_schema_rollup.sql to build a fresh, clean schema and seed initial data.
Test Execution: Your tests run against this clean, isolated schema.
This approach is faster, more reliable, and removes the need for sudo access within the CI pipeline.
gitea-runner@projectium:~$ pm2 install pm2-logrotate
[PM2][Module] Installing NPM pm2-logrotate module
[PM2][Module] Calling [NPM] to install pm2-logrotate ...
added 161 packages in 5s
21 packages are looking for funding
run `npm fund` for details
npm notice
npm notice New patch version of npm available! 11.6.3 -> 11.6.4
npm notice Changelog: https://github.com/npm/cli/releases/tag/v11.6.4
npm notice To update run: npm install -g npm@11.6.4
npm notice
[PM2][Module] Module downloaded
[PM2][WARN] Applications pm2-logrotate not running, starting...
[PM2] App [pm2-logrotate] launched (1 instances)
Module: pm2-logrotate
$ pm2 set pm2-logrotate:max_size 10M
$ pm2 set pm2-logrotate:retain 30
$ pm2 set pm2-logrotate:compress false
$ pm2 set pm2-logrotate:dateFormat YYYY-MM-DD_HH-mm-ss
$ pm2 set pm2-logrotate:workerInterval 30
$ pm2 set pm2-logrotate:rotateInterval 0 0 \* \* _
$ pm2 set pm2-logrotate:rotateModule true
Modules configuration. Copy/Paste line to edit values.
[PM2][Module] Module successfully installed and launched
[PM2][Module] Checkout module options: `$ pm2 conf`
┌────┬───────────────────────────────────┬─────────────┬─────────┬─────────┬──────────┬────────┬──────┬───────────┬──────────┬──────────┬──────────┬──────────┐
│ id │ name │ namespace │ version │ mode │ pid │ uptime │ ↺ │ status │ cpu │ mem │ user │ watching │
├────┼───────────────────────────────────┼─────────────┼─────────┼─────────┼──────────┼────────┼──────┼───────────┼──────────┼──────────┼──────────┼──────────┤
│ 2 │ flyer-crawler-analytics-worker │ default │ 0.0.0 │ fork │ 3846981 │ 7m │ 5 │ online │ 0% │ 55.8mb │ git… │ disabled │
│ 11 │ flyer-crawler-api │ default │ 0.0.0 │ fork │ 3846987 │ 7m │ 0 │ online │ 0% │ 59.0mb │ git… │ disabled │
│ 12 │ flyer-crawler-worker │ default │ 0.0.0 │ fork │ 3846988 │ 7m │ 0 │ online │ 0% │ 54.2mb │ git… │ disabled │
└────┴───────────────────────────────────┴─────────────┴─────────┴─────────┴──────────┴────────┴──────┴───────────┴──────────┴──────────┴──────────┴──────────┘
Module
┌────┬──────────────────────────────┬───────────────┬──────────┬──────────┬──────┬──────────┬──────────┬──────────┐
│ id │ module │ version │ pid │ status │ ↺ │ cpu │ mem │ user │
├────┼──────────────────────────────┼───────────────┼──────────┼──────────┼──────┼──────────┼──────────┼──────────┤
│ 13 │ pm2-logrotate │ 3.0.0 │ 3848878 │ online │ 0 │ 0% │ 20.1mb │ git… │
└────┴──────────────────────────────┴───────────────┴──────────┴──────────┴──────┴──────────┴──────────┴──────────┘
gitea-runner@projectium:~$ pm2 set pm2-logrotate:max_size 10M
[PM2] Module pm2-logrotate restarted
[PM2] Setting changed
Module: pm2-logrotate
$ pm2 set pm2-logrotate:max_size 10M
$ pm2 set pm2-logrotate:retain 30
$ pm2 set pm2-logrotate:compress false
$ pm2 set pm2-logrotate:dateFormat YYYY-MM-DD_HH-mm-ss
$ pm2 set pm2-logrotate:workerInterval 30
$ pm2 set pm2-logrotate:rotateInterval 0 0 _ \* _
$ pm2 set pm2-logrotate:rotateModule true
gitea-runner@projectium:~$ pm2 set pm2-logrotate:retain 14
[PM2] Module pm2-logrotate restarted
[PM2] Setting changed
Module: pm2-logrotate
$ pm2 set pm2-logrotate:max_size 10M
$ pm2 set pm2-logrotate:retain 14
$ pm2 set pm2-logrotate:compress false
$ pm2 set pm2-logrotate:dateFormat YYYY-MM-DD_HH-mm-ss
$ pm2 set pm2-logrotate:workerInterval 30
$ pm2 set pm2-logrotate:rotateInterval 0 0 _ \* \*
$ pm2 set pm2-logrotate:rotateModule true
gitea-runner@projectium:~$
## dev server setup:
Here are the steps to set up the development environment on Windows using Podman with an Ubuntu container:
1. Install Prerequisites on Windows
Install WSL 2: Podman on Windows relies on the Windows Subsystem for Linux. Install it by running wsl --install in an administrator PowerShell.
Install Podman Desktop: Download and install Podman Desktop for Windows.
2. Set Up Podman
Initialize Podman: Launch Podman Desktop. It will automatically set up its WSL 2 machine.
Start Podman: Ensure the Podman machine is running from the Podman Desktop interface.
3. Set Up the Ubuntu Container
- Pull Ubuntu Image: Open a PowerShell or command prompt and pull the latest Ubuntu image:
podman pull ubuntu:latest
- Create a Podman Volume: Create a volume to persist node_modules and avoid installing them every time the container starts.
podman volume create node_modules_cache
- Run the Ubuntu Container: Start a new container with the project directory mounted and the necessary ports forwarded.
- Open a terminal in your project's root directory on Windows.
- Run the following command, replacing D:\gitea\flyer-crawler.projectium.com\flyer-crawler.projectium.com with the full path to your project:
podman run -it -p 3001:3001 -p 5173:5173 --name flyer-dev -v "D:\gitea\flyer-crawler.projectium.com\flyer-crawler.projectium.com:/app" -v "node_modules_cache:/app/node_modules" ubuntu:latest
-p 3001:3001: Forwards the backend server port.
-p 5173:5173: Forwards the Vite frontend server port.
--name flyer-dev: Names the container for easy reference.
-v "...:/app": Mounts your project directory into the container at /app.
-v "node_modules_cache:/app/node_modules": Mounts the named volume for node_modules.
4. Configure the Ubuntu Environment
You are now inside the Ubuntu container's shell.
- Update Package Lists:
apt-get update
- Install Dependencies: Install curl, git, and nodejs (which includes npm).
apt-get install -y curl git
curl -sL https://deb.nodesource.com/setup_20.x | bash -
apt-get install -y nodejs
- Navigate to Project Directory:
cd /app
- Install Project Dependencies:
npm install
5. Run the Development Server
- Start the Application:
npm run dev
6. Accessing the Application
- Frontend: Open your browser and go to http://localhost:5173.
- Backend: The frontend will make API calls to http://localhost:3001.
Managing the Environment
- Stopping the Container: Press Ctrl+C in the container terminal, then type exit.
- Restarting the Container:
podman start -a -i flyer-dev
## for me:
cd /mnt/d/gitea/flyer-crawler.projectium.com/flyer-crawler.projectium.com
podman run -it -p 3001:3001 -p 5173:5173 --name flyer-dev -v "$(pwd):/app" -v "node_modules_cache:/app/node_modules" ubuntu:latest
rate limiting
respect the AI service's rate limits, making it more stable and robust. You can adjust the GEMINI_RPM environment variable in your production environment as needed without changing the code.
[Add license information here]

Some files were not shown because too many files have changed in this diff Show More