one moar time - we can do it?

notes-to-ai-3-genai.txt

@@ -14,3 +14,222 @@ End-of-Life Date: All support for this repository (including bug fixes) will per
 We encourage all users to begin planning their migration to the Google Generative AI SDK to ensure continued access to the latest capabilities and support.
 what does this mean to you about the correct new version of Google Generative AI SDK is ??
 NO CODE
+
+
+
+
+
+
+
+
+
+remove "Avatar URL" from "Create an Account" panel
+
+ - also, "Register" button is not very obvious - all buttons in this panel need better definition
+
+ - no "Confirm Password" field - to ensure passwords are typed correctly
+
+ the "Upload New Flyer" area is not using the application stadard colour scheme
+
+
+Now, can you improve the styling of the tabs in the authenticated view?
+
+
+after being logged in, and doing a page refresh, the "Log In" button is displayed at the top right, not the user icon indicating they are logged in
+
+Can you refactor the drag-and-drop logic in `FlyerUploader` into a reusable hook?
+
+
+
+
+current GenAI plan has a limit of 5 requests per minute
+
+current GenAI plan has a limit of 5 requests per minute - can we set up a ENV variable for rate limits, and enforce rate limits in the flyer processing flow
+
+also, can we limit flyer unit tests to not hit the real GenAI endpoint
+
+
+
+how best to store DEV passwords?
+
+[start:dev] /app/node_modules/passport-jwt/lib/strategy.js:45
+[start:dev]         throw new TypeError('JwtStrategy requires a secret or key');
+[start:dev]               ^
+[start:dev]
+[start:dev] TypeError: JwtStrategy requires a secret or key
+[start:dev]     at new JwtStrategy (/app/node_modules/passport-jwt/lib/strategy.js:45:15)
+[start:dev]     at <anonymous> (/app/src/routes/passport.routes.ts:215:14)
+[start:dev]     at ModuleJob.run (node:internal/modules/esm/module_job:325:25)
+[start:dev]     at async ModuleLoader.import (node:internal/modules/esm/loader:606:24)
+[start:dev]     at async asyncRunEntryPointWithESMLoader (node:internal/modules/run_main:117:5)
+
+
+
+split test and prod
+
+
+suggestions + suggestion review area in admin - AI currated 
+
+
+
+
+Review all `*.routes.ts` files and refactor them to use the new, simplified error handling without manual 404 checks.
+
+Review all `*.routes.ts` files and refactor them to use the new `NotFoundError` class instead of manual 404 checks.
+
+Now, can you refactor all the route handlers in `user.routes.ts` and `admin.routes.ts` to remove the old `error.message.includes('not found')` checks?
+
+
+Now, refactor all the route handlers in `user.routes.ts` and `admin.routes.ts` to remove the old `error.message.includes('not found')` checks.
+
+
+
+
+
+Okay, this is a significant and excellent step towards a more robust deployment strategy! Splitting your production environment into distinct test and production environments is crucial for ensuring stability and catching issues before they impact live users.
+
+Here's a detailed plan to achieve this, without any code changes yet, focusing on the logical steps and infrastructure considerations:
+
+Project Goal: Split Production into Test + Production Environments
+New Domain: https://flyer-crawler-test.projectium.com Deployment Strategy: Split the existing deploy.yml into two distinct jobs: deploy-to-test and deploy-to-production.
+
+Phase 1: Infrastructure Setup (Outside Gitea Workflows)
+This phase involves setting up the necessary server-side components for the new test environment.
+
+DNS Configuration:
+
+Create a new A record for flyer-crawler-test.projectium.com that points to the IP address of your existing projectium.com server. This makes the new domain resolvable.
+Nginx Configuration:
+
+On your Ubuntu co-location server, create a new Nginx server block configuration file (e.g., /etc/nginx/sites-available/flyer-crawler-test.projectium.com.conf).
+Configure this new server block to:
+Listen on port 80 and 443 for flyer-crawler-test.projectium.com.
+Serve static files from a new root directory, e.g., /var/www/flyer-crawler-test.projectium.com.
+Proxy API requests (e.g., /api) to a new PM2 process running the backend for the test environment (e.g., http://localhost:3002 if production is on 3001).
+Set up SSL certificates for flyer-crawler-test.projectium.com (e.g., using Certbot).
+Enable the new Nginx site and reload Nginx.
+
+Database Setup:
+
+Create a new PostgreSQL database specifically for the test environment, e.g., flyer-crawler-test. This should be distinct from your production database (flyer-crawler-prod).
+Ensure your existing DB_USER (or a new dedicated test user) has full access to this new flyer-crawler-test database.
+Important: This new database should be completely isolated from the production database.
+File System Preparation:
+
+Create the new deployment directory on the server: /var/www/flyer-crawler-test.projectium.com.
+Inside this directory, create necessary subdirectories like flyer-images, flyer-images/icons, flyer-images/archive to match the production setup.
+
+PM2 Configuration (Initial):
+
+Plan for distinct PM2 process names for the test environment, e.g., flyer-crawler-api-test and flyer-crawler-worker-test. This will allow PM2 to manage both environments independently.
+
+Phase 2: Gitea Secrets Management
+To maintain separation and security, new secrets will be needed for the test environment.
+
+Create New Gitea Repository Secrets:
+DB_DATABASE_TEST: The name of your new test database (e.g., flyer-crawler-test).
+REDIS_PASSWORD_TEST: If you use a separate Redis instance or password for the test environment. If not, you might reuse REDIS_PASSWORD_PROD but it's generally better to isolate.
+VITE_GOOGLE_GENAI_API_KEY_TEST: If you have a separate API key for the test frontend/backend.
+JWT_SECRET_TEST: A different JWT secret for the test environment.
+Consider if DB_HOST, DB_USER, DB_PASSWORD should also be duplicated for the test environment, even if they point to the same server, for future flexibility and isolation. For this plan, we'll assume the same DB credentials but a different DB name.
+
+
+
+
+items completed : 
+ - DNS Configuration: Done
+ - Nginx Configuration: Done
+ - Database Setup: Done - I think our current setup already uses a test database, with its own user
+ - File System Preparation: /var/www/flyer-crawler-test.projectium.com created - I'll let the deploy-to-test to create the needed directories
+ - PM2 Configuration (Initial): NOT DONE
+ - Create New Gitea Repository Secrets: DONE
+ - Phase 3: Gitea Workflow: Next
+
+NO CODE
+
+
+
+
+Phase 3: Gitea Workflow (deploy.yml) Modifications Plan
+This is where the existing deploy.yml will be refactored into two distinct jobs.
+
+Rename Existing Job:
+
+Rename the current deploy job to deploy-production.
+Create New Job: deploy-to-test:
+
+This job will mirror the deploy-production job but target the new test environment.
+Trigger: Initially, it can be triggered on push to main alongside deploy-production. Later, you might consider a develop branch for test deployments.
+Environment Variables:
+Use the newly created _TEST secrets for database, Redis, API keys, and JWT secrets.
+Set FRONTEND_URL to https://flyer-crawler-test.projectium.com.
+Set VITE_API_BASE_URL to https://flyer-crawler-test.projectium.com/api.
+Deployment Path: All rsync and cd commands will target /var/www/flyer-crawler-test.projectium.com.
+PM2 Process Names: Use distinct PM2 process names (e.g., flyer-crawler-api-test, flyer-crawler-worker-test) when stopping, starting, or reloading PM2 processes.
+Database Schema Check/Update: This job will perform its own schema check and update against the flyer-crawler-test database using DB_DATABASE_TEST.
+Testing: The existing Run All Tests and Generate Merged Coverage Report step should be moved into this deploy-to-test job. This ensures that the test environment is thoroughly validated upon deployment.
+Coverage Report Deployment: Deploy the coverage report to flyer-crawler-test.projectium.com/coverage.
+Modify deploy-to-production Job:
+
+Ensure all environment variables, deployment paths, PM2 process names, and database interactions continue to point exclusively to the production environment (flyer-crawler.projectium.com, flyer-crawler-prod database).
+Remove Testing Steps: The Run All Tests and Generate Merged Coverage Report step should be removed from this job, as testing will now be handled by the deploy-to-test job.
+Dependency (Optional but Recommended): Consider adding a needs: deploy-to-test dependency to the deploy-to-production job. This would ensure that the test environment deployment (and its associated tests) must succeed before the production deployment can even start.
+Refine manual-db-backup.yml, manual-db-reset.yml, manual-db-restore.yml:
+
+These workflows currently target the production database.
+Option A (Simpler): Keep them as-is, explicitly for production. If you need similar functionality for test, create new workflows (e.g., manual-db-backup-test.yml).
+Option B (More Flexible): Modify them to accept an input parameter (e.g., environment: 'prod' or environment: 'test') and dynamically select the database name and secrets based on this input. This would require more complex scripting within the workflow.
+For now, assume Option A for simplicity, meaning these workflows remain production-specific.
+Phase 4: Post-Deployment Considerations
+Monitoring and Logging:
+
+Ensure your monitoring and logging tools are configured to differentiate between the test and production environments.
+Set up alerts specific to each environment.
+Access Control:
+
+Consider if access to the test environment should be restricted (e.g., IP whitelisting, basic auth) compared to the public production environment.
+Data Management:
+
+Establish a process for refreshing the test database with (sanitized) production data periodically, if needed for realistic testing.
+This plan provides a clear roadmap for separating your environments. The next step would be to implement these changes in your Gitea workflow files and server configurations.
+
+
+
+RULES: 1: if you do not have a file that you need, stop, and request it immediately. 2: never remove logging or comments 3: never use 'any' or 'unknown' to solve possible typescript issues 4: when creating new files, output there entire path in your explanation, to make it easier to know where to save those new files and directories to 5: add comments when appropriate to help ensure ideas persist 6: Your knowledge of package version, like nodejs, is always old, like a year or more old - ask me for the best version to use, as your knowledge is incomplete 7: Stop making predictions and/or guessing at solutions. Focus on adding logging and debugging to issues that are not solved right away. 8: Do not make obsequious statements - we're here to do a job, not get patted on the shoulder for insignificant achievements. 9: Provide me with the npm command to execute rather than wanting to edit the package.json file
+
+
+
+
+RULES:
+if you do not have a file that you need, stop, and request it immediately.
+never remove logging or comments
+you cannot ever use 'any' or 'unknown' to solve possible typescript issues
+when creating new files, output there entire path in your explanation, to make it easier to know where to save those new files and directories to
+add comments when you can, as that will help ensure ideas persist into the app
+Your knowledge of package version, like nodejs, is always old, like a year or more old - ask me for the best version to use, as your knowledge is incomplete
+Stop making predictions and/or guessing at solutions. Focus on adding logging and debugging to issues that are not solved right away.
+Do not make obsequious statements - we're here to do a job, not get patted on the shoulder for insignificant achievements.
+Provide me with the npm command to execute rather than wanting to edit the package.json file. That is not the correct way to handle a package update.
+
+do not assume success before testing proves success - never remove logging or debugging before success has been proved by testing
+
+
+provide code changes in DIFF format
+
+
+add as much logging and debugging as possible to possible to fix these issues quickly
+
+
+
+apply the new validation in ADR-003 to src/routes/admin.routes.ts
+
+
+
+
+
+
+
+The failures are happening because Zod's z.string() validation checks the type before checks the constraints (like .min()). When a field is missing (undefined), Zod returns a generic "Expected string, received undefined" type error, which doesn't match your custom .min(1, "Message") error message, and the required_error option is causing Type errors in your environment.
+The robust architectural solution is to use z.preprocess() to normalize "missing" (undefined or null) values into an empty string "" before the validator runs. This forces the validation to fail the .min(1) check instead of the type check, guaranteeing your custom error message is always used for both missing and empty fields.
+Here is the fix applied to both route files using a helper function requiredString.
+src/routes/auth.routes.ts