diff --git a/docs/coolify.md b/docs/coolify.md index 3b97f72..2913b02 100644 --- a/docs/coolify.md +++ b/docs/coolify.md @@ -1,5 +1,11 @@ # Deploying SpomeniQR on Coolify +> **Canonical reference:** this guide is maintained for background/context. The +> **verified production walkthrough** (working Traefik setup, v3 `HostRegexp` +> routing labels, Docker 29 compatibility, and a dedicated **wildcard subdomain +> certificate** section) lives in **[coolifyDeployFinal.md](./coolifyDeployFinal.md)**. +> Start there for a new deployment; keep this file's notes below in mind as well. + This guide covers deploying SpomeniQR to a VPS using [Coolify](https://coolify.io/) — a self-hosting platform similar to Heroku/Vercel that manages Docker deployments, SSL, and databases. ## Prerequisites diff --git a/docs/coolifyDeployFinal.md b/docs/coolifyDeployFinal.md new file mode 100644 index 0000000..c52044a --- /dev/null +++ b/docs/coolifyDeployFinal.md @@ -0,0 +1,652 @@ +# Deploying SpomeniQR to Coolify — Final Production Reference + +This document is the complete, verified guide for deploying **SpomeniQR** +(a Next.js multi-tenant memorial app) to a VPS managed by +[Coolify](https://coolify.io/). It reflects the **working production setup** for +`testbed.mk` including wildcard subdomains, a manual wildcard TLS certificate +issued via `acme.sh` + Contabo DNS, and the Traefik `coolify-proxy` that Coolify +runs. + +It supersedes `docs/coolify.md` and `docs/deploy.md` for the Coolify target. + +--- + +## Table of Contents + +1. [Working stack snapshot](#1-working-stack-snapshot) +2. [Prerequisites](#2-prerequisites) +3. [DNS configuration](#3-dns-configuration) +4. [Install Coolify](#4-install-coolify) +5. [Create the project & database](#5-create-the-project--database) +6. [Add the application](#6-add-the-application) +7. [Environment variables](#7-environment-variables) +8. [Domain & subdomain routing (Traefik labels)](#8-domain--subdomain-routing-traefik-labels) +9. [Subdomain certificates (wildcard TLS)](#9-subdomain-certificates-wildcard-tls) +10. [Deploy the application](#10-deploy-the-application) +11. [Verify everything](#11-verify-everything) +12. [Troubleshooting](#12-troubleshooting) +13. [Maintenance & commands](#13-maintenance--commands) + +--- + +## 1. Working stack snapshot + +The production composition that is known to work: + +| Piece | Value | +|-------|-------| +| Proxy | Coolify's **Traefik** (service `coolify-proxy`) | +| Proxy image | `traefik:v3.6.x` (must be **v3.6+**, see [§12.1](#121-traefik-cannot-talk-to-docker-29) — a plain `v3.1` is **broken**) | +| App container | built from this repo's `Dockerfile` (Next.js standalone), listens on `:3000` | +| Database | Coolify-managed PostgreSQL 16, hostname `spomeniqr-db` | +| Wildcard routing | Traefik `HostRegexp` labels on the app container (v3 syntax) | +| Wildcard TLS | `acme.sh` + **Contabo DNS-01** cert for `testbed.mk` + `*.testbed.mk`, served as Traefik **default certificate** | +| Subdomain logic | handled **in-app** from the `Host` header (`src/middleware.ts`) — no `X-Subdomain` proxy header needed | + +``` +Internet + │ + ├── testbed.mk ─────────────► Traefik (coolify-proxy) ──► app:3000 + │ │ + └── *.testbed.mk ──────────────────────┤ (HostRegexp label) + └─► serves wildcard default cert (acme.sh) +``` + +The app derives the memorial subdomain directly from the request `Host` header +(`getSubdomain()` in `src/middleware.ts`), rewrites `/` → `/`, and +renders `src/app/[subdomain]/page.tsx`. **The proxy only needs to route every +host under `testbed.mk` to the app and terminate TLS — it does not need to set +any custom header.** + +--- + +## 2. Prerequisites + +- A VPS running a recent Linux (Ubuntu 22.04+ used here), **≥2 GB RAM**. +- **Docker 29.x** is what triggers the critical Traefik compatibility step + below. If your VPS only has Docker ≤28, Traefik `v3.1` still works, but the + production setup documented here assumes Docker 29 and pins Traefik `v3.6`. +- Domain `testbed.mk` registered, with DNS nameserver access (Contabo DNS in + this project, used by the wildcard cert's DNS-01 challenge). +- A Contabo S3 bucket (`monuments-images`) with public-read policy (see + `docs/deploy.md` §4) and a Clerk **production** instance. + +--- + +## 3. DNS configuration + +Create two DNS records at your provider: + +| Type | Name | Value | TTL | +|------|------|-------|-----| +| A | `@` | `YOUR_VPS_IP` | 300 | +| A | `*` | `YOUR_VPS_IP` | 300 | + +- `@` makes `testbed.mk` resolve → VPS. +- `*` (wildcard) makes **every** `*.testbed.mk` resolve → VPS. Required for + tenant subdomains and for the wildcard cert's DNS-01 **ACME validation + domain** (`_acme-challenge.testbed.mk`). + +Verify: + +```bash +dig testbed.mk +short +dig random.testbed.mk +short # both must print YOUR_VPS_IP +``` + +> The DNS-01 challenge for a wildcard cert needs an `_acme-challenge.testbed.mk` +> **TXT** record which the DNS client creates automatically via API. The +> Contabo API credentials in `dns_contabo.sh` must therefore have permission to +> manage `testbed.mk` records. + +--- + +## 4. Install Coolify + +Use the official installer (`coolify.io/docs/installation`) or the Vagrant-free +server method. On a fresh VPS: + +```bash +curl -fsSL https://coolify.io/install | bash +``` + +Coolify will install Docker and its own stack (`coolify`, `coolify-db`, +`coolify-redis`, `coolify-realtime`, `coolify-sentinel`, `coolify-proxy`, …). +After installation it manages: + +- its proxy configuration under **`/data/coolify/proxy/`** +- its application source under **`/data/coolify/source/`** + +Finish the installer in the browser and log in to the dashboard. + +--- + +## 5. Create the project & database + +1. Dashboard → **+ Add New Project** → name it `SpomeniQR`. +2. Inside the project → **+ Add New Resource → Database → PostgreSQL**: + - Name: `spomeniqr-db` + - PostgreSQL version: **16** + - Database name: `monuments` + - Username: `postgres` + - Password: strong, unique +3. **Deploy**. Copy the **Internal Connection String**, e.g.: + `postgresql://postgres:PASSWORD@spomeniqr-db:5432/monuments` + (the host is the Coolify-internal service name `spomeniqr-db`, not `localhost`). + +--- + +## 6. Add the application + +1. Inside the project → **+ Add New Resource → Application**. +2. Select **Public Repository** (or **Private** with a deploy key). + - Name: `spomeniqr` + - Repository URL: your Git URL + - Branch: `main` + - **Build Pack: Docker** (uses the repo's `Dockerfile`; verified path). +3. Wire the database to the app on the shared Coolify network (`quick connect` + button or ensure both are on `coolify` network). + +The repo `Dockerfile` already handles `prisma generate`, `next build`, +`prisma migrate deploy`, and provisioning the `super` admin on startup, so no +custom build/start commands are needed. + +--- + +## 7. Environment variables + +In the app's **Environment Variables** panel add (mark **every** `NEXT_PUBLIC_*` +as a **Build Variable**): + +```env +# Database — Coolify internal hostname +DATABASE_URL=postgresql://postgres:PASSWORD@spomeniqr-db:5432/monuments + +# Clerk +NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY=pk_live_... +CLERK_SECRET_KEY=sk_live_... +NEXT_PUBLIC_CLERK_FAPI_HOST=clerk.testbed.mk # build var, only if using custom frontend API domain +NEXT_PUBLIC_CLERK_SIGN_IN_URL=/sign-in +NEXT_PUBLIC_CLERK_SIGN_UP_URL=/sign-up +NEXT_PUBLIC_CLERK_SIGN_IN_FALLBACK_REDIRECT_URL=/ +NEXT_PUBLIC_CLERK_SIGN_UP_FALLBACK_REDIRECT_URL=/ + +# Contabo S3 +S3_ENDPOINT=https://eu2.contabostorage.com +S3_REGION=eu-2 +S3_ACCESS_KEY_ID=your-access-key +S3_SECRET_ACCESS_KEY=your-secret-key +S3_BUCKET_NAME=monuments-images + +# App +NEXT_PUBLIC_APP_URL=https://testbed.mk +NEXT_PUBLIC_APP_DOMAIN=testbed.mk + +# Admin session signing secret (openssl rand -hex 32) +ADMIN_SESSION_SECRET=your-random-64-char-secret + +# Super-admin (bcrypt hash, NOT plaintext) +SUPER_ADMIN_USERNAME=super +SUPER_ADMIN_PASSWORD_HASH=$2b$12$REPLACE_WITH_BCRYPT_HASH + +# Node +NODE_ENV=production +``` + +**Critical rules** + +- `NEXT_PUBLIC_*` variables are inlined at `next build`. Changing one requires a + **redeploy** (not just a restart). `NEXT_PUBLIC_CLERK_FAPI_HOST` in particular + must be a build var or the build-time CSP blocks Clerk JS. +- `SUPER_ADMIN_PASSWORD_HASH` is passed **literally** by Coolify — paste the raw + `$2b$12$…` hash with no `$$` escaping (that escaping is only for `env_file` + Compose usage). Generate with: + `node -e "import('bcryptjs').then(b => b.default.hash('YOUR_PASSWORD', 12).then(console.log))"` +- Server-only vars (`CLERK_SECRET_KEY`, `DATABASE_URL`, `ADMIN_SESSION_SECRET`, + `SUPER_ADMIN_PASSWORD_HASH`, `S3_*`) stay as normal runtime vars. +- `ADMIN_SESSION_SECRET` missing ⇒ super-admin login returns **500** at boot. + +--- + +## 8. Domain & subdomain routing (Traefik labels) + +Coolify's proxy in this project is **Traefik**. Routing to the app is defined by +**Docker labels in the repo's `docker-compose.yaml`**, which Coolify applies +when it deploys the container. Keep that file in sync in the repo: + +```yaml +services: + app: + build: + context: . + dockerfile: Dockerfile + restart: unless-stopped + env_file: + - .env + networks: + - coolify + labels: + - traefik.enable=true + - traefik.docker.network=coolify + - traefik.http.routers.testbed-root.rule=Host(`testbed.mk`) + - traefik.http.routers.testbed-root.entryPoints=https + - traefik.http.routers.testbed-root.service=spomeniqr-svc + - traefik.http.routers.testbed-root.tls=true + - traefik.http.routers.testbed-wildcard.rule=HostRegexp(`^[a-z0-9-]+\.testbed\.mk$$`) + - traefik.http.routers.testbed-wildcard.entryPoints=https + - traefik.http.routers.testbed-wildcard.service=spomeniqr-svc + - traefik.http.routers.testbed-wildcard.tls=true + - traefik.http.routers.testbed-root-http.rule=Host(`testbed.mk`) + - traefik.http.routers.testbed-root-http.entryPoints=http + - traefik.http.routers.testbed-root-http.middlewares=redirect-to-https + - traefik.http.routers.testbed-root-http.service=spomeniqr-svc + - traefik.http.routers.testbed-wildcard-http.rule=HostRegexp(`^[a-z0-9-]+\.testbed\.mk$$`) + - traefik.http.routers.testbed-wildcard-http.entryPoints=http + - traefik.http.routers.testbed-wildcard-http.middlewares=redirect-to-https + - traefik.http.routers.testbed-wildcard-http.service=spomeniqr-svc + - traefik.http.services.spomeniqr-svc.loadbalancer.server.port=3000 + - traefik.http.middlewares.gzip.compress=true + - traefik.http.middlewares.redirect-to-https.redirectscheme.scheme=https + +networks: + coolify: + external: true +``` + +**Traefik v3 `HostRegexp` — critical syntax note** + +- ✅ v3-valid: `HostRegexp(`^[a-z0-9-]+\.testbed\.mk$`)` +- ❌ v2 (removed in v3): `HostRegexp(`{subdomain:[a-zA-Z0-9-]+}.testbed.mk`)` + +The v2 named-capture form silently fails to parse under Traefik 3, so the +wildcard router is **not created** and subdomains fall through to Coolify's +default **503** page ("server not available") even though the apex works. **Do +not** reintroduce the v2 form. + +In `docker-compose.yaml` the trailing `$` anchor must be written as `$$` so YAML +interpolation yields a literal `$`. After editing, the container must be +**recreated** for the new labels to take effect (a plain restart is not enough). + +**Effect of the two HTTPS routers** + +- `Host(\`testbed.mk\`)` → apex landing page. +- `HostRegexp(\`^[a-z0-9-]+\.testbed\.mk$\`)` → **every** subdomain + (`perop`, `test-memorial`, …) → same `app:3000`. The app's middleware then + maps the subdomain to a memorial page. +- Both are `tls=true` **without a certresolver**, so Traefik uses the + **default certificate** — which is the wildcard cert from + [§9 Subdomain certificates](#9-subdomain-certificates-wildcard-tls). No + `tls.certresolver` label is needed on these routers. + +> Because Coolify's own default routers can also be generated from the UI +> `Domains` panel, prefer adding `testbed.mk` / `*.testbed.mk` there **or** +> relying solely on these labels — avoid having both apply the same rule, which +> creates duplicate-router ambiguity. + +--- + +## 9. Subdomain certificates (wildcard TLS) + +This section is dedicated to provisioning and renewing the TLS certificate that +secures **all** `*.testbed.mk` subdomains, since it is the part that is easiest +to get wrong. + +### 9.1 Why a wildcard cert + +The app issues an arbitrary number of tenant subdomains +(`.testbed.mk`). It is impractical to obtain an individual +certificate per subdomain. A single wildcard certificate for `*.testbed.mk` +covers every present and future tenant. Because it is issued as Traefik's +**default certificate**, any router with `tls=true` and no explicit +certresolver (which is exactly how §8 is configured) is served the wildcard +cert automatically. + +### 9.2 Challenge type & provider + +Wildcard certs can **only** be validated with the **DNS-01** challenge (an +`_acme-challenge.testbed.mk` TXT record — there is no HTTP path to validate +`.testbed.mk` itself). This project uses `acme.sh` with a custom **Contabo DNS** +client (`dns_contabo.sh`) because the domain is hosted on Contabo DNS. + +### 9.3 Required credentials + +Obtain from your Contabo customer account: + +| Variable | Purpose | +|----------|---------| +| `CONTABO_CLIENT_ID` | OAuth2 Client ID | +| `CONTABO_CLIENT_SECRET` | OAuth2 Client Secret | +| `CONTABO_API_USER` | Contabo account email | +| `CONTABO_API_PASSWORD` | Contabo account password | + +`dns_contabo.sh` reads these from the environment / acme.sh account conf and +uses Contabo's API (`https://api.contabo.com/v1`) to create and delete the +`_acme-challenge` TXT records needed for validation. + +### 9.4 Install `dns_contabo.sh` + +1. Install `acme.sh` (non-root or root): + ```bash + curl https://get.acme.sh | sh -s email=you@example.com + ``` +2. Place `dns_contabo.sh` where acme.sh finds API plugins. It is written to be + self-contained (uses `acme.sh` built-ins `_post`/`_info`/`_err`), so the + cleanest approach is: + ```bash + mkdir -p ~/.acme.sh/dnsapi + cp /path/to/dns_contabo.sh ~/.acme.sh/dnsapi/ + chmod +x ~/.acme.sh/dnsapi/dns_contabo.sh + ``` + +### 9.5 Export credentials + +```bash +export CONTABO_CLIENT_ID="..." +export CONTABO_CLIENT_SECRET="..." +export CONTABO_API_USER="you@example.com" +export CONTABO_API_PASSWORD="..." +``` + +### 9.6 Issue the wildcard certificate + +```bash +# Covers the apex AND every subdomain +~/.acme.sh/acme.sh --issue \ + --dns dns_contabo \ + -d testbed.mk \ + -d '*.testbed.mk' +``` + +acme.sh stores the result in `~/.acme.sh/testbed.mk_ecc/`: +`fullchain.cer`, `testbed.mk.key`, `testbed.mk.cer`, `ca.cer`. + +> **Validation**: the `${DOMAIN}_ecc` directory must contain `testbed.mk.cer` +> and a key. If verification fails, check the DNS TXT records +> (`dig +short _acme-challenge.testbed.mk TXT`) and that the Contabo credentials +> have record permissions on the zone. + +### 9.7 Traefik default-certificate config (Coolify proxy) + +The proxy (`coolify-proxy`, image `traefik:v3.6`) mounts the host directory +`/data/coolify/proxy/` at `/traefik` (see [§12.2](#122-proxy-mount-layout)). +Two things must be provisioned there: + +**a) Copy the cert files into Coolify's certs directory:** + +```bash +COOLIFY_CERT_DIR="/data/coolify/proxy/certs" +mkdir -p "$COOLIFY_CERT_DIR" +cp ~/.acme.sh/testbed.mk_ecc/fullchain.cer "$COOLIFY_CERT_DIR/testbed.mk.cert" +cp ~/.acme.sh/testbed.mk_ecc/testbed.mk.key "$COOLIFY_CERT_DIR/testbed.mk.key" +chmod 644 "$COOLIFY_CERT_DIR/testbed.mk.cert" +chmod 600 "$COOLIFY_CERT_DIR/testbed.mk.key" +``` + +**b) Create the dynamic config** `/data/coolify/proxy/dynamic/testbed.mk-wildcard.yaml` +(loaded by the proxy's file provider; note paths are **inside the container**, +i.e. under `/traefik/…`): + +```yaml +tls: + certificates: + - certFile: /traefik/certs/testbed.mk.cert + keyFile: /traefik/certs/testbed.mk.key + stores: + default: + defaultCertificate: + certFile: /traefik/certs/testbed.mk.cert + keyFile: /traefik/certs/testbed.mk.key +``` + +The `defaultCertificate` under `tls.stores.default` makes this the cert Traefik +serves for any TLS router that has no explicit certresolver — including both +`testbed-root` and `testbed-wildcard` (§8). A change here requires a proxy +reload (restarting `coolify-proxy` picks it up). + +### 9.8 Install with acme.sh for auto-renewal + +Register the copy + reload steps so acme.sh performs them automatically every +renewal (certs expire every ~90 days): + +```bash +~/.acme.sh/acme.sh --install-cert -d testbed.mk -d '*.testbed.mk' \ + --fullchain-file "/data/coolify/proxy/certs/testbed.mk.cert" \ + --key-file "/data/coolify/proxy/certs/testbed.mk.key" \ + --reloadcmd "docker restart coolify-proxy" +``` + +- `--fullchain-file`/`--key-file` tell acme.sh to **copy** the renewed certs + into Coolify's certs directory on every renewal. +- `--reloadcmd` restarts the proxy so Traefik reloads the files. +- acme.sh installs its own cron job for renewal, so no separate crontab is + needed. + +Optionally, a dedicated renewal hook script can be used (kept in +`~/.acme.sh/renewal-hooks/testbed.mk-coolify-reload.sh`) that copies both files +and runs `docker restart coolify-proxy`; advantages are clearer logging and a +manual script you can run to force a reload after a manual cert update. + +### 9.9 Verify the wildcard cert is live + +```bash +# SANs must list both testbed.mk and *.testbed.mk +openssl x509 -in /data/coolify/proxy/certs/testbed.mk.cert -noout -text | grep -A2 "Subject Alternative Name" + +# Traefik default cert actually offered on a subdomain +echo | openssl s_client -connect perop.testbed.mk:443 -servername perop.testbed.mk 2>/dev/null \ + | openssl x509 -noout -subject -issuer + +# Next renewal date +~/.acme.sh/acme.sh --list | grep testbed.mk +``` + +### 9.10 Renewal policy summary + +| Item | Value | +|------|-------| +| Issuer | Let's Encrypt via `acme.sh` (DNS-01, `dns_contabo`) | +| Covered | `testbed.mk` + `*.testbed.mk` | +| Lifetime | 90 days; auto-renewed by acme.sh cron | +| Destination | `/data/coolify/proxy/certs/testbed.mk.{cert,key}` | +| Reload | `--reloadcmd docker restart coolify-proxy` | +| Served as | Traefik **default certificate** (`tls.stores.default.defaultCertificate`) | + +> **Do NOT** rely on Coolify's built-in Let's Encrypt `letsencrypt` certresolver +> for the wildcard. Coolify's default uses an HTTP-01 challenge, which cannot +> validate a wildcard, and Traefik v3 does not auto-provision per-subdomain certs +> from `HostRegexp` routers. The manual DNS-01 wildcard + default-cert approach +> is the correct one for this app. + +--- + +## 10. Deploy the application + +1. Ensure `[§7 env vars]` are saved and all `NEXT_PUBLIC_*` are marked as build + variables **before** the first build. +2. Click **Deploy**. Watch the build log to confirm, in order: + - dependencies installed, + - `npx prisma generate`, + - `next build`, + - `npx prisma migrate deploy`, + - `[seed] Super-admin 'super' provisioned.`, + - server listening on port **3000**. +3. If the routing (Traefik) labels changed (e.g. §8 edits in the repo), triggers + a **recreate** so the new labels apply. + +--- + +## 11. Verify everything + +```bash +# Apex +curl -s -o /dev/null -w "%{http_code}\n" https://testbed.mk # 200 + +# Subdomain / wildcard routing + default cert +curl -s -o /dev/null -w "%{http_code}\n" https://perop.testbed.mk # 200 + +# TLS on a subdomain uses the wildcard cert (no browser error) +echo | openssl s_client -connect perop.testbed.mk:443 -servername perop.testbed.mk 2>/dev/null \ + | openssl x509 -noout -subject + +# Proxy healthy + no docker-provider errors +docker logs coolify-proxy --since 1m 2>&1 | grep -iE "too old|error" | tail + +# API health +curl -s https://testbed.mk/api/check-subdomain?slug=test | python3 -m json.tool + +# Admin login (expects {"success":true}) +curl -s -X POST https://testbed.mk/api/admin/login -H "Content-Type: application/json" \ + -d '{"username":"super","password":"YOUR_PASSWORD"}' +``` + +Browser checks + +- `https://testbed.mk` → landing page. +- Publish a memorial with subdomain `perop`, then `https://perop.testbed.mk` + → the memorial page with a green lock (no certificate warning). +- DevTools → Network: no CSP violation for `clerk.testbed.mk/npm/...` when the + custom Clerk frontend domain is used. + +--- + +## 12. Troubleshooting + +### 12.1 Traefik cannot talk to Docker 29 ("client version 1.24 is too old") + +**Symptom**: `coolify-proxy` logs repeat: + +``` +ERR Failed to retrieve information of the docker client ... +error="Error response from daemon: client version 1.24 is too old. +Minimum supported API version is 1.40, please upgrade your client" +providerName=docker +``` + +and **every** domain (apex + subdomains) returns 503. + +**Cause**: Docker 29 raised its minimum API version (here **1.40**); Traefik +`v3.1` runs a Docker client pinned to **API 1.24** and never negotiates, so the +Docker provider — which discovers the app's label routers — is dead. It is not a +config problem. Coolify's default proxy (image `traefik:v3.1`) hits this on a +Docker 29 host. + +**Fix**: pin the proxy image to **Traefik v3.6+** (adds Docker API auto- +negotiation). The proxy compose lives at `/data/coolify/proxy/docker-compose.yml` +(the service is named `traefik`): + +```bash +sed -i "s@traefik:v3.1@traefik:v3.6@" /data/coolify/proxy/docker-compose.yml +grep 'image:' /data/coolify/proxy/docker-compose.yml +docker compose -f /data/coolify/proxy/docker-compose.yml up -d --force-recreate traefik +docker exec coolify-proxy traefik version # expect 3.6.x +``` + +Confirm the provider is healthy and routing resumes: + +```bash +docker logs coolify-proxy --since 30s 2>&1 | grep -i "too old" # should be empty +curl -s -o /dev/null -w "%{http_code}\n" https://testbed.mk # 200 +``` + +A future Coolify **update may regenerate the proxy compose back to `v3.1`**. +Re-apply the `sed` above after any Coolify update, or (as a durable stopgap) +lower the daemon's minimum API so the 1.24 client is accepted by adding +`"min-api-version": "1.24"` to `/etc/docker/daemon.json` and restarting Docker: + +```bash +# /etc/docker/daemon.json: { ..., "min-api-version": "1.24" } +systemctl restart docker +``` + +Prefer the v3.6 image fix — it touches only the proxy, not the whole daemon. + +### 12.2 Proxy mount layout + +`coolify-proxy` mounts the host directory `/data/coolify/proxy/` at `/traefik`. +Everything the proxy reads must therefore be expressed with in-container paths: + +- cert files inside the container: `/traefik/certs/*` +- dynamic config inside the container: `/traefik/dynamic/*` +- ACME store: `/traefik/acme.json` + +Verify the mount and files: + +```bash +docker inspect coolify-proxy --format '{{json .Mounts}}' +docker exec coolify-proxy ls -la /traefik/certs /traefik/dynamic +``` + +### 12.3 Subdomain shows "server not available" (503) while apex works + +- **Most common: v2 `HostRegexp` label.** Ensure both + `testbed-wildcard` and `testbed-wildcard-http` use the v3 form + `HostRegexp(\`^[a-z0-9-]+\.testbed\.mk$$\`)` (see §8). Recreate the container + after editing labels. +- **Docker 29 / Traefik v3.1**: see §12.1. +- **DNS**: `dig perop.testbed.mk +short` must return the VPS IP. +- **Duplicate routers**: if you also added `*.testbed.mk` in Coolify's UI + `Domains`, you may have overlapping routers — prefer one method. + +### 12.4 Subdomain loads but shows a certificate error + +- Wildcard default cert not loaded / stale. Check §9.7 layout and paths, then + `docker restart coolify-proxy`. +- Cert files not renewed: rerun `~/.acme.sh/acme.sh --renew -d testbed.mk -d '*.testbed.mk' --force` + and check the SANs (must include `*.testbed.mk`). + +### 12.5 Build fails with Prisma errors + +`DATABASE_URL` must use the internal hostname `spomeniqr-db:5432`, not +`localhost`, and the app + database must be on the shared Coolify network. + +### 12.6 Super-admin login fails + +- 401: `SUPER_ADMIN_PASSWORD_HASH` pasted with `$$` escaping → re-paste the raw + `$2b$12$…` hash and redeploy. +- 500 / `ADMIN_SESSION_SECRET env var is not set` → set `ADMIN_SESSION_SECRET` + (runtime) and restart. +- Check boot log for `[seed] Super-admin 'super' provisioned.` + +### 12.7 CSP blocks Clerk JS (auth buttons dead) + +Set `NEXT_PUBLIC_CLERK_FAPI_HOST` (e.g. `clerk.testbed.mk`) as a **build +variable** and **redeploy** — the CSP is generated at build time. + +--- + +## 13. Maintenance & commands + +| Action | Command / location | +|--------|--------------------| +| Redeploy app | Project → Application → **Deploy** | +| Re-apply proxy v3.6 pin | `sed -i 's@traefik:v3.1@traefik:v3.6@' /data/coolify/proxy/docker-compose.yml && docker compose -f /data/coolify/proxy/docker-compose.yml up -d --force-recreate traefik` | +| Restart proxy | `docker restart coolify-proxy` | +| Proxy logs | `docker logs coolify-proxy -f` | +| App logs | Project → Application → **Logs** / `docker logs -f` | +| Force cert renewal | `~/.acme.sh/acme.sh --renew -d testbed.mk -d '*.testbed.mk' --force` | +| Check cert status | `~/.acme.sh/acme.sh --list` | +| Migrations | auto on container start; manual: `docker exec npx prisma migrate deploy` | +| DB backup | Coolify UI database backup, or `pg_dump` against `spomeniqr-db` | +| Update app | push to Git; Coolify redeploys (or click **Deploy**) | + +--- + +### Environment variables quick reference + +| Variable | Req | Notes | +|----------|-----|-------| +| `DATABASE_URL` | Yes | Coolify-internal, host `spomeniqr-db` | +| `NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY` | Yes | build var; `pk_live_…` | +| `NEXT_PUBLIC_CLERK_FAPI_HOST` | No* | build var; custom Clerk domain | +| `CLERK_SECRET_KEY` | Yes | runtime; `sk_live_…` | +| `NEXT_PUBLIC_CLERK_*` paths | Yes | build vars (see §7) | +| `S3_ENDPOINT` / `S3_REGION` | Yes | Contabo object storage | +| `S3_ACCESS_KEY_ID` / `S3_SECRET_ACCESS_KEY` | Yes | Contabo API keys | +| `S3_BUCKET_NAME` | Yes | `monuments-images` | +| `NEXT_PUBLIC_APP_URL` | Yes | build var; `https://testbed.mk` | +| `NEXT_PUBLIC_APP_DOMAIN` | Yes | build var; `testbed.mk` | +| `ADMIN_SESSION_SECRET` | Yes | runtime; `openssl rand -hex 32` | +| `SUPER_ADMIN_USERNAME` / `SUPER_ADMIN_PASSWORD_HASH` | No | bcrypt hash, literal | +| `NODE_ENV` | Yes | `production` | + +\* only when using a custom Clerk frontend API domain (`clerk.testbed.mk`). \ No newline at end of file