# Deploying SpomeniQR to Coolify — Final Production Reference

This document is the complete, verified guide for deploying **SpomeniQR**
(a Next.js multi-tenant memorial app) to a VPS managed by
[Coolify](https://coolify.io/). It reflects the **working production setup** for
`testbed.mk` including wildcard subdomains, a manual wildcard TLS certificate
issued via `acme.sh` + Contabo DNS, and the Traefik `coolify-proxy` that Coolify
runs.

It supersedes `docs/coolify.md` and `docs/deploy.md` for the Coolify target.

---

## Table of Contents

1. [Working stack snapshot](#1-working-stack-snapshot)
2. [Prerequisites](#2-prerequisites)
3. [DNS configuration](#3-dns-configuration)
4. [Install Coolify](#4-install-coolify)
5. [Create the project & database](#5-create-the-project--database)
6. [Add the application](#6-add-the-application)
7. [Environment variables](#7-environment-variables)
8. [Domain & subdomain routing (Traefik labels)](#8-domain--subdomain-routing-traefik-labels)
9. [Subdomain certificates (wildcard TLS)](#9-subdomain-certificates-wildcard-tls)
10. [Deploy the application](#10-deploy-the-application)
11. [Verify everything](#11-verify-everything)
12. [Troubleshooting](#12-troubleshooting)
13. [Maintenance & commands](#13-maintenance--commands)

---

## 1. Working stack snapshot

The production composition that is known to work:

| Piece | Value |
|-------|-------|
| Proxy | Coolify's **Traefik** (service `coolify-proxy`) |
| Proxy image | `traefik:v3.6.x` (must be **v3.6+**, see [§12.1](#121-traefik-cannot-talk-to-docker-29) — a plain `v3.1` is **broken**) |
| App container | built from this repo's `Dockerfile` (Next.js standalone), listens on `:3000` |
| Database | Coolify-managed PostgreSQL 16, hostname `spomeniqr-db` |
| Wildcard routing | Traefik `HostRegexp` labels on the app container (v3 syntax) |
| Wildcard TLS | `acme.sh` + **Contabo DNS-01** cert for `testbed.mk` + `*.testbed.mk`, served as Traefik **default certificate** |
| Subdomain logic | handled **in-app** from the `Host` header (`src/middleware.ts`) — no `X-Subdomain` proxy header needed |

```
Internet
  │
  ├── testbed.mk ─────────────► Traefik (coolify-proxy) ──► app:3000
  │                                      │
  └── *.testbed.mk ──────────────────────┤  (HostRegexp label)
                                         └─► serves wildcard default cert (acme.sh)
```

The app derives the memorial subdomain directly from the request `Host` header
(`getSubdomain()` in `src/middleware.ts`), rewrites `/` → `/<subdomain>`, and
renders `src/app/[subdomain]/page.tsx`. **The proxy only needs to route every
host under `testbed.mk` to the app and terminate TLS — it does not need to set
any custom header.**

---

## 2. Prerequisites

- A VPS running a recent Linux (Ubuntu 22.04+ used here), **≥2 GB RAM**.
- **Docker 29.x** is what triggers the critical Traefik compatibility step
  below. If your VPS only has Docker ≤28, Traefik `v3.1` still works, but the
  production setup documented here assumes Docker 29 and pins Traefik `v3.6`.
- Domain `testbed.mk` registered, with DNS nameserver access (Contabo DNS in
  this project, used by the wildcard cert's DNS-01 challenge).
- A Contabo S3 bucket (`monuments-images`) with public-read policy (see
  `docs/deploy.md` §4) and a Clerk **production** instance.

---

## 3. DNS configuration

Create two DNS records at your provider:

| Type | Name | Value | TTL |
|------|------|-------|-----|
| A    | `@`  | `YOUR_VPS_IP` | 300 |
| A    | `*`  | `YOUR_VPS_IP` | 300 |

- `@` makes `testbed.mk` resolve → VPS.
- `*` (wildcard) makes **every** `*.testbed.mk` resolve → VPS. Required for
  tenant subdomains and for the wildcard cert's DNS-01 **ACME validation
  domain** (`_acme-challenge.testbed.mk`).

Verify:

```bash
dig testbed.mk +short
dig random.testbed.mk +short      # both must print YOUR_VPS_IP
```

> The DNS-01 challenge for a wildcard cert needs an `_acme-challenge.testbed.mk`
> **TXT** record which the DNS client creates automatically via API. The
> Contabo API credentials in `dns_contabo.sh` must therefore have permission to
> manage `testbed.mk` records.

---

## 4. Install Coolify

Use the official installer (`coolify.io/docs/installation`) or the Vagrant-free
server method. On a fresh VPS:

```bash
curl -fsSL https://coolify.io/install | bash
```

Coolify will install Docker and its own stack (`coolify`, `coolify-db`,
`coolify-redis`, `coolify-realtime`, `coolify-sentinel`, `coolify-proxy`, …).
After installation it manages:

- its proxy configuration under **`/data/coolify/proxy/`**
- its application source under **`/data/coolify/source/`**

Finish the installer in the browser and log in to the dashboard.

---

## 5. Create the project & database

1. Dashboard → **+ Add New Project** → name it `SpomeniQR`.
2. Inside the project → **+ Add New Resource → Database → PostgreSQL**:
   - Name: `spomeniqr-db`
   - PostgreSQL version: **16**
   - Database name: `monuments`
   - Username: `postgres`
   - Password: strong, unique
3. **Deploy**. Copy the **Internal Connection String**, e.g.:
   `postgresql://postgres:PASSWORD@spomeniqr-db:5432/monuments`
   (the host is the Coolify-internal service name `spomeniqr-db`, not `localhost`).

---

## 6. Add the application

1. Inside the project → **+ Add New Resource → Application**.
2. Select **Public Repository** (or **Private** with a deploy key).
   - Name: `spomeniqr`
   - Repository URL: your Git URL
   - Branch: `main`
   - **Build Pack: Docker** (uses the repo's `Dockerfile`; verified path).
3. Wire the database to the app on the shared Coolify network (`quick connect`
   button or ensure both are on `coolify` network).

The repo `Dockerfile` already handles `prisma generate`, `next build`,
`prisma migrate deploy`, and provisioning the `super` admin on startup, so no
custom build/start commands are needed.

---

## 7. Environment variables

In the app's **Environment Variables** panel add (mark **every** `NEXT_PUBLIC_*`
as a **Build Variable**):

```env
# Database — Coolify internal hostname
DATABASE_URL=postgresql://postgres:PASSWORD@spomeniqr-db:5432/monuments

# Clerk
NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY=pk_live_...
CLERK_SECRET_KEY=sk_live_...
NEXT_PUBLIC_CLERK_FAPI_HOST=clerk.testbed.mk      # build var, only if using custom frontend API domain
NEXT_PUBLIC_CLERK_SIGN_IN_URL=/sign-in
NEXT_PUBLIC_CLERK_SIGN_UP_URL=/sign-up
NEXT_PUBLIC_CLERK_SIGN_IN_FALLBACK_REDIRECT_URL=/
NEXT_PUBLIC_CLERK_SIGN_UP_FALLBACK_REDIRECT_URL=/

# Contabo S3
S3_ENDPOINT=https://eu2.contabostorage.com
S3_REGION=eu-2
S3_ACCESS_KEY_ID=your-access-key
S3_SECRET_ACCESS_KEY=your-secret-key
S3_BUCKET_NAME=monuments-images

# App
NEXT_PUBLIC_APP_URL=https://testbed.mk
NEXT_PUBLIC_APP_DOMAIN=testbed.mk

# Admin session signing secret (openssl rand -hex 32)
ADMIN_SESSION_SECRET=your-random-64-char-secret

# Super-admin (bcrypt hash, NOT plaintext)
SUPER_ADMIN_USERNAME=super
SUPER_ADMIN_PASSWORD_HASH=$2b$12$REPLACE_WITH_BCRYPT_HASH

# Node
NODE_ENV=production
```

**Critical rules**

- `NEXT_PUBLIC_*` variables are inlined at `next build`. Changing one requires a
  **redeploy** (not just a restart). `NEXT_PUBLIC_CLERK_FAPI_HOST` in particular
  must be a build var or the build-time CSP blocks Clerk JS.
- `SUPER_ADMIN_PASSWORD_HASH` is passed **literally** by Coolify — paste the raw
  `$2b$12$…` hash with no `$$` escaping (that escaping is only for `env_file`
  Compose usage). Generate with:
  `node -e "import('bcryptjs').then(b => b.default.hash('YOUR_PASSWORD', 12).then(console.log))"`
- Server-only vars (`CLERK_SECRET_KEY`, `DATABASE_URL`, `ADMIN_SESSION_SECRET`,
  `SUPER_ADMIN_PASSWORD_HASH`, `S3_*`) stay as normal runtime vars.
- `ADMIN_SESSION_SECRET` missing ⇒ super-admin login returns **500** at boot.

---

## 8. Domain & subdomain routing (Traefik labels)

Coolify's proxy in this project is **Traefik**. Routing to the app is defined by
**Docker labels in the repo's `docker-compose.yaml`**, which Coolify applies
when it deploys the container. Keep that file in sync in the repo:

```yaml
services:
  app:
    build:
      context: .
      dockerfile: Dockerfile
    restart: unless-stopped
    env_file:
      - .env
    networks:
      - coolify
    labels:
      - traefik.enable=true
      - traefik.docker.network=coolify
      - traefik.http.routers.testbed-root.rule=Host(`testbed.mk`)
      - traefik.http.routers.testbed-root.entryPoints=https
      - traefik.http.routers.testbed-root.service=spomeniqr-svc
      - traefik.http.routers.testbed-root.tls=true
      - traefik.http.routers.testbed-wildcard.rule=HostRegexp(`^[a-z0-9-]+\.testbed\.mk$$`)
      - traefik.http.routers.testbed-wildcard.entryPoints=https
      - traefik.http.routers.testbed-wildcard.service=spomeniqr-svc
      - traefik.http.routers.testbed-wildcard.tls=true
      - traefik.http.routers.testbed-root-http.rule=Host(`testbed.mk`)
      - traefik.http.routers.testbed-root-http.entryPoints=http
      - traefik.http.routers.testbed-root-http.middlewares=redirect-to-https
      - traefik.http.routers.testbed-root-http.service=spomeniqr-svc
      - traefik.http.routers.testbed-wildcard-http.rule=HostRegexp(`^[a-z0-9-]+\.testbed\.mk$$`)
      - traefik.http.routers.testbed-wildcard-http.entryPoints=http
      - traefik.http.routers.testbed-wildcard-http.middlewares=redirect-to-https
      - traefik.http.routers.testbed-wildcard-http.service=spomeniqr-svc
      - traefik.http.services.spomeniqr-svc.loadbalancer.server.port=3000
      - traefik.http.middlewares.gzip.compress=true
      - traefik.http.middlewares.redirect-to-https.redirectscheme.scheme=https

networks:
  coolify:
    external: true
```

**Traefik v3 `HostRegexp` — critical syntax note**

- ✅ v3-valid: `HostRegexp(`^[a-z0-9-]+\.testbed\.mk$`)`
- ❌ v2 (removed in v3): `HostRegexp(`{subdomain:[a-zA-Z0-9-]+}.testbed.mk`)`

The v2 named-capture form silently fails to parse under Traefik 3, so the
wildcard router is **not created** and subdomains fall through to Coolify's
default **503** page ("server not available") even though the apex works. **Do
not** reintroduce the v2 form.

In `docker-compose.yaml` the trailing `$` anchor must be written as `$$` so YAML
interpolation yields a literal `$`. After editing, the container must be
**recreated** for the new labels to take effect (a plain restart is not enough).

**Effect of the two HTTPS routers**

- `Host(\`testbed.mk\`)` → apex landing page.
- `HostRegexp(\`^[a-z0-9-]+\.testbed\.mk$\`)` → **every** subdomain
  (`perop`, `test-memorial`, …) → same `app:3000`. The app's middleware then
  maps the subdomain to a memorial page.
- Both are `tls=true` **without a certresolver**, so Traefik uses the
  **default certificate** — which is the wildcard cert from
  [§9 Subdomain certificates](#9-subdomain-certificates-wildcard-tls). No
  `tls.certresolver` label is needed on these routers.

> Because Coolify's own default routers can also be generated from the UI
> `Domains` panel, prefer adding `testbed.mk` / `*.testbed.mk` there **or**
> relying solely on these labels — avoid having both apply the same rule, which
> creates duplicate-router ambiguity.

---

## 9. Subdomain certificates (wildcard TLS)

This section is dedicated to provisioning and renewing the TLS certificate that
secures **all** `*.testbed.mk` subdomains, since it is the part that is easiest
to get wrong.

### 9.1 Why a wildcard cert

The app issues an arbitrary number of tenant subdomains
(`<subdomain>.testbed.mk`). It is impractical to obtain an individual
certificate per subdomain. A single wildcard certificate for `*.testbed.mk`
covers every present and future tenant. Because it is issued as Traefik's
**default certificate**, any router with `tls=true` and no explicit
certresolver (which is exactly how §8 is configured) is served the wildcard
cert automatically.

### 9.2 Challenge type & provider

Wildcard certs can **only** be validated with the **DNS-01** challenge (an
`_acme-challenge.testbed.mk` TXT record — there is no HTTP path to validate
`.testbed.mk` itself). This project uses `acme.sh` with a custom **Contabo DNS**
client (`dns_contabo.sh`) because the domain is hosted on Contabo DNS.

### 9.3 Required credentials

Obtain from your Contabo customer account:

| Variable | Purpose |
|----------|---------|
| `CONTABO_CLIENT_ID` | OAuth2 Client ID |
| `CONTABO_CLIENT_SECRET` | OAuth2 Client Secret |
| `CONTABO_API_USER` | Contabo account email |
| `CONTABO_API_PASSWORD` | Contabo account password |

`dns_contabo.sh` reads these from the environment / acme.sh account conf and
uses Contabo's API (`https://api.contabo.com/v1`) to create and delete the
`_acme-challenge` TXT records needed for validation.

### 9.4 Install `dns_contabo.sh`

1. Install `acme.sh` (non-root or root):
   ```bash
   curl https://get.acme.sh | sh -s email=you@example.com
   ```
2. Place `dns_contabo.sh` where acme.sh finds API plugins. It is written to be
   self-contained (uses `acme.sh` built-ins `_post`/`_info`/`_err`), so the
   cleanest approach is:
   ```bash
   mkdir -p ~/.acme.sh/dnsapi
   cp /path/to/dns_contabo.sh ~/.acme.sh/dnsapi/
   chmod +x ~/.acme.sh/dnsapi/dns_contabo.sh
   ```

### 9.5 Export credentials

```bash
export CONTABO_CLIENT_ID="..."
export CONTABO_CLIENT_SECRET="..."
export CONTABO_API_USER="you@example.com"
export CONTABO_API_PASSWORD="..."
```

### 9.6 Issue the wildcard certificate

```bash
# Covers the apex AND every subdomain
~/.acme.sh/acme.sh --issue \
  --dns dns_contabo \
  -d testbed.mk \
  -d '*.testbed.mk'
```

acme.sh stores the result in `~/.acme.sh/testbed.mk_ecc/`:
`fullchain.cer`, `testbed.mk.key`, `testbed.mk.cer`, `ca.cer`.

> **Validation**: the `${DOMAIN}_ecc` directory must contain `testbed.mk.cer`
> and a key. If verification fails, check the DNS TXT records
> (`dig +short _acme-challenge.testbed.mk TXT`) and that the Contabo credentials
> have record permissions on the zone.

### 9.7 Traefik default-certificate config (Coolify proxy)

The proxy (`coolify-proxy`, image `traefik:v3.6`) mounts the host directory
`/data/coolify/proxy/` at `/traefik` (see [§12.2](#122-proxy-mount-layout)).
Two things must be provisioned there:

**a) Copy the cert files into Coolify's certs directory:**

```bash
COOLIFY_CERT_DIR="/data/coolify/proxy/certs"
mkdir -p "$COOLIFY_CERT_DIR"
cp ~/.acme.sh/testbed.mk_ecc/fullchain.cer "$COOLIFY_CERT_DIR/testbed.mk.cert"
cp ~/.acme.sh/testbed.mk_ecc/testbed.mk.key "$COOLIFY_CERT_DIR/testbed.mk.key"
chmod 644 "$COOLIFY_CERT_DIR/testbed.mk.cert"
chmod 600 "$COOLIFY_CERT_DIR/testbed.mk.key"
```

**b) Create the dynamic config** `/data/coolify/proxy/dynamic/testbed.mk-wildcard.yaml`
(loaded by the proxy's file provider; note paths are **inside the container**,
i.e. under `/traefik/…`):

```yaml
tls:
  certificates:
    - certFile: /traefik/certs/testbed.mk.cert
      keyFile: /traefik/certs/testbed.mk.key
  stores:
    default:
      defaultCertificate:
        certFile: /traefik/certs/testbed.mk.cert
        keyFile: /traefik/certs/testbed.mk.key
```

The `defaultCertificate` under `tls.stores.default` makes this the cert Traefik
serves for any TLS router that has no explicit certresolver — including both
`testbed-root` and `testbed-wildcard` (§8). A change here requires a proxy
reload (restarting `coolify-proxy` picks it up).

### 9.8 Install with acme.sh for auto-renewal

Register the copy + reload steps so acme.sh performs them automatically every
renewal (certs expire every ~90 days):

```bash
~/.acme.sh/acme.sh --install-cert -d testbed.mk -d '*.testbed.mk' \
  --fullchain-file "/data/coolify/proxy/certs/testbed.mk.cert" \
  --key-file "/data/coolify/proxy/certs/testbed.mk.key" \
  --reloadcmd "docker restart coolify-proxy"
```

- `--fullchain-file`/`--key-file` tell acme.sh to **copy** the renewed certs
  into Coolify's certs directory on every renewal.
- `--reloadcmd` restarts the proxy so Traefik reloads the files.
- acme.sh installs its own cron job for renewal, so no separate crontab is
  needed.

Optionally, a dedicated renewal hook script can be used (kept in
`~/.acme.sh/renewal-hooks/testbed.mk-coolify-reload.sh`) that copies both files
and runs `docker restart coolify-proxy`; advantages are clearer logging and a
manual script you can run to force a reload after a manual cert update.

### 9.9 Verify the wildcard cert is live

```bash
# SANs must list both testbed.mk and *.testbed.mk
openssl x509 -in /data/coolify/proxy/certs/testbed.mk.cert -noout -text | grep -A2 "Subject Alternative Name"

# Traefik default cert actually offered on a subdomain
echo | openssl s_client -connect perop.testbed.mk:443 -servername perop.testbed.mk 2>/dev/null \
  | openssl x509 -noout -subject -issuer

# Next renewal date
~/.acme.sh/acme.sh --list | grep testbed.mk
```

### 9.10 Renewal policy summary

| Item | Value |
|------|-------|
| Issuer | Let's Encrypt via `acme.sh` (DNS-01, `dns_contabo`) |
| Covered | `testbed.mk` + `*.testbed.mk` |
| Lifetime | 90 days; auto-renewed by acme.sh cron |
| Destination | `/data/coolify/proxy/certs/testbed.mk.{cert,key}` |
| Reload | `--reloadcmd docker restart coolify-proxy` |
| Served as | Traefik **default certificate** (`tls.stores.default.defaultCertificate`) |

> **Do NOT** rely on Coolify's built-in Let's Encrypt `letsencrypt` certresolver
> for the wildcard. Coolify's default uses an HTTP-01 challenge, which cannot
> validate a wildcard, and Traefik v3 does not auto-provision per-subdomain certs
> from `HostRegexp` routers. The manual DNS-01 wildcard + default-cert approach
> is the correct one for this app.

---

## 10. Deploy the application

1. Ensure `[§7 env vars]` are saved and all `NEXT_PUBLIC_*` are marked as build
   variables **before** the first build.
2. Click **Deploy**. Watch the build log to confirm, in order:
   - dependencies installed,
   - `npx prisma generate`,
   - `next build`,
   - `npx prisma migrate deploy`,
   - `[seed] Super-admin 'super' provisioned.`,
   - server listening on port **3000**.
3. If the routing (Traefik) labels changed (e.g. §8 edits in the repo), triggers
   a **recreate** so the new labels apply.

---

## 11. Verify everything

```bash
# Apex
curl -s -o /dev/null -w "%{http_code}\n" https://testbed.mk          # 200

# Subdomain / wildcard routing + default cert
curl -s -o /dev/null -w "%{http_code}\n" https://perop.testbed.mk    # 200

# TLS on a subdomain uses the wildcard cert (no browser error)
echo | openssl s_client -connect perop.testbed.mk:443 -servername perop.testbed.mk 2>/dev/null \
  | openssl x509 -noout -subject

# Proxy healthy + no docker-provider errors
docker logs coolify-proxy --since 1m 2>&1 | grep -iE "too old|error" | tail

# API health
curl -s https://testbed.mk/api/check-subdomain?slug=test | python3 -m json.tool

# Admin login (expects {"success":true})
curl -s -X POST https://testbed.mk/api/admin/login -H "Content-Type: application/json" \
  -d '{"username":"super","password":"YOUR_PASSWORD"}'
```

Browser checks

- `https://testbed.mk` → landing page.
- Publish a memorial with subdomain `perop`, then `https://perop.testbed.mk`
  → the memorial page with a green lock (no certificate warning).
- DevTools → Network: no CSP violation for `clerk.testbed.mk/npm/...` when the
  custom Clerk frontend domain is used.

---

## 12. Troubleshooting

### 12.1 Traefik cannot talk to Docker 29 ("client version 1.24 is too old")

**Symptom**: `coolify-proxy` logs repeat:

```
ERR Failed to retrieve information of the docker client ...
error="Error response from daemon: client version 1.24 is too old.
Minimum supported API version is 1.40, please upgrade your client"
providerName=docker
```

and **every** domain (apex + subdomains) returns 503.

**Cause**: Docker 29 raised its minimum API version (here **1.40**); Traefik
`v3.1` runs a Docker client pinned to **API 1.24** and never negotiates, so the
Docker provider — which discovers the app's label routers — is dead. It is not a
config problem. Coolify's default proxy (image `traefik:v3.1`) hits this on a
Docker 29 host.

**Fix**: pin the proxy image to **Traefik v3.6+** (adds Docker API auto-
negotiation). The proxy compose lives at `/data/coolify/proxy/docker-compose.yml`
(the service is named `traefik`):

```bash
sed -i "s@traefik:v3.1@traefik:v3.6@" /data/coolify/proxy/docker-compose.yml
grep 'image:' /data/coolify/proxy/docker-compose.yml
docker compose -f /data/coolify/proxy/docker-compose.yml up -d --force-recreate traefik
docker exec coolify-proxy traefik version   # expect 3.6.x
```

Confirm the provider is healthy and routing resumes:

```bash
docker logs coolify-proxy --since 30s 2>&1 | grep -i "too old"   # should be empty
curl -s -o /dev/null -w "%{http_code}\n" https://testbed.mk      # 200
```

A future Coolify **update may regenerate the proxy compose back to `v3.1`**.
Re-apply the `sed` above after any Coolify update, or (as a durable stopgap)
lower the daemon's minimum API so the 1.24 client is accepted by adding
`"min-api-version": "1.24"` to `/etc/docker/daemon.json` and restarting Docker:

```bash
# /etc/docker/daemon.json:  { ..., "min-api-version": "1.24" }
systemctl restart docker
```

Prefer the v3.6 image fix — it touches only the proxy, not the whole daemon.

### 12.2 Proxy mount layout

`coolify-proxy` mounts the host directory `/data/coolify/proxy/` at `/traefik`.
Everything the proxy reads must therefore be expressed with in-container paths:

- cert files inside the container: `/traefik/certs/*`
- dynamic config inside the container: `/traefik/dynamic/*`
- ACME store: `/traefik/acme.json`

Verify the mount and files:

```bash
docker inspect coolify-proxy --format '{{json .Mounts}}'
docker exec coolify-proxy ls -la /traefik/certs /traefik/dynamic
```

### 12.3 Subdomain shows "server not available" (503) while apex works

- **Most common: v2 `HostRegexp` label.** Ensure both
  `testbed-wildcard` and `testbed-wildcard-http` use the v3 form
  `HostRegexp(\`^[a-z0-9-]+\.testbed\.mk$$\`)` (see §8). Recreate the container
  after editing labels.
- **Docker 29 / Traefik v3.1**: see §12.1.
- **DNS**: `dig perop.testbed.mk +short` must return the VPS IP.
- **Duplicate routers**: if you also added `*.testbed.mk` in Coolify's UI
  `Domains`, you may have overlapping routers — prefer one method.

### 12.4 Subdomain loads but shows a certificate error

- Wildcard default cert not loaded / stale. Check §9.7 layout and paths, then
  `docker restart coolify-proxy`.
- Cert files not renewed: rerun `~/.acme.sh/acme.sh --renew -d testbed.mk -d '*.testbed.mk' --force`
  and check the SANs (must include `*.testbed.mk`).

### 12.5 Build fails with Prisma errors

`DATABASE_URL` must use the internal hostname `spomeniqr-db:5432`, not
`localhost`, and the app + database must be on the shared Coolify network.

### 12.6 Super-admin login fails

- 401: `SUPER_ADMIN_PASSWORD_HASH` pasted with `$$` escaping → re-paste the raw
  `$2b$12$…` hash and redeploy.
- 500 / `ADMIN_SESSION_SECRET env var is not set` → set `ADMIN_SESSION_SECRET`
  (runtime) and restart.
- Check boot log for `[seed] Super-admin 'super' provisioned.`

### 12.7 CSP blocks Clerk JS (auth buttons dead)

Set `NEXT_PUBLIC_CLERK_FAPI_HOST` (e.g. `clerk.testbed.mk`) as a **build
variable** and **redeploy** — the CSP is generated at build time.

---

## 13. Maintenance & commands

| Action | Command / location |
|--------|--------------------|
| Redeploy app | Project → Application → **Deploy** |
| Re-apply proxy v3.6 pin | `sed -i 's@traefik:v3.1@traefik:v3.6@' /data/coolify/proxy/docker-compose.yml && docker compose -f /data/coolify/proxy/docker-compose.yml up -d --force-recreate traefik` |
| Restart proxy | `docker restart coolify-proxy` |
| Proxy logs | `docker logs coolify-proxy -f` |
| App logs | Project → Application → **Logs** / `docker logs <app-container> -f` |
| Force cert renewal | `~/.acme.sh/acme.sh --renew -d testbed.mk -d '*.testbed.mk' --force` |
| Check cert status | `~/.acme.sh/acme.sh --list` |
| Migrations | auto on container start; manual: `docker exec <app> npx prisma migrate deploy` |
| DB backup | Coolify UI database backup, or `pg_dump` against `spomeniqr-db` |
| Update app | push to Git; Coolify redeploys (or click **Deploy**) |

---

### Environment variables quick reference

| Variable | Req | Notes |
|----------|-----|-------|
| `DATABASE_URL` | Yes | Coolify-internal, host `spomeniqr-db` |
| `NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY` | Yes | build var; `pk_live_…` |
| `NEXT_PUBLIC_CLERK_FAPI_HOST` | No* | build var; custom Clerk domain |
| `CLERK_SECRET_KEY` | Yes | runtime; `sk_live_…` |
| `NEXT_PUBLIC_CLERK_*` paths | Yes | build vars (see §7) |
| `S3_ENDPOINT` / `S3_REGION` | Yes | Contabo object storage |
| `S3_ACCESS_KEY_ID` / `S3_SECRET_ACCESS_KEY` | Yes | Contabo API keys |
| `S3_BUCKET_NAME` | Yes | `monuments-images` |
| `NEXT_PUBLIC_APP_URL` | Yes | build var; `https://testbed.mk` |
| `NEXT_PUBLIC_APP_DOMAIN` | Yes | build var; `testbed.mk` |
| `ADMIN_SESSION_SECRET` | Yes | runtime; `openssl rand -hex 32` |
| `SUPER_ADMIN_USERNAME` / `SUPER_ADMIN_PASSWORD_HASH` | No | bcrypt hash, literal |
| `NODE_ENV` | Yes | `production` |

\* only when using a custom Clerk frontend API domain (`clerk.testbed.mk`).