first commit

2026-10-09 02:48:31 +07:00
commit 8db22bfaa6
17 changed files with 1722 additions and 0 deletions
@@ -0,0 +1,84 @@
+# Deploy: stack `sicabe-host` (tanpa build)
+
+Satu compose untuk satu host SiCABE, memuat empat service:
+
+| Service | Image | Profile | Di host mana |
+| --- | --- | --- | --- |
+| `docker-socket-proxy` | `tecnativa/docker-socket-proxy` | – | **semua** host |
+| `config-agent` | `<registry>/config-agent` | – | **semua** host |
+| `gateway` (nginx) | `nginx:alpine` | – | **semua** host |
+| `sicabe-control-panel` | `<registry>/control-panel` | `panel` | **satu** host (panel) |
+
+Compose di sini **tidak membangun image** — semua diambil dari registry. Untuk
+build & push image ke registry privat, pakai `deploy/panel/` + `scripts/build-image.sh`.
+
+## 1. Build & push image (sekali, dari mesin dev)
+
+```bash
+./scripts/build-image.sh --all --push --registry registry.example.com/sicabe
+# → registry.example.com/sicabe/control-panel:<git-sha> + :latest
+# → registry.example.com/sicabe/config-agent:<git-sha> + :latest
+```
+
+`--all` membangun **panel + config-agent** dengan tag yang sama (agar satu
+`SICABE_TAG` cukup di stack). Login registry dulu bila perlu: `docker login <registry>`.
+
+## 2. Siapkan env (per host)
+
+```bash
+cd deploy/stack
+cp env.example .env       # isi GATEWAY_API_KEY, SICABE_REGISTRY, SICABE_DEPLOY_DIR
+```
+
+Di host **panel**, isi juga bagian bawah (`PANEL_URL`, `NEXTAUTH_SECRET`,
+`KEYCLOAK_CLIENT_SECRET`, `SICABE_HOSTS`, dst).
+
+## 3. Jalankan
+
+```bash
+# SEMUA host SiCABE (docker-proxy + config-agent + gateway):
+docker compose up -d
+
+# HANYA di host panel — sekaligus menjalankan panelnya:
+docker compose --profile panel up -d
+```
+
+Cek: `docker compose ps` dan `docker compose logs -f gateway`.
+
+## Update image
+
+```bash
+# setelah build+push tag baru:
+SICABE_TAG=<tag-baru> docker compose pull        # (di host panel: + --profile panel)
+docker compose up -d        # recreate
+```
+
+## Menghentikan / menghapus
+
+`docker compose down` hanya menghapus service **pada profile yang aktif**. Karena
+itu, di host panel jalankan dengan profile yang sama:
+
+```bash
+# host panel:
+docker compose --profile panel down      # hapus SEMUA (termasuk panel)
+
+# host non-panel:
+docker compose down
+```
+
+> Kalau `down` dijalankan tanpa `--profile panel` di host panel, container
+> `sicabe-control-panel` akan tertinggal sebagai *orphan*. Compose akan
+> memperingatkan; hapus dengan `--profile panel down` atau `docker rm -f`.
+
+## Catatan
+
+- **Panel memakai `network_mode: host`** karena gateway bind `127.0.0.1` (tak
+  terjangkau dari bridge Docker; sekaligus SAN sertifikat Trino `localhost`
+  cocok). Port panel = `PANEL_PORT` (default 3000).
+- **Template nginx dipakai bersama** dari `../host-gateway/nginx/` (tanpa
+  duplikasi). Setelah mengubah template, jalankan `docker compose restart gateway`
+  (nginx men-substitute template saat start).
+- **Menambah host baru**: jalankan `docker compose up -d` (tanpa profile) di host
+  itu, lalu daftarkan di panel (**Settings → Hosts & Agents**) dan **Scan**.
+- Compose ini **menggantikan** `deploy/host-gateway/` + `deploy/panel/` sebagai
+  artefak deploy. Keduanya tetap ada untuk **build lokal**/dev.
@@ -0,0 +1,178 @@
+# =============================================================================
+# sicabe-host — SATU compose untuk satu host SiCABE.
+#
+# Berisi: docker-socket-proxy + config-agent + gateway(nginx) + control-panel.
+#
+# PROFILE:
+#   - tanpa profile  → docker-socket-proxy + config-agent + gateway
+#                      (WAJIB di SEMUA host SiCABE)
+#   - profile panel  → + sicabe-control-panel
+#                      (hanya di SATU host: host panel)
+#
+# Pemakaian:
+#   # di SEMUA host SiCABE:
+#   docker compose up -d
+#
+#   # di HOST PANEL (selain di atas):
+#   docker compose --profile panel up -d
+#
+# Env: salin `env.example` → `.env` (compose membacanya otomatis).
+# Image TIDAK di-build di sini — pakai image dari registry:
+#   SICABE_REGISTRY=<registry>  SICABE_TAG=<tag>   (lihat scripts/build-image.sh --all --push)
+#
+# Lihat README.md.
+# =============================================================================
+
+name: sicabe-host
+
+services:
+  # ── Docker Engine API (ACL ketat, internal saja) ──────────────────────────
+  docker-proxy:
+    image: tecnativa/docker-socket-proxy:v0.5.0
+    container_name: sicabe-docker-proxy
+    environment:
+      CONTAINERS: "1"
+      POST: "1"
+      INFO: "1"
+      SYSTEM: "1"
+      EXEC: "0"
+      SWARM: "0"
+      SERVICES: "0"
+      SECRETS: "0"
+      CONFIGS: "0"
+      NETWORKS: "0"
+      VOLUMES: "0"
+      IMAGES: "0"
+      BUILD: "0"
+      GRPC: "0"
+    volumes:
+      - /var/run/docker.sock:/var/run/docker.sock:ro
+    expose:
+      - "2375"
+    read_only: true
+    tmpfs:
+      - /tmp
+      - /run
+    restart: unless-stopped
+    logging:
+      driver: json-file
+      options:
+        max-size: "10m"
+        max-file: "3"
+
+  # ── Konfigurasi platform per host (internal saja) ─────────────────────────
+  config-agent:
+    image: ${SICABE_REGISTRY:-sicabe}/config-agent:${SICABE_TAG:-latest}
+    container_name: sicabe-config-agent
+    # Jalan sebagai UID/GID host agar file yang ditulis (katalog Trino, data OPA)
+    # tetap milik user host, bukan root.
+    user: "${AGENT_UID:-1000}:${AGENT_GID:-1000}"
+    environment:
+      AGENT_API_KEY: ${GATEWAY_API_KEY:?GATEWAY_API_KEY wajib diisi (env)}
+      AGENT_PORT: "8080"
+      PYTHONDONTWRITEBYTECODE: "1"
+      SICABE_ENV_FILE: /config/.env
+      TRINO_CATALOG_DIR: /config/config/trino/catalog
+      DBT_PROJECT_DIR: /config/config/airflow/dbt/sicabe
+      OPA_DATA_FILE: /config/config/opa/data.json
+      # Nama file PEM di dalam /ca (lihat AGENT_CA_HOST_PATH). Diperlukan agar
+      # panel bisa memverifikasi TLS Trino (self-signed).
+      AGENT_CA_FILE: ${AGENT_CA_FILE:-/ca/trino.crt}
+      # Sumber daftar container untuk scan (docker-proxy internal).
+      DOCKER_PROXY_URL: http://docker-proxy:2375
+    volumes:
+      - ${SICABE_DEPLOY_DIR:?set SICABE_DEPLOY_DIR (env)}:/config:ro
+      - ${SICABE_DEPLOY_DIR:?}/config/trino/catalog:/config/config/trino/catalog:rw
+      - ${SICABE_DEPLOY_DIR:?}/config/opa:/config/config/opa:rw
+      # CA TLS (GET /ca): direktori berisi PEM → /ca. WAJIB diisi di host
+      # panel agar TLS Trino (self-signed) bisa diverifikasi.
+      - ${AGENT_CA_HOST_PATH:-${SICABE_DEPLOY_DIR:?}}:/ca:ro
+    expose:
+      - "8080"
+    read_only: true
+    tmpfs:
+      - /tmp
+    restart: unless-stopped
+    logging:
+      driver: json-file
+      options:
+        max-size: "10m"
+        max-file: "3"
+
+  # ── Gateway nginx: satu-satunya port ke host ──────────────────────────────
+  gateway:
+    image: nginx:1.30.4-alpine3.24
+    container_name: sicabe-host-gateway
+    depends_on:
+      - docker-proxy
+      - config-agent
+    environment:
+      GATEWAY_API_KEY: ${GATEWAY_API_KEY:?GATEWAY_API_KEY wajib diisi (env)}
+    volumes:
+      # Template nginx dipakai bersama dari deploy/host-gateway (tanpa duplikasi).
+      - ../host-gateway/nginx/gateway.conf.template:/etc/nginx/templates/default.conf.template:ro
+    ports:
+      - "${GATEWAY_BIND_IP:-127.0.0.1}:${GATEWAY_PORT:-22375}:80"
+    restart: unless-stopped
+    logging:
+      driver: json-file
+      options:
+        max-size: "10m"
+        max-file: "3"
+
+  # ── Control panel — HANYA di host panel (profile "panel") ────────────────
+  panel:
+    profiles: ["panel"]
+    image: ${SICABE_REGISTRY:-sicabe}/control-panel:${SICABE_TAG:-latest}
+    container_name: sicabe-control-panel
+    depends_on:
+      - gateway
+    # Host network: gateway host bind 127.0.0.1 → tak terjangkau dari bridge,
+    # dan SAN sertifikat Trino (localhost) cocok. Port via PORT.
+    network_mode: host
+    environment:
+      NODE_ENV: production
+      HOSTNAME: "0.0.0.0"
+      PORT: ${PANEL_PORT:-3000}
+      # ── Wajib (panel gagal-cepat bila kosong) ──
+      NEXTAUTH_URL: ${PANEL_URL:-}
+      NEXTAUTH_SECRET: ${NEXTAUTH_SECRET:-}
+      KEYCLOAK_CLIENT_SECRET: ${KEYCLOAK_CLIENT_SECRET:-}
+      SICABE_HOSTS: ${SICABE_HOSTS:-}
+      # ── Login lokal (hybrid, opsional) ──
+      # Akun lokal berlaku juga di produksi. Pakai varian _B64 (bcrypt memuat
+      # '$' yang bisa di-interpolasi Compose). Kosong = hanya SSO.
+      PANEL_LOCAL_USER: ${PANEL_LOCAL_USER:-}
+      PANEL_LOCAL_PASSWORD_HASH: ${PANEL_LOCAL_PASSWORD_HASH:-}
+      PANEL_LOCAL_PASSWORD_HASH_B64: ${PANEL_LOCAL_PASSWORD_HASH_B64:-}
+      PANEL_LOCAL_ROLE: ${PANEL_LOCAL_ROLE:-}
+      # Identitas Trino untuk sesi lokal (tanpa JWT) — service account Basic.
+      TRINO_SERVICE_ACCOUNT: ${TRINO_SERVICE_ACCOUNT:-}
+      # ── Opsional (ada default di kode) ──
+      KEYCLOAK_URL: ${KEYCLOAK_URL:-}
+      KEYCLOAK_CLIENT_ID: ${KEYCLOAK_CLIENT_ID:-sicabe-control-panel}
+      KEYCLOAK_WEB_URL: ${KEYCLOAK_WEB_URL:-}
+      TRINO_URL: ${TRINO_URL:-}
+      TRINO_WEB_URL: ${TRINO_WEB_URL:-}
+      NESSIE_URL: ${NESSIE_URL:-}
+      AIRFLOW_URL: ${AIRFLOW_URL:-}
+      STARROCKS_HOST: ${STARROCKS_HOST:-}
+      STARROCKS_PORT: ${STARROCKS_PORT:-}
+      STARROCKS_WEB_URL: ${STARROCKS_WEB_URL:-}
+      OPENMETADATA_URL: ${OPENMETADATA_URL:-}
+      GRAFANA_URL: ${GRAFANA_URL:-}
+      MINIO_CONSOLE_URL: ${MINIO_CONSOLE_URL:-}
+      SICABE_COMPOSE_PROJECTS: ${SICABE_COMPOSE_PROJECTS:-sicabe}
+      SICABE_CONFIG_FILE: data/config.json
+      SICABE_STATE_FILE: data/state.json
+    volumes:
+      - panel-data:/app/data
+    restart: unless-stopped
+    logging:
+      driver: json-file
+      options:
+        max-size: "10m"
+        max-file: "3"
+
+volumes:
+  panel-data:
@@ -0,0 +1,77 @@
+# =============================================================================
+# Env untuk stack sicabe-host (satu file untuk SEMUA host).
+# Salin ke .env lalu isi nilainya:  cp env.example .env
+#
+# Dipakai oleh: docker-socket-proxy + config-agent + gateway (semua host),
+# dan control-panel (hanya host panel / profile "panel").
+# =============================================================================
+
+# ── Image dari registry (WAJIB) ──────────────────────────────────────────────
+# Hasil: ./scripts/build-image.sh --all --push --registry <registry>
+# Kosongkan SICABE_REGISTRY bila memakai image lokal (mis. saat uji).
+#SICABE_REGISTRY=registry.example.com/sicabe
+SICABE_TAG=latest
+
+# ── Gateway (WAJIB, semua host) ──────────────────────────────────────────────
+# Satu API key untuk docker-proxy + config-agent (diperiksa nginx).
+GATEWAY_API_KEY=
+# Bind ke 127.0.0.1 (default, aman) atau IP internal. JANGAN ke internet.
+GATEWAY_BIND_IP=127.0.0.1
+GATEWAY_PORT=22375
+
+# ── Config-agent (WAJIB, semua host) ─────────────────────────────────────────
+# Direktori deploy sicabe-doc di host ini (berisi .env, config/trino/catalog, ...).
+SICABE_DEPLOY_DIR=/path/ke/sicabe-doc/deploy
+# UID/GID host yang menulis file (katalog/OPA) — agar tidak root-owned.
+AGENT_UID=1000
+AGENT_GID=1000
+# CA TLS Trino — WAJIB di host panel agar panel memverifikasi sertifikat Trino
+# (self-signed). Isi direktori yang memuat PEM-nya (dihasilkan
+# ./scripts/export-trino-ca.sh → <panel>/.ca/trino.crt).
+# Di host NON-panel boleh dikosongkan (panel-lah yang butuh CA).
+AGENT_CA_HOST_PATH=/path/ke/sicabe-control-panel/.ca
+AGENT_CA_FILE=/ca/trino.crt
+
+# =============================================================================
+# Di bawah ini HANYA untuk host panel (profile "panel").
+# Boleh dibiarkan kosong di host non-panel (compose tetap valid).
+# =============================================================================
+
+# ── Panel: wajib ─────────────────────────────────────────────────────────────
+# URL publik panel (harus sama dengan yang diakses browser & redirect URI Keycloak).
+PANEL_URL=http://localhost:3000
+PANEL_PORT=3000
+# openssl rand -base64 32
+NEXTAUTH_SECRET=
+
+# ── Login lokal (hybrid dengan SSO) — OPSIONAL ───────────────────────────────
+# Akun lokal tanpa Keycloak; password = bcrypt hash. Pakai varian _B64 di sini
+# karena Compose meng-interpolasi '$' (bcrypt selalu memuat '$').
+#   node -e "console.log(Buffer.from(require('bcryptjs').hashSync('PASSWORD',10)).toString('base64'))"
+#PANEL_LOCAL_USER=admin
+#PANEL_LOCAL_PASSWORD_HASH_B64=
+#PANEL_LOCAL_ROLE=data-admin
+# Identitas Trino utk sesi lokal (tanpa JWT) — service account (Basic).
+#TRINO_SERVICE_ACCOUNT=svc_dbt
+# Client Keycloak panel (scripts/create-keycloak-client.sh).
+KEYCLOAK_CLIENT_SECRET=dev-keycloak-secret
+# Host/agent yang dikelola panel. WAJIB ada `configUrl` (config-agent).
+# Di host panel (network host) gateway lokal = 127.0.0.1:${GATEWAY_PORT}.
+# Format JSON tanpa kutip pembungkus (hindari masalah parsing .env).
+SICABE_HOSTS=[{"name":"local-gw","proxyUrl":"http://127.0.0.1:22375","apiKey":"CHANGE_ME","configUrl":"http://127.0.0.1:22375","configApiKey":"CHANGE_ME","publicHost":"localhost","enabled":true}]
+
+# ── Panel: opsional (ada default di kode) ────────────────────────────────────
+KEYCLOAK_URL=http://127.0.0.1:12080
+KEYCLOAK_CLIENT_ID=sicabe-control-panel
+KEYCLOAK_WEB_URL=http://localhost:12080
+TRINO_URL=https://localhost:12443
+TRINO_WEB_URL=https://localhost:12443
+NESSIE_URL=http://127.0.0.1:12120
+AIRFLOW_URL=http://127.0.0.1:12082
+STARROCKS_HOST=127.0.0.1
+STARROCKS_PORT=12930
+STARROCKS_WEB_URL=http://localhost:12030
+OPENMETADATA_URL=http://127.0.0.1:12085
+GRAFANA_URL=http://127.0.0.1:12300
+MINIO_CONSOLE_URL=http://localhost:12901
+SICABE_COMPOSE_PROJECTS=sicabe