platform/sicabe-core
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

first commit

Browse Source
This commit is contained in:
subroto
2026-10-09 02:35:14 +07:00
commit 4ca7a9a58e
108 changed files with 23035 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
deploy/README.md
+697
View File
@@ -0,0 +1,697 @@
# eBdesk SiCABE — `deploy` (Installer Hasil Merge Eksperimen)
**SiCABE** (Solusi Integrasi Cerdas Adaptif Big Data Enterprise) adalah platform
data modern 5-lapisan (lakehouse) untuk analisis harga dan logistik kebutuhan
pokok. Folder `deploy/` adalah **installer hasil merge** dari:
- `deploy-legacy/` — base installer lama (yang sudah berjalan baik),
- `experiments/nessie-dremio/` — **Nessie** (ganti Polaris) + **OPA** + **Dremio**,
- `experiments/starrocks-minio/` — StarRocks **shared-data** (FE + Compute Node),
- `experiments/starrocks-oidc/` — **LDAP group provider**.
> `deploy-legacy/` (base lama) **tidak diubah** — tetap menjadi referensi/rollback.
> Dokumen rencana & mapping: [`../reviews/sicabe-07-merge-experiments-plan.md`](../reviews/sicabe-07-merge-experiments-plan.md).
## Arsitektur
```
L1 INGESTION Airflow (base) ── dbt/Sling ──► Trino
L2 STORAGE MinIO (base) + Nessie (JDBC2/PostgreSQL) + OpenLDAP (identitas)
L3 PROCESSING Trino 483 ──► Nessie catalog; authz Trino ──► OPA
L4 SERVING StarRocks shared-data (FE + CN) | Dremio OSS 25.2.0
L5 GOVERNANCE Keycloak (user lokal) · OpenMetadata · Grafana
IDENTITY OpenLDAP (sumber grup untuk Trino & StarRocks)
DATABASE PostgreSQL shared: keycloak | nessie | airflow | openmetadata_db
```
### Dependency Graph (urutan deploy)
```
Phase 1: Keycloak + OpenLDAP ──────── (tidak ada dependensi)
Phase 2: MinIO ◄──── Nessie ───────── (MinIO dulu, baru Nessie)
Phase 3: Trino ◄──── Nessie + OPA ─── (katalog Nessie, authz OPA)
Phase 4: StarRocks FE + CN (+ Dremio opt-in) ── (shared-data; engine paralel)
Phase 5: Airflow ◄──── Postgres ───── (PostgreSQL sebagai backend DB)
Phase 6: OpenMetadata ◄──── Grafana ─
```
### Matriks Install (Fase × Layer)
**A — Service yang baru diinstall per fase:**
| Fase | Database | Identity | L1 Ingestion | L2 Storage & Catalog | L3 Processing | L4 Serving | L5 Governance |
|---|---|---|---|---|---|---|---|
| **Phase 1** | `postgres` | `openldap` | — | — | — | — | `keycloak` |
| **Phase 2** | — | — | — | `minio`, `minio-init`*, `nessie` | — | — | — |
| **Phase 3** | — | — | — | — | `opa`, `trino` | — | — |
| **Phase 4** | — | — | — | — | — | `starrocks-fe`, `starrocks-cn` | — |
| **Phase 5** | — | — | `airflow-init`, `airflow-webserver`, `airflow-dag-processor`, `airflow-scheduler` | — | — | — | — |
| **Phase 6** | — | — | — | — | — | — | `openmetadata-opensearch`, `openmetadata-migrate`*, `openmetadata-server`, `grafana` |
`*` one-shot (selesai lalu exit): `minio-init` (buat bucket), `openmetadata-migrate` (migrasi schema OM).
**B — Kondisi aktif kumulatif setelah fase N (full deploy `./install.sh`):**
| Fase | Database | Identity | L1 Ingestion | L2 Storage & Catalog | L3 Processing | L4 Serving | L5 Governance |
|---|---|---|---|---|---|---|---|
| **1** | `postgres` | `openldap` | — | — | — | — | `keycloak` |
| **2** | `postgres` | `openldap` | — | `minio`, `nessie` | — | — | `keycloak` |
| **3** | `postgres` | `openldap` | — | `minio`, `nessie` | `opa`, `trino` | — | `keycloak` |
| **4** | `postgres` | `openldap` | — | `minio`, `nessie` | `opa`, `trino` | `starrocks-fe`, `starrocks-cn` | `keycloak` |
| **5** | `postgres` | `openldap` | `airflow-*` | `minio`, `nessie` | `opa`, `trino` | `starrocks-*` | `keycloak` |
| **6** | `postgres` | `openldap` | `airflow-*` | `minio`, `nessie` | `opa`, `trino` | `starrocks-*` | `keycloak`, `openmetadata-*`, `grafana` |
> `./install.sh --phase N` menjalankan profil `phaseN`. Phase 1–4 kumulatif; phase 5–6 lebih sempit
> (disarankan full deploy berurutan). `bootstrap.sh` (9 step) dijalankan **setelah** install.
>
> **Dremio bersifat opt-in**: tidak ikut `--profile all` / `phaseN` (installer-nya
> menolak heap < 4 GB sehingga boros RAM di dev). Aktifkan dengan
> `docker compose --profile dremio up -d dremio`. `bootstrap.sh` step 9 akan
> dilewati bila container Dremio tidak berjalan.
## Stack
| Layer | Komponen | Versi | Catatan |
|-------|----------|-------|---------|
| Database | PostgreSQL | 15 | 1 instance, DB terpisah: keycloak/nessie/airflow/openmetadata_db |
| L2 Catalog | **Project Nessie** | 0.108.4-java | JDBC2 di PostgreSQL; Iceberg REST `/iceberg/` |
| L2 Storage | MinIO | RELEASE.2025-09-07 | Bucket `iceberg-data` (warehouse + StarRocks) |
| L3 Processing | Trino | 483 | Katalog `iceberg` via Nessie |
| L3 Authz | **OPA** | latest | Deny-by-default; policy Rego + ACL `config/opa/data.json` |
| L4 Serving | StarRocks | 4.0.13 | **shared-data** (FE + CN) |
| L4 Serving | **Dremio OSS** *(opt-in)* | 25.2.0 | Engine paralel; auth lokal |
| L5 IAM | Keycloak | 26.7 | OIDC; user lokal |
| Identity | **OpenLDAP** | 1.5.0 | Sumber grup (`data-admin`, dst.) |
| L1 Ingestion | Airflow | **3.3.2** | Sling → Nessie Iceberg REST; **image ber-plugin OpenMetadata** (`/pluginsv2/`); **provider MySQL** (koneksi ke server MySQL eksternal) |
| L5 Metadata | OpenMetadata | **2.0.3** | Konektor Trino + Airflow; jalur DQ = Great Expectations (§8.9) |
| L5 Search | OpenSearch | **3.8.0** | Backend pencarian OM 2.0.x (security plugin dimatikan) |
| L5 Observability | Grafana | 13.1 | Provisioning file |
### Kebutuhan CPU (AVX2)
Hanya **satu** komponen di stack ini yang mensyaratkan **AVX2**: **StarRocks CN (BE)**. Build
resmi StarRocks memakai `USE_AVX2=ON`, dan CPU tanpa AVX2 membuat container-nya **gagal start**
dengan `SIGILL` (illegal instruction) — bukan sekadar lambat, dan saat ini **tidak ada paket
resmi non-AVX2**. StarRocks **FE** tidak terpengaruh (berbasis JVM). **AVX-512 tidak wajib** di
komponen mana pun.
Komponen lain berjalan di x86-64 biasa: Trino (hanya butuh 64-bit Linux + Java 25), OpenSearch
(kita hanya memakai pencarian teks/BM25 — AVX2/AVX-512 hanya relevan untuk plugin k-NN yang
tidak dipakai), OpenMetadata/Keycloak/Nessie (JDK 21), Airflow (CPython 3.11), serta
MinIO/PostgreSQL/Grafana/Prometheus. Dremio opt-in tidak mendokumentasikan syarat ISA.
Periksa CPU calon server **sebelum** deploy:
```bash
grep -o -m1 avx2 /proc/cpuinfo # satu-satunya gate nyata di stack ini
lscpu | grep -o -E 'avx2|avx512[a-z]*|sse4_2|bmi2|popcnt' | sort -u
```
Tanpa AVX2, opsinya: lewati Fase 4 (StarRocks) dan pakai Dremio sebagai engine serving, atau
kompilasi StarRocks sendiri dengan `build.sh --without-avx2`. Catatan arsitektur: **ppc64le**
hanya didukung Trino, PostgreSQL, Prometheus, dan Airflow — komponen lain butuh x86-64 atau ARM64.
## Peta Port (prefix 12xxx)
Aturan: **host port = `12` + 3 digit port kanonik**. Port internal container tidak berubah.
| Service | Internal | Host |
|---|---|---|
| Grafana | 3000 | **12300** |
| Dremio Flight | 32010 | **12011** |
| Dremio JDBC | 31010 | **12010** |
| StarRocks FE HTTP | 8030 | **12030** |
| Keycloak | 8080 | **12080** |
| Airflow | 8080 | **12082** |
| OpenMetadata | 8585 | **12085** |
| Nessie | 19120 | **12120** |
| OPA | 8181 | **12181** |
| Trino HTTPS | 8443 | **12443** |
| OpenSearch | 9200 | **12920** |
| MinIO API | 9000 | **12900** |
| MinIO Console | 9001 | **12901** |
| StarRocks FE MySQL | 9030 | **12930** |
| Dremio UI | 9047 | **12947** |
| OpenLDAP | 389 | **12389** |
## Prasyarat
| Tool | Minimal | Catatan |
|------|---------|---------|
| Docker Engine | 24.x+ | `docker --version` |
| Docker Compose | v2.x+ | `docker compose version` |
| OpenSSL | 1.1+ | generate password `.env` |
| curl | 7.x+ | bootstrap & test |
| python3 | 3.8+ | parsing JSON |
| MinIO Client (mc) | latest | opsional |
| MySQL Client | 8.x+ | opsional (StarRocks) |
## Quick Start
```bash
cd deploy
chmod +x install.sh bootstrap.sh test-verification.sh
# 1. Deploy 6 fase (~15 menit)
./install.sh
# 2. Bootstrap & integrasi (Keycloak, OpenLDAP, MinIO, StarRocks, Airflow, OM)
./bootstrap.sh
# 3. Verifikasi
./scripts/verify-phase1-iam.sh # gate kontrak IAM (Phase 1) — sebelum lanjut fase berikutnya
./scripts/verify-phase2-storage.sh # gate kontrak storage & catalog (Phase 2)
./scripts/verify-phase3-processing.sh # gate kontrak processing: Trino + OPA (Phase 3)
./scripts/verify-phase4-serving.sh # gate kontrak serving: StarRocks shared-data (Phase 4)
./scripts/verify-phase5-ingestion.sh # gate kontrak ingestion: Airflow + connections (Phase 5)
./scripts/verify-phase6-governance.sh # gate kontrak governance: OpenMetadata + Grafana (Phase 6)
# 3b. Setelah restart StarRocks FE — pulihkan dynamic config (auth chain OIDC)
./scripts/starrocks-reapply-dynamic-config.sh
# 3c. Isi katalog OpenMetadata (agent ingestion) — tanpa ini halaman Explore kosong
./scripts/ingest-openmetadata.sh
./test-verification.sh
./test-verification.sh --layer L2 # MinIO + Nessie
./test-verification.sh --layer E2E # End-to-end smoke test
# 3d. ALUR DATA (urutan ini penting)
# Ingest DULU, baru DAG:
./scripts/ingest-openmetadata.sh --all # buat entitas tabel di katalog OM
docker exec sicabe-airflow-webserver airflow dags test sicabe_batch_daily
#
# Mengapa urutannya begitu: action GE (OpenMetadataValidationAction) menempelkan
# test suite/test case ke ENTITAS TABEL di OM. Sebelum ingest, entitas itu belum ada,
# sehingga hasil Great Expectations tidak muncul di OM walau tugas DAG-nya sukses.
# Catatan: bagian LINEAGE dari --all membaca artefak dbt (target/manifest.json) yang
# baru ada SETELAH DAG berjalan — untuk lineage utuh jalankan `--dbt-only` sekali lagi
# setelah DAG, atau ulangi `--all`.
#
# 4. (Opsional) Seed tabel contoh Iceberg — namespace bronze + bronze.orders
./scripts/seed-sample-data.sh
# 5. (Opsional) Seed SUMBER PostgreSQL untuk DAG — sicabe_data.public.orders
# WAJIB bila ingin menjalankan DAG 'sicabe_batch_daily' (task ingest_to_bronze)
./scripts/seed-source-postgres.sh
```
> Memakai komponen yang **sudah ada** (Keycloak/LDAP/MinIO/Nessie/Trino/OPA, atau Postgres
> milik DBA)? Arahkan endpoint-nya di `.env` — `KEYCLOAK_URL`, `LDAP_URL`, `S3_ENDPOINT`,
> `NESSIE_URL`, `TRINO_URL`, `OPA_URL` (atau `EXTERNAL_SERVICES="postgres ..."`). `install.sh`
**Menempatkan komponen kita di host lain (multi-host).** Beda dari komponen EXISTING:
di sini komponen tetap milik klaster ini, hanya dipindah host. Override variabel
`*_HOST` di `.env` dengan alamat host pemiliknya — default-nya nama container, sehingga
**single-host tidak berubah**:
| Variabel | Default | Untuk |
|---|---|---|
| `POSTGRES_HOST` | `postgres` | Keycloak, Nessie, Airflow |
| `MINIO_HOST` | `minio` | Nessie, minio-init |
| `KEYCLOAK_HOST` | `keycloak` | Keycloak backend URL, Grafana OAuth |
| `OPENMETADATA_HOST` | `openmetadata-server` | `OM_API_URL`, `SERVER_HOST_API_URL` |
| `AIRFLOW_WEBSERVER_HOST` | `airflow-webserver` | `AIRFLOW__WEBSERVER__BASE_URL`, pipeline client |
| `STARROCKS_FE_HOST` / `STARROCKS_CN_HOST` | `starrocks-fe` / `starrocks-cn` | registrasi CN, healthcheck |
| `OPENSEARCH_HOST` | `openmetadata-opensearch` | `ELASTICSEARCH_HOST` OpenMetadata |
> File `config/**` yang dibaca **container** (`.properties`/`.yaml`/`.cfg`/`.json`/`.rego`)
> tidak bisa ekspansi env. Untuk multi-host jalankan `scripts/render-config.sh` yang
> menyalinnya ke `generated/host-<x>/config/` sambil mengganti hostname terdaftar:
>
> ```bash
> ./scripts/render-config.sh --out generated/host-compute/config --dry-run \
> --map postgres=10.20.0.12 --map nessie=10.20.0.10
> ```
>
> Penggantian hanya pada konteks jaringan (setelah `//`, setelah `@`, atau `host:port`)
> sehingga path filesystem seperti `/etc/trino/` dan baris komentar tidak ikut berubah.
> Rencana lengkap: `reviews/sicabe-10-multi-host-installer-plan.md`.
> ⚠️ **Pilot multi-host di host yang SAMA tidak boleh dua stack berjalan bersamaan.**
> Memisah network/port saja **tidak cukup** — RAM tidak akan cukup (host pilot 15 GB,
> satu stack penuh ±10 GB). **Matikan stack lama lebih dulu** sebelum menyalakan hasil
> `generated/host-<x>/`.
**Trino dipisah coordinator/worker.** Sejak 2026-09-28 Trino bukan lagi satu service:
`trino-coordinator` (port host `12443`, HTTPS + OAuth2/Basic, client-facing) dan
`trino-worker` (tanpa port host; mengeksekusi task, `discovery.uri` ke coordinator).
Klien **tidak berubah** (`TRINO_URL` tetap `https://localhost:12443`). Untuk restart:
`./install.sh --restart trino-coordinator` (atau `trino-worker`). Perubahan **nama
service** mewajibkan regenerasi sertifikat: `./scripts/generate-trino-keystore.sh`
(SAN memuat `trino-coordinator`), lalu restart coordinator.
### Menyiapkan server baru (Ubuntu 24.04)
Dua script, dijalankan **di tiap server** sebelum instalasi:
```bash
sudo ./scripts/prepare-host.sh --dry-run # lihat rencananya dulu (tidak mengubah apa pun)
sudo ./scripts/prepare-host.sh # paket apt + Docker + sysctl + precheck
./scripts/prefetch-images.sh # tarik semua image lebih dulu
```
**`prepare-host.sh`** memasang paket yang benar-benar dipakai script deploy
(`htpasswd`/apache2-utils, `ldapsearch`/ldap-utils, klien `mysql`, `openssl`, `curl`,
`python3`+PyYAML), memasang Docker Engine + compose v2 (lewati dengan `--no-docker`),
menetapkan `vm.max_map_count=262144` (**syarat keras OpenSearch**), dan memeriksa
**AVX2** (wajib `starrocks-cn`), jam/NTP, RAM, disk, serta port `12xxx` yang akan dipakai.
Flag `--swap-off` (opsional) mematikan swap sesuai rekomendasi OpenSearch.
**`prefetch-images.sh`** mengambil daftar image dari `docker compose config --images`
(**tidak** di-hardcode, jadi selalu sinkron dengan compose). Default: **14 image ≈ 6,2 GB**
(terukur). Pilihan: `--all` (default) · `--host <nama>` (hanya image host itu, dari
`generated/host-<x>/`) · `--list` (lihat saja).
Tiga image **dikecualikan secara default** dan ditampilkan sebagai catatan
("dikecualikan: …") supaya tidak terkesan terlupa:
| Dikecualikan | Alasan | Mengaktifkan |
|---|---|---|
| `dremio/dremio-oss:25.2.0` | Dremio **opt-in** di compose (default nonaktif) | `--with-dremio` |
| `apache/airflow:slim-3.3.2-python3.11` | hanya base untuk build lokal Airflow | `--with-base` |
| `sicabe/airflow:3.3.2-om203-ol` | **dibangun lokal**; sejak 2026-10-02 **juga tersedia di registry** | `docker pull` dari registry (di bawah), atau build-sekali-salin |
> 📌 **Sejak 2026-10-02 image ini tersedia di registry internal** — jadi host lain bisa
> **menariknya**, tidak harus membangun:
> ```bash
> docker pull registry.sicabe.com/airflow:3.3.2-om203-ol
> # opsi A (disarankan): arahkan compose ke nama registry — tanpa retag
> export AIRFLOW_IMAGE=registry.sicabe.com/airflow:3.3.2-om203-ol
> # opsi B: beri nama yang diharapkan compose
> docker tag registry.sicabe.com/airflow:3.3.2-om203-ol sicabe/airflow:3.3.2-om203-ol
> ```
> ⚠️ `docker-compose.yml` memakai `${AIRFLOW_IMAGE:-sicabe/airflow:3.3.2-om203-ol}` —
> jadi setelah `pull`, **pilih salah satu** dari dua opsi di atas; kalau tidak, compose
> akan mencari tag `sicabe/...` yang belum ada di host itu.
> ⚠️ **`sicabe/airflow:3.3.2-om203-ol` dibangun lokal** (base `apache/airflow:slim-3.3.2-python3.11`),
> jadi **tidak tersedia di Docker Hub**. ✅ Sejak 2026-10-02 fetch bisa lewat registry
> internal (lihat blok di atas); alur build-sekali-salin di bawah tetap tersedia untuk
> host yang **tidak punya akses registry**:
> ```bash
> # di SATU host
> docker build -t sicabe/airflow:3.3.2-om203-ol -f docker/airflow/Dockerfile.332 docker/airflow
> ```
> 📌 Sejak BATCH B (review R18, 2026-10-01) varian image **berdampingan**:
> `Dockerfile.332` (+ `requirements.332.{in,lock}`) = Airflow 3.3.2 / Python 3.11 —
> **yang dipakai sekarang**; `Dockerfile.slim` (+ `requirements.{in,lock}`) tetap
> disimpan sebagai jalur rollback ke Airflow 3.3.1.
> 📌 **2026-10-02 — adopsi opsi C**: tag **`-ol`** = image di atas **+ provider
> OpenLineage + shim `eventTime`**. OpenLineage **AKTIF** mengirim lineage task
> Airflow ke OpenMetadata (rute langsung, tanpa Marquez).
>
> - **Transport dibaca dari KONEKSI Airflow** `openlineage_config` (bukan `.env`),
> sehingga token **tidak** tersebar di berkas. `./bootstrap.sh` membuatnya
> bersama `openmetadata_lineage_api` (password = token bot `lineage-bot`,
> `JWTTokenExpiry=Unlimited`).
> - Aktif/mati: `AIRFLOW_OPENLINEAGE_CONFIG_CONN_ID` (default `openlineage_config`;
> kosongkan untuk mematikan). ⚠️ Koneksinya **harus sudah ada** — pemuatan
> konfigurasi provider tidak dibungkus try/except (bisa menggagalkan task).
> - ⚠️ `AIRFLOW__OPENLINEAGE__EXECUTE_IN_THREAD=true` **wajib**; tanpa itu proses
> emisi HANG lalu dimatikan dan tidak ada event yang keluar.
> - ⚠️ Shim `ol_time_shim.ZuluHttpTransport` **wajib** untuk OM 2.0.3: provider
> memakai `eventTime` berakhiran `+00:00`, sedangkan OM hanya menerima `Z`
> (tanpa shim → `400 Invalid request format`).
> - Lineage **tabel + kolom** muncul untuk task yang memakai **operator SQL**
> (mis. `SQLExecuteQueryOperator`). Task dbt/Sling via `BashOperator` tidak
> menghasilkan lineage dari jalur ini — dbt tetap dari ingestion dbt.
> - ⚠️ OM 2.0.3 mencatat error berulang `null value in column "updatedat"`
> saat mencoba auto-create entitas pipeline (bug OM; tidak berbahaya).
> - Rincian: `experiments/lineage-airflow-om/README.md`.
> Rollback: `sicabe/airflow:3.3.2-om203` (tanpa OpenLineage).
> ./scripts/prefetch-images.sh --save-airflow /tmp/airflow-image.tar # ±835 MB
> # di host lain
> scp /tmp/airflow-image.tar <host>:/tmp/
> ./scripts/prefetch-images.sh --load-airflow /tmp/airflow-image.tar
> ```
### Menempatkan komponen ke beberapa host (Fase 1)
```bash
cp cluster.example.yml cluster.yml # sesuaikan address tiap host
./install.sh --generate # → generated/host-<x>/ (gitignored)
./install.sh --generate --blueprint b5 # preset 5 host (blueprint B)
./install.sh --sync # salin deploy/ (termasuk .env) ke host lain (dari control)
./install.sh --sync --host state --dry-run # lihat saja (tanpa menyalin)
./install.sh --host compute --dry-run # lihat perintahnya tanpa menjalankan
```
**Distribusi ke host lain (`--sync`).** Dijalankan di **host control** (master installer).
Menyalin seluruh `deploy/` — **termasuk `.env` dan `generated/`** — ke tiap host lain di
`cluster.yml` via `rsync` over SSH:
```bash
./install.sh --sync # ke SEMUA host lain (host ini dideteksi dari alamat lokal)
./install.sh --sync --host state # hanya satu host
./install.sh --sync --user admin --dest '~/sicabe-doc/deploy' # SSH user & tujuan
./install.sh --sync --delete # hapus di tujuan berkas yang tak ada di sumber
```
Host yang sedang dijalankan **dikecualikan otomatis** (kecocokan `hostname -I` dengan
`address` di inventory); bila tak cocok, ia menyinkron ke semua host. `.env` **wajib ikut**
(rahasia bersama — OIDC client secret harus identik lintas host). Setelah sync, di tiap host
jalankan `./scripts/prefetch-images.sh --host <nama>` lalu `./install.sh --host <nama>`.
Tanpa `cluster.yml`, installer berperilaku **persis** seperti single-host (tidak ada
yang di-generate). Hasil generate berisi per host: `docker-compose.yml` (hanya service
milik host itu), `.env` (endpoint → alamat host), dan `config/` ter-render.
Uji: `./scripts/test-inventory.sh` (validasi inventory) · `./scripts/test-generate.sh`
(semantik hasil generate vs single-host). Rencana lengkap:
`reviews/sicabe-10-multi-host-installer-plan.md`.
**Endpoint per host di skrip (multi-host).** Skrip gate/verifikasi memakai helper
`sicabe_load_env`/`sicabe_local_url`/`sicabe_local_host`, jadi jalankan dengan env host agar
endpoint menunjuk **ALAMAT HOST PEMILIK** (bukan `localhost`):
```bash
./scripts/verify-phase3-processing.sh --env-file generated/host-compute/.env
./scripts/verify-phase6-governance.sh --env-file generated/host-control/.env
./install.sh --status --host compute # status + endpoint host compute
```
`--env-file` juga dihormati lewat env `SICABE_ENV_FILE` — dipakai otomatis oleh
`bootstrap.sh --host`, sehingga sub-script (`seed-*`/`ingest-*`/`reapply`) mewarisi env host
dan tidak lagi menimpanya dengan `deploy/.env`. **Single-host tidak berubah** (tetap
`localhost`) tanpa opsi ini.
> **Skrip host-scoped (multi-host).** Skrip yang `docker exec` container harus dijalankan **DI
> host pemilik container** — mis. `seed-source-postgres.sh` di host **`state`**,
> `seed-starrocks.sh` di **`compute`**, `seed-native-dq.sh` di **`control`**. Bila dijalankan
> di host lain, skrip keluar rapi (kode **3**) + instruksi, bukan error; `bootstrap.sh`
> menampilkannya sebagai WARN. Gate (`verify-phase*`) otomatis **SKIP** cek container milik
> host lain (bukan FAIL).
>
> ```bash
> # di host state:
> ./scripts/seed-source-postgres.sh --env-file generated/host-state/.env
> # di host compute:
> ./scripts/seed-starrocks.sh --env-file generated/host-compute/.env
> ```
> **TLS Trino lintas-host.** Saat `--generate`, keystore + cert Trino dibangkitkan **sekali**
> dengan **SAN alamat IP tiap host**, lalu **disalin identik ke tiap bundle** (keypair harus
> sama lintas host). Jadi klien yang terhubung via IP (mis. ingestion **OpenMetadata**) tetap
> **memverifikasi TLS** — tak perlu `verify: false`. **`DBT_TRINO_HOST`** juga di-set ke IP
> pemilik Trino sehingga dbt (di Airflow, host lain) menyambung lintas-host. Ubah alamat host
> → `./install.sh --generate` → `--sync` → recreate `trino-coordinator` **dan** service Airflow.
> ⚠️ **Setelah mengubah file yang di-MOUNT (cert/config), container WAJIB di-restart manual.**
> `./install.sh --host <x>` menjalankan `docker compose up -d`, yang **hanya** recreate bila
> *definisi* compose berubah — **bukan** bila isi file bind-mount berubah. Contoh: regenerasi
> cert Trino → `docker restart sicabe-trino-coordinator` (di host Trino) dan service Airflow
> (di host control). Cek cepat: fingerprint cert yang **disajikan** vs yang **dipakai**
> container harus sama:
> ```bash
> # disajikan (dari host lain): echo | openssl s_client -connect <ip-trino>:8443 -servername trino 2>/dev/null | openssl x509 -noout -fingerprint -sha256
> # dipakai (di container airflow):
> docker exec sicabe-airflow-webserver openssl x509 -in /etc/ssl/trino/trino.crt -noout -fingerprint -sha256
> ```
> **Endpoint komponen SE-HOST memakai nama service.** `.env` bundle kini menulis `*_HOST`
> komponen **se-host** sebagai **nama service** (mis. `OPENMETADATA_HOST=openmetadata-server`)
> dan komponen **beda host** sebagai **alamat IP**. Sebab port container (mis. OM 8585) tidak
> diterbitkan di host; memakai IP host justru "connection refused" (regresi OpenLineage).
> ⚠️ **`./install.sh --host <nama>` dijalankan DI host itu**, dan hanya setelah stack
> lama dimatikan bila host-nya sama. `depends_on` lintas host sudah dibuang generator —
> penggantinya `preflight` yang **belum ada** (Fase 2), jadi urutan start antar host
> masih manual.
> dan `test-verification.sh` akan **melewati container** komponen tersebut, dan `bootstrap.sh`
> beserta gate fase memakai endpoint yang sama. Detail: `docs/sicabe-install-phases-detail.md`
> §2.8–§2.12.
>
> `scripts/seed-sample-data.sh` (idempoten) membuat namespace + tabel contoh
> `iceberg.bronze.orders` (6 baris) agar catalog tidak kosong dan langsung
> terlihat di Trino, Dremio, dan StarRocks.
>
> `scripts/seed-source-postgres.sh` (idempoten) menyiapkan **sumber** di
> PostgreSQL: database `sicabe_data` + tabel `public.orders` (6 baris). Ini yang
> dibaca task `ingest_to_bronze` (Sling: `POSTGRES_DB.public.orders` →
> `ICEBERG bronze.orders`). Tanpa ini DAG `sicabe_batch_daily` gagal dengan
> pesan database `sicabe_data` tidak ditemukan.
> **Diagram alur instalasi** — `install.sh` (6 fase) → `bootstrap.sh` (9 step) → stack siap:
> [Mermaid `sicabe-install-sequence.mmd`](../docs/diagrams/sicabe-install-sequence.mmd) ·
> [versi interaktif `.drawio`](../docs/diagrams/sicabe-install-sequence.drawio).
> Regenerasi `.drawio` dari `.mmd`:
> `bash docs/diagrams/generate-drawio-from-mermaid.sh sicabe-install-sequence.mmd`
> **Detail per fase & bootstrap step** (prasyarat/dependency, container yang
> dinyalakan, aksi **configure/seed/verify** tiap step):
> [`docs/sicabe-install-phases-detail.md`](../docs/sicabe-install-phases-detail.md)
>
> **Memakai layanan existing?** Variabel `.env` yang perlu ditambahkan:
> Keycloak/LDAP → §2.8 · MinIO/S3 & Nessie → §2.9 pada dokumen yang sama.
> Untuk **IAM existing** (dikelola pihak lain/tanpa akses admin): daftar permintaan ke
> admin → Keycloak [`docs/keycloak-existing-requirements.md`](../docs/keycloak-existing-requirements.md) ·
> OpenLDAP [`docs/openldap-existing-requirements.md`](../docs/openldap-existing-requirements.md).
Deploy per phase:
```bash
./install.sh --phase 1 # Keycloak + OpenLDAP + PostgreSQL
./install.sh --phase 2-4 # MinIO + Nessie + Trino + OPA + StarRocks FE+CN
./install.sh --status
./install.sh --destroy
# Dremio opt-in — tidak ikut phase mana pun (butuh RAM ~4 GB lebih)
docker compose --profile dremio up -d dremio
```
### URL browser-facing (issuer OIDC, redirect, UI)
Enam variabel berisi URL yang dilihat **browser**, bukan lalu lintas container:
`KEYCLOAK_PUBLIC_URL`, `GRAFANA_PUBLIC_URL`, `OPENMETADATA_PUBLIC_URL`,
`AIRFLOW_PUBLIC_URL`, `TRINO_PUBLIC_URL`, `MINIO_CONSOLE_PUBLIC_URL`.
| Kondisi | Nilai `*_PUBLIC_URL` |
|---|---|
| `tls.mode: none` (default) | `http://<alamat host yang menjalankan komponen>:<port>` — diisi otomatis oleh `install.sh --generate`, **sama di semua bundle** |
| `tls.mode: edge` / `full` | `https://<label>.<domain>` dari `cluster.domain` |
| tanpa `cluster.yml` (single-host) | `http://localhost:<port>` |
**Keycloak 26.x (hostname v2):** yang menentukan issuer, discovery, dan redirect adalah
**`KC_HOSTNAME`**, dan isinya **URL lengkap** (`http://host:port`) — diambil dari
`KEYCLOAK_PUBLIC_URL`. Jangan pakai `KC_HOSTNAME_URL`: opsi hostname v1 itu **dihapus di
Keycloak 26.0.0** sehingga tidak berefek; `KC_HOSTNAME_BACKEND_URL` bahkan bukan opsi
Keycloak. `KC_HOSTNAME_BACKCHANNEL_DYNAMIC=true` membuat container lain tetap bisa
memakai nama internal `keycloak:8080` untuk token/JWKS.
**Akses HTTP dari alamat non-lokal (mis. Tailscale).** `sslRequired` adalah atribut
**realm** (default Keycloak `external`) dan dinilai dari **alamat sumber koneksi**, bukan
header Host. Menurut Keycloak, "lokal" = loopback + RFC1918 (`10/8`, `172.16/12`,
`192.168/16`) + link-local + IPv6 `fc00::/7` — sehingga **`100.64.0.0/10` (CGNAT/Tailscale)
dianggap eksternal** dan admin console gagal dengan `HTTPS required`, padahal akses via
`localhost` normal. Keycloak 26 **tidak punya** opsi server-level untuk ini (tidak ada
`KC_HTTPS_REQUIRED`), jadi harus per realm: `bootstrap.sh --step 1` menyetelnya untuk
**`master` dan realm Sicabe** dari `KEYCLOAK_SSL_REQUIRED` (`none` saat `tls.mode: none`,
`external` saat ada TLS/Traefik).
**Mengubah alamat publik berarti mengubah `iss`** → semua konsumen harus ikut dan
**semua pengguna login ulang**:
- Trino: `issuer` + `auth-url` memakai `${ENV:KEYCLOAK_PUBLIC_URL}` (token/JWKS tetap
ke `keycloak:8080`) — lihat `config/trino/config-coordinator.properties`;
- redirect URI + web origin klien: disinkronkan otomatis oleh `bootstrap.sh step 1`
(digabung, idempoten — termasuk client lama yang masih localhost-only);
- Grafana (`GF_SERVER_ROOT_URL`) dan Airflow (`AIRFLOW__WEBSERVER__BASE_URL`) mengikuti
`*_PUBLIC_URL` masing-masing.
Gejala bila salah: browser dibuang ke `localhost`, atau login ditolak dengan
`Invalid parameter: redirect_uri`.
### Ingress + TLS (langkah 2 / T-20)
Traefik berjalan **hanya pada profile `edge`**, dan hanya ikut ke bundle host saat
`tls.mode` != `none`. Setiap host menjalankan Traefik-nya sendiri, dengan rute diambil dari
label container (provider docker) — jadi daftar rute = label pada layanan.
```bash
# 1. inventory: tls.mode: edge (domain: sicabe.com)
# 2. install.sh --generate && install.sh --sync -> distribusi bundle ke host lain
# 3. isi kredensial DNS di .env tiap host (contoh Cloudflare: token Zone:DNS:Edit)
# CF_DNS_API_TOKEN=...
# 4. jalankan ingress
docker compose --profile all --profile edge up -d
```
| Variabel | Arti |
|---|---|
| `CLUSTER_DOMAIN` | domain rute `Host()` (label dari `PUBLIC_URL_SERVICES`: kc/grafana/om/airflow/trino/minio) — hanya di host ingress |
| `tls.ingress_host` | (inventory, opsional) host yang menjalankan Traefik; default pemilik Keycloak |
| `TRAEFIK_BIND_ADDR` | alamat bind port 80/443 — diisi generator dengan alamat host (bukan semua antarmuka) |
| `TRAEFIK_ACME_CASERVER` | **staging** secara default; produksi: `https://acme-v02.api.letsencrypt.org/directory` |
| `CF_DNS_API_TOKEN` | kredensial DNS-01 Cloudflare |
Catatan penting:
- **Tanpa kredensial DNS, Traefik tetap start** dan memakai sertifikat self-signed bawaan,
jadi routing masih bisa diuji: `curl -k -H 'Host: kc.<domain>' https://<host>/`.
Log ACME akan menyebut variabel yang kurang secara eksplisit.
- `acme.json` **wajib mode 0600** (Traefik menolak start bila terlalu terbuka). Compose
memakai named volume, sehingga Traefik membuatnya sendiri dengan mode benar.
- **Ingress dijalankan SATU host saja** (default: host yang menjalankan Keycloak —
`control`; bisa dipatok dengan `tls.ingress_host`). Alasannya bukan hanya kesederhanaan:
apex (`sicabe.com`) dan wildcard (`*.sicabe.com`) memakai **nama TXT DNS-01 yang sama**
(`_acme-challenge.<domain>`), sehingga beberapa Traefik yang menerbitkan wildcard yang
sama akan **saling bertabrakan** — gejalanya `403 Incorrect TXT record ... (and 1 more)`.
Batas Let's Encrypt juga hanya **5 sertifikat per himpunan nama per minggu**.
- Host ingress mengambil rute komponen **lokal** dari label container, dan rute komponen di
**host lain** dari `config/traefik/dynamic.yml` (provider file) yang dibangkitkan generator —
mis. `trino` → `https://<alamat-compute>:12443`, `minio` → `http://<alamat-state>:12901`.
Traefik hanya melihat container di host-nya sendiri, jadi rute lintas host **wajib** lewat
berkas dinamis.
- Karena ingress terpusat, **A-record cukup satu wildcard** `*.sicabe.com` → alamat host
ingress (tidak perlu per label). Port 80/443 hanya perlu dibuka di host ingress.
- Label `traefik.*` tetap tertulis di semua bundle (inert tanpa Traefik) — tidak berbahaya,
tapi jangan menjalankan `--profile edge` di host non-ingress karena akan menggandakan rute.
- Trino internal memakai HTTPS (sertifikat self-signed) -> Traefik memakai `serversTransport`
dari `config/traefik/dynamic.yml` dengan `insecureSkipVerify` **hanya untuk Trino**.
`serversTransports` **tidak bisa** didefinisikan lewat label Docker (menghasilkan error
runtime `servers transport not found`).
- `tls.mode: edge` membuat generator mengisi `KC_PROXY_HEADERS=xforwarded`,
`TRINO_PROCESS_FORWARDED=true`, dan `KEYCLOAK_SSL_REQUIRED=external`. Sebaliknya pada
`none` semuanya kosong/`false` — Trino dan Keycloak **menolak** header `X-Forwarded-*`
dari klien, jadi host bisa dipalsukan tidak.
## Detail per Komponen
### Phase 1 — Keycloak + OpenLDAP
- Realm `sicabe`; OIDC clients: `trino`, `starrocks`, `airflow`, `openmetadata`, `grafana`.
- Users lokal: `admin`, `engineer`, `analyst`, `viewer`.
- **OpenLDAP** = sumber kebenaran **grup** (`data-admin`, `data-engineer`,
`data-analyst`, `data-viewer`). User tidak difederasikan — autentikasi via
Keycloak, resolusi grup via LDAP (opsi 1).
- Akses: `http://localhost:12080/admin/sicabe/console`.
### Phase 2 — Storage: MinIO + Nessie
- MinIO bucket `iceberg-data` (dipakai Nessie warehouse + StarRocks shared-data).
- **Nessie** menyimpan version store di **PostgreSQL** (`nessie` DB, schema `nessie`,
`NESSIE_VERSION_STORE_TYPE=JDBC2`). Tabel Nessie di-auto-create.
- Warehouse: `s3://iceberg-data/warehouse`; S3 MinIO path-style, kredensial server-side.
- Akses: `http://localhost:12120` (`/api/v2`, `/iceberg/`).
### Phase 3 — Processing: Trino + OPA
- Katalog `iceberg` via Nessie (`iceberg.catalog.type=nessie`).
- **Authz OPA** (`access-control.name=opa`, deny-by-default). Policy:
`config/opa/policies/trino.rego`; ACL: `config/opa/data.json`.
- **Group provider LDAP** (`group-provider.name=ldap`) → grup `data-*`.
- **Auth hybrid** (`http-server.authentication.type=oauth2,password`):
- **End user**: OAuth2 SSO Keycloak (Web UI + JDBC) di `https://localhost:12443`.
- **Service/pipeline**: **HTTP Basic** service account (`svc_dbt`, `svc_seed`,
`svc_airflow`, `svc_om`) via `password-authenticator` file
(`config/trino/password.db`, bcrypt; di-generate `bootstrap.sh`/`install.sh`,
pra-syarat `htpasswd`). Otorisasi via OPA `data.users` (user→role).
- **HTTPS-only**: HTTP polos **dimatikan** (`allow-insecure-over-http=false`) —
`http://…:12081` tidak lagi di-expose. Semua akses via `https://localhost:12443`
(self-signed → `curl -k` / terima peringatan browser).
- ⚠️ **Web UI OAuth2 wajib HTTPS** — Trino mengalihkan `/ui/` ke
`/ui/static/disabled.html` bila request tidak secure.
### Phase 4 — Serving: StarRocks (shared-data) + Dremio
**StarRocks shared-data:**
- FE `run_mode=shared_data`, `cloud_native_storage_type=S3`; data tabel internal
disimpan di MinIO via **storage volume** `s3_volume`.
- Cluster = **FE + CN** (Compute Node, `ALTER SYSTEM ADD COMPUTE NODE`).
- OIDC: `CREATE SECURITY INTEGRATION sr_oidc`; **LDAP group provider** `sr_ldap`;
`authentication_chain = native,sr_oidc`; `GRANT role TO EXTERNAL GROUP`.
- External catalog `iceberg_catalog` → **Nessie** (`http://nessie:19120/iceberg/`).
- Akses: `mysql -h localhost -P 12930 -u root -p`.
**Dremio OSS:**
- Engine paralel di atas Nessie yang sama; auth **lokal** (first-run admin via UI).
- Source Nessie: endpoint `http://nessie:19120/api/v2`, MinIO path-style
(`fs.s3a.path.style.access=true`, `dremio.s3.compat=true`).
- Akses: `http://localhost:12947`. Detail: `config/dremio/README.md`.
### Phase 5 — Ingestion: Airflow
- Sling menarik PostgreSQL → Bronze (Iceberg via **Nessie Iceberg REST**).
- dbt via Trino: Bronze → Silver → Gold. Great Expectations validasi.
- Akses: `http://localhost:12082`.
### Phase 6 — Governance: OpenMetadata + Grafana
- OpenMetadata: DB `openmetadata_db`; konektor Trino + Airflow.
- Grafana: datasource provisioning file. Akses `http://localhost:12300`.
## RBAC
| User | Password | Role | Akses |
|------|----------|------|-------|
| `admin` | `KEYCLOAK_ADMIN_PASSWORD` | admin | Penuh (OPA superuser, StarRocks `sr_admin`) |
| `engineer` | `engineer123` | engineer | OPA write+ddl, StarRocks `sr_engineer` |
| `analyst` | `analyst123` | analyst | OPA select (per-kolom), StarRocks `sr_analyst` |
| `viewer` | `viewer123` | viewer | Terbatas (StarRocks tanpa grant → fail-closed) |
Pemetaan grup → akses:
- **Trino/OPA**: grup LDAP `data-*` → `config/opa/data.json` `roles.<grup>.tables`.
- **StarRocks**: external group `data-*` → role `sr_*` (`GRANT ... TO EXTERNAL GROUP`).
## Perbedaan dari `deploy-legacy/`
| Aspek | `deploy-legacy/` (base lama) | `deploy/` (merge) |
|---|---|---|
| Iceberg catalog | Polaris | **Nessie** (JDBC2/Postgres) |
| Trino authz | `rules.json` | **OPA** |
| Trino group provider | plugin Keycloak | **LDAP** |
| StarRocks mode | shared-nothing (FE+BE) | **shared-data (FE+CN)** |
| StarRocks storage | disk lokal | **S3/MinIO (storage volume)** |
| Engine paralel | — | **Dremio OSS** |
| Identitas | user+grup di Keycloak | user di Keycloak, **grup di OpenLDAP** |
| Project/network | `sicabe` / `172.28.0.0/16` | `sicabe` / `172.28.0.0/16` (sama) |
| Host port | 3xxx/8xxx/9xxx | **12xxx** |
## Catatan Penting
- **`run_mode=shared_data` bersifat final** — tidak bisa diubah setelah first
start; `deploy/` mulai dari volume bersih.
- **Nessie tidak mendukung Iceberg spec v3** → selalu `format-version=2`.
- **Dremio OSS** memakai auth lokal & penyimpanan volume (bukan PostgreSQL).
- **Tulis lintas engine** (Trino/StarRocks/Dremio) di Nessie sebaiknya bergantian,
bukan simultan.
- **`.env`** di-generate `install.sh` (JANGAN commit).
- Container & project memakai **`sicabe-*`** (sama dengan `deploy-legacy/`).
Karena nama container/project sama, **jangan jalankan `deploy/` dan
`deploy-legacy/` bersamaan** — `deploy-legacy/` hanya arsip/referensi.
- Host port tetap **12xxx**.
## Troubleshooting (ringkas)
| Gejala | Solusi |
|---|---|
| Nessie unhealthy | Cek DB `nessie` + schema `nessie` ada; cek `NESSIE_DB_PASSWORD` |
| Trino query ditolak semua | Cek OPA reachable (`curl localhost:12181/v1/data/trino/allow`); cek `config/opa/data.json` |
| StarRocks CN tidak terdaftar | `docker logs sicabe-starrocks-cn`; `SHOW COMPUTE NODES` di FE |
| StarRocks S3 400/301 | Pastikan path-style di storage volume + `cn.conf`/env S3 |
| Login OIDC StarRocks gagal | `required_issuer` harus host yang dipakai login (`http://localhost:12080/...`); redirect URI client `starrocks` harus memuat `http://starrocks-fe:8030/api/oauth2` |
| Dremio tidak melihat Nessie | Buat source Nessie manual (first-run); MinIO path-style + `dremio.s3.compat=true` |
| Sling gagal ke katalog | Sling tidak mendukung `catalog_type=nessie`; gunakan `catalog_type: rest` + `rest_uri: http://nessie:19120/iceberg/` |
Detail tambahan: `experiments/*/README.md` dan `config/*/README.md`.
## Referensi
> **Catatan versi & perilaku terverifikasi** — [`docs/sicabe-catatan-versi-dan-perilaku.md`](../docs/sicabe-catatan-versi-dan-perilaku.md):
> hal-hal yang pernah menggigit dan nilai yang benar (Keycloak 26 hostname/sslRequired/cluster
> health, Traefik v3 label/file-provider/ACME, token DNS-01 Cloudflare, env OM & MinIO),
> termasuk **Traefik membuang container yang unhealthy** yang membuat komponen hilang dari
> ingress tanpa error.
- Rencana merge: `../reviews/sicabe-07-merge-experiments-plan.md`
- [Project Nessie](https://projectnessie.org/guides/iceberg-rest/)
- [Trino OPA access control](https://trino.io/docs/current/security/opa-access-control.html)
- [StarRocks shared-data](https://docs.starrocks.io/docs/deployment/deploy_shared_data_manually/)
- [Dremio OSS](https://docs.dremio.com/)
- [Sling Iceberg](https://docs.slingdata.io/connections/datalake-connections/iceberg)
---
*deploy — hasil merge `deploy-legacy/` + 3 eksperimen (2026-09-10).*
deploy/bootstrap.sh
Executable
+1655
View File
File diff suppressed because it is too large Load Diff
deploy/cluster.example.yml
+78
View File
@@ -0,0 +1,78 @@
# =============================================================================
# cluster.example.yml — inventory multi-host SiCABE (CONTOH; salin ke cluster.yml)
# -----------------------------------------------------------------------------
# Fase 1 R10 §4.3/§4.5 + taksonomi 7-tier R11.
#
# CARA PAKAI
# cp cluster.example.yml cluster.yml # lalu sesuaikan address
# ./install.sh --generate # hasil ke generated/host-<x>/
#
# Penting: berkas `cluster.yml` bersifat OPSIONAL. Bila TIDAK ADA, installer
# berjalan seperti sekarang (single-host) — `docker-compose.yml` dipakai langsung
# dan tidak ada berkas di `generated/` yang dibuat.
#
# PRECEDENCE (tertinggi → terendah)
# 1. components.<nama>.host (override eksplisit)
# 2. hosts.<host>.tiers (tier → host)
# 3. preset (a3 | b5) — hanya mengisi default tiers
# =============================================================================
version: 1
cluster:
name: sicabe-prod
env: production
# domain = basis nama BROWSER-FACING (lihat blok `tls` di bawah). Isi nama domain
# saja — tanpa skema, tanpa garis miring. Untuk mode `none` nilainya tidak dipakai.
domain: sicabe.com
# ─── TLS (langkah 1: baru SKEMA; Traefik + sertifikat menyusul di step 2) ─────
# none (DEFAULT) — perilaku sekarang: http, IP + port 12xxx. Bidang internal
# plaintext. Cocok untuk lab/uji tertutup.
# edge — browser memakai https://<nama>.<domain> lewat ingress;
# bidang internal (container↔container) TETAP plaintext.
# Pilihan paling murah untuk pilot: satu sertifikat wildcard,
# dan naik ke `full` nanti TIDAK mengubah URL/issuer.
# full — seluruh segmen TLS (butuh kepercayaan CA di semua klien).
# tls:
# mode: edge # contoh: aktifkan ingress berbasis nama
tls:
mode: none
# a3 = 3 host (default) · b5 = 5 host (coord terpisah + 2 engine)
preset: a3
hosts:
control:
address: 10.20.0.10
# tier = kelompok peran (R11 §3.1). Preset a3 sudah mengisi ini; ditulis
# eksplisit agar terbaca. `control` mengelompokkan komponen ringan.
tiers: [identity, policy, pipeline, governance]
compute:
address: 10.20.0.11
# coord = trino-coordinator + starrocks-fe · engine = trino-worker + starrocks-cn
tiers: [coord, engine]
state:
address: 10.20.0.12
# state WAJIB tepat satu host dan tidak boleh digabung dengan `engine`.
# Berisi: postgres, minio (+minio-init), openmetadata-opensearch.
tiers: [state]
components:
# Override penempatan per komponen (prioritas tertinggi). Contoh: pindahkan
# StarRocks FE ke host control bila compute sudah terlalu padat.
# starrocks-fe: { host: control }
# Dremio default NONAKTIF. Bila diaktifkan: taruh di host bertier `engine`
# (butuh RAM ~4 GB lebih — cek kapasitas host).
dremio: { enabled: false }
# Contoh blueprint b5 (5 host): hapus komentar dan GANTI `preset` di atas.
# preset: b5
# hosts:
# control: { address: 10.20.0.10, tiers: [identity, policy, pipeline, governance] }
# coord: { address: 10.20.0.11, tiers: [coord] }
# engine-1: { address: 10.20.0.12, tiers: [engine] }
# engine-2: { address: 10.20.0.13, tiers: [engine] }
# state: { address: 10.20.0.14, tiers: [state] }
# Catatan b5: dua host `engine` menjalankan service `trino-worker` dan
# `starrocks-cn` masing-masing satu — jadi tiap host engine punya node sendiri.
deploy/cluster.p1.yml
+77
View File
@@ -0,0 +1,77 @@
# =============================================================================
# cluster.p1.yml — simulasi "P1": host EXISTING (dikelola pihak lain) + cluster 3 host
# -----------------------------------------------------------------------------
# Skenario P1:
# - Host `existing` menjalankan IAM/storage fase 1–2 SENDIRI (single-host):
# keycloak, openldap, postgres, minio, nessie → port HOST 12xxx dipublikasikan
# (12080 / 12389 / 12900 / 12120 …). TIDAK di-deploy dari berkas ini.
# - Cluster (control/compute/state) menjalankan SISANYA dan memakai keycloak/openldap/
# minio/nessie EXISTING lewat alamat DI ATAS + port yang dipublikasikan.
#
# CARA PAKAI
# # di control (sumber .env cluster), TAMBAHKAN ke deploy/.env:
# KEYCLOAK_INTERNAL_PORT=12080 # <-- port HOST keycloak existing (bukan 8080)
# OPENLDAP_INTERNAL_PORT=12389
# MINIO_INTERNAL_PORT=12900
# NESSIE_INTERNAL_PORT=12120
# # POSTGRES_INTERNAL_PORT tetap 5432 (postgres milik cluster di `state`)
#
# # WAJIB: SALIN nilai IAM & storage existing ke .env cluster SEBELUM --generate
# # (bundle MENYALIN .env). Kalau beda → OIDC / LDAP bind / S3 GAGAL:
# LDAP_ADMIN_PASSWORD=... # = password bind di host existing
# TRINO_CLIENT_SECRET=... STARROCKS_CLIENT_SECRET=... AIRFLOW_CLIENT_SECRET=...
# OPENMETADATA_CLIENT_SECRET=... GRAFANA_CLIENT_SECRET=...
# MINIO_ROOT_USER=... MINIO_ROOT_PASSWORD=... # = kredensial MinIO existing (S3)
# # Ambil dari .env host existing (atau ./scripts/print-*.sh di host itu).
# # Diubah SETELAH --generate → ulangi --generate + --sync (bundle embed nilai lama).
#
# ./install.sh --generate --inventory cluster.p1.yml
# ./install.sh --sync
# ./install.sh --host control # dan compute, state
#
# PENTING (host EXISTING): set `KEYCLOAK_SSL_REQUIRED=none` di .env host existing lalu
# jalankan `./bootstrap.sh` (idempoten). Dengan default `external`, HTTP dari alamat
# Tailscale 100.64/10 — dianggap NON-lokal oleh Keycloak — DITOLAK ("HTTPS required"),
# sehingga cluster tak bisa OIDC walau jaringan lancar. Lihat juga
# docs/keycloak-existing-requirements.md §2.
#
# PENTING (host EXISTING) — S3: set `S3_ENDPOINT=http://<alamat-existing>:12900` di .env
# host existing lalu recreate nessie. Sling memakai iceberg-go yang meminta
# `X-Iceberg-Access-Delegation: vended-credentials`; Nessie meng-vend `s3.endpoint` dari
# konfigurasinya (`minio:9000` internal) dan endpoint-vended MENANG atas `s3_endpoint` di
# ~/.sling/env.yaml → klien di host lain menembak `minio:9000` (gagal). Mengarahkan
# S3_ENDPOINT existing ke alamat publik membuat yang di-vend reachable.
#
# CATATAN: host `existing` hanya sebagai PENANDA tempat komponen — JANGAN deploy
# `generated/host-existing/` (itu akan mengarahkan keycloak/nessie-nya ke postgres
# cluster). Host existing tetap instalasi single-host fase 1–2.
# =============================================================================
version: 1
cluster:
name: sicabe-p1
env: simulation
tls:
mode: none
preset: a3
hosts:
# Host EXISTING (pihak lain) — hanya untuk resolusi endpoint & kelengkapan inventory.
existing:
address: 100.93.54.108
tiers: [identity] # keycloak, openldap, nessie
control:
address: 100.112.76.124
tiers: [policy, pipeline, governance]
compute:
address: 100.123.123.71
tiers: [coord, engine]
state:
address: 100.73.229.107
tiers: [state]
components:
minio: { host: existing } # minio ada di host existing (bukan cluster)
dremio: { enabled: false }
deploy/cluster.yml
+78
View File
@@ -0,0 +1,78 @@
# =============================================================================
# cluster.example.yml — inventory multi-host SiCABE (CONTOH; salin ke cluster.yml)
# -----------------------------------------------------------------------------
# Fase 1 R10 §4.3/§4.5 + taksonomi 7-tier R11.
#
# CARA PAKAI
# cp cluster.example.yml cluster.yml # lalu sesuaikan address
# ./install.sh --generate # hasil ke generated/host-<x>/
#
# Penting: berkas `cluster.yml` bersifat OPSIONAL. Bila TIDAK ADA, installer
# berjalan seperti sekarang (single-host) — `docker-compose.yml` dipakai langsung
# dan tidak ada berkas di `generated/` yang dibuat.
#
# PRECEDENCE (tertinggi → terendah)
# 1. components.<nama>.host (override eksplisit)
# 2. hosts.<host>.tiers (tier → host)
# 3. preset (a3 | b5) — hanya mengisi default tiers
# =============================================================================
version: 1
cluster:
name: sicabe-prod
env: production
# domain = basis nama BROWSER-FACING (lihat blok `tls` di bawah). Isi nama domain
# saja — tanpa skema, tanpa garis miring. Untuk mode `none` nilainya tidak dipakai.
domain: sicabe.com
# ─── TLS (langkah 1: baru SKEMA; Traefik + sertifikat menyusul di step 2) ─────
# none (DEFAULT) — perilaku sekarang: http, IP + port 12xxx. Bidang internal
# plaintext. Cocok untuk lab/uji tertutup.
# edge — browser memakai https://<nama>.<domain> lewat ingress;
# bidang internal (container↔container) TETAP plaintext.
# Pilihan paling murah untuk pilot: satu sertifikat wildcard,
# dan naik ke `full` nanti TIDAK mengubah URL/issuer.
# full — seluruh segmen TLS (butuh kepercayaan CA di semua klien).
# tls:
# mode: edge # contoh: aktifkan ingress berbasis nama
tls:
mode: edge
# a3 = 3 host (default) · b5 = 5 host (coord terpisah + 2 engine)
preset: a3
hosts:
control:
address: 100.112.76.124
# tier = kelompok peran (R11 §3.1). Preset a3 sudah mengisi ini; ditulis
# eksplisit agar terbaca. `control` mengelompokkan komponen ringan.
tiers: [identity, policy, pipeline, governance]
compute:
address: 100.123.123.71
# coord = trino-coordinator + starrocks-fe · engine = trino-worker + starrocks-cn
tiers: [coord, engine]
state:
address: 100.73.229.107
# state WAJIB tepat satu host dan tidak boleh digabung dengan `engine`.
# Berisi: postgres, minio (+minio-init), openmetadata-opensearch.
tiers: [state]
components:
# Override penempatan per komponen (prioritas tertinggi). Contoh: pindahkan
# StarRocks FE ke host control bila compute sudah terlalu padat.
# starrocks-fe: { host: control }
# Dremio default NONAKTIF. Bila diaktifkan: taruh di host bertier `engine`
# (butuh RAM ~4 GB lebih — cek kapasitas host).
dremio: { enabled: false }
# Contoh blueprint b5 (5 host): hapus komentar dan GANTI `preset` di atas.
# preset: b5
# hosts:
# control: { address: 10.20.0.10, tiers: [identity, policy, pipeline, governance] }
# coord: { address: 10.20.0.11, tiers: [coord] }
# engine-1: { address: 10.20.0.12, tiers: [engine] }
# engine-2: { address: 10.20.0.13, tiers: [engine] }
# state: { address: 10.20.0.14, tiers: [state] }
# Catatan b5: dua host `engine` menjalankan service `trino-worker` dan
# `starrocks-cn` masing-masing satu — jadi tiap host engine punya node sendiri.
deploy/config/airflow/airflow.cfg
+56
View File
@@ -0,0 +1,56 @@
# =============================================================================
# Apache Airflow — Configuration (Airflow 3.3.x)
# Referensi arsitektur: bab 4.5.1 (Apache Airflow — Orchestrator)
# =============================================================================
# ⚠️ Airflow 3.3 memakai SimpleAuthManager (bukan FabAuthManager).
# Config [api] auth_backends dan [webserver] auth_backend dari Airflow 2.x
# TIDAK berlaku lagi — dihapus agar task runner bisa autentikasi internal.
# =============================================================================
[core]
executor = LocalExecutor
simple_auth_manager_users = admin:Admin
simple_auth_manager_passwords_file = /opt/airflow/simple_auth_manager_passwords.json
parallelism = 32
max_active_tasks_per_dag = 16
dagbag_import_timeout = 30
load_examples = False
default_timezone = Asia/Jakarta
[database]
# ⚠️ ConfigParser does NOT expand ${VAR} syntax — this is a placeholder only.
# The actual connection string is set via AIRFLOW__DATABASE__SQL_ALCHEMY_CONN
# environment variable in docker-compose.yml.
sql_alchemy_conn = postgresql+psycopg2://airflow:PLACEHOLDER@postgres:5432/airflow
sql_alchemy_pool_enabled = True
sql_alchemy_pool_size = 10
sql_alchemy_max_overflow = 20
[webserver]
# base_url internal (container) — task runner connect ke sini
base_url = http://airflow-webserver:8080
default_ui_timezone = Asia/Jakarta
expose_config = True
[scheduler]
parsing_processes = 4
max_threads = 4
scheduler_heartbeat_sec = 5
[dag_processor]
# Airflow 3.x: dag_dir_list_interval dipindah dari [scheduler] ke sini
# (menghindari DeprecationWarning saat start).
refresh_interval = 30
[logging]
remote_logging = False
log_level = INFO
[email]
email_backend = airflow.utils.email.send_email_smtp
[smtp]
smtp_host = localhost
smtp_port = 25
smtp_timeout = 30
smtp_mail_from = airflow@sicabe.local
deploy/config/airflow/dags/dag_sitomat_gold_build.py
+96
View File
@@ -0,0 +1,96 @@
"""sitomat — DAG-3 (gold build) — TAHAP A: CONNECTIVITY SMOKE DAG (Airflow 3.x).
HONESTY (baca dulu):
DAG ini HANYA memvalidasi konektivitas sitomat <-> Airflow (validate DAG,
trigger run, poll status). Ia TIDAK membangun gold dan **TIDAK** memanggil
callback `register_gold` — jadi **tidak ada artefak gold yang diklaim**.
Tahap B akan menggantinya dengan transform nyata: baca silver (via Trino),
tulis gold, lalu callback ke sitomat. Lihat README.md di folder yang sama.
Airflow 3.x: dataclass import publik adalah `airflow.sdk` (bukan
`airflow.decorators`/`airflow.models`), dan `schedule_interval` sudah dihapus
-> gunakan `schedule=None` untuk DAG yang hanya dipicu lewat REST.
"""
from __future__ import annotations
import logging
import pendulum
from airflow.sdk import dag, get_current_context, task
log = logging.getLogger(__name__)
# Kunci conf yang WAJIB dikirim sitomat saat trigger (lihat F16 `flattenTriggerConf`).
REQUIRED_CONF_KEYS = (
"correlationId",
"dataProductBuildId",
"dataProductId",
"useCaseId",
)
@dag(
dag_id="sitomat_gold_build", # samakan dengan env `GOLD_DAG_ID` di sitomat
description="sitomat F16 DAG-3 — connectivity smoke (tidak membangun gold)",
schedule=None, # Airflow 3: hanya dipicu lewat REST (POST /api/v2/.../dagRuns)
start_date=pendulum.datetime(2026, 1, 1, tz="UTC"),
catchup=False,
tags=["sitomat", "f16", "connectivity"],
)
def sitomat_gold_build():
@task
def parse_conf() -> dict:
"""Baca `dag_run.conf` dari sitomat dan validasi kunci wajib."""
ctx = get_current_context()
conf = dict((ctx["dag_run"].conf or {})) if ctx else {}
missing = [key for key in REQUIRED_CONF_KEYS if not conf.get(key)]
if missing:
# Gagal jujur (BUKAN sukses palsu): conf tidak lengkap.
raise ValueError(f"conf sitomat tidak lengkap, hilang: {missing}")
silver_refs = [ref for ref in str(conf.get("silverRefs", "")).split(",") if ref]
log.info(
"sitomat conf OK: correlationId=%s build=%s product=%s useCase=%s "
"sources=%s silverRefs=%d",
conf.get("correlationId"),
conf.get("dataProductBuildId"),
conf.get("dataProductId"),
conf.get("useCaseId"),
conf.get("sourceCount"),
len(silver_refs),
)
if not silver_refs:
log.warning(
"silverRefs kosong — Tahap B (build nyata) belum bisa membaca sumber."
)
return {
"correlationId": conf.get("correlationId"),
"dataProductBuildId": conf.get("dataProductBuildId"),
"silverRefs": silver_refs,
}
@task
def connectivity_report(payload: dict) -> None:
"""TAHAP A: sengaja hanya melaporkan. TIDAK menulis gold.
Tahap B (ganti task ini):
1. baca `payload["silverRefs"]` via Trino (koneksi `trino_default`),
2. rakit & tulis tabel gold (Iceberg/Nessie),
3. POST ke sitomat:
{SITOMAT_BASE_URL}/data-products/builds/{dataProductBuildId}/gold
header: `X-API-Key: <kunci scope pipeline>`
body: {"goldRef": ..., "checksum": ..., "rowCount": ...}
Simpan URL & kunci sebagai Airflow Variable/Connection (JANGAN hardcode).
"""
log.info(
"sitomat DAG-3 connectivity OK untuk build=%s (Tahap A: tanpa gold)",
payload.get("dataProductBuildId"),
)
connectivity_report(parse_conf())
sitomat_gold_build()
deploy/config/airflow/dags/dag_sitomat_gold_build_prod.py
+344
View File
@@ -0,0 +1,344 @@
"""
DAG-3 PRODUKSI — bangun GOLD untuk sebuah Data Product, lalu lapor ke sitomat.
Fitur F16/F15 (ADR-0004..0008). sitomat TIDAK memindahkan byte: DAG ini yang
membangun artefak di data lake (Iceberg lewat Trino) dan memanggil callback.
Kontrak
-------
Trigger conf (dikirim F15 `createGoldBuildTriggerPort`; SEMUA nilai STRING):
correlationId, dataProductBuildId, dataProductId, useCaseId,
datasetVersionIds (dipisah koma), silverRefs (dipisah koma; posisinya
sejajar dengan datasetVersionIds, slot kosong = sumber tanpa ref),
sourceCount
Callback WAJIB (tanpa ini build sitomat tetap 'building'):
POST {SITOMAT_BASE_URL}/data-products/builds/{dataProductBuildId}/gold
Header: X-API-Key: <kunci F09-A scope `pipeline`>
Body: {"goldRef": "...", "checksum": "sha256:...", "rowCount": N}
Konfigurasi — Airflow Variable dulu, lalu env, lalu default:
sitomat_base_url SITOMAT_BASE_URL http://172.27.228.103:13000
sitomat_api_key SITOMAT_API_KEY (WAJIB) kunci scope `pipeline`
trino_url TRINO_URL https://sicabe-trino:8443
trino_user TRINO_USER svc_seed
trino_password TRINO_PASSWORD (WAJIB)
gold_target GOLD_TARGET iceberg.gold.dp_<dataProductId>
gold_sql GOLD_SQL opsional; bila kosong dirakit dari silverRefs
gold_tls_verify GOLD_TLS_VERIFY false (sertifikat internal self-signed)
checksum_max_rows CHECKSUM_MAX_ROWS 100000 (0 = tanpa batas)
**Presedensi:** `silverRefs` dari conf adalah jalur NYATA dan menang; `gold_sql`
= **fallback** untuk dataset yang belum menyimpan `artifact_ref`.
**Kegagalan DILAPORKAN, bukan menggantung:** build NYATA yang sumbernya belum
punya `artifact_ref` (atau tanpa sumber sama sekali) **GAGAL dengan sengaja** —
DAG menolak membangun produk parsial dan task `report_failure` memanggil
`POST …/fail`, sehingga build/produk/use case menjadi `failed`. Hanya run
DIAGNOSTIK (`bun run check:airflow`) yang berhenti diam tanpa callback.
Cakupan OPA: stack ini memakai allowlist per-tabel (`default allow := false`).
Menulis ke tabel gold BARU mensyaratkan policy OPA memberi grup `data-engineer`
akses pada schema gold/silver (lihat README). Bila belum, set `GOLD_TARGET` ke
tabel yang sudah diizinkan (mis. `iceberg.gold.orders_gold`).
"""
from __future__ import annotations
import hashlib
import json
import os
import re
import requests
import urllib3
from airflow.sdk import dag, get_current_context, task
# Sertifikat internal self-signed: jangan banjiri log dengan peringatan.
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
DEFAULT_BASE_URL = "http://172.27.228.103:13000"
DEFAULT_TRINO_URL = "https://sicabe-trino:8443"
DEFAULT_TRINO_USER = "svc_seed"
DEFAULT_CHECKSUM_MAX_ROWS = 100000
# Nama identifier SQL yang aman; apa pun di luar ini DITOLAK (anti-injeksi).
IDENTIFIER = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*$")
# ── Pengaturan (Airflow Variable → env → default) ────────────────────────────
def _variable_get(key: str) -> str | None:
"""Baca Airflow Variable `key`; `None` bila tidak ada / tidak tersedia."""
Variable = None
try:
from airflow.sdk import Variable # Airflow 3 SDK
except Exception: # pragma: no cover - tergantung versi Airflow
try:
from airflow.models import Variable # fallback klasik
except Exception:
return None
try:
return Variable.get(key)
except Exception:
return None
def setting(name: str, default: str | None = None) -> str | None:
"""Airflow Variable `name` (huruf kecil) → env `NAME` (huruf besar) → default."""
value = os.environ.get(name.upper())
if value:
return value
variable = _variable_get(name.lower())
return variable if variable else default
# ── Util murni ───────────────────────────────────────────────────────────────
def qualified(identifier: str) -> str:
"""Validasi `tabel` / `schema.tabel` / `catalog.schema.tabel`. Raise bila bukan."""
raw = identifier.strip().strip('"').strip("`")
parts = [part.strip().strip('"').strip("`") for part in raw.split(".")]
if not 1 <= len(parts) <= 3 or not all(IDENTIFIER.match(part) for part in parts):
raise ValueError(f"nama tabel tidak valid: {identifier!r}")
return ".".join(parts)
def safe_suffix(value: str) -> str:
"""Jadikan nilai bebas menjadi sufiks identifier yang aman."""
cleaned = re.sub(r"[^A-Za-z0-9_]", "_", value or "x")
if not cleaned or cleaned[0].isdigit():
cleaned = f"t_{cleaned}"
return cleaned
def _as_int(value) -> int:
"""Integer non-negatif dari conf (semua nilai conf adalah STRING)."""
try:
return max(0, int(str(value).strip()))
except (TypeError, ValueError):
return 0
def target_and_sql(conf: dict, sources: list[str]) -> tuple[str, str]:
"""(target_fqn, sql_ctas) dari conf + pengaturan.
`GOLD_SQL` (Variable `gold_sql`) adalah escape hatch yang disengaja: dipakai
bila `silverRefs` kosong (mis. versi silver lama belum menyimpan `artifact_ref`)
ATAU untuk mengganti transform. Tanpa keduanya, tidak ada yang bisa dibangun.
"""
configured = setting("gold_target")
target = qualified(
configured
or f"iceberg.gold.dp_{safe_suffix(str(conf.get('dataProductId', 'x')))}"
)
explicit = setting("gold_sql")
# PRESEDENSI: sumber dari conf (`silverRefs`) adalah jalur NYATA dan menang;
# `gold_sql` hanya FALLBACK operator untuk dataset yang belum punya
# `artifact_ref`. (Kebalikannya membuat override global menelan lineage.)
if sources:
select = " UNION ALL ".join(f"SELECT * FROM {source}" for source in sources)
elif explicit:
select = explicit.strip().rstrip(";")
else:
raise ValueError("tidak ada silverRefs dan tidak ada gold_sql — tidak ada yang dibangun")
return target, f"CREATE OR REPLACE TABLE {target} AS {select}"
# ── Klien Trino REST minimal (/v1/statement + paging nextUri) ─────────────────
class Trino:
"""Klien Trino HTTPS berbatas; tidak ada dependensi di luar `requests`."""
def __init__(self, url: str, user: str, password: str | None, verify: bool) -> None:
self.url = url.rstrip("/")
self.session = requests.Session()
self.session.verify = verify
self.session.auth = (user, password)
self.session.headers.update({"X-Trino-User": user, "Content-Type": "text/plain"})
def pages(self, sql: str):
"""Iterasi (columns, rows) per halaman; raise bila Trino melaporkan error."""
response = self.session.post(f"{self.url}/v1/statement", data=sql, timeout=300)
response.raise_for_status()
payload = response.json()
for _ in range(10_000): # pengaman halaman, bukan batas logis
error = payload.get("error")
if error:
raise RuntimeError(f"trino error: {error.get('message', error)}")
columns = [col.get("name") for col in payload.get("columns", [])]
yield columns, payload.get("data", [])
next_uri = payload.get("nextUri")
if not next_uri:
return
payload = self.session.get(next_uri, timeout=300).json()
raise RuntimeError("trino: terlalu banyak halaman tanpa akhir")
def execute(self, sql: str) -> None:
"""Jalankan statement tanpa peduli baris (mis. CTAS)."""
for _ in self.pages(sql):
pass
def scalar(self, sql: str):
"""Nilai pertama baris pertama."""
for _columns, rows in self.pages(sql):
if rows:
return rows[0][0]
return None
def checksum(self, table: str, max_rows: int) -> str:
"""sha256 deterministik atas header + baris (JSON ringkas), opsional dibatasi."""
digest = hashlib.sha256()
header_written = False
seen = 0
for columns, rows in self.pages(f"SELECT * FROM {table}"):
if not header_written and columns:
digest.update((",".join(columns) + "\n").encode("utf-8"))
header_written = True
for row in rows:
if max_rows and seen >= max_rows:
return f"sha256:{digest.hexdigest()} (bounded:{max_rows})"
digest.update(
(json.dumps(row, separators=(",", ":"), default=str) + "\n").encode("utf-8")
)
seen += 1
return f"sha256:{digest.hexdigest()}"
def trino_client() -> Trino:
password = setting("trino_password")
if not password:
raise RuntimeError("trino_password belum diatur (Airflow Variable atau TRINO_PASSWORD)")
return Trino(
setting("trino_url", DEFAULT_TRINO_URL) or DEFAULT_TRINO_URL,
setting("trino_user", DEFAULT_TRINO_USER) or DEFAULT_TRINO_USER,
password,
str(setting("gold_tls_verify", "false")).lower() in {"1", "true", "yes"},
)
# ── DAG ──────────────────────────────────────────────────────────────────────
@dag(
dag_id="sitomat_gold_build_prod",
schedule=None,
catchup=False,
tags=["sitomat", "f16", "gold"],
doc_md=__doc__,
)
def sitomat_gold_build_prod():
@task
def build_gold() -> dict:
"""CTAS gold dari silverRefs; kembalikan identitas artefak."""
conf = get_current_context()["dag_run"].conf or {}
build_id = str(conf.get("dataProductBuildId", "")).strip()
# Run DIAGNOSTIK (`bun run check:airflow`) tidak punya build nyata.
diagnostic = not build_id or build_id == "airflow-check"
source_count = _as_int(conf.get("sourceCount"))
sources = [
qualified(raw)
for raw in str(conf.get("silverRefs") or "").split(",")
if raw.strip()
]
missing = max(0, source_count - len(sources))
explicit = setting("gold_sql")
# Run diagnostik tanpa sumber: tidak ada yang dibangun, tidak ada callback.
if diagnostic and not sources and not explicit:
print("[sitomat] run diagnostik tanpa sumber — tidak membangun, tidak callback")
return {"built": False}
# PRODUK PARSIAL DITOLAK. Sumber yang belum punya `artifact_ref` TIDAK
# boleh diam-diam dihilangkan (itu menghasilkan gold yang salah). Untuk
# build NYATA kita GAGAL di sini, sehingga task `report_failure`
# melaporkannya ke sitomat — bukan menggantung `building` selamanya.
if missing > 0 and not explicit:
raise ValueError(
f"build {build_id}: {missing} dari {source_count} sumber tidak punya "
"artifact_ref — menolak membangun produk parsial"
)
target, sql = target_and_sql(conf, sources)
client = trino_client()
print(f"[sitomat] membangun {target} dari {len(sources)} sumber")
client.execute(sql)
row_count = int(client.scalar(f"SELECT count(*) FROM {target}") or 0)
checksum = client.checksum(target, int(setting("checksum_max_rows", str(DEFAULT_CHECKSUM_MAX_ROWS))))
print(f"[sitomat] selesai: {target} rows={row_count} checksum={checksum}")
return {
"built": True,
"goldRef": target,
"rowCount": row_count,
"checksum": checksum,
"buildId": str(conf.get("dataProductBuildId", "")),
}
@task
def register_gold(build: dict) -> None:
"""Laporkan artefak ke sitomat (callback `register_gold`)."""
if not build.get("built"):
return
base = (setting("sitomat_base_url", DEFAULT_BASE_URL) or DEFAULT_BASE_URL).rstrip("/")
api_key = setting("sitomat_api_key")
if not api_key:
raise RuntimeError("sitomat_api_key belum diatur (Airflow Variable atau SITOMAT_API_KEY)")
url = f"{base}/data-products/builds/{build['buildId']}/gold"
response = requests.post(
url,
json={
"goldRef": build["goldRef"],
"checksum": build["checksum"],
"rowCount": build["rowCount"],
},
headers={"X-API-Key": api_key},
timeout=60,
)
if response.status_code >= 300:
raise RuntimeError(
f"callback sitomat gagal: HTTP {response.status_code} {response.text[:300]}"
)
print(f"[sitomat] callback OK ({response.status_code}): {response.text[:200]}")
@task(trigger_rule="one_failed")
def report_failure() -> None:
"""Laporkan kegagalan ke sitomat agar build TIDAK menggantung `building`.
Best-effort: kegagalan pelaporan tidak boleh menutupi error asli DAG.
"""
context = get_current_context()
conf = context["dag_run"].conf or {}
build_id = str(conf.get("dataProductBuildId", "")).strip()
# Run diagnostik (`bun run check:airflow`) tidak punya build nyata.
if not build_id or build_id == "airflow-check":
print("[sitomat] tidak ada build nyata untuk dilaporkan (mode konektivitas)")
return
base = (setting("sitomat_base_url", DEFAULT_BASE_URL) or DEFAULT_BASE_URL).rstrip("/")
api_key = setting("sitomat_api_key")
if not api_key:
print("[sitomat] sitomat_api_key tidak diatur — kegagalan TIDAK dilaporkan")
return
cause = f"DAG-3 sitomat_gold_build_prod failed (run {context['dag_run'].run_id})"
try:
response = requests.post(
f"{base}/data-products/builds/{build_id}/fail",
json={"cause": cause},
headers={"X-API-Key": api_key},
timeout=60,
)
print(f"[sitomat] failure callback -> HTTP {response.status_code}")
except Exception as error: # noqa: BLE001 - never mask the original failure
print(f"[sitomat] failure callback GAGAL (diabaikan): {error}")
built = build_gold()
registered = register_gold(built)
# `one_failed` + kedua upstream: lapor bila SALAH SATU task gagal.
[built, registered] >> report_failure()
sitomat_gold_build_prod()
deploy/config/airflow/dags/example_sicabe_pipeline.py
+213
View File
@@ -0,0 +1,213 @@
"""
# =============================================================================
# eBdesk SiCABE — Contoh DAG Pipeline End-to-End
#
# Referensi arsitektur:
# bab 5.4 (Ingestion → Processing: Pipeline Integration)
# bab 8.1 (Skenario End-to-End: Batch Harian)
# bab 8.2 (Skenario End-to-End: CDC Real-Time)
# =============================================================================
Alur:
1. [Ingestion] Sling tarik data dari PostgreSQL → Bronze (Iceberg via Nessie)
2. [Processing] dbt via Trino: Bronze → Silver → Gold
3. [Governance] Great Expectations validasi output pipeline DI ICEBERG
Prerequisites (dibuat oleh bootstrap.sh):
- Airflow connection: minio_s3 (object storage)
- Sling env: ~/.sling/env.yaml (koneksi POSTGRES_DB + ICEBERG)
- dbt project: /opt/airflow/dbt/sicabe/ (bind-mount dari config/airflow/dbt) dengan model bronze_to_silver, silver_to_gold, incremental
- Tabel sumber: public.orders di database sicabe_data (PostgreSQL)
- Kredensial Trino service account: env DBT_TRINO_USER / DBT_TRINO_PASSWORD
Catatan penting (jangan diubah tanpa alasan):
- Task dbt memakai `env=DBT_TASK_ENV, append_env=True`. Tanpa `append_env=True`,
`env=` MENGGANTI seluruh environment sehingga PATH & HOME hilang dan `dbt`
(terpasang di ~/.local/bin) tidak ditemukan: "dbt: command not found".
- Validator Great Expectations membaca tabel Iceberg LEWAT TRINO (HTTPS 8443,
service account), bukan tabel sumber di PostgreSQL — supaya hasil "LULUS"
benar-benar membuktikan output pipeline (bronze/gold), bukan data mentah.
GE tidak punya datasource Trino → dipakai Pandas datasource dari hasil query.
"""
from datetime import datetime, timedelta
import os
from airflow import DAG
from airflow.providers.standard.operators.bash import BashOperator
from airflow.providers.standard.operators.python import PythonOperator
def _run_ge_to_om(table):
"""Jalankan validasi Great Expectations DAN kirim hasilnya ke OpenMetadata.
Runner adalah SATU sumber logika validasi (dipakai juga di luar DAG) dan
memakai action resmi OpenMetadata untuk GE 1.x, sehingga hasilnya muncul
sebagai Data Quality Test Suite/Test Case di OpenMetadata — TERMASUK ketika
GAGAL. Sebelumnya hasil GE hanya berhenti di log Airflow.
Fail-fast tetap dipertahankan: exit code non-nol → task FAILED.
"""
import subprocess
import sys
script = os.getenv(
"GE_OM_RUNNER", "/opt/airflow/great_expectations/validate_to_openmetadata.py"
)
proc = subprocess.run(
[sys.executable, script, "--table", table], capture_output=True, text=True
)
for stream in (proc.stdout, proc.stderr):
if stream:
print(stream[-2000:])
if proc.returncode != 0:
raise RuntimeError(
f"Data quality check GAGAL untuk {table} (hasil tetap dikirim ke OpenMetadata)"
)
def validate_gold_data():
"""Validasi GOLD (output akhir pipeline) — `gold.orders_gold` → hasil ke OM."""
_run_ge_to_om("gold.orders_gold")
def validate_bronze_data():
"""Validasi BRONZE (hasil ingestion Sling) — `bronze.orders` → hasil ke OM."""
_run_ge_to_om("bronze.orders")
# Env untuk task dbt: autentikasi Trino via SERVICE ACCOUNT (HTTP Basic) langsung
# ke HTTPS 8443 (lihat config/airflow/dbt/profiles.yml). Password dari env
# DBT_TRINO_PASSWORD (di-set docker-compose untuk service Airflow).
DBT_TASK_ENV = {
"DBT_TRINO_USER": os.getenv("DBT_TRINO_USER", "svc_dbt"),
"DBT_TRINO_PASSWORD": os.getenv("DBT_TRINO_PASSWORD", ""),
}
# =============================================================================
# Default Arguments
# =============================================================================
default_args = {
"owner": "engineer",
"depends_on_past": False,
"email_on_failure": True,
"email_on_retry": False,
"retries": 2,
"retry_delay": timedelta(minutes=5),
"execution_timeout": timedelta(hours=2),
}
# =============================================================================
# DAG — Batch Harian (bab 8.1)
# =============================================================================
BATCH_DAG_ID = "sicabe_batch_daily"
BATCH_SCHEDULE = "0 2 * * *" # Setiap hari jam 02:00
with DAG(
dag_id=BATCH_DAG_ID,
default_args=default_args,
description="SiCABE: Batch harian — Ingestion → Processing → Governance",
schedule=BATCH_SCHEDULE,
start_date=datetime(2026, 1, 1),
catchup=False,
tags=["sicabe", "batch", "bronze", "silver", "gold"],
) as dag:
# ─── Step 1: Ingestion — Tarik data ke Bronze (Iceberg) ──────────────
ingest_to_bronze = BashOperator(
task_id="ingest_to_bronze",
bash_command=(
'sling run --src-conn POSTGRES_DB '
'--src-stream public.orders '
'--tgt-conn ICEBERG '
'--tgt-object bronze.orders '
'--mode full-refresh '
'--stdout'
),
)
# ─── Step 2: Processing — Transform Bronze → Silver ──────────────────
bronze_to_silver = BashOperator(
task_id="bronze_to_silver",
bash_command=(
'cd /opt/airflow/dbt/sicabe && '
'export DBT_PROFILES_DIR=/opt/airflow/dbt && '
'dbt run --select bronze_to_silver --profile trino --target prod'
),
env=DBT_TASK_ENV,
append_env=True,
)
# ─── Step 3: Processing — Transform Silver → Gold ────────────────────
silver_to_gold = BashOperator(
task_id="silver_to_gold",
bash_command=(
'cd /opt/airflow/dbt/sicabe && '
'export DBT_PROFILES_DIR=/opt/airflow/dbt && '
'dbt run --select silver_to_gold --profile trino --target prod'
),
env=DBT_TASK_ENV,
append_env=True,
)
# ─── Step 4: Governance — Validasi GOLD di Iceberg ───────────────────
validate_gold = PythonOperator(
task_id="validate_gold",
python_callable=validate_gold_data,
)
# ─── Dependency Chain ────────────────────────────────────────────────
ingest_to_bronze >> bronze_to_silver >> silver_to_gold >> validate_gold
# =============================================================================
# DAG — CDC Real-Time (bab 8.2)
# =============================================================================
CDC_DAG_ID = "sicabe_cdc_realtime"
CDC_SCHEDULE = "*/15 * * * *" # Setiap 15 menit
with DAG(
dag_id=CDC_DAG_ID,
default_args=default_args,
description="SiCABE: CDC real-time — Incremental ingestion + processing",
schedule=CDC_SCHEDULE,
start_date=datetime(2026, 1, 1),
catchup=False,
tags=["sicabe", "cdc", "incremental", "realtime"],
) as dag:
# ─── Step 1: CDC Ingestion via Sling ─────────────────────────────────
cdc_ingest = BashOperator(
task_id="cdc_ingest",
bash_command=(
'sling run --src-conn POSTGRES_DB '
'--src-stream public.orders '
'--tgt-conn ICEBERG '
'--tgt-object bronze.orders '
'--mode incremental '
'--primary-key id '
'--update-key updated_at '
'--stdout'
),
)
# ─── Step 2: dbt incremental update ──────────────────────────────────
dbt_incremental = BashOperator(
task_id="dbt_incremental",
bash_command=(
'cd /opt/airflow/dbt/sicabe && '
'export DBT_PROFILES_DIR=/opt/airflow/dbt && '
'dbt run --select incremental --profile trino --target prod'
),
env=DBT_TASK_ENV,
append_env=True,
)
# ─── Step 3: Validasi BRONZE di Iceberg ──────────────────────────────
cdc_validate = PythonOperator(
task_id="cdc_validate",
python_callable=validate_bronze_data,
)
cdc_ingest >> dbt_incremental >> cdc_validate
deploy/config/airflow/dags/sample_sql_lineage.py
+58
View File
@@ -0,0 +1,58 @@
# =============================================================================
# SAMPLE — Lineage otomatis via operator SQL (tanpa inlets/outlets)
# =============================================================================
# Tujuan : membuktikan bahwa task yang memakai operator SQL memperoleh lineage
# tabel SECARA OTOMATIS dari teks SQL, tanpa perlu menuliskan
# inlets/outlets.
#
# Alur lineage yang diharapkan:
# iceberg.bronze.orders ---> iceberg.silver.orders_clean
#
# Cara kerja:
# Provider OpenLineage mem-parsing SQL task (lihat
# airflow.providers.openlineage.utils.sql_hook_lineage) dan mengekstrak
# tabel input/output, lalu mengirimnya ke OpenMetadata.
#
# Prasyarat di Airflow:
# 1. Koneksi `trino_ol` (type trino) menunjuk ke Trino koordinator.
# 2. Transport OpenLineage aktif + shim eventTime (lihat shim/ol_time_shim.py).
# 3. AIRFLOW__OPENLINEAGE__EXECUTE_IN_THREAD=true (tanpa ini emisi HANG).
#
# Catatan: DAG ini tidak memakai inlets/outlets secara sengaja.
# =============================================================================
from datetime import datetime
from airflow import DAG
from airflow.providers.common.sql.operators.sql import SQLExecuteQueryOperator
CONN_ID = "trino_ol"
CLEAN_SQL = """
CREATE TABLE IF NOT EXISTS iceberg.silver.orders_clean AS
SELECT
id,
order_date,
komoditas,
harga_per_kg,
stok_kg,
region
FROM iceberg.bronze.orders
WHERE id IS NOT NULL
"""
with DAG(
dag_id="sample_sql_lineage",
description="SAMPLE: lineage otomatis via operator SQL (bronze.orders -> silver.orders_clean)",
start_date=datetime(2026, 1, 1),
schedule=None, # dijalankan manual
catchup=False,
tags=["sample", "lineage", "openlineage", "sql"],
) as dag:
# Lineage bronze.orders -> silver.orders_clean diekstrak otomatis dari SQL ini.
clean_orders = SQLExecuteQueryOperator(
task_id="clean_orders",
conn_id=CONN_ID,
sql=CLEAN_SQL,
)
deploy/config/airflow/dags/sicabe_dq_gold_native.py
+9
View File
@@ -0,0 +1,9 @@
"""
This file has been generated from dag_runner.j2
"""
from airflow import DAG
from openmetadata_managed_apis.workflows import workflow_factory
workflow = workflow_factory.WorkflowFactory.create("/opt/airflow/dag_generated_configs/sicabe_dq_gold_native.json")
workflow.generate_dag(globals())
dag = workflow.get_dag()
deploy/config/airflow/dbt/profiles.yml
+39
View File
@@ -0,0 +1,39 @@
# =============================================================================
# eBdesk SiCABE — dbt Profiles (Trino Connection)
# =============================================================================
# Service/pipeline memakai SERVICE ACCOUNT via HTTP Basic (dbt-trino method
# 'ldap' = username/password Basic). Koneksi LANGSUNG ke Trino HTTPS
# (self-signed) → cert: false (lewati verifikasi TLS) untuk dev.
# Password dari env DBT_TRINO_PASSWORD.
# =============================================================================
trino:
target: prod
outputs:
prod:
type: trino
method: ldap
user: "{{ env_var('DBT_TRINO_USER', 'svc_dbt') }}"
password: "{{ env_var('DBT_TRINO_PASSWORD', '') }}"
database: iceberg
schema: bronze
host: "{{ env_var('DBT_TRINO_HOST', 'trino-coordinator') }}"
port: 8443
threads: 4
http_scheme: https
cert: false
suppress_cert_warning: true
dev:
type: trino
method: ldap
user: "{{ env_var('DBT_TRINO_USER', 'svc_dbt') }}"
password: "{{ env_var('DBT_TRINO_PASSWORD', '') }}"
database: iceberg
schema: bronze
host: "{{ env_var('DBT_TRINO_HOST', 'localhost') }}"
port: 12443
threads: 2
http_scheme: https
cert: false
suppress_cert_warning: true
deploy/config/airflow/dbt/sicabe/dbt_project.yml
+56
View File
@@ -0,0 +1,56 @@
# =============================================================================
# eBdesk SiCABE — dbt Project Configuration
# =============================================================================
# Referensi arsitektur: bab 8.1 (Batch Harian), bab 8.2 (CDC Real-Time)
# Pipeline: Bronze → Silver → Gold
# =============================================================================
name: "sicabe"
version: "1.0.0"
config-version: 2
require-dbt-version: [">=1.8.0", "<2.0.0"]
profile: "trino"
model-paths: ["models"]
analysis-paths: ["analyses"]
test-paths: ["tests"]
seed-paths: ["seeds"]
macro-paths: ["macros"]
snapshot-paths: ["snapshots"]
target-path: "target"
clean-targets:
- "target"
- "dbt_packages"
models:
sicabe:
bronze_to_silver:
+materialized: table
+schema: silver
+database: iceberg
silver_to_gold:
+materialized: table
+schema: gold
+database: iceberg
incremental:
+materialized: incremental
+schema: bronze
+database: iceberg
+unique_key: "id"
+incremental_strategy: merge
# ───────────────────────────────────────────────────────────────────────
# WAJIB: relasi sementara harus berupa TABEL, bukan VIEW.
#
# Katalog Iceberg Trino di stack ini bertipe `nessie`, dan Trino TIDAK
# mendukung view di sana. Tanpa setelan ini, setiap run incremental gagal:
# TrinoUserError(NOT_SUPPORTED):
# "createView is not supported for Iceberg Nessie catalogs"
#
# Sebabnya: untuk strategi `merge`/`append`, dbt-trino membuat relasi
# sementara sebagai VIEW. `views_enabled: false` membuatnya memakai tabel.
# Dibuktikan 2026-09-22: MERGE (1 rows) berhasil untuk baris baru maupun
# baris yang diubah (upsert). Lihat reviews/sicabe-15-transform-service.md.
# ───────────────────────────────────────────────────────────────────────
+views_enabled: false
deploy/config/airflow/dbt/sicabe/macros/generate_schema_name.sql
+15
View File
@@ -0,0 +1,15 @@
-- =============================================================================
-- eBdesk SiCABE — dbt Macro: generate_schema_name
-- =============================================================================
-- Override default schema naming agar +schema menghasilkan nama persis
-- (bukan {default}_{custom}). Contoh: +schema: gold → schema "gold"
-- =============================================================================
{% macro generate_schema_name(custom_schema_name, node) -%}
{%- set default_schema = target.schema -%}
{%- if custom_schema_name is none -%}
{{ default_schema }}
{%- else -%}
{{ custom_schema_name | trim }}
{%- endif -%}
{%- endmacro %}
deploy/config/airflow/dbt/sicabe/models/bronze_to_silver/orders_silver.sql
+36
View File
@@ -0,0 +1,36 @@
-- =============================================================================
-- eBdesk SiCABE — dbt Model: Bronze → Silver
-- =============================================================================
-- Transformasi data mentah (bronze.orders) menjadi data bersih (silver.orders):
-- - Normalisasi tipe data
-- - Filter data valid (harga > 0, stok >= 0)
-- - Standarisasi format tanggal
-- =============================================================================
WITH bronze_orders AS (
SELECT
id,
CAST(order_date AS DATE) AS order_date,
komoditas,
CAST(harga_per_kg AS DOUBLE) AS harga_per_kg,
CAST(stok_kg AS BIGINT) AS stok_kg,
region,
created_at,
updated_at
FROM {{ source('iceberg', 'orders') }}
)
SELECT
id,
order_date,
komoditas,
harga_per_kg,
stok_kg,
region,
-- Nilai total per transaksi
ROUND(harga_per_kg * stok_kg, 2) AS total_value,
created_at,
updated_at
FROM bronze_orders
WHERE harga_per_kg > 0
AND stok_kg >= 0
deploy/config/airflow/dbt/sicabe/models/incremental/orders_incremental.sql
+29
View File
@@ -0,0 +1,29 @@
-- =============================================================================
-- eBdesk SiCABE — dbt Model: Incremental (CDC)
-- =============================================================================
-- Incremental update untuk data bronze.orders:
-- - Hanya memproses baris baru/berubah (berdasarkan updated_at)
-- - Materialized: incremental dengan unique_key = id
-- =============================================================================
{{ config(
materialized='incremental',
unique_key='id',
incremental_strategy='merge'
) }}
SELECT
id,
CAST(order_date AS DATE) AS order_date,
komoditas,
CAST(harga_per_kg AS DOUBLE) AS harga_per_kg,
CAST(stok_kg AS BIGINT) AS stok_kg,
region,
created_at,
updated_at
FROM {{ source('iceberg', 'orders') }}
{% if is_incremental() %}
-- Hanya ambil data yang baru/berubah sejak run terakhir
WHERE updated_at > (SELECT MAX(updated_at) FROM {{ this }})
{% endif %}
deploy/config/airflow/dbt/sicabe/models/silver_to_gold/orders_gold.sql
+35
View File
@@ -0,0 +1,35 @@
-- =============================================================================
-- eBdesk SiCABE — dbt Model: Silver → Gold
-- =============================================================================
-- Agregasi data bersih (silver.orders) menjadi data analitik (gold.orders):
-- - Rata-rata harga per komoditas per hari
-- - Total stok per komoditas
-- - Ranking komoditas berdasarkan nilai transaksi
-- =============================================================================
WITH silver_orders AS (
SELECT
id,
order_date,
komoditas,
harga_per_kg,
stok_kg,
region,
total_value
FROM {{ ref('orders_silver') }}
)
SELECT
order_date,
komoditas,
region,
COUNT(*) AS jumlah_transaksi,
ROUND(AVG(harga_per_kg), 2) AS avg_harga_per_kg,
ROUND(MIN(harga_per_kg), 2) AS min_harga_per_kg,
ROUND(MAX(harga_per_kg), 2) AS max_harga_per_kg,
SUM(stok_kg) AS total_stok_kg,
ROUND(SUM(total_value), 2) AS total_nilai,
-- Harga rata-rata tertimbang (weighted average)
ROUND(SUM(harga_per_kg * stok_kg) / NULLIF(SUM(stok_kg), 0), 2) AS weighted_avg_harga
FROM silver_orders
GROUP BY order_date, komoditas, region
deploy/config/airflow/dbt/sicabe/models/sources.yml
+32
View File
@@ -0,0 +1,32 @@
# =============================================================================
# eBdesk SiCABE — dbt Sources
# =============================================================================
# Mendefinisikan sumber data di Iceberg (via Trino catalog)
# =============================================================================
version: 2
sources:
- name: iceberg
database: iceberg
schema: bronze
tables:
- name: orders
description: "Data mentah orders dari PostgreSQL (via Sling)"
columns:
- name: id
description: "Primary key"
- name: order_date
description: "Tanggal order"
- name: komoditas
description: "Nama komoditas"
- name: harga_per_kg
description: "Harga per kilogram"
- name: stok_kg
description: "Stok dalam kilogram"
- name: region
description: "Wilayah"
- name: created_at
description: "Waktu dibuat"
- name: updated_at
description: "Waktu diupdate"
deploy/config/airflow/great_expectations/checkpoints/cdc_quality_checkpoint.json
+43
View File
@@ -0,0 +1,43 @@
{
"name": "cdc_quality_checkpoint",
"config_version": 4.0,
"template_name": null,
"run_name_template": "cdc-quality-%Y%m%d-%H%M%S",
"expectation_suite_name": "bronze_orders_suite",
"batch_request": {
"datasource_name": "trino_iceberg",
"data_connector_name": "default_inferred_data_connector_name",
"data_asset_name": "bronze.orders",
"batch_spec_passthrough": {}
},
"action_list": [
{
"name": "store_validation_result",
"action": {
"class_name": "StoreValidationResultAction"
}
},
{
"name": "update_data_docs",
"action": {
"class_name": "UpdateDataDocsAction"
}
}
],
"evaluation_parameters": {},
"runtime_configuration": {},
"validations": [
{
"batch_request": {
"datasource_name": "trino_iceberg",
"data_connector_name": "default_inferred_data_connector_name",
"data_asset_name": "bronze.orders"
},
"expectation_suite_name": "bronze_orders_suite",
"action_list": []
}
],
"profilers": [],
"ge_cloud_id": null,
"expectation_suite_ge_cloud_id": null
}
deploy/config/airflow/great_expectations/checkpoints/gold_quality_checkpoint.json
+43
View File
@@ -0,0 +1,43 @@
{
"name": "gold_quality_checkpoint",
"config_version": 4.0,
"template_name": null,
"run_name_template": "gold-quality-%Y%m%d-%H%M%S",
"expectation_suite_name": "gold_orders_suite",
"batch_request": {
"datasource_name": "trino_iceberg",
"data_connector_name": "default_inferred_data_connector_name",
"data_asset_name": "gold.orders",
"batch_spec_passthrough": {}
},
"action_list": [
{
"name": "store_validation_result",
"action": {
"class_name": "StoreValidationResultAction"
}
},
{
"name": "update_data_docs",
"action": {
"class_name": "UpdateDataDocsAction"
}
}
],
"evaluation_parameters": {},
"runtime_configuration": {},
"validations": [
{
"batch_request": {
"datasource_name": "trino_iceberg",
"data_connector_name": "default_inferred_data_connector_name",
"data_asset_name": "gold.orders"
},
"expectation_suite_name": "gold_orders_suite",
"action_list": []
}
],
"profilers": [],
"ge_cloud_id": null,
"expectation_suite_ge_cloud_id": null
}
deploy/config/airflow/great_expectations/expectations/bronze_orders_suite.json
+42
View File
@@ -0,0 +1,42 @@
{
"expectation_suite_name": "bronze_orders_suite",
"ge_cloud_id": null,
"meta": {
"created_by": "sicabe-bootstrap",
"created_at": "2026-08-31"
},
"expectations": [
{
"expectation_type": "expect_table_row_count_to_be_between",
"kwargs": {
"min_value": 1,
"max_value": 100000
},
"meta": {}
},
{
"expectation_type": "expect_column_values_to_not_be_null",
"kwargs": {
"column": "id"
},
"meta": {}
},
{
"expectation_type": "expect_column_values_to_not_be_null",
"kwargs": {
"column": "komoditas"
},
"meta": {}
},
{
"expectation_type": "expect_column_values_to_be_between",
"kwargs": {
"column": "harga_per_kg",
"min_value": 0,
"max_value": 1000000
},
"meta": {}
}
],
"data_asset_type": null
}
deploy/config/airflow/great_expectations/expectations/gold_orders_suite.json
+44
View File
@@ -0,0 +1,44 @@
{
"expectation_suite_name": "gold_orders_suite",
"ge_cloud_id": null,
"meta": {
"created_by": "sicabe-bootstrap",
"created_at": "2026-08-31"
},
"expectations": [
{
"expectation_type": "expect_table_row_count_to_be_between",
"kwargs": {
"min_value": 1,
"max_value": 100000
},
"meta": {}
},
{
"expectation_type": "expect_column_values_to_not_be_null",
"kwargs": {
"column": "komoditas"
},
"meta": {}
},
{
"expectation_type": "expect_column_values_to_be_between",
"kwargs": {
"column": "harga_per_kg",
"min_value": 0,
"max_value": 1000000
},
"meta": {}
},
{
"expectation_type": "expect_column_values_to_be_between",
"kwargs": {
"column": "stok_kg",
"min_value": 0,
"max_value": 100000000
},
"meta": {}
}
],
"data_asset_type": null
}
deploy/config/airflow/great_expectations/great_expectations.yml
+43
View File
@@ -0,0 +1,43 @@
# =============================================================================
# eBdesk SiCABE — Great Expectations Configuration
# =============================================================================
# Referensi arsitektur: bab 8.1 (Batch Harian), bab 8.2 (CDC Real-Time)
# Validasi kualitas data di layer Gold sebelum dikonsumsi aplikasi.
# =============================================================================
config_version: 4.0
datasources:
trino_iceberg:
class_name: Datasource
execution_engine:
class_name: SqlAlchemyExecutionEngine
connection_string: "trino://{{ env_var('GE_TRINO_USER', 'admin') }}:{{ env_var('GE_TRINO_PASSWORD', '') }}@{{ env_var('GE_TRINO_HOST', 'trino-coordinator') }}:8080/iceberg/gold"
data_connectors:
default_inferred_data_connector_name:
class_name: InferredAssetSqlDataConnector
include_schema_name: true
introspection_directives:
schema_name: gold
expectations_store:
class_name: ExpectationsStore
store_backend:
class_name: TupleFilesystemStoreBackend
base_directory: expectations/
validations_store:
class_name: ValidationsStore
store_backend:
class_name: TupleFilesystemStoreBackend
base_directory: validations/
data_docs_sites:
local:
class_name: SiteBuilder
show_how_to_buttons: true
store_backend:
class_name: TupleFilesystemStoreBackend
base_directory: uncommitted/data_docs/local_site/
site_index_builder:
class_name: DefaultSiteIndexBuilder
deploy/config/airflow/great_expectations/om/config.yml
+18
View File
@@ -0,0 +1,18 @@
# =============================================================================
# Konfigurasi koneksi OpenMetadata untuk GE Action (dibaca oleh ometa_config_handler)
# =============================================================================
# Direktori ini berisi "config.yml" karena action membacanya sebagai TEMPLATE Jinja:
# create_jinja_environment(<dir>) + render_template(env) → <dir>/config.yml
# `env('VAR')` disediakan oleh handler, sehingga token TIDAK ditulis di repo.
#
# Dipakai oleh: metadata/great_expectations/action.py (OpenMetadata >= 2.0)
# OpenMetadataValidationAction(config_file_path="/opt/airflow/great_expectations/om", ...)
# Di OpenMetadata 1.13.x modul action-nya berbeda (generasi 1xx) dan dihapus di
# 2.0; rincian migrasi ada di deploy/output/merge-om20x-rollback.md.
#
# Jalankan dengan OM_JWT_TOKEN tersedia di environment container.
# =============================================================================
hostPort: "{{ env('OM_API_URL') }}"
authProvider: openmetadata
securityConfig:
jwtToken: "{{ env('OM_JWT_TOKEN') }}"
deploy/config/airflow/great_expectations/validate_to_openmetadata.py
+300
View File
@@ -0,0 +1,300 @@
#!/usr/bin/env python
# =============================================================================
# eBdesk SiCABE — Jalankan Great Expectations DAN kirim hasilnya ke OpenMetadata
# =============================================================================
# Jalur A (GE -> OpenMetadata): hasil GE tidak berhenti di log Airflow.
# Action resmi OpenMetadata (metadata.great_expectations.action) dipasang pada
# checkpoint GE 1.x, sehingga setiap kali checkpoint dijalankan, hasilnya dikirim
# ke OpenMetadata dan muncul sebagai Data Quality Test Suite/Test Case dengan
# platform "GreatExpectations".
#
# Catatan versi: OM 1.13.x memakai modul action generasi lama (kelas berakhiran
# "1xx"); di OM 2.0 modul itu dihapus dan kelasnya kini bernama
# OpenMetadataValidationAction di metadata.great_expectations.action.
# Rincian migrasinya: deploy/output/merge-om20x-rollback.md.
#
# ⚠️ Sejak OM 2.0 action MENOLAK batch yang bukan dari database relasional:
# `_get_checkpoint_batch_spec()` melempar ValueError bila batch_spec tidak
# memuat kunci SQL (table_name/schema_name/query/temp_table_schema_name).
# Karena itu ekspektasi dijalankan di Trino lewat DATASOURCE SQL — bukan atas
# pandas dataframe — sehingga batch_spec memuat schema_name + table_name dan
# action dapat menemukan tabel OpenMetadata-nya.
#
# Dijalankan DI DALAM container Airflow (butuh great_expectations + metadata):
# docker exec -e OM_JWT_TOKEN=<token> -e DBT_TRINO_PASSWORD=<pw> \
# sicabe-airflow-webserver python /opt/airflow/great_expectations/validate_to_openmetadata.py
#
# Exit code: 0 = semua ekspektasi lulus; 1 = ADA yang gagal (task Airflow akan
# FAILED) — hasilnya TETAP terkirim ke OpenMetadata sebelum exit.
# =============================================================================
import os
import sys
import great_expectations as gx
import trino
from great_expectations.core.expectation_suite import ExpectationSuite
from metadata.great_expectations.action import OpenMetadataValidationAction
OM_CONFIG_DIR = os.getenv("OM_CONFIG_DIR", "/opt/airflow/great_expectations/om")
# OM_API_URL = alamat OM dari DALAM jaringan container (bukan OPENMETADATA_URL yang
# dipakai dari host). Dipakai juga oleh template config action (om/config.yml).
OM_API_URL = os.getenv("OM_API_URL", "http://openmetadata-server:8585/api")
DATABASE_SERVICE = os.getenv("OM_DATABASE_SERVICE", "trino_sicabe")
DATABASE_NAME = os.getenv("OM_DATABASE_NAME", "iceberg")
TRINO_CERT = os.getenv("TRINO_CA_BUNDLE", "/etc/ssl/trino/trino.crt")
# (schema, tabel, daftar ekspektasi)
CHECKS = [
("bronze", "orders", [
gx.expectations.ExpectTableRowCountToBeBetween(min_value=1, max_value=1000000),
gx.expectations.ExpectColumnValuesToNotBeNull(column="id"),
gx.expectations.ExpectColumnValuesToNotBeNull(column="komoditas"),
gx.expectations.ExpectColumnValuesToBeBetween(
column="harga_per_kg", min_value=0, max_value=1000000),
]),
("silver", "orders_silver", [
gx.expectations.ExpectTableRowCountToBeBetween(min_value=1, max_value=1000000),
gx.expectations.ExpectColumnValuesToNotBeNull(column="id"),
gx.expectations.ExpectColumnValuesToBeBetween(
column="total_value", min_value=0, max_value=1000000000000),
]),
("gold", "orders_gold", [
gx.expectations.ExpectTableRowCountToBeBetween(min_value=1, max_value=1000000),
gx.expectations.ExpectColumnValuesToNotBeNull(column="komoditas"),
gx.expectations.ExpectColumnValuesToBeBetween(
column="avg_harga_per_kg", min_value=0, max_value=1000000),
gx.expectations.ExpectColumnValuesToBeBetween(
column="total_nilai", min_value=0, max_value=1000000000000),
]),
]
def _trino_connection():
user = os.getenv("DBT_TRINO_USER", "svc_dbt")
password = os.getenv("DBT_TRINO_PASSWORD")
if not password:
raise RuntimeError("DBT_TRINO_PASSWORD tidak diset")
return trino.dbapi.connect(
host=os.getenv("DBT_TRINO_HOST", "trino-coordinator"),
port=int(os.getenv("DBT_TRINO_PORT", "8443")),
http_scheme="https",
user=user,
catalog=DATABASE_NAME,
auth=trino.auth.BasicAuthentication(user, password),
verify=TRINO_CERT,
)
def _trino_sqlalchemy_url():
"""URL SQLAlchemy untuk datasource SQL Great Expectations (tanpa password).
Password TIDAK ditulis di URL — dikirim lewat objek auth pada connect_args,
sehingga tidak muncul di connection string, log, atau pesan error.
"""
user = os.getenv("DBT_TRINO_USER", "svc_dbt")
host = os.getenv("DBT_TRINO_HOST", "trino-coordinator")
port = os.getenv("DBT_TRINO_PORT", "8443")
return f"trino://{user}@{host}:{port}/{DATABASE_NAME}"
def _trino_connect_args():
"""Argumen koneksi untuk dialect SQLAlchemy Trino.
Penting: dialect Trino TIDAK membaca `http_scheme`/`verify` dari query URL —
keduanya harus diberikan sebagai connect_args. `verify` juga harus berupa
path sertifikat (bukan JSON string seperti bila lewat query URL).
"""
user = os.getenv("DBT_TRINO_USER", "svc_dbt")
password = os.getenv("DBT_TRINO_PASSWORD")
if not password:
raise RuntimeError("DBT_TRINO_PASSWORD tidak diset")
return {
"http_scheme": "https",
"verify": TRINO_CERT,
"auth": trino.auth.BasicAuthentication(user, password),
}
def _assert_table_not_empty(schema, table):
"""Fail-fast bila tabel kosong: tidak ada yang bermakna untuk divalidasi."""
row = None
with _trino_connection() as conn:
cursor = conn.cursor()
cursor.execute(f"SELECT count(*) FROM {schema}.{table}")
row = cursor.fetchone()
if not row or not row[0]:
raise RuntimeError(f"Tabel {schema}.{table} kosong — tidak ada yang divalidasi")
def _build_checkpoint(context, schema, table, expectations):
"""Satu checkpoint per tabel: suite + batch definition SQL + action OM."""
_assert_table_not_empty(schema, table)
suite_name = f"{schema}_{table}_suite"
data_source = context.data_sources.add_or_update_sql(
name=f"trino_{schema}_{table}",
connection_string=_trino_sqlalchemy_url(),
kwargs={"connect_args": _trino_connect_args()},
)
asset = data_source.add_table_asset(name=table, table_name=table, schema_name=schema)
batch_definition = asset.add_batch_definition_whole_table(f"bd_{schema}_{table}")
suite = ExpectationSuite(name=suite_name)
for expectation in expectations:
suite.add_expectation(expectation)
context.suites.add_or_update(suite)
# ValidationDefinition memerlukan BatchDefinition (bukan Batch) + suite.
validation_definition = context.validation_definitions.add(
gx.ValidationDefinition(name=f"vd_{schema}_{table}", data=batch_definition, suite=suite)
)
action = OpenMetadataValidationAction(
config_file_path=OM_CONFIG_DIR,
database_service_name=DATABASE_SERVICE,
database_name=DATABASE_NAME,
schema_name=schema,
table_name=table,
)
checkpoint = context.checkpoints.add(
gx.Checkpoint(
name=f"cp_{schema}_{table}",
validation_definitions=[validation_definition],
actions=[action],
)
)
# Batch "whole table" dijalankan di Trino (tanpa batch_parameters).
return checkpoint
def _ensure_om_token():
"""Token OpenMetadata untuk action: pakai OM_JWT_TOKEN, atau login sendiri.
Login sendiri membuat skrip aman dipakai pada jadwal (DAG) — tidak bergantung
pada token sementara yang ditempel di environment.
"""
if os.getenv("OM_JWT_TOKEN"):
return True
# Template config action membaca OM_API_URL → pastikan selalu terisi.
os.environ.setdefault("OM_API_URL", OM_API_URL)
email = os.getenv("OPENMETADATA_ADMIN_EMAIL", "admin@open-metadata.org")
password = os.getenv("OPENMETADATA_ADMIN_PASSWORD")
import base64
import json
import urllib.request
# DUA kandidat, MAKSIMAL DUA percobaan (jangan diulang dalam loop — akun admin bisa
# terkunci): password .env, lalu default 'admin' yang dipakai OM 2.0.x sampai
# bootstrap step 7 menulis hash bcrypt. Sama seperti ingest-openmetadata.sh.
candidates = [c for c in dict.fromkeys([password or "", "admin"]) if c]
err_last = ""
for pw in candidates:
body = json.dumps({"email": email,
"password": base64.b64encode(pw.encode()).decode()}).encode()
req = urllib.request.Request(f"{OM_API_URL}/v1/users/login", data=body,
headers={"Content-Type": "application/json"})
try:
token = json.load(urllib.request.urlopen(req, timeout=20)).get("accessToken", "")
except Exception as exc: # noqa: BLE001
err_last = str(exc)
token = ""
if token:
os.environ["OM_JWT_TOKEN"] = token
return True
print(f"ERROR: login ke OpenMetadata gagal: {err_last or 'tanpa token'}")
print(" dicoba: password .env + 'admin' — pastikan: ./bootstrap.sh --step 7")
return False
def _suite_exists_in_om(schema: str, table: str):
"""Apakah test suite untuk tabel ini ADA di OpenMetadata?
Mengembalikan nama suite bila ada, False bila tidak, None bila tidak dapat
diverifikasi.
Dua hal penting yang membuat versi pertama fungsi ini SALAH:
1. `checkpoint.run()` hanya melaporkan status CHECKPOINT (validasi expectation),
bukan status pengiriman action — jadi tidak boleh disimpulkan dari situ.
2. Action OpenMetadataValidationAction menamai test suite menurut FQN TABEL:
<service>.<database>.<schema>.<table>.testSuite
BUKAN menurut nama ExpectationSuite GE (`<schema>_<table>_suite`). Memeriksa
nama GE-nya menghasilkan false negative: DQ sudah ada di OM tapi dilaporkan
tidak ada.
"""
token = os.environ.get("OM_JWT_TOKEN", "")
if not token:
return None
import json
import urllib.request
req = urllib.request.Request(
f"{OM_API_URL}/v1/dataQuality/testSuites?limit=200",
headers={"Authorization": f"Bearer {token}"})
try:
d = json.load(urllib.request.urlopen(req, timeout=20))
except Exception as exc: # noqa: BLE001
print(f"[GE] verifikasi ke OM gagal: {exc}")
return None
names = {s.get("name") for s in (d.get("data") or [])}
table_fqn = f"{DATABASE_SERVICE}.{DATABASE_NAME}.{schema}.{table}"
for expected in (f"{table_fqn}.testSuite", table_fqn):
if expected in names:
return expected
# toleran terhadap variasi penamaan: suite apa pun yang berawalan FQN tabel ini
for n in names:
if n and n.startswith(table_fqn):
return n
return False
def main():
only = None
if len(sys.argv) > 2 and sys.argv[1] == "--table":
only = sys.argv[2] # format: schema.table
if not _ensure_om_token():
print("ERROR: token OpenMetadata tidak tersedia (set OM_JWT_TOKEN, atau "
"OPENMETADATA_ADMIN_PASSWORD untuk login otomatis)")
return 1
checks = [c for c in CHECKS if only is None or f"{c[0]}.{c[1]}" == only]
if not checks:
print(f"ERROR: tidak ada pemeriksaan untuk '{only}'")
return 1
context = gx.get_context(mode="ephemeral")
checkpoints = [_build_checkpoint(context, s, t, e) for s, t, e in checks]
failed = 0
not_in_om = 0
for (schema, table, _expectations), checkpoint in zip(checks, checkpoints):
result = checkpoint.run()
success = bool(getattr(result, "success", False))
in_om = _suite_exists_in_om(schema, table)
if in_om is None:
print(f"[GE] {checkpoint.name}: checkpoint success={success}; "
f"keberadaan di OpenMetadata TIDAK dapat diverifikasi")
elif in_om:
print(f"[GE] {checkpoint.name}: checkpoint success={success} "
f"→ test suite '{in_om}' ADA di OpenMetadata")
else:
print(f"[GE] {checkpoint.name}: checkpoint success={success} "
f"TAPI belum ada test suite untuk {DATABASE_SERVICE}.{DATABASE_NAME}.{schema}.{table}")
print("[GE] penyebab tersering: entitas TABEL-nya belum ada di katalog OM.")
print("[GE] urutan yang benar: ./scripts/ingest-openmetadata.sh --all DULU,")
print("[GE] baru DAG -- karena action GE menempel ke entitas tabel di OM.")
not_in_om += 1
if not success:
failed += 1
total = len(checkpoints)
print(f"[GE] ringkasan: {total - failed}/{total} checkpoint lulus"
+ (f"; {not_in_om} suite TIDAK ADA di OpenMetadata" if not_in_om else ""))
# PENTING: `success` hanyalah status CHECKPOINT, bukan status pengiriman. Pesan lama
# ("success=True → hasil dikirim ke OpenMetadata") menyesatkan persis karena itu.
# Suite yang tidak ada di OM juga membuat task gagal, supaya penyebabnya terlihat
# di sini alih-alih muncul sebagai kegagalan gate beberapa langkah kemudian.
return 1 if (failed or not_in_om) else 0
if __name__ == "__main__":
sys.exit(main())
deploy/config/dremio/README.md
+83
View File
@@ -0,0 +1,83 @@
# Dremio OSS — Konfigurasi `deploy/`
Dremio ditambahkan sebagai **query engine paralel** di atas catalog Nessie yang
sama dengan Trino & StarRocks (membuktikan dua engine berbagi satu catalog tanpa
korupsi metadata).
| Item | Nilai |
|---|---|
| Image | `dremio/dremio-oss:${DREMIO_VERSION:-25.2.0}` |
| Web UI | `http://localhost:${DREMIO_UI_PORT:-12947}` |
| JDBC | `${DREMIO_JDBC_PORT:-12010}` → container `31010` |
| Arrow Flight | `${DREMIO_FLIGHT_PORT:-12011}` → container `32010` |
| Auth | **Lokal** (bukan OIDC — Dremio OSS tidak menyediakan OIDC) |
| Penyimpanan internal | Volume `dremio-data` (bukan PostgreSQL) |
## First-run & Source Nessie — otomatis via `bootstrap.sh` (Step 9)
`bootstrap.sh` **Step 9** menangani first-run admin + registrasi source Nessie
secara idempoten (aman diulang):
1. Tunggu Dremio siap.
2. Buat admin first-run bila belum ada — `PUT /apiv2/bootstrap/firstuser`
(body flat, tanpa auth; `400` = sudah pernah dibuat → dilewati).
3. Panggil `scripts/dremio/register-nessie-source.sh` untuk source Nessie.
Kredensial default (override lewat `.env`):
| Variabel | Default |
|---|---|
| `DREMIO_USER` | `admin` |
| `DREMIO_PASSWORD` | `dremio_admin_DevOnly1` |
| `DREMIO_ADMIN_EMAIL` | `admin@sicabe.local` |
Jalankan hanya step ini:
```bash
cd deploy
./bootstrap.sh --step 9
```
### Manual (alternatif)
1. Buka `http://localhost:12947` → buat akun admin pertama (form first-run).
2. Daftarkan source Nessie lewat script:
```bash
cd deploy
set -a; source config/environment.defaults.sh; source .env; set +a
DREMIO_USER=<admin> DREMIO_PASSWORD='<password>' \
./scripts/dremio/register-nessie-source.sh
```
Script: login (`/apiv2/login`, header `Authorization: _dremio<token>`) → cek
existing (`/api/v3/catalog/by-path/<name>`) → `POST /api/v3/catalog`
(`type=NESSIE`). Idempotent (source sudah ada → sukses tanpa buat ulang).
> **Temuan (diuji 2026-09-10, Dremio OSS 25.2.0):** klaim docs bahwa Catalog API
> "unable to retrieve or manage Nessie sources" (DX-64994) **tidak akurat** —
> create **dan** GET by-path berhasil. Kunci config yang benar:
> - **JANGAN** sertakan `storageType` (ditolak: `Invalid value found at: storageType`).
> - `awsRootPath` format **`/bucket/path`** (TANPA prefix `s3://`), mis. `/iceberg-data/warehouse`.
> - `credentialType=ACCESS_KEY`, `awsAccessKey`/`awsAccessSecret`, `secure=false`.
> - `propertyList`: `fs.s3a.path.style.access=true`, `fs.s3a.endpoint=minio:9000`,
> `dremio.s3.compat=true`.
Verifikasi query (data yang sama ditulis Trino):
```sql
SELECT * FROM nessie.demo.t ORDER BY id; -- 2 baris: alpha, beta
```
### Alternatif: via UI
**Add Source → Nessie** dengan field: endpoint `http://nessie:19120/api/v2`,
Authentication `None`, Storage AWS S3/MinIO (path-style, endpoint `minio:9000`),
access key/secret MinIO.
## Catatan
- **Dua engine berbagi satu Nessie catalog**: tulis dari Dremio (CTAS/INSERT)
langsung terlihat di Trino dan sebaliknya. Lakukan tulis secara bergantian
(bukan simultan) untuk menghindari konflik concurrency Nessie.
- Nessie **tidak mendukung Iceberg spec v3** → gunakan `format-version=2`.
- Referensi: `experiments/nessie-dremio/README.md` (H2 terbukti).
deploy/config/environment.defaults.sh
+388
View File
@@ -0,0 +1,388 @@
#!/usr/bin/env bash
# =============================================================================
# eBdesk SiCABE — Environment Defaults
# Sourced by install.sh and bootstrap.sh
# Override any value by exporting the variable before running install.sh
# or by creating deploy/.env with: export VAR=value
# =============================================================================
# ─── Global ─────────────────────────────────────────────────────────────────
export SICABE_ENV="${SICABE_ENV:-development}"
export COMPOSE_PROJECT_NAME="${COMPOSE_PROJECT_NAME:-sicabe}"
export SICABE_DOMAIN="${SICABE_DOMAIN:-localhost}"
export TIMEZONE="${TIMEZONE:-Asia/Jakarta}"
# ─── Secrets (ALWAYS override in production!) ───────────────────────────────
# Password statis agar konsisten saat docker-compose memproses env vars.
# Di production, overriede semua nilai berikut di file .env dengan password kuat.
# openssl rand -hex 16 > .env
# PostgreSQL: 1 instance shared untuk semua service (beda database)
export POSTGRES_PASSWORD="${POSTGRES_PASSWORD:-sicabe_pg_super_DevOnly}"
export KEYCLOAK_DB_PASSWORD="${KEYCLOAK_DB_PASSWORD:-sicabe_kc_db_DevOnly}"
export NESSIE_DB_PASSWORD="${NESSIE_DB_PASSWORD:-sicabe_nessie_db_DevOnly}"
export AIRFLOW_DB_PASSWORD="${AIRFLOW_DB_PASSWORD:-sicabe_af_db_DevOnly}"
export OPENMETADATA_DB_PASSWORD="${OPENMETADATA_DB_PASSWORD:-sicabe_om_db_DevOnly}"
export KEYCLOAK_ADMIN_PASSWORD="${KEYCLOAK_ADMIN_PASSWORD:-admin123}"
export MINIO_ROOT_PASSWORD="${MINIO_ROOT_PASSWORD:-minio12345}"
export AIRFLOW_ADMIN_PASSWORD="${AIRFLOW_ADMIN_PASSWORD:-airflow123}"
export GRAFANA_ADMIN_PASSWORD="${GRAFANA_ADMIN_PASSWORD:-grafana123}"
export OPENMETADATA_ADMIN_PASSWORD="${OPENMETADATA_ADMIN_PASSWORD:-openmetadata123}"
# OpenLDAP (identity — sumber grup untuk Trino & StarRocks)
export LDAP_ADMIN_PASSWORD="${LDAP_ADMIN_PASSWORD:-ldap_admin_DevOnly}"
export LDAP_CONFIG_PASSWORD="${LDAP_CONFIG_PASSWORD:-ldap_config_DevOnly}"
# OIDC Client secrets untuk setiap komponen (wajib sinkron dengan konfigurasi
# Trino, Grafana, dll. yang menggunakan nilai hardcoded di file .properties)
export TRINO_CLIENT_SECRET="${TRINO_CLIENT_SECRET:-trino-secret}"
export STARROCKS_CLIENT_SECRET="${STARROCKS_CLIENT_SECRET:-starrocks-secret}"
export AIRFLOW_CLIENT_SECRET="${AIRFLOW_CLIENT_SECRET:-airflow-secret}"
export OPENMETADATA_CLIENT_SECRET="${OPENMETADATA_CLIENT_SECRET:-om-secret}"
export GRAFANA_CLIENT_SECRET="${GRAFANA_CLIENT_SECRET:-grafana-secret}"
# ─── Network ────────────────────────────────────────────────────────────────
export HTTP_PORT="${HTTP_PORT:-80}"
export HTTPS_PORT="${HTTPS_PORT:-443}"
# Catatan: PostgreSQL TIDAK diexpose ke host (internal network saja).
# Untuk debugging dari host: docker exec -it sicabe-postgres psql -U sicabe_admin -d keycloak
# ─── Skema Port Host (prefix 12xxx) ─────────────────────────────────────────
# Aturan: host port = "12" + 3 digit port kanonik. Port internal container tetap.
# Memudahkan pelatihan: seluruh endpoint ada di blok 12000-12999.
# ─── Keycloak (Phase 1 — Governance: IAM) ──────────────────────────────────
export KEYCLOAK_VERSION="${KEYCLOAK_VERSION:-26.7}"
export KEYCLOAK_PORT="${KEYCLOAK_PORT:-12080}"
export KEYCLOAK_DB="${KEYCLOAK_DB:-keycloak}"
export KEYCLOAK_DB_USER="${KEYCLOAK_DB_USER:-keycloak}"
export KEYCLOAK_DB_HOST="${KEYCLOAK_DB_HOST:-postgres}"
# CATATAN: KC_HOSTNAME_URL sudah DIHAPUS di Keycloak 26.0.0 — jangan dipasang lagi
# (dulu diekspor di sini, dan tidak berpengaruh apa pun). Yang menentukan issuer,
# discovery, dan redirect sekarang: KC_HOSTNAME = URL publik lengkap, yang di compose
# diisi dari KEYCLOAK_PUBLIC_URL (lihat docker-compose.yml service keycloak).
export SICABE_REALM="${SICABE_REALM:-sicabe}"
export KEYCLOAK_ADMIN="${KEYCLOAK_ADMIN:-admin}"
# ─── Identity — OpenLDAP (Phase 1b) ─────────────────────────────────────────
# LDAP = sumber kebenaran GRUP (Trino & StarRocks group provider).
# User tetap didefinisikan di realm Keycloak (autentikasi OIDC).
export OPENLDAP_VERSION="${OPENLDAP_VERSION:-1.5.0}"
export OPENLDAP_PORT="${OPENLDAP_PORT:-12389}"
export LDAP_ORGANISATION="${LDAP_ORGANISATION:-SiCABE}"
export LDAP_DOMAIN="${LDAP_DOMAIN:-sicabe.local}"
export LDAP_ADMIN_DN="${LDAP_ADMIN_DN:-cn=admin,dc=sicabe,dc=local}"
export LDAP_BASE_DN="${LDAP_BASE_DN:-dc=sicabe,dc=local}"
export LDAP_USERS_DN="${LDAP_USERS_DN:-ou=users,dc=sicabe,dc=local}"
export LDAP_GROUPS_DN="${LDAP_GROUPS_DN:-ou=groups,dc=sicabe,dc=local}"
# ─── MinIO (Phase 2 — Storage) ──────────────────────────────────────────────
# Image MinIO HANYA tersedia di quay.io — repo Docker Hub `minio/minio` dan
# `minio/mc` sudah tidak ada (pull gagal: "repository does not exist").
export MINIO_VERSION="${MINIO_VERSION:-RELEASE.2025-09-07T16-13-09Z}"
export MINIO_MC_VERSION="${MINIO_MC_VERSION:-RELEASE.2025-08-13T08-35-41Z}"
export MINIO_PORT="${MINIO_PORT:-12900}"
export MINIO_CONSOLE_PORT="${MINIO_CONSOLE_PORT:-12901}"
export MINIO_ROOT_USER="${MINIO_ROOT_USER:-minioadmin}"
export MINIO_BUCKET_ICEBERG_DATA="${MINIO_BUCKET_ICEBERG_DATA:-iceberg-data}"
# Apache Nessie (Phase 2 — Storage: Metadata Catalog)
export NESSIE_VERSION="${NESSIE_VERSION:-0.108.4-java}"
export NESSIE_PORT="${NESSIE_PORT:-12120}"
# Nessie persistence database (PostgreSQL) — mengikuti konvensi shared Postgres
export NESSIE_DB="${NESSIE_DB:-nessie}"
export NESSIE_DB_USER="${NESSIE_DB_USER:-nessie}"
# ─── Alamat antar-container (multi-host) ────────────────────────────────────
# B1 (Review 10 §4.4): hostname yang dilihat CONTAINER saat menghubungi komponen
# lain. Kontrak: DEFAULT = nama container di docker-compose.yml → SINGLE-HOST
# TIDAK BERUBAH (dibuktikan diff `docker compose config` sha256 identik).
# Untuk menyebar komponen ke beberapa host, override ke alamat host pemilik:
# POSTGRES_HOST=10.20.0.12 MINIO_HOST=10.20.0.12 KEYCLOAK_HOST=10.20.0.10
# OPENMETADATA_HOST=10.20.0.10 STARROCKS_FE_HOST=10.20.0.11
# Dibedakan dari komponen EXISTING milik sistem lain (lihat bagian di bawah):
# yang itu memakai *_URL / S3_ENDPOINT + EXTERNAL_SERVICES.
export POSTGRES_HOST="${POSTGRES_HOST:-postgres}"
export MINIO_HOST="${MINIO_HOST:-minio}"
export KEYCLOAK_HOST="${KEYCLOAK_HOST:-keycloak}"
export OPENMETADATA_HOST="${OPENMETADATA_HOST:-openmetadata-server}"
export AIRFLOW_WEBSERVER_HOST="${AIRFLOW_WEBSERVER_HOST:-airflow-webserver}"
export STARROCKS_FE_HOST="${STARROCKS_FE_HOST:-starrocks-fe}"
export STARROCKS_CN_HOST="${STARROCKS_CN_HOST:-starrocks-cn}"
export OPENSEARCH_HOST="${OPENSEARCH_HOST:-openmetadata-opensearch}"
# ─── URL BROWSER-FACING (tls.mode != none) ──────────────────────────────────
# Dipakai HANYA untuk apa yang dilihat BROWSER (issuer OIDC, redirect, link UI) —
# BUKAN untuk lalu lintas container (itu memakai `*_HOST`/alamat di atas).
# Default = nilai sekarang, jadi tanpa perubahan apa pun perilakunya identik.
# Saat `tls.mode` = edge/full, generator menimpanya menjadi https://<label>.<domain>.
# Perhatian: `iss` token memuat SKEMA, jadi berpindah none -> edge mengubah
# issuer dan semua user harus login ulang (lihat R10 §5.7).
export KEYCLOAK_PUBLIC_URL="${KEYCLOAK_PUBLIC_URL:-http://localhost:${KEYCLOAK_PORT:-12080}}"
# Kebijakan SSL realm Keycloak. Default Keycloak `external` menuntut HTTPS untuk
# permintaan dari alamat NON-lokal — dan alamat Tailscale 100.64.0.0/10 (CGNAT)
# TERMASUK non-lokal, sehingga admin console via http://<tailscale>:12080 gagal
# dengan "HTTPS required". Keycloak 26 tidak punya opsi server-level untuk ini
# (tidak ada KC_HTTPS_REQUIRED), jadi harus per realm. Diisi `external` oleh
# generator saat tls.mode != none (ada Traefik/TLS di depan).
# Default `external` = aman: admin console via HTTP hanya diizinkan dari alamat lokal.
# Generator menulis `none` HANYA bila tls.mode != none... (mode `none` pada multi-host,
# yaitu HTTP polos ala dev) — di situ akses dari alamat Tailscale/CGNAT memang perlu
# diizinkan. Single-host (tanpa cluster.yml) tidak melewati generator, jadi ia memakai
# default ini dan tetap aman.
export KEYCLOAK_SSL_REQUIRED="${KEYCLOAK_SSL_REQUIRED:-external}"
# Cache Keycloak: `ispn` (default, registry di DB) atau `local` untuk instalasi
# SATU Keycloak — mematikan klaster + pemeriksaan cluster health.
export KEYCLOAK_CACHE="${KEYCLOAK_CACHE:-ispn}"
# ── Ingress / TLS (langkah 2, T-20) ──────────────────────────────────────────
# Traefik berjalan hanya pada profile `edge` (docker compose --profile edge).
# Tiap host menerbitkan sertifikatnya SENDIRI: acme.json tidak boleh dibagi antar
# host (Traefik tidak punya penyimpanan ACME terdistribusi), dan batas Let's
# Encrypt 5 sertifikat identik/minggu sebaiknya dihindari dengan staging dulu.
export CLUSTER_DOMAIN="${CLUSTER_DOMAIN:-sicabe.com}"
export TRAEFIK_VERSION="${TRAEFIK_VERSION:-v3.7}"
export TRAEFIK_HTTP_PORT="${TRAEFIK_HTTP_PORT:-80}"
export TRAEFIK_HTTPS_PORT="${TRAEFIK_HTTPS_PORT:-443}"
# Alamat bind port 80/443. Di multi-host diisi generator dengan alamat host,
# supaya ingress tidak terbuka di semua antarmuka.
export TRAEFIK_BIND_ADDR="${TRAEFIK_BIND_ADDR:-0.0.0.0}"
export TRAEFIK_ACME_EMAIL="${TRAEFIK_ACME_EMAIL:-admin@${CLUSTER_DOMAIN:-sicabe.com}}"
export TRAEFIK_DNS_PROVIDER="${TRAEFIK_DNS_PROVIDER:-cloudflare}"
# STAGING dulu (batas longgar). Produksi:
# https://acme-v02.api.letsencrypt.org/directory
# Jeda sebelum LE memeriksa TXT. Apex + wildcard memakai nama TXT yang SAMA
# (_acme-challenge.<domain>), jadi propagasi perlu waktu — default Traefik 0s.
export TRAEFIK_ACME_PROPAGATION_DELAY="${TRAEFIK_ACME_PROPAGATION_DELAY:-30s}"
# Catatan lego/Cloudflare (opsional, diset sebagai env container bila perlu):
# CLOUDFLARE_PROPAGATION_TIMEOUT (default 120), CLOUDFLARE_POLLING_INTERVAL (2), CLOUDFLARE_TTL (120)
export TRAEFIK_ACME_CASERVER="${TRAEFIK_ACME_CASERVER:-https://acme-staging-v02.api.letsencrypt.org/directory}"
# Cloudflare DNS-01: token API dengan izin Zone:DNS:Edit. WAJIB diisi sebelum
# sertifikat bisa diterbitkan; kosong = Traefik tetap start dan memakai
# sertifikat self-signed bawaan (routing masih bisa diuji).
export CF_DNS_API_TOKEN="${CF_DNS_API_TOKEN:-}"
export GRAFANA_PUBLIC_URL="${GRAFANA_PUBLIC_URL:-http://localhost:${GRAFANA_PORT:-12300}}"
# URL browser-facing: default localhost (nama internal airflow-webserver:8080 tidak
# bisa diresolusi browser). Bundle multi-host mengisinya dengan alamat host pemilik.
export AIRFLOW_PUBLIC_URL="${AIRFLOW_PUBLIC_URL:-http://localhost:${AIRFLOW_PORT:-12082}}"
export OPENMETADATA_PUBLIC_URL="${OPENMETADATA_PUBLIC_URL:-http://localhost:${OPENMETADATA_PORT:-12085}}"
export TRINO_PUBLIC_URL="${TRINO_PUBLIC_URL:-https://localhost:${TRINO_HTTPS_PORT:-12443}}"
export MINIO_CONSOLE_PUBLIC_URL="${MINIO_CONSOLE_PUBLIC_URL:-http://localhost:${MINIO_CONSOLE_PORT:-12901}}"
# ─── Endpoint Object Storage & Nessie ───────────────────────────────────────
# Default = container lokal hasil deploy ini (perilaku tidak berubah). Untuk
# memakai penyimpanan objek (S3/MinIO) atau Nessie EXISTING, override di .env:
# S3_ENDPOINT → dilihat CONTAINER (Trino, Nessie, StarRocks)
# NESSIE_URL → dilihat CONTAINER (Trino, Sling, StarRocks)
# NESSIE_HOST/PORT → dilihat HOST (skrip verifikasi)
# Contoh:
# S3_ENDPOINT=https://s3.corp.internal
# S3_ACCESS_KEY=... S3_SECRET_KEY=... S3_REGION=ap-southeast-1
# NESSIE_URL=https://nessie.corp.internal
export S3_ENDPOINT="${S3_ENDPOINT:-http://minio:9000}"
export S3_REGION="${S3_REGION:-us-east-1}"
# CATATAN: S3_ACCESS_KEY / S3_SECRET_KEY sengaja TIDAK di-default di sini.
# Bila di-default dari MINIO_ROOT_*, nilainya terkunci pada nilai default sebelum
# .env di-source (mis. minio12345) sehingga berbeda dari kredensial sebenarnya.
# Semua pemakaian memakai fallback saat-pakai: ${S3_ACCESS_KEY:-${MINIO_ROOT_USER:-...}}.
export NESSIE_URL="${NESSIE_URL:-http://nessie:19120}"
export NESSIE_HOST="${NESSIE_HOST:-localhost}"
export NESSIE_PORT="${NESSIE_PORT:-12120}"
# ─── Trino (Phase 3 — Processing) ───────────────────────────────────────────
export TRINO_VERSION="${TRINO_VERSION:-483}"
# TRINO_PORT = port HTTP polos historis. Trino TIDAK melayani HTTP polos
# (allow-insecure-over-http=false) dan port ini tidak di-expose. Dipakai gate Fase 3
# (verify-phase3-processing.sh) sebagai target uji-negatif "HTTP polos harus mati".
export TRINO_PORT="${TRINO_PORT:-12081}"
export TRINO_HTTPS_PORT="${TRINO_HTTPS_PORT:-12443}"
# Endpoint dari sisi HOST (gate verifikasi + bootstrap step 4). Untuk Trino
# EXISTING: TRINO_URL=https://trino.corp:8443 dan TRINO_TLS_VERIFY=1 (verifikasi TLS aktif;
# default 0 = lewati verifikasi, hanya untuk sertifikat self-signed dev).
export TRINO_URL="${TRINO_URL:-https://localhost:${TRINO_HTTPS_PORT:-12443}}"
export TRINO_TLS_VERIFY="${TRINO_TLS_VERIFY:-0}"
# Service account (HTTP Basic) untuk service/pipeline. Password di-hash (bcrypt)
# ke config/trino/password.db oleh bootstrap.sh/install.sh — pra-syarat: htpasswd.
export TRINO_SVC_SEED_PASSWORD="${TRINO_SVC_SEED_PASSWORD:-svc_seed_dev}"
export TRINO_SVC_DBT_PASSWORD="${TRINO_SVC_DBT_PASSWORD:-svc_dbt_dev}"
export TRINO_SVC_AIRFLOW_PASSWORD="${TRINO_SVC_AIRFLOW_PASSWORD:-svc_airflow_dev}"
export TRINO_SVC_OM_PASSWORD="${TRINO_SVC_OM_PASSWORD:-svc_om_dev}"
# ─── StarRocks (Phase 4 — Serving) ──────────────────────────────────────────
export STARROCKS_VERSION="${STARROCKS_VERSION:-4.0.13}"
export STARROCKS_FE_PORT="${STARROCKS_FE_PORT:-12930}"
export STARROCKS_HTTP_PORT="${STARROCKS_HTTP_PORT:-12030}"
# Endpoint dari sisi HOST (gate verifikasi + bootstrap step 5). Untuk StarRocks
# EXISTING: STARROCKS_HOST=starrocks.corp (dan STARROCKS_PORT=9030, port MySQL
# StarRocks — bukan port publish compose 12930).
export STARROCKS_HOST="${STARROCKS_HOST:-localhost}"
export STARROCKS_PORT="${STARROCKS_PORT:-${STARROCKS_FE_PORT:-12930}}"
# ─── Airflow (Phase 5 — Ingestion) ──────────────────────────────────────────
export AIRFLOW_VERSION="${AIRFLOW_VERSION:-3.3.2}"
export AIRFLOW_PORT="${AIRFLOW_PORT:-12082}"
# Endpoint dari sisi HOST (gate Fase 5, test-verification). Untuk Airflow
# EXISTING: AIRFLOW_HOST=airflow.corp (+ AIRFLOW_PORT sesuai) → install.sh tidak
# menyalakan service Airflow, dan uji container menjadi SKIP.
export AIRFLOW_HOST="${AIRFLOW_HOST:-localhost}"
# Secret key untuk JWT internal Airflow (task runner auth) — WAJIB sama di semua container
export AIRFLOW_SECRET_KEY="${AIRFLOW_SECRET_KEY:-sicabe_airflow_jwt_DevOnly_0123456789abcdef}"
export AIRFLOW_DB="${AIRFLOW_DB:-airflow}"
export AIRFLOW_DB_USER="${AIRFLOW_DB_USER:-airflow}"
export AIRFLOW_DB_HOST="${AIRFLOW_DB_HOST:-postgres}"
export AIRFLOW_EXECUTOR="${AIRFLOW_EXECUTOR:-LocalExecutor}"
# Image Airflow kustom. Sejak OM 2.0.2 image ini memuat extra
# `openmetadata-ingestion[airflow]` + plugin `openmetadata-managed-apis` yang
# disajikan di /pluginsv2/ — inilah yang menghidupkan tombol "Test Connection"
# di UI OpenMetadata (§8.11). Tag lama `3.3.1-slim` (tanpa plugin) tetap ada
# sebagai jalur rollback: set AIRFLOW_IMAGE=sicabe/airflow:3.3.1-slim.
# BATCH B (review R18, 2026-10-01): naik ke 3.3.2 (OM 2.0.3 mem-pin Airflow 3.3.2).
# Jalur rollback: sicabe/airflow:3.3.1-om202 (atau 3.3.1-om202-presqlalchemy).
# ⚠️ Image 3.3.2 di Python 3.13 DITOLAK pada uji go/no-go B3 (dbt mati) — yang
# dipakai adalah varian Python 3.11. Lihat deploy/output/b3-hasil-uji.md.
# 2026-10-02: tag `-ol` = 3.3.2 + provider **OpenLineage** (permintaan pemilik
# proyek). Provider itu INERT tanpa backend OpenLineage (belum ada di stack) —
# jadi ini mengaktifkan kemampuan, belum menghasilkan lineage.
# Rollback: sicabe/airflow:3.3.2-om203 (tanpa OpenLineage).
export AIRFLOW_IMAGE="${AIRFLOW_IMAGE:-sicabe/airflow:3.3.2-om203-ol}"
# ─── Image refs yang bisa diarahkan ke MIRROR/registry internal ──────────────
# Repositori asal bisa berubah kebijakan: mis. image community MinIO kini menuntut
# autentikasi → `pull` anonim GAGAL ("unauthorized"). Arahkan ke registry/mirror
# internal via var FULL-IMAGE di bawah, TANPA menyunting docker-compose.yml.
# Bila var ini tidak diset, default dibangun dari *_VERSION di atas (perilaku lama).
export MINIO_IMAGE="${MINIO_IMAGE:-quay.io/minio/minio:${MINIO_VERSION}}"
export MINIO_MC_IMAGE="${MINIO_MC_IMAGE:-quay.io/minio/mc:${MINIO_MC_VERSION}}"
export NESSIE_IMAGE="${NESSIE_IMAGE:-ghcr.io/projectnessie/nessie:${NESSIE_VERSION}}"
# ─── OpenLineage → OpenMetadata (eksperimen experiments/lineage-airflow-om) ──
# Provider OpenLineage sudah terpasang di image di atas, tetapi INERT sampai
# diarahkan ke OpenMetadata. Konfigurasi transport dibaca dari KONEKSI Airflow
# `openlineage_config` (bukan dari env) supaya token lineage TIDAK perlu
# ditulis di deploy/.env — lihat blok koneksi di bootstrap.sh.
#
# AKTIF sejak 2026-10-02 (adopsi opsi C — keputusan pemilik proyek).
# ⚠️ PRASYARAT: koneksi `openlineage_config` + `openmetadata_lineage_api` harus
# SUDAH ADA. Pemuatan konfigurasi provider TIDAK dibungkus try/except, jadi
# bila CONFIG_CONN_ID diisi sedangkan koneksinya belum ada, task dapat GAGAL.
# Koneksi dibuat oleh `./bootstrap.sh` (langkah "Setup connections").
# Mematikan kembali : AIRFLOW_OPENLINEAGE_CONFIG_CONN_ID= (kosongkan).
export AIRFLOW_OPENLINEAGE_CONFIG_CONN_ID="${AIRFLOW_OPENLINEAGE_CONFIG_CONN_ID:-openlineage_config}"
# Namespace job OpenLineage — label asal job; tidak memengaruhi pemetaan tabel
# (pemetaan tabel berjalan otomatis lewat nama dataset + service OM).
export AIRFLOW_OPENLINEAGE_NAMESPACE="${AIRFLOW_OPENLINEAGE_NAMESPACE:-airflow_sicabe}"
export AIRFLOW__CORE__PARALLELISM="${AIRFLOW__CORE__PARALLELISM:-4}"
# Catatan (R18/A2): AIRFLOW_CONN_MINIO_ACCESS_KEY & AIRFLOW_CONN_MINIO_SECRET_KEY
# DIHAPUS dari berkas ini. Keduanya tidak pernah dikonsumsi skrip/compose mana pun
# (diverifikasi dengan grep seluruh repo) dan nilainya TIDAK SAH untuk MinIO
# produksi — pengujian langsung di experiments/airflow332 menghasilkan
# SignatureDoesNotMatch (access key `minioadmin`, sedangkan secret-nya hanya
# 10 karakter padahal password root 32 karakter).
# Jalur resmi kredensial MinIO adalah S3_ACCESS_KEY / S3_SECRET_KEY dengan
# fallback saat-pakai ke MINIO_ROOT_* (lihat blok "S3 / MinIO" di atas) — itulah
# yang dipakai bootstrap.sh untuk membangun connection `minio_s3`, dan cara itu
# sudah terverifikasi bekerja. Ini sekaligus menutup item R02-L4.
# Catatan: AIRFLOW_CONN_TRINO_HOST/PORT DIHAPUS pada review Fase 5 — tidak pernah
# dikonsumsi skrip/compose mana pun, dan connection 'trino_default' yang dulu
# dibangun darinya sudah tidak disediakan (tidak dipakai DAG; versi lamanya
# menunjuk http://trino:8080 tanpa kredensial → 403). Lihat §7.6.
# ─── OpenMetadata (Phase 6 — Governance: Metadata) ─────────────────────────
# BATCH B (review R18): 2.0.2 -> 2.0.3 (wajib seversi dengan openmetadata-ingestion
# di image Airflow). Migrasi OM bersifat SATU ARAH — rollback = pulihkan dump
# (lihat deploy/output/merge-airflow332-rollback.md).
export OPENMETADATA_VERSION="${OPENMETADATA_VERSION:-2.0.3}"
export OPENMETADATA_PORT="${OPENMETADATA_PORT:-12085}"
export OPENMETADATA_DB="${OPENMETADATA_DB:-openmetadata_db}"
export OPENMETADATA_DB_USER="${OPENMETADATA_DB_USER:-openmetadata}"
# Endpoint dari sisi HOST (gate Fase 6, bootstrap step 7, test-verification).
# Untuk OpenMetadata EXISTING: OPENMETADATA_URL=http://om.corp:8585
export OPENMETADATA_URL="${OPENMETADATA_URL:-http://localhost:${OPENMETADATA_PORT:-12085}}"
# ─── Grafana (Phase 6 — Governance: Observability) ─────────────────────────
export GRAFANA_VERSION="${GRAFANA_VERSION:-13.1}"
export GRAFANA_PORT="${GRAFANA_PORT:-12300}"
# Endpoint dari sisi HOST. Untuk Grafana EXISTING: GRAFANA_URL=http://grafana.corp:3000
export GRAFANA_URL="${GRAFANA_URL:-http://localhost:${GRAFANA_PORT:-12300}}"
# Prometheus — sumber metrik untuk datasource Grafana (fase 6).
export PROMETHEUS_VERSION="${PROMETHEUS_VERSION:-v3.6.0}"
export PROMETHEUS_PORT="${PROMETHEUS_PORT:-12090}"
# statsd-exporter — metrik Airflow (StatsD/UDP 9125) → Prometheus :9102 (M2 observabilitas).
export STATSD_EXPORTER_VERSION="${STATSD_EXPORTER_VERSION:-v0.28.0}"
export STATSD_EXPORTER_PORT="${STATSD_EXPORTER_PORT:-12091}"
# jmx-exporter — saturasi Trino (JMX RMI) → Prometheus :9404 (M3 observabilitas).
export JMX_EXPORTER_VERSION="${JMX_EXPORTER_VERSION:-latest}"
# ─── Port tambahan hasil merge eksperimen (Fase 3-6) ────────────────────────
export OPENSEARCH_PORT="${OPENSEARCH_PORT:-12920}"
export OPA_VERSION="${OPA_VERSION:-1.20.2}"
export OPA_PORT="${OPA_PORT:-12181}"
# Endpoint OPA dari sisi HOST (gate verifikasi + bootstrap step 4). Untuk OPA
# EXISTING: OPA_URL=http://opa.corp:8181 (policy path tetap /v1/data/trino/allow).
export OPA_URL="${OPA_URL:-http://localhost:${OPA_PORT:-12181}}"
export OPENLDAP_PORT="${OPENLDAP_PORT:-12389}"
export DREMIO_UI_PORT="${DREMIO_UI_PORT:-12947}"
export DREMIO_JDBC_PORT="${DREMIO_JDBC_PORT:-12010}"
export DREMIO_FLIGHT_PORT="${DREMIO_FLIGHT_PORT:-12011}"
# ─── Endpoint IAM (Keycloak & OpenLDAP) ─────────────────────────────────────
# Default = container lokal hasil deploy ini. Untuk memakai IAM EXISTING,
# override di .env (tiga variabel):
#
# KEYCLOAK_URL=http://kc.internal:8080 # dijangkau dari HOST (bootstrap/verify)
# LDAP_URL=ldap://ldap.internal:389 # dijangkau dari CONTAINER (Trino & StarRocks)
# LDAP_HOST=ldap.internal # dijangkau dari HOST (bootstrap/verify)
# LDAP_PORT=389
#
# Dua alamat diperlukan karena pada mode lokal hostname-nya berbeda:
# container melihat `openldap:389`, host melihat `localhost:12389`.
export KEYCLOAK_URL="${KEYCLOAK_URL:-http://localhost:${KEYCLOAK_PORT:-12080}}"
export LDAP_URL="${LDAP_URL:-ldap://openldap:389}"
export LDAP_HOST="${LDAP_HOST:-localhost}"
export LDAP_PORT="${LDAP_PORT:-${OPENLDAP_PORT:-12389}}"
# Dremio admin dibuat otomatis oleh bootstrap.sh (Step 9) via REST API first-run,
# lalu dipakai script scripts/dremio/register-nessie-source.sh untuk source Nessie.
export DREMIO_URL="${DREMIO_URL:-http://localhost:12947}"
# ─── Port INTERNAL (container ↔ container) — bisa diparameterkan ─────────────
# Port yang dipakai ANTAR-container untuk menjangkau layanan; default = port
# container standar. Ubah HANYA bila layanan yang dipakai berjalan di port berbeda
# (mis. Keycloak/Postgres EXISTING) — cukup di .env, TANPA menyunting docker-compose.yml.
# Layanan milik cluster sendiri selalu memakai DEFAULT ini (hostname service + port container),
# jadi mengubahnya hanya relevan untuk layanan eksternal berport non-standar.
export POSTGRES_INTERNAL_PORT="${POSTGRES_INTERNAL_PORT:-5432}"
export KEYCLOAK_INTERNAL_PORT="${KEYCLOAK_INTERNAL_PORT:-8080}"
export MINIO_INTERNAL_PORT="${MINIO_INTERNAL_PORT:-9000}"
export NESSIE_INTERNAL_PORT="${NESSIE_INTERNAL_PORT:-19120}"
export OPENLDAP_INTERNAL_PORT="${OPENLDAP_INTERNAL_PORT:-389}"
export OPA_INTERNAL_PORT="${OPA_INTERNAL_PORT:-8181}"
# ─── Komponen EXISTING (dikelola sistem lain) ────────────────────────────────
# Bila sebuah komponen sudah dikelola sistem lain, container-nya TIDAK perlu
# (dan tidak boleh) dinyalakan oleh deploy ini. install.sh menyimpulkan ini
# secara OTOMATIS dari endpoint di atas: bila nilainya berbeda dari nilai lokal
# docker-compose.yml, komponen itu dianggap eksternal.
#
# keycloak lokal http://localhost:12080 → jadikan eksternal: KEYCLOAK_URL
# openldap lokal ldap://openldap:389 → LDAP_URL
# minio lokal http://minio:9000 → S3_ENDPOINT (+ minio-init dilewati)
# nessie lokal http://nessie:19120 → NESSIE_URL
# trino lokal https://localhost:12443 → TRINO_URL
# opa lokal http://localhost:12181 → OPA_URL
#
# Postgres tidak punya variabel endpoint → set eksplisit bila dikelola DBA.
# EXTERNAL_SERVICES menang atas auto-deteksi; LOCAL_SERVICES memaksa kembali lokal.
# Nama service mengikuti docker-compose.yml (mis. starrocks-fe, airflow-webserver).
export EXTERNAL_SERVICES="${EXTERNAL_SERVICES:-}"
export LOCAL_SERVICES="${LOCAL_SERVICES:-}"
export DREMIO_USER="${DREMIO_USER:-admin}"
export DREMIO_PASSWORD="${DREMIO_PASSWORD:-dremio_admin_DevOnly1}"
export DREMIO_ADMIN_EMAIL="${DREMIO_ADMIN_EMAIL:-admin@sicabe.local}"
# ─── StarRocks credentials (root + aplikasi) ────────────────────────────────
export STARROCKS_ROOT_PASSWORD="${STARROCKS_ROOT_PASSWORD:-starrocks_root_DevOnly}"
export STARROCKS_APP_PASSWORD="${STARROCKS_APP_PASSWORD:-starrocks_app_DevOnly}"
deploy/config/grafana/dashboards/dashboards.yaml
+16
View File
@@ -0,0 +1,16 @@
# =============================================================================
# Grafana — Provisioning Dashboards
# Referensi arsitektur: bab 4.6.3.3 (Setup dashboard bawaan)
# =============================================================================
apiVersion: 1
providers:
- name: "SiCABE Data Quality"
orgId: 1
folder: "SiCABE"
type: file
disableDeletion: false
editable: true
updateIntervalSeconds: 60
options:
path: /etc/grafana/provisioning/dashboards/sicabe
deploy/config/grafana/dashboards/sicabe/sicabe_hosts.json
+260
View File
@@ -0,0 +1,260 @@
{
"title": "SiCABE — Host Docker (node_exporter)",
"uid": "sicabe-hosts",
"version": 1,
"timezone": "Asia/Jakarta",
"schemaVersion": 36,
"refresh": "30s",
"time": { "from": "now-6h", "to": "now" },
"tags": ["sicabe", "host", "node_exporter"],
"panels": [
{
"id": 1,
"title": "Host Up",
"type": "stat",
"datasource": "Prometheus",
"gridPos": { "h": 8, "w": 4, "x": 0, "y": 0 },
"description": "Status node_exporter per host (:9100, pra-syarat host di luar installer). 1 = up, 0 = down. Job: node.",
"options": {
"reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false },
"colorMode": "background",
"graphMode": "none",
"textMode": "value_and_name",
"orientation": "vertical"
},
"fieldConfig": {
"defaults": {
"unit": "short",
"mappings": [
{
"type": "value",
"options": {
"0": { "text": "DOWN", "color": "red", "index": 1 },
"1": { "text": "UP", "color": "green", "index": 0 }
}
}
],
"thresholds": {
"mode": "absolute",
"steps": [
{ "color": "red", "value": null },
{ "color": "green", "value": 1 }
]
}
},
"overrides": []
},
"targets": [
{ "refId": "A", "expr": "up{job=\"node\"}", "legendFormat": "{{instance}}" }
]
},
{
"id": 2,
"title": "CPU Usage %",
"type": "timeseries",
"datasource": "Prometheus",
"gridPos": { "h": 8, "w": 10, "x": 4, "y": 0 },
"description": "Pemakaian CPU per host = 100 × (1 - idle), rata-rata jendela 5m.",
"fieldConfig": {
"defaults": {
"unit": "percent",
"min": 0,
"max": 100,
"custom": { "drawStyle": "line", "fillOpacity": 10, "lineWidth": 1, "showPoints": "never" }
},
"overrides": []
},
"options": {
"legend": { "displayMode": "table", "placement": "bottom", "calcs": ["mean", "max"] },
"tooltip": { "mode": "multi", "sort": "desc" }
},
"targets": [
{
"refId": "A",
"expr": "100 * (1 - avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[5m])))",
"legendFormat": "{{instance}}"
}
]
},
{
"id": 3,
"title": "Load Average (1/5/15m)",
"type": "timeseries",
"datasource": "Prometheus",
"gridPos": { "h": 8, "w": 10, "x": 14, "y": 0 },
"description": "Load average 1/5/15 menit per host. Bandingkan dengan jumlah CPU; load > jumlah CPU = jenuh.",
"fieldConfig": {
"defaults": {
"unit": "short",
"custom": { "drawStyle": "line", "fillOpacity": 0, "lineWidth": 1, "showPoints": "never" }
},
"overrides": []
},
"options": {
"legend": { "displayMode": "table", "placement": "bottom", "calcs": ["mean", "max"] },
"tooltip": { "mode": "multi", "sort": "desc" }
},
"targets": [
{ "refId": "A", "expr": "node_load1", "legendFormat": "{{instance}} load1" },
{ "refId": "B", "expr": "node_load5", "legendFormat": "{{instance}} load5" },
{ "refId": "C", "expr": "node_load15", "legendFormat": "{{instance}} load15" }
]
},
{
"id": 4,
"title": "Memory Used %",
"type": "timeseries",
"datasource": "Prometheus",
"gridPos": { "h": 8, "w": 8, "x": 0, "y": 8 },
"description": "Pemakaian memori per host = 100 × (1 - MemAvailable/MemTotal). Ambang: 80% warning, 90% critical.",
"fieldConfig": {
"defaults": {
"unit": "percent",
"min": 0,
"max": 100,
"custom": { "drawStyle": "line", "fillOpacity": 10, "lineWidth": 1, "showPoints": "never" },
"thresholds": {
"mode": "absolute",
"steps": [
{ "color": "green", "value": null },
{ "color": "orange", "value": 80 },
{ "color": "red", "value": 90 }
]
}
},
"overrides": []
},
"options": {
"legend": { "displayMode": "table", "placement": "bottom", "calcs": ["mean", "max"] },
"tooltip": { "mode": "multi", "sort": "desc" }
},
"targets": [
{
"refId": "A",
"expr": "100 * (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)",
"legendFormat": "{{instance}}"
}
]
},
{
"id": 5,
"title": "Disk I/O",
"type": "timeseries",
"datasource": "Prometheus",
"gridPos": { "h": 8, "w": 8, "x": 8, "y": 8 },
"description": "Laju baca/tulis disk per host (rata-rata 5m, semua device digabung).",
"fieldConfig": {
"defaults": {
"unit": "Bps",
"custom": { "drawStyle": "line", "fillOpacity": 5, "lineWidth": 1, "showPoints": "never" }
},
"overrides": []
},
"options": {
"legend": { "displayMode": "table", "placement": "bottom", "calcs": ["mean", "max"] },
"tooltip": { "mode": "multi", "sort": "desc" }
},
"targets": [
{
"refId": "A",
"expr": "sum by (instance) (rate(node_disk_read_bytes_total[5m]))",
"legendFormat": "{{instance}} read"
},
{
"refId": "B",
"expr": "sum by (instance) (rate(node_disk_written_bytes_total[5m]))",
"legendFormat": "{{instance}} write"
}
]
},
{
"id": 6,
"title": "Network Traffic",
"type": "timeseries",
"datasource": "Prometheus",
"gridPos": { "h": 8, "w": 8, "x": 16, "y": 8 },
"description": "Laju RX/TX per host, antarmuka fisik saja (lo/veth/docker/bridge dikecualikan).",
"fieldConfig": {
"defaults": {
"unit": "Bps",
"custom": { "drawStyle": "line", "fillOpacity": 5, "lineWidth": 1, "showPoints": "never" }
},
"overrides": []
},
"options": {
"legend": { "displayMode": "table", "placement": "bottom", "calcs": ["mean", "max"] },
"tooltip": { "mode": "multi", "sort": "desc" }
},
"targets": [
{
"refId": "A",
"expr": "sum by (instance) (rate(node_network_receive_bytes_total{device!~\"lo|veth.*|docker.*|br-.*|virbr.*\"}[5m]))",
"legendFormat": "{{instance}} RX"
},
{
"refId": "B",
"expr": "sum by (instance) (rate(node_network_transmit_bytes_total{device!~\"lo|veth.*|docker.*|br-.*|virbr.*\"}[5m]))",
"legendFormat": "{{instance}} TX"
}
]
},
{
"id": 7,
"title": "Disk Usage % (per host & mount)",
"type": "table",
"datasource": "Prometheus",
"gridPos": { "h": 8, "w": 16, "x": 0, "y": 16 },
"description": "Persentase pemakaian tiap mount. Filesystem pseudo (tmpfs/overlay/squashfs/ramfs) dikecualikan. Ambang: 80% warning, 90% critical.",
"fieldConfig": {
"defaults": {
"unit": "percent",
"decimals": 1,
"thresholds": {
"mode": "absolute",
"steps": [
{ "color": "green", "value": null },
{ "color": "orange", "value": 80 },
{ "color": "red", "value": 90 }
]
}
},
"overrides": [
{
"matcher": { "id": "byName", "options": "Value" },
"properties": [
{ "id": "custom.cellOptions", "value": { "type": "color-background", "mode": "gradient" } }
]
}
]
},
"options": { "showHeader": true },
"targets": [
{
"refId": "A",
"expr": "100 * (1 - node_filesystem_avail_bytes{fstype!~\"tmpfs|overlay|squashfs|ramfs\"} / node_filesystem_size_bytes{fstype!~\"tmpfs|overlay|squashfs|ramfs\"})",
"format": "table",
"instant": true
}
]
},
{
"id": 8,
"title": "Uptime",
"type": "stat",
"datasource": "Prometheus",
"gridPos": { "h": 8, "w": 8, "x": 16, "y": 16 },
"description": "Lama host hidup sejak boot = time() - node_boot_time_seconds.",
"options": {
"reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false },
"colorMode": "none",
"graphMode": "none",
"textMode": "value_and_name",
"orientation": "vertical"
},
"fieldConfig": { "defaults": { "unit": "dtdurations" }, "overrides": [] },
"targets": [
{ "refId": "A", "expr": "time() - node_boot_time_seconds", "legendFormat": "{{instance}}" }
]
}
]
}
deploy/config/grafana/dashboards/sicabe/sicabe_overview.json
+159
View File
@@ -0,0 +1,159 @@
{
"title": "SiCABE — Overview Lapisan & Tata Kelola Data",
"version": 6,
"timezone": "Asia/Jakarta",
"schemaVersion": 36,
"refresh": "30s",
"time": { "from": "now-6h", "to": "now" },
"panels": [
{
"id": 1,
"title": "DAG Failure Rate (Ingestion Layer)",
"type": "stat",
"datasource": "Prometheus",
"gridPos": { "h": 8, "w": 8, "x": 0, "y": 0 },
"description": "KPI #7 — rasio DAG run gagal (jendela 1 jam). Ambang: 5% warning, 15% critical. Sumber: airflow_dagrun_duration_{failed,success}_count (Airflow StatsD → statsd-exporter).",
"fieldConfig": {
"defaults": {
"unit": "percent",
"min": 0,
"thresholds": {
"mode": "absolute",
"steps": [
{ "color": "green", "value": null },
{ "color": "orange", "value": 5 },
{ "color": "red", "value": 15 }
]
}
},
"overrides": []
},
"targets": [
{ "refId": "A", "expr": "100 * (sum(rate(airflow_dagrun_duration_failed_count[1h])) or vector(0)) / clamp_min((sum(rate(airflow_dagrun_duration_failed_count[1h])) or vector(0)) + (sum(rate(airflow_dagrun_duration_success_count[1h])) or vector(0)), 0.0001)" }
]
},
{
"id": 2,
"title": "Disk Usage (MinIO — Storage Layer)",
"type": "gauge",
"datasource": "MinIO Metrics",
"gridPos": { "h": 8, "w": 8, "x": 8, "y": 0 },
"description": "KPI #1 — kapasitas terpakai (%) = 1 - usable_free/usable_total. Ambang: 80% warning, 90% critical.",
"fieldConfig": {
"defaults": {
"unit": "percent",
"min": 0,
"max": 100,
"thresholds": {
"mode": "absolute",
"steps": [
{ "color": "green", "value": null },
{ "color": "orange", "value": 80 },
{ "color": "red", "value": 90 }
]
}
},
"overrides": []
},
"targets": [
{ "refId": "A", "expr": "(1 - minio_cluster_capacity_usable_free_bytes / minio_cluster_capacity_usable_total_bytes) * 100" }
]
},
{
"id": 3,
"title": "Serving Query Latency P99 (StarRocks)",
"type": "gauge",
"datasource": "Prometheus",
"gridPos": { "h": 8, "w": 8, "x": 16, "y": 0 },
"description": "KPI #6 — latensi query P99 StarRocks FE. Ambang: 500 ms warning, 2000 ms critical.",
"fieldConfig": {
"defaults": {
"unit": "ms",
"thresholds": {
"mode": "absolute",
"steps": [
{ "color": "green", "value": null },
{ "color": "orange", "value": 500 },
{ "color": "red", "value": 2000 }
]
}
},
"overrides": []
},
"targets": [
{ "refId": "A", "expr": "starrocks_fe_query_latency_ms{quantile=\"0.99\"}" }
]
},
{
"id": 4,
"title": "Trino — Saturasi Query (coordinator)",
"type": "timeseries",
"datasource": "Prometheus",
"gridPos": { "h": 9, "w": 12, "x": 0, "y": 8 },
"description": "KPI #4 (SATURASI) — antrean & kegagalan query Trino coordinator. ⚠️ P99 LATENCY tidak di sini: tidak tersedia via JMX/StatsD (metrik turunan, di luar lingkup M1–M3). Sumber: JMX → trino-jmx-exporter.",
"targets": [
{ "refId": "A", "legendFormat": "running query", "expr": "trino_execution_querymanager_runningqueries" },
{ "refId": "B", "legendFormat": "queued query", "expr": "trino_execution_querymanager_queuedqueries" },
{ "refId": "C", "legendFormat": "failed /s", "expr": "sum(rate(trino_execution_querymanager_failedqueries_totalcount[5m]))" }
]
},
{
"id": 5,
"title": "S3 API Latency (MinIO) — % panggilan lambat",
"type": "timeseries",
"datasource": "MinIO Metrics",
"gridPos": { "h": 9, "w": 12, "x": 12, "y": 8 },
"description": "KPI #2 — rasio panggilan S3 yang lebih lambat dari ambang (jendela 5m). Warning 50 ms; critical ~250 ms (bucket terdekat — MinIO tidak menyediakan bucket 200 ms). Basis metrik minio_s3_requests_ttfb_seconds_distribution bertipe gauge (bukan histogram), sehingga dipakai rasio barbucket, bukan histogram_quantile.",
"fieldConfig": {
"defaults": { "unit": "percent", "min": 0 },
"overrides": []
},
"targets": [
{
"refId": "A",
"legendFormat": "% > 50 ms (warning)",
"expr": "100 * (1 - sum(rate(minio_s3_requests_ttfb_seconds_distribution{le=\"0.050\"}[5m])) / sum(rate(minio_s3_requests_ttfb_seconds_distribution{le=\"+Inf\"}[5m])))"
},
{
"refId": "B",
"legendFormat": "% > 250 ms (critical)",
"expr": "100 * (1 - sum(rate(minio_s3_requests_ttfb_seconds_distribution{le=\"0.250\"}[5m])) / sum(rate(minio_s3_requests_ttfb_seconds_distribution{le=\"+Inf\"}[5m])))"
}
]
},
{
"id": 6,
"title": "Airflow — Durasi DAG & Task",
"type": "timeseries",
"datasource": "Prometheus",
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 17 },
"description": "KPI #5 (perkiraan) — durasi pipeline & task. Airflow pada StatsD ini TIDAK memecah durasi per-task (hanya summary global `airflow_task_duration`), jadi durasi dbt (silver_to_gold/bronze_to_silver) belum bisa diisolasi. Perbaikan menyusul: statsd mapping config (lihat Review 19). DAG P90 = durasi `sicabe_batch_daily`.",
"targets": [
{
"refId": "A",
"legendFormat": "rata-rata task (dtk)",
"expr": "sum(rate(airflow_task_duration_sum[1h])) / clamp_min(sum(rate(airflow_task_duration_count[1h])), 0.0001)"
},
{
"refId": "B",
"legendFormat": "DAG sicabe_batch_daily P90 (dtk)",
"expr": "airflow_dagrun_duration_success_sicabe_batch_daily{quantile=\"0.9\"}"
}
]
},
{
"id": 7,
"title": "Trino — Query Throughput (coordinator)",
"type": "timeseries",
"datasource": "Prometheus",
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 17 },
"description": "Laju siklus hidup query Trino coordinator (per detik, jendela 5m). Sumber: JMX → trino-jmx-exporter (domain trino.execution).",
"targets": [
{ "refId": "A", "legendFormat": "submitted /s", "expr": "sum(rate(trino_execution_querymanager_submittedqueries_totalcount[5m]))" },
{ "refId": "B", "legendFormat": "completed /s", "expr": "sum(rate(trino_execution_querymanager_completedqueries_totalcount[5m]))" },
{ "refId": "C", "legendFormat": "failed /s", "expr": "sum(rate(trino_execution_querymanager_failedqueries_totalcount[5m]))" },
{ "refId": "D", "legendFormat": "canceled /s", "expr": "sum(rate(trino_execution_querymanager_canceledqueries_totalcount[5m]))" }
]
}
]
}
deploy/config/grafana/datasources/datasources.yaml
+42
View File
@@ -0,0 +1,42 @@
# =============================================================================
# eBdesk SiCABE — Grafana datasource (hasil provisioning)
# =============================================================================
# Datasource bertipe `prometheus` WAJIB menunjuk server Prometheus
# (`http://prometheus:9090`). Menunjuk `http://minio:9000` akan selalu gagal
# ("error querying the Prometheus API") karena MinIO adalah API S3, bukan
# server Prometheus — metrik MinIO diambil Prometheus lewat scrape
# (lihat config/prometheus/prometheus.yml), bukan dibaca langsung oleh Grafana.
#
# Datasource OpenMetadata DIHAPUS pada review Fase 6: plugin
# `grafana-opendatasource-http` TIDAK ADA di katalog Grafana (404) sehingga
# selalu "Plugin not registered". Tidak ada plugin resmi Grafana → OpenMetadata;
# arah yang didukung OpenMetadata adalah sebaliknya (OM meng-ingest Grafana).
# Bila data OM ingin ditampilkan di Grafana, pakai datasource HTTP generik
# (mis. Infinity) + token API OpenMetadata.
# =============================================================================
apiVersion: 1
# Grafana provisioning hanya MENAMBAH/MENGUBAH — datasource yang dihapus dari
# berkas ini TIDAK ikut terhapus. Karena itu penghapusan ditulis eksplisit di
# bawah (idempoten, aman dijalankan berulang).
deleteDatasources:
- name: OpenMetadata
orgId: 1
datasources:
- name: Prometheus
type: prometheus
access: proxy
url: http://prometheus:9090
isDefault: true
editable: false
# Metrik MinIO tetap memakai instance Prometheus yang sama (job_name: minio),
# dinamai terpisah karena dipakai panel "Storage Usage" di dashboard.
- name: MinIO Metrics
type: prometheus
access: proxy
url: http://prometheus:9090
editable: false
jsonData:
tlsSkipVerify: true
deploy/config/keycloak/realm-export.json
+160
View File
@@ -0,0 +1,160 @@
{
"//": "⚠️ File referensi — gunakan bootstrap.sh untuk deployment otomatis.",
"//2": "JSON tidak mendukung shell variable expansion. Ubah nilai secara manual jika perlu import via UI.",
"realm": "sicabe",
"displayName": "eBdesk SiCABE",
"enabled": true,
"sslRequired": "none",
"registrationAllowed": false,
"loginWithEmailAllowed": true,
"resetPasswordAllowed": true,
"accessTokenLifespan": 3600,
"ssoSessionMaxLifespan": 86400,
"clients": [
{
"clientId": "trino",
"name": "Trino — Query Engine",
"description": "OIDC client untuk Trino (Processing Layer)",
"enabled": true,
"protocol": "openid-connect",
"clientAuthenticatorType": "client-secret",
"secret": "trino-secret",
"standardFlowEnabled": true,
"serviceAccountsEnabled": true,
"publicClient": false,
"redirectUris": ["http://localhost:*", "https://localhost:*"],
"webOrigins": ["http://localhost:*", "https://localhost:*"]
},
{
"clientId": "starrocks",
"name": "StarRocks — OLAP Engine",
"description": "OIDC client untuk StarRocks (Serving Layer)",
"enabled": true,
"protocol": "openid-connect",
"clientAuthenticatorType": "client-secret",
"secret": "starrocks-secret",
"standardFlowEnabled": true,
"serviceAccountsEnabled": true,
"publicClient": false,
"redirectUris": ["http://localhost:12030/*", "http://starrocks-fe:8030/api/oauth2"],
"webOrigins": ["http://localhost:12030"]
},
{
"clientId": "airflow",
"name": "Apache Airflow — Orchestrator",
"description": "OIDC client untuk Airflow (Ingestion Layer)",
"enabled": true,
"protocol": "openid-connect",
"clientAuthenticatorType": "client-secret",
"secret": "airflow-secret",
"standardFlowEnabled": true,
"serviceAccountsEnabled": true,
"publicClient": false,
"redirectUris": ["http://localhost:12082/*"],
"webOrigins": ["http://localhost:12082"]
},
{
"clientId": "openmetadata",
"name": "OpenMetadata — Metadata Platform",
"description": "OIDC client untuk OpenMetadata (Governance Layer)",
"enabled": true,
"protocol": "openid-connect",
"clientAuthenticatorType": "client-secret",
"secret": "om-secret",
"standardFlowEnabled": true,
"serviceAccountsEnabled": true,
"publicClient": false,
"redirectUris": ["http://localhost:12085/*"],
"webOrigins": ["http://localhost:12085"]
},
{
"clientId": "grafana",
"name": "Grafana — Observability",
"description": "OIDC client untuk Grafana (Governance Layer)",
"enabled": true,
"protocol": "openid-connect",
"clientAuthenticatorType": "client-secret",
"secret": "grafana-secret",
"standardFlowEnabled": true,
"serviceAccountsEnabled": true,
"publicClient": false,
"redirectUris": ["http://localhost:12300/*"],
"webOrigins": ["http://localhost:12300"]
}
],
"roles": {
"realm": [
{
"name": "admin",
"description": "Administrator — akses penuh ke seluruh komponen"
},
{
"name": "engineer",
"description": "Data Engineer — akses ke Ingestion, Processing, Storage"
},
{
"name": "analyst",
"description": "Data Analyst — akses baca ke Serving dan Governance"
},
{
"name": "viewer",
"description": "Viewer — akses baca terbatas ke dashboard dan metadata"
}
]
},
"users": [
{
"username": "admin",
"enabled": true,
"email": "admin@sicabe.local",
"firstName": "Administrator",
"lastName": "SiCABE",
"credentials": [{ "type": "password", "value": "admin123", "temporary": false }],
"realmRoles": ["admin"],
"clientRoles": {
"trino": ["admin"],
"starrocks": ["admin"],
"airflow": ["admin"],
"openmetadata": ["admin"],
"grafana": ["Admin"]
}
},
{
"username": "engineer",
"enabled": true,
"email": "engineer@sicabe.local",
"firstName": "Data",
"lastName": "Engineer",
"credentials": [{ "type": "password", "value": "engineer123", "temporary": false }],
"realmRoles": ["engineer"]
},
{
"username": "analyst",
"enabled": true,
"email": "analyst@sicabe.local",
"firstName": "Data",
"lastName": "Analyst",
"credentials": [{ "type": "password", "value": "analyst123", "temporary": false }],
"realmRoles": ["analyst"]
},
{
"username": "viewer",
"enabled": true,
"email": "viewer@sicabe.local",
"firstName": "Viewer",
"lastName": "SiCABE",
"credentials": [{ "type": "password", "value": "viewer123", "temporary": false }],
"realmRoles": ["viewer"]
}
],
"scopeMappings": [
{
"client": "trino",
"roles": ["admin", "engineer"]
},
{
"client": "airflow",
"roles": ["admin", "engineer"]
}
]
}
deploy/config/minio/buckets.sh
Executable
+30
View File
@@ -0,0 +1,30 @@
#!/usr/bin/env bash
# =============================================================================
# MinIO — Inisialisasi Bucket dan Policy
# Referensi arsitektur: bab 4.2.1 (MinIO — Object Storage)
# - 4.2.1.7 Buat bucket awal (iceberg-data)
# - 4.2.1.8 Buat service account dan policy akses
# =============================================================================
set -euo pipefail
MINIO_ALIAS="${MINIO_ALIAS:-sicabe}"
MINIO_ENDPOINT="${MINIO_ENDPOINT:-http://minio:9000}"
MINIO_ROOT_USER="${MINIO_ROOT_USER:-minioadmin}"
MINIO_ROOT_PASSWORD="${MINIO_ROOT_PASSWORD:?err}"
MINIO_BUCKET_ICEBERG_DATA="${MINIO_BUCKET_ICEBERG_DATA:-iceberg-data}"
echo "[MinIO] Configuring alias..."
mc alias set "${MINIO_ALIAS}" "${MINIO_ENDPOINT}" "${MINIO_ROOT_USER}" "${MINIO_ROOT_PASSWORD}"
# ─── Buat bucket awal (4.2.1.7) ──────────────────────────────────────────────
echo "[MinIO] Creating buckets..."
for bucket in "${MINIO_BUCKET_ICEBERG_DATA}"; do
if mc ls "${MINIO_ALIAS}/${bucket}" >/dev/null 2>&1; then
echo "[MinIO] Bucket '${bucket}' already exists, skipping."
else
mc mb "${MINIO_ALIAS}/${bucket}"
echo "[MinIO] Bucket '${bucket}' created."
fi
done
echo "[MinIO] Buckets initialized successfully."
deploy/config/nessie/README.md
+82
View File
@@ -0,0 +1,82 @@
# Nessie — Konfigurasi `deploy/`
Nessie menggantikan Apache Polaris sebagai **Iceberg REST catalog** pada
`deploy/`. Konfigurasi diberikan sepenuhnya melalui **environment variable
di `docker-compose.yml`** (bukan file `application.properties`), dengan alasan:
- Kredensial (password PostgreSQL, access key MinIO) harus berasal dari `.env`
yang di-generate `install.sh` — mengikuti pola komponen lain di SiCABE.
- Env var adalah bentuk yang didokumentasikan resmi untuk image Nessie dan tidak
bergantung pada lokasi mount file config di dalam image.
## Versi
| Item | Nilai |
|---|---|
| Image | `ghcr.io/projectnessie/nessie:${NESSIE_VERSION:-0.108.4-java}` |
| Port API/Iceberg REST | `19120` (host `${NESSIE_PORT:-12120}`) |
| Port management (health/metrics) | `9000` (internal, tidak dipublikasikan) |
| Path Iceberg REST | `/iceberg/` (default warehouse, branch `main`) |
| Path Nessie REST v2 | `/api/v2` |
## Persistensi: JDBC2 di PostgreSQL
Mengikuti **konvensi shared PostgreSQL** SiCABE (lihat `config/postgres/init/01-create-dbs.sh`):
```yaml
NESSIE_VERSION_STORE_TYPE: "JDBC2"
NESSIE_VERSION_STORE_PERSIST_JDBC_DATASOURCE: "postgresql"
QUARKUS_DATASOURCE_POSTGRESQL_DB_KIND: "postgresql"
QUARKUS_DATASOURCE_POSTGRESQL_ACTIVE: "true"
QUARKUS_DATASOURCE_POSTGRESQL_DEVSERVICES_ENABLED: "false"
QUARKUS_DATASOURCE_POSTGRESQL_JDBC_URL: "jdbc:postgresql://postgres:5432/nessie?currentSchema=nessie"
QUARKUS_DATASOURCE_POSTGRESQL_USERNAME: "nessie"
QUARKUS_DATASOURCE_POSTGRESQL_PASSWORD: "${NESSIE_DB_PASSWORD}"
```
> **Penting**: Nessie **tidak** memakai Flyway. Database `nessie` **dan** schema
> `nessie` harus dibuat lebih dulu (dilakukan `01-create-dbs.sh`); tabel-tabel
> Nessie di-auto-create saat startup. Driver PostgreSQL sudah ter-bundle di image.
> **Catatan versi**: `JDBC` (tanpa `2`) masih ada tetapi **deprecated** di 0.108.x —
> gunakan `JDBC2`.
## Warehouse & S3/MinIO (server-side)
```yaml
NESSIE_CATALOG_DEFAULT_WAREHOUSE: "warehouse"
NESSIE_CATALOG_WAREHOUSES_WAREHOUSE_LOCATION: "s3://${MINIO_BUCKET_ICEBERG_DATA:-iceberg-data}/warehouse"
NESSIE_CATALOG_SERVICE_S3_DEFAULT_OPTIONS_ENDPOINT: "http://minio:9000"
NESSIE_CATALOG_SERVICE_S3_DEFAULT_OPTIONS_REGION: "us-east-1"
NESSIE_CATALOG_SERVICE_S3_DEFAULT_OPTIONS_AUTH_TYPE: "STATIC"
NESSIE_CATALOG_SERVICE_S3_DEFAULT_OPTIONS_PATH_STYLE_ACCESS: "true"
NESSIE_CATALOG_SERVICE_S3_DEFAULT_OPTIONS_ACCESS_KEY: "urn:nessie-secret:quarkus:minio.creds"
MINIO_CREDS_NAME: "${MINIO_ROOT_USER}"
MINIO_CREDS_SECRET: "${MINIO_ROOT_PASSWORD}"
```
> **Penting**: nama secret harus memakai **titik** (`minio.creds`), bukan hyphen.
> Quarkus env mapping tidak men-decode hyphen dengan benar (temuan eksperimen
> nessie-dremio) → error `Missing access key and secret for STATIC authentication mode`.
- Warehouse memakai bucket `iceberg-data` (konvensi base) dengan prefix `warehouse/`.
- `default-options.` **wajib** untuk Nessie ≥ 0.92.
- Kredensial S3 server-side **tidak diekspos ke klien**; engine (Trino/StarRocks/Dremio)
mengakses MinIO dengan kredensialnya sendiri.
- `external-endpoint` sengaja tidak di-set karena semua klien berada di dalam
network Docker yang sama (`minio:9000`).
## Healthcheck
Image Nessie tidak menyediakan `curl`/`wget`, sehingga healthcheck memakai
`bash /dev/tcp` ke **management port 9000**:
```yaml
test: ["CMD", "bash", "-c", "exec 3<>/dev/tcp/localhost/9000 && printf 'GET /q/health/ready HTTP/1.0\\r\\n\\r\\n' >&3 && cat <&3 | grep -q '200 OK'"]
```
## Referensi
- https://projectnessie.org/nessie-0-108-4/configuration/
- https://projectnessie.org/guides/iceberg-rest/
- Cache riset: `.tmp/external-context/nessie/`
deploy/config/opa/data.json
+37
View File
@@ -0,0 +1,37 @@
{
"roles": {
"data-admin": {
"superuser": true
},
"data-engineer": {
"tables": {
"bronze_orders": { "select": true, "write": true, "ddl": true, "columns": "*" },
"bronze_orders_incremental": { "select": true, "write": true, "ddl": true, "columns": "*" },
"silver_orders_silver": { "select": true, "write": true, "ddl": true, "columns": "*" },
"gold_orders_gold": { "select": true, "write": true, "ddl": true, "columns": "*" }
}
},
"data-catalog": {
"tables": {
"bronze_orders": { "select": true, "columns": "*" },
"bronze_orders_incremental": { "select": true, "columns": "*" },
"silver_orders_silver": { "select": true, "columns": "*" },
"silver_orders_clean": { "select": true, "columns": "*" },
"gold_orders_gold": { "select": true, "columns": "*" }
}
},
"data-analyst": {
"tables": {
"gold_orders_gold": { "select": true, "columns": "*" },
"silver_orders_silver": { "select": true, "columns": ["id", "total_value"] }
}
},
"data-viewer": {}
},
"users": {
"svc_dbt": "data-engineer",
"svc_airflow": "data-engineer",
"svc_seed": "data-engineer",
"svc_om": "data-catalog"
}
}
deploy/config/opa/policies/trino.rego
+270
View File
@@ -0,0 +1,270 @@
# =============================================================================
# OPA — policy Trino access-control (package trino) — OPA-ONLY
# -----------------------------------------------------------------------------
# Trino (access-control.name=opa) mengirim satu request per operasi:
# POST http://opa:8181/v1/data/trino/allow
# input.context.identity = { "user": "...", "groups": [...] }
# input.action.operation = "SELECT" | "INSERT" | ...
# input.action.resource = { catalogName, schemaName, tableName, columns }
# OPA mengembalikan { "result": { "allow": true|false } }.
#
# Grup user berasal dari LDAP group provider Trino (OpenLDAP, cn = data-*).
# Izin didefinisikan sebagai data di data.json → data.roles.<grup>.tables.<schema_table>.
# grant = { select: bool, write: bool, ddl: bool, columns: "*" | [..] }
# - data-admin : superuser (semua operasi)
# - select : SELECT/SHOW COLUMNS/DESCRIBE
# - write : INSERT/UPDATE/DELETE/MERGE
# - ddl : CREATE/DROP/ALTER/... TABLE
# Kolom: bila columns != "*", SELECT hanya boleh menyebut kolom tsb.
# deny-by-default.
# =============================================================================
package trino
import rego.v1
default allow := false
# Normalisasi operasi: uppercase, buang spasi & underscore.
op := upper(replace(replace(trim_space(input.action.operation), " ", ""), "_", ""))
identity_groups := input.context.identity.groups
identity_user := input.context.identity.user
# Role efektif = grup (dari LDAP) ∪ role user (data.users) — untuk service account
# yang tidak terdaftar di LDAP (mis. svc_dbt, svc_om dari password file Basic).
effective_roles contains r if {
some g in identity_groups
r := g
}
effective_roles contains r if {
r := data.users[identity_user]
}
# Grup 'data-admin' = superuser (bypass semua pembatasan data/sistem).
is_super if {
some r in effective_roles
r == "data-admin"
}
# ── CREATE/DROP SCHEMA (DDL level-schema) ───────────────────────────────────
# Trino mengirim operasi ini tanpa resource tabel → tak bisa memakai grant
# per-tabel di op_caps. Izinkan untuk peran engineering (data-admin/engineer).
allow if {
op in {"CREATESCHEMA", "DROPSCHEMA"}
some r in effective_roles
r in {"data-admin", "data-engineer"}
}
# ── Pemetaan operasi -> kemampuan yang dibutuhkan ───────────────────────────
# SELECTFROMCOLUMNS TIDAK ada di sini (ditangani khusus + cek kolom).
op_caps := {
"SHOWCOLUMNS": "select",
"DESCRIBE": "select",
"INSERTINTOTABLE": "write",
"UPDATE": "write",
"DELETE": "write",
"MERGE": "write",
"CREATETABLE": "ddl",
"DROPTABLE": "ddl",
"TRUNCATETABLE": "ddl",
"ALTERTABLE": "ddl",
"RENAMETABLE": "ddl",
"ADDCOLUMN": "ddl",
"DROPCOLUMN": "ddl",
"RENAMECOLUMN": "ddl",
"COMMENTTABLE": "ddl",
}
# ── Operasi metadata yang diizinkan utk user terautentikasi ────────────────
meta_op(opx) if startswith(opx, "SHOW")
meta_op(opx) if opx in {"USE", "SETSESSION", "RESETSESSION", "SETROLE", "SHOWROLES", "SHOWCURRENTROLES"}
# ── Eksekusi query (level sistem) ───────────────────────────────────────────
allow if op == "EXECUTEQUERY"
# ── Operasi metadata SHOW/USE/... utk user terautentikasi ──────────────────
allow if meta_op(op)
# ── Superuser ───────────────────────────────────────────────────────────────
allow if is_super
# ── Schema sistem/metadata (listing/autocomplete; isi data tetap terbatas) ─
allow if {
op == "SELECTFROMCOLUMNS"
res := input.action.resource
res.table.catalogName == "system"
res.table.schemaName in {"jdbc", "metadata", "information_schema"}
}
allow if {
op == "SELECTFROMCOLUMNS"
res := input.action.resource
res.table.schemaName == "information_schema"
}
# ── Akses & listing catalog / schema ────────────────────────────────────────
allow if {
op in {"ACCESSCATALOG", "FILTERCATALOGS"}
res := input.action.resource
res.catalog.name in {"iceberg", "system"}
}
allow if {
op == "FILTERSCHEMAS"
res := input.action.resource
res.schema.catalogName in {"iceberg", "system"}
}
allow if {
op == "FILTERTABLES"
res := input.action.resource
res.table.catalogName == "system"
res.table.schemaName in {"jdbc", "metadata", "information_schema"}
}
# ── Operasi data/metadata per tabel (non-SELECT) ────────────────────────────
# dbt membuat tabel sementara <name>__dbt_tmp (dan backup <name>__dbt_backup):
# cocokkan ke grant tabel dasarnya.
dbt_suffixes := {"__dbt_tmp", "__dbt_backup"}
has_dbt_suffix(name) if {
some s in dbt_suffixes
endswith(name, s)
}
base_table_name(name) := n if {
some s in dbt_suffixes
endswith(name, s)
n := substring(name, 0, count(name) - count(s))
}
base_table_name(name) := name if {
not has_dbt_suffix(name)
}
table_id(schema, table) := sprintf("%s_%s", [schema, base_table_name(table)])
allow if {
cap := op_caps[op]
res := input.action.resource
tid := table_id(res.table.schemaName, res.table.tableName)
some r in effective_roles
grant := data.roles[r].tables[tid]
grant[cap] == true
}
# ── SELECT ... (per-kolom) ──────────────────────────────────────────────────
# Semua kolom yang diminta harus diizinkan oleh grant.columns.
cols_all_ok(columns, cols) if columns == "*"
cols_all_ok(columns, cols) if {
columns != "*"
every c in cols {
c in columns
}
}
allow if {
op == "SELECTFROMCOLUMNS"
res := input.action.resource
res.table.catalogName == "iceberg"
tid := table_id(res.table.schemaName, res.table.tableName)
cols := res.table.columns
count(cols) > 0
some r in effective_roles
grant := data.roles[r].tables[tid]
grant.select == true
cols_all_ok(grant.columns, cols)
}
# ── SELECT tanpa kolom eksplisit (mis. count(*)) ────────────────────────────
# Trino mengirim columns: [] untuk count(*)/agregasi tanpa kolom. Izinkan bila
# user punya 'select' pada tabel tsb (tidak membocorkan isi kolom). Tanpa rule
# ini, count(*) selalu ditolak karena rule SELECT di atas butuh count(cols) > 0.
allow if {
op == "SELECTFROMCOLUMNS"
res := input.action.resource
res.table.catalogName == "iceberg"
tid := table_id(res.table.schemaName, res.table.tableName)
count(res.table.columns) == 0
some r in effective_roles
grant := data.roles[r].tables[tid]
grant.select == true
}
# ── FilterColumns — daftar kolom utk metadata (information_schema/SHOW COLUMNS)
# Trino memakai operasi ini untuk memfilter kolom yang boleh DILIHAT. Tanpa rule
# ini (deny-by-default) SEMUA kolom tersembunyi dari information_schema,
# `SHOW COLUMNS`, dan `DESCRIBE` — padahal SELECT-nya sendiri diizinkan. Akibat
# nyata: katalog metadata (mis. ingestion OpenMetadata) melihat tabel TANPA kolom.
# Batas per-kolom tetap dihormati lewat cols_all_ok (least privilege).
allow if {
op == "FILTERCOLUMNS"
res := input.action.resource
res.table.schemaName == "information_schema"
}
allow if {
op == "FILTERCOLUMNS"
res := input.action.resource
res.table.catalogName == "system"
}
allow if {
op == "FILTERCOLUMNS"
res := input.action.resource
tid := table_id(res.table.schemaName, res.table.tableName)
some r in effective_roles
grant := data.roles[r].tables[tid]
grant.select == true
grant.columns == "*"
}
allow if {
op == "FILTERCOLUMNS"
res := input.action.resource
tid := table_id(res.table.schemaName, res.table.tableName)
cols := res.table.columns
count(cols) > 0
some r in effective_roles
grant := data.roles[r].tables[tid]
grant.select == true
cols_all_ok(grant.columns, cols)
}
# ── Listing tabel (FilterTables): tabel terlihat bila ada grant apapun ─────
any_cap(grant) if {
some cap in ["select", "write", "ddl"]
grant[cap] == true
}
allow if {
op == "FILTERTABLES"
res := input.action.resource
res.table.catalogName == "iceberg"
tid := table_id(res.table.schemaName, res.table.tableName)
some r in effective_roles
grant := data.roles[r].tables[tid]
any_cap(grant)
}
allow if {
op == "SetSystemSessionProperty"
}
# ── DEV/EVAL: data-engineer bebas DDL + tulis di schema bronze ───────────────
# Diminta eksplisit (2026-09-22) untuk mempermudah ingestion (Sling) dan
# transform (dbt) di layer bronze.
#
# ⚠️ Cakupan blok ini SENGAJA LEBIH LUAS dari least-privilege per-tabel:
# 1. Berlaku untuk OPERASI APA PUN yang menargetkan tabel di schema bronze —
# termasuk operasi level-kolom yang dikirim Trino untuk UPDATE/MERGE, yang
# tidak punya entri di op_caps sehingga sebelumnya jatuh ke deny-by-default.
# Tanpa ini, `MERGE` (upsert) selalu gagal: "Access Denied: Cannot update
# columns [...]".
# 2. Berlaku untuk tabel apa pun, termasuk tabel sementara `<nama>_tmp` milik
# Sling. Tabel itu tidak akan cocok dengan grant biasa karena policy hanya
# men-strip sufiks milik dbt (`__dbt_tmp`, `__dbt_backup`).
#
# Efek: tabel baru di bronze tidak lagi perlu baris grant di data.json.
# Mempersempit kembali: hapus blok ini, atau tambahkan syarat daftar nama tabel
# eksplisit. Layer lain (silver/gold) TIDAK terpengaruh — tetap least-privilege.
allow if {
res := input.action.resource
res.table.schemaName in {"bronze", "silver", "gold"}
some r in effective_roles
r == "data-engineer"
}
deploy/config/openldap/seed.ldif
+93
View File
@@ -0,0 +1,93 @@
# =============================================================================
# Seed OpenLDAP (osixia) — deploy-merged
# ou=users → admin, engineer, analyst, viewer
# ou=groups → data-admin, data-engineer, data-analyst, data-viewer
#
# Sumber kebenaran GRUP untuk:
# - Trino → LDAP group provider (Fase 3)
# - StarRocks → LDAP group provider (Fase 4)
#
# Catatan: user juga terdaftar lokal di realm Keycloak (autentikasi OIDC).
# LDAP menyediakan pemetaan user → grup. Password LDAP di sini adalah nilai
# dev (selaras dengan realm-export.json) untuk kemungkinan bind per-user.
# =============================================================================
dn: ou=users,dc=sicabe,dc=local
objectClass: organizationalUnit
ou: users
dn: ou=groups,dc=sicabe,dc=local
objectClass: organizationalUnit
ou: groups
# ─── Users ──────────────────────────────────────────────────────────────────
dn: uid=admin,ou=users,dc=sicabe,dc=local
objectClass: inetOrgPerson
objectClass: posixAccount
uid: admin
cn: Administrator
sn: SiCABE
mail: admin@sicabe.local
userPassword: admin123
uidNumber: 1000
gidNumber: 1000
homeDirectory: /home/admin
dn: uid=engineer,ou=users,dc=sicabe,dc=local
objectClass: inetOrgPerson
objectClass: posixAccount
uid: engineer
cn: Data Engineer
sn: SiCABE
mail: engineer@sicabe.local
userPassword: engineer123
uidNumber: 1001
gidNumber: 1001
homeDirectory: /home/engineer
dn: uid=analyst,ou=users,dc=sicabe,dc=local
objectClass: inetOrgPerson
objectClass: posixAccount
uid: analyst
cn: Data Analyst
sn: SiCABE
mail: analyst@sicabe.local
userPassword: analyst123
uidNumber: 1002
gidNumber: 1002
homeDirectory: /home/analyst
dn: uid=viewer,ou=users,dc=sicabe,dc=local
objectClass: inetOrgPerson
objectClass: posixAccount
uid: viewer
cn: Viewer
sn: SiCABE
mail: viewer@sicabe.local
userPassword: viewer123
uidNumber: 1003
gidNumber: 1003
homeDirectory: /home/viewer
# ─── Groups ─────────────────────────────────────────────────────────────────
dn: cn=data-admin,ou=groups,dc=sicabe,dc=local
objectClass: groupOfNames
cn: data-admin
member: uid=admin,ou=users,dc=sicabe,dc=local
dn: cn=data-engineer,ou=groups,dc=sicabe,dc=local
objectClass: groupOfNames
cn: data-engineer
member: uid=engineer,ou=users,dc=sicabe,dc=local
dn: cn=data-analyst,ou=groups,dc=sicabe,dc=local
objectClass: groupOfNames
cn: data-analyst
member: uid=analyst,ou=users,dc=sicabe,dc=local
dn: cn=data-viewer,ou=groups,dc=sicabe,dc=local
objectClass: groupOfNames
cn: data-viewer
member: uid=viewer,ou=users,dc=sicabe,dc=local
deploy/config/openmetadata/ingest-dbt.yaml
+55
View File
@@ -0,0 +1,55 @@
# =============================================================================
# eBdesk SiCABE — Ingestion agent OpenMetadata: dbt (LINEAGE bronze→silver→gold)
# =============================================================================
# Mengapa file ini ada:
# Ingestion metadata (`ingest-trino.yaml`) hanya mengambil tabel & kolom — TIDAK
# menghasilkan lineage, karena lineage berasal dari definisi transformasi, bukan
# dari katalog. Transformasi bronze→silver→gold dijalankan oleh dbt, sehingga
# lineage-nya diambil dari ARTEFAK dbt:
# target/manifest.json (grafik model: ref()/source() → relasi antar tabel)
# target/catalog.json (skema kolom per model → lineage antar KOLOM)
# target/run_results.json(hasil eksekusi/test)
#
# Artefak dibuat di dalam container Airflow (project di /home/airflow/dbt/sicabe):
# dbt docs generate --profile trino --target prod
#
# ⚠️ Catatan: `target/` berada di layer container (bukan bind-mount), jadi artefak
# HILANG bila container Airflow di-recreate → jalankan `dbt docs generate` lagi
# sebelum ingestion dbt.
#
# Jalankan (butuh OM_JWT_TOKEN):
# ./scripts/ingest-openmetadata.sh --workflow config/openmetadata/ingest-dbt.yaml
# =============================================================================
source:
type: dbt
# Service database tempat model dbt ditautkan (harus sama dengan service Trino).
serviceName: trino_sicabe
sourceConfig:
config:
# Enum-nya huruf besar: DbtConfigType.DBT = 'DBT' (bukan 'dbt').
type: DBT
dbtConfigSource:
dbtConfigType: local
# Artefak dbt dibaca dari DBT_TARGET_PATH (lihat docker-compose.yml), BUKAN dari
# <project>/target di dalam bind-mount repo — bind-mount itu tidak bisa ditulis
# container (uid host vs uid 50000), sehingga artefaknya memang tidak pernah ada
# di sana dan ingestion lineage selalu menghasilkan 0 relasi.
dbtManifestFilePath: /tmp/dbt/target/manifest.json
dbtCatalogFilePath: /tmp/dbt/target/catalog.json
dbtRunResultsFilePath: /tmp/dbt/target/run_results.json
dbtUpdateDescriptions: true
dbtUpdateOwners: true
includeTags: false
searchAcrossDatabases: false
sink:
type: metadata-rest
config: {}
workflowConfig:
loggerLevel: INFO
openMetadataServerConfig:
hostPort: http://openmetadata-server:8585/api
authProvider: openmetadata
securityConfig:
jwtToken: ${OM_JWT_TOKEN}
deploy/config/openmetadata/ingest-trino.yaml
+71
View File
@@ -0,0 +1,71 @@
# =============================================================================
# eBdesk SiCABE — Ingestion agent OpenMetadata: Trino (Iceberg via Nessie)
# =============================================================================
# Menjelaskan mengapa file ini ada:
# Membuat SERVICE di OpenMetadata hanya menyimpan konfigurasi koneksi. Metadata
# sesungguhnya (database → schema → tabel → kolom) baru diambil oleh AGENT
# (ingestion pipeline). Halaman Explore membaca search index, dan index itu
# hanya terisi setelah agent berjalan. Tanpa file ini, service `trino_sicabe`
# ada tetapi Explore kosong.
#
# Catatan skema (penting, berbeda dari skema API):
# - di API OM 1.13.1 field TLS 'verifySSL' DITOLAK; di Ingestion Framework
# yang benar adalah `verify` (string), di sini "false" karena sertifikat Trino
# self-signed.
# - `username` wajib di level config; `authType` memakai BasicAuth (username+password).
# - Identitas agent = `svc_om` dengan role `data-catalog`: baca-saja
# (select + kolom "*" untuk SEMUA tabel) sehingga katalog Explore LENGKAP,
# namun TANPA hak write/ddl. (Sebelumnya identitas ini memakai `data-analyst`
# sehingga katalog hanya sebagian.)
#
# Jalankan (butuh TRINO_SVC_OM_PASSWORD & OM_JWT_TOKEN di environment):
# metadata ingest -c ingest-trino.yaml
# =============================================================================
source:
type: trino
serviceName: trino_sicabe
serviceConnection:
config:
type: Trino
scheme: trino
hostPort: trino-coordinator:8443
username: svc_om
# BasicAuth hanya berisi `password` (username ada di level config) —
# sesuai contoh resmi 'metadata ingest' untuk Trino.
authType:
password: ${TRINO_SVC_OM_PASSWORD}
catalog: iceberg
# `verify` adalah path CA bundle (string) — OM hanya meneruskannya bila
# truthy, dan klien Trino memperlakukannya sebagai path. Memakai sertifikat
# Trino (diekspor scripts/generate-trino-keystore.sh, di-mount ke
# /etc/ssl/trino) membuat TLS DIVERIFIKASI, bukan dimatikan.
verify: /etc/ssl/trino/trino.crt
supportsMetadataExtraction: true
supportsProfiler: true
supportsQueryComment: true
# Batasi ke namespace yang dikelola (bronze/silver/gold). Karena ini
# pattern di level KONEKSI, filter schema juga tersedia di sourceConfig.
schemaFilterPattern:
includes:
- bronze
- silver
- gold
sourceConfig:
config:
type: DatabaseMetadata
markDeletedTables: false
includeTables: true
includeViews: true
includeTags: false
sink:
type: metadata-rest
config: {}
workflowConfig:
loggerLevel: INFO
openMetadataServerConfig:
hostPort: http://openmetadata-server:8585/api
authProvider: openmetadata
securityConfig:
jwtToken: ${OM_JWT_TOKEN}
deploy/config/postgres/init/01-create-dbs.sh
Executable
+95
View File
@@ -0,0 +1,95 @@
#!/bin/bash
# =============================================================================
# eBdesk SiCABE — PostgreSQL Multi-Database Init
# =============================================================================
# Script ini dijalankan OTOMATIS oleh PostgreSQL Docker entrypoint pada
# FIRST STARTUP saja (saat volume data masih kosong).
#
# Membuat database dan user terpisah untuk setiap komponen SiCABE:
# - keycloak → Governance (IAM)
# - nessie → Storage (Metadata Catalog: Iceberg REST + version store)
# - airflow → Ingestion (Orchestrator)
# - openmetadata_db → Governance (Metadata Platform)
#
# Variable environment yang harus tersedia (di-pass ke container postgres):
# POSTGRES_USER, POSTGRES_PASSWORD (superuser — sudah built-in)
# KEYCLOAK_DB_PASSWORD, NESSIE_DB_PASSWORD
# AIRFLOW_DB_PASSWORD, OPENMETADATA_DB_PASSWORD
# =============================================================================
set -e
# ─── Fungsi: Buat database, user, dan grant schema permissions ────────────
create_service_db() {
local db_name="$1"
local db_user="$2"
local db_pass="$3"
echo "[SiCABE] Creating database '${db_name}' with user '${db_user}'..."
# Create user (if not exists) — connect to POSTGRES_DB (keycloak)
psql -v ON_ERROR_STOP=1 --username "${POSTGRES_USER}" --dbname "${POSTGRES_DB}" <<-EOSQL
DO \$\$
BEGIN
IF NOT EXISTS (SELECT FROM pg_catalog.pg_roles WHERE rolname = '${db_user}') THEN
CREATE ROLE ${db_user} LOGIN PASSWORD '${db_pass}';
END IF;
END
\$\$;
EOSQL
# Create database (if not exists) with owner
psql -v ON_ERROR_STOP=1 --username "${POSTGRES_USER}" --dbname "${POSTGRES_DB}" <<-EOSQL
SELECT 'CREATE DATABASE ${db_name} OWNER ${db_user}'
WHERE NOT EXISTS (SELECT FROM pg_database WHERE datname = '${db_name}')\gexec
EOSQL
# Grant all privileges on the database
psql -v ON_ERROR_STOP=1 --username "${POSTGRES_USER}" --dbname "${db_name}" <<-EOSQL
GRANT ALL ON SCHEMA public TO ${db_user};
GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO ${db_user};
GRANT ALL PRIVILEGES ON ALL SEQUENCES IN SCHEMA public TO ${db_user};
ALTER SCHEMA public OWNER TO ${db_user};
EOSQL
echo "[SiCABE] ✓ Database '${db_name}' siap."
}
# ─── Validasi env vars ─────────────────────────────────────────────────────
if [[ -z "${POSTGRES_USER}" ]]; then
echo "[SiCABE] ERROR: POSTGRES_USER tidak terdefinisi!"
exit 1
fi
# ─── Buat database & user untuk setiap service ─────────────────────────────
# Urutan: sesuai fase deployment
# Phase 1 — Governance (IAM)
create_service_db "keycloak" "keycloak" "${KEYCLOAK_DB_PASSWORD}"
# Phase 2 — Storage (Metadata Catalog)
create_service_db "nessie" "nessie" "${NESSIE_DB_PASSWORD}"
# Nessie (JDBC2 version store) membutuhkan schema dibuat lebih dulu.
# Tabel-tabel Nessie akan di-auto-create oleh server saat startup.
psql -v ON_ERROR_STOP=1 --username "${POSTGRES_USER}" --dbname "nessie" <<-EOSQL
CREATE SCHEMA IF NOT EXISTS nessie AUTHORIZATION nessie;
EOSQL
# Phase 5 — Ingestion (Orchestrator)
create_service_db "airflow" "airflow" "${AIRFLOW_DB_PASSWORD}"
# Phase 6 — Governance (Metadata Platform)
create_service_db "openmetadata_db" "openmetadata" "${OPENMETADATA_DB_PASSWORD}"
# Pipeline — database SUMBER contoh untuk DAG `sicabe_batch_daily`
# (task `ingest_to_bronze` membaca public.orders via Sling). Dimiliki superuser
# (POSTGRES_USER) sehingga tidak perlu role tambahan. Tabel + 6 baris contoh
# diisi oleh `scripts/seed-source-postgres.sh` (dipanggil bootstrap step 6).
psql -v ON_ERROR_STOP=1 --username "${POSTGRES_USER}" --dbname "${POSTGRES_DB}" <<-EOSQL
SELECT 'CREATE DATABASE sicabe_data OWNER ${POSTGRES_USER}'
WHERE NOT EXISTS (SELECT FROM pg_database WHERE datname = 'sicabe_data')\gexec
EOSQL
echo "[SiCABE] ✓ Database 'sicabe_data' siap (sumber contoh pipeline)."
echo "[SiCABE] ✅ Semua database dan user berhasil dibuat."
deploy/config/prometheus/prometheus.yml
+91
View File
@@ -0,0 +1,91 @@
# =============================================================================
# eBdesk SiCABE — Konfigurasi Prometheus (Fase 6: Observability)
# =============================================================================
# Mengapa Prometheus ini ada:
# Dashboard Grafana punya panel bertipe metrik (mis. minio_bucket_usage_object_total).
# Datasource Grafana bertipe `prometheus` HARUS menunjuk SERVER Prometheus — versi
# sebelumnya menunjuk `http://minio:9000` (API S3 MinIO) sehingga selalu
# "400 Bad Request: error querying the Prometheus API". Prometheus ini yang
# meng-scrape metrik, lalu Grafana membacanya.
#
# Target yang tersedia di stack ini:
# - prometheus : self-scrape
# - minio : metrik publik (MINIO_PROMETHEUS_AUTH_TYPE=public di compose)
# - starrocks-fe : FE mengekspos /metrics pada port HTTP 8030
#
# Sudah tersedia (M2/M3 observabilitas — lihat reviews/sicabe-19-observability-metrics-plan.md):
# - airflow : Airflow 3 mengirim metrik lewat StatsD → statsd-exporter:9102 (job `airflow`)
# - trino : JMX coordinator (RMI :9080) → sidecar trino-jmx-exporter:9404 (job `trino`)
# - node : node_exporter di SETIAP host (:9100). node_exporter adalah PRA-SYARAT HOST
# (di luar installer); daftar host dibaca dari `targets/node.yml` via file_sd
# (digenerate dari cluster.yml bila ada — lihat lib/cluster/generate.py).
# =============================================================================
global:
scrape_interval: 30s
evaluation_interval: 30s
scrape_configs:
- job_name: prometheus
static_configs:
- targets: ["localhost:9090"]
# MinIO memecah metrik per endpoint — satu job per metrics_path.
# /cluster : 197 metrik (server/cluster)
# /bucket : 140 metrik — termasuk 'minio_bucket_usage_object_total' yang
# dipakai panel "Storage Usage" di dashboard Grafana. Metrik itu
# TIDAK ada di /cluster, sehingga tanpa job ini panel menampilkan
# "No data" walaupun target-nya 'up'.
- job_name: minio
metrics_path: /minio/v2/metrics/cluster
scheme: http
static_configs:
- targets: ["minio:9000"]
- job_name: minio-bucket
metrics_path: /minio/v2/metrics/bucket
scheme: http
static_configs:
- targets: ["minio:9000"]
- job_name: starrocks-fe
metrics_path: /metrics
scheme: http
static_configs:
- targets: ["starrocks-fe:8030"]
# Airflow → StatsD → statsd-exporter (M2 observabilitas).
# Metrik Airflow (dagrun duration, task duration, scheduler heartbeat) muncul di
# :9102 setelah AIRFLOW__METRICS__STATSD_ON diaktifkan pada service Airflow.
- job_name: airflow
metrics_path: /metrics
scheme: http
static_configs:
- targets: ["statsd-exporter:9102"]
# Trino → JMX (RMI :9080) → sidecar trino-jmx-exporter:9404 (M3 observabilitas).
# Metrik saturasi coordinator (running/queued/failed query) + JVM.
- job_name: trino
metrics_path: /metrics
scheme: http
static_configs:
- targets: ["trino-jmx-exporter:9404"]
# node_exporter di setiap host (:9100) — PRA-SYARAT HOST (bukan service compose).
# Daftar host dibaca dari file ini dan bisa diubah TANPA restart Prometheus
# (file_sd memantau perubahan berkas; refresh_interval hanya cadangan).
#
# - single-host : targets/node.yml default = "host.docker.internal:9100"
# - multi-host : digenerate dari cluster.yml oleh `./install.sh --generate`
# (semua host, alamat:9100, label host=<nama>)
#
# Label `host` dari berkas target dipromosikan ke semua seri; relabel di bawah
# menyalinnya ke `instance` supaya panel menampilkan NAMA host, bukan alamat.
- job_name: node
metrics_path: /metrics
scheme: http
file_sd_configs:
- files: ["/etc/prometheus/targets/node.yml"]
refresh_interval: 30s
relabel_configs:
- source_labels: [host]
target_label: instance
deploy/config/prometheus/targets/node.yml
+36
View File
@@ -0,0 +1,36 @@
# =============================================================================
# node_exporter targets — job `node` (Prometheus file_sd)
# =============================================================================
# node_exporter adalah PRA-SYARAT HOST: dipasang di SETIAP host (port 9100),
# DI LUAR installer SiCABE. Berkas ini HANYA memberi tahu Prometheus host mana
# yang harus di-scrape.
#
# FORMAT (file_sd): daftar objek { targets: [...], labels: {...} }.
# - `targets` : "<alamat-host>:9100"
# - `labels.host` : NAMA host — disalin ke label `instance` oleh relabel_configs
# di prometheus.yml, sehingga panel menampilkan nama (bukan IP).
#
# CARA PAKAI
# - Single-host (default): biarkan apa adanya. `host.docker.internal` menunjuk
# Docker host lewat `extra_hosts: host-gateway` (lihat docker-compose.yml).
# - Multi-host: TIDAK perlu disunting tangan bila memakai cluster.yml —
# `./install.sh --generate` menulis ulang berkas ini di bundle host Prometheus
# dengan SELURUH host (alamat:9100, label host=<nama>).
# - Tanpa cluster.yml tapi ada host tambahan: tambahkan satu entri per host
# (contoh di bawah). Perubahan terbaca otomatis tanpa restart Prometheus.
# =============================================================================
- targets: ["host.docker.internal:9100"]
labels:
host: local
- targets: ["100.123.123.71:9100"]
labels:
host: darkworld
# Contoh penambahan host (hapus komentar):
# - targets: ["10.20.0.11:9100"]
# labels:
# host: compute
# - targets: ["10.20.0.12:9100"]
# labels:
# host: state
deploy/config/starrocks/cn.conf
+32
View File
@@ -0,0 +1,32 @@
# =============================================================================
# StarRocks — Compute Node (CN) Configuration — SHARED-DATA
# =============================================================================
# CN TIDAK memakai parameter run_mode (hanya FE). Data utama tersimpan di
# object storage; CN hanya menyimpan cache lokal di storage_root_path.
# Start via ./bin/start_cn.sh (bukan start_be.sh).
#
# Diregistrasi ke FE dengan: ALTER SYSTEM ADD COMPUTE NODE "starrocks-cn:9050";
# Referensi: docs.starrocks.io/docs/deployment/deploy_shared_data_manually/
# =============================================================================
sys_log_level = INFO
# Thrift server port utk koneksi FE
be_port = 9060
# HTTP server port
be_http_port = 8040
# Heartbeat port utk koneksi FE (diregistrasi sbg <host>:9050)
heartbeat_service_port = 9050
# Brpc port komunikasi internal
brpc_port = 8060
# Starlet port — komunikasi CN ↔ object storage (shared-data)
starlet_port = 9070
# Direktori data cache lokal (shared-data). Volume cn-cache dipasang di sini.
storage_root_path = /opt/starrocks/cn/cache
# Cluster configuration (subnet network deploy-merged)
priority_networks = 172.28.0.0/16
deploy/config/starrocks/fe.conf
+42
View File
@@ -0,0 +1,42 @@
# =============================================================================
# StarRocks — Frontend (FE) Configuration — SHARED-DATA
# =============================================================================
# Mode shared-data: data tabel internal disimpan di object storage S3/MinIO
# (bukan di disk lokal). Dua baris di bawah WAJIB di-set sebelum first start
# dan TIDAK bisa diubah via SQL.
#
# Penyimpanan diatur via storage volume SQL:
# CREATE STORAGE VOLUME s3_volume TYPE=S3 ...; SET s3_volume AS DEFAULT STORAGE VOLUME;
#
# External catalog Iceberg (via Nessie) tetap dikonfigurasi via SQL.
# Referensi: docs.starrocks.io/docs/deployment/deploy_shared_data_manually/
# =============================================================================
# --- Mode shared-data (WAJIB, sebelum first start) ---
run_mode = shared_data
cloud_native_storage_type = S3
# Meta data directory
meta_dir = /opt/starrocks/fe/data
# HTTP server port (redirect_url OIDC: http://<fe>:8030/api/oauth2)
http_port = 8030
# Thrift server port for CN connections
rpc_port = 9020
# MySQL server port for client connections
query_port = 9030
# Edit log port for FE cluster communication
edit_log_port = 9010
# Port internal FE untuk shared-data metadata (default 6090)
cloud_native_meta_port = 6090
# JVM configuration
JAVA_HOME = /usr/lib/jvm/java-17-openjdk-amd64
JAVA_OPTS = -Xms512m -Xmx1024m -XX:+UseG1GC
# Cluster configuration (subnet network deploy-merged)
priority_networks = 172.28.0.0/16
deploy/config/traefik/dynamic.yml
+14
View File
@@ -0,0 +1,14 @@
# Konfigurasi dinamis Traefik (provider file).
#
# serversTransports TIDAK dapat didefinisikan lewat label Docker — percobaan memakai
# label menghasilkan error runtime:
# "error building proxy for server URL https://...: servers transport not found ..."
# Karena itu didefinisikan di berkas ini.
#
# trino-insecure: Trino internal memakai HTTPS (http-server.allow-insecure-over-http=false)
# dengan sertifikat self-signed untuk nama container, jadi Traefik perlu melewati
# verifikasi HANYA untuk backend Trino — bukan global.
http:
serversTransports:
trino-insecure:
insecureSkipVerify: true
deploy/config/trino/access-control.properties
+17
View File
@@ -0,0 +1,17 @@
# =============================================================================
# Trino — Access Control via OPA (built-in, deny-by-default)
# =============================================================================
# access-control.name=opa (built-in Trino). Policy Rego di OPA
# (config/opa/policies/trino.rego) menerima input dari Trino dan memutuskan
# per-objek/per-kolom secara LOKAL (ACL di config/opa/data.json).
#
# Grup user berasal dari LDAP group provider (lihat group-provider.properties),
# lalu dicocokkan dengan data.roles.<grup> di data.json.
#
# OPA: http://opa:8181/v1/data/trino/allow
# =============================================================================
access-control.name=opa
opa.policy.uri=http://opa:8181/v1/data/trino/allow
# Selama debugging biarkan true; matikan saat sudah stabil (log request OPA).
opa.log-requests=true
deploy/config/trino/catalog/iceberg.properties
+30
View File
@@ -0,0 +1,30 @@
# =============================================================================
# Trino Iceberg Catalog — Project Nessie (Nessie REST API v2)
# =============================================================================
# Catalog ini menghubungkan Trino ke tabel Iceberg yang dikelola Nessie
# (versioning git-like). Data file + metadata disimpan di MinIO (S3-compatible).
#
# Catatan: Nessie belum mendukung Iceberg spec v3 → selalu format-version=2
# (default Trino = 2, dibiarkan).
# =============================================================================
# --- Connector ---
connector.name=iceberg
# --- Catalog Type: Nessie ---
iceberg.catalog.type=nessie
iceberg.nessie-catalog.uri=${ENV:NESSIE_URL}/api/v2
iceberg.nessie-catalog.ref=main
iceberg.nessie-catalog.default-warehouse-dir=s3://${ENV:MINIO_BUCKET_ICEBERG_DATA}/warehouse
# --- S3 Native Filesystem (MinIO / S3 existing) ---
fs.s3.enabled=true
s3.endpoint=${ENV:S3_ENDPOINT}
s3.path-style-access=true
s3.aws-access-key=${ENV:S3_ACCESS_KEY}
s3.aws-secret-key=${ENV:S3_SECRET_KEY}
s3.region=${ENV:S3_REGION}
# --- Table Defaults ---
iceberg.file-format=PARQUET
iceberg.compression-codec=ZSTD
deploy/config/trino/certs/trino.crt
+20
View File
@@ -0,0 +1,20 @@
-----BEGIN CERTIFICATE-----
MIIDVTCCAj2gAwIBAgIUETRg8AYPjCnKkFChXiQiP62JG9YwDQYJKoZIhvcNAQEL
BQAwEDEOMAwGA1UEAwwFdHJpbm8wHhcNMjYwOTI4MTIwOTMxWhcNMjgxMjMxMTIw
OTMxWjAQMQ4wDAYDVQQDDAV0cmlubzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCC
AQoCggEBAMyEX/NwRqEuT3SweBZeHcUxFKgeFEY89trx9D2nkuP7OMiHRFyeKfuL
Kmyyyta9QQDU1EU1cHyhgHwhXLzFxJ6QnlLgSxOP/hTQlF6qL2dvMgKDzBmo51/I
e6nwNYlYC9wzOPLNVdqZkqraSk2xmnnMB4fM9L+xisZJOQ7VpprzTNIXigus7q8M
CsWPXu+dr9XJSKYeqOvV5jIQGhSE7yyb3zC2nHG/qQ3YiBZc7OMtkUti+UT2/oGK
tErYxBj8uCtsqLGCFjpGoioOIFAUmqriBzSrSrOwzkkeF3mI5cok4P4mBmwQcRZk
3TBrQW8HrCB8oVq3kNgsuM8dRJZ3NzsCAwEAAaOBpjCBozAdBgNVHQ4EFgQUvcvH
CUrnQiy1yYFCsrrcNcYukmswHwYDVR0jBBgwFoAUvcvHCUrnQiy1yYFCsrrcNcYu
kmswDwYDVR0TAQH/BAUwAwEB/zBQBgNVHREESTBHggV0cmlub4IJbG9jYWxob3N0
ggxzaWNhYmUtdHJpbm+CEXRyaW5vLWNvb3JkaW5hdG9yggx0cmluby13b3JrZXKH
BH8AAAEwDQYJKoZIhvcNAQELBQADggEBAMrDCPDqEN/d+LhQ3bNoiVBdjoLemML5
G+Torrv7VIcD2VDdUbWQ7WfzXmXS0WTuzVbqD5I6Fus1NK2GFKB0q2kP9whOrmqu
lE6Sv2b1g38H3PnjzWQbY34lNTRzJr/PmxvxgTs1GIWFu4fcd7GsNocF3ZciLf/5
4Ky51U8JG3ukzPjkFSdAY03aZ2d2Q2VqZkmLJyHcjJDH4+u3zveEbUxSSTgM9Kzx
NxaYiqXeUzd65Y+9ALsECuDMQGLkMl9acmgJIO+vH/3sfM9WiDksSN9WJhR3R2UX
B5zp85Reh+ehobxQsbIZU+IXR6cEy8SfgQNzxlFVyOQ1/V4MgFckFNw=
-----END CERTIFICATE-----
deploy/config/trino/config-coordinator.properties
+67
View File
@@ -0,0 +1,67 @@
# =============================================================================
# Trino COORDINATOR — config.properties
# -----------------------------------------------------------------------------
# Dipisah dari worker sejak awal (tier R11: coord). Coordinator melayani KLIEN
# (HTTPS 8443 + OAuth2/Basic) dan mengatur penjadwalan; worker yang mengeksekusi
# task (node-scheduler.include-coordinator=false).
#
# Klien TIDAK melihat worker; worker mendaftar lewat discovery.uri di bawah.
# Perubahan dari konfigurasi single-node sebelumnya:
# node-scheduler.include-coordinator : true → false
# discovery.uri : http://localhost:8080 → http://trino-coordinator:8080
# =============================================================================
coordinator=true
node-scheduler.include-coordinator=false
http-server.http.port=8080
# HTTPS untuk Web UI OAuth2 — OAuth2 Web UI WAJIB secure request (TLS),
# jika tidak Trino mengalihkan /ui/ ke /ui/static/disabled.html.
# HTTP (8080) TIDAK lagi melayani tanpa kredensial (allow-insecure-over-http=false);
# klien service/pipeline lewat nginx proxy → HTTPS.
http-server.https.enabled=true
http-server.https.port=8443
http-server.https.keystore.path=/etc/trino/keystore.p12
http-server.https.keystore.key=trino_ssl_dev
# HTTP 8080 tetap dipakai untuk komunikasi INTERNAL coordinator↔worker
# (worker mendaftar ke coordinator lewat endpoint ini).
discovery.uri=http://trino-coordinator:8080
catalog.management=${ENV:CATALOG_MANAGEMENT}
# Authentication: hybrid — OAuth2 (end user, Keycloak SSO) + password/Basic
# (service account). Keduanya berlaku di HTTPS (8443). HTTP polos DIMATIKAN
# (allow-insecure-over-http=false) → klien service lewat nginx proxy (HTTP→HTTPS).
http-server.authentication.type=oauth2,password
http-server.authentication.allow-insecure-over-http=false
# Di belakang reverse proxy TLS (Traefik), Trino harus memercayai X-Forwarded-* supaya
# skema publik (https) dan alamat klien terbaca benar — termasuk untuk redirect OAuth.
# Nilainya dari env: diisi `true` oleh generator saat tls.mode != none, `false` bila tidak.
# Trino GAGAL START bila variabel ini tidak ada, jadi compose selalu mendefinisikannya.
http-server.process-forwarded=${ENV:TRINO_PROCESS_FORWARDED}
# Issuer = URL PUBLIK Keycloak — wajib SAMA PERSIS dengan iss claim JWT.
# Diambil dari KEYCLOAK_PUBLIC_URL (bukan ditulis tetap): nilainya berubah antara
# single-host (localhost) dan multi-host tanpa TLS (http://<alamat-host-control>:12080).
# Kalau di-hardcode, issuer di klaster multi-host tidak cocok dan token ditolak.
http-server.authentication.oauth2.issuer=${ENV:KEYCLOAK_PUBLIC_URL}/realms/sicabe
# Matikan OIDC discovery — Trino tidak bisa mengakses URL publik dari dalam container.
# Endpoint di-set manual: browser pakai URL publik, server-ke-server pakai keycloak:8080.
http-server.authentication.oauth2.oidc.discovery=false
# Endpoint browser-facing (auth redirect) — harus URL publik juga
http-server.authentication.oauth2.auth-url=${ENV:KEYCLOAK_PUBLIC_URL}/realms/sicabe/protocol/openid-connect/auth
# Endpoint server-side (token exchange & JWKS) — reachable dari container
http-server.authentication.oauth2.token-url=http://keycloak:${ENV:KEYCLOAK_INTERNAL_PORT}/realms/sicabe/protocol/openid-connect/token
http-server.authentication.oauth2.jwks-url=http://keycloak:${ENV:KEYCLOAK_INTERNAL_PORT}/realms/sicabe/protocol/openid-connect/certs
http-server.authentication.oauth2.client-id=trino
http-server.authentication.oauth2.client-secret=${ENV:TRINO_CLIENT_SECRET}
# Username Trino = preferred_username (bukan 'sub') agar cocok dengan uid LDAP
# pada LDAP group provider.
http-server.authentication.oauth2.principal-field=preferred_username
# Token Keycloak client 'trino' membawa aud='account'; trust audience ini.
http-server.authentication.oauth2.additional-audiences=account
# Web UI: aktifkan OAuth2 untuk UI. Web UI OAuth2 WAJIB HTTPS (lihat blok https
# di atas) — bila request tidak secure, Trino mengalihkan /ui/ ke disabled.html.
# allow-insecure-over-http hanya berlaku untuk REST API/JDBC, BUKAN Web UI.
web-ui.authentication.type=oauth2
# Shared secret untuk internal communication (required saat auth enabled)
# WAJIB identik dengan worker — tanpa ini worker ditolak coordinator.
internal-communication.shared-secret=trino-internal-secret-dev-only
deploy/config/trino/config-worker.properties
+18
View File
@@ -0,0 +1,18 @@
# =============================================================================
# Trino WORKER — config.properties
# -----------------------------------------------------------------------------
# Dipisah dari coordinator sejak awal (tier R11: engine). Worker TIDAK melayani
# klien: tidak ada HTTPS, tidak ada autentikasi OAuth2/Basic, tidak ada port yang
# di-expose ke host. Worker hanya:
# 1. mendaftar & menerima task dari coordinator lewat HTTP 8080,
# 2. membaca katalog yang sama (etc/catalog/*.properties).
#
# `internal-communication.shared-secret` WAJIB identik dengan coordinator —
# inilah yang mengautentikasi worker (bukan password/OAuth2 klien).
# =============================================================================
coordinator=false
http-server.http.port=8080
# Daftar ke coordinator (bukan localhost — worker bukan coordinator-nya sendiri).
discovery.uri=http://trino-coordinator:8080
internal-communication.shared-secret=trino-internal-secret-dev-only
deploy/config/trino/group-provider.properties
+28
View File
@@ -0,0 +1,28 @@
# =============================================================================
# Trino — LDAP Group Provider
# =============================================================================
# Grup user di-resolve dari OpenLDAP (source of truth grup). User Trino
# = preferred_username dari Keycloak (lihat config.properties principal-field).
# Nama grup yang dikembalikan = cn (mis. data-admin, data-engineer, ...),
# lalu dicocokkan dengan data.roles.<grup> di config/opa/data.json.
# =============================================================================
group-provider.name=ldap
# Alamat LDAP dari environment (LDAP_URL) supaya bisa diarahkan ke LDAP EXISTING.
# Default compose: ldap://openldap:389 (container lokal).
ldap.url=${ENV:LDAP_URL}
ldap.allow-insecure=true
# DN akun BIND (bukan harus admin): boleh akun read-only dgn hak baca ke ou=users/ou=groups.
# Diarahkan dari .env (LDAP_ADMIN_DN) supaya LDAP EXISTING dgn bind DN berbeda bisa dipakai.
# Env-nya WAJIB diteruskan ke container (lihat docker-compose.yml trino-coordinator).
ldap.admin-user=${ENV:LDAP_ADMIN_DN}
ldap.admin-password=${ENV:LDAP_ADMIN_PASSWORD}
ldap.user-base-dn=ou=users,dc=sicabe,dc=local
ldap.user-search-filter=(uid={0})
ldap.group-name-attribute=cn
# Filtering mode: cari group di ou=groups yg member-nya = DN user
ldap.use-group-filter=true
ldap.group-base-dn=ou=groups,dc=sicabe,dc=local
ldap.group-search-member-attribute=member
deploy/config/trino/jmx-exporter.yml
+31
View File
@@ -0,0 +1,31 @@
# =============================================================================
# eBdesk SiCABE — konfigurasi sidecar `trino-jmx-exporter` (M3 observabilitas)
# =============================================================================
# Dibaca `jmx_prometheus_standalone.jar` (image bitnami/jmx-exporter):
# java -jar jmx_prometheus_standalone.jar 9404 /config/config.yml
#
# hostPort : target JMX coordinator via RMI — HARUS sama dengan port &
# java.rmi.server.hostname di config/trino/jvm-coordinator.config.
#
# includeObjectNames : PEMBATAS CARDINALITY. Tanpa ini (`rules: ".*"` saja)
# jmx-exporter memaparkan SEMUA MBean (~9.700 metrik) — termasuk
# io.trino.filesystem.s3, io.airlift.*, org.eclipse.jetty.*, java.lang.*,
# trino.server, trino.security, dst yang tidak dipakai.
# Di sini dibatasi ke domain EKSEKUSI Trino yang DIPAKAI saja:
# trino.execution:* → QueryManager, QueryExecution, TaskManager
# trino.execution.executor.* → task executor (timesharing/dedicated)
# trino.execution.resourcegroups:* → resource group
# SENGAJA DIKECUALIKAN: trino.execution.scheduler[.faulttolerant]*
# (≈1.614 metrik statistik penjadwalan FT — stack ini tidak memakai
# fault-tolerant execution). Tambahkan kembali bila FT dipakai.
# (Konsekuensi: metrik JVM `java_lang_memory_*` juga tidak ada.)
---
hostPort: trino-coordinator:9080
lowercaseOutputName: true
lowercaseOutputLabelNames: true
includeObjectNames:
- "trino.execution:*"
- "trino.execution.executor.*:*"
- "trino.execution.resourcegroups:*"
rules:
- pattern: ".*"
deploy/config/trino/jvm-coordinator.config
+32
View File
@@ -0,0 +1,32 @@
# =============================================================================
# Trino — JVM Tuning
# Referensi arsitektur: bab 4.3.1.7
# =============================================================================
-server
# Heap COORDINATOR: ringan (penjadwalan + auth); task dieksekusi worker.
-Xmx512m
-XX:+UseG1GC
-XX:G1HeapRegionSize=32M
-XX:+UseGCOverheadLimit
-XX:+ExplicitGCInvokesConcurrent
-XX:+HeapDumpOnOutOfMemoryError
-XX:+ExitOnOutOfMemoryError
-XX:HeapDumpPath=/data/trino/heapdump
# Diperlukan oleh Trino 450 untuk Java < 22.0.2 (JDK-8329528)
-XX:+UnlockDiagnosticVMOptions
-XX:G1NumCollectionsKeepPinned=10000000
-Djdk.attach.allowAttachSelf=true
-Djava.security.egd=file:/dev/urandom
# ─── JMX remote (M3 observabilitas, Review 19) ───────────────────────────────
# Dibaca sidecar `trino-jmx-exporter` via RMI. Port 9080 hanya dalam jaringan
# docker `sicabe-network` (tidak ada `ports:` yang mengeksposnya ke host), karena
# itu authenticate=false dapat diterima di sini.
# ⚠️ `java.rmi.server.hostname` WAJIB = nama container (bukan default = ID container),
# agar stub RMI bisa diresolusi sidecar di jaringan yang sama.
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=9080
-Dcom.sun.management.jmxremote.rmi.port=9080
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.local.only=false
-Djava.rmi.server.hostname=trino-coordinator
deploy/config/trino/jvm-worker.config
+19
View File
@@ -0,0 +1,19 @@
# =============================================================================
# Trino — JVM Tuning
# Referensi arsitektur: bab 4.3.1.7
# =============================================================================
-server
# Heap WORKER: menampung eksekusi task (scan, join, agregasi).
-Xmx1G
-XX:+UseG1GC
-XX:G1HeapRegionSize=32M
-XX:+UseGCOverheadLimit
-XX:+ExplicitGCInvokesConcurrent
-XX:+HeapDumpOnOutOfMemoryError
-XX:+ExitOnOutOfMemoryError
-XX:HeapDumpPath=/data/trino/heapdump
# Diperlukan oleh Trino 450 untuk Java < 22.0.2 (JDK-8329528)
-XX:+UnlockDiagnosticVMOptions
-XX:G1NumCollectionsKeepPinned=10000000
-Djdk.attach.allowAttachSelf=true
-Djava.security.egd=file:/dev/urandom
deploy/config/trino/keystore.p12
BIN
View File
Binary file not shown.
deploy/config/trino/log.properties
+6
View File
@@ -0,0 +1,6 @@
# =============================================================================
# Trino — Logging Configuration
# =============================================================================
io.trino=INFO
io.trino.server.security=DEBUG
io.trino.plugin.iceberg=INFO
deploy/config/trino/password-authenticator.properties
+13
View File
@@ -0,0 +1,13 @@
# =============================================================================
# Trino — Password authenticator (HTTP Basic) untuk SERVICE ACCOUNT
# -----------------------------------------------------------------------------
# Dipakai bersama OAuth2 (lihat config.properties: type=oauth2,password):
# - End user : OAuth2 (Keycloak SSO) → HTTPS Web UI/JDBC
# - Service : HTTP Basic (file bcrypt) → lewat nginx proxy (HTTP→HTTPS)
#
# password.db di-generate oleh bootstrap.sh/install.sh dari .env.
# Pra-syarat: `htpasswd` tersedia di host. JANGAN commit (lihat .gitignore).
# =============================================================================
password-authenticator.name=file
file.password-file=/etc/trino/password.db
file.refresh-period=5s
deploy/config/trino/resource-groups.json
+68
View File
@@ -0,0 +1,68 @@
{
"rootGroups": [
{
"name": "global",
"softMemoryLimit": "80%",
"hardConcurrencyLimit": 100,
"maxQueued": 1000,
"schedulingPolicy": "weighted",
"jmxExport": true,
"subGroups": [
{
"name": "data_definition",
"softMemoryLimit": "10%",
"hardConcurrencyLimit": 5,
"maxQueued": 100,
"schedulingWeight": 1
},
{
"name": "data_analyst",
"softMemoryLimit": "30%",
"hardConcurrencyLimit": 20,
"maxQueued": 200,
"schedulingWeight": 2
},
{
"name": "data_engineer",
"softMemoryLimit": "40%",
"hardConcurrencyLimit": 30,
"maxQueued": 300,
"schedulingWeight": 3
},
{
"name": "pipeline",
"softMemoryLimit": "20%",
"hardConcurrencyLimit": 10,
"maxQueued": 100,
"schedulingWeight": 1
}
]
}
],
"selectors": [
{
"userGroup": "admin",
"group": "global.data_engineer"
},
{
"source": ".*pipeline.*",
"group": "global.pipeline"
},
{
"queryType": "SELECT",
"group": "global.data_analyst"
},
{
"queryType": "DATA_DEFINITION",
"group": "global.data_definition"
},
{
"user": "admin",
"group": "global.data_engineer"
},
{
"source": ".*",
"group": "global.data_analyst"
}
]
}
deploy/config/trino/resource-groups.properties
+6
View File
@@ -0,0 +1,6 @@
# =============================================================================
# Trino — Resource Groups & Query Queuing
# Referensi arsitektur: bab 4.3.1.5
# =============================================================================
resource-groups.configuration-manager=file
resource-groups.config-file=/etc/trino/resource-groups.json
deploy/docker-compose.yml
+1415
View File
File diff suppressed because it is too large Load Diff
deploy/docker/airflow/Dockerfile
+114
View File
@@ -0,0 +1,114 @@
# syntax=docker/dockerfile:1
# =============================================================================
# eBdesk SiCABE — Custom Airflow Image (strategi S1: SATU environment)
# =============================================================================
# Base: apache/airflow:3.3.1-python3.11
# - Python 3.11 → masuk matriks uji openmetadata-ingestion 1.13.x
# - Airflow 3.3.1 (terbaru)
#
# Yang di-bake:
# - apache-airflow 3.3.1 + providers (trino, amazon, http)
# - dbt-core 1.11.x + dbt-trino, great_expectations
# - openmetadata-ingestion==1.13.1.0 (match server OM 1.13.1; TANPA extra [airflow])
# - Sling CLI (binary Go — BUKAN paket pip `sling`)
#
# Tiga aturan penting:
# 1. ❌ JANGAN pakai extra `openmetadata-ingestion[airflow]` — extra itu mem-pin
# `apache-airflow==3.2.1` dan langsung merusak resolver.
# 2. ⚠️ Beberapa dependensi OM hanya tersedia sebagai **sdist** (tanpa wheel):
# `antlr4-python3-runtime==4.9.2`, `dsnparse==0.1.15`, dst. Dengan build
# isolation, pip MENGGANTUNG sangat lama; karena itu dipakai
# `--no-build-isolation` (build hanya beberapa detik) + backend build disiapkan.
# 3. ✅ `dbt-core<1.12` (lihat requirements.in): 1.12.x menarik
# `dbt-core-experimental-parser` yang sdist-only.
# Instalasi memakai `--constraint requirements.lock` (hasil `uv pip compile`)
# agar tidak ada backtracking.
#
# Catatan `pip check` (NON-FATAL, sengaja): setelah langkah (3) tersisa 2 konflik
# dengan komponen yang TIDAK dipakai DAG SiCABE:
# - aiobotocore -> botocore>=1.43.3 vs botocore 1.41.6 (OM: boto3~=1.41.5)
# - providers-mysql -> pymysql<1.2 vs pymysql 1.2.0
# Bila kelak DAG memakai salah satunya, pindahkan toolchain-nya ke environment
# terpisah (venv) — lihat strategi S2 di reviews/sicabe-10.
#
# Regenerasi lock bila dependensi berubah:
# docker run --rm -v "$PWD/deploy/docker/airflow:/w" apache/airflow:3.3.1-python3.11 \
# bash -lc 'pip -q install uv && uv pip compile /w/requirements.in --output-file /w/requirements.lock'
#
# Build (konteks = folder ini, karena COPY requirements.*):
# docker build -t sicabe/airflow:3.3.1 -f deploy/docker/airflow/Dockerfile deploy/docker/airflow
# =============================================================================
FROM apache/airflow:3.3.1-python3.11
ARG AIRFLOW_VERSION=3.3.1
ARG SLING_VERSION=1.5.23
USER airflow
COPY requirements.in /tmp/requirements.in
COPY requirements.lock /tmp/requirements.lock
# ─── (1) Backend build untuk paket sdist ────────────────────────────────────
# Dipakai oleh langkah (2) karena build isolation dimatikan.
RUN --mount=type=cache,target=/home/airflow/.cache/pip,uid=50000 \
pip install --user --no-compile \
"setuptools" "wheel" "hatchling" "hatch-vcs" "poetry-core" "flit-core" "setuptools-scm"
# ─── (2) Seluruh dependensi dari lock ───────────────────────────────────────
# --no-build-isolation: kunci utama. Sebagian paket (antlr4-python3-runtime,
# dsnparse, …) hanya sdist; dengan isolasi pip menggantung sangat lama.
# --constraint: pin dari lock → tidak ada backtracking.
# --no-compile: tidak menulis .pyc (menghemat puluhan MB; Python tetap jalan,
# hanya sedikit lebih lambat saat import pertama).
# Cache mount pip → rebuild tidak mengunduh ulang seluruh wheel.
RUN --mount=type=cache,target=/home/airflow/.cache/pip,uid=50000 \
pip install --user --no-build-isolation --no-compile \
--constraint /tmp/requirements.lock \
-r /tmp/requirements.in
# ─── (3) Buang providers-google + dependensi yatim-piatunya ─────────────────
# SiCABE hanya memakai provider trino/amazon/http. `apache-airflow-providers-google`
# (bawaan base image) menarik paket besar: ray (~186 MB), pyarrow, litellm,
# scikit-learn, google-api-python-client, google-cloud-*, pandas-gbq, dsb.
# `pip-autoremove` menghapus paket yang tidak lagi dibutuhkan paket lain.
# Hasil uji: ~/.local 2.5G -> 1.8G (−0.7 GB); airflow/dbt/GE/OM ingestion tetap OK.
#
# ⚠️ Bila kelak butuh Google (mis. Workspace/Gmail/Drive), pasang provider-nya
# kembali — provider Airflow tidak dipecah per produk, jadi akan menarik
# seluruh dependensi google lagi (≈ +0.7 GB).
RUN --mount=type=cache,target=/home/airflow/.cache/pip,uid=50000 \
pip install --no-compile pip-autoremove \
&& pip-autoremove apache-airflow-providers-google -y \
&& pip uninstall -y pip-autoremove
# ─── Verifikasi inti (fatal bila gagal) ─────────────────────────────────────
# `python -B` → jangan tulis .pyc saat verifikasi (menghemat ~50 MB layer).
RUN python -B -c "import airflow; print('airflow', airflow.__version__)" \
&& python -B -c "import dbt, great_expectations; print('dbt + great_expectations OK')"
# ─── Verifikasi OM (non-fatal: hanya melaporkan) ────────────────────────────
RUN ( python -B -c "import metadata; print('openmetadata-ingestion OK')" \
|| echo '[WARN] modul `metadata` gagal di-import — cek nama paket OM' ) \
&& ( command -v metadata >/dev/null && echo 'CLI metadata OK' \
|| echo '[WARN] CLI `metadata` tidak ditemukan' ) \
&& ( pip check || echo '[WARN] pip check melaporkan ketidaksesuaian (lihat log)' )
# ─── Sling CLI (binary Go) ──────────────────────────────────────────────────
# Tarball ±99 MB. Jaringan lambat → timeout tinggi + retry + resume (-C -),
# dan hasil unduhan disimpan di cache mount agar percobaan ulang tidak mulai
# dari nol. Nama file memuat versi supaya cache tidak salah pakai.
RUN --mount=type=cache,target=/tmp/sling-dl,uid=50000 \
( tar -tzf "/tmp/sling-dl/sling-${SLING_VERSION}.tar.gz" >/dev/null 2>&1 \
|| curl -fL --retry 5 --retry-delay 5 --retry-all-errors \
--max-time 2400 --connect-timeout 15 --continue-at - \
"https://github.com/slingdata-io/sling-cli/releases/download/v${SLING_VERSION}/sling_linux_amd64.tar.gz" \
-o "/tmp/sling-dl/sling-${SLING_VERSION}.tar.gz" ) \
&& mkdir -p "${HOME}/.local/bin" \
&& tar -xzf "/tmp/sling-dl/sling-${SLING_VERSION}.tar.gz" -C /tmp/ \
&& ( command -v strip >/dev/null && strip /tmp/sling 2>/dev/null || true ) \
&& chmod +x /tmp/sling \
&& mv /tmp/sling "${HOME}/.local/bin/sling" \
&& "${HOME}/.local/bin/sling" --version
USER airflow
deploy/docker/airflow/Dockerfile.332
+239
View File
@@ -0,0 +1,239 @@
# syntax=docker/dockerfile:1
# =============================================================================
# eBdesk SiCABE — Custom Airflow Image 3.3.2 / Python 3.11 (BERDAMPINGAN)
# =============================================================================
# VARIAN BARU (review R18 / BATCH B). Berdampingan dengan `Dockerfile.slim`
# (3.3.1/py3.11) — yang lama TIDAK diubah, sehingga rollback hanya soal menunjuk
# tag image. Rencana: reviews/sicabe-18-merge-airflow332-plan.md
# Asal-usul & bukti validasi: experiments/airflow332/ (H1–H9 terbukti)
#
# ⚠️ REVISI PENTING (B3 → B2 revisi): percobaan pertama memakai base
# **Python 3.13** dan DITOLAK pada uji go/no-go: `dbt debug`/`dbt compile`
# GAGAL di 3.13 —
# mashumaro → typing_extensions.Annotated.__class_getitem__
# → AttributeError: __class_getitem__ (typing.py 3.13)
# dan TIDAK ADA versi `mashumaro` yang sekaligus memenuhi batas dependensi kita
# (`collate-data-diff <3.11`, `dbt-core <3.15`) dan berfungsi di 3.13.
# Bukti lengkap: deploy/output/b3-hasil-uji.md
# ➡️ Karena itu base dikembalikan ke **Python 3.11** — tumpukan yang sudah
# terbukti di produksi. Tujuan BATCH B (Airflow 3.3.2 + OM 2.0.3 + provider
# MySQL) tetap tercapai, karena pin OM memaksa 3.3.2 terlepas dari Python.
#
# Perbedaan dari Dockerfile.slim:
# 1. base : slim-3.3.1-python3.11 -> slim-3.3.2-python3.11
# 2. provider MySQL : tidak ada -> DIPASANG (keputusan K3), dengan mysqlclient
# dibangun di STAGE BUILDER + libmariadb3 sebagai runtime
# 3. openmetadata : 2.0.2.0 -> 2.0.3.0, extra `[airflow]` DILEPAS
# 4. sqlalchemy<2.1 : PIN WAJIB (hasil A1) — lihat requirements.332.in
# 5. paket tambahan (2026-10-06): statsd (metrik StatsD), duckdb,
# pyiceberg[s3fs,pyarrow]
#
# Base: apache/airflow:slim-3.3.2-python3.11
# Debian 12 (bookworm) · Python 3.11.16 · TANPA gcc (maka stage builder wajib)
#
# Fakta yang sudah diverifikasi pada base ini:
# - Python.h ADA di /usr/python/include/python3.11/Python.h → mysqlclient BISA
# dibangun. Karena itu `python3-dev` Debian TIDAK dipasang.
# - libmariadb3 tersedia (1:10.11.19+maria~deb12); `libmariadb3t64` TIDAK ada.
# - mysqlclient terbukti terbangun di base 3.3.2 (di 3.13:
# mysqlclient-2.3.0-cp313-cp313-linux_x86_64.whl; di 3.11 ekuivalen cp311).
# - `pip install --user` DITERIMA di base 3.11 (di base 3.13 ia DITOLAK karena
# VIRTUAL_ENV=/home/airflow/.local). Berkas ini TIDAK memakai `--user`:
# hasilnya sama, karena venv-nya memang di ~/.local.
#
# ⚠️ URUTAN PEMASANGAN YANG MENENTUKAN (jangan diubah tanpa alasan):
# mysqlclient tidak punya wheel Linux → bila dibiarkan, pip akan mencoba
# membangunnya dari sdist di stage akhir yang TIDAK punya compiler → gagal.
# Karena itu: bangun wheel di stage builder, lalu pasang dari wheelhouse LEBIH
# DULU (sebelum install utama).
#
# Regenerasi lock (CARA YANG DIPAKAI — menghasilkan diff minimal):
# (1) ambil lock produksi 3.3.1 sebagai dasar constraint
# (2) BUANG dari constraint paket yang SENGAJA dinaikkan:
# grep -vE '^(apache-airflow|openmetadata-|collate-)' requirements.lock > c.lock
# (tanpa ini, constraint justru memblokir kenaikan 3.3.2 / OM 2.0.3 yang kita inginkan)
# (3) resolve:
# docker run --rm -v "$PWD/deploy/docker/airflow:/w" -v /path/c.lock:/c.lock:ro \
# apache/airflow:slim-3.3.2-python3.11 \
# bash -lc 'pip -q install uv && uv pip compile /w/requirements.332.in --constraint /c.lock'
# Hasil: hanya 44 baris paket berubah vs produksi (Airflow, provider, OM, collate,
# + MySQL) — jauh lebih kecil daripada resolusi bebas.
# ⚠️ Constraints RESMI Airflow 3.3.2 TIDAK dapat dipakai bersama OM 2.0.3.0
# (bentrok tabulate==0.10.0 vs 0.9.0 dan kubernetes==36.0.3 vs <36).
#
# Build (konteks = folder ini, karena COPY requirements.332.*):
# docker build -t sicabe/airflow:3.3.2-om203-ol \
# -f deploy/docker/airflow/Dockerfile.332 deploy/docker/airflow
# =============================================================================
# ─── STAGE 1 — bangun mysqlclient (compiler hanya ada di sini) ──────────────
FROM apache/airflow:slim-3.3.2-python3.11 AS mysqlwheel
USER root
# gcc + pkg-config + header MariaDB. SENGAJA TANPA `python3-dev` (lihat catatan atas).
# ⚠️ `/wheelhouse` harus DIBUAT dan dimiliki `airflow`; tanpa itu build gagal:
# PermissionError: [Errno 13] Permission denied: '/wheelhouse'
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
gcc pkg-config default-libmysqlclient-dev \
&& rm -rf /var/lib/apt/lists/* \
&& mkdir -p /wheelhouse && chown airflow:0 /wheelhouse
USER airflow
RUN --mount=type=cache,target=/home/airflow/.cache/pip,uid=50000 \
pip wheel --no-cache-dir --wheel-dir /wheelhouse "mysqlclient>=2.2.5" \
&& ls -la /wheelhouse
# ─── STAGE 2 — image akhir (tanpa compiler) ────────────────────────────────
FROM apache/airflow:slim-3.3.2-python3.11
ARG SLING_VERSION=1.5.23
USER root
# Library RUNTIME untuk ekstensi mysqlclient (libmariadb.so.3). Tanpa paket ini:
# ImportError: libmariadb.so.3: cannot open shared object file
RUN apt-get update \
&& apt-get install -y --no-install-recommends libmariadb3 \
&& rm -rf /var/lib/apt/lists/* \
&& ldconfig
USER airflow
COPY --from=mysqlwheel --chown=airflow:0 /wheelhouse /wheelhouse
COPY requirements.332.in /tmp/requirements.in
COPY requirements.332.lock /tmp/requirements.lock
# ─── (1) mysqlclient dari WHEELHOUSE, lebih dulu ────────────────────────────
# `--no-index` + `--find-links` memaksa pip memakai wheel hasil stage builder,
# sehingga tidak pernah mencoba membangun sdist di stage tanpa compiler ini.
RUN --mount=type=cache,target=/home/airflow/.cache/pip,uid=50000 \
pip install --no-compile --no-index --find-links /wheelhouse \
--constraint /tmp/requirements.lock mysqlclient
# ─── (2) Backend build untuk paket sdist pure-Python ────────────────────────
RUN --mount=type=cache,target=/home/airflow/.cache/pip,uid=50000 \
pip install --no-compile \
"setuptools" "wheel" "hatchling" "hatch-vcs" "poetry-core" "flit-core" "setuptools-scm"
# ─── (3) Dependensi dari lock ───────────────────────────────────────────────
RUN --mount=type=cache,target=/home/airflow/.cache/pip,uid=50000 \
pip install --no-build-isolation --no-compile \
--constraint /tmp/requirements.lock \
-r /tmp/requirements.in
# ─── Verifikasi inti (FATAL bila gagal) ─────────────────────────────────────
RUN python -B -c "import airflow; print('airflow', airflow.__version__)"
# Kelima klien koneksi harus dapat diimpor (inti eksperimen airflow332).
RUN python -B -c "import MySQLdb; print('MySQLdb (mysqlclient) OK ->', MySQLdb.version_info)" \
&& python -B -c "import mysql.connector; print('mysql-connector-python OK ->', mysql.connector.__version__)" \
&& python -B -c "import pymysql; print('pymysql OK')" \
&& python -B -c "import psycopg2; print('psycopg2 OK ->', psycopg2.__version__)" \
&& python -B -c "import trino; print('trino client OK ->', trino.__version__)"
# Bukti mysqlclient benar-benar DIMUAT (bukan sekadar terpasang): impor modul C-nya
# sudah memuat libmariadb.so.3 — kalau library runtime tidak ada, perintah gagal.
RUN python -B -c "import MySQLdb, MySQLdb._mysql; print('modul C mysqlclient dimuat:', MySQLdb._mysql.__file__)" \
&& ( ldconfig -p 2>/dev/null | grep -i mariadb || echo '[WARN] libmariadb tidak terlihat di ldconfig' )
RUN python -B -c "import dbt, great_expectations; print('dbt + great_expectations OK')" \
&& python -B -c "import airflow.providers.mysql; print('provider mysql OK')"
# ─── Provider OpenLineage (FATAL bila gagal) ────────────────────────────────
# Ditambahkan 2026-10-02 atas permintaan pemilik proyek. Verifikasi memastikan:
# 1. paket openlineage-python terpasang
# 2. provider Airflow-nya dapat diimpor
# 3. entry point provider benar-benar terdaftar
# ⚠️ Provider ini INERT tanpa backend: hanya aktif bila transport/endpoint
# OpenLineage dikonfigurasi (OPENLINEAGE_URL + OPENLINEAGE_API_KEY, dst).
# Belum ada backend OpenLineage di stack ini — jadi ini mengaktifkan KEMAMPUAN.
# Catatan pemeriksaan: di Airflow 3, NAMA entry point provider selalu
# `provider_info` (yang membedakan adalah VALUE/module path-nya). Percobaan
# pertama saya memeriksa `e.name` -> positif palsu. Yang fatal di sini cukup
# impor keduanya; info entry point hanya DICETAK (tidak di-assert) agar tidak rapuh.
RUN python -B -c "import openlineage; print('openlineage-python OK')" \
&& python -B -c "import airflow.providers.openlineage; print('provider openlineage import OK')" \
&& python -B -c "import importlib.metadata as m; print('versi provider:', m.version('apache-airflow-providers-openlineage')); vals=[e.value for e in m.entry_points(group='apache_airflow_provider')]; print('entry point provider:', [v for v in vals if 'lineage' in v.lower()] or '(nama/value tidak memuat kata lineage — normal di Airflow 3)')"
# ─── Shim eventTime OpenLineage (FATAL bila gagal) ───────────────────────────
# Ditemukan di experiments/lineage-airflow-om: OpenMetadata 2.0.3 menolak event
# yang `eventTime`-nya berakhiran `+00:00` (bentuk yang dihasilkan
# datetime.isoformat() untuk UTC) dengan
# 400 {"code":400,"message":"Invalid request format"}
# Shim ZuluHttpTransport menormalkan `+00:00` -> `Z` sebelum event dikirim.
# Ditempatkan di direktori khusus + PYTHONPATH agar dapat diimpor sebagai
# `ol_time_shim.ZuluHttpTransport` dari proses task maupun subprosesnya.
COPY --chown=airflow:0 ol_time_shim.py /opt/airflow/ol_shim/ol_time_shim.py
# Catatan: image dasar `apache/airflow:slim-3.3.2-python3.11` TIDAK mendefinisikan
# PYTHONPATH (diverifikasi dengan `docker run --entrypoint env`), jadi tidak ada
# nilai lama yang tertimpa. Jangan menulis "${PYTHONPATH}" di sini: BuildKit
# memperingatkannya sebagai variabel tak terdefinisi.
ENV PYTHONPATH="/opt/airflow/ol_shim"
RUN python -B -c "from ol_time_shim import ZuluHttpTransport; print('shim eventTime OpenLineage DAPAT DIMUAT:', ZuluHttpTransport)"
# ─── Penjaga regresi dbt (FATAL) — pelajaran dari uji B3 ────────────────────
# `import dbt` dan `dbt --version` TIDAK CUKUP: pada percobaan Python 3.13
# keduanya LOLOS, tetapi `dbt debug`/`dbt compile` gagal dengan
# mashumaro -> typing_extensions.Annotated.__class_getitem__
# -> AttributeError: __class_getitem__ (typing.py)
# Pemeriksaan di bawah memakai PEMAKAIAN YANG SAH (`Annotated[tipe, anotasi]`),
# yang menempuh jalur kode yang sama dengan dbt saat mem-parse model — sehingga
# kerusakan jenis itu tertangkap saat BUILD.
# ⚠️ Percobaan pertama penjaga ini SALAH dan menghasilkan positif palsu:
# memanggil `Annotated.__class_getitem__((int,))` dengan SATU argumen ->
# TypeError "Annotated[...] should be used with at least two arguments".
# (Catatan: galat itu sekaligus membuktikan fungsinya ADA di py3.11.)
# Catatan: `dbt debug` penuh tidak bisa dijalankan di sini karena butuh koneksi
# Trino; pemeriksaan ini adalah proksi yang tepat dan murah.
RUN python -B -c "import typing_extensions as te; te.Annotated[int, 'meta']" \
&& python -B -c "import importlib.metadata as m; print('jalur Annotated OK -> dbt dapat berjalan; mashumaro', m.version('mashumaro'))"
# ─── Plugin OM harus benar-benar DAPAT DIMUAT (FATAL) ───────────────────────
# Pelajaran dari experiments/airflow332: `import openmetadata_managed_apis` BERHASIL,
# tetapi entry point gagal `load()` karena rantai
# flask_appbuilder -> sqlalchemy_utils -> SQLAlchemy 2.1
# (AttributeError: 'ScalarAttributeImpl')
# → plugin MATI saat Airflow start; tombol "Test Connection" hilang; log hanya
# berbunyi "Failed to import plugin openmetadata_managed_apis".
# ⚠️ Verifikasi lama (hanya mengimpor MODUL) LOLOS pada kondisi rusak itu — karena
# itu di sini entry point benar-benar dimuat, dan kegagalannya FATAL.
# Ditulis SATU BARIS: kode Python multi-baris dalam `python -c "..."` tanpa `\`
# akan dibaca Docker sebagai instruksi baru ("unknown instruction: import").
RUN python -B -c "import flask_appbuilder; print('flask_appbuilder import OK')" \
&& python -B -c "import sqlalchemy as sa; print('sqlalchemy', sa.__version__)" \
&& python -B -c "import importlib.metadata as m; eps=[e for e in m.entry_points(group='airflow.plugins') if 'openmetadata_managed_apis' in e.name]; assert eps, 'entry point openmetadata_managed_apis tidak ada'; cls=eps[0].load(); print('plugin openmetadata_managed_apis DAPAT DIMUAT:', cls)"
# ─── Paket tambahan 2026-10-06 (FATAL bila gagal) ───────────────────────────
# 1) statsd — WAJIB agar Airflow memakai StatsClient (StatsD → statsd-exporter).
# Tanpa ini Airflow jatuh ke NoStatsLogger dan TIDAK ada metrik terkirim.
RUN python -B -c "import statsd; print('statsd OK (StatsClient aktif)')"
# 2) duckdb — analitik lokal di task DAG.
RUN python -B -c "import duckdb; print('duckdb OK ->', duckdb.__version__)"
# 3) pyiceberg[s3fs,pyarrow] — S3FileIO via s3fs + tabel Arrow.
# Impor ketiganya SEKALIGUS: pyiceberg tanpa s3fs/pyarrow kehilangan extra-nya.
RUN python -B -c "import pyiceberg, pyarrow, s3fs; print('pyiceberg OK ->', pyiceberg.__version__); print('pyarrow OK ->', pyarrow.__version__); print('s3fs OK ->', s3fs.__version__)"
# ─── Verifikasi non-fatal (melaporkan saja) ─────────────────────────────────
RUN ( python -B -c "import metadata; print('openmetadata-ingestion OK')" \
|| echo '[WARN] modul `metadata` gagal di-import' ) \
&& ( python -B -c "import openmetadata_managed_apis; print('plugin openmetadata-managed-apis OK')" \
|| echo '[WARN] plugin openmetadata-managed-apis gagal di-import' ) \
&& ( python -B -c "import airflow.providers.postgres; print('provider postgres OK')" \
|| echo '[WARN] provider postgres' ) \
&& ( command -v metadata >/dev/null && echo 'CLI metadata OK' \
|| echo '[WARN] CLI metadata tidak ada' ) \
&& ( pip check || echo '[WARN] pip check melaporkan ketidaksesuaian (lihat log)' )
# ─── Sling CLI (binary Go — BUKAN paket pip `sling`) ────────────────────────
RUN --mount=type=cache,target=/tmp/sling-dl,uid=50000 \
( tar -tzf "/tmp/sling-dl/sling-${SLING_VERSION}.tar.gz" >/dev/null 2>&1 \
|| curl -fL --retry 5 --retry-delay 5 --retry-all-errors \
--max-time 2400 --connect-timeout 15 --continue-at - \
"https://github.com/slingdata-io/sling-cli/releases/download/v${SLING_VERSION}/sling_linux_amd64.tar.gz" \
-o "/tmp/sling-dl/sling-${SLING_VERSION}.tar.gz" ) \
&& mkdir -p "${HOME}/.local/bin" \
&& tar -xzf "/tmp/sling-dl/sling-${SLING_VERSION}.tar.gz" -C /tmp/ \
&& chmod +x /tmp/sling \
&& mv /tmp/sling "${HOME}/.local/bin/sling" \
&& "${HOME}/.local/bin/sling" --version
USER airflow
deploy/docker/airflow/Dockerfile.slim
+133
View File
@@ -0,0 +1,133 @@
# syntax=docker/dockerfile:1
# =============================================================================
# eBdesk SiCABE — Custom Airflow Image (base SLIM)
# =============================================================================
# Base: apache/airflow:slim-3.3.1-python3.11
# 1.15 GB · hanya 5 provider (standard, common-compat, common-io, common-sql, smtp)
# vs apache/airflow:3.3.1-python3.11 → 3.23 GB · 30 provider
#
# Yang di-bake:
# - provider tambahan: trino, amazon, http, postgres (standard/common-* dari base)
# - dbt-core 1.11.x + dbt-trino, great_expectations
# - openmetadata-ingestion[airflow,pii-processor]==2.0.2.0 + plugin
# openmetadata-managed-apis==2.0.2.0, disajikan Airflow di /pluginsv2/
# (inilah yang menghidupkan tombol Test Connection — §8.11)
# Extra `pii-processor` (presidio + spacy) WAJIB agar DAG yang dibangkitkan
# plugin untuk pipeline TestSuite bisa diimpor Airflow — tanpa itu:
# No module named 'presidio_analyzer' (§8.10)
# Konsekuensi: image +390 MB dan pandas diturunkan ke 2.3.3 (extra minta pandas<3).
# - Sling CLI (binary Go)
#
# Catatan penting:
# 1. ✅ Extra `openmetadata-ingestion[airflow]` DIPAKAI sejak OM 2.0.2. Di OM 2.0.2
# extra itu mem-pin apache-airflow==3.3.1 (sama dengan base image ini), sedang
# di OM 1.13.x mem-pin 3.2.1 sehingga menurunkan Airflow. Diuji di experiments/om20x.
# 2. ⚠️ Sebagian paket hanya sdist (antlr4-python3-runtime, dsnparse, …) → wajib
# `--no-build-isolation`. Keduanya **pure Python** sehingga TIDAK butuh gcc
# (base slim memang tidak punya gcc).
# 3. ✅ `dbt-core<1.12` (1.12.x menarik dbt-core-experimental-parser, sdist-only).
# 4. ✅ `--constraint requirements.lock` → tanpa backtracking.
# 5. ⚠️ Base slim TIDAK punya psycopg2, padahal Postgres adalah metadata DB
# Airflow (postgresql+psycopg2://) → disediakan oleh provider `postgres`.
# 6. `strip` tidak ada di slim → binary Sling dibiarkan apa adanya (255 MB).
#
# Provider MySQL — TIDAK dipasang (disengaja):
# `apache-airflow-providers-mysql` tidak dipakai SiCABE — koneksi Airflow hanya
# aws/trino/postgres/http, dan OpenMetadata memakai konektor + pymysql miliknya
# sendiri (`metadata/ingestion/source/database/mysql` + `pymysql==1.2.0`).
# Selain itu provider ini butuh `mysqlclient` yang TIDAK punya wheel Linux
# (sdist → butuh compiler, sedangkan base slim tanpa gcc), sehingga hook-nya
# (MySQLdb) tetap tidak berfungsi walau provider dipasang.
# Bila kelak benar-benar dibutuhkan, bangun mysqlclient di stage builder:
# FROM apache/airflow:slim-3.3.1-python3.11 AS mysqlwheel
# USER root
# RUN apt-get update && apt-get install -y --no-install-recommends \
# gcc pkg-config default-libmysqlclient-dev python3-dev \
# && rm -rf /var/lib/apt/lists/*
# USER airflow
# RUN pip wheel --wheel-dir /wheelhouse "mysqlclient>=2.2.5"
# lalu di stage akhir: COPY --from=mysqlwheel /wheelhouse /wheelhouse +
# pip install --no-index --find-links /wheelhouse mysqlclient,
# serta apt-get install libmariadb3 (lib runtime).
#
# Regenerasi lock:
# docker run --rm -v "$PWD/deploy/docker/airflow:/w" apache/airflow:slim-3.3.1-python3.11 \
# bash -lc 'pip -q install uv && uv pip compile /w/requirements.in --output-file /w/requirements.lock'
#
# Build (konteks = folder ini, karena COPY requirements.*):
# docker build -t sicabe/airflow:3.3.1-om202 -f deploy/docker/airflow/Dockerfile.slim deploy/docker/airflow
# =============================================================================
FROM apache/airflow:slim-3.3.1-python3.11
ARG SLING_VERSION=1.5.23
USER airflow
COPY requirements.in /tmp/requirements.in
COPY requirements.lock /tmp/requirements.lock
# ─── (1) Backend build untuk paket sdist pure-Python ────────────────────────
RUN --mount=type=cache,target=/home/airflow/.cache/pip,uid=50000 \
pip install --user --no-compile \
"setuptools" "wheel" "hatchling" "hatch-vcs" "poetry-core" "flit-core" "setuptools-scm"
# ─── (2) Dependensi dari lock (termasuk provider postgres → psycopg2-binary) ─
RUN --mount=type=cache,target=/home/airflow/.cache/pip,uid=50000 \
pip install --user --no-build-isolation --no-compile \
--constraint /tmp/requirements.lock \
-r /tmp/requirements.in
# ─── Verifikasi inti (fatal bila gagal) ─────────────────────────────────────
# `python -B` → jangan tulis .pyc (menghemat puluhan MB layer).
RUN python -B -c "import airflow; print('airflow', airflow.__version__)" \
&& python -B -c "import psycopg2; print('psycopg2', psycopg2.__version__)" \
&& python -B -c "import dbt, great_expectations; print('dbt + great_expectations OK')"
# ─── Plugin OM harus benar-benar DAPAT DIMUAT (FATAL) ───────────────────────
# Pelajaran dari experiments/airflow332 (reviews/sicabe-18, A1):
# `import openmetadata_managed_apis` BERHASIL, tetapi entry point-nya gagal
# `load()` karena rantai flask_appbuilder -> sqlalchemy_utils -> SQLAlchemy 2.1
# (AttributeError: module 'sqlalchemy.orm.attributes' has no attribute
# 'ScalarAttributeImpl'). Akibatnya plugin MATI saat Airflow start: tombol
# "Test Connection" / pipeline service client (§8.11) hilang, dan log hanya
# berbunyi "Failed to import plugin openmetadata_managed_apis" — tanpa
# menyebut SQLAlchemy sama sekali.
# ⚠️ Verifikasi lama (hanya mengimpor MODUL) LOLOS pada kondisi rusak itu.
# Karena itu di sini entry point benar-benar dimuat, dan kegagalannya FATAL:
# lebih baik build berhenti daripada menghasilkan image yang tombolnya mati.
# Catatan: ditulis SATU BARIS — kode Python multi-baris dalam `python -c "..."`
# tanpa `\` akan dibaca Docker sebagai instruksi baru ("unknown instruction").
RUN python -B -c "import flask_appbuilder; print('flask_appbuilder import OK')" \
&& python -B -c "import sqlalchemy as sa; print('sqlalchemy', sa.__version__)" \
&& python -B -c "import importlib.metadata as m; eps=[e for e in m.entry_points(group='airflow.plugins') if 'openmetadata_managed_apis' in e.name]; assert eps, 'entry point openmetadata_managed_apis tidak ada'; cls=eps[0].load(); print('plugin openmetadata_managed_apis DAPAT DIMUAT:', cls)"
# ─── Verifikasi OM & provider (non-fatal: hanya melaporkan) ─────────────────
RUN ( python -B -c "import metadata; print('openmetadata-ingestion OK')" \
|| echo '[WARN] modul `metadata` gagal di-import' ) \
&& ( python -B -c "import openmetadata_managed_apis; print('plugin openmetadata-managed-apis OK')" \
|| echo '[WARN] plugin openmetadata-managed-apis gagal di-import' ) \
&& python -B -c "import importlib.metadata as m; print('airflow.plugins entry points:', [e.name for e in m.entry_points(group='airflow.plugins')])" \
&& ( python -B -c "import importlib.metadata as m; assert 'openmetadata_managed_apis' in [e.name for e in m.entry_points(group='airflow.plugins')]" \
|| echo '[WARN] plugin TIDAK terdaftar pada entry point airflow.plugins' ) \
&& ( python -B -c "import airflow.providers.postgres; print('provider postgres OK')" \
|| echo '[WARN] provider postgres' ) \
&& ( command -v metadata >/dev/null && echo 'CLI metadata OK' \
|| echo '[WARN] CLI metadata tidak ada' ) \
&& ( pip check || echo '[WARN] pip check melaporkan ketidaksesuaian (lihat log)' )
# ─── Sling CLI (binary Go — BUKAN paket pip `sling`) ────────────────────────
# Tarball ±94 MB. Timeout tinggi + retry + resume; di-cache agar tidak mengulang.
RUN --mount=type=cache,target=/tmp/sling-dl,uid=50000 \
( tar -tzf "/tmp/sling-dl/sling-${SLING_VERSION}.tar.gz" >/dev/null 2>&1 \
|| curl -fL --retry 5 --retry-delay 5 --retry-all-errors \
--max-time 2400 --connect-timeout 15 --continue-at - \
"https://github.com/slingdata-io/sling-cli/releases/download/v${SLING_VERSION}/sling_linux_amd64.tar.gz" \
-o "/tmp/sling-dl/sling-${SLING_VERSION}.tar.gz" ) \
&& mkdir -p "${HOME}/.local/bin" \
&& tar -xzf "/tmp/sling-dl/sling-${SLING_VERSION}.tar.gz" -C /tmp/ \
&& chmod +x /tmp/sling \
&& mv /tmp/sling "${HOME}/.local/bin/sling" \
&& "${HOME}/.local/bin/sling" --version
USER airflow
deploy/docker/airflow/ol_time_shim.py
+44
View File
@@ -0,0 +1,44 @@
"""Shim transport OpenLineage — perbaiki `eventTime` `+00:00` -> `Z`.
Kenapa ada:
OpenMetadata 2.0.3 hanya menerima `eventTime` berakhiran `Z`. Provider
OpenLineage Airflow membangun `eventTime` dengan `datetime.isoformat()`
(lihat airflow/providers/openlineage/plugins/adapter.py) yang untuk datetime
UTC menghasilkan akhiran `+00:00`. Akibatnya OM menjawab
400 {"code":400,"message":"Invalid request format"}
untuk SETIAP event (pesan berasal dari JsonMappingExceptionMapper —
kegagalan Jackson memetakan JSON ke OpenLineageRunEvent).
Cara pakai:
Transport kustom didukung resmi oleh provider OpenLineage lewat `type`
berupa path kelas Python (openlineage/client/transport/factory.py ->
import_from_string). Konfigurasikan transport seperti biasa, tetapi
`type` diarahkan ke kelas ini, mis.:
{"type": "ol_time_shim.ZuluHttpTransport",
"url": "http://openmetadata-server:8585/api/v1/openlineage/",
"endpoint": "lineage",
"auth": {"type": "airflow_connection_api_key",
"conn_id": "openmetadata_lineage_api"}}
Referensi: experiments/lineage-airflow-om/{FASE3-HASIL.md,shim/ol_time_shim.py}
"""
from __future__ import annotations
from openlineage.client.transport.http import HttpConfig, HttpTransport
class ZuluHttpTransport(HttpTransport):
"""HttpTransport yang menormalkan `eventTime` ke bentuk `Z` (UTC)."""
kind = "zuluhttp"
config_class = HttpConfig
def emit(self, event): # type: ignore[no-untyped-def]
et = getattr(event, "eventTime", None)
if isinstance(et, str) and et.endswith("+00:00"):
event.eventTime = et[:-6] + "Z"
return super().emit(event)
deploy/docker/airflow/requirements.332.in
+99
View File
@@ -0,0 +1,99 @@
# =============================================================================
# eBdesk SiCABE — dependensi image Airflow 3.3.2 / Python 3.11 (BERDAMPINGAN)
# =============================================================================
# SUMBER (input) untuk `uv pip compile`. Jangan tambahkan pin hasil ke sini —
# hasilnya ada di requirements.332.lock.
# Berdampingan dengan `requirements.in` (3.3.1) yang TIDAK diubah.
# Rencana & bukti: reviews/sicabe-18-merge-airflow332-plan.md · experiments/airflow332/
#
# Perubahan dari requirements.in (3.3.1):
# 1. apache-airflow 3.3.1 -> 3.3.2
# 2. + apache-airflow-providers-mysql (keputusan K3 — sebelumnya sengaja tidak ada)
# 3. openmetadata-* 2.0.2.0 -> 2.0.3.0 ; extra `[airflow]` DILEPAS
# 4. + sqlalchemy<2.1 (PIN WAJIB, hasil langkah A1)
#
# ⚠️ CONSTRAINT: resolusi ini di-resolve BEBAS (tanpa constraints resmi Airflow),
# karena constraints resmi 3.3.2 TIDAK dapat dipakai bersama OpenMetadata 2.0.3.0:
# constraints tabulate==0.10.0 vs OM tabulate==0.9.0 (keras)
# constraints kubernetes==36.0.3 vs OM kubernetes<36
# Jalur `--override` sudah dicoba dan TIDAK menang atas constraints.
# Preseden: lock produksi 3.3.1 pun di-resolve bebas (ia memakai tabulate 0.9.0).
# =============================================================================
apache-airflow==3.3.2
# Provider Airflow yang dipakai DAG / konektor
apache-airflow-providers-trino
apache-airflow-providers-amazon
apache-airflow-providers-http
# WAJIB: Postgres adalah metadata DB Airflow (postgresql+psycopg2://).
apache-airflow-providers-postgres
# ⚠️ BARU (keputusan K3): untuk konek ke server MySQL eksternal. Provider ini
# menarik EMPAT klien (mysqlclient, mysql-connector-python, aiomysql, pymysql)
# sehingga ketiga rute Extra `client` tersedia. `mysqlclient` tidak punya wheel
# Linux → dibangun di stage builder Dockerfile.332 + libmariadb3 sebagai runtime.
apache-airflow-providers-mysql
# OpenLineage — memancarkan event lineage dari task Airflow.
# ⚠️ INERT tanpa backend: provider ini hanya aktif bila transport/endpoint
# OpenLineage dikonfigurasi (mis. OPENLINEAGE_URL + OPENLINEAGE_API_KEY, atau
# transport lain). Di stack ini BELUM ada backend OpenLineage, jadi penambahan
# ini mengaktifkan KEMAMPUAN — belum menghasilkan lineage sampai dikonfigurasi.
# Dependensinya ringan dan hanya batas bawah: attrs>=22.2 (kita punya 26.1.0),
# providers-common-sql/common-compat (sudah ada), plus dua paket baru:
# openlineage-python + openlineage-integration-common.
apache-airflow-providers-openlineage
# ─── PIN WAJIB: melindungi plugin OpenMetadata dari kerusakan SENYAP ─────────
# SQLAlchemy 2.1 mematahkan rantai:
# flask_appbuilder -> sqlalchemy_utils -> sqlalchemy.orm.attributes
# AttributeError: module 'sqlalchemy.orm.attributes' has no attribute
# 'ScalarAttributeImpl'
# Akibatnya plugin `openmetadata-managed-apis` GAGAL DIMUAT saat Airflow start
# (log hanya "Failed to import plugin openmetadata_managed_apis"), sehingga tombol
# "Test Connection" dan jalur pipeline service client (install-phases §8.11) mati.
# ⚠️ Berkas constraints RESMI Airflow pun TIDAK mem-pin sqlalchemy — jadi hanya
# pin eksplisit inilah yang melindungi (ditemukan di experiments/airflow332).
sqlalchemy<2.1
# Transformasi & data quality
# dbt-core<1.12: 1.12.x menarik `dbt-core-experimental-parser` yang sdist-only.
dbt-core<1.12
dbt-trino>=1.8,<2.0
great_expectations>=1.0,<2.0
# Metadata ingestion
# ⚠️ Extra `[airflow]` DILEPAS (keputusan pemilik proyek). Dampaknya TERUKUR di H7:
# produksi (dengan extra) 18/18 modul · eksperimen (tanpa extra) 13/18
# yang hilang: tornado, elasticsearch, opensearchpy,
# providers-elasticsearch, providers-opensearch
# TETAP ADA (yang menentukan): workflow_factory, api.routes.deploy, pygtrie,
# attrs, werkzeug, starlette, python-multipart, otlp, presidio, spacy
# → jalur DAG ingestion metadata TETAP UTUH; OM 2.x hanya punya 2 sink
# (`file`, `metadata_rest`) dan keduanya impor bersih.
# Bila kelak memakai pipeline yang meng-ingest DARI Elasticsearch/OpenSearch,
# tambahkan paket spesifik di atas — bukan mengembalikan seluruh extra.
# Extra `[pii-processor]` TETAP dipasang (presidio + spacy) agar DAG pipeline DQ
# yang dibangkitkan plugin dapat diimpor Airflow. Versi WAJIB sama dengan server.
openmetadata-ingestion[pii-processor]==2.0.3.0
# Plugin REST API OpenMetadata yang disajikan Airflow di prefix /pluginsv2/
# (yang menghidupkan tombol "Test Connection"). Paket ini TIDAK mem-pin Airflow
# (`apache-airflow>=2.2.2`) sehingga netral terhadap pilihan versi kita.
openmetadata-managed-apis==2.0.3.0
# Lineage dbt (bronze→silver→gold) di OpenMetadata — tanpa ini workflow dbt gagal
# dengan "No module named 'collate_dbt_artifacts_parser'".
collate-dbt-artifacts-parser==0.1.4
# ─── Ditambahkan 2026-10-06 atas permintaan pemilik proyek ──────────────────
# 1) statsd — WAJIB agar Airflow memakai StatsClient (StatsD → statsd-exporter).
# Tanpa paket ini Airflow jatuh ke NoStatsLogger:
# "Could not configure StatsClient: No module named 'statsd'"
# sehingga TIDAK ada metrik yang terkirim (menutup caveat non-permanen M2).
statsd
# 2) duckdb — analitik lokal di dalam task DAG.
duckdb
# 3) pyiceberg[s3fs,pyarrow] — akses tabel Iceberg langsung (S3FileIO via s3fs)
# + konversi tabel Arrow. Menarik pyarrow, s3fs, dan aiobotocore (lihat lock).
pyiceberg[s3fs,pyarrow]
deploy/docker/airflow/requirements.332.lock
+1697
View File
File diff suppressed because it is too large Load Diff
deploy/docker/airflow/requirements.in
+71
View File
@@ -0,0 +1,71 @@
# =============================================================================
# eBdesk SiCABE — dependensi top-level image Airflow (strategi S1: satu env)
# =============================================================================
# File ini adalah SUMBER (input) untuk `uv pip compile`. Jangan tambahkan pin
# hasil ke sini — hasilnya ada di requirements.lock.
#
# Regenerasi lock (lihat juga header deploy/docker/airflow/Dockerfile):
# docker run --rm -v "$PWD/deploy/docker/airflow:/w" apache/airflow:3.3.1-python3.11 \
# bash -lc 'pip -q install uv && uv pip compile /w/requirements.in --output-file /w/requirements.lock'
#
# ✅ Extra `[airflow]` WAJIB dipakai sejak OpenMetadata 2.0.2.
# Di OM 1.13.x extra itu mem-pin `apache-airflow==3.2.1` sehingga menurunkan
# Airflow; di OM 2.0.2 extra yang sama mem-pin `apache-airflow==3.3.1` — sama
# persis dengan versi terbaru dan base image kita. Diuji di experiments/om20x.
# =============================================================================
apache-airflow==3.3.1
# Provider Airflow yang dipakai DAG / konektor
apache-airflow-providers-trino
apache-airflow-providers-amazon
apache-airflow-providers-http
# WAJIB: Postgres adalah metadata DB Airflow (postgresql+psycopg2://) — tanpa
# driver-nya container gagal start. Base `slim` TIDAK membawanya.
apache-airflow-providers-postgres
# ─── PIN WAJIB: melindungi plugin OpenMetadata dari kerusakan SENYAP ─────────
# SQLAlchemy 2.1 mematahkan rantai:
# flask_appbuilder -> sqlalchemy_utils -> sqlalchemy.orm.attributes
# AttributeError: module 'sqlalchemy.orm.attributes' has no attribute
# 'ScalarAttributeImpl'
# Akibatnya plugin `openmetadata-managed-apis` GAGAL DIMUAT saat Airflow start
# (log hanya berbunyi "Failed to import plugin openmetadata_managed_apis"),
# sehingga tombol "Test Connection" dan jalur pipeline service client
# (docs/sicabe-install-phases-detail.md §8.11) mati — dengan pesan yang TIDAK
# menyebut SQLAlchemy sama sekali.
# Lock saat ini memakai 2.0.54 HANYA karena di-resolve ketika 2.1 belum ada;
# tanpa pin ini, regenerasi lock berikutnya akan menarik 2.1.x dan mematahkannya.
# ⚠️ Berkas constraints RESMI Airflow pun TIDAK mem-pin sqlalchemy — jadi hanya
# pin eksplisit inilah yang melindungi.
# Ditemukan & dibuktikan di experiments/airflow332 — lihat reviews/sicabe-18.
sqlalchemy<2.1
# Transformasi & data quality
# dbt-core<1.12: 1.12.x menarik `dbt-core-experimental-parser` yang sdist-only
# (tanpa wheel) sehingga memaksa build dari source. 1.11.x tidak.
dbt-core<1.12
dbt-trino>=1.8,<2.0
great_expectations>=1.0,<2.0
# Metadata ingestion
# WAJIB sama dengan versi server OpenMetadata (deploy/ = 2.0.2).
# Extra `[airflow]` membawa plugin-side deps + provider elasticsearch/opensearch.
# Extra `[pii-processor]` (presidio + spacy) dibutuhkan agar DAG yang DIBANGKITKAN
# plugin OpenMetadata untuk pipeline TestSuite bisa diimpor Airflow: DAG tsb
# mengimpor workflow ingestion yang menyentuh presidio_analyzer. Tanpa extra ini
# deploy pipeline gagal dengan:
# Failed to import dag_file [.../sicabe_dq_gold_native.py]: No module named 'presidio_analyzer'
openmetadata-ingestion[airflow,pii-processor]==2.0.2.0
# Plugin REST API OpenMetadata yang disajikan Airflow di prefix /pluginsv2/.
# Inilah yang di-probe server OM (GET /pluginsv2/api/v2/openmetadata/health-auth)
# dan yang menghidupkan tombol "Test Connection" di UI — lihat
# docs/sicabe-install-phases-detail.md §8.11.
openmetadata-managed-apis==2.0.2.0
# Lineage dbt (bronze→silver→gold) di OpenMetadata.
# Extra `[dbt]` dari openmetadata-ingestion memerlukan paket ini untuk membaca
# artefak dbt (manifest.json/catalog.json). Tanpa ini, workflow dbt gagal dengan
# "No module named 'collate_dbt_artifacts_parser'" dan lineage tidak terbentuk.
collate-dbt-artifacts-parser==0.1.4
deploy/docker/airflow/requirements.lock
+1188
View File
File diff suppressed because it is too large Load Diff
deploy/install.sh
Executable
+997
View File
@@ -0,0 +1,997 @@
#!/usr/bin/env bash
# =============================================================================
# eBdesk SiCABE — Installer
# =============================================================================
# Sesuai urutan deployment berdasarkan Dependency Graph (bab 3.1 & 12.2):
# Phase 1: Governance (IAM) → Keycloak
# Phase 2: Storage → MinIO → Nessie
# Phase 3: Processing → Trino
# Phase 4: Serving → StarRocks
# Phase 5: Ingestion → Airflow
# Phase 6: Governance (Lan) → OpenMetadata → Grafana
#
# Usage:
# ./install.sh # Deploy semua phase
# ./install.sh --phase 1 # Deploy hanya phase tertentu
# ./install.sh --phase 1-3 # Deploy phase 1 sampai 3
# ./install.sh --status # Cek status semua komponen
# ./install.sh --destroy # Hentikan dan hapus semua container
# ./install.sh --help # Tampilkan bantuan
# =============================================================================
set -euo pipefail
# ─── Base Directory ─────────────────────────────────────────────────────────
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
cd "${SCRIPT_DIR}"
# ─── Shared Library ──────────────────────────────────────────────────────────
source "${SCRIPT_DIR}/lib/common.sh"
# ─── Color & Styling (overrides untuk install.sh) ────────────────────────────
STEP="${BOLD}[${NC}"
# ─── Generate .env jika belum ada ──────────────────────────────────────────
# KECUALI mode multi-host (--host): rahasia untuk host itu sudah ada di
# generated/host-<x>/.env. Membuat .env acak di server ini BERBAHAYA karena rahasianya
# akan BEDA dari host lain (client secret OIDC, password DB, service account), dan
# berkas itu akan dipakai diam-diam oleh pemanggilan TANPA --host (bootstrap.sh,
# test-verification.sh, gate fase) sehingga muncul kegagalan login yang sulit dilacak.
HOST_MODE_ARG=0
ENV_WAS_CREATED=0
for _arg in "$@"; do
[[ "${_arg}" == "--host" ]] && HOST_MODE_ARG=1
done
if [[ ${HOST_MODE_ARG} -eq 1 && ! -f "${SCRIPT_DIR}/.env" ]]; then
echo -e "${INFO} Mode --host: .env lokal TIDAK dibuat (memakai generated/host-<x>/.env)"
fi
if [[ ${HOST_MODE_ARG} -eq 0 && ! -f "${SCRIPT_DIR}/.env" ]]; then
echo -e "${INFO} Membuat .env dengan password acak..."
echo -e "${WARN} Password tersimpan di .env — JANGAN commit file ini ke git!"
# PERHATIAN: Password PostgreSQL hanya diproses SAAT PERTAMA container dibuat.
# Jika sudah pernah deploy lalu mengubah password di .env, jalankan:
# docker compose down -v && docker volume rm sicabe_postgres-data
# untuk menghapus volume lama.
cat > "${SCRIPT_DIR}/.env" <<EOF
# eBdesk SiCABE — Environment File (auto-generated by install.sh)
# Override nilai di bawah sesuai kebutuhan production.
# PostgreSQL: 1 instance shared untuk semua service (beda database)
POSTGRES_PASSWORD=$(openssl rand -hex 16)
KEYCLOAK_DB_PASSWORD=$(openssl rand -hex 16)
NESSIE_DB_PASSWORD=$(openssl rand -hex 16)
AIRFLOW_DB_PASSWORD=$(openssl rand -hex 16)
OPENMETADATA_DB_PASSWORD=$(openssl rand -hex 16)
KEYCLOAK_ADMIN_PASSWORD=$(openssl rand -hex 8)
MINIO_ROOT_PASSWORD=$(openssl rand -hex 16)
AIRFLOW_ADMIN_PASSWORD=$(openssl rand -hex 8)
AIRFLOW_SECRET_KEY=$(openssl rand -hex 32)
TRINO_SVC_SEED_PASSWORD=$(openssl rand -hex 8)
TRINO_SVC_DBT_PASSWORD=$(openssl rand -hex 8)
TRINO_SVC_AIRFLOW_PASSWORD=$(openssl rand -hex 8)
TRINO_SVC_OM_PASSWORD=$(openssl rand -hex 8)
GRAFANA_ADMIN_PASSWORD=$(openssl rand -hex 8)
# OM 2.0.x menegakkan kebijakan password (min 8 karakter + huruf besar + huruf
# kecil + digit + karakter spesial). `openssl rand -hex` hanya menghasilkan
# 0-9a-f sehingga TIDAK memenuhi kebijakan itu — password harus dirakit agar
# keempat kelas karakter dijamin ada.
OPENMETADATA_ADMIN_PASSWORD="$(openssl rand -hex 12)Aa1!"
LDAP_ADMIN_PASSWORD=$(openssl rand -hex 8)
LDAP_CONFIG_PASSWORD=$(openssl rand -hex 8)
# OIDC Client Secrets (harus sinkron dengan config .properties/.yaml files)
TRINO_CLIENT_SECRET=$(openssl rand -hex 16)
STARROCKS_CLIENT_SECRET=$(openssl rand -hex 16)
AIRFLOW_CLIENT_SECRET=$(openssl rand -hex 16)
OPENMETADATA_CLIENT_SECRET=$(openssl rand -hex 16)
GRAFANA_CLIENT_SECRET=$(openssl rand -hex 16)
EOF
chmod 600 "${SCRIPT_DIR}/.env"
echo -e "${OK} .env dibuat dengan password acak."
ENV_WAS_CREATED=1
fi
# ─── Load Default Environment ───────────────────────────────────────────────
source "${SCRIPT_DIR}/config/environment.defaults.sh"
# ─── Load .env jika ada ─────────────────────────────────────────────────────
if [[ -f "${SCRIPT_DIR}/.env" ]]; then
set -a
source "${SCRIPT_DIR}/.env"
set +a
echo -e "${INFO} Loaded overrides from .env"
fi
# ─── Fungsi Bantuan ──────────────────────────────────────────────────────────
usage() {
echo -e "${BOLD}Usage:${NC}"
echo " $0 # Deploy semua phase (1-6)"
echo " $0 --phase N # Deploy phase tertentu (1-6)"
echo " $0 --phase N-M # Deploy range phase (contoh: 1-3)"
echo " $0 --status [--host <nama>] # Cek status container (--host: pakai bundle host)"
echo " $0 --logs [service] # Lihat logs service (atau semua)"
echo " $0 --destroy # Hentikan dan hapus semua container"
echo " $0 --reset-ldap # Reset volume OpenLDAP (seed ulang user/grup)"
echo " $0 --restart [service] # Restart service tertentu"
echo ""
echo -e "${BOLD}Multi-host (Fase 1):${NC}"
echo " $0 --generate [--inventory F] [--blueprint a3|b5]"
echo " # Hasilkan generated/host-<x>/ dari cluster.yml"
echo " $0 --host <nama> [--dry-run] # Jalankan compose untuk satu host (di host itu!)"
echo " $0 --preflight [--inventory F]"
echo " # Cek keterjangkauan lintas host (tanpa deploy)"
echo " $0 --sync [--host <nama>] [--dest DIR] [--user U] [--delete] [--dry-run]"
echo " # Salin deploy/ (termasuk .env) ke host lain (dari control)"
echo " $0 --help # Tampilkan bantuan ini"
echo ""
echo -e "${BOLD}Phase:${NC}"
echo " 1 = Keycloak (Governance: IAM)"
echo " 2 = MinIO + Nessie (Storage)"
echo " 3 = Trino (Processing)"
echo " 4 = StarRocks (Serving)"
echo " 5 = Airflow (Ingestion)"
echo " 6 = OpenMetadata + Grafana (Governance: Metadata & Observability)"
echo ""
echo -e "${BOLD}Komponen existing (dikelola sistem lain):${NC}"
echo " Arahkan endpoint di .env → container komponen itu TIDAK dinyalakan:"
echo " KEYCLOAK_URL · LDAP_URL · S3_ENDPOINT · NESSIE_URL · TRINO_URL · OPA_URL"
echo " Override eksplisit (mis. Postgres milik DBA):"
echo " EXTERNAL_SERVICES=\"keycloak postgres\" LOCAL_SERVICES=\"minio\""
exit 0
}
# ─── Fungsi Prasyarat ─────────────────────────────────────────────────────────
check_prerequisites() {
echo -e "${INFO} Memeriksa prasyarat..."
# Docker
if ! command -v docker &>/dev/null; then
echo -e "${ERR} Docker tidak ditemukan. Install Docker terlebih dahulu."
echo " https://docs.docker.com/engine/install/"
exit 1
fi
echo -e "${OK} Docker: $(docker --version)"
# Docker Compose
if command -v docker-compose &>/dev/null; then
DOCKER_COMPOSE="docker-compose"
elif docker compose version &>/dev/null; then
DOCKER_COMPOSE="docker compose"
else
echo -e "${ERR} Docker Compose tidak ditemukan."
exit 1
fi
echo -e "${OK} Docker Compose: $(${DOCKER_COMPOSE} version 2>/dev/null || docker compose version)"
# mc (MinIO client) — opsional untuk bootstrap
if command -v mc &>/dev/null; then
echo -e "${OK} MinIO Client (mc): tersedia"
else
echo -e "${WARN} MinIO Client (mc) tidak ditemukan. Beberapa fitur bootstrap mungkin terbatas."
fi
# OpenSSL
if ! command -v openssl &>/dev/null; then
echo -e "${ERR} OpenSSL tidak ditemukan. Install openssl."
exit 1
fi
# curl
if ! command -v curl &>/dev/null; then
echo -e "${ERR} curl tidak ditemukan."
exit 1
fi
echo -e "${OK} curl tersedia"
# htpasswd (httpd-tools) — generate password.db service account Trino
if ! command -v htpasswd &>/dev/null; then
echo -e "${ERR} htpasswd tidak ditemukan (dibutuhkan untuk password service account Trino)."
echo " Install: Debian/Ubuntu → sudo apt-get install -y apache2-utils"
echo " RHEL/Fedora → sudo dnf install -y httpd-tools"
echo " macOS → brew install httpd"
exit 1
fi
echo -e "${OK} htpasswd: tersedia"
# python3 (untuk JSON parsing di bootstrap.sh & test-verification.sh)
if ! command -v python3 &>/dev/null; then
echo -e "${WARN} python3 tidak ditemukan. Beberapa fungsi bootstrap dan test mungkin terbatas."
echo " Install python3 atau pastikan tersedia di PATH."
else
echo -e "${OK} python3: tersedia"
fi
# Pastikan direktori config ada
for dir in config/keycloak config/trino/catalog config/starrocks config/grafana/datasources config/grafana/dashboards config/airflow config/nessie config/opa config/openldap; do
if [[ ! -d "${SCRIPT_DIR}/${dir}" ]]; then
echo -e "${WARN} Direktori ${dir} tidak ditemukan, membuat..."
mkdir -p "${SCRIPT_DIR}/${dir}"
fi
done
echo -e "${OK} Semua prasyarat terpenuhi."
echo ""
}
# ─── Generate password.db service account Trino (bcrypt) ─────────────────────
# Pra-syarat: htpasswd. Harus dijalankan SEBELUM Trino start (bind-mount file
# harus ada; jika tidak, Docker membuat DIRECTORY di path tsb).
generate_trino_password_db() {
local pw_file="${SCRIPT_DIR}/config/trino/password.db"
# Wajib FILE (bukan direktori): Docker membuat DIRECTORY bila sumber bind-mount
# tidak ada saat `up`. `sicabe_ensure_file_mount` membuang sisa direktori & membuat file.
sicabe_ensure_file_mount "${pw_file}" 644
local pair user pass
for pair in \
"svc_seed:${TRINO_SVC_SEED_PASSWORD:?}" \
"svc_dbt:${TRINO_SVC_DBT_PASSWORD:?}" \
"svc_airflow:${TRINO_SVC_AIRFLOW_PASSWORD:?}" \
"svc_om:${TRINO_SVC_OM_PASSWORD:?}"; do
user="${pair%%:*}"; pass="${pair#*:}"
htpasswd -nbB -C 10 "${user}" "${pass}" | sed '/^$/d' >> "${pw_file}"
done
chmod 644 "${pw_file}"
echo -e "${OK} password.db di-generate: $(cut -d: -f1 "${pw_file}" | paste -sd, -)"
}
# ─── Deteksi komponen EXISTING (dikelola sistem lain) ────────────────────────
# Prinsip: docker-compose.yml hanya untuk komponen yang MEMANG dijalankan di sini.
# Komponen yang endpoint-nya sudah diarahkan ke luar tidak dinyalakan, sehingga
# tidak ada container kembar / bentrok port dengan sistem yang sudah ada.
# Lihat config/environment.defaults.sh untuk tabel variabelnya.
# Nilai "lokal" (sesuai docker-compose.yml) untuk tiap komponen.
local_endpoint_of() {
case "$1" in
keycloak) echo "http://localhost:${KEYCLOAK_PORT:-12080}" ;;
openldap) echo "ldap://openldap:389" ;;
minio) echo "http://minio:9000" ;;
nessie) echo "http://nessie:19120" ;;
trino) echo "https://localhost:${TRINO_HTTPS_PORT:-12443}" ;;
opa) echo "http://localhost:${OPA_PORT:-12181}" ;;
*) echo "" ;;
esac
}
# Endpoint yang sedang dikonfigurasi (.env / defaults).
configured_endpoint_of() {
case "$1" in
keycloak) echo "${KEYCLOAK_URL:-}" ;;
openldap) echo "${LDAP_URL:-}" ;;
minio) echo "${S3_ENDPOINT:-}" ;;
nessie) echo "${NESSIE_URL:-}" ;;
trino) echo "${TRINO_URL:-}" ;;
opa) echo "${OPA_URL:-}" ;;
*) echo "" ;;
esac
}
# Daftar service yang dikelola EKSTERNAL (spasi-separated, unik).
# Nama LOGIS -> nama service sebenarnya.
#
# Filter komponen eksternal mencocokkan NAMA SERVICE (`starrocks-fe`, `trino-worker`,
# …). Tanpa pemetaan ini, menulis nama logis di EXTERNAL_SERVICES/LOCAL_SERVICES
# diam-diam TIDAK berefek: `EXTERNAL_SERVICES=starrocks` tidak melewati apa pun, dan
# `LOCAL_SERVICES=starrocks` tidak memaksa apa pun. Dulu hanya `trino` yang istimewa,
# sehingga orang yang menulis `starrocks` merasa setelannya diabaikan.
expand_service_names() { # <nama-logis|nama-service>...
local n out=""
for n in "$@"; do
case "${n}" in
trino) out+=" trino-coordinator trino-worker " ;;
starrocks) out+=" starrocks-fe starrocks-cn " ;;
airflow) out+=" airflow-init airflow-webserver airflow-scheduler airflow-dag-processor " ;;
openmetadata) out+=" openmetadata-server openmetadata-opensearch " ;;
*) out+=" ${n} " ;;
esac
done
printf '%s' "${out# }"
}
external_services() {
local out=" $(expand_service_names ${EXTERNAL_SERVICES:-}) " svc cfg local_val
for svc in keycloak openldap minio nessie trino opa; do
cfg="$(configured_endpoint_of "${svc}")"
local_val="$(local_endpoint_of "${svc}")"
if [[ -n "${cfg}" && -n "${local_val}" && "${cfg}" != "${local_val}" ]]; then
out+=" ${svc} "
fi
done
# S3 eksternal ⇒ minio-init (pembuat bucket lokal) tidak berlaku.
if [[ "${out}" == *" minio "* ]]; then
out+=" minio-init "
fi
# Trino kini DUA service (coordinator + worker): nama logis `trino` dipetakan ke
# keduanya agar deteksi eksternal benar-benar melewati container-nya.
if [[ "${out}" == *" trino "* ]]; then
out=" $(echo "${out}" | sed 's/ trino / /') "
out+=" trino-coordinator trino-worker "
fi
# StarRocks: FE & CN dikelola sistem lain bila hostnya bukan localhost.
if [[ -n "${STARROCKS_HOST:-}" && "${STARROCKS_HOST}" != "localhost" && "${STARROCKS_HOST}" != "127.0.0.1" ]]; then
out+=" starrocks-fe starrocks-cn "
fi
# Airflow: seluruh service fase 5 dikelola sistem lain bila hostnya bukan localhost.
if [[ -n "${AIRFLOW_HOST:-}" && "${AIRFLOW_HOST}" != "localhost" && "${AIRFLOW_HOST}" != "127.0.0.1" ]]; then
out+=" airflow-init airflow-webserver airflow-dag-processor airflow-scheduler "
fi
# OpenMetadata / Grafana: dikelola sistem lain bila URL host-nya bukan localhost.
if [[ -n "${OPENMETADATA_URL:-}" && "${OPENMETADATA_URL}" != "http://localhost:${OPENMETADATA_PORT:-12085}" ]]; then
out+=" openmetadata-opensearch openmetadata-migrate openmetadata-server "
fi
if [[ -n "${GRAFANA_URL:-}" && "${GRAFANA_URL}" != "http://localhost:${GRAFANA_PORT:-12300}" ]]; then
out+=" grafana "
fi
# LOCAL_SERVICES: paksa komponen tertentu tetap dijalankan lokal.
# Nama logis (`trino`, `starrocks`, `airflow`, `openmetadata`) diuraikan oleh
# expand_service_names() supaya cocok dengan nama service yang sebenarnya.
local s
for s in $(expand_service_names ${LOCAL_SERVICES:-}); do
out=" $(echo "${out}" | sed "s/ ${s} / /g") "
done
# PENTING (regresi e099b95): JANGAN pakai `grep -v '^$'` di sini. Saat TIDAK ada
# service eksternal, input hanya baris kosong → grep tidak menemukan baris cocok →
# exit 1 → dengan `set -euo pipefail` assignment `EXT_SVCS="$(external_services)"`
# MEMATIKAN install.sh tepat setelah banner (tanpa pesan error). `awk 'NF'` setara
# (buang baris kosong) tetapi SELALU exit 0.
echo "${out}" | tr ' ' '\n' | awk 'NF' | sort -u | paste -sd' ' -
}
is_external() {
[[ " $(external_services) " == *" $1 "* ]]
}
# Service yang harus dinyalakan untuk sebuah fase:
# kandidat dari profil compose DIKURANGI komponen eksternal.
phase_services() {
local phase="$1" svc out=""
local ext=" $(external_services) "
while read -r svc; do
[[ -z "${svc}" ]] && continue
if [[ "${ext}" == *" ${svc} "* ]]; then
continue
fi
out+=" ${svc}"
done < <(${DOCKER_COMPOSE} -f "${SCRIPT_DIR}/docker-compose.yml" --profile "phase${phase}" config --services 2>/dev/null)
echo "${out# }"
}
# Naikkan container satu fase.
# - Tanpa komponen eksternal: perilaku lama (`--profile phaseN up -d`) supaya
# depends_on tetap mengatur urutan + menunggu healthcheck.
# - Dengan komponen eksternal: hanya service lokal, `--no-deps` agar depends_on
# tidak menyeret service yang dikelola sistem lain. Postgres (bila lokal)
# dinaikkan lebih dulu supaya dependennya tidak menabrak DB yang belum siap.
deploy_phase_compose() {
local phase="$1" svcs
svcs="$(phase_services "${phase}")"
# Dir bind-mount yang absen di git (config/airflow/sling hanya env.yaml gitignored):
# bila tidak ada, Docker (root) membuatnya sebagai ROOT → operator non-root gagal
# menulis env.yaml lewat bootstrap. Pastikan ada & user-owned sebelum `up`.
mkdir -p "${SCRIPT_DIR}/config/airflow/sling"
if [[ -z "$(external_services)" ]]; then
${DOCKER_COMPOSE} -f "${SCRIPT_DIR}/docker-compose.yml" --profile "phase${phase}" up -d
return 0
fi
if [[ -z "${svcs}" ]]; then
echo -e "${INFO} Semua komponen fase ${phase} dikelola eksternal — tidak ada container yang dinyalakan."
return 0
fi
echo -e "${INFO} Mode existing — container fase ${phase} yang dinyalakan: ${BOLD}${svcs}${NC}"
if [[ " ${svcs} " == *" postgres "* ]]; then
${DOCKER_COMPOSE} -f "${SCRIPT_DIR}/docker-compose.yml" --profile all up -d --no-deps postgres
wait_for_container "sicabe-postgres" 60
fi
# shellcheck disable=SC2086
${DOCKER_COMPOSE} -f "${SCRIPT_DIR}/docker-compose.yml" --profile all up -d --no-deps ${svcs}
}
# Tunggu komponen: bila eksternal, tunggu endpoint eksternalnya (bukan localhost).
wait_component() {
local svc="$1" local_url="$2" timeout="$3" ext_url="${4:-}"
if is_external "${svc}"; then
if [[ "${ext_url}" == http* ]]; then
echo -e "${INFO} ${svc} dikelola eksternal — menunggu ${ext_url}"
wait_for_service "${svc} (eksternal)" "${ext_url}" "${timeout}"
else
echo -e "${INFO} ${svc} dikelola eksternal — lewati penantian (diverifikasi gate fase)"
fi
return 0
fi
wait_for_service "${svc}" "${local_url}" "${timeout}"
}
# ─── Fungsi Deploy per Phase ─────────────────────────────────────────────────
deploy_phase() {
local phase="$1"
local compose_file="${SCRIPT_DIR}/docker-compose.yml"
case "${phase}" in
1)
echo -e "${STEP} Phase 1${NC} — Governance (IAM): Keycloak"
deploy_phase_compose 1
wait_component "keycloak" "http://localhost:${KEYCLOAK_PORT:-12080}/health/ready" 120 \
"${KEYCLOAK_URL:-}/health/ready"
if is_external openldap; then
echo -e "${INFO} OpenLDAP dikelola eksternal (LDAP_URL=${LDAP_URL}) — lewati penantian container"
else
wait_for_container "sicabe-openldap" 60
fi
echo -e "${OK} Phase 1 selesai — Keycloak ${KEYCLOAK_URL}, OpenLDAP ${LDAP_URL}"
;;
2)
echo -e "${STEP} Phase 2${NC} — Storage: MinIO + Nessie"
deploy_phase_compose 2
wait_component "minio" "http://localhost:${MINIO_PORT:-12900}/minio/health/live" 60
wait_component "nessie" "http://localhost:${NESSIE_PORT:-12120}/api/v2/config" 90 \
"http://${NESSIE_HOST:-localhost}:${NESSIE_PORT:-12120}/api/v2/config"
echo -e "${OK} Phase 2 selesai — object storage ${S3_ENDPOINT}, Nessie ${NESSIE_URL}"
;;
3)
echo -e "${STEP} Phase 3${NC} — Processing: Trino + OPA"
if is_external trino; then
echo -e "${INFO} Trino dikelola eksternal (${TRINO_URL}) — password.db lokal tidak dibuat/dipakai"
else
generate_trino_password_db
fi
deploy_phase_compose 3
wait_component "trino" "https://localhost:${TRINO_HTTPS_PORT:-12443}/v1/info" 90 \
"${TRINO_URL:-}/v1/info"
echo -e "${OK} Phase 3 selesai — Trino ${TRINO_URL} (OAuth2 + Basic) · OPA ${OPA_URL}"
;;
4)
echo -e "${STEP} Phase 4${NC} — Serving: StarRocks ${BOLD}${NC}"
deploy_phase_compose 4
if is_external starrocks-fe; then
echo -e "${INFO} StarRocks dikelola eksternal — lewati penantian start"
else
echo -e "${INFO} StarRocks FE membutuhkan waktu ~2 menit untuk start..."
sleep 30
fi
echo -e "${OK} Phase 4 selesai — StarRocks FE :${STARROCKS_FE_PORT:-12930}, CN (shared-data)"
;;
5)
echo -e "${STEP} Phase 5${NC} — Ingestion: Airflow ${BOLD}${NC}"
# SimpleAuthManager membuka file ini dengan mode a+ → wajib ADA & writable (666).
# File ini TIDAK di-track git (lihat .gitignore) → generate dari .env di sini
# agar Docker tidak membuat bind-mount file ini sebagai DIRECTORY (fresh clone).
if is_external airflow-webserver; then
echo -e "${INFO} Airflow dikelola eksternal — lewati pembuatan file auth lokal"
else
local pw_file="${SCRIPT_DIR}/config/airflow/simple_auth_manager_passwords.json"
sicabe_ensure_file_mount "${pw_file}" 666
printf '{"admin": "%s"}\n' "${AIRFLOW_ADMIN_PASSWORD:?AIRFLOW_ADMIN_PASSWORD belum diset di .env}" > "${pw_file}"
chmod 666 "${pw_file}"
fi
deploy_phase_compose 5
wait_component "airflow-webserver" "http://localhost:${AIRFLOW_PORT:-12082}/api/v2/monitor/health" 120
echo -e "${OK} Phase 5 selesai — Airflow siap di http://localhost:${AIRFLOW_PORT:-12082}"
;;
6)
echo -e "${STEP} Phase 6${NC} — Governance (Lanjutan): OpenMetadata + Grafana ${BOLD}${NC}"
deploy_phase_compose 6
wait_component "openmetadata-server" "http://localhost:${OPENMETADATA_PORT:-12085}/health" 180
wait_component "grafana" "http://localhost:${GRAFANA_PORT:-12300}/api/health" 60
echo -e "${OK} Phase 6 selesai — OpenMetadata :${OPENMETADATA_PORT:-12085}, Grafana :${GRAFANA_PORT:-12300}"
;;
*)
echo -e "${ERR} Phase tidak valid: ${phase}. Gunakan 1-6."
exit 1
;;
esac
}
# ─── Status ──────────────────────────────────────────────────────────────────
show_status() {
# `--status [--host <nama>]`: dengan --host memakai bundle host itu
# (compose + .env hasil generate) sehingga endpoint menunjuk ALAMAT HOST,
# bukan localhost (perbaikan bug multi-host).
local st_host="${1:-}"
local compose_file="${SCRIPT_DIR}/docker-compose.yml"
if [[ -n "${st_host}" ]]; then
compose_file="${SCRIPT_DIR}/generated/host-${st_host}/docker-compose.yml"
local henv="${SCRIPT_DIR}/generated/host-${st_host}/.env"
if [[ ! -f "${compose_file}" || ! -f "${henv}" ]]; then
echo -e "${ERR} Host '${st_host}' belum digenerate (${compose_file})."
echo " Jalankan dulu: ./install.sh --generate"
return 1
fi
set -a; source "${henv}"; set +a
fi
echo -e "${BOLD}${BLUE}═══════════════════════════════════════════════════════${NC}"
echo -e "${BOLD} STATUS KOMPONEN eBdesk SiCABE${NC}"
[[ -n "${st_host}" ]] && echo -e "${BOLD} (host: ${st_host})${NC}"
echo -e "${BOLD}${BLUE}═══════════════════════════════════════════════════════${NC}"
echo ""
if ! ${DOCKER_COMPOSE} -f "${compose_file}" ps &>/dev/null; then
echo -e "${INFO} Tidak ada container SiCABE yang berjalan."
return
fi
echo -e "${BOLD}Container Status:${NC}"
${DOCKER_COMPOSE} -f "${compose_file}" ps --all 2>/dev/null || \
docker ps --filter "name=sicabe-" --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"
echo ""
echo -e "${BOLD}Endpoint Access:${NC}"
# Alamat dasar: alamat host (mode --host) atau localhost (single-host).
local ea="localhost"
[[ -n "${st_host}" ]] && ea="${SICABE_HOST_ADDRESS:-localhost}"
# `*_PUBLIC_URL` (diisi generator) MENANG bila ada. Untuk layanan tanpa URL publik,
# ambil HOST PEMILIK dari variabel endpoint bundle (mode --host), jatuh ke ea.
local h_kc="$ea" h_minio="$ea" h_sr="$ea" h_afs="$ea" h_om="$ea" h_ldap="$ea" h_opa="$ea" h_nessie="$ea"
if [[ -n "${st_host}" ]]; then
h_kc="${KEYCLOAK_HOST:-$ea}"
h_minio="${MINIO_HOST:-$ea}"
h_sr="${STARROCKS_FE_HOST:-$ea}"
h_afs="${AIRFLOW_WEBSERVER_HOST:-$ea}"
h_om="${OPENMETADATA_HOST:-$ea}"
if [[ -n "${NESSIE_URL:-}" ]]; then h_nessie="${NESSIE_URL#*://}"; h_nessie="${h_nessie%%[:/]*}"; fi
if [[ -n "${OPA_URL:-}" ]]; then h_opa="${OPA_URL#*://}"; h_opa="${h_opa%%[:/]*}"; fi
if [[ -n "${LDAP_URL:-}" ]]; then h_ldap="${LDAP_URL#*://}"; h_ldap="${h_ldap%%[:/]*}"; fi
fi
echo -e " Keycloak ${CYAN}${KEYCLOAK_PUBLIC_URL:-http://${h_kc}:${KEYCLOAK_PORT:-12080}}${NC}"
echo -e " OpenLDAP ${CYAN}ldap://${h_ldap}:${OPENLDAP_PORT:-12389}${NC}"
echo -e " MinIO API ${CYAN}http://${h_minio}:${MINIO_PORT:-12900}${NC}"
echo -e " MinIO Console ${CYAN}${MINIO_CONSOLE_PUBLIC_URL:-http://${h_minio}:${MINIO_CONSOLE_PORT:-12901}}${NC}"
echo -e " Nessie ${CYAN}http://${h_nessie}:${NESSIE_PORT:-12120}${NC}"
echo -e " Trino ${CYAN}${TRINO_PUBLIC_URL:-https://${ea}:${TRINO_HTTPS_PORT:-12443}}${NC} (OAuth2 + Basic, HTTPS-only)"
echo -e " OPA ${CYAN}http://${h_opa}:${OPA_PORT:-12181}${NC}"
echo -e " StarRocks FE ${CYAN}${h_sr}:${STARROCKS_FE_PORT:-12930}${NC} (MySQL protocol)"
echo -e " Dremio ${CYAN}http://${ea}:${DREMIO_UI_PORT:-12947}${NC}"
echo -e " Airflow ${CYAN}${AIRFLOW_PUBLIC_URL:-http://${h_afs}:${AIRFLOW_PORT:-12082}}${NC}"
echo -e " OpenMetadata ${CYAN}${OPENMETADATA_PUBLIC_URL:-http://${h_om}:${OPENMETADATA_PORT:-12085}}${NC}"
echo -e " Grafana ${CYAN}${GRAFANA_PUBLIC_URL:-http://${ea}:${GRAFANA_PORT:-12300}}${NC}"
}
# ─── Logs ─────────────────────────────────────────────────────────────────────
show_logs() {
local compose_file="${SCRIPT_DIR}/docker-compose.yml"
if [[ $# -eq 0 ]]; then
${DOCKER_COMPOSE} -f "${compose_file}" logs --tail=50 -f
else
${DOCKER_COMPOSE} -f "${compose_file}" logs --tail=100 -f "$@"
fi
}
# ─── Destroy ─────────────────────────────────────────────────────────────────
destroy() {
echo -e "${YELLOW}${BOLD}⚠️ PERINGATAN: Ini akan menghapus semua container dan volume!${NC}"
echo -e "${YELLOW}Data di volume akan dihapus jika menggunakan --volumes.${NC}"
echo ""
echo -n "Lanjutkan? (y/N): "
read -r confirm
if [[ "${confirm}" != "y" && "${confirm}" != "Y" ]]; then
echo -e "${INFO} Dibatalkan."
exit 0
fi
local compose_file="${SCRIPT_DIR}/docker-compose.yml"
# destroy() tidak memanggil check_prerequisites → DOCKER_COMPOSE bisa belum
# di-set. Deteksi di sini agar tidak "unbound variable" (set -u).
if [[ -z "${DOCKER_COMPOSE:-}" ]]; then
if docker compose version &>/dev/null; then
DOCKER_COMPOSE="docker compose"
elif docker-compose version &>/dev/null; then
DOCKER_COMPOSE="docker-compose"
else
echo -e "${ERR} Docker Compose tidak ditemukan."
exit 1
fi
fi
echo -e "${INFO} Menghentikan container..."
# Semua service di balik `profiles:` → WAJIB `--profile all`, jika tidak
# `down` tidak menyentuh container apa pun (no-op).
${DOCKER_COMPOSE} -f "${compose_file}" --profile all down --remove-orphans 2>/dev/null || true
echo ""
echo -n "Hapus semua volume data? (y/N): "
read -r confirm_vol
if [[ "${confirm_vol}" == "y" || "${confirm_vol}" == "Y" ]]; then
echo -e "${INFO} Menghapus volume..."
${DOCKER_COMPOSE} -f "${compose_file}" --profile all down -v 2>/dev/null || true
docker volume prune -f --filter "label=com.docker.compose.project=sicabe" 2>/dev/null || true
fi
echo -e "${OK} Destroy selesai."
}
# ─── Reset LDAP ──────────────────────────────────────────────────────────────
# Menghapus volume OpenLDAP agar config/openldap/seed.ldif di-apply ulang oleh
# bootstrap.sh. Diperlukan saat nama user/grup berubah, karena seeding di-skip
# bila OU users sudah ada (idempotent).
reset_ldap() {
local compose_file="${SCRIPT_DIR}/docker-compose.yml"
local project="${COMPOSE_PROJECT_NAME:-sicabe}"
if [[ -z "${DOCKER_COMPOSE:-}" ]]; then
if docker compose version &>/dev/null; then DOCKER_COMPOSE="docker compose"
elif docker-compose version &>/dev/null; then DOCKER_COMPOSE="docker-compose"
else echo -e "${ERR} Docker Compose tidak ditemukan."; exit 1; fi
fi
echo -n "Reset OpenLDAP (hapus volume user+group, seed ulang via bootstrap)? (y/N): "
read -r confirm
if [[ "${confirm}" != "y" && "${confirm}" != "Y" ]]; then
echo -e "${INFO} Dibatalkan."
exit 0
fi
echo -e "${INFO} Menghentikan OpenLDAP..."
${DOCKER_COMPOSE} -f "${compose_file}" --profile all stop openldap 2>/dev/null || true
echo -e "${INFO} Menghapus volume ${project}_openldap-data & ${project}_openldap-config..."
docker volume rm "${project}_openldap-data" "${project}_openldap-config" 2>/dev/null || true
echo -e "${INFO} Menjalankan ulang OpenLDAP..."
${DOCKER_COMPOSE} -f "${compose_file}" --profile all up -d openldap 2>/dev/null || true
echo -e "${OK} OpenLDAP di-reset. Jalankan './bootstrap.sh' (Step 3) untuk seed ulang."
}
# ─── Main ────────────────────────────────────────────────────────────────────
main() {
local MODE="all"
local PHASE_START=1
local PHASE_END=6
local EXTRA_ARGS=()
if [[ $# -gt 0 ]]; then
# ── Perintah multi-host (Fase 1) — ditangani lebih dulu ──────────────────
# Catatan: `DOCKER_COMPOSE` baru di-set oleh check_prerequisites(), jadi perintah
# yang butuh Docker memanggilnya lebih dulu; `--generate` justru TIDAK butuh Docker
# (hanya python3+PyYAML) sehingga tidak dipaksa.
case "${1:-}" in
--generate)
local inv="${SCRIPT_DIR}/cluster.yml" bp=""
shift
while [[ $# -gt 0 ]]; do
case "$1" in
--inventory) inv="${2:-}"; shift 2 ;;
--blueprint) bp="${2:-}"; shift 2 ;;
*) echo -e "${ERR} Argumen tidak dikenal untuk --generate: $1"; exit 1 ;;
esac
done
if ! command -v python3 &>/dev/null; then
echo -e "${ERR} python3 wajib untuk --generate."
exit 1
fi
if ! python3 -c 'import yaml' &>/dev/null; then
echo -e "${ERR} PyYAML wajib untuk --generate (pip install pyyaml)."
exit 1
fi
generate_hosts "${inv}" "${bp}" && exit 0
exit 1
;;
--preflight)
local pinv="${SCRIPT_DIR}/cluster.yml"
shift
while [[ $# -gt 0 ]]; do
case "$1" in
--inventory) pinv="${2:-}"; shift 2 ;;
*) echo -e "${ERR} Argumen tidak dikenal untuk --preflight: $1"; exit 1 ;;
esac
done
if [[ ! -f "${pinv}" ]]; then
echo -e "${ERR} Inventory tidak ditemukan: ${pinv}"
echo " Mulai dari contoh: cp cluster.example.yml cluster.yml"
exit 1
fi
python3 "${SCRIPT_DIR}/lib/cluster/preflight.py" "${pinv}" \
--compose "${SCRIPT_DIR}/docker-compose.yml" --deploy "${SCRIPT_DIR}"
exit $?
;;
--sync)
sync_hosts "$@"
exit $?
;;
--host)
local hname="${2:-}" hdry=0
if [[ -z "${hname}" ]]; then
echo -e "${ERR} Sebutkan nama host (lihat: --generate)."
exit 1
fi
shift 2
[[ "${1:-}" == "--dry-run" ]] && hdry=1
check_prerequisites # menyetel DOCKER_COMPOSE + memvalidasi Docker
run_host "${hname}" "${hdry}" && exit 0
exit 1
;;
esac
case "$1" in
--help|-h) usage ;;
--status)
check_prerequisites
local st_host=""
[[ "${2:-}" == "--host" ]] && st_host="${3:-}"
show_status "${st_host}"; exit 0 ;;
--logs) check_prerequisites; shift; show_logs "$@"; exit 0 ;;
--destroy) destroy; exit 0 ;;
--reset-ldap) reset_ldap; exit 0 ;;
--restart)
check_prerequisites
local compose_file="${SCRIPT_DIR}/docker-compose.yml"
if [[ $# -lt 2 ]]; then
echo -e "${ERR} Sebutkan service yang akan direstart."
echo " $0 --restart keycloak"
exit 1
fi
${DOCKER_COMPOSE} -f "${compose_file}" restart "${@:2}"
exit $?
;;
--phase)
if [[ $# -lt 2 ]]; then
echo -e "${ERR} Sebutkan phase (1-6 atau range 1-3)."
exit 1
fi
MODE="phase"
if [[ "$2" == *-* ]]; then
PHASE_START="${2%-*}"
PHASE_END="${2#*-}"
else
PHASE_START="$2"
PHASE_END="$2"
fi
shift 2
EXTRA_ARGS=("$@")
;;
*)
echo -e "${ERR} Argumen tidak dikenal: $1"
usage
;;
esac
fi
# ─── Mulai Deployment ──────────────────────────────────────────────────────
echo -e "${BOLD}${BLUE}"
echo "╔═══════════════════════════════════════════════════════╗"
echo "║ eBdesk SiCABE — Deployment Script ║"
echo "║ Arsitektur 5 Lapisan — Urutan Berdasar Dependensi ║"
echo "╚═══════════════════════════════════════════════════════╝"
echo -e "${NC}"
# Komponen yang dikelola sistem lain → container-nya TIDAK dinyalakan.
local EXT_SVCS
EXT_SVCS="$(external_services)"
if [[ -n "${EXT_SVCS}" ]]; then
echo -e "${INFO} Komponen dikelola EKSTERNAL: ${BOLD}${EXT_SVCS}${NC}"
echo " Container komponen tsb tidak dinyalakan; endpoint diambil dari .env."
echo " Verifikasi setelah bootstrap: ./scripts/verify-phase1-iam.sh dst."
echo ""
fi
check_prerequisites
if [[ "${MODE}" == "all" ]]; then
echo -e "${BOLD}Mode: Full Deployment (Semua Phase 1-6)${NC}"
echo ""
# Phase 1: Governance (IAM)
echo -e "${STEP}╔═══ Phase 1/6: Governance (IAM) — Keycloak ═══╗${NC}"
deploy_phase 1
# Phase 2: Storage
echo -e "${STEP}╔═══ Phase 2/6: Storage — MinIO → Nessie ═══╗${NC}"
deploy_phase 2
# Phase 3: Processing
echo -e "${STEP}╔═══ Phase 3/6: Processing — Trino ═══╗${NC}"
deploy_phase 3
# Phase 4: Serving
echo -e "${STEP}╔═══ Phase 4/6: Serving — StarRocks ═══╗${NC}"
deploy_phase 4
# Phase 5: Ingestion
echo -e "${STEP}╔═══ Phase 5/6: Ingestion — Airflow ═══╗${NC}"
deploy_phase 5
# Phase 6: Governance (Lanjutan)
echo -e "${STEP}╔═══ Phase 6/6: Governance (Lan) — OpenMetadata → Grafana ═══╗${NC}"
deploy_phase 6
echo ""
echo -e "${BOLD}${GREEN}═══════════════════════════════════════════════════════${NC}"
echo -e "${BOLD}${GREEN} ✅ SEMUA PHASE SELESAI${NC}"
echo -e "${BOLD}${GREEN} Jalankan ./bootstrap.sh untuk inisialisasi & integrasi${NC}"
echo -e "${BOLD}${GREEN} Jalankan ./test-verification.sh untuk verifikasi${NC}"
echo -e "${BOLD}${GREEN}═══════════════════════════════════════════════════════${NC}"
echo ""
elif [[ "${MODE}" == "phase" ]]; then
echo -e "${BOLD}Mode: Phase ${PHASE_START} sampai ${PHASE_END}${NC}"
for p in $(seq "${PHASE_START}" "${PHASE_END}"); do
deploy_phase "${p}"
done
echo -e "${OK} Phase ${PHASE_START}-${PHASE_END} selesai!"
fi
show_status
}
# ─── Multi-host (Fase 1, R10 §4.5) ───────────────────────────────────────────
# Membangkitkan artefak deploy per host ke `generated/host-<x>/`.
generate_hosts() {
local inv="$1" blueprint="$2" args=()
if [[ ! -f "${inv}" ]]; then
echo -e "${ERR} Inventory tidak ditemukan: ${inv}"
echo " Mulai dari contoh: cp cluster.example.yml cluster.yml (lalu sesuaikan address)"
return 1
fi
# Rahasia untuk SEMUA host diambil dari .env mesin ini (keputusan S1: satu .env
# bersama). Bila .env baru saja dibuat otomatis, rahasianya BARU dan harus menjadi
# satu-satunya sumber: distribusikan generated/ ke semua host, dan JANGAN membuat
# .env terpisah di tiap host (client secret OIDC akan tidak cocok -> login gagal).
if [[ "${ENV_WAS_CREATED:-0}" -eq 1 ]]; then
echo -e "${WARN} .env BARU SAJA dibuat di mesin ini (password acak)."
echo " .env ini adalah sumber rahasia untuk SEMUA host."
echo " → distribusikan hasil generated/ (di dalamnya rahasia sudah tertanam) ke tiap host;"
echo " → JANGAN membuat .env terpisah di host lain (OIDC client secret harus identik)."
fi
[[ -n "${blueprint}" ]] && args+=("${blueprint}")
echo -e "${INFO} Membangkitkan artefak per host dari ${inv}..."
python3 "${SCRIPT_DIR}/lib/cluster/generate.py" render \
"${inv}" "${SCRIPT_DIR}/docker-compose.yml" "${SCRIPT_DIR}/.env" \
"${SCRIPT_DIR}/generated" "${SCRIPT_DIR}/scripts/render-config.sh" "${args[@]}"
}
# Menjalankan compose untuk SATU host dari hasil generate.
run_host() {
# Catatan bash: JANGAN menggabung `local name="$1" ... dir="...${name}"` dalam satu
# baris — dengan `set -u`, bash meng-expand seluruh kata sebelum assignment sehingga
# `${name}` masih unbound ("name: unbound variable"). Pisahkan per baris.
local name="$1"
local dry="$2"
local dir="${SCRIPT_DIR}/generated/host-${name}"
if [[ ! -d "${dir}" ]]; then
echo -e "${ERR} Host '${name}' belum digenerate (tidak ada ${dir})."
echo " Jalankan dulu: $0 --generate [--inventory cluster.yml]"
return 1
fi
# Guard: berkas runtime bind-mount WAJIB FILE (Docker membuat DIRECTORY bila sumber
# hilang → mis. Airflow IsADirectoryError). Isi ditulis setelah .env host di-source.
sicabe_ensure_file_mount "${dir}/config/trino/password.db" 644
echo -e "${WARN} Jalankan ini HANYA di host '${name}'."
echo -e "${WARN} Pilot berdampingan di host yang SAMA tidak cukup dipisah network/port:"
echo -e "${WARN} RAM tidak cukup untuk dua stack — matikan stack lama lebih dulu (README)."
# ── PRECEDENCE (bug nyata yang dilaporkan dari server) ──────────────────────
# Compose memberi prioritas pada variabel **SHELL** di atas `--env-file`. Skrip ini
# sudah men-source config/environment.defaults.sh, yang mengekspor
# POSTGRES_HOST=postgres (default single-host) dan kawan-kawan. Akibatnya nilai dari
# bundle host KALAH, container mencari hostname `postgres` yang hanya ada di host
# lain -> java.net.UnknownHostException: postgres, walaupun --env-file sudah benar.
# Karena itu .env host di-source lebih dulu (set -a) supaya nilai host yang menang.
set -a
# shellcheck disable=SC1091
. "${dir}/.env"
set +a
# Berkas runtime bind-mount WAJIB FILE (Docker membuat DIRECTORY bila sumber hilang →
# mis. Airflow SimpleAuthManager IsADirectoryError). Bundle normal sudah memuatnya;
# jaring pengaman ini menulis ulang isi dari .env host.
local _pwp="${dir}/config/airflow/simple_auth_manager_passwords.json"
sicabe_ensure_file_mount "${_pwp}" 666
printf '{"admin": "%s"}\n' "${AIRFLOW_ADMIN_PASSWORD:?AIRFLOW_ADMIN_PASSWORD tidak ada di ${dir}/.env}" > "${_pwp}"
chmod 666 "${_pwp}"
# ── INGRESS (T-20) ─────────────────────────────────────────────────────────
# Service `traefik` berada di profile `edge`. Bila bundle host ini memuatnya
# (artinya tls.mode != none), profile itu WAJIB ikut: tanpa itu Traefik tidak pernah
# start sehingga semua rute publik tidak ada — walaupun label sudah terpasang di
# tiap layanan dan sertifikat sudah bisa diterbitkan. Dideteksi dari BERKAS bundle,
# bukan dari variabel, supaya tidak bisa meleset antara mode none dan edge.
local profiles="--profile all"
if grep -qE '^ traefik:' "${dir}/docker-compose.yml" 2>/dev/null; then
profiles="--profile all --profile edge"
fi
if [[ "${dry}" == "1" ]]; then
echo -e "${INFO} (dry-run) perintah yang akan dijalankan:"
echo " ${DOCKER_COMPOSE} -f ${dir}/docker-compose.yml --env-file ${dir}/.env ${profiles} up -d"
return 0
fi
${DOCKER_COMPOSE} -f "${dir}/docker-compose.yml" --env-file "${dir}/.env" ${profiles} up -d
}
# ─── Multi-host: distribusi bundle ke host lain (Fase 1) ─────────────────────
# Menyalin direktori deploy/ (TERMASUK .env + generated/) ke host lain via rsync
# over SSH. Dijalankan di host CONTROL (master installer). Target = semua host di
# cluster.yml KECUALI host ini (dideteksi dari kecocokan alamat lokal) — atau satu
# host saja via --host. Aman diulang (rsync idempoten).
sync_hosts() {
local inv="${SCRIPT_DIR}/cluster.yml" dest="~/sicabe-doc/deploy" user="${USER:-root}" \
only_host="" do_delete=0 dry=0
shift
while [[ $# -gt 0 ]]; do
case "$1" in
--inventory) inv="${2:-}"; shift 2 ;;
--dest) dest="${2:-}"; shift 2 ;;
--user) user="${2:-}"; shift 2 ;;
--host) only_host="${2:-}"; shift 2 ;;
--delete) do_delete=1; shift ;;
--dry-run) dry=1; shift ;;
*) echo -e "${ERR} Argumen tidak dikenal untuk --sync: $1"; exit 1 ;;
esac
done
if [[ ! -f "${inv}" ]]; then
echo -e "${ERR} Inventory tidak ditemukan: ${inv}"
echo " Mulai dari contoh: cp cluster.example.yml cluster.yml"
exit 1
fi
if ! command -v rsync &>/dev/null; then
echo -e "${ERR} rsync wajib terpasang (mis. sudo apt-get install -y rsync)."
exit 1
fi
local pairs
pairs="$(python3 "${SCRIPT_DIR}/lib/cluster/inventory.py" hosts "${inv}" 2>/dev/null)" || {
echo -e "${ERR} Gagal membaca inventory (${inv})."; exit 1; }
if [[ -z "${pairs}" ]]; then
echo -e "${ERR} Inventory tidak memuat host."; exit 1
fi
# Tentukan host INI dari kecocokan alamat lokal; bila tak cocok → sinkron ke SEMUA.
local self="" lip n a
lip="$(hostname -I 2>/dev/null || true)"
while read -r n a; do
[[ -z "${n}" ]] && continue
[[ " ${lip} " == *" ${a} "* ]] && self="${n}"
done <<< "${pairs}"
# Kumpulkan target: kecuali host ini; hormati --host.
local targets=()
while read -r n a; do
[[ -z "${n}" ]] && continue
[[ -n "${self}" && "${n}" == "${self}" ]] && continue
[[ -n "${only_host}" && "${n}" != "${only_host}" ]] && continue
targets+=("${n}|${a}")
done <<< "${pairs}"
echo -e "${BOLD}${BLUE}Distribusi deploy/ ke host lain${NC}"
echo " sumber : ${SCRIPT_DIR}/"
echo " tujuan : ${user}@<host>:${dest}"
echo " host ini : ${self:-<tak dikenal — sinkron ke semua>}"
[[ ${do_delete} -eq 1 ]] && echo " mode : --delete (hapus berkas tujuan yang tak ada di sumber)"
echo ""
if [[ ${#targets[@]} -eq 0 ]]; then
echo -e "${WARN} Tidak ada host tujuan (host ini=${self:-?}; --host=${only_host:-—})."
return 0
fi
local t name addr okc=0 failc=0
for t in "${targets[@]}"; do
name="${t%%|*}"; addr="${t##*|}"
echo -e "${STEP} → ${name} (${user}@${addr}:${dest})${NC}"
local rargs=(-az --human-readable -e "ssh -o ConnectTimeout=8" --exclude '.git' --exclude 'output')
[[ ${do_delete} -eq 1 ]] && rargs+=(--delete)
[[ ${dry} -eq 1 ]] && rargs+=(--dry-run)
# Buat direktori tujuan bila belum ada (`~` diekspansi oleh shell REMOTE).
rargs+=(--rsync-path "mkdir -p ${dest} && rsync")
if rsync "${rargs[@]}" "${SCRIPT_DIR}/" "${user}@${addr}:${dest}/"; then
echo -e " ${OK} ${name}: tersinkron"
okc=$((okc + 1))
else
echo -e " ${ERR} ${name}: GAGAL — periksa SSH/rsync ke ${addr}" >&2
failc=$((failc + 1))
fi
done
echo ""
echo -e "${BOLD}Ringkasan${NC}"
echo " berhasil: ${okc} · gagal: ${failc}"
echo " Langkah berikutnya DI tiap host tujuan:"
echo " ./scripts/prefetch-images.sh --host <nama> && ./install.sh --host <nama>"
[[ ${failc} -eq 0 ]]
}
# ─── Jalankan ─────────────────────────────────────────────────────────────────
main "$@"
deploy/lib/cluster/generate.py
+899
View File
@@ -0,0 +1,899 @@
#!/usr/bin/env python3
"""generate.py — render artefak deploy per host dari docker-compose.yml + cluster.yml.
Fase 1 R10 §4.5 (**Varian A: render, bukan overlay asli Compose**). Alasan: merge
overlay tidak dapat MENGHAPUS service, sedangkan kita perlu menyisakan hanya
service milik satu host.
Yang dihasilkan `generated/host-<x>/`:
1. `docker-compose.yml` — hanya service milik host itu.
2. `.env` — endpoint menunjuk ALAMAT HOST pemilik komponen (B1) + seluruh isi
`.env` sumber (rahasia wajib identik lintas host, S1 R10 §4.12).
3. `config/` — hasil `render-config.sh` (hostname container → alamat host), karena
berkas di `config/**` tidak dapat melakukan ekspansi env saat runtime.
`depends_on` lintas host DIBUANG (penggantinya `preflight`, Fase 2); intra-host
dipertahankan supaya urutan + healthcheck tetap berlaku di dalam satu host.
Hasil di `generated/` bersifat ARTEFAK (gitignored) — jangan disunting tangan.
Perintah:
generate.py render <inventory> <base-compose> <base-env> <out-base> <render-config.sh>
"""
from __future__ import annotations
import json
import shutil
import subprocess
import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent))
import inventory # noqa: E402 (modul sebelah, sengaja)
try:
import yaml
except ImportError: # pragma: no cover
sys.stderr.write("PyYAML wajib tersedia\n")
raise SystemExit(2)
# Variabel endpoint (B1) → komponen pemiliknya. Dipakai untuk mengarahkan
# container ke ALAMAT HOST, bukan nama container (nama container hanya resolve
# di host yang sama).
ENDPOINT_VARS = {
"POSTGRES_HOST": "postgres",
"MINIO_HOST": "minio",
"KEYCLOAK_HOST": "keycloak",
"OPENMETADATA_HOST": "openmetadata-server",
"AIRFLOW_WEBSERVER_HOST": "airflow-webserver",
"STARROCKS_FE_HOST": "starrocks-fe",
"STARROCKS_CN_HOST": "starrocks-cn",
"OPENSEARCH_HOST": "openmetadata-opensearch",
# Var SISI-HOST yang belum ada padanannya, dipakai `scripts/verify-phase*` (mis. Trino
# & Grafana hanya punya *_PUBLIC_URL). Nilainya = alamat host pemilik. CATATAN:
# STARROCKS_HOST/AIRFLOW_HOST sengaja TIDAK ditulis di sini — skrip memakainya sebagai
# penanda komponen EXTERNAL (default `localhost`); mengisinya dengan alamat host akan
# salah menandai komponen lokal sebagai external.
"TRINO_HOST": "trino-coordinator",
# Dipakai dbt (profiles.yml: env_var DBT_TRINO_HOST) — lintas-host butuh ALAMAT IP.
"DBT_TRINO_HOST": "trino-coordinator",
"GRAFANA_HOST": "grafana",
"LDAP_HOST": "openldap",
"NESSIE_HOST": "nessie",
}
# Sebagian konsumen memakai bentuk URL (bukan `*_HOST`), dan berkas config yang
# dibaca container TIDAK dapat ekspansi env → URL-nya harus ikut diarahkan di sini,
# kalau tidak Trino/Sling/StarRocks lintas host akan mencari nama container yang
# hanya ada di host lain. (Komponen pemilik tetap diambil dari inventory.)
URL_ENDPOINT_VARS = {
"S3_ENDPOINT": ("minio", "http://{addr}:{port}"),
"NESSIE_URL": ("nessie", "http://{addr}:{port}"),
"LDAP_URL": ("openldap", "ldap://{addr}:{port}"),
"OPA_URL": ("opa", "http://{addr}:{port}"),
}
# Port container untuk var URL di atas — bisa ditimpa di .env (mis. layanan EXISTING
# berport non-standar). Default = port standar → perilaku lama tidak berubah.
URL_ENDPOINT_PORTS = {
"S3_ENDPOINT": ("MINIO_INTERNAL_PORT", 9000),
"NESSIE_URL": ("NESSIE_INTERNAL_PORT", 19120),
"LDAP_URL": ("OPENLDAP_INTERNAL_PORT", 389),
"OPA_URL": ("OPA_INTERNAL_PORT", 8181),
}
# URL BROWSER-FACING saat tls.mode = none (tanpa proxy/TLS): dialamatkan ke HOST yang
# MENJALANKAN komponen, bukan ke localhost maupun ke IP host lokal. Nilainya diambil
# dari host pemilik sehingga SAMA di semua bundle — wajib, karena `iss`/issuer OIDC
# harus satu nilai; kalau tiap host menulis alamatnya sendiri, validator akan tolak.
# Port di sini mengikuti skema 12xxx (sama seperti URL_ENDPOINT_VARS di bawah).
PUBLIC_URL_ENDPOINTS = {
"KEYCLOAK_PUBLIC_URL": ("keycloak", "http://{addr}:12080"),
"GRAFANA_PUBLIC_URL": ("grafana", "http://{addr}:12300"),
"OPENMETADATA_PUBLIC_URL": ("openmetadata-server", "http://{addr}:12085"),
"AIRFLOW_PUBLIC_URL": ("airflow-webserver", "http://{addr}:12082"),
"TRINO_PUBLIC_URL": ("trino-coordinator", "https://{addr}:12443"),
"MINIO_CONSOLE_PUBLIC_URL": ("minio", "http://{addr}:12901"),
}
# SENGAJA TIDAK diarahkan ke IP: KEYCLOAK_URL / TRINO_URL dan var `*_URL` lain yang
# bersifat identitas (issuer OIDC harus stabil — lihat R10 §4.10.1: pakai DNS
# internal, bukan IP). Itu milik operator dan harus sudah benar di `cluster.yml`
# (domain) / `.env`.
# Hostname container yang dikenali render-config.sh → nama komponen inventory.
RENDER_HOSTS = (
"postgres", "minio", "nessie", "keycloak", "openldap", "opa",
"trino-coordinator", "trino-worker", "starrocks-fe", "starrocks-cn",
"airflow-webserver", "openmetadata-server", "openmetadata-opensearch",
"prometheus", "grafana", "dremio", "statsd-exporter", "trino-jmx-exporter",
)
# Catatan: co-location ditangani render_config_map() — komponen SE-HOST TIDAK ditulis ke
# alamat host (biar tetap nama container, satu jaringan Docker). Maka daftar ini boleh
# lengkap; yang penting hostname-nya ada di KNOWN_HOSTS render-config.sh.
# node_exporter = PRA-SYARAT HOST (dipasang di luar installer, port 9100). Semua
# host di cluster.yml dianggap menjalankannya; generate.py menulis daftar target
# file_sd (job `node`) ke bundle host pemilik Prometheus.
NODE_EXPORTER_PORT = 9100
def load_compose(path: str) -> dict:
with open(path, encoding="utf-8") as fh:
return yaml.safe_load(fh) or {}
def keep_only(services: dict, wanted: set[str]) -> dict:
return {name: spec for name, spec in services.items() if name in wanted}
def prune_depends_on(services: dict) -> list[str]:
"""Buang `depends_on` lintas host; kembalikan daftar yang dibuang."""
dropped: list[str] = []
for name, spec in services.items():
deps = spec.get("depends_on")
if deps is None:
continue
if isinstance(deps, dict):
gone = set(deps) - set(services)
kept: object = {k: v for k, v in deps.items() if k in services}
else:
gone = set(deps) - set(services)
kept = [d for d in deps if d in services]
dropped += [f"{name} → {g}" for g in sorted(gone)]
if kept:
spec["depends_on"] = kept
else:
spec.pop("depends_on", None)
return dropped
def used_volumes(services: dict) -> set[str]:
used: set[str] = set()
for spec in services.values():
for vol in spec.get("volumes", []) or []:
if isinstance(vol, str):
src = vol.split(":", 1)[0]
elif isinstance(vol, dict):
src = vol.get("source", "")
else:
continue
if src and not src.startswith((".", "/", "~")):
used.add(src)
return used
def build_compose(base: dict, wanted: set[str]) -> tuple[dict, list[str]]:
services = keep_only(base["services"], wanted)
dropped = prune_depends_on(services)
declared = base.get("volumes") or {}
out: dict = {"services": services}
if base.get("name"):
out["name"] = base["name"]
out["volumes"] = {n: s for n, s in declared.items() if n in used_volumes(services)}
for key in ("networks", "configs", "secrets"):
if key in base:
out[key] = base[key]
return out, dropped
def write_compose(path: Path, doc: dict) -> None:
path.parent.mkdir(parents=True, exist_ok=True)
with open(path, "w", encoding="utf-8") as fh:
fh.write("# DIBANGKITKAN OTOMATIS — jangan disunting tangan.\n")
fh.write("# Sumber: docker-compose.yml + cluster.yml · regenerasi: ./install.sh --generate\n")
yaml.safe_dump(doc, fh, sort_keys=False, default_flow_style=False, allow_unicode=True)
def _is_topology_var(line: str) -> bool:
"""Baris .env yang menjadi milik INVENTORY (bukan isi .env sumber).
Kalau tidak disaring, `.env` sumber ikut menyalin `EXTERNAL_SERVICES`/
`*_HOST` milik single-host → di dotenv yang MENANG adalah yang TERAKHIR,
sehingga nilai dari inventory bisa hilang diam-diam (bug yang ditemukan saat
uji: `EXTERNAL_SERVICES` muncul dua kali).
"""
key = line.split("=", 1)[0].strip()
owned = {
"EXTERNAL_SERVICES", "LOCAL_SERVICES", "KEYCLOAK_SSL_REQUIRED",
"KC_PROXY_HEADERS", "TRINO_PROCESS_FORWARDED", "CLUSTER_DOMAIN", "TRAEFIK_BIND_ADDR",
*ENDPOINT_VARS, *URL_ENDPOINT_VARS,
*inventory.PUBLIC_URL_SERVICES, # KEYCLOAK_PUBLIC_URL, GRAFANA_PUBLIC_URL, …
}
return key in owned or key.startswith("SICABE_HOST_")
def render_env(base_env: Path, out_env: Path, host: str, spec: dict, endpoints: list[str]) -> None:
lines: list[str] = []
if base_env.is_file():
for line in base_env.read_text(encoding="utf-8").splitlines():
if _is_topology_var(line):
continue # ditentukan inventory, bukan .env sumber
lines.append(line)
else:
lines.append("# (tidak ada .env sumber — rahasia wajib dilengkapi manual)")
lines += [
"",
"# ── Ditambahkan generator multi-host (Fase 1) — variabel topologi ──",
f"SICABE_HOST_NAME={host}",
f"SICABE_HOST_ADDRESS={spec['address']}",
f"SICABE_HOST_TIERS={','.join(spec['tiers'])}",
]
lines += endpoints
out_env.parent.mkdir(parents=True, exist_ok=True)
out_env.write_text("\n".join(lines) + "\n", encoding="utf-8")
# Port KONTAINER yang harus terjangkau lintas host (Fase 2). Variabel `*_HOST`
# hanya memuat NAMA host (tanpa port), jadi port diambil dari referensi compose =
# port container (5432, 8080, …). Selama port container itu tidak terbuka di alamat
# host, container di host lain tidak bisa menjangkau — jadi generator menambahkannya.
#
# Dikunci ke ALAMAT HOST (mis. `10.20.0.12:5432:5432`), bukan `0.0.0.0`, supaya hanya
# terjangkau lewat NIC privat host itu (R10 §4.8). Ini SENGAJA hanya di artefak
# multi-host: instalasi single-host tidak pernah membuka port internal ini, dan
# `docker-compose.yml` tetap byte-identik (invarian Fase 0 terjaga).
# Eksposur untuk manusia/klien tetap memakai port host `12xxx` yang sudah ada.
INTERNAL_PORTS: dict[str, list[int]] = {
"postgres": [5432],
"minio": [9000],
"nessie": [19120],
"keycloak": [8080],
"openldap": [389],
"opa": [8181],
# 8080 = worker mendaftar (discovery) · 8443 = klien lintas host (OM ingestion,
# task dbt, native DQ) — keduanya dibutuhkan begitu coordinator & worker beda host.
"trino-coordinator": [8080, 8443],
"starrocks-fe": [9030, 8030],
"starrocks-cn": [9050, 8040, 8060],
"openmetadata-opensearch": [9200],
"openmetadata-server": [8585],
"airflow-webserver": [8080],
# Sidecar JMX di host `coord`: Prometheus (di host `governance`) men-scrape 9404
# LINTAS host → portnya WAJIB diterbitkan di alamat host pemilik.
"trino-jmx-exporter": [9404],
}
def _reachable_on_host_port(entry, cport: int, address: str) -> bool:
"""Apakah `entry` benar-benar membuka port container `cport` DI alamat host?
Jebakan yang sempat lolos: `12080:8080` memetakan port host 12080 → container
8080. Itu TIDAK membuat port 8080 terjangkau di host, sehingga container di host
lain tetap tidak bisa menjangkau. Yang dihitung hanya pemetaan yang port host-nya
SAMA dengan port container (mis. `addr:8080:8080`).
"""
if isinstance(entry, str):
parts = entry.split(":")
if len(parts) == 2:
host_ip, host_port = None, parts[0]
elif len(parts) == 3:
host_ip, host_port = parts[0], parts[1]
else:
return False
if host_port != str(cport):
return False
return host_ip in (None, "", address)
if isinstance(entry, dict):
published = entry.get("published")
return (
entry.get("target") == cport
and str(published) == str(cport)
and entry.get("host_ip") in (None, "", address)
)
return False
def tier_implied_components(placement: dict, plan: dict) -> set[str]:
"""Komponen yang PASTI dikonsumsi lintas host walau tidak terlihat di env/config.
`trino-coordinator` selalu dipakai host bertier `pipeline` (task dbt Airflow) dan
`governance` (ingestion OpenMetadata). Referensinya tidak dapat ditemukan dengan
pemindaian env/config karena `ingest-trino.yaml` **disalin ke container saat
runtime** (docker cp), bukan di-mount — celah ini ditemukan saat menguji preflight.
"""
needed: set[str] = set()
owners = placement.get("trino-coordinator") or []
if not owners:
return needed
owner = owners[0]
for host, spec in plan.items():
if host == owner:
continue
if {"pipeline", "governance"} & set(spec["tiers"]):
needed.add("trino-coordinator")
break
return needed
def config_referenced_components(deploy_dir: Path, base: dict, plan: dict,
placement: dict, known: set[str]) -> set[str]:
"""Komponen yang dirujuk dari file CONFIG yang di-mount oleh service di host lain.
Deteksi berbasis environment saja tidak cukup: Trino tidak memakai `*_HOST` untuk
OAuth2, melainkan `config/trino/config-coordinator.properties`
(`http://keycloak:8080/...`). Tanpa pemindaian ini, port Keycloak tidak dibuka dan
Trino di host lain gagal mengambil JWKS.
"""
import re
needed: set[str] = set()
for host, spec in plan.items():
for svc in spec["services"]:
for vol in base["services"].get(svc, {}).get("volumes") or []:
src = vol.split(":", 1)[0] if isinstance(vol, str) else vol.get("source", "")
if not src.startswith("./config/"):
continue
path = deploy_dir / src[2:]
files = list(path.rglob("*")) if path.is_dir() else [path]
for f in files:
if not f.is_file():
continue
try:
text = f.read_text(encoding="utf-8", errors="ignore")
except OSError:
continue
for comp in known:
if re.search(r"(?<![\w.-])" + re.escape(comp) + r"(?![\w-])", text):
owners = placement.get(comp) or []
if owners and host not in owners:
needed.add(comp)
return needed
def cross_host_components(base: dict, placement: dict, plan: dict) -> set[str]:
"""Komponen yang PUNYA konsumen di host lain → perlu port internal terbuka.
Tanpa perhitungan ini generator membuka port untuk semua komponen, dan di
blueprint a3 itu langsung menabrak: `keycloak` dan `airflow-webserver` sama-sama
meminta `<control>:8080` (dua service, satu port host) → `compose up` gagal
'port is already allocated'. Padahal `airflow-webserver` hanya dikonsumsi
OpenMetadata yang berada di host yang sama → tidak perlu port host sama sekali.
Bonus keamanan: port internal yang tidak dibutuhkan tidak pernah dibuka.
"""
var_to_component = {v: c for v, c in ENDPOINT_VARS.items()}
var_to_component.update({v: c for v, (c, _t) in URL_ENDPOINT_VARS.items()})
needed: set[str] = set()
for host, spec in plan.items():
for svc in spec["services"]:
source = base["services"].get(svc, {})
text = json.dumps(source.get("environment") or {})
for var, component in var_to_component.items():
if var not in text:
continue
owners = placement.get(component) or []
if owners and host not in owners:
needed.add(component)
return needed
def required_cross_host(base: dict, deploy_dir: Path, plan: dict, placement: dict) -> set[str]:
"""Gabungan SEMUA alasan sebuah komponen butuh port lintas host.
Satu fungsi supaya generator dan preflight TIDAK bisa berbeda pendapat — memisahkan
perhitungan ini sempat membuat preflight melewatkan trino-coordinator begitu aturan
baru ditambahkan ke generator.
"""
known = set(base["services"]) & set(INTERNAL_PORTS)
return (
cross_host_components(base, placement, plan)
| config_referenced_components(deploy_dir, base, plan, placement, known)
| tier_implied_components(placement, plan)
)
def add_internal_ports(services: dict, address: str, needed: set[str]) -> list[str]:
"""Tambahkan binding port internal ke alamat host; kembalikan yang ditambahkan."""
added: list[str] = []
for name, spec in services.items():
if name not in needed:
continue
want = INTERNAL_PORTS.get(name)
if not want:
continue
ports = spec.setdefault("ports", [])
for cport in want:
if any(_reachable_on_host_port(e, cport, address) for e in ports):
continue
ports.append(f"{address}:{cport}:{cport}")
added.append(f"{name}:{cport}")
return added
def component_placements(doc: dict) -> dict[str, list[str]]:
return inventory.resolve(doc)
def extra_bind_dirs(base: dict) -> list[str]:
"""Bind-mount relatif selain `config/` yang harus ikut disalin ke tiap host.
Contoh: `./scripts/airflow:/opt/airflow/scripts:ro`. Tanpa disalin, direktori
generated tidak mandiri — Docker akan membuat direktori KOSONG di host tujuan
(skrip Airflow hilang diam-diam). Ditemukan oleh test-generate.sh.
"""
found: set[str] = set()
for spec in base["services"].values():
for vol in spec.get("volumes", []) or []:
if isinstance(vol, str):
src = vol.split(":", 1)[0]
elif isinstance(vol, dict):
src = vol.get("source", "")
else:
continue
if src.startswith("./") and not src.startswith("./config/"):
found.add(src[2:])
return sorted(found)
def copy_extra_binds(deploy_dir: Path, host_dir: Path, paths: list[str]) -> list[str]:
"""Salin bind relatif non-config (mis. `scripts/airflow`) ke direktori host."""
import shutil
copied: list[str] = []
for rel in paths:
src = deploy_dir / rel
if not src.exists():
continue
dst = host_dir / rel
dst.parent.mkdir(parents=True, exist_ok=True)
if src.is_dir():
shutil.copytree(src, dst, dirs_exist_ok=True)
else:
shutil.copy2(src, dst)
copied.append(rel)
return copied
def endpoint_lines(doc: dict, host: str, plan: dict) -> list[str]:
"""Baris `VAR=alamat|nama` untuk host ini.
SE-HOST → **nama service** (satu jaringan Docker). Port container TIDAK diterbitkan
di host (mis. OM 8585), jadi menulis alamat host justru "connection refused"
(regresi: Airflow/OpenLineage → `http://<ip-control>:8585/api`). Beda host → alamat
host pemilik (port internal diterbitkan generator untuk itu).
"""
placement = component_placements(doc)
lines: list[str] = []
for var, component in ENDPOINT_VARS.items():
owners = placement.get(component)
if not owners:
continue
if host in owners:
lines.append(f"{var}={component}") # SE-HOST → nama service
else:
lines.append(f"{var}={plan[owners[0]]['address']}") # LINTAS host → alamat
disabled = [n for n, v in inventory.enabled_components(doc).items() if v is False]
if disabled:
# Komponen `enabled: false` = tidak dijalankan di host mana pun → installer
# harus melewatinya (install.sh: EXTERNAL_SERVICES menang atas auto-deteksi).
lines.append('EXTERNAL_SERVICES="%s"' % " ".join(disabled))
return lines
def public_url_lines(doc: dict, plan: dict | None = None) -> list[str]:
"""Baris `*_PUBLIC_URL=...` — URL yang dilihat BROWSER (issuer OIDC, redirect, UI),
bukan lalu lintas container.
tls.mode != none : `https://<label>.<domain>` (inventory.public_urls).
tls.mode == none : `http(s)://<alamat HOST PEMILIK>:<port>`. Sebelumnya mode ini
mengembalikan daftar kosong, sehingga .env host memakai default
`http://localhost:<port>` — dan browser di host lain dibuang ke
localhost (issuer OIDC pun localhost). Sekarang alamatnya diambil
dari host yang menjalankan komponen, jadi akses lintas host benar
tanpa perlu nilai berbeda per bundle.
"""
by_domain = inventory.public_urls(doc)
if by_domain:
return [f"{var}={url}" for var, url in sorted(by_domain.items())]
if plan is None:
return []
placement = component_placements(doc)
resolved: dict[str, str] = {}
for var, (component, tmpl) in PUBLIC_URL_ENDPOINTS.items():
owners = placement.get(component)
if not owners:
continue
# Sengaja memakai alamat PEMILIK (bukan host lokal) dan urutan yang pasti,
# supaya nilainya identik di setiap bundle.
resolved[var] = tmpl.format(addr=plan[sorted(owners)[0]]["address"])
return [f"{var}={url}" for var, url in sorted(resolved.items())]
def tls_policy_lines(doc: dict) -> list[str]:
"""Kebijakan keamanan turunan `tls.mode` (saat ini hanya realm Keycloak).
Keycloak DEFAULT `sslRequired=external` menuntut HTTPS untuk permintaan dari alamat
NON-lokal. Alamat Tailscale `100.64.0.0/10` (CGNAT) TERMASUK non-lokal bagi Keycloak
(`isLocal` = loopback + RFC1918 + link-local + `fc00::/7`), sehingga admin console di
`http://<alamat-tailscale>:12080` gagal dengan "HTTPS required" — dan Keycloak 26
TIDAK punya opsi server-level untuk ini (tidak ada `KC_HTTPS_REQUIRED`), hanya per
realm. Karena itu nilainya diteruskan ke bootstrap.sh.
tls.mode `none` -> `none` (HTTP polos; instalasi dev/simulasi)
tls.mode `edge`/`full` -> `external` (ada Traefik/TLS di depan)
"""
edge = inventory.tls_mode(doc) != "none"
return [
f"KEYCLOAK_SSL_REQUIRED={'external' if edge else 'none'}",
# Di belakang Traefik, Keycloak WAJIB memercayai X-Forwarded-* — tanpa itu
# permintaan HTTPS dari proxy dianggap HTTP (redirect salah / tuntutan HTTPS).
# Sengaja KOSONG saat tanpa proxy: memasangnya membuat Keycloak memercayai
# header yang bisa dipalsukan klien.
f"KC_PROXY_HEADERS={'xforwarded' if edge else ''}",
# Trino: X-Forwarded-Proto supaya skema publik benar saat di belakang proxy.
f"TRINO_PROCESS_FORWARDED={'true' if edge else 'false'}",
]
# Rute untuk komponen publik yang TIDAK berjalan di host ingress. Traefik hanya melihat
# container di host-nya sendiri, jadi komponen di host lain dirutekan lewat berkas
# dinamis (provider file) ke PORT HOST-nya (12xxx), bukan port container.
# Nilai: (label rute, URL, butuh serverstransport-insecure?)
REMOTE_ROUTE_TARGETS = {
"keycloak": ("kc", "http://{addr}:12080", False),
"grafana": ("grafana", "http://{addr}:12300", False),
"openmetadata-server": ("om", "http://{addr}:12085", False),
"airflow-webserver": ("airflow", "http://{addr}:12082", False),
"trino-coordinator": ("trino", "https://{addr}:12443", True),
"minio": ("minio", "http://{addr}:12901", False),
}
def ingress_host(doc: dict, plan: dict) -> str:
"""Host yang menjalankan ingress Traefik (T-20).
Default: host yang menjalankan `keycloak` (tier identity — pada preset a3 itu
`control`), supaya tidak perlu setelan tambahan. Bisa dipatok eksplisit lewat
`tls.ingress_host`. Kosong bila tls.mode `none` (memang tanpa ingress).
"""
if inventory.tls_mode(doc) == "none":
return ""
explicit = ((doc.get("tls") or {}).get("ingress_host") or "").strip()
if explicit:
return explicit if explicit in plan else ""
owners = sorted(component_placements(doc).get("keycloak") or [])
if owners:
return owners[0]
return sorted(plan)[0] if plan else ""
def remote_route_doc(doc: dict, plan: dict) -> dict | None:
"""Isi berkas dinamis untuk rute komponen yang ada DI HOST LAIN.
Mengembalikan None bila tidak ada rute jarak jauh (semua komponen publik memang
berada di host ingress).
"""
ing = ingress_host(doc, plan)
if not ing:
return None
domain = inventory.cluster_domain(doc)
placement = component_placements(doc)
routers: dict = {}
services: dict = {}
for component, (label, tmpl, needs_insecure) in REMOTE_ROUTE_TARGETS.items():
owners = sorted(placement.get(component) or [])
if not owners or ing in owners:
continue # sudah ditangani label container di host ini
owner = owners[0]
router = {
"rule": f"Host(`{label}.{domain}`)",
"entryPoints": ["websecure"],
"service": label,
"tls": {
"certResolver": "le",
"domains": [{"main": domain, "sans": [f"*.{domain}"]}],
},
}
lb: dict = {"servers": [{"url": tmpl.format(addr=plan[owner]["address"])}]}
if needs_insecure:
lb["serversTransport"] = "trino-insecure"
routers[label] = router
services[label] = {"loadBalancer": lb}
if not routers:
return None
# serversTransport wajib ikut di berkas ini: berkas dinamis yang dibangkitkan
# MENGGANTIKAN salinan dasar, jadi definisinya harus ada di sini juga.
return {
"http": {
"serversTransports": {"trino-insecure": {"insecureSkipVerify": True}},
"routers": routers,
"services": services,
}
}
def node_targets_doc(plan: dict) -> list[dict]:
"""Berkas target file_sd untuk job `node`: satu entri `<alamat>:9100` per host.
node_exporter adalah PRA-SYARAT HOST (di luar installer), sehingga SEMUA host di
cluster.yml dianggap menjalankannya di port 9100. Label `host` disalin ke
`instance` oleh relabel_configs di prometheus.yml (panel menampilkan nama host).
"""
return [
{"targets": [f"{spec['address']}:{NODE_EXPORTER_PORT}"], "labels": {"host": name}}
for name, spec in sorted(plan.items())
]
def ingress_lines(doc: dict, host: str, plan: dict) -> list[str]:
"""Variabel ingress untuk .env host (T-20).
`CLUSTER_DOMAIN` dipakai label rute Traefik (Host(`<label>.<domain>`)), dan
`TRAEFIK_BIND_ADDR` membatasi port 80/443 ke alamat host ini — bukan semua antarmuka.
Hanya ditulis saat ada TLS/ingress; tanpa itu Traefik memang tidak dijalankan.
"""
if inventory.tls_mode(doc) == "none":
return []
if host != ingress_host(doc, plan):
return []
return [
f"CLUSTER_DOMAIN={inventory.cluster_domain(doc)}",
f"TRAEFIK_BIND_ADDR={plan[host]['address']}",
]
def _endpoint_port(var, env_path):
"""Port container untuk var URL (dari .env, fallback default URL_ENDPOINT_PORTS)."""
spec = URL_ENDPOINT_PORTS.get(var)
if not spec:
return ""
env_name, default = spec
val = _source_env_value(env_path, env_name) if env_path else None
return val if val else str(default)
def url_endpoint_lines(doc: dict, host: str, plan: dict, env_path=None) -> list[str]:
"""Baris `VAR=<url>` untuk konsumen bentuk URL.
SE-HOST → **nama service** (mis. `http://nessie:19120`, `http://opa:8181`);
LINTAS host → alamat IP pemilik (port internalnya diterbitkan generator).
Konsisten dengan endpoint_lines(): port container tak diterbitkan di host.
Port diambil dari .env (mis. NESSIE_INTERNAL_PORT) bila di-override.
"""
placement = component_placements(doc)
lines: list[str] = []
for var, (component, template) in URL_ENDPOINT_VARS.items():
owners = placement.get(component)
if not owners:
continue
addr = component if host in owners else plan[owners[0]]["address"]
lines.append(f"{var}={template.format(addr=addr, port=_endpoint_port(var, env_path))}")
return lines
def render_config_map(doc: dict, host: str, plan: dict) -> list[str]:
"""Peta hostname container → alamat, untuk render-config.sh (khusus host ini).
HANYA komponen yang dimiliki HOST LAIN yang dipetakan. Komponen SE-HOST dibiarkan
memakai NAMA container (satu jaringan Docker) — menulisnya ke alamat host justru
RUSAK: port container sering tak diterbitkan di host. Regresi nyata: `prometheus`
(se-host dengan grafana) ditulis `http://<ip-host>:9090` di datasources.yaml →
Grafana "connection refused"; `statsd-exporter` (se-host dengan prometheus) serupa.
"""
placement = component_placements(doc)
args: list[str] = []
for component in RENDER_HOSTS:
owners = placement.get(component)
if not owners:
continue
if host in owners:
continue # SE-HOST → biarkan nama container
args += ["--map", f"{component}={plan[owners[0]]['address']}"]
return args
def cmd_render(argv: list[str]) -> int:
if len(argv) < 7:
sys.stderr.write(__doc__ or "")
return 2
inv_path, compose_path, env_path, out_base, renderer = argv[2:7]
preset_override = argv[7] if len(argv) > 7 else ""
doc = inventory.load(inv_path)
if preset_override:
doc["preset"] = preset_override
doc = inventory.apply_preset(doc)
errors = inventory.validate(doc)
if errors:
sys.stderr.write("cluster.yml tidak valid — jalankan 'inventory.py validate' untuk detail:\n")
for e in errors:
sys.stderr.write(f" ✗ {e}\n")
return 1
base = load_compose(compose_path)
extra_binds = extra_bind_dirs(base)
plan = inventory.plan(doc)
out_root = Path(out_base)
placement = component_placements(doc)
needed = required_cross_host(base, Path(compose_path).parent, plan, placement)
total_dropped: dict[str, list[str]] = {}
port_internal_all: set[str] = set()
# Trino cert/keystore: dibangkitkan SEKALI (SAN = ALAMAT HOST semua host), lalu
# disalin IDENTIK ke tiap bundle. WAJIB sekali — kalau tiap host beda keypair, cert
# klien (OM ingestion, dll.) tak cocok dgn cert server → verifikasi TLS gagal.
#
# DIPERSISTENKAN di deploy/.trino-keystore/ (gitignored): kalau cert dibuat BARU tiap
# `--generate`, server Trino yang belum di-recreate menyajikan cert LAMA → klien
# (GE/OM) "self-signed certificate". Stabil kecuali daftar SAN (alamat host) berubah.
_ks_cache = Path(__file__).resolve().parents[2] / ".trino-keystore"
_ks_cache.mkdir(parents=True, exist_ok=True)
_trino_ks_src = _ks_cache
_san = ",".join(f"IP:{plan[h]['address']}" for h in sorted(plan) if plan[h].get("address"))
_ks_script = Path(renderer).resolve().parent / "generate-trino-keystore.sh"
_marker = _ks_cache / "san.txt"
_need_regen = (not (_ks_cache / "keystore.p12").is_file()
or not _marker.is_file()
or _marker.read_text(encoding="utf-8").strip() != _san)
if _san and _ks_script.is_file() and _need_regen:
subprocess.run([str(_ks_script), "--out", str(_trino_ks_src), "--san-extra", _san],
capture_output=True, text=True)
try:
_marker.write_text(_san, encoding="utf-8")
except OSError:
pass
for host, spec in plan.items():
host_dir = out_root / f"host-{host}"
# 1) compose
# Ingress: setiap host menjalankan Traefik-nya sendiri saat ada TLS/proxy
# (satu sertifikat per host — acme.json tidak bisa dibagi antar host).
# Tanpa TLS, service traefik memang tidak ikut.
wanted = set(spec["services"])
# Ingress satu host: HANYA host ingress yang menjalankan Traefik. Ini juga
# menghilangkan perebutan record TXT DNS-01 (dulu tiap host menerbitkan
# wildcard yang sama) dan menekan pemakaian batas sertifikat Let's Encrypt.
ing = ingress_host(doc, plan)
if ing and host == ing:
wanted.add("traefik")
compose_out, dropped = build_compose(base, wanted)
# Fase 2: buka port internal ke alamat host ini agar komponen di host lain
# dapat menjangkau (pengganti ketiadaan satu bridge network lintas host).
added_ports = add_internal_ports(compose_out["services"], spec["address"], needed)
port_internal_all.update(added_ports)
write_compose(host_dir / "docker-compose.yml", compose_out)
total_dropped[host] = dropped
# 2) .env
render_env(
Path(env_path), host_dir / ".env", host, spec,
endpoint_lines(doc, host, plan)
+ url_endpoint_lines(doc, host, plan, env_path)
+ public_url_lines(doc, plan)
+ tls_policy_lines(doc)
+ ingress_lines(doc, host, plan),
)
# 3) config ter-render
# 3a) bind relatif non-config (mis. scripts/airflow) → bundle mandiri
copied = copy_extra_binds(Path(compose_path).parent, host_dir, extra_binds)
# 3b) config ter-render melalui render-config.sh
# Bersihkan target berkas runtime yang mungkin berwujud DIREKTORI (sisa `up`
# Docker saat sumber bind-mount belum ada). Tanpa ini render-config GAGAL
# ("tulis gagal") dan generator berhenti SEBELUM menulis berkas runtime.
for _rel in ("airflow/simple_auth_manager_passwords.json",
"trino/password.db"):
_t = host_dir / "config" / _rel
if _t.is_dir():
shutil.rmtree(_t)
rc = subprocess.run(
[renderer, "--out", str(host_dir / "config"), *render_config_map(doc, host, plan)],
capture_output=True, text=True,
)
# Berkas dinamis rute komponen di host lain (hanya di host ingress).
# Ditulis SETELAH render-config karena berkas ini menggantikan salinan dasar.
remote = remote_route_doc(doc, plan)
if remote is not None and host == ing:
(host_dir / "config" / "traefik").mkdir(parents=True, exist_ok=True)
with open(host_dir / "config" / "traefik" / "dynamic.yml", "w", encoding="utf-8") as fh:
fh.write("# DIBANGKITKAN OTOMATIS oleh install.sh --generate — jangan disunting tangan.\n")
fh.write("# Ingress satu host: rute untuk komponen yang TIDAK ada di host ini.\n")
yaml.safe_dump(remote, fh, sort_keys=False, allow_unicode=True)
# Target node_exporter (job `node`, file_sd) — hanya di host pemilik
# Prometheus. Ditulis SETELAH render-config karena menggantikan salinan dasar.
if host in (placement.get("prometheus") or []):
tdir = host_dir / "config" / "prometheus" / "targets"
tdir.mkdir(parents=True, exist_ok=True)
with open(tdir / "node.yml", "w", encoding="utf-8") as fh:
fh.write("# DIBANGKITKAN OTOMATIS dari cluster.yml oleh install.sh --generate\n")
fh.write("# — jangan disunting tangan. Target node_exporter (job `node`) semua host.\n")
yaml.safe_dump(node_targets_doc(plan), fh, sort_keys=False, allow_unicode=True)
if rc.returncode != 0:
sys.stderr.write(f"[host-{host}] render-config GAGAL:\n{rc.stdout}{rc.stderr}")
return 1
# Salin cert/keystore Trino (dibangkitkan SEKALI dgn SAN IP semua host) ke bundle
# ini — keypair/cert IDENTIK lintas host agar verifikasi TLS server↔klien lolos.
if (_trino_ks_src / "keystore.p12").is_file():
(host_dir / "config" / "trino" / "certs").mkdir(parents=True, exist_ok=True)
shutil.copy2(_trino_ks_src / "keystore.p12",
host_dir / "config" / "trino" / "keystore.p12")
shutil.copy2(_trino_ks_src / "certs" / "trino.crt",
host_dir / "config" / "trino" / "certs" / "trino.crt")
# Berkas runtime gitignored (Airflow auth, Trino password.db) tidak ikut
# render-config → tulis eksplisit agar compose tak me-mount sumber hilang (DIRECTORY).
_write_bundle_runtime_files(host_dir, Path(env_path))
print(f"host-{host:9s} {spec['address']:15s} services={len(spec['services'])} "
f"depends_on_dibuang={len(dropped)} bind_disalin={len(copied)} "
f"port_internal={len(added_ports)}")
print(f"komponen butuh port lintas host: {', '.join(sorted(needed)) or '—'}")
if port_internal_all:
print(f"port internal dibuka: {len(port_internal_all)} ("
+ ", ".join(sorted(port_internal_all)) + ")")
# ringkasan untuk di-review
for host, dropped in total_dropped.items():
for item in dropped:
print(f" [host-{host}] depends_on lintas host dibuang: {item}")
print(json.dumps({h: len(s["services"]) for h, s in plan.items()}, sort_keys=True))
return 0
def _source_env_value(env_path, key):
"""Ambil satu nilai dari .env sumber (kutip luar dibuang). None bila tak ada."""
try:
text = Path(env_path).read_text(encoding="utf-8")
except OSError:
return None
for line in text.splitlines():
line = line.strip()
if not line or line.startswith("#") or "=" not in line:
continue
name, _, value = line.partition("=")
if name.strip() != key:
continue
value = value.strip()
if len(value) >= 2 and value[0] == value[-1] and value[0] in "\"'":
value = value[1:-1]
return value
return None
def _write_bundle_runtime_files(host_dir, env_path):
"""Tulis berkas runtime yang TIDAK di-track git ke bundle host.
compose generated me-mount path ini sebagai FILE. Bila file tidak ada, Docker
membuat DIRECTORY di host → service gagal (Airflow SimpleAuthManager:
IsADirectoryError; Trino: password.db bukan file). ``render-config`` tidak
menyalinnya (gitignored), jadi harus ditulis eksplisit di sini.
"""
cfg = host_dir / "config"
# Dir bind-mount yang ABSEN di git (config/airflow/sling hanya memuat env.yaml yang
# gitignored) → Docker (daemon root) membuatnya sebagai ROOT saat `up` → operator
# non-root tak bisa menulis env.yaml (bootstrap gagal). Pastikan ada & user-owned.
(cfg / "airflow" / "sling").mkdir(parents=True, exist_ok=True)
# 1) Airflow SimpleAuthManager — {"admin": <password>} (mode a+ → wajib writable)
pw_json = cfg / "airflow" / "simple_auth_manager_passwords.json"
pw_json.parent.mkdir(parents=True, exist_ok=True)
if pw_json.is_dir():
shutil.rmtree(pw_json)
admin_pw = _source_env_value(env_path, "AIRFLOW_ADMIN_PASSWORD")
if admin_pw is not None:
pw_json.write_text(json.dumps({"admin": admin_pw}) + "\n", encoding="utf-8")
elif not pw_json.is_file():
pw_json.write_text("{}\n", encoding="utf-8")
pw_json.chmod(0o666)
# 2) Trino password.db — byte-identik dari sumber bila ada; kalau tidak, FILE kosong
# (yang penting BUKAN direktori; bootstrap mengisinya).
src_db = Path(env_path).resolve().parent / "config" / "trino" / "password.db"
dst_db = cfg / "trino" / "password.db"
dst_db.parent.mkdir(parents=True, exist_ok=True)
if dst_db.is_dir():
shutil.rmtree(dst_db)
if src_db.is_file():
shutil.copy2(src_db, dst_db)
elif not dst_db.is_file():
dst_db.write_text("", encoding="utf-8")
def main(argv: list[str]) -> int:
if len(argv) < 2:
sys.stderr.write(__doc__ or "")
return 2
if argv[1] == "render":
return cmd_render(argv)
sys.stderr.write(f"perintah tak dikenal: {argv[1]}\n")
return 2
if __name__ == "__main__":
raise SystemExit(main(sys.argv))
deploy/lib/cluster/inventory.py
+365
View File
@@ -0,0 +1,365 @@
#!/usr/bin/env python3
"""inventory.py — cluster.yml: parse, validasi, resolusi penempatan (Fase 1 multi-host).
Ini otak Fase 1 (R10 §4.3/§4.5, R11 §3–§5). Satu implementasi otoritatif dipakai
oleh `lib/cluster/generate.sh` dan `install.sh`; tidak ada logika penempatan yang
diduplikasi di shell.
Model:
- Tier (7): state, identity, policy, coord, engine, pipeline, governance.
Tiap service compose dipetakan ke tepat satu tier (TIER_SERVICES).
- Preset (a3/b5) hanya mengisi DEFAULT `hosts.<h>.tiers`; komponen diturunkan
dari tier-nya. Override eksplisit `components.<c>.host` menang.
- Tier `engine` boleh >1 host (scale-out): komponennya muncul di SEMUA host
bertier engine. Tier lain tepat 1 host.
Perintah:
inventory.py validate <cluster.yml> # exit 0 = valid; cetak ringkasan/error
inventory.py hosts <cluster.yml> # daftar host
inventory.py plan <cluster.yml> # JSON: host -> {address, tiers, services}
inventory.py components <cluster.yml> # komponen -> host (untuk dokumentasi)
"""
from __future__ import annotations
import json
import sys
from pathlib import Path
try:
import yaml
except ImportError: # pragma: no cover
sys.stderr.write("PyYAML wajib tersedia (python3 -c 'import yaml')\n")
raise SystemExit(2)
# ─── Taksonomi (R11 §3.1) ───────────────────────────────────────────────────
# Setiap service compose milik TEPAT SATU tier.
TIER_SERVICES: dict[str, list[str]] = {
"state": ["postgres", "minio", "minio-init", "openmetadata-opensearch"],
"identity": ["keycloak", "openldap", "nessie"],
"policy": ["opa"],
"coord": ["trino-coordinator", "starrocks-fe", "trino-jmx-exporter"],
"engine": ["trino-worker", "starrocks-cn", "dremio"],
"pipeline": [
"airflow-volume-init",
"airflow-init",
"airflow-webserver",
"airflow-dag-processor",
"airflow-scheduler",
],
# `prometheus` menyusul R11 (belum ada di taksonomi 7-tier). Ditempatkan di
# `governance`: pada blueprint A maupun B ia berada di host yang sama, jadi
# granularitas tier tidak mengubah penempatan.
# `statsd-exporter` (M2 observabilitas) ikut `governance` bersama Prometheus;
# pada a3/b5 tier ini se-host dengan `pipeline` (Airflow), sehingga UDP
# StatsD Airflow→statsd-exporter tetap lokal.
"governance": ["openmetadata-migrate", "openmetadata-server", "grafana", "prometheus", "statsd-exporter"],
}
VALID_TIERS = tuple(TIER_SERVICES)
# ─── TLS & URL publik (langkah 1: hanya skema/var; BELUM mengaktifkan TLS) ──
# `none` = perilaku sekarang (http, IP+port 12xxx). `edge` = browser memakai nama
# domain lewat ingress, bidang internal tetap plaintext. `full` = seluruh segmen TLS.
# Default `none` supaya TANPA cluster.yml perilaku single-host tidak berubah.
TLS_MODES = ("none", "edge", "full")
# Layanan yang punya URL BROWSER-FACING → label subdomain di bawah `domain`.
# Dipakai juga oleh ingress (step 2) sebagai daftar rute Host(), jadi nama di sini
# adalah satu-satunya sumber kebenaran penamaan.
PUBLIC_URL_SERVICES: dict[str, str] = {
"KEYCLOAK_PUBLIC_URL": "kc",
"GRAFANA_PUBLIC_URL": "grafana",
"OPENMETADATA_PUBLIC_URL": "om",
"AIRFLOW_PUBLIC_URL": "airflow",
"TRINO_PUBLIC_URL": "trino",
"MINIO_CONSOLE_PUBLIC_URL": "minio",
}
# Blueprint: default tiers per host (R11 §4.1/§4.2).
PRESETS: dict[str, dict[str, list[str]]] = {
"a3": {
"control": ["identity", "policy", "pipeline", "governance"],
"compute": ["coord", "engine"],
"state": ["state"],
},
"b5": {
"control": ["identity", "policy", "pipeline", "governance"],
"coord": ["coord"],
"engine-1": ["engine"],
"engine-2": ["engine"],
"state": ["state"],
},
}
# Komponen stateful: tidak boleh berpindah host tanpa migrasi volume (R10 R7).
PINNED = ("postgres", "minio", "openmetadata-opensearch", "starrocks-fe", "starrocks-cn", "dremio")
def tier_of(service: str) -> str | None:
"""Tier pemilik sebuah service compose."""
for tier, services in TIER_SERVICES.items():
if service in services:
return tier
return None
def load(path: str) -> dict:
try:
with open(path, encoding="utf-8") as fh:
return yaml.safe_load(fh) or {}
except yaml.YAMLError as exc:
raise SystemExit(f"cluster.yml bukan YAML yang sah: {exc}") from exc
def apply_preset(doc: dict) -> dict:
"""Isi `hosts` dari preset bila belum didefinisikan eksplisit.
Preset tak dikenal TIDAK menghentikan program di sini — biarkan `validate()`
yang melaporkannya (dulu ini KeyError → traceback, bukan pesan yang berguna).
"""
preset = doc.get("preset") or "none"
if preset not in PRESETS:
return doc
hosts = doc.setdefault("hosts", {})
for name, tiers in PRESETS[preset].items():
hosts.setdefault(name, {})
hosts[name].setdefault("tiers", list(tiers))
return doc
def resolve(doc: dict) -> dict[str, list[str]]:
"""Komponen → daftar host (tier `engine` bisa >1 host)."""
hosts = doc.get("hosts") or {}
tiers_by_service = {svc: tier for tier, svcs in TIER_SERVICES.items() for svc in svcs}
explicit = {name: spec.get("host") for name, spec in (doc.get("components") or {}).items()}
placement: dict[str, list[str]] = {}
for service, tier in tiers_by_service.items():
if service in explicit and explicit[service]:
placement[service] = [explicit[service]]
continue
owners = [h for h, spec in hosts.items() if tier in (spec.get("tiers") or [])]
if owners:
placement[service] = sorted(owners)
return placement
def tls_mode(doc: dict) -> str:
"""Mode TLS dari inventory (`none` bila tidak ditulis)."""
return (doc.get("tls") or {}).get("mode") or "none"
def cluster_domain(doc: dict) -> str:
return ((doc.get("cluster") or {}).get("domain") or "").strip()
def public_urls(doc: dict) -> dict[str, str]:
"""URL browser-facing turunan dari tls.mode + domain.
Kosong bila mode `none` — dalam mode itu pemakaian tetap IP+port 12xxx seperti
sekarang, sehingga perilaku single-host TIDAK berubah.
"""
mode = tls_mode(doc)
domain = cluster_domain(doc)
if mode == "none" or not domain:
return {}
scheme = "https" if mode in ("edge", "full") else "http"
return {var: f"{scheme}://{label}.{domain}" for var, label in PUBLIC_URL_SERVICES.items()}
def enabled_components(doc: dict) -> dict[str, bool]:
return {
name: bool(spec.get("enabled", True))
for name, spec in (doc.get("components") or {}).items()
}
def validate(doc: dict) -> list[str]:
"""Validasi R10 §4.3. Mengembalikan daftar error (kosong = valid)."""
errors: list[str] = []
preset = doc.get("preset") or "none"
if preset not in ("none", *PRESETS):
errors.append(f"preset '{preset}' tidak dikenal (pakai: none|{'|'.join(PRESETS)})")
# TLS & domain (langkah 1)
mode = (doc.get("tls") or {}).get("mode") or "none"
if mode not in TLS_MODES:
errors.append(f"tls.mode '{mode}' tidak dikenal (pakai: {'|'.join(TLS_MODES)})")
domain = ((doc.get("cluster") or {}).get("domain") or "").strip()
if mode != "none":
if not domain:
errors.append(f"tls.mode '{mode}' memerlukan cluster.domain (mis. sicabe.com)")
elif "://" in domain or "/" in domain or "." not in domain:
errors.append(f"cluster.domain '{domain}' tidak sah — isi nama domain saja "
f"(mis. sicabe.com), tanpa skema atau garis miring")
# Ingress satu host (T-20): opsional — default host yang menjalankan keycloak.
ing = ((doc.get("tls") or {}).get("ingress_host") or "").strip()
if ing:
if mode == "none":
errors.append("tls.ingress_host tidak bermakna bila tls.mode 'none' "
"(tanpa Traefik/TLS) — hapus atau ubah tls.mode")
elif ing not in (doc.get("hosts") or {}):
errors.append(f"tls.ingress_host '{ing}' bukan nama host yang dikenal "
f"(host: {', '.join(sorted((doc.get('hosts') or {})))})")
hosts = doc.get("hosts") or {}
if not hosts:
errors.append("'hosts' kosong — inventory harus mendefinisikan minimal satu host")
return errors
# (f) tiap host punya address + tier yang sah
for name, spec in hosts.items():
if not spec.get("address"):
errors.append(f"host '{name}': 'address' wajib diisi")
tiers = spec.get("tiers") or []
if not tiers:
errors.append(f"host '{name}': minimal satu tier")
for tier in tiers:
if tier not in VALID_TIERS:
errors.append(f"host '{name}': tier '{tier}' tidak dikenal (sah: {', '.join(VALID_TIERS)})")
# (d) tier state tepat 1 host · R11 aturan 2 (state selalu host sendiri)
state_hosts = [h for h, s in hosts.items() if "state" in (s.get("tiers") or [])]
if len(state_hosts) != 1:
errors.append(
f"tier 'state' harus di TEPAT 1 host, ditemukan {len(state_hosts)}: "
f"{state_hosts or '—'} (I/O-bound & pinned, R11 §3.2)"
)
# (3) tier coord tepat 1 host (OSS Trino = 1 coordinator)
coord_hosts = [h for h, s in hosts.items() if "coord" in (s.get("tiers") or [])]
if len(coord_hosts) != 1:
errors.append(f"tier 'coord' harus di TEPAT 1 host, ditemukan {len(coord_hosts)}: {coord_hosts or '—'}")
# engine: boleh >1 (scale-out) — hanya info bila tidak ada
if not [h for h, s in hosts.items() if "engine" in (s.get("tiers") or [])]:
errors.append("tidak ada host bertier 'engine' — tidak ada tempat eksekusi Trino/StarRocks")
# anti-affinity R11 §3.2: state ⊥ engine
for h, s in hosts.items():
t = set(s.get("tiers") or [])
if "state" in t and "engine" in t:
errors.append(f"host '{h}': tier 'state' dan 'engine' tidak boleh satu host (anti-affinity)")
# (a) komponen menunjuk host yang ada · (c) endpoint hanya ke komponen enabled
components = doc.get("components") or {}
for name, spec in components.items():
host = spec.get("host")
if host and host not in hosts:
errors.append(f"komponen '{name}': host '{host}' tidak ada di 'hosts'")
if tier_of(name) is None and name not in ("dremio",):
errors.append(f"komponen '{name}' tidak ada di taksonomi tier (TIER_SERVICES)")
# (c) tiap komponen harus bisa ditempatkan
placement = resolve(doc)
for service in (svc for svcs in TIER_SERVICES.values() for svc in svcs):
if service not in placement:
errors.append(
f"komponen '{service}' tidak bisa ditempatkan — tidak ada host bertier "
f"'{tier_of(service)}' dan tidak ada components.{service}.host"
)
# stateful: cegah pindah host tanpa migrasi (peringatan, bukan error)
return errors
def plan(doc: dict) -> dict[str, dict]:
"""host → {address, tiers, services} untuk generator."""
hosts = doc.get("hosts") or {}
placement = resolve(doc)
enabled = enabled_components(doc)
result: dict[str, dict] = {}
for name, spec in hosts.items():
result[name] = {
"address": spec.get("address"),
"tiers": spec.get("tiers") or [],
"services": [],
}
for service, owners in placement.items():
if enabled.get(service) is False:
continue
for owner in owners:
if owner in result:
result[owner]["services"].append(service)
for name in result:
result[name]["services"].sort()
return result
def warnings(doc: dict) -> list[str]:
out: list[str] = []
placement = resolve(doc)
for service in PINNED:
hosts = placement.get(service) or []
if len(hosts) > 1:
out.append(f"'{service}' stateful berada di >1 host {hosts} — butuh volume terpisah per host")
return out
def cmd_validate(path: str) -> int:
doc = apply_preset(load(path))
errors = validate(doc)
warns = warnings(doc)
if errors:
print(f"cluster.yml TIDAK valid ({len(errors)} masalah):")
for e in errors:
print(f" ✗ {e}")
return 1
p = plan(doc)
print(f"cluster.yml valid — preset '{doc.get('preset') or 'none'}'")
for host, spec in p.items():
print(f" {host:9s} {spec['address']:15s} tiers=[{','.join(spec['tiers'])}] "
f"services={len(spec['services'])}")
for w in warns:
print(f" ⚠ {w}")
return 0
def cmd_plan(path: str) -> int:
doc = apply_preset(load(path))
errors = validate(doc)
if errors:
sys.stderr.write("cluster.yml tidak valid — jalankan 'validate' untuk detail\n")
return 1
print(json.dumps(plan(doc), indent=2, sort_keys=True))
return 0
def cmd_components(path: str) -> int:
doc = apply_preset(load(path))
for service, owners in sorted(resolve(doc).items()):
print(f"{service:24s} -> {','.join(owners)}")
return 0
def cmd_hosts(path: str) -> int:
doc = apply_preset(load(path))
for name, spec in (doc.get("hosts") or {}).items():
print(f"{name} {spec.get('address')}")
return 0
def main(argv: list[str]) -> int:
if len(argv) < 3:
sys.stderr.write(__doc__ or "")
return 2
command, path = argv[1], argv[2]
if not Path(path).is_file():
sys.stderr.write(f"tidak ditemukan: {path}\n")
return 2
commands = {
"validate": cmd_validate,
"plan": cmd_plan,
"components": cmd_components,
"hosts": cmd_hosts,
}
if command not in commands:
sys.stderr.write(f"perintah tak dikenal: {command}\n")
return 2
return commands[command](path)
if __name__ == "__main__":
raise SystemExit(main(sys.argv))
deploy/lib/cluster/preflight.py
+233
View File
@@ -0,0 +1,233 @@
#!/usr/bin/env python3
"""preflight.py — verifikasi kesiapan lintas host SEBELUM deploy (Fase 2, R10 §4.7).
Pengganti `depends_on` lintas host: generator MEMBUANG `depends_on` yang menyeberang
host (Compose tidak bisa menyalakan service di host lain), sehingga tidak ada lagi yang
menjamin urutan/ketersediaan antar host. Preflight menutup celah itu dengan memeriksa
**dari sisi jaringan**: apakah tiap port yang dibutuhkan benar-benar terjangkau di
alamat host pemiliknya.
Yang diperiksa:
1. Inventory valid (aturan penempatan R10 §4.3 / R11 §3.2).
2. Port internal yang dibutuhkan lintas host (perhitungan yang SAMA dengan generator —
diimpor, bukan disalin) benar-benar terbuka di alamat host pemiliknya.
3. TLS Trino coordinator (`8443`) menyelesaikan handshake.
4. Selisih waktu antar host (header `Date` HTTP) — JWT/`notBefore` sensitif, jadi
skew besar = kegagalan login yang sulit dilacak.
Pemakaian:
preflight.py <cluster.yml> [--compose docker-compose.yml] [--deploy DIR] [--timeout N]
Keluaran mengikuti konvensi gate repo: [PASS]/[FAIL]/[WARN]/[SKIP] + Ringkasan + recap.
Exit 1 bila ada FAIL.
"""
from __future__ import annotations
import json
import socket
import ssl
import sys
from datetime import datetime, timezone
from email.utils import parsedate_to_datetime
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent))
import generate # noqa: E402
import inventory # noqa: E402
GRN = "\033[32m"; RED = "\033[31m"; YLW = "\033[33m"; BLD = "\033[1m"; NC = "\033[0m"
class Report:
def __init__(self) -> None:
self.passed = 0
self.failed = 0
self.warned = 0
self.skipped = 0
self.fail_msgs: list[str] = []
def ok(self, msg: str) -> None:
print(f" {GRN}[PASS]{NC} {msg}"); self.passed += 1
def bad(self, msg: str) -> None:
print(f" {RED}[FAIL]{NC} {msg}"); self.failed += 1; self.fail_msgs.append(msg)
def warn(self, msg: str) -> None:
print(f" {YLW}[WARN]{NC} {msg}"); self.warned += 1
def skip(self, msg: str) -> None:
print(f" {YLW}[SKIP]{NC} {msg}"); self.skipped += 1
def recap(self) -> int:
print()
print(f"{BLD}Ringkasan:{NC} {self.passed} PASS · {self.failed} FAIL · "
f"{self.warned} WARN · {self.skipped} SKIP")
if self.fail_msgs:
print(f"\n{BLD}Yang gagal ({len(self.fail_msgs)}):{NC}")
for i, m in enumerate(self.fail_msgs, 1):
print(f" {i}. {m}")
return 1 if self.failed else 0
def tcp_open(host: str, port: int, timeout: float) -> tuple[bool, str]:
try:
with socket.create_connection((host, port), timeout=timeout):
return True, ""
except socket.timeout:
return False, "timeout"
except ConnectionRefusedError:
return False, "connection refused (port tertutup)"
except OSError as exc:
return False, str(exc)
def tls_handshake(host: str, port: int, timeout: float) -> tuple[bool, str]:
"""Handshake TLS tanpa verifikasi sertifikat (deploy ini self-signed).
Yang diuji hanya: TLS tersedia dan handshake selesai. Verifikasi nama/host
sengaja tidak dilakukan di sini — itu urusan klien masing-masing
(mis. OM memakai `verify: /etc/ssl/trino/trino.crt`).
"""
ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE
try:
with socket.create_connection((host, port), timeout=timeout) as raw:
with ctx.wrap_socket(raw, server_hostname=host) as tls:
_ = tls.version()
return True, ""
except OSError as exc:
return False, str(exc)
def http_date(host: str, port: int, timeout: float) -> datetime | None:
"""Ambil header `Date` dari respons HTTP paling sederhana (tanpa dependensi)."""
try:
with socket.create_connection((host, port), timeout=timeout) as sock:
sock.sendall(f"HEAD / HTTP/1.0\r\nHost: {host}\r\n\r\n".encode())
data = sock.recv(2048).decode("latin-1", errors="ignore")
except OSError:
return None
for line in data.splitlines():
if line.lower().startswith("date:"):
try:
return parsedate_to_datetime(line.split(":", 1)[1].strip())
except (ValueError, TypeError):
return None
return None
def clock_skew(host: str, port: int, timeout: float, report: Report) -> None:
remote = http_date(host, port, timeout)
if remote is None:
return
if remote.tzinfo is None:
remote = remote.replace(tzinfo=timezone.utc)
skew = abs((datetime.now(timezone.utc) - remote).total_seconds())
if skew > 60:
report.warn(f"{host}: selisih waktu {skew:.0f}s (>60s) — JWT/OIDC sensitif; "
f"pastikan NTP aktif di host itu")
else:
report.ok(f"{host}: selisih waktu {skew:.0f}s (aman)")
def main(argv: list[str]) -> int:
if len(argv) < 2:
sys.stderr.write(__doc__ or "")
return 2
inv_path = argv[1]
compose_path = "docker-compose.yml"
deploy_dir = Path(".")
timeout = 3.0
rest = argv[2:]
while rest:
flag = rest.pop(0)
if flag == "--compose" and rest:
compose_path = rest.pop(0)
elif flag == "--deploy" and rest:
deploy_dir = Path(rest.pop(0))
elif flag == "--timeout" and rest:
timeout = float(rest.pop(0))
else:
sys.stderr.write(f"argumen tak dikenal: {flag}\n")
return 2
report = Report()
print(f"{BLD}Preflight lintas host{NC}")
print(f" inventory : {inv_path}")
print(f" compose : {compose_path}")
print(f" {YLW}preflight hanya MEMERIKSA — tidak menyalakan apa pun. Urutan start "
f"antar host tetap manual (belum ada orkestrator).{NC}")
print()
# ── 1. inventory ────────────────────────────────────────────────────────
print(f"{BLD}A. Inventory{NC}")
doc = inventory.apply_preset(inventory.load(inv_path))
errors = inventory.validate(doc)
if errors:
for e in errors:
report.bad(f"inventory: {e}")
return report.recap()
plan = inventory.plan(doc)
report.ok(f"inventory valid — preset '{doc.get('preset') or 'none'}' · {len(plan)} host")
for host, spec in plan.items():
report.ok(f" {host:9s} {spec['address']:15s} tiers=[{','.join(spec['tiers'])}]")
# ── 2. port internal yang dibutuhkan lintas host ─────────────────────────
print(f"\n{BLD}B. Port lintas host{NC}")
base = generate.load_compose(compose_path)
placement = inventory.resolve(doc)
needed = generate.required_cross_host(base, deploy_dir, plan, placement)
if not needed:
report.skip("tidak ada komponen yang dikonsumsi dari host lain")
for component in sorted(needed):
owners = placement.get(component) or []
ports = generate.INTERNAL_PORTS.get(component) or []
if len(owners) > 1:
report.skip(f"{component}: di {len(owners)} host (scale-out) — "
f"periksa manual {owners}")
continue
if not owners:
report.bad(f"{component}: tidak ada host pemilik"); continue
address = plan[owners[0]]["address"]
for port in ports:
open_ok, why = tcp_open(address, port, timeout)
if open_ok:
report.ok(f"{component} {address}:{port} terjangkau")
else:
report.bad(f"{component} {address}:{port} TIDAK terjangkau ({why})")
# ── 3. TLS Trino coordinator ─────────────────────────────────────────────
print(f"\n{BLD}C. TLS Trino coordinator{NC}")
trino_owners = placement.get("trino-coordinator") or []
if trino_owners:
addr = plan[trino_owners[0]]["address"]
handshake_ok, why = tls_handshake(addr, 8443, timeout)
if handshake_ok:
report.ok(f"TLS handshake coordinator {addr}:8443 berhasil")
else:
report.bad(f"TLS handshake coordinator {addr}:8443 gagal ({why})")
clock_skew(addr, 8443, timeout, report)
else:
report.skip("tidak ada trino-coordinator di inventory")
# ── 4. waktu antar host (lewat layanan HTTP) ─────────────────────────────
print(f"\n{BLD}D. Waktu antar host{NC}")
checked = 0
for component in ("keycloak", "nessie", "opa"):
owners = placement.get(component) or []
port = {"keycloak": 8080, "nessie": 19120, "opa": 8181}[component]
if len(owners) != 1:
continue
addr = plan[owners[0]]["address"]
if tcp_open(addr, port, timeout)[0]:
clock_skew(addr, port, timeout, report)
checked += 1
if checked == 0:
report.skip("tidak ada layanan HTTP yang terjangkau untuk memeriksa waktu "
"(pastikan NTP aktif di semua host)")
return report.recap()
if __name__ == "__main__":
raise SystemExit(main(sys.argv))
deploy/lib/common.sh
+234
View File
@@ -0,0 +1,234 @@
#!/usr/bin/env bash
# =============================================================================
# eBdesk SiCABE — Common Library (shared by install.sh, bootstrap.sh, test-verification.sh)
# =============================================================================
# ─── Color & Styling ─────────────────────────────────────────────────────────
export RED='\033[0;31m'
export GREEN='\033[0;32m'
export YELLOW='\033[1;33m'
export BLUE='\033[0;34m'
export CYAN='\033[0;36m'
export BOLD='\033[1m'
export NC='\033[0m'
export INFO="${CYAN}[INFO]${NC}"
export OK="${GREEN}[OK]${NC}"
export WARN="${YELLOW}[WARN]${NC}"
export ERR="${RED}[ERROR]${NC}"
export STEP="${BOLD}[STEP]${NC}"
export PASS="${GREEN}[PASS]${NC}"
export FAIL="${RED}[FAIL]${NC}"
# ─── Named Constants (Timeouts) ─────────────────────────────────────────────-
# Default max wait times in seconds (override via env var)
export WAIT_SHORT="${WAIT_SHORT:-30}" # ~30s — quick services
export WAIT_MEDIUM="${WAIT_MEDIUM:-60}" # ~60s — most services
export WAIT_LONG="${WAIT_LONG:-120}" # ~120s — slow services (Keycloak, StarRocks)
export WAIT_EXTRA="${WAIT_EXTRA:-180}" # ~180s — OpenMetadata
export SLEEP_STANDARD="${SLEEP_STANDARD:-2}" # polling interval
export SLEEP_SLOW="${SLEEP_SLOW:-5}" # slow polling (StarRocks)
# ─── Helper: Wait for HTTP endpoint ─────────────────────────────────────────
# Usage: wait_for_service "name" "http://url/health" [max_attempts]
# Selalu return 0 (non-fatal): readiness wait saja. Kegagalan nyata ditangkap
# test-verification.sh. Return non-zero akan mematikan script di bawah `set -e`.
wait_for_service() {
local name="$1"
local url="$2"
local max_attempts="${3:-${WAIT_MEDIUM}}"
local attempt=0
local sleep_time="${SLEEP_STANDARD}"
echo -e "${INFO} Menunggu ${name} siap (maks ${max_attempts}s)..."
while [[ ${attempt} -lt ${max_attempts} ]]; do
if curl -skf "${url}" >/dev/null 2>&1; then
echo -e "${OK} ${name} siap!"
return 0
fi
attempt=$((attempt + sleep_time))
sleep "${sleep_time}"
done
echo -e "${WARN} ${name} belum siap dalam ${max_attempts}s. Lanjutkan... (cek nanti: curl ${url})"
return 0
}
# ─── Helper: Wait for container readiness via Docker inspect ────────────────
# Usage: wait_for_container "container_name" [max_attempts]
wait_for_container() {
local container="$1"
local max_attempts="${2:-${WAIT_MEDIUM}}"
local attempt=0
local sleep_time="${SLEEP_STANDARD}"
echo -e "${INFO} Menunggu container ${container} running..."
while [[ ${attempt} -lt ${max_attempts} ]]; do
if docker ps --format "{{.Names}}" | grep -q "${container}"; then
echo -e "${OK} Container ${container} running."
return 0
fi
attempt=$((attempt + sleep_time))
sleep "${sleep_time}"
done
echo -e "${WARN} Container ${container} belum running dalam ${max_attempts}s."
return 1
}
# ─── Helper: JSON field extractor ───────────────────────────────────────────
# Usage: extract_json "field_name" < <json_string>
# Requires python3
extract_json() {
local field="$1"
python3 -c "import sys,json; print(json.load(sys.stdin).get('${field}',''))" 2>/dev/null || echo ""
}
# ─── Helper: Check if a command exists ──────────────────────────────────────
check_command() {
local cmd="$1"
local name="${2:-$1}"
if command -v "${cmd}" &>/dev/null; then
echo -e "${OK} ${name}: tersedia ($(command -v "${cmd}"))"
return 0
else
echo -e "${WARN} ${name} tidak ditemukan."
return 1
fi
}
# ─── Helper: Print section header ───────────────────────────────────────────
print_section() {
local title="$1"
echo ""
echo -e "${BOLD}${BLUE}═══════════════════════════════════════════════════════${NC}"
echo -e "${BOLD} ${title}${NC}"
echo -e "${BOLD}${BLUE}═══════════════════════════════════════════════════════${NC}"
echo ""
}
# ─── Helper: Muat berkas env (dukung mode multi-host) ───────────────────────
# Prioritas sumber env: --env-file <berkas> > $SICABE_ENV_FILE > <base>/.env
#
# KENAPA: di multi-host, endpoint (POSTGRES_HOST, TRINO_URL, KEYCLOAK_URL, …) BERBEDA
# per host dan ada di `generated/host-<x>/.env` — BUKAN di `deploy/.env`. Tanpa ini
# skrip memakai nilai single-host (localhost) dan memeriksa/menyentuh endpoint salah.
#
# Dipakai: `sicabe_load_env "${SCRIPT_DIR}" "$@"` — GANTI blok `source .env` inline.
# - `--env-file <berkas>` dipilih (relatif dihitung dari <base>).
# - `$SICABE_ENV_FILE` dihormati agar sub-script yang dipanggil `bootstrap.sh --host`
# otomatis mewarisi env host tanpa menimpa dengan deploy/.env.
# - `source` berjalan di shell PEMANGGIL (fungsi tidak membuat subshell), jadi
# variabel tetap terlihat setelah fungsi kembali.
sicabe_load_env() {
local _sicabe_base="$1"; shift
local _sicabe_override="${SICABE_ENV_FILE:-}" _sicabe_prev="" _sicabe_a
for _sicabe_a in "$@"; do
[[ "${_sicabe_prev}" == "--env-file" ]] && _sicabe_override="${_sicabe_a}"
_sicabe_prev="${_sicabe_a}"
done
local _sicabe_envf="${_sicabe_override:-${_sicabe_base}/.env}"
[[ "${_sicabe_envf}" != /* ]] && _sicabe_envf="${_sicabe_base}/${_sicabe_envf}"
if [[ ! -f "${_sicabe_envf}" ]]; then
if [[ -n "${_sicabe_override}" ]]; then
echo -e "${ERR} env-file tidak ditemukan: ${_sicabe_envf}" >&2
return 1
fi
return 0
fi
set -a
# shellcheck disable=SC1090
source "${_sicabe_envf}"
set +a
# Direktori bundle (config + docker-compose.yml + .env) — dipakai helper host-aware
# untuk tahu container mana milik host ini (lihat sicabe_host_expects_container).
SICABE_BUNDLE_DIR="$(dirname "${_sicabe_envf}")"
[[ -n "${_sicabe_override}" ]] && echo -e "${INFO} env-file: ${_sicabe_envf} (mode host)" >&2
return 0
}
# ─── Helper: URL endpoint efektif (sadar EXTERNAL & multi-host) ─────────────
# Urutan: (1) <url_var> eksplisit NON-localhost (komponen EXTERNAL) → pakai apa adanya;
# (2) multi-host (bundle memuat SICABE_HOST_NAME) → scheme://<host_var>:<port>;
# (3) selain itu → scheme://localhost:<port> (single-host).
# Dipakai `scripts/verify-phase*` agar endpoint menunjuk ALAMAT HOST di multi-host,
# tanpa merusak dukungan komponen existing (single-host tetap localhost; external
# tetap memakai URL yang diberikan operator).
sicabe_local_url() { # <scheme> <host_var> <port> [url_var]
local _slu_s="$1" _slu_hv="$2" _slu_p="$3" _slu_uv="${4:-}" _slu_u="" _slu_h=""
[[ -n "${_slu_uv}" ]] && _slu_u="${!_slu_uv:-}"
if [[ -n "${_slu_u}" && "${_slu_u}" != *localhost* && "${_slu_u}" != *127.0.0.1* ]]; then
echo "${_slu_u}"; return 0
fi
_slu_h="${!_slu_hv:-}"
# Remote ⟺ nilai berupa ALAMAT (mengandung titik: IP pemilik, mis. 100.112.76.124).
# Nama layanan tanpa titik (mis. `airflow-webserver`, `keycloak`, `openmetadata-server`)
# = komponen CO-LOCATED → dari host dijangkau via localhost. Uji LAMA ("≠ localhost")
# keliru menganggap nama layanan sebagai remote → `http://airflow-webserver:12082`
# (nama container + port HOST) yang tak bisa diakses dari host.
if [[ -n "${SICABE_HOST_NAME:-}" && "${_slu_h}" == *.* && "${_slu_h}" != "127.0.0.1" ]]; then
echo "${_slu_s}://${_slu_h}:${_slu_p}"; return 0
fi
echo "${_slu_s}://localhost:${_slu_p}"
}
# ─── Helper: HOST endpoint efektif (sadar EXTERNAL & multi-host) ────────────
# Urutan: (1) <fallback_var> bila NON-localhost (override operator / komponen EXTERNAL);
# (2) multi-host (SICABE_HOST_NAME) → <host_var>;
# (3) selain itu → localhost.
# Dipakai untuk variabel yang di skrip SEKALIGUS menjadi penanda external
# (mis. STARROCKS_HOST/AIRFLOW_HOST): penanda tetap memakai nilai operator, sedangkan
# koneksi memakai host efektif ini.
sicabe_local_host() { # <host_var> [fallback_var]
local _slh_hv="$1" _slh_fb="${2:-}" _slh_f=""
if [[ -n "${_slh_fb}" ]]; then
_slh_f="${!_slh_fb:-}"
# Remote = ALAMAT (mengandung titik). Nama layanan tanpa titik = co-located → localhost.
if [[ "${_slh_f}" == *.* && "${_slh_f}" != "127.0.0.1" ]]; then
echo "${_slh_f}"; return 0
fi
fi
local _slh_h="${!_slh_hv:-}"
if [[ -n "${SICABE_HOST_NAME:-}" && "${_slh_h}" == *.* && "${_slh_h}" != "127.0.0.1" ]]; then
echo "${_slh_h}"; return 0
fi
echo "localhost"
}
# ─── Helper: apakah container milik HOST INI? ───────────────────────────────
# Single-host (SICABE_HOST_NAME kosong) → selalu true (semua lokal).
# Multi-host (dijalankan dgn `--env-file generated/host-<x>/.env`) → true hanya bila
# container ada di bundle host ini. Dipakai gate agar container host LAIN di-SKIP
# (bukan FAIL) — sama seperti mode host di test-verification.sh.
# Daftar container di-cache (1× `docker compose config` per proses).
sicabe_host_expects_container() { # <nama-container>
[[ -z "${SICABE_HOST_NAME:-}" ]] && return 0
local _shc_dir="${SICABE_BUNDLE_DIR:-.}"
if [[ -z "${SICABE_BUNDLE_CONTAINERS+x}" ]]; then
SICABE_BUNDLE_CONTAINERS="$(docker compose -f "${_shc_dir}/docker-compose.yml" \
--env-file "${_shc_dir}/.env" --profile all config 2>/dev/null \
| python3 -c "import sys,yaml;d=yaml.safe_load(sys.stdin) or {};print('\n'.join((s.get('container_name') or '') for s in (d.get('services') or {}).values()))" \
2>/dev/null || true)"
fi
printf '%s\n' "${SICABE_BUNDLE_CONTAINERS}" | grep -qx "$1"
}
# ─── Helper: wajibkan container LOKAL (kalau bukan → instruksi + kode 3) ─────
# Dipakai skrip seed/reapply yang `docker exec` container tertentu. Di multi-host,
# bila container itu milik host LAIN, skrip harus dijalankan DI host pemiliknya.
sicabe_require_local_container() { # <container> <nama-komponen> <host-contoh>
sicabe_host_expects_container "$1" && return 0
echo -e "${WARN} '${1}' (${2}) berjalan di host LAIN — skrip ini harus dijalankan DI host itu." >&2
echo -e "${WARN} contoh: ./scripts/${0##*/} --env-file generated/host-${3}/.env (di host ${3})" >&2
return 3
}
# ─── Helper: pastikan sumber bind-mount berupa FILE (bukan direktori) ─────────
# Docker membuat DIRECTORY bila sumber bind-mount sebuah file TIDAK ADA saat `up`
# (bind mount ke path non-existent di host). Akibatnya service gagal, mis.
# Airflow SimpleAuthManager: IsADirectoryError '/opt/airflow/simple_auth_manager_pa...'.
# Bila path sudah berupa direktori (sisa `up` sebelumnya) → buang; lalu pastikan ada file.
sicabe_ensure_file_mount() { # <path> [mode]
local path="$1" mode="${2:-644}"
if [[ -d "${path}" ]]; then rm -rf "${path}"; fi
[[ -e "${path}" ]] || : > "${path}"
chmod "${mode}" "${path}" 2>/dev/null || true
}
deploy/output/a1-verifikasi-lock.txt
+14
View File
@@ -0,0 +1,14 @@
# A1 (R18) — verifikasi pin sqlalchemy<2.1 tidak menggeser paket
# tanggal : 2026-10-01 21:04:37
# metode : uv pip compile requirements.in --constraint <lock lama>
# base : apache/airflow:slim-3.3.1-python3.11
# lock lama : 1188 baris · md5 5936cf19f36004fd
# lock baru : 1650 baris
# baris PAKET berbeda : 0
# sqlalchemy : sqlalchemy==2.0.54 -> sqlalchemy==2.0.54
#
# KESIMPULAN
# 0 paket berubah — pin ini PREVENTIF (mencegah regenerasi lock mendatang
# menarik SQLAlchemy 2.1). Lock dibiarkan utuh karena isi paketnya identik;
# perbedaannya hanya baris komentar anotasi constraint (-c /old.lock).
# Perlindungan ada di requirements.in (sumber), bukan di lock.
deploy/output/b3-hasil-uji.md
+114
View File
@@ -0,0 +1,114 @@
# Hasil Uji B3 — Titik Go/No-Go BATCH B (review R18)
> **Tanggal**: 2026-10-01 · **Image diuji**: `sicabe/airflow:3.3.2-om203-py313` (ID `33990a6c2b88`)
> **Metode**: stack eksperimen terisolasi (`airflow332-exp`, port 12184/12185) + config & DAG
> produksi di-mount **read-only** + uji koneksi/OPA/dbt dari container sekali pakai.
> ⚠️ Produksi **tidak tersentuh** sepanjang uji.
---
## 1. VERDICT: 🔴 **NO-GO untuk Python 3.13**
BATCH B **tidak boleh dilanjutkan** dengan base Python 3.13. Penyebab tunggal: **dbt tidak
berfungsi** di Python 3.13 pada tumpukan dependensi kita — dan dbt adalah tulang punggung
lapisan silver/gold.
**Yang tetap sahih**: seluruh hasil B3 lainnya, serta A1 (pin `sqlalchemy<2.1`), A2, dan B1.
**Yang perlu diulang**: hanya **B2** (build image) dengan base Python **3.11**.
---
## 2. Yang LOLOS ✅
| Uji | Hasil |
|---|---|
| **DAG produksi (4 berkas, dari volume produksi)** | ✅ `dag_sitomat_gold_build.py`, `dag_sitomat_gold_build_prod.py` **impor bersih**. `sicabe_dq_gold_native.py` berhasil dimuat sampai tahap *generate workflow* (gagal hanya karena `openmetadata-server` tidak ter-resolve dari container uji saya — bukan masalah versi) |
| **Plugin OpenMetadata** | ✅ `/pluginsv2/api/v2/openmetadata/health-auth` → **200**; `/pluginsv2/` → **200** |
| **Trino** | ✅ `SELECT 1` → `[[1]]`; `SHOW TABLES FROM iceberg.bronze` → `orders`, `orders_incremental` |
| **MinIO** | ✅ bucket `['iceberg-data']` via `S3Hook` |
| **MySQL (klien eksternal)** | ✅ `('8.0.40', 'asdp')` via `MySQLdb`/`MySqlHook` |
| **PostgreSQL** | ✅ Postgres 15.18 via `PostgresHook` |
| **OPA** | ✅ aturan bronze utuh (`bronze_orders`: ddl/select/write) + keputusan INSERT bronze = **true** |
| **Metadata DB / migrasi** | ✅ `airflow db migrate` bersih; OM 2.0.3 (API `{"version":"2.0.3"}`) |
---
## 3. Yang GAGAL 🔴 — dbt
### 3.1 Gejala
```
$ dbt debug --profile trino --target prod
AttributeError: __class_getitem__
mashumaro/core/meta/helpers.py:624 → typing_extensions.Annotated.__class_getitem__
/usr/python/lib/python3.13/typing.py:548
```
`dbt compile` gagal dengan galat yang sama.
⚠️ **`dbt --version` justru berhasil** — sehingga pemeriksaan dangkal (`import dbt`,
`dbt --version`) **tidak menangkap** kerusakan ini.
### 3.2 Akar masalah & tiga versi yang diuji
| `mashumaro` | dbt (`debug`/`compile`) | `pip check` |
|---|---|---|
| **3.10** (maksimum yang diizinkan `collate-data-diff <3.11`) | ❌ **GAGAL** | ✅ **bersih** ← *rusak, tapi terlihat sehat* |
| **3.17** | ✅ **debug OK ("All checks passed!") + compile OK** (SQL `orders_silver` ter-compile) | ⚠️ melanggar `dbt-core <3.15` **dan** `collate-data-diff <3.11` |
| **3.23** | ✅ berhasil | ⚠️ melanggar keduanya |
### 3.3 Kesimpulan teknis
**Tidak ada versi `mashumaro` yang sekaligus (a) memenuhi batas dependensi kita dan
(b) berfungsi di Python 3.13.** Yang mengikat ketat adalah `collate-data-diff` (dipakai jalur
lineage dbt OpenMetadata) yang menuntut `<3.11.0`, sedangkan perbaikan kompatibilitas 3.13
berada di versi ≥ ~3.11.
➡️ Jadi masalahnya bukan pilihan paket, melainkan **kombinasi Python 3.13 + tumpukan ini**.
### 3.4 Pelajaran metodologis (penting)
`pip check` **bersih** pada konfigurasi yang **tidak berfungsi**. Artinya:
> Batas dependensi yang dideklarasikan ≠ kebenaran fungsional.
Uji kelayakan untuk tumpukan ini **wajib menjalankan perintah nyata** (mis. `dbt debug`,
`dbt compile`), bukan hanya mengimpor modul atau membaca `--version`.
---
## 4. Koreksi atas rekomendasi K1
Rekomendasi awal saya (K1 = Python **3.13**) dinilai **salah** dan dicabut. Dasarnya waktu itu
adalah "artefak eksperimen sudah tervalidasi end-to-end" — padahal validasi itu **tidak
lengkap**: eksperimen memeriksa `import dbt` dan `dbt --version`, tetapi **tidak pernah
menjalankan `dbt` sungguhan**. B3 — yang memang dirancang sebagai gerbang — yang menemukannya.
**K1 direvisi menjadi Python 3.11**, dengan alasan:
- tujuan BATCH B adalah **Airflow 3.3.2 + OM 2.0.3 + provider MySQL**, bukan Python 3.13;
- `openmetadata-ingestion[airflow]` 2.0.3.0 mem-pin `apache-airflow==3.3.2` **terlepas dari
Python** → 3.3.2 di Python 3.11 tetap koheren;
- Python 3.11 mempertahankan seluruh dependensi lain di set yang **sudah terbukti di produksi**
→ risiko jauh lebih rendah.
---
## 5. Langkah berikutnya (B2 revisi)
| Sub | Isi |
|---|---|
| B2r.1 | `Dockerfile.332` → base `slim-3.3.2-python3.11` (stage builder `mysqlclient` disesuaikan dengan kondisi header Python 3.11 pada base itu) |
| B2r.2 | Regenerasi lock 3.3.2/py3.11 **dengan lock 3.3.1 sebagai `--constraint`** (platform identik → diff minimal; teknik yang terbukti di A1) |
| B2r.3 | Build ulang image (tag tetap `sicabe/airflow:3.3.2-om203-py313`, menimpa image yang no-go) |
| B3r | **Ulangi B3 dengan `dbt debug` + `dbt compile` sebagai syarat WAJIB** (bukan opsional) |
---
## 6. Catatan kesalahan uji saya (agar tidak terulang)
| # | Kesalahan | Akibat |
|---|---|---|
| 1 | Memakai `airflow dags list` untuk menguji folder yang di-mount | `dags list` membaca **metadata DB**, bukan folder → uji pertama tidak sah (diperbaiki dengan impor langsung tiap berkas DAG) |
| 2 | Menguji DAG dari container yang tidak berada di jaringan yang tepat | `openmetadata-server` tidak ter-resolve → kegagalan palsu |
| 3 | Mount dbt **read-only** | dbt perlu menulis `logs/` → keluar `exit 2` **tanpa keluaran sama sekali** (menyesatkan) |
| 4 | `pip install "mashumaro>=3.9,<3.18"` tanpa `--upgrade` | pip tidak menaikkan (3.10 sudah memenuhi rentang) → uji sempat tidak konklusif |
| 5 | Salah mount point volume config OM | kegagalan palsu pada DAG DQ |
| 6 | Pemeriksaan awal hanya `import dbt` / `dbt --version` | kerusakan dbt lolos sampai B3 |
deploy/output/b3r-hasil-uji.md
+72
View File
@@ -0,0 +1,72 @@
# Hasil B3 ULANG — Go/No-Go BATCH B pada Image Python 3.11 (review R18)
> **Tanggal**: 2026-10-01 · **Image diuji**: `sicabe/airflow:3.3.2-om203` (ID `a02620e1d0a8`)
> — hasil **B2 revisi** (base Python 3.11), menggantikan image py3.13 yang ditolak.
> **Metode**: sama dengan B3 pertama — stack eksperimen terisolasi + config & DAG produksi
> read-only + container sekali pakai untuk koneksi/dbt. ⚠️ Produksi tidak tersentuh.
---
## 1. VERDICT: 🟢 **GO**
BATCH B boleh dilanjutkan ke B4. Penyebab no-go sebelumnya (dbt mati di Python 3.13)
**teratasi** dengan kembali ke Python 3.11.
---
## 2. Hasil lengkap
| # | Uji | Hasil |
|---|---|---|
| 1 | `airflow version` | ✅ **3.3.2** |
| 2 | **`dbt debug`** | ✅ **"Connection test: OK connection ok" + "All checks passed!"** |
| 3 | **`dbt compile --select orders_silver`** | ✅ compiles (SQL model terbentuk) |
| 4 | DAG produksi `dag_sitomat_gold_build.py` | ✅ impor bersih |
| 5 | DAG produksi `dag_sitomat_gold_build_prod.py` | ✅ impor bersih |
| 6 | DAG `sicabe_dq_gold_native.py` | ✅ dimuat sampai tahap *generate workflow*; gagal hanya karena `openmetadata-server` tidak ter-resolve dari container uji (bukan masalah versi) |
| 7 | `airflow dags list-import-errors` | ✅ kosong |
| 8 | Plugin OM `/pluginsv2/api/v2/openmetadata/health-auth` | ✅ **HTTP 200** |
| 9 | Plugin OM `/pluginsv2/` | ✅ **HTTP 200** |
| 10 | Trino (`TrinoHook`) | ✅ `SELECT 1` → `[[1]]`; bronze: `orders`, `orders_incremental` |
| 11 | MinIO (`S3Hook`) | ✅ bucket `['iceberg-data']`, `check_for_bucket` True |
| 12 | MySQL klien eksternal (`MySqlHook`) | ✅ `('8.0.40', 'asdp')`; tabel `passenger_productions`, `trx_booking` |
| 13 | OPA (aturan bronze `data-engineer`) | ✅ utuh (`bronze_orders`: ddl/select/write) |
Verifikasi dari **build** (step #18–#24): 5 klien DB, `mysqlclient` C termuat +
`libmariadb.so.3`, provider mysql, `dbt + great_expectations`, `flask_appbuilder` +
**plugin OM dapat dimuat** (`RestApiPlugin`), `pip check` bersih.
---
## 3. Ringkasan perjalanan B2 (tiga percobaan)
| Percobaan | Masalah | Penyelesaian |
|---|---|---|
| **1** | Build memakai Python 3.13 → **dbt mati** (mashumaro/Annotated) | Diputuskan pindah ke Python 3.11 (lihat `b3-hasil-uji.md`) |
| **2** | Build py3.11: pip install 62 menit (jaringan PyPI ~60–80 kB/s) lalu gagal di **penjaga regresi yang saya tulis** | Penjaga dipanggil dengan 1 argumen → `TypeError: Annotated[...] should be used with at least two arguments`. Diperbaiki menjadi `te.Annotated[int, 'meta']`. ⚠️ Galat itu justru membuktikan jalur Annotated **ada** di py3.11 |
| **3** | — | ✅ **BERHASIL** (langkah pip ter-cache → selesai cepat) |
Catatan operasional: build percobaan 1 sempat **dibatalkan** karena client `docker build`
ikut mati saat shell pengawas dihentikan (`nohup` tidak cukup — perlu `setsid`).
---
## 4. Konsekuensi & langkah berikutnya
- **Image target BATCH B**: `sicabe/airflow:3.3.2-om203` (ID `a02620e1d0a8`, 3.2 GB).
- Image py3.13 (`3.3.2-om203-py313`) **tidak dipakai** — disimpan sebagai bukti no-go.
- `Dockerfile.332` + `requirements.332.in/.lock` sudah final untuk py3.11; lock hanya
**44 baris** berbeda dari produksi.
- Berikutnya: **B4** — naikkan `AIRFLOW_IMAGE` + `OPENMETADATA_VERSION` di
`environment.defaults.sh` lalu `docker compose up -d`.
- Bukti rollback: `deploy/output/merge-airflow332-rollback.md` (dump sudah diuji pulih).
---
## 5. Kesalahan uji saya pada B3r (agar tidak terulang)
| # | Kesalahan | Akibat |
|---|---|---|
| 1 | Penjaga regresi `Annotated.__class_getitem__((int,))` — argumen kurang satu | **Positif palsu**: build gagal padahal sehat |
| 2 | Uji MinIO hanya meng-`source deploy/.env`, tanpa `environment.defaults.sh` | `MINIO_ROOT_USER` kosong → `NoCredentialsError` (bukan masalah image) |
| 3 | `pgrep -f` mencocokkan perintah saya sendiri | PID yang dilaporkan tidak sahih saat memantau build |
deploy/output/merge-airflow332-rollback.md
+178
View File
@@ -0,0 +1,178 @@
# Rollback — Merge Eksperimen `airflow332` ke `deploy/` (BATCH B / review R18)
> **Dibuat**: 2026-10-01 · **Oleh**: langkah **B1** dari `reviews/sicabe-18-merge-airflow332-plan.md`
> **Status**: 🟢 **Backup siap & TERBUKTI DAPAT DIPULIHKAN.** BATCH B **belum dijalankan**.
> ⚠️ **Berkas ini adalah satu-satunya jalan pulang** untuk langkah B4 (migrasi OpenMetadata
> bersifat **satu arah** — lihat `docs/sicabe-install-phases-detail.md` §8.11.2).
---
## 1. Keadaan yang dibekukan (baseline sebelum BATCH B)
| Item | Nilai |
|---|---|
| Tanggal baseline | 2026-10-01 |
| Airflow | **3.3.1** (`sicabe/airflow:3.3.1-om202`) |
| OpenMetadata | **2.0.2** (`openmetadata/server:2.0.2`) |
| OpenSearch | `opensearchproject/opensearch:3.8.0` |
| Gate `test-verification.sh` | **60 PASS / 0 FAIL / 2 SKIP** |
| Container produksi | 20 (Airflow webserver/scheduler/dag-processor, OM, OpenSearch, dst) |
---
## 2. Berkas backup
Lokasi: `deploy/output/merge-airflow332-backup/` — **gitignored** (berisi data produksi).
| Berkas | Isi | Ukuran | MD5 |
|---|---|---|---|
| `openmetadata_db.dump` | dump `openmetadata_db` (format `-Fc`) | 1.9 MB | `a8495339165cfbc286ea7e90490c12bb` |
| `airflow.dump` | dump `airflow` (format `-Fc`) | 248 KB | `4a834cc43307b296b58c250909749d2a` |
| `environment.defaults.sh.before-B` | salinan `deploy/config/environment.defaults.sh` | 22 KB | `2d341e008af43a7b` |
Ukuran DB saat baseline: `openmetadata_db` = **36 MB**, `airflow` = **12 MB**.
---
## 3. Bukti backup DAPAT DIPULIHKAN (dua kali diuji)
| Uji | Hasil |
|---|---|
| Restore `openmetadata_db` → DB coretan `_restorecheck_om` | **204 tabel** asli = **204 tabel** hasil restore ✅ |
| Restore `airflow` → DB coretan `_restorecheck_af` | **5 DAG** asli = **5 DAG** hasil restore ✅ |
| DB coretan setelah uji | dihapus ✅ (produksi tidak tersentuh: 20 container, OM HTTP 200) |
Perintah yang dipakai (nilai persis):
```bash
# contoh untuk openmetadata_db
docker exec -e PGPASSWORD=<dari deploy/.env> sicabe-postgres \
psql -U sicabe_admin -d postgres -c "CREATE DATABASE _restorecheck_om OWNER openmetadata;"
docker cp deploy/output/merge-airflow332-backup/openmetadata_db.dump sicabe-postgres:/tmp/check.dump
docker exec -e PGPASSWORD=<…> sicabe-postgres \
pg_restore -U sicabe_admin -d _restorecheck_om --no-owner --no-privileges /tmp/check.dump
docker exec -e PGPASSWORD=<…> sicabe-postgres \
psql -U sicabe_admin -d postgres -c "DROP DATABASE _restorecheck_om;"
```
---
## 4. Digest image (untuk kembali ke versi lama)
| Image | Image ID (sha256, dipotong) |
|---|---|
| `sicabe/airflow:3.3.1-om202` (produksi saat ini) | `afc0421a0490…` |
| `sicabe/airflow:3.3.1-om202-presqlalchemy` (tag rollback A1) | `afc0421a0490…` (identik — A1 tidak rebuild) |
| **`sicabe/airflow:3.3.2-om203`** (target BATCH B — Python 3.11, hasil B2 revisi) | `a02620e1d0a8…` (3.2 GB) |
| ~~`sicabe/airflow:3.3.2-om203-py313`~~ — **DITOLAK** di B3 (dbt mati di Python 3.13) | `33990a6c2b88…` (tidak dipakai) |
| `openmetadata/server:2.0.2` | `3b6f52aeb30f…` |
| `openmetadata/server:2.0.3` (target BATCH B) | `5f4d52bca2d7…` |
| `opensearchproject/opensearch:3.8.0` | `fafe3fc35870…` |
---
## 5. PROSEDUR ROLLBACK
### 5.1 Kasus paling umum — B4/B5 gagal, skema OM masih sehat
Cukup kembalikan versi, tanpa menyentuh DB:
```bash
cd /home/broto/sicabe-doc/deploy
# (1) kembalikan versi ke baseline
# AIRFLOW_IMAGE -> sicabe/airflow:3.3.1-om202
# OPENMETADATA_VERSION -> 2.0.2
# (atau pulihkan berkas utuh: cp output/merge-airflow332-backup/environment.defaults.sh.before-B \
# config/environment.defaults.sh)
# (2) jalankan ulang
docker compose up -d airflow-webserver airflow-scheduler airflow-dag-processor \
openmetadata-server openmetadata-migrate
# (3) verifikasi
./test-verification.sh # harapan: 60 PASS / 0 FAIL / 2 SKIP
```
### 5.2 Kasus berat — migrasi OM 2.0.3 merusak skema
⚠️ **Urutannya penting**: OM produksi harus **berhenti** dulu sebelum restore, kalau tidak
koneksi aktif akan mengganggu `DROP/CREATE DATABASE`.
```bash
cd /home/broto/sicabe-doc/deploy
# (1) HENTIKAN OpenMetadata (wajib, sebelum menyentuh DB-nya)
docker stop sicabe-openmetadata
# (2) pulihkan versi image di config (OPENMETADATA_VERSION=2.0.2)
cp output/merge-airflow332-backup/environment.defaults.sh.before-B config/environment.defaults.sh
# (3) restore openmetadata_db dari dump
docker exec -e PGPASSWORD=<dari deploy/.env> sicabe-postgres \
psql -U sicabe_admin -d postgres -c "DROP DATABASE IF EXISTS openmetadata_db;"
docker exec -e PGPASSWORD=<…> sicabe-postgres \
psql -U sicabe_admin -d postgres -c "CREATE DATABASE openmetadata_db OWNER openmetadata;"
docker cp output/merge-airflow332-backup/openmetadata_db.dump sicabe-postgres:/tmp/om.dump
docker exec -e PGPASSWORD=<…> sicabe-postgres \
pg_restore -U sicabe_admin -d openmetadata_db --no-owner --no-privileges /tmp/om.dump
# (4) mulai ulang dengan image LAMA
docker compose up -d openmetadata-server
# (5) verifikasi
curl -s http://localhost:12085/api/v1/system/version # harapan: {"version":"2.0.2",...}
```
### 5.3 Bila metadata Airflow perlu dipulihkan (mis. migrasi DB Airflow bermasalah)
```bash
docker stop sicabe-airflow-webserver sicabe-airflow-scheduler sicabe-airflow-dag-processor
docker exec -e PGPASSWORD=<…> sicabe-postgres psql -U sicabe_admin -d postgres \
-c "DROP DATABASE IF EXISTS airflow;" -c "CREATE DATABASE airflow OWNER airflow;"
docker cp output/merge-airflow332-backup/airflow.dump sicabe-postgres:/tmp/af.dump
docker exec -e PGPASSWORD=<…> sicabe-postgres \
pg_restore -U sicabe_admin -d airflow --no-owner --no-privileges /tmp/af.dump
docker compose up -d airflow-webserver airflow-scheduler airflow-dag-processor
```
### 5.4 Yang TIDAK perlu di-rollback
- **OpenSearch** — indeksnya **derived**: OM membangun ulang isinya. Tidak ada dump khusus
untuk itu, dan itu memang disengaja.
- **`deploy-legacy/`** — di luar lingkup (K6).
- **Image** — tidak dihapus; dua versi hidup berdampingan, jadi rollback hanya soal menunjuk
tag lama.
---
## 6. Setelah rollback
1. Catat penyebab kegagalan di berkas ini (bagian bawah) atau di `reviews/sicabe-18…`.
2. Jangan jalankan B4 lagi sebelum penyebabnya dipahami.
3. Indeks/tabel OM hasil migrasi yang tertinggal **tidak** dibiarkan: pastikan restore
benar-benar menggantikan `openmetadata_db` (bukan menumpuk).
---
## 7. Catatan keamanan
⚠️ Berkas dump berisi **data produksi** (termasuk baris `user_entity` — hash kredensial, dan
mungkin data pribadi). Perlakuan:
- **gitignored** (`deploy/output/merge-airflow332-backup/`) — jangan pernah di-commit;
- hapus setelah BATCH B dinyatakan stabil (dan simpan hanya bila memang diperlukan sebagai
arsip, di lokasi terbatas akses).
---
## Riwayat
| Tanggal | Kejadian |
|---|---|
| 2026-10-01 | B1 dijalankan: dump dibuat, **diuji pulih (204=204 tabel; 5=5 DAG)**, digest & config dicatat, dokumen ini dibuat. BATCH B belum dimulai. |
| 2026-10-01 | **B2** dijalankan: image `sicabe/airflow:3.3.2-om203-py313` dibangun **dari `deploy/`** (berdampingan — `Dockerfile.332` + `requirements.332.{in,lock}`), ID `33990a6c2b88…`. Seluruh verifikasi fatal lolos: 5 klien DB, `mysqlclient` benar-benar termuat, `sqlalchemy 2.0.54` (pin A1), **plugin OM dapat dimuat**, `pip check` bersih. Produksi **tetap** `3.3.1-om202` — belum tersentuh. |
| 2026-10-01 | **B3 (go/no-go) = NO-GO** untuk Python 3.13: `dbt debug`/`dbt compile` gagal (mashumaro → `Annotated.__class_getitem__`). Tidak ada versi `mashumaro` yang memenuhi batas dependensi kita **dan** berfungsi di 3.13. Bukti: `b3-hasil-uji.md` |
| 2026-10-01 | **B2 revisi**: base dikembalikan ke **Python 3.11**; lock di-resolve ulang dengan lock 3.3.1 sebagai constraint **tersaring** (`^apache-airflow`, `^openmetadata-`, `^collate-` dibuang) → hanya **44 baris** berbeda dari produksi. Image baru: **`sicabe/airflow:3.3.2-om203`** ID `a02620e1d0a8…` (3.2 GB) |
| 2026-10-01 | **B3 ULANG = GO** 🟢: `dbt debug` **"All checks passed"**, `dbt compile` OK, DAG produksi impor bersih, plugin `/pluginsv2/` **200**, Trino/MinIO/MySQL OK, OPA utuh. Bukti: `b3r-hasil-uji.md`. Produksi tetap `3.3.1-om202` — siap untuk **B4** |
| 2026-10-02 | **B4 dijalankan (produksi naik)**: migrasi OM 2.0.3 **exit 0**; `airflow-init` (DB migrate 3.3.2) **exit 0**; OM server + 3 service Airflow di-recreate ke versi baru — semuanya **healthy**. Image: `sicabe/airflow:3.3.2-om203`, `openmetadata/server:2.0.3` |
| 2026-10-02 | **B5 verifikasi = LOLOS** ✅: `airflow version` **3.3.2**; OM API **2.0.3**; plugin `/pluginsv2/…health-auth` **200**; **0 error impor** DAG; `MySQLdb` + provider mysql tersedia di produksi; **GATE 60 PASS / 0 FAIL / 2 SKIP** (identik baseline). **B6 (rollback) tidak diperlukan.** |
| — | ⚠️ **Dump DISIMPAN** sampai dinyatakan stabil. Bila beberapa hari produksi berjalan normal: hapus `deploy/output/merge-airflow332-backup/` (§7) — atau arsipkan di lokasi terbatas akses |
deploy/output/merge-om20x-rollback.md
+114
View File
@@ -0,0 +1,114 @@
# Runbook Rollback — Integrasi OpenMetadata 2.0.2 ke `deploy/`
**Dibuat**: 2026-09-16 · **Tugas**: `merge-om20x` (subtask 01)
**Tag titik kembali**: `pre-merge-om20x` @ `5627353`
Runbook ini adalah dokumen yang ditulis tangan (bukan keluaran uji otomatis), sengaja
diletakkan di `deploy/output/` mengikuti penugasan subtask 01. Isinya harus tetap
akurat selama integrasi berlangsung.
---
## 1. Kapan runbook ini dipakai
Saat integrasi ke OpenMetadata 2.0.2 + OpenSearch 3.x + Airflow ber-plugin gagal dan
kita memutuskan kembali ke kondisi 1.13.1 yang stabil (sebelumnya gate Fase 1-6 hijau,
`test-verification.sh` = 60 PASS / 0 FAIL / 2 SKIP).
## 2. Kondisi baseline (terverifikasi sebelum perubahan apa pun)
| Aspek | Nilai |
|---|---|
| git HEAD | `5627353` (tag `pre-merge-om20x`) |
| OpenMetadata server | **1.13.1** (`openmetadata/server:1.13.1`) |
| OpenSearch | **2.19.6** (`opensearchproject/opensearch:2.19.6`) |
| Airflow image | **`sicabe/airflow:3.3.1-slim`** (id `15ae04b69228`), Airflow 3.3.1 |
| Plugin OpenMetadata di Airflow | **TIDAK ADA** — `/api/v1/openmetadata/health` → 404 |
| Data OM | database `openmetadata_db` di volume `sicabe_postgres-data` |
| Data OpenSearch | **tidak di volume** — ada di layer container (index derived) |
Catatan penting soal "plugin tidak ada": di Airflow deploy,
`/pluginsv2/api/v2/openmetadata/health-auth` menjawab **200 tetapi `content-type: text/html`**
(HTML aplikasi Airflow) — itu fallback UI, bukan respons plugin. Bukti plugin absen
adalah `/api/v1/openmetadata/health` → **404**. Jangan tertipu angka 200 di path
`/pluginsv2/`.
## 3. Cara memverifikasi kita benar-benar sudah kembali ke baseline
```bash
cd /home/broto/sicabe-doc
git rev-parse --short HEAD # harus 5627353 (atau turunannya yg setara)
docker inspect sicabe-openmetadata --format '{{.Config.Image}}' # openmetadata/server:1.13.1
docker inspect sicabe-opensearch --format '{{.Config.Image}}' # opensearchproject/opensearch:2.19.6
docker inspect sicabe-airflow-webserver --format '{{.Config.Image}}' # sicabe/airflow:3.3.1-slim
curl -s -o /dev/null -w '%{http_code}\n' -u "admin:$AIRFLOW_ADMIN_PASSWORD" \
http://localhost:12082/api/v1/openmetadata/health # harus 404
cd deploy && ./test-verification.sh # harus 60 PASS / 0 FAIL / 2 SKIP
```
## 4. Langkah rollback
### 4.1 Kembalikan berkas konfigurasi
```bash
cd /home/broto/sicabe-doc
git checkout pre-merge-om20x -- deploy/ docs/
git status -s # tinjau: hanya berkas deploy/docs yang kembali
```
Jangan `git reset --hard` — commit hasil eksperimen (`f47945d`, `5627353`) dan
`experiments/om20x/` tidak perlu ikut hilang.
### 4.2 Nyalakan ulang container lama
```bash
cd deploy
set -a && . ./.env && set +a
docker compose --profile all up -d --force-recreate \
openmetadata-opensearch openmetadata-migrate openmetadata-server \
airflow-init airflow-webserver airflow-scheduler airflow-dag-processor
```
Image lama **tidak boleh dihapus**: `openmetadata/server:1.13.1`,
`opensearchproject/opensearch:2.19.6`, `sicabe/airflow:3.3.1-slim`.
OpenSearch akan membuat index baru dari nol saat container dibuat ulang
(tidak ada volume; index memang derived dan OM akan mem-populasinya kembali).
### 4.3 Bila database OM sudah telanjur dikosongkan (subtask 12)
Setelah `openmetadata_db` di-drop, **data katalog OM tidak bisa dipulihkan dari
deploy** — tidak ada dump (keputusan sadar: jaring pengaman cukup tag git). Yang bisa
dilakukan: membuat ulang database kosong lalu membiarkan OM 1.13.1 membangun skemanya,
lalu mengisi ulang katalog dari sumber yang reproducible:
```bash
# buat ulang database kosong (pola sama dgn deploy/config/postgres/init/01-create-dbs.sh)
docker exec sicabe-postgres psql -U postgres -c "DROP DATABASE IF EXISTS openmetadata_db;"
docker exec sicabe-postgres psql -U postgres -c "CREATE DATABASE openmetadata_db OWNER openmetadata;"
cd deploy
docker compose --profile all up -d --force-recreate openmetadata-migrate # migrate schema 1.13.1
docker compose --profile all up -d openmetadata-server
./scripts/ingest-openmetadata.sh # katalog Trino + lineage dbt
# lalu jalankan DAG utk mengirim ulang hasil Great Expectations
```
## 5. Yang bisa dan tidak bisa dipulihkan (jujur)
| Bisa dipulihkan | Tidak bisa dipulihkan |
|---|---|
| Versi komponen (kembali 1.13.1 / 2.19.6 / image slim) | Isi katalog OM bila DB sudah di-drop (reproducible, bukan mustahil) |
| Konfigurasi (`deploy/`, `docs/`) via tag git | Riwayat DQ (test case + hasil) — dibangun ulang oleh DAG |
| Lineage dbt | Dashboard/artefak yang hanya ada di OM |
Alasan ini dapat diterima: seluruh isi katalog berasal dari
`scripts/ingest-openmetadata.sh` + DAG `sicabe_batch_daily`, jadi bisa dibangun ulang.
## 6. Titik tidak-balik (penting)
| Setelah subtask | Kondisi |
|---|---|
| 05 (+07) | Airflow memakai image ber-plugin; rollback = kembalikan 4 baris `AIRFLOW_IMAGE` |
| 10 | OpenSearch 3.x; index lama sudah hilang saat container dibuat ulang |
| **12** | **`openmetadata_db` di-drop → satu-satunya titik yang kehilangan data** |
| 13-15 | OM 2.0.2 jalan dgn password admin dari `.env` |
deploy/output/r18-6-e2e-hasil.md
+106
View File
@@ -0,0 +1,106 @@
# Hasil R18-6 — Uji End-to-End: OM membangkitkan DAG ingestion → Airflow 3.3.2
> **Tanggal**: 2026-10-01/02 · **Lingkungan**: PRODUKSI (Airflow 3.3.2 + OM 2.0.3)
> **Tujuan**: membuktikan jalur "OpenMetadata mengirim DAG ke Airflow" secara **utuh** —
> bukan hanya tingkat impor (itu sudah diukur di H7) — dan sekaligus menutup pertanyaan H7
> secara definitif.
> ⚠️ Uji ini **membuat** pipeline sementara di OM produksi lalu **menghapusnya kembali**.
---
## 1. VERDICT: 🟢 **BERHASIL PENUH — H7 tertutup secara definitif**
| Tahap | Hasil |
|---|---|
| OM membuat ingestion pipeline (`metadata`) untuk `trino_sicabe` | ✅ dibuat |
| OM men-deploy DAG ke Airflow (lewat plugin `/pluginsv2/`) | ✅ **HTTP 200** |
| DAG muncul di Airflow 3.3.2 | ✅ `r18_e2e_ingestion.py` |
| **Error impor** | ✅ **0** (`No data found`) |
| Trigger dari OM menjalankan DAG | ✅ HTTP 200 (percobaan ke-2) |
| **`dag_run`** | ✅ **`success`** (17 detik) |
| **`task_instance` (`ingestion_task`)** | ✅ **`success`** |
**Ringkasan dari log task (bukti terkuat):**
```
Processed records: 4 · Updated records: 0 · Errors: 0 · Success %: 100.0
Processed records: 5 · Updated records: 0 · Errors: 0 · Success %: 100.0
Workflow Success %: 100.0
Workflow finished in time: 6s 668.015ms
Done.
```
➡️ **Kesimpulan**: paket yang hilang akibat melepas extra `[airflow]`
(`tornado`, `elasticsearch`, `opensearchpy`, `providers-elasticsearch`,
`providers-opensearch`) **tidak mengganggu jalur ingestion metadata**. H7 kini bukan lagi
"terukur di tingkat impor", melainkan **terbukti sampai eksekusi**.
---
## 2. TEMUAN BARU — plugin memvalidasi LEBIH KETAT daripada server OM
Ini temuan integrasi yang belum tercatat sebelumnya, dan menjelaskan kegagalan deploy yang
sempat terjadi tiga kali:
```
POST /v1/services/ingestionPipelines/deploy/<id> → 400
{"error":"Extra parameter 'serviceName'"}
{"error":"Extra parameter 'markDeletedTablesFromFilterOnly'"}
{"error":"Extra parameter 'serviceConnections'"}
```
**Artinya**: pipeline yang **diterima** server OM saat dibuat (`POST /v1/services/ingestionPipelines`
→ 201) bisa **ditolak saat deploy** oleh plugin di sisi Airflow. Validasi plugin lebih ketat
daripada skema server.
**Yang bekerja** — `sourceConfig.config` **paling minimal**:
```json
{ "airflowConfig": { "scheduleInterval": "0 0 * * *", "retries": 0, "startDate": <epoch-ms> },
"name": "<nama>",
"pipelineType": "metadata",
"service": { "id": "<id service>", "type": "databaseService" },
"sourceConfig": { "config": { "type": "DatabaseMetadata" } } }
```
- **Jangan** menambahkan `serviceName`, `serviceConnections`, `includeTables`,
`markDeletedTablesFromFilterOnly`, dsb. pada `DatabaseMetadata` — ditolak sebagai
*extra parameter*.
- Koneksi diambil dari **service record** yang sudah ada (dibuat `bootstrap.sh`) — dan
**bekerja** ✅. (Catatan: ini berbeda dari jalur SDK yang punya masalah koneksi tersimpan
kehilangan secret → `401`; pada jalur OM-orchestrated koneksi tersimpan justru dipakai
dengan benar.)
- Untuk pipeline **TestSuite**, `serviceConnections` justru **diterima** (seeder DQ
memakainya) — jadi aturan "extra parameter" itu **per tipe pipeline**, bukan global.
---
## 3. Perilaku lain yang terkonfirmasi
| Perilaku | Bukti |
|---|---|
| **Trigger gagal di percobaan pertama**, berhasil di percobaan kedua | DAG baru dibuat → `dag-processor` belum selesai mem-parsing. Perlu jeda ±20 detik (seeder DQ sudah mengantisipasi ini) |
| `airflow dags list` membaca **metadata DB**, bukan folder | Sempat menyesatkan saat uji folder mount |
| CLI Airflow 3: `airflow dags list-runs` **bukan** bentuk yang valid | Status run paling andal dibaca dari tabel `dag_run`/`task_instance` di metadata DB |
| Menghapus pipeline (`hardDelete`) menghapus **berkas DAG** dari volume | File `r18_e2e_ingestion.py` hilang; catatan DAG di DB menjadi *stale* pada siklus parse berikutnya |
---
## 4. Kebersihan setelah uji
| Item | Status |
|---|---|
| Pipeline `r18_e2e_ingestion` di OM | ✅ **dihapus** (HTTP 200, `hardDelete`) |
| Berkas DAG di volume `airflow-dags` | ✅ **hilang** (kembali 4 berkas seperti semula) |
| Catatan DAG di metadata DB Airflow | ⏳ menjadi *stale* otomatis pada siklus parse berikutnya (berkasnya sudah tidak ada) |
| DAG produksi lain | ✅ tidak tersentuh |
---
## 5. Kesalahan saya selama R18-6 (agar tidak terulang)
| # | Kesalahan | Akibat |
|---|---|---|
| 1 | Menyusun `sourceConfig.config` dengan field lengkap ala UI | Deploy ditolak 3× (`Extra parameter …`) — seharusnya mulai dari yang **minimal** |
| 2 | Menganggap semua tipe pipeline menerima field yang sama | `serviceConnections` diterima di TestSuite, **ditolak** di DatabaseMetadata |
| 3 | Memakai `airflow dags list-runs` | Bentuk CLI berbeda di Airflow 3 → pakai tabel DB |
deploy/scripts/airflow/entrypoint.sh
Executable
+97
View File
@@ -0,0 +1,97 @@
#!/usr/bin/env bash
# =============================================================================
# eBdesk SiCABE — Airflow Entrypoint Wrapper
# =============================================================================
# 1. Install sling CLI (binary, not available via pip)
# 2. Delegate ke Airflow entrypoint asli untuk service start
# =============================================================================
set -euo pipefail
# ─── Percayai sertifikat TLS Trino (dev, self-signed) ───────────────────────
# Sebagian klien MEMVERIFIKASI TLS dan tidak punya opsi mematikannya — contohnya
# Great Expectations / trino client (`verify=True`) dan DQ native OpenMetadata.
# Alih-alih `append` ke certifi (dgn guard marker) — yang MEMBUAT certifi tetap
# memegang cert LAMA setelah cert Trino di-regenerasi — kita bangun ULANG bundle
# CA (certifi + cert Trino) setiap start, lalu arahkan REQUESTS_CA_BUNDLE /
# SSL_CERT_FILE ke sana. Verifikasi TLS TETAP aktif.
CA_SRC="/etc/ssl/trino/trino.crt"
SICABE_CA="/tmp/sicabe-ca.pem"
if [[ -f "${CA_SRC}" ]]; then
CERTIFI_BUNDLE="$(python -c 'import certifi;print(certifi.where())' 2>/dev/null || true)"
if [[ -n "${CERTIFI_BUNDLE}" && -f "${CERTIFI_BUNDLE}" ]]; then
if cat "${CERTIFI_BUNDLE}" "${CA_SRC}" > "${SICABE_CA}" 2>/dev/null; then
export REQUESTS_CA_BUNDLE="${SICABE_CA}"
export SSL_CERT_FILE="${SICABE_CA}"
echo "[SiCABE] CA bundle (certifi + Trino) → ${SICABE_CA} (selalu dibangun ulang)."
else
echo "[SiCABE] WARNING: gagal membangun CA bundle — lanjut."
fi
fi
fi
# ─── Install sling CLI jika belum ada (NON-BLOCKING, fail-fast) ─────────────
# Install ke ~/.local/bin (writable oleh user airflow, sudah di PATH)
if ! command -v sling &>/dev/null; then
echo "[SiCABE] Installing sling CLI (timeout 15s)..."
SLING_VER="1.5.23"
mkdir -p "${HOME}/.local/bin"
if curl -fL --max-time 15 --connect-timeout 5 \
"https://github.com/slingdata-io/sling-cli/releases/download/v${SLING_VER}/sling_linux_amd64.tar.gz" \
-o /tmp/sling.tar.gz 2>/dev/null && tar -xzf /tmp/sling.tar.gz -C /tmp/ 2>/dev/null && [[ -f /tmp/sling ]]; then
chmod +x /tmp/sling
mv /tmp/sling "${HOME}/.local/bin/sling"
echo "[SiCABE] Sling CLI v${SLING_VER} installed: $(sling --version 2>&1)"
else
echo "[SiCABE] WARNING: sling CLI download gagal — lanjut tanpa sling (DAG akan warning)."
fi
rm -rf /tmp/sling.tar.gz /tmp/sling 2>/dev/null || true
fi
# ─── Copy dbt project ke home airflow (writable) ────────────────────────────
# Mount /opt/airflow/dbt adalah read-only (host bind). dbt perlu menulis
# target/, logs/, dbt_packages/. Copy ke ~/dbt yang writable oleh airflow.
if [[ -d /opt/airflow/dbt/sicabe && ! -d "${HOME}/dbt/sicabe" ]]; then
echo "[SiCABE] Copying dbt project ke ${HOME}/dbt..."
mkdir -p "${HOME}/dbt"
cp -r /opt/airflow/dbt/sicabe "${HOME}/dbt/sicabe"
cp /opt/airflow/dbt/profiles.yml "${HOME}/dbt/profiles.yml"
echo "[SiCABE] dbt project siap di ${HOME}/dbt/sicabe"
fi
# ─── Copy Great Expectations ke home airflow (writable) ─────────────────────
if [[ -d /opt/airflow/great_expectations && ! -d "${HOME}/great_expectations" ]]; then
echo "[SiCABE] Copying Great Expectations ke ${HOME}/great_expectations..."
cp -r /opt/airflow/great_expectations "${HOME}/great_expectations"
echo "[SiCABE] Great Expectations siap di ${HOME}/great_expectations"
fi
# ─── LINK sling env.yaml ke ~/.sling (SELALU ikut file yang di-mount) ───────
# WAJIB symlink, bukan copy: bootstrap men-regenerasi /opt/airflow/sling/env.yaml
# SETELAH container start (host-aware), sehingga salinan hasil `cp` menjadi STALE
# (gejala: Sling "lookup postgres ... server misbehaving"). Symlink → perubahan
# langsung terbaca tanpa restart. ~/.sling/ tetap writable untuk .sling.db.
# NON-FATAL: jangan sampai crash-loop container (set -e).
# Dibuat TANPA syarat berkas ada: bila env.yaml belum ada saat start (bootstrap jalan
# SETELAH container), symlink "menggantung" lalu VALID begitu berkas muncul. Dulu ada
# guard `[[ -f … ]]` → symlink tak dibuat → Sling "could not find connection POSTGRES_DB".
mkdir -p "${HOME}/.sling"
rm -f "${HOME}/.sling/env.yaml" 2>/dev/null || true
ln -s /opt/airflow/sling/env.yaml "${HOME}/.sling/env.yaml" \
&& echo "[SiCABE] Sling env.yaml -> symlink ${HOME}/.sling/env.yaml" \
|| echo "[SiCABE] WARNING: gagal symlink sling env.yaml — lanjut."
# ─── dbt-trino http_scheme ──────────────────────────────────────────────────
# Sejak image di-build dari paket resmi (tanpa patch), http_scheme sudah HTTPS
# lewat config/airflow/dbt/profiles.yml — tidak ada patch yang perlu dibalik.
# ─── Delegasikan ke entrypoint asli Airflow ─────────────────────────────────
# Airflow official entrypoint ada di /entrypoint
if [[ -x /entrypoint ]]; then
exec /entrypoint "$@"
elif [[ -x /usr/bin/docker-entrypoint.sh ]]; then
exec /usr/bin/docker-entrypoint.sh "$@"
else
echo "[SiCABE] WARNING: Airflow entrypoint tidak ditemukan, menjalankan langsung..."
exec "$@"
fi
deploy/scripts/airflow/run-om-native-dq.py
+194
View File
@@ -0,0 +1,194 @@
#!/usr/bin/env python
# =============================================================================
# eBdesk SiCABE — JALUR B: Data Quality NATIVE OpenMetadata (OM yang menjalankan)
# =============================================================================
# Bedanya dengan Jalur A (Great Expectations):
# A: GE yang MENJALANKAN ekspektasi → hasilnya DIKIRIM ke OM
# (test case ber-platform "GreatExpectations").
# B: OPENMETADATA yang MENJALANKAN test (definisi bawaan OM:
# tableRowCountToBeBetween, columnValuesToBeNotNull, columnValuesToBeBetween)
# → hasilnya langsung tersimpan di OM (platform "OpenMetadata").
#
# Catatan implementasi (dua hal yang harus begini, sudah diverifikasi):
# 1. HARUS lewat `TestRunner.for_table`/`from_yaml` — `TestRunner(...)` mentah
# tidak memuat entity tabel sehingga run() gagal "Table entity not provided".
# 2. Koneksi data HARUS dibawa eksplisit lewat workflow (serviceConnection +
# verify=<sertifikat Trino>). Bila mengandalkan koneksi yang TERSIMPAN di OM,
# eksekusi gagal 401 'Invalid credentials' pada OM 1.13.1. `metadata ingest`
# CLI juga bukan jalur eksekusi TestSuite (registry source-nya terpisah).
#
# Jalankan DI DALAM container Airflow:
# docker exec -e OPENMETADATA_ADMIN_PASSWORD=... \
# sicabe-airflow-webserver python /opt/airflow/scripts/run-om-native-dq.py
#
# Exit code: 0 = semua test lulus; 1 = ada test gagal/aborted.
# =============================================================================
import base64
import json
import os
import sys
import urllib.request
from metadata.generated.schema.entity.services.connections.metadata.openMetadataConnection import (
OpenMetadataConnection,
)
from metadata.generated.schema.security.client.openMetadataJWTClientConfig import (
OpenMetadataJWTClientConfig,
)
from metadata.ingestion.ometa.ometa_api import OpenMetadata
from metadata.sdk.data_quality.runner import TestRunner
from metadata.sdk.data_quality.tests import (
ColumnValuesToBeBetween,
ColumnValuesToBeNotNull,
TableRowCountToBeBetween,
)
OM_API_URL = os.getenv("OM_API_URL", "http://openmetadata-server:8585/api")
SERVICE = os.getenv("OM_DATABASE_SERVICE", "trino_sicabe")
CATALOG = os.getenv("OM_DATABASE_NAME", "iceberg")
TRINO_HOSTPORT = os.getenv("OM_TRINO_HOSTPORT", "trino-coordinator:8443")
TRINO_CERT = os.getenv("TRINO_CA_BUNDLE", "/etc/ssl/trino/trino.crt")
# Workflow TestSuite: membawa koneksi Trino eksplisit (username + password +
# verify=sertifikat) supaya eksekusi terautentikasi dan TLS tetap DIVERIFIKASI.
WORKFLOW_TEMPLATE = """
source:
type: TestSuite
serviceName: {service}
sourceConfig:
config:
type: TestSuite
entityFullyQualifiedName: {table}
# WAJIB 'serviceConnections' (plural, list) — DQ source membaca field ini via
# _load_yaml_service_connections(). Memakai 'serviceConnection' (tunggal)
# diabaikan, sehingga eksekusi jatuh ke koneksi TERSIMPAN di OM yang
# kehilangan secret → HTTP 401 'Invalid credentials' (lihat TODO upstream
# open-metadata#21259). Dengan bentuk ini koneksi dari YAML yang dipakai.
serviceConnections:
- serviceName: {service}
serviceConnection:
config:
type: Trino
scheme: trino
hostPort: {hostport}
username: svc_om
authType:
password: "{password}"
catalog: {catalog}
verify: {cert}
sink:
type: metadata-rest
config: {{}}
workflowConfig:
loggerLevel: INFO
openMetadataServerConfig:
hostPort: {om_api}
authProvider: openmetadata
securityConfig:
jwtToken: "{token}"
"""
# Test NATIVE (dijalankan OM) — definisi bawaan OM, bukan ekspektasi GE.
TABLES = {
f"{SERVICE}.{CATALOG}.bronze.orders": [
TableRowCountToBeBetween(min_count=1, max_count=1000000),
ColumnValuesToBeNotNull(column="id"),
ColumnValuesToBeNotNull(column="komoditas"),
ColumnValuesToBeBetween(column="harga_per_kg", min_value=0, max_value=1000000),
],
f"{SERVICE}.{CATALOG}.silver.orders_silver": [
TableRowCountToBeBetween(min_count=1, max_count=1000000),
ColumnValuesToBeNotNull(column="id"),
ColumnValuesToBeBetween(column="total_value", min_value=0, max_value=1000000000000),
],
f"{SERVICE}.{CATALOG}.gold.orders_gold": [
TableRowCountToBeBetween(min_count=1, max_count=1000000),
ColumnValuesToBeNotNull(column="komoditas"),
ColumnValuesToBeBetween(column="avg_harga_per_kg", min_value=0, max_value=1000000),
ColumnValuesToBeBetween(column="total_nilai", min_value=0, max_value=1000000000000),
],
}
def _token():
"""Token OM: dari OM_JWT_TOKEN, atau login pakai password admin.
Dua kandidat: OPENMETADATA_ADMIN_PASSWORD lalu 'admin' — default yang dipakai
OM 2.0.x sampai bootstrap step 7 menyetel hash-nya dari .env. MAKSIMAL DUA
percobaan dan tidak diulang dalam loop: akun admin OM bisa terkunci.
Catatan: 401 pada kandidat pertama adalah hal NORMAL (mis. step 7 belum jalan),
jadi urlopen-nya dibungkus try/except — tanpa itu HTTPError 401 menggagalkan
seluruh skrip.
"""
if os.getenv("OM_JWT_TOKEN"):
return os.environ["OM_JWT_TOKEN"]
email = os.getenv("OPENMETADATA_ADMIN_EMAIL", "admin@open-metadata.org")
for password in (os.getenv("OPENMETADATA_ADMIN_PASSWORD"), "admin"):
if not password:
continue
body = json.dumps({
"email": email,
"password": base64.b64encode(password.encode()).decode(),
}).encode()
req = urllib.request.Request(f"{OM_API_URL}/v1/users/login", data=body,
headers={"Content-Type": "application/json"})
try:
tok = json.load(urllib.request.urlopen(req, timeout=20)).get("accessToken")
except Exception: # noqa: BLE001 — kandidat pertama gagal = hal biasa
tok = None
if tok:
return tok
return None
def main():
token = _token()
trino_password = os.getenv("TRINO_SVC_OM_PASSWORD")
if not token:
print("ERROR: token OpenMetadata tidak tersedia (OM_JWT_TOKEN / "
"OPENMETADATA_ADMIN_PASSWORD)")
return 1
if not trino_password:
print("ERROR: TRINO_SVC_OM_PASSWORD tidak diset — koneksi Trino tidak bisa dibuat")
return 1
client = OpenMetadata(
OpenMetadataConnection(
hostPort=OM_API_URL,
authProvider="openmetadata",
securityConfig=OpenMetadataJWTClientConfig(jwtToken=token),
)
)
failed = 0
total = 0
for table_fqn, tests in TABLES.items():
print(f"[DQ-native] tabel {table_fqn} — {len(tests)} test")
workflow = WORKFLOW_TEMPLATE.format(
service=SERVICE, hostport=TRINO_HOSTPORT, password=trino_password,
catalog=CATALOG, cert=TRINO_CERT, table=table_fqn, om_api=OM_API_URL,
token=token,
)
# use_connection_from_yaml=True → pakai koneksi dari workflow (bukan yang
# tersimpan di OM), sehingga autentikasi Trino bekerja.
runner = TestRunner.from_yaml(
yaml_string=workflow, use_connection_from_yaml=True, client=client
)
for test in tests:
runner.add_test(test)
for result in runner.run():
total += 1
status = str(getattr(result.testCaseResult, "testCaseStatus", None))
name = getattr(result, "name", None) or getattr(result, "fqn", "test")
print(f" - {name}: {status.split('.')[-1]}")
if status.split(".")[-1] != "Success":
failed += 1
print(f"[DQ-native] ringkasan: {total - failed}/{total} test lulus")
return 1 if failed else 0
if __name__ == "__main__":
sys.exit(main())
deploy/scripts/airflow/seed-om-native-dq.py
+210
View File
@@ -0,0 +1,210 @@
#!/usr/bin/env python
# =============================================================================
# eBdesk SiCABE — seed contoh Data Quality NATIVE OpenMetadata (Jalur B)
# =============================================================================
# Mengapa skrip ini ada:
# Test case native TIDAK bisa diandalkan lewat TestRunner SDK yang berjalan di
# klien: jalur itu memakai koneksi TERSIMPAN yang kehilangan secret sehingga
# selalu berakhir `Aborted` dengan `401 Access Denied` (bug upstream, masih ada
# di OM 2.0.2 — kredensial sudah dipastikan valid).
# Jalur yang bekerja: definisi test case dibuat lewat REST API, lalu eksekusi
# DIPICU OpenMetadata lewat pipeline service client (plugin → Airflow). Koneksi
# Trino dibawa EKSPLISIT di `sourceConfig.serviceConnections`, sehingga tidak
# bergantung pada koneksi tersimpan.
#
# Idempoten: test case diperiksa berdasarkan nama; pipeline diperiksa berdasarkan
# nama. Menjalankan ulang tidak menggandakan apa pun.
#
# Dipanggil oleh: scripts/seed-native-dq.sh (dari host) — atau manual:
# docker exec -e OM_SERVICE=trino_sicabe ... sicabe-airflow-webserver \
# python /opt/airflow/scripts/seed-om-native-dq.py [--trigger]
#
# Exit code: 0 = semua langkah yang diminta berhasil.
# =============================================================================
import base64
import json
import os
import sys
import urllib.error
import urllib.request
OM_API_URL = os.getenv("OM_API_URL", "http://openmetadata-server:8585/api")
ADMIN_EMAIL = os.getenv("OPENMETADATA_ADMIN_EMAIL", "admin@open-metadata.org")
SERVICE = os.getenv("OM_DATABASE_SERVICE", "trino_sicabe")
CATALOG = os.getenv("OM_DATABASE_NAME", "iceberg")
TABLE_FQN = os.getenv("OM_DQ_SEED_TABLE", f"{SERVICE}.{CATALOG}.gold.orders_gold")
PIPELINE_NAME = os.getenv("OM_DQ_SEED_PIPELINE", "sicabe_dq_gold_native")
SCHEDULE = os.getenv("OM_DQ_SEED_SCHEDULE", "0 3 * * *")
TRINO_HOSTPORT = os.getenv("OM_TRINO_HOSTPORT", "trino-coordinator:8443")
TRINO_USER = os.getenv("DBT_TRINO_USER", "svc_om")
TRINO_CERT = os.getenv("TRINO_CA_BUNDLE", "/etc/ssl/trino/trino.crt")
# Contoh test native. `parameterValues` HARUS memakai nama parameter definisi
# (tableRowCountToBeBetween: minValue/maxValue) dan columnValuesToBeNotNull
# tidak punya parameter sama sekali — kolomnya ditentukan lewat entityLink.
TEST_CASES = [
{
"name": "native_row_count_between",
"testDefinition": "tableRowCountToBeBetween",
"entityLink": f"<#E::table::{TABLE_FQN}>",
"parameterValues": [
{"name": "minValue", "value": "1"},
{"name": "maxValue", "value": "1000000"},
],
},
{
"name": "native_komoditas_not_null",
"testDefinition": "columnValuesToBeNotNull",
"entityLink": f"<#E::table::{TABLE_FQN}::columns::komoditas>",
},
]
def _request(method, path, payload=None, token=None):
"""Panggil REST API OpenMetadata; kembalikan (status, body terurai)."""
headers = {"Content-Type": "application/json"}
if token:
headers["Authorization"] = f"Bearer {token}"
data = json.dumps(payload).encode() if payload is not None else None
req = urllib.request.Request(f"{OM_API_URL}{path}", data=data, headers=headers, method=method)
try:
with urllib.request.urlopen(req, timeout=60) as resp:
raw = resp.read()
return resp.status, (json.loads(raw) if raw else {})
except urllib.error.HTTPError as exc:
raw = exc.read()[:300]
try:
return exc.code, json.loads(raw)
except Exception: # noqa: BLE001
return exc.code, raw.decode(errors="replace")
def login():
"""Token admin: password dikirim base64 (wajib sejak OM 2.0.x).
Dua kandidat: OPENMETADATA_ADMIN_PASSWORD, lalu 'admin' — default yang dipakai
OM 2.0.x sampai bootstrap step 7 menyetel hash-nya dari .env. MAKSIMAL DUA
percobaan dan tidak pernah diulang dalam loop: akun admin OM bisa terkunci.
"""
last = "tidak ada kandidat password"
for password in (os.getenv("OPENMETADATA_ADMIN_PASSWORD"), "admin"):
if not password:
continue
body = {"email": ADMIN_EMAIL,
"password": base64.b64encode(password.encode()).decode()}
status, res = _request("POST", "/v1/users/login", body)
if status == 200 and res.get("accessToken"):
return res["accessToken"]
last = f"HTTP {status}: {str(res)[:120]}"
raise RuntimeError(f"login gagal (password .env dan 'admin'): {last}")
def ensure_test_cases(token):
"""Buat test case native yang belum ada. Kembalikan jumlah yang dibuat."""
status, page = _request("GET", "/v1/dataQuality/testCases?limit=200", token=token)
if status != 200:
raise RuntimeError(f"daftar test case tidak terbaca (HTTP {status})")
existing = {tc.get("name") for tc in page.get("data", [])}
created = 0
for spec in TEST_CASES:
if spec["name"] in existing:
print(f" [SKIP] test case sudah ada: {spec['name']}")
continue
status, res = _request("POST", "/v1/dataQuality/testCases", spec, token=token)
if status in (200, 201):
created += 1
print(f" [OK] test case dibuat: {spec['name']}")
else:
raise RuntimeError(f"gagal membuat {spec['name']} (HTTP {status}): {str(res)[:200]}")
return created
def _pipeline_payload(suite_id):
"""Payload pipeline TestSuite dengan koneksi Trino EKSPLISIT (anti-401)."""
return {
"name": PIPELINE_NAME,
"pipelineType": "TestSuite",
"service": {"id": suite_id, "type": "testSuite"},
"sourceConfig": {"config": {
"type": "TestSuite",
"entityFullyQualifiedName": TABLE_FQN,
"serviceConnections": [{
"serviceName": SERVICE,
"serviceConnection": {"config": {
"type": "Trino", "scheme": "trino", "hostPort": TRINO_HOSTPORT,
"username": TRINO_USER,
"authType": {"password": os.getenv("TRINO_SVC_OM_PASSWORD", "")},
"catalog": CATALOG, "verify": TRINO_CERT,
}},
}],
}},
"airflowConfig": {"scheduleInterval": SCHEDULE},
}
def ensure_pipeline(token):
"""Buat pipeline TestSuite bila belum ada. Kembalikan (id, dibuat_baru)."""
status, suites = _request("GET", "/v1/dataQuality/testSuites?limit=50", token=token)
if status != 200:
raise RuntimeError(f"daftar suite tidak terbaca (HTTP {status})")
suite = next((s for s in suites.get("data", []) if TABLE_FQN.rsplit(".", 1)[0] in s.get("name", "")), None)
if suite is None:
raise RuntimeError(f"test suite untuk '{TABLE_FQN}' belum ada — jalankan ingestion dulu")
status, page = _request("GET", "/v1/services/ingestionPipelines?limit=50", token=token)
for pipe in (page.get("data") or []) if status == 200 else []:
if pipe.get("name") == PIPELINE_NAME:
print(f" [SKIP] pipeline sudah ada: {PIPELINE_NAME} (id {pipe['id']})")
return pipe["id"], False
status, res = _request("POST", "/v1/services/ingestionPipelines",
_pipeline_payload(suite["id"]), token=token)
if status not in (200, 201):
raise RuntimeError(f"gagal membuat pipeline (HTTP {status}): {str(res)[:200]}")
print(f" [OK] pipeline dibuat: {PIPELINE_NAME} (id {res['id']})")
return res["id"], True
def deploy_and_trigger(token, pipeline_id):
"""Deploy lalu trigger pipeline; trigger diulang sekali bila DAG belum siap."""
import time
status, res = _request("POST", f"/v1/services/ingestionPipelines/deploy/{pipeline_id}", token=token)
print(f" [{'OK' if status == 200 else 'GAGAL'}] deploy -> HTTP {status}")
if status != 200:
raise RuntimeError(f"deploy gagal: {str(res)[:200]}")
for attempt in (1, 2):
status, res = _request("POST", f"/v1/services/ingestionPipelines/trigger/{pipeline_id}", token=token)
print(f" [{'OK' if status == 200 else 'GAGAL'}] trigger -> HTTP {status} (percobaan {attempt})")
if status == 200:
return
# DAG baru dibuat: dag-processor bisa belum memparsingnya.
time.sleep(20)
raise RuntimeError("trigger gagal setelah 2 percobaan")
def main():
trigger = "--trigger" in sys.argv
print(f"[seed-native-dq] tabel={TABLE_FQN} pipeline={PIPELINE_NAME}")
token = login()
print(f" [OK] login {ADMIN_EMAIL}")
created = ensure_test_cases(token)
pipeline_id, new_pipeline = ensure_pipeline(token)
if trigger:
deploy_and_trigger(token, pipeline_id)
print(f"[seed-native-dq] selesai: {created} test case baru, pipeline "
f"{'baru' if new_pipeline else 'sudah ada'}"
f"{', deploy+trigger dijalankan' if trigger else ''}")
return 0
if __name__ == "__main__":
try:
sys.exit(main())
except Exception as exc: # noqa: BLE001
print(f"ERROR: {exc}")
sys.exit(1)
deploy/scripts/dremio/register-nessie-source.sh
Executable
+108
View File
@@ -0,0 +1,108 @@
#!/usr/bin/env bash
# =============================================================================
# Dremio — Register Nessie catalog source via REST API
# =============================================================================
# Membuat source "Nessie" di Dremio OSS lewat API (bukan UI).
#
# ⚠️ Catatan: dokumentasi Dremio menyatakan Catalog API "unable to retrieve or
# manage Nessie sources" (DX-64994). Namun TERBUKTI (Dremio OSS 25.2.0, diuji
# 2026-09-10) bahwa POST /api/v3/catalog dengan type=NESSIE BERHASIL, dan
# GET /api/v3/catalog/by-path/<name> juga bekerja. Kunci config:
# - JANGAN pakai "storageType" (ditolak: "Invalid value found at: storageType")
# - awsRootPath format "/bucket/path" (TANPA prefix s3://)
#
# Pemakaian:
# export DREMIO_USER=admin DREMIO_PASSWORD='<password>'
# ./scripts/dremio/register-nessie-source.sh
#
# Env (opsional, ada default):
# DREMIO_URL default http://localhost:12947
# DREMIO_SOURCE_NAME default nessie
# NESSIE_ENDPOINT default ${NESSIE_URL}/api/v2 (http://nessie:19120/api/v2)
# MINIO_ENDPOINT default host dari S3_ENDPOINT (tanpa skema)
# AWS_ROOT_PATH default /<bucket>/warehouse
# MINIO_ROOT_USER / MINIO_ROOT_PASSWORD (fallback: S3_ACCESS_KEY / S3_SECRET_KEY)
# =============================================================================
set -euo pipefail
DREMIO_URL="${DREMIO_URL:-http://localhost:12947}"
DREMIO_USER="${DREMIO_USER:-admin}"
DREMIO_PASSWORD="${DREMIO_PASSWORD:?set DREMIO_PASSWORD}"
DREMIO_SOURCE_NAME="${DREMIO_SOURCE_NAME:-nessie}"
NESSIE_ENDPOINT="${NESSIE_ENDPOINT:-${NESSIE_URL:-http://nessie:19120}/api/v2}"
# MINIO_ENDPOINT dipakai tanpa skema (Dremio memisahkan host:port) → buang skema.
_minio_host="${MINIO_ENDPOINT:-${S3_ENDPOINT:-http://minio:9000}}"
MINIO_ENDPOINT="${_minio_host#*://}"
AWS_ROOT_PATH="${AWS_ROOT_PATH:-/${MINIO_BUCKET_ICEBERG_DATA:-iceberg-data}/warehouse}"
MINIO_ROOT_USER="${MINIO_ROOT_USER:-${S3_ACCESS_KEY:-minioadmin}}"
MINIO_ROOT_PASSWORD="${MINIO_ROOT_PASSWORD:-${S3_SECRET_KEY:?set MINIO_ROOT_PASSWORD atau S3_SECRET_KEY}}"
echo "==> Login Dremio (${DREMIO_URL}) sebagai '${DREMIO_USER}'..."
LOGIN_RESP=$(curl -s -X POST "${DREMIO_URL}/apiv2/login" \
-H "Content-Type: application/json" \
-d "{\"userName\":\"${DREMIO_USER}\",\"password\":\"${DREMIO_PASSWORD}\"}")
TOKEN=$(echo "${LOGIN_RESP}" | python3 -c "import sys,json;print(json.load(sys.stdin).get('token',''))" 2>/dev/null || echo "")
if [[ -z "${TOKEN}" ]]; then
echo "ERR: login gagal. Respons: ${LOGIN_RESP}"
exit 1
fi
AUTH="Authorization: _dremio${TOKEN}"
echo " OK: token diperoleh."
# Cek apakah source sudah ada (best-effort; GET Nessie mungkin tidak didukung)
EXIST_CODE=$(curl -s -o /dev/null -w "%{http_code}" -H "${AUTH}" \
"${DREMIO_URL}/api/v3/catalog/by-path/${DREMIO_SOURCE_NAME}" || echo "000")
if [[ "${EXIST_CODE}" == "200" ]]; then
echo "OK: source '${DREMIO_SOURCE_NAME}' sudah ada."
exit 0
fi
echo " Source belum ada (GET by-path -> HTTP ${EXIST_CODE}). Membuat..."
BODY=$(cat <<JSON
{
"entityType": "source",
"name": "${DREMIO_SOURCE_NAME}",
"type": "NESSIE",
"config": {
"nessieEndpoint": "${NESSIE_ENDPOINT}",
"nessieAuthType": "NONE",
"awsRootPath": "${AWS_ROOT_PATH}",
"credentialType": "ACCESS_KEY",
"awsAccessKey": "${MINIO_ROOT_USER}",
"awsAccessSecret": "${MINIO_ROOT_PASSWORD}",
"secure": false,
"asyncEnabled": true,
"isCachingEnabled": false,
"propertyList": [
{ "name": "fs.s3a.path.style.access", "value": "true" },
{ "name": "fs.s3a.endpoint", "value": "${MINIO_ENDPOINT}" },
{ "name": "dremio.s3.compat", "value": "true" }
]
}
}
JSON
)
echo "==> POST /api/v3/catalog (type=NESSIE)..."
CREATE_CODE=$(curl -s -o /tmp/dremio_create_resp.json -w "%{http_code}" \
-X POST "${DREMIO_URL}/api/v3/catalog" \
-H "${AUTH}" -H "Content-Type: application/json" -d "${BODY}")
echo " HTTP ${CREATE_CODE}"
cat /tmp/dremio_create_resp.json 2>/dev/null; echo
case "${CREATE_CODE}" in
200|201)
echo "✅ Source '${DREMIO_SOURCE_NAME}' dibuat via API."
;;
409)
echo "✅ Source '${DREMIO_SOURCE_NAME}' sudah ada (409 Conflict)."
;;
*)
echo "⚠️ API menolak pembuatan source Nessie (HTTP ${CREATE_CODE})."
echo " Sesuai docs (DX-64994), Catalog API tidak mengelola Nessie source."
echo " Buat manual via UI: http://localhost:12947 → Add Source → Nessie"
echo " endpoint: ${NESSIE_ENDPOINT}"
echo " storage : AWS S3 / MinIO (path-style), endpoint ${MINIO_ENDPOINT}"
exit 2
;;
esac
deploy/scripts/generate-trino-keystore.sh
Executable
+88
View File
@@ -0,0 +1,88 @@
#!/usr/bin/env bash
# =============================================================================
# eBdesk SiCABE — Generate keystore TLS Trino (+ ekspor sertifikat untuk klien)
# =============================================================================
# Mengapa perlu SAN (Subject Alternative Name)?
# Keystore lama hanya berisi CN=localhost TANPA SAN. Klien yang memverifikasi
# TLS (bukan mematikannya) tidak dapat memakai sertifikat itu untuk hostname
# `trino` (nama service di jaringan compose) → verifikasi gagal. Sertifikat di
# sini menerbitkan SAN: trino, localhost, sicabe-trino, 127.0.0.1 sehingga
# verifikasi BERJALAN (bukan dimatikan).
#
# Diperlukan oleh:
# - Trino: /etc/trino/keystore.p12 (password di config.properties)
# - Klien yang memverifikasi (mis. ingestion OpenMetadata): sertifikat PEM hasil
# ekspor di config/trino/certs/trino.crt
#
# ⚠️ Ini keystore DEVELOPMENT (di-commit). Untuk produksi: generate ulang, jangan
# commit private key-nya, dan ganti `http-server.https.keystore.key`.
#
# Usage:
# ./scripts/generate-trino-keystore.sh
# TRINO_KEYSTORE_PASSWORD=rahasia ./scripts/generate-trino-keystore.sh
# =============================================================================
set -uo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
# shellcheck source=../lib/common.sh
source "${SCRIPT_DIR}/lib/common.sh"
# Opsi:
# --out <dir> tulis <dir>/keystore.p12 + <dir>/certs/trino.crt (default config/trino)
# --san-extra <csv> tambahkan entri SAN, mis. "IP:10.0.0.11,IP:10.0.0.12" — dipakai
# multi-host supaya klien yang terhubung via ALAMAT HOST tetap
# memverifikasi TLS (cert tak hanya valid utk nama container).
OUT_DIR="${SCRIPT_DIR}/config/trino"
SAN_EXTRA=""
while [[ $# -gt 0 ]]; do
case "$1" in
--out) OUT_DIR="${2:-}"; shift 2 ;;
--san-extra) SAN_EXTRA="${2:-}"; shift 2 ;;
-h|--help) sed -n '20,23p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//'; exit 0 ;;
*) echo -e "${ERR} Argumen tidak dikenal: $1"; exit 2 ;;
esac
done
KEYSTORE="${OUT_DIR}/keystore.p12"
CERT_DIR="${OUT_DIR}/certs"
CERT="${CERT_DIR}/trino.crt"
PROPS="${SCRIPT_DIR}/config/trino/config-coordinator.properties"
# Password keystore: env menang, kalau tidak ambil dari config.properties
# (Trino memakai SATU password untuk keystore & key).
KS_PASSWORD="${TRINO_KEYSTORE_PASSWORD:-}"
if [[ -z "${KS_PASSWORD}" ]]; then
KS_PASSWORD="$(sed -n 's/^http-server.https.keystore.key=//p' "${PROPS}" 2>/dev/null | head -1)"
fi
if [[ -z "${KS_PASSWORD}" ]]; then
echo -e "${ERR} Password keystore tidak diketahui (isi TRINO_KEYSTORE_PASSWORD)."
exit 1
fi
SAN="DNS:trino,DNS:localhost,DNS:sicabe-trino,DNS:trino-coordinator,DNS:trino-worker,IP:127.0.0.1"
[[ -n "${SAN_EXTRA}" ]] && SAN="${SAN},${SAN_EXTRA}"
mkdir -p "${CERT_DIR}"
TMP_KEY="$(mktemp)"
trap 'rm -f "${TMP_KEY}"' EXIT
echo -e "${INFO} Menerbitkan sertifikat self-signed (SAN: ${SAN})..."
if ! openssl req -x509 -newkey rsa:2048 -nodes -days 825 \
-keyout "${TMP_KEY}" -out "${CERT}" \
-subj "/CN=trino" -addext "subjectAltName=${SAN}" >/dev/null 2>&1; then
echo -e "${ERR} Gagal membuat sertifikat (openssl tidak tersedia?)."
exit 1
fi
echo -e "${INFO} Membungkus ke PKCS12 (${KEYSTORE##*/})..."
if ! openssl pkcs12 -export -in "${CERT}" -inkey "${TMP_KEY}" \
-name trino -out "${KEYSTORE}" -passout "pass:${KS_PASSWORD}" >/dev/null 2>&1; then
echo -e "${ERR} Gagal membuat keystore PKCS12."
exit 1
fi
chmod 644 "${KEYSTORE}" "${CERT}"
echo -e "${OK} Keystore dibuat: ${KEYSTORE}"
echo -e "${OK} Sertifikat diekspor: ${CERT}"
openssl x509 -in "${CERT}" -noout -subject -dates 2>/dev/null | sed 's/^/ /'
openssl x509 -in "${CERT}" -noout -ext subjectAltName 2>/dev/null | sed 's/^/ /'
echo -e "${INFO} Restart Trino coordinator agar sertifikat baru dipakai: ./install.sh --restart trino-coordinator"
deploy/scripts/ingest-openmetadata.sh
Executable
+190
View File
@@ -0,0 +1,190 @@
#!/usr/bin/env bash
# =============================================================================
# eBdesk SiCABE — Jalankan agent ingestion OpenMetadata
# =============================================================================
# Mengapa script ini ada:
# Membuat SERVICE di OpenMetadata hanya menyimpan konfigurasi koneksi. Metadata
# (database → schema → tabel → kolom) baru terisi setelah AGENT dijalankan, dan
# halaman Explore membaca hasilnya dari search index. Tanpa agent: Explore kosong.
#
# LINEAGE (bronze→silver→gold) TIDAK dihasilkan oleh ingestion metadata: lineage
# berasal dari definisi transformasi, jadi diambil dari ARTEFAK dbt lewat workflow
# terpisah (`ingest-dbt.yaml`). Keduanya dijalankan oleh `--all`.
#
# Script memakai Ingestion Framework yang sudah ada di container Airflow
# (`metadata` CLI) — tidak perlu memasang apa pun.
#
# ⚠️ Cakupan metadata mengikuti ACL identitas agent (workflow memakai `svc_om`,
# role `data-catalog` = baca-saja seluruh tabel).
#
# Usage:
# ./scripts/ingest-openmetadata.sh # metadata (Trino) saja
# ./scripts/ingest-openmetadata.sh --all # metadata + lineage dbt
# ./scripts/ingest-openmetadata.sh --dbt-only # lineage dbt saja
# ./scripts/ingest-openmetadata.sh --workflow config/openmetadata/ingest-dbt.yaml
# =============================================================================
set -uo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
# shellcheck source=../lib/common.sh
source "${SCRIPT_DIR}/lib/common.sh"
# shellcheck source=../config/environment.defaults.sh
source "${SCRIPT_DIR}/config/environment.defaults.sh"
sicabe_load_env "${SCRIPT_DIR}" "$@"
OM_URL="${OPENMETADATA_URL:-http://localhost:${OPENMETADATA_PORT:-12085}}"
OM_ADMIN_EMAIL="${OPENMETADATA_ADMIN_EMAIL:-admin@open-metadata.org}"
OM_ADMIN_PASS="${OPENMETADATA_ADMIN_PASSWORD:-openmetadata123}"
SERVICE_NAME="${OPENMETADATA_TRINO_SERVICE:-trino_sicabe}"
AIRFLOW_CONTAINER="${AIRFLOW_INGEST_CONTAINER:-sicabe-airflow-webserver}"
REMOTE_DIR="/tmp/om-ingest"
DBT_PROJECT_DIR="${DBT_PROJECT_DIR:-/opt/airflow/dbt/sicabe}"
# Di multi-host, pakai workflow TER-RENDER dari bundle host (hostname → alamat host);
# sumber `config/**` masih memakai nama container (single-host). `SICABE_BUNDLE_DIR`
# diset sicabe_load_env(): single-host = deploy/ (config sumber), multi-host =
# generated/host-<x>/ (config ter-render).
CONFIG_DIR="${SICABE_BUNDLE_DIR:-${SCRIPT_DIR}}/config"
WF_METADATA="${CONFIG_DIR}/openmetadata/ingest-trino.yaml"
WF_DBT="${CONFIG_DIR}/openmetadata/ingest-dbt.yaml"
MODE="metadata"
CUSTOM_WORKFLOW=""
while [[ $# -gt 0 ]]; do
case "$1" in
--env-file) shift 2 ;; # sudah dipakai sicabe_load_env() di atas
--all) MODE="all"; shift ;;
--dbt-only) MODE="dbt"; shift ;;
--workflow) CUSTOM_WORKFLOW="$2"; shift 2 ;;
*) echo "Argumen tidak dikenal: $1"; exit 1 ;;
esac
done
# ── Helper ──────────────────────────────────────────────────────────────────
om_ws() { # om_ws <method> <path> [body]
local method="$1" path="$2" body="${3:-}"
if [[ -n "${body}" ]]; then
curl -s --max-time 30 -X "${method}" "${OM_URL}${path}" \
-H "Authorization: Bearer ${TOKEN}" -H "Content-Type: application/json" -d "${body}" 2>/dev/null
else
curl -s --max-time 30 -X "${method}" "${OM_URL}${path}" -H "Authorization: Bearer ${TOKEN}" 2>/dev/null
fi
}
run_workflow() { # run_workflow <path-workflow> <label>
local wf="$1" label="$2" remote
remote="${REMOTE_DIR}/$(basename "${wf}")"
if [[ ! -f "${wf}" ]]; then
echo -e "${ERR} Workflow tidak ditemukan: ${wf}"
return 1
fi
echo -e "${INFO} [${label}] menyalin ${wf##*/} ke container..."
docker exec "${AIRFLOW_CONTAINER}" mkdir -p "${REMOTE_DIR}"
docker cp "${wf}" "${AIRFLOW_CONTAINER}:${remote}" >/dev/null
echo -e "${INFO} [${label}] menjalankan ingestion..."
local out
out="$(docker exec -e TRINO_SVC_OM_PASSWORD="${TRINO_SVC_OM_PASSWORD:-}" -e OM_JWT_TOKEN="${TOKEN}" \
"${AIRFLOW_CONTAINER}" metadata ingest -c "${remote}" 2>&1)"
echo "${out}" | sed 's/\x1b\[[0-9;]*m//g' |
grep -iE "Processed records|Errors:|Success %|Workflow Success|Error running node|ERROR" | tail -8
if ! echo "${out}" | grep -q "Workflow Success %: 100.0"; then
echo -e "${ERR} [${label}] ingestion tidak 100% sukses — periksa keluaran di atas."
return 1
fi
return 0
}
# ── Prasyarat ───────────────────────────────────────────────────────────────
for f in "${WF_METADATA}" "${WF_DBT}"; do
[[ -f "${f}" ]] || { echo -e "${ERR} Workflow tidak ditemukan: ${f}"; exit 1; }
done
if ! docker ps --format "{{.Names}}" 2>/dev/null | grep -q "^${AIRFLOW_CONTAINER}$"; then
echo -e "${ERR} Container '${AIRFLOW_CONTAINER}' tidak berjalan (butuh Ingestion Framework)."
exit 1
fi
echo -e "${INFO} Login ke OpenMetadata (${OM_ADMIN_EMAIL})..."
# Dua kandidat, MAKSIMAL DUA percobaan — jangan diulang dalam loop (akun admin bisa
# terkunci): password .env, lalu default 'admin' yang dipakai OM 2.0.x sampai
# bootstrap step 7 menyetel hash-nya. `curl -s` TANPA -f, agar kegagalan HTTP tidak
# mematikan skrip di bawah `set -e` — penanganannya ada di blok if berikutnya.
TOKEN=""
for TRY_PASS in "${OM_ADMIN_PASS}" "admin"; do
[[ -n "${TRY_PASS}" ]] || continue
PW64="$(printf '%s' "${TRY_PASS}" | base64)"
TOKEN="$(curl -s -X POST "${OM_URL}/api/v1/users/login" \
-H "Content-Type: application/json" \
-d "{\"email\":\"${OM_ADMIN_EMAIL}\",\"password\":\"${PW64}\"}" 2>/dev/null |
python3 -c "import sys,json; print(json.load(sys.stdin).get('accessToken',''))" 2>/dev/null || true)"
[[ -n "${TOKEN}" ]] && break
done
if [[ -z "${TOKEN}" ]]; then
echo -e "${ERR} Gagal login ke OpenMetadata di ${OM_URL} (dicoba: password .env + 'admin')"
echo " pastikan admin sudah diset: ./bootstrap.sh --step 7"
exit 1
fi
echo -e "${OK} Token diperoleh."
RC=0
if [[ -n "${CUSTOM_WORKFLOW}" ]]; then
run_workflow "${CUSTOM_WORKFLOW}" "custom" || RC=1
else
[[ "${MODE}" == "metadata" || "${MODE}" == "all" ]] && { run_workflow "${WF_METADATA}" "metadata" || RC=1; }
if [[ "${MODE}" == "dbt" || "${MODE}" == "all" ]]; then
# Artefak dbt wajib ada di container (hilang bila container di-recreate,
# karena berada di layer container, bukan bind-mount).
# Artefak dbt ditulis ke DBT_TARGET_PATH (default /tmp/dbt/target — lihat
# docker-compose.yml). DULU diperiksa di ${DBT_PROJECT_DIR}/target, yaitu langsung
# di dalam bind-mount repo yang TIDAK bisa ditulis container (uid 50000 vs uid host),
# sehingga pemeriksaan ini selalu melaporkan "belum ada" walaupun artefaknya ada.
DBT_TARGET_PATH="${DBT_TARGET_PATH:-/tmp/dbt/target}"
if ! docker exec "${AIRFLOW_CONTAINER}" test -f "${DBT_TARGET_PATH}/manifest.json" 2>/dev/null; then
echo -e "${WARN} Artefak dbt belum ada di container (${DBT_TARGET_PATH})."
echo -e "${INFO} Buat dulu (butuh kredensial dbt). Perhatikan DBT_TARGET_PATH/DBT_LOG_PATH:"
echo " docker exec -e DBT_TRINO_USER=svc_dbt -e DBT_TRINO_PASSWORD=\$TRINO_SVC_DBT_PASSWORD \\"
echo " -e DBT_PROFILES_DIR=/opt/airflow/dbt -e DBT_TARGET_PATH=${DBT_TARGET_PATH} \\"
echo " -e DBT_LOG_PATH=/tmp/dbt/logs -w ${DBT_PROJECT_DIR} ${AIRFLOW_CONTAINER} \\"
echo " bash -lc 'dbt docs generate --profile trino --target prod'"
RC=1
else
run_workflow "${WF_DBT}" "lineage dbt" || RC=1
fi
fi
fi
# ── Bukti hasil ─────────────────────────────────────────────────────────────
echo -e "${INFO} Memverifikasi hasil di OpenMetadata..."
SUMMARY="$(om_ws GET "/api/v1/tables?service=${SERVICE_NAME}&fields=columns&limit=50" |
python3 -c "
import sys,json
try:
d=json.load(sys.stdin).get('data',[])
print('{0}|{1}'.format(len(d), sum(len(t.get('columns') or []) for t in d)))
except Exception: print('0|0')" 2>/dev/null)"
TABLES="${SUMMARY%%|*}"; COLUMNS="${SUMMARY##*|}"
if [[ "${TABLES}" =~ ^[0-9]+$ ]] && [[ ${TABLES} -gt 0 ]]; then
echo -e "${OK} ${TABLES} tabel ter-indeks (${COLUMNS} kolom) — muncul di Explore."
else
echo -e "${WARN} Belum ada tabel ter-indeks untuk '${SERVICE_NAME}'."
RC=1
fi
# Lineage (ditampilkan bila workflow dbt dijalankan)
if [[ "${MODE}" == "dbt" || "${MODE}" == "all" ]]; then
EDGES="$(om_ws GET "/api/v1/lineage/getLineage?fqn=${SERVICE_NAME}.iceberg.bronze.orders&upstreamDepth=0&downstreamDepth=3" |
python3 -c "
import sys,json
try:
d=json.load(sys.stdin)
print(len(d.get('downstreamEdges') or []))
except Exception: print('0')" 2>/dev/null)"
if [[ "${EDGES}" =~ ^[0-9]+$ ]] && [[ ${EDGES} -gt 0 ]]; then
echo -e "${OK} Lineage terbentuk: ${EDGES} relasi downstream dari bronze.orders."
else
echo -e "${WARN} Lineage belum terbentuk (0 relasi) — periksa keluaran ingestion dbt."
RC=1
fi
fi
exit ${RC}
deploy/scripts/prefetch-images.sh
Executable
+336
View File
@@ -0,0 +1,336 @@
#!/usr/bin/env bash
# =============================================================================
# prefetch-images.sh — tarik SEMUA image Docker lebih dulu (agar install offline-ish)
# -----------------------------------------------------------------------------
# Daftar image TIDAK di-hardcode: diambil dari `docker compose config --images`,
# sehingga selalu sinkron dengan docker-compose.yml (termasuk image yang berada di
# balik profil, mis. dremio).
#
# CATATAN PENTING — satu image dibangun LOKAL:
# sicabe/airflow:3.3.2-om203-ol dibangun dari docker/airflow/Dockerfile.332
# (base apache/airflow:slim-3.3.2-python3.11). Ia TIDAK ada di registry publik,
# sehingga `pull` dari Docker Hub tidak mungkin.
# ✅ SEJAK 2026-10-02 image ini TERSEDIA di REGISTRY INTERNAL:
# registry.sicabe.com/airflow:3.3.2-om203-ol
# Script ini OTOMATIS ikut menariknya bila `AIRFLOW_IMAGE` menunjuk REGISTRY
# (host memuat titik, mis. `registry.sicabe.com/airflow:…`) — cukup set
# AIRFLOW_IMAGE=registry.sicabe.com/airflow:3.3.2-om203-ol
# di .env. Bila `AIRFLOW_IMAGE` dibiarkan nama LOKAL polos (`sicabe/airflow:…`),
# image DIKECUALIKAN dari pull dan wajib dibangun/di-retag manual (alur di bawah).
# Untuk host TANPA akses registry, tetap ada alur build-sekali-salin:
# 1) di SATU host: ./scripts/prefetch-images.sh
# docker build -t sicabe/airflow:3.3.2-om203-ol \
# -f docker/airflow/Dockerfile.332 docker/airflow
# ./scripts/prefetch-images.sh --save-airflow /tmp/airflow.tar
# 2) salin: scp /tmp/airflow.tar host2:/tmp/ ; scp … host3:/tmp/
# 3) di host lain: ./scripts/prefetch-images.sh --load-airflow /tmp/airflow.tar
#
# Pemakaian:
# ./scripts/prefetch-images.sh [--all | --host <nama>] [--list] [--no-base]
# ./scripts/prefetch-images.sh --save-airflow /tmp/airflow.tar
# ./scripts/prefetch-images.sh --load-airflow /tmp/airflow.tar
# =============================================================================
set -uo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
DEPLOY_DIR="$(cd "${SCRIPT_DIR}/.." && pwd)"
RED=$'\e[31m'; GRN=$'\e[32m'; YLW=$'\e[33m'; BLD=$'\e[1m'; NC=$'\e[0m'
# PENTING: semua diagnostik ke STDERR. stdout hanya untuk DATA (daftar image),
# karena collect_images() ditangkap dengan $( ) — pesan info yang ikut tertangkap
# akan dianggap nama image (bug yang ditemukan saat menguji --list).
ok() { echo -e " ${GRN}[OK]${NC} $*" >&2; }
inf() { echo -e " ${YLW}[..]${NC} $*" >&2; }
warn() { echo -e " ${YLW}[WARN]${NC} $*" >&2; }
err() { echo -e " ${RED}[!!]${NC} $*" >&2; }
MODE="all" # all | host
HOST_NAME=""
LIST_ONLY=0
# Dremio (opt-in di compose), image build lokal, dan base build Airflow DEFAULT
# DIKELUARKAN dari daftar prefetch. Ketiganya diaktifkan lewat flag bila memang
# diperlukan (mis. deployment yang menjalankan dremio / akan build Airflow).
WITH_BASE=0
WITH_DREMIO=0
SAVE_AIRFLOW=""
LOAD_AIRFLOW=""
usage() { sed -n '2,28p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//'; exit 0; }
parse_args() {
while [[ $# -gt 0 ]]; do
case "$1" in
--all) MODE="all" ;;
--host) MODE="host"; HOST_NAME="${2:-}"; shift ;;
--list) LIST_ONLY=1 ;;
--no-base) WITH_BASE=0 ;; # dipertahankan: kini defaultnya sudah tanpa base
--with-base) WITH_BASE=1 ;;
--with-dremio) WITH_DREMIO=1 ;;
--save-airflow) SAVE_AIRFLOW="${2:-}"; shift ;;
--load-airflow) LOAD_AIRFLOW="${2:-}"; shift ;;
-h|--help) usage ;;
*) err "Argumen tidak dikenal: $1"; exit 2 ;;
esac
shift
done
}
require_docker() {
if ! command -v docker &>/dev/null; then
err "docker tidak ditemukan — jalankan ./scripts/prepare-host.sh lebih dulu."
exit 1
fi
}
# ─── Daftar image (tidak di-hardcode) ───────────────────────────────────────
compose_images() { # <file> [--env-file <f>] [--profile …]…
local file="$1"; shift
local out err_file
# stderr JANGAN ditelan: kalau kosong, tampilkan penyebabnya (pelajaran dari gate
# Fase 5 yang sempat menyembunyikan error compose sehingga gagal sulit dilacak).
#
# Catatan bug: versi pertama memakai `local err` + `trap … RETURN`. Trap RETURN
# berjalan SETELAH variabel local dibersihkan sehingga ${err} menjadi unbound, dan
# nama `err` juga menutupi FUNGSI err(). Kini tanpa trap + bernama err_file.
err_file="$(mktemp)"
out="$(docker compose -f "${file}" "$@" config --images 2>"${err_file}" | sort -u)"
if [[ -z "${out}" && -s "${err_file}" ]]; then
warn "docker compose gagal mendaftar image:"
sed 's/^/ /' "${err_file}" >&2
fi
rm -f "${err_file}"
printf '%s\n' "${out}"
}
# Isi ENV_ARGS (variabel global) dengan argumen env untuk compose.
# Bila .env BELUM ADA (server masih bersih — .env dibuat install.sh saat instalasi),
# pakai default dari config/environment.defaults.sh: ke-16 variabel yang diwajibkan
# compose (:?err) semuanya ada di sana, dan nilainya TIDAK memengaruhi daftar image
# (image di-pin di docker-compose.yml).
#
# PENTING: fungsi ini HARUS dipanggil di shell saat ini — bukan di dalam $( ) — supaya
# `set -a; source defaults` benar-benar berlaku untuk proses compose kita. Versi pertama
# saya memanggilnya dalam command substitution (subshell), jadi sourcing-nya sia-sia.
ENV_ARGS=()
resolve_env() { # <envfile>
local envfile="$1"
if [[ -f "${envfile}" ]]; then
ENV_ARGS=(--env-file "${envfile}")
return 0
fi
warn "berkas ${envfile} belum ada — memakai default config/environment.defaults.sh"
echo " (.env dibuat oleh install.sh saat instalasi; tidak memengaruhi daftar image)" >&2
set -a
# shellcheck disable=SC1091
. "${DEPLOY_DIR}/config/environment.defaults.sh"
set +a
ENV_ARGS=()
}
collect_images() {
local file envfile
if [[ "${MODE}" == "host" ]]; then
[[ -z "${HOST_NAME}" ]] && { err "--host memerlukan nama host."; exit 2; }
file="${DEPLOY_DIR}/generated/host-${HOST_NAME}/docker-compose.yml"
if [[ ! -f "${file}" ]]; then
err "Bundle host belum ada: ${file}"
echo " Jalankan dulu: ./install.sh --generate (atau --generate --inventory cluster.yml)"
exit 1
fi
envfile="${DEPLOY_DIR}/generated/host-${HOST_NAME}/.env"
[[ -f "${envfile}" ]] || envfile="${DEPLOY_DIR}/.env"
inf "sumber daftar image: bundle host '${HOST_NAME}'"
resolve_env "${envfile}"
compose_images "${file}" ${ENV_ARGS[@]+"${ENV_ARGS[@]}"} --profile all
else
local extra=()
if [[ ${WITH_DREMIO} -eq 1 ]]; then
extra=(--profile dremio)
inf "sumber daftar image: docker-compose.yml (profil all + dremio)"
else
inf "sumber daftar image: docker-compose.yml (profil all; dremio dikeluarkan)"
fi
resolve_env "${DEPLOY_DIR}/.env"
compose_images "${DEPLOY_DIR}/docker-compose.yml" ${ENV_ARGS[@]+"${ENV_ARGS[@]}"} \
--profile all ${extra[@]+"${extra[@]}"}
fi
}
# Base image Airflow diambil dari Dockerfile agar tidak ter-hardcode di sini.
airflow_base_image() {
local df="${DEPLOY_DIR}/docker/airflow/Dockerfile.slim"
[[ -f "${df}" ]] || return 1
grep -m1 '^FROM ' "${df}" | awk '{print $2}'
}
airflow_built_image() {
# shellcheck disable=SC1091
( set -a; [[ -f "${DEPLOY_DIR}/config/environment.defaults.sh" ]] && . "${DEPLOY_DIR}/config/environment.defaults.sh"
[[ -f "${DEPLOY_DIR}/.env" ]] && . "${DEPLOY_DIR}/.env"
echo "${AIRFLOW_IMAGE:-sicabe/airflow:3.3.2-om203-ol}" ) 2>/dev/null | tail -1
}
# Host registry dari nama image — untuk petunjuk diagnosa saat pull gagal.
# Image tanpa '/' atau dengan komponen pertama tanpa titik berasal dari Docker Hub
# (endpoint API-nya registry-1.docker.io).
registry_host() {
# Dipecah DUA baris: `local x="$1" y="${x%%/*}"` mengevaluasi RHS `y` SEBELUM `x`
# di-assign → "unbound variable" di bawah `set -u` (bug laten yang baru muncul saat
# fungsi ini dipanggil di jalur normal).
local img="$1"
local first="${img%%/*}"
# WAJIB ada '/' dulu: tanpa itu, "postgres:15" membuat ${img%%/*} = seluruh string
# dan tanda ':' adalah TAG, bukan port registry (bug yang ditemukan saat menguji).
if [[ "${img}" == */* && ( "${first}" == *.* || "${first}" == *:* || "${first}" == "localhost" ) ]]; then
echo "${first}"
else
echo "registry-1.docker.io"
fi
}
image_size_mb() {
docker image inspect -f '{{.Size}}' "$1" 2>/dev/null | awk '{printf "%d", $1/1024/1024}'
}
# ─── Aksi: simpan / muat image Airflow yang dibangun lokal ──────────────────
do_save_airflow() {
local built; built="$(airflow_built_image)"
if ! docker image inspect "${built}" &>/dev/null; then
err "Image '${built}' belum ada — bangun dulu:"
echo " docker build -t ${built} -f docker/airflow/Dockerfile.332 docker/airflow"
exit 1
fi
inf "menyimpan '${built}' ke ${SAVE_AIRFLOW} (berisi seluruh layer, termasuk base)"
docker save -o "${SAVE_AIRFLOW}" "${built}" && \
ok "tersimpan ($(du -h "${SAVE_AIRFLOW}" | cut -f1)) — salin ke host lain lalu pakai --load-airflow"
}
do_load_airflow() {
[[ -f "${LOAD_AIRFLOW}" ]] || { err "berkas tidak ditemukan: ${LOAD_AIRFLOW}"; exit 1; }
inf "memuat image dari ${LOAD_AIRFLOW}"
docker load -i "${LOAD_AIRFLOW}" && ok "image dimuat"
local built; built="$(airflow_built_image)"
docker image inspect "${built}" &>/dev/null && ok "'${built}' tersedia" || warn "image '${built}' belum terlihat"
}
# ─── Aksi: tarik image ──────────────────────────────────────────────────────
do_pull() {
local images base built built_local
images="$(collect_images)"
if [[ -z "${images}" ]]; then
err "daftar image kosong — periksa docker-compose.yml / bundle host."
exit 1
fi
base="$(airflow_base_image || true)"
built="$(airflow_built_image)"
# Image Airflow: dikecualikan HANYA bila namanya menunjuk LOKAL polos
# (`sicabe/airflow:…`, tidak ada host registry). Bila `AIRFLOW_IMAGE` menunjuk
# REGISTRY (mis. `registry.sicabe.com/airflow:…`, host memuat titik), image
# IKUT di-pull seperti image lain — sejak image dipublikasikan ke registry internal.
# Deteksi memakai registry_host(): nama Docker Hub polos -> "registry-1.docker.io".
built_local=0
if [[ -n "${built}" && "$(registry_host "${built}")" == "registry-1.docker.io" ]]; then
built_local=1
# Keluarkan dari daftar (bukan sekadar dilewati) supaya daftar = yang benar-benar diunduh.
images="$(printf '%s\n' "${images}" | grep -vxF "${built}" || true)"
fi
echo
echo -e "${BLD}Daftar image (${MODE})${NC}"
printf '%s\n' "${images}" | sed 's/^/ - /'
[[ -n "${base}" && ${WITH_BASE} -eq 1 ]] && echo " - ${base} (base build Airflow)"
if [[ -n "${built}" && ${built_local} -eq 1 ]]; then
echo " (dikecualikan: ${built} — dibangun lokal, tidak ada di Docker Hub; tersedia di registry internal)"
fi
[[ ${WITH_DREMIO} -eq 0 ]] && echo " (dikecualikan: dremio/dremio-oss — opt-in, aktifkan dengan --with-dremio)"
if [[ ${LIST_ONLY} -eq 1 ]]; then
echo
inf "--list: tidak ada yang ditarik."
return 0
fi
echo
echo -e "${BLD}Menarik image${NC}"
local failed=() okc=0
while IFS= read -r img; do
[[ -z "${img}" ]] && continue
# Image yang dibangun lokal tidak ada di registry — jangan dicoba di-pull.
if [[ ${built_local} -eq 1 && "${img}" == "${built}" ]]; then
inf "${img} — dibangun lokal, dilewati (lihat alur build-sekali-salin di bawah)"
continue
fi
if docker image inspect "${img}" &>/dev/null && [[ "${img}" != *":latest" ]]; then
ok "${img} (sudah ada)"
okc=$((okc + 1))
continue
fi
inf "pull ${img} …"
# Output pull ditangkap ke berkas, BUKAN dibuang: tanpa ini kegagalan hanya
# berbunyi "GAGAL" tanpa sebab (dialami saat minio/quay.io gagal di server).
local perr; perr="$(mktemp)"
if docker pull "${img}" >"${perr}" 2>&1; then
ok "${img}"
okc=$((okc + 1))
else
warn "GAGAL: ${img}"
tail -3 "${perr}" | sed 's/^/ /' >&2
local reg; reg="$(registry_host "${img}")"
echo " hint: periksa akses ke registry '${reg}' → curl -sI https://${reg}/v2/" >&2
failed+=("${img}")
fi
rm -f "${perr}"
done <<< "${images}"
if [[ -n "${base}" && ${WITH_BASE} -eq 1 ]]; then
if docker image inspect "${base}" &>/dev/null; then
ok "${base} (sudah ada)"
elif docker pull -q "${base}" >/dev/null 2>&1; then
ok "${base} (base build Airflow)"
else
warn "GAGAL: ${base} — build Airflow lokal akan mengunduh ulang"
failed+=("${base}")
fi
fi
echo
echo -e "${BLD}Ringkasan${NC}"
echo " berhasil/sudah ada : ${okc}"
if [[ ${#failed[@]} -gt 0 ]]; then
echo " gagal : ${#failed[@]}"
printf ' - %s\n' "${failed[@]}"
fi
# Ukuran total image yang terdaftar (perkiraan kebutuhan disk)
local total=0 size
while IFS= read -r img; do
size="$(image_size_mb "${img}")"
[[ -n "${size}" ]] && total=$((total + size))
done <<< "${images}"
echo " ukuran total : ~${total} MB (image yang terdaftar)"
if [[ ${built_local} -eq 1 ]] && ! docker image inspect "${built}" &>/dev/null; then
echo
warn "'${built}' BELUM ada dan ia TIDAK dapat di-pull (dibangun lokal)."
echo " Alur build-sekali-salin (opsi b):"
echo " di SATU host : docker build -t ${built} -f docker/airflow/Dockerfile.332 docker/airflow"
echo " ./scripts/prefetch-images.sh --save-airflow /tmp/airflow.tar"
echo " host lain : scp /tmp/airflow.tar <host>:/tmp/"
echo " ./scripts/prefetch-images.sh --load-airflow /tmp/airflow.tar"
fi
[[ ${#failed[@]} -eq 0 ]]
}
main() {
parse_args "$@"
require_docker
if [[ -n "${SAVE_AIRFLOW}" ]]; then do_save_airflow; exit $?; fi
if [[ -n "${LOAD_AIRFLOW}" ]]; then do_load_airflow; exit $?; fi
echo -e "${BLD}Prefetch image SiCABE${NC}"
do_pull
}
main "$@"
deploy/scripts/prepare-host.sh
Executable
+316
View File
@@ -0,0 +1,316 @@
#!/usr/bin/env bash
# =============================================================================
# prepare-host.sh — siapkan server Ubuntu 24.04 (noble) untuk instalasi SiCABE
# -----------------------------------------------------------------------------
# Menyiapkan satu host sebelum `install.sh` dijalankan:
# 1. paket apt yang benar-benar dipakai script deploy (bukan container)
# 2. Docker Engine + plugin compose v2 (opsional --no-docker)
# 3. sysctl vm.max_map_count=262144 → SYARAT KERAS OpenSearch
# 4. (opsional --swap-off) matikan swap → rekomendasi OpenSearch
# 5. prometheus-node-exporter (:9100) → PRA-SYARAT dashboard Host
# (opsional --no-node-exporter)
# 6. precheck: AVX2, jam, RAM, disk, port yang akan dipakai
#
# Pemakaian:
# sudo ./scripts/prepare-host.sh --dry-run # lihat rencana, TIDAK mengubah apa pun
# sudo ./scripts/prepare-host.sh # jalankan
# sudo ./scripts/prepare-host.sh --no-docker # bila Docker sudah dipasang manual
# sudo ./scripts/prepare-host.sh --swap-off # ikut mematikan swap
# sudo ./scripts/prepare-host.sh --no-node-exporter # jangan pasang node_exporter
#
# Sifat: IDEMPOTEN — aman dijalankan berulang.
# =============================================================================
set -uo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
DEPLOY_DIR="$(cd "${SCRIPT_DIR}/.." && pwd)"
DRY_RUN=0
WITH_DOCKER=1
SWAP_OFF=0
WITH_NODE_EXPORTER=1
RED=$'\e[31m'; GRN=$'\e[32m'; YLW=$'\e[33m'; BLD=$'\e[1m'; NC=$'\e[0m'
ok() { echo -e " ${GRN}[OK]${NC} $*"; }
inf() { echo -e " ${YLW}[..]${NC} $*"; }
warn() { echo -e " ${YLW}[WARN]${NC} $*"; }
err() { echo -e " ${RED}[!!]${NC} $*" >&2; }
# Jalankan perintah; pada --dry-run cukup dicetak.
run() {
if [[ ${DRY_RUN} -eq 1 ]]; then
echo " [dry-run] $*"
return 0
fi
"$@"
}
usage() {
sed -n '2,22p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//'
exit 0
}
parse_args() {
while [[ $# -gt 0 ]]; do
case "$1" in
--dry-run) DRY_RUN=1 ;;
--no-docker) WITH_DOCKER=0 ;;
--swap-off) SWAP_OFF=1 ;;
--no-node-exporter) WITH_NODE_EXPORTER=0 ;;
-h|--help) usage ;;
*) err "Argumen tidak dikenal: $1"; exit 2 ;;
esac
shift
done
}
# ─── 0. OS & hak akses ──────────────────────────────────────────────────────
check_os() {
echo -e "${BLD}A. Sistem${NC}"
if [[ ! -r /etc/os-release ]]; then
err "/etc/os-release tidak terbaca — script ini untuk Ubuntu."; exit 1
fi
# shellcheck disable=SC1091
. /etc/os-release
echo " OS: ${PRETTY_NAME:-?} (codename: ${VERSION_CODENAME:-?})"
if [[ "${ID:-}" != "ubuntu" ]]; then
warn "Bukan Ubuntu (ID=${ID:-?}) — script ini diuji pada Ubuntu 24.04; lanjutkan hati-hati."
elif [[ "${VERSION_CODENAME:-}" != "noble" ]]; then
warn "Bukan 24.04/noble (codename=${VERSION_CODENAME:-?}) — nama paket bisa berbeda."
else
ok "Ubuntu 24.04 (noble) — sesuai target"
fi
if [[ ${EUID} -ne 0 && ${DRY_RUN} -eq 0 ]]; then
err "Perlu root: jalankan dengan sudo."
exit 1
fi
[[ ${EUID} -ne 0 ]] && inf "berjalan tanpa root (dry-run) — hanya memeriksa"
return 0
}
# ─── 1. Paket apt ───────────────────────────────────────────────────────────
# Daftar ini berasal dari pemakaian nyata di deploy/, bukan tebakan:
# htpasswd (apache2-utils) → password.db Trino (bootstrap/install)
# ldapsearch (ldap-utils) → bootstrap step 3 + verify-phase1
# mysql (client) → seed/reapply StarRocks + verify-phase4
# openssl → keystore Trino
# curl → hampir semua script
# python3 + PyYAML → lib/cluster/*.py + bootstrap
# ca-certificates/gnupg → repo & TLS
PKGS=(ca-certificates curl gnupg openssl apache2-utils ldap-utils python3 python3-yaml git unzip)
install_packages() {
echo
echo -e "${BLD}B. Paket apt${NC}"
# Klien MySQL: nama paket berbeda antar rilis → coba berurutan, laporkan yang dipakai.
local mysql_candidates=(default-mysql-client mysql-client mariadb-client) picked=""
run apt-get update -qq
run apt-get install -y --no-install-recommends "${PKGS[@]}"
for cand in "${mysql_candidates[@]}"; do
if apt-cache show "${cand}" >/dev/null 2>&1; then
run apt-get install -y --no-install-recommends "${cand}"
picked="${cand}"
break
fi
done
if [[ -n "${picked}" ]]; then
ok "paket terpasang (klien MySQL: ${picked})"
else
warn "klien MySQL tidak ditemukan di repo — seed/reapply StarRocks butuh binary 'mysql'"
fi
# MinIO client (mc) dipakai verify-phase2 & bootstrap, tapi tidak selalu ada di repo.
if command -v mc &>/dev/null; then
ok "mc tersedia"
else
warn "mc (MinIO client) belum ada — sebagian cek Fase 2 & bootstrap akan dilewati."
echo " Pasang manual bila diinginkan, mis.:"
echo " curl -sSL https://dl.min.io/client/mc/release/linux-amd64/mc -o /usr/local/bin/mc && chmod +x /usr/local/bin/mc"
fi
# node_exporter (:9100) — PRA-SYARAT dashboard Host (job `node` di Prometheus).
# Best-effort: paket ada di repo Ubuntu (universe). Bila tak tersedia, dashboard
# Host hanya menampilkan target DOWN — tidak fatal.
if [[ ${WITH_NODE_EXPORTER} -eq 0 ]]; then
inf "node_exporter dilewati (--no-node-exporter)"
elif command -v prometheus-node-exporter &>/dev/null; then
ok "node_exporter sudah tersedia (:9100)"
elif apt-cache show prometheus-node-exporter >/dev/null 2>&1; then
run apt-get install -y --no-install-recommends prometheus-node-exporter
ok "node_exporter terpasang & listen :9100 (di-scrape Prometheus job 'node')"
else
warn "paket prometheus-node-exporter tidak ada di repo — dashboard Host akan kosong"
fi
}
# ─── 2. Docker ──────────────────────────────────────────────────────────────
install_docker() {
echo
echo -e "${BLD}C. Docker${NC}"
if [[ ${WITH_DOCKER} -eq 0 ]]; then
inf "--no-docker: dilewati"
elif command -v docker &>/dev/null && docker compose version &>/dev/null; then
ok "Docker + compose v2 sudah tersedia ($(docker --version))"
else
inf "memasang Docker Engine + plugin compose dari repo resmi Docker"
run install -m 0755 -d /etc/apt/keyrings
# kunci GPG repo Docker
if [[ ${DRY_RUN} -eq 0 ]]; then
curl -fsSL https://download.docker.com/linux/ubuntu/gpg \
| gpg --dearmor -o /etc/apt/keyrings/docker.gpg
chmod a+r /etc/apt/keyrings/docker.gpg
# shellcheck disable=SC1091
. /etc/os-release
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \
https://download.docker.com/linux/ubuntu ${VERSION_CODENAME} stable" \
> /etc/apt/sources.list.d/docker.list
apt-get update -qq
apt-get install -y --no-install-recommends \
docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
else
echo " [dry-run] tambah repo Docker + pasang docker-ce, docker-compose-plugin, …"
fi
fi
# Pengguna non-root: supaya `docker` bisa dipakai tanpa sudo setelah re-login.
local target_user="${SUDO_USER:-}"
if [[ -n "${target_user}" && "${target_user}" != "root" && ${DRY_RUN} -eq 0 ]]; then
if id -nG "${target_user}" | grep -qw docker; then
ok "pengguna ${target_user} sudah di grup docker"
else
run usermod -aG docker "${target_user}"
warn "pengguna ${target_user} ditambahkan ke grup docker — LOGOUT/LOGIN agar berlaku"
fi
fi
if [[ ${DRY_RUN} -eq 0 ]] && command -v docker &>/dev/null; then
if systemctl is-active --quiet docker; then
ok "layanan docker aktif"
else
run systemctl enable --now docker
fi
fi
}
# ─── 3. sysctl (SYARAT KERAS OpenSearch) ────────────────────────────────────
configure_sysctl() {
echo
echo -e "${BLD}D. sysctl — vm.max_map_count${NC}"
local want=262144 have
have="$(sysctl -n vm.max_map_count 2>/dev/null || echo 0)"
if [[ "${have}" -ge "${want}" ]]; then
ok "vm.max_map_count=${have} (cukup)"
return 0
fi
warn "vm.max_map_count=${have} < ${want} — OpenSearch GAGAL start bila kurang (syarat keras)"
local f=/etc/sysctl.d/99-sicabe.conf
if [[ ${DRY_RUN} -eq 1 ]]; then
echo " [dry-run] tulis 'vm.max_map_count=${want}' ke ${f} lalu sysctl --system"
return 0
fi
echo "vm.max_map_count=${want}" > "${f}"
sysctl --system >/dev/null 2>&1 || true
ok "ditetapkan vm.max_map_count=$(sysctl -n vm.max_map_count 2>/dev/null) (persist: ${f})"
}
# ─── 4. Swap (opsional, rekomendasi OpenSearch) ─────────────────────────────
configure_swap() {
echo
echo -e "${BLD}E. Swap${NC}"
if [[ ${SWAP_OFF} -eq 0 ]]; then
inf "--swap-off tidak diberikan: swap dibiarkan (OpenSearch hanya merekomendasikan)"
return 0
fi
if [[ -z "$(swapon --show 2>/dev/null)" ]]; then
ok "swap sudah mati"
return 0
fi
if [[ ${DRY_RUN} -eq 1 ]]; then
echo " [dry-run] swapoff -a + nonaktifkan entri swap di /etc/fstab (dengan backup)"
return 0
fi
cp -a /etc/fstab "/etc/fstab.sicabe-bak-$(date +%Y%m%d%H%M%S)"
swapoff -a
sed -i 's|^\([^#].*[[:space:]]swap[[:space:]].*\)$|# \1|' /etc/fstab
ok "swap dimatikan (fstab dibackup; entri swap dikomentari)"
warn "pastikan RAM cukup — tanpa swap, OOM langsung membunuh proses"
}
# ─── 5. Precheck ────────────────────────────────────────────────────────────
precheck() {
echo
echo -e "${BLD}F. Precheck${NC}"
# AVX2: syarat keras starrocks-cn (tanpa ini container gagal SIGILL)
if grep -qo -m1 avx2 /proc/cpuinfo 2>/dev/null; then
ok "CPU mendukung AVX2 (syarat starrocks-cn)"
else
warn "AVX2 TIDAK terdeteksi — container starrocks-cn akan gagal (SIGILL)."
echo " Jalankan install tanpa Fase 4, atau pindahkan starrocks-cn ke host ber-AVX2."
fi
# Jam: OIDC/JWT sensitif terhadap skew (preflight juga memeriksanya)
if command -v timedatectl &>/dev/null; then
local ntp
ntp="$(timedatectl show -p NTPSynchronized --value 2>/dev/null || echo "")"
case "${ntp}" in
yes) ok "jam tersinkronisasi (NTP aktif)" ;;
no) warn "jam BELUM tersinkron — aktifkan NTP (timedatectl set-ntp true) sebelum install" ;;
*) inf "status NTP tidak dapat dibaca di lingkungan ini — verifikasi manual di server nyata" ;;
esac
fi
# RAM & disk
local ram_gb disk_gb
ram_gb="$(awk '/MemTotal/{printf "%d", $2/1024/1024}' /proc/meminfo 2>/dev/null || echo 0)"
disk_gb="$(df -BG --output=avail /var/lib/docker 2>/dev/null | tail -1 | tr -dc '0-9')"
[[ -z "${disk_gb}" ]] && disk_gb="$(df -BG --output=avail / | tail -1 | tr -dc '0-9')"
echo " RAM: ${ram_gb} GB · disk tersedia: ${disk_gb:-?} GB"
[[ "${ram_gb}" -lt 8 ]] 2>/dev/null && warn "RAM < 8 GB: sesuaikan penempatan komponen (lihat reviews/sicabe-09…)"
[[ "${disk_gb:-0}" -lt 20 ]] 2>/dev/null && warn "disk < 20 GB: image + volume bisa tidak cukup (siapkan ±20 GB untuk image)"
# Port yang akan dipakai: dibaca dari config/environment.defaults.sh agar sinkron
local ports=""
if [[ -f "${DEPLOY_DIR}/config/environment.defaults.sh" ]]; then
ports="$(grep -oE ':-(12[0-9]{3})\}' "${DEPLOY_DIR}/config/environment.defaults.sh" \
| tr -dc '0-9\n' | sort -un | paste -sd' ' -)"
fi
if [[ -n "${ports}" ]]; then
local busy="" p
for p in ${ports}; do
if (exec 3<>"/dev/tcp/127.0.0.1/${p}") 2>/dev/null; then busy+=" ${p}"; fi
exec 3<&- 2>/dev/null
done
if [[ -z "${busy}" ]]; then
ok "semua port 12xxx yang dibutuhkan masih bebas"
else
warn "port sudah terpakai:${busy}"
echo " Bila itu stack SiCABE yang lama: matikan dulu (RAM tidak cukup untuk dua stack)."
fi
else
inf "daftar port tidak terbaca dari environment.defaults.sh — lewati pemeriksaan port"
fi
}
main() {
parse_args "$@"
echo -e "${BLD}Persiapan host untuk SiCABE${NC}"
echo " deploy dir : ${DEPLOY_DIR}"
[[ ${DRY_RUN} -eq 1 ]] && echo -e " ${YLW}MODE DRY-RUN — tidak ada perubahan sistem${NC}"
echo
check_os
install_packages
install_docker
configure_sysctl
configure_swap
precheck
echo
echo -e "${BLD}Ringkasan${NC}"
echo " Langkah berikutnya di host ini:"
echo " 1. salin direktori deploy/ ke host ini"
echo " 2. ./scripts/prefetch-images.sh # tarik semua image lebih dulu"
echo " 3. ./install.sh # atau ./install.sh --host <nama>"
echo " 4. ./install.sh --preflight (cek keterjangkauan lintas host)"
}
main "$@"
deploy/scripts/print-keycloak-requirements.sh
Executable
+81
View File
@@ -0,0 +1,81 @@
#!/usr/bin/env bash
# =============================================================================
# eBdesk SiCABE — print-keycloak-requirements.sh
# -----------------------------------------------------------------------------
# Mencetak KEBUTUHAN Keycloak (nilai konkret dari .env) untuk diserahkan ke
# administrator Keycloak — pasangan dari docs/keycloak-existing-requirements.md.
#
# Read-only: tidak mengubah apa pun; hanya menampilkan.
#
# Usage:
# ./scripts/print-keycloak-requirements.sh
# ./scripts/print-keycloak-requirements.sh --env-file generated/host-control/.env
# ./scripts/print-keycloak-requirements.sh --mask # sembunyikan client secret
#
# Keluaran bisa disalin ke tiket/email ke pengelola Keycloak.
# =============================================================================
set -uo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
# shellcheck source=../lib/common.sh
source "${SCRIPT_DIR}/lib/common.sh"
# shellcheck source=../config/environment.defaults.sh
source "${SCRIPT_DIR}/config/environment.defaults.sh"
MASK=0
ARGS=()
for _a in "$@"; do
case "${_a}" in
--mask) MASK=1 ;;
*) ARGS+=("${_a}") ;;
esac
done
sicabe_load_env "${SCRIPT_DIR}" "${ARGS[@]}"
REALM="${SICABE_REALM:-sicabe}"
KC_PUB="${KEYCLOAK_PUBLIC_URL:-http://localhost:${KEYCLOAK_PORT:-12080}}"
SSL_REQ="${KEYCLOAK_SSL_REQUIRED:-external}"
SR_HTTP_PORT="${STARROCKS_HTTP_PORT:-12030}"
# Tampilkan secret: nilai asli, atau '********' bila --mask.
show_secret() { # <value> <dev-default>
local v="${1:-}" d="${2:-}"
if [[ -z "${v}" ]]; then echo "(KOSONG!)"; return; fi
if [[ "${MASK}" == "1" ]]; then echo "********"; return; fi
if [[ "${v}" == "${d}" ]]; then echo "${v} ← DEFAULT DEV, ganti di produksi"; else echo "${v}"; fi
}
printf '%s\n' "================================================================================"
printf '%s\n' " KEBUTUHAN KEYCLOAK (existing) — dari: ${SICABE_ENV_FILE:-${SCRIPT_DIR}/.env}"
printf '%s\n' "================================================================================"
printf ' Realm : %s\n' "${REALM}"
printf ' Issuer : %s/realms/%s\n' "${KC_PUB}" "${REALM}"
printf ' sslRequired : %s (external bila ada TLS; none utk http lokal)\n' "${SSL_REQ}"
printf '\n'
printf '%s\n' " Realm roles : admin engineer analyst viewer"
printf '%s\n' " Users (opsional) : admin engineer analyst viewer"
printf '%s\n' " (atau petakan user organisasi ke 4 role di atas)"
printf '\n'
printf '%s\n' " OIDC clients — SEMUA: type=OpenID Connect, client authentication ON,"
printf '%s\n' " Standard flow ON, Service accounts ON (publicClient=false)"
printf '\n'
printf ' %-13s %-34s %s\n' "clientId" "client secret (harus sama)" "redirect URI"
printf ' %-13s %-34s %s\n' "------------" "---------------------------" "--------------------------"
printf ' %-13s %-34s %s/*\n' "trino" "$(show_secret "${TRINO_CLIENT_SECRET:-}" "trino-secret")" "${TRINO_PUBLIC_URL:-}"
printf ' %-13s %-34s %s\n' "starrocks" "$(show_secret "${STARROCKS_CLIENT_SECRET:-}" "starrocks-secret")" "http://<host-starrocks>:${SR_HTTP_PORT}/api/oauth2"
printf ' %-13s %-34s %s/*\n' "airflow" "$(show_secret "${AIRFLOW_CLIENT_SECRET:-}" "airflow-secret")" "${AIRFLOW_PUBLIC_URL:-}"
printf ' %-13s %-34s %s/*\n' "openmetadata" "$(show_secret "${OPENMETADATA_CLIENT_SECRET:-}" "om-secret")" "${OPENMETADATA_PUBLIC_URL:-}"
printf ' %-13s %-34s %s/*\n' "grafana" "$(show_secret "${GRAFANA_CLIENT_SECRET:-}" "grafana-secret")" "${GRAFANA_PUBLIC_URL:-}"
printf '\n'
printf '%s\n' " Semua client juga sertakan: http://localhost:* dan https://localhost:*"
printf '\n'
printf '%s\n' " Protocol mapper (WAJIB tiap client): Audience 'account' (oidc-audience-mapper)"
printf '%s\n' " → Tanpa ini Trino MENOLAK token (Trino memverifikasi aud=account)."
printf '%s\n' " StarRocks: tambahan audience 'starrocks' (required_audience=starrocks)."
printf '\n'
printf '%s\n' " Kembalikan ke SiCABE: nama realm, issuer URL, dan 5 client secret di atas."
if [[ "${MASK}" == "0" ]]; then
printf '\n'
printf '%s\n' " ⚠ KELUARAN MEMUAT SECRET — kirim lewat kanal aman; jangan tempel di tiket publik."
fi
printf '%s\n' "================================================================================"
deploy/scripts/print-openldap-requirements.sh
Executable
+79
View File
@@ -0,0 +1,79 @@
#!/usr/bin/env bash
# =============================================================================
# eBdesk SiCABE — print-openldap-requirements.sh
# -----------------------------------------------------------------------------
# Mencetak KEBUTUHAN OpenLDAP (nilai konkret dari .env) untuk diserahkan ke
# administrator LDAP — pasangan dari docs/openldap-existing-requirements.md.
#
# Read-only: hanya menampilkan.
#
# Usage:
# ./scripts/print-openldap-requirements.sh
# ./scripts/print-openldap-requirements.sh --env-file generated/host-control/.env
# ./scripts/print-openldap-requirements.sh --mask # sembunyikan bind password
# =============================================================================
set -uo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
# shellcheck source=../lib/common.sh
source "${SCRIPT_DIR}/lib/common.sh"
# shellcheck source=../config/environment.defaults.sh
source "${SCRIPT_DIR}/config/environment.defaults.sh"
MASK=0
ARGS=()
for _a in "$@"; do
case "${_a}" in
--mask) MASK=1 ;;
*) ARGS+=("${_a}") ;;
esac
done
sicabe_load_env "${SCRIPT_DIR}" "${ARGS[@]}"
BASE_DN="${LDAP_BASE_DN:-dc=sicabe,dc=local}"
USERS_DN="${LDAP_USERS_DN:-ou=users,${BASE_DN}}"
GROUPS_DN="${LDAP_GROUPS_DN:-ou=groups,${BASE_DN}}"
BIND_DN="${LDAP_ADMIN_DN:-cn=admin,${BASE_DN}}"
LDAP_URL="${LDAP_URL:-ldap://openldap:389}"
LDAP_HOST="${LDAP_HOST:-localhost}"
LDAP_PORT="${LDAP_PORT:-389}"
show_secret() { # <value> <dev-default>
local v="${1:-}" d="${2:-}"
if [[ -z "${v}" ]]; then echo "(KOSONG!)"; return; fi
if [[ "${MASK}" == "1" ]]; then echo "********"; return; fi
if [[ "${v}" == "${d}" ]]; then echo "${v} ← DEFAULT DEV, ganti di produksi"; else echo "${v}"; fi
}
printf '%s\n' "================================================================================"
printf '%s\n' " KEBUTUHAN OPENLDAP (existing) — dari: ${SICABE_ENV_FILE:-${SCRIPT_DIR}/.env}"
printf '%s\n' "================================================================================"
printf '%s\n' " Peran LDAP di SiCABE = SUMBER GRUP (bukan autentikasi; auth = Keycloak OIDC)."
printf '\n'
printf ' URL (dari container) : %s\n' "${LDAP_URL}"
printf ' host:port (dari host): %s:%s\n' "${LDAP_HOST}" "${LDAP_PORT}"
printf '\n'
printf ' Base DN : %s\n' "${BASE_DN}"
printf ' OU users : %s\n' "${USERS_DN}"
printf ' OU groups : %s\n' "${GROUPS_DN}"
printf '\n'
printf ' Akun BIND (buat baca) : %s\n' "${BIND_DN}"
printf ' → boleh akun READ-ONLY, bukan harus admin.\n'
printf ' password : %s\n' "$(show_secret "${LDAP_ADMIN_PASSWORD:-}" "ldap_admin_DevOnly")"
printf '\n'
printf '%s\n' " Grup (WAJIB, objectClass=groupOfNames, atribut cn + member=DN user):"
printf '%s\n' " data-admin data-engineer data-analyst data-viewer"
printf '%s\n' " → nama ini HARUS sama dengan kunci roles di config/opa/data.json"
printf '%s\n' " DAN permitted_groups StarRocks."
printf '\n'
printf '%s\n' " User (objectClass inetOrgPerson + posixAccount):"
printf '%s\n' " uid HARUS = username Keycloak (preferred_username) ← krusial"
printf '%s\n' " atribut minimal: uid, cn, sn, mail, uidNumber, gidNumber, homeDirectory"
printf '\n'
printf '%s\n' " Kembalikan ke SiCABE: LDAP_URL, host:port, bind DN + password, base DN/OU,"
printf '%s\n' " dan konfirmasi bahwa uid user = username Keycloak & keanggotaan grup benar."
if [[ "${MASK}" == "0" ]]; then
printf '\n'
printf '%s\n' " ⚠ KELUARAN MEMUAT PASSWORD — kirim lewat kanal aman; jangan tempel di tiket publik."
fi
printf '%s\n' "================================================================================"
deploy/scripts/render-config.sh
Executable
+195
View File
@@ -0,0 +1,195 @@
#!/usr/bin/env bash
# ─────────────────────────────────────────────────────────────────────────────
# render-config.sh — render config/** untuk deploy multi-host (Fase 0/1, R10 §4.9)
#
# MASALAH: file config/** yang dibaca CONTAINER (.properties/.yaml/.cfg/.json/rego)
# TIDAK dapat melakukan ekspansi env saat runtime, berbeda dari docker-compose.yml.
# Pada deploy multi-host, hostname nama container (mis. `nessie:19120`) harus
# menjadi alamat host sebenarnya (mis. `10.20.0.10:19120`).
#
# SOLUSI (Opsi (i) R10 §4.9): `config/` dibiarkan LITERAL (single-host tidak
# tersentuh); script ini menyalinnya ke direktori output sambil mengganti
# hostname TERDAFTAR, menghasilkan `generated/host-<x>/config/` (gitignored).
#
# PRINSIP KEAMANAN
# 1. Hanya hostname yang di-`--map` yang diganti → tidak ada kejutan.
# 2. Penggantian ANCHORED: `host` diganti hanya bila diikuti `:` atau `/`
# (posisi target jaringan). Karena itu `KC_DB: postgres` (dialek DB),
# `DB_SCHEME: postgresql`, dan `postgres:15` (tag image di compose) TIDAK
# ikut berubah.
# 3. `config/` sumber tidak pernah ditulis — hanya dibaca.
# 4. Baris KOMENTAR (`#`) tidak ditulis ulang: komentar sering menyebut nilai
# default lokal, dan menimpanya akan membuat komentar itu justru salah.
#
# PEMAKAIAN
# # Uji tanpa menulis (perlihatkan rencana):
# ./scripts/render-config.sh --out /tmp/render --dry-run \
# --map postgres=10.20.0.12 --map nessie=10.20.0.10
#
# # Render sungguhan:
# ./scripts/render-config.sh --out generated/host-compute/config \
# --map postgres=10.20.0.12 --map minio=10.20.0.12
#
# # Fase 1 akan menambah: --inventory cluster.yml --host <nama>
# ─────────────────────────────────────────────────────────────────────────────
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
DEPLOY_DIR="$(cd "${SCRIPT_DIR}/.." && pwd)"
SRC_CONFIG="${DEPLOY_DIR}/config"
OUT_DIR=""
DRY_RUN=0
declare -A MAP=()
# Hostname yang SAH dipetakan ke alamat host. Daftar sengaja tetap (bukan bebas)
# supaya salah ketik --map tidak diam-diam menulis ulang file config.
KNOWN_HOSTS=(
postgres minio nessie keycloak openldap opa
trino trino-coordinator trino-worker
starrocks-fe starrocks-cn airflow-webserver
openmetadata-server openmetadata-opensearch dremio grafana prometheus
# Sidecar observabilitas (M2/M3): target di config/prometheus dipetakan ke alamat
# HOST-nya saat multi-host. WAJIB ada di sini — generate.py meng-`--map` setiap
# RENDER_HOSTS, dan hostname yang tak terdaftar membuat render-config menolak,
# sehingga `install.sh --generate` GAGAL TOTAL (bug M2/M3 yang diperbaiki di sini).
statsd-exporter trino-jmx-exporter
)
# Tidak di-render: skrip yang sanggup ekspansi env sendiri + dokumentasi.
# environment.defaults.sh — di-source shell, memakai ${VAR:-...}
# buckets.sh — skrip mc, memakai ${MINIO_ENDPOINT:-...}
SKIP_FILES=(environment.defaults.sh buckets.sh)
SKIP_PATTERNS=("*.md")
RED=$'\e[31m'; GRN=$'\e[32m'; YLW=$'\e[33m'; BLD=$'\e[1m'; NC=$'\e[0m'
ok() { echo -e " ${GRN}[OK]${NC} $*"; }
inf() { echo -e " ${YLW}[..]${NC} $*"; }
err() { echo -e " ${RED}[!!]${NC} $*" >&2; }
usage() {
sed -n '2,30p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//'
echo
echo "Hostname yang dikenal: ${KNOWN_HOSTS[*]}"
}
is_known_host() {
local h="$1" k
for k in "${KNOWN_HOSTS[@]}"; do [[ "${k}" == "${h}" ]] && return 0; done
return 1
}
add_map() {
local pair="$1" host addr
if [[ "${pair}" != *"="* ]]; then
err "--map harus berformat host=alamat (diberikan: '${pair}')"; exit 2
fi
host="${pair%%=*}"; addr="${pair#*=}"
[[ -n "${host}" && -n "${addr}" ]] || { err "--map kosong: '${pair}'"; exit 2; }
is_known_host "${host}" || { err "hostname '${host}' tidak dikenal (lihat --help)"; exit 2; }
MAP["${host}"]="${addr}"
}
parse_args() {
while [[ $# -gt 0 ]]; do
case "$1" in
--out) OUT_DIR="${2:-}"; shift 2 ;;
--map) add_map "${2:-}"; shift 2 ;;
--dry-run) DRY_RUN=1; shift ;;
-h|--help) usage; exit 0 ;;
*) err "Argumen tidak dikenal: $1"; usage; exit 2 ;;
esac
done
[[ -n "${OUT_DIR}" ]] || { err "--out wajib diisi"; exit 2; }
[[ ${#MAP[@]} -gt 0 ]] || { err "minimal satu --map host=alamat wajib diisi"; exit 2; }
}
# Satu pass perl untuk semua host. Anchor diperketat ke TIGA konteks jaringan —
# tanpa ini, path filesystem seperti `/etc/trino/` atau `/data/trino/` ikut
# terkorupsi (hostname `trino` diikuti `/`):
# 1. setelah skema URL `//` → http://nessie:19120, jdbc:postgresql://postgres:5432
# 2. setelah `@` (JDBC) → user:pw@postgres:5432
# 3. bare `host:port` → nessie:19120, starrocks-fe:8030 (bukan bagian token lain)
# Baris komentar (`#`) tidak ditulis ulang.
render_file() {
local src="$1" dst="$2"
local -a pairs=()
local k
for k in "${!MAP[@]}"; do pairs+=("${k}=${MAP[$k]}"); done
perl -e '
my ($src, $dst, @pairs) = @ARGV;
my %map = map { split(/=/, $_, 2) } @pairs;
my $alt = join "|", sort { length($b) <=> length($a) } map { quotemeta } keys %map;
open(my $in, "<", $src) or die "buka gagal: $src\n";
open(my $out, ">", $dst) or die "tulis gagal: $dst\n";
while (my $l = <$in>) {
unless ($l =~ /^\s*#/) {
$l =~ s{(?<=//)(${alt})(?=[:/])}{$map{$1}}g; # 1. setelah skema //
$l =~ s{(?<=@)(${alt})(?=[:/])}{$map{$1}}g; # 2. setelah @ (JDBC)
$l =~ s{(?<![\w/.:-])(${alt})(?=:\d)}{$map{$1}}g; # 3. bare host:port
# 4. nilai BARE untuk key yang jelas host (host, hostname, *_HOST).
# `host: postgres` (Sling) tidak punya ':' atau '/', sehingga aturan 1-3
# tidak mengenainya -> di host lain Sling mencari nama container yang tidak
# ada (kelas bug yang sama dengan DB_HOST OpenMetadata). Hanya key ber-'host'
# yang disentuh, supaya `type: postgres` / `KC_DB: postgres` (dialek) aman.
$l =~ s{^(\s*(?:-\s*)?[A-Za-z_.\-]*(?:host|hostname)[A-Za-z_.\-]*\s*[:=]\s*)(${alt})(\s*)$}{$1 . $map{$2} . $3}gie;
}
print $out $l;
}
close $in; close $out;
' "${src}" "${dst}" "${pairs[@]}"
}
# Jumlah baris sumber yang berubah di file tujuan.
count_subs() {
local src="$1" dst="$2"
diff "${src}" "${dst}" 2>/dev/null | grep -c '^<' || true
}
should_skip() {
local name="$1" pat skip
for skip in "${SKIP_FILES[@]}"; do [[ "${name}" == "${skip}" ]] && return 0; done
for pat in "${SKIP_PATTERNS[@]}"; do
# shellcheck disable=SC2053
[[ "${name}" == ${pat} ]] && return 0
done
return 1
}
# Render satu file; cetak jumlah baris berubah bila ada.
process_file() {
local src="$1" rel="${1#${SRC_CONFIG}/}" dst="$2" n tmp
if [[ ${DRY_RUN} -eq 1 ]]; then
tmp="$(mktemp)"; render_file "${src}" "${tmp}"; n="$(count_subs "${src}" "${tmp}")"; rm -f "${tmp}"
else
dst="${OUT_DIR}/${rel}"
mkdir -p "$(dirname "${dst}")"
render_file "${src}" "${dst}"
n="$(count_subs "${src}" "${dst}")"
fi
CHANGED_TOTAL=$((CHANGED_TOTAL + n))
if [[ "${n}" -gt 0 ]]; then ok "${rel} — ${n} baris"; FILES_CHANGED=$((FILES_CHANGED + 1)); fi
}
main() {
parse_args "$@"
echo -e "${BLD}Render config/** untuk multi-host${NC}"
echo " sumber : ${SRC_CONFIG}"
echo " tujuan : ${OUT_DIR}$([[ ${DRY_RUN} -eq 1 ]] && echo ' (DRY-RUN)')"
echo -n " peta : "; for k in "${!MAP[@]}"; do printf '%s→%s ' "${k}" "${MAP[$k]}"; done; echo; echo
[[ ${DRY_RUN} -eq 0 ]] && mkdir -p "${OUT_DIR}"
FILES_CHANGED=0; CHANGED_TOTAL=0
local src scanned=0 skipped=0
while IFS= read -r -d '' src; do
scanned=$((scanned + 1))
if should_skip "$(basename "${src}")"; then skipped=$((skipped + 1)); continue; fi
process_file "${src}" ""
done < <(find "${SRC_CONFIG}" -type f -print0 | sort -z)
echo
inf "dipindai ${scanned} file · ${FILES_CHANGED} berubah (${CHANGED_TOTAL} baris) · ${skipped} dilewati"
echo -e " ${GRN}${BLD}Selesai.${NC} config/ sumber tidak tersentuh."
}
main "$@"
deploy/scripts/seed-native-dq.sh
Executable
+69
View File
@@ -0,0 +1,69 @@
#!/usr/bin/env bash
# =============================================================================
# eBdesk SiCABE — seed contoh Data Quality NATIVE OpenMetadata (Jalur B)
# =============================================================================
# Membuat contoh test case native + pipeline TestSuite yang membawa koneksi Trino
# eksplisit, lalu (opsional) deploy + trigger supaya hasilnya masuk ke OM.
#
# Mengapa lewat pipeline, bukan TestRunner SDK: lihat header
# scripts/airflow/seed-om-native-dq.py dan docs/sicabe-install-phases-detail.md
# §8.10 — jalur SDK memakai koneksi tersimpan yang kehilangan secret (bug
# upstream) sehingga selalu `Aborted`.
#
# Usage:
# ./scripts/seed-native-dq.sh # buat definisi + pipeline saja
# ./scripts/seed-native-dq.sh --trigger # sekaligus deploy + trigger
#
# Idempoten: test case & pipeline yang sudah ada dilewati.
# Dijalankan dari host; pekerjaannya dieksekusi DI DALAM container Airflow.
# =============================================================================
set -euo pipefail
# ⚠️ JALUR B TERBLOKIR UPSTREAM — banner ini disengaja.
# DQ native OpenMetadata (Jalur B) memakai koneksi tersimpan yang kehilangan secret saat
# dibangun runner, sehingga test SELALU berakhir `Aborted` (docs/sicabe-install-phases-detail.md
# §8.10). Karena itu skrip ini TIDAK dipanggil bootstrap.sh / seed-sample-data.sh /
# ingest-openmetadata.sh: ia opt-in untuk eksperimen. Gate fase 6 memperlakukannya sebagai
# SKIP, bukan FAIL. Untuk DQ yang benar-benar bekerja, pakai Jalur A (Great Expectations)
# pada task validate_gold / cdc_validate di DAG sicabe_batch_daily.
echo "⚠️ seed-native-dq.sh = Jalur B (DQ native OM) yang TERBLOKIR bug upstream (§8.10)."
echo " Hasil test akan 'Aborted'. Untuk DQ yang bekerja: task validate_gold/cdc_validate (Jalur A)."
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
DEPLOY_DIR="$(dirname "${SCRIPT_DIR}")"
# shellcheck source=../lib/common.sh
. "${DEPLOY_DIR}/lib/common.sh" 2>/dev/null || true
# shellcheck source=../config/environment.defaults.sh
. "${DEPLOY_DIR}/config/environment.defaults.sh" 2>/dev/null || true
# .env memuat password admin OM + password service account Trino.
# Multi-host: dukung --env-file / SICABE_ENV_FILE seperti skrip lain.
sicabe_load_env "${DEPLOY_DIR}" "$@" || exit 1
AIRFLOW_CONTAINER="${AIRFLOW_CONTAINER:-sicabe-airflow-webserver}"
WORKER="/opt/airflow/scripts/seed-om-native-dq.py"
# Multi-host: Airflow ada di host `control`; `docker exec` bersifat lokal.
sicabe_require_local_container "${AIRFLOW_CONTAINER}" "Airflow webserver" "control" || exit $?
for var in OPENMETADATA_ADMIN_PASSWORD TRINO_SVC_OM_PASSWORD; do
if [[ -z "${!var:-}" ]]; then
echo "GAGAL: ${var} kosong — periksa ${DEPLOY_DIR}/.env" >&2
exit 1
fi
done
echo "== seed contoh DQ native =="
# Buang `--env-file <berkas>` dari argumen yang diteruskan ke worker Python
# (sudah dipakai sicabe_load_env; worker tak mengenalnya).
WORKER_ARGS=(); _snq_prev=""
for _snq_a in "$@"; do
if [[ "${_snq_prev}" == "--env-file" ]]; then _snq_prev=""; continue; fi
if [[ "${_snq_a}" == "--env-file" ]]; then _snq_prev="--env-file"; continue; fi
WORKER_ARGS+=("${_snq_a}")
done
docker exec \
-e OPENMETADATA_ADMIN_PASSWORD="${OPENMETADATA_ADMIN_PASSWORD}" \
-e TRINO_SVC_OM_PASSWORD="${TRINO_SVC_OM_PASSWORD}" \
"${AIRFLOW_CONTAINER}" python "${WORKER}" "${WORKER_ARGS[@]+"${WORKER_ARGS[@]}"}"
deploy/scripts/seed-sample-data.sh
Executable
+169
View File
@@ -0,0 +1,169 @@
#!/usr/bin/env bash
# =============================================================================
# eBdesk SiCABE — Seed Sample Data: Iceberg bronze.orders
# =============================================================================
# Membuat namespace + tabel contoh `iceberg.bronze.orders` (8 kolom, sesuai
# dbt sources.yml) lalu mengisinya dengan 6 baris contoh. Idempoten: aman
# dijalankan ulang (schema/table IF NOT EXISTS, INSERT hanya bila kosong).
#
# Autentikasi: OIDC password-grant (client 'trino') ke Keycloak → Bearer JWT.
# ⚠️ Password-grant bersifat dev-only (bukan untuk produksi).
#
# Pemakaian:
# cd deploy
# set -a; source config/environment.defaults.sh; source .env; set +a
# ./scripts/seed-sample-data.sh
#
# Env (opsional, ada default):
# TRINO_HTTPS_PORT (12443), KEYCLOAK_PORT (12080), SICABE_REALM (sicabe),
# TRINO_CLIENT_SECRET, KEYCLOAK_ADMIN_PASSWORD, SEED_USER (admin),
# SEED_CATALOG (iceberg), SEED_SCHEMA (bronze), SEED_TABLE (orders)
# =============================================================================
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
cd "${SCRIPT_DIR}"
# ─── Shared Library & Environment ───────────────────────────────────────────
source "${SCRIPT_DIR}/lib/common.sh"
source "${SCRIPT_DIR}/config/environment.defaults.sh"
sicabe_load_env "${SCRIPT_DIR}" "$@"
# Service account (HTTP Basic) — langsung ke Trino HTTPS (self-signed).
# Multi-host: Trino ada di host `compute` → endpoint host-aware (single-host tetap localhost).
TRINO_URL="$(sicabe_local_url https TRINO_HOST "${TRINO_HTTPS_PORT:-12443}" TRINO_URL)"
SEED_USER="${SEED_USER:-svc_seed}"
SEED_PASSWORD="${SEED_PASSWORD:-${TRINO_SVC_SEED_PASSWORD:-}}"
SEED_CATALOG="${SEED_CATALOG:-iceberg}"
SEED_SCHEMA="${SEED_SCHEMA:-bronze}"
SEED_TABLE="${SEED_TABLE:-orders}"
FQN="${SEED_CATALOG}.${SEED_SCHEMA}.${SEED_TABLE}"
usage_seed() {
echo "Usage: $0 [--user <trino-user>] [--help]"
echo " Seed tabel contoh '${FQN}' (idempoten)."
}
# ─── Auth: service account (HTTP Basic) ─────────────────────────────────────
check_auth() {
[[ -n "${SEED_USER}" && -n "${SEED_PASSWORD}" ]]
}
# ─── Submit SQL ke Trino + polling nextUri (Trino 483) ──────────────────────
# Cetak baris data (jika ada); gagal (return 1) bila Trino melaporkan error.
trino_exec() {
local sql="$1" resp next err i=0
resp=$(curl -skf -X POST "${TRINO_URL}/v1/statement" \
-u "${SEED_USER}:${SEED_PASSWORD}" \
-H "Content-Type: text/plain" \
-d "${sql}" 2>/dev/null) || { echo -e "${ERR} submit ke Trino gagal."; return 1; }
while :; do
err=$(echo "${resp}" | python3 -c "import sys,json;d=json.load(sys.stdin);e=d.get('error');print(e.get('message','') if e else '')" 2>/dev/null || true)
if [[ -n "${err}" ]]; then
echo -e "${ERR} ${err}"
return 1
fi
echo "${resp}" | python3 -c "import sys,json;d=json.load(sys.stdin);[print('\t',r) for r in (d.get('data') or [])]" 2>/dev/null || true
next=$(echo "${resp}" | python3 -c "import sys,json;print(json.load(sys.stdin).get('nextUri',''))" 2>/dev/null || true)
[[ -z "${next}" ]] && break
i=$((i + 1))
if [[ ${i} -gt 60 ]]; then
echo -e "${WARN} Polling Trino melebihi batas waktu."
return 1
fi
sleep 1
resp=$(curl -skf -u "${SEED_USER}:${SEED_PASSWORD}" "${next}" 2>/dev/null) || return 1
done
return 0
}
# ─── Ambil satu nilai skalar dari Trino ─────────────────────────────────────
trino_value() {
local sql="$1" resp next val i=0
resp=$(curl -skf -X POST "${TRINO_URL}/v1/statement" \
-u "${SEED_USER}:${SEED_PASSWORD}" \
-H "Content-Type: text/plain" \
-d "${sql}" 2>/dev/null) || return 1
while :; do
val=$(echo "${resp}" | python3 -c "import sys,json;d=json.load(sys.stdin);r=d.get('data') or [];print(r[0][0] if r else '')" 2>/dev/null || true)
[[ -n "${val}" ]] && { echo "${val}"; return 0; }
next=$(echo "${resp}" | python3 -c "import sys,json;print(json.load(sys.stdin).get('nextUri',''))" 2>/dev/null || true)
[[ -z "${next}" ]] && { echo "${val}"; return 0; }
i=$((i + 1)); [[ ${i} -gt 60 ]] && return 1
sleep 1
resp=$(curl -skf -u "${SEED_USER}:${SEED_PASSWORD}" "${next}" 2>/dev/null) || return 1
done
}
# ─── DDL + seed (idempoten) ─────────────────────────────────────────────────
seed_table() {
echo -e "${INFO} [1/3] Buat namespace '${SEED_CATALOG}.${SEED_SCHEMA}'..."
trino_exec "CREATE SCHEMA IF NOT EXISTS ${SEED_CATALOG}.${SEED_SCHEMA}" \
&& echo -e "${OK} Namespace '${SEED_SCHEMA}' siap."
echo -e "${INFO} [2/3] Buat tabel '${FQN}'..."
trino_exec "CREATE TABLE IF NOT EXISTS ${FQN} (
id bigint,
order_date date,
komoditas varchar,
harga_per_kg double,
stok_kg bigint,
region varchar,
created_at timestamp(6),
updated_at timestamp(6)
) WITH (format = 'PARQUET')" \
&& echo -e "${OK} Tabel '${FQN}' siap."
echo -e "${INFO} [3/3] Isi data contoh (bila kosong)..."
local count
count=$(trino_value "SELECT count(*) FROM ${FQN}") || count="0"
if [[ "${count}" -gt 0 ]]; then
echo -e "${OK} Tabel sudah berisi ${count} baris (skip seed)."
return 0
fi
trino_exec "INSERT INTO ${FQN} VALUES
(1, DATE '2026-09-01', 'beras', 12000.0, 500, 'jakarta', TIMESTAMP '2026-09-01 08:00:00', TIMESTAMP '2026-09-01 08:00:00'),
(2, DATE '2026-09-01', 'gula', 15000.0, 300, 'bandung', TIMESTAMP '2026-09-01 09:00:00', TIMESTAMP '2026-09-01 09:00:00'),
(3, DATE '2026-09-02', 'minyak', 18000.0, 200, 'surabaya', TIMESTAMP '2026-09-02 08:00:00', TIMESTAMP '2026-09-02 08:00:00'),
(4, DATE '2026-09-02', 'beras', 12500.0, 450, 'jakarta', TIMESTAMP '2026-09-02 10:00:00', TIMESTAMP '2026-09-02 10:00:00'),
(5, DATE '2026-09-03', 'tepung', 11000.0, 350, 'medan', TIMESTAMP '2026-09-03 08:00:00', TIMESTAMP '2026-09-03 08:00:00'),
(6, DATE '2026-09-03', 'gula', 15500.0, 250, 'bandung', TIMESTAMP '2026-09-03 11:00:00', TIMESTAMP '2026-09-03 11:00:00')" \
&& echo -e "${OK} 6 baris contoh dimasukkan."
}
# ─── Verifikasi akhir ───────────────────────────────────────────────────────
verify_seed() {
echo -e "${INFO} Verifikasi:"
trino_exec "SELECT id, komoditas, harga_per_kg, stok_kg, region FROM ${FQN} ORDER BY id"
}
# ─── MAIN ───────────────────────────────────────────────────────────────────
main() {
while [[ $# -gt 0 ]]; do
case "$1" in
--env-file) shift 2 ;; # sudah dipakai sicabe_load_env() di atas
--user) SEED_USER="$2"; shift 2 ;;
--help|-h) usage_seed; exit 0 ;;
*) echo -e "${ERR} Argumen tidak dikenal: $1"; usage_seed; exit 1 ;;
esac
done
echo -e "${BOLD}${BLUE}═══ Seed Sample Data — Iceberg '${FQN}' ═══${NC}"
echo -e "${INFO} Autentikasi Trino (service account '${SEED_USER}')..."
if ! check_auth; then
echo -e "${ERR} SEED_PASSWORD / TRINO_SVC_SEED_PASSWORD belum diset."
exit 1
fi
echo -e "${OK} Kredensial service account siap."
seed_table
verify_seed
echo -e "${OK} ${GREEN}Seed selesai — '${FQN}' siap dipakai.${NC}"
}
main "$@"
deploy/scripts/seed-source-postgres.sh
Executable
+99
View File
@@ -0,0 +1,99 @@
#!/usr/bin/env bash
# =============================================================================
# eBdesk SiCABE — Seed Sumber PostgreSQL: sicabe_data.public.orders
# =============================================================================
# Membuat database `sicabe_data` + tabel `public.orders` (8 kolom, konsisten
# dengan dbt sources.yml dan tabel Iceberg `bronze.orders`) lalu mengisinya
# dengan 6 baris contoh.
#
# Tabel ini adalah SUMBER untuk tugas DAG `sicabe_batch_daily.ingest_to_bronze`
# (Sling: POSTGRES_DB.public.orders → ICEBERG bronze.orders).
#
# CATATAN: `scripts/seed-sample-data.sh` mengisi sisi Iceberg (bronze.orders)
# secara langsung — script INI mengisi sisi PostgreSQL (sumbernya), sehingga
# jalur Sling → Iceberg → dbt → GE dapat berjalan end-to-end.
#
# Idempoten: aman dijalankan berulang (CREATE ... IF NOT EXISTS; INSERT hanya
# bila tabel kosong).
#
# Pemakaian:
# cd deploy
# ./scripts/seed-source-postgres.sh
#
# Env opsional (ada default):
# PG_CONTAINER (sicabe-postgres), SEED_SOURCE_DB (sicabe_data),
# SEED_SOURCE_TABLE (orders), POSTGRES_USER (sicabe_admin)
# =============================================================================
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
# shellcheck source=../lib/common.sh
source "${SCRIPT_DIR}/lib/common.sh"
# shellcheck source=../config/environment.defaults.sh
source "${SCRIPT_DIR}/config/environment.defaults.sh"
sicabe_load_env "${SCRIPT_DIR}" "$@"
PG_CONTAINER="${PG_CONTAINER:-sicabe-postgres}"
SEED_DB="${SEED_SOURCE_DB:-sicabe_data}"
SEED_TABLE="${SEED_SOURCE_TABLE:-orders}"
PG_USER="${POSTGRES_USER:-sicabe_admin}"
# Multi-host: PostgreSQL ada di host `state` dan `docker exec` bersifat lokal →
# bila bukan di host ini, beri instruksi & keluar rapi (kode 3), bukan gagal.
sicabe_require_local_container "${PG_CONTAINER}" "PostgreSQL" "state" || exit $?
on_err() { echo -e "${ERR} Gagal pada baris $1."; exit 1; }
trap 'on_err $LINENO' ERR
# psql di dalam container (tanpa password: koneksi lokal container).
pg() { docker exec -i "${PG_CONTAINER}" psql -v ON_ERROR_STOP=1 -qtA -U "${PG_USER}" "$@"; }
echo -e "${BOLD}${BLUE}=== Seed Sumber PostgreSQL: ${SEED_DB}.public.${SEED_TABLE} ===${NC}"
# ─── [1/3] Database ─────────────────────────────────────────────────────────
echo -e "${INFO} [1/3] Database '${SEED_DB}'..."
if [[ "$(pg -d postgres -c "SELECT 1 FROM pg_database WHERE datname='${SEED_DB}'")" == "1" ]]; then
echo -e "${OK} Database '${SEED_DB}' sudah ada."
else
pg -d postgres -c "CREATE DATABASE ${SEED_DB} OWNER ${PG_USER}" >/dev/null
echo -e "${OK} Database '${SEED_DB}' dibuat."
fi
# ─── [2/3] Tabel sumber ─────────────────────────────────────────────────────
echo -e "${INFO} [2/3] Tabel 'public.${SEED_TABLE}'..."
pg -d "${SEED_DB}" -c "CREATE TABLE IF NOT EXISTS public.${SEED_TABLE} (
id bigint PRIMARY KEY,
order_date date,
komoditas varchar(64),
harga_per_kg double precision,
stok_kg bigint,
region varchar(64),
created_at timestamp,
updated_at timestamp
)" >/dev/null
echo -e "${OK} Tabel 'public.${SEED_TABLE}' siap."
# ─── [3/3] Data contoh ──────────────────────────────────────────────────────
echo -e "${INFO} [3/3] Data contoh (bila kosong)..."
COUNT="$(pg -d "${SEED_DB}" -c "SELECT count(*) FROM public.${SEED_TABLE}")"
if [[ "${COUNT}" -gt 0 ]]; then
echo -e "${OK} Sudah berisi ${COUNT} baris (skip seed)."
else
pg -d "${SEED_DB}" -c "INSERT INTO public.${SEED_TABLE} VALUES
(1, DATE '2026-09-01', 'beras', 12000.0, 500, 'jakarta', TIMESTAMP '2026-09-01 08:00:00', TIMESTAMP '2026-09-01 08:00:00'),
(2, DATE '2026-09-01', 'gula', 15000.0, 300, 'bandung', TIMESTAMP '2026-09-01 09:00:00', TIMESTAMP '2026-09-01 09:00:00'),
(3, DATE '2026-09-02', 'minyak', 18000.0, 200, 'surabaya', TIMESTAMP '2026-09-02 08:00:00', TIMESTAMP '2026-09-02 08:00:00'),
(4, DATE '2026-09-02', 'beras', 12500.0, 450, 'jakarta', TIMESTAMP '2026-09-02 10:00:00', TIMESTAMP '2026-09-02 10:00:00'),
(5, DATE '2026-09-03', 'tepung', 11000.0, 350, 'medan', TIMESTAMP '2026-09-03 08:00:00', TIMESTAMP '2026-09-03 08:00:00'),
(6, DATE '2026-09-03', 'gula', 15500.0, 250, 'bandung', TIMESTAMP '2026-09-03 11:00:00', TIMESTAMP '2026-09-03 11:00:00')" >/dev/null
echo -e "${OK} 6 baris contoh dimasukkan."
fi
# ─── Verifikasi ─────────────────────────────────────────────────────────────
echo -e "${INFO} Verifikasi (sumber PostgreSQL):"
pg -d "${SEED_DB}" -c "SELECT id, order_date, komoditas, harga_per_kg, stok_kg, region
FROM public.${SEED_TABLE} ORDER BY id" \
| while IFS='|' read -r a b c d e f; do printf ' %-3s %-12s %-8s %-10s %-6s %s\n' "$a" "$b" "$c" "$d" "$e" "$f"; done
echo -e "${OK} Selesai. Lanjutkan dengan DAG 'sicabe_batch_daily' (task ingest_to_bronze)."
deploy/scripts/seed-starrocks.sh
Executable
+175
View File
@@ -0,0 +1,175 @@
#!/usr/bin/env bash
# =============================================================================
# eBdesk SiCABE — Seed StarRocks: External Catalog (Nessie) + Native Shared Tables
# =============================================================================
# Idempoten:
# 1. Pastikan external catalog 'iceberg_catalog' → Nessie (query Iceberg).
# 2. Buat database native 'sicabe' di default_catalog.
# 3. Buat tabel native shared-data 'sicabe.orders' + 'sicabe.products'
# (otomatis CLOUD_NATIVE via default storage volume 's3_volume' → MinIO).
# 4. Seed data contoh (hanya bila kosong).
# 5. GRANT object privileges ke role (RBAC internal — tanpa ini user OIDC
# bisa login tapi tabel tidak terlihat).
#
# Pemakaian:
# cd deploy
# set -a; source config/environment.defaults.sh; source .env; set +a
# ./scripts/seed-starrocks.sh
#
# Env (opsional):
# STARROCKS_FE_CONTAINER (sicabe-starrocks-fe), STARROCKS_ROOT_PASSWORD,
# SEED_DB (sicabe)
# =============================================================================
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
cd "${SCRIPT_DIR}"
source "${SCRIPT_DIR}/lib/common.sh"
source "${SCRIPT_DIR}/config/environment.defaults.sh"
sicabe_load_env "${SCRIPT_DIR}" "$@"
SR_CONTAINER="${STARROCKS_FE_CONTAINER:-sicabe-starrocks-fe}"
SR_PASSWORD="${STARROCKS_ROOT_PASSWORD:?set STARROCKS_ROOT_PASSWORD}"
SEED_DB="${SEED_DB:-sicabe}"
# Multi-host: StarRocks FE ada di host `compute`; `docker exec` bersifat lokal.
sicabe_require_local_container "${SR_CONTAINER}" "StarRocks FE" "compute" || exit $?
# Jalankan SQL via mysql client di dalam container FE (selalu tersedia di image).
sr_sql() {
docker exec "${SR_CONTAINER}" mysql -h 127.0.0.1 -P 9030 -u root -p"${SR_PASSWORD}" -N -e "$1" 2>/dev/null
}
# ─── 1. External catalog Nessie (idempoten) ─────────────────────────────────
ensure_external_catalog() {
echo -e "${INFO} [1/5] Pastikan external catalog 'iceberg_catalog' → Nessie..."
if sr_sql "SHOW CATALOGS;" | grep -q "iceberg_catalog"; then
echo -e "${OK} Catalog 'iceberg_catalog' sudah ada."
return 0
fi
sr_sql "CREATE EXTERNAL CATALOG iceberg_catalog PROPERTIES (
'type' = 'iceberg',
'iceberg.catalog.type' = 'rest',
'iceberg.catalog.uri' = '${NESSIE_URL:-http://nessie:19120}/iceberg/',
'iceberg.catalog.warehouse' = 'warehouse',
'iceberg.catalog.rest.nested-namespace-enabled' = 'true',
'aws.s3.endpoint' = '${S3_ENDPOINT:-http://minio:9000}',
'aws.s3.access_key' = '${S3_ACCESS_KEY:-${MINIO_ROOT_USER:-minioadmin}}',
'aws.s3.secret_key' = '${S3_SECRET_KEY:-${MINIO_ROOT_PASSWORD}}',
'aws.s3.enable_path_style_access' = 'true',
'aws.s3.region' = '${S3_REGION:-us-east-1}'
);" && echo -e "${OK} Catalog 'iceberg_catalog' dibuat."
}
# ─── 2. Database native ─────────────────────────────────────────────────────
ensure_native_db() {
echo -e "${INFO} [2/5] Database native '${SEED_DB}'..."
sr_sql "CREATE DATABASE IF NOT EXISTS ${SEED_DB};" \
&& echo -e "${OK} Database '${SEED_DB}' siap."
}
# ─── 3. Tabel native shared-data (→ MinIO via default storage volume) ───────
ensure_native_tables() {
echo -e "${INFO} [3/5] Tabel native shared-data (→ MinIO via 's3_volume')..."
sr_sql "CREATE TABLE IF NOT EXISTS ${SEED_DB}.orders (
id INT NOT NULL,
item VARCHAR(50),
customer VARCHAR(50),
qty INT,
price DECIMAL(10,2),
ts DATETIME
) DUPLICATE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 3
PROPERTIES (\"replication_num\" = \"1\");" \
&& echo -e "${OK} Tabel '${SEED_DB}.orders' siap."
sr_sql "CREATE TABLE IF NOT EXISTS ${SEED_DB}.products (
id INT NOT NULL,
name VARCHAR(50),
category VARCHAR(50),
price DECIMAL(10,2)
) DUPLICATE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 3
PROPERTIES (\"replication_num\" = \"1\");" \
&& echo -e "${OK} Tabel '${SEED_DB}.products' siap."
}
# ─── 4. Seed data (hanya bila kosong) ───────────────────────────────────────
seed_rows() {
echo -e "${INFO} [4/5] Seed data contoh (bila kosong)..."
local cnt
cnt=$(sr_sql "SELECT count(*) FROM ${SEED_DB}.orders;" || true); cnt="${cnt:-0}"
if [[ "${cnt}" -gt 0 ]]; then
echo -e "${OK} '${SEED_DB}.orders' sudah berisi ${cnt} baris (skip)."
else
sr_sql "INSERT INTO ${SEED_DB}.orders VALUES
(1,'laptop','alice',2,1500.00,'2026-09-01 09:00:00'),
(2,'monitor','budi',1,300.50,'2026-09-02 10:30:00'),
(3,'keyboard','citra',3,75.25,'2026-09-03 11:00:00');" \
&& echo -e "${OK} 3 baris → '${SEED_DB}.orders'."
fi
local pcnt
pcnt=$(sr_sql "SELECT count(*) FROM ${SEED_DB}.products;" || true); pcnt="${pcnt:-0}"
if [[ "${pcnt}" -gt 0 ]]; then
echo -e "${OK} '${SEED_DB}.products' sudah berisi ${pcnt} baris (skip)."
else
sr_sql "INSERT INTO ${SEED_DB}.products VALUES
(1,'laptop','elektronik',1500.00),
(2,'monitor','elektronik',300.50),
(3,'keyboard','aksesoris',75.25),
(4,'kursi','furnitur',450.00);" \
&& echo -e "${OK} 4 baris → '${SEED_DB}.products'."
fi
}
# ─── 5. GRANT object privileges ke role (RBAC internal) ─────────────────────
# Role di-map dari grup eksternal (bootstrap Step 5: GRANT role TO EXTERNAL
# GROUP). Tanpa object privilege, user OIDC bisa login tapi tidak melihat tabel.
grant_roles() {
echo -e "${INFO} [5/5] GRANT object privileges ke role (RBAC internal)..."
local role
for role in sr_admin sr_engineer sr_analyst sr_viewer; do
sr_sql "CREATE ROLE ${role};" >/dev/null 2>&1 || true
done
sr_sql "GRANT SELECT ON ALL TABLES IN DATABASE ${SEED_DB} TO ROLE sr_analyst;" >/dev/null 2>&1 \
&& echo -e "${OK} sr_analyst: SELECT ${SEED_DB}.*"
sr_sql "GRANT SELECT,INSERT,UPDATE,DELETE ON ALL TABLES IN DATABASE ${SEED_DB} TO ROLE sr_engineer;" >/dev/null 2>&1 \
&& echo -e "${OK} sr_engineer: RW ${SEED_DB}.*"
sr_sql "GRANT SELECT,INSERT,UPDATE,DELETE,ALTER,DROP ON ALL TABLES IN ALL DATABASES TO ROLE sr_admin;" >/dev/null 2>&1 \
&& echo -e "${OK} sr_admin: ALL (semua database)"
echo -e "${INFO} sr_viewer: tanpa grant (fail-closed)"
}
# ─── Verifikasi ─────────────────────────────────────────────────────────────
verify() {
echo -e "${INFO} Verifikasi:"
echo " Catalogs:"
sr_sql "SHOW CATALOGS;" | sed 's/^/ /'
echo " Native tables (default_catalog.${SEED_DB}):"
sr_sql "SHOW TABLES FROM ${SEED_DB};" | sed 's/^/ /'
echo " ${SEED_DB}.orders:"
sr_sql "SELECT id,item,customer,qty,price FROM ${SEED_DB}.orders ORDER BY id;" | sed 's/^/ /'
echo " External (Nessie) iceberg_catalog.bronze.orders:"
sr_sql "SELECT id,komoditas,harga_per_kg FROM iceberg_catalog.bronze.orders ORDER BY id;" | sed 's/^/ /'
}
# ─── MAIN ───────────────────────────────────────────────────────────────────
main() {
echo -e "${BOLD}${BLUE}═══ Seed StarRocks — catalog Nessie + tabel native shared (${SEED_DB}) ═══${NC}"
if ! docker ps --format '{{.Names}}' | grep -q "^${SR_CONTAINER}$"; then
echo -e "${ERR} Container '${SR_CONTAINER}' tidak running."
exit 1
fi
ensure_external_catalog
ensure_native_db
ensure_native_tables
seed_rows
grant_roles
verify
echo -e "${OK} ${GREEN}Seed StarRocks selesai.${NC}"
}
main "$@"
deploy/scripts/starrocks-reapply-dynamic-config.sh
Executable
+135
View File
@@ -0,0 +1,135 @@
#!/usr/bin/env bash
# =============================================================================
# eBdesk SiCABE — StarRocks: terapkan DYNAMIC CONFIG (hilang saat FE restart)
# =============================================================================
# Mengapa terpisah dan BUKAN di fe.conf?
# `authentication_chain` harus memuat `sr_oidc`, tetapi integrasi `sr_oidc`
# baru dibuat lewat SQL (bootstrap step 5). Pada boot pertama FE, integrasi itu
# BELUM ada sehingga menaruhnya di fe.conf akan gagal. Nilainya karena itu
# bersifat runtime: harus diterapkan ULANG setiap kali FE restart.
#
# Yang diterapkan:
# 1. authentication_chain = native,<sr_oidc> → tanpa ini login OIDC mati
# (integrasi tetap "ada" di SHOW SECURITY INTEGRATIONS, jadi sulit terlihat).
# 2. activate_all_roles_on_login = TRUE → role otomatis aktif saat login.
#
# Skrip ini:
# - idempoten (aman dipanggil berkali-kali),
# - memverifikasi lebih dulu bahwa security integration ada (kalau tidak,
# berhenti dengan pesan agar menjalankan bootstrap step 5),
# - memverifikasi hasilnya setelah diterapkan (exit != 0 bila gagal).
#
# Usage:
# ./scripts/starrocks-reapply-dynamic-config.sh
# # StarRocks existing:
# STARROCKS_HOST=starrocks.corp STARROCKS_PORT=9030 \
# STARROCKS_ROOT_PASSWORD=... ./scripts/starrocks-reapply-dynamic-config.sh
# =============================================================================
set -uo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
# shellcheck source=../lib/common.sh
source "${SCRIPT_DIR}/lib/common.sh"
# shellcheck source=../config/environment.defaults.sh
source "${SCRIPT_DIR}/config/environment.defaults.sh"
sicabe_load_env "${SCRIPT_DIR}" "$@"
STARROCKS_HOST="${STARROCKS_HOST:-localhost}"
STARROCKS_PORT="${STARROCKS_PORT:-${STARROCKS_FE_PORT:-12930}}"
# Multi-host: arahkan ke host pemilik StarRocks FE (single-host tetap localhost).
STARROCKS_HOST="$(sicabe_local_host STARROCKS_FE_HOST STARROCKS_HOST)"
SR_USER="${STARROCKS_USER:-root}"
SR_PW="${STARROCKS_ROOT_PASSWORD:-}"
OIDC_NAME="${STARROCKS_OIDC_INTEGRATION:-sr_oidc}"
SR_MYSQL_IMAGE="${SR_MYSQL_IMAGE:-mysql:8}"
if command -v mysql >/dev/null 2>&1; then
SR_MODE="host"
elif docker image inspect "${SR_MYSQL_IMAGE}" >/dev/null 2>&1; then
SR_MODE="container"
else
SR_MODE="none"
fi
# sr_sql <sql> → hasil; kode keluar bukan-0 bila koneksi/kliens gagal.
sr_sql() {
case "${SR_MODE}" in
host)
MYSQL_PWD="${SR_PW}" mysql --protocol=TCP -h "${STARROCKS_HOST}" -P "${STARROCKS_PORT}" \
-u "${SR_USER}" -N -B -e "$1" 2>/dev/null
;;
container)
docker run --rm -i --network "${SICABE_NETWORK:-sicabe-network}" \
-e MYSQL_PWD="${SR_PW}" "${SR_MYSQL_IMAGE}" \
mysql --protocol=TCP -h "${STARROCKS_HOST}" -P "${STARROCKS_PORT}" -u "${SR_USER}" \
-N -B -e "$1" 2>/dev/null
;;
*) return 1 ;;
esac
}
apply_dynamic_config() {
echo -e "${INFO} [1/2] authentication_chain = native,${OIDC_NAME}"
sr_sql "ADMIN SET FRONTEND CONFIG (\"authentication_chain\" = \"native,${OIDC_NAME}\");" >/dev/null
echo -e "${INFO} [2/2] activate_all_roles_on_login = TRUE"
sr_sql "SET GLOBAL activate_all_roles_on_login = TRUE;" >/dev/null
}
main() {
print_section "StarRocks — terapkan dynamic config (auth chain OIDC)"
if [[ "${SR_MODE}" == "none" ]]; then
echo -e "${ERR} Klien MySQL tidak tersedia di host dan image ${SR_MYSQL_IMAGE} belum ada."
echo " Install klien: sudo apt-get install -y mysql-client (atau tarik image mysql)"
exit 1
fi
if [[ -z "${SR_PW}" ]]; then
echo -e "${ERR} STARROCKS_ROOT_PASSWORD kosong (isi di .env)."
exit 1
fi
if [[ "$(sr_sql 'SELECT 1')" != "1" ]]; then
echo -e "${ERR} Tidak bisa terhubung ke StarRocks di ${STARROCKS_HOST}:${STARROCKS_PORT}."
exit 1
fi
echo -e "${OK} Terhubung ke ${STARROCKS_HOST}:${STARROCKS_PORT}"
# Prasyarat: integrasi harus sudah ada — kalau tidak, menyetel chain akan error.
if ! sr_sql "SHOW SECURITY INTEGRATIONS;" | grep -q "${OIDC_NAME}"; then
echo -e "${ERR} Security integration '${OIDC_NAME}' belum ada."
echo " Jalankan dulu: ./bootstrap.sh --step 5"
exit 1
fi
echo -e "${OK} Security integration '${OIDC_NAME}' ada (prasyarat terpenuhi)"
apply_dynamic_config
# Verifikasi hasil — jangan lapor sukses tanpa bukti.
local chain roles
chain="$(sr_sql "ADMIN SHOW FRONTEND CONFIG LIKE 'authentication_chain';" | awk -F'\t' '{print $3}')"
roles="$(sr_sql "SHOW VARIABLES LIKE 'activate_all_roles_on_login';" | awk -F'\t' '{print $2}')"
local rc=0
if [[ "${chain}" == *"${OIDC_NAME}"* ]]; then
echo -e "${OK} authentication_chain = ${chain}"
else
echo -e "${ERR} authentication_chain = ${chain:-tak terbaca} — gagal memuat ${OIDC_NAME}"
rc=1
fi
if [[ "${roles}" == "true" ]]; then
echo -e "${OK} activate_all_roles_on_login = ${roles}"
else
echo -e "${ERR} activate_all_roles_on_login = ${roles:-tak terbaca} (diharapkan true)"
rc=1
fi
if [[ ${rc} -eq 0 ]]; then
echo -e "${OK} ${GREEN}Dynamic config diterapkan.${NC}"
echo -e "${INFO} Verifikasi menyeluruh: ./scripts/verify-phase4-serving.sh"
else
echo -e "${ERR} Sebagian dynamic config gagal diterapkan."
fi
exit ${rc}
}
main "$@"
deploy/scripts/test-bootstrap-host.sh
Executable
+168
View File
@@ -0,0 +1,168 @@
#!/usr/bin/env bash
# =============================================================================
# test-bootstrap-host.sh — penjagaan untuk `bootstrap.sh --host` (Fase 2b/2c)
# -----------------------------------------------------------------------------
# Latar: `generate_runtime_credentials()` menulis TIGA berkas ke `config/` BERSAMA.
# Dalam mode `--host`, endpoint-nya milik host itu, sehingga berkas bersama tertimpa
# alamat host lain — dan itu TIDAK terlihat di `git status` karena
# `config/airflow/sling/env.yaml` memang gitignored. Dampaknya: deploy single-host
# yang sedang berjalan menunjuk alamat yang tidak ada.
#
# Uji ini punya dua lapis:
# A. STATIS — membaca bootstrap.sh dan memastikan ketiga penulisan memakai
# `${cfg_base}` (bukan `${SCRIPT_DIR}/config` langsung). Tidak butuh
# apa pun, selalu jalan.
# B. DINAMIS — bila stack SiCABE sedang berjalan: hash ketiga berkas bersama,
# jalankan `bootstrap.sh --host state` (hanya step 2 = pastikan bucket,
# idempoten), lalu pastikan hash TIDAK berubah.
#
# Jalankan: ./scripts/test-bootstrap-host.sh
# =============================================================================
set -uo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
DEPLOY_DIR="$(cd "${SCRIPT_DIR}/.." && pwd)"
GRN=$'\e[32m'; RED=$'\e[31m'; YLW=$'\e[33m'; BLD=$'\e[1m'; NC=$'\e[0m'
PASS_N=0; FAIL_N=0
ok() { printf " ${GRN}✅${NC} %s\n" "$1"; PASS_N=$((PASS_N + 1)); }
bad() { printf " ${RED}❌${NC} %s\n" "$1"; FAIL_N=$((FAIL_N + 1)); }
inf() { printf " ${YLW}[..]${NC} %s\n" "$1"; }
SHARED_FILES=(
"config/airflow/sling/env.yaml"
"config/airflow/simple_auth_manager_passwords.json"
"config/trino/password.db"
)
echo -e "${BLD}Uji bootstrap.sh --host${NC}"
echo
echo "── A. penjagaan statis (tanpa dependensi) ──"
# Ekstrak badan fungsi generate_runtime_credentials.
body="$(awk '/^generate_runtime_credentials\(\) \{/{f=1} f{print} f&&/^\}/{exit}' "${DEPLOY_DIR}/bootstrap.sh")"
if [[ -z "${body}" ]]; then
bad "fungsi generate_runtime_credentials() tidak ditemukan"
else
ok "fungsi generate_runtime_credentials() ditemukan"
for target in 'airflow/sling' 'airflow/simple_auth_manager_passwords.json' 'trino/password.db'; do
if grep -q 'local .*="\${cfg_base}/'"${target}"'"' <<<"${body}"; then
ok "penulisan ${target} memakai \${cfg_base}"
else
bad "penulisan ${target} TIDAK memakai \${cfg_base} (rawan menimpa config bersama)"
fi
done
if grep -q 'HOST_NAME' <<<"${body}" && grep -q 'generated/host-' <<<"${body}"; then
ok "cfg_base dialihkan ke bundle host saat HOST_NAME diisi"
else
bad "cfg_base tidak dialihkan berdasarkan HOST_NAME"
fi
# Tidak boleh ada penulisan langsung ke ${SCRIPT_DIR}/config (selain default cfg_base).
if grep -nE '(cat >|: >|>>) *"\$\{SCRIPT_DIR\}/config' <<<"${body}"; then
bad "masih ada penulisan langsung ke \${SCRIPT_DIR}/config"
else
ok "tidak ada penulisan langsung ke \${SCRIPT_DIR}/config"
fi
fi
# step_tiers harus memetakan seluruh step 1..9. Bentuk pattern bisa berkelompok
# (mis. `1|3)`, `4|5)`, `7|8)`), jadi diambil hanya bagian sebelum ')' lalu dipecah
# pada '|' — regex "baris diawali angka" akan salah untuk bentuk berkelompok.
patterns="$(awk '/^step_tiers\(\) \{/{f=1} f{print} f&&/^\}/{exit}' "${DEPLOY_DIR}/bootstrap.sh" \
| grep -E '\)' | sed 's/).*//' | tr '|' '\n' | tr -d ' ')"
missing=""
for s in 1 2 3 4 5 6 7 8 9; do
grep -qx "${s}" <<<"${patterns}" || missing+=" ${s}"
done
if [[ -z "${missing}" ]]; then
ok "step_tiers() memetakan step 1..9"
else
bad "step_tiers() belum memetakan step:${missing}"
fi
echo
echo "── B. penjagaan dinamis (butuh stack SiCABE berjalan) ──"
if ! docker ps --format '{{.Names}}' 2>/dev/null | grep -q '^sicabe-'; then
inf "tidak ada container sicabe- yang berjalan — lapis dinamis DILEWATI"
echo
printf " ${YLW}Ringkasan:${NC} %s PASS · %s FAIL (lapis dinamis dilewati)\n" "${PASS_N}" "${FAIL_N}"
[[ ${FAIL_N} -eq 0 ]] && exit 0 || exit 1
fi
GEN_DIR="${DEPLOY_DIR}/generated"
created_gen=0
if [[ ! -f "${GEN_DIR}/host-state/.env" ]]; then
inf "bundle host belum ada — generate dulu (sementara, akan dihapus)"
if python3 "${DEPLOY_DIR}/lib/cluster/generate.py" render \
"${DEPLOY_DIR}/cluster.example.yml" "${DEPLOY_DIR}/docker-compose.yml" \
"${DEPLOY_DIR}/.env" "${GEN_DIR}" "${DEPLOY_DIR}/scripts/render-config.sh" \
>/dev/null 2>&1; then
created_gen=1
else
bad "gagal menyiapkan bundle host untuk uji dinamis"
GEN_DIR=""
fi
fi
if [[ -n "${GEN_DIR}" ]]; then
declare -A before=()
for f in "${SHARED_FILES[@]}"; do
[[ -f "${DEPLOY_DIR}/${f}" ]] && before["${f}"]="$(sha256sum "${DEPLOY_DIR}/${f}" | cut -c1-16)"
done
inf "menjalankan: bootstrap.sh --host state (hanya step 2 — pastikan bucket)"
"${DEPLOY_DIR}/bootstrap.sh" --host state >/tmp/test-bootstrap-host.log 2>&1 || true
touched=0
for f in "${SHARED_FILES[@]}"; do
[[ -z "${before[${f}]:-}" ]] && continue
now="$(sha256sum "${DEPLOY_DIR}/${f}" | cut -c1-16)"
if [[ "${before[${f}]}" != "${now}" ]]; then
bad "${f} BERUBAH setelah --host state (bug config bersama kembali)"
touched=1
fi
done
[[ ${touched} -eq 0 ]] && ok "ketiga berkas config bersama tidak berubah (sha256 identik)"
# Kredensial host harus mendarat di bundle host.
if [[ -f "${GEN_DIR}/host-state/config/airflow/sling/env.yaml" ]]; then
ok "kredensial runtime ditulis ke bundle host (generated/host-state/config/...)"
else
bad "kredensial runtime TIDAK ditemukan di bundle host"
fi
fi
[[ ${created_gen} -eq 1 ]] && rm -rf "${GEN_DIR}" && inf "artefak uji generated/ dihapus"
rm -f /tmp/test-bootstrap-host.log
# ── install.sh --host: profil `edge` ikut saat bundle memuat traefik (T-20) ─────
# Traefik berada di profile `edge`. Bila run_host hanya memakai `--profile all`,
# Traefik tidak pernah start sehingga SELURUH rute publik tidak ada — padahal label
# sudah terpasang di setiap layanan dan sertifikat sudah bisa diterbitkan.
run_host_body="$(sed -n '/^run_host()/,/^}/p' "${DEPLOY_DIR}/install.sh")"
if grep -q 'traefik:' <<<"${run_host_body}" && grep -q 'profile edge' <<<"${run_host_body}"; then
ok "run_host menyertakan profile edge bila bundle memuat traefik"
else
bad "run_host tidak mendeteksi profile edge (Traefik tidak akan pernah start)"
fi
# ── install.sh: nama LOGIS harus diuraikan di EXTERNAL_SERVICES/LOCAL_SERVICES ──
# Tanpa ini, `starrocks` (dan dulu hanya `trino` yang istimewa) diam-diam tidak
# berefek: filter mencocokkan nama SERVICE (`starrocks-fe`, `starrocks-cn`).
if grep -q 'expand_service_names()' "${DEPLOY_DIR}/install.sh" \
&& grep -q 'starrocks-fe starrocks-cn' "${DEPLOY_DIR}/install.sh" \
&& grep -q 'expand_service_names ${LOCAL_SERVICES' "${DEPLOY_DIR}/install.sh" \
&& grep -q 'expand_service_names ${EXTERNAL_SERVICES' "${DEPLOY_DIR}/install.sh"; then
ok "nama logis diuraikan di EXTERNAL_SERVICES & LOCAL_SERVICES (starrocks/trino/…)"
else
bad "pemetaan nama logis hilang — setelan seperti LOCAL_SERVICES=starrocks akan diabaikan"
fi
echo
if [[ ${FAIL_N} -eq 0 ]]; then
echo -e " ${GRN}${BLD}SEMUA LULUS${NC} (${PASS_N} cek)"
exit 0
fi
echo -e " ${RED}${BLD}GAGAL${NC}: ${FAIL_N} masalah · ${PASS_N} lulus"
exit 1
deploy/scripts/test-generate.sh
Executable
+814
View File
@@ -0,0 +1,814 @@
#!/usr/bin/env bash
# =============================================================================
# test-generate.sh — verifikasi generator multi-host (Fase 1 R10 §4.5)
# -----------------------------------------------------------------------------
# Yang dibuktikan (bukan sekadar "berkasnya ada"):
# 1. `docker compose config` VALID untuk setiap host hasil generate.
# 2. Service yang ikut **identik secara semantik** dengan single-host pada semua
# field (image, command, ports, healthcheck, profiles, volumes, ...) — ini yang
# menangkap kerusakan akibat round-trip YAML (mis. `"true"` jadi boolean).
# 3. Perbedaan `environment` HANYA pada variabel topologi yang memang diarahkan
# generator (alamat host) — tidak ada kebocoran lain.
# 4. `depends_on` lintas host benar-benar HABIS; intra-host tetap.
# 5. Tidak ada variabel topologi yang DUPLIKAT di `.env` (dotenv: yang terakhir
# menang → duplikat = nilai inventory bisa hilang diam-diam).
#
# Jalankan: ./scripts/test-generate.sh [cluster.yml]
# =============================================================================
set -uo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
DEPLOY_DIR="$(cd "${SCRIPT_DIR}/.." && pwd)"
INVENTORY="${1:-${DEPLOY_DIR}/cluster.example.yml}"
OUT="$(mktemp -d)"
trap 'rm -rf "${OUT}"' EXIT
GRN=$'\e[32m'; RED=$'\e[31m'; BLD=$'\e[1m'; NC=$'\e[0m'
PASS_N=0; FAIL_N=0
ok() { printf " ${GRN}✅${NC} %s\n" "$1"; PASS_N=$((PASS_N + 1)); }
bad() { printf " ${RED}❌${NC} %s\n" "$1"; FAIL_N=$((FAIL_N + 1)); }
echo -e "${BLD}Uji generator multi-host (lib/cluster/generate.py)${NC}"
echo " inventory: ${INVENTORY}"
echo
# ─── 0. konsistensi RENDER_HOSTS ⊆ KNOWN_HOSTS ──────────────────────────────
# generate.py meng-`--map` SETIAP entri RENDER_HOSTS, sedangkan render-config.sh
# hanya menerima hostname di KNOWN_HOSTS. Bila berbeda, `install.sh --generate`
# gagal total (bug M2/M3: statsd-exporter/trino-jmx-exporter). Dijaga di sini agar
# tidak terulang diam-diam.
map_gap="$(python3 - "${DEPLOY_DIR}" <<'PY'
import re, sys
d = sys.argv[1]
gen = open(f"{d}/lib/cluster/generate.py", encoding="utf-8").read()
rc = open(f"{d}/scripts/render-config.sh", encoding="utf-8").read()
render_hosts = set(re.findall(r'"([^"]+)"', re.search(r"RENDER_HOSTS = \((.*?)\)", gen, re.S).group(1)))
# Blok KNOWN_HOSTS berakhir pada baris ")" tersendiri; komentar dibuang sebelum split
# (komentar boleh memuat tanda kurung, mis. "(M2/M3)", yang merusak regex sederhana).
block = re.search(r"KNOWN_HOSTS=\((.*?)\n\)", rc, re.S).group(1)
known = set()
for line in block.splitlines():
known.update(line.split("#", 1)[0].split())
print(" ".join(sorted(render_hosts - known)))
PY
)"
if [[ -z "${map_gap}" ]]; then
ok "RENDER_HOSTS ⊆ KNOWN_HOSTS (render-config menerima semua --map)"
else
bad "RENDER_HOSTS memuat hostname yang TIDAK ada di KNOWN_HOSTS: ${map_gap}"
fi
# ─── 0b. Trino group-provider memakai LDAP_ADMIN_DN (bukan DN literal) ───────
# LDAP existing bisa punya bind DN ≠ cn=admin,dc=sicabe,dc=local. Bila
# group-provider.properties menulis DN literal, Trino mem-bind dengan DN SALAH →
# user terautentikasi tetapi tanpa grup. Pastikan ia memakai ${ENV:LDAP_ADMIN_DN}
# DAN compose meneruskan env itu ke trino-coordinator (Trino gagal start bila tak ada).
gp="${DEPLOY_DIR}/config/trino/group-provider.properties"
if grep -q 'ldap.admin-user=${ENV:LDAP_ADMIN_DN}' "${gp}" 2>/dev/null; then
ok "group-provider.properties: ldap.admin-user dari \${ENV:LDAP_ADMIN_DN}"
else
bad "group-provider.properties: ldap.admin-user BUKAN \${ENV:LDAP_ADMIN_DN} (bind DN existing tak akan jalan)"
fi
if awk '/^ trino-coordinator:/{f=1;next} f && /^ [a-zA-Z0-9_-]+:/{exit} f' \
"${DEPLOY_DIR}/docker-compose.yml" | grep -q 'LDAP_ADMIN_DN:'; then
ok "docker-compose: trino-coordinator menerima LDAP_ADMIN_DN"
else
bad "docker-compose: trino-coordinator TIDAK menerima LDAP_ADMIN_DN (Trino gagal start)"
fi
# ─── 0c. Port internal terparameter (bukan literal) ──────────────────────────
# Port container harus bisa di-override via .env (mis. layanan existon berport
# non-standar) tanpa menyunting docker-compose.yml.
dc="${DEPLOY_DIR}/docker-compose.yml"
miss=""
grep -q '${POSTGRES_INTERNAL_PORT:-5432}' "${dc}" || miss="${miss} POSTGRES"
grep -q '${KEYCLOAK_INTERNAL_PORT:-8080}' "${dc}" || miss="${miss} KEYCLOAK"
grep -q '${MINIO_INTERNAL_PORT:-9000}' "${dc}" || miss="${miss} MINIO"
[[ -z "${miss}" ]] && ok "docker-compose: port internal terparameter (POSTGRES/KEYCLOAK/MINIO)" \
|| bad "docker-compose: port internal BUKAN parameter:${miss}"
hard="$(grep -nE '\$\{(POSTGRES_HOST|KEYCLOAK_HOST|MINIO_HOST)[^}]*\}:(5432|8080|9000)' "${dc}" || true)"
[[ -z "${hard}" ]] && ok "docker-compose: tak ada port internal literal di var *_HOST" \
|| bad "docker-compose: masih ada port literal: ${hard}"
grep -q '${ENV:KEYCLOAK_INTERNAL_PORT}' "${DEPLOY_DIR}/config/trino/config-coordinator.properties" \
&& ok "Trino config: token/jwks-url pakai \${ENV:KEYCLOAK_INTERNAL_PORT}" \
|| bad "Trino config: port Keycloak masih literal 8080"
grep -q ':{port}' "${DEPLOY_DIR}/lib/cluster/generate.py" \
&& ok "generate.py: URL_ENDPOINT_VARS memakai {port} (dari .env)" \
|| bad "generate.py: URL_ENDPOINT_VARS masih port literal"
# ─── 0d. Image ref terparameter (bisa diarahkan ke mirror/registry internal) ─
# Repositori asal bisa mengubah kebijakan (mis. MinIO community butuh auth →
# pull anonim GAGAL). Var *_IMAGE (full ref) harus ada agar bisa di-mirror tanpa
# menyunting docker-compose.yml.
for pair in "MINIO_IMAGE:quay.io/minio/minio" "MINIO_MC_IMAGE:quay.io/minio/mc" "NESSIE_IMAGE:ghcr.io/projectnessie/nessie"; do
var="${pair%%:*}"; repo="${pair#*:}"
if grep -q "\${${var}:-${repo}" "${DEPLOY_DIR}/docker-compose.yml"; then
ok "docker-compose: image ${repo} lewat \${${var}}"
else
bad "docker-compose: image ${repo} masih hardcoded (tidak bisa di-mirror)"
fi
done
# ─── 1. generate ────────────────────────────────────────────────────────────
if ! python3 "${DEPLOY_DIR}/lib/cluster/generate.py" render \
"${INVENTORY}" "${DEPLOY_DIR}/docker-compose.yml" "${DEPLOY_DIR}/.env" \
"${OUT}" "${DEPLOY_DIR}/scripts/render-config.sh" > "${OUT}/generate.log" 2>&1; then
bad "generate gagal:"; sed 's/^/ /' "${OUT}/generate.log" | head -10; exit 1
fi
ok "generate berhasil ($(grep -c '^host-' "${OUT}/generate.log") host)"
HOSTS="$(grep '^host-' "${OUT}/generate.log" | awk '{print $1}' | sed 's/^host-//')"
# ─── 1b. target node_exporter (job `node`, file_sd) di bundle Prometheus ─────
# node_exporter = PRA-SYARAT HOST (di luar installer), jadi generate.py menulis
# daftar target SELURUH host dari cluster.yml ke bundle host pemilik Prometheus.
nodeyml="${OUT}/host-control/config/prometheus/targets/node.yml"
if [[ -f "${nodeyml}" ]]; then
nsum="$(python3 - "${nodeyml}" <<'PY'
import sys, yaml
data = yaml.safe_load(open(sys.argv[1], encoding="utf-8")) or []
hosts = sorted((e.get("labels") or {}).get("host", "") for e in data)
addrs = sorted(t for e in data for t in (e.get("targets") or []))
print(f"{len(hosts)}|{','.join(hosts)}|{','.join(addrs)}")
PY
)"
ncount="${nsum%%|*}"; rest="${nsum#*|}"; nhosts="${rest%%|*}"; naddrs="${rest##*|}"
exp_hosts="$(echo ${HOSTS} | tr ' ' '\n' | sort | paste -sd, -)"
nhost_total="$(echo ${HOSTS} | wc -w | tr -d ' ')"
if [[ "${ncount}" == "${nhost_total}" && "${nhosts}" == "${exp_hosts}" ]]; then
ok "node targets: ${ncount} host digenerate di bundle Prometheus (${nhosts})"
else
bad "node targets: ${ncount} host (harus ${nhost_total}); host=${nhosts} (harus ${exp_hosts})"
fi
if echo "${naddrs}" | tr ',' '\n' | grep -qx '10.20.0.12:9100'; then
ok 'node targets: format "<alamat>:9100" per host (contoh 10.20.0.12:9100)'
else
bad "node targets: format alamat:9100 salah (${naddrs})"
fi
else
bad "node targets: ${nodeyml} tidak digenerate"
fi
# ─── 2. compose valid per host + berkas runtime bind-mount berupa FILE ──────
for h in ${HOSTS}; do
d="${OUT}/host-${h}"
if docker compose -f "${d}/docker-compose.yml" --env-file "${d}/.env" --profile all config -q 2>"${OUT}/err-${h}"; then
ok "host-${h}: docker compose config valid"
else
bad "host-${h}: compose TIDAK valid"; head -3 "${OUT}/err-${h}" | sed 's/^/ /'
fi
# WAJIB FILE (bukan direktori): Docker membuat DIRECTORY bila sumber bind-mount
# hilang → Airflow SimpleAuthManager IsADirectoryError / Trino gagal auth.
for rf in config/airflow/simple_auth_manager_passwords.json config/trino/password.db; do
[[ -f "${d}/${rf}" ]] && ok "host-${h}: ${rf} berupa file" \
|| bad "host-${h}: ${rf} BUKAN file (bind-mount akan jadi DIRECTORY)"
done
done
# ─── 3. .env: tidak ada variabel topologi ganda ─────────────────────────────
TOPO='EXTERNAL_SERVICES|LOCAL_SERVICES|POSTGRES_HOST|MINIO_HOST|KEYCLOAK_HOST|OPENMETADATA_HOST|AIRFLOW_WEBSERVER_HOST|STARROCKS_FE_HOST|STARROCKS_CN_HOST|OPENSEARCH_HOST|S3_ENDPOINT|NESSIE_URL|LDAP_URL|OPA_URL'
for h in ${HOSTS}; do
dup="$(grep -E "^(${TOPO})=" "${OUT}/host-${h}/.env" | cut -d= -f1 | sort | uniq -d | paste -sd' ' -)"
[[ -z "${dup}" ]] && ok "host-${h}: .env tanpa variabel topologi ganda" \
|| bad "host-${h}: .env punya variabel ganda: ${dup}"
done
# ─── 4 & 5. perbandingan semantik dengan render single-host ─────────────────
docker compose -f "${DEPLOY_DIR}/docker-compose.yml" --env-file "${DEPLOY_DIR}/.env" \
--profile all config --format json > "${OUT}/base.json" 2>/dev/null
if [[ ! -s "${OUT}/base.json" ]]; then
bad "tidak bisa render single-host (base) sebagai pembanding"; exit 1
fi
for h in ${HOSTS}; do
d="${OUT}/host-${h}"
docker compose -f "${d}/docker-compose.yml" --env-file "${d}/.env" \
--profile all config --format json > "${OUT}/host-${h}.json" 2>/dev/null
done
PLAN_JSON="$(python3 "${DEPLOY_DIR}/lib/cluster/inventory.py" plan "${INVENTORY}")"
COMP_JSON="$(python3 "${DEPLOY_DIR}/lib/cluster/inventory.py" components "${INVENTORY}" | python3 -c "
import sys, json
d = {}
for line in sys.stdin:
name, _, hosts = line.strip().partition(' -> ')
name = name.strip() # keluaran CLI mem-pad nama (kolom 24)
if name: d[name] = hosts.strip().split(',')
print(json.dumps(d))")"
# ─── 3b. target Prometheus lintas-host → portnya HARUS diterbitkan ───────────
# Regresi nyata: prometheus.yml memakai `trino-jmx-exporter:9404` (→ IP host coord
# saat generate) tetapi 9404 TIDAK diterbitkan → job `trino` DOWN di multi-host.
# Guard: tiap target `static_configs` yang menunjuk ALAMAT HOST harus punya binding
# `<addr>:<port>:<port>` di bundle host pemilik. Target berupa NAMA container
# (mis. `statsd-exporter:9102`) harus SE-HOST dengan Prometheus (host-control).
pym="${OUT}/host-control/config/prometheus/prometheus.yml"
if [[ -f "${pym}" ]]; then
prom_issues="$(python3 - "${OUT}" "${PLAN_JSON}" <<'PY'
import json, sys, yaml
from pathlib import Path
out = Path(sys.argv[1]); plan = json.loads(sys.argv[2])
addr2host = {v.get("address"): h for h, v in plan.items()}
doc = yaml.safe_load((out / "host-control" / "config" / "prometheus" / "prometheus.yml").read_text()) or {}
def ports_of(hostdir):
comp = yaml.safe_load((out / hostdir / "docker-compose.yml").read_text()) or {}
found = set()
for svc in (comp.get("services") or {}).values():
for e in svc.get("ports") or []:
if isinstance(e, dict):
found.add(f"{e.get('host_ip','')}:{e.get('published')}:{e.get('target')}")
else:
found.add(str(e))
return found
issues = []
for job in doc.get("scrape_configs", []):
if "file_sd_configs" in job:
continue # node_exporter = proses di HOST (:9100), bukan container
for sc in job.get("static_configs", []) or []:
for t in sc.get("targets", []) or []:
if ":" not in t:
continue
a, p = t.rsplit(":", 1)
if a in ("localhost", "127.0.0.1"):
continue
if a in addr2host:
owner = addr2host[a]
if f"{a}:{p}:{p}" not in ports_of(f"host-{owner}"):
issues.append(f"job {job.get('job_name')}: {t} lintas-host tetapi {a}:{p} TIDAK diterbitkan di host-{owner}")
else:
ctrl = yaml.safe_load((out / "host-control" / "docker-compose.yml").read_text()) or {}
if a not in (ctrl.get("services") or {}):
issues.append(f"job {job.get('job_name')}: {t} bukan alamat host & bukan container di host-control")
print("\n".join(issues))
PY
)"
if [[ -z "${prom_issues}" ]]; then
ok "prometheus.yml: target lintas-host menunjuk port yang diterbitkan"
else
bad "prometheus.yml target bermasalah:"
echo "${prom_issues}" | sed 's/^/ /'
fi
else
bad "prometheus.yml tidak ada di bundle control: ${pym}"
fi
# ─── 3c. config TIDAK boleh menunjuk ALAMAT HOST SENDIRI pada port tak-terbit ─
# Kelas bug: komponen SE-HOST ditulis ke alamat host (mis. grafana→prometheus jadi
# `http://<ip>:9090`) padahal port container itu TIDAK diterbitkan di host → connection
# refused. Komponen se-host seharusnya tetap memakai NAMA container.
coloc_issues="$(python3 - "${OUT}" "${PLAN_JSON}" <<'PY'
import json, re, sys, yaml
from pathlib import Path
out = Path(sys.argv[1]); plan = json.loads(sys.argv[2])
issues = []
for host, spec in plan.items():
addr = spec.get("address")
hd = out / f"host-{host}"
compf = hd / "docker-compose.yml"
if not addr or not compf.is_file():
continue
comp = yaml.safe_load(compf.read_text()) or {}
published = set()
for svc in (comp.get("services") or {}).values():
for e in svc.get("ports") or []:
if isinstance(e, dict):
published.add(str(e.get("published")))
elif isinstance(e, str):
parts = e.split(":")
if len(parts) == 2:
published.add(parts[0])
elif len(parts) == 3:
published.add(parts[1])
cfg = hd / "config"
if not cfg.is_dir():
continue
for f in cfg.rglob("*"):
if not f.is_file():
continue
try:
text = f.read_text(encoding="utf-8", errors="ignore")
except OSError:
continue
# 9100 = node_exporter: proses di HOST (pra-syarat), BUKAN container → wajar tak
# ada di `ports:` compose. Port host-level lain bisa ditambah ke sini.
host_level_ports = {"9100"}
for m in re.finditer(re.escape(addr) + r":(\d+)", text):
port = m.group(1)
if port in host_level_ports:
continue
if port not in published:
issues.append(f"host-{host}: {f.relative_to(hd)} menunjuk alamat HOST SENDIRI {addr}:{port} (port tak diterbitkan)")
print("\n".join(sorted(set(issues))))
PY
)"
if [[ -z "${coloc_issues}" ]]; then
ok "config: tidak ada rujukan ke alamat host sendiri pada port tak-terbit"
else
bad "config menunjuk alamat host sendiri pada port tak-terbit:"
echo "${coloc_issues}" | sed 's/^/ /'
fi
sem_out="$(python3 - "${OUT}" "${HOSTS}" "${DEPLOY_DIR}" "${PLAN_JSON}" "${COMP_JSON}" <<'PY'
import json, re, sys
from pathlib import Path
out = Path(sys.argv[1]); hosts = sys.argv[2].split(); deploy = Path(sys.argv[3])
plan = json.loads(sys.argv[4]); comps = json.loads(sys.argv[5])
base = json.loads((out / "base.json").read_text())["services"]
COMPARE = ("image", "entrypoint", "user", "restart", "profiles",
"deploy", "networks", "ulimits", "cap_add")
def split_ports(ports, addr):
"""Pisahkan binding port internal multi-host (host_ip == alamat host) dari dasar.
Fase 2 menambahkan `alamat:cport:cport` HANYA di artefak multi-host (port container
yang harus terjangkau lintas host, mis. postgres 5432). Binding itu tidak ada di
compose dasar, jadi harus dipisahkan sebelum membandingkan — sekaligus diperiksa
bentuknya (host port == container port, dikunci ke alamat host).
"""
base, internal = [], []
for p in ports or []:
if isinstance(p, dict):
if p.get("host_ip") == addr:
internal.append(p)
else:
base.append(p)
elif isinstance(p, str) and p.startswith(addr + ":"):
internal.append(p)
else:
base.append(p)
return base, internal
def addr_of(host: str) -> str:
return plan.get(host, {}).get("address", "")
def name_addr(host: str) -> dict:
"""hostname container → alamat, dengan aturan yang sama seperti generate.py."""
out_map = {}
for name, owners in comps.items():
if not owners:
continue
owner = host if host in owners else owners[0]
if owner in plan: # `plan` di-key oleh NAMA HOST
out_map[name] = plan[owner]["address"]
return out_map
def normalize(value, mapping: dict):
"""Samakan nama container DAN alamat host menjadi satu placeholder.
Yang diuji: satu-satunya perbedaan nilai dasar vs per-host adalah **identitas
host** yang dipakai (nama container ↔ alamat host). Pendekatan "rekonstruksi
nilai yang diharapkan" dicoba lebih dulu dan rapuh — gagal pada bentuk
berkutip (`"starrocks-cn"`) dan salah menandai nilai `postgres` yang memang
dialek DB, bukan endpoint. Normalisasi dua arah menghindari keduanya.
"""
if isinstance(value, str):
s = value
for _name, addr in mapping.items():
s = s.replace(addr, "<HOST>")
for name in mapping:
s = re.sub(r"(?<![\w.-])" + re.escape(name) + r"(?![\w-])", "<HOST>", s)
return s
if isinstance(value, list):
return [normalize(v, mapping) for v in value]
if isinstance(value, dict):
return {k: normalize(v, mapping) for k, v in value.items()}
return value
# Kunci yang nilainya URL BROWSER-FACING. Nilainya memang berbeda antara single-host
# (localhost) dan multi-host (alamat host yang menjalankan komponen) — itu DISENGAJA,
# justru itu inti perbaikannya. Untuk kunci ini `localhost` diperlakukan sama seperti
# alamat host, sehingga yang masih terdeteksi sebagai masalah hanyalah perbedaan
# sebenarnya (skema, port, atau path berubah).
PUBLIC_URL_KEYS = (
"KC_HOSTNAME", "KC_HOSTNAME_URL",
"GF_SERVER_ROOT_URL", "AIRFLOW__WEBSERVER__BASE_URL",
"KEYCLOAK_PUBLIC_URL", "GRAFANA_PUBLIC_URL", "OPENMETADATA_PUBLIC_URL",
"AIRFLOW_PUBLIC_URL", "TRINO_PUBLIC_URL", "MINIO_CONSOLE_PUBLIC_URL",
# Konsumen browser-facing turunan URL publik (T-21). Disertakan supaya beda
# localhost (single-host) vs alamat host (multi-host) tidak dianggap regresi,
# sementara beda port/skema tetap tertangkap.
"OPENMETADATA_SERVER_URL", "MINIO_BROWSER_REDIRECT_URL", "MINIO_SERVER_URL",
"OIDC_SERVER_URL", "OIDC_CALLBACK", "AUTHENTICATION_CALLBACK_URL",
)
def normalize_env_key(key: str, value, mapping: dict):
s = normalize(value, mapping)
if isinstance(s, str) and key in PUBLIC_URL_KEYS:
s = re.sub(r"(?<![\w.-])localhost(?![\w-])", "<HOST>", s)
return s
def norm_volumes(vols, root: Path):
result = []
for v in vols or []:
item = dict(v)
src = item.get("source", "") or ""
prefix = str(root) + "/"
if src.startswith(prefix): src = src[len(prefix):]
item["source"] = src
result.append(item)
return result
issues, checked, changed_env, internal_count = [], 0, 0, 0
warn_skip = 0
for h in hosts:
host_root = out / f"host-{h}"
doc = json.loads((out / f"host-{h}.json").read_text())
svcs = doc["services"]; names = set(svcs); mapping = name_addr(h)
for name, spec in svcs.items():
ref = base.get(name)
if ref is None:
issues.append(f"host-{h}: service '{name}' TIDAK ada di compose dasar"); continue
for key in COMPARE:
if spec.get(key) != ref.get(key):
issues.append(f"host-{h}/{name}: field '{key}' berbeda dari single-host")
# ports: binding internal multi-host dipisahkan dulu, lalu bentuknya diperiksa
addr = addr_of(h)
hb, hi = split_ports(spec.get("ports"), addr)
rb, ri = split_ports(ref.get("ports"), addr)
if hb != rb:
issues.append(f"host-{h}/{name}: ports dasar berbeda dari single-host")
if ri:
issues.append(f"host-{h}/{name}: compose DASAR sudah punya binding internal? {ri}")
for entry in hi:
if isinstance(entry, str):
parts = entry.split(":")
ok = len(parts) == 3 and parts[0] == addr and parts[1] == parts[2]
if not ok:
issues.append(f"host-{h}/{name}: binding internal salah bentuk: {entry}")
elif str(entry.get("target")) != str(entry.get("published")) or entry.get("host_ip") != addr:
issues.append(f"host-{h}/{name}: binding internal salah bentuk: {entry}")
internal_count += len(hi)
if norm_volumes(spec.get("volumes"), host_root) != norm_volumes(ref.get("volumes"), deploy):
issues.append(f"host-{h}/{name}: 'volumes' berbeda (di luar prefix direktori)")
for key in ("command", "healthcheck", "entrypoint"):
if normalize(spec.get(key), mapping) != normalize(ref.get(key), mapping):
issues.append(f"host-{h}/{name}: '{key}' berbeda di luar identitas host")
dd = spec.get("depends_on", {}) or {}; db = ref.get("depends_on", {}) or {}
if set(dd) - set(db):
issues.append(f"host-{h}/{name}: depends_on menambah {sorted(set(dd)-set(db))}")
if set(dd) - names:
issues.append(f"host-{h}/{name}: depends_on masih lintas host {sorted(set(dd)-names)}")
ea = spec.get("environment", {}) or {}; eb = ref.get("environment", {}) or {}
if set(ea) != set(eb):
issues.append(f"host-{h}/{name}: kunci environment berbeda "
f"(hilang={sorted(set(eb)-set(ea))}, tambahan={sorted(set(ea)-set(eb))})")
for k in set(ea) & set(eb):
if normalize_env_key(k, ea[k], mapping) != normalize_env_key(k, eb[k], mapping):
issues.append(f"host-{h}/{name}: env '{k}' berbeda di luar identitas host\n"
f" dasar={eb[k]!r}\n host ={ea[k]!r}")
elif eb[k] != ea[k]:
changed_env += 1
checked += 1
print(f"checked={checked}")
print(f"env_diarahkan={changed_env}")
print(f"binding_internal={internal_count}")
for i in issues:
print(f"issue={i}")
PY
)"
checked="$(echo "${sem_out}" | sed -n 's/^checked=//p')"
internal="$(echo "${sem_out}" | sed -n 's/^binding_internal=//p')"
issues="$(echo "${sem_out}" | grep -c '^issue=' || true)"
if [[ "${issues}" -eq 0 ]]; then
ok "semantik: ${checked} service identik dengan single-host (kecuali variabel topologi)"
ok "port internal multi-host: ${internal} binding (dikunci ke alamat host, host port == container port)"
else
bad "semantik: ${issues} masalah"
echo "${sem_out}" | grep '^issue=' | sed 's/^issue=/ /' | head -12
fi
# ─── 6. URL browser-facing menurut tls.mode ────────────────────────────────
# Mode `none` (contoh) tidak boleh menghasilkan URL publik sama sekali; mode `edge`
# harus menghasilkan https://<label>.<domain> untuk enam UI, TANPA membocorkan nama
# domain ke variabel internal (container tetap memakai alamat).
echo
echo "── URL browser-facing (tls.mode) ──"
# Jebakan bash: `grep -c` mencetak "0" SEKALIGUS keluar status 1, sehingga
# `|| echo 0` menambah baris kedua ("0\n0") dan [[ ]] jadi error sintaks.
# Pakai `|| true` lalu default-nya.
none_urls="$(grep -c '_PUBLIC_URL=' "${OUT}/host-control/.env" 2>/dev/null || true)"
none_urls="${none_urls:-0}"
if [[ "${none_urls}" -eq 6 ]]; then
ok "mode none: 6 URL publik di .env host"
else
bad "mode none: ${none_urls} URL publik (harus 6)"
fi
# Mode none = tanpa TLS/proxy → browser memakai alamat HOST, jadi URL publik:
# (a) harus menunjuk alamat host, BUKAN localhost atau nama service internal
# (dulu defaultnya localhost / airflow-webserver:8080 → browser host lain
# dibuang ke localhost, dan issuer OIDC ikut salah);
# (b) harus SAMA di setiap bundle — issuer OIDC hanya boleh satu nilai.
none_bad=""
while IFS= read -r line; do
[[ -z "${line}" ]] && continue
val="${line#*=}"
[[ "${val}" =~ ^https?://([0-9]{1,3}\.){3}[0-9]{1,3}:[0-9]+$ ]] || none_bad+=" ${line}"
done < <(grep '_PUBLIC_URL=' "${OUT}/host-control/.env" 2>/dev/null || true)
if [[ -z "${none_bad}" ]]; then
ok "mode none: URL publik memakai alamat host (bukan localhost/nama internal)"
else
bad "mode none: URL publik bukan alamat host:${none_bad}"
fi
for f in "${OUT}"/host-*/.env; do
grep '_PUBLIC_URL=' "${f}" 2>/dev/null | sort > "${f}.urls" || true
done
if diff -q "${OUT}/host-control/.env.urls" "${OUT}/host-state/.env.urls" >/dev/null 2>&1 \
&& diff -q "${OUT}/host-control/.env.urls" "${OUT}/host-compute/.env.urls" >/dev/null 2>&1; then
ok "mode none: URL publik identik di semua bundle (issuer satu nilai)"
else
bad "mode none: URL publik BEDA antar bundle"
fi
# KC_HOSTNAME harus berisi URL publik itu — opsi inilah yang menentukan issuer.
kc_pub="$(grep '^KEYCLOAK_PUBLIC_URL=' "${OUT}/host-control/.env" 2>/dev/null | cut -d= -f2- || true)"
kc_resolved="$(docker compose -f "${OUT}/host-control/docker-compose.yml" \
--env-file "${OUT}/host-control/.env" --profile all config 2>/dev/null \
| python3 -c "
import sys, yaml
d = yaml.safe_load(sys.stdin) or {}
print(((d.get('services') or {}).get('keycloak') or {}).get('environment', {}).get('KC_HOSTNAME', ''))" 2>/dev/null || true)"
if [[ -n "${kc_pub}" && "${kc_resolved}" == "${kc_pub}" ]]; then
ok "mode none: KC_HOSTNAME = URL publik Keycloak (${kc_pub})"
else
bad "mode none: KC_HOSTNAME='${kc_resolved}' != KEYCLOAK_PUBLIC_URL='${kc_pub}'"
fi
# T-21 di mode none: console MinIO memakai URL host pemilik (state), bukan localhost.
rmc="$(docker compose -f "${OUT}/host-state/docker-compose.yml" --env-file "${OUT}/host-state/.env" \
--profile all config 2>/dev/null | python3 -c "
import sys, yaml
d = yaml.safe_load(sys.stdin) or {}
e = ((d.get('services') or {}).get('minio') or {}).get('environment', {}) or {}
print(e.get('MINIO_BROWSER_REDIRECT_URL', ''))" 2>/dev/null || true)"
if [[ "${rmc}" == "http://10.20.0.12:12901" ]]; then
ok "mode none: MINIO_BROWSER_REDIRECT_URL = alamat host state"
else
bad "mode none: MINIO_BROWSER_REDIRECT_URL='${rmc}' (harus http://10.20.0.12:12901)"
fi
# T-20: tanpa TLS/ingress, service traefik TIDAK ikut (single-host tetap utuh)
if grep -q '^ traefik:' "${OUT}/host-control/docker-compose.yml" 2>/dev/null; then
bad "mode none: service traefik ikut padahal tidak ada ingress"
else
ok "mode none: service traefik tidak ikut (single-host tidak berubah)"
fi
if grep -qx 'TRINO_PROCESS_FORWARDED=false' "${OUT}/host-control/.env" 2>/dev/null; then
ok "mode none: TRINO_PROCESS_FORWARDED=false (menolak header X-Forwarded-* dari klien)"
else
bad "mode none: TRINO_PROCESS_FORWARDED bukan false"
fi
# sslRequired: mode none = HTTP polos -> realm Keycloak tidak boleh menuntut HTTPS.
# Alamat Tailscale 100.64.0.0/10 dianggap NON-lokal oleh Keycloak, sehingga default
# `external` membuat admin console gagal dengan "HTTPS required".
if grep -qx 'KEYCLOAK_SSL_REQUIRED=none' "${OUT}/host-control/.env" 2>/dev/null; then
ok "mode none: KEYCLOAK_SSL_REQUIRED=none (bukan external)"
else
bad "mode none: KEYCLOAK_SSL_REQUIRED bukan none"
fi
if grep -q 'KC_HOSTNAME_URL' "${OUT}/host-control/docker-compose.yml" 2>/dev/null; then
bad "compose masih memuat KC_HOSTNAME_URL (opsi dihapus di Keycloak 26, tidak berefek)"
else
ok "compose tidak memuat KC_HOSTNAME_URL (benar: dihapus di KC 26)"
fi
EDGE_INV="${OUT}/edge.yml"
cat > "${EDGE_INV}" <<'YAML'
version: 1
cluster:
name: sicabe-prod
env: production
domain: sicabe.com
preset: a3
tls:
mode: edge
hosts:
control: { address: 10.20.0.10, tiers: [identity, policy, pipeline, governance] }
compute: { address: 10.20.0.11, tiers: [coord, engine] }
state: { address: 10.20.0.12, tiers: [state] }
components:
dremio: { enabled: false }
YAML
EDGE_OUT="${OUT}/edge"
if python3 "${DEPLOY_DIR}/lib/cluster/generate.py" render "${EDGE_INV}" \
"${DEPLOY_DIR}/docker-compose.yml" "${DEPLOY_DIR}/.env" "${EDGE_OUT}" \
"${DEPLOY_DIR}/scripts/render-config.sh" >/dev/null 2>&1; then
edge_env="${EDGE_OUT}/host-control/.env"
edge_n="$(grep -c '_PUBLIC_URL=' "${edge_env}" 2>/dev/null || true)"
edge_n="${edge_n:-0}"
[[ "${edge_n}" -eq 6 ]] && ok "mode edge: 6 URL publik di .env" \
|| bad "mode edge: ${edge_n} URL publik (harus 6)"
for pair in "KEYCLOAK_PUBLIC_URL=https://kc.sicabe.com" \
"GRAFANA_PUBLIC_URL=https://grafana.sicabe.com" \
"OPENMETADATA_PUBLIC_URL=https://om.sicabe.com" \
"AIRFLOW_PUBLIC_URL=https://airflow.sicabe.com" \
"TRINO_PUBLIC_URL=https://trino.sicabe.com" \
"MINIO_CONSOLE_PUBLIC_URL=https://minio.sicabe.com"; do
grep -qx "${pair}" "${edge_env}" && ok "mode edge: ${pair%%=*}" \
|| bad "mode edge: ${pair} tidak ada"
done
# ── T-20 (ingress/TLS): service Traefik + rute + kebijakan proxy ────────────
svc_labels() { # <host> <service>
docker compose -f "${EDGE_OUT}/host-$1/docker-compose.yml" --env-file "${EDGE_OUT}/host-$1/.env" \
--profile all --profile edge config 2>/dev/null | python3 -c "
import sys, yaml
d = yaml.safe_load(sys.stdin) or {}
lab = (((d.get('services') or {}).get('$2') or {}).get('labels')) or []
if isinstance(lab, dict):
for k, v in lab.items():
print(k + '=' + str(v))
else:
for x in lab:
print(x)" 2>/dev/null || true
}
host_port_of() { # <host> <service> <container_port>
docker compose -f "${EDGE_OUT}/host-$1/docker-compose.yml" --env-file "${EDGE_OUT}/host-$1/.env" \
--profile all --profile edge config 2>/dev/null | python3 -c "
import sys, yaml
d = yaml.safe_load(sys.stdin) or {}
for e in (((d.get('services') or {}).get('$2') or {}).get('ports') or []):
if isinstance(e, dict) and str(e.get('target')) == '$3':
print(e.get('host_ip')); break" 2>/dev/null || true
}
# Ingress SATU host (default: pemilik keycloak = control). Ini yang menghilangkan
# perebutan record TXT DNS-01 (dulu tiap host menerbitkan wildcard yang sama) dan
# menekan pemakaian batas sertifikat Let's Encrypt.
grep -q '^ traefik:' "${EDGE_OUT}/host-control/docker-compose.yml" 2>/dev/null \
&& ok "mode edge: service traefik ada di host ingress (control)" \
|| bad "mode edge: service traefik TIDAK ada di host ingress"
for h in compute state; do
grep -q '^ traefik:' "${EDGE_OUT}/host-${h}/docker-compose.yml" 2>/dev/null \
&& bad "mode edge: service traefik ikut di host ${h} (harus hanya ingress)" \
|| ok "mode edge: host ${h} tanpa Traefik (rute jarak jauh ditangani ingress)"
done
for pt in 80 443; do
got="$(host_port_of control traefik "${pt}")"
[[ "${got}" == "10.20.0.10" ]] && ok "mode edge: traefik :${pt} terikat ke 10.20.0.10" \
|| bad "mode edge: traefik :${pt} terikat ke '${got}' (harus 10.20.0.10)"
done
# CLUSTER_DOMAIN hanya perlu di host ingress (label rute)
grep -qx 'CLUSTER_DOMAIN=sicabe.com' "${EDGE_OUT}/host-control/.env" 2>/dev/null \
&& ok "mode edge: CLUSTER_DOMAIN ada di host ingress" \
|| bad "mode edge: CLUSTER_DOMAIN tidak ada di host ingress"
grep -q '^CLUSTER_DOMAIN=' "${EDGE_OUT}/host-compute/.env" 2>/dev/null \
&& bad "mode edge: CLUSTER_DOMAIN ikut di host non-ingress" \
|| ok "mode edge: host non-ingress tanpa CLUSTER_DOMAIN"
# Rute untuk komponen di HOST LAIN ditulis di berkas dinamis (file provider),
# karena Traefik hanya melihat container di host-nya sendiri.
dyn="${EDGE_OUT}/host-control/config/traefik/dynamic.yml"
if [[ -f "${dyn}" ]]; then
ok "mode edge: berkas dinamis ingress dibuat"
grep -q 'https://10.20.0.11:12443' "${dyn}" \
&& ok "mode edge: rute trino -> compute:12443 (https/backend)" \
|| bad "mode edge: rute trino tidak menunjuk compute:12443"
grep -q 'http://10.20.0.12:12901' "${dyn}" \
&& ok "mode edge: rute minio -> state:12901 (console)" \
|| bad "mode edge: rute minio tidak menunjuk state:12901"
grep -q 'serversTransport: trino-insecure' "${dyn}" \
&& ok "mode edge: rute trino memakai serversTransport insecure (sertifikat internal)" \
|| bad "mode edge: rute trino tanpa serversTransport"
grep -qE 'kc\.sicabe\.com|grafana\.sicabe\.com' "${dyn}" \
&& bad "mode edge: komponen lokal ikut ditulis di berkas dinamis (harus lewat label)" \
|| ok "mode edge: berkas dinamis hanya memuat rute jarak jauh"
else
bad "mode edge: berkas dinamis ingress tidak dibuat"
fi
# Rute Host() per layanan memakai domain dari bundle (bukan jatuh ke `localhost`).
for spec in "control:keycloak:kc" "control:grafana:grafana" "control:openmetadata-server:om" \
"control:airflow-webserver:airflow"; do
h="${spec%%:*}"; rest="${spec#*:}"; svc="${rest%%:*}"; label="${rest##*:}"
svc_labels "${h}" "${svc}" | grep -qF "Host(\`${label}.sicabe.com\`)" \
&& ok "mode edge: rute $svc = Host(\`${label}.sicabe.com\`)" \
|| bad "mode edge: rute $svc tidak memakai domain bundle"
done
# Trino internal HTTPS -> backend https + serversTransport dari provider file
# (transport TIDAK bisa lewat label; lewat label muncul error runtime).
svc_labels compute trino-coordinator | grep -qF "loadbalancer.server.scheme=https" \
&& ok "mode edge: backend Trino memakai https" \
|| bad "mode edge: backend Trino tidak diset https"
svc_labels compute trino-coordinator | grep -qF "serverstransport=trino-insecure@file" \
&& ok "mode edge: Trino memakai serversTransport dari provider file" \
|| bad "mode edge: serversTransport Trino salah"
# Jeda propagasi DNS-01: apex + wildcard menulis nama TXT yang SAMA
# (_acme-challenge.<domain>) -> tanpa jeda, LE bisa membaca record lama.
if docker compose -f "${EDGE_OUT}/host-control/docker-compose.yml" \
--env-file "${EDGE_OUT}/host-control/.env" --profile all --profile edge config 2>/dev/null \
| grep -q 'dnschallenge.propagation.delayBeforeChecks='; then
ok "mode edge: ACME memakai propagation.delayBeforeChecks (jeda propagasi DNS)"
else
bad "mode edge: ACME tanpa jeda propagasi DNS"
fi
grep -q 'insecureSkipVerify: true' "${DEPLOY_DIR}/config/traefik/dynamic.yml" 2>/dev/null \
&& ok "mode edge: berkas dinamis mendefinisikan insecureSkipVerify untuk Trino" \
|| bad "mode edge: config/traefik/dynamic.yml tidak berisi insecureSkipVerify"
if grep -qx 'KEYCLOAK_SSL_REQUIRED=external' "${edge_env}" 2>/dev/null; then
ok "mode edge: KEYCLOAK_SSL_REQUIRED=external (ada TLS/proxy di depan)"
else
bad "mode edge: KEYCLOAK_SSL_REQUIRED bukan external"
fi
# Kelas bug "hostname bare": config yang memakai nilai bare untuk key host
# (mis. Sling `host: postgres`) tidak punya ':' atau '/', sehingga sempat TIDAK
# tersubstitusi -> di host lain mencari nama container yang tidak ada. Sekaligus
# pastikan nilai non-host yang kebetulan sama (dialek DB) TIDAK ikut berubah.
sling="${EDGE_OUT}/host-control/config/airflow/sling/env.yaml"
if [[ -f "${sling}" ]]; then
grep -q "host: 10.20.0.12" "${sling}" \
&& ok "config: nilai bare key 'host' tersubstitusi (sling 5432 -> alamat state)" \
|| bad "config: 'host: postgres' di sling TIDAK tersubstitusi"
grep -q "type: postgres" "${sling}" \
&& ok "config: 'type: postgres' (dialek) TIDAK ikut berubah" \
|| bad "config: nilai dialek 'type: postgres' ikut berubah (berbahaya)"
else
warn_skip=$((warn_skip + 1))
fi
# DB_HOST OpenMetadata: sempat literal "postgres" dan menyebabkan
# java.net.UnknownHostException di container openmetadata-migrate.
# CATATAN: compose hasil generate SENGAJA tetap memuat ${POSTGRES_HOST:-postgres}
# (resolusi terjadi saat `up` lewat .env host) — berbeda dari config/** yang
# di-render menjadi nilai literal. Jadi yang diuji ada DUA:
# (a) nilainya sudah terparameterisasi (bukan literal "postgres"), dan
# (b) hasil RESOLUSI-nya benar-benar alamat host state.
om_cfg="${EDGE_OUT}/host-control/docker-compose.yml"
grep -q 'DB_HOST: ${POSTGRES_HOST:-postgres}' "${om_cfg}" \
&& ok "compose: DB_HOST OpenMetadata terparameterisasi (bukan literal)" \
|| bad "compose: DB_HOST OpenMetadata masih literal 'postgres'"
resolved_om="$(docker compose -f "${om_cfg}" --env-file "${EDGE_OUT}/host-control/.env" \
--profile all config 2>/dev/null | grep -m1 'DB_HOST:' | awk '{print $2}')"
[[ "${resolved_om}" == "10.20.0.12" ]] \
&& ok "compose: DB_HOST ter-resolve ke alamat state (${resolved_om})" \
|| bad "compose: DB_HOST ter-resolve '${resolved_om}' (harusnya alamat state 10.20.0.12)"
# Precedence nilai: Compose memprioritaskan variabel SHELL di atas --env-file,
# sedangkan install.sh mengekspor default (POSTGRES_HOST=postgres) sebelum memanggil
# compose. Bundle host HARUS tetap menang setelah .env-nya di-source -- inilah yang
# run_host lakukan. Tanpa itu container mencari hostname `postgres` yang hanya ada di
# host lain: java.net.UnknownHostException (bug yang dilaporkan dari server).
resolved_prec="$(set -a; . "${DEPLOY_DIR}/config/environment.defaults.sh"; set +a
set -a; . "${EDGE_OUT}/host-control/.env"; set +a
docker compose -f "${EDGE_OUT}/host-control/docker-compose.yml" \
--env-file "${EDGE_OUT}/host-control/.env" --profile all config 2>/dev/null \
| grep -m1 'DB_HOST:' | awk '{print $2}')"
if [[ "${resolved_prec}" == "10.20.0.12" ]]; then
ok "precedence: nilai bundle host menang atas default yang diekspor (DB_HOST=${resolved_prec})"
else
bad "precedence: DB_HOST ter-resolve '${resolved_prec}' — default menimpa bundle host"
fi
# variabel internal TIDAK boleh memakai nama domain (container pakai alamat)
# Helper: nilai env satu service pada compose host ter-resolve.
pub_env() { # <host> <service> <key>
docker compose -f "${EDGE_OUT}/host-$1/docker-compose.yml" --env-file "${EDGE_OUT}/host-$1/.env" \
--profile all config 2>/dev/null | python3 -c "
import sys, yaml
d = yaml.safe_load(sys.stdin) or {}
e = ((d.get('services') or {}).get('$2') or {}).get('environment', {}) or {}
print(e.get('$3', ''))" 2>/dev/null || true
}
# T-21: URL publik tersambung ke konsumen browser-facing-nya (bukan hanya ditulis di .env).
om_srv="$(pub_env control openmetadata-server OPENMETADATA_SERVER_URL)"
if [[ "${om_srv}" == "https://om.sicabe.com" ]]; then
ok "mode edge: OPENMETADATA_SERVER_URL = URL publik (link email/notifikasi)"
else
bad "mode edge: OPENMETADATA_SERVER_URL='${om_srv}' (harus https://om.sicabe.com)"
fi
mc_redir="$(pub_env state minio MINIO_BROWSER_REDIRECT_URL)"
if [[ "${mc_redir}" == "https://minio.sicabe.com" ]]; then
ok "mode edge: MINIO_BROWSER_REDIRECT_URL = URL publik (console)"
else
bad "mode edge: MINIO_BROWSER_REDIRECT_URL='${mc_redir}' (harus https://minio.sicabe.com)"
fi
# INVRASIAN: SERVER_HOST_API_URL dipakai sisi SERVER (Airflow/ingestion memanggil balik
# OM) -> wajib tetap alamat internal; kalau ikut URL publik, ingestion rusak.
om_api="$(pub_env control openmetadata-server SERVER_HOST_API_URL)"
if [[ -n "${om_api}" && "${om_api}" != *"sicabe.com"* ]]; then
ok "mode edge: SERVER_HOST_API_URL tetap internal (${om_api})"
else
bad "mode edge: SERVER_HOST_API_URL='${om_api}' — harus internal, bukan URL publik"
fi
if grep -E '^(KEYCLOAK_HOST|NESSIE_URL|S3_ENDPOINT|OPA_URL|LDAP_URL|POSTGRES_HOST)=' "${edge_env}" | grep -q 'sicabe\.com'; then
bad "mode edge: nama domain bocor ke variabel internal"
else
ok "mode edge: variabel internal tetap pakai alamat (bukan domain)"
fi
else
bad "mode edge: generate gagal"
fi
echo
if [[ ${FAIL_N} -eq 0 ]]; then
echo -e " ${GRN}${BLD}SEMUA LULUS${NC} (${PASS_N} grup cek)"
exit 0
fi
echo -e " ${RED}${BLD}GAGAL${NC}: ${FAIL_N} masalah · ${PASS_N} lulus"
exit 1
deploy/scripts/test-inventory.sh
Executable
+212
View File
@@ -0,0 +1,212 @@
#!/usr/bin/env bash
# =============================================================================
# test-inventory.sh — uji validator inventory multi-host (Fase 1)
# -----------------------------------------------------------------------------
# Menguji `lib/cluster/inventory.py` secara negatif (harus DITOLAK) dan positif
# (harus DITERIMA). Sengaja banyak kasus negatif: validator yang selalu setuju
# tidak berguna — setiap aturan penempatan R10 §4.3 / R11 §3.2 diuji terpisah.
#
# Jalankan: ./scripts/test-inventory.sh
# =============================================================================
set -uo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
DEPLOY_DIR="$(cd "${SCRIPT_DIR}/.." && pwd)"
INV="${DEPLOY_DIR}/lib/cluster/inventory.py"
TMP="$(mktemp -d)"
trap 'rm -rf "${TMP}"' EXIT
GRN=$'\e[32m'; RED=$'\e[31m'; BLD=$'\e[1m'; NC=$'\e[0m'
PASS_N=0; FAIL_N=0
check() { # <harapan: reject|accept> <nama> <yaml>
local expect="$1" name="$2" yaml="$3" rc out
printf '%b' "$yaml" > "${TMP}/t.yml"
out="$(python3 "${INV}" validate "${TMP}/t.yml" 2>&1)"; rc=$?
if [[ "${expect}" == "reject" ]]; then
if [[ ${rc} -ne 0 && "${out}" != *"Traceback"* ]]; then
printf " ${GRN}✅ ditolak${NC} %-42s → %s\n" "${name}" "$(echo "${out}" | sed -n '2p' | cut -c1-66)"
PASS_N=$((PASS_N + 1))
else
printf " ${RED}❌ MASALAH${NC} %-42s rc=%s (harus ditolak tanpa traceback)\n" "${name}" "${rc}"
echo "${out}" | head -4 | sed 's/^/ /'
FAIL_N=$((FAIL_N + 1))
fi
else
if [[ ${rc} -eq 0 ]]; then
printf " ${GRN}✅ diterima${NC} %-41s\n" "${name}"
PASS_N=$((PASS_N + 1))
else
printf " ${RED}❌ DITOLAK (harusnya valid)${NC} %s\n" "${name}"
echo "${out}" | head -4 | sed 's/^/ /'
FAIL_N=$((FAIL_N + 1))
fi
fi
}
echo -e "${BLD}Uji validator inventory (lib/cluster/inventory.py)${NC}"
echo
echo "── harus DITOLAK ──"
check reject "state dipecah ke 2 host" 'hosts:\n a:\n address: 1.1.1.1\n tiers: [state]\n b:\n address: 2.2.2.2\n tiers: [state, engine]\n'
check reject "coord di 2 host" 'hosts:\n a:\n address: 1.1.1.1\n tiers: [coord, engine]\n b:\n address: 2.2.2.2\n tiers: [coord, state]\n'
check reject "state + engine satu host (anti-affinity)" 'hosts:\n a:\n address: 1.1.1.1\n tiers: [state, engine, coord]\n'
check reject "preset tak dikenal" 'preset: a9\nhosts:\n a:\n address: 1.1.1.1\n tiers: [state]\n b:\n address: 2.2.2.2\n tiers: [engine, coord]\n'
check reject "tier tak dikenal" 'hosts:\n a:\n address: 1.1.1.1\n tiers: [state, engine, coord, banjar]\n'
check reject "address kosong" 'hosts:\n a:\n tiers: [state, engine, coord]\n'
check reject "components.host tidak ada di hosts" 'hosts:\n a:\n address: 1.1.1.1\n tiers: [state]\n b:\n address: 2.2.2.2\n tiers: [engine, coord]\ncomponents:\n grafana: { host: tidak-ada }\n'
check reject "tanpa host bertier engine" 'hosts:\n a:\n address: 1.1.1.1\n tiers: [state, coord]\n'
check reject "hosts kosong" 'preset: a3\nhosts: {}\n'
check reject "YAML rusak" 'hosts:\n a: [ini: bukan: yaml\n'
echo
echo "── harus DITOLAK (TLS & domain) ──"
check reject "tls.mode tak dikenal" 'tls:
mode: tls
cluster:
domain: sicabe.com
preset: a3
hosts:
control: { address: 1.1.1.1, tiers: [identity, policy, pipeline, governance] }
compute: { address: 1.1.1.2, tiers: [coord, engine] }
state: { address: 1.1.1.3, tiers: [state] }
'
check reject "edge tanpa domain" 'tls:
mode: edge
preset: a3
hosts:
control: { address: 1.1.1.1, tiers: [identity, policy, pipeline, governance] }
compute: { address: 1.1.1.2, tiers: [coord, engine] }
state: { address: 1.1.1.3, tiers: [state] }
'
check reject "domain memuat skema" 'tls:
mode: edge
cluster:
domain: https://sicabe.com
preset: a3
hosts:
control: { address: 1.1.1.1, tiers: [identity, policy, pipeline, governance] }
compute: { address: 1.1.1.2, tiers: [coord, engine] }
state: { address: 1.1.1.3, tiers: [state] }
'
check reject "domain tanpa titik" 'tls:
mode: full
cluster:
domain: sicabe
preset: a3
hosts:
control: { address: 1.1.1.1, tiers: [identity, policy, pipeline, governance] }
compute: { address: 1.1.1.2, tiers: [coord, engine] }
state: { address: 1.1.1.3, tiers: [state] }
'
echo
echo "── harus DITERIMA ──"
check accept "cluster.example.yml (preset a3)" "$(cat "${DEPLOY_DIR}/cluster.example.yml")"
check accept "edge + domain sah" 'tls:
mode: edge
cluster:
domain: sicabe.com
preset: a3
hosts:
control: { address: 1.1.1.1, tiers: [identity, policy, pipeline, governance] }
compute: { address: 1.1.1.2, tiers: [coord, engine] }
state: { address: 1.1.1.3, tiers: [state] }
'
check accept "preset b5 (5 host, tanpa tiers eksplisit)" 'preset: b5\nhosts:\n control: { address: 10.0.0.10 }\n coord: { address: 10.0.0.11 }\n engine-1: { address: 10.0.0.12 }\n engine-2: { address: 10.0.0.13 }\n state: { address: 10.0.0.14 }\n'
echo
echo "── distribusi penempatan (plan) ──"
plan_out="$(python3 "${INV}" plan "${DEPLOY_DIR}/cluster.example.yml" 2>&1)"
plan_check() { # <nama> <host> <service-yang-harus-ada>
local name="$1" host="$2" svc="$3"
if echo "${plan_out}" | python3 -c "
import json,sys
p=json.load(sys.stdin)
sys.exit(0 if '$svc' in p.get('$host',{}).get('services',[]) else 1)"; then
printf " ${GRN}✅${NC} %-46s %s ∈ %s\n" "$name" "$svc" "$host"; PASS_N=$((PASS_N + 1))
else
printf " ${RED}❌${NC} %-46s %s TIDAK di %s\n" "$name" "$svc" "$host"; FAIL_N=$((FAIL_N + 1))
fi
}
plan_check "a3: coordinator di compute" compute trino-coordinator
plan_check "a3: worker di compute" compute trino-worker
plan_check "a3: postgres di state" state postgres
plan_check "a3: minio di state" state minio
plan_check "a3: keycloak di control" control keycloak
plan_check "a3: grafana di control" control grafana
plan_check "a3: airflow-webserver di control" control airflow-webserver
plan_not_in() { # <nama> <host> <service-yang-harus-TIDAK-ada>
local name="$1" host="$2" svc="$3"
if echo "${plan_out}" | python3 -c "
import json,sys
p=json.load(sys.stdin)
sys.exit(0 if '$svc' not in p.get('$host',{}).get('services',[]) else 1)"; then
printf " ${GRN}✅${NC} %-46s %s ∉ %s\n" "${name}" "${svc}" "${host}"; PASS_N=$((PASS_N + 1))
else
printf " ${RED}❌${NC} %-46s %s ADA di %s\n" "${name}" "${svc}" "${host}"; FAIL_N=$((FAIL_N + 1))
fi
}
plan_not_in "a3: postgres tidak di control" control postgres
plan_not_in "a3: trino-worker tidak di control" control trino-worker
plan_not_in "a3: dremio off (default) tidak ditempatkan" compute dremio
# b5: tier engine >1 host → worker & CN harus muncul di KEDUA host engine
printf 'preset: b5\nhosts:\n control: { address: 10.0.0.10 }\n coord: { address: 10.0.0.11 }\n engine-1: { address: 10.0.0.12 }\n engine-2: { address: 10.0.0.13 }\n state: { address: 10.0.0.14 }\n' > "${TMP}/b5.yml"
b5_out="$(python3 "${INV}" plan "${TMP}/b5.yml" 2>&1)"
b5_check() { # <host> <service>
if echo "${b5_out}" | python3 -c "
import json,sys
p=json.load(sys.stdin)
sys.exit(0 if '$2' in p.get('$1',{}).get('services',[]) else 1)"; then
printf " ${GRN}✅${NC} b5: %-14s %s\n" "$2 ∈ $1" ""; PASS_N=$((PASS_N + 1))
else
printf " ${RED}❌${NC} b5: %s TIDAK di %s\n" "$2" "$1"; FAIL_N=$((FAIL_N + 1))
fi
}
b5_check engine-1 trino-worker
b5_check engine-2 trino-worker
b5_check engine-1 starrocks-cn
b5_check engine-2 starrocks-cn
b5_check coord trino-coordinator
if echo "${b5_out}" | python3 -c "
import json,sys
p=json.load(sys.stdin)
sys.exit(0 if 'trino-worker' not in p.get('coord',{}).get('services',[]) else 1)"; then
printf " ${GRN}✅${NC} b5: trino-worker tidak di coord (coord ⊥ engine)\n"; PASS_N=$((PASS_N + 1))
else
printf " ${RED}❌${NC} b5: trino-worker ada di coord\n"; FAIL_N=$((FAIL_N + 1))
fi
echo "── URL browser-facing turunan (tls.mode) ──"
url_case() { # <nama> <yaml> <harapan: kosong|ada> <var>
local name="$1" yaml="$2" expect="$3" var="$4" out
printf '%b' "$yaml" > "${TMP}/tls.yml"
out="$(python3 -c "
import sys; sys.path.insert(0,'${DEPLOY_DIR}/lib/cluster')
import inventory
doc = inventory.apply_preset(inventory.load('${TMP}/tls.yml'))
print('|'.join(f'{k}={v}' for k,v in sorted(inventory.public_urls(doc).items())))" 2>/dev/null)"
if [[ "${expect}" == "kosong" ]]; then
if [[ -z "${out}" ]]; then printf " ${GRN}✅${NC} %s\n" "${name}"; PASS_N=$((PASS_N+1))
else printf " ${RED}❌${NC} %s — seharusnya kosong, dapat: %s\n" "${name}" "${out}"; FAIL_N=$((FAIL_N+1)); fi
else
if [[ "${out}" == *"${var}=${expect}"* ]]; then printf " ${GRN}✅${NC} %s\n" "${name}"; PASS_N=$((PASS_N+1))
else printf " ${RED}❌${NC} %s — tidak menemukan %s=%s di: %s\n" "${name}" "${var}" "${expect}" "${out}"; FAIL_N=$((FAIL_N+1)); fi
fi
}
A3='tls:\n mode: none\ncluster:\n domain: sicabe.com\npreset: a3\nhosts:\n control: { address: 1.1.1.1, tiers: [identity, policy, pipeline, governance] }\n compute: { address: 1.1.1.2, tiers: [coord, engine] }\n state: { address: 1.1.1.3, tiers: [state] }\n'
EDGE='tls:\n mode: edge\ncluster:\n domain: sicabe.com\npreset: a3\nhosts:\n control: { address: 1.1.1.1, tiers: [identity, policy, pipeline, governance] }\n compute: { address: 1.1.1.2, tiers: [coord, engine] }\n state: { address: 1.1.1.3, tiers: [state] }\n'
FULL='tls:\n mode: full\ncluster:\n domain: contoh.org\npreset: a3\nhosts:\n control: { address: 1.1.1.1, tiers: [identity, policy, pipeline, governance] }\n compute: { address: 1.1.1.2, tiers: [coord, engine] }\n state: { address: 1.1.1.3, tiers: [state] }\n'
url_case "mode none -> TIDAK ada URL publik" "${A3}" kosong ""
url_case "mode edge -> Keycloak https" "${EDGE}" "https://kc.sicabe.com" "KEYCLOAK_PUBLIC_URL"
url_case "mode edge -> Grafana https" "${EDGE}" "https://grafana.sicabe.com" "GRAFANA_PUBLIC_URL"
url_case "mode edge -> MinIO console https" "${EDGE}" "https://minio.sicabe.com" "MINIO_CONSOLE_PUBLIC_URL"
url_case "mode full -> memakai domain sendiri" "${FULL}" "https://om.contoh.org" "OPENMETADATA_PUBLIC_URL"
echo
if [[ ${FAIL_N} -eq 0 ]]; then
echo -e " ${GRN}${BLD}SEMUA LULUS${NC} (${PASS_N} cek)"
exit 0
fi
echo -e " ${RED}${BLD}GAGAL${NC}: ${FAIL_N} bermasalah · ${PASS_N} lulus"
exit 1
deploy/scripts/test-shell-safety.sh
Executable
+86
View File
@@ -0,0 +1,86 @@
#!/usr/bin/env bash
# =============================================================================
# test-shell-safety.sh — jaga "jebakan set -e + grep" di skrip deploy
# -----------------------------------------------------------------------------
# JEBAKAN: pipeline yang memuat `grep` mengembalikan 1 saat TIDAK ada baris cocok.
# Di skrip ber-`set -euo pipefail`, hasil pipeline itu MEMATIKAN skrip DIAM-DIAM.
#
# Regresi nyata: e099b95 membuat `external_services()` di install.sh berakhir
# ... | grep -v '^$' | sort -u | paste ...
# sehingga `./install.sh` (single-host tanpa service eksternal) keluar exit 1
# TEPAT setelah banner. Perbaikan: `awk 'NF'` (buang baris kosong, selalu exit 0).
#
# YANG DIPERIKSA (hanya skrip ber-`set -e`):
# 1. `| grep -v` / `| grep -c` — mode FILTER data yang lazim exit 1 saat kosong,
# dan jarang dipakai sebagai uji boolean. Lain halnya `grep -q`/`-E` yang
# memang sering dipakai di `if`, sehingga TIDAK ditandai.
# 2. Idiom `grep -v '^$'` — pemicu langsung regresi di atas.
#
# Baik (1) maupun (2) DILEWATI bila barisnya jelas kondisional — `if/elif/while/
# until/!` atau memuat `&&`/`||` — karena status exit-nya memang dikonsumsi.
#
# Jalankan: ./scripts/test-shell-safety.sh
# =============================================================================
set -uo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
DEPLOY_DIR="$(cd "${SCRIPT_DIR}/.." && pwd)"
GRN=$'\e[32m'; RED=$'\e[31m'; BLD=$'\e[1m'; NC=$'\e[0m'
PASS_N=0; FAIL_N=0
ok() { printf " ${GRN}✅${NC} %s\n" "$1"; PASS_N=$((PASS_N+1)); }
bad() { printf " ${RED}❌${NC} %s\n" "$1"; FAIL_N=$((FAIL_N+1)); }
# Baris ini "aman" dari set -e karena status exit-nya dikonsumsi (kondisi operator)?
is_conditional() {
local s="$1"
case "${s}" in
if\ *|elif\ *|while\ *|until\ *|'!'*) return 0 ;;
esac
[[ "${s}" == *"&&"* || "${s}" == *"||"* || "${s}" == *"|| true"* ]] && return 0
return 1
}
echo -e "${BLD}Uji keamanan shell (jebakan set -e + grep)${NC}"
echo
checked=0
while IFS= read -r f; do
# Hanya skrip ber-`set -e` (opsi memuat huruf 'e'); `set -uo pipefail` tanpa -e
# tidak berisiko karena tak ada yang mematikan skrip.
grep -qE '^[[:space:]]*set -[a-zA-Z]*e' "$f" 2>/dev/null || continue
checked=$((checked+1))
rel="${f#"${DEPLOY_DIR}/"}"
file_bad=0
# (1) pipe ke grep -v / grep -c
while IFS=: read -r ln text; do
[[ -z "${ln}" ]] && continue
stripped="$(printf '%s' "${text}" | sed 's/^[[:space:]]*//')"
is_conditional "${stripped}" && continue
bad "${rel}:${ln} — pipe ke 'grep -v/-c' (exit 1 saat kosong → set -e mematikan)"
printf ' %s\n' "${stripped}"
file_bad=1
done < <(grep -nE '\|[[:space:]]*grep[[:space:]]+-[a-zA-Z]*[vc]' "$f" 2>/dev/null || true)
# (2) idiom terlarang grep -v '^$' (baris komentar & kondisional dikecualikan)
while IFS=: read -r ln text; do
[[ -z "${ln}" ]] && continue
stripped="$(printf '%s' "${text}" | sed 's/^[[:space:]]*//')"
[[ "${stripped}" == '#'* ]] && continue
is_conditional "${stripped}" && continue
bad "${rel}:${ln} — idiom terlarang grep -v '^\$' (pakai awk 'NF')"
file_bad=1
done < <(grep -nF "grep -v '^$'" "$f" 2>/dev/null || true)
[[ ${file_bad} -eq 0 ]] && ok "${rel}: bersih"
done < <(find "${DEPLOY_DIR}" -name '*.sh' -not -path '*/generated/*' | sort)
echo
echo " skrip ber-set -e diperiksa: ${checked}"
if [[ ${FAIL_N} -eq 0 ]]; then
echo -e " ${GRN}${BLD}SEMUA LULUS${NC} (${PASS_N} cek)"
exit 0
fi
echo -e " ${RED}${BLD}GAGAL${NC}: ${FAIL_N} masalah · ${PASS_N} lulus"
exit 1
deploy/scripts/verify-phase1-iam.sh
Executable
+478
View File
@@ -0,0 +1,478 @@
#!/usr/bin/env bash
# =============================================================================
# eBdesk SiCABE — Verifikasi Phase 1 (IAM): Keycloak + OpenLDAP
#
# Gate "kontrak IAM": memastikan konfigurasi Keycloak & LDAP benar-benar SIAP
# dipakai fase berikutnya (Trino+OPA, StarRocks, Airflow, OpenMetadata, Grafana)
# — termasuk saat memakai Keycloak/LDAP EXISTING (bukan container deploy ini).
#
# Yang diperiksa:
# A. Keycloak statis : realm, roles, 5 OIDC client, audience mapper, redirect URI
# B. Keycloak fungsional : login tiap user → preferred_username & roles
# C. LDAP : ou=users/ou=groups + pemetaan user→grup
# (filter PERSIS seperti config/trino/group-provider.properties)
# D. Cross-check : user Keycloak == uid LDAP ;
# grup LDAP == roles di config/opa/data.json == permitted_groups StarRocks
# E. Client secret : nilai tersimpan di Keycloak == nilai env deploy
#
# Exit code: 0 = semua gate lolos (WARN boleh ada), 1 = ada FAIL.
#
# Usage:
# ./scripts/verify-phase1-iam.sh
# # menunjuk IAM existing:
# KEYCLOAK_URL=http://kc.corp:8080 LDAP_HOST=ldap.corp LDAP_PORT=389 \
# ./scripts/verify-phase1-iam.sh
# =============================================================================
set -uo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
# shellcheck source=../lib/common.sh
source "${SCRIPT_DIR}/lib/common.sh"
# shellcheck source=../config/environment.defaults.sh
source "${SCRIPT_DIR}/config/environment.defaults.sh"
sicabe_load_env "${SCRIPT_DIR}" "$@"
# Config RELEVAN: mode host (--env-file) = BUNDLE host (ter-render), bukan deploy/config/ sumber.
if [[ -n "${SICABE_BUNDLE_DIR:-}" ]]; then CONFIG_DIR="${SICABE_BUNDLE_DIR}/config"; else CONFIG_DIR="${SCRIPT_DIR}/config"; fi
KC="$(sicabe_local_url http KEYCLOAK_HOST "${KEYCLOAK_PORT:-12080}" KEYCLOAK_URL)"
REALM="${SICABE_REALM:-sicabe}"
LDAP_HOST="${LDAP_HOST:-localhost}"
LDAP_PORT="${LDAP_PORT:-389}"
LDAP_ADMIN_DN="${LDAP_ADMIN_DN:-cn=admin,dc=sicabe,dc=local}"
LDAP_BASE_DN="${LDAP_BASE_DN:-dc=sicabe,dc=local}"
USERS_DN="${LDAP_USERS_DN:-ou=users,${LDAP_BASE_DN}}"
GROUPS_DN="${LDAP_GROUPS_DN:-ou=groups,${LDAP_BASE_DN}}"
REQUIRED_CLIENTS=("trino" "starrocks" "airflow" "openmetadata" "grafana")
REQUIRED_ROLES=("admin" "engineer" "analyst" "viewer")
REQUIRED_USERS=("admin" "engineer" "analyst" "viewer")
AUDIENCE_MAPPER="audience-account"
PASS_N=0
FAIL_N=0
WARN_N=0
pass() { echo -e " ${PASS} $1"; PASS_N=$((PASS_N + 1)); }
FAIL_MSGS=()
fail() { echo -e " ${FAIL} $1"; FAIL_N=$((FAIL_N + 1)); FAIL_MSGS+=("$1"); }
warn() { echo -e " ${WARN} $1"; WARN_N=$((WARN_N + 1)); }
# ─────────────────────────────────────────────────────────────────────────────
# Keycloak — helper
# ─────────────────────────────────────────────────────────────────────────────
kc_token() {
curl -sf -X POST "${KC}/realms/master/protocol/openid-connect/token" \
-d client_id=admin-cli \
-d "username=${KEYCLOAK_ADMIN:-admin}" \
-d "password=${KEYCLOAK_ADMIN_PASSWORD}" \
-d grant_type=password 2>/dev/null |
python3 -c "import sys,json;print(json.load(sys.stdin).get('access_token',''))" 2>/dev/null || echo ""
}
kc_get() {
curl -sf -H "Authorization: Bearer ${KC_TOKEN}" \
"${KC}/admin/realms/${REALM}$1" 2>/dev/null
}
kc_user_login_probe() {
# → "preferred_username|roles". Membaca **id_token** (scope=openid): sejak
# Keycloak 26 access token bersifat "lightweight" sehingga tidak lagi memuat
# preferred_username / realm_access.
local user="$1" pw="$2"
curl -s -X POST "${KC}/realms/${REALM}/protocol/openid-connect/token" \
-d client_id=admin-cli -d "username=${user}" -d "password=${pw}" \
-d grant_type=password -d scope=openid 2>/dev/null |
python3 -c "
import sys,json,base64
try:
r=json.load(sys.stdin)
except Exception:
print(''); raise SystemExit
tok=r.get('id_token') or r.get('access_token') or ''
if not tok:
print(''); raise SystemExit
p=tok.split('.')[1]; p+='='*(-len(p)%4)
try:
c=json.loads(base64.urlsafe_b64decode(p))
except Exception:
print(''); raise SystemExit
roles=c.get('realm_access',{}).get('roles') or []
print(c.get('preferred_username',''), '|', ','.join(roles))
" 2>/dev/null || echo ""
}
# Deteksi "lightweight access token" (Keycloak 26+): access token tanpa
# preferred_username. Tidak fatal, tetapi perlu diketahui karena komponen yang
# membaca klaim dari ACCESS token (bukan id_token) harus menyesuaikan.
kc_access_token_is_lightweight() {
local user="$1" pw="$2"
curl -s -X POST "${KC}/realms/${REALM}/protocol/openid-connect/token" \
-d client_id=admin-cli -d "username=${user}" -d "password=${pw}" \
-d grant_type=password 2>/dev/null |
python3 -c "
import sys,json,base64
try:
t=json.load(sys.stdin).get('access_token','')
p=t.split('.')[1]; p+='='*(-len(p)%4)
c=json.loads(base64.urlsafe_b64decode(p))
except Exception:
print('unknown'); raise SystemExit
print('lightweight' if 'preferred_username' not in c else 'full')
" 2>/dev/null || echo "unknown"
}
check_keycloak_static() {
echo -e "${BOLD}A. Keycloak — statis (realm, roles, client, mapper)${NC}"
local discovery
discovery=$(curl -sf "${KC}/realms/${REALM}/.well-known/openid-configuration" 2>/dev/null || echo "")
if [[ -n "${discovery}" ]]; then
pass "OIDC discovery realm '${REALM}' OK ($(echo "${discovery}" | python3 -c "import sys,json;print(json.load(sys.stdin).get('issuer',''))" 2>/dev/null))"
else
fail "OIDC discovery realm '${REALM}' tidak merespon — cek ${KC}/realms/${REALM}"
return 1
fi
if [[ "${HAVE_ADMIN:-0}" -eq 0 ]]; then
warn "Tanpa admin Keycloak — cek realm/roles/client/mapper DILEWATI (didelegasikan ke admin IdP)."
return 0
fi
local roles
roles=$(kc_get "/roles" | python3 -c "import sys,json;print(' '.join(r['name'] for r in json.load(sys.stdin)))" 2>/dev/null || echo "")
local r missing_roles=""
for r in "${REQUIRED_ROLES[@]}"; do
[[ " ${roles} " == *" ${r} "* ]] || missing_roles+=" ${r}"
done
if [[ -z "${missing_roles}" ]]; then
pass "Roles lengkap: ${REQUIRED_ROLES[*]}"
else
fail "Role belum ada:${missing_roles}"
fi
local c cid mappers
for c in "${REQUIRED_CLIENTS[@]}"; do
cid=$(kc_get "/clients?clientId=${c}" | python3 -c "import sys,json;d=json.load(sys.stdin);print(d[0]['id'] if d else '')" 2>/dev/null || echo "")
if [[ -z "${cid}" ]]; then
fail "Client OIDC '${c}' tidak ada"
continue
fi
mappers=$(kc_get "/clients/${cid}/protocol-mappers/models" | python3 -c "import sys,json;print(' '.join(m['name'] for m in json.load(sys.stdin)))" 2>/dev/null || echo "")
if [[ " ${mappers} " == *" ${AUDIENCE_MAPPER} "* ]]; then
pass "Client '${c}' ada + audience mapper '${AUDIENCE_MAPPER}'"
else
fail "Client '${c}' TANPA audience mapper '${AUDIENCE_MAPPER}' (Trino OIDC akan menolak token)"
fi
done
local sr_redirect
sr_redirect=$(kc_get "/clients?clientId=starrocks" | python3 -c "
import sys,json
d=json.load(sys.stdin)
print(' '.join(d[0].get('redirectUris',[])) if d else '')" 2>/dev/null || echo "")
if [[ " ${sr_redirect} " == *"starrocks-fe:8030"* ]]; then
pass "Client 'starrocks' punya redirect URI internal FE"
else
warn "Client 'starrocks' belum memuat redirect internal (http://starrocks-fe:8030/api/oauth2)"
fi
return 0
}
check_client_secrets() {
echo -e "${BOLD}E. Client secret — Keycloak vs env deploy${NC}"
if [[ "${HAVE_ADMIN:-0}" -eq 0 ]]; then
warn "Tanpa admin Keycloak — cek secret via Admin API DILEWATI (diganti cek client_credentials §F)."
return 0
fi
local pairs=("trino:${TRINO_CLIENT_SECRET}" "starrocks:${STARROCKS_CLIENT_SECRET}"
"airflow:${AIRFLOW_CLIENT_SECRET}" "openmetadata:${OPENMETADATA_CLIENT_SECRET}"
"grafana:${GRAFANA_CLIENT_SECRET}")
local pair cid expected actual bad=0
for pair in "${pairs[@]}"; do
cid="${pair%%:*}"
expected="${pair#*:}"
actual=$(kc_get "/clients?clientId=${cid}" | python3 -c "
import sys,json
d=json.load(sys.stdin)
print(d[0].get('secret','') if d else '')" 2>/dev/null || echo "")
if [[ -z "${actual}" ]]; then
warn "Client '${cid}' tidak ditemukan — lewati cek secret"
elif [[ "${actual}" == "${expected}" ]]; then
pass "Secret '${cid}' sinkron dengan env deploy"
else
fail "Secret '${cid}' BEDA dari env deploy (komponen akan gagal login)"
bad=$((bad + 1))
fi
done
[[ ${bad} -eq 0 ]] || warn "Samakan secret di .env dengan nilai di Keycloak."
}
check_keycloak_users_login() {
echo -e "${BOLD}B. Keycloak — fungsional (login tiap user)${NC}"
local user pw claim preferred
for user in "${REQUIRED_USERS[@]}"; do
case "${user}" in
admin) pw="${KEYCLOAK_ADMIN_PASSWORD}" ;;
*) pw="${user}123" ;;
esac
claim=$(kc_user_login_probe "${user}" "${pw}")
preferred="${claim%% |*}"
if [[ "${preferred}" == "${user}" ]]; then
pass "Login '${user}' OK (preferred_username=${preferred})"
elif [[ -n "${preferred}" ]]; then
fail "Login '${user}' OK tapi preferred_username='${preferred}' — harus sama dengan uid LDAP"
elif [[ "${HAVE_ADMIN:-0}" -eq 0 ]]; then
warn "Login '${user}' tak bisa diverifikasi (password user dikelola admin IdP) — dilewati"
else
fail "Login '${user}' gagal (password realm di luar default → isi manual)"
fi
done
local lt
lt=$(kc_access_token_is_lightweight "engineer" "engineer123")
if [[ "${lt}" == "lightweight" ]]; then
warn "Access token 'lightweight' (Keycloak 26): tanpa preferred_username/realm_access di access token."
warn " Komponen yang membaca klaim dari access token harus memakai id_token / userinfo."
fi
}
# ─────────────────────────────────────────────────────────────────────────────
# LDAP — helper (container lokal atau LDAP existing)
# ─────────────────────────────────────────────────────────────────────────────
ldap_is_local_container() {
docker ps --format "{{.Names}}" 2>/dev/null | grep -q "^${LDAP_CONTAINER_NAME:-sicabe-openldap}$"
}
ldap_endpoint() {
if ldap_is_local_container; then
echo "ldap://localhost"
else
echo "ldap://${LDAP_HOST}:${LDAP_PORT}"
fi
}
ldap_run() {
if ldap_is_local_container; then
docker exec -i "${LDAP_CONTAINER_NAME:-sicabe-openldap}" "$@"
else
"$@"
fi
}
ldap_search() {
ldap_run ldapsearch -LLL -x -H "$(ldap_endpoint)" \
-D "${LDAP_ADMIN_DN}" -w "${LDAP_ADMIN_PASSWORD}" "$@" 2>/dev/null
}
# Catatan: JANGAN menyalurkan keluaran ldap_search ke `grep -q`. Dengan
# `set -o pipefail`, `grep -q` yang berhenti lebih awal membuat `docker exec`
# kena SIGPIPE (exit 141) sehingga pipa dilaporkan gagal padahal cocok.
# Karena itu hasil ditampung dulu ke variabel.
ldap_entry_exists() {
local base="$1" filter="$2" attr="${3:-cn}" out
out=$(ldap_search -b "${base}" -s one "${filter}" "${attr}")
[[ -n "${out}" ]]
}
ldap_dn_exists() {
local out
out=$(ldap_search -b "$1" -s base "(objectClass=*)" "*")
[[ -n "${out}" ]]
}
check_ldap_structure() {
echo -e "${BOLD}C. LDAP — struktur & pemetaan user→grup${NC}"
if ! ldap_is_local_container && ! command -v ldapsearch >/dev/null 2>&1; then
fail "ldapsearch tidak ada di host (mode LDAP existing). Install ldap-utils."
return 1
fi
local result
result=$(ldap_search -b "${LDAP_BASE_DN}" -s one "(objectClass=*)" dn)
if [[ -n "${result}" ]]; then
pass "LDAP terjangkau di $(ldap_endpoint) — base '${LDAP_BASE_DN}'"
else
fail "LDAP tidak terjangkau / bind gagal di $(ldap_endpoint)"
return 1
fi
local u g dn
for u in "${REQUIRED_USERS[@]}"; do
dn="uid=${u},${USERS_DN}"
if ldap_dn_exists "${dn}"; then
pass "User LDAP '${u}' ada"
else
fail "User LDAP '${u}' tidak ada (${dn}) — Trino tidak akan menemukan grupnya"
fi
done
local user_groups
for u in "${REQUIRED_USERS[@]}"; do
user_groups=$(ldap_search -b "${GROUPS_DN}" "(member=uid=${u},${USERS_DN})" cn |
sed -n 's/^cn: //p' | tr '\n' ',' | sed 's/,$//')
if [[ -n "${user_groups}" ]]; then
pass " ${u} → ${user_groups}"
else
fail " ${u} tidak berada di grup mana pun (otorisasi akan selalu ditolak)"
fi
done
for g in data-admin data-engineer data-analyst data-viewer; do
ldap_entry_exists "${GROUPS_DN}" "(cn=${g})" cn ||
warn "Grup LDAP '${g}' tidak ada"
done
}
# ─────────────────────────────────────────────────────────────────────────────
# D. Cross-check antar-sistem
# ─────────────────────────────────────────────────────────────────────────────
check_cross_reference() {
echo -e "${BOLD}D. Cross-check antar-sistem${NC}"
if [[ "${HAVE_ADMIN:-0}" -eq 1 ]]; then
local kc_users
kc_users=$(kc_get "/users?max=200" |
python3 -c "import sys,json;print(' '.join(sorted(u['username'] for u in json.load(sys.stdin))))" 2>/dev/null || echo "")
local u ldap_missing=""
for u in "${REQUIRED_USERS[@]}"; do
[[ " ${kc_users} " == *" ${u} "* ]] || ldap_missing+=" ${u}"
done
if [[ -z "${ldap_missing}" ]]; then
pass "User Keycloak ⊇ user wajib (${REQUIRED_USERS[*]})"
else
fail "User Keycloak tidak punya:${ldap_missing} — tidak akan bisa login OIDC"
fi
else
warn "Tanpa admin Keycloak — cek daftar user DILEWATI (verifikasi user lewat login §B)."
fi
local opa_roles
opa_roles=$(python3 -c "
import json
print(' '.join(sorted(json.load(open('${CONFIG_DIR}/opa/data.json'))['roles'])))" 2>/dev/null || echo "")
if [[ -n "${opa_roles}" ]]; then
pass "Roles di OPA (data.json): ${opa_roles}"
else
fail "Tidak bisa membaca roles dari config/opa/data.json"
fi
# Role dapat diberikan lewat dua jalur: (a) grup LDAP dengan cn sama, atau
# (b) pemetaan service account di data.users (untuk akun yang tidak ada di LDAP).
local svc_roles
svc_roles=$(python3 -c "
import json
print(' '.join(sorted(set(json.load(open('${CONFIG_DIR}/opa/data.json')).get('users', {}).values()))))" 2>/dev/null || echo "")
local g no_ldap="" svc_only=""
for g in ${opa_roles}; do
if ldap_entry_exists "${GROUPS_DN}" "(cn=${g})" cn; then
continue
fi
if [[ " ${svc_roles} " == *" ${g} "* ]]; then
svc_only+=" ${g}"
else
no_ldap+=" ${g}"
fi
done
if [[ -z "${no_ldap}" ]]; then
pass "Semua role OPA dapat diberikan (via grup LDAP atau pemetaan service account)"
[[ -n "${svc_only}" ]] && pass " role khusus service account:${svc_only}"
else
fail "Role OPA tanpa grup LDAP & tanpa pemetaan service:${no_ldap}"
fi
local sr_groups
sr_groups=$(grep -oE "'permitted_groups' = '[^']+'" "${SCRIPT_DIR}/bootstrap.sh" 2>/dev/null |
head -1 | sed "s/.*= '//; s/'//" | tr ',' ' ')
if [[ -n "${sr_groups}" ]]; then
local grp mismatch=""
for grp in ${sr_groups}; do
[[ " ${opa_roles} " == *" ${grp} "* ]] || mismatch+=" ${grp}"
done
if [[ -z "${mismatch}" ]]; then
pass "permitted_groups StarRocks ⊆ roles OPA"
else
warn "permitted_groups StarRocks memuat grup yang tidak ada di OPA:${mismatch}"
fi
else
warn "Tidak bisa membaca permitted_groups StarRocks dari bootstrap.sh"
fi
}
# F. client_credentials — TANPA admin: membuktikan klien ADA + secret SINKRON +
# audience mapper jalan (aud=account). Ini pengganti cek statis §A/§E saat IAM
# dikelola pihak lain (tidak ada akses admin). Butuh serviceAccountsEnabled=true.
check_client_credentials() {
echo -e "${BOLD}F. Keycloak — client_credentials (klien + secret + audience; tanpa admin)${NC}"
local pairs=("trino:${TRINO_CLIENT_SECRET}" "starrocks:${STARROCKS_CLIENT_SECRET}"
"airflow:${AIRFLOW_CLIENT_SECRET}" "openmetadata:${OPENMETADATA_CLIENT_SECRET}"
"grafana:${GRAFANA_CLIENT_SECRET}")
local pair cid secret tok aud
for pair in "${pairs[@]}"; do
cid="${pair%%:*}"; secret="${pair#*:}"
tok=$(curl -s -X POST "${KC}/realms/${REALM}/protocol/openid-connect/token" \
-d grant_type=client_credentials -d "client_id=${cid}" -d "client_secret=${secret}" 2>/dev/null |
python3 -c "import sys,json;print(json.load(sys.stdin).get('access_token',''))" 2>/dev/null || echo "")
if [[ -z "${tok}" ]]; then
fail "client_credentials '${cid}' GAGAL (klien tak ada / secret beda / service account OFF)"
continue
fi
aud=$(printf '%s' "${tok}" | python3 -c "
import sys,json,base64
t=sys.stdin.read().strip(); p=t.split('.')[1]; p+='='*(-len(p)%4)
d=json.loads(base64.urlsafe_b64decode(p)); a=d.get('aud') or []
print(','.join(a if isinstance(a,list) else [a]))" 2>/dev/null || echo "")
if [[ " ${aud} " == *" account "* ]]; then
pass "client_credentials '${cid}' OK + aud=account (mapper audience ada, secret sinkron)"
else
warn "client_credentials '${cid}' OK tapi aud tanpa 'account' (aud='${aud}') — cek audience mapper"
fi
done
}
# ─────────────────────────────────────────────────────────────────────────────
# Main
# ─────────────────────────────────────────────────────────────────────────────
main() {
print_section "Verifikasi Phase 1 — Kontrak IAM (Keycloak + LDAP)"
echo -e " Keycloak : ${CYAN}${KC}${NC} (realm ${REALM})"
echo -e " LDAP : ${CYAN}$(ldap_endpoint)${NC}"
echo ""
KC_TOKEN="$(kc_token)"
HAVE_ADMIN=1
if [[ -z "${KC_TOKEN}" ]]; then
HAVE_ADMIN=0
echo -e "${WARN} Tanpa akses admin Keycloak (${KC}) — lanjut mode READ-ONLY."
echo -e " ${INFO} Cek statis realm/role/user/client di Keycloak DILEWATI (didelegasikan ke admin IdP)."
echo -e " ${INFO} Tetap diperiksa: OIDC discovery · client_credentials (klien ada + secret sinkron +"
echo -e " ${INFO} aud=account) · LDAP · cross-check lokal (OPA/StarRocks)."
fi
check_keycloak_static
check_keycloak_users_login
check_client_credentials
check_client_secrets
check_ldap_structure
check_cross_reference
echo ""
echo -e "${BOLD}Ringkasan:${NC} ${GREEN}${PASS_N} PASS${NC} · ${RED}${FAIL_N} FAIL${NC} · ${YELLOW}${WARN_N} WARN${NC}"
if [[ ${FAIL_N} -gt 0 ]]; then
echo ""
echo -e "${BOLD}Yang gagal (${FAIL_N}):${NC}"
local _i=1 _m
for _m in "${FAIL_MSGS[@]}"; do
echo -e " ${RED}${_i}.${NC} ${_m}"
_i=$((_i + 1))
done
echo -e "${INFO} Perbaiki item di atas, lalu jalankan ulang: ./scripts/${0##*/}"
fi
if [[ ${FAIL_N} -eq 0 ]]; then
echo -e "${OK} ${GREEN}Kontrak IAM Phase 1 terpenuhi — aman lanjut ke fase berikutnya.${NC}"
exit 0
fi
echo -e "${ERR} Ada ${FAIL_N} gate yang gagal — perbaiki sebelum melanjutkan."
exit 1
}
main "$@"

Some files were not shown because too many files have changed in this diff Show More