docs: add real-world production deployment walkthrough section
This commit is contained in:
+1
-1
@@ -28,7 +28,7 @@ flowchart TD
|
||||
| :---: | :--- | :--- | :--- |
|
||||
| **1** | [**Arsitektur Platform**](./architecture.md) | Software Architect, Data Architect | Rancang bangun Single-Port Appliance (Port 14000), isolasi perimeter loopback, arsitektur modul stateless, dan integrasi GitOps Airflow 3. |
|
||||
| **2** | [**Panduan Deployment & Build**](./deployment_guide.md) | DevOps, System Administrator | Spesifikasi hardware ultra-ringan (~100MB RAM), instalasi sekali-klik via `setup.sh`, build image mandiri, dan upgrade runbook. |
|
||||
| **3** | [**Panduan Integrasi Airflow External**](./panduan_integrasi_existing_airflow.md) | Data Platform Engineer, DevOps | Cara menghubungkan MERICA ke Airflow existing via **Metode 1 (Tanpa Rebuild)** atau **Metode 2 (Build Ulang Image Resmi)**, serta pengelolaan starter pipelines. |
|
||||
| **3** | [**Panduan Integrasi Airflow External**](./panduan_integrasi_existing_airflow.md) | Data Platform Engineer, DevOps | Metode penyiapan Airflow existing (Tanpa Rebuild vs Build Ulang), starter pipelines, serta **Studi Kasus Nyata di Server Produksi**. |
|
||||
| **4** | [**Panduan Pengguna (User Guide)**](./user_guide.md) | Data Engineer, Analytics Engineer | Panduan visual drag-and-drop di Canvas, katalog komponen lakehouse, GWS discovery, DB introspection & safe preview, dlt REST API, dan Shift-Left dbt testing. |
|
||||
|
||||
---
|
||||
|
||||
@@ -192,6 +192,128 @@ Daftar DAGs Anda akan langsung tampil secara visual dan siap digunakan!
|
||||
|
||||
---
|
||||
|
||||
## 🏢 5. Studi Kasus Implementasi Nyata (Real-World Deployment Walkthrough)
|
||||
|
||||
Bagian ini menyajikan contoh implementasi nyata ketika Anda menghubungkan MERICA ke server produksi yang telah menjalankan Apache Airflow dan stack data dengan port kustom (non-standar).
|
||||
|
||||
### Skenario Lapangan:
|
||||
Sebuah server telah menjalankan kluster Airflow dan infrastruktur data dengan kondisi:
|
||||
* **Kontainer Airflow**: `sicabe-airflow-scheduler`, `sicabe-airflow-webserver` (Port Host `12082:8080`), dan `sicabe-airflow-dag-processor` sudah berstatus `Up 7 days` (tidak boleh restart lama).
|
||||
* **Penyimpanan Objek MinIO S3**: Port Host `8443`.
|
||||
* **Katalog IAM Keycloak**: Port Host `12080`.
|
||||
* **PostgreSQL MetaDB Shared**: Berjalan pada host dengan port standar atau custom.
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
subgraph MERICA_CONTAINER ["MERICA Standalone (Port 14000)"]
|
||||
UI["Web UI Canvas\n(Port 14000)"]
|
||||
BE["FastAPI Engine\n(127.0.0.1:8000)"]
|
||||
UI --- BE
|
||||
end
|
||||
|
||||
subgraph EXISTING_STACK ["Existing Data Stack (Server SiCABE / Enterprise)"]
|
||||
AF_WEB["Airflow Webserver\n(Port Host :12082)"]
|
||||
AF_SCHED["Airflow Scheduler\n(Injected Libs)"]
|
||||
AF_PROC["DAG Processor\n(Injected Libs)"]
|
||||
S3["MinIO S3\n(Port Host :8443)"]
|
||||
PG["PostgreSQL MetaDB\n(Port :5432 / Custom)"]
|
||||
end
|
||||
|
||||
BE -->|REST API :12082| AF_WEB
|
||||
BE -->|Introspection & Safe Preview| PG
|
||||
BE -->|Storage Connection| S3
|
||||
AF_SCHED -.->|Eksekusi Task Ingestion| S3
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Langkah Praktis Penerapan di Server:
|
||||
|
||||
#### 1. Inspeksi Kontainer & Port Aktif
|
||||
Periksa nama kontainer Airflow dan pemetaan port host di server Anda:
|
||||
```bash
|
||||
docker ps --filter "name=airflow" --format "table {{.Names}}\t{{.Image}}\t{{.Status}}\t{{.Ports}}"
|
||||
```
|
||||
*Catat nama kontainer scheduler dan dag processor serta port host webserver (misal: `0.0.0.0:12082->8080/tcp`).*
|
||||
|
||||
#### 2. Injeksi Pustaka Engine ke Kontainer Berjalan (Tanpa Restart)
|
||||
Gunakan `docker exec` untuk memasang dependensi engine (`astronomer-cosmos`, `dbt-postgres`, `duckdb`, `pyiceberg`) langsung ke worker/scheduler tanpa mematikan layanan:
|
||||
```bash
|
||||
# Injeksi ke Scheduler
|
||||
docker exec -u airflow <nama_kontainer_scheduler> pip install --no-cache-dir \
|
||||
astronomer-cosmos==1.15.1 dbt-postgres==1.11.0 apache-airflow-providers-git==1.0.0 \
|
||||
duckdb==1.1.3 pyiceberg[s3fs,pyarrow]==0.8.0 dlt[duckdb,filesystem]==1.6.1
|
||||
|
||||
# Injeksi ke DAG Processor
|
||||
docker exec -u airflow <nama_kontainer_dag_processor> pip install --no-cache-dir \
|
||||
astronomer-cosmos==1.15.1 dbt-postgres==1.11.0 apache-airflow-providers-git==1.0.0 \
|
||||
duckdb==1.1.3 pyiceberg[s3fs,pyarrow]==0.8.0 dlt[duckdb,filesystem]==1.6.1
|
||||
|
||||
# Verifikasi instalasi berhasil
|
||||
docker exec <nama_kontainer_scheduler> python -c "import cosmos, duckdb, pyiceberg; print('✅ Airflow Engine Ready!')"
|
||||
```
|
||||
|
||||
#### 3. Penanganan Komunikasi Jaringan (Docker Networking)
|
||||
Tentukan bagaimana kontainer MERICA berkomunikasi dengan layanan existing:
|
||||
* **Kasus A — Satu Mesin (Single-Host)**:
|
||||
Gunakan `host.docker.internal` (didukung di Linux via `extra_hosts` pada compose) atau gunakan IP gateway Docker host (umumnya `172.17.0.1`).
|
||||
* **Kasus B — Beda Server (Multi-Node)**:
|
||||
Ganti `host.docker.internal` dengan IP private / LAN server Airflow Anda (contoh: `http://192.168.10.50:12082`).
|
||||
* **Kasus C — Shared Docker Network**:
|
||||
Jika ingin menggunakan nama kontainer langsung (contoh: `http://sicabe-airflow-webserver:8080`), hubungkan kontainer MERICA ke network yang sama dengan Airflow melalui `docker-compose.yml`:
|
||||
```yaml
|
||||
networks:
|
||||
default:
|
||||
name: sicabe_default
|
||||
external: true
|
||||
```
|
||||
|
||||
#### 4. Contoh Konfigurasi `.env` Riil yang Teruji
|
||||
Di direktori `/opt/merica`, buat berkas `.env` dengan konfigurasi seperti berikut:
|
||||
|
||||
```bash
|
||||
# --- MERICA Appliance ---
|
||||
MERICA_IMAGE=registry.sicabe.com/merica:1.0.0
|
||||
MERICA_PORT=14000
|
||||
ENVIRONMENT=production
|
||||
|
||||
# --- External Airflow Cluster (Port Kustom 12082) ---
|
||||
AIRFLOW_WEB_URL=http://host.docker.internal:12082
|
||||
AIRFLOW_API_URL=http://host.docker.internal:12082
|
||||
AIRFLOW_USERNAME=admin
|
||||
AIRFLOW_PASSWORD=1310cc9a318f75cf
|
||||
|
||||
# --- PostgreSQL MetaDB (Introspection & Preview) ---
|
||||
POSTGRES_HOST=host.docker.internal
|
||||
POSTGRES_PORT=5432
|
||||
POSTGRES_DB=airflow
|
||||
POSTGRES_USER=airflow
|
||||
POSTGRES_PASSWORD=a7bda1593a75d0ff0a8e51002c62c1f3
|
||||
|
||||
# --- External MinIO S3 (Port Kustom 8443) ---
|
||||
S3_ENDPOINT_URL=http://host.docker.internal:8443
|
||||
S3_ACCESS_KEY=minioadmin
|
||||
S3_SECRET_KEY=942a43b2a0e371545dfa0bff75381d49
|
||||
|
||||
# --- Pipeline Storage SSOT ---
|
||||
PIPELINES_DIR=./pipelines
|
||||
```
|
||||
|
||||
#### 5. Eksekusi & Validasi Konektivitas
|
||||
1. Nyalakan kontainer MERICA:
|
||||
```bash
|
||||
bash ./setup.sh
|
||||
# atau: docker compose up -d
|
||||
```
|
||||
2. Uji endpoint kesehatan:
|
||||
```bash
|
||||
curl -s http://localhost:14000/api/v1/health | jq .
|
||||
```
|
||||
3. Akses antarmuka web MERICA di browser: `http://<ip-server>:14000`.
|
||||
4. Anda dapat langsung menguji fitur **Database Introspection & Safe Preview** yang terhubung ke PostgreSQL, serta melihat tautan langsung ke Airflow Webserver di port `12082`.
|
||||
|
||||
---
|
||||
|
||||
## 🧭 Navigasi Dokumen Terkait
|
||||
|
||||
* Pahami Rancang Bangun: [🏛️ Arsitektur Platform](architecture.md)
|
||||
|
||||
Reference in New Issue
Block a user