201 lines
8.8 KiB
Markdown
201 lines
8.8 KiB
Markdown
|
# 📘 Panduan Integrasi MERICA Standalone dengan Existing Apache Airflow
|
||
|
|
|
||
|
|
> **Navigasi Cepat**: [🏠 Pusat Dokumentasi](index.md) | [🏛️ Arsitektur Platform](architecture.md) | [📦 Panduan Deployment](deployment_guide.md) | [🎨 Panduan Pengguna Canvas](user_guide.md)
|
||
|
|
|
||
|
|
Dokumen panduan ini dirancang khusus bagi **Data Engineer, Platform Engineer, dan DevOps** yang ingin menginstal **MERICA secara mandiri (*standalone appliance*)** dan menghubungkannya ke kluster **Apache Airflow yang sudah ada (*existing enterprise cluster*)**.
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## 🏛️ 1. Gambaran Umum Arsitektur
|
||
|
|
|
||
|
|
MERICA (*Managed Engine for Robust Ingestion & Cloud Airflow*) berfungsi sebagai **Visual Low-Code DAG Builder** berbasis web. MERICA tidak mengeksekusi pipeline berat secara langsung; tugas eksekusi, pemantauan, dan retry task sepenuhnya didelegasikan ke kluster Apache Airflow eksternal Anda.
|
||
|
|
|
||
|
|
```mermaid
|
||
|
|
flowchart LR
|
||
|
|
subgraph STANDALONE ["MERICA Standalone Host (Port 14000)"]
|
||
|
|
UI["ReactFlow Canvas UI"]
|
||
|
|
API["FastAPI Backend Gateway"]
|
||
|
|
PIPES["Mounted Pipelines SSOT\n(./pipelines/)"]
|
||
|
|
UI --- API
|
||
|
|
API <--> PIPES
|
||
|
|
end
|
||
|
|
|
||
|
|
subgraph AIRFLOW_HOST ["Existing Apache Airflow Cluster"]
|
||
|
|
AF_WEBSERVER["Airflow Web UI / API\n(Port 14080 / Domain)"]
|
||
|
|
AF_SCHEDULER["Airflow Scheduler & Workers"]
|
||
|
|
AF_DAG_PROC["DAG Processor\n(AIP-66 Git DAG Bundles)"]
|
||
|
|
AF_SCHEDULER --- AF_DAG_PROC
|
||
|
|
end
|
||
|
|
|
||
|
|
STANDALONE -.->|"Deploy DAG (Git / YAML)"| AIRFLOW_HOST
|
||
|
|
API -.->|"Direct Link Out"| AF_WEBSERVER
|
||
|
|
```
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## 🚀 2. Dua Metode Penyiapan Sisi Existing Airflow
|
||
|
|
|
||
|
|
Untuk dapat menjalankan pipeline modern yang dirancang di MERICA (termasuk dbt Medallion, deduplikasi DuckDB, dan Shift-Left Quality Gates), worker dan DAG processor Airflow Anda membutuhkan beberapa pustaka tambahan (seperti `astronomer-cosmos` dan `dbt-postgres`).
|
||
|
|
|
||
|
|
Pilih salah satu dari **2 Metode** berikut sesuai kebijakan infrastruktur Anda:
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
### METODE 1: Modifikasi Langsung Tanpa Build Ulang (*Zero-Build / Runtime Injection*)
|
||
|
|
|
||
|
|
> [!TIP]
|
||
|
|
> **Gunakan metode ini jika**: Anda ingin uji coba cepat, tidak memiliki akses untuk mem-build image di private registry, atau dilarang me-rebuild Dockerfile Airflow existing.
|
||
|
|
|
||
|
|
#### Opsi 1A: Injeksi Runtime Otomatis via `_PIP_ADDITIONAL_REQUIREMENTS` (Rekomendasi Persisten)
|
||
|
|
Image resmi Apache Airflow memiliki hook entrypoint otomatis. Pustaka akan terpasang setiap kali kontainer dinyalakan tanpa perlu build Dockerfile:
|
||
|
|
1. Buka berkas `.env` atau `docker-compose.yml` di server Airflow existing Anda.
|
||
|
|
2. Tambahkan variabel lingkungan ini pada service Airflow (`airflow-scheduler`, `airflow-dag-processor`, dan `airflow-worker`):
|
||
|
|
```yaml
|
||
|
|
environment:
|
||
|
|
_PIP_ADDITIONAL_REQUIREMENTS: "astronomer-cosmos==1.15.1 dbt-postgres==1.11.0 apache-airflow-providers-git==1.0.0 duckdb==1.1.3 pyiceberg[s3fs,pyarrow]==0.8.0"
|
||
|
|
```
|
||
|
|
3. Restart service Airflow:
|
||
|
|
```bash
|
||
|
|
docker compose restart airflow-scheduler airflow-dag-processor
|
||
|
|
```
|
||
|
|
|
||
|
|
#### Opsi 1B: `docker exec` Langsung ke Kontainer (Aktif dalam 15 Detik Tanpa Restart)
|
||
|
|
Jika kontainer Airflow sedang running dalam jam kerja dan Anda dilarang me-restart kontainer:
|
||
|
|
```bash
|
||
|
|
# 1. Install di Scheduler (user airflow)
|
||
|
|
docker exec -u airflow <container_scheduler> pip install --no-cache-dir \
|
||
|
|
astronomer-cosmos==1.15.1 dbt-postgres==1.11.0 apache-airflow-providers-git==1.0.0
|
||
|
|
|
||
|
|
# 2. Install di DAG Processor
|
||
|
|
docker exec -u airflow <container_dag_processor> pip install --no-cache-dir \
|
||
|
|
astronomer-cosmos==1.15.1 dbt-postgres==1.11.0 apache-airflow-providers-git==1.0.0
|
||
|
|
```
|
||
|
|
|
||
|
|
#### Verifikasi Keberhasilan Instalasi:
|
||
|
|
```bash
|
||
|
|
docker exec <container_scheduler> python -c "import cosmos; print('✅ Cosmos Version:', cosmos.__version__)"
|
||
|
|
```
|
||
|
|
|
||
|
|
#### Bagaimana Menghubungkan DAGs & Custom Operators Tanpa Build Ulang?
|
||
|
|
Seluruh operator custom (`helpers/duckdb_operator.py`, `helpers/dbt_trino_operator.py`, `helpers/ge_lakehouse_operator.py`, dll.) serta `dag_factory.py` **tidak perlu dibake ke image Airflow**.
|
||
|
|
Cukup sambungkan folder `dags/` ke Airflow Anda:
|
||
|
|
* **Via AIP-66 Git DAG Bundles (Airflow 3)**: Airflow otomatis meng-clone dari repositori `merica-pipelines.git`.
|
||
|
|
* **Via Shared Volume Mount**: Mount direktori `dags/` ke `/opt/airflow/dags` di kontainer Airflow.
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
### METODE 2: Build Ulang Image Resmi (*Official Baked Image*)
|
||
|
|
|
||
|
|
> [!IMPORTANT]
|
||
|
|
> **Gunakan metode ini jika**: Server produksi organisasi Anda berada di jaringan terisolasi (*air-gapped / tanpa akses internet luar*) atau standar DevOps mewajibkan seluruh dependensi paten di dalam image registry resmi.
|
||
|
|
|
||
|
|
#### 1. Buat Dockerfile Airflow Resmi
|
||
|
|
Buat berkas `Dockerfile.airflow` untuk registry organisasi Anda:
|
||
|
|
|
||
|
|
```dockerfile
|
||
|
|
FROM registry.sicabe.com/airflow:3.3.2-om203-ol
|
||
|
|
|
||
|
|
LABEL maintainer="BPKP Data Lakehouse Team <dlh@bpkp.go.id>"
|
||
|
|
LABEL description="Official Airflow 3 Engine Image with Baked MERICA Operators & Cosmos"
|
||
|
|
LABEL version="3.3.2-merica-v1.1"
|
||
|
|
|
||
|
|
USER airflow
|
||
|
|
|
||
|
|
# Instalasi dependensi engine MERICA langsung ke layer image
|
||
|
|
RUN pip install --no-cache-dir \
|
||
|
|
astronomer-cosmos==1.15.1 \
|
||
|
|
dbt-postgres==1.11.0 \
|
||
|
|
apache-airflow-providers-git==1.0.0 \
|
||
|
|
duckdb==1.1.3 \
|
||
|
|
pyiceberg[s3fs,pyarrow]==0.8.0 \
|
||
|
|
pyarrow==17.0.0 \
|
||
|
|
s3fs==2024.9.0 \
|
||
|
|
fsspec==2024.9.0 \
|
||
|
|
dlt[duckdb,filesystem]==1.6.1
|
||
|
|
|
||
|
|
# Validasi kompilasi paket di dalam image
|
||
|
|
RUN python -c "import cosmos; import duckdb; import pyiceberg; print('✅ All MERICA Dependencies Baked!')"
|
||
|
|
```
|
||
|
|
|
||
|
|
#### 2. Build dan Push ke Private Registry
|
||
|
|
```bash
|
||
|
|
docker build -t registry.sicabe.com/airflow:3.3.2-merica-v1.1 -f Dockerfile.airflow .
|
||
|
|
docker push registry.sicabe.com/airflow:3.3.2-merica-v1.1
|
||
|
|
```
|
||
|
|
|
||
|
|
#### 3. Terapkan Image Baru di Kluster Airflow
|
||
|
|
Ubah tag image pada deployment Airflow:
|
||
|
|
```yaml
|
||
|
|
image: registry.sicabe.com/airflow:3.3.2-merica-v1.1
|
||
|
|
```
|
||
|
|
Lalu deploy ulang kontainer (`docker compose up -d` atau `kubectl rollout restart deployment airflow`).
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## 🎨 3. Fitur "Anti-Blank Canvas" (Jaminan DAGs List Langsung Muncul)
|
||
|
|
|
||
|
|
### Kenapa Fitur Ini Ada?
|
||
|
|
MERICA adalah backend *stateless file-based*. Jika kontainer berdiri sendiri tanpa mount volume, sistem akan membaca folder kosong sehingga daftar pipeline di halaman web akan kosong (*blank*), yang sering kali menimbulkan kebingungan bagi pengguna baru.
|
||
|
|
|
||
|
|
### Cara Kerja Starter Seed Pipelines:
|
||
|
|
Di dalam paket rilis `release/merica/`, disediakan direktori `./pipelines/`:
|
||
|
|
* `pipelines/configs/`: Menyimpan spesifikasi YAML Airflow (`e2e_modernized_ingestion_flow.yaml`, `e2e_medallion_permits_flow.yaml`, `e2e_orders_master_medallion_flow.yaml`).
|
||
|
|
* `pipelines/metadata/`: Menyimpan data visual layout node & edge ReactFlow (`*.json`).
|
||
|
|
* `pipelines/dbt/`: Menyimpan model SQL dbt Medallion.
|
||
|
|
|
||
|
|
Saat kontainer `merica` dijalankan via `docker compose up -d`, direktori `./pipelines` ini otomatis di-mount ke `/app/merica-pipelines`. Begitu browser dibuka di `http://localhost:14000`, **3 pipeline data lakehouse modern langsung muncul secara visual** dan siap dieksplorasi atau diduplikasi!
|
||
|
|
|
||
|
|
### Cara Mengganti Starter Pipelines dengan Git Organisasi Anda (Opsi GitOps SSOT):
|
||
|
|
Jika tim Anda menggunakan pendekatan GitOps di mana MERICA dan Airflow 3 tersinkronisasi langsung ke repositori Git yang sama (`merica-pipelines.git`):
|
||
|
|
1. Clone repositori Git di samping folder `release/merica`:
|
||
|
|
```bash
|
||
|
|
git clone https://repo.sicabe.com/merica-dlh/merica-pipelines.git ../merica-pipelines
|
||
|
|
```
|
||
|
|
2. Pada berkas `.env`, aktifkan variabel yang sudah disediakan di template (cukup buka tanda komentar):
|
||
|
|
```bash
|
||
|
|
# Ganti dari ./pipelines ke path clone repositori Git Anda:
|
||
|
|
PIPELINES_DIR=../merica-pipelines
|
||
|
|
```
|
||
|
|
3. Parameter referensi Git DAG Bundle di sisi Airflow 3 (`AIP-66`):
|
||
|
|
```bash
|
||
|
|
GIT_PIPELINE_BUNDLE_NAME=merica-pipeline
|
||
|
|
GIT_PIPELINE_REPO_URL=https://repo.sicabe.com/merica-dlh/merica-pipelines.git
|
||
|
|
GIT_PIPELINE_REPO_BRANCH=main
|
||
|
|
GIT_PIPELINE_REPO_SUBDIR=dags
|
||
|
|
AIRFLOW_DAG_BUNDLE_REFRESH_INTERVAL=30
|
||
|
|
```
|
||
|
|
4. Restart MERICA: `docker compose up -d`. Seluruh pipeline tim Anda akan langsung tersinkronkan secara *hot-reload*!
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## ⚡ 4. Panduan Operasional Cepat (Runbook 3 Menit)
|
||
|
|
|
||
|
|
### Langkah 1: Siapkan Konfigurasi Lingkungan
|
||
|
|
```bash
|
||
|
|
cd release/merica
|
||
|
|
cp .env.example .env
|
||
|
|
```
|
||
|
|
Sesuaikan parameter `AIRFLOW_WEB_URL` dan `POSTGRES_HOST` mengarah ke alamat server Airflow & Database Anda.
|
||
|
|
|
||
|
|
### Langkah 2: Jalankan Script Setup Sekali Klik
|
||
|
|
```bash
|
||
|
|
bash ./setup.sh
|
||
|
|
```
|
||
|
|
|
||
|
|
### Langkah 3: Buka Antarmuka MERICA
|
||
|
|
Akses antarmuka web di browser:
|
||
|
|
* **URL**: [`http://localhost:14000`](http://localhost:14000)
|
||
|
|
* **Username**: `admin`
|
||
|
|
* **Password**: `admin`
|
||
|
|
|
||
|
|
Daftar DAGs Anda akan langsung tampil secara visual dan siap digunakan!
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## 🧭 Navigasi Dokumen Terkait
|
||
|
|
|
||
|
|
* Pahami Rancang Bangun: [🏛️ Arsitektur Platform](architecture.md)
|
||
|
|
* Cara Install Appliance: [📦 Panduan Deployment & Build](deployment_guide.md)
|
||
|
|
* Panduan Mendesain Pipeline di Canvas: [🎨 Panduan Pengguna Canvas](user_guide.md)
|
||
|
|
* Kembali ke Indeks: [🏠 Pusat Dokumentasi](index.md)
|