97 lines
4.6 KiB
Markdown
97 lines
4.6 KiB
Markdown
# 🌊 MERICA Pipeline Bundles (Release Delivery v1.0)
|
|
|
|
> **Declarative Lakehouse Pipelines, dbt Transformations, and Data Contracts**
|
|
> *Repositori Resmi Definisi Data Pipeline untuk Pengguna / Data Engineer (Self-Hosted Git Delivery).*
|
|
|
|
---
|
|
|
|
## 📌 1. Ikhtisar Repositori
|
|
|
|
Repositori **`merica-pipeline`** adalah **Single Source of Truth (SSOT)** untuk seluruh definisi pipeline data, model transformasi dbt, aturan tata kelola metadata, dan pengujian kualitas data pada platform **MERICA Data Lakehouse**.
|
|
|
|
Melalui arsitektur **Git DAG Bundles (AIP-66)** Apache Airflow 3, setiap berkas konfigurasi YAML di repositori ini secara otomatis disinkronkan ke Airflow runtime dalam interval **30 detik** tanpa memerlukan restart layanan atau penyalinan berkas manual ke server engine.
|
|
|
|
---
|
|
|
|
## 📦 2. Struktur Direktori Repositori
|
|
|
|
```text
|
|
release/merica-pipeline/
|
|
├── configs/ # Deklarasi pipeline ingestion kanonikal (YAML)
|
|
│ ├── example_permits_medallion_dbt_postgres.yaml # Referensi End-to-End Medallion dbt
|
|
│ ├── ingestion_api_spku_air_quality.yaml # IoT Real-time Air Quality (Sliding Window)
|
|
│ ├── ingestion_db_environmental_permits_cdc.yaml # DB Transactional CDC (Sling)
|
|
│ └── ingestion_gws_waste_reports.yaml # Google Workspace Spreadsheet Sanitizer
|
|
├── dbt/ # Transformasi Medallion data mart (SQL)
|
|
│ └── models/
|
|
│ ├── silver/ # Model pembersihan & deduplikasi
|
|
│ └── gold/ # Model data mart & kepatuhan pelaporan
|
|
├── metadata/ # JSON metadata tata kelola OpenMetadata
|
|
│ ├── example_permits_medallion_dbt_postgres.json
|
|
│ └── ingestion_api_spku_air_quality.json
|
|
├── schemas/ # Kontrak skema JSON & validator
|
|
├── samples/ # Sampel dummy dataset untuk pengujian
|
|
│ ├── waste_reports_raw.csv # Sampel CSV kotor
|
|
│ ├── spku_realtime.json # Sampel JSON sensor IoT
|
|
│ └── seed_amdal_permits.sql # SQL seed data transaksional
|
|
├── docs/ # Panduan penulisan pipeline untuk Data Engineer
|
|
│ ├── pipeline_specifications.md
|
|
│ ├── dbt_modeling_guide.md
|
|
│ └── adding_new_pipeline.md
|
|
└── README.md # Panduan utama repositori pipeline
|
|
```
|
|
|
|
---
|
|
|
|
## 🎯 3. Katalog 4 Pipeline Bawaan (Canonical Use Cases)
|
|
|
|
| Nama Pipeline | File Konfigurasi | Engine & Operator | Lapisan Medallion |
|
|
| :--- | :--- | :--- | :--- |
|
|
| **AMDAL CDC Ingestion** | `ingestion_db_environmental_permits_cdc.yaml` | Sling CLI Log-Based CDC | PostgreSQL ➔ Bronze S3 (Parquet) |
|
|
| **GWS Waste Reports** | `ingestion_gws_waste_reports.yaml` | GWS Sanitizer Operator | Google Sheets ➔ Bronze Parquet |
|
|
| **SPKU IoT Air Quality** | `ingestion_api_spku_air_quality.yaml` | REST API Sliding Window | HTTP REST ➔ Bronze S3 (JSON) |
|
|
| **Reference Medallion dbt** | `example_permits_medallion_dbt_postgres.yaml` | Sling + Great Expectations + dbt | Bronze ➔ Silver ➔ Gold Marts |
|
|
|
|
---
|
|
|
|
## 🛠️ 4. Panduan Menambahkan Pipeline Baru
|
|
|
|
Untuk menambahkan pipeline baru ke dalam sistem:
|
|
|
|
1. **Buat Berkas Konfigurasi YAML di Direktori `configs/`**:
|
|
Contoh: `configs/ingestion_my_dataset.yaml`
|
|
```yaml
|
|
pipeline_id: "ingestion_my_dataset"
|
|
description: "Pipeline Ingestion Data Operasional Baru"
|
|
schedule_interval: "@daily"
|
|
tags: ["operational", "bronze"]
|
|
source:
|
|
type: "postgres"
|
|
connection_id: "conn_postgres_source_db"
|
|
table: "public.my_table"
|
|
target:
|
|
bucket: "dlh-bronze"
|
|
format: "parquet"
|
|
```
|
|
|
|
2. **(Opsional) Tambahkan Transformasi dbt di Direktori `dbt/models/`**:
|
|
Tambahkan file SQL di `silver/` atau `gold/` untuk mendefinisikan agregasi analitik.
|
|
|
|
3. **Commit & Push ke Server Git**:
|
|
```bash
|
|
git add configs/ingestion_my_dataset.yaml
|
|
git commit -m "feat(pipeline): add new operational dataset pipeline"
|
|
git push origin main
|
|
```
|
|
|
|
4. **Verifikasi di Airflow Webserver (Port 14080)**:
|
|
Dalam waktu maksimum 30 detik, DAG baru dengan ID `dag_ingestion_my_dataset` akan muncul otomatis di antarmuka Airflow tanpa restart scheduler!
|
|
|
|
---
|
|
|
|
## 📚 5. Dokumentasi Teknis Terkait
|
|
|
|
* 📑 [docs/pipeline_specifications.md](docs/pipeline_specifications.md) — Spesifikasi lengkap schema sintaksis YAML pipeline.
|
|
* 🧊 [docs/dbt_modeling_guide.md](docs/dbt_modeling_guide.md) — Panduan pemodelan data Medallion (Bronze, Silver, Gold).
|
|
* 🚀 [docs/adding_new_pipeline.md](docs/adding_new_pipeline.md) — Tutorial langkah-demi-langkah bagi Data Engineer.
|