feat: initial commit for merica-pipeline delivery

2026-10-08 19:33:15 +07:00
commit 42fedab752
46 changed files with 5484 additions and 0 deletions
@@ -0,0 +1,96 @@
+# 🌊 MERICA Pipeline Bundles (Release Delivery v1.0)
+
+> **Declarative Lakehouse Pipelines, dbt Transformations, and Data Contracts**  
+> *Repositori Resmi Definisi Data Pipeline untuk Pengguna / Data Engineer (Self-Hosted Git Delivery).*
+
+---
+
+## 📌 1. Ikhtisar Repositori
+
+Repositori **`merica-pipeline`** adalah **Single Source of Truth (SSOT)** untuk seluruh definisi pipeline data, model transformasi dbt, aturan tata kelola metadata, dan pengujian kualitas data pada platform **MERICA Data Lakehouse**.
+
+Melalui arsitektur **Git DAG Bundles (AIP-66)** Apache Airflow 3, setiap berkas konfigurasi YAML di repositori ini secara otomatis disinkronkan ke Airflow runtime dalam interval **30 detik** tanpa memerlukan restart layanan atau penyalinan berkas manual ke server engine.
+
+---
+
+## 📦 2. Struktur Direktori Repositori
+
+```text
+release/merica-pipeline/
+├── configs/                    # Deklarasi pipeline ingestion kanonikal (YAML)
+│   ├── example_permits_medallion_dbt_postgres.yaml # Referensi End-to-End Medallion dbt
+│   ├── ingestion_api_spku_air_quality.yaml          # IoT Real-time Air Quality (Sliding Window)
+│   ├── ingestion_db_environmental_permits_cdc.yaml  # DB Transactional CDC (Sling)
+│   └── ingestion_gws_waste_reports.yaml             # Google Workspace Spreadsheet Sanitizer
+├── dbt/                        # Transformasi Medallion data mart (SQL)
+│   └── models/
+│       ├── silver/             # Model pembersihan & deduplikasi
+│       └── gold/               # Model data mart & kepatuhan pelaporan
+├── metadata/                   # JSON metadata tata kelola OpenMetadata
+│   ├── example_permits_medallion_dbt_postgres.json
+│   └── ingestion_api_spku_air_quality.json
+├── schemas/                    # Kontrak skema JSON & validator
+├── samples/                    # Sampel dummy dataset untuk pengujian
+│   ├── waste_reports_raw.csv   # Sampel CSV kotor
+│   ├── spku_realtime.json      # Sampel JSON sensor IoT
+│   └── seed_amdal_permits.sql  # SQL seed data transaksional
+├── docs/                       # Panduan penulisan pipeline untuk Data Engineer
+│   ├── pipeline_specifications.md
+│   ├── dbt_modeling_guide.md
+│   └── adding_new_pipeline.md
+└── README.md                   # Panduan utama repositori pipeline
+```
+
+---
+
+## 🎯 3. Katalog 4 Pipeline Bawaan (Canonical Use Cases)
+
+| Nama Pipeline | File Konfigurasi | Engine & Operator | Lapisan Medallion |
+| :--- | :--- | :--- | :--- |
+| **AMDAL CDC Ingestion** | `ingestion_db_environmental_permits_cdc.yaml` | Sling CLI Log-Based CDC | PostgreSQL ➔ Bronze S3 (Parquet) |
+| **GWS Waste Reports** | `ingestion_gws_waste_reports.yaml` | GWS Sanitizer Operator | Google Sheets ➔ Bronze Parquet |
+| **SPKU IoT Air Quality** | `ingestion_api_spku_air_quality.yaml` | REST API Sliding Window | HTTP REST ➔ Bronze S3 (JSON) |
+| **Reference Medallion dbt** | `example_permits_medallion_dbt_postgres.yaml` | Sling + Great Expectations + dbt | Bronze ➔ Silver ➔ Gold Marts |
+
+---
+
+## 🛠️ 4. Panduan Menambahkan Pipeline Baru
+
+Untuk menambahkan pipeline baru ke dalam sistem:
+
+1. **Buat Berkas Konfigurasi YAML di Direktori `configs/`**:
+   Contoh: `configs/ingestion_my_dataset.yaml`
+   ```yaml
+   pipeline_id: "ingestion_my_dataset"
+   description: "Pipeline Ingestion Data Operasional Baru"
+   schedule_interval: "@daily"
+   tags: ["operational", "bronze"]
+   source:
+     type: "postgres"
+     connection_id: "conn_postgres_source_db"
+     table: "public.my_table"
+   target:
+     bucket: "dlh-bronze"
+     format: "parquet"
+   ```
+
+2. **(Opsional) Tambahkan Transformasi dbt di Direktori `dbt/models/`**:
+   Tambahkan file SQL di `silver/` atau `gold/` untuk mendefinisikan agregasi analitik.
+
+3. **Commit & Push ke Server Git**:
+   ```bash
+   git add configs/ingestion_my_dataset.yaml
+   git commit -m "feat(pipeline): add new operational dataset pipeline"
+   git push origin main
+   ```
+
+4. **Verifikasi di Airflow Webserver (Port 14080)**:
+   Dalam waktu maksimum 30 detik, DAG baru dengan ID `dag_ingestion_my_dataset` akan muncul otomatis di antarmuka Airflow tanpa restart scheduler!
+
+---
+
+## 📚 5. Dokumentasi Teknis Terkait
+
+* 📑 [docs/pipeline_specifications.md](docs/pipeline_specifications.md) — Spesifikasi lengkap schema sintaksis YAML pipeline.
+* 🧊 [docs/dbt_modeling_guide.md](docs/dbt_modeling_guide.md) — Panduan pemodelan data Medallion (Bronze, Silver, Gold).
+* 🚀 [docs/adding_new_pipeline.md](docs/adding_new_pipeline.md) — Tutorial langkah-demi-langkah bagi Data Engineer.