merica-dlh/merica-pipelines
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(release): prepare clean public open-source MERICA pipeline bundles repository

Browse Source
This commit is contained in:
MERICA Admin
2026-10-09 01:01:44 +07:00
parent 42fedab752
commit 3aa0d15462
6 changed files with 468 additions and 71 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.gitignore
+59 -8
View File
@@ -1,16 +1,67 @@
# Python
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class
*.so
.pytest_cache/
# dbt
target/
dbt_packages/
logs/
# C extensions
*.so
# Local Environment
.env
# Distribution / packaging
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
*.egg-info/
.installed.cfg
*.egg
# Virtual environments
venv/
.venv/
ENV/
env/
# IDE & OS files
.idea/
.vscode/
*.swp
*.swo
*~
.DS_Store
Thumbs.db
# Environment variables & secrets
.env
.env.*
!.env.example
*.pem
*.key
*.token
credentials.json
service_account*.json
# Airflow local state & logs
airflow.db
unittests.cfg
airflow.cfg
logs/
*.log
# dbt temporary & target files
dbt/target/
dbt/dbt_packages/
dbt/logs/
# Great Expectations local stores
uncommitted/
great_expectations/uncommitted/
LICENSE
+120
View File
@@ -0,0 +1,120 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work.
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to use, reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
patent license to make, have made, use, offer to sell, sell, import,
and otherwise transfer the Work.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License.
END OF TERMS AND CONDITIONS
README.md
+85 -62
View File
@@ -1,96 +1,119 @@
# 🌊 MERICA Pipeline Bundles (Release Delivery v1.0)
# 📦 MERICA Pipeline Bundles (Public Open-Source Data Lakehouse Assets)
> **Declarative Lakehouse Pipelines, dbt Transformations, and Data Contracts**
> *Repositori Resmi Definisi Data Pipeline untuk Pengguna / Data Engineer (Self-Hosted Git Delivery).*
> **Repositori Resmi**: **Single Source of Truth (SSOT)** Aset Konfigurasi Pipeline, Model dbt Medallion, Metadata, & Custom Operators untuk **MERICA Data Lakehouse**
> **Status**: **General Availability (GA) v1.0 — Production Ready & Open-Source**
> **Standar Integrasi**: Apache Airflow 3 Native Git DAG Bundles (`GitDagBundle` AIP-66) & dbt Core
---
## 📌 1. Ikhtisar Repositori
## 🏛️ 1. Ikhtisar Repositori
Repositori **`merica-pipeline`** adalah **Single Source of Truth (SSOT)** untuk seluruh definisi pipeline data, model transformasi dbt, aturan tata kelola metadata, dan pengujian kualitas data pada platform **MERICA Data Lakehouse**.
Repositori **`merica-pipeline`** adalah **Single Source of Truth (SSOT)** publik yang memisahkan seluruh logika bisnis data pipelines dari runtime platform MERICA Studio dan Airflow Engine.
Melalui arsitektur **Git DAG Bundles (AIP-66)** Apache Airflow 3, setiap berkas konfigurasi YAML di repositori ini secara otomatis disinkronkan ke Airflow runtime dalam interval **30 detik** tanpa memerlukan restart layanan atau penyalinan berkas manual ke server engine.
Dengan arsitektur **Git DAG Bundles (AIP-66)** Apache Airflow 3:
* **Zero-Credential / Anonymous Clone**: Karena repositori ini bersifat publik/open, Airflow 3 DAG Processor dapat melakukan kloning dan sinkronisasi otomatis tanpa memerlukan username, password, atau personal access token (PAT).
* **Hot-Reload Tanpa Restart**: Setiap berkas konfigurasi YAML di repositori ini secara otomatis disinkronkan ke Airflow runtime dalam interval **30 detik** tanpa mematikan atau me-restart scheduler/webserver.
* **Canvas Interaktif MERICA Studio**: Konfigurasi YAML di repositori ini langsung dibaca dan divisualisasikan menjadi node-edge pipeline di antarmuka web MERICA Studio Canvas.
---
## 📦 2. Struktur Direktori Repositori
## 📁 2. Struktur Direktori Repositori
```text
release/merica-pipeline/
├── configs/ # Deklarasi pipeline ingestion kanonikal (YAML)
│ ├── example_permits_medallion_dbt_postgres.yaml # Referensi End-to-End Medallion dbt
│ ├── ingestion_api_spku_air_quality.yaml # IoT Real-time Air Quality (Sliding Window)
merica-pipeline/
├── LICENSE # Lisensi Apache 2.0 Open-Source
├── README.md # Dokumentasi utama repositori
├── RELEASE_NOTES.md # Catatan rilis resmi v1.0 GA
├── TESTING.md # Panduan pengujian lokal & CI/CD
├── requirements.txt # Dependensi testing Python (pytest, ruff)
├── configs/ # Konfigurasi pipeline deklaratif (*.yaml)
│ ├── example_permits_medallion_dbt_postgres.yaml # Referensi E2E Medallion dbt
│ ├── ingestion_api_spku_air_quality.yaml # IoT Real-time Air Quality
│ ├── ingestion_db_environmental_permits_cdc.yaml # DB Transactional CDC (Sling)
│ └── ingestion_gws_waste_reports.yaml # Google Workspace Spreadsheet Sanitizer
├── dbt/ # Transformasi Medallion data mart (SQL)
│ └── ingestion_gws_waste_reports.yaml # Google Sheets Extractor & Sanitizer
├── dags/ # Airflow 3 DAG Factory & Custom Operators
│ ├── dag_factory.py # Dynamic DAG generator berbasis YAML deklaratif
│ └── helpers/ # Multi-engine execution operators
│ ├── base_lakehouse_operator.py
│ ├── duckdb_operator.py
│ ├── pyiceberg_operator.py
│ ├── pyspark_operator.py
│ ├── ge_lakehouse_operator.py
│ ├── dbt_trino_operator.py
│ ├── sling_helper.py
│ ├── storage_helper.py
│ ├── api_helper.py
│ ├── gws_helper.py
│ ├── merica_alert_hook.py
│ └── sitomat_callback_operator.py
├── dbt/ # Model SQL dbt Core Medallion
│ ├── dbt_project.yml # Konfigurasi dbt project
│ ├── profiles.yml # Koneksi profile dbt
│ └── models/
│ ├── silver/ # Model pembersihan & deduplikasi
│ └── gold/ # Model data mart & kepatuhan pelaporan
├── metadata/ # JSON metadata tata kelola OpenMetadata
│ ├── example_permits_medallion_dbt_postgres.json
│ └── ingestion_api_spku_air_quality.json
├── schemas/ # Kontrak skema JSON & validator
├── samples/ # Sampel dummy dataset untuk pengujian
│ ├── waste_reports_raw.csv # Sampel CSV kotor
│ ├── spku_realtime.json # Sampel JSON sensor IoT
│ └── seed_amdal_permits.sql # SQL seed data transaksional
├── docs/ # Panduan penulisan pipeline untuk Data Engineer
│ ├── pipeline_specifications.md
│ ├── dbt_modeling_guide.md
│ └── adding_new_pipeline.md
└── README.md # Panduan utama repositori pipeline
│ ├── silver/ # Deduplikasi CDC & Data Cleansing
│ └── gold/ # Data Mart, ISPU harian, & Agregasi KPI
├── metadata/ # Metadata visual posisi node kanvas (*.json)
├── samples/ # Dummy sample dataset untuk validasi lokal
├── schemas/ # Kontrak skema dataset (.gitkeep)
├── docs/ # Panduan teknis penulisan pipeline
└── tests/ # Automated unit tests untuk seluruh operator
```
---
## 🎯 3. Katalog 4 Pipeline Bawaan (Canonical Use Cases)
## 📋 3. Katalog 4 Pipeline Bawaan (Canonical Use Cases)
| Nama Pipeline | File Konfigurasi | Engine & Operator | Lapisan Medallion |
| Nama Pipeline | Domain | Jadwal | Engine & Alur Medallion |
| :--- | :--- | :--- | :--- |
| **AMDAL CDC Ingestion** | `ingestion_db_environmental_permits_cdc.yaml` | Sling CLI Log-Based CDC | PostgreSQL ➔ Bronze S3 (Parquet) |
| **GWS Waste Reports** | `ingestion_gws_waste_reports.yaml` | GWS Sanitizer Operator | Google Sheets ➔ Bronze Parquet |
| **SPKU IoT Air Quality** | `ingestion_api_spku_air_quality.yaml` | REST API Sliding Window | HTTP REST ➔ Bronze S3 (JSON) |
| **Reference Medallion dbt** | `example_permits_medallion_dbt_postgres.yaml` | Sling + Great Expectations + dbt | Bronze ➔ Silver ➔ Gold Marts |
| `example_permits_medallion_dbt_postgres` | Perizinan Lingkungan | `0 8 * * *` | **E2E Medallion**: Sling CDC $\rightarrow$ GE Bronze $\rightarrow$ dbt Silver $\rightarrow$ GE Silver $\rightarrow$ dbt Gold $\rightarrow$ Callback. |
| `ingestion_db_environmental_permits_cdc` | AMDAL / UKL-UPL | `0 */2 * * *` | **Log CDC**: Replikasi intensif Postgres DB $\rightarrow$ S3 Bronze Parquet $\rightarrow$ DuckDB Silver $\rightarrow$ dbt Gold. |
| `ingestion_api_spku_air_quality` | Kualitas Udara (SPKU) | `0 4 * * *` | **Sliding Window**: REST API JSON $\rightarrow$ Bronze S3 $\rightarrow$ DuckDB Iceberg $\rightarrow$ GE Suite $\rightarrow$ dbt Gold Mart. |
| `ingestion_gws_waste_reports` | Sampah & Limbah B3 | `0 4 * * *` | **GWS Sanitizer**: Google Sheets $\rightarrow$ Sanitasi Regex $\rightarrow$ S3 Bronze $\rightarrow$ PyIceberg Silver $\rightarrow$ dbt Gold. |
---
## 🛠️ 4. Panduan Menambahkan Pipeline Baru
## ⚙️ 4. Cara Penggunaan Bersama MERICA Standalone Appliance
Untuk menambahkan pipeline baru ke dalam sistem:
Cukup arahkan variabel lingkungan di `.env` file **MERICA Standalone** ke URL repositori ini:
1. **Buat Berkas Konfigurasi YAML di Direktori `configs/`**:
Contoh: `configs/ingestion_my_dataset.yaml`
```yaml
pipeline_id: "ingestion_my_dataset"
description: "Pipeline Ingestion Data Operasional Baru"
schedule_interval: "@daily"
tags: ["operational", "bronze"]
source:
type: "postgres"
connection_id: "conn_postgres_source_db"
table: "public.my_table"
target:
bucket: "dlh-bronze"
format: "parquet"
```bash
# Di dalam merica-standalone/.env:
GIT_PIPELINE_REPO_URL=https://github.com/<your-username-or-org>/merica-pipeline.git
GIT_PIPELINE_REPO_BRANCH=main
GIT_PIPELINE_REPO_SUBDIR=dags
# Karena repositori ini publik/open, kosongkan kredensial git:
GIT_PIPELINE_REPO_USER=
GIT_PIPELINE_REPO_TOKEN=
```
2. **(Opsional) Tambahkan Transformasi dbt di Direktori `dbt/models/`**:
Tambahkan file SQL di `silver/` atau `gold/` untuk mendefinisikan agregasi analitik.
Ketika stack MERICA Standalone dinyalakan (`./scripts/01_install_stack.sh`), Airflow 3 DAG Processor akan otomatis meng-clone repositori ini dan menyajikan seluruh DAG di web UI ([http://localhost:14080](http://localhost:14080)) serta di MERICA Studio Canvas ([http://localhost:14000](http://localhost:14000)).
3. **Commit & Push ke Server Git**:
---
## 🛠️ 5. Menambahkan Pipeline Baru
1. **Tambahkan file konfigurasi YAML** di `configs/<nama_pipeline>.yaml`.
2. **(Opsional) Tambahkan model dbt** di `dbt/models/silver/` atau `dbt/models/gold/`.
3. **Commit & Push**:
```bash
git add configs/ingestion_my_dataset.yaml
git commit -m "feat(pipeline): add new operational dataset pipeline"
git add configs/<nama_pipeline>.yaml
git commit -m "feat(pipeline): add new pipeline"
git push origin main
```
4. **Verifikasi di Airflow Webserver (Port 14080)**:
Dalam waktu maksimum 30 detik, DAG baru dengan ID `dag_ingestion_my_dataset` akan muncul otomatis di antarmuka Airflow tanpa restart scheduler!
4. **Selesai**: Dalam waktu 30 detik, DAG baru langsung aktif dan terdeteksi di Airflow dan MERICA Studio.
---
## 📚 5. Dokumentasi Teknis Terkait
## 🧪 6. Menjalankan Unit Tests Lokal
* 📑 [docs/pipeline_specifications.md](docs/pipeline_specifications.md) — Spesifikasi lengkap schema sintaksis YAML pipeline.
* 🧊 [docs/dbt_modeling_guide.md](docs/dbt_modeling_guide.md) — Panduan pemodelan data Medallion (Bronze, Silver, Gold).
* 🚀 [docs/adding_new_pipeline.md](docs/adding_new_pipeline.md) — Tutorial langkah-demi-langkah bagi Data Engineer.
```bash
pip install -r requirements.txt
pytest tests/ -v
```
---
## 📄 Lisensi
Proyek ini dilisensikan di bawah ketentuan [Apache License 2.0](LICENSE).
RELEASE_NOTES.md
+100
View File
@@ -0,0 +1,100 @@
# 📦 Catatan Rilis: MERICA Pipeline Bundles v1.0 (General Availability)
> **Versi Rilis**: `v1.0.0 (GA)`
> **Tanggal Rilis**: 07 Oktober 2026
> **Repositori**: `merica-pipeline-bundles.git` (Commit: `604456c`)
> **Target Pengguna**: Tim Data Engineer, Analis Data DLH, Walidata BPKP, dan Pengembang Studio
> **Standar Kompatibilitas**: Apache Airflow 3.3.2 Native `GitDagBundle`, dbt Core 1.10+, Great Expectations 0.18+
---
## 🌟 1. Sorotan Rilis (Release Highlights)
Repositori **`merica-pipeline-bundles`** diinisialisasi sebagai repositori terpisah (*decoupled repository*) untuk menampung seluruh aset analitik data lakehouse di lingkungan produksi, memisahkan logika bisnis dari kode sumber mesin platform.
### Poin Utama Rilis v1.0 GA:
* **4 Konfigurasi Deklaratif Pipeline Kanonikal (`configs/*.yaml`)**: Difokuskan murni pada use case strategis Dinas Lingkungan Hidup (DLH), mencakup replikasi CDC izin lingkungan (AMDAL), telemetri IoT kualitas udara (SPKU), digitalisasi rekapitulasi limbah Google Sheets, serta pipeline referensi Medallion harian.
* **Zero-Dummy Policy**: Seluruh 6 artefak pengujian dummy ad-hoc telah dibersihkan secara permanen dari repositori produksi.
* **8 Model SQL Transformasi Medallion (`dbt/models/`)**: 4 model Silver Clean (deduplikasi CDC, parsing regex, sanitasi batas sensor, dan sliding window lookback) serta 4 model Gold Mart (KPI perizinan harian, agregasi limbah, publikasi ISPU harian, dan tabel fakta multidimensi).
* **Integrasi Great Expectations Native Deklaratif**: Seluruh kontrak skema data dan aturan kualitas diekspresikan secara deklaratif di dalam konfigurasi YAML (`rules:` dan `queries:`), diuji secara otomatis di Bronze dan Silver layer.
* **Mekanisme GitOps Otomatis (Airflow 3 `GitDagBundle`)**: Setiap perubahan yang disimpan melalui kanvas visual MERICA Studio langsung di-commit ke repositori ini dan disinkronkan secara otomatis oleh Airflow 3 DAG Processor dalam interval 30 detik tanpa restart kontainer.
* **Metadata Kanvas Visual (`metadata/*.json`)**: Menyimpan topologi visual node ReactFlow untuk pengalaman visual pipeline builder dua arah (bidirectional visual sync).
---
## 🔄 2. Peningkatan dari Hasil Sandbox Riset (`experimental/`)
Selama fase inkubasi di sandbox `experimental/`, aset analitik data sebelumnya tercampur dengan kode engine platform. Pada rilis v1.0 GA ini:
1. **Pemisahan Total Logika Bisnis & Platform Engine**:
* Walidata DLH dapat memperbarui rumus analitik SQL atau aturan batas kewajaran sensor tanpa menyentuh repositori platform `merica-beta`.
2. **Standardisasi Format Great Expectations**:
* Menghilangkan seluruh skrip imperatif Python/psycopg2 inline. Rule validasi kini menggunakan format standar deklaratif:
```yaml
rules:
- expectation: expect_column_values_to_not_be_null
kwargs: { column: "permit_id" }
- expectation: expect_column_values_to_be_between
kwargs: { column: "pm25", min_value: 0, max_value: 500 }
```
3. **Penyelarasan Model dbt dengan Dynamic Bundle Loader**:
* Model SQL dbt kini menggunakan relasi modular dan referensi standar `{{ source(...) }}` dan `{{ ref(...) }}` yang dipindai secara dinamis oleh platform engine.
---
## 📋 3. Inventarisasi Aset Rilis v1.0 GA
### A. Daftar 4 Konfigurasi Pipeline Kanonikal (`configs/`)
1. `example_permits_medallion_dbt_postgres.yaml`: Pipeline referensi Medallion harian lengkap (Sling CDC $\rightarrow$ GE Bronze $\rightarrow$ dbt Silver $\rightarrow$ GE Silver $\rightarrow$ dbt Gold $\rightarrow$ SITOMAT Callback).
2. `ingestion_db_environmental_permits_cdc.yaml`: Replikasi CDC 15-menitan untuk data transaksional izin lingkungan hidup (AMDAL/UKL-UPL).
3. `ingestion_api_spku_air_quality.yaml`: Pipeline ELT near-realtime (5 menit) telemetri sensor kualitas udara dan penyusunan ISPU harian.
4. `ingestion_gws_waste_reports.yaml`: Ekstraksi harian (02:00 WIB) spreadsheet laporan limbah dan sampah perkotaan dengan forward-fill merged cells.
### B. Daftar 8 Model Transformasi dbt Medallion (`dbt/models/`)
* **Lapisan Silver Clean**:
* `silver_environmental_permits_clean.sql`: Deduplikasi idempoten `ROW_NUMBER()` berdasarkan `updated_at`.
* `silver_gws_waste_reports_clean.sql`: Sanitasi regex string numerik bercampur teks dan normalisasi tanggal.
* `silver_spku_sensor_clean.sql`: Pengalihan nilai sensor di luar batas kewajaran (*sensor sanity*) ke `NULL`.
* `stg_spku_realtime.sql`: Staging query dengan sliding lookback window 3 jam.
* **Lapisan Gold Mart**:
* `gold_permits_summary_daily.sql`: Ringkasan metrik persetujuan izin dan kepatuhan SLA pemrosesan.
* `gold_waste_management_summary.sql`: Agregasi volume sampah organik vs anorganik per kecamatan.
* `gold_ispu_daily_summary.sql`: Publikasi harian nilai ISPU per parameter pencemar untuk masyarakat.
* `fct_spku_daily_mart.sql`: Star schema fakta pengukuran sensor kualitas udara untuk BI / dashboard.
---
## 🛠️ 4. Panduan Kontribusi & Authoring Pipeline Baru
Bagi Walidata atau Data Engineer yang ingin menambahkan atau memperbarui pipeline:
### Opsi A: Melalui UI Canvas MERICA Studio (Direkomendasikan)
1. Buka MERICA Studio di `http://<host>:8950`.
2. Susun atau modifikasi visual pipeline pada kanvas.
3. Klik tombol **Publish Pipeline**. Backend akan membuat komit Git baru secara otomatis.
### Opsi B: Melalui Git Langsung
1. Buat branch fitur dari `development`:
```bash
git checkout -b feature/nama-pipeline-baru
```
2. Tambahkan berkas konfigurasi di `configs/nama_pipeline.yaml` dan model SQL di `dbt/models/...`.
3. Validasi sintaksis YAML:
```bash
python3 -c "import yaml; yaml.safe_load(open('configs/nama_pipeline.yaml'))"
```
4. Commit dan push ke remote repository:
```bash
git commit -m "feat(pipeline): add new pipeline for ..."
git push origin feature/nama-pipeline-baru
```
5. Buat Merge Request ke branch `development`.
---
## 📚 5. Dokumentasi Teknis Terkait
* 🧪 [TESTING.md](file:///mnt/c/project/bpkp/dlh/tech/merica-pipeline-bundles/TESTING.md) — Panduan Validasi & Pengujian Asset Pipeline
* 📑 [Dokumen 39: Panduan Implementasi Dual-Repo Git DAG Bundles](file:///mnt/c/project/bpkp/dlh/tech/docs/39_panduan_implementasi_dual_repo_git_dag_bundles_dan_ci_cd_merica_beta.md)
* 🚀 [Dokumen 40: Laporan Resmi Rilis GA v1.0 & Katalog Pipeline Bundles](file:///mnt/c/project/bpkp/dlh/tech/docs/40_laporan_resmi_rilis_merica_beta_v1_dan_katalog_pipeline_bundles.md)
* 🧪 [Dokumen 41: Panduan & Laporan Pengujian Riil E2E](file:///mnt/c/project/bpkp/dlh/tech/docs/41_panduan_dan_laporan_pengujian_riil_e2e_merica_beta_dan_pipeline_bundles.md)
TESTING.md
+103
View File
@@ -0,0 +1,103 @@
# 🧪 Panduan Validasi & Pengujian: MERICA Pipeline Bundles
> **Repositori**: `merica-pipeline-bundles.git`
> **Status**: **Production Ready (4 Canonical Pipelines & 8 dbt Models)**
> **Integrasi**: Apache Airflow 3 Native `GitDagBundle` (Interval: 30s)
> **Kebijakan**: **Zero-Dummy Policy**
---
## 📌 1. Gambaran Umum Pengujian Asset Pipeline
Repositori ini menyimpan logika analitik data lakehouse yang dipantau secara langsung oleh Airflow 3 DAG Processor. Sebelum aset pipeline di-merge atau di-push ke branch produksi (`development` / `main`), seluruh aset wajib melewati validasi:
1. **Validasi Sintaksis YAML**: Menjamin tidak ada kesalahan indentasi atau parsing pada file konfigurasi pipeline.
2. **Validasi Kontrak Aturan Great Expectations**: Memastikan deklarasi `rules:` dan `queries:` valid dan memiliki parameter wajib.
3. **Validasi Sintaksis Model dbt SQL**: Memastikan model transformasi Silver dan Gold memiliki struktur query yang kompatibel.
4. **Zero-Dummy Compliance Check**: Memastikan tidak ada berkas eksperimen atau dummy pipeline yang tersimpan di `configs/` atau `metadata/`.
---
## ⚡ 2. Perintah Validasi Lokal
Jalankan perintah berikut di root repositori `merica-pipeline-bundles/`:
### A. Validasi Seluruh Konfigurasi YAML
```bash
python3 -c "
import glob, yaml, sys
files = sorted(glob.glob('configs/*.yaml'))
print(f'Memvalidasi {len(files)} berkas konfigurasi...')
for f in files:
try:
data = yaml.safe_load(open(f))
pid = data.get('pipeline_id', 'UNKNOWN')
print(f' ✓ {f} -> Pipeline ID: {pid} (VALID)')
except Exception as e:
print(f' ✗ ERROR pada {f}: {e}', file=sys.stderr)
sys.exit(1)
print('Seluruh konfigurasi YAML valid 100%!')
"
```
### B. Validasi Keberadaan 4 Pipeline Produksi Kanonikal
```bash
python3 -c "
import glob, yaml, sys
expected = {
'example_permits_medallion_dbt_postgres',
'ingestion_db_environmental_permits_cdc',
'ingestion_gws_waste_reports',
'ingestion_api_spku_air_quality'
}
actual = {yaml.safe_load(open(f)).get('pipeline_id') for f in glob.glob('configs/*.yaml')}
diff = actual - expected
if diff:
print(f'PERINGATAN: Ditemukan pipeline non-standar: {diff}', file=sys.stderr)
sys.exit(1)
missing = expected - actual
if missing:
print(f'PERINGATAN: Pipeline kanonikal hilang: {missing}', file=sys.stderr)
sys.exit(1)
print(f'Audit Zero-Dummy Sukses: Tepat 4 pipeline kanonikal terdaftar ({len(actual)}/4).')
"
```
### C. Validasi Model dbt SQL
```bash
# Periksa keberadaan 4 model Silver dan 4 model Gold
ls -la dbt/models/silver/*.sql
ls -la dbt/models/gold/*.sql
```
---
## 🛡️ 3. Kebijakan Zero-Dummy Pipeline
Repositori produksi ini diaudit ketat terhadap aturan **Zero-Dummy Policy**:
* Dilarang menempatkan file konfigurasi dengan prefix `test_` atau penamaan ad-hoc seperti `pipeline_2507` atau `pipeline_4984`.
* Seluruh pipeline harus memiliki identitas semantik yang mencerminkan fungsi bisnis DLH:
* `example_permits_medallion_dbt_postgres.yaml`
* `ingestion_db_environmental_permits_cdc.yaml`
* `ingestion_gws_waste_reports.yaml`
* `ingestion_api_spku_air_quality.yaml`
---
## 🔄 4. Verifikasi End-to-End dengan Airflow 3 `GitDagBundle`
Saat perubahan di-push ke remote Git:
1. Airflow 3 DAG Processor akan menarik pembaruan dalam interval maksimal 30 detik.
2. Buka dashboard Airflow di `http://<airflow-host>:8088`.
3. Pastikan DAG baru muncul di daftar DAGs dengan bundle tag `dlh-pipeline-bundle`.
4. Periksa log DAG Processor untuk memastikan tidak ada pesan *Import Error*:
```bash
docker compose logs --tail=50 airflow-dag-processor | grep -i "error"
```
---
## 📚 5. Referensi Terkait
* 🧪 [Dokumen 41: Panduan & Laporan Pengujian Riil E2E](file:///mnt/c/project/bpkp/dlh/tech/docs/41_panduan_dan_laporan_pengujian_riil_e2e_merica_beta_dan_pipeline_bundles.md)
* 📋 [RELEASE_NOTES.md](file:///mnt/c/project/bpkp/dlh/tech/merica-pipeline-bundles/RELEASE_NOTES.md)
schemas/.gitkeep
View File