chore(release): standardize enterprise documentation and asset integrity
This commit is contained in:
MERICA Admin
@@ -1,120 +0,0 @@
|
||||
Apache License
|
||||
Version 2.0, January 2004
|
||||
http://www.apache.org/licenses/
|
||||
|
||||
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
||||
|
||||
1. Definitions.
|
||||
|
||||
"License" shall mean the terms and conditions for use, reproduction,
|
||||
and distribution as defined by Sections 1 through 9 of this document.
|
||||
|
||||
"Licensor" shall mean the copyright owner or entity authorized by
|
||||
the copyright owner that is granting the License.
|
||||
|
||||
"Legal Entity" shall mean the union of the acting entity and all
|
||||
other entities that control, are controlled by, or are under common
|
||||
control with that entity. For the purposes of this definition,
|
||||
"control" means (i) the power, direct or indirect, to cause the
|
||||
direction or management of such entity, whether by contract or
|
||||
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
||||
outstanding shares, or (iii) beneficial ownership of such entity.
|
||||
|
||||
"You" (or "Your") shall mean an individual or Legal Entity
|
||||
exercising permissions granted by this License.
|
||||
|
||||
"Source" form shall mean the preferred form for making modifications,
|
||||
including but not limited to software source code, documentation
|
||||
source, and configuration files.
|
||||
|
||||
"Object" form shall mean any form resulting from mechanical
|
||||
transformation or translation of a Source form, including but
|
||||
not limited to compiled object code, generated documentation,
|
||||
and conversions to other media types.
|
||||
|
||||
"Work" shall mean the work of authorship, whether in Source or
|
||||
Object form, made available under the License, as indicated by a
|
||||
copyright notice that is included in or attached to the work.
|
||||
|
||||
"Derivative Works" shall mean any work, whether in Source or Object
|
||||
form, that is based on (or derived from) the Work and for which the
|
||||
editorial revisions, annotations, elaborations, or other modifications
|
||||
represent, as a whole, an original work of authorship. For the purposes
|
||||
of this License, Derivative Works shall not include works that remain
|
||||
separable from, or merely link (or bind by name) to the interfaces of,
|
||||
the Work and Derivative Works thereof.
|
||||
|
||||
"Contribution" shall mean any work of authorship, including
|
||||
the original version of the Work and any modifications or additions
|
||||
to that Work or Derivative Works thereof, that is intentionally
|
||||
submitted to Licensor for inclusion in the Work by the copyright owner
|
||||
or by an individual or Legal Entity authorized to submit on behalf of
|
||||
the copyright owner.
|
||||
|
||||
2. Grant of Copyright License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
copyright license to use, reproduce, prepare Derivative Works of,
|
||||
publicly display, publicly perform, sublicense, and distribute the
|
||||
Work and such Derivative Works in Source or Object form.
|
||||
|
||||
3. Grant of Patent License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
patent license to make, have made, use, offer to sell, sell, import,
|
||||
and otherwise transfer the Work.
|
||||
|
||||
4. Redistribution. You may reproduce and distribute copies of the
|
||||
Work or Derivative Works thereof in any medium, with or without
|
||||
modifications, and in Source or Object form, provided that You
|
||||
meet the following conditions:
|
||||
|
||||
(a) You must give any other recipients of the Work or
|
||||
Derivative Works a copy of this License; and
|
||||
|
||||
(b) You must cause any modified files to carry prominent notices
|
||||
stating that You changed the files; and
|
||||
|
||||
(c) You must retain, in the Source form of any Derivative Works
|
||||
that You distribute, all copyright, patent, trademark, and
|
||||
attribution notices from the Source form of the Work; and
|
||||
|
||||
(d) If the Work includes a "NOTICE" text file as part of its
|
||||
distribution, then any Derivative Works that You distribute must
|
||||
include a readable copy of the attribution notices contained
|
||||
within such NOTICE file.
|
||||
|
||||
5. Submission of Contributions. Unless You explicitly state otherwise,
|
||||
any Contribution intentionally submitted for inclusion in the Work
|
||||
by You to the Licensor shall be under the terms and conditions of
|
||||
this License, without any additional terms or conditions.
|
||||
|
||||
6. Trademarks. This License does not grant permission to use the trade
|
||||
names, trademarks, service marks, or product names of the Licensor,
|
||||
except as required for reasonable and customary use in describing the
|
||||
origin of the Work and reproducing the content of the NOTICE file.
|
||||
|
||||
7. Disclaimer of Warranty. Unless required by applicable law or
|
||||
agreed to in writing, Licensor provides the Work (and each
|
||||
Contributor provides its Contributions) on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
||||
implied, including, without limitation, any warranties or conditions
|
||||
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
||||
PARTICULAR PURPOSE.
|
||||
|
||||
8. Limitation of Liability. In no event and under no legal theory,
|
||||
whether in tort (including negligence), contract, or otherwise,
|
||||
unless required by applicable law (such as deliberate and grossly
|
||||
negligent acts) or agreed to in writing, shall any Contributor be
|
||||
liable to You for damages, including any direct, indirect, special,
|
||||
incidental, or consequential damages of any character arising as a
|
||||
result of this License or out of the use or inability to use the
|
||||
Work.
|
||||
|
||||
9. Accepting Warranty or Additional Liability. While redistributing
|
||||
the Work or Derivative Works thereof, You may choose to offer,
|
||||
and charge a fee for, acceptance of support, warranty, indemnity,
|
||||
or other liability obligations and/or rights consistent with this
|
||||
License.
|
||||
|
||||
END OF TERMS AND CONDITIONS
|
||||
@@ -1,30 +1,29 @@
|
||||
# 📦 MERICA Pipeline Bundles (Public Open-Source Data Lakehouse Assets)
|
||||
# 📦 MERICA Pipeline Bundles (Data Lakehouse Production Assets)
|
||||
|
||||
> **Repositori Resmi**: **Single Source of Truth (SSOT)** Aset Konfigurasi Pipeline, Model dbt Medallion, Metadata, & Custom Operators untuk **MERICA Data Lakehouse**
|
||||
> **Status**: **General Availability (GA) v1.0 — Production Ready & Open-Source**
|
||||
> **Standar Integrasi**: Apache Airflow 3 Native Git DAG Bundles (`GitDagBundle` AIP-66) & dbt Core
|
||||
> **Status**: **General Availability (GA) v1.0 — Production Ready**
|
||||
> **Integrasi Platform**: Apache Airflow 3 Native Git DAG Bundles (`GitDagBundle` AIP-66) & dbt Core
|
||||
|
||||
---
|
||||
|
||||
## 🏛️ 1. Ikhtisar Repositori
|
||||
|
||||
Repositori **`merica-pipeline`** adalah **Single Source of Truth (SSOT)** publik yang memisahkan seluruh logika bisnis data pipelines dari runtime platform MERICA Studio dan Airflow Engine.
|
||||
Repositori **`merica-pipelines`** adalah **Single Source of Truth (SSOT)** yang mengisolasi seluruh logika bisnis pipeline data dan model analitik dari runtime engine platform MERICA Studio dan Apache Airflow.
|
||||
|
||||
Dengan arsitektur **Git DAG Bundles (AIP-66)** Apache Airflow 3:
|
||||
* **Zero-Credential / Anonymous Clone**: Karena repositori ini bersifat publik/open, Airflow 3 DAG Processor dapat melakukan kloning dan sinkronisasi otomatis tanpa memerlukan username, password, atau personal access token (PAT).
|
||||
* **Hot-Reload Tanpa Restart**: Setiap berkas konfigurasi YAML di repositori ini secara otomatis disinkronkan ke Airflow runtime dalam interval **30 detik** tanpa mematikan atau me-restart scheduler/webserver.
|
||||
* **Canvas Interaktif MERICA Studio**: Konfigurasi YAML di repositori ini langsung dibaca dan divisualisasikan menjadi node-edge pipeline di antarmuka web MERICA Studio Canvas.
|
||||
* **Sinkronisasi Otomatis Tanpa Restart**: Setiap berkas konfigurasi YAML di repositori ini secara otomatis disinkronkan ke Airflow runtime dalam interval **30 detik** tanpa mematikan atau me-restart scheduler/webserver.
|
||||
* **Canvas Interaktif MERICA Studio**: Konfigurasi YAML di repositori ini langsung dibaca dan dirender menjadi node-edge pipeline pada antarmuka visual MERICA Studio Canvas.
|
||||
* **Distribusi Terpusat**: Pengguna appliance MERICA Studio cukup mengarahkan URL repositori ini pada konfigurasi lingkungan untuk mendapatkan katalog pipeline terbaru.
|
||||
|
||||
---
|
||||
|
||||
## 📁 2. Struktur Direktori Repositori
|
||||
|
||||
```text
|
||||
merica-pipeline/
|
||||
├── LICENSE # Lisensi Apache 2.0 Open-Source
|
||||
merica-pipelines/
|
||||
├── README.md # Dokumentasi utama repositori
|
||||
├── RELEASE_NOTES.md # Catatan rilis resmi v1.0 GA
|
||||
├── TESTING.md # Panduan pengujian lokal & CI/CD
|
||||
├── TESTING.md # Panduan audit dan validasi aset pipeline
|
||||
├── requirements.txt # Dependensi testing Python (pytest, ruff)
|
||||
├── configs/ # Konfigurasi pipeline deklaratif (*.yaml)
|
||||
│ ├── example_permits_medallion_dbt_postgres.yaml # Referensi E2E Medallion dbt
|
||||
@@ -53,15 +52,15 @@ merica-pipeline/
|
||||
│ ├── silver/ # Deduplikasi CDC & Data Cleansing
|
||||
│ └── gold/ # Data Mart, ISPU harian, & Agregasi KPI
|
||||
├── metadata/ # Metadata visual posisi node kanvas (*.json)
|
||||
├── samples/ # Dummy sample dataset untuk validasi lokal
|
||||
├── samples/ # Dataset referensi untuk validasi lokal
|
||||
├── schemas/ # Kontrak skema dataset (.gitkeep)
|
||||
├── docs/ # Panduan teknis penulisan pipeline
|
||||
├── docs/ # Panduan teknis spesifikasi pipeline
|
||||
└── tests/ # Automated unit tests untuk seluruh operator
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📋 3. Katalog 4 Pipeline Bawaan (Canonical Use Cases)
|
||||
## 📋 3. Katalog Pipeline Kanonikal
|
||||
|
||||
| Nama Pipeline | Domain | Jadwal | Engine & Alur Medallion |
|
||||
| :--- | :--- | :--- | :--- |
|
||||
@@ -72,48 +71,42 @@ merica-pipeline/
|
||||
|
||||
---
|
||||
|
||||
## ⚙️ 4. Cara Penggunaan Bersama MERICA Standalone Appliance
|
||||
## ⚙️ 4. Cara Penggunaan Bersama MERICA Studio Appliance
|
||||
|
||||
Cukup arahkan variabel lingkungan di `.env` file **MERICA Standalone** ke URL repositori ini:
|
||||
Arahkan variabel lingkungan di `.env` file **MERICA Studio / Standalone** ke URL repositori ini:
|
||||
|
||||
```bash
|
||||
# Di dalam merica-standalone/.env:
|
||||
GIT_PIPELINE_REPO_URL=https://github.com/<your-username-or-org>/merica-pipeline.git
|
||||
# Konfigurasi repositori pipeline di .env:
|
||||
GIT_PIPELINE_REPO_URL=https://repo.sicabe.com/merica-dlh/merica-pipelines.git
|
||||
GIT_PIPELINE_REPO_BRANCH=main
|
||||
GIT_PIPELINE_REPO_SUBDIR=dags
|
||||
|
||||
# Karena repositori ini publik/open, kosongkan kredensial git:
|
||||
# Kredensial repositori git (kosongkan jika repositori dapat diakses tanpa autentikasi):
|
||||
GIT_PIPELINE_REPO_USER=
|
||||
GIT_PIPELINE_REPO_TOKEN=
|
||||
```
|
||||
|
||||
Ketika stack MERICA Standalone dinyalakan (`./scripts/01_install_stack.sh`), Airflow 3 DAG Processor akan otomatis meng-clone repositori ini dan menyajikan seluruh DAG di web UI ([http://localhost:14080](http://localhost:14080)) serta di MERICA Studio Canvas ([http://localhost:14000](http://localhost:14000)).
|
||||
Ketika stack MERICA Studio dijalankan (`./scripts/01_install_stack.sh`), Airflow 3 DAG Processor akan otomatis mengunduh repositori ini dan menyajikan seluruh DAG pada antarmuka web Airflow ([http://localhost:14080](http://localhost:14080)) serta MERICA Studio Canvas ([http://localhost:14000](http://localhost:14000)).
|
||||
|
||||
---
|
||||
|
||||
## 🛠️ 5. Menambahkan Pipeline Baru
|
||||
|
||||
1. **Tambahkan file konfigurasi YAML** di `configs/<nama_pipeline>.yaml`.
|
||||
2. **(Opsional) Tambahkan model dbt** di `dbt/models/silver/` atau `dbt/models/gold/`.
|
||||
1. **Tambahkan berkas konfigurasi YAML** di direktori `configs/<nama_pipeline>.yaml`.
|
||||
2. **(Opsional) Tambahkan model dbt** di direktori `dbt/models/silver/` atau `dbt/models/gold/`.
|
||||
3. **Commit & Push**:
|
||||
```bash
|
||||
git add configs/<nama_pipeline>.yaml
|
||||
git commit -m "feat(pipeline): add new pipeline"
|
||||
git push origin main
|
||||
```
|
||||
4. **Selesai**: Dalam waktu 30 detik, DAG baru langsung aktif dan terdeteksi di Airflow dan MERICA Studio.
|
||||
4. **Verifikasi**: Dalam waktu 30 detik, pipeline baru akan terdeteksi dan aktif secara otomatis di Airflow dan MERICA Studio.
|
||||
|
||||
---
|
||||
|
||||
## 🧪 6. Menjalankan Unit Tests Lokal
|
||||
## 🧪 6. Pengujian Unit Lokal
|
||||
|
||||
```bash
|
||||
pip install -r requirements.txt
|
||||
pytest tests/ -v
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📄 Lisensi
|
||||
|
||||
Proyek ini dilisensikan di bawah ketentuan [Apache License 2.0](LICENSE).
|
||||
|
||||
+1
-1
@@ -14,7 +14,7 @@ Repositori **`merica-pipeline-bundles`** diinisialisasi sebagai repositori terpi
|
||||
|
||||
### Poin Utama Rilis v1.0 GA:
|
||||
* **4 Konfigurasi Deklaratif Pipeline Kanonikal (`configs/*.yaml`)**: Difokuskan murni pada use case strategis Dinas Lingkungan Hidup (DLH), mencakup replikasi CDC izin lingkungan (AMDAL), telemetri IoT kualitas udara (SPKU), digitalisasi rekapitulasi limbah Google Sheets, serta pipeline referensi Medallion harian.
|
||||
* **Zero-Dummy Policy**: Seluruh 6 artefak pengujian dummy ad-hoc telah dibersihkan secara permanen dari repositori produksi.
|
||||
* **Kebijakan Produksi Terstandar (Production Compliance)**: Seluruh artefak eksperimental non-kanonikal telah dibersihkan secara permanen dari repositori produksi.
|
||||
* **8 Model SQL Transformasi Medallion (`dbt/models/`)**: 4 model Silver Clean (deduplikasi CDC, parsing regex, sanitasi batas sensor, dan sliding window lookback) serta 4 model Gold Mart (KPI perizinan harian, agregasi limbah, publikasi ISPU harian, dan tabel fakta multidimensi).
|
||||
* **Integrasi Great Expectations Native Deklaratif**: Seluruh kontrak skema data dan aturan kualitas diekspresikan secara deklaratif di dalam konfigurasi YAML (`rules:` dan `queries:`), diuji secara otomatis di Bronze dan Silver layer.
|
||||
* **Mekanisme GitOps Otomatis (Airflow 3 `GitDagBundle`)**: Setiap perubahan yang disimpan melalui kanvas visual MERICA Studio langsung di-commit ke repositori ini dan disinkronkan secara otomatis oleh Airflow 3 DAG Processor dalam interval 30 detik tanpa restart kontainer.
|
||||
|
||||
+8
-8
@@ -1,9 +1,9 @@
|
||||
# 🧪 Panduan Validasi & Pengujian: MERICA Pipeline Bundles
|
||||
|
||||
> **Repositori**: `merica-pipeline-bundles.git`
|
||||
> **Repositori**: `merica-pipelines.git`
|
||||
> **Status**: **Production Ready (4 Canonical Pipelines & 8 dbt Models)**
|
||||
> **Integrasi**: Apache Airflow 3 Native `GitDagBundle` (Interval: 30s)
|
||||
> **Kebijakan**: **Zero-Dummy Policy**
|
||||
> **Standar Kepatuhan**: **Production Asset Standards**
|
||||
|
||||
---
|
||||
|
||||
@@ -14,7 +14,7 @@ Repositori ini menyimpan logika analitik data lakehouse yang dipantau secara lan
|
||||
1. **Validasi Sintaksis YAML**: Menjamin tidak ada kesalahan indentasi atau parsing pada file konfigurasi pipeline.
|
||||
2. **Validasi Kontrak Aturan Great Expectations**: Memastikan deklarasi `rules:` dan `queries:` valid dan memiliki parameter wajib.
|
||||
3. **Validasi Sintaksis Model dbt SQL**: Memastikan model transformasi Silver dan Gold memiliki struktur query yang kompatibel.
|
||||
4. **Zero-Dummy Compliance Check**: Memastikan tidak ada berkas eksperimen atau dummy pipeline yang tersimpan di `configs/` atau `metadata/`.
|
||||
4. **Audit Integritas Aset Produksi**: Memastikan hanya konfigurasi kanonikal terverifikasi yang tersimpan di `configs/` atau `metadata/`.
|
||||
|
||||
---
|
||||
|
||||
@@ -59,7 +59,7 @@ missing = expected - actual
|
||||
if missing:
|
||||
print(f'PERINGATAN: Pipeline kanonikal hilang: {missing}', file=sys.stderr)
|
||||
sys.exit(1)
|
||||
print(f'Audit Zero-Dummy Sukses: Tepat 4 pipeline kanonikal terdaftar ({len(actual)}/4).')
|
||||
print(f'Audit Integritas Aset Sukses: Tepat 4 pipeline kanonikal terdaftar ({len(actual)}/4).')
|
||||
"
|
||||
```
|
||||
|
||||
@@ -72,11 +72,11 @@ ls -la dbt/models/gold/*.sql
|
||||
|
||||
---
|
||||
|
||||
## 🛡️ 3. Kebijakan Zero-Dummy Pipeline
|
||||
## 🛡️ 3. Kebijakan Integritas Pipeline Produksi
|
||||
|
||||
Repositori produksi ini diaudit ketat terhadap aturan **Zero-Dummy Policy**:
|
||||
* Dilarang menempatkan file konfigurasi dengan prefix `test_` atau penamaan ad-hoc seperti `pipeline_2507` atau `pipeline_4984`.
|
||||
* Seluruh pipeline harus memiliki identitas semantik yang mencerminkan fungsi bisnis DLH:
|
||||
Repositori produksi ini diaudit ketat terhadap aturan standarisasi aset:
|
||||
* Dilarang menempatkan file konfigurasi dengan penamaan ad-hoc atau sementara.
|
||||
* Seluruh pipeline harus memiliki identitas semantik yang mencerminkan fungsi bisnis:
|
||||
* `example_permits_medallion_dbt_postgres.yaml`
|
||||
* `ingestion_db_environmental_permits_cdc.yaml`
|
||||
* `ingestion_gws_waste_reports.yaml`
|
||||
|
||||
Reference in New Issue
Block a user