feat: initial commit for merica-pipeline delivery
This commit is contained in:
MERICA Admin
@@ -0,0 +1,84 @@
|
||||
# 🚀 Panduan Menambahkan Pipeline Baru (Authoring Tutorial)
|
||||
|
||||
Dokumen ini merupakan panduan praktis langkah-demi-langkah bagi Data Engineer untuk membuat, menguji, dan mempublikasikan data pipeline baru ke platform MERICA.
|
||||
|
||||
---
|
||||
|
||||
## Alur Kerja Penambahan Pipeline
|
||||
|
||||
```
|
||||
┌────────────────────────────────────────────────────────────────────────────────────────┐
|
||||
│ DATA PIPELINE AUTHORING WORKFLOW │
|
||||
├───────────┬──────────────────────────────────┬─────────────────────────────────────────┤
|
||||
│ Langkah │ Tindakan Pengembang │ Lokasi Berkas │
|
||||
├───────────┼──────────────────────────────────┼─────────────────────────────────────────┤
|
||||
│ 1 │ Buat Deklarasi Pipeline YAML │ `configs/ingestion_<nama_pipeline>.yaml`│
|
||||
│ 2 │ (Opsional) Buat Model dbt │ `dbt/models/silver/` atau `gold/` │
|
||||
│ 3 │ (Opsional) Registrasi Metadata │ `metadata/<nama_pipeline>.json` │
|
||||
│ 4 │ Validasi Sintaksis Lokal │ Pemeriksaan struktur YAML │
|
||||
│ 5 │ Git Commit & Push │ Server Git On-Premise │
|
||||
│ 6 │ Verifikasi Auto-Discovery │ Airflow Web UI (Port 14080) │
|
||||
└───────────┴──────────────────────────────────┴─────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Langkah 1: Menyiapkan File YAML
|
||||
|
||||
Buat file baru di direktori `configs/`, misalnya `configs/ingestion_sensor_kebisingan.yaml`:
|
||||
|
||||
```yaml
|
||||
pipeline_id: "ingestion_sensor_kebisingan"
|
||||
pipeline_name: "Sensor Kebisingan Kota Realtime"
|
||||
description: "Ingestion data kebisingan akustik dari jaringan sensor IoT lingkungan."
|
||||
schedule_interval: "*/15 * * * *" # Berjalan setiap 15 menit
|
||||
tags:
|
||||
- "iot"
|
||||
- "sensor"
|
||||
- "kebisingan"
|
||||
|
||||
source:
|
||||
type: "api"
|
||||
connection_id: "conn_spku_api_service"
|
||||
endpoint: "/v1/noise/telemetry"
|
||||
method: "GET"
|
||||
lookback_window_hours: 1
|
||||
response_path: "data.records"
|
||||
|
||||
target:
|
||||
bucket: "dlh-bronze"
|
||||
format: "parquet"
|
||||
partition_by: "ingest_date"
|
||||
|
||||
governance:
|
||||
owner: "bidang_ppkl@dlh.id"
|
||||
tier: "Tier2"
|
||||
domain: "Pengendalian Pencemaran Lingkungan"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Langkah 2: Menyiapkan Model Transformasi dbt (Bila Perlu)
|
||||
|
||||
Jika pipeline ini memerlukan pembersihan atau agregasi lanjutan:
|
||||
1. Buat model Silver: `dbt/models/silver/silver_sensor_kebisingan_clean.sql`
|
||||
2. Buat model Gold: `dbt/models/gold/gold_noise_hourly_exceedance.sql`
|
||||
|
||||
---
|
||||
|
||||
## Langkah 3: Commit dan Push ke Git Server
|
||||
|
||||
Commit perubahan ke branch aktif (misal `main`):
|
||||
```bash
|
||||
git add configs/ingestion_sensor_kebisingan.yaml
|
||||
git commit -m "feat(pipeline): register noise sensor telemetry pipeline"
|
||||
git push origin main
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Langkah 4: Verifikasi di Airflow 3 Runtime
|
||||
|
||||
Tunggu paling lama **30 detik**. Buka antarmuka Airflow pada `http://localhost:14080`:
|
||||
* DAG `dag_ingestion_sensor_kebisingan` akan muncul secara otomatis.
|
||||
* Anda dapat memicu eksekusi manual via tombol **Trigger DAG (Play button)** untuk menguji aliran data dari sumber ke S3 Bronze.
|
||||
@@ -0,0 +1,86 @@
|
||||
# 🧊 Panduan Pemodelan Data Medallion dbt (Lakehouse Transformations)
|
||||
|
||||
Dokumen ini menjelaskan struktur dan konvensi penulisan model transformasi **dbt** (Data Build Tool) pada platform MERICA Data Lakehouse.
|
||||
|
||||
---
|
||||
|
||||
## 1. Konvensi Penamaan Model Medallion
|
||||
|
||||
Transformasi dbt diorganisasikan ke dalam subdirektori `dbt/models/`:
|
||||
|
||||
```text
|
||||
dbt/models/
|
||||
├── silver/
|
||||
│ ├── silver_environmental_permits_clean.sql
|
||||
│ ├── silver_gws_waste_reports_clean.sql
|
||||
│ └── silver_spku_sensor_clean.sql
|
||||
└── gold/
|
||||
├── gold_permits_summary_daily.sql
|
||||
├── gold_waste_management_summary.sql
|
||||
└── gold_ispu_daily_summary.sql
|
||||
```
|
||||
|
||||
| Layer | Prefix Penamaan | Tujuan & Prinsip Transformasi |
|
||||
| :--- | :--- | :--- |
|
||||
| **Silver** | `silver_` atau `int_` | **Pembersihan & Deduplikasi**: Casting tipe data yang konsisten, penanganan nilai null, pemfilteran duplikasi menggunakan `ROW_NUMBER() OVER (PARTITION BY pk ORDER BY updated_at DESC) WHERE rn = 1`. |
|
||||
| **Gold** | `gold_` atau `fct_` | **Data Marts & Agregasi**: Menggabungkan dimensi dan fakta untuk metrik kepatuhan, laporan harian, KPI eksekutif, dan dashboard visualisasi. |
|
||||
|
||||
---
|
||||
|
||||
## 2. Contoh Struktur Model Silver
|
||||
|
||||
Berkas `dbt/models/silver/silver_environmental_permits_clean.sql`:
|
||||
```sql
|
||||
{{ config(
|
||||
materialized='table',
|
||||
schema='silver'
|
||||
) }}
|
||||
|
||||
WITH source_data AS (
|
||||
SELECT
|
||||
permit_number,
|
||||
UPPER(TRIM(company_name)) AS company_name,
|
||||
permit_type,
|
||||
CAST(issued_date AS DATE) AS issued_date,
|
||||
UPPER(TRIM(status)) AS status,
|
||||
updated_at,
|
||||
ROW_NUMBER() OVER (
|
||||
PARTITION BY permit_number
|
||||
ORDER BY updated_at DESC
|
||||
) AS row_num
|
||||
FROM {{ source('bronze', 'environmental_permits') }}
|
||||
)
|
||||
|
||||
SELECT
|
||||
permit_number,
|
||||
company_name,
|
||||
permit_type,
|
||||
issued_date,
|
||||
status,
|
||||
updated_at
|
||||
FROM source_data
|
||||
WHERE row_num = 1
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. Contoh Struktur Model Gold
|
||||
|
||||
Berkas `dbt/models/gold/gold_permits_summary_daily.sql`:
|
||||
```sql
|
||||
{{ config(
|
||||
materialized='table',
|
||||
schema='gold'
|
||||
) }}
|
||||
|
||||
SELECT
|
||||
permit_type,
|
||||
status,
|
||||
COUNT(permit_number) AS total_permits,
|
||||
MIN(issued_date) AS earliest_issued,
|
||||
MAX(issued_date) AS latest_issued,
|
||||
NOW() AS calculated_at
|
||||
FROM {{ ref('silver_environmental_permits_clean') }}
|
||||
GROUP BY permit_type, status
|
||||
ORDER BY total_permits DESC
|
||||
```
|
||||
@@ -0,0 +1,96 @@
|
||||
# 📑 Spesifikasi Sintaksis Konfigurasi Pipeline (YAML Specification)
|
||||
|
||||
Dokumen ini menjelaskan struktur kontrak data deklaratif (*declarative pipeline contract*) untuk berkas-berkas konfigurasi YAML di repositori **`merica-pipeline/configs/`**.
|
||||
|
||||
---
|
||||
|
||||
## 1. Skema Tingkat Atas (Top-Level Schema)
|
||||
|
||||
Setiap file pipeline YAML mendefinisikan properti berikut:
|
||||
|
||||
```yaml
|
||||
pipeline_id: "string" # Unik, identifier teknis pipeline (snake_case)
|
||||
pipeline_name: "string" # Nama yang dapat dibaca manusia (Human-readable)
|
||||
description: "string" # Deskripsi fungsional pipeline
|
||||
schedule_interval: "cron/expr" # Jadwal crontab atau preset (@daily, @hourly, None)
|
||||
tags: # Array string untuk pengelompokan di UI
|
||||
- "category"
|
||||
- "layer"
|
||||
|
||||
source: # Konfigurasi sumber data hulu
|
||||
type: "postgres|gws|api"
|
||||
connection_id: "conn_id"
|
||||
# Parameter spesifik sumber...
|
||||
|
||||
target: # Konfigurasi tujuan hilir
|
||||
bucket: "dlh-bronze"
|
||||
format: "parquet|json|iceberg"
|
||||
# Parameter spesifik target...
|
||||
|
||||
governance: # Metadata tata kelola (OpenMetadata)
|
||||
owner: "nama_tim_atau_email"
|
||||
tier: "Tier1|Tier2|Tier3|Tier4|Tier5"
|
||||
domain: "nama_domain_bisnis"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. Parameter Berdasarkan Tipe Sumber (*Source Types*)
|
||||
|
||||
### 2.1 Sumber Database Transaksional (`type: "postgres"`)
|
||||
```yaml
|
||||
source:
|
||||
type: "postgres"
|
||||
connection_id: "conn_postgres_source_db"
|
||||
table: "public.environmental_permits"
|
||||
mode: "incremental" # "full-refresh" atau "incremental"
|
||||
primary_key: "permit_number"
|
||||
update_key: "updated_at" # Kolom watermark waktu
|
||||
```
|
||||
|
||||
### 2.2 Sumber Spreadsheet Google Workspace (`type: "gws"`)
|
||||
```yaml
|
||||
source:
|
||||
type: "gws"
|
||||
connection_id: "conn_gws_service_account"
|
||||
spreadsheet_id: "1A2B3C..."
|
||||
sheet_name: "Laporan Sampah"
|
||||
header_row: 1
|
||||
cleaning_rules:
|
||||
fill_empty: "ffill" # Forward fill merged cells
|
||||
clean_numeric_columns:
|
||||
- "total_volume"
|
||||
- "biaya_retribusi"
|
||||
```
|
||||
|
||||
### 2.3 Sumber Telemetri REST API (`type: "api"`)
|
||||
```yaml
|
||||
source:
|
||||
type: "api"
|
||||
connection_id: "conn_spku_api_service"
|
||||
endpoint: "/v1/sensors/realtime"
|
||||
method: "GET"
|
||||
lookback_window_hours: 3 # Menarik data dari (now - 3h) s/d now
|
||||
response_path: "data.readings"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. Integrasi Validasi Kualitas (Great Expectations)
|
||||
|
||||
Anda dapat menyematkan aturan validasi kualitas data deklaratif langsung di dalam blok YAML:
|
||||
|
||||
```yaml
|
||||
quality_gate:
|
||||
engine: "great_expectations"
|
||||
on_failure: "warn|fail" # "warn": catat log; "fail": hentikan pipeline
|
||||
expectations:
|
||||
- expectation_type: "expect_column_values_to_not_be_null"
|
||||
kwargs:
|
||||
column: "permit_number"
|
||||
- expectation_type: "expect_column_values_to_be_between"
|
||||
kwargs:
|
||||
column: "pm25"
|
||||
min_value: 0.0
|
||||
max_value: 500.0
|
||||
```
|
||||
Reference in New Issue
Block a user