feat: initial commit for merica-pipeline delivery

2026-10-08 19:33:15 +07:00
commit 42fedab752
46 changed files with 5484 additions and 0 deletions
@@ -0,0 +1,86 @@
+# 🧊 Panduan Pemodelan Data Medallion dbt (Lakehouse Transformations)
+
+Dokumen ini menjelaskan struktur dan konvensi penulisan model transformasi **dbt** (Data Build Tool) pada platform MERICA Data Lakehouse.
+
+---
+
+## 1. Konvensi Penamaan Model Medallion
+
+Transformasi dbt diorganisasikan ke dalam subdirektori `dbt/models/`:
+
+```text
+dbt/models/
+├── silver/
+│   ├── silver_environmental_permits_clean.sql
+│   ├── silver_gws_waste_reports_clean.sql
+│   └── silver_spku_sensor_clean.sql
+└── gold/
+    ├── gold_permits_summary_daily.sql
+    ├── gold_waste_management_summary.sql
+    └── gold_ispu_daily_summary.sql
+```
+
+| Layer | Prefix Penamaan | Tujuan & Prinsip Transformasi |
+| :--- | :--- | :--- |
+| **Silver** | `silver_` atau `int_` | **Pembersihan & Deduplikasi**: Casting tipe data yang konsisten, penanganan nilai null, pemfilteran duplikasi menggunakan `ROW_NUMBER() OVER (PARTITION BY pk ORDER BY updated_at DESC) WHERE rn = 1`. |
+| **Gold** | `gold_` atau `fct_` | **Data Marts & Agregasi**: Menggabungkan dimensi dan fakta untuk metrik kepatuhan, laporan harian, KPI eksekutif, dan dashboard visualisasi. |
+
+---
+
+## 2. Contoh Struktur Model Silver
+
+Berkas `dbt/models/silver/silver_environmental_permits_clean.sql`:
+```sql
+{{ config(
+    materialized='table',
+    schema='silver'
+) }}
+
+WITH source_data AS (
+    SELECT 
+        permit_number,
+        UPPER(TRIM(company_name)) AS company_name,
+        permit_type,
+        CAST(issued_date AS DATE) AS issued_date,
+        UPPER(TRIM(status)) AS status,
+        updated_at,
+        ROW_NUMBER() OVER (
+            PARTITION BY permit_number 
+            ORDER BY updated_at DESC
+        ) AS row_num
+    FROM {{ source('bronze', 'environmental_permits') }}
+)
+
+SELECT 
+    permit_number,
+    company_name,
+    permit_type,
+    issued_date,
+    status,
+    updated_at
+FROM source_data
+WHERE row_num = 1
+```
+
+---
+
+## 3. Contoh Struktur Model Gold
+
+Berkas `dbt/models/gold/gold_permits_summary_daily.sql`:
+```sql
+{{ config(
+    materialized='table',
+    schema='gold'
+) }}
+
+SELECT 
+    permit_type,
+    status,
+    COUNT(permit_number) AS total_permits,
+    MIN(issued_date) AS earliest_issued,
+    MAX(issued_date) AS latest_issued,
+    NOW() AS calculated_at
+FROM {{ ref('silver_environmental_permits_clean') }}
+GROUP BY permit_type, status
+ORDER BY total_permits DESC
+```