merica-pipelines/docs/dbt_modeling_guide.md

🧊 Panduan Pemodelan Data Medallion dbt (Lakehouse Transformations)

Dokumen ini menjelaskan struktur dan konvensi penulisan model transformasi dbt (Data Build Tool) pada platform MERICA Data Lakehouse.

---

1. Konvensi Penamaan Model Medallion

Transformasi dbt diorganisasikan ke dalam subdirektori dbt/models/:

dbt/models/
├── silver/
│   ├── silver_environmental_permits_clean.sql
│   ├── silver_gws_waste_reports_clean.sql
│   └── silver_spku_sensor_clean.sql
└── gold/
    ├── gold_permits_summary_daily.sql
    ├── gold_waste_management_summary.sql
    └── gold_ispu_daily_summary.sql

Layer	Prefix Penamaan	Tujuan & Prinsip Transformasi
Silver	silver_ atau int_	Pembersihan & Deduplikasi: Casting tipe data yang konsisten, penanganan nilai null, pemfilteran duplikasi menggunakan ROW_NUMBER() OVER (PARTITION BY pk ORDER BY updated_at DESC) WHERE rn = 1.
Gold	gold_ atau fct_	Data Marts & Agregasi: Menggabungkan dimensi dan fakta untuk metrik kepatuhan, laporan harian, KPI eksekutif, dan dashboard visualisasi.

---

2. Contoh Struktur Model Silver

Berkas dbt/models/silver/silver_environmental_permits_clean.sql:

{{ config(
    materialized='table',
    schema='silver'
) }}

WITH source_data AS (
    SELECT 
        permit_number,
        UPPER(TRIM(company_name)) AS company_name,
        permit_type,
        CAST(issued_date AS DATE) AS issued_date,
        UPPER(TRIM(status)) AS status,
        updated_at,
        ROW_NUMBER() OVER (
            PARTITION BY permit_number 
            ORDER BY updated_at DESC
        ) AS row_num
    FROM {{ source('bronze', 'environmental_permits') }}
)

SELECT 
    permit_number,
    company_name,
    permit_type,
    issued_date,
    status,
    updated_at
FROM source_data
WHERE row_num = 1

---

3. Contoh Struktur Model Gold

Berkas dbt/models/gold/gold_permits_summary_daily.sql:

{{ config(
    materialized='table',
    schema='gold'
) }}

SELECT 
    permit_type,
    status,
    COUNT(permit_number) AS total_permits,
    MIN(issued_date) AS earliest_issued,
    MAX(issued_date) AS latest_issued,
    NOW() AS calculated_at
FROM {{ ref('silver_environmental_permits_clean') }}
GROUP BY permit_type, status
ORDER BY total_permits DESC