87 lines
2.3 KiB
Markdown
87 lines
2.3 KiB
Markdown
|
# 🧊 Panduan Pemodelan Data Medallion dbt (Lakehouse Transformations)
|
||
|
|
|
||
|
|
Dokumen ini menjelaskan struktur dan konvensi penulisan model transformasi **dbt** (Data Build Tool) pada platform MERICA Data Lakehouse.
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## 1. Konvensi Penamaan Model Medallion
|
||
|
|
|
||
|
|
Transformasi dbt diorganisasikan ke dalam subdirektori `dbt/models/`:
|
||
|
|
|
||
|
|
```text
|
||
|
|
dbt/models/
|
||
|
|
├── silver/
|
||
|
|
│ ├── silver_environmental_permits_clean.sql
|
||
|
|
│ ├── silver_gws_waste_reports_clean.sql
|
||
|
|
│ └── silver_spku_sensor_clean.sql
|
||
|
|
└── gold/
|
||
|
|
├── gold_permits_summary_daily.sql
|
||
|
|
├── gold_waste_management_summary.sql
|
||
|
|
└── gold_ispu_daily_summary.sql
|
||
|
|
```
|
||
|
|
|
||
|
|
| Layer | Prefix Penamaan | Tujuan & Prinsip Transformasi |
|
||
|
|
| :--- | :--- | :--- |
|
||
|
|
| **Silver** | `silver_` atau `int_` | **Pembersihan & Deduplikasi**: Casting tipe data yang konsisten, penanganan nilai null, pemfilteran duplikasi menggunakan `ROW_NUMBER() OVER (PARTITION BY pk ORDER BY updated_at DESC) WHERE rn = 1`. |
|
||
|
|
| **Gold** | `gold_` atau `fct_` | **Data Marts & Agregasi**: Menggabungkan dimensi dan fakta untuk metrik kepatuhan, laporan harian, KPI eksekutif, dan dashboard visualisasi. |
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## 2. Contoh Struktur Model Silver
|
||
|
|
|
||
|
|
Berkas `dbt/models/silver/silver_environmental_permits_clean.sql`:
|
||
|
|
```sql
|
||
|
|
{{ config(
|
||
|
|
materialized='table',
|
||
|
|
schema='silver'
|
||
|
|
) }}
|
||
|
|
|
||
|
|
WITH source_data AS (
|
||
|
|
SELECT
|
||
|
|
permit_number,
|
||
|
|
UPPER(TRIM(company_name)) AS company_name,
|
||
|
|
permit_type,
|
||
|
|
CAST(issued_date AS DATE) AS issued_date,
|
||
|
|
UPPER(TRIM(status)) AS status,
|
||
|
|
updated_at,
|
||
|
|
ROW_NUMBER() OVER (
|
||
|
|
PARTITION BY permit_number
|
||
|
|
ORDER BY updated_at DESC
|
||
|
|
) AS row_num
|
||
|
|
FROM {{ source('bronze', 'environmental_permits') }}
|
||
|
|
)
|
||
|
|
|
||
|
|
SELECT
|
||
|
|
permit_number,
|
||
|
|
company_name,
|
||
|
|
permit_type,
|
||
|
|
issued_date,
|
||
|
|
status,
|
||
|
|
updated_at
|
||
|
|
FROM source_data
|
||
|
|
WHERE row_num = 1
|
||
|
|
```
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## 3. Contoh Struktur Model Gold
|
||
|
|
|
||
|
|
Berkas `dbt/models/gold/gold_permits_summary_daily.sql`:
|
||
|
|
```sql
|
||
|
|
{{ config(
|
||
|
|
materialized='table',
|
||
|
|
schema='gold'
|
||
|
|
) }}
|
||
|
|
|
||
|
|
SELECT
|
||
|
|
permit_type,
|
||
|
|
status,
|
||
|
|
COUNT(permit_number) AS total_permits,
|
||
|
|
MIN(issued_date) AS earliest_issued,
|
||
|
|
MAX(issued_date) AS latest_issued,
|
||
|
|
NOW() AS calculated_at
|
||
|
|
FROM {{ ref('silver_environmental_permits_clean') }}
|
||
|
|
GROUP BY permit_type, status
|
||
|
|
ORDER BY total_permits DESC
|
||
|
|
```
|