merica-pipelines/docs/pipeline_specifications.md

# 📑 Spesifikasi Sintaksis Konfigurasi Pipeline (YAML Specification)

Dokumen ini menjelaskan struktur kontrak data deklaratif (*declarative pipeline contract*) untuk berkas-berkas konfigurasi YAML di repositori **`merica-pipelines/configs/`**.

---

## 1. Skema Tingkat Atas (Top-Level Schema)

Setiap file pipeline YAML mendefinisikan properti berikut:

```yaml
pipeline_id: "string"           # Unik, identifier teknis pipeline (snake_case)
pipeline_name: "string"         # Nama yang dapat dibaca manusia (Human-readable)
description: "string"           # Deskripsi fungsional pipeline
schedule_interval: "cron/expr"  # Jadwal crontab atau preset (@daily, @hourly, None)
tags:                           # Array string untuk pengelompokan di UI
  - "category"
  - "layer"

source:                         # Konfigurasi sumber data hulu
  type: "postgres|gws|api"
  connection_id: "conn_id"
  # Parameter spesifik sumber...

target:                         # Konfigurasi tujuan hilir
  bucket: "dlh-bronze"
  format: "parquet|json|iceberg"
  # Parameter spesifik target...

governance:                     # Metadata tata kelola (OpenMetadata)
  owner: "nama_tim_atau_email"
  tier: "Tier1|Tier2|Tier3|Tier4|Tier5"
  domain: "nama_domain_bisnis"
```

---

## 2. Parameter Berdasarkan Tipe Sumber (*Source Types*)

### 2.1 Sumber Database Transaksional (`type: "postgres"`)
```yaml
source:
  type: "postgres"
  connection_id: "conn_postgres_source_db"
  table: "public.environmental_permits"
  mode: "incremental"          # "full-refresh" atau "incremental"
  primary_key: "permit_number"
  update_key: "updated_at"     # Kolom watermark waktu
```

### 2.2 Sumber Spreadsheet Google Workspace (`type: "gws"`)
```yaml
source:
  type: "gws"
  connection_id: "conn_gws_service_account"
  spreadsheet_id: "1A2B3C..."
  sheet_name: "Laporan Sampah"
  header_row: 1
  cleaning_rules:
    fill_empty: "ffill"        # Forward fill merged cells
    clean_numeric_columns:
      - "total_volume"
      - "biaya_retribusi"
```

### 2.3 Sumber Telemetri REST API (`type: "api"`)
```yaml
source:
  type: "api"
  connection_id: "conn_spku_api_service"
  endpoint: "/v1/sensors/realtime"
  method: "GET"
  lookback_window_hours: 3     # Menarik data dari (now - 3h) s/d now
  response_path: "data.readings"
```

---

## 3. Integrasi Validasi Kualitas (Great Expectations)

Anda dapat menyematkan aturan validasi kualitas data deklaratif langsung di dalam blok YAML:

```yaml
quality_gate:
  engine: "great_expectations"
  on_failure: "warn|fail"      # "warn": catat log; "fail": hentikan pipeline
  expectations:
    - expectation_type: "expect_column_values_to_not_be_null"
      kwargs:
        column: "permit_number"
    - expectation_type: "expect_column_values_to_be_between"
      kwargs:
        column: "pm25"
        min_value: 0.0
        max_value: 500.0
```