merica-pipelines/docs/pipeline_specifications.md

📑 Spesifikasi Sintaksis Konfigurasi Pipeline (YAML Specification)

Dokumen ini menjelaskan struktur kontrak data deklaratif (declarative pipeline contract) untuk berkas-berkas konfigurasi YAML di repositori merica-pipeline/configs/.

---

1. Skema Tingkat Atas (Top-Level Schema)

Setiap file pipeline YAML mendefinisikan properti berikut:

pipeline_id: "string"           # Unik, identifier teknis pipeline (snake_case)
pipeline_name: "string"         # Nama yang dapat dibaca manusia (Human-readable)
description: "string"           # Deskripsi fungsional pipeline
schedule_interval: "cron/expr"  # Jadwal crontab atau preset (@daily, @hourly, None)
tags:                           # Array string untuk pengelompokan di UI
  - "category"
  - "layer"

source:                         # Konfigurasi sumber data hulu
  type: "postgres|gws|api"
  connection_id: "conn_id"
  # Parameter spesifik sumber...

target:                         # Konfigurasi tujuan hilir
  bucket: "dlh-bronze"
  format: "parquet|json|iceberg"
  # Parameter spesifik target...

governance:                     # Metadata tata kelola (OpenMetadata)
  owner: "nama_tim_atau_email"
  tier: "Tier1|Tier2|Tier3|Tier4|Tier5"
  domain: "nama_domain_bisnis"

---

2. Parameter Berdasarkan Tipe Sumber (Source Types)

2.1 Sumber Database Transaksional (type: "postgres")

source:
  type: "postgres"
  connection_id: "conn_postgres_source_db"
  table: "public.environmental_permits"
  mode: "incremental"          # "full-refresh" atau "incremental"
  primary_key: "permit_number"
  update_key: "updated_at"     # Kolom watermark waktu

2.2 Sumber Spreadsheet Google Workspace (type: "gws")

source:
  type: "gws"
  connection_id: "conn_gws_service_account"
  spreadsheet_id: "1A2B3C..."
  sheet_name: "Laporan Sampah"
  header_row: 1
  cleaning_rules:
    fill_empty: "ffill"        # Forward fill merged cells
    clean_numeric_columns:
      - "total_volume"
      - "biaya_retribusi"

2.3 Sumber Telemetri REST API (type: "api")

source:
  type: "api"
  connection_id: "conn_spku_api_service"
  endpoint: "/v1/sensors/realtime"
  method: "GET"
  lookback_window_hours: 3     # Menarik data dari (now - 3h) s/d now
  response_path: "data.readings"

---

3. Integrasi Validasi Kualitas (Great Expectations)

Anda dapat menyematkan aturan validasi kualitas data deklaratif langsung di dalam blok YAML:

quality_gate:
  engine: "great_expectations"
  on_failure: "warn|fail"      # "warn": catat log; "fail": hentikan pipeline
  expectations:
    - expectation_type: "expect_column_values_to_not_be_null"
      kwargs:
        column: "permit_number"
    - expectation_type: "expect_column_values_to_be_between"
      kwargs:
        column: "pm25"
        min_value: 0.0
        max_value: 500.0