72 lines
4.7 KiB
Markdown
72 lines
4.7 KiB
Markdown
# 📚 Pusat Dokumentasi & Panduan — MERICA
|
|
|
|
Selamat datang di pusat dokumentasi resmi **MERICA** (*Managed Engine for Robust Ingestion & Cloud Airflow*) untuk distribusi produksi mandiri (*Clean Production Appliance*).
|
|
|
|
Dokumentasi ini dirancang terisolasi (*self-contained*) dan menggunakan tautan relatif berbasis Git untuk memudahkan penelusuran baik langsung di repositori Git (GitLab/GitHub), peramban lokal, maupun editor teks.
|
|
|
|
---
|
|
|
|
## 🗺️ Alur Penelusuran Dokumen (Recommended Reading Flow)
|
|
|
|
Ikuti urutan membaca di bawah ini sesuai peran dan kebutuhan implementasi Anda:
|
|
|
|
```mermaid
|
|
flowchart TD
|
|
START(["Mulai di Sini: docs/index.md"]) --> Q{"Apa Kebutuhan Anda?"}
|
|
|
|
Q -->|"Memahami Rancang Bangun & Komponen"| D1["1. Arsitektur Platform\n(architecture.md)"]
|
|
Q -->|"Melakukan Instalasi Appliance Mandiri"| D2["2. Panduan Deployment & Build\n(deployment_guide.md)"]
|
|
Q -->|"Menghubungkan ke Cluster Airflow Existing"| D3["3. Panduan Integrasi Airflow\n(panduan_integrasi_existing_airflow.md)"]
|
|
Q -->|"Merancang Pipeline Data di Canvas"| D4["4. Panduan Pengguna Canvas\n(user_guide.md)"]
|
|
|
|
D1 --> D2
|
|
D2 --> D3
|
|
D3 --> D4
|
|
```
|
|
|
|
| Urutan | Dokumen | Target Pembaca | Fokus Utama |
|
|
| :---: | :--- | :--- | :--- |
|
|
| **1** | [**Arsitektur Platform**](./architecture.md) | Software Architect, Data Architect | Rancang bangun Single-Port Appliance (Port 14000), isolasi perimeter loopback, arsitektur modul stateless, dan integrasi GitOps Airflow 3. |
|
|
| **2** | [**Panduan Deployment & Build**](./deployment_guide.md) | DevOps, System Administrator | Spesifikasi hardware ultra-ringan (~100MB RAM), instalasi sekali-klik via `setup.sh`, build image mandiri, dan upgrade runbook. |
|
|
| **3** | [**Panduan Integrasi Airflow External**](./panduan_integrasi_existing_airflow.md) | Data Platform Engineer, DevOps | Cara menghubungkan MERICA ke Airflow existing via **Metode 1 (Tanpa Rebuild)** atau **Metode 2 (Build Ulang Image Resmi)**, serta pengelolaan starter pipelines. |
|
|
| **4** | [**Panduan Pengguna (User Guide)**](./user_guide.md) | Data Engineer, Analytics Engineer | Panduan visual drag-and-drop di Canvas, katalog komponen lakehouse, GWS discovery, DB introspection & safe preview, dlt REST API, dan Shift-Left dbt testing. |
|
|
|
|
---
|
|
|
|
## ⚡ Matriks Port & Referensi Cepat Jaringan
|
|
|
|
| Layanan | Port Host | Protokol | Visibilitas | Keterangan |
|
|
| :--- | :---: | :---: | :--- | :--- |
|
|
| **MERICA Gateway (Nginx)** | `14000` | HTTP | Eksternal (Publik) | Pintu masuk tunggal: melayani Web Canvas UI (`/*`) dan reverse proxy API (`/api/*`). |
|
|
| **MERICA Backend (FastAPI)** | `8000` | HTTP | Internal Loopback (`127.0.0.1`) | **Tidak diekspos ke host**. Menjalankan engine validasi, parser YAML, dan compiler dbt. |
|
|
| **Apache Airflow 3 API Server** | `14080` | HTTP | Eksternal | Airflow Web UI & REST API eksekusi pipeline. |
|
|
| **PostgreSQL 15 MetaDB** | `14432` | TCP/PG | Eksternal | Database metadata relasional Lakehouse / Airflow. |
|
|
| **MinIO S3 Storage (Console)** | `14090` / `14091` | HTTP | Eksternal | Penyimpanan objek Bronze Parquet & Silver Iceberg. |
|
|
| **OpenMetadata Governance** | `14585` | HTTP | Eksternal | Portal katalog metadata, data lineage, dan data quality. |
|
|
|
|
---
|
|
|
|
## 📁 Struktur Direktori Paket Rilis
|
|
|
|
```text
|
|
release/merica/
|
|
├── .env.example # Template konfigurasi variabel lingkungan
|
|
├── Dockerfile # Resep build mandiri Nginx + FastAPI (supervisord)
|
|
├── README.md # Ringkasan cepat & petunjuk mulai 2 menit
|
|
├── docker-compose.yml # Compose Standalone Appliance (Port 14000)
|
|
├── docker-compose.all-in-one.yml # Compose demo All-in-One (dengan Airflow tertanam)
|
|
├── setup.sh # Skrip instalasi sekali-klik otomatis
|
|
├── docs/ # Pusat dokumentasi terisolasi
|
|
│ ├── index.md # Halaman indeks & panduan navigasi ini
|
|
│ ├── architecture.md # Rancang bangun teknis platform
|
|
│ ├── deployment_guide.md # Panduan instalasi, build, dan pemeliharaan
|
|
│ ├── panduan_integrasi_existing_airflow.md # Integrasi kluster Airflow eksternal
|
|
│ └── user_guide.md # Panduan visual perancangan pipeline di Canvas
|
|
├── pipelines/ # Starter Seed Pipelines (Template Awal)
|
|
│ ├── configs/ # Kontrak deklaratif YAML Airflow
|
|
│ ├── metadata/ # Posisi visual node & edge kanvas ReactFlow
|
|
│ └── dbt/ # Proyek dbt Medallion (Silver & Gold models)
|
|
├── backend/ # Kode sumber backend FastAPI
|
|
└── frontend/ # Kode sumber antarmuka React Canvas UI
|
|
```
|