feat: initial commit for merica-pipeline delivery

2026-10-08 19:33:15 +07:00
commit 42fedab752
46 changed files with 5484 additions and 0 deletions
@@ -0,0 +1,23 @@
+# ==============================================================================
+# MERICA Data Lakehouse - Airflow Multi-Engine Pinned Dependencies
+# Target Runtime: Apache Airflow 3.3.2 / Python 3.11 (registry.sicabe.com/airflow:3.3.2-om203-ol)
+# ==============================================================================
+
+# Fast OLAP Ingestion & Transformation Engine (Embedded / Zero-Bloat)
+duckdb==1.1.3
+
+# Apache Iceberg Lakehouse Table Management & Metadata Tracking
+pyiceberg[s3fs,pyarrow]==0.8.0
+
+# High-Performance Columnar Memory & Storage Format
+pyarrow==17.0.0
+
+# S3 File System & Filesystem Spec for MinIO/Lakehouse Object Storage
+s3fs==2024.9.0
+fsspec==2024.9.0
+
+# NOTE ON PYSPARK:
+# PySpark and Java JDK are deliberately EXCLUDED from this image to prevent bloat (>4GB)
+# and avoid OOM risks on Airflow Scheduler / DAG Processor.
+# Distributed PySpark jobs are dispatched externally via Livy REST API (http://spark-master:8998)
+# or SparkSubmitOperator pointing to an external cluster.