84 lines
3.1 KiB
Markdown
84 lines
3.1 KiB
Markdown
|
# Dremio OSS — Konfigurasi `deploy/`
|
||
|
|
|
||
|
|
Dremio ditambahkan sebagai **query engine paralel** di atas catalog Nessie yang
|
||
|
|
sama dengan Trino & StarRocks (membuktikan dua engine berbagi satu catalog tanpa
|
||
|
|
korupsi metadata).
|
||
|
|
|
||
|
|
| Item | Nilai |
|
||
|
|
|---|---|
|
||
|
|
| Image | `dremio/dremio-oss:${DREMIO_VERSION:-25.2.0}` |
|
||
|
|
| Web UI | `http://localhost:${DREMIO_UI_PORT:-12947}` |
|
||
|
|
| JDBC | `${DREMIO_JDBC_PORT:-12010}` → container `31010` |
|
||
|
|
| Arrow Flight | `${DREMIO_FLIGHT_PORT:-12011}` → container `32010` |
|
||
|
|
| Auth | **Lokal** (bukan OIDC — Dremio OSS tidak menyediakan OIDC) |
|
||
|
|
| Penyimpanan internal | Volume `dremio-data` (bukan PostgreSQL) |
|
||
|
|
|
||
|
|
## First-run & Source Nessie — otomatis via `bootstrap.sh` (Step 9)
|
||
|
|
|
||
|
|
`bootstrap.sh` **Step 9** menangani first-run admin + registrasi source Nessie
|
||
|
|
secara idempoten (aman diulang):
|
||
|
|
|
||
|
|
1. Tunggu Dremio siap.
|
||
|
|
2. Buat admin first-run bila belum ada — `PUT /apiv2/bootstrap/firstuser`
|
||
|
|
(body flat, tanpa auth; `400` = sudah pernah dibuat → dilewati).
|
||
|
|
3. Panggil `scripts/dremio/register-nessie-source.sh` untuk source Nessie.
|
||
|
|
|
||
|
|
Kredensial default (override lewat `.env`):
|
||
|
|
|
||
|
|
| Variabel | Default |
|
||
|
|
|---|---|
|
||
|
|
| `DREMIO_USER` | `admin` |
|
||
|
|
| `DREMIO_PASSWORD` | `dremio_admin_DevOnly1` |
|
||
|
|
| `DREMIO_ADMIN_EMAIL` | `admin@sicabe.local` |
|
||
|
|
|
||
|
|
Jalankan hanya step ini:
|
||
|
|
|
||
|
|
```bash
|
||
|
|
cd deploy
|
||
|
|
./bootstrap.sh --step 9
|
||
|
|
```
|
||
|
|
|
||
|
|
### Manual (alternatif)
|
||
|
|
|
||
|
|
1. Buka `http://localhost:12947` → buat akun admin pertama (form first-run).
|
||
|
|
2. Daftarkan source Nessie lewat script:
|
||
|
|
|
||
|
|
```bash
|
||
|
|
cd deploy
|
||
|
|
set -a; source config/environment.defaults.sh; source .env; set +a
|
||
|
|
DREMIO_USER=<admin> DREMIO_PASSWORD='<password>' \
|
||
|
|
./scripts/dremio/register-nessie-source.sh
|
||
|
|
```
|
||
|
|
|
||
|
|
Script: login (`/apiv2/login`, header `Authorization: _dremio<token>`) → cek
|
||
|
|
existing (`/api/v3/catalog/by-path/<name>`) → `POST /api/v3/catalog`
|
||
|
|
(`type=NESSIE`). Idempotent (source sudah ada → sukses tanpa buat ulang).
|
||
|
|
|
||
|
|
> **Temuan (diuji 2026-09-10, Dremio OSS 25.2.0):** klaim docs bahwa Catalog API
|
||
|
|
> "unable to retrieve or manage Nessie sources" (DX-64994) **tidak akurat** —
|
||
|
|
> create **dan** GET by-path berhasil. Kunci config yang benar:
|
||
|
|
> - **JANGAN** sertakan `storageType` (ditolak: `Invalid value found at: storageType`).
|
||
|
|
> - `awsRootPath` format **`/bucket/path`** (TANPA prefix `s3://`), mis. `/iceberg-data/warehouse`.
|
||
|
|
> - `credentialType=ACCESS_KEY`, `awsAccessKey`/`awsAccessSecret`, `secure=false`.
|
||
|
|
> - `propertyList`: `fs.s3a.path.style.access=true`, `fs.s3a.endpoint=minio:9000`,
|
||
|
|
> `dremio.s3.compat=true`.
|
||
|
|
|
||
|
|
Verifikasi query (data yang sama ditulis Trino):
|
||
|
|
```sql
|
||
|
|
SELECT * FROM nessie.demo.t ORDER BY id; -- 2 baris: alpha, beta
|
||
|
|
```
|
||
|
|
|
||
|
|
### Alternatif: via UI
|
||
|
|
|
||
|
|
**Add Source → Nessie** dengan field: endpoint `http://nessie:19120/api/v2`,
|
||
|
|
Authentication `None`, Storage AWS S3/MinIO (path-style, endpoint `minio:9000`),
|
||
|
|
access key/secret MinIO.
|
||
|
|
|
||
|
|
## Catatan
|
||
|
|
|
||
|
|
- **Dua engine berbagi satu Nessie catalog**: tulis dari Dremio (CTAS/INSERT)
|
||
|
|
langsung terlihat di Trino dan sebaliknya. Lakukan tulis secara bergantian
|
||
|
|
(bukan simultan) untuk menghindari konflik concurrency Nessie.
|
||
|
|
- Nessie **tidak mendukung Iceberg spec v3** → gunakan `format-version=2`.
|
||
|
|
- Referensi: `experiments/nessie-dremio/README.md` (H2 terbukti).
|