Artikel ini akan membagikan panduan step-by-step untuk membangun sistem log monitoring menggunakan OpenSearch. Kita akan penyimpanan log (centralized log) dari container Docker dengan memanfaatkan Fluent Bit sebagai agent dan Data Prepper sebagai middleware dan log tersebut akan disimpan di OpenSearch.
Sebelum melanjutkan pastikan sudah memiliki Server/VM/Host/Node yang terinstall OpenSearch, Data Prepper dan Install Agent Fluent Bit di node Docker yang ingin di collect lognya untuk dikirim ke opensearch.
Table of Contents
Usecase
Sebagai usecase, disini saya akan menggunakan docker container n8n yang berjalan diatas docker lalu log container n8n tersebut akan di kirim ke opensearch menggunakan Fluent Bit melalui Data Prepper.
Setup Fluent Bit
Pastikan Fluent Bit sudah terinstall di node atau server docker. Selanjutnya, tambahkan parser berikut di file /etc/fluent-bit/parser.conf untuk melakukan extract id container docker.
[PARSER]
Name extract_container_id
Format regex
Regex /var/lib/docker/containers/(?<container_id>[^/]+)/.*
Langkah berikutnya lakukan konfigurasi fluent-bit.conf dengan konfigurasi dibawah ini.
# =========================================================
# Config docker container n8n
# =========================================================
[INPUT]
Name tail
Tag docker-n8n
Path /var/lib/docker/containers/c25bd49d4a53*/*-json.log,/var/lib/docker/containers/a6d6b0a20cb3*/*-json.log,/var/lib/docker/containers/75ed7b1de490*/*-json.log
Parser docker
DB /var/log/docker_hephaestus.db
Mem_Buf_Limit 20MB
Path_Key filepath
Skip_Long_Lines Off
Buffer_Chunk_Size 512k
Buffer_Max_Size 5M
[FILTER]
Name parser
Match docker-n8n
Key_Name filepath
Parser extract_container_id
Reserve_Data On
[FILTER]
Name modify
Match docker-n8n
Condition Key_value_matches container_id ^c25bd49d4a53.*
Add container_name n8n-n8n
[FILTER]
Name modify
Match docker-n8n
Condition Key_value_matches container_id ^a6d6b0a20cb3.*
Add container_name n8n-postgres
[FILTER]
Name modify
Match docker-n8n
Condition Key_value_matches container_id ^75ed7b1de490.*
Add container_name n8n-mysql
[FILTER]
Name modify
Match docker-n8n
Add hostname docker-honet
Add ip_address 192.168.201.18
Add environment production
[OUTPUT]
Name http
Match docker-n8n
Host 10.10.5.87
Port 9992
URI /log/ingest
Format json
Setup Data Prepper
Pastikan Data Prepper sudah terinstall, jika belum silakan lakukan install dan setup Data Prepper terlebih dahulu. Jika sudah, lanjutkan untuk membuat pipeline berikut di directory pipelines.
Pastikan untuk sourcenya menggunakan http lalu port samakan seperti Fluent Bit. Karena Data Prepper akan menerima log melalui endpoint dengan port tersebut.
docker-n8n-pipeline:
source:
http:
port: 9992
ssl: false
processor:
- trim_string:
with_keys:
- "log"
- date:
from_time_received: true
destination: "timestamp_ingest"
sink:
- opensearch:
hosts: [ "https://10.10.5.89:9220" ]
username: "admin"
password: "admin"
insecure: true
index: "docker-logs-n8n-%{yyyy.MM.dd}"
Restart Service Data Prepper & Fluent Bit
Setelah itu lakukan restart service Data Prepper dan Fluent Bit dan pastikan tidak ada error. Di Fluent Bit, ketika berhasil mengirim log ke Data Prepper akan ada log seperti berikut:
2026-07-17T16:02:16.487295+07:00 docker-honet fluent-bit[3442469]: [2026/07/17 16:02:16.485] [ info] [output:http:http.1] 10.10.5.87:9992, HTTP status=200
2026-07-17T16:02:16.487581+07:00 docker-honet fluent-bit[3442469]: 200 OK
2026-07-17T16:02:21.485858+07:00 docker-honet fluent-bit[3442469]: [2026/07/17 16:02:21.484] [ info] [output:http:http.1] 10.10.5.87:9992, HTTP status=200
2026-07-17T16:02:21.486040+07:00 docker-honet fluent-bit[3442469]: 200 OK
2026-07-17T16:02:26.477000+07:00 docker-honet fluent-bit[3442469]: [2026/07/17 16:02:26.475] [ info] [output:http:http.1] 10.10.5.87:9992, HTTP status=200
2026-07-17T16:02:26.477338+07:00 docker-honet fluent-bit[3442469]: 200 OK
2026-07-17T16:02:31.476122+07:00 docker-honet fluent-bit[3442469]: [2026/07/17 16:02:31.474] [ info] [output:http:http.1] 10.10.5.87:9992, HTTP status=200
2026-07-17T16:02:31.476289+07:00 docker-honet fluent-bit[3442469]: 200 OK
2026-07-17T16:02:36.477089+07:00 docker-honet fluent-bit[3442469]: [2026/07/17 16:02:36.475] [ info] [output:http:http.1] 10.10.5.87:9992, HTTP status=200
2026-07-17T16:02:36.477302+07:00 docker-honet fluent-bit[3442469]: 200 OK
2026-07-17T16:02:41.476982+07:00 docker-honet fluent-bit[3442469]: [2026/07/17 16:02:41.475] [ info] [output:http:http.1] 10.10.5.87:9992, HTTP status=200
2026-07-17T16:02:41.477238+07:00 docker-honet fluent-bit[3442469]: 200 OK
2026-07-17T16:02:51.475104+07:00 docker-honet fluent-bit[3442469]: [2026/07/17 16:02:51.473] [ info] [output:http:http.1] 10.10.5.87:9992, HTTP status=200
2026-07-17T16:02:51.475949+07:00 docker-honet fluent-bit[3442469]: 200 OK
2026-07-17T16:02:56.475342+07:00 docker-honet fluent-bit[3442469]: [2026/07/17 16:02:56.474] [ info] [output:http:http.1] 10.10.5.87:9992, HTTP status=200
2026-07-17T16:02:56.475574+07:00 docker-honet fluent-bit[3442469]: 200 OK
2026-07-17T16:03:01.477516+07:00 docker-honet fluent-bit[3442469]: [2026/07/17 16:03:01.475] [ info] [output:http:http.1] 10.10.5.87:9992, HTTP status=200
2026-07-17T16:03:01.477949+07:00 docker-honet fluent-bit[3442469]: 200 OK
Check Index OpenSearch
Setelah itu, lakukan check index dengan cara curl atau melihat langsung melalui OpenSearch Dashboard. Jika menggunakan curl bisa menggunakan command berikut:
curl -X GET "https://10.10.5.89:9220/docker-logs-n8n-*/_search?pretty" \
-u "admin:admin" \
--insecure
Jika melalui OpenSearch dashboard bisa akses menu devtools lalu jalankan query berikut:
GET docker-logs-n8n-*/_search
