Mengirim Metrics Linux OpenTelemetry ke OpenSearch menggunakan Data Prepper

Menggunakan OpenTelemetry untuk sebagai collector host metrics di linux, ini bisa digunakan pada sistem operasi linux. Seperti Node Exporter yang biasa digunakan untuk Prometheus, opentelemetry collector juga bisa untuk collect host metrics, seperti mengumpulkan metrik utilisasi sistem (host metrics) seperti CPU, Memori, Disk, dan Jaringan.

Saat ini saya menggunakan usecase untuk mengirim metricsnya ke OpenSearch melalui Data Prepper. Data Prepper sebagai middleware (penerima) sebelum dikirimkan ke OpenSearch.

Table of Contents

Install dan Konfigurasi OpenTelemetry Collector

Check versi Terbaru OpenTelemetry

Lakukan check versi latest OpenTelemetry dari repo OpenTelemetry dengan menjalankan perintah dibawah ini.

LATEST_VERSION=$(curl -s https://api.github.com/repos/open-telemetry/opentelemetry-collector-releases/releases/latest | grep '"tag_name":' | sed -E 's/.*"v([^"]+)".*/\1/')
    echo "Mengunduh OpenTelemetry Collector Contrib versi: $LATEST_VERSION"

Download OpenTelemetry versi latest

Setelah sudah melakukan pengecekan versi latest OpenTelemetry, download versi latest OpenTelemetry dengan perintah dibawah.

Ubuntu, Debian dan turunannya

Jika menggunakan distro linux ubuntu, debian dan turunannya bisa gunakan perintah dibawah:

wget "https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v${LATEST_VERSION}/otelcol-contrib_${LATEST_VERSION}_linux_amd64.deb"

RHEL, Rocky dan turunannya

Jika menggunakan distro linux RHEL, Rocky dan turunannya bisa gunakan perintah dibawah:

wget "https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v${LATEST_VERSION}/otelcol-contrib_${LATEST_VERSION}_linux_amd64.rpm"

Install OpenTelemetry

Setelah proses download selesai, lakukan installasi dengan perintah dibawah ini.

Ubuntu, Debian dan turunannya

sudo dpkg -i "otelcol-contrib_${LATEST_VERSION}_linux_amd64.deb"

RHEL, Rocky dan turunannya

sudo rpm -ivh "otelcol-contrib_${LATEST_VERSION}_linux_amd64.rpm"

Konfigurasi OpenTelemetry

Setelah proses install sudah selesai, lakukan konfigurasi dengan melakukan edit pada file config.yml yang ada di directory /etc/otelcol-contrib/

karena usecasenya disini saya ingin store datanya ke OpenSearch melalui Data Prepper, maka untuk konfigurasi exporter endpoint diarahkan ke URL Endpoint Data Prepper di IP Address dan port 10.10.5.87:9902

receivers:
  hostmetrics:
    collection_interval: 30s
    scrapers:
      cpu:
      disk:
      filesystem:
      load:
      memory:
      network:
      process:
      processes:
      paging:

processors:
  resource:
    attributes:
      - key: hostname
        value: "docker-honet"
        action: insert
      - key: ip_address
        value: "192.168.201.18"
        action: insert

exporters:
  otlp:
    endpoint: "10.10.5.87:9902"
    tls:
      insecure: true

service:
  pipelines:
    metrics:
      receivers: [hostmetrics]
      processors: [resource]
      exporters: [otlp]

Ada beberapa metrics yang harus diaktifkan jika ingin menggunakan metrics tersebut, untuk melakukan enablenya bisa dengan merubah konfigurasi openTelemetry Collector (config.yml) dengan konfigurasi berikut ini:

Pada bagian receivers ada perbedaan yang signifikan yaitu ada beberapa metrics yang harus diberikan parameter true agar metrics tersebut bisa muncul di index OpenSearch.

receivers:
  # ==========================================
  # FOR METRICS
  # ==========================================
  hostmetrics:
    collection_interval: 30s
    scrapers:
      cpu:
        metrics:
          system.cpu.logical.count:
            enabled: true
          system.cpu.time:
            enabled: true
          system.cpu.frequency:
            enabled: true
          system.cpu.physical.count:
            enabled: true
          system.cpu.utilization:
            enabled: true

      disk:
        metrics:
          system.disk.io:
            enabled: true
          system.disk.io_time:
            enabled: true
          system.disk.merged:
            enabled: true
          system.disk.operation_time:
            enabled: true
          system.disk.operations:
            enabled: true
          system.disk.pending_operations:
            enabled: true
          system.disk.weighted_io_time:
            enabled: true

      filesystem:
        metrics:
          system.filesystem.inodes.usage:
            enabled: true
          system.filesystem.usage:
            enabled: true
          system.filesystem.utilization:
            enabled: true

      load:
        metrics:
          system.cpu.load_average.15m:
            enabled: true
          system.cpu.load_average.1m:
            enabled: true
          system.cpu.load_average.5m:
            enabled: true

      memory:
        metrics:
          system.memory.usage:
            enabled: true
          system.linux.memory.available:
            enabled: true
          system.linux.memory.dirty:
            enabled: true
          system.memory.limit:
            enabled: true
          system.memory.linux.hugepages.limit:
            enabled: true
          system.memory.linux.hugepages.page_size:
            enabled: true
          system.memory.linux.hugepages.reserved:
            enabled: true
          system.memory.linux.hugepages.surplus:
            enabled: true
          system.memory.linux.hugepages.usage:
            enabled: true
          system.memory.linux.hugepages.utilization:
            enabled: true
          system.memory.linux.shared:
            enabled: true
          system.memory.page_size:
            enabled: true
          system.memory.utilization:
            enabled: true

      network:
        metrics:
          system.network.connections:
            enabled: true
          system.network.dropped:
            enabled: true
          system.network.errors:
            enabled: true
          system.network.io:
            enabled: true
          system.network.packets:
            enabled: true

      paging:
        metrics:
          system.paging.faults:
            enabled: true
          system.paging.operations:
            enabled: true
          system.paging.usage:
            enabled: true
          system.paging.utilization:
            enabled: true
            
      processes:
        metrics:
          system.processes.count:
            enabled: true
          system.processes.created:
            enabled: true

      process:
        mute_process_name_error: true
        mute_process_exe_error: true
        mute_process_io_error: true
        metrics:
          process.cpu.time:
            enabled: true
          process.disk.io:
            enabled: true
          process.memory.usage:
            enabled: true
          process.memory.virtual:
            enabled: true
          process.context_switches:
            enabled: true
          process.cpu.utilization:
            enabled: true
          process.disk.operations:
            enabled: true
          process.handles:
            enabled: true
          process.memory.utilization:
            enabled: true
          process.threads:
            enabled: true
          process.open_file_descriptors:
            enabled: false
          process.signals_pending:
            enabled: false
          process.uptime:
            enabled: true

processors:
  attributes:
    actions:
      - key: "environment"
        value: "production"
        action: insert
      - key: "hostname"
        value: "dev-w1.onyx.id"
        action: insert
      - key: "ip_address"
        value: "172.17.8.82"
        action: insert

exporters:
  otlp/node1: 
    endpoint: "172.17.8.147:4911"
    tls:
      insecure: true
  otlp/node2: 
    endpoint: "10.10.5.87:4911"
    tls:
      insecure: true

service:
  pipelines:
    # --- Metrics
    metrics:
      receivers: [hostmetrics]
      processors: [attributes]
      exporters: [otlp/node1, otlp/node2]

Start Service OpenTelemetry Collector

Setelah konfigurasi sudah selesai, lakukan start service OpenTelemetry Collector dengan perintah berikut:

systemctl start otelcol-contrib
systemctl status otelcol-contrib

Konfigurasi Pipeline Data Prepper

Membuat dan Konfigurasi Pipeline Data Prepper

Buat pipeline Data Prepper di directory pipeline pada working directory Data Prepper, untuk services Data Prepper saya terinstall atau working directory ada di /opt/data-prepper/pipeline

Tambahkan konfigurasi pipeline Data Prepper dibawah ini, pastikan untuk port sourcenya sesuai dengan konfigurasi Exporter OpenTelemetry yang sudah di konfigurasi sebelumnya.

metrics-pipeline:
  source:
    otel_metrics_source:
      port: 9902       
      ssl: false
      
  processor:
    - date:
        from_time_received: true
        destination: "timestamp_ingest"
        
  sink:
    - opensearch:
        hosts: ["https://10.x.x.x:9220"]
        username: "admin"
        password: "xxxxxx"
        insecure: true
        index: "opentelemetry-metrics-raw-%{yyyy.MM.dd}"

Start/Restart Services Data Prepper

Setelah sudah melakukan konfigurasi pipeline Data Prepper lakukan start atau restart service Data Prepper.

systemctl start data-prepper
systemctl status data-prepper

Catatan: Pastikan tidak ada error pada log Data Prepper dan OpenTelemetry Collector sebelum check data di OpenSearch

Check Data di OpenSearch

Berikutnya check data di index OpenSearch, apakah datanya sudah masuk dengan menggunakan perintah curl atau query langsung dari dashboard OpenSearch

GET fluentbit-metrics-node-*/_search

curl -k -u 'admin:admin' -X GET "https://10.10.5.89:9220/fluentbit-metrics-node-*/_search?pretty"

List Metrics

Berikut list metrics yang disediakan oleh OpenTelemetry

process.cpu.time
process.memory.usage
process.memory.virtual
process.cpu.utilization
system.cpu.time
process.context_switches
process.paging.faults
process.memory.utilization
process.threads
process.uptime
system.disk.io
system.disk.merged
system.disk.operation_time
system.disk.operations
system.network.connections
system.filesystem.usage
system.disk.io_time
system.disk.pending_operations
system.disk.weighted_io_time
system.memory.usage
system.filesystem.inodes.usage
system.network.dropped
system.network.errors
system.network.io
system.network.packets
process.disk.io
system.cpu.utilization
system.cpu.load_average.15m
system.cpu.load_average.1m
system.cpu.load_average.5m
system.memory.utilization
system.processes.count
system.cpu.frequency
system.paging.operations
system.cpu.logical.count
system.filesystem.utilization
system.paging.faults
system.paging.usage
system.paging.utilization
process.disk.operations
system.memory.linux.hugepages.usage
system.cpu.physical.count
system.processes.created
system.linux.memory.available
system.memory.limit
system.linux.memory.dirty
system.memory.linux.hugepages.limit
system.memory.linux.hugepages.page_size
system.memory.linux.hugepages.reserved
system.memory.linux.hugepages.surplus
system.memory.linux.shared
system.memory.page_size
process.open_file_descriptors
process.signals_pending

Refrensi:

https://opentelemetry.io/
https://docs.github.com/en/[email protected]/admin/monitoring-and-managing-your-instance/monitoring-your-instance/opentelemetry-metrics/configuring-opentelemetry-for-your-instance
https://github.com/open-telemetry