Fluent Bit

Fluent Bit integration with PacketAI

Input

The following section defines the fluent-bit input section. for more details see fluent-bit input here

[INPUT]
    Name        tail
    # setting the path for the file to tail
    Path        /var/log/**/*.log
    Tag   mac_logs
[INPUT]
    Name        cpu
    # setting the path for the file to tail
    Tag   my_cpu_metrics
    Interval_Sec 10

Filters

We use filters to add some PacketAI metadata to easily process. It's important to have packetai.cluster_name and packetai.app_name. These 2 fields help us to quickly add logpatterns and more easy to manage opensearch indexes.

// Some code
[FILTER]
    Name record_modifier
    Match *_logs //here it's matches tag: mac_logs, it's impoortant that you tag your fluent-bit inputs
    Record hostname ${HOSTNAME}
    Record packetai.cluster_name macos
    Record packetai.app_name varlogs # For metrics this is optional.

Output

The output section is used to send the logs/metrics to PacketAI.

Logs

For logs the below output configuration is being used. We need to configure the Host, Header, and Match sections.

The Host varies depending on your packetai URL. Please check with PacketAI team if you are not sure which URL to use.

Headers X-PAI-TOKEN and X-PAI-IID needs to get it from packetai URL. After login, you can navigate to Deploy PacketAI/Agent/logstash click on credentials It will show X-PAI-TOKEN and X-PAI-IID, copy these values and replace YOUR_PAI_TOKEN and YOUR_PAI_IID.

[OUTPUT]
    Name  http
    Port 443
    Tls On
    Host vector-ingester-logpatterns.packetai.co # Check your URL, it might be different from this
    URI /fluent/log
    Header X-PAI-TOKEN YOUR_PAI_TOKEN # This token needs to get it from packetai UI section
    Header X-PAI-IID YOUR_PAI_IID
    Format json
    json_date_format iso8601
    json_date_key timestamp
    Match *_logs

Metrics

Similarly for Metrics we need configure the Host, Header, and Match sections.

The Host varies depending on your packetai URL. Please check with PacketAI team if you are not sure which URL to use.

Headers X-PAI-TOKEN and X-PAI-IID needs to get it from packetai URL. After login, you can navigate to Deploy PacketAI/Agent/logstash click on credentials It will show X-PAI-TOKEN and X-PAI-IID, copy these values and replace YOUR_PAI_TOKEN and YOUR_PAI_IID.

// Some code

[OUTPUT]
    Name  http
    Port 443
    Tls On
    Host vector-ingester-logpatterns.packetai.co # Check your URL, it might be different from this
    URI /fluent/metric
    Header X-PAI-TOKEN YOUR_PAI_TOKEN # This token needs to get it from packetai UI section
    Header X-PAI-IID YOUR_PAI_IID # This token needs to get it from packetai UI section
    Format json
    json_date_format iso8601
    json_date_key timestamp
    Match *_metrics

Example configuration

[INPUT]
    name            cpu
    tag             my_cpu_metrics
    Interval_Sec 10

[FILTER]
    Name record_modifier
    Match *
    Record hostname ${HOSTNAME}
    Record packetai.cluster_name macos
    Record packetai.app_name macos

[OUTPUT]
    Name  http
    Port 443
    Tls On
    Host vector-ingester-logpatterns.packetai.co
    URI /fluent/metric
    Header X-PAI-TOKEN YOUR_PAI_TOKEN
    Header X-PAI-IID YOUR_PAI_ID
    Format json
    json_date_format iso8601
    json_date_key timestamp
    Match *_metrics