Using Logstash agent

This document describes how to send logs to PacketAI using the logstash http output plugin

Setting keys, cluster name and access control

In order to complete the setup, there are 2 main things to configure :

1. Obtain your keys YOUR_PAI_TOKEN and YOUR_PAI_IID by going to https://logpatterns.packetai.co/deploy/agent. Pick any of the integrations to find your keys.

2. Replace yourclustername with your cluster name in the mutate block (see below). ⚠️Note that even if you are not using a proper cluster, you must provide a value for this variable.

Input conf

Logstash supports several input plugins, the list of input plugins it supports can be found here. The following file input configuration is provided as an example.

input { 
    file {
        path => [
            "/var/log/*.log",
            "/var/log/*/*.log"
        ]
    }
}

Filter conf

Logstash supports several plugins for filtering logs, the more details about several filters (plugins) available here. PacketAI needs some additional fields needs to be attached to each message, see the below mutate section. We need to map the each application message to the following field, [kubernetes][controller][name] . for example if we have field application_name in the message, we could map [kubernetes][controller][name] => [application_name]. The following fields can be static

mutate {
    add_field => {
        "[kubernetes][namespace]" => "user" #don't change this value
        "[kubernetes][controller][type]" => "Deployment" #don't change this value
        "[kubernetes][controller][name]" => "test" #this needs to be mapped dynamically, it could be your application name
        "[packetai][cluster_name]" => "yourclustername" #needs to be update by user
    }
}

Output conf

Logstash several output plugins, more details can be found here. PacketAI use http output plugin of logstash to send the messages to packetai. The http output plugin code snippet can be found below. we need to replace the YOUR_PAI_TOKEN_HERE and YOUR_PAI_IID_HERE with the correct values.

output {
  http {
    id => "packetai_logstash_ingester"
    url => "https://vector-ingester-logpatterns.packetai.co/vector/log"
    http_method => "post"
    headers => ["X-PAI-TOKEN","YOUR_PAI_TOKEN_HERE","X-PAI-IID","YOUR_PAI_IID_HERE"]
    format => "json_batch"
    http_compression => true
  }
}

Sample logstash configuration

Please replace the YOUR_PAI_TOKEN_HERE and YOUR_PAI_IID_HERE with the correct values from the PacketAI account.

input { 
    file {
        path => [
            "/var/log/*.log",
            "/var/log/*/*.log"
        ]
    }
}
filter {
    ruby {
        #Here We are setting logfilename as app_name
        code => 'event.set("app_name",event.get("[log][file][path]").split("/")[-1][0..-5].gsub(/[^0-9a-zA-Z]/i, ""))'
    }
    mutate {
        add_field => {
            "[kubernetes][namespace]" => "user"
            "[kubernetes][controller][type]" => "Deployment"
            "[kubernetes][controller][name]" => "%{app_name}" # here we are setting the app_name as kubernetes.controller.name
            "[packetai][cluster_name]" => "prod"
        }
    }
}
output {
  http {
    id => "packetai_logstash_ingester"
    url => "https://vector-ingester-logpatterns.packetai.co/vector/log"
    http_method => "post"
    headers => ["X-PAI-TOKEN","YOUR_PAI_TOKEN_HERE","X-PAI-IID","YOUR_PAI_IID_HERE"]
    format => "json_batch"
    http_compression => true
  }
}

Validate your configuration

It is advised to test and validate your logstash configuration with below command

logstash --config.test_and_exit -f logstash-simple.conf