Metricbeat

Using metricbeat to send the metrics to PacketAI

Download Metricbeat

Linux

Download metricbeat using below bash script: https://www.elastic.co/downloads/past-releases/metricbeat-8-4-3

curl https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-8.4.3-linux-x86_64.tar.gz -o metricbeat.tar.gz
tar -xf metricbeat.tar.gz
mv metricbeat-8.4.3-linux-x86_64 metricbeat
rm metricbeat.tar.gz

Windows

Download metricbeat using Powershell script:

$ProgressPreference = 'SilentlyContinue'
Invoke-WebRequest -Uri https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-8.4.3-windows-x86_64.zip -OutFile mbeat.zip
Expand-Archive .\mbeat.zip
mv ./mbeat/metricbeat-8.4.3-windows-x86_64/ ./metricbeat
rm -r ./mbeat
rm ./mbeat.zip
cd ./metricbeat

Configure Metricbeat

Edit metricbeat.yml, use the below metricbeat configuration, we need to customise this according to your requirements, explained in details about each section below. The complete metricbeat configuration can be found here

metricbeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: true

  # Period on which files under path should be checked for changes
  #reload.period: 10s
setup.template.enabled: false
setup.dashboards.enabled: false
setup.ilm.enabled: false

setup.kibana:

# ================================== Outputs ===================================

# Configure what output to use when sending the data collected by the beat.
#output.console:
 # pretty: true
# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  allow_older_versions: true
  hosts: ["beats-ingester-gcpdev.packetai.co:443"]
  protocol: https
  path: /elasticsearch/mb
  compression_level: 9
  index: "es"
  headers:
    X-PAI-IID: YOUR_PAI_IID
    X-PAI-TOKEN: YOUR_PAI_TOKEN

# ================================= Processors =================================

# Configure processors to enhance or manipulate events generated by the beat.

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~
  - add_fields:
      fields:
        clusterName: YOUR_CLUSTER_NAME
      target: fields

metricbeat modules:

• To enable the windows module, please run the below command.

  • metricbeat.exe modules enable windows

  • There are multiple modules available on metricbeat, they can be enabled to similar to the above command, the list of available modules are listed here.

  • The list of enabled modules are available in modules.d directory

  • To disable a module we need to run this command: metricbeat.exe modules disable windows

output.elasticsearch:

• hosts: Make sure that your hosts entries are correct. hosts: ["beats-ingester-logpatterns.packetai.co:443"] change URL if your PacketAI URL's are different

• compression_level is between 0 to 9, 0 being no compression at all, and 9 being best compression, we would suggest to use the value 6. higher compression values means higher cpu usage and lower network usage.

• headers sub section needs to update according to you PAI_IID and PAI_TOKEN. you can get them on packetai after login, and under the section of Deploy PacketAI / Agent

processors:

• We need to modify the YOUR_CLUSTER_NAME with appropriate cluster name. clusterName can be used to manage the retention period on PacketAI managed. Please note that clusterName cannot contain character "-". Currently this field is mandatory.

• add_cloud_metadata: ~ is optional, this adds the metadata of the cloud. i.e. region, zone, machine_id etc...

• add_docker_metadata is optional, this also docker metadata, docker container name, image name, docker labels, etc.... This extra metadata will increase the index size at PacketAI.

Installation of Metricbeat

Linux:

We could run the metricbeat with the below command to start the metricbeat, we need to install a service, if we want to automatically start the metricbeat service on system startup.

./metricbeat -c metricbeat.yml

systemd service file (metricbeat.service): Here we are assuming that metricbeat is located at /opt/metricbeat, if the metricbeat is located somewhere please update the metricbeat.service file to reflect the same.

# metricbeat.service
[Unit]
Description=metricbeat
After=syslog.target network.target remote-fs.target nss-lookup.target

[Service]
Type=simple
ExecStart=/opt/metricbeat/metricbeat -c metricbeat.yml
Restart=on-failure
WorkingDirectory=/opt/metricbeat

[Install]
WantedBy=multi-user.target

copy the file to /etc/systemd/system directory, and run the below commands. In

// Some code
systemctl enable metricbeat
systemctl start metricbeat

Windows:

In order to install the metricbeat on Windows, we need to run the below command, in metricbeat folder.

./install-service-metricbeat.ps1
Start-Service metricbeat