# Filebeat ### Download the Filebeat #### Linux Download the filebeat using below bash script: ```bash curl https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.4.3-linux-x86_64.tar.gz -o filebeat.tar.gz tar -xf filebeat.tar.gz mv filebeat-8.4.3-linux-x86_64 filebeat rm filebeat.tar.gz ``` #### Windows Download the filebeat using Powershell script: {% code overflow="wrap" %} ```powershell $ProgressPreference = 'SilentlyContinue' Invoke-WebRequest -Uri https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.4.3-windows-x86_64.zip -OutFile fbeat.zip Expand-Archive .\fbeat.zip mv ./fbeat/filebeat-8.4.3-windows-x86_64/ ./filebeat rm -r ./fbeat rm ./fbeat.zip cd ./filebeat ``` {% endcode %} ### Configure Filebeat Edit the filebeat.yml, use the below filebeat configuration, we need to customise this according to your requirements, explained in details about each section below. The complete filebeat configuration can be found [here](https://www.elastic.co/guide/en/beats/filebeat/current/configuring-howto-filebeat.html) ```yaml filebeat.inputs: # This section is to monitor which files on your machine and their paths. - type: filestream id: wifi-filestream-id enabled: true # Paths that should be crawled and fetched. Glob based paths. paths: - /var/log/wifi.log fields: appName: wifi tail_files: true - type: filestream id: system-filestream-id enabled: true # Paths that should be crawled and fetched. Glob based paths. paths: - /var/log/system.log fields: appName: system tail_files: true - type: filestream id: fsck-filestream-id enabled: true # Paths that should be crawled and fetched. Glob based paths. paths: - /var/log/fsck*.log fields: appName: fsck tail_files: true # ============================== Filebeat modules ============================== filebeat.config.modules: # Glob pattern for configuration loading path: ${path.config}/modules.d/*.yml # Set to true to enable config reloading reload.enabled: true # Disable template, dashboards, index management, don't change these values to true setup.template.enabled: false setup.dashboards.enabled: false setup.ilm.enabled: false # ---------------------------- Elasticsearch Output ---------------------------- output.elasticsearch: # Array of hosts to connect to. allow_older_versions: true hosts: ["beats-ingester-logpatterns.packetai.co:443"] protocol: https path: /elasticsearch/fb compression_level: 6 index: "index" headers: X-PAI-IID: YOUR_PAI_IID X-PAI-TOKEN: YOUR_PAI_TOKEN # ================================= Processors ================================= processors: - add_host_metadata: when.not.contains.tags: forwarded - add_cloud_metadata: ~ - add_docker_metadata: ~ - add_fields: fields: clusterName: YOUR_CLUSTER_NAME target: fields ``` **`filebeat.inputs`**: * This section is to monitor the list of log files on your host. we need to mention the `paths` which takes array of files with regular expression (glob), make sure that `id` is unique, and set the `enabled` to true to start the monitoring of the log lines. * We could define the `appName` to each log file, this can be helpful to filter the logs at packetai. When we set the `tail_files` to `true` for not sending the logs from the beginning of the log file. Here `type: filestream` defines that this is a filestream type monitoring. `filebeat.inputs` is an array where we could monitor multiple log files. **`output.elasticsearch`**`:` * `hosts`: Make sure that your hosts entries are correct. `hosts: ["beats-ingester-logpatterns.packetai.co:443"]` change if your PacketAI API's are different * compression\_level is between 0 to 9, 0 being no compression at all, and 9 being best compression, we would suggest to use the value 6. higher compression values means higher cpu usage and lower network usage. * `headers` sub section needs to update according to you PAI\_II*D and PA*I\_TOKEN. you can get them on packetai after login, and under the section of `Deploy PacketAI / Agent` **`processors:`** * We need to modify the YOUR\_CLUSTER\_NAME *with* appropriate cluster name*.* clusterName can be used to manage the retention period on PacketAI managed. * add\_*cloud\_metadata: \~* is optional, this adds the metadata of the cloud. i.e. region, zone, machine\_id etc... * add\_docker\_metadata is optional, this also docker metadata, docker container name, image name, docker labels, etc.... This extra metadata will increase the index size at PacketAI. ### Installation of Filebeat **Linux:** We could run the filebeat with the below command to start the filebeat, we need to install a service, if we want to automatically start the filebeat service on system startup. ``` ./filebeat -c filebeat.yml ``` systemd service file (`filebeat.service`): Here we are assuming that filebeat is located at /opt/filebeat, if the filebeat is located somewhere please update the filebeat.service file to reflect the same. ```systemd # filebeat.service [Unit] Description=filebeat After=syslog.target network.target remote-fs.target nss-lookup.target [Service] Type=simple ExecStart=/opt/filebeat/filebeat -c filebeat.yml Restart=on-failure WorkingDirectory=/opt/filebeat [Install] WantedBy=multi-user.target ``` copy the file to `/etc/systemd/system` directory, and run the below commands. In ``` // Some code systemctl enable filebeat systemctl start filebeat ``` **Windows:** In order to install the filebeat on Windows, we need to run the below command, in filebeat folder. ``` ./install-service-filebeat.ps1 Start-Service filebeat ``` --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://packetai.gitbook.io/packetai-autonomous-it-monitoring/set-up/filebeat.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.