Using Logstash with BindPlane OP

BindPlane OP and the BindPlane Agent can be used to collect data from your Logstash agents. This allows you to start taking advantage of BindPlane OP without the need to re-instrument your collectors at the edge.

Step 1: Update your output stanza of the logstash conf.d files on your Logstash agents

Caveats

BindPlane expects the output from Logstash to be in JSON format. It depends on the codec => json_lines configuration, such as in the examples below to work as expected.

Example output stanza

This output stanza sends to a BindPlane agent installed on a host with the IP 10.10.1.5, and configured to listen on port 2255 (the default).

1output {
2  tcp {
3    codec => json_lines # this is required otherwise it will send eveything in a single line
4    host => "10.10.1.5"
5    port => 2255
6  }
7}

Below are a pair of sample logstash conf.d files. After adding these, or modifying the output stanza of existing ones, restart the logstash service.

Config for collecting from /var/log files using the logstash file plugin

1# Sample Logstash configuration for creating a simple
2# file -> logstash -> otel
3input {
4  file {
5    id => "logstash_var_log_plugin"
6    path => ["/var/log/messages", "/var/log/*", "/var/log/**/*"]
7    exclude => ["*.gz", "*.journal", "daemon.log", "logstash*.log"]
8  }
9}
10
11output {
12  tcp {
13    codec => json_lines # this is required otherwise it will send eveything in a single line
14    host => "localhost"
15    port => 2255
16  }
17}

Config for collecting from a json formatted log file using the logstash beats plugin

1# The # character at the beginning of a line indicates a comment. Use
2# comments to describe your configuration.
3
4# filebeats -> logstash -> otel
5
6input {
7  beats {
8    id => "logstash_beats_otel"
9    port => "5044"
10  }
11}
12# The filter part of this file is commented out to indicate that it is
13# optional.
14# filter {
15#
16# }
17output {
18  tcp {
19    codec => json_lines # this is required otherwise it will send eveything in a single line
20    host => "localhost"
21    port => 2255
22  }
23}

Step 2: Deploy a BindPlane Agent as an aggregator

This is the agent you will be routing data through and is what will be managed by BindPlane OP. In a production environment, this is likely to be a fleet of agents behind a load balancer. See our Collector Sizing and Scaling docs for more details on determining your collector architecture.

Step 3: Build the Configuration

1. Create a new configuration
2. Add the Logstash Source and configure it to receive from your Logstash agents (as shown below)

3. Add a destination of your choice and configure it.

Step 4: Transform the Data

Once you have verified data is flowing through the BindPlane Agent to your destination without issue, you can now start re-routing data to different destinations and inserting processors into your pipeline to reduce the amount of data you are sending.