Kafka as Event Bus

important

🚧 This feature is only available in BindPlane OP Enterprise. Learn more here.

Enable Kafka as the Event Bus

Kafka can be used as the event bus for BindPlane OP Enterprise and is a good option for distributed on-prem deployments.

Prerequisites

In order to use Kafka as the event bus we need to ensure a topic exists. The below command can be used to generate the topic, just replace <YOUR_KAFKA_SERVER> with a broker address. Auto topic creation can also be used if configured in your Kafka environment.

shell

1kafka-topics.sh --create --topic bindplane-op-message-bus --partitions 1 --replication-factor 1 --bootstrap-server <YOUR_KAFKA_SERVER>

note

Ordering is required for BindPlane OP events so the topic must only have 1 partition.

Configuration

In order to use Kafka as the event bus the eventBus.type field must be set to kafka and the eventBus.kafka config must be filled out.

Here is an example of a full configuration using Kafka as the event bus.

yaml

1name: default
2apiVersion: bindplane.observiq.com/v1
3auth:
4    # A random uuid which is used as a shared secret between bindplane and
5    # deployed agents.
6    secretKey: your-secret-key
7
8    # Basic auth should use a username other than
9    # admin along with a secure password.
10    username: admin
11    password: password
12
13    # A random uuid which is used for generating web ui session cookies.
14    sessionSecret: your-session-secret
15network:
16    # Listen on port 3001, all interfaces.
17    host: 0.0.0.0
18    port: "3001"
19
20    # Endpoint for which clients and collectors will interfact
21    # with the server's http interface.
22    remoteURL: http://bindplane.c.bindplane.internal:3001
23store:
24    bbolt:
25    	path: /var/lib/bindplane/storage/bindplane.db
26eventBus:
27    type: kafka
28    kafka:
29      # A list of all brokers that are in the Kafka cluster
30    	brokers:
31      	- broker1:9092
32        - broker2:9092
33      authType: none
34      # protocolVersion should be specified as MAJOR.MINOR.PATCH and should match your version of Kafka
35      protocolVersion: '3.5.0'
36logging:
37    filePath: /var/log/bindplane/bindplane.log

Configuration Parameters

Parameter Name	Description	Required	Default Value
`brokers`	List of brokers to connect to.	:heavy-check-mark:
`protocolVersion`	The protocol version the Kafka brokers are using. Should be specified as `MAJOR.MINOR.PATCH`. This usually matches the version of the brokers.	:heavy-check-mark:
`topic`	The name of the topic to use. See Prerequisites.	:heavy-check-mark:	`bindplane-op-message-bus`
`authType`	The type of authentication to use. Valid values are `none`, `plainText`, and `sasl`.	:heavy-check-mark:	`none`
`plainText`	Authentication configuration when `authType` is `plainText`. See PlainText Auth.
`sasl`	Authentication configuration when `authType` is `sasl`. See SASL Auth.
`enableTLS`	Signals whether or not to use TLS.		`false`
`tls`	TLS Configuration when `enableTLS` is `true`. See TLS Configuration.

PlainText Auth

Below are the configuration parameters when using PlainText authentication.

Parameter Name	Description	Required	Default Value
`plainText.username`	The username for authentication.	:heavy-check-mark:
`plainText.password`	The password for the user.	:heavy-check-mark:

Here is an example configuration using PlainText authentication:

yaml

1name: default
2apiVersion: bindplane.observiq.com/v1
3auth:
4    # A random uuid which is used as a shared secret between bindplane and
5    # deployed agents.
6    secretKey: your-secret-key
7
8    # Basic auth should use a username other than
9    # admin along with a secure password.
10    username: admin
11    password: password
12
13    # A random uuid which is used for generating web ui session cookies.
14    sessionSecret: your-session-secret
15network:
16    # Listen on port 3001, all interfaces.
17    host: 0.0.0.0
18    port: "3001"
19
20    # Endpoint for which clients and collectors will interfact
21    # with the server's http interface.
22    remoteURL: http://bindplane.c.bindplane.internal:3001
23store:
24    bbolt:
25    	path: /var/lib/bindplane/storage/bindplane.db
26eventBus:
27    type: kafka
28    kafka:
29      # A list of all brokers that are in the Kafka cluster
30    	brokers:
31      	- broker1:9092
32        - broker2:9092
33      # protocolVersion should be specified as MAJOR.MINOR.PATCH and should match your version of Kafka
34      protocolVersion: '3.5.0'
35      # PlainText Authentication Configuration
36      authType: plainText
37      plainText:
38      	username: 'user1'
39        password: 'password1234'
40logging:
41    filePath: /var/log/bindplane/bindplane.log

SASL Authentication

Below are the configuration parameters when using SASL authentication.

Parameter Name	Description	Required	Default Value
`sasl.username`	The username for authentication.	:heavy-check-mark:
`sasl.password`	The password for the user.	:heavy-check-mark:
`sasl.mechanism`	The mechanism by which to use SASL authentication. Valid values are `plain`, `scramSHA256`, and `scramSHA512`.	:heavy-check-mark:	`plain`
`sasl.version`	The version of SASL authentication to use. Valid values are `0` or `1`.	:heavy-check-mark:	`0`

Here is an example configuration using SASL authentication:

yaml

1name: default
2apiVersion: bindplane.observiq.com/v1
3auth:
4    # A random uuid which is used as a shared secret between bindplane and
5    # deployed agents.
6    secretKey: your-secret-key
7
8    # Basic auth should use a username other than
9    # admin along with a secure password.
10    username: admin
11    password: password
12
13    # A random uuid which is used for generating web ui session cookies.
14    sessionSecret: your-session-secret
15network:
16    # Listen on port 3001, all interfaces.
17    host: 0.0.0.0
18    port: "3001"
19
20    # Endpoint for which clients and collectors will interfact
21    # with the server's http interface.
22    remoteURL: http://bindplane.c.bindplane.internal:3001
23store:
24    bbolt:
25    	path: /var/lib/bindplane/storage/bindplane.db
26eventBus:
27    type: kafka
28    kafka:
29      # A list of all brokers that are in the Kafka cluster
30    	brokers:
31      	- broker1:9092
32        - broker2:9092
33      # protocolVersion should be specified as MAJOR.MINOR.PATCH and should match your version of Kafka
34      protocolVersion: '3.5.0'
35      # SASL Authentication Configuration
36      authType: sasl
37      sasl:
38      	username: 'user1'
39        password: 'password1234'
40        mechanism: 'scramSHA256'
41logging:
42    filePath: /var/log/bindplane/bindplane.log

TLS Configuration

Below are the configuration parameters when using TLS.

Parameter Name	Description	Default
`tls.tlsCert`	Path to the TLS certificate file.
`tls.tlsKey`	Path to the TLS key file.
`tls.tlsCa`	Path to the TLS certificate authority file.
`tls.tlsSkipVerify`	If `true` use TLS without verification.	`false`

Here is an example configuration using TLS:

yaml

1name: default
2apiVersion: bindplane.observiq.com/v1
3auth:
4    # A random uuid which is used as a shared secret between bindplane and
5    # deployed agents.
6    secretKey: your-secret-key
7
8    # Basic auth should use a username other than
9    # admin along with a secure password.
10    username: admin
11    password: password
12
13    # A random uuid which is used for generating web ui session cookies.
14    sessionSecret: your-session-secret
15network:
16    # Listen on port 3001, all interfaces.
17    host: 0.0.0.0
18    port: "3001"
19
20    # Endpoint for which clients and collectors will interfact
21    # with the server's http interface.
22    remoteURL: http://bindplane.c.bindplane.internal:3001
23store:
24    bbolt:
25    	path: /var/lib/bindplane/storage/bindplane.db
26eventBus:
27    type: kafka
28    kafka:
29      # A list of all brokers that are in the Kafka cluster
30    	brokers:
31      	- broker1:9092
32        - broker2:9092
33      # protocolVersion should be specified as MAJOR.MINOR.PATCH and should match your version of Kafka
34      protocolVersion: '3.5.0'
35      authType: none
36      enableTLS: true
37      tls:
38      	tlsCert: /opt/kafka.crt
39        tlsKey: /opt/kafka.key
40logging:
41    filePath: /var/log/bindplane/bindplane.log

Kafka as Event Bus

How to setup BindPlane OP to use Kafka as its event bus

important

Enable Kafka as the Event Bus

Prerequisites

note

Configuration

Configuration Parameters

PlainText Auth

SASL Authentication

TLS Configuration