ClickHouse

Description

This ClickHouse destination can be used to send metrics, logs, and traces to a ClickHouse server.

Supported Types

Metrics	Logs	Traces	Bindplane Agent
✓	✓	✓	`v1.41.0`+

Configuration

Field	Description
Telemetry Types	The kind of telemetry that will be sent to the ClickHouse server. Can be any combination of logs, metrics, and traces.
Protocol	The kind of protocol to be used when sending to the ClickHouse server. Can be TCP, HTTP, or HTTPS. See this ClickHouse documentation for more information.
Endpoint	The endpoint to use to send telemetry data to ClickHouse. Supports multiple endpoints. See this ClickHouse documentation for more information.
Username	Username to use to authenticate with the ClickHouse server. See this ClickHouse documentation for more information.
Password	Password to use to authenticate with the ClickHouse server. See this ClickHouse documentation for more information.
Database	Name of the database to use when interacting with the ClickHouse server. Creates the database if it does not already exist.
Logs Table Name	Name of the table inside `Database` in ClickHouse to store log data in. Creates the table if it does not already exist.
Metrics Table Name	Name of the table inside `Database` in ClickHouse to store metric data in. Creates the table if it does not already exist.
Traces Table Name	Name of the table inside `Database` in ClickHouse to store traces data in. Creates the table if it does not already exist.
TTL	The data time-to-live, for example '30m'. 0 means no TTL. Make sure the telemetry sent has a timestamp field.
Timeout	Timeout for each attempt to send data to ClickHouse.
Connection Parameters	Additional connection parameters with map format. Used as query parameters in the URL. See this ClickHouse documentation for more information.

important

Metrics stored in ClickHouse are grouped by their type (sum, gauge, etc.) and stored in tables specific to that type by appending _type to the table name. For example, using the example configuration below, gauge metrics would be stored in bpop.bp_metrics_gauge. To read more, see this documentation.

note

TLS can be configured using Connection Parameters. To do so, add an entry called secure with the value true. An example can be seen below.

warning

The agent will fail to start if it cannot reach the ClickHouse server on start up. If the ClickHouse server becomes unreachable while the agent is already running, it will continue running while reporting an error.

The agent can also fail to start if configured to use TTL, but the telemetry being sent lacks a Timestamp field.

This destination supports the following retry and queuing settings:

Sending Queue	Persistent Queue	Retry on Failure
✓	✓	✓

Example Configuration

Here we set a single endpoint that uses the TCP protocol. We are sending metrics, logs, and traces, so we have them selected and table names for them. We also utilize authentication and are setting compression and TLS via the connection parameters. Finally, we have the sending queue, persistent queue, and retry on failure enabled as well.

Web Interface

observIQ docs - ClickHouse Destination - image 1

observIQ docs - ClickHouse Destination - image 2

Standalone Destination

yaml

1apiVersion: bindplane.observiq.com/v1
2kind: Destination
3metadata:
4  id: clickhouse
5  name: clickhouse
6spec:
7  type: clickhouse
8  parameters:
9    - name: telemetry_types
10      value: ['Logs', 'Metrics', 'Traces']
11    - name: protocol
12      value: tcp
13    - name: endpoint
14      value: ['0.0.0.0:9000']
15    - name: username
16      value: admin
17    - name: password
18      value: admin
19    - name: database
20      value: bpop
21    - name: logs_table_name
22      value: bp_logs
23    - name: metrics_table_name
24      value: bp_metrics
25    - name: traces_table_name
26      value: bp_traces
27    - name: connection_params
28      value:
29        compress: gzip
30        secure: true
31    - name: retry_on_failure_enabled
32      value: 'true'
33    - name: sending_queue_enabled
34      value: 'true'
35    - name: persistent_queue_enabled
36      value: 'true'

Updated 6 months ago