Snowflake

Description

This destination can send logs, metrics, and traces to Snowflake, a cloud data warehouse service.

Supported Types

MetricsLogsTracesBindPlane Agent
v1.45.0+

Prerequisites

The following guide will detail how to get a user initialized in Snowflake that can be used with this destination. It is meant to get telemetry flowing with limited time spent configuring. If you'd like to take an alternative approach, check out the exporter documentation on this topic here.

Snowflake has a variety of ways to connect to it, but these instructions will be tailored for "Classic Console" as all accounts have access to it.

Before starting, log in to Classic Console using a user that has access to the ACCOUNTADMIN role or another role in you Snowflake account that has permission to grant privileges and create users. If the default role is not the required one, then you'll need to assume that role using this SQL command (replace the role as needed):

sql
1ASSUME ROLE "ACCOUNTADMIN";

These instructions will grant privileges to one of the default roles Snowflake is initialized with, SYSADMIN. If you want to grant privileges to a different role then just switch out SYSADMIN for your role in the SQL commands.

1. Grant Warehouse Usage

First, we need to grant the USAGE privilege to the SYSADMIN role on the data warehouse telemetry data will be stored in. Run this SQL command next (replace TEST with your warehouse name):

sql
1GRANT USAGE ON WAREHOUSE "TEST" TO ROLE "SYSADMIN";

2. Grant Create Database Privilege

Next the SYSADMIN role needs to be granted the ability to create databases in the Snowflake account. Run the following SQL to do so:

SQL
1GRANT CREATE DATABASE ON ACCOUNT TO ROLE "SYSADMIN";

3. Create New User For BindPlane

Now a new user needs to be created that the BindPlane Agent can login as. The user should also have the default role assigned as SYSADMIN, although it isn't necessary.

Note: If the default role is not assigned, then the exporter will need to be configured with the correct role to work.

Remember the login name and password you use and configure the destination with these values. Replace the user, password, and login name in the following SQL to match yours:

sql
1CREATE USER BP_AGENT PASSWORD="password" LOGIN_NAME="BP_AGENT" DEFAULT_ROLE="SYSADMIN";

4. Grant Privilege to SYSADMIN Role

Even though the default role was set as SYSADMIN we still need to grant the new account permission to it. This can be done using the next SQL command (replace user as needed):

sql
1GRANT ROLE "SYSADMIN" TO USER BP_AGENT;

Now we have a Snowflake user with the correct permissions to be able to create a database, schemas, and tables and also use the configured warehouse to store telemetry data in.

Configuration

FieldDescription
TelemetryThe kinds of telemetry to send to Snowflake.
Account IdentifierThe account identifier for the Snowflake account that data should be sent to.
WarehouseTHe Snowflake warehouse that telemetry data should be stored in.
UsernameUsername the exporter will use to authenticate with Snowflake.
PasswordPassword the exporter will use to authenticate with Snowflake.
DatabaseThe Snowflake database that telemetry schemas will be stored in. Will be created if it doesn't exist.
Log SchemaThe schema that the log table will be stored in. Will be created if it doesn't exist.
Log TableThe table that logs will be stored in. Will be created if it doesn't exist.
Metric SchemaThe schema that the metric tables will be stored in. Will be created if it doesn't exist.
Metric TableThe prefix used for metric tables. Tables are created if they don't exist. See this exporter documentation for more.
Trace SchemaThe schema that the trace table will be stored in. Will be created if it doesn't exist.
Trace TableThe table that traces will be stored in. Will be created if it doesn't exist.
RoleThe Snowflake role the exporter should use. Only required if the default role of the provided credentials does not have correct privileges.
ParametersAdditional optional parameters the exporter should use when connecting to Snowflake. This option is generally not required. See this Snowflake documentation for more.

This destination supports the following retry and queuing settings:

Sending QueuePersistent QueueRetry on Failure

Example Configuration

Here we will configure this destination to send all telemetry types to a Snowflake account using the default values for database resources. We'll also configure the sending queue, persistent queue, and retry on failure.

Web Interface

observIQ docs - Snowflake Destination - image 1 observIQ docs - Snowflake Destination - image 2

Standalone Destination

yaml
1apiVersion: BindPlane.observiq.com/v1
2kind: Destination
3metadata:
4  id: snowflake
5  name: snowflake
6spec:
7  type: snowflake
8  parameters:
9    - name: telemetry
10      value: ['Logs', 'Metrics', 'Traces']
11    - name: account_id
12      value: 'my_account_id'
13    - name: warehouse
14      value: 'my_warehouse'
15    - name: username
16      value: 'my_user'
17    - name: password
18      value: 'my_pass'
19    - name: sending_queue_enabled
20      value: true
21    - name: persistent_queue_enabled
22      value: true
23    - name: retry_on_failure_enabled
24      value: true