Amazon Managed Prometheus

Description

This Amazon Managed Prometheus destination can be used to send metrics to an Amazon Managed Prometheus (AMP) workspace in AWS.

Supported Types

MetricsLogsTracesBindPlane Agent
v1.41.0+

Prerequisites

Getting the BindPlane Agent authenticated and authorized with AWS requires completing the following steps. Before starting, make sure you are logged in to AWS Console and have permission to create users, create roles, and generate access tokens.

1. Create an AWS User the exporter can use.

To start, head to the Identity and Access Management (IAM) service and under Access management navigate to Users. There should be an option to Create user on this page. This is the user the exporter and BindPlane Agent will be authenticated as, so name it appropriately. This new user will not need access to AWS Management Console, so leave this option unselected. For permissions, you don't need to specify any at this point since the user will be assuming a role with the required permissions. Once the user is created, navigate to its summary page and copy the ARN (we will need it in the next step).

2. Create an AWS Role the exporter can assume.

Now we need to create an AWS Role the exporter will assume. This is what will allow the exporter/BindPlane Agent to send data to the AMP workspace. To do this, head to IAM and under Access management navigate to Roles and select Create role. Under the first step, Select trusted entity, we are defining which AWS resource will be allowed to use this role, which we want to be the user we created in the first step. Select Custom trust policy. We want to edit just the Principal JSON field to contain the ARN copied in the first step. It should look similar to this:

JSON
1"Principal": {
2    "AWS": "arn:aws:iam::123456789:user/BindPlane"
3}

The next step in creating the role is Add permissions. Search for "Prometheus" in the search bar and a number of permissions should show up. The exporter only needs AmazonPrometheusRemoteWriteAccess, so select that. In the next area, give the role an appropriate name and description and finish with Create role.

As an alternative to the AWS pre-defined AmazonPrometheusRemoteWriteAccess permission policy, you may also create your own permission policy. To read more, see this AWS documentation. If taking this approach, be sure the exporter will still have write access to the desired AMP Workspace or the BindPlane Agent will fail.

3. Give BindPlane Agent access keys.

Now that the BindPlane Agent will have access to the AMP workspace, it needs access keys so it can access AWS as the user that was created for it in step 1. Head back to IAM and Users and select the user created in step 1. Now select the Security credentials tab and find the Access keys section. For the first step of creating a key, select Other and continue to the next step. Provide an appropriate description and select Create access key. Make note of or download the Access key and Secret access key values as AWS won't show these again.

Now that the access keys have been created, we need to create environment variables for them on the same machine as the BindPlane Agent. The environment variables to add are AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, which correspond to the values retrieved early in this step.

You can modify the collector's environment variables by configuring a systemd override. Run sudo systemctl edit observiq-otel-collector and add the keys:

text
1[Service]
2Environment=AWS_ACCESS_KEY_ID=******
3Environment=AWS_SECRET_ACCESS_KEY=******

Now the BindPlane Agent will be able to authenticate with AWS as the user that was created back in step 1.

For more information on specifying the AWS credentials for the BindPlane Agent, please see this AWS documentation. See this AWS documentation for alternative ways of using AWS environment variables for you specific environment.

4. Retrieve Amazon Managed Prometheus workspace endpoint.

The last pre-requisite step to configuring the Amazon Managed Prometheus destination is retrieving the remote write endpoint. If you haven't already created an Amazon Managed Prometheus workspace, search for "Amazon Prometheus" in the AWS console search bar and initialize one. With a workspace created, navigate to All workspaces inside of the Amazon Prometheus AWS service. Select the workspace you wish to send metrics to. On the summary page, copy the value for Endpoint - remote write URL. This is the endpoint you'll use when configuring the destination.

Configuration

FieldDescription
EndpointThe remote write endpoint to send metrics to the Amazon Managed Prometheus workspace. See this AMP documentation for more.
RegionThe region the Amazon Managed Prometheus workspace is located in. One location it can be found is in the given endpoint. See this AMP documentation for more.
AWS RoleThe Amazon Resource Name (ARN) for the AWS role the exporter will assume. The role should have permission to write to Amazon Managed Prometheus. See this AMP documentation for more.
STS RegionThe region to use for assuming the AWS Role. Useful for cross region authentication (i.e. if the agent is located in a different region than the AMP workspace). See this AMP documentation for more.
Session NameOptional name to give the session when the exporter assumes the AWS Role. Useful to differentiate sessions when multiple principals could assume the given role. See this AWS Documentation for more.
CompressionCompression algorithm to use when sending data to Amazon Managed Prometheus.
NamespacePrefix to attach to each metric name. See this Prometheus documentation for more.
Max Batch SizeMaximum size, in bytes, for a batch of metrics to be sent to the AMP endpoint. If a batch is larger than this limit, it will be broken up into multiple batches.
Convert AttributesWhether or not to convert all resource attributes to metric attributes.
Unit & Type SuffixesWhether or not to attach the metric unit and metric type to the end of the metric name.
Created MetricWhether or not a "_created" metric is exported for Summary, Histogram, and Monotonic Sum metric points when "StartTimeUnixNano" is set.
HTTP HeadersAdditional headers to attach to each HTTP Request. The following headers cannot be changed: Content-Encoding, Content-Type, X-Prometheus-Remote-Write-Version, and User-Agent.
External LabelsLabel names and values to be attached as metric attributes. See this Prometheus documentation for more.
Enable TLSWhether or not to use TLS.
Strict TLS VerifyWhether or not to use Strict TLS Certification Verification.
Certificate Authority FileCertificate authority used to validate TLS certificates. Not required if the collector's operating system already trusts the certificate authority.
Enable Mutual TLSWhether ot not to use Mutual TLS.
Client Certification FileA TLS certificate used for client authentication, if Mutual TLS is enabled.
Client Private Key FileA TLS private key used for client authentication, if Mutual TLS is enabled.
Enable Remote QueueWhether or not to enable a remote write queue. This helps balance outgoing results.
Remote Write Queue SizeThe number of metrics that can be queued.
Remote Write Queue ConsumersThe minimum number of workers to use to fan out the outgoing requests.

This destination supports the following retry and queuing settings:

Sending QueuePersistent QueueRetry on Failure

Example Configuration

Here we configure the destination to send to AMP by providing the endpoint, region, and arn. Some advanced configuration options we make use of include compression, namespace, max batch size, convert attributes, unit and type suffixes, created metric, and external labels by defining a label called "bp_agent" with a value of "agent1". We also enable TLS, Strict TLS, and Mutual TLS and provide a certificate authority file, client certificate file, and client key file. We also have a remote write queue and Retry on Failure enabled with their respective default configurations.

Web Interface

observIQ docs - Amazon Managed Prometheus Destination - image 1 observIQ docs - Amazon Managed Prometheus Destination - image 2

Standalone Destination

yaml
1apiVersion: BindPlane.observiq.com/v1
2kind: Destination
3metadata:
4  id: aws_managed_prometheus
5  name: aws_managed_prometheus
6spec:
7  type: aws_managed_prometheus
8  parameters:
9    - name: endpoint
10      value: some_aws_amp_workspace_endpoint
11    - name: region
12      value: us-east-2
13    - name: arn
14      value: some_aws_role_arn
15    - name: compression
16      value: snappy
17    - name: namespace
18      value: bp_amp
19    - name: max_batch_size_bytes
20      value: 3000000
21    - name: enable_resource_to_telemetry_conversion
22      value: true
23    - name: add_unit_and_type
24      value: true
25    - name: export_created_metric
26      value: true
27    - name: external_labels
28      value:
29        bp_agent: agent1
30    - name: enable_tls
31      value: true
32    - name: strict_tls_verify
33      value: true
34    - name: ca_file
35      value: some_ca_file
36    - name: mutual_tls
37      value: true
38    - name: cert_file
39      value: some_cert_file
40    - name: key_file
41      value: some_key_file
42    - name: enable_remote_write_queue
43      value: true
44    - name: retry_on_failure_enabled
45      value: true