Ship AWS Cloudwatch Logs to Any Destination with OpenTelemetry

With observIQ’s latest contributions to OpenTelemetry, you can now use free, open-source tools to easily aggregate logs across your entire infrastructure to any or multiple analysis tools. The easiest way to use the latest OpenTelemetry tools is with observIQ’s distribution of the OpenTelemetry collector. You can find it here.

In this blog, we cover how to use OpenTelemetry to ship logs from AWS Cloudwatch – you can use the AWS Cloudwatch receiver to ship logs to many popular analysis tools, including Google Cloud, New Relic, OTLP, Grafana, and more.

What signals matter?

AWS Cloudwatch is AWS’s native logging solution. Lambda function, EC2 instance, and EKS logs end up in Cloudwatch. If your environment includes sources outside of AWS, or you want to analyze or retain your logs in a different tool, OpenTelemetry allows easy, vendor-agnostic data management.

• Amazon EKS Logs in Cloudwatch include:
  • API Server Component Logs
  • Audit Logs
  • Authenticator Logs
  • Controller Manager Logs
  • Scheduler Logs
• AWS Lambda Logs are generated by functions you create to return logs. Some common examples include:
  • RequestID logs
  • Duration logs
  • Memory size and allocation logs
• EC2 Instances provide flexible computing resources in AWS cloud. EC2 Logs depend on what computing processes you are using.

Related Content: OpenTelemetry in Production: A Primer

Installing the Receiver

Suppose you don’t already have an OpenTelemetry collector built with the latest AWS Cloudwatch receiver installed. In that case, we suggest using the observIQ OpenTelemetry Collector distro, including the AWS Cloudwatch receiver (and many others). Installation is simple with our one-line installer. Come back to this blog after running the install command on your source.

Configuring the Receiver

You can just navigate to your OpenTelemetry configuration file. If you’re using the observIQ Collector, you’ll find it in one of the following locations:

• /opt/observiq-otel-collector/config.yaml (Linux)
• C:\Program Files\Google\Cloud Operations\Ops Agent\config\config.yaml (Windows)

Edit the configuration file to include the AWS Cloudwatch receiver as shown below:

1awscloudwatch:
2 region: us-west-1
3 logs:
4   poll_interval: 1m
5   groups:
6     autodiscover:
7       limit: 100
8       prefix: /aws/eks/

Below are a few editable fields you can add or adjust in the config file.

• Resource Attributes
  • aws.region
  • cloudwatch.log.group.name
  • Cloudwatch.log.stream
• Log Attributes
  • ID

Related Content: How to enrich data with OpenTelemetry

Viewing the logs collected

If you follow the steps detailed above, the following AWS Cloudwatch logs will now be delivered to your specified destination. observIQ’s monitoring technology is a game changer for organizations that care about performance and efficiency. If you’re using Oracle DB, our solutions can significantly impact your infrastructure monitoring. Follow this space to keep up with all our future posts and simplified configurations for various sources. For questions, requests, and suggestions, contact our support team at support@observIQ.com. Join our open-source observability community Slack Channel.