Ship AWS Cloudwatch Logs to Any Destination with OpenTelemetry

With observIQ’s latest contributions to OpenTelemetry, you can now use free open source tools to easily aggregate logs across your entire infrastructure to any or multiple analysis tools. The easiest way to use the latest OpenTelemetry tools is with observIQ’s distribution of the OpenTelemetry collector. You can find it here.

In this blog, we cover how to use OpenTelemetry to ship logs from AWS Cloudwatch – you can use the AWS Cloudwatch receiver to ship logs to many popular analysis tools, including Google Cloud, New Relic, OTLP, Grafana, and more.

What signals matter?

AWS Cloudwatch is AWS’s native logging solution. Lambda function, EC2 instance, and EKS logs end up in Cloudwatch. If your environment includes sources outside of AWS, or you want to analyze or retain your logs in a different tool, OpenTelemetry allows easy, vendor-agnostic data management.

• Amazon EKS Logs in Cloudwatch include:
  • API Server Component Logs
  • Audit Logs
  • Authenticator Logs
  • Controller Manager Logs
  • Scheduler Logs
• AWS Lambda Logs are generated by functions you create to return logs. Some common examples include:
  • RequestID logs
  • Duration logs
  • Memory size and allocation logs
• EC2 Instances provide flexible computing resources in AWS cloud. EC2 Logs depend on what compute processes you are using.

Installing the Receiver

If you don’t already have an OpenTelemetry collector built with the latest AWS Cloudwatch receiver installed, we suggest using the observIQ OpenTelemetry Collector distro that includes the AWS Cloudwatch receiver (and many others). Installation is simple with our one-line installer. Come back to this blog after running the install command on your source.

Configuring the Receiver

Navigate to your OpenTelemetry configuration file. If you’re using the observIQ Collector, you’ll find it in one of the following location:

• /opt/observiq-otel-collector/config.yaml (Linux)
• C:\Program Files\Google\Cloud Operations\Ops Agent\config\config.yaml (Windows)

Edit the configuration file to include the AWS Cloudwatch receiver as shown below:

1awscloudwatch:
2 region: us-west-1
3 logs:
4   poll_interval: 1m
5   groups:
6     autodiscover:
7       limit: 100
8       prefix: /aws/eks/

Below are a few editable fields you can add or adjust in the config file.

• Resource Attributes
  • aws.region
  • cloudwatch.log.group.name
  • Cloudwatch.log.stream
• Log Attributes
  • ID

You can also use OpenTelemetry to enrich log data – check out our blog on how to enrich data with OpenTelemetry.

Viewing the logs collected

If you followed the steps detailed above, the following AWS Cloudwatch logs will now be delivered to your specified destination.observIQ’s monitoring technology is a game changer for organizations that care about performance and efficiency. If you’re using Oracle DB, our solutions can make a significant difference in your infrastructure monitoring. Follow this space to keep up with all our future posts and simplified configurations for various sources. For questions, requests, and suggestions, reach out to our support team at support@observIQ.com. Join our open source observability community Slack Channel.