How to Ship AWS Cloudwatch Logs to Any Destination with OpenTelemetry

Observability and log management are needed for a strong IT strategy. Two essential tools for these purposes are AWS CloudWatch and OpenTelemetry. AWS Cloudwatch provides real-time data and insights into AWS-powered applications' health, performance, and efficiency. On the other hand, OpenTelemetry is an open-source observability framework that assists developers in creating, gathering, and exporting telemetry data (such as traces, metrics, and logs) for analysis.

Our team has recently contributed to OpenTelemetry, making it easier to gather logs from your entire infrastructure using free, open-source tools. You can access the latest OpenTelemetry capabilities through observIQ's distribution of the OpenTelemetry Collector, available here.

This blog will teach you how to use OpenTelemetry to send logs from AWS Cloudwatch. You can use the AWS Cloudwatch receiver to send logs to popular analysis tools like Google Cloud, New Relic, OTLP, Grafana, and more.

What signals matter?

AWS CloudWatch is AWS’s primary logging solution. It collects logs from Lambda functions, EC2 instances, and EKS. If your system involves sources outside of AWS, or you need to analyze or store logs in a different tool, OpenTelemetry can help you manage data across different vendors.

• Amazon EKS Logs in CloudWatch include:
  • API Server Component Logs
  • Audit Logs
  • Authenticator Logs
  • Controller Manager Logs
  • Scheduler Logs
• AWS Lambda Logs are generated by functions you create. Examples include RequestID logs, Duration logs, and Memory size and allocation logs.
• EC2 Instances provide flexible computing resources in the AWS cloud. The logs generated by EC2 depend on your specific computing processes.

Related Content: OpenTelemetry in Production: A Primer

Installing the Receiver

If you do not have the latest AWS CloudWatch receiver installed with an OpenTelemetry Collector, we suggest using the observIQ OpenTelemetry Collector distribution. This distribution includes the AWS CloudWatch receiver and many others. The Installation is simple with our one-line installer. After running the installation command on your source, come back to this blog for more guidance.

Configuring the Receiver

To set up the Receiver, you can just open your OpenTelemetry configuration file. If you use the observIQ Collector, look for it in the following locations:

• /opt/observiq-otel-collector/config.yaml (Linux)
• C:\Program Files\Google\Cloud Operations\Ops Agent\config\config.yaml (Windows)

Edit the configuration file to include the AWS Cloudwatch receiver as shown below:

1awscloudwatch:
2 region: us-west-1
3 logs:
4   poll_interval: 1m
5   groups:
6     autodiscover:
7       limit: 100
8       prefix: /aws/eks/

Below are some fields you can add or change in the config file:

• Resource Attributes
  • aws.region
  • cloudwatch.log.group.name
  • Cloudwatch.log.stream
• Log Attributes
  • ID

Related Content: How to enrich data with OpenTelemetry

Viewing and Analyzing Collected Logs

To start receiving the specific AWS Cloudwatch logs, simply follow the steps outlined above. If you use Oracle DB, our solutions can significantly enhance your infrastructure monitoring.

Start Using OpenTelemetry

Want to improve your observability and log management? Test the latest OpenTelemetry tools with our version of the OpenTelemetry Collector here.

Stay updated on our future posts and simplified configurations for different sources. If you have questions, requests, or suggestions, contact our support team. You can also join our open-source observability community Slack Channel.