While installing AWS CLI is not required in order to collect logs using the AWS Cloudwatch source type, it still provides an easier means of authentication. With AWS Cloudwatch, users are required to provide some form of authentication. This can be either profile credentials or environment variables that provide access keys for user accounts. Setting these credentials up can prove tedious and confusing. Luckily, AWS CLI can generate the user's profile/credentials with the
aws configure command. Here is the AWS CLI Getting Started Guide, which outlines a few prerequisites.
- Creating an IAM user account
- Required Permissions
- The user does not require console access
- Required Permissions
- Create an access key ID and secret access key
Credential and Config Files
AWS Authentication utilizes a user profile specified in the user's home directory at
.aws/credentials. Each profile's credentials should include, at minimum, the profile name, access key, and secret access key.
1[profile_name] 2aws_access_key_id=****** 3aws_secret_access_key=******
In addition to the credentials file, there is also a
.aws/config. This includes less sensitive configuration options such as the region, output format, etc. A typical entry in the
config file should look as such.
1[profile_name] 2region=us-west-2 3output=json
More information on AWS Configuration and Credentials
Alternatively, AWS Environment variables can be specified to override a credentials file.
You can modify the collector's environment variables by configuring a Systemd override.
sudo systemctl edit observiq-otel-collector and add your access key, secret key, and region.
1[Service] 2Environment=AWS_ACCESS_KEY_ID=****** 3Environment=AWS_SECRET_ACCESS_KEY=****** 4Environment=AWS_DEFAULT_REGION=us-west-2
Then restart the collector after making environment changes.
- Once logged in, Select the
Configstab at the top of the Bindplane Home page.
- Select a pre-existing config or create a new one.
- Add a new source and select
- After configuring credentials using the AWS CLI on the collector system, using the default values in the source form should enable the collector to collect logs from Cloudwatch. If credentials were configured using environment variables, you will need to leave the
- Click save configuration.
- Add the destination type of your choosing.
- Apply configuration to the desired agent.
- Voila. Logs should be collecting
|region*||us-east-1||The AWS recognized region string.|
|profile *||"default"||The AWS profile is used to authenticate; if none is specified, the default is chosen from the list of profiles.|
|credential_type||profile||Determines whether to pull credentials from a |
|discovery_type||AutoDiscover||Configuration for Log Groups. By default, all Log Groups and Log Streams will be collected.|
|limit||50||Limits the number of discovered log groups.|
|prefix||""||A prefix for log groups to limit the number of log groups discovered.|
|names||||A list of full log stream names to filter the discovered log groups to collect from.|
|prefixes||||A list of prefixes to filter the discovered log groups to collect from.|
|named_groups||||Configuration for Log Groups. By default, all Log Groups and Log Streams will be collected.|
|imds_endpoint||""||A way of specifying a custom URL to be used by the EC2 IMDS client to validate the session.|
|poll_interval||1||The duration of waiting in between requests (minutes).|
|max_events_per_request||50||The maximum number of events to process per request to Cloudwatch.|
When starting with an AWS Cloudwatch set to its default values, you should see log collection from all log groups with no filtering of log streams. The default polling interval for Cloudwatch is 1 minute, so there may be a delay before seeing any logs coming through.
When using Discovery Type
AutoDiscover, there are some optional parameters that can be added to / filter the amount of logs collected.
limit: limits the number of discovered log groups(default = 50).
prefix: Prefix for log groups to limit the number of log groups discovered
- If omitted, all log groups up to the limit will be collected.
names: A list of full log stream names to filter the discovered log groups to collect from.
prefixes: A list of log stream prefixes to filter the discovered log groups to collect from.
This Discovery Type filters logs by listing only the desired log groups to collect from and omitting any other log groups. When selecting this Discovery Type, at least one log group is required otherwise no logs would be collected. When listing log groups the
ID field of each log group instance should match the full name of the group.
Named also provides
names parameters for each listed log group that filters out the listed log streams. These parameters should be listed underneath each log group's IDs as they are unique to each individual log group.