How to collect metrics and logs for NGINX using the OpsAgent

Why use OpsAgent?

The Ops agent is Google’s recommended agent for collating your application’s telemetry data, and forwarding them to GCP for visualization, alerting and monitoring. The Ops agent collates logs and a metrics collector into one single powerhouse. Some of the key advantages of using the Ops agent are outlined below:

• Ability to monitor and parse logs that are written to file or console
• It is a lightweight agent
• The installation and configuration is extremely simple and seamless.

How to install?

Before you install:

1. Identify the most recent version of the Ops agent from Google Cloud documentation. Please note that to monitor Nginx, the Ops agent version installed on the VM should be ver 2.1.0 or higher
2. Get the deployment link for the VM’s operating system

You may install the Ops agent in the following methods:

• Use the gcloud/ agent policies to install the agent to a fleet of VMs at once.
• Use automation tools such as Ansible, Terraform, etc. to install the agent of a fleet of VMs
• Install the agent to a single VM from the Google Cloud Console

In this post, we detail the steps to install the Ops agent on a single VM(that has Nginx installed) from the Google Cloud Console.

Copy the installation code from Google’s documentation linked here. Please ensure that you have the installation code for the VM’s operating system. In this case, for a Linux Debian 10 VM the installation code is:

curl -sSO https://dl.google.com/cloudagents/add-google-cloud-ops-agent-repo.sh

sudo bash add-google-cloud-ops-agent-repo.sh –also-install

Broken image

Configuring the Ops agent to collect Nginx logs and metrics

In order to collect logs and metrics, these components must be enabled in the ops agent config.

Access the Ops agent’s config file in the /etc/google-cloud-ops-agent/config.yaml directory and add the following configuration:

1logging: 
2 receivers:
3  nginx_access:
4   type:nginx_access
5  nginx_error:
6   type:nginx_error
7 service:
8  pipelines:
9   nginx:
10    receivers:
11     - nginx_access
12     - nginx_error
13metrics:
14 receivers:
15  nginx:
16   type:nginx
17   stub_status_url: http://127.0.0.1:80/status
18  pipelines:
19   nginx:
20    receivers:
21   - nginx

Broken image

Viewing Nginx logs and metrics

View the logs ingested from Nginx under the Logs Explorer

Broken image

View the metrics forwarded to Google Cloud Monitoring

Broken image

What logs are ingested from Nginx?

The Ops agents ingests and forwards the following logs to the Logs viewer. There are two types of Nginx logs, access logs and error logs.

Sample Error Log:

Broken image

Sample Access log:

Broken image

Nginx Access logs:

1. Request type: The protocol used in every request sent is logged.
2. Referrer header: Nginx blocks access to a site if the request received has invalid values in the header field. The contents of this header field are logged.
3. Client IP address: This log contains the IP address of the client from which the request is received.
4. HTTP Method: The HTTP method in the request header.
5. Request URL: The URL in the request received
6. Response size: The size of the response sent to the client from Nginx.
7. HTTP status code: The HTTP status code sent as part of the response to the client
8. User agent header: The contents of the user agent proxy sent to the client
9. JSON payload username: The authenticated username sending the request
10. Timestamp: The time that the request is received

What metrics does the Ops agent collect from the instance?

Nginx collects and forwards the following metrics from Nginx

1. Requests received
2. Connections accepted based on requests
3. Connections handled by Nginx
4. The current connections that are active

Broken image

The ease of use of our telemetry solutions are a must try for any organization looking to refine and expand their observability. Reach out to our customer support team with your questions and requests.