How to set up Stanza as the log agent for your GCP?

Stanza is a robust log agent. GCP users can use Stanza for ingesting large volumes of log data. Before we dive into the configuration steps, here’s a matrix detailing the functional differences between all the common log agents used by GCP users.

Stanza was built as a modernized version of FluentD, Fluentbit, and Logstash. GCP users now have the ability to install Stanza to their VMs/ GKE clusters to ingest logs and route them to GCP log explorer. In this post we detail the steps for installing Stanza to Linux, Windows and Kubernetes environments and viewing the logs in the GCP log explorer.

Stanza for a Linux VM in GCP:

1. Single line installation command: In your VM, run the following single line installation command for Stanza

sh -c "$(curl -fsSlL https://github.com/observiq/stanza/releases/latest/download/unix-install.sh)" unix-install.sh

Stanza installation for Linux

2. Once the installation is complete, the following message displays. It provides the commands for starting and stopping Stanza. In addition, it shows the path for the config file.

Post installation instructions

3. Check if Stanza is running using

ps -ef | grep stanza

Check if Stanza is available on the VM

4. Open the config.yaml using the command vi config.yaml. In this example, we use the vi editor, alternatively any linux/unix editor could be used to edit and save the config file.

5. Comment everything except the following in the config.yaml and save the file

type: file_input include: file path output: example_output id: example_output type: google_cloud_output

Change the config.yaml to route logs to the GCP Logs Explorer

6. After the config file is saved, stop and start Stanza, using the following commands and very if the service is running.

systemctl stop stanza systemctl start stanza ps -ef | grep stanza

Stop and Start Stanza after changing the config file

7. Run a search query as shown below to verify if the logs are sent to GCP cloud and it is available for view in GCP’s log Explorer.

Stanza for a Windows VM in GCP:

1. Access the command line and run the single line installation command for Windows.

[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12 ; Invoke-Expression ((New-Object net.webclient).DownloadString('https://github.com/observiq/stanza/releases/latest/download/windows-install.ps1')); Log-Agent-Install

Windows VM in GCP log agent installation

2. Check if Stanza is running using

ps -ef | grep stanza

3. Make the following edits to the config.yaml

type: file_input include: file path output: example_output id: example_output type: google_cloud_output

4. After the config file is saved, stop and start Stanza, using the following commands.

1net stop stanza
2net start stanza

5. Verify if the logs are routed to the correct path and are available in GCP’s log explorer. To do this, in the Log Explorer console enter the following query:

label.file.name: ”stanza.log” stanza

Windows logs in GCP

Stanza for ingesting logs from GKE clusters:

1. As a prerequisite, follow the instructions in the link to create credentials secret on a JSON file.

2. Download and add the following files to the bucket in GCP. The files are available at

https://github.com/observIQ/stanza/tree/7537876b255cf727106e8b6ca6622c8a07f36c57/examples/k8s/events

3. Run the following command in the GKE cluster to create the service account. Our sample application is running an e-commerce application. For test purposes use this application linked here.

kubectl apply -f service_account.yaml

4. Run the following command to create the config map.

kubectl apply -f config.yaml

5. Run the following command to deploy the agent to the GKE cluster

kubectl apply -f deployment.yaml

6. Here’s a sample of the expected configuration if you choose to do it manually.

https://github.com/observIQ/stanza/blob/master/examples/k8s/daemonset/daemonset_gke.yaml

7. Verify in the logs explorer that Stanza ingests GKE cluster logs for the application or the test application.

Verify that Stanza ingests the GKE cluster logs

Stanza for ingesting logs from MySQL databases in GCP:

1. Use the following default configurations to begin ingesting logs.

- enable_error_log: true enable_general_log: false enable_mariadb_audit_log: false enable_slow_log: false error_log_path: /var/log/mysql/error.log start_at: end type: mysql - type: google_cloud_output

2. Verify that the logs are ingested from the database in the logs explorer.

View the MySQL logs ingested in Log Explorer

Stanza is a lightweight logs ingestor and transporter. Clubbed with the great features of GCP, you should be able to set up flawless end-to-end observability for your applications. Try this out and write to us with your requests/ suggestions.