How to set up Stanza as the log agent for your GCP?

by Deepa Ramachandra on November 12, 2021

Stanza is a robust log agent. GCP users can use Stanza for ingesting large volumes of log data. Before we dive into the configuration steps, here’s a matrix detailing the functional differences between all the common log agents used by GCP users.Chart comparing the specifications of four common logging agents, FluentD, FluentBit, Logstash, and Stanza. The chart depicts in objective terms the advantages of Stanza over the other agents

Stanza was built as a modernized version of FluentD, Fluentbit, and Logstash. GCP users now have the ability to install Stanza to their VMs/ GKE clusters to ingest logs and route them to GCP log explorer. In this post we detail the steps for installing Stanza to Linux, Windows and Kubernetes environments and viewing the logs in the GCP log explorer.

 

Stanza for a Linux VM in GCP:

1. Single line installation command: In your VM, run the following single line installation command for Stanza

sh -c "$(curl -fsSlL https://github.com/observiq/stanza/releases/latest/download/unix-install.sh)" unix-install.sh
Install log agent for Linux VM in GCP
Stanza installation for Linux

2. Once the installation is complete, the following message displays. It provides the commands for starting and stopping Stanza. In addition, it shows the path for the config file.

Log agent installation for linux complete in GCP VM
Post installation instructions

3. Check if Stanza is running using

 ps -ef | grep stanza
Verify if log agent is installed for Linux
Check if Stanza is available on the VM

4. Open the config.yaml using the command vi config.yaml.  In this example, we use the vi editor, alternatively any linux/unix  editor could be used to edit and save the config file. 

5. Comment everything except the following in the config.yaml and save the file

type: file_input

include: file path

output: example_output

id: example_output

type: google_cloud_output 
Linux log agent installed for GCP VM configuration
Change the config.yaml to route logs to the GCP Logs Explorer

 

6. After the config file is saved, stop and start Stanza, using the following commands and very if the service is running.

systemctl stop stanza

systemctl start stanza

ps -ef | grep stanza
Stop and Start Log Agent in GCP
Stop and Start Stanza after changing the config file

7. Run a search query as shown below to verify if the logs are sent to GCP cloud and it is available for view in GCP’s log Explorer.

Verify Linux logs in GCP

Stanza for a Windows VM in GCP:

1. Access the command line and run the single line installation command for Windows.

[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12 ; Invoke-Expression ((New-Object net.webclient).DownloadString('https://github.com/observiq/stanza/releases/latest/download/windows-install.ps1')); Log-Agent-Install
Windows log agent installation
Windows VM in GCP log agent installation

2. Check if Stanza is running using

ps -ef | grep stanza

3. Make the following edits to the config.yaml

type: file_input

include: file path

output: example_output

id: example_output

type: google_cloud_output 

Log agent configuration for windows logs in GCP

4. After the config file is saved, stop and start Stanza, using the following commands.

net stop stanza 

net start stanza 

Config Windows event logs in GCP

5. Verify if the logs are routed to the correct path and are available in GCP’s log explorer. To do this, in the Log Explorer console enter the following query:

label.file.name: ”stanza.log” stanza
Windows event logs in GCP via Stanza
Windows logs in GCP

Stanza for ingesting logs from GKE clusters:

1. As a prerequisite, follow the instructions in the link to create credentials secret on a JSON file.

2. Download and add the following files to the bucket in GCP. The files are available at

 https://github.com/observIQ/stanza/tree/7537876b255cf727106e8b6ca6622c8a07f36c57/examples/k8s/events

Config files in GCP bucket

3. Run the following command in the GKE cluster to create the service account. Our sample application is running an e-commerce application. For test purposes use this application linked here. 

kubectl apply -f service_account.yaml

4. Run the following command to create the config map. 

kubectl apply -f config.yaml

5. Run the following command to deploy the agent to the GKE cluster

kubectl apply -f deployment.yaml

6. Here’s a sample of the expected configuration if you choose to do it manually.

https://github.com/observIQ/stanza/blob/master/examples/k8s/daemonset/daemonset_gke.yaml

7. Verify in the logs explorer that Stanza ingests GKE cluster logs for the application or the test application.  

GKE Cluster logs stanza log agent GCP
Verify that Stanza ingests the GKE cluster logs

Stanza for  ingesting logs from MySQL databases in GCP:

1. Use the following default configurations to begin ingesting logs. 

- enable_error_log: true

  enable_general_log: false

  enable_mariadb_audit_log: false

  enable_slow_log: false

  error_log_path: /var/log/mysql/error.log

  start_at: end

  type: mysql

- type: google_cloud_output

2. Verify that the logs are ingested from the database in the logs explorer.

MySQL logs in GCP through Stanza
View the MySQL logs ingested in Log Explorer

Stanza is a lightweight logs ingestor and transporter. Clubbed with the great features of GCP, you should be able to set up flawless end-to-end observability for your applications. Try this out and write to us with your requests/ suggestions.

Sign Up for the observIQ Cloud Beta

Download the Splunk Solution Brief

Sign Up to receive updates on our products

observIQ Support

For support on observIQ Cloud, please contact:

support@observIQ.com

For the Open Source Log Agent, community-based support is available on our:

GitHub Repository

Sign Up for Our Newsletter