Stanza is a robust log agent. GCP users can use Stanza for ingesting large volumes of log data. Before we dive into the configuration steps, here’s a matrix detailing the functional differences between all the common log agents used by GCP users.
Stanza was built as a modernized version of FluentD, Fluentbit, and Logstash. GCP users now have the ability to install Stanza to their VMs/ GKE clusters to ingest logs and route them to GCP log explorer. In this post we detail the steps for installing Stanza to Linux, Windows and Kubernetes environments and viewing the logs in the GCP log explorer.
Stanza for a Linux VM in GCP:
1. Single line installation command: In your VM, run the following single line installation command for Stanza
sh -c "$(curl -fsSlL https://github.com/observiq/stanza/releases/latest/download/unix-install.sh)" unix-install.sh
2. Once the installation is complete, the following message displays. It provides the commands for starting and stopping Stanza. In addition, it shows the path for the config file.
3. Check if Stanza is running using
ps -ef | grep stanza
4. Open the config.yaml using the command vi config.yaml. In this example, we use the vi editor, alternatively any linux/unix editor could be used to edit and save the config file.
5. Comment everything except the following in the config.yaml and save the file
type: file_input include: file path output: example_output id: example_output type: google_cloud_output
6. After the config file is saved, stop and start Stanza, using the following commands and very if the service is running.
systemctl stop stanza systemctl start stanza ps -ef | grep stanza
7. Run a search query as shown below to verify if the logs are sent to GCP cloud and it is available for view in GCP’s log Explorer.
Stanza for a Windows VM in GCP:
1. Access the command line and run the single line installation command for Windows.
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12 ; Invoke-Expression ((New-Object net.webclient).DownloadString('https://github.com/observiq/stanza/releases/latest/download/windows-install.ps1')); Log-Agent-Install
2. Check if Stanza is running using
ps -ef | grep stanza
3. Make the following edits to the config.yaml
type: file_input include: file path output: example_output id: example_output type: google_cloud_output
4. After the config file is saved, stop and start Stanza, using the following commands.
net stop stanza net start stanza
5. Verify if the logs are routed to the correct path and are available in GCP’s log explorer. To do this, in the Log Explorer console enter the following query:
label.file.name: ”stanza.log” stanza
Stanza for ingesting logs from GKE clusters:
1. As a prerequisite, follow the instructions in the link to create credentials secret on a JSON file.
2. Download and add the following files to the bucket in GCP. The files are available at
https://github.com/observIQ/stanza/tree/7537876b255cf727106e8b6ca6622c8a07f36c57/examples/k8s/events
3. Run the following command in the GKE cluster to create the service account. Our sample application is running an e-commerce application. For test purposes use this application linked here.
kubectl apply -f service_account.yaml
4. Run the following command to create the config map.
kubectl apply -f config.yaml
5. Run the following command to deploy the agent to the GKE cluster
kubectl apply -f deployment.yaml
6. Here’s a sample of the expected configuration if you choose to do it manually.
https://github.com/observIQ/stanza/blob/master/examples/k8s/daemonset/daemonset_gke.yaml
7. Verify in the logs explorer that Stanza ingests GKE cluster logs for the application or the test application.
Stanza for ingesting logs from MySQL databases in GCP:
1. Use the following default configurations to begin ingesting logs.
- enable_error_log: true enable_general_log: false enable_mariadb_audit_log: false enable_slow_log: false error_log_path: /var/log/mysql/error.log start_at: end type: mysql - type: google_cloud_output
2. Verify that the logs are ingested from the database in the logs explorer.
Stanza is a lightweight logs ingestor and transporter. Clubbed with the great features of GCP, you should be able to set up flawless end-to-end observability for your applications. Try this out and write to us with your requests/ suggestions.