Logging Gitlab Runners for MacOS and Linux

by Deepa Ramachandra on September 14, 2021

Gitlab is the DevOps lifecycle tool of choice for most application developers. It was developed to offer continuous integration and deployment pipeline features on an open-source licensing model.

GitLab Runner is an open-source application that is integrated within the GitLab CI/ CD pipeline to automate running jobs in the pipeline. It is written in GoLang, making it platform agnostic. It is installed onto any supported operating system, a locally hosted application environment, or within a container. A Gitlab runner executes jobs outlined in the. gitlab-ci.yml file and sends the results back to GitLab.

Gitlab build runner common workflow

The. GitLab-ci.yml file is installed in the root directory of GitLab’s repository.  The most common stages in a Gitlab workflow are outlined below:     

  • Pre-build – These are the set of operations that are to be performed before the application build commences when there are new code changes/ commits. As a precursor to this set of operations, all libraries and tools that are required to create the build are executed. 
  • Build – These are the set of executable operations that create a new build of the application with the new changes/ commits. 
  • Test – A set of tests to ensure that the new build of the application is functioning as expected. 
  • Deploy – A set of executable operations that follow a successful test run to move the new build to the designated production or other environments. 

Advantages of using Gitlab build runners

Before we dive into logging build runners, it is necessary to know the reason for implementing a Gitlab build runner:

  • To streamline the otherwise complicated application development and deployment process with a declarative approach, instead of one large deployment, the process of deployment is handled in a trimmed-down, repetitive and automated series of tasks that create interdependent builds automatically – Thus making every software build automated, saving the developers time and effort spent on running and testing builds. 
  • Easily validating the code commits and changes. Both development and DevOps teams have full control over build management irrespective of the application model. 
  • A more systematic process for validating code changes ahead of merge via automated tests. The test run could be anything from a functional test to a more enhanced application performance test. This ensures that the application is continuously tested to meet functional and business requirement standards and to easily identify all possible application defects and security-related vulnerabilities.
  • Gitlab is built in GoLang, making it a multi-platform CI solution, working well across operating systems such as Windows, MacOS, and Linux. It is also language agnostic, offering support for most commonly used development languages such as Java, Ruby, PHP, etc. 
  • The declarative process of the Gitlab runner enables developers to build security into each automated process ensuring access control and management across the CI pipeline. User access control is established at the job level and branch level, giving way to establishing compliance practices in the automated build runs. It is also easier to ensure code compliance to set standards with built-in automated code quality checks. 
  • Gitlab build runners offer a strong integrative approach to both upstream and downstream processes. This works advantageous for developers who want to strike a balance between cloud-native solutions and integrations. Processes such as source code control, project management, versioning, artifact repository management, compliance adherence, etc. are built into the CI pipeline. 
  • The most critical use of all is the increased observability into the CI pipeline execution. As builds are run Gitlab logs all events to the designated location. Identifying build errors, code quality issues, test failures, and other application metrics are at a more granular level. 

Gitlab build runners for MacOS and Linux

Gitlab released Beta versions of custom build cloud runners for MacOS and Linux operating systems. The build cloud runner for Linux is aimed at managing Linux-based applications in the GCP environment while the MacOS version targets build cloud management in an Apple devices-based ecosystem. 

The build cloud runners are architecturally different from the other Gitlab shared runners. Gitlab’s detailed posts offer more information about the architecture and performance enhancements of the build cloud runners. 

Logging build cloud runners

Gitlab logs the outcome of every job as it processes them. By default, Gitlab sends the job logs from the GitLab Runner in chunks. For efficiency, Gitlab recommends incremental logging, in which Redis is used as a temporary cache for job logs. Designated object storage in the log archiving process ensures that the logs are rotated as the CI pipeline executes new builds. The common logging workflow of Gitlab:

  • During the patching phase, Gitlab logs every job execution status onto the file storage in the stored path.
  • During the overwriting phase, Gitlab logs the status of all completed jobs onto the file storage in the stored path. 
  • During the archiving phase, Gitlab moves all logs of a completed build to the artifacts folder in the stored path. 
  • During the unloading phase, Gitlab moves all logs from the artifacts folders to the configured object storage in the selected path. 

When using incremental logging, the first two steps of the logging workflow change. Gitlab uses Redis to store chunks of logs and a preset persistent store to save the logs instead of the file storage. In this case, Redis is used as a first-class storage option that retains the log data until it reaches a set limit. Once that threshold is reached, the chunk of the log is transmitted to the persistent store, from there it is moved to object storage at the end of the stipulated build run time. 

observIQ solution

While the logging option offered by Gitlab build runners are elaborate, it is most productive and useful to have the logs ingested into a log management tool with an effective log analysis option. observIQ offers that solution. With observIQ you can have your Gitlab build runner logs ingested seamlessly, parsed, indexed, and aligned graphically for extensive analysis. To check out our solution, signup for a free account and try it out. If you have any comments or suggestions, send us feedback. We love making plug-in requests possible. 

Sign Up for the observIQ Cloud Beta

Download the Splunk Solution Brief

Sign Up to receive updates on our products

observIQ Support

For support on observIQ Cloud, please contact:

support@observIQ.com

For the Open Source Log Agent, community-based support is available on our:

GitHub Repository

Sign Up for Our Newsletter