Logging GitLab Runners for MacOS and Linux

GitLab is the DevOps lifecycle tool of choice for most application developers. It was developed to offer continuous integration and deployment pipeline features on an open-source licensing model.

GitLab Runner is an open-source application integrated within the GitLab CI/ CD pipeline to automate running jobs in the pipeline. It is written in GoLang, making the platform agnostic. It is installed on any supported operating system, in a locally hosted application environment, or within a container. A Gitlab runner executes jobs outlined in the. gitlab-ci.yml file and sends the results back to GitLab.

GitLab Build Runner Common Workflow

The. GitLab-ci.yml file is installed in the root directory of GitLab’s repository. The most common stages in a Gitlab workflow are outlined below:

• Pre-build – These are the operations to be performed before the application build commences when there are new code changes/ commits. All libraries and tools required to create the build are executed as a precursor to this set of operations.
• Build – These executable operations create a new application build with the latest changes/ commits.
• Test – A set of tests to ensure that the new build of the application is functioning as expected.
• Deploy – A set of executable operations that follow a successful test run to move the new build to the designated production or other environments.

Related Content: How to Manage Sensitive Log Data

Advantages of Using GitLab Build Runners

Before we dive into logging build runners, it is necessary to know the reason for implementing a Gitlab build runner:

• To streamline the otherwise complicated application development and deployment process with a declarative approach, instead of one large deployment, the process of deployment is handled in a trimmed-down, repetitive, and automated series of tasks that create interdependent builds automatically – Thus making every software build automated, saving the developers time and effort spent on running and testing builds.
• They are quickly validating the code commits and changes. Both development and DevOps teams have complete control over build management, irrespective of the application model.
• A more systematic process for validating code changes ahead of merge via automated tests. The test run could be anything from a functional test to a more enhanced application performance test. This ensures the application is continuously tested to meet functional and business requirement standards and quickly identifies all possible application defects and security-related vulnerabilities.
• Gitlab is built in GoLang, making it a multi-platform CI solution that works well across operating systems such as Windows, MacOS, and Linux. It is also language agnostic, offering support for the most commonly used development languages such as Java, Ruby, PHP, etc.
• The declarative process of the Gitlab runner enables developers to build security into each automated process, ensuring access control and management across the CI pipeline. User access control is established at the job and branch levels, establishing compliance practices in the automated build runs. Ensuring code compliance by setting standards with built-in automated code quality checks is also easier.
• Gitlab builds runners and offers a robust integrative approach to upstream and downstream processes. This works advantageous for developers who want to balance cloud-native solutions and integrations. Processes such as source code control, project management, versioning, artifact repository management, compliance adherence, etc., are built into the CI pipeline.
• The most critical use is increased observability in the CI pipeline execution. As builds are run, GitLab logs all events to the designated location. Identifying build errors, code quality issues, test failures, and other application metrics is at a more granular level.

Related Content: Reducing Log Volume with Log-based Metrics

GitLab Build Runners for MacOS and Linux

GitLab released Beta versions of custom-built cloud runners for MacOS and Linux operating systems. The build cloud runner for Linux aims to manage Linux-based applications in the GCP environment, while the MacOS version targets build cloud management in an Apple devices-based ecosystem.

The build cloud runners are architecturally different from the other Gitlab shared runners. Gitlab’s detailed posts offer more information about the architecture and performance enhancements of the build cloud runners.

Logging Build Cloud Runners

Gitlab logs the outcome of every job as it processes them. By default, Gitlab sends the job logs from the GitLab Runner in chunks. For efficiency, Gitlab recommends incremental logging, in which Redis is used as a temporary cache for job logs. Designated object storage in the log archiving process ensures that the logs are rotated as the CI pipeline executes new builds. The standard logging workflow of Gitlab:

• During the patching phase, Gitlab logs every job execution status onto the file storage in the stored path.
• During the overwriting phase, Gitlab logs the status of all completed jobs onto the file storage in the stored path.
• During the archiving phase, Gitlab moves all completed build logs to the artifacts folder in the stored path.
• During the unloading phase, Gitlab moves all logs from the artifacts folders to the configured object storage in the selected path.

The first two steps of the logging workflow change when using incremental logging. Gitlab uses Redis to store chunks of logs and a preset persistent store to save the logs instead of using file storage. In this case, Redis is a first-class storage option that retains the log data until it reaches a set limit. Once that threshold is reached, the chunk of the log is transmitted to the persistent store, and from there, it is moved to object storage at the end of the stipulated build run time.

observIQ Solution

While the logging options offered by Gitlab build runners are elaborate, having the logs ingested into a log management tool with an effective log analysis option is most productive and valuable. observIQ offers that solution. With observIQ, you can have your GitLab build runner logs ingested seamlessly, parsed, indexed, and aligned graphically for extensive analysis. To check out our solution, sign up for a free account and try it out. If you have any comments or suggestions, could you send us feedback? We love making plug-in requests possible.