Skip to content

Gitlab-CI Observability Best Practices

GitLab

GitLab is a web-based Git repository management tool developed by GitLab Inc. under the MIT License, with wiki and issue tracking capabilities. It is built on top of Git for code management.

CI/CD

CI/CD stands for Continuous Integration (CI), Continuous Delivery (CD), and Continuous Deployment (CD).
Continuous Integration focuses on merging the work of individual developers into a shared code repository, often several times a day. Its primary goal is to detect integration errors early, enabling teams to work more closely and collaborate better.
Continuous Delivery aims to minimize the friction inherent in deployment or release processes. It typically automates each step of the build and deployment pipeline so that code releases can be performed safely at any time (ideally).
Continuous Deployment is a higher level of automation where code changes are automatically built and deployed whenever significant changes are made.
Some CI/CD tools include:

  1. Jenkins

  2. GitLab CI

  3. Travis CI
  4. GoCD

GitLab CI

GitLab CI/CD (hereafter referred to as GitLab CI) is a CI/CD system built on GitLab. Although relatively new in the CI/CD space, it has already achieved a leading position in the Forrester Wave for Continuous Integration tools. It allows developers to configure CI/CD pipelines in their projects via a .gitlab-ci.yml file. After a commit, the system can automatically or manually execute jobs to complete CI/CD operations. Its configuration is very simple: the CI Runner is written in Go and packaged as a single file, so you only need a Runner program and an execution platform for running jobs (e.g., bare metal + SSH, Docker, or Kubernetes; Docker is recommended because it is easy to set up) to run a complete CI/CD system.

Guance

“Guance” is a cloud-era observability platform. It includes modules such as infrastructure, logs, metrics, events, Application Performance Monitoring (APM), Real User Monitoring (RUM), Synthetic Monitoring, and system-level security checks. It provides full-stack data analysis and insights for the Logging, Metrics, and Tracing data generated by these modules. It fully covers H5, iOS, Android, and Mini Programs, supports complete tracking of user access behavior and real experience, and provides views and data for page performance, resource calls, error alerts, and business access. Linked with distributed tracing, it helps you gain real-time insight into application performance and the real needs behind each request. Flexible scenario layouts, rich chart options, and drag-and-drop interaction make it easy to build your own dashboards. A unified query method supports configuring various types of data, making it simple and easy to use.

DataKit

DataKit is an open-source collection tool from Guance. Open-source repository: https://github.com/DataFlux-cn/datakit

Background

As microservices become increasingly popular, enterprises are gradually transitioning from monolithic architectures to microservices architectures. A characteristic of microservices is the large number of engineering modules, making deployment relatively complex. While CI/CD tools can well integrate delivery and deployment, it is difficult to statistically analyze issues that arise during the deployment process. Leveraging Guance's powerful observability and custom view capabilities helps you effectively view and analyze problems encountered during continuous deployment.

Architecture Flow

image.png

  1. Developer commits and pushes code

  2. GitLab Runner registers with GitLab

  3. GitLab trigger triggers GitLab CI execution
  4. After GitLab CI completes, it triggers a webhook to push data to DataKit
  5. DataKit tags and pushes the data to the Guance platform

Prerequisites

Enable GitLab CI in DataKit

Edit gitlab.conf

cd conf.d/gitlab cp gitlab.conf.sample gitlab.conf

Full content of gitlab.conf:

[[inputs.gitlab]]
## set true if you need to collect metric from url below
enable_collect = false

## param type: string - default: http://127.0.0.1:80/-/metrics
prometheus_url = "http://127.0.0.1:80/-/metrics"

## param type: string - optional: time units are "ms", "s", "m", "h" - default: 10s
interval = "10s"

## datakit can listen to gitlab ci data at /v1/gitlab when enabled
enable_ci_visibility = true

## extra tags for gitlab-ci data.
## these tags will not overwrite existing tags.
[inputs.gitlab.ci_extra_tags]
# some_tag = "some_value"
# more_tag = "some_other_value"

## extra tags for gitlab metrics
[inputs.gitlab.tags]
# some_tag = "some_value"
# more_tag = "some_other_value"

Parameter description:

  • enable_collect: false # Disable metric collection
  • prometheus_url: Metric collection address
  • enable_ci_visibility: true # Enable GitLab CI visibility

Restart DataKit

datakit --restart

GitLab Installation and Configuration

Skip if already installed.

Install GitLab via Docker

docker run --name=gitlab -d -p 8899:8899 -p 2443:443 --restart always \ --volume /data/midsoftware/gitlab/config:/etc/gitlab \ --volume /data/midsoftware/gitlab/logs:/var/log/gitlab \ --volume /data/midsoftware/gitlab/data:/var/opt/gitlab \ docker.io/gitlab/gitlab-ce

Port description:

Port Description
8899 GitLab UI port
2443 GitLab SSL port

Modify configuration file: gitlab.rb

# Access URL
external_url 'http://192.168.91.11:8899'

# Set timeout, default 10 (in seconds)
gitlab_rails['webhook_timeout'] = 60

Restart GitLab

docker restart gitlab

Check GitLab version

[root@middle config]# docker exec -it gitlab cat /opt/gitlab/embedded/service/gitlab-rails/VERSION
14.6.2

View GitLab initial password

[root@middle config]# docker exec -it gitlab cat /etc/gitlab/initial_root_password |grep Password
#          2. Password hasn't been changed manually, either via UI or via command line.
Password: yBY9toQ0SJ8fxh3mndHPzfWclVUDZ/J8e8O4bDsal2E=

The account is root. Log in via browser at http://ip:8899 and change the password.

2022-01-14-12-02-04-image.png

Create your first project

Menu -> Projects -> Your Projects -> New Project -> Create blank project. Fill in the project name.

image.png

GitLab Runner Installation and Configuration

Skip if already installed.

Install GitLab Runner via Docker

docker run -d --name gitlab-runner --restart always \
    -v /data/midsoftware/gitlab-runner/config:/etc/gitlab-runner \
    -v /var/run/docker.sock:/var/run/docker.sock \
    gitlab/gitlab-runner:latest

GitLab Runner Registration Token

GitLab Runner does not currently support global configuration, so the Runner token must be found in the project. Go to the newly created project -> Settings -> Runners, and copy the token. It will be needed for the next step of Runner registration.

image.png

Register GitLab Runner with GitLab

docker run --rm -v /data/midsoftware/gitlab-runner/config:/etc/gitlab-runner gitlab/gitlab-runner register \
  --non-interactive \
  --executor "docker" \
  --docker-image alpine:latest \
  --url "http://192.168.91.11:8899" \
  --registration-token "U6uhCZGPrZ7tGs6aV8rY" \
  --description "gitlab-runner" \
  --tag-list "docker,localMachine" \
  --run-untagged="true" \
  --locked="false" \
  --access-level="not_protected"

Parameter description:

Parameter Value Description
executor docker Can use other executors or omit this parameter. If deploying to Docker environment, Docker is recommended.
docker-image alpine:latest Docker image version, used together with executor.
url http://192.168.91.11:8899 GitLab access URL
registration-token token GitLab admin token
description gitlab-runner Description
tag-list docker,localMachine Tags can be used to select the corresponding executor.

For more parameters, refer to: https://docs.gitlab.com/runner/configuration/advanced-configuration.html

Configure GitLab Webhook

Go to the newly created project, select Settings -> Webhooks, fill in the URL, check Pipeline events, and save.

image.png

Description:

URL: http:///v1/gitlab

Enable Job events

Enable Pipeline events

You can also click Test -> select Pipeline events, which will trigger a pipeline event and push the data to the configured webhook URL. Check the status to verify the process.

image.png

Write .gitlab-ci.yml

Go to the newly created project, select CI/CD -> Editor.

image.png

Fill in the script content as follows:

# Set the execution image
image: busybox:latest

# The entire pipeline has two stages
stages:
  - build
  - test

before_script:
  - echo "Before script section"

after_script:
  - echo "After script section"


build_job:
  stage: build
  only:
    - master
  script:
    - echo "Writing content to cache"
    - sleep 80s
    # - d ps 

test_job:
  stage: test
  script:
    - echo "Reading content from cache"

After saving, the CI/CD pipeline is automatically triggered. The process will push data to the configured webhook URL.

View Webhook Push Records

image.png

A status of 200 indicates success.

Guance

After the pipeline is successfully pushed, you can use the Guance platform to visually observe the overall execution status of the pipeline through dashboards and explorers.

CI Explorer

Use the CI menu in the explorer to view gitlab_pipeline and gitlab_job details.

gitlab_pipeline:

image.png

gitlab_job:

image.png

Click on a detail to view the flame graph and job list.

Flame graph:

image.png

CI Overview

The CI overview provides a view of GitLab CI pipeline and job execution, such as pipeline success rate, execution time, job success rate, and execution time.

image.png

Feedback

Is this page helpful?