GitLab
采集 GitLab 运行数据并以指标的方式上报到观测云。
配置¶
首先需要打开 GitLab 服务的数据采集功能和设置白名单,具体操作见后续分段。
GitLab 设置完成后,对 DataKit 进行配置。注意,根据 GitLab 版本和配置不同,采集到的数据可能存在差异。
进入 DataKit 安装目录下的 conf.d/samples 目录,复制 gitlab.conf.sample 并命名为 gitlab.conf。示例如下:
[[inputs.gitlab]]
## set true if you need to collect metric from url below
enable_collect = true
## param type: string - default: http://127.0.0.1:80/-/metrics
prometheus_url = "http://127.0.0.1:80/-/metrics"
## param type: string - optional: time units are "ms", "s", "m", "h" - default: 10s
interval = "10s"
## datakit can listen to gitlab ci data at /v1/gitlab when enabled
enable_ci_visibility = true
## Set true to enable election
election = true
## extra tags for gitlab-ci data.
## these tags will not overwrite existing tags.
[inputs.gitlab.ci_extra_tags]
# some_tag = "some_value"
# more_tag = "some_other_value"
## extra tags for gitlab metrics
[inputs.gitlab.tags]
# some_tag = "some_value"
# more_tag = "some_other_value"
配置好后,重启 DataKit 即可。
目前可以通过 ConfigMap 方式注入采集器配置来开启采集器。
GitLab 开启数据采集功能¶
GitLab 需要开启 Prometheus 数据采集功能,开启方式如下(以英文页面为例):
- 以管理员账号登陆己方 GitLab 页面
- 转到
Admin Area>Settings>Metrics and profiling - 选择
Metrics - Prometheus,点击Enable Prometheus Metrics并且save change - 重启 GitLab 服务
详情见官方配置文档。
配置数据访问端白名单¶
只开启数据采集功能还不够,GitLab 对于数据管理十分严格,需要再配置访问端的白名单。开启方式如下:
- 修改 GitLab 配置文件
/etc/gitlab/gitlab.rb,找到gitlab_rails['monitoring_whitelist'] = ['::1/128']并在该数组中添加 DataKit 的访问 IP(通常情况为 DataKit 所在主机的 IP,如果 GitLab 运行在容器中需根据实际情况添加) - 重启 GitLab 服务
详情见官方配置文档。
开启 GitLab CI 可视化¶
通过配置 GitLab Webhook,可以实现 GitLab CI 可视化。
在 GitLab 左边栏,转到 Settings ➔ Webhooks 中,将 URL 配置为第一步的 API 地址,Trigger 配置 Job events 和 Pipeline events 两项,点击 Add webhook 确认添加;
指标¶
以下所有数据采集,默认会追加全局选举 tag,也可以在配置中指定其它标签:
- 可以在配置中通过
[inputs.gitlab.tags]为 GitLab 指标数据指定其它标签:
- 可以在配置中通过
[inputs.gitlab.ci_extra_tags]为 GitLab CI 数据指定其它标签:
注意:为了确保 GitLab CI 功能正常,为 GitLab CI 数据指定的 extra tags 不会覆盖其数据中已有的标签(GitLab CI 标签列表见下)。
gitlab¶
GitLab runtime metrics
| Tags & Fields | Description |
|---|---|
| action ( tag) |
Action |
| controller ( tag) |
Controller |
| feature_category ( tag) |
Feature category |
| storage ( tag) |
Storage |
| banzai_cacheless_render_real_duration_seconds_count | The count of duration of rendering Markdown into HTML when cached output exists Type: float Unit: time,s |
| banzai_cacheless_render_real_duration_seconds_sum | The sum of duration of rendering Markdown into HTML when cached output exists Type: float Unit: time,s |
| cache_misses_total | The cache read miss count Type: float Unit: - |
| cache_operation_duration_seconds_count | The count of cache access time Type: float Unit: time,s |
| cache_operation_duration_seconds_sum | The count of cache access time Type: float Unit: time,s |
| cache_operations_total | The count of cache access time Type: float Unit: - |
| rack_requests_total | The rack request count Type: float Unit: - |
| redis_client_requests_duration_seconds_count | The count of redis request latency, excluding blocking commands Type: float Unit: time,s |
| redis_client_requests_duration_seconds_sum | The sum of redis request latency, excluding blocking commands Type: float Unit: time,s |
| redis_client_requests_total | Number of Redis client requests Type: float Unit: - |
| sql_duration_seconds_count | The total SQL execution time, excluding SCHEMA operations and BEGIN / COMMIT Type: float Unit: time,s |
| sql_duration_seconds_sum | The sum of SQL execution time, excluding SCHEMA operations and BEGIN / COMMIT Type: float Unit: time,s |
| transaction_cache_read_hit_count_total | The counter for cache hits for Rails cache calls Type: float Unit: count |
| transaction_cache_read_miss_count_total | The counter for cache misses for Rails cache calls Type: float Unit: count |
| transaction_db_cached_count_total | The counter for db cache Type: float Unit: count |
| transaction_db_count_total | The counter for db Type: float Unit: count |
| transaction_duration_seconds_count | The count of duration for all transactions (gitlab_transaction_* metrics) Type: float Unit: time,s |
| transaction_duration_seconds_sum | The sum of duration for all transactions (gitlab_transaction_* metrics) Type: float Unit: time,s |
| transaction_new_redis_connections_total | The counter for new Redis connections Type: float Unit: - |
| transaction_view_duration_total | The duration for views Type: float Unit: - |
gitlab_base¶
GitLab programming language level metrics
| Tags & Fields | Description |
|---|---|
| rails_queue_duration_seconds_count | The counter for latency between GitLab Workhorse forwarding a request to Rails Type: float Unit: time,s |
| rails_queue_duration_seconds_sum | The sum for latency between GitLab Workhorse forwarding a request to Rails Type: float Unit: time,s |
| ruby_gc_duration_seconds_count | The count of time spent by Ruby in GC Type: float Unit: time,s |
| ruby_gc_duration_seconds_sum | The sum of time spent by Ruby in GC Type: float Unit: time,s |
| ruby_sampler_duration_seconds_total | The time spent collecting stats Type: float Unit: time,s |
gitlab_http¶
GitLab HTTP metrics
| Tags & Fields | Description |
|---|---|
| method ( tag) |
方法 |
| status ( tag) |
状态码 |
| http_health_requests_total | Number of health requests Type: float Unit: - |
| http_request_duration_seconds_count | The counter for request duration Type: float Unit: time,s |
| http_request_duration_seconds_sum | The sum for request duration Type: float Unit: time,s |
日志¶
gitlab_pipeline¶
GitLab Pipeline event metrics
| Tags & Fields | Description |
|---|---|
| author_email ( tag) |
Author email |
| ci_status ( tag) |
CI type |
| commit_sha ( tag) |
The commit SHA of the most recent commit of the code that triggered the Pipeline |
| object_kind ( tag) |
Event type, in this case Pipeline |
| operation_name ( tag) |
Operation name |
| pipeline_name ( tag) |
Pipeline name |
| pipeline_source ( tag) |
Sources of Pipeline triggers |
| pipeline_url ( tag) |
Pipeline URL |
| ref ( tag) |
Branches involved |
| repository_url ( tag) |
Repository URL |
| resource ( tag) |
Project name |
| commit_message | The message attached to the most recent commit of the code that triggered the Pipeline. Type: string Unit: - |
| created_at | Millisecond timestamp of Pipeline creation Type: int Unit: timeStamp,msec |
| duration | Pipeline duration (microseconds) Type: int Unit: time,μs |
| event_raw | The raw JSON body of the webhook event Type: string Unit: N/A |
| finished_at | Millisecond timestamp of the end of the Pipeline Type: int Unit: timeStamp,msec |
| message | The message attached to the most recent commit of the code that triggered the Pipeline. Same as commit_message Type: string Unit: N/A |
| pipeline_id | Pipeline id Type: string Unit: - |
| queued_duration | Pipeline queued duration Type: int Unit: time,ms |
gitlab_job¶
GitLab Job Event metrics
| Tags & Fields | Description |
|---|---|
| build_commit_sha ( tag) |
The commit SHA corresponding to build |
| build_failure_reason ( tag) |
Build failure reason |
| build_name ( tag) |
Build name |
| build_repo_name ( tag) |
Repository name corresponding to build |
| build_stage ( tag) |
Build stage |
| build_status ( tag) |
Build status |
| object_kind ( tag) |
Event type, in this case Job |
| project_name ( tag) |
Project name |
| sha ( tag) |
The commit SHA corresponding to build |
| user_email ( tag) |
User email |
| build_commit_message | The message attached to the most recent commit of the code that triggered the build Type: string Unit: - |
| build_duration | Build duration (microseconds) Type: int Unit: time,μs |
| build_finished_at | Millisecond timestamp of the end of build Type: int Unit: timeStamp,msec |
| build_id | build id Type: string Unit: - |
| build_started_at | Millisecond timestamp of the start of build Type: int Unit: timeStamp,msec |
| event_raw | The raw JSON body of the webhook event Type: string Unit: N/A |
| message | The message attached to the most recent commit of the code that triggered the build. Same as build_commit_message Type: string Unit: - |
| pipeline_id | Pipeline id for build Type: string Unit: - |
| project_id | Project id for build Type: string Unit: - |
| queued_duration | Job queued duration Type: int Unit: time,ms |
| runner_id | Runner id for build Type: string Unit: - |