GitLab
采集 GitLab 运行数据并以指标的方式上报到观测云。
配置¶
首先需要打开 GitLab 服务的数据采集功能和设置白名单,具体操作见后续分段。
GitLab 设置完成后,对 DataKit 进行配置。注意,根据 GitLab 版本和配置不同,采集到的数据可能存在差异。
进入 DataKit 安装目录下的 conf.d/samples 目录,复制 gitlab.conf.sample 并命名为 gitlab.conf。示例如下:
[[inputs.gitlab]]
    ## set true if you need to collect metric from url below
    enable_collect = true
    ## param type: string - default: http://127.0.0.1:80/-/metrics
    prometheus_url = "http://127.0.0.1:80/-/metrics"
    ## param type: string - optional: time units are "ms", "s", "m", "h" - default: 10s
    interval = "10s"
    ## datakit can listen to gitlab ci data at /v1/gitlab when enabled
    enable_ci_visibility = true
    ## Set true to enable election
    election = true
    ## extra tags for gitlab-ci data.
    ## these tags will not overwrite existing tags.
    [inputs.gitlab.ci_extra_tags]
    # some_tag = "some_value"
    # more_tag = "some_other_value"
    ## extra tags for gitlab metrics
    [inputs.gitlab.tags]
    # some_tag = "some_value"
    # more_tag = "some_other_value"
配置好后,重启 DataKit 即可。
目前可以通过 ConfigMap 方式注入采集器配置来开启采集器。
GitLab 开启数据采集功能¶
GitLab 需要开启 Prometheus 数据采集功能,开启方式如下(以英文页面为例):
- 以管理员账号登陆己方 GitLab 页面
- 转到 Admin Area>Settings>Metrics and profiling
- 选择 Metrics - Prometheus,点击Enable Prometheus Metrics并且save change
- 重启 GitLab 服务
详情见官方配置文档。
配置数据访问端白名单¶
只开启数据采集功能还不够,GitLab 对于数据管理十分严格,需要再配置访问端的白名单。开启方式如下:
- 修改 GitLab 配置文件 /etc/gitlab/gitlab.rb,找到gitlab_rails['monitoring_whitelist'] = ['::1/128']并在该数组中添加 DataKit 的访问 IP(通常情况为 DataKit 所在主机的 IP,如果 GitLab 运行在容器中需根据实际情况添加)
- 重启 GitLab 服务
详情见官方配置文档。
开启 GitLab CI 可视化¶
通过配置 GitLab Webhook,可以实现 GitLab CI 可视化。
在 GitLab 左边栏,转到 Settings ➔ Webhooks 中,将 URL 配置为第一步的 API 地址,Trigger 配置 Job events 和 Pipeline events 两项,点击 Add webhook 确认添加;
指标¶
以下所有数据采集,默认会追加全局选举 tag,也可以在配置中指定其它标签:
- 可以在配置中通过 [inputs.gitlab.tags]为 GitLab 指标数据指定其它标签:
- 可以在配置中通过 [inputs.gitlab.ci_extra_tags]为 GitLab CI 数据指定其它标签:
注意:为了确保 GitLab CI 功能正常,为 GitLab CI 数据指定的 extra tags 不会覆盖其数据中已有的标签(GitLab CI 标签列表见下)。
gitlab¶
GitLab runtime metrics
| Tags & Fields | Description | 
|---|---|
| action ( tag) | Action | 
| controller ( tag) | Controller | 
| feature_category ( tag) | Feature category | 
| storage ( tag) | Storage | 
| banzai_cacheless_render_real_duration_seconds_count | The count of duration of rendering Markdown into HTML when cached output exists Type: float Unit: time,s | 
| banzai_cacheless_render_real_duration_seconds_sum | The sum of duration of rendering Markdown into HTML when cached output exists Type: float Unit: time,s | 
| cache_misses_total | The cache read miss count Type: float Unit: - | 
| cache_operation_duration_seconds_count | The count of cache access time Type: float Unit: time,s | 
| cache_operation_duration_seconds_sum | The count of cache access time Type: float Unit: time,s | 
| cache_operations_total | The count of cache access time Type: float Unit: - | 
| rack_requests_total | The rack request count Type: float Unit: - | 
| redis_client_requests_duration_seconds_count | The count of redis request latency, excluding blocking commands Type: float Unit: time,s | 
| redis_client_requests_duration_seconds_sum | The sum of redis request latency, excluding blocking commands Type: float Unit: time,s | 
| redis_client_requests_total | Number of Redis client requests Type: float Unit: - | 
| sql_duration_seconds_count | The total SQL execution time, excluding SCHEMA operations and BEGIN / COMMIT Type: float Unit: time,s | 
| sql_duration_seconds_sum | The sum of SQL execution time, excluding SCHEMA operations and BEGIN / COMMIT Type: float Unit: time,s | 
| transaction_cache_read_hit_count_total | The counter for cache hits for Rails cache calls Type: float Unit: count | 
| transaction_cache_read_miss_count_total | The counter for cache misses for Rails cache calls Type: float Unit: count | 
| transaction_db_cached_count_total | The counter for db cache Type: float Unit: count | 
| transaction_db_count_total | The counter for db Type: float Unit: count | 
| transaction_duration_seconds_count | The count of duration for all transactions (gitlab_transaction_* metrics) Type: float Unit: time,s | 
| transaction_duration_seconds_sum | The sum of duration for all transactions (gitlab_transaction_* metrics) Type: float Unit: time,s | 
| transaction_new_redis_connections_total | The counter for new Redis connections Type: float Unit: - | 
| transaction_view_duration_total | The duration for views Type: float Unit: - | 
gitlab_base¶
GitLab programming language level metrics
| Tags & Fields | Description | 
|---|---|
| rails_queue_duration_seconds_count | The counter for latency between GitLab Workhorse forwarding a request to Rails Type: float Unit: time,s | 
| rails_queue_duration_seconds_sum | The sum for latency between GitLab Workhorse forwarding a request to Rails Type: float Unit: time,s | 
| ruby_gc_duration_seconds_count | The count of time spent by Ruby in GC Type: float Unit: time,s | 
| ruby_gc_duration_seconds_sum | The sum of time spent by Ruby in GC Type: float Unit: time,s | 
| ruby_sampler_duration_seconds_total | The time spent collecting stats Type: float Unit: time,s | 
gitlab_http¶
GitLab HTTP metrics
| Tags & Fields | Description | 
|---|---|
| method ( tag) | 方法 | 
| status ( tag) | 状态码 | 
| http_health_requests_total | Number of health requests Type: float Unit: - | 
| http_request_duration_seconds_count | The counter for request duration Type: float Unit: time,s | 
| http_request_duration_seconds_sum | The sum for request duration Type: float Unit: time,s | 
日志¶
gitlab_pipeline¶
GitLab Pipeline event metrics
| Tags & Fields | Description | 
|---|---|
| author_email ( tag) | Author email | 
| ci_status ( tag) | CI type | 
| commit_sha ( tag) | The commit SHA of the most recent commit of the code that triggered the Pipeline | 
| object_kind ( tag) | Event type, in this case Pipeline | 
| operation_name ( tag) | Operation name | 
| pipeline_name ( tag) | Pipeline name | 
| pipeline_source ( tag) | Sources of Pipeline triggers | 
| pipeline_url ( tag) | Pipeline URL | 
| ref ( tag) | Branches involved | 
| repository_url ( tag) | Repository URL | 
| resource ( tag) | Project name | 
| commit_message | The message attached to the most recent commit of the code that triggered the Pipeline. Type: string Unit: - | 
| created_at | Millisecond timestamp of Pipeline creation Type: int Unit: timeStamp,msec | 
| duration | Pipeline duration (microseconds) Type: int Unit: time,μs | 
| event_raw | The raw JSON body of the webhook event Type: string Unit: N/A | 
| finished_at | Millisecond timestamp of the end of the Pipeline Type: int Unit: timeStamp,msec | 
| message | The message attached to the most recent commit of the code that triggered the Pipeline. Same as commit_message Type: string Unit: N/A | 
| pipeline_id | Pipeline id Type: string Unit: - | 
| queued_duration | Pipeline queued duration Type: int Unit: time,ms | 
gitlab_job¶
GitLab Job Event metrics
| Tags & Fields | Description | 
|---|---|
| build_commit_sha ( tag) | The commit SHA corresponding to build | 
| build_failure_reason ( tag) | Build failure reason | 
| build_name ( tag) | Build name | 
| build_repo_name ( tag) | Repository name corresponding to build | 
| build_stage ( tag) | Build stage | 
| build_status ( tag) | Build status | 
| object_kind ( tag) | Event type, in this case Job | 
| project_name ( tag) | Project name | 
| sha ( tag) | The commit SHA corresponding to build | 
| user_email ( tag) | User email | 
| build_commit_message | The message attached to the most recent commit of the code that triggered the build Type: string Unit: - | 
| build_duration | Build duration (microseconds) Type: int Unit: time,μs | 
| build_finished_at | Millisecond timestamp of the end of build Type: int Unit: timeStamp,msec | 
| build_id | build id Type: string Unit: - | 
| build_started_at | Millisecond timestamp of the start of build Type: int Unit: timeStamp,msec | 
| event_raw | The raw JSON body of the webhook event Type: string Unit: N/A | 
| message | The message attached to the most recent commit of the code that triggered the build. Same as build_commit_message Type: string Unit: - | 
| pipeline_id | Pipeline id for build Type: string Unit: - | 
| project_id | Project id for build Type: string Unit: - | 
| queued_duration | Job queued duration Type: int Unit: time,ms | 
| runner_id | Runner id for build Type: string Unit: - |