Managing Configuration with Git¶
How Git Works¶
Git is a technology for version control, the same as SVN. For more information, see here.
Git components are divided into Git Server and Git Client. Running on the remote server is the Git Server, the remote repository. In the local (or Kubernetes container). The following words "local" all mean this. ) is running the Git Client, the local copy.
The content managed by Git is divided into local copy and remote warehouse. Changes are submitted locally as a copy during commit operations and to the remote repository only when push operations are performed.
Create A Git Repository¶
You can generally create a Git repository using New Project
in Github/GitLab.
After creating the Git repository, you can get an address like http://github.com/path/to/repository.git
, through which the Git Client pushes or pulls the content.
Git Operation Flow¶
Generally Git operation flow is roughly as follows:
Step 1: Add the change file. Such as:
Step 2: Explain this change and submit it to the local copy (commit operation). Such as:
Step 3: Commit the changes to the remote repository (push operation). Such as:
Directory Requirements for Git Repositories¶
gitrepos/repo-name/conf.d
is used to place collector configuration files with unrestricted subdirectories (datakit.conf
is not managed bygitrepos
)gitrepos/repo-name/pipeline
is used to put pipeline scripts, and only.p
in the first tier of this directory will take effect, and none of its subdirectories will take effectgitrepos/repo-name/python.d
for python scripts
Submit A conf File and Directory¶
The following is an example of the ClickHouse collector.
Step 1: Switch to the /root
directory and use the git clone http://github.com/path/to/repository.git
command to pull the remote repository locally.
Select the collector you want to open, here is ClickHouse. Copy [Datakit 安装目录]/conf.d/db/ClickHousev1.conf.sample
to the /root/repository
directory above.
Note: All collector configuration file samples are in the [Datakit 安装目录]/conf.d
directory.
The file name is removed from .sample
, and the file structure is as follows:
According to their actual situation, modify the ClickHousev1.conf
configuration, save.
Step 2: Commit changes to the remote repository.
git add ClickHousev1.conf # Add change file
git commit -m "new ClickHousev1.conf" # Add change description
git push origin master # Commit changes to remote repository
At this point, the edited ClickHousev1.conf
file has been successfully pushed to the remote repository.
Configure the Repository on the DataKit¶
The demonstration here adopts the host mode, which is not suitable for Kubernetes environment. Operations in the Kubernetes environment are described separately below.
The Git authentication method demonstrated here is user name and password.
Step 1: You need to turn on the gitrepos
functionality in datakit.conf
.
Find git_repos
in datakit.conf
to configure as follows:
[git_repos]
pull_interval = "1m" # Pull updates every minute
[[git_repos.repo]]
enable = true # Open to pull this Git branch.
url = "http://username:password@github.com/path/to/repository.git" # User name/password authentication is used.
branch = "master" # The name of the branch to pull. Usually master.
Step 2: After configuration, restart DataKit.
Step 3: Observe whether Git has pulled updates and loaded the configuration.
You can observe whether the newly added/modified collector is effective:
Update and Pull Warehouse¶
We have a local copy in /root/repository
above. There we made some modifications to the ClickHousev1.conf
file.
Submit after modification is completed:
git add ClickHousev1.conf # Add change file
git commit -m "modify ClickHousev1.conf" # Add change description
git push origin master # Commit changes to remote repository
After the submission is completed. DataKit pulls according to the interval set by pull_interval
in the configuration, and when the interval expires, it automatically pulls the latest ClickHousev1.conf
and makes it effective.
Git Uses in Kubernetes¶
Because of the particularity of Kubernetes environment, the installation/configuration mode with environment variable passing is the simplest.
The git authentication method is user name and password.
When installing in Kubernetes, you need to set the following environment variables to bring Git configuration information into it:
Environment Variable Name | Environment Variable Value |
---|---|
ENV_GIT_URL | http://username:password@github.com/path/to/repository.git |
ENV_GIT_BRANCH | master |
ENV_GIT_INTERVAL | 1m |
For more information on the configuration under Datakit's Kubernetes environment, see this document.
FAQ¶
Error: Authentication Required¶
This error may be reported in the following situations.
If SSH is used:
- The key provided is wrong;
If you are using HTTP:
- The user name and password provided are wrong;
- The protocol of git address is incorrect;
For example, if the original address is https://username:password@github.com/path/to/repository.git
, and then it is written as http://username:password@github.com/path/to/repository.git
, that is, if https
is changed to http
, this error will also be reported.