Offline Deployment¶
In some cases, the target machine does not have a public access exit, so you can install DataKit offline as follows.
Agent Installation¶
If there is a machine in the intranet that can access the external, a proxy can be deployed at the node to proxy the access traffic of the intranet machine through the machine.
At present, DataKit has a inner proxy collector; The same goal can also be achieved through Nginx forward proxy function. The basic network structure is as follows:
Preconditions¶
- Install a DataKit on a machine with a public network exit in the normal installation mode, and turn on the proxy collector on the DataKit, assuming that the proxy collector is located in Datakit IP 1.2. 3.4, with the following configuration:
- Or Nginx ready to configure the forward proxy
- Use the DataKit proxy
Add the environment variable HTTPS_PROXY="1.2.3.4:9530"
, and the installation command is as follows:
DK_DATAWAY=https://openway.guance.com?token=<TOKEN> HTTPS_PROXY=http://1.2.3.4:9530 bash -c 'eval "$(curl -L https://static.guance.com/datakit/install.sh)"'
- Using the Nginx proxy
Add the environment variable DK_PROXY_TYPE="nginx"; DK_NGINX_IP="1.2.3.4";
, and the installation command is as follows:
- Using the DataKit proxy
Add the environment variable $env:HTTPS_PROXY="1.2.3.4:9530"
, and the installation command is as follows:
Remove-Item -ErrorAction SilentlyContinue Env:DK_*;
$env:DK_DATAWAY="https://openway.guance.com?token=<TOKEN>";
$env:HTTPS_PROXY="1.2.3.4:9530";
Set-ExecutionPolicy Bypass -scope Process -Force;
Import-Module bitstransfer;
start-bitstransfer -ProxyUsage Override -ProxyList $env:HTTPS_PROXY -source https://static.guance.com/datakit/install.ps1 -destination .install.ps1;
powershell ./.install.ps1;
- Using the Nginx proxy
Add the environment variable $env:DK_PROXY_TYPE="nginx"; $env:DK_NGINX_IP="1.2.3.4";
, and the installation command is as follows:
Remove-Item -ErrorAction SilentlyContinue Env:DK_*;
$env:DK_DATAWAY="https://openway.guance.com?token=<TOKEN>";
$env:DK_NGINX_IP="1.2.3.4";
$env:DK_PROXY_TYPE="nginx";
Set-ExecutionPolicy Bypass -scope Process -Force;
Import-Module bitstransfer;
start-bitstransfer -ProxyUsage Override -ProxyList $env:DK_NGINX_IP -source https://static.guance.com/datakit/install.ps1 -destination .install.ps1;
powershell ./.install.ps1;
Note: Other setup parameter settings are the same as normal setup.
Full Offline Installation¶
When there is no external network in the environment, the installation package can only be downloaded from the public network to the internal network by moving the hard disk (U disk).
There are two strategies to choose from for full offline installation:
- Simple mode: Directly copy the files in the U disk to each host and install DataKit. However, Simple Mode currently does not support setting through environment variables during installation.
- Advanced mode: Deploy a Nginx on the intranet and build a file server with Nginx instead of static.guance.com.
Simple Mode¶
The address of the following files can be downloaded through wget and other download tools, or directly enter the corresponding URL to download in the browser.
Attention
When downloading from Safari browser, the suffix name may be different (for example, downloading the . tar.gz
file to . tar
), which will cause the installation to fail. It is recommended to download with Chrome browser.
-
Download the packet data.tar.gz first, which is the same for every platform.
-
Then download more installers as below:
After downloading, you should have a few files as below (<OS-ARCH>
here refers to the platform-specific installation package):
datakit-<OS-ARCH>.tar.gz
dk_upgrader-<OS-ARCH>.tar.gz
installer-<OS-ARCH>
orinstaller-<OS-ARCH>.exe
data.tar.gz
Copy these files to the corresponding machine (via USB flash drive or scp
and other commands).
Attention
It is crucial to download each of these files completely. They may not be reusable between different versions. For example, the installer program behaves differently across various Datakit versions because it may adjust the default configurations of Datakit, which can have varying degrees of additions and deletions. It is best to use the installer program corresponding to version 1.2.3 of Datakit for the installation or upgrade of Datakit 1.2.3.
Installation¶
If you are performing an offline install of the lite version of Datakit, you need to specify the installation package with a
_lite
suffix, such asdatakit_lite-linux-amd64-1.63.0.tar.gz
.
To run with root privileges:
Upgrade¶
If you are performing an offline upgrade of the lite version of Datakit, you need to specify the installation package with a
_lite
suffix, such asdatakit_lite-linux-amd64-1.63.0.tar.gz
.
To run with root privileges:
How to Specify More Configuration Parameters for Offline Installation
During online installation, we can specify some default parameters through environment variables DK_XXX=YYY
. These default parameters actually take effect through the install.sh script (on Windows, it's install.ps1). However, these environment variables are ineffective for the installation program installer-xxx. We can only use the command-line arguments of installer-xxx to add these options. By using the following command, we can find out the parameters supported by the installation program:
For example, the Dataway address we specified above is set through the --dataway
option. Additionally, these extra command-line parameter settings are only effective in installation mode and do not take effect in (offline) upgrade mode.
Advanced Mode¶
DataKit is currently installed on the public web, and all binary data and installation scripts are downloaded from the static.guance.com site. For machines that cannot access the site, you can replace the static.guance.com site by deploying a file server on the intranet.
The network traffic topology of advanced mode is as follows:
Prepare a machine that can be accessed on the intranet, install Nginx on the machine, and download (or copy) the files required for DataKit installation to the Nginx server, so that other machines can download the installation files from the Nginx file server to complete the installation.
- Setting up the Nginx file server {#nginx-config}
Add configuration in nginx.conf
server {
listen 8080;
server_name _;
## Map to the following directory
location / {
root /;
autoindex on;
autoindex_exact_size off;
autoindex_localtime on;
charset utf-8,gbk;
}
}
Restart Nginx:
- Download the files to the /datakit directory where the Nginx server is located, taking wget downloading the Linux AMD64 platform installation package as an example:
#!/bin/bash
mkdir -p /datakit
wget -P /datakit https://static.guance.com/datakit/install.sh
wget -P /datakit https://static.guance.com/datakit/version
wget -P /datakit https://static.guance.com/datakit/data.tar.gz
wget -P /datakit https://static.guance.com/datakit/installer-linux-amd64-1.63.0
wget -P /datakit https://static.guance.com/datakit/datakit-linux-amd64-1.63.0.tar.gz
wget -P /datakit https://static.guance.com/datakit/datakit_lite-linux-amd64-1.63.0.tar.gz
wget -P /datakit https://static.guance.com/datakit/dk_upgrader-linux-amd64.tar.gz
# Download other toolkits: sources is the installation package used to turn on the RUM sourcemap function. If this function is not turned on, you can choose not to download it.
sources=(
"/datakit/sourcemap/jdk/OpenJDK11U-jdk_x64_mac_hotspot_11.0.16_8.tar.gz"
"/datakit/sourcemap/jdk/OpenJDK11U-jdk_aarch64_mac_hotspot_11.0.15_10.tar.gz"
"/datakit/sourcemap/jdk/OpenJDK11U-jdk_x64_linux_hotspot_11.0.16_8.tar.gz"
"/datakit/sourcemap/jdk/OpenJDK11U-jdk_aarch64_linux_hotspot_11.0.16_8.tar.gz"
"/datakit/sourcemap/R8/commandlinetools-mac-8512546_simplified.tar.gz"
"/datakit/sourcemap/R8/commandlinetools-linux-8512546_simplified.tar.gz"
"/datakit/sourcemap/proguard/proguard-7.2.2.tar.gz"
"/datakit/sourcemap/ndk/android-ndk-r22b-x64-mac-simplified.tar.gz"
"/datakit/sourcemap/ndk/android-ndk-r25-x64-linux-simplified.tar.gz"
"/datakit/sourcemap/atosl/atosl-darwin-x64"
"/datakit/sourcemap/atosl/atosl-darwin-arm64"
"/datakit/sourcemap/atosl/atosl-linux-x64"
"/datakit/sourcemap/atosl/atosl-linux-arm64"
)
mkdir -p /datakit/sourcemap/jdk \
/datakit/sourcemap/R8 \
/datakit/sourcemap/proguard \
/datakit/sourcemap/ndk \
/datakit/sourcemap/atosl
for((i=0;i<${#sources[@]};i++)); do
wget https://static.guance.com${sources[$i]} -O ${sources[$i]}
done
Attention
You must append suffix .exe to the download link of Installer
on Windows, for example: https://static.guance.com/datakit/installer-windows-386-1.63.0.exe for Windows 32bit and
https://static.guance.com/datakit/installer-windows-amd64-1.63.0.exe for Windows 64bit.
Install¶
On the intranet machine, point it to the Nginx file server by setting DK_INSTALLER_BASE_URL
:
Remove-Item -ErrorAction SilentlyContinue Env:DK_*;
$env:DK_DATAWAY="https://openway.guance.com?token=<TOKEN>";
$env:DK_INSTALLER_BASE_URL="http://[Nginx-Server]:8080/datakit";
$env:HTTPS_PROXY="1.2.3.4:9530";
Set-ExecutionPolicy Bypass -scope Process -Force;
Import-Module bitstransfer;
start-bitstransfer -source ${DK_INSTALLER_BASE_URL}/install.ps1 -destination .install.ps1;
powershell ./.install.ps1;
So far, the offline installation is complete. Note that HTTPS_PROXY is additionally set here.
Upgrade¶
If there is a new version of DataKit, you can download it as above and execute the following command to upgrade:
$env:DK_INSTALLER_BASE_URL="http://<nginxServer>:8080/datakit";
Remove-Item -ErrorAction SilentlyContinue Env:DK_*;
$env:DK_UPGRADE="1";
Set-ExecutionPolicy Bypass -scope Process -Force;
Import-Module bitstransfer;
start-bitstransfer -source ${DK_INSTALLER_BASE_URL}/install.ps1 -destination .install.ps1;
powershell ./.install.ps1;
Kubernetes Offline Deployment¶
Bash Script Assisted Installation¶
Here is a simple script to help you complete the tasks of password free login, file distribution and image decompression.
datakit_tools.sh
(Stand-alone open)
#!/bin/bash
# Please modify the host IP to be password-free
host_ip=(
10.200.14.112
10.200.14.113
10.200.14.114
)
# Please change the login password
psd='123.com'
menu() {
echo -e "\e[33m------Please select the required operation------\e[0m"
echo -e "\e[33m1. Set SSH remote keyless login\e[0m"
echo -e "\e[33m2. Scp remote transfer file\e[0m"
echo -e "\e[33m3. Remote decompression image\e[0m"
read -p "Please enter an option:" num
}
# Send ssh-copy-id to the host
SSH-COPY(){
yum install -y expect
ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
for i in ${host_ip[@]}
do
/usr/bin/expect<<EOF
spawn ssh-copy-id ${i}
expect {
"(yes/no)" {send "yes\r";exp_continue}
"password" {send "${psd}\r"}
}
expect eof
EOF
done
}
SCP(){
read -p "Please enter the file name to transfer (multiple files can be passed in): " file_name
for i in ${host_ip[@]}
do
for j in ${file_name[@]}
do
echo -e "\e[33m------${i}---${j}------\e[0m"
scp ${j} root@${i}:/root/
done
done
}
SSH(){
read -p "Please enter the file name to extract: " file_name
# Remotely unzip image packets
for i in ${host_ip[@]}
do
echo -e "\e[33m------${i}------\e[0m"
# ssh root@${i} "docker load -i ${file_name}"
ssh root@${i} " ctr -n=k8s.io image import ${file_name} "
done
}
#menu
CASE(){
case ${num} in
1)
SSH-COPY
echo -e "\e[33m------------------------------------------------------\e[0m"
;;
2)
SCP
echo -e "\e[33m------------------------------------------------------\e[0m"
;;
3)
SSH
echo -e "\e[33m------------------------------------------------------\e[0m"
;;
*)
echo -e "\e[31mPlease enter the number in the option{1|2|3}\e[0m"
esac
}
menu
CASE
read -p "Do you want to continue with the list operation? [y/n]:" a
while [ "${a}" == "y" ]
do
menu
CASE
read -p "Do you want to continue with the list operation? [y/n]:" a
continue
done
# You need to modify the host IP and login password in the script, and then complete the operation according to the guidance.
chmod +x datakit_tools.sh
./datakit_tools.sh
Agent Installation¶
If there is a machine in the intranet that can connect to the internet, you can deploy a nginx server on this node to use as the image acquisition.
- Download
datakit.yaml
and DataKit image files
- Download the DataKit image and make it into a package
# Pull the image of the amd64 architecture and make it into an image package
docker pull --platform amd64 pubrepo.guance.com/datakit/datakit:1.63.0
docker save -o datakit-amd64-1.63.0.tar pubrepo.guance.com/datakit/datakit:1.63.0
mv datakit-amd64-1.63.0.tar /home/guance
# Pull the image of the arm64 architecture and make it into an image package
docker pull --platform arm64 pubrepo.guance.com/datakit/datakit:1.63.0
docker save -o datakit-arm64-1.63.0.tar pubrepo.guance.com/datakit/datakit:1.63.0
mv datakit-arm64-1.63.0.tar /home/guance
# Check whether the image architecture is correct
docker image inspect pubrepo.guance.com/datakit/datakit:1.63.0 |grep Architecture
- Modify Nginx configuration agent
/etc/nginx/nginx.conf
#user nobody;
worker_processes 1;
#error_log logs/error.log;
#error_log logs/error.log notice;
#error_log logs/error.log info;
#pid logs/nginx.pid;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
#log_format main '$remote_addr - $remote_user [$time_local] "$request" '
# '$status $body_bytes_sent "$http_referer" '
# '"$http_user_agent" "$http_x_forwarded_for"';
#access_log logs/access.log main;
sendfile on;
#tcp_nopush on;
#keepalive_timeout 0;
keepalive_timeout 65;
#gzip on;
server {
listen 8080;
server_name localhost;
root /home/guance;
autoindex on;
#charset koi8-r;
#access_log logs/host.access.log main;
#error_page 404 /404.html;
# redirect server error pages to the static page /50x.html
#
# proxy the PHP scripts to Apache listening on 127.0.0.1:80
#
#location ~ \.php$ {
# proxy_pass http://127.0.0.1;
#}
# pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
#
#location ~ \.php$ {
# root html;
# fastcgi_pass 127.0.0.1:9000;
# fastcgi_index index.php;
# fastcgi_param SCRIPT_FILENAME /scripts$fastcgi_script_name;
# include fastcgi_params;
#}
# deny access to .htaccess files, if Apache's document root
# concurs with nginx's one
#
#location ~ /\.ht {
# deny all;
#}
}
# another virtual host using mix of IP-, name-, and port-based configuration
#
#server {
# listen 8000;
# listen somename:8080;
# server_name somename alias another.alias;
# location / {
# root html;
# index index.html index.htm;
# }
#}
# HTTPS server
#
#server {
# listen 443 ssl;
# server_name localhost;
# ssl_certificate cert.pem;
# ssl_certificate_key cert.key;
# ssl_session_cache shared:SSL:1m;
# ssl_session_timeout 5m;
# ssl_ciphers HIGH:!aNULL:!MD5;
# ssl_prefer_server_ciphers on;
# location / {
# root html;
# index index.html index.htm;
# }
#}
}
- Other intranet machines execute commands.
wget http://<nginx-server-ip>:8080/datakit.yaml
wget http://<nginx-server-ip>:8080/datakit-amd64-1.63.0.tar
- Unzip image command
# docker
docker load -i /k8sdata/datakit/datakit-amd64-1.63.0.tar
# containerd
ctr -n=k8s.io image import /k8sdata/datakit/datakit-amd64-1.63.0.tar
- Start DataKit container
Full Offline Installation¶
When there is no external network in the environment, the installation package needs be downloaded from the public network to the internal network through mobile hard disk (U disk).
- Unzip image command
# docker
docker load -i datakit-amd64-1.63.0.tar
# containerd
ctr -n=k8s.io image import datakit-amd64-1.63.0.tar
- The cluster controller executes the start command
- Update command