Hybrid Cloud Deployment Plan¶
Overview¶
“Guance” offers two versions: a SaaS service version and a private deployment version.
The SaaS service version provides fast and easy access. The central platform handles all services, and customers only need to install DataKit on their side and enable the required data collection to use all monitoring features of “Guance”. For customers with larger application cluster monitoring needs or those whose data cannot be easily sent out of the internal network, “Guance” offers a private deployment version.
“Guance” Deployment Implementation¶
The deployment implementation of “Guance” consists of three parts: 1. DataKit deployment and data collection implementation on the monitored objects; 2. Implementation of scenarios, views, anomaly detection, etc., within “Guance”; 3. If using the private deployment version of “Guance”, an additional independent deployment of “Guance” is required.
After installing DataKit on the host objects of the monitored user system and enabling the corresponding data collection, the data must be reported to the “Guance” center for visual insights, correlated analysis of metrics, logs, traces, RUM, etc., and anomaly detection. The “Guance” center can choose to use the SaaS version or the private deployment version.
SaaS Version of “Guance”¶
The SaaS version features low access and usage costs, and fast implementation (only requiring installation and enabling of DataKit in the customer's application environment). It is suitable for monitoring small to medium-sized application clusters. Its characteristics include low initial investment, pay-as-you-go billing, and no upfront resource investment.
For the specific implementation of DataKit, refer to the "DataKit Deployment Plan" section later.
Private Deployment Version of “Guance”¶
For customers with larger scales or higher requirements for system and data control, the private deployment version can be used, providing exclusive resource usage. This approach has a higher initial investment, requiring an upfront resource procurement fee. For example, deploying on Alibaba Cloud incurs an annual resource consumption cost of at least 200,000 to 300,000 RMB (depending on the customer's total data scale).
Private Deployment Architecture¶
For the private deployment version of “Guance”, we are responsible for installation, deployment, and subsequent maintenance and upgrades for the customer. To ensure the security of the customer's application system, the network planning of IT resources should isolate the “Guance” from the monitored user application service cluster. Only one-way access from the monitored objects to the “Guance” resources should be allowed. This can be achieved using VPC isolation, security group isolation, or even deploying “Guance” in a separate cloud account, effectively safeguarding the data security of the user's system.
The “Guance” center supports deployment on various clouds as well as on-premises hosts. Due to differences in cloud products supported by different clouds and system differences in various customer environments, the entire system adopts a cloud-native microservice architecture, using Kubernetes as the deployment foundation to ensure consistency of the runtime environment and eliminate system differences.
Architecturally, all data such as hosts, application logs, and traces from user systems are collected by DataKit and reported to the “Guance” center via DataWay. If the data volume is huge and a single DataWay cannot handle it, DataWay supports clustered deployment for horizontal scaling.
To ensure balanced and non-blocking data reporting from clients, the “Guance” center uses the NSQ message queue (a decentralized, open-source messaging system with automatic node registration and discovery). All data first enters the NSQ message queue, and Kodo then consumes the data in order. After processing, time-series metrics data is written to InfluxDB, and log text data is written to Elasticsearch.
Deployment on the Cloud¶
For the deployment of “Guance”, cloud deployment is the preferred recommendation. All required hardware resources should preferably use cloud products supported by the cloud provider, as these products inherently ensure the reliability, stability, and disaster recovery of the infrastructure.
Taking Alibaba Cloud as an example, Kubernetes uses Alibaba Cloud's Container Service for Kubernetes (ACK), InfluxDB uses Alibaba Cloud's Time Series InfluxDB Edition, Elasticsearch uses Alibaba Cloud's big data cloud product Elasticsearch, MySQL uses Alibaba Cloud's ApsaraDB RDS for MySQL, and Redis uses ApsaraDB for Redis.
Deployment on On-Premises Hosts¶
“Guance” also supports deployment on on-premises hosts. All basic components need to be set up independently. The deployment of the “Guance” center must also be based on Kubernetes.
First, a Kubernetes cluster must be deployed on the on-premises hosts. All basic components such as InfluxDB, Elasticsearch, MySQL, and Redis can be deployed as containers within the Kubernetes cluster. To improve system stability, it is recommended to deploy these components in a separate container cluster from the “Guance” deployment.
DataKit Deployment Plan¶
The deployment and implementation of DataKit are independent of the architecture of the “Guance” center. The target workspace for data ingestion, whether connecting to the SaaS version of “Guance” or the private deployment version, is identical.
In other words, once DataKit is implemented, it can seamlessly switch the target “Guance” center by simply changing the data gateway address.
DataKit Basic Architecture¶
DataKit supports three major platforms: Linux, macOS, and Windows. After installing DataKit on the monitored host, basic host metrics collection is enabled by default, such as CPU, Mem, Disk, DiskIO, and System. Additional data collection can be enabled. For specific configuration methods and a list of collection sources, refer to the document Collection Source Configuration.
After installation, DataKit opens port 9529 by default (the default listening port can be modified in the datakit.conf file) as an HTTP data ingestion service port. Data collected by various external collection sources (Metric, Log, Tracing, RUM, Security, etc.) is sent to DataKit via the HTTP interface on port 9529. After data cleansing and formatting, it is reported to the “Guance” center.
Distributed Tracing Deployment¶
DataKit supports data ingestion from open-source tracing frameworks such as Skywalking, Jaeger, Zipkin, and ddTrace. The output address of the tracing framework agent should be configured to the DataKit tracing routing address.
The best deployment practice is to deploy DataKit on each application server to better unify and perform correlated analysis of server host metrics, application logs, syslog, and application service tracing data.
Real User Monitoring (RUM) Deployment¶
For information on how to ingest Real User Monitoring (RUM) data, refer to the document Real User Monitoring.
The deployment method requires deploying a DataKit (RUM DataKit) on the user's application server side. The application client introduces the corresponding RUM SDK for each platform. User access data, such as page requests, resource requests, and JS error information from the client, is preprocessed and formatted by the RUM DataKit before being uploaded to “Guance”.
Kubernetes Deployment¶
In the era of cloud-native microservices architecture, a large number of applications are adopting microservices. Kubernetes, as a cloud-native container orchestration and scheduling tool, is increasingly used. DataKit also supports containerized deployment.
DaemonSet Method¶
Deploying DataKit in Kubernetes using the DaemonSet method has two main advantages:
- Simple deployment: regardless of the size of the service cluster, there is no need to deploy individually on each node.
- Combined with the high scalability of Kubernetes clusters, DataKit can be automatically started on newly scaled-out nodes without manual intervention.
Cluster Deployment Without Public Network Access¶
In some server-side scenarios, not all service hosts can access the public network. For hosts without public network access, the DataKit cannot directly report data to “Guance”.
A separate host with public network access capability can be set up as a Proxy within the cluster. All DataKit instances in the cluster report data to “Guance” through this Proxy.
DataKit itself supports proxy cascading. You can install DataKit on this public network host, enable the Proxy function, and configure the data addresses of all DataKit instances in the cluster to point to this Proxy DataKit. The monitoring data of internal network hosts is then reported to the “Guance” center through this DataKit with public network access.
Additionally, other proxy solutions such as Nginx, HAProxy, or H5 can be used to route internal network data through a proxy to the public network.