Logs¶

In modern IT infrastructure, systems can generate thousands of log events per minute. These logs follow specific formats, usually contain timestamp information, and are output by servers to different files such as system logs, application logs, and security logs. Since logs are scattered across various server nodes, when a system failure occurs, operations personnel need to log into multiple servers separately to review logs, a process that significantly increases the complexity and time cost of troubleshooting.

Faced with massive log data, teams often encounter data management challenges:

Which logs should be sent to the log management platform in real time?

Which can be archived?

If filtering is performed during the data collection phase, critical failure information might be missed or valuable data could be accidentally deleted, creating hidden risks for subsequent problem investigation.

To address these challenges, building a centralized log management platform is essential. Through powerful log collection capabilities, log data from distributed environments can be uniformly reported to the workspace, enabling centralized storage, audit monitoring, intelligent alerts, and in-depth analysis of logs. This approach avoids the risk of data loss that may result from pre-filtering, while also significantly improving fault diagnosis efficiency through a unified search interface and correlation analysis functions.

Guance's log functionality is designed based on this concept. It transforms originally isolated log data into a "connector" that runs through the entire observability system, enabling operations teams to proactively grasp the system status and quickly locate the root cause of issues in emergencies, thereby achieving a shift from passive response to proactive prevention in operations models.

Getting Started¶

Collection and Integration¶

Guance provides flexible log collection solutions through the DataKit collector. You can choose the appropriate method based on your environment:

• Host Log Collection: After installing DataKit on the server, specify the log file path through the collector configuration to collect text-format log files.

• K8s Environment Collection: Deploy DataKit as a DaemonSet in a Kubernetes cluster to automatically collect logs from container standard output (stdout/stderr).

• Receive Third-Party Logs: Supports receiving log data from tools such as Fluentd, Logstash, and Kafka via HTTP/S or TCP protocols, compatible with existing technology stacks.

Processing and Parsing¶

Guance provides complete log processing capabilities through Pipeline:

• Structured Parsing: Use Grok patterns and regular expressions to extract key fields such as status codes and timestamps from raw log text.

• Data Standardization: Convert unstructured log text into a unified standardized format, laying the foundation for subsequent analysis and querying.

Query and Analysis¶

Guance provides powerful query and analysis functions through the Log Explorer:

• Supports precise filtering queries and data retrieval based on fields, enabling quick location of target logs.

• Intelligent Analysis: Automatically identify log patterns through pattern analysis and display data trends through visual charts.

• Troubleshooting: Supports viewing log context information and performing correlation analysis with corresponding traces and infrastructure metrics.

• Team Collaboration: Provides data snapshot functionality, supporting secure team collaboration and knowledge sharing.

Monitoring and Alerting¶

Guance provides intelligent log monitoring and alerting capabilities:

• Intelligent Monitoring and Alerting: Create monitors based on log data to achieve real-time detection of anomalies and second-level alert notifications, ensuring issues are discovered and handled promptly.

• Fine-Grained Cost Governance: For massive log scenarios, supports filtering invalid logs at the collection end through blacklists, and optimizing storage costs at the storage end through multi-index and tiered storage policies.

Storage and Archiving¶

Guance provides a comprehensive log storage management solution:

• Permissions and Security: Through data access functionality, different roles can be configured with the scope of log data they are authorized to access. Meanwhile, using sensitive data scanning functionality, sensitive information in logs (such as ID numbers, keys) can be proactively discovered and handled to ensure data compliance.

• Data Forwarding and Archiving: Supports long-term archiving of log data to Guance object storage, or real-time forwarding to external storage systems to meet data backup, auditing, or further processing needs.