Update Log (2022)¶

This document records the update content of Guance for each online release, including DataKit, Guance best practices, Guance integration documents, and Guance.

December 29, 2022¶

Guance Update¶

Added MFA Authentication Management¶

Guance provides MFA authentication management, which helps add an extra layer of security protection beyond account usernames and passwords. After enabling MFA authentication, you will need to perform a second verification during login, helping you enhance your account's security. For more details, refer to the documentation MFA Management.

Added Workspace IP Access Login Whitelist¶

Guance supports configuring an IP whitelist for workspaces to restrict visitor users. After enabling the IP whitelist, only sources from the whitelist IPs can log in normally, and all other requests will be denied access. For more details, refer to the documentation IP Whitelist.

Note: The IP whitelist is only configurable by administrators and owners. Simultaneously, "owners" are not restricted by the IP whitelist access limits.

Added Service Network Topology¶

In K8S environments, you can use the Service network topology diagram to view request relationships, response times, error rates, etc., between different Services within the K8S environment. When a Service has connection issues, you can click to view the corresponding logs of that Service to locate its problems. For more details, refer to the documentation Service Network.

Other Functionality Optimization¶

• Pipeline usage optimization
• Log clustering analysis supports custom fields
• Ultra-large logs after slicing support viewing associated logs
• Backup log display optimization
• Sensitive field data authorization supports all types of data except metrics

DataKit Update¶

• Prometheus collector supports collecting data via Unix Socket

• Allows running DataKit as a non-root user

• Optimized httpflow protocol judgment in eBPF

• Optimized Datakit installation and upgrade commands on Windows
• Optimized Pythond encapsulation
• Pipeline provides more detailed operation error information
• Pipeline reftable provides a localized storage implementation based on SQLite

For more DataKit updates, refer to DataKit Version History.

Intelligent Inspection Update¶

Official Intelligent Inspection Discontinued¶

The official intelligent inspections for memory leaks, application performance detection, and disk usage have been discontinued starting today. Users can configure self-built inspection scripts through Self-built Inspections.

Added Cloud Account Instance Dimension Billing Inspection¶

Cloud account instance dimension billing inspection helps users manage abnormal fee warnings at the cloud service instance level, predict fee situations, and provide high-growth and high-consumption instance prompts and bill visualization capabilities, supporting multi-dimensional visualization of cloud service resource consumption.

Added AliCloud Preemptible Instance Survival Inspection¶

Due to the fluctuating market price of preemptible instances, it is necessary to specify a bidding model when creating preemptible instances. Preemptible instances can only be successfully created if the real-time market price for the specified instance specification is lower than the bid and inventory is sufficient. Therefore, inspecting preemptible instances is particularly important for cloud assets. Through inspections, when it is detected that a preemptible instance is about to be released, it will prompt the latest prices of all available zones for the current specification of preemptible instances, along with historical prices of the preemptible instance and appropriate handling recommendations.

For more intelligent inspection updates, refer to Intelligent Inspection Update Log.

December 15, 2022¶

Guance Update¶

Added Profile and Network Billing¶

Guance added Profile and Network billing items. Profile counts the number of application performance Profiles reported on the day in the workspace, and Network counts the number of hosts (host) reporting network data on the day in the workspace. For more details, refer to the documentation Billing Method.

Guance Registration Process Optimization¶

Guance simplified the registration process, helping you quickly complete registration and start using Guance. For more details, refer to the documentation Register Guance Commercial Plan.

Added Warroom Feature¶

Guance provides the "Warroom" dashboard, where you can oversee key metric trends across all workspaces, helping you quickly understand the critical data situation of each workspace and promptly identify and resolve abnormal issues. For more details, refer to the documentation Key Metric Management.

Added Dashboard Carousel Function¶

Guance provides a carousel function for multiple related business dashboards. After setting it up, you can display them on large screens. For more details, refer to the documentation Carousel Dashboard.

Workspace Binding to External SLS Index¶

Guance newly supports binding SLS indexes, helping you quickly view and analyze your log data. Additionally, Guance adds support for field mapping functions, assisting you in performing cluster analysis by mapping SLS index fields. For more details, refer to the documentation Log Index.

Trace Viewer Chart Display Optimization¶

In the trace viewer, new trend charts for "Span Quantity" and "Error Span Quantity" have been added. You can quickly view the trend indicators for Span quantity, Error Span quantity, request quantity, Error request quantity, and response time through the viewer chart.

Other Functionality Optimization¶

• View variable interaction optimization
• Interaction optimization for built-in view configuration binding
• Workflow optimization for workspace arrears lockout
• User experience optimization for workspace invitations
• Help documentation added an evaluation system, including better and worse reviews
• Monitoring batch functionality optimization
• SLO added enable/disable functionality
• Fuzzy match Wildcard left * match query limit optimization

DataKit Update¶

• Added Golang Profiling integration
• logfwd supports injecting image fields via LOGFWD_TARGET_CONTAINER_IMAGE
• Trace collector:
  • Optimized error-stack/error-message format issues
  • Adjusted SkyWalking compatibility to support the full 8.X series
• eBPF httpflow added pid/process_name fields, optimized kernel version support
• datakit.yaml adjustments recommended, update yaml suggested
• GPU card collection supports remote mode

For more DataKit updates, refer to DataKit Version History.

Intelligent Inspection Update¶

Added Intelligent Inspection Guide Page¶

Guance default supports three intelligent inspection templates: memory leaks, disk usage, and application performance detection. After this update, existing workspaces support manually deleting the default intelligent inspection templates, while simultaneously prompting the system-wide offline time; newly created workspaces no longer default provide these three intelligent inspections, adding a beginner guide tip.

Intelligent Inspection Components Support Configurable Custom Jump Links¶

When configuring self-built inspections, you can set custom jump links. Refer to the documentation Self-built Inspection for how to configure self-built inspections.

For more intelligent inspection updates, refer to Intelligent Inspection Update Log.

Best Practices Update¶

• Insight
  • Scenario (Scene) - SpringBoot Project with External Tomcat Scene Link Observability.

For more best practice updates, refer to Best Practice Version History.

December 1, 2022¶

Guance Update¶

Added Application Performance New Service List Function¶

Application Performance Monitoring Service List supports real-time viewing of ownership, dependencies, performance, associated dashboards, and correlation analysis for different services, quickly identifying and resolving service performance issues, helping teams efficiently build and manage large-scale end-to-end distributed applications. For more details, refer to the documentation Service List.

Dashboard Chart Usage Experience Optimization¶

• Added manual input for chart units, colors, and aliases. You can customize preset units, colors, and aliases according to current standards, such as unit input formats being aggregation functions(metrics), like last(usage_idle)
• Added Y-axis configuration for time-series charts, allowing you to manually configure the maximum and minimum values of the Y-axis
• Added grouping display for time-series charts and pie charts. After enabling, only label values are displayed in the legend
• Added gradient intervals for hexagonal charts, China maps, and world maps, including automatic and custom options
• Column chart and histogram display optimization

For more details, refer to the documentation Visualization Charts.

View Variable Usage Experience Optimization¶

• Added support for multi-selection in view variables
• Added whether to set multiple default values when configuring view variables

For more details, refer to the documentation View Variables.

User Access Monitoring Session Viewer Adjustment¶

In the User Access Monitoring Session Viewer, added "Session" and "All Records" viewing lists:

• Session: Counts deduplicated Session (session) data within the current time range, displaying the latest Session (session) data by default;
• All Records: Displays all Session (session) reporting data, where the same Session (session) may correspond to multiple records.

Note: In the Session session list, if the user-added display column does not exist in the current Session data, there may be blank data in the session list. For more details, refer to the documentation Session (Session).

Events Added Mobile Jump Options¶

After receiving alert event notifications, clicking "Go to Guance View" takes you to the Guance mobile app to view detailed information about abnormal events.

Other Functionality Optimization¶

• Beginner guide page optimization, supports minimization
• User access monitoring application list optimization, added customizable interval switching queries for the time widget
• Host Label interaction optimization

DataKit Update¶

• Added Python Profiling integration
• Pythond added custom event reporting functionality

• netstat supports specific port metric collection

• Optimized Redis collector CPU usage collection, added new metric fields

• Optimized logfwd collector configuration
• Fully collected host object fields, added network, disk-related fields

For more DataKit updates, refer to DataKit Version History.

Intelligent Inspection Update¶

• Kubernetes Pod Abnormal Restart Inspection
• MySQL Performance Inspection
• Server-side Application Error Inspection
• Memory Leak Inspection
• Disk Usage Rate Inspection
• Application Performance Inspection
• Frontend Application Log Error Inspection

For more intelligent inspection updates, refer to Intelligent Inspection Update Log.

Best Practices Update¶

• Monitoring Monitoring
  • Application Performance Monitoring (APM) - Performance Optimization - Utilizing async-profiler to Optimize Application Performance

For more best practice updates, refer to Best Practice Version History.

November 17, 2022¶

Guance Update¶

Pipeline Usage Experience Optimization¶

• Pipeline supports multi-select filtering condition configuration

• Supports setting any one Pipeline script as the "Default Pipeline Script". If the current data type does not match other Pipeline scripts when matching Pipeline processing, the data will be processed according to the rules of the default Pipeline script.

• Categorized script functions

Event Optimization¶

• Supports writing user-defined events, for more details refer to the documentation Write Event Data via openAPI.

• Added quick filters on the left side of the unresolved events viewer

Custom Viewer Supports Selecting More Types of Data¶

When creating a new viewer in scenarios, supports selecting log, application performance, user access, security inspection, Profile these five types of data, cannot change after saving.

Member Management Added Member Grouping Function¶

Added member grouping function in member management, you can edit member groups, one member can support being in multiple groups; while configuring notification objects for alert strategies, supports configuring group tags for members.

Optimized Built-in View Binding Function¶

In the scenario "Built-in Views" - "User Views", added support for fuzzy matching when binding fields, after field binding, you can associate and view corresponding built-in views in the viewer.

Added One-click Import/Export Dashboards, Custom Viewers, Monitors Within Workspaces¶

In "Management" - "Basic Settings", added one-click import/export JSON configurations for dashboards, custom viewers, monitors, etc., within workspaces. You can import dashboards, custom viewers, monitors, etc., from other workspaces into the current workspace.

Application Performance Monitoring Log Page Added Custom Associated Fields¶

In the details page of the Application Performance Monitoring, in the associated log page, if you have administrator-level or higher permissions, you can customize associated fields. You can click the settings button to the right of the associated fields, select the fields you need to associate in the pop-up dialog box, support manual input, drag-and-drop order adjustment, confirm to complete the configuration.

Other Functionality Optimization¶

• When using fuzzy matching and fuzzy unmatching in filter conditions, supports left * matching
• Infrastructure analysis dimension optimization, last reporting time of the host in the host details page
• Supports users quickly logging into their last single sign-on
• Blacklist application performance monitoring added support for filtering "all services"
• Process upgraded to Commercial Plan for users registered at the "China Zone 4 (Guangzhou)" site
• Field descriptions support viewing in quick filters, display columns, etc.

DataKit Update¶

• Added SNMP Collector
• Added IPMI Collector
• Added bulk injection DDTrace-Java tool
• Latest DDTrace-Java SDK enhanced SQL desensitization function
• Remote Pipeline optimization
  • Pipeline supports source mapping relationship configuration, facilitating batch configuration between Pipelines and data sources
  • Pipeline provides function classification information, facilitating remote Pipeline writing
• Optimized Kafka Message Subscription, no longer limited to obtaining SkyWalking related data, also supports speed limits, multi-version coverage, sampling, and load balancing settings
• Alleviated short lifecycle Pod log collection issues by providing additional configuration parameters (ENV_INPUT_CONTAINER_LOGGING_SEARCH_INTERVAL)
• Pure container environment supports label-based configuration for log collection within containers
• Added Pipeline functions
  • sample(): Sampling function
  • b64enc(): Base64 encoding function
  • b64dec(): Base64 decoding function
  • append(): List append function

For more DataKit updates, refer to DataKit Version History.

Best Practices Update¶

• Insight

  • Guance Tips (Skills) - DataKit HTTPS Configuration

• Monitoring Monitoring

  • Application Performance Monitoring (APM) - Common Parameter Usage for ddtrace
  • Application Performance Monitoring (APM) - ddtrace-api Usage Guide
  • Application Performance Monitoring (APM) - ddtrace Sampling
  • Application Performance Monitoring (APM) - ddtrace Log Association

For more best practice updates, refer to Best Practice Version History.

November 11, 2022¶

Guance Added "China Zone 4 (Guangzhou)" Site¶

Guance supports multi-site registration and login, adding the "China Zone 4 (Guangzhou)" site, combining the existing "China Zone 1 (Hangzhou)", "China Zone 2 (Ningxia)", "Overseas Zone 1 (Oregon)", comprehensively covering different regions domestically and internationally, providing users with comprehensive data observability.

It is important to note that accounts and data between different Guance sites are independent, unable to share or migrate data mutually. You can choose a suitable site for registration and login based on your resource usage situation. Currently, Guance supports the following four sites.

November 3, 2022¶

Guance Update¶

Added Guance and SLS Joint Solution¶

Guance added the SLS storage solution, supporting Alibaba Cloud SLS users to quickly use Guance for data viewing and analysis. During Guance commercial plan registration/upgrade, choosing "Alibaba Cloud Account Settlement" allows you to select the SLS storage solution, storing data under your own Alibaba Cloud account for exclusive data storage. For more details, refer to the documentation SLS Storage.

The joint solution between Guance and SLS supports binding log indices to synchronize data from your Alibaba Cloud account to Guance workspace for querying and analysis. In Guance workspace "Logs" - "Index", click "Bind Index" to synchronize log data from the index. For more details, refer to the documentation Log Index.

Optimized Beginner Guide Page¶

Guance added beginner guides for scenarios, events, infrastructure, metrics, logs, application performance monitoring, user access monitoring, availability monitoring, security inspections, CI visualization, and monitoring of various functional modules, optimizing the welcome homepage of the workspace to make it easier for users to understand Guance and use Guance through the beginner guide.

Added Three Intelligent Inspection Configuration Documents¶

• Cloud Account Billing Inspection: Helps users manage budget warnings for cloud services, abnormal fee warnings, predicted cost situations, and provides visualization capabilities, supporting multi-dimensional visualization of cloud service resource consumption;
• Frontend Application Log Error Inspection: Helps discover new error messages (clustered Error Messages) appearing in frontend applications over the past hour, helping developers and operations personnel promptly fix code, avoiding continuous harm to customer experience over time;
• AliCloud Asset Inspection: Helps users gain more understanding of product performance status from cloud vendors.

Added Trace Error Tracking Viewer¶

In Application Performance Monitoring, added an error tracking viewer to help you quickly view similar errors and their distribution in the trace, quickly locating performance issues. For more details, refer to the documentation Trace Tracking.

Optimized Time Series Graph and Overview Graph Period Comparison Function¶

Period comparison functionality is no longer linked with locked time. By default, the button is visible and turned off. After enabling period comparison, supported dimensions include four options: hour (compared with one hour ago), day (compared with one day ago), week (compared with one week ago), month (compared with one month ago). For more details, refer to the documentation Period Comparison.

Other Functionality Optimization¶

• Added "Save Snapshot" button in the "Settings" section of dashboards/notebooks/viewers;

• Added more options to the time widget;

• Pipeline and blacklist functions added import, batch export, batch delete functions;

• Intelligent inspection added dashboard, bar chart components;
• Integration menu Func page optimization.

DataKit Update¶

• Improved Prometheus ecosystem compatibility, added ServiceMonitor and PodMonitor collection identification

• Added Java Profiling via async-profiler

• eBPF collection added interval parameter, facilitating adjusting the amount of collected data

• All remote collectors default to using their collection address as the value of the host field, avoiding possible misinterpretation of the host field during remote collection
• APM data collected by DDTrace can automatically extract error-related fields, facilitating better APM error tracking in the center
• MySQL collector added extra fields Com_commit/Com_rollback collection

For more DataKit updates, refer to DataKit Version History.

Best Practices Update¶

• Monitoring Monitoring

  • Application Performance Monitoring (APM) - Kafka Observability Best Practices

• Cloud Platform Integration

  • Alibaba Cloud - Alibaba Cloud ACK Integration Guance

For more best practice updates, refer to Best Practice Version History.

October 20, 2022¶

Guance Update¶

Monitoring Optimization¶

Added Outlier Detection¶

Outlier detection detects whether the metrics/statistics of the monitored objects in a specific group deviate significantly using algorithms. If there is inconsistency exceeding a certain degree, an outlier detection anomaly event will be generated for subsequent alert tracking. For more details, refer to the documentation Outlier Detection.

Mutation Detection Logic Optimization¶

The new mutation detection compares the absolute or relative (%) changes of the same metric between two different time periods to determine if an anomaly occurs. It is widely used to track peaks or data changes in a certain metric, generating more precise events when anomalies occur. For more details, refer to the documentation Mutation Detection.

Interval Detection Logic Optimization¶

The new interval detection performs anomaly detection on the selected metric data within the selected detection interval time range. When the proportion of anomalous data points exceeds the set percentage, an interval detection anomaly event is generated. This is mainly applied to monitor stable-trend data/metrics. For more details, refer to the documentation Interval Detection.

Other Optimizations¶

• Adjusted read-only member permissions, supporting viewing configurations for monitors, intelligent inspections, SLOs, mute management, and alert strategies
• Changed the selection for "Trigger Data Gap Events" and "Trigger Recovery Events" from trigger/recovery configuration to time range configuration, supporting manual input

• Supports configuring monitors based on "Event" data

• Alert configuration added "Information" event notification level selection

Scenario Chart Optimization¶

Added Histogram Chart Component¶

A histogram, also known as a quality distribution chart, is used to represent the distribution of data and is a common statistical chart. Generally, the horizontal axis represents the data range, and the vertical axis represents the distribution situation. For more details, refer to the documentation Histogram.

Chart Function Optimization¶

• Overview chart, rectangular tree map, funnel chart added time slice functionality
• Time series chart "View Similar Trend Metrics" adjusted from supporting only metric queries to supporting all data types, including logs, application performance, user access, etc.
• Leaderboard supports viewing all contents exceeding the chart width
• Honeycomb chart display optimization

Viewer Optimization¶

Viewer Supports Analysis Mode¶

In log, application performance monitoring, user access monitoring, availability testing, security inspection, CI viewers, a new "Analysis" function has been added. Infrastructure adjusted "Grouping" to "Analysis," supporting multi-dimensional analysis and statistics based on 1-3 labels, reflecting the distribution characteristics and trends of data under different dimensions and at different times. In analysis mode, multiple data chart analyses are supported, including time series charts, leaderboards, pie charts, and rectangular tree maps. For more details, refer to the documentation Viewer Analysis Mode.

Optimized Related Log Viewing Experience¶

In infrastructure, application performance monitoring, user access monitoring, CI visualization viewer detail pages, optimized related log viewing experience. Default displays log content according to the "maximum display rows" and "display columns" set in the log viewer, supporting customization adjustments via "display columns."

Log Index Optimization¶

After setting the index in "Logs" - "Index," related log queries added an index option, defaulting to index default. You can query and analyze log data based on the index you set, involving functions including log viewers, custom scene viewers, chart log queries, log monitors, etc. For more details, refer to the documentation Log Index.

Optimized Metric Analysis Chart Query¶

Added time intervals and legend options in metric analysis, adjusting the display order of aggregation functions in the legend. For more details, refer to Metric Analysis.

Other Functional Optimization¶

• Added dropdown menu options in the user access monitoring application list to help users quickly switch and view different application data
• Added Pipelines shortcuts in the metrics, user access monitoring, application performance monitoring, infrastructure, security inspection directories

DataKit Update¶

• DataKit collector configuration and Pipeline support synchronization through configuration centers like etcd/Consul
• Prometheus Remote Write Optimization
  • Collection supports filtering tags via regular expressions
  • Supports filtering metric set names via regular expressions
• Pipeline Optimization
  • Optimized grok() and json() functions so they execute trim-space operations by default
  • Added match() function
  • Added cidr() function
• Process collector added detailed fields for open file lists
• Enhanced external data (T/R/L) disk caching and queue processing
• Monitor added overuse warning: At the bottom of the monitor, if the current space usage exceeds the quota, there will be a red text Beyond Usage warning
• Optimized log collection position function, mounting files externally on the host in container environments to avoid losing original position records after DataKit restarts
• Optimized sparse log scenario collection delay issues

More DataKit updates can be found in DataKit Version History.

Best Practices Update¶

• Monitoring Monitoring

  • Infrastructure Monitoring (ITIM) - Ansible Batch Processing Practical Combat.

• Cloud Native

  • Logs - Guance Collecting Amazon ECS Logs.

More best practice updates can be found in Best Practice Version History.

September 29, 2022¶

Guance Update¶

Added Field Management¶

Guance supports unified management of field data in the current workspace, including system fields and custom fields. You can view field descriptions in scenario chart queries, detector metrics in monitors, and simple query modes in DQL queries, helping you quickly understand field meanings and apply them. The figure below shows viewing the host field description in a scenario chart query. For more details, refer to the documentation Field Management.

Optimized Metric Functionality¶

Metric Viewer Transformation¶

The original "Metric Viewer" has been changed to "Metric Analysis," supporting users to visualize query data in time series chart forms (line charts, area charts, bar charts). For more details, refer to Metric Analysis.

Optimized Metric Management¶

• Supports searching "metric sets" and "metrics" in "Metric Management";

• Supports modifying the "unit" and "description" of metrics in "Metric Management";

• Supports viewing tag description information in "Metric Management".

For more details, refer to Metric Management.

Added Export Event Content in PDF Format¶

On the event details page, the "Jump to Monitor" button has added text explanations. Clicking exports both JSON Files and PDF Files, retrieving all key data corresponding to the current event.

Monitor Adjustment¶

• Monitor names do not support customization. The event title input generates the monitor name synchronously;
• Monitors support "info" event generation logic. Trigger conditions can choose to enable or disable information: normal detection results also generate events;
• Monitor configuration steps adjusted to: Detection Configuration - Event Notification - Association.

Note: In the latest version, the "Monitor Name" will be generated synchronously after entering the "Event Title." There might be inconsistencies between "Monitor Names" and "Event Titles" in old monitors. To enhance your experience, please sync to the latest version as soon as possible, supporting one-click replacement of event titles.

Silent Rule Supports Dynamic Configuration¶

In silent management, optimized silent scope, supporting multiple options such as monitors, intelligent inspections, self-built inspections, SLOs, alert strategies, and added tag options.

Note: Silent scope is mandatory, tags are optional. Only when both "silent scope" and "tags" conditions are met simultaneously will the silence take effect. For example, if the silent scope selects application performance detection and disk usage, and the tag selects: host:izbp152ke14timzud0du15z, then alerts for either application performance detection and host:izbp152ke14timzud0du15z or disk usage and host:izbp152ke14timzud0du15z will not send notifications to alert notification recipients.

Optimized Pipeline Configuration Page¶

In text processing (Pipeline), supports adding multiple sample parsing tests, fixing the issue of incorrect one-click sample acquisition for metric data. For more details, refer to Text Processing (Pipeline).

Other Functional Optimization¶

• Notes added global lock time configuration. After configuring the global lock time, all charts on the note page display data according to the locked time.
• Unresolved event queries modified to show data from the last 48 hours, supporting manual refresh
• User access monitoring supports overriding logic for same-name user views
• User access monitoring Web, WeChat Mini Program, and uniapp-developed Mini Programs initialization parameters added isIntakeUrl configuration, used to determine whether to collect corresponding resource data based on request resource URLs, defaulting to collect all
• Intelligent inspections added support for self-built inspections
• Generate metric page operation column adjusted, added "Open in Metric Analysis" and "Open in Metric Management" operation icons
• Management navigation position adjusted, SSO management moved to member management, notification target management moved to monitoring, built-in views moved to scenarios

Best Practices Update¶

• Cloud Native: Best Practices for Multiple Kubernetes Cluster Metrics Collection.

More best practice updates can be found in Best Practice Version History.

September 15, 2022¶

Guance Update¶

Added Log Viewer DQL Search Mode¶

In the log viewer, you can switch to DQL manual input query mode by clicking the toggle small icon , supporting custom input filtering and search conditions.

• Filtering Conditions: Supports arbitrary combinations of and / or, uses () parentheses to indicate search priority, supports = , != and other operators;
• Search Conditions: Supports searching logs via DQL function query_string().

For more details, refer to the documentation DQL Search.

Optimized Application Performance Monitoring¶

Trace Viewer Detail Page Optimization¶

• Optimized detail page layout display, added duration, http method, http url, http status code, TraceId, etc.;
• Flame graph added thumbnail, double-click Span zoom display, click service name highlights corresponding Span;
• Span list added search, click select Span switches to flame graph synchronized display;
• Service call relationship added search

For more details, refer to the documentation Trace Viewer.

Profile Viewer Detail Page Optimization¶

In the Profile Viewer detail page, added quick operations, including search, copy, and click to view methods. For more details, refer to the documentation Quick Operation Instructions.

Optimized Monitor Event Notification Content Editing Mode¶

When creating a monitor, the event notification content triggered by satisfying conditions supports inputting and previewing markdown formatted text information, supporting the use of template variables.

Added Periodic Silence Configuration in Silence Management¶

Silence management added periodic silence functionality, allowing selection of "Only Once" or "Repeat" in the silence time. For more details, refer to Silence Management.

Other Functional Optimization¶

• Scenario dashboard grouped display optimization, scenario notes optimized text component Markdown format
• Infrastructure viewer display optimization, added tooltip information for display columns such as CPU usage rate and MEM usage rate; "Pods", "Deployments", "Services" viewer detail pages added viewing and copying corresponding YAML files

• Metric viewer deleted list view mode, retained tiled view mode and mixed view mode

• Log multi-index supports jumping to view

• Quick filter top 5 values in the viewer support viewing proportions, time field format in the viewer optimized, default display format is 2022/09/15 20:53:40
• Trace viewer time series chart added legend display, quick filter added HTTP-related fields

DataKit Update¶

• Added automatic cloud synchronization feature, no longer requiring manual specification of cloud vendor
• Supports syncing k8s labels as tags to pod metrics and logs
• Supports collecting various yaml information from k8s onto corresponding object data
• Trace collection supports automatically extracting some key meta information
• Installation process supports specifying installation source addresses to simplify offline installation flow
• Pipeline new features:
  • Added for loop/dictionary/array support
  • Added arithmetic expression support
  • Pipeline error information will be displayed on the collected data
  • If the time field segmentation fails, supports automatically correcting the time field(time) to avoid console pages failing to display time
  • Added len() function

For more DataKit updates, refer to DataKit Version History.

Best Practices Update¶

• Cloud Native: Best Practices for Multiple Kubernetes Cluster Metrics Collection.

More best practice updates can be found in Best Practice Version History.

September 1, 2022¶

Guance Update¶

Added Log Viewer DQL Search Mode¶

In the log viewer, you can switch to DQL manual input query mode by clicking the toggle small icon .

• Filter Conditions: Supports arbitrary combinations of and / or, uses () parentheses to indicate execution search priority, supports =, != and other operators;
• Search Conditions: Supports searching logs via DQL function query_string().

For more details, refer to the documentation DQL Search.

Optimized Application Performance Monitoring¶

Trace Viewer Details Page Optimization¶

• Optimized details page layout display, added duration, http method, http url, http status code, TraceId, etc.;
• Flame graph added thumbnail, double-click Span zoom display, click service name highlights corresponding Span;
• Span list added search, click select Span switches to flame graph synchronized display;
• Service call relationship added search

For more details, refer to the documentation Trace Viewer.

Profile Viewer Details Page Optimization¶

In the Profile Viewer details page, added quick operations, including search, copy, and click to view methods. For more details, refer to the documentation Quick Operation Instructions.

Optimized Monitor Event Notification Content Editing Mode¶

When creating a monitor, the event notification content triggered by meeting conditions supports inputting and previewing markdown formatted text information, supporting the use of template variables.

Added Periodic Silence Configuration in Silence Management¶

Silence management added periodic silence functionality, allowing selection of "Only Once" or "Repeat". For more details, refer to Silence Management.

Other Functional Optimization¶

• Scenario dashboard grouped display optimization, scenario notes optimized text component Markdown format
• Infrastructure viewer display optimization, added tooltip information for display columns such as CPU usage rate and MEM usage rate; "Pods", "Deployments", "Services" viewer detail pages added viewing and copying corresponding YAML files

• Metric viewer deleted list view mode, retained tiled view mode and mixed view mode

• Log multi-index supports jumping to view

• Quick filter top 5 values in the viewer support viewing proportions, viewer time field format optimized, default display format is 2022/09/15 20:53:40
• Trace viewer time series chart added legend display, quick filter added HTTP-related fields

DataKit Update¶

• Increased automatic cloud synchronization function, no longer needs manual designation of cloud vendors
• Supports syncing k8s labels as tags to pod metrics and logs
• Supports collecting various yaml information from k8s onto corresponding object data
• Trace collection supports automatically extracting some key meta information
• Installation process supports specifying installation source addresses, simplifying the offline installation process
• Pipeline new features:
  • Added for loop/dictionary/array support
  • Added arithmetic expression support
  • Pipeline error information will be displayed on the collected data
  • If the time field segmentation fails, supports automatically correcting the time field(time), to avoid console pages failing to display time
  • Added len() function

More DataKit updates can be referenced in DataKit Version History.

Best Practices Update¶

• Monitoring Monitoring- Middleware (MIDDLEWARE) - Insight MySQL.

More best practice updates can be referenced in Best Practice Version History.

September 15, 2022¶

Guance Update¶

Added Intelligent Inspection Functionality¶

Intelligent inspections are based on Guance's intelligent detection algorithms, supporting automatic detection of infrastructure and application issues, helping users discover problems occurring during the operation of IT systems. Through root cause analysis, quickly locate the reasons for abnormal issues; through Guance's intelligent prediction algorithm, help users foresee potential problems in infrastructure and applications, evaluate the impact level of problems on system operations, better determine the priority of troubleshooting work, and reduce the uncertainty of the troubleshooting process.

Intelligent inspections currently support three inspection templates:

• Memory Leakage: Detects if there is a memory leak issue with the current workspace hosts
• Disk Usage Rate: Detects if there is excessively high disk usage rate for the current workspace hosts
• Application Performance Detection: Detects if there are fluctuations in QPS, average response time, P90 response time, and error rates for services in the current workspace

Optimized Viewer Search and Filtering Functions¶

1. Viewer Search Added "Not Wildcard Reverse Fuzzy Match", i.e., not matching fuzzy query results.

1. Quick Filtering in the Viewer Added Workspace-level and Personal-level Filtering Methods

2. Workspace-level Filter Items: Configured by administrators/owners, click the "Settings" button next to the quick filter to configure workspace-level filter items

3. Personal-level Filter Items: All members can configure local browser-based quick filter items, click the "Edit" on the right side of the quick filter to configure personal-level filter items

For more details, refer to the documentation Viewer Search and Filtering.

Added Modification of URL Time Range for Data Queries¶

Guance supports modifying the time parameter in the browser's URL directly to perform data queries within the current workspace viewer/dashboard with a time range, supporting seconds, minutes, hours, and days as units, such as time=30s, time=20m, time=6h, time=2d, etc. As shown in the figure below, modifying time=2h in the browser displays data from the last 2 hours. For more details, refer to the documentation URL Time Range.

Added Dashboard View Variable Log, Application Performance, User Access, Security Inspection Data Source Configuration¶

Dashboard view variables added "Log," "Application Performance," "User Access," and "Security Inspection" data source configurations, while optimizing the interaction for creating and editing dashboard view variables.

• In dashboards without added view variables, a "Add View Variable" button is added to the top navigation bar, allowing you to add view variables by clicking it.
• If the dashboard has already added view variables, the name of the view variable will be displayed below the top navigation bar. Click the "Edit" button on the right to enter the view variable editing page.

For more details, see the documentation View Variables.

Optimized Chart Query Interaction¶

• Clicking the "Separator Bar" between the chart and the query allows up and down dragging.
• Clicking the "Drag" button to the left of the query statement allows up and down dragging to adjust the order of the queries.
• The scrollbar only affects the query statement area and does not affect the chart.

Added User Access Metric Detection Event Notification Template Variables¶

In the "User Access Metric Detection (RUM)" monitor, when configuring notification events, additional template variables are supported besides the general ones. For more details, refer to the documentation Event Title, Content Template.

Optimized One-click Open Link in Event Content¶

When configuring a monitor, adding links in the event content allows you to open a new page by clicking the link in the "Event Content" on the event details page after the event is triggered.

DataKit Update¶

• The default timeout duration for prom collectors is 3 seconds.
• Log-related issues fixed:
  • Added log_read_offset field for log collection
  • Fixed the bug where logs were not correctly readAll after rotate
• Container collection-related issue fixes:
  • Fixed compatibility issues with environment variable NODE_NAME
  • Prom collection in Kubernetes changed to serial mode, node distributed collection
  • Appended corresponding yaml information to Pod objects
• Pipeline-related updates:
  • Optimized execution steps of Pipeline
  • grok() and json() functions now execute trim-space operations by default
• DDTrace-related updates:
  • Fixed potential goroutine leakage problems
  • Supports configuring disk cache to alleviate memory usage problems
• Other Bug Fixes:
  • Optimized line protocol construction
  • Removed regular cleaning of tail data in log collection to alleviate possible log truncation issues

More DataKit updates can be referenced in DataKit Version History.

July 21, 2022¶

Guance Update¶

Added Intelligent Inspection Functionality¶

Intelligent inspection based on Guance intelligent detection algorithms supports automatic detection of infrastructure and application issues, helping users discover problems that occur during the operation of IT systems. Through root cause analysis, quickly locate the reasons for abnormal issues; through Guance intelligent prediction algorithms, help users foresee potential problems in infrastructure and applications, evaluate the impact level of problems on system operation, better determine the priority of troubleshooting work, reduce the uncertainty of the troubleshooting process.

Currently, intelligent inspections support three inspection templates:

• Memory Leak: Detects if there is a memory leak problem with the current workspace host
• Disk Usage Rate: Detects if there is an excessively high disk usage rate problem with the current workspace host
• Application Performance Detection: Detects if there are fluctuations in service QPS, average response time, P90 response time, and error rates in the current workspace

Optimized Viewer Search and Filtering Functions¶

1. Viewer search added "not wildcard reverse fuzzy match", i.e., does not match fuzzy query results.

1. Viewer quick filtering added space-level and personal-level filtering methods

2. Space-level filter items: Configured by administrators/owners, click the "Settings" button next to the quick filter to configure space-level filter items

3. Personal-level filter items: All members can configure local browser quick filter items, click the "Edit" on the right side of the quick filter to configure personal-level filter items

For more details, refer to the documentation Viewer Search and Filtering.

Added Modify Time Range in URL for Data Queries¶

Guance supports directly modifying the time parameter in the browser's URL to perform data queries within the current workspace viewer/dashboard with a time range, supporting seconds, minutes, hours, and days as units, such as time=30s, time=20m, time=6h, time=2d, etc. As shown in the figure below, modifying time=2h in the browser displays data from the last 2 hours. For more details, refer to the documentation URL Time Range.

Added Dashboard View Variable Log, Application Performance, User Access, Security Inspection Data Source Configuration¶

Dashboard view variables added "Log," "Application Performance," "User Access," and "Security Inspection" data source configurations, while optimizing the interaction for creating and editing dashboard view variables.

• In dashboards without added view variables, a "Add View Variable" button is added to the top navigation bar, allowing you to add view variables by clicking it.
• If the dashboard has already added view variables, the name of the view variable will be directly displayed below the top navigation bar. Click the "Edit" button on the right to enter the view variable editing page.

For more details, check the documentation View Variables.

Optimized Chart Query Interaction¶

• Clicking the "Separator Bar" between the chart and the query allows up and down dragging.
• Clicking the "Drag" button to the left of the query statement allows up and down dragging to adjust the query order.
• The scrollbar only affects the query statement area and does not affect the chart.

Added User Access Metric Detection Event Notification Template Variables¶

In the "User Access Metric Detection (RUM)" monitor configuration, apart from the general template variables, the following template variables are additionally supported. For more details, refer to the documentation Event Title, Content Template.

Optimized One-click Open Link in Event Content¶

When configuring a monitor, adding links in the event content allows you to open a new page by clicking the link in the "Event Content" on the event details page after the event is triggered.

DataKit Updates¶

• Adjusted global tag behavior to avoid splitting tags in election-type collections
• SQLServer collector added election support
• Line protocol filter supports all data types
• 9529 HTTP service added timeout mechanism
• MySQL
  • dbm metric set name adjustment
  • service field conflict issues
• Added container_runtime_name field to container objects to differentiate container names at different levels
• Redis adjusted slowlog collection, changing its data to log storage
• Optimized TDEngine collection
• Enhanced Containerd log collection
• Pipeline increased Profile data support
• Container/Pod log collection supports adding tags via Label/Annotation
• Fixed Jenkins CI data collection time precision issue
• Fixed Tracing resource-type value inconsistency issue
• eBPF added HTTPS support
• Fixed possible crash issue in log collector
• Fixed prom collector leakage issue
• Supported configuring IO disk cache via environment variables
• Added Kubernetes CRD support

More DataKit updates can be referenced in DataKit Version History.

Best Practices Update¶

• Integration
  • Skywalking collects JVM observability best practices
  • Minio observability best practices

More best practice updates can be referenced in Best Practice Version History.

June 21, 2022¶

Guance Update¶

Guance Help Documentation Fully Updated¶

To enhance the reading experience of Guance help documentation, Guance help documentation has been migrated to Guance domain, allowing you to view Guance help documentation more simply and quickly. The new Guance help documentation address is: https://docs.guance.com/en.

Added Profile Observability¶

Profile supports collecting dynamic performance data of applications running under different language environments like Java / Python, helping users view CPU, memory, IO performance issues. Collecting profile data requires installing DataKit first and configuring the Profile collector. After configuration, DataKit will upload collected profile data to your Guance workspace, allowing you to understand your program code performance through the real-time data viewer of Profile. For more details, refer to the documentation Profile.

Full Text Analysis Processing Coverage for All Data Types via Pipeline¶

Text processing (Pipeline) is used for data parsing. By defining parsing rules, various data types are cut into structured data that meet our requirements. In Guance workspace "Management" - "Text Processing (Pipeline)", click "Create Pipeline" to create a new pipeline file. For more details, refer to the documentation Text Processing (Pipeline).

Added Deployment Network Details and Network Distribution¶

Deployment network supports viewing network traffic between Deployments. It supports viewing source IP to destination IP network traffic and data connection situations based on IP/port, visualizing them in real-time to help businesses understand their business system's network operating status, quickly analyze, track, and locate fault problems, preventing or avoiding business problems caused by network performance degradation or interruption.

After successful collection of Deployment network data, it is reported to Guance console. You can view Deployment network performance monitoring data information in the "Network" section of the "Infrastructure" - "Container" - "Deployment" detail page; in "Infrastructure" - "Network" - "Deployment", you can view the network distribution and data connections of all Deployments within the workspace. For more details, refer to Deployment Network.

Optimized Event Detection Dimension Jump to Other Viewers¶

In the unresolved event viewer, you can click the detection dimension to view related containers, processes, logs, traces, RUM, availability tests, security inspections, CI, etc. If the related viewer has no relevant data, the corresponding jump link will be grayed out and unclickable.

Added Log Viewer JSON-formatted message Information Search¶

The log viewer added searching for JSON-formatted log content (message), with the search format being: @key.key:value.

Note: JSON search only supports workspaces created after this feature launch.

Added Custom Input app_id Information When Creating New Applications in User Access Monitoring¶

Added custom application ID functionality. When creating a new application using user access monitoring, you can custom input app_id to generate a unique application ID identifier within the current workspace, which can be used for distinguishing application types, data upload matching, etc.

• Application Name (Required): Used to identify the current application implementing user access monitoring.
• Application ID Identifier (Optional): A unique application ID identifier within the current workspace, supporting customization for data upload matching; the application ID identifier can have up to 20 characters, only supporting input of uppercase and lowercase letters.
• Application ID: If you fill in the application ID identifier when creating an application, after clicking Create Application, the filled-in application ID identifier information will be concatenated before the application ID for easy querying and filtering of applications.

Optimized Process Detection to Infrastructure Object Detection¶

Process detection optimized to infrastructure object monitoring, added selection for hosts, containers, processes, Pods, Deployments, Replicaset, Jobs, resource catalogs, etc., basic objects used to monitor infrastructure object data within the workspace. For more details, refer to the documentation Infrastructure Object Detection.

Other Functional Optimization¶

• Added CPU usage rate and memory usage fill metrics in the cell mode of the infrastructure POD viewer
• Optimized log blacklist configuration. Supports manually entering log sources as log blacklist sources
• Optimized time component query for service list data in application performance monitoring
• Optimized K8S DataKit installation guidance text, automatically adding the current workspace token to the configuration DataWay data gateway address
• Optimized monitor configuration UI style

DataKit Updates¶

2022/06/21¶

• gitrepo supports passwordless mode
• prom collector
  • Supports log mode collection
  • Supports configuring HTTP request headers
• Supports collecting container logs longer than 16KB
• Supports TDEngine collector
• Pipeline
  • Supports XML parsing
  • Remote debugging supports multiple data types
  • Supports calling external Pipeline scripts via the use() function in Pipeline
• Added IP library (MaxMindIP) support
• Added DDTrace Profile integration
• Containerd log collection supports filtering rule configuration via image and K8s Annotation
• Document library overall switch

2022/06/16¶

• Log collection supports recording collection position, avoiding data loss due to DataKit restarts, etc.
• Adjusted Pipeline settings for handling different data types
• Supports receiving SkyWalking metric data
• Optimized log blacklist debugging function:
  • Displays filtered points in Monitor
  • Adds a .filter file in the datakit/data directory to record fetched filters
• Monitor adds display of DataKit opened file count
• DataKit compiler upgraded to golang 1.18.3

2022/06/07¶

• Added TCP/UDP port detection collector
• DataKit and DataWay added DNS detection, supporting DataWay DNS dynamic switching
• eBPF L4/L7 traffic data added k8s deployment name field
• Optimized OpenTelemetry metric data
• ElasticSearch added AWS OpenSearch support
• Line Protocol Limit relaxed string length limit to 32MB
• prom collector added extra configuration, supporting ignoring specified tag=value matches to reduce unnecessary time series
• Sink added Jaeger support
• Kubernetes-related metric collection defaults to all off to avoid excessive time series problems
• DataKit Monitor added dynamic discovery (e.g., prom) of collector list refresh

More DataKit updates can be referenced in DataKit Version History.

Best Practices Update¶

• APM
  • GraalVM and Spring Native Project Achieving Trace Observability
• Integration
  • Host Observability Best Practices (Linux)

More best practice updates can be referenced in Best Practice Version History.

Integration Template Update¶

Added Documents¶

• Alibaba Cloud
  • Alibaba Cloud NAT
  • Alibaba Cloud CDN

Added Views¶

• Alibaba Cloud
  • Alibaba Cloud NAT
  • Alibaba Cloud CDN

June 6, 2022¶

Guance Billing Update¶

Guance billing optimizes the Time Series billing logic and Metric Data Retention Policy. The original 300 items for 3 yuan has been reduced to 1000 items for 3 yuan. At the same time, metrics data added 3-day, 7-day, and 14-day data retention policies, supporting Custom Data Retention Policies.

Time Series counts the total number of combinations based on tags for the metrics data reported in the current workspace. Data retention policy refers to the data storage duration, which is the data retention time for data reported to the current workspace, and data exceeding the storage duration will be automatically deleted.

Time Series is fully counted, meaning the daily generated time series will be cumulatively added during the data retention period. The longer the data retention policy, the higher the cost of time series.

This optimization reduces the cost of time series and shortens the data retention policy for metrics, allowing you to flexibly adjust the data retention policy for metrics and save costs.

Guance Update¶

Added Jenkins CI Observability¶

Guance added Jenkins CI observability. You can directly view the CI results in Jenkins through Guance's CI visualization function. CI is continuous integration. When developers push code, if they encounter problems, they can view all CI pipelines and their success rates, failure reasons, specific failed stages in Guance, providing code update assurance. For more details, refer to CI Visualization.

Added Custom Viewer Chart Synchronous Search¶

Custom viewers added a synchronous search toggle, deciding whether search conditions affect chart queries, defaulting to enabled. When there is content in the search box, turning off the toggle restores the chart query to its default state; turning on the toggle makes the chart query affected by the selected filter content.

Added Network Topology and Service Topology Drill-down Analysis¶

In the infrastructure network topology diagram, clicking the host/Pod icon and then clicking "View Upstream/Downstream" allows you to view the upstream/downstream node associations of the current node. In the upstream/downstream nodes, clicking the top-left "Return to Overview" returns to the original network topology diagram, and searching or filtering in the search box filters associated upstream/downstream nodes, displaying matching upstream/downstream node associations according to the search or filter results. For more details, refer to the documentation Network Upstream/Downstream.

In the service topology diagram, clicking the service icon and then clicking "View Upstream/Downstream" allows you to view the upstream/downstream service associations of the current service. In the upstream/downstream services, clicking the top-left "Return to Overview" returns to the original service topology diagram, and searching or filtering in the search box filters associated upstream/downstream services, displaying matching upstream/downstream service associations according to the search or filter results. For more details, refer to the documentation Service Upstream/Downstream.

Added Deletion of Resource Catalog Data and Indexes¶

Guance supports owners and administrators deleting specified resource catalog classifications and all resource catalogs, entering "Management" - "Basic Settings", clicking "Delete Resource Catalog" afterward, choosing the deletion method for resource catalogs deletes the corresponding object data.

• Specified Resource Catalog Classification: Only deletes data under the selected object classification, does not delete indexes
• All Resource Catalogs: Deletes all resource catalog data and indexes

Note: Once all resource catalogs are deleted, all data and indexes uploaded to infrastructure customization will be completely deleted and cannot be recovered, requiring re-reporting of all configured resource catalog data. Each day limits five deletions of all resource catalogs.

For more resource catalog introductions, refer to the documentation Customization.

Added Viewer Snapshot Viewing Entry¶

In Guance metrics, logs, events, application performance monitoring, user access monitoring, cloud dial testing, security inspection, CI visualization, etc., viewers, after saving snapshots, you can directly click the view snapshot icon in the top-right corner to slide out and view already saved snapshots.

• Supports searching snapshots by keywords in snapshot names for fuzzy matching
• The first one is the default view, does not support sharing, copying links, or deletion functions
• Except for the first default view, other snapshots support sharing, copying links, and deletion functions, clicking the "Snapshot Name" opens the corresponding data copy in the current viewer

Added Viewer Filter Condition Editing Function¶

In Guance viewers, when filtering using "field:value" in the search bar, you can click "field:value" to edit and modify "field:value" and filter using the modified result.

Optimized User Access View Viewer Associated Trace as Fetch/XHR¶

In Guance user access monitoring View Viewer detail pages, switching to "Fetch/XHR" supports viewing every network request sent by the user to the backend application during user access, including occurrence time, request trace, and duration.

If the network request has a corresponding trace_id, there will be a prompt icon before the request, clicking the request will redirect to the corresponding trace detail page.

Added High-performance Mode for Chart Data Loading¶

Guance supports high-performance mode for chart data loading, defaulting to off. You can enable it by clicking the bottom-left account and selecting "High-performance Mode". After enabling high-performance mode, all charts do not dynamically load; instead, they load simultaneously when entering the page, meaning that when scrolling beyond the current page to view, the charts have already loaded completely and can be viewed directly.

Note: Enabling high-performance mode only applies to the current user viewing charts.

Added Alert Configuration Event Notification Levels¶

Alert configuration supports custom selection of event notification levels, including critical, important, warning, recovery, data gap, data gap recovery, and considering data gap as recovery, supporting multi-selection, one-click clearing of options, clearing options after clearing, requiring manual selection of corresponding values. For more alert configurations, refer to the documentation Alert Settings.

Other Functional Optimizations¶

• Scenario dashboards support hiding/showing large titles
• Optimized event details page type text display
• Added sorting function by fields in infrastructure list views
• Log viewer added hide distribution chart button
• Viewer supports searching display columns by keyword, supports setting display columns as preset fields, allowing direct display of uploaded data after cutting fields via Pipeline and reporting data.
• Added one-click view of corresponding integration documents in built-in template libraries and built-in views to help you quickly configure corresponding collectors
• Besides supporting binding built-in views to trace services, applications, log sources, projects, and labels in built-in views, added support for custom key and value binding to related views, also supporting binding built-in views in service side-sliding detail pages
• Optimized Lark robot notification targets, supporting custom security key checks
• When configuring monitors, if the configured data range is smaller than the detection frequency, a prompt is triggered indicating potential data gaps

DataKit Updates (2022/05/26)¶

• Pipeline adjustments allow all data types to be processed via Pipeline configuration
• grok() supports directly extracting fields as specified types, eliminating the need for additional cast() function conversions
• Pipeline added multiline string support, enhancing readability for long strings (e.g., grok regular expression cuts) by writing them across multiple lines
• Each Pipeline's operational status can be directly viewed via datakit monitor -V
• Added Kubernetes Pod object CPU/memory metrics
• Helm added more Kubernetes version installation compatibility
• Optimized OpenTelemetry, added JSON support for HTTP protocol
• DataKit automatically records correction actions when auto-correcting line protocols, facilitating data debugging
• Removed all string metrics from time-series data
• In DaemonSet installations, if the election namespace is configured, all participating collectors will add specific tags (election_namespace) to their data
• CI observability added Jenkins support

Breaking Changes¶

For Docker container log collection, the host (Node) _varl/lib_ path needs to be mounted into DataKit (because Docker logs default to the host’s _var/lib/_). In _datakit.yaml_, add the following configurations to volumeMounts and volumes:

volumeMounts:
- mountPath: /var/lib
  name: lib

# Omitted other parts...

volumes:
- hostPath:
    path: /var/lib
  name: lib

Best Practices Updates¶

• APM
  • Based on Guance, use SkyWalking to achieve联动analysis of RUM, APM, and logs
• Monitoring Best Practices
  • Construction of OpenTelemetry Observability
  • OpenTelemetry to Jaeger, Grafana, ELK
  • OpenTelemetry to Grafana
  • OpenTelemetry to Guance
• Guance Tips
  • OpenTelemetry Sampling Best Practices

Integration Template Updates¶

Added Documents and Views¶

• Data Collection
  • Opentelemetry Collector
• Container Orchestration
  • Kubernetes Scheduler
  • Kubernetes Controller Manager
  • Kubernetes API Server
  • Kubernetes Kubelet

Added Views¶

• Container Orchestration
  • Kubernetes Nodes Overview
• Middleware
  • JVM Kubernetes

May 19, 2022¶

Guance Update¶

Optimized Guance Commercial Plan Registration Process¶

Guance registration provides Free Plan and Commercial Plan registration options, allowing you to register a Guance account according to actual needs. Guance supports pay-as-you-go billing, providing an out-of-the-box cloud platform for comprehensive observability.

Added Scenario Dashboard User View Template Library¶

Guance includes over 60 built-in system view templates, requiring no configuration, ready-to-use, meeting various monitoring scenario needs. You can customize views as user view templates to create dashboards with one click.

Added Custom Viewer Log Source and Filtering Linkage in Scenarios¶

The data scope of the custom viewer in scenarios added filtering functionality, based on log sources, filtering fields under that log source, supporting matching multiple fields, and combining with search and grouping for further log data filtering.

Added Copy Event Details Page Content as JSON Format¶

On the event details page, supports clicking the "Copy Complete Event Content" button to obtain all key data corresponding to the current event. If a monitor is configured with an associated dashboard, you can click the "Associated Dashboard" button to jump to the corresponding dashboard.

In the "Associated Information" of the event details page, if "Log Detection" configures multiple expression queries (same object type), the associated information supports tab switching for multiple expression queries. If there are two expression queries A and B, then the relationship information includes A and B tabs for switching views.

Added Log Data Desensitization Processing¶

Guance added log data desensitization processing. After data collection and reporting to the Guance workspace, some data may contain sensitive information, such as IP addresses and user information. This part of the information can be desensitized by configuring sensitive fields.
Note:

• Desensitized data is only accessible to workspace administrators and above; standard and read-only members cannot view desensitized information.
• Configuring sensitive fields is only operable by workspace administrators and above; standard and read-only members can only view configured sensitive fields.

Optimized Log Viewer and Detail Pages¶

Guance Log Viewer defaults to showing "time" and "message" fields. This optimization supports hiding the "message" field display.

In the log detail page, log content automatically displays Json and text viewing modes based on the message type. If the log does not have a message field, the log content part will not be displayed. Log content supports expanding/collapsing, defaulting to expanded state, collapsing shows only one row height.

Extended fields display all related fields of the log, supporting "Copy" and "Add to Filter" for quick filtering and viewing.

Added Network Data Monitoring Monitors¶

Network Data Monitoring is used to monitor network performance metrics data within the workspace. By setting threshold ranges, alerts are triggered when the metrics reach thresholds. "Guance" supports setting alerts and custom alert levels for individual metrics. In Monitors, click + Create Monitor, select Network Data Monitoring, and enter the detection rule configuration page.

Optimized Built-in View Binding Function¶

Guance built-in views include system views and user views. This optimization cancels custom binding of system views as viewer views, only supporting binding user views as viewer views. If you need to bind system views, you can clone system views as user views first. If system views and user views have the same name, user views are prioritized in the viewer. For how to bind user views as viewer views, refer to the documentation Bind Built-in Views.

Other Functional Optimizations¶

• Paid Plans and Bills Added Pre-purchase Card Balance
• Infrastructure Detail Style Optimization
• Trace Detail Page Attribute Wrap Display Optimization
• Monitor Configuration Template Variable Display Optimization
• Added Quick Access, DQL Query and Snapshot Menus Moved Under Quick Access
• Guance Management Backend Added Template Management Category Information

DataKit Updates (2022/5/12)¶

• eBPF Added arm64 Support
• Line Protocol Construction Supports Automatic Correction
• DataKit Main Configuration Added Example Configuration
• Prometheus Remote Write Supports Tag Renaming
• Merged Community Edition DataKit Existing Features, Mainly Including Sinker Functionality and Filebeat Collector
• Adjusted Container Log Collection, Directly Supporting Containerd Container stdout/stderr Log Collection
• Adjusted Hostname Acquisition Strategy in DaemonSet Mode
• Trace Collector Supports Filtering Resources (resource) by Service Name (service) Wildcards

More DataKit updates can be referenced in DataKit Version History.

Best Practices Updates¶

• Cloud Native
  • Utilize Guance to Initiate Rancher Observability Journey
• Microservices Observability Best Practices
  • Kubernetes Cluster Application Using SkyWalking to Collect Trace Data
  • Kubernetes Cluster Logs Reported to Same Node's DataKit Best Practices
• Gitlab-CI Observability Best Practices
  • Gitlab-CI Observability Best Practices

More best practice updates can be referenced in Best Practice Version History.

Integration Template Updates¶

Added Documents and Views¶

• Middleware
  • Resin
  • Beats
• Host System
  • Procstat

Added Views¶

• Container Orchestration
  • Istio Service
• Alibaba Cloud
  • ASM Service

May 6, 2022¶

Guance Update¶

Optimized Guance Commercial Plan Upgrade Process¶

Guance upgrades to the commercial plan default to opening Guance Billing Center Account Settlement, supporting changing the settlement method to cloud account settlement, including Alibaba Cloud Account and AWS Cloud Account settlement methods.

Added Process, Log, Trace Details Page Related Network¶

Guance Process, Log, Trace detail pages added related network data analysis, supporting viewing source host/source process service to target network traffic and data connection situations based on IP/port, visualizing them in real-time to help enterprises understand the network operating status of their business systems, quickly analyzing, tracking, and locating fault problems, preventing or avoiding business problems caused by network performance degradation or interruptions.

Scene Module Optimization¶

Optimized Dashboard, Removed Edit Mode¶

In the scene dashboard top navigation bar, removed the "Edit" button, added "Add Chart" to add new charts to the dashboard. After adding the chart, click the top-right "Complete Adding" to finish.

In Charts, click the "Settings" button, choose "Modify" to edit the chart.

Added Chart Link Display Switch¶

Guance supports built-in links and custom links in charts, helping you jump from the current chart to the target page. Built-in links are default links provided by Guance for charts, mainly based on the current query time range and group labels, helping you view corresponding logs, processes, containers, and traces. The built-in link display switch is default off and can be enabled during chart editing; custom links, once created, have the display switch default on. For more details, refer to the documentation Chart Links.

Optimized DQL Query and Simple Query Conversion¶

Click the toggle button to the right of DQL Query to switch DQL Query to simple query.
Note: Switching DQL Query to Simple Query, if unable to parse or incomplete parsing:

• In Simple Query without making changes, switching back to DQL Query will show the previous DQL query statement;
• Adjusting the query statement in Simple Query, switching back to DQL Query will parse according to the latest Simple Query.

For more applications of DQL Query and Simple Query, refer to the documentation Chart Query.

Monitor and Event Module Optimization¶

Added Event Associated Information¶

Guance supports viewing related information triggering the current event, such as viewing logs related to the triggered event. This associated information only supports four types of monitors producing events: log detection, security inspection anomaly detection, process anomaly detection, and availability data detection.

Added Data Gap Event Title and Content Configuration¶

Guance monitors "Threshold Detection", "Water Level Detection", "Interval Detection", "Mutation Detection", "Process Anomaly Detection", "Application Performance Indicator Detection", "User Access Indicator Detection" added data gap event title and content configuration, defaulting to non-editable, becoming editable when selecting to trigger data gap events, supporting the use of pre-configured template variables, detailed reference Event Title/Content Template.

Optimized Availability Data Detection¶

Guance monitor Availability Data Detection, optimized to support selecting HTTP, TCP, ICMP, WEBSOCKET probing types.

Optimized Alert Notification Templates, Added Associated Jump Links¶

Emails, DingTalk, WeChat, Lark received alert notifications include "Guance jump link", clicking directly jumps to the corresponding Guance event details, with a time range of 15 minutes prior to the current time, i.e., for an event at 18:45:00, clicking the link redirects to the event details page with a fixed time range of 4.20 18:30:00 ~ 4.20 18:45:00. More alert notifications can be referenced in the documentation Alert Settings.

Other Functional Optimizations¶

• Optimized service servicemap metric query performance
• Added numerical field support > | >= | < | <= | [] 5 kinds of writing methods in viewers
• Added label cascading filtering support in metric viewers
• Optimized DQL query return error prompts

DataKit Updates¶

• The filtering function of the process collector only applies to metric collection, object collection is unaffected
• Optimized DataKit sending DataWay timeout issues
• Optimized Gitlab collector
• Fixed log collection truncation issues
• Fixed various trace collectors reload partial configuration not taking effect issues

More DataKit updates can be referenced in DataKit Version History.

Integration Template Updates¶

Added Redis Sentinel Data Storage Integration Documents and Views¶

Redis-sentinel observation scenes mainly display Redis cluster, slaves, and node distribution information.

April 26, 2022¶

Guance Update¶

Optimized SSO Single Sign-On¶

Guance supports users initiating SSO single sign-on based on workspaces. Users log in by entering their company email to obtain the corresponding SSO login, achieving the corresponding verification login. In Guance workspace "Management" - "SSO Management" - "Enable", you can set SSO single sign-on for employees. The main optimizations include:

• A workspace updated from supporting creation of multiple identity providers to only supporting configuration of one SSO single sign-on, defaulting to treating the last updated SAML2.0 configuration as the final single sign-on entry point
• If multiple workspaces configure the same identity provider (IdP) data, after SSO single sign-on, you can switch to view data from the corresponding workspaces
• During SSO single sign-on configuration, the "User Whitelist" configuration is replaced with "Email Domain", requiring only the suffix domain of the email to match the user email suffix configured in the identity provider (IdP) for single sign-on
• SSO single sign-on enablement, configuration updates, and deletions support email notifications and audit event generation
• SSO single sign-on users support deletion and editing, editing can upgrade permissions to "Administrator"

More SSO single sign-on details can be referenced in the documentation SSO Management.

April 21, 2022¶

Guance Community Edition Launch¶

Guance Community Edition provides teachers, students, cloud computing enthusiasts, and other community users with a simple-to-obtain yet fully functional productized local deployment platform. Welcome to apply for free and download for trial use, building your own Guance platform and experiencing the complete product features.

Guance Update¶

Added Gitlab CI Observability¶

Guance supports visualizing the process and results of Gitlab's built-in CI. You can directly view the CI results in Gitlab through Guance's CI visualization function. The CI process is continuous integration, where developers encounter problems while pushing code, allowing you to view all CI pipelines and their success rates, failure reasons, specific failed stages in Guance, helping ensure code update reliability. For more details, refer to CI Viewer.

Added Online Help Obus Assistant¶

Guance Obus Assistant supports quickly viewing foundational introductions, advanced guides, best practices, DataKit, Func, etc., documents within workspaces, by clicking provided keywords or directly inputting keywords in the search bar for searching, helping you quickly obtain relevant document explanations. For more details, refer to the documentation Help.

Added Dashboard Set Refresh Frequency¶

Guance supports setting a refresh frequency in scenario dashboards. The initial default refresh frequency is 30 seconds, supporting options of 10 seconds, 30 seconds, and 60 seconds. If the time widget is "Paused," it will no longer refresh.

Added Process 48-hour Replay¶

Guance infrastructure processes support viewing process data collected in the last ten minutes. Clicking the time range allows you to view the 48-hour replay of the process. After dragging, refreshing pauses, and the time displayed is: [Start Time - End Time], with the query time range being 5 minutes. Clicking the "Play" button or refreshing the page returns to viewing the process from the last 10 minutes.

Added Application Performance Global Overview, Service Classification Filtering, Service Topology Diagram Distinguishing Environment and Version¶

Application Performance Monitoring added a global performance overview view. You can view online service counts, P90 service response time, service maximum impact time, service error counts, service error rate statistics on the overview page. Additionally, you can view the top 10 rankings for P90 service, resource, operation response times, as well as top 10 rankings for service error rate, resource 5xx error rate, and resource 4xx error rate.

In the application performance service list, you can classify filter by clicking the service type icon. Clicking again restores full viewing.

In the service list, switching to "Topology Map" mode allows you to view the call relationships between different services. Supports drawing trace topology maps based on two dimensions: service (service) and service environment version (service+env+version). Enabling "Distinguish Environment and Version" will draw the service topology map according to different environment versions. For example, in a canary release, enabling environment and version allows you to view service calls under different environment versions.

Optimized SSO Single Sign-On Configuration¶

SSO single sign-on configuration adjusts user whitelist to email domain, used to verify if the email suffix entered at the single sign-on matches, matching emails can obtain SSO login links online. For more SSO configuration details, refer to the documentation SSO Management.

Other Functional Optimizations¶

• Added "All Sources" option in the associated logs of the trace detail page
• Added reverse selection support for metric filtering, adjusted aggregation function position
• Optimized log, application performance, user access, security inspection generating metrics, the selected time in "Frequency" also serves as the aggregation period
• Canceled the workspace owner transfer permission function in Guance deployment edition, supports backend management settings
• Optimized alert notification SMS templates
• Optimized availability monitoring new probing list, supports directly selecting HTTP, TCP, ICMP, WEBSOCKET probes

DataKit Updates¶

• Pipeline module fixed Grok dynamic multi-line pattern issues
• DaemonSet optimized Helm installation, added pprof environment variable configuration support, all default enabled collectors in DaemonSet support environment variable configuration
• Tracing collector initially supports Pipeline data processing.
• Probing collector added a mechanism for failing tasks to exit
• Logs added unknown level (status), for logs not specified with a level are set to unknown
• Container collector fixes:
  • Fixed cluster field naming issues
  • Fixed namespace field naming issues
  • In container log collection, if Pod Annotation does not specify the log source, then DataKit will deduce the log source based on this priority
  • Object reporting is no longer restricted by the 32KB character length limit (due to Annotation content exceeding 32KB), all Kubernetes objects have deleted annotation

More DataKit updates can be referenced in DataKit Version History.

Best Practices Updates¶

• Microservices Observability Best Practices
  • Full-service mesh microservices architecture from development to canary release best practices (Part 1)
  • Full-service mesh microservices architecture from development to canary release best practices (Part 2)
  • Full-service mesh microservices architecture from development to canary release best practices (Part 3)
• Monitoring Best Practices
  • JAVA OOM Exception Observability Best Practices

More best practice updates can be referenced in Best Practice Version History.

Integration Template Updates¶

Added Documents¶

• Application Performance Monitoring (APM)
  • Node.JS
  • Ruby
• Middleware
  • RocketMQ
• Container Orchestration
  • Istio
  • Kube State Metrics
• Data Storage
  • Aerospike

Added Views¶

• Container Orchestration
  • Kubernetes Overview by Pods
  • Istio Mesh
  • Istio Control Plane
• Alibaba Cloud
  • Alibaba Cloud ASM Mesh
  • Alibaba Cloud ASM Control Plane
  • Alibaba Cloud ASM Workload
• Middleware
  • RocketMQ

April 8, 2022¶

Guance Billing Update¶

Added Alibaba Cloud Account Settlement Method¶

In Guance Billing Center "Manage Workspace", you can view all workspaces bound to the account, modify the settlement method of already bound workspaces, and switch between Guance Billing Center account, Amazon Cloud account, and Alibaba Cloud account settlements via "Change Settlement Method".

Guance Update¶

Added DQL Query Viewer¶

DQL is a language specifically developed for Guance, with simple syntax for easy use, allowing data queries in Guance workspaces or terminal devices using DQL language.

In Guance workspace, click the menu bar's "DQL Query" to open the DQL Query Viewer, or you can directly open DQL Query via the shortcut key Alt+Q. The DQL Query Viewer supports table and JSON return results, saving historical query records for 7 days.

Availability Monitoring Added TCP/ICMP/Websocket Probe Protocols¶

Guance supports custom probe tasks. By creating probe tasks based on different protocols such as HTTP, TCP, ICMP, WEBSOCKET, it comprehensively monitors network performance, quality, and stability of data transmission to various services in different regions and carriers.

Added Infrastructure Network Module¶

In infrastructure, the original host network Map and Pod network Map moved from the host and container modules to the newly added "Network" module, supporting viewing host and Pod network Maps. The Pod network Map fill indicators added seventh-layer network indicators: requests per second, error rate, and average response time. For more details, refer to the documentation Network.

Infrastructure Container Pod Added HTTP Seventh-Layer Network Data Display¶

Infrastructure container Pod added HTTP seventh-layer network data collection and display. After successful Pod network data collection, it is reported to Guance workspace. In "Infrastructure" - "Container" - "Pod" detail page's "Network", you can view all Pod network performance monitoring data information within the workspace. For more details, refer to the documentation Pod Network.

Added Quick Filter "Reverse Selection" and "Reset" Functions in Viewers¶

In any viewer's "Quick Filter", when selecting field filter content, you can perform "Reverse Selection" or "Reset". "Reverse Selection" means the selected field filter content is not displayed (clicking "Reverse Selection" again returns to the selected state), and "Reset" clears the filter conditions. For more details, refer to Log Viewer Quick Filter.

Optimized Log Blacklist¶

Guance supports filtering out logs that meet certain conditions by setting up a log blacklist, meaning that after configuring the log blacklist, logs that meet the criteria will no longer be reported to Guance workspace, helping users save on log data storage costs.

Log blacklist optimization supports matching all log sources, supporting two blacklist configuration methods:

• Trigger blacklist filtering if any one of the filtering conditions is met
• Trigger blacklist filtering if all filtering conditions are met

For more blacklist configurations, refer to the documentation Log Blacklist.

Other Functional Optimizations¶

• Added span quantity statistics in trace detail pages
• Optimized timeline drawing method for traces associated with hosts
• Optimized overview chart time slicing, canceling options; if previous overview charts had time slicing enabled, after optimization, it defaults to disabling time slicing
• Optimized browser scaling scenarios where entering edit mode prevented switching between combined chart queries for different charts
• Optimized manual pause of page refresh in log viewers, preventing automatic refresh when scrolling to the top

DataKit Updates¶

• Increased memory limits for runtime on host installations, supporting memory limit configuration during installation,
• CPU collector added load5s metric
• Supports Guance optimized log blacklist functionality, adjusting monitor layout, adding blacklist filtering situation display
• DaemonSet installation added Helm support, added DaemonSet installation best practices
• eBPF added HTTP protocol collection, host installations no longer install eBPF collectors by default (specific installation instructions required), DaemonSet installations unaffected

More DataKit updates can be referenced in DataKit Version History.

Guance Mobile App Updates¶

Added site login capabilities, optimized scene and event viewers, maintaining the same access experience as the web version.

• Supports users selecting accounts corresponding to sites, logging in via account password or verification code.
• Supports users viewing all or any source log data
• Supports users viewing all dashboards under the current space and switching via dropdown menus among "All Dashboards", "My Favorites", "Imported Projects", "My Creations", and "Frequently Browsed" to quickly filter and find corresponding dashboards.
• Supports users viewing all unresolved event contents triggered by anomaly detection libraries in the Event viewer via "All"; via "Mine" events, view unresolved events notified to users via email, DingTalk bots, Enterprise WeChat bots, Webhooks, etc.

Best Practices Updates¶

• Guance Tips
  • Multi-microservice project performance observability practice
  • ddtrace advanced usage
  • Mapping DataKit service using ExternalName in Kubernetes clusters
• Access (Integration) Best Practices
  • Best Practices for OpenTelemetry Trace Data Access
• Microservices Observability Best Practices
  • Best Practices for Achieving Microservices Observability Based on Alibaba Cloud ASM

For more best practice updates, refer to Best Practice Version History.

Integration Template Updates¶

Added Alibaba Cloud PolarDB Oracle Integration Documents, Views, and Monitors¶

Alibaba Cloud PolarDB Oracle indicator display includes CPU usage rate, memory usage rate, network traffic, connection count, IOPS, TPS, data disk size, etc.

Added Alibaba Cloud PolarDB PostgreSQL Integration Documents, Views, and Monitors¶

Alibaba Cloud PolarDB PostgreSQL indicator display includes CPU usage rate, memory usage rate, network traffic, connection count, IOPS, TPS, data disk size, etc.

Added Alibaba Cloud RDS SQLServer Integration Documents, Views, and Detection Libraries¶

Alibaba Cloud RDS SQLServer indicator display includes CPU usage rate, disk usage rate, IOPS, network bandwidth, TPS, QPS, etc.

Added DataKit Integration Documents, Views, and Monitors¶

DataKit performance indicator display includes CPU usage rate, memory information, running time, log records, etc.

Added Nacos Integration Documents, Views¶

Nacos performance indicator display: Nacos online duration, Nacos config long link count, Nacos config count, Service Count, http request count, etc.

March 24, 2022¶

Guance Site Updates¶

Guance supports multi-site login and registration, adding the "Overseas Zone 1 (Oregon)" site. The original "China Zone 1 (Alibaba Cloud)" changed to "China Zone 1 (Hangzhou)", and "China Zone 2 (AWS)" changed to "China Zone 2 (Ningxia)".

Accounts and data between different sites are independent, unable to share or migrate data mutually. You can choose a suitable site for registration and login based on your resource usage situation. Currently, Guance supports the following three sites. For how to select a site, refer to the documentation Guance Site Instructions.

Guance Updates¶

Added Workspace Data Authorization¶

Guance supports authorizing multiple workspace data to the current workspace via data authorization, querying and displaying through the chart components of the scenario dashboard and notes. If there are multiple workspaces, after configuring data authorization, you can view data from all workspaces in one workspace. For more configuration details, refer to the documentation Data Authorization.

1. Configure the workspaces whose data needs to be authorized in "Management" - "Data Authorization".

2. After obtaining data authorization in the workspace, open "Scenarios" - "Dashboard or Notes", select the chart component, and in the "Settings" section, choose the authorized workspace under "Workspace". Then, you can view and analyze data from the authorized workspace via Chart Queries.

Added Saving Online Pipeline Sample Test Data¶

Guance Pipeline supports both custom and official library types:

• After writing custom Pipeline script rules, you can input log sample data for testing to verify whether your configured parsing rules are correct. After saving the custom Pipeline, the log sample test data is synchronized.
• The official Pipeline library comes with multiple log sample test data. Before "Cloning," you can select the log sample test data that fits your needs. After modifying and saving the cloned Pipeline, the log sample test data is synchronized.

For more details about online Pipeline functions, refer to the documentation Pipelines.

Optimized Resource Directory Viewer¶

In Guance workspace, through "Infrastructure" - "Custom" - "Add Object Classification", you can create new object classifications and resource directory classification names and object fields.

After adding the resource directory classification, you can report custom data via the Func Function Processing Platform. For how to report data to Guance workspace via Func, refer to the documentation Resource Directory Data Reporting.

Optimized Snapshot Sharing Supporting Permanently Valid Links¶

Snapshot sharing supports setting validity periods, choosing either "48 Hours" or "Permanently Valid." In the snapshot list, click the share button to configure advanced settings like "Hide Top Bar" in the pop-up dialog box. For more snapshot sharing details, refer to the documentation Snapshots.

Note: Permanently valid sharing can easily pose data security risks, so please use with caution.

Optimized Chart Time Intervals¶

In the chart settings of the scenario dashboard, selecting "Auto Align" for the time interval, during preview, an option for time intervals appears in the top-right corner of the chart, allowing you to choose the time interval based on your actual situation to view your data.

Optimized Process, Application Performance, User Access Detection Data Gap Trigger Strategies¶

In Guance monitoring functional modules, when configuring process anomaly detection, application performance indicator detection, and user access indicator detection monitors, the data gap status supports three configurations: "Trigger Data Gap Events", "Trigger Recovery Events", "Do Not Trigger Events", requiring manual configuration of data gap handling strategies.

Other Functional Optimizations¶

• Optimized DataKit, Func Installation Guide Pages
• Optimized full display of single log in the log viewer
• Added NOT combination association search in viewers
• Optimized member permission display editing

DataKit Updates¶

• Added DataKit command-line completion functionality, helping you with command prompts and parameter completions during terminal operations
• Allowed upgrading DataKit to non-stable versions to experience the latest experimental features. If you are in a production environment, please upgrade with caution
• Preliminarily supported data collection for Kubernetes/Containerd architectures
• Network probing added TCP/UDP/ICMP/Websocket several protocol supports
• Adjusted Remote Pipeline storage in DataKit locally, avoiding file name case issues caused by different file systems
• Pipeline added decode() function, which can avoid configuring encoding in the log collector by achieving encoding conversion in Pipeline; add_pattern() increased scope management

More DataKit updates can be referenced in DataKit Version History.

Best Practices Updates¶

• Scenario Best Practices
  • RUM Data Reporting DataKit Cluster Best Practices
• Log Best Practices
  • Pod Log Collection Best Practices

More best practice updates can be referenced in Best Practice Version History.

Integration Template Updates¶

Added Alibaba Cloud PolarDB Mysql Integration Documents, Views, and Detection Libraries¶

Alibaba Cloud PolarDB Mysql indicator display includes CPU usage rate, memory hit rate, network traffic, connection count, QPS, TPS, read-only node delay, etc.

March 10, 2022¶

Guance Billing Update¶

Added Guance Billing Prepaid Cards¶

Guance prepaid cards support purchasing via account cash balance, applicable to all Guance consumption modes, including pay-as-you-go and annual packages. Log into Guance Billing Center, click "Manage Prepaid Cards" to enter the prepaid card management page for purchase. After purchasing and paying for the prepaid card, invoices are issued according to the actual payment amount. For more details, refer to Prepaid Card Management.

Guance Update¶

Added User Access Monitoring Resource (resource), Action (action), Long Task (long_task), Error (error) Viewers¶

User access monitoring viewers help you view and analyze detailed information about user access to applications. In Guance workspace, opening "User Access Monitoring", clicking on any application allows you to understand each user session, page performance, resources, long tasks, errors, and delays affecting users through the "Viewer", helping you fully understand and improve the operational status and usage of the application, enhancing user experience.

Guance user access monitoring viewers include session (session), view (page), resource (resource), action (action), long_task (long task), error (error).

Added Pod Network Details and Network Distribution¶

Pod network supports viewing network traffic between Pods. It supports viewing source IP to target IP network traffic and data connection situations based on IP/port, visualizing them in real-time to help enterprises understand the network operating status of their business systems, quickly analyzing, tracking, and locating fault problems, preventing or avoiding business problems caused by network performance degradation or interruptions.

After successful Pod network data collection, it is reported to the Guance console. In "Infrastructure" - "Container" - "Pod" detail page's "Network", you can view all Pod network performance monitoring data information within the workspace. For more details, refer to Pod Network.

In "Infrastructure" - "Container" - "Pod", clicking the small icon of the network distribution diagram in the top-left corner switches to viewing Pod network distribution. In the "Network Distribution Diagram", you can visually query network traffic between Pods in the current workspace, quickly analyzing TCP latency, TCP fluctuations, TCP retransmission counts, TCP connection counts, and TCP closure counts between different Pods. For more details, refer to Pod Network Distribution Diagram.

DataKit Updates¶

• DataKit collector added support for SkyWalking, Jaeger, Zipkin data configuration sampling strategies.
• DataKit collector added support for OpenTelemetry data access.
• DataKit documentation library added an introduction to overall log collection, including four methods: obtaining logs from disk files, acquiring logs by calling environment APIs, remotely pushing logs to DataKit, and Sidecar-style log collection.

Breaking Changes¶

2022/03/22

• Significant adjustments were made to Tracing data collection, involving several incompatible aspects:
  • DDtrace original conf-configured ignore_resources field needs to be changed to close_resource, and the field type changes from the original array ([...]) form to dictionary array (map[string][...]) form
  • Original data tags type in DDTrace changed to source_type

2022/03/04

• If the old version of DataKit had RUM functionality enabled, after upgrading, you need to reinstall the IP library, as the old version's IP library will no longer function.

2021/12/30

• Old versions of DataKit via datakit --version no longer push new upgrade commands. Directly use the following commands:

• Linux/Mac:

DK_UPGRADE=1 bash -c "$(curl -L https://static.guance.com/datakit/install.sh)"

• Windows

$env:DK_UPGRADE="1"; Set-ExecutionPolicy Bypass -scope Process -Force; Import-Module bitstransfer; start-bitstransfer -source https://static.guance.com/datakit/install.ps1 -destination .install.ps1; powershell .install.ps1;

More DataKit updates can be referenced in DataKit Version History.

SDK Updates¶

User access monitoring compatible with Opentracing protocol trace tracking tools, Web, Mini Program, Android, iOS SDK supports data linkage of OTEL, SkyWalking, Jaeger, etc., trace tracking tools.

Best Practices Updates¶

• Custom Access Best Practices
  • Best Practices for Quickly Getting Started with Pythond Collector
  • Alibaba Cloud "Cloud Monitor Data" Integration Best Practices
• Log Best Practices
  • Logback Socket Log Collection Best Practices

More best practice updates can be referenced in Best Practice Version History.

Scene Template Updates¶

Added MySQL Database Viewer Template to Custom Scene Viewer¶

The custom scene viewer in Guance added a MySQL database viewer template, helping you quickly set up a MySQL log viewer. In Guance workspace "Scene" - "Viewer" - "Built-in Viewer Template", click "MySQL Viewer Template" to directly create a MySQL log viewer. If related logs have already been collected, you can view and analyze data through this log viewer.

Integration Template Updates¶

Added Host System EthTool Integration Documents and Views¶

EthTool indicators include network interface inbound/outbound traffic, inbound/outbound data packets, discarded data packets, etc.

Added Host System Conntrack Integration Documents and Views¶

Conntrack performance indicators include successfully searched entries, inserted packet counts, connection counts, etc.

February 22, 2022¶

Added Log Configuration Pipeline Script¶

Pipeline is used for log data parsing, defining parsing rules to cut diverse format logs into structured data meeting our requirements. Guance provides three log Pipeline text processing methods:

• DataKit: After installing DataKit on the server, configure DataKit's log collector and corresponding pipeline files in terminal tools to process text data;
• DCA: DataKit Control APP, a desktop client application for DataKit, requires installation first. After installation, you can view and edit DataKit's default log pipeline files and manually add pipeline files;
• Pipelines: Supports manually configuring and viewing log pipeline files in Guance workspace without logging into the DataKit server for operations.

Added IFrame Chart Component¶

Guance added the IFrame chart component, supporting configuration of https or http link addresses. You can directly input external addresses in the IFrame URL to view, or use template variables in the IFrame URL for viewing. More configuration details can be referenced in the documentation IFrame.

Added Event Details Historical Records, Associated SLO¶

Observation optimizes the basic attributes, status & trend, and associated events layout on the event details page, and adds historical records and associated SLOs, viewable by clicking the event name in the abnormal event list.

Added event history records, supporting viewing detection object hosts, abnormal/recovery times, and duration.

If SLO is configured in the monitor, you can view associated SLOs, including SLO name, achievement rate, remaining quota, target, etc.

Added Default Absolute Time Enabled When Saving Snapshots¶

Guance added default absolute time enabled when saving snapshots.

• If absolute time is selected when saving snapshots, the shared link will display the absolute time of the saved snapshot. For example, if you choose the last 15 minutes and open the snapshot link at 14:00, it will show the data of the previous absolute time;
• If absolute time is deselected when saving snapshots, the shared link will display the absolute time of the saved snapshot. For example, if you choose the last 15 minutes and open the snapshot link at 14:00, it will show the data from 13:45 ~ 14:00.

For more snapshot sharing details, refer to the documentation Snapshots.

Optimized Data Gap Trigger Event Configuration and Trigger Condition Unit Prompts for Monitors¶

Guance added three data gap status configurations: "Trigger Data Gap Events", "Trigger Recovery Events", "Do Not Trigger Events".

• Metric class data monitors require manual configuration of data gap handling strategies;
• Log class data monitors default to selecting the "Trigger Recovery Events" strategy, do not require data gap configuration, and obtain the data gap recovery event cycle from the "Normal" condition.

Optimized Chart Query Expression Calculation Units¶

Guance optimized the logic for chart query expression calculation units. If query A has a unit, the result of query A plus a number will also carry the same unit. For example, if A’s unit is KB, then A+100’s unit is also KB. For more details, refer to the documentation Chart Query.

Added "Time Series" Pay-As-You-Go Mode¶

Guance added a "Time Series" pay-as-you-go mode and optimized the "DataKit + Time Series" pay-as-you-go mode. Specific billing modes can be referenced in the documentation Pay-As-You-Go.

Other Optimization Features¶

• Chart query data sources for logs, application performance, security inspections, and networks support full selection (*);
• Chart query text, button styles, and text prompts optimized;
• Workspace operation buttons iconified, such as edit, delete, etc.;
• Other UI display optimizations

January 20, 2022¶

Added Open API and API Key Management¶

"Guance" supports obtaining and updating Guance workspace data by calling Open API interfaces. Before calling the API interface, you need to create an API Key as the authentication method. For more details, refer to the documentation API Key Management.

Added Metric Dictionary and Metric Unit Management¶

After collecting metric data, you can view all collected metric sets and their metrics and labels in Guance workspace's "Metric Dictionary" and manually set units for custom metric data. For more details, refer to the documentation Metric Dictionary.

• Metrics help you understand the overall availability of the system, such as server CPU usage, website load times, etc., combined with Guance provided logs and trace tracking, helping you quickly locate and resolve faults.
• Labels help you associate data. Guance supports uploading all metrics, logs, and trace data to the workspace, associating queries by tagging the collected data, allowing you to perform correlation analysis and discover and resolve potential risks.

Added Funnel Chart to Scenario Charts¶

Funnel charts are generally suitable for normative, long-term, and multi-stage process analyses. Comparing data across different stages can intuitively highlight issues. Additionally, funnel charts are suitable for website business flow analysis, displaying the final conversion rate of users from entering the website to making a purchase, as well as the conversion rate at each step.

Added Saving Dashboards to Built-in Views and Binding Label Data¶

After creating a dashboard view, you can click the "Settings" button and choose "Save to Built-in View" to save the dashboard view to the "User View" in Built-in Views.

When saving a dashboard view to Built-in Views, you can choose binding relationships, selecting label binding. After saving to Built-in Views, you can view the saved dashboard view in Guance workspace "Management" - "Built-in Views" - "User Views". Simultaneously, since the binding relationship label:* was set, you can view the bound built-in views on the host/container detail pages where "Label Attributes" have been set in infrastructure. For more details, refer to the documentation Save Dashboard as Built-in View.

Added Pod Network Details and Network Distribution¶

Pod network supports viewing network traffic between Pods. It supports viewing network traffic and data connections between source IP and destination IP based on IP/port, visualizing them in real-time to help businesses understand the network operating status of their business systems, quickly analyzing, tracking, and locating fault problems, preventing or avoiding business problems due to network performance degradation or interruption.

After successful Pod network data collection, it is reported to Guance console. In "Infrastructure" - "Container" - "Pod" detail page's "Network", you can view all Pod network performance monitoring data information within the workspace. For more details, refer to Pod Network.

In "Infrastructure" - "Container" - "Pod", clicking the small icon of the network distribution diagram in the top-left corner switches to viewing Pod network distribution. In the "Network Distribution Diagram", you can visually query network traffic between Pods in the current workspace, quickly analyzing TCP latency, TCP fluctuations, TCP retransmission counts, TCP connection counts, and TCP closure counts between different Pods. For more details, refer to Pod Network Distribution Diagram.

DataKit Updates¶

• DataKit collector added support for SkyWalking, Jaeger, Zipkin data configuration sampling strategies.
• DataKit collector added support for OpenTelemetry data access.
• DataKit documentation library added a document introducing overall log collection, including four methods: obtaining logs from disk files, acquiring logs by calling environment APIs, remote pushing logs to DataKit, and Sidecar-style log collection.

Breaking Changes¶

2022/03/22

• This update made significant adjustments to Tracing data collection, involving several incompatible aspects:
  • DDtrace original conf-configured ignore_resources field needs to be changed to close_resource, and the field type changes from the original array ([...]) form to dictionary array (map[string][...]) form
  • DDTrace original data tag type field changed to source_type

2022/03/04

• If the old version of DataKit had RUM functionality enabled, after upgrading, you need to reinstall the IP library, as the old version's IP library will no longer function.

2021/12/30

• The old version of DataKit via datakit --version no longer pushes new upgrade commands. Directly use the following commands:

• Linux/Mac:

DK_UPGRADE=1 bash -c "$(curl -L https://static.guance.com/datakit/install.sh)"

• Windows

$env:DK_UPGRADE="1"; Set-ExecutionPolicy Bypass -scope Process -Force; Import-Module bitstransfer; start-bitstransfer -source https://static.guance.com/datakit/install.ps1 -destination .install.ps1; powershell .install.ps1;

More DataKit updates can be referenced in DataKit Version History.

SDK Updates¶

User access monitoring is compatible with Opentracing protocol trace tracking tools. Web, Mini Program, Android, iOS SDKs support data linkage of OTEL, SkyWalking, Jaeger, etc., trace tracking tools.

Best Practices Updates¶

• Custom Access Best Practices
  • Quick Start Pythond Collector Best Practices
  • Alibaba Cloud “Cloud Monitor Data” Integration Best Practices
• Log Best Practices
  • Logback Socket Log Collection Best Practices

More best practice updates can be referenced in Best Practice Version History.

Scene Template Updates¶

Added MySQL Database Viewer Template to Custom Scene Viewer¶

The custom scene viewer in Guance added a MySQL database viewer template, helping you quickly set up a MySQL log viewer. In Guance workspace “Scene” - “Viewer” - “Built-in Viewer Template”, click “MySQL Viewer Template” to directly create a MySQL log viewer. If related logs have already been collected, you can directly view and analyze data through this log viewer.

Integration Template Updates¶

Added Host System EthTool Integration Documents and Views¶

EthTool indicators include network interface inbound/outbound traffic, inbound/outbound data packets, discarded data packets, etc.

Added Host System Conntrack Integration Documents and Views¶

Conntrack performance indicators include successful search entry counts, inserted packet counts, connection counts, etc.

February 22, 2022¶

Added Log Configuration Pipeline Script¶

Pipeline is used for log data parsing. By defining parsing rules, heterogeneous format logs are split into structured data that meets our requirements. Guance provides three log Pipeline text processing methods:

• DataKit: After installing DataKit on the server, configure DataKit's log collector and corresponding pipeline file in terminal tools to handle text data;
• DCA: DataKit Control APP, the desktop client application for DataKit, requires installation first. After installation, you can view and edit DataKit's default log pipeline files and manually add pipeline files;
• Pipelines: Supports manually configuring and viewing log pipeline files in Guance workspace without needing to log into the DataKit server for operations.

Added IFrame Chart Component¶

Guance added the IFrame chart component, supporting configuration of https or http link addresses. You can directly input external addresses in the IFrame URL to view, or use template variables in the IFrame URL for viewing. For more configuration details, refer to the documentation IFrame.

Added Event Detail Historical Records, Associated SLO¶

Observation optimized the layout of basic attributes, status & trends, and associated events on the event detail page, and added historical records and associated SLOs. Clicking the event name in the anomaly event list allows you to view.

Added event historical records, supporting viewing detection object hosts, anomaly/recovery timesand duration.

If SLO is configured in the monitor, you can view associated SLOs, including SLO name, achievement rate, remaining quota, target, etc.

Added Default Absolute Time Enabled When Saving Snapshots¶

Guance added default absolute time enabled when saving snapshots.

• If absolute time is selected when saving snapshots, the shared link will display the absolute time of the saved snapshot. For example, if you choose the last 15 minutes and open the snapshot link at 14:00, it will show the data of the previous absolute time;
• If absolute time is deselected when saving snapshots, the shared link will display the absolute time of the saved snapshot. For example, if you choose the last 15 minutes and open the snapshot link at 14:00, it will show the data from 13:45 ~ 14:00.

For more snapshot sharing details, refer to the documentation Snapshots.

Optimized Monitor Data Gap Trigger Event Configuration and Trigger Condition Unit Prompts¶

Guance added three data gap status configurations "Trigger Data Gap Events", "Trigger Recovery Events", "Do Not Trigger Events".

• Metric type data monitors require manual configuration of data gap handling strategies;
• Log type data monitors default to selecting the "Trigger Recovery Events" strategy and do not require data gap configuration, obtaining the cycle for "Normal" conditions as the data gap recovery event cycle;

Optimized Chart Query Expression Calculation Units¶

Guance optimized the logic for chart query expression calculation units. If query A has a unit, the result of query A with numbers also carries the same unit. For example, if A's unit is KB, then A+100’s unit is also KB. For more details, refer to the documentation Chart Query.

Added "Time Series" Pay-As-You-Go Mode¶

Guance added a "Time Series" pay-as-you-go mode and optimized the "DataKit + Time Series" pay-as-you-go mode. Specific billing modes can be referenced in the documentation Pay-As-You-Go.

Other Optimization Features¶

• Chart query data sources for logs, application performance, security inspections, and networks support full selection (*);
• Chart query text, button styles, and text prompts optimized;
• Workspace operation buttons iconified, such as edit, delete, etc.;
• Other UI display optimizations

January 20, 2022¶

Added Open API and API Key Management¶

"Guance" supports obtaining and updating Guance workspace data by calling Open API interfaces. Before calling API interfaces, you need to create an API Key as the authentication method. For more details, refer to the documentation API Key Management.

Added Metric Dictionary and Metric Unit Management¶

After collecting metric data, you can view all collected metric sets and their metrics and tags in Guance workspace's "Metric Dictionary" and manually set units for custom metric data. For more details, refer to the documentation Metric Dictionary.

• Metrics help you understand the overall availability of the system, such as server CPU usage, website load times, etc., combined with Guance provided logs and trace tracking, helping you quickly locate and resolve faults.
• Tags help you associate data. Guance supports uploading all metrics, logs, and trace data to the workspace, associating queries by tagging the collected data, allowing you to perform correlation analysis and discover and resolve potential risks.

Added Funnel Chart to Scenario Charts¶

Funnel charts are generally suitable for normative, long-term, multi-stage process analyses. Comparing data across different stages can intuitively highlight issues. Additionally, funnel charts are suitable for website business flow analysis, displaying the final conversion rate of users from entering the website to making a purchase, as well as the conversion rate at each step.

Added Saving Dashboards to Built-in Views and Binding Label Data¶

After creating dashboard views, you can click the "Settings" button and choose "Save to Built-in View" to save the dashboard view to the "User View" in Built-in Views.

When saving dashboard views to Built-in Views, you can select binding relationships, choosing label binding. After saving to Built-in Views, you can view the saved dashboard view in Guance workspace "Management" - "Built-in Views" - "User Views". Simultaneously, since the binding relationship label:* was set, you can view the bound built-in views on host/container detail pages where "Label Attributes" have been set in infrastructure. For more details, refer to the documentation Save Dashboard as Built-in View.

Added Container Detail Page Associated Pod¶

In the container detail page, you can view basic information and performance indicator statuses within the selected time component range related to Pods (associated fields: pod_name). For more container-related queries, refer to the documentation Container.
Note: To view related Pods in the container details, the field "pod_name" needs to match; otherwise, you cannot view the related Pod page in the container details.

Added Monitor Group Management¶

Guance added grouping functionality supporting meaningful monitor combinations during setup, allowing filtering of corresponding monitors via "Groups", facilitating group management of various monitors.

Note: - Each monitor must select a group during creation, defaulting to "Default Group"; - When a group is deleted, monitors under that group will automatically be categorized under the "Default Group".

Added Log Viewer, Table Chart, Log Stream Chart Formatting Configuration¶

Guance added formatting configurations allowing you to hide sensitive log content or highlight necessary log content, and perform quick filtering by replacing original log content. Supports configuring formatting in the log viewer, table chart, and log stream chart.

Optimized Silence Management, Added Disable/Enable Rules¶

Guance added disable/enable functionality for silence rules, helping you quickly disable/enable silence tasks. For more details, refer to the documentation Silence Management.

• Enable: Silence rules follow the normal process
• Disable: Silence rules are ineffective; if silence notifications are set and the "xx minutes before start" notification action has not been executed, the notification will not execute

Note: Both enabling/disabling rules generate operational audit events, which can be viewed in Guance workspace "Management" - "Basic Settings" under operational audits.

Added Log Pipeline User Manual¶

Guance added a user manual for log pipelines, helping you understand how to use DataKit's built-in debugging tools to assist in writing Pipeline scripts.

# Debugging script example
datakit --pl datakit.p --txt '2022-01-12T18:40:51.962+0800 WARN diskio diskio/input.go:320 Error gathering disk info: open /run/udev/data/b252:1: no such file or directory'

# Successful extraction example
Extracted data(drop: false, cost: 3.108038ms):
{
  "code": "diskio/input.go:320",
  "level": "WARN",
  "message": "2022-01-12T18:40:51.962+0800 WARN diskio diskio/input.go:320 Error gathering disk info: open /run/udev/data/b252:1: no such file or directory",
  "module": "diskio",
  "msg": "Error gathering disk info: open /run/udev/data/b252:1: no such file or directory",
  "time": 1641984051962000000
}

Added Outer Functions to DQL¶

Guance added two outer functions rate() and irate().

• rate(): Calculates the average change rate of a certain metric over a specific time range. Suitable for alarms and slowly moving counters.
• irate(): Calculates the instantaneous change rate of a certain metric over a specific time range, suitable for drawing volatile, fast-changing counters.

For more details, refer to the documentation DQL Outer Functions.