Update Log (2021)¶
This document records the update content description of each online release of Guance, including DataKit, Guance best practices, Guance integration documentation, and Guance.
December 30, 2021¶
Optimization of binding built-in views¶
"Guance" supports associating built-in views with different link services, applications, log sources, projects, etc., by binding fields. According to the bound fields, you can view the associated built-in views (system views, user views) on the corresponding Explorer detail pages. Supported Explorers include custom scene Explorers, infrastructure, logs, links, user access, security checks, availability, etc.
Optimization of log Explorer¶
- In the log Explorer, the left source list is expanded by default when logging in for the first time and can be manually collapsed. The Explorer will remember the last state by default.
- In the log Explorer, click the settings button on the left to manually add filter fields for quick filtering.
- In the log data list, you can use "hovering the mouse" over the log content to expand and view all the log content. Clicking the "copy" button copies the entire log content to the clipboard. When expanding, if possible, the system will format the log JSON; otherwise, it displays the log content normally.
Optimization of associated display on Explorer detail pages¶
The associated display on the log, infrastructure, link, security check Explorer detail pages has been optimized. This includes associated queries for hosts, metrics, links, containers, logs, Pods, etc. Two types of associated displays are supported: fixed display and non-fixed display (which requires displaying based on whether the associated field is included).
Take the log Explorer detail page's associated host as an example. The associated fields are "host", and in the log details, you need to match the "host" field to view related hosts; otherwise, you cannot see the related host page in the log details. After matching the fields, you can view the basic information and metric performance status of the host under the associated host.
-
Attribute view: includes basic host information, integration operation status. If cloud host collection is enabled, you can also view cloud vendor information.
-
Metric view: You can view the performance metric views of related hosts within the default 15 minutes. Click "Open this view" to go to the built-in view, where you can customize and modify the host view via cloning and save it as a user view. The user view can be viewed by binding it on the log detail page.
Optimization of snapshot sharing¶
Snapshot sharing supports hiding the top bar of the shared page. In the snapshot list, click the share button, and in the pop-up dialog box, you can set advanced options like "hide the top bar".
Optimization of chart query result showing thousand separators¶
Guance dashboard chart query results support automatically adding thousand separators to the data display. If a unit is set, the data format will follow the set unit, and after setting, you can preview the result.
Other optimization features¶
- Optimized performance of the host network topology diagram query, added target domain name display in the network topology diagram node. That is, if the target host is not in the current workspace but the target domain name exists and the port of the target domain name is less than 10000, then the target domain name will be displayed in the topology diagram.
- Optimized logic for pinning favorite dashboards and notes:
- Unfavorited dashboards are sorted in reverse order of creation time
- Favorited dashboards are sorted in reverse order of favoriting time
- Favorited dashboards are displayed above unfavorited dashboards
- Optimized cascading query logic for view variables, i.e., when two variables are linked, selecting
*
in the previous one does not filter the linked variable, showing all value options. - Added help documentation links to the collapsed navigation bar on the left.
December 16, 2021¶
New similarity metric query for time series charts¶
In analysis mode of the time series chart, you can select "chart query" as trend lines/columns for metric queries and view similar trend metrics. By selecting an absolute time range, you can query similar metric trends within that space.
New associated data statistics display on Explorer detail pages¶
Added data volume statistics for related logs, links, containers, etc., in the Explorer detail pages. These statistics can be directly obtained in the tab titles of the Explorer detail pages, including infrastructure, logs, application performance, user access, security checks, etc.
New export function for Explorer charts¶
Guance supports exporting any chart from the Guance space Explorer to a dashboard, note, or clipboard.
New Explorer search match¶
Guance supports wildcard searches in the Explorer search bar using formats like "host:cc*", which can be used for filtering log data of hosts with names starting similarly. Additionally, a "delete" button has been added on the right side of the Explorer search bar to support one-click deletion of entered text.
Optimization of log Explorer and log-related data query¶
- Through the "container" below the log details page in "Guance", you can view all container (CONTAINERS) data related to the log's host within the last ten minutes.
- Logs related to trace and container data support filtering by "trace_id" or "host".
- Log details content supports two viewing modes: Json and text, automatically displaying format based on the message type.
- Through approximate text Explorer, you can view the details of associated logs after Pattern clustering.
- Hover tips for log Explorer have removed colors.
Optimization of silence strategy, supporting setting silence for monitoring rules¶
Guance allows users to manage all silence rules for the current space through "Silence Management" in the "Monitoring" feature. "Guance" provides three types of silences: host silence, monitor silence, and group silence, supporting silence management for different hosts, monitors, and groups, ensuring that silent objects do not send alarm notifications to any notification targets during the silence period.
Optimization of monitor management¶
Duplicate monitors in the monitor list may affect users' quick location of monitoring objects. "Guance" supports choosing whether to create duplicate monitors from templates. Users can also customize the export/deletion of monitors through the newly added "Batch Management" tool.
Optimization of view variable query display¶
- Default values have been added to dashboard view variables, allowing previews of all variables in the current view variable and selection of initial variables to be viewed by default on the dashboard.
- Dragging view variables with mouse hover (hover) to adjust their order is supported.
- Removed the "Settings" button for view variables, allowing direct use of "Sorting", "Hiding", and "Deleting" functions in the view variable list.
Optimization of locked time settings for dashboards and notes¶
Guance supports adding "More" options to customize locked time ranges for locked time settings on dashboards and notes. After this optimization, the locked time component and chart component functions remain consistent.
Optimization of Help Center entry¶
In the Guance Help Center, you can quickly view product documentation, product updates, blogs, live review sessions, and more. You can also join our official service group by scanning a code.
December 2, 2021¶
New owner member role¶
Guance has added an owner role for the current workspace. Currently, four workspace member permissions are supported, including "owner", "administrator", "standard member", and "read-only member", respectively constraining the management, configuration, operation, and viewing permissions of different member categories. The "owner" role has the highest operation permission, able to designate the "administrator" for the current space and perform any management, configuration, operation, and data browsing. In Guance workspace's "Manage"-"Member Management"-"Edit", you can edit and update member permissions.
Scene adds custom Explorer¶
"Guance" provides a quick-to-build Explorer based on the log data range in the scene, supporting space members to jointly build Explorers based on custom scenes and customized viewing needs. Completed "Explorers" can be exported and shared with others, sharing the Explorer template.
New export of user view to dashboard¶
"Guance" supports exporting user-created views from the "Built-in View" to a JSON file, which can be used for importing scenarios or built-in views into different workspaces.
Optimization of host network observability, new network flow data view list¶
"Guance" supports displaying different visual charts based on selected protocols in the "Network" section of the "Host" details page in the infrastructure section, and customizing the display fields of the network flow data list.
By clicking "View Network Flow Data", you can view network flow data from the last two days by default, including time, source IP/port, destination IP/port, source host, transmission direction, protocol, bytes sent, etc. Custom display fields or adding filter conditions are supported, filtering all string-type keyword fields.
New sourcemap feature¶
When releasing applications in production environments, to prevent issues such as code leakage, files are generally transformed and compressed during packaging processes. Although these measures ensure code security, they also result in error stack information being obfuscated, making it difficult to directly locate problems, complicating subsequent Bug troubleshooting.
To address these issues, "Guance" provides sourcemap functionality for web applications, enabling the deobfuscation of code for easier error identification and faster problem resolution.
New security inspection data analysis for host association¶
"Guance" supports viewing security inspection data related to the host within the past day and performing keyword searches, multi-tag filtering, and data sorting on these security inspection data.
New installation guide for DataKit collector on Kubernetes¶
"Guance" supports adding a guide for installing the DataKit collector on a Kubernetes cluster in the workspace "Integration"-"DataKit".
Optimization of user access monitoring associated link queries¶
"Guance" supports adding attributes to the current filter or copying them (copying the tag to the local clipboard) on the link details page and user access performance details page (user access monitoring Viewer).
Optimization of registration upgrade process¶
Guance has added an option to directly choose to activate the Guance billing platform during account registration and optimized the upgrade process to the agile version after logging into the workspace,
Price reduction adjustment¶
Guance has adopted the latest technology to optimize data storage costs, hence reducing pricing and introducing a tiered billing model based on data retention strategies, including log data, APM Trace, and RUM PV billing items.
Additionally, to assist enterprises in observing IT infrastructure and application systems comprehensively at more favorable prices, Guance has launched three packages based on different enterprise development stages: startup acceleration package, entrepreneurship development package, and enterprise standard package, along with stackable traffic packages for enterprises to use according to their own needs and stacking with packages.
Other optimization features¶
- User access monitoring adds LCP, FID, CLS, FCP related detection indicators. In the Guance workspace "Monitoring"-Monitors, click "+Create Monitor", select "User Access Indicator Detection", and enter the configuration page for detection rules.
- Simple queries in Monitors change grouping to detection dimensions, which determine the dimension on which detection rules trigger, i.e., the triggering object. "Guance" supports adding multiple detection dimensions, and triggers an alert if any detection dimension's metric meets the alarm condition.
- Navigation menu display optimization, the navigation bar in the Guance workspace collapses with secondary menus displayed flush with the top for navigation names.
November 18, 2021¶
New SSO login¶
Guance supports users creating identity providers within the workspace. During login, users input their company email to obtain the corresponding SSO login link, click the SSO login link to achieve the corresponding authentication login. In the Guance workspace "Management"-"SSO Management"-"Create Identity Provider", you can set up SSO single sign-on for employees.
New SLO charts on dashboards and notes¶
SLO charts have been added to scenario dashboards and notes. SLO charts can directly select and display configured monitored SLOs. Selecting different SLO names synchronously displays SLO data results.
New custom addition of grouping fields for infrastructure¶
In Guance infrastructure Explorer, click the settings button next to the grouping, and you can custom add grouping tags. After adding, you can choose to group and aggregate display according to the added grouping tags.
Note: Only administrators can add custom grouping tags.
New SLO notification silence configuration¶
If an event is not very urgent but has a high frequency of alarm notifications, you can reduce the notification frequency by setting notification silence. After setting notification silence, events will continue to occur, but notifications will no longer be sent, and the generated events will be stored in event management.
Chart link optimization¶
Guance supports adding built-in links and custom links to charts. Through links, you can jump from the current chart to the target page and pass data information by modifying the corresponding variable values in the link using template variables, completing data linkage. Chart links support opening from a new page, current page, or slide-over detail page, achieving linked analysis.
Price reduction adjustment¶
- Daily DataKit price reduced to 3 yuan per unit
- Availability API testing adjusted to 1 yuan per 10,000 tests
Other optimization features¶
- Launch free new version in China region 1 (Alibaba Cloud): Free Plan split into Alibaba Cloud login and AWS login. For Alibaba Cloud login, timeline data retention policy is 30 days, other data retention policy is 7 days; for AWS login, all data retention policies are 1 day;
- Optimize SLO deduction logic: Monitors in disabled status will not be included in the deduction scope;
- Add dashboard label feature: When creating/modifying a dashboard view, you can add custom labels to dashboards to help with quick filtering;
- Optimize log highlight display: When searching logs in the search bar, only the matched keywords in the returned list are highlighted;
- Optimize Explorer search: Add help documentation links in the Explorer search dropdown;
- Optimize Markdown format for text charts: Add support for unordered lists and ordered lists, optimizing display format.
November 4, 2021¶
New scene notes¶
In the scene, you can create multiple notes for summary reports, supporting the insertion of real-time visual charts for data analysis, inserting text documents for explanations, combining charts and documents for data analysis and summary reports; supporting the export of infrastructure, logs, application performance, user access, etc., Explorer data to notes for data analysis; supporting sharing notes with all workspace members, retaining anomaly data analysis, helping with retrospection, location, and problem-solving.
Scene view upgraded to dashboard¶
Guance's scene view has been upgraded to a dashboard, and adjustments have been made to the dashboard functions and layout. In the scene, you can create multiple dashboards to build data insight scenarios, support searching dashboards via keywords, support adding visualization charts to dashboards for data analysis, and support quickly filtering and finding corresponding dashboards from "My Favorites", "Import Projects", "My Creations", and "Frequently Browsed".
Export to dashboard and notes added to Explorer¶
Export to dashboard and notes have been added to infrastructure, log monitoring, application performance monitoring, availability monitoring, and security inspection Explorers.
Exception detection library upgraded to monitoring¶
The original "exception detection library" has been upgraded to "monitoring", and the names of "exception detection library" and "detection rules" have changed. Specific changes are as follows:
- Monitors (originally referred to as "anomaly detection rules") provide "threshold detection", "log detection", and various other monitoring methods, allowing users to configure detection rules and trigger conditions themselves and receive alarm notifications promptly.
- Templates (originally referred to as "built-in detection libraries"), Guance includes various ready-to-use availability monitoring templates, supporting one-click creation of Docker, Elasticsearch, Host, Redis monitors. After successfully creating a template, the corresponding official monitors are automatically added to the current workspace.
- Groups (originally referred to as "custom detection libraries"), the grouping function supports you to create meaningful monitor combinations, facilitating grouped management of various monitors.
New monitoring SLO function, supports export to dashboard
¶
"Monitoring" has added SLO monitoring, supporting scoring the quality of service provided by system service providers (Providers) in the cloud era, comparing the corresponding SLI (Service Level Indicator) against target needs. Additionally, "Guance" supports exporting SLO as a view to the dashboard for synchronized SLO monitoring on the dashboard.
Other optimization features¶
- Chart optimization:
- Overview chart removes group selection;
- Time series, pie, and bar chart style adjustments;
- Time slicing adds tooltip information;
- Chart json supports editing, linked with query/setting, validates input json, shows error messages if errors exist;
- Style optimization in analysis mode of time series charts;
- Billing added prompt for paid plans, choose settlement method aliyun or aws, bill list directly shows jump to corresponding cloud service console to view bill information.
October 21, 2021¶
New host network distribution chart¶
A new host network distribution chart has been added, providing infrastructure network observability. In "Infrastructure"-"Host", click the small icon in the upper left corner to switch to view the host network distribution situation. In "Network Distribution Chart", you can visually query the network traffic between hosts in the current workspace, quickly analyzing TCP latency, TCP fluctuations, TCP retransmission counts, TCP connection counts, and TCP close counts between different hosts.
New tracking function for user access monitoring¶
A new "tracking" function has been added to user access monitoring. It supports users creating tracking tasks via "user access monitoring" to monitor custom link traces in real time. By pre-setting link traces, you can filter link data centrally, accurately query user access experiences, and promptly discover vulnerabilities, anomalies, and risks.
New JSON format query for scene charts¶
When editing charts in "Scene", each correct chart query corresponds to a JSON text. It supports parsing between JSON texts and chart queries within the workspace, helping you understand the chart drawing details.
DCA desktop client application optimization¶
UI display of the DCA desktop client has been optimized, adding host status.
- online: indicates normal data reporting, allowing DCA to view the running status and configuration of DataKit;
- unknown: indicates remote management configuration is not enabled, or not within the same local area network;
- offline: indicates the host has not reported data for over 10 minutes, or the host name has been modified. If the host has not reported data normally for over 24 hours, the host record will be removed from the list.
Event-host correlation query and detection rule query optimization¶
The event module has been optimized to support correlated host queries, allowing real-time monitoring of logs, containers, processes, links, and inspection events related to the event within the selected time component range. It also supports viewing the associated detection rules with one click and querying other triggered events under the corresponding detection rules.
Adjustment of billing method¶
- Choose billing settlement method based on node login method, including CloudCare account, Aliyun, AWS. Among them, CloudCare account settlement is the general method, Aliyun login nodes can choose Aliyun account settlement method (under implementation), AWS login nodes can choose AWS account settlement method (already implemented).
- Removed Session count and Browser test count billing dimensions, removed Session count and Browser test count statistics and views in "Paid Plans and Billing", added security inspection count statistics graph.
Other optimization features¶
- Optimized scene chart addition function, removed the "Add Chart" button in "Edit" mode, all addable chart types are selectable at the top of the scene;
- Optimized scene chart grouping function, supports deleting groups only and keeping existing charts entering the default group;
- Added regional ranking switch for China map/World map, default off regional ranking;
- Optimized Explorer query method, merged search and filter bars, supports keyword search, label field filtering, and associated search based on labels, fields, and text.
September 28, 2021¶
New host network performance monitoring¶
Host network performance monitoring supports viewing network traffic between host services, containers, and any tagged infrastructure. It supports viewing source host to destination network traffic and data connections based on IP/port, and real-time visualization helps businesses understand the operational status of their business systems' networks, analyze, track, and locate fault problems, preventing or avoiding business issues caused by network performance degradation or interruption.
After successful collection of host network data, it is reported to the Guance console. In the "network" section of the "Host" details page under "Infrastructure", you can view all network performance monitoring data information within the workspace.
New chart linking queries¶
In the scene view, you can view the DQL query statement, associated links, etc., of the chart by single-clicking the mouse, or perform associated queries for relevant logs, containers, processes, links, etc., under different grouping query tags. When the chart query contains host (host) information, you can view the related host monitoring view. For example, clicking any chart will allow you to view the DQL query statement, set links, and associated queries for related logs, containers, processes, links, host monitoring views, etc.
New rectangular tree map¶
The rectangular tree map is used to visualize the percentage distribution of metric data under different groups. The color of the rectangular tree map represents the group's tag (Tag), and the area represents the metric data. When multiple metric queries are added, you can choose the main displayed metric, and the size of the area is displayed based on the selected metric data results, Top/Bottom, and quantity. For example, this figure uses a visual way to display the CPU usage rate ranking of hosts.
New associated query for application performance and user access monitoring¶
Within the link detail of application performance monitoring, you can view associated (same trace_id) real user access experience data through the "Related Views" at the top of the page. This feature not only helps you view application performance data from proactive monitoring but also allows you to quickly gain insights into real user access situations. More details about application performance monitoring can be found in the Link Analysis document.
New absolute time range for infrastructure queries¶
Infrastructure has added absolute time ranges for "Host", "Container", "Process" queries, defaulting to displaying host data from the last 24 hours, container data from the last ten minutes, and process data from the last ten minutes, which can be refreshed manually. Hovering the mouse over an offline host allows you to view host offline handling prompts.
Optimization of chart comparison function¶
The chart comparison function has been optimized, supporting comparisons of time-series charts, overview charts, etc., with concurrent data. You can choose year-over-year (comparison of the same time point in adjacent time periods) or quarter-over-quarter (comparison of adjacent time periods). Based on the chart lock time, you can choose quarter-over-quarter, daily quarter-over-quarter, weekly quarter-over-quarter, monthly quarter-over-quarter, weekly year-over-year, monthly year-over-year, etc.
Optimization of SLIMIT in chart queries¶
By default, SLIMIT is added to grouped chart queries, used to limit the number of points or series returned by time-series charts. One chart limits returning 10,000 points, and one query returns up to 10 timelines.
Cloud dial tests adjusted to availability monitoring¶
Cloud dial tests are divided into API dial tests and Browser dial tests, adjusted to availability monitoring. After the adjustment, API dial tests are retained. Other functions such as overview analysis, Viewer analysis, self-built node functions remain unchanged. More details can be found in the availability monitoring document.
Price reduction adjustment¶
- Log price reduced from 1 yuan per 1 million to 0.5 yuan per 1 million
- User access monitoring price removed Session billing dimension, adjusted to 0.3 yuan per 1,000 PVs
For more details on pricing and data expiration strategies, please refer to the pay-as-you-go document.
September 9, 2021¶
DataFlux renamed to "Guance"¶
In the cloud computing era, observability is increasingly adopted and valued by more and more enterprises. As an observability platform for the cloud era launched by ZYUN Technology, DataFlux has been officially renamed to "Guance" to make it easier for enterprises and users to understand. The website is "guance.com".
At the same time, we have designed a new logo for "Guance", using letters C and G as the basis to form a radar-like wave graphic, representing detection and observation characteristics, fully fitting the features of observability, and reflecting the core safeguard system built by "Guance" for enterprises and users based on data.
New approximate text Viewer for logs¶
"Guance" supports similarity calculation analysis on the message
field of raw log data. Based on the selected time range in the upper right corner, it fixes the current time period and retrieves 10,000 log data entries for approximate analysis within that time period. It aggregates logs with high similarity, extracts and statistically analyzes common Patterns, helping quickly discover abnormal logs and locate problems.
Pattern Clustering Details
Clicking on the Pattern clustering list will draw out the current complete list of Pattern clusters. The returned data list is sorted in reverse chronological order, supporting the display of the latest 50 log data entries, which can be shown in 1 line, 3 lines, or 10 lines.
New Container Jobs and Cron Jobs Viewer¶
"Guance" has added a "Jobs" and "Cron Jobs" Viewer for containers. In the left object list under "Containers" in "Infrastructure", you can select the corresponding Viewer to view the running status and service capability metrics of Jobs and Cron Jobs in Kubernetes, thereby enabling real-time monitoring of Kubernetes clusters and deployed resources. Refer to the Containers document for more details.
New DCA Desktop Client Application¶
DCA, DataKit Control APP, is a desktop client application aimed at conveniently managing already installed and configured collectors. It supports viewing the integration list, configuration file management, Pipeline management, and online viewing of integration configuration documentation. You can remotely connect to DataKit through DCA, make online changes to collectors, and save updates to take effect immediately.
In the "Guance" workspace, click "Integrations"-"DCA" sequentially to download the installation package. After downloading, you can install it on your computer to remotely manage DataKit.
New Mobile APP¶
DataFlux Mobile helps you view log data, scene views, etc., from Guance on mobile devices anytime and anywhere, and receive all alarm events notified via email, DingTalk bot, Enterprise WeChat bot, Webhook, etc.
In the Guance workspace, you can download the DataFlux Mobile APP through "Integrations"-"Mobile".
New label filtering function for metrics, optimized three viewing modes for metrics¶
The DataKit collector will append the tag host=<DataKit host name>
by default to all collected data,
Tags are a set of properties identifying the data point collection object. Tags consist of tag names and tag values, and in the "Metrics" page, you can choose to view the corresponding metric view for the selected tag value. As shown in the figure below: filter the host in the tag bar and view its usage_system
, usage_total
, usage_user
, usage_iowait
metric views.
On the left side of the "Metrics" page, you can choose to switch the metric viewing mode, supporting three viewing modes: tile mode, mixed mode, list mode, with tile mode selected by default.
Anomaly detection optimization¶
The new window (window) function is used in anomaly detection rule queries and chart queries, meaning support for executing statistical calculations on each record with the selected time interval as the window (record set) and the detection frequency as the offset.
• When performing chart queries, support adding window functions, meaning using the selected time interval as the window (supports selecting 1 minute, 5 minutes, 15 minutes, 30 minutes, 1 hour, 3 hours), combined with aggregation functions to execute statistical calculations on each record.
• When configuring anomaly detection rule settings, the new window function is added as the plotting display area for query results, returning real-time anomaly detection metric data used to trigger alerts.
The new percent() function is used in the aggregation functions of anomaly rule queries and chart queries. Support adding the percent() function in aggregation functions, including p50 (median value), p75 (value at 75% position), p90 (value at 90% position), p99 (value at 99% position).
The new label filtering is used in anomaly rule queries and chart queries, supporting selecting host label properties for filtered display. Before selecting labels in the query, you need to set label properties for the host in "Infrastructure"-"Host".
Optimization of unresolved event display style¶
In the unresolved event list, you can preview the window function of the event for the last 6 hours:
• The dashed border display effect represents the time period affected by the anomaly event
• When the detection library rule type is threshold, log, application performance indicator, user access indicator detection, security inspection, anomalous process, cloud dial test detection, you can view related anomaly detection indicator data according to the color blocks corresponding to different alert levels, including critical, error, warning.
• When the detection library rule type is mutation, interval, water level, you can quickly identify the current event trigger time point based on the "vertical line" on the chart.
You can view the status distribution, DQL function, and window function line chart of anomaly events on the event details page.
• Status distribution: Displays the event status (critical, important, warning, data interruption) within the selected time range (default displays the last 6 hours)
• DQL query statement: Returns real-time indicator data based on the custom query statement of anomaly detection rules, default displaying the last 6 hours of real-time indicator data
• Window function: Based on anomaly detection rules, executes statistical calculations on each record using the selected time range as the window (record set) and the detection frequency as the offset, returning real-time anomaly detection indicator data used to trigger alerts. Defaults to displaying the last 6 hours of real-time anomaly detection indicator data
Optimization of scene view analysis¶
In preview mode, double-clicking the chart enlarges it to enter analysis mode for viewing. In edit mode, double-clicking the chart enters edit mode for adding, deleting, or modifying charts.
Optimization of infrastructure binding built-in views¶
When selecting built-in views, duplicate-named views are filtered out. When system views and user views have the same name, user views are prioritized for display, meaning only user views can be selected. To select system views, you need to rename or delete duplicate-named user views.
Optimization of billing method, new cloud dial test billing, adjustment of data expiration strategy¶
- New PV count statistics: Session count and PV count are billed based on the lower actual cost
- New cloud dial test task count billing: Open self-built node management for cloud dial tests, supporting any space administrator to create new dial test nodes globally; cloud dial test billing dimensions will charge based on the number of calls to current workspace cloud dial test tasks; Free Plan workspaces can create up to 5 dial test tasks and only support "China region" dial test nodes; Agile Plan and higher version workspaces support creating more dial test tasks and using more foreign dial test nodes.
August 26, 2021¶
DataFlux Studio¶
New chart query label filtering¶
DataFlux supports selecting host label properties for filtered display. Before selecting labels in the chart query, you need to set label properties for the host in "Infrastructure"-"Host". As shown in the figure below, there are two hosts with the "production" label property, so data from two hosts is displayed.
New infrastructure container Cluster, Replicate Set, Node Viewer¶
DataFlux has added Cluster, Replicate Set, Node Viewers for containers, supporting viewing the operational status and service capability metrics of Cluster, Replicate Set, Node in Kubernetes, thereby enabling real-time monitoring of Kubernetes clusters and deployed resources.
New quick filtering for user access monitoring View Viewer¶
DataFlux supports adding custom filter fields in the View Viewer by clicking the settings button on the left. After selecting the fields to add, click "Complete" to proceed.
Note: If the added filter field represents a time field, you can filter by selecting a time range to view the corresponding page content. For example, loading_time (page load time).
New navigation menu help entry, new multiple Viewer data collection guidance entry¶
DataFlux has added a help documentation entry in the navigation menu. When no data is uploaded to the Viewer, DataFlux provides a quick access entry to "How to Start Data Collection" to help you start real-time monitoring in different Viewers promptly. This includes how to enable log monitoring, how to enable cloud dial tests, how to collect host objects, etc.
Log Viewer optimization¶
DataFlux has optimized the log Viewer, using different colors to highlight different contents of logs, allowing you to quickly obtain key data information from logs, supporting light and dark themes.
Log Content | Light Theme | Dark Theme |
---|---|---|
Date (log occurrence time) | Yellow | Light Yellow |
Keywords (HTTP protocol related, e.g., GET) | Green | Light Green |
Text (string with quotes) | Blue | Light Blue |
Default (text without any identifiers) | Black | Gray |
Numbers (log status codes, e.g., 404) | Purple | Light Purple |
Security Inspection Overview Statistics Optimization¶
DataFlux supports customizing the view of the security inspection overview page. By clicking the "Jump" button, you can jump to the corresponding built-in view of the overview page for viewing, editing, copying, and exporting.
Workspace Switch Optimization¶
DataFlux supports switching to other workspaces or creating new workspaces by clicking the "Current Space" name.
System Theme Color Switch Optimization¶
DataFlux has added a "Auto" option for system theme colors, allowing automatic switching of theme colors based on computer appearance settings.
Other Optimization Features¶
- Optimize service details in application performance monitoring, clicking the resource name in the service details page directly jumps to "Traces" to view the corresponding resource's trace situation;
- Optimize performance details in the user access monitoring View Viewer, clicking error data (data marked as "Error" in the View list) allows you to view the corresponding detailed error;
- Optimize tag name and tag value display in the metric Viewer;
- Optimize X-axis time display in time series charts, default time range is the last 15 minutes, X-axis segments are divided by minute intervals; last day, X-axis segments are divided by hour intervals; last 7 days, X-axis segments are divided by day intervals;
- Optimize expression queries, when expression queries contain multiple query statements, grouping tags must remain consistent;
- Optimize host integration run status, in "Infrastructure"-"Host"-"Integration Run Status", add DataKit version information for installed hosts;
- Optimize snapshot sharing, support sharing snapshots with anyone who has the share link, or encrypted sharing with people who have the link and key.
August 12, 2021¶
DataFlux Studio¶
Add 3 Permission Roles, Optimize Scene View Permissions¶
DataFlux currently supports defining three workspace member permissions, including "Administrator", "Standard Member", and "Read-Only Member", separately constraining the management, operation, and viewing permissions of different member categories. "Administrators" can customize the permission scope of members, such as limiting read-only members' accessible scene views, limiting standard members' editable scene views, etc.
In the DataFlux workspace's "Management"-"Member Management"-"Modify", you can edit and update member permissions.
New Infrastructure Host Detail View Connection Tracking and File Layout Optimization¶
DataFlux supports viewing host system information in the infrastructure host detail page, including host name, operating system, processor, memory, network, hard disk, connection tracking count, file handle count, etc.
New Scene View Snapshot Save Function¶
In the DataFlux workspace, go to "Scene"-"Node", click the "Settings" button in the top navigation bar, and select "Save Snapshot" to save the data content displayed in the current view as a snapshot.
Additionally, DataFlux supports saving snapshots quickly via page function buttons and shortcut keys (Ctrl+K). In scene views and all Viewers, using the shortcut key Ctrl+K will quickly pop up the [Save Snapshot] dialog boxEnter a name and click [OK] to add the current page as a new snapshot.
Note: If there are other shortcut keys in your background program that conflict with the shortcut key Ctrl+K, you will not be able to use the shortcut key function for "Save Snapshot".
New Chart and Snapshot Sharing Management¶
DataFlux supports unified management of charts and snapshots shared within the current space. For example, after sharing a chart in the "Scene View", you can view the chart sharing list within the current space through "Management"-"Share Management"-"Chart Sharing" and perform operations such as viewing the chart, checking embed codes, and canceling sharing.
New User Email Change, Mobile Unbinding Function¶
In the DataFlux workspace, click the bottom-left "Account"-"Settings"-"Basic Settings" to change or unbind mobile and email.
New Custom User Access Monitoring SDK Data Collection Content¶
DataFlux supports customizing the data content collected by the user access monitoring SDK. By customizing user identifiers, sessions, and adding extra data TAGs, it helps users locate and analyze data under specific scenarios by setting different types of identifiers.
New Free Quota for Paid Version Session, Task Scheduling, Trace¶
DataFlux's paid version adds a daily gift of 1 million log data entries per DataKit, 100 Session counts, 10,000 task scheduling times, and 1 million Trace counts.
Optimize Scene Chart Analysis Mode¶
DataFlux supports previewing object data interaction changes within four times the query cycle through the timeline below the chart in the analysis mode of time series charts. You can also select the time range displayed on the time series chart by dragging the timeline. For example: if the current time is 14:00 and the time range selected is [10:00-11:00], then the timeline range will be [08:00-12:00].
Note: A single query statement in a time series chart can return up to 10 timelines, i.e., based on the results of the grouping (group by) conditions, only the first 10 timelines are displayed if there are more than 10 timelines.
Optimize Infrastructure Host and Container Honeycomb Charts¶
Hovering over host and container objects displays the host/container name, CPU usage rate, and MEM usage rate.
Optimize Log Source List Display Based on Viewer Time Range¶
DataFlux supports filtering log source lists based on the selected viewer time range. For example, selecting a 15-minute time range in the upper-right corner of the "Log Viewer" in the DataFlux workspace will only display log sources with data in the last 15 minutes on the left log list.
Optimize System Theme Color Switch¶
DataFlux supports switching system theme colors, divided into light and dark modes. Click the account at the bottom-left and choose the desired theme under "System Theme" to switch directly.
Other Optimization Features¶
- Webhook custom notification sending content type changed from text format to JSON format.
- Added an alert silence option of 15 minutes for anomaly detection library alert settings, meaning no notifications for the same alert within a 15-minute time range.
- Adjusted the display style of unresolved event list pages, changing log alert event statistics display from table charts to time-series charts.
August 4, 2021¶
DataFlux Studio¶
m
New Container Services and Deployments Viewer¶
DataFlux supports comprehensive display of container statuses from different dimensions through "Container", "Pod", "Services", and "Deployments" Viewers.
In the workspace "Infrastructure"-"Container", you can switch to "Services" via the data type filter bar in the upper-left corner to view detailed information of all retained Services in the space, including Service name, service type, Cluster IP, External IP, runtime duration, etc.
Switching to "Deployments" allows you to view detailed information of all retained Deployments in the space, including Deployment name, available replicas, upgraded replicas, readiness, runtime duration, etc.
July 29, 2021¶
DataFlux Studio¶
New Global Label Functionality for Infrastructure¶
DataFlux has added a global Label functionality for infrastructure, supporting users to add custom tags to infrastructure objects such as hosts, containers, processes, and filter and display infrastructure objects with the same tag via the added tags.
In the infrastructure object list, clicking the object name will pull out the detail page to add tags "Label Properties" for the object.
After adding custom tags, you can filter and display objects with the same tags in the object list via the added tags.
New SMS Notification Target; Added SMS Cost Statistics to Paid Plans and Billing¶
DataFlux supports managing notification targets for alerts, currently supporting adding DingTalk bots, Enterprise WeChat bots, Lark bots, Webhook customizations, email groups, and SMS groups.
Note:
- Members who need SMS notifications must first be invited to join the workspace via "Management" - "Member Management" before they can be selected.
- DataFlux Free Plan does not include SMS notifications; other versions offer SMS notifications at 100 messages/10 yuan, billed daily.
New Slide Show Mode for Scene Views¶
DataFlux has added slide show playback for scene views, supporting browsing all node views under specified scenes using slide show mode and customizing the interval for slide show playback.
New Workspace Token Replacement Function¶
DataFlux supports current space administrators replacing tokens within the space and customizing the expiration time of the current token. Enter "Management" - "Basic Settings" - "Replace Token", select the expiration time and confirm "Replace". After confirmation, DataFlux will automatically generate a new token, and the old token will expire within the specified time.
Note:
- After replacing the token, the original token will expire within the specified time. If any proxy uses this token, data reporting will stop immediately, so please check the integration configuration promptly;
- Replacing the token will trigger "Operation Events" and "Notifications".
New Metric DQL Query Mode¶
DataFlux supports switching "Search" to "DQL Query Mode" in the workspace "Metrics", allowing direct input of DQL statements for queries. The query allows you to view corresponding tags and metrics on the left and visualized charts returned by the query statement on the right.
Optimize Multi-line Display in Log Viewer¶
DataFlux supports choosing "1 line", "3 lines", or "10 lines" in the "Settings" of the Log Viewer to view complete log content.
Optimize Service and Trace Switching in Application Performance Monitoring, Optimize Response Time Line Chart¶
DataFlux supports default retention of current filtering conditions (including: environment, version) and time range when switching between "Service" and "Trace" Viewers in application performance monitoring. For example: In application performance monitoring "Service", select "Environment" as production_a
and "Time Range" as last 30 minutes
.
Switching to application performance monitoring "Trace", the filtering conditions remain consistent with "Service".
Optimize DataKit Collector Installation Guide¶
DataFlux supports custom installation options when installing DataKit, such as cloud vendor, namespace, global tag, etc.; DataKit offline installation supports directly downloading offline installation packages for different operating systems by clicking.
Optimize Flame Graph Sync and Async Call Display¶
DataFlux's flame graph in application performance monitoring supports viewing detailed data of service sync and async call chain performance. For example, through the flame graph, you can clearly see which requests were performed asynchronously, when they started, when they ended, and how much time they took in total.
Optimize Field Alias Customization for Viewer Display¶
DataFlux supports customizing aliases for fields displayed in Viewers. For example, in the infrastructure host object, users can modify field aliases in "Add Display Columns", and after modification, the aliases can be viewed in the Viewer and details.
Optimize Self-built Node Selection for Cloud Dial Tests¶
DataFlux splits the geographical location of self-built nodes for cloud dial tests into two filter boxes: country and province (or city). If the country filter box selects China, the next filter box will filter out the province list; if the country selects non-China, the next filter box will filter and display the city list. The filter box defaults to showing 20 items and supports fuzzy search.
Optimize DQL Query Functions¶
DataFlux DQL adds various query functions such as exists, wildcard, with_labels, etc.
July 15, 2021¶
DataFlux Studio¶
New Third-party Development Tool Integration for User Access¶
DataFlux adds support for integrating mini-programs developed using the uniapp development framework. By introducing the SDK file, you can monitor the performance indicators, error logs, and resource request data of the mini-program and report them to the DataFlux platform.
New Data Gap Anomaly Detection¶
DataFlux adds data gap anomaly detection, triggering events based on custom-configured detection cycles. Data gap events generated after the configuration takes effect are uniformly stored in the event list.
When the data gap detection rule takes effect, the first detection of a data gap and continuous data gaps will not trigger a data gap alert event; if data is detected and a data gap occurs within the configured custom detection cycle, a data gap alert event will be triggered.
New Built-in Detection Libraries for Elasticsearch, Redis, and Docker¶
DataFlux currently supports 4 built-in detection libraries: Elasticsearch, Redis, Host, and Docker. No manual configuration is required, allowing you to use them out-of-the-box. Clicking on the corresponding built-in detection library directly provides more details.
- Redis Detection Library: Includes abnormal increase in the number of client connections waiting for blocking commands
- Host Monitoring: Includes multiple data metrics such as host CPU, memory, disk, etc.
- Elasticsearch Detection Library: Includes ES JVM heap memory, thread pool, search query load anomalies, etc.
- Docker Monitoring: Includes multiple data metrics such as container CPU, memory, etc.
New "All Traces Viewer" for Application Performance¶
DataFlux provides three trace filtering viewing lists: "All Spans", "Top Spans", and "Traceid". Span represents a logical work unit within a given time period in a distributed system, and multiple spans form a trace trajectory.
- All Spans: Displays all span data collected within the currently selected time range.
- Top Spans: Filters and displays all span data entering for the first time within the currently selected time range.
- All Traces: Filters and displays all trace data containing initial top-level spans within the currently selected time range.
New Topology Call Relationship Diagram for Trace Details
DataFlux adds a corresponding service topology call relationship diagram in the application performance monitoring details page to view the call relationships between various services.
New User Access Error Details Page¶
DataFlux supports viewing user access error details in the user access details page, including error data information, error type, and error occurrence time during the user access session.
Clicking on the error information allows you to jump to the corresponding error details page.
New Security Inspection Overview Statistics¶
In "Security Inspection"-"Overview", you can filter hosts, security inspection levels, and security inspection categories to view the overview of security inspection events occurring on different hosts, including the quantity of security inspection events of different levels and visualized chart analysis, as well as rankings of security inspection events of different categories and rules.
New Security Inspection Event Handling Suggestions¶
In the security inspection Viewer, click the inspection event you want to view. In the pulled-out details page, you can view handling suggestions for this security inspection event, including the theoretical basis of the security inspection event, risk items, audit methods, remedial measures, etc.
New Secondary Navigation Bar¶
DataFlux adds support for displaying secondary navigation by hovering the mouse, thereby shortening the time users spend finding target data.
New Member Management Search Function¶
DataFlux supports searching for members quickly in "Member Management" based on email or name matching the current workspace members.
New Binding of Built-in Views to Infrastructure Objects¶
DataFlux supports current space administrators binding built-in views (system views, user views) to the details pages of infrastructure objects. As shown in the figure below, the built-in system view "Disk Monitoring View" has been bound.
Optimize Event List Display by Detection Library Type¶
DataFlux supports displaying event information in the event list based on different detection library types.
- When the detection library rule type is threshold, mutation, interval, water level, application performance indicator, user access indicator detection, the configured trigger conditions display corresponding baselines and their corresponding information, including critical, error, warning.
- When the detection library rule type is log, security inspection, anomalous process, cloud dial test detection, it queries all log data related to anomalous events.
As shown in the figure below, there are three different detection library types, including threshold detection, security inspection detection, and process data detection.
Optimize Deletion of Metric Sets¶
DataFlux supports administrators deleting metric sets within the space. Enter "Management"-"Basic Settings", click "Delete Metric Set", query and select the metric set name by entering keywords, and click "Confirm" to enter the deletion queue and wait for deletion. Deleting a metric set generates system notification events, such as users creating delete metric set tasks, successful execution of delete metric set tasks, failed execution of delete metric set tasks, etc.
Optimize Query Time Range for Built-in Views¶
Change the query time range for built-in views, including "Host Monitoring" views and "Process" views on the details pages of infrastructure objects, traces, and logs.
- When viewing trace and log details pages, "Host Monitoring" defaults to displaying performance data of associated hosts within the selected time component range.
- When viewing infrastructure object (host, container) details pages, "Host Monitoring" defaults to displaying host performance data within the last 24 hours.
- When viewing trace details pages, "Process" defaults to displaying real-time process data 10 minutes before and after the trace occurrence time.
- When viewing log details pages, "Process" defaults to displaying real-time process data 10 minutes before and after the log generation time.
Optimize Quick Filter Interaction in Viewers¶
If the current field value is too long in the quick filter of each Viewer in DataFlux, the exceeding part is displayed as ...
, and moving the mouse over the current field value area extends the display according to the field value length.
July 1, 2021¶
DataFlux Studio¶
New Unresolved Event Statistics Viewer¶
DataFlux adds a new unresolved event statistics Viewer. In the Events section of the DataFlux workspace, by switching the Viewer in the upper-left corner, you can view all unresolved events continuously triggered in the space, as well as the data volume statistics and alarm information details of unresolved events under different alarm levels.
New Chart Link Function¶
DataFlux adds link functions for time series charts, bar charts, overview charts, dashboards, pie charts, ranking charts, scatter plots, bubble charts, maps, honeycomb charts. Chart links help you jump from the current chart to the target page, supporting modifying the corresponding variable values in the link via template variables to pass data information and complete data linkage.
Chart links support obtaining template variables via queries, adding internal and external links, opening in new pages, current pages, and slide-out detail pages, and adding alias displays for links.
After adding the link, clicking the chart allows you to view clickable "Custom Links."
New Binding of Links and Built-in Views for Application Performance Monitoring¶
DataFlux supports adding built-in views, including built-in views and linkable web addresses, in individual service details. In the service list, click "Settings" to add one or more jump link addresses or built-in views and customize names.
After adding built-in views, you can view the corresponding built-in pages by checking service details.
New PV/UV Overview Statistics for User Access Monitoring Scenarios¶
DataFlux adds PV/UV overview statistics for user access monitoring in Web, iOS, Android, and Mini Program scenarios. You can quickly compare PV/UV differences and understand the actual number of accessing users.
New Notification Target: Lark Bot¶
DataFlux supports managing notification targets for alerts, currently supporting adding emails, DingTalk bots, Enterprise WeChat bots, Lark bots, and Webhook customizations.
Optimize Infrastructure Resource Catalog¶
Infrastructure data for objects other than hosts, containers, and processes can be reported to "Infrastructure"-"Custom" via DataFlux Function for viewing and analysis. Supports binding built-in views to resource catalog data and manually adding tags for filtering.
Reporting resource catalog data requires first installing and connecting DataKIt and Function, then reporting data to DataKit via Function, and finally DataKit reporting data to the DataFlux workspace.
Optimize Detection Rules for Anomaly Detection Library¶
DataFlux adds recovery alert configuration for anomaly events; adds support for regular expression matching styles adjustment for filtering conditions, optimizing =~
to match
, adding not match
; adds support for non-numeric data matching in trigger conditions, adding match
, not match
regular expression matching, and between
.
Optimize Associated Logs in Viewer Detail Pages¶
DataFlux supports filtering associated logs based on trace_id
by default in the associated log page within the side panel of the trace details page, viewing log details associated with the same trace_id
.
Within the trace details, clicking "Logs" below the flame graph allows you to view associated logs based on the host and trace ID (trace_id
field) related to the current trace. Additionally, you can perform keyword searches and multi-tag filtering on logs. If you need to view more detailed log content, you can click the log content to jump to the log details page, or click the jump button to open the log page.
New log association page added to Viewer.
Optimize Viewer Search Style¶
Adjust search styles for new scenes, metrics, application performance monitoring, user access monitoring, anomaly detection library, host mute management, integrations, management/built-in views, snapshots, etc.
Deployment Version Supports Modifying Data Retention Duration¶
The basic information of workspace management adds the ability to change data storage strategies.
June 17, 2021¶
DataFlux Studio¶
New Log Analysis Mode¶
DataFlux supports grouping large amounts of raw data based on 1-3 tags through the grouping function of logs, reflecting the distribution characteristics and trends of grouped data at different times. DataFlux supports three ways to browse grouped data: line chart, area chart, bar chart.
New Log Filtering¶
New log filtering function (log blacklist), supporting setting filter rules to filter out logs that meet the conditions, meaning after configuring log filtering, logs that meet the criteria will no longer be reported to the DataFlux workspace, helping users save on log data storage costs.
In "Logs"-"Filter Rules"-"New Rule", select "Log Source", add one or more relational "and" data screening conditions to filter the collected log data. Log data that meets the screening conditions will not be reported to the dashboard.
New Save Metric Browse Snapshots, Add Snapshot Link Copy Function¶
DataFlux supports creating quick-access data copies for metric viewers, which are uniformly saved in the "Snapshot" directory; a new feature generates sharable snapshot links, allowing sharing of snapshot links with other members within the space, enabling quick reproduction of immediate copied data information and restoring data to a certain point in time and a certain data display logic.
New Pod Viewer for Containers¶
DataFlux adds a Pod Viewer under "Infrastructure"-"Containers". Through the filter in the upper-left corner, you can switch to "Pod" to view detailed information of all collected Pods, including Pod name, belonging node Node, contained one or more containers, start status, running status, and related metrics. You can also switch to the honeycomb chart for viewing.
Clicking on a Pod in the honeycomb chart allows you to view the Pod's details, including Pod name, namespace, visual monitoring charts, etc.
New Free Quota Warning Notifications¶
Added warning notifications for Free Plan usage quotas and notifications when the quota is used up, supporting email and system notifications. Learn more about message notifications. System message notifications can be viewed in the DataFlux workspace "Account"-"Notifications".
New Chart Sharing¶
DataFlux supports chart sharing, which can be embedded in code outside of DataFlux for visualization data display and analysis. The shared charts will change according to changes in the DataFlux charts. Charts shared in the scene view are uniformly stored in "Integrations"-"Embedded Sharing".
Optimize Chart Queries and Styles and Settings¶
Adjust and optimize the query, style, and setting layout of visual charts in the scene view, adding functions such as hiding chart titles, time slice settings, precision settings for overview chart data, baseline settings for ranking charts, color fill settings for maps/honeycomb charts.
Optimize Integrations, Add DataKit and Function Installation Guides, Add Embedded Sharing¶
DataFlux supports viewing all supported collectors, DataKit installation instructions, DataKit upgrade commands, Function installation instructions, and embedded sharing chart codes in the integrations.
In the DataFlux workspace, click the "Integrations" menu on the left to view all supported collectors. Clicking on a collector displays its configuration details.
In the DataFlux workspace, click "Integrations"-"DataKit" to obtain DataKit installation instructions, including Linux, Windows, MacOS three major operating systems and offline installation methods. If DataKit is already installed but not the latest version, you can obtain the update script here under "DataKit Upgrade".
In the DataFlux workspace, click "Integrations"-"Function" to obtain Function installation instructions.
Optimize Host Details Page¶
In "Infrastructure"-"Host", clicking on the host name in the list pulls out the details page to view basic information, processor, memory, network, hard disk, etc. Supports automatic conversion of data units, and supports displaying different host monitoring views based on different operating systems.
June 3, 2021¶
DataFlux Studio¶
New Browser Log Collection¶
DataFlux supports actively sending different levels of log data (corresponding source:browser_log
metric type log data) to DataFlux via web browsers or JavaScript clients.
New Session Viewer for User Access Monitoring¶
DataFlux's user access monitoring Viewer adds a new feature to view user access behavior monitoring data (session data) related to Web/Mini Program/Android/iOS applications and session data details. In the workspace, open "User Access Monitoring"-"Select Any Application"-"Viewer", you can switch to the Session Viewer (Session). The Session Viewer statistically tracks the entire session's duration, number of pages, number of operations, number of errors, etc. Clicking on a single session allows you to view the detailed records under the entire session. By viewing and analyzing session data from multiple dimensions, you can better understand real user access data situations and help improve application performance experience.
New Browser Dial Test, Cloud Dial Test Anomaly Detection Template, and Optimized Self-built Nodes¶
- Browser Dial Test
DataFlux adds browser dial testing. In DataFlux's "Cloud Dial Test", click "New"-"Browser Dial Test" to create a new Browser dial test task, obtaining real-time user access experience data for Web pages, including page load time, resource load time, etc.
- Anomaly Detection Template
"Cloud Dial Test Data Detection" monitors cloud dial test data within the workspace. By setting threshold (boundary) ranges for specified data volumes produced by dial test tasks within a certain time period, an alert is triggered when the data volume reaches the threshold range. You can also customize alert levels, triggering different levels of alert events when the specified data volume reaches different threshold ranges.
- Self-built Nodes
DataFlux's cloud dial test self-built nodes add "Node Code", used to obtain the code for the node information, ensuring that the node code is not duplicated within the current space.
New Registration Welcome Page¶
A welcome page is displayed after registering and logging into a DataFlux account, where you can view a DataFlux introduction video and scan a code to enter the DataFlux WeChat and DingTalk service groups.
New Support for DataFlux Func in Free Plan, Adjust Billing Items¶
All versions of DataFlux fully support the DataFlux Func platform for custom functions. DataFlux Func is a script development, management, and execution platform based on Python and similar to Serverless. At the same time, to reduce user costs for using DataFlux, DataFlux adjusts the billing logic for application performance monitoring and user access monitoring. After the adjustment, application performance monitoring counts trace quantities, while user access monitoring counts daily session quantities.
Optimize Generated Metrics¶
Add user performance monitoring, user access monitoring, and security inspection generated metrics, optimizing log-generated metrics. Generating metrics can be completed in three steps: data filtering, data querying, generating metrics, including configuring dimensions, aggregation time, defining aggregation frequency, metric set, and metric name to generate metrics. Examples are as follows:
- Application Performance Monitoring
- User Access Monitoring
- Security Inspection
Optimize Built-in Views, Add Clone Function¶
Built-in views are divided into "System Views" and "User Views", supporting cloning from "System Views" to "User Views", and allowing system views and user views to have the same name. Clicking "View" on a system view jumps to the system view details.
Click "Clone" to open the clone view dialog box, click "Confirm" to create the corresponding user view based on the current view template.
Successfully created, jump to the created user view page.
If user views and system views have the same name and the system view is applied to the Explorer detail view, such as "Infrastructure-Host-Host Details" under the host view, the user view will replace the system view and be displayed in the host view.
Optimize View Variables¶
DataFlux supports adding global variables to views. When you want to dynamically filter charts in a view, you can choose to use view variables to achieve this. After configuring view variables, you can select whether to enable "Default Match All" via "Advanced Settings".
After enabling, you can view the comprehensive view of all hosts via *
, and manually select variables for filtered viewing.
Optimize Explorers¶
- When single-column information is not fully displayed, hover delay for 1 second shows all tooltip information;
- When searching key content in the Explorer, the matched content in the list is highlighted;
- Tags on the details page support copy function; host tags support ["View Related Logs"]["View Related Containers"]["View Related Processes"]["View Related Traces"]["View Related Inspections"];
- Statistical charts for log, application performance monitoring, user access monitoring, cloud dial tests, security inspections are changed to status statistics;
- New help links for starting user performance monitoring, user access monitoring, security inspections.
May 20, 2021¶
DataFlux Studio¶
New Snapshot Saving¶
DataFlux supports creating snapshots for infrastructure, logs, events, application performance monitoring, user access monitoring, cloud dial tests, security inspections. Through snapshots, you can quickly view and analyze data and anomalies from different time periods and labels.
New Infrastructure Host Integration Interaction¶
The "Integration Operation Status" of infrastructure hosts displays the operation status of related collectors for the host. There are two states for the operation status: normally running integrations and errored integrations. Normally running integrations are displayed as 'light blue' by default, and errored integrations are displayed as 'red' by default and support viewing error information by clicking.
At the same time, clicking on integrations with built-in view symbols (as shown in the figure below) supports "View Monitoring View". Clicking opens the relevant built-in view.
New Self-built Nodes, Optimized Cloud Dial Test Node Display¶
"Cloud Dial Test" supports custom data collection through the DataFlux Studio console. By creating a probe point, you can periodically monitor websites, domains, backend interfaces, etc., based on the HTTP protocol, analyzing site quality at any time. DataFlux has covered 16 nodes globally, categorized by domestic, overseas, and self-built nodes. At the same time, it supports self-built dial test nodes, ensuring service availability.
New Process, Application Performance Monitoring, and User Access Monitoring Check Rules¶
"Process Anomaly Detection" is used to monitor process data within the workspace, supporting setting alert triggers for one or more fields of process data. For example: you can set alerts based on the 'host' field being 'izaqbin' and the 'state' field being 'sleep' appearing a certain number of times.
"User Access Indicator Detection" is used to monitor the indicator data of "User Access Monitoring" within the workspace. By setting threshold ranges, an alert is triggered when the indicator reaches the threshold, supporting setting alerts for individual indicators and customizing alert levels.
"Application Performance Indicator Detection" is used to monitor the indicator data of "Application Performance Monitoring" within the workspace. By setting threshold ranges, an alert is triggered when the indicator reaches the threshold, supporting setting alerts for individual indicators and customizing alert levels.
Optimize Integration¶
Log into the DataFlux workspace and go to the "Integration" page to view all supported collectors. You can also click the "Quick Get DataKit Installation Command" in the upper-right corner to directly obtain the DataKit installation command for installation. After installation, you can start more collectors to collect data metrics.
Optimize Log Metric Generation¶
Log metric generation adds log source selection, allowing you to generate metrics by selecting log sources, filtering logs, configuring dimensions, aggregation frequency, defining metric sets, metric names, and aggregation rules. [Learn more about log metric generation]
Optimize Log Viewer¶
After log data is collected to DataFlux, you can view all log sources and their contents in the "Logs" section of the DataFlux Studio console. It supports viewing log details by selecting log sources via the search bar.
Optimize User Access Monitoring Viewer¶
On the "Performance" page of the user access monitoring details, you can view frontend page performance, including page load time, content rendering time, interaction time, input latency, etc. As shown in the figure below, the LCP (Largest Content Paint) indicator reached 5.43 seconds, while the recommended time is within 2.5 seconds, indicating that the page loading speed needs optimization. It supports filtering and searching to help users quickly locate resources and content.
Switching to "Trace" allows you to view requests sent to the backend application, including the occurrence time, content, and duration of invoked resources. For example, you can view every ajax request returned by the xhr object.
Switching to "Errors" allows you to quickly locate frontend errors through detailed information, including error messages, error types, and occurrence times.
Optimize Built-in Views¶
Built-in views are divided into two types: system views and user views, which can be applied in scenes.
System views provide integrated view templates already provided by the system, supporting modifications, exports, and restorations. Users can directly "Export" system views or "Modify" system views. By "Restoring" system views, modified system views can be converted back to the original system views. After restoration, they cannot be recovered.
User views are user-defined views saved as templates, supporting modifications, exports, and deletions. In "Built-in Views"-"User Views", click "Create User View" to create user-defined view templates. Specific configurations can refer to Scenes and Views.
Optimize Chart Locked Time Optimization¶
Dashboard views support locking the time range of charts, unaffected by the global time component. After successful settings, the upper-right corner of the chart will display the user-set time, such as "xx minutes", "xx hours", "xx days".
Optimize Nested Functions¶
All chart components' nested functions are optimized into built-in functions, no longer calling DataFlux Func functions, implemented via DQL, and the time interval option for nested functions is deleted.
May 7, 2021¶
DataFlux Studio¶
New Save Snapshots¶
DataFlux snapshots support creating snapshots for infrastructure, logs, events, application performance monitoring, user access monitoring, cloud dial tests, security inspections. Through snapshots, you can quickly view and analyze data and anomalies from different time periods and labels.
New Infrastructure Host Integration Interaction¶
The "Integration Operation Status" of infrastructure hosts displays the operation status of related collectors for the host. There are two states for the operation status: normally running integrations and errored integrations. Normally running integrations are displayed as 'light blue' by default, and errored integrations are displayed as 'red' by default and support viewing error information by clicking.
At the same time, clicking on integrations with built-in view symbols (as shown in the figure below) supports "View Monitoring View". Clicking opens the corresponding built-in view.
New Self-built Nodes, Optimized Cloud Dial Test Node Display¶
"Cloud Dial Test" supports custom data collection through the DataFlux Studio console. By creating a probe point, you can periodically monitor websites, domains, backend interfaces, etc., based on the HTTP protocol, analyzing site quality at any time. DataFlux has covered 16 nodes globally, categorized by domestic, overseas, and self-built nodes. At the same time, it supports self-built dial test nodes, ensuring service availability.
New Process, Application Performance Monitoring, and User Access Monitoring Check Rules¶
"Process Anomaly Detection" is used to monitor process data within the workspace, supporting setting alert triggers for one or more fields of process data. For example: you can set alerts based on the 'host' field being 'izaqbin' and the 'state' field being 'sleep' appearing a certain number of times.
"User Access Indicator Detection" is used to monitor the indicator data of "User Access Monitoring" within the workspace. By setting threshold ranges, an alert is triggered when the indicator reaches the threshold, supporting setting alerts for individual indicators and customizing alert levels.
"Application Performance Indicator Detection" is used to monitor the indicator data of "Application Performance Monitoring" within the workspace. By setting threshold ranges, an alert is triggered when the indicator reaches the threshold, supporting setting alerts for individual indicators and customizing alert levels.
¶
- Fixed frequent collection of Aliyun CMS data collector causing some other collectors to freeze issues.
March 25, 2021¶
DataFlux Studio¶
New Cloud Service Association¶
DataFlux supports centrally managing cloud service account AccessKey information through "Cloud Association". After configuration, you can regularly synchronize cloud host information under the affiliated cloud account in [Infrastructure - Host], displaying all cloud host information, such as CPU, memory, network, disk, etc.
If you no longer need the association, you can delete the already associated Alibaba Cloud RAM sub-account AK information. Deleting the association information triggers an association invalidation notification event, which can be viewed in Events - Notification Events. For more details on cloud associations, refer to the document Cloud Association.
New Audit for Operation Events¶
DataFlux supports auditing operation events within the workspace, storing operation events uniformly in Events - "Operation Events".
New Host Muting¶
If you need to perform operations such as stress testing on a single cloud host, it may trigger frequent alarms. For such alarm notifications, you can temporarily enable host muting functionality. In "Infrastructure" - "Host", click on the cloud host details to select the host muting time.
All muted hosts can be viewed in "Anomaly Detection Library" - "Host Mute Management". After deleting a muted host, you can re-receive alarm notifications for the corresponding host.
New Notification Target: Enterprise WeChat Bot¶
DataFlux supports users managing notification targets for sending alerts, currently supporting adding emails, DingTalk bots, Enterprise WeChat bots, and Webhook customizations.
Optimize Anomaly Detection Library¶
DataFlux includes various built-in detection libraries, ready-to-use. It supports multiple built-in detection libraries such as hosts, after enabling them, you can receive related anomaly event alerts immediately.
In the "Detection Library", you can "Import/Export Detection Libraries". After configuring all detection rules, you can export the detection library via JSON files, and the exported detection library can be imported into other detection libraries. When importing the detection library, it verifies whether duplicate detection items exist, allowing you to "Replace" or "Skip".
Optimize Object Viewers and Details¶
- Host objects add operating system small icon display
- Container objects add container type small icon display
- Process side panel detail pages add "Start Time" display
Optimize Data Storage Strategy, Paid Plans, and Billing¶
- Basic information of workspace management adds change data storage strategy
- Workspace management usage statistics optimize to paid plans and billing
April 7, 2021¶
DataFlux Studio¶
Add View Variable Object Mapping¶
When using "Object Mapping", you must first define a view variable based on object class fields. After configuring object mapping in "View Variables", you need to use the mapped tag fields as grouping in "Chart Queries" and enable "Field Mapping" in "Chart Settings". Then, you can view the set object mapping display in the view chart.
Optimize Event Grouping¶
DataFlux supports grouping "Anomaly Events" by "Detection Items", aggregating related events. By viewing "Aggregated Events", you can quickly obtain the aggregated event list based on the detection item, allowing you to quickly get all events triggered by the detection item. By switching to the aggregated event list, you can view the details of these events.
Optimize Detection Item Queries¶
In the "Anomaly Detection Library", the "Detection Rules" "Detection Metrics" only allow adding one query by default, supporting switching between "Simple Query" and "Expression Query"; uniformly change the "Detection Rules" "Trigger Conditions" M1
to Result
; support adding AS
aliases to queries.
DataKit (v1.1.4-rc2)¶
- Fixed frequent collection of Aliyun CMS data collector causing some other collectors to freeze issues.
March 25, 2021¶
DataFlux Studio¶
New Cloud Service Association¶
DataFlux supports centrally managing cloud service account AccessKey information through "Cloud Association". After configuration, you can regularly synchronize cloud host information under the affiliated cloud account in [Infrastructure - Host], displaying all cloud host information, such as CPU, memory, network, disk, etc.
If you no longer need the association, you can delete the already associated Alibaba Cloud RAM sub-account AK information. Deleting the association information triggers an association invalidation notification event, which can be viewed in Events - Notification Events. For more details on cloud associations, refer to the document Cloud Association.
New Operation Event Audit¶
DataFlux supports auditing operation events within the workspace, storing operation events uniformly in Events - "Operation Events".
New Host Muting¶
If you need to perform operations such as stress testing on a single cloud host, it may trigger frequent alarms. For such alarm notifications, you can temporarily enable host muting functionality. In "Infrastructure" - "Host", click on the cloud host details to select the host muting time.
All muted hosts can be viewed in "Anomaly Detection Library" - "Host Mute Management". After deleting a muted host, you can re-receive alarm notifications for the corresponding host.
New Notification Target: Enterprise WeChat Bot¶
DataFlux supports users managing notification targets for sending alerts, currently supporting adding emails, DingTalk bots, Enterprise WeChat bots, and Webhook customizations.
Optimize Event Viewer, Add Event Group Aggregation Viewing¶
In the event list viewer, through grouping functions, DataFlux supports quickly aggregating and statistically analyzing related events based on group labels. For example, in "Anomaly Events", based on "Detection Item" grouping, you can quickly obtain all events triggered by the detection item. By switching to the aggregated event list, you can view the details of these events.
Optimize Link Service Filtering and Topology Chart¶
In the "Service" list of "Application Performance Monitoring", you can switch the list to topology chart mode to view the call relationships between various services. It supports filtering displays based on different performance indicators and customizing link service performance indicator color ranges.
Other Optimization Features¶
- Optimized log, event, infrastructure, application performance monitoring, user access detection viewer searches and exports, added export to views. Optimized log, event, link, infrastructure-container, infrastructure-process detail page label filtering, associated query analysis.
- Optimized time component, optimized simple query and expression query switching in scene views (overview chart, dashboard, ranking chart)
- Optimized offline host display: when the host is offline, CPU usage rate, memory usage rate, load are not displayed, values are filled with " - "
- Optimized anomaly detection library detection rules, added water level detection, interval detection, and mutation detection.
DataKit (v1.1.4-rc1)¶
- Added more information to the message field in the process collector for full-text search convenience
- Host object collector supports custom tags, facilitating cloud attribute synchronization
DataKit (v1.1.4-rc0)¶
- Added file collector, dial test collector, and HTTP packet collector
- Built-in support for ActiveMQ/Kafka/RabbitMQ/gin (Gin HTTP access logs)/Zap (third-party log framework) log cutting
- Enriched [http://localhost:9529/stats] page statistical information, added such as collection frequency (n/min), amount of data collected each time, etc.
- DataKit itself increases certain cache space (invalid upon restart), avoiding data loss due to occasional network reasons
- Improved Pipeline date conversion function, enhancing accuracy. Additionally, added more Pipeline functions (parse_duration()/parse_date())
- Trace data increased more business fields (project/env/version/http_method/http_status_code)
- Various detailed improvements to other collectors
March 4, 2021¶
DataFlux Studio¶
1. Optimize Time Series Charts, Bar Charts, Rankings, Log Stream Charts, Alert Statistics Charts, Object List Charts¶
- Time Series Charts: Added "Display Density", including "Low: 60 points", "Medium: 180 points", "High: 360 points", "Very High: 720 points". Added "Stacked Display", only supported for bar charts, default off.
- Rankings: Chart display optimization
- Bar Charts: Chart display optimization
- Log Stream Charts: Added search box, supporting searching content in messages
- Alert Statistics Charts: Query display optimization
- Object List Charts: Query display optimization
2. New Automatic Unit Conversion¶
- Default query result values automatically convert units following "Scientific Notation K, M, B", retaining two decimal places (1 thousand = 1K, 1 million = 1M, 1 billion = 1B)
- Renminbi automatically converts units "Yuan, Ten Thousand Yuan, Hundred Million Yuan", retaining two decimal places
- After setting units, cancel scientific notation, displaying according to user-set units
3. New Object Management¶
DataFlux supports user management of notification targets for sending alerts, currently supporting adding DingTalk bot notification configurations and Webhook custom notification configurations.
4. Adjust Event, Log, Trace, Application Monitoring, Object Host, Object Container, Object Process, Other Object Viewers, Supporting Exporting CSV Documents, Maximum Support Exporting Last 5000 Data Entries.¶
- Events
DataFlux supports collecting all event data generated based on the "Anomaly Detection Library". Users can view and search all event data within the workspace on the Events page.
- Object Hosts
DataFlux supports collecting object host data. In the host list under "Objects", you can view and search all host object data information.
Clicking on the host name in the list pulls out the details page to view detailed information about the host object, including basic information, CPU, memory, network, hard disk, etc.
- Object Containers
DataFlux supports collecting object container data. In the container list under "Objects", you can view and search all container object data information. Clicking on the container name in the list pulls out the details page to view detailed information about the container object.
- Object Processes
DataFlux supports collecting object process data. In the process list under "Objects", you can view and search all process object data information. Clicking on the process name in the list pulls out the details page to view detailed information about the process object.
- Other Objects
DataFlux supports collecting other object data, such as Alibaba Cloud ECS, etc. In the other list under "Objects", you can view and search all other object data information. Clicking on the other object name in the list pulls out the details page to view detailed information about the other object.
- Logs
DataFlux supports collecting various formats of log data, including system logs, Nginx logs, Apache logs, and many other types of log data, allowing quick viewing and analysis of all log data within the current workspace.
- Traces
DataFlux supports viewing service links and trace lists in "Trace Monitoring". In the service list, you can view or search all services receiving trace data. In the trace list, you can query trace data, export trace data, view trace details, and perform comprehensive analysis of trace performance through flame graphs and span lists.
DataFlux supports both light and dark theme displays. Here, we take the dark theme as an example.
Service List
Trace Topology
Trace List
Flame Graph
- Application Monitoring
DataFlux supports collecting Web, Android, iOS, and Mini Program application data and provides application monitoring scenarios, allowing quick viewing and analysis of application conditions. Taking Web applications as an example:
Web Viewer
Web Application Details
February 2, 2021¶
DataFlux Studio¶
- New area chart style for time series charts
- New object host, container, process, new host honeycomb chart
- Optimize anomaly detection, new built-in detection library
-
Optimize events, objects, logs, traces, application monitoring, objects host, objects container, objects process, other object viewers, support exporting CSV documents, maximum support exporting last 5000 data entries.
-
Events
DataFlux supports collecting all event data generated based on the "Anomaly Detection Library". Users can view and search all event data within the workspace on the Events page.
- Object Host
DataFlux supports collecting object host data. In the host list under "Objects", you can view and search all host object data information.
Clicking on the host name in the list pulls out the details page to view detailed information about the host object, including basic information, CPU, memory, network, hard disk, etc.
- Object Container
DataFlux supports collecting object container data. In the container list under "Objects", you can view and search all container object data information. Clicking on the container name in the list pulls out the details page to view detailed information about the container object.
- Object Process
DataFlux supports collecting object process data. In the process list under "Objects", you can view and search all process object data information. Clicking on the process name in the list pulls out the details page to view detailed information about the process object.
- Other Objects
DataFlux supports collecting other object data, such as Alibaba Cloud ECS, etc. In the other list under "Objects", you can view and search all other object data information. Clicking on the other object name in the list pulls out the details page to view detailed information about the other object.
- Logs
DataFlux supports collecting various formats of log data, including system logs, Nginx logs, Apache logs, and many other types of log data, allowing quick viewing and analysis of all log data within the current workspace.
- Traces
DataFlux supports viewing service links and trace lists in "Trace Monitoring". In the service list, you can view or search all services receiving trace data. In the trace list, you can query trace data, export trace data, view trace details, and perform comprehensive analysis of trace performance through flame graphs and span lists.
DataFlux supports both light and dark theme displays. Here, we take the dark theme as an example.
Service List
Trace Topology
Trace List
Flame Graph
- Application Monitoring
DataFlux supports collecting Web, Android, iOS, and Mini Program application data and provides application monitoring scenarios, allowing quick viewing and analysis of application conditions. Taking Web applications as an example:
Web Viewer
Web Application Details
February 2, 2021¶
DataFlux Studio¶
- Added area chart style for time series charts
- Added object host, container, process, added host honeycomb chart
- Optimized anomaly detection, added built-in detection library
- Optimized events, objects, logs, traces, application monitoring, object host, object container, object process, other object viewers, functions and pages
- Optimized log stream chart queries
- Optimized ranking, dashboard, overview chart queries, added expression query mode.
- Optimized visual chart queries, simple queries and DQL queries support mutual switching
- Optimized grouping logic, including scatter plots, bubble charts, maps, honeycomb charts, table charts
- Optimized filtering conditions support real-time filtering
DataKit¶
- Added host object data collection
- Added container object data collection
- Added process object data collection
January 14, 2021¶
DataFlux Studio¶
- Added Explorer layout, optimized data structure, supporting querying and analyzing modules such as objects/links/RUM/logs
- Added Mini Program application monitoring
- Added time series charts, chart types can choose line charts or bar charts
- Adjusted chart UI
- Adjusted log stream chart queries and settings
- Optimized bar chart, ranking chart, table chart related settings
- Threshold detection adjusted to DQL UI query mode