Change Log (2020)¶

This document records the update contents of each release of Guance, including DataKit, Guance best practices, Guance integration documents, and Guance.

December 29, 2020¶

DataFlux Studio¶

• Added built-in views
• Added chart type: Image
• Added support for stacked bar charts and percentage stacked bar charts in bar charts
• Added unit conversion: traffic, bandwidth, timestamp
• Optimized data gateway page, added option to enable host monitoring
• Optimized DQL query, DQL functions, DQL command-line client
• Optimized chart query methods, supporting normal queries, expression queries, DQL queries
• Optimized view variable management method, supporting DQL queries
• Optimized preview/edit view, added chart interaction
• Optimized scene creation, supporting creation based on view templates
• Optimized object monitoring view, supporting binding views
• Relationship diagrams can be saved as charts with optimized drawing interactions
• Optimized node tree click interactions

DataKit¶

• Added WeChat Mini Program application monitoring data collection
• Added default enabled host monitoring configuration file

December 15, 2020¶

DataFlux Studio¶

• Added RUM application monitoring module, supporting Web, iOS, Android application monitoring
• Added DQL query, supporting remote CLI
• Added composite chart, supporting up to 5 chart combinations
• Added Top List chart, supporting bar chart and table chart display methods
• Added chart links, supporting line chart, pie chart, bar chart, table chart to jump from the current page to the target page
• Added chart title description
• Added map chart metrics Top5 ranking and ascending/descending sorting
• Optimized scatter plot, added bubble chart, supporting using bubble size to represent data size

DataKit¶

• Added Web, iOS, Android application monitoring data collection

November 26, 2020¶

DataFlux Studio¶

• Adjusted billing model to pay-as-you-go
• Added usage statistics module for viewing version information and usage of charged items
• Added object addition feature, supporting resource catalog attributes and tags
• Added relationship diagram function to objects
• Added quick filters for logs, events, traces, supporting user-defined tags
• Official website added billing-related landing pages

DataKit¶

• Added Alibaba Cloud, Tencent Cloud, Huawei Cloud, Docker container object collection

November 5, 2020¶

DataFlux Studio¶

• Time component added relative time selection and data refresh time adjustment
• Supported setting different time ranges for different views
• Added label filtering functionality for objects and traces, allowing data filtering
• Added two new chart types: object list chart, alert statistics chart
• Log stream chart added unit settings and display column addition function
• Added metric query copy function
• Anomaly detection added log detection
• Optimized event page data loading speed
• Optimized several interactions and front-end interfaces to improve user experience

October 15, 2020¶

DataFlux Studio¶

• Optimized detail pages for logs, events, objects for clearer and more intuitive viewing of detailed information
• Added search functionality for view variables
• Added chart copy function, supporting copying charts and pasting into any scenario
• Adjusted object data structure
• Fixed several style issues, improving chart loading speed

DataKit¶

• Optimized CSV/Excel metric collection
• Added Huawei Cloud cloud monitoring metrics, TiDB metrics collection

September 22, 2020¶

DataFlux Studio¶

• Added "Dark Mode" theme

September 17, 2020¶

DataFlux Studio¶

• Added monitoring view functionality to objects, usable for IT infrastructure monitoring
• Added view variables for object types

September 8, 2020¶

DataFlux Studio¶

• Added metric search function
• Added breadcrumb function
• Optimized several interactions and front-end interfaces to improve user experience
• Fixed discovered bugs

August 20, 2020¶

DataFlux Studio¶

• Optimized scene creation process, adjusting the display effect of the node tree
• Optimized the process and UI for creating new nodes
• Added source configuration function to view templates, supporting automatic generation of source configurations by importing template files.
• Added a new chart type: honeycomb chart
• Fixed discovered bugs

August 6, 2020¶

DataFlux Studio¶

• Added service topology diagram function in "Trace Analysis" to display service call relationships
• Added list settings function for "Logs", "Objects", and "Traces", supporting custom display columns and column names; added display settings function, supporting hiding the "filter bar" and "statistics chart"
• Added ticket function, allowing users to submit tickets for questions and suggestions based on encountered issues

DataKit¶

• Added 15 data sources: ASP .NET, .NET CLR, Druid, Envoy, Ansible, Kibana, Microsoft Exchange Server, ModBus, MQTT Consumer, Nfsstat, Puppet Agent, systemd_utils, Kong monitoring metrics, WeChat Mini Program operation metrics, and AWS CloudWatch logs and events collection

July 16, 2020¶

DataFlux Studio¶

• Adjusted left navigation menu, adding log, trace analysis, object functional modules
• Added log collection and management functions, supporting "extract labels" and "generate metrics" based on log data
• Added trace analysis function for analyzing and managing trace data, suitable for APM monitoring
• Added object collection and analysis functions, supporting manual addition of objects
• Renamed "Key Events" to Events, restructured business logic, adjusted display interface, supporting aggregation of related events; supporting display of relevant metrics line charts triggered by events; supporting creation of events based on charts in analysis mode
• Added 4 categories of view classifications in scenes: "Log View", "Event View", "Trace View", and "Object View", supporting saving logs, events, traces, objects to corresponding views for querying and analysis
• Added associated analysis function for tags. Supporting querying related metrics, logs, objects, events, and traces based on tags
• Renamed "Metric Browser" to "Metrics", added tiled mode, supporting simultaneous viewing of multiple metrics line charts
• Renamed "Trigger" to "Anomaly Detection", adjusted "Basic Detection" to "Threshold Detection", "Advanced Detection" to "Custom Detection Function", and added four anomaly detection types: event detection, mutation detection, range detection, water level detection
• Removed "Trigger History" module, alert history data will be directly saved to Events
• Added a new chart type: log stream chart, supporting adding log data tables to views
• In the Metrics Query UI mode, added influxdb's built-in transformation functions
• Added 5 view templates: etcd monitoring view, CoreDNS monitoring view, Traefik monitoring view, Hadoop monitoring view, RaspberryPi (Raspberry Pi) monitoring view
• Other optimizations and adjustments
• Optimized navigation menu collapse/expand styles
• Added verified status indicators to integrated pages, including "Verified" and "Beta" states
• Optimized derivative node sorting, defaulting to alphabetical order by node name
• Optimized UI design for several interfaces, added hover display effects for view charts
• SaaS edition user registration added enterprise/team information

DataFlux Admin Console¶

• Adjusted DataWay gateway usage logic, moved DataWay installation to the admin backend, all workspace members can use it
• Supported setting different data retention periods for different types of data

DataKit¶

• DataKit added 15 data sources: Containerd, GitLab, Yarn, Harbor, Tailf, Rsyslog, Jira, AWS CloudWatch Log, RaspberryPi (Raspberry Pi), Go runtime metrics, Neo4j monitoring metrics, directory monitoring metrics, Alibaba Cloud security metrics, Baidu Index, and host object data collection
• DataKit supported collecting trace data through Zipkin and Jaeger
• Adjusted DataKit installation commands, DataKit installation supports more platforms, also supports offline installation
• Adjusted DataKit related configurations, some collectors have field adjustments, not recommended to copy the original conf.d directory to the new collector
• Organized DataKit configuration file directory structure and default installation path

DataWay¶

• DataWay added interfaces for writing log data, object data, and adjusted the event data write interface

May 7, 2020¶

DataFlux Studio¶

• Charts added scatter plots, displaying data distribution
• In the Metric Browser, "Measurement" added the "Configure Automatic Aggregation Rules" function, supporting automatic downgrading of collected metrics according to predefined aggregation rules
• Removed dataset mode in views, optimized metric browsing interaction, added table mode when browsing metrics
• Optimized the display effect of integrated components
• Fixed bugs where login would automatically log out after a period of time and tag filter conditions were not saved in some cases during new query creation, improving product usability

DataFlux Admin Console¶

• Workspaces added data warehouse configuration, supporting synchronization of specified metrics to the data warehouse
• Workspaces supported configuring automatic aggregation of metrics

DataFlux f(x)¶

• Updated "DataFlux.f(x) Package Learning" to the latest version
• Authorization link GET access supports flat form (?kwargs_key=value) and simplified form (?key=value)

April 29, 2020¶

Data Collection¶

• DataKit added 6 data sources: Oracle monitoring metrics, CoreDNS, uWSGI, Traefik, Solr, and Alibaba Cloud RDS slow query logs. Added monitoring metrics collection for Express projects
• Added four view templates: self-built MySQL monitoring, Alibaba Cloud PolarDB MySQL monitoring, Alibaba Cloud PolarDB PostgreSQL monitoring, and Alibaba Cloud RDS for PostgreSQL monitoring; optimized two view templates: MySQL monitoring, PostgreSQL monitoring

April 21, 2020¶

DataFlux Studio¶

• Added a new chart type: custom chart, supporting drawing charts based on custom background images
• Basic charts added an "automatic alignment of time intervals" function. The system calculates time intervals based on selected time ranges and aggregation times, then dynamically adjusts queries according to preset time intervals (including 1 minute, 5 minutes, 10 minutes, 30 minutes, 1 hour, etc.)
• Charts added adaptive view window functionality. When only one chart exists in the view, support selecting "fill view" to make the chart adapt to the entire view window
• When adding nodes, derived nodes support specifying metrics and node icons
• Adjusted the classification and naming of the left navigation bar
• Basic settings support viewing the current workspace version
• Adjusted the data gateway interface for SAAS editions
• Fixed several bugs, improving system stability

DataKit (v1.0.0-109-g13b2ced)¶

• Added timezone, StatsD, SSH, lighttpd, etcd, and Alibaba Cloud pricing six data sources
• Supported CentOS 6.7 and Ubuntu 14.04 and above operating systems
• DataKit supported cascading and Lua script cleaning

April 14, 2020¶

DataFlux Studio¶

• DataFlux online added product version restrictions, all newly created workspaces defaulted to Free Plan (Time Series 500 entries, data retention 1 day, trigger rule count 1, trigger rule validity 7 days)
• Restructured and optimized the underlying logic of triggers; trigger rules now support customizing alert message content for data gaps, added advanced detection features, supporting development of advanced detection functions on the DataFlux f(x) platform
• Optimized chart loading speed and SQL queries
• Fixed issues with alias and color display errors in charts
• Fixed issue where metrics could not be deleted
• Fixed issue where metrics with English periods in their names could not be queried

DataFlux Admin Console¶

• Adjusted "Add Workspace" logic, removed "Select Existing Database" and "Set Data Permissions" functions, all newly created workspaces automatically create new databases
• New workspaces support setting "data retention duration" and "trigger history retention duration"

DataFlux.f(x)¶

• Added online update functionality for official scripts, no longer requiring export and import of official scripts
• Official script library added simple testing support (requires installing the latest official library)
• Editor uses strictly monospaced fonts, ensuring 1 Chinese character equals 2 English characters in width
• Fixed known issues and improved system stability

DataWay (V1.0.1-401-ga917b59)¶

• MongoDB/Redis/MySQL connection pool functionality added to Lua modules (enabled by default)
• Removed correctness validation for $flow_, $alert metrics
• DataWay added data reporting security authentication mechanism, all deployed DataWays must be upgraded, otherwise they cannot continue to be used

April 3, 2020¶

• Line charts, bar charts, pie charts' "Settings" support setting colors, and support setting colors for different groups after grouping
• Chart settings support customizing metric units
• Fixed Safari compatibility issues
• Optimized side navigation bar

March 31, 2020¶

DataFlux Studio¶

• Help Center added search functionality, supporting quick keyword location
• Scene nodes added sharing functionality, supporting viewing share history, editing shares, canceling shares, and viewing share QR codes
• Baseline functionality adjusted, moving "Baseline" from "Query" and "Anomaly Detection" to "Line Chart" and "Bar Chart" settings
• "Metric Browser" supports viewing "Data Types" of metrics
• Added 37 view templates for IT operations scenarios
• Adjusted chart query rules and UI styles, supporting multiple fields selection from one metric set, supporting nested aggregation functions
• Optimized scene node tree, recording expanded state selections; fixed selection issues
• Optimized left navigation bar, supporting collapsed display, increasing chart visualization area
• Optimized "Custom SQL" function in metric queries, supporting point limit and time limit returns
• Optimized tag key, tag value, measurement, fieldkey query performance

DataFlux Mobile¶

• Added "Service Agreement" confirmation function, agreeing is required to use the app
• Added node import function, supporting direct viewing of node charts

Data Collection¶

• Added PostgreSQL, MongoDB, network bypass packet capture (pcap), and Alibaba Cloud CDN data sources

March 16, 2020¶

DataFlux Studio¶

• Supported users exiting workspaces; administrators can dissolve workspaces
• In chart analysis mode, line charts support adjusting query conditions; support key events aggregated by time display; support key events list display
• Added overview chart, line chart, bar chart year-over-year comparison function
• Added two chart types: China map, world map
• In new charts, aliases support formatted display, including "only show group name", "only show metric name", etc.
• When creating child nodes, parent node tag filters are not inherited
• Support locking chart time ranges, unaffected by global time ranges
• Optimized table chart, supporting grouped display query methods
• DataFlux now supports Warehouse functionality
• Fixed issues with pie chart, table chart aggregation modes, dashboard, bar chart grouping mode queries
• Optimized integration component filtering methods
• Fixed bug where trigger rules sometimes did not generate trigger history

DataFlux Admin Console¶

• Adding workspaces supports "automatically create new database" and "select existing database" options
• Default workspace names unified to [username]'s workspace

DataFlux.f(x)¶

• "Simple Debug Panel" now supports browsing workspace lists via built-in DataWay
• "Script List" added "Quick View Panel", allowing simultaneous viewing of another script code while editing
• "DataWay Operation Object" now supports data writing with specified Token
• "Script Editor" no longer allows multiple windows to edit the same script
• "@DFF.API" now supports cagetory='check' option for marking check functions
• Fixed known issues and improved system stability

Data Collection¶

• Added 5 new DataWay APIs: /v1/config, /v1/reload, /v1/lua, /v1/lua/list, /v1/license
• Optimized /v1/write/metrics API functionality
• DataWay upgrade now supports merging old and new configuration files, including parameters like remote_host, collect_second, lua_worker, access_key, secret_key that can be passed via command line
• DataWay installation added two parameter supports: DW_ENABLE_CONFIG_API and DW_CONFIG_PASSWORD, for enabling config API and setting config API password. If config API is enabled without specifying a password, installation will fail
• Adjusted DataWay Cache cleanup behavior: limited single cleanup quantity, increased cleanup frequency
• All Lua files unified under <DataWay-installation-directory>/lua, old versions require manual move to this directory, necessary routing configuration changes may be required
• No longer support absolute path Lua configurations, if configured in old versions, manual handling is required
• DataKit allows specifying token via X-Token header to send data to public DataWay

March 13, 2020¶

Data Collection¶

• Added DataWay Android SDK, DataWay iOS SDK, DataWay Python SDK, DataWay JavaScript SDK

March 11, 2020¶

Data Collection¶

• DataKit (v1.0.0-44-gaa4a656) added support for Azure Monitor, Zabbix, Prometheus Exporter, NSQ monitoring metrics, network port scanning, Kubernetes monitoring data, GitHub, and Java middleware (Cassandra, Hadoop-HDFS, Java JVM, JBoss, Kafka, Tomcat, Weblogic, BitBucket) monitoring data collection
• DataWay added support for Logstash, Prometheus data collectors
• Released CSV data collector
• Added Oracle data collection support

March 5, 2020¶

DataFlux Studio¶

• Metric sets support deletion

DataFlux Admin Console¶

• Backend account creation for admin and member accounts added password strength requirements

February 28, 2020¶

DataFlux Studio¶

• Chart queries added custom function mode, supporting development of custom query functions using DataFlux.f(x)
• View variables added sorting function, supporting sorting and previewing variable values during creation and modification
• Optimized SQL mode, supporting direct mapping of UI mode to SQL mode, adding execute button and error feedback
• Optimized chart query function interaction, adding function explanation prompts
• Optimized overview chart value retrieval, supporting selection of aggregation algorithms
• Fixed null data in table charts causing interface non-display issues
• Fixed bar chart top settings sorting issues

DataFlux Admin Console¶

• Management backend accounts added role function, divided into admin and developer roles, developers can only log into DataFlux.f(x) platform

DataFlux.f(x)¶

• Added data source structure browser
• Code editor can switch between published, editing, DIFF views
• Enhanced UI/API asynchronous call support
• Added query function support (category=query)
• Updated built-in official script packages
• Fixed known issues and improved system stability

Data Collection¶

• DataKit added Alibaba Cloud Log Service (SLS) data collection, Alibaba Cloud ActionTrail data collection, Kafka Consumer data collection, route tracing metrics collection, Kubernetes monitoring metrics, Kube Inventory monitoring metrics collection, HTTP performance metrics collection
• Added Oracle data collection
• DataWay compatible with DataX data synchronization tool

February 20, 2020¶

DataFlux Studio¶

• Added command component, supporting creation and settings of command components in charts
• Optimized chart addition interaction and chart categorization, including basic and advanced charts
• Added PNG image export functionality for charts
• Overview chart added mapping function
• Integration added scene category filtering
• When adding or modifying scenes, supported setting scene data ranges
• Optimized chart data display, retaining two decimal places for all float type data
• Supported adding custom icons to nodes
• Trigger rules added action function triggering
• Added DataFlux EBA Agent Builder, which can package and generate dedicated collectors for enterprise behavior analysis scenarios
• System deployment supported setting Continuous Query rules for metric downgrading

DataKit¶

• Added support for agent configuration
• Fixed bugs related to Alibaba Cloud cost collection
• Optimized collection and reporting performance

February 15, 2020¶

DataFlux Studio¶

• Fixed occasional white screen issues in views
• Fixed inability to input custom parameters in prediction functions in analysis mode
• Fixed bar chart not displaying in SQL mode
• Fixed generated scene QR codes in private deployments not scannable on mobile devices
• Fixed previous and next page buttons not clickable in table charts, returning to first page after timed refresh, duplicate data suppression, pagination not displayed when empty after pagination

DataFlux.f(x)¶

• Script editor added read-only mode, entering edit mode upon clicking start editing
• Fixed API interface failure when script functions returned non-standard JSON
• Fixed text and display issues, added page prompts

February 10, 2020¶

• Added scene template templates, supporting creation based on templates, and exporting created scenes as templates
• Scenes support sharing, sharable via DataFlux APP or mini program on mobile devices
• Views support adding flowcharts, videos, real-time video streams
• Added chart combination functionality, combining bar charts with line charts, overview charts with line charts
• Chart query transformation functions support selecting custom transformation functions developed using DataFlux f(x)
• Released DataFlux mobile app, supporting scanning QR codes to view scenes after publication
• Bar charts support Top List display method
• Optimized view display page, supporting dynamic chart loading
• Optimized Dataway management interface monitoring views
• Released DataWay API documentation and SDK
• Metric sets support multiple RP (retention policy)

January 10, 2020¶

• Added Telegraf collector configuration documentation
• DataKit collector added support for snmp, syslog, exec, varnish, NSQ Consumer data sources
• Optimized query performance when too many data points are returned
• Optimized configuration logic for string type metrics when setting trigger rules: removed unsupported aggregation functions for strings; optimized setting of trigger conditions for string count aggregations
• Fixed incorrect redirection after pinning scenes to home page
• Fixed misalignment issues in bar charts with multiple queries
• Fixed data gap issues after deleting custom expressions in new queries