Skip to content

Changelog

1.28.1 (April 22, 2024)

This release is a hotfix that addresses the following issue:

  • Fixed an issue where some crashes were causing the drop of data (#2193).

1.28.0 (April 17, 2024)

This release is an iterative update with the following main changes:

New Features

  • Added cache_get()/cache_set()/http_request() functions to Pipeline, which extend external data sources for Pipeline (#2128).
  • Support for collecting Kubernetes system resource Prometheus metrics has been added, currently in an experimental phase (#2032).
    • Certain cloud-hosted Kubernetes may not be collectible as they have disabled the corresponding feature authorization.

Bug Fixes

  • Fixed the filter logic issue for container logs (#2188).

Feature Enhancements

  • PrometheusCRD-ServiceMonitor now supports TLS configuration (#2168).
  • Improved network interface information collection under Bond mode (#1877).
  • Further optimized Windows Event collection performance (#2172).
  • Optimized field information extraction in Jaeger APM data collection (#2174).
  • Added the log_file_inode field to log collection.
  • New point-pool configuration to optimize Datakit's memory usage under high load scenarios (#2034).
    • Refactored some Datakit modules to optimize garbage collection (GC) overhead, which may slightly increase memory usage under low-load conditions (the additional memory is mainly used for the memory pool).
  • Other documentation adjustments and minor optimizations (#2191/#2189/#2185/#2181/#2180).

1.27.0 (April 3, 2024)

New Features

  • Introduced the Pipeline Offload collector, specialized for centralized processing of Pipelines (#1917).
  • Supported BPF-based HTTP/HTTP2/gRPC network data collection to cover lower versions of Linux kernels (#2017).

Bug Fixes

  • Fixed the default timestamp disorder issue in Point construction (#2163).
  • Fixed potential crashes in Kubernetes collection (#2176).
  • Fixed Node.js Profiling collection issues (#2149).

Feature Enhancements

  • Prometheus Remote Write collection now supports attributing measurement name through metric prefix (#2165).
  • Improved Datakit's own metrics by adding statistics for Goroutine crashes in each module (#2173).
  • Enhanced the bug report feature to support direct upload of info files to OSS (#2170).
  • Optimized the performance of Windows Event collection (#2155).
  • Improved the historical position recording feature in log collection (#2156).
  • Dial testing now supports the option to disable internal network probing (#2142).
  • Various miscellaneous updates and documentation improvements (#2154/#2148/#1975/#2164).

1.26.1 (2024/03/27)

This release is a hotfix release that addresses the following issues:

  • Fixed an issue with Redis not supporting TLS (#2161)
  • Fixed an issue with Trace data timestamps (#2162)
  • Fixed an issue with vmalert writing to Prometheus Remote Write (#2153)

1.26.0 (2024/03/20)

New Features

  • Added Doris collector (#2137)

Bug Fixes

  • Fixed an issue with DDTrace header sampling leading to repeated sampling (#2131)
  • Fixed an problem with missing tags in SQLServer custom collection (#2144)
  • Resolved duplicate collection issue with Kubernetes Events (#2145)
  • Corrected inaccurate container count collection in Kubernetes (#2146)

Enhancements

  • Added upgrade program configuration in datakit.conf, also included fields related to the upgrade program in the host object collector (#2124)
  • Improved bug report feature, attaching self-error information in the appendix (#2132)
  • Optimized TLS settings for MySQL collector and default collector configuration file (#2134)
  • Enhanced logic for host-cloud synchronization global tag configuration, allowing tags synced from the cloud to not be added to global-host-tags (#2136)
  • Added redis-cli command in Datakit image for easier collection of big-key/hot-key in Redis (#2138)
  • Added offset/partition field in data collected from Kafka-MQ (#2140)
  • Miscellaneous updates and documentation enhancements (#2133/#2143)

Version 1.25.0 (2024/03/06)

This release is an iteration release, with the following updates:

New Features

  • Added new HTTP APIs to update global tags dynamically (#2076)
  • Added collection for Kubernetes PersistentVolume / PersistentVolumeClaim resources(and need additional settings for RBAC) (#2109)

Bug Fixes

  • Fixed SkyWalking RUM root-span issue (#2131)
  • Fixed incomplete Windows Event collection issue (#2118)
  • Fixed missing host field in Pinpoint collection (#2114)
  • Fixed RabbitMQ metrics collection issue (#2108)
  • Fixed compatibility issues with older versions of OpenTelemetry (#2089)
  • Fixed line parsing error for Containerd logs (#2121)

Enhancements

  • Improved handling of count data in StatsD by defaulting to floating-point values (#2127)
  • Collector container support for Docker versions 1.24 and above (#2112)
  • Optimized SQLServer collector (#2105)
  • Improved Health Check collector (#2105)
  • Updated default time values for log collection (#2116)
  • Added environment variable ENV_INPUT_CONTAINER_DISABLE_COLLECT_KUBE_JOB to disable Kubernetes Job resource collection (#2129)
  • Updated a batch of built-in dashboard for collectors:
    • ssh (#2125)
    • etcd (#2101)
  • Miscellaneous updates and documentation enhancements (#2119/#2123/#2115/#2113)

1.24.0(2024/01/24)

This release is an iteration release, with the following updates:

New Features

Bug Fixes

  • Fixed a crash issue in Windows Event collector (#2087)
  • Fixed issues with data recording functionality and improved related documentation (#2092)
  • Fixed an issue with DDTrace multi-trace propagation (#2093)
  • Fixed truncation issue in Socket log collection (#2095)
  • Fixed residual main configuration file during Datakit upgrade (#2096)
  • Fixed script overwrite issue during update (#2085)

Feature Enhancements

  • Optimized resource limitation functionality during non-root-user Linux host installation (#2011)
  • Improved matching performance for Sink and blacklist, significantly reducing memory consumption (10X) (#2077)
  • Log Streaming add support for FireLens (#2090)
  • Added log_read_lines field in Log Forward log collection (#2098)
  • Optimized handling of tag cluster_name_k8s in K8s (#2099)
  • Added restart count metric (restarts) in K8s Pod metric
  • Optimized measurement kubernetes by adding container statistics
  • Optimized Kubelet metric collection

1.23.1(2024/01/12)

This release is a Hotfix release, which fixes the following issues:

  • Fix Datakit service error under Windows

1.23.0(2024/01/11)

This release is an iteration release, with the following updates:

New Features

  • Support configuring any collector's configure via environment variable (ENV_DATAKIT_INPUTS) for Kubernetes deployment (#2068)
  • Container collector now supports more fine-grained configuration by converting Kubernetes object labels to tags (#2064)
    • ENV_INPUT_CONTAINER_EXTRACT_K8S_LABEL_AS_TAGS_V2_FOR_METRIC: support converting labels to tags for metric data
    • ENV_INPUT_CONTAINER_EXTRACT_K8S_LABEL_AS_TAGS_V2: support converting labels to tags for non-metric data (e.g. objects/logging)

Bug Fixes

  • Fixed errors with deployment and daemonset fields in container collector (#2081)
  • Fixed issue where container log collection would lose the last few lines of logs when a container briefly ran and exited (#2082)
  • Fixed slow query SQL time error in Oracle collector (#2079)
  • Fixed issue with instance setting in Prom collector (#2084)

Enhancements

  • Enhanced Prometheus Remote Write collection (#2069)
  • eBPF collection now supports setting resource usage (#2075)
  • Optimized profiling data collection (#2083)
  • MongoDB collector now supports separate configuration for username and password (#2073)
  • SQLServer collector now supports configuring instance name (#2074)
  • Optimized dashboard and monitors of ElasticSearch collector (#2058)
  • KafkaMQ collector now supports multi-threaded mode (#2051)
  • SkyWalking collector now supports Meter data type (#2078)
  • Updated some collector documentation and other bug fixes (#2074/#2067)
  • Optimized upgrade command for Proxy installation (#2033)

1.22.0 (2023/12/28)

This release is an iteration release, with the following updates:

New Features

  • Added OceanBase custom SQL collection(#2046)
  • Added blacklist/whitelist for Prometheus Remote(#2053)
  • Added node_name tag to Kubernetes resource count collection (only supported for Pod resources)(#2057)
  • Added cpu_limit_millicores/mem_limit/mem_used_percent_base_limit fields to Kubernetes Pod metrics
  • Added bpf-netlog plugin to eBPF collector(#2017)
  • Add environments configure for recorder under Kubernetes

Bug Fixes

  • Fixed zombie process issue with external collector(#2063)
  • Fixed conflicts with container log tags(#2066)
  • Fixed failure to retrieve virtual NIC information(#2050)
  • Fixed issues with Pipeline Refer Table and IPDB(#2045)

Improvements

  • Improved field/tag extraction(whitelist) for DDTrace and OTEL(#2056)
  • Improved SQL retrieval for SQLServer collector's sqlserver_lock_dead metric(#2049)
  • Update SDK for PostgreSQL collector(#2044)
  • Update default configuration for ElasticSearch collector(#2048)
  • Added additional ENV configuration options during K8s installation(#2025)
  • Optimized DataKit exported Prometheus metrics
  • Updated integration documentation for some collectors

1.21.1(2023/12/21)

This release is a Hotfix release, which fixes the following issues:

  • Fixed issue of Prometheus Remote Write not adding Datakit host tags to keep compatibility with older configurations(#2055)
  • Fixed issue of default log collection in a batch of middleware not including host tags
  • Fixed issue on remove color characters within Chinese characters while collecting logging

1.21.0 (2023/12/14)

This release is an iteration release, with the following updates:

New Features

Bug Fixes

  • Fixed version detection issue of PostgreSQL collector (#2040)
  • Fixed account permission setting issue of ElasticSearch collector (#2036)
  • Fixed directory crash issue of Host Dir collector (#2037)

Improvements

  • DDTrace collector removed duplicate tags in message.Mate (#2010)
  • Optimized path search strategy for logs inside containers (#2027)
  • Added datakit_version field and set collection time to the start time of the task for dial testing collector (#2029)
  • Removed datakit export command to decrease Datakit binary package size (#2024)
  • Added time series count for debugging collector configuration (#2016)
  • Profile collection now uses disk caching to implement asynchronous reporting (#2041)
  • Update install script under Windows (#2026)
  • Updated a batch of built-in dashboard and monitors for collectors

Breaking Changes

  • DDTrace collector no longer extracts all fields by default, which may result in missing data for some page's custom fields. Specific fields can be extracted by writing a Pipeline or using the new JSON lookup syntax (message@json.meta.xxx).

1.20.1(2023/12/07)

This release is a Hotfix release, which fixes the following issues:

Bug fix

  • Fixed DDTrace sampling bug
  • Fixed error_message lost information
  • Fixed Kubernetes Pod data collection bug

1.20.0(2023/11/30)

This release is an iterative release with the following updates:

New addition

  • Redis collector added hotkey info(#2019)
  • Command datakit monitor add playing support for metrics from Bug Report(#2001)
  • Oracle collector added custom queries(#1929)
  • Container logging files support wildcard match(#2004)
  • Kubernetes Pod add network and storage info(#2022)
  • RUM added filtering for session replays data(#1945)

Fix

  • Fixed cgroup panic error in some environments(#2003)
  • Fixed Windows installation script execution failure under PowerShell(#1997)
  • Fix disk cache default enabled bug(#2023)
  • Update naming for Prometheus metrics from Kubernetes Auto-Discovery(#2015)

Function optimization

  • Optimized built-in dashboard and monitor for MySQL/PostgreSQL/SQLServer(#2008/#2007/#2013/#2024)
  • Optimized Prom collector's metrics name(#2014)
  • Optimized Proxy collector and release basic benchmark(#1988)
  • Container logging tags add support for Pod Labels(#2006)
  • Set NODE_LOCAL as the default mode when collecting Kubernetes data(and need additional settings for RBAC)(#2025)
  • Optimized tracing handle(on memory usage)(#1966)
  • Update PinPoint collector(#1947)
  • Enable dropping message to save storage for APM(#2021)

1.19.2(2023/11/20)

This release is a Hotfix release, which fixes the following issues:

Bug fix{#cl-1.19.2-fix}

  • Fix diskcache bug that drop data on session replay
  • Add Prometheus metrics on collecting Kubernetes related data

1.19.1(2023/11/17)

This release is a Hotfix release, which fixes the following issues:

Bug fix{#cl-1.19.1-fix}

  • Fix bug on open diskcache(issue)

1.19.0(2023/11/16)

This release is an iterative release with the following updates:

New addition

Fix

  • Fixed invalid resource limits for old Windows(#1987)
  • Fix dialtesting ICMP issues(#1998)

Function optimization

  • Optimized statsd collection(#1995)
  • Optimized Datakit installation script(#1979)
  • Optimize MySQL dashboard(#1974)
  • Add more Prometheus metrics, such as Golang runtime(#1971/#1969)
  • Update documents and unit test optimization(#1952/#1993)
  • Improved Redis collector and added more metrics(#1940)
  • Allow to add packet(ASCII text only) detection in TCP dial testing(#1934)
  • Optimized installation for non-root users:
    • Ignore ulimit setup failure(#1991)
    • Add documentation on features(eBPF) that run under non-root user(#1989)
    • Update the requirements for non-root installation(#1990)
  • Add support for MongoDB old version 2.8.0(#1985)
  • Add support for RabbitMQ old versions (3.6.X/3.7.X)(#1944)
  • Add support to collecting Pod metrics via kubelet instead of Metric Server(#1972)
  • Add support to set measurement name on collecting Prometheus metrics under Kubernetes(#1970)

Compatible adjustment

  • Remove feature that write point data to Datakit local files(#1738)

1.18.0(2023/11/02)

This release is an iterative release with the following updates:

New addition

  • Added OceanBase Collector(#1924)

Fix

  • Fixed compatibility of large Tag values in Tracing data, now adjusted to 32MB(#1932)
  • Fix RUM session replay dirty data issue(#1958)
  • Fixed indicator information export issue(#1953)
  • Fix the v2 version protocol build error

Function optimization

  • Added mount points and other indicators in host directory Collection and Disk Collection(#1941)
  • KafkaMQ supports OpenTelemetry Tracing for data processing(#1887)
  • Added more information collection in Bug Report(#1908)
  • Improved self-index exposure during Prom collection(#1951)
  • Update default IP library to support IPv6(#1957)
  • Update image name Download address is pubrepo.guance.com(#1949)
  • Optimized log capture file location function(#1961)
  • Kubernetes
    • Support Node-Local Pod collection(metric&object) to relieve pressure on election nodes(#1960)
    • Container log collector supports more filtering(#1959)
    • Added Service related metric collecting(#1948)
    • Support for selecting labels from PodMonitor and ServiceMonitor(#1963)
    • Support for converting Node labels to tags on Node object(#1962)

Compatible adjustment

  • Kubernetes no longer collects CPU&memory metrics for Pods created by Job/CronJob(#1964)

1.17.3(2023/10/31)

This release is a Hotfix release, which fixes the following issues:

Bug fix{#cl-1.17.3-fix}

  • Fix Pipeline not working for logging(#1954)
  • Fix eBPF not working under arm64(#1955)

1.17.2(2023/10/27)

This release is a Hotfix release, which fixes the following issues:

Bug fix{#cl-1.17.2-fix}

  • Fix logging input that missing host global tags(#1942)
  • Fix RUM session replay uploading(#1943)
  • Fix point encoding error on non-UTF8 string

1.17.1(2023/10/26)

This release is a Hotfix release, which fixes the following issues:

Bug fix{#cl-1.17.1-fix}

  • Fix dialtesting bug that do not upload data(#1931)

New features

  • eBPF can also build APM data to trace process/thread relationship under Linux(#1835)
  • Pipeline add new function pt_name(#1937)

Features Optimizations

  • Optimize point build to save CPU and memory(#1792)

1.17.0(2023/10/19)

This release is an iterative release, mainly including the following updates:

New features

  • Added cpu_limt for Pod (#1913)
  • Added New Relic tracing collector (#1834)

Bug fixes

  • Fixed the memory issue that may be caused by too long single-line data in the log (#1923)
  • Fixed an issue where disk collector failed to obtain the disk mount point (#1919)
  • Fixed the issue of inconsistent service names in helm and yaml (#1910)
  • Fixed the missing agentid field in pinpoint spans (#1897)
  • Fixed the bugs in goroutine group (#1893)
  • Fixed the empty data of MongoDB collector (#1884)
  • Fixed a large number of 408 and 500 status codes in the request of the rum collector (#1915)

Function optimization

  • Optimized the exit logic of logfwd to avoid program exit due to configuration errors affecting business pods (#1922)
  • Optimized the ElasticSearch collector, add index metric set elasticsearch_indices_stats for shard and replica metrics (#1921)
  • Added the disk integration test (#1920)
  • DataKit monitor supports HTTPS (#1909)
  • Added slow query logs for Oracle collector (#1906)
  • Optimized collector point implementation (#1900)
  • Added detection of authorization for MongoDB collector integration test (#1885)

1.16.1(2023/10/09)

Bug fixes

  • Container(#1895)
    • Fixed failed to get CPU metrics
    • Fixed bug on handing multi-line logging text under containerd
  • Fixed Prom collector eat too many memory bug(#1905)

Breaking Changes

  • Dash(-) will no longer be replaced with _ in all tracing collectors. This change was made to avoid problems when associate tracing and logging with these dash-named keys(#1903)
  • All Prometheus exporter collector by default uses streaming mode to avoid eat too much memory on collecting large exporter URLs.

1.16.0(2023/09/21)

This release is an iterative release, mainly including the following updates:

New features

  • Added the Neo4j collector (#1846)
  • The RUM collector has added API for uploading, deleting, and verifying sourcemap files, and removed the sourcemap upload and deletion API from the DCA service (#1860)
  • Added a monitoring view and detection libraries for the IBM Db2 collector（#1862）

Bug fixes

  • Fixed an issue where environment variable ENV_GLOBAL_HOST_TAGS couldn't fetch the hostname of the machine by __datakit_hostname (#1874)
  • Fixed an issue where the open_files field was missing from the metrics data of the host_processes collector (#1875)
  • Fixed an issue where the Pinpoint collector had a large number of empty resources and was using too much memory (#1857 #1849)

Function optimization

  • Optimized the efficiency of Kubernetes metrics collection and object collection (#1854)
  • Optimized the metrics output of log collection (#1881)
  • The Kubernetes Node object collector has added two new fields: unschedulable and node_ready (#1886)
  • The Oracle collector now supports Linux ARM64 architecture (#1859)
  • The logstreaming collector has added integration tests (#1570)
  • The Datakit development documentation added content about IBM Db2 collector (#1870)
  • Improve the documentation of the Kafka and MongoDB collectors (#1883)
  • When creating a monitoring account for MySQL, MySQL 8.0+ now defaults to use the caching_sha2_password encryption method (#1882)
  • Optimized the syslog file size issue in the bug report command (#1872)

Breaking Changes

  • Removed the sourcemap file upload and deletion API from the DCA service and moved them to the RUM collector

1.15.1(2023/09/12)

Bug fix

  • Fix the bug of repeated collection of logfwd

1.15.0 (2023/09/07)

This release is an iterative release, mainly including the following updates:

New features

Bug fixes

  • Fix the double star(**) problem of container acquisition configuration include/exclude (#1855)
  • Fixed field error in Kubernetes service object data

Function optimization

Compatibility adjustments

1.14.2(2023/09/04)

Bug fixes

  • Fix instance tag missing for Prometheus metrics on Kubernetes Pod's Annotation
  • Fix Kubernetes pod missing bug

Bug fixes

  • Optimize Prometheus metrics collecting(streaming collection) in Kubernetes to avoid possible large memory usage(#1853/#1845)

  • Fix colored logging

    • For Kubernetes, the environment key is ENV_INPUT_CONTAINER_LOGGING_REMOVE_ANSI_ESCAPE_CODES

1.14.0 (2023/08/24)

This release is an iterative release, mainly including the following updates:

New features

  • Added collector NetFlow (#1821)
  • Added Filter(Blacklist) Debugger (#1787)
  • Added Kubernetes StatefulSet metrics and object collection, added replicas_desired object field (#1822)
  • Added DK_LITE environment variable for installing DataKit Lite (#123)

Bug fixes

  • Fixed the problem that Container and Kubernetes collection did not add HostTags and ElectionTags correctly (#1833)
  • Fixed MySQL the problem that the indicator cannot be collected when the custom collection Tags is empty (#1835)

Function optimization

  • Added the process_count metric in the System collector to indicate the number of processes on the current machine (#1838)
  • Remove open_files_list field in Process collector (#1838)
  • Added the handling case of index loss in the collector document of host object (#1838)
  • Optimize the Datakit view and improve the Datakit Prometheus indicator documentation
  • Optimize the mount method of Pod/container log collection (#1844)
  • Add Process and System collector integration tests (#1841/#1842)
  • Optimize etcd integration tests (#1847)
  • Upgrade Golang 1.19.12 (#1516)
  • Added Install DataKit via ash command(#123)
  • RUM supports custom indicator set, the default indicator set adds telemetry (#1843)

Compatibility adjustments

  • Remove the Sinker function on the Datakit side and transfer its function to Dataway side implementation (#1801)
  • Remove pasued and condition fields from Kubernetes Deployment metrics data, and add object data paused field

1.13.2 (2023/08/15)

Bug fixes

  • Fix MySQL custom collection failure. (#1831)
  • Fix Prometheus Export has Service scope and execution errors. (#1828)
  • Unexpected HTTP response codes and delays with the eBPF collector. (#1829)

Function optimization

  • Improve the value of the image field for container collection. (#1830)
  • MySQL integration test optimization to improve test speed. (#1826)

1.13.1 (2023/08/11)

  • Fix container log source field naming issue (#1827)

1.13.0 (2023/08/10)

This release is an iterative release, mainly including the following updates:

New features

  • Host Object Collector supports debug commands(#1802)
  • KafkaMQ adds support for external plugin handle function(#1797)
  • Input container supports cri-o runtime(#1763)
  • Pipeline adds create_point function for metrics(#1803)
  • Added PHP profiling support(#1811)

Bug fixes

  • Fix Cat collector NPE exception.
  • Fix the http response_download time of the dial test collector. (#1820)
  • Fixed the problem that containerd log collection did not splice partial logs normally. (#1825)
  • Fix eBPF collector ebpf-conntrack plug-in probe failure. (#1793)

Function optimization

  • Bug-report command optimization (#1810)
  • The RabbitMQ collector supports multiple simultaneous runs. (#1756)
  • Host object grabber tweaks. Remove the state field. (#1802)
  • Optimize the error reporting mechanism. Solve the problem that the eBPF collector cannot report errors. (#1802)
  • The Oracle external collector has been added to send information to the center in the event of an error. (#1802)
  • Optimize Pythond documentation, add module not found solution case. (#1807)
  • Added global tag integration test cases for some collectors. (#1791)
  • Optimize Oracle integration tests. (#1802)
  • OpenTelemetry adds metrics sets and dashboards.
  • Adjust the k8s event field. (#1766)
  • Added new container collection field. (#1819)
  • eBPF collector adds flow field to httpflow. (#1790)

1.12.3 (2023/08/03)

  • Fixed the problem of delayed release of log collection files under Windows (#1805)
  • Fix the problem that the head log of the new container is not collected
  • Fixed several regular expressions that could cause crashes (#1781)
  • Fix the problem that the installation package is too large (#1804)
  • Fix the problem that the log collector may fail to open the disk cache

1.12.2 (2023/07/31)

  • Fix OpenTelemetry Metric and Trace routing configuration issues

1.12.1 (2023/07/28)

  • Fix the old version of DDTrace Python Profile access problem (#1800)

1.12.0 (2023/07/27)

This release is an iterative release, mainly including the following updates:

New features

  • HTTP API Add sourcemap file upload (#1782)
  • Added support for .net Profiling access (#1772)
  • Added Couchbase collector (#1717)

Bug fixes

  • Fix the problem that the owner field is missing in the dial test collector (#1789)
  • Fixed the missing host problem of the DDTrace collector, and changed the tag collection of various Traces to a blacklist mechanism 1 (#1776)
  • Fix RUM API cross domain issue (#1785)

Function optimization

  • Optimize SNMP collector encryption algorithm identification method; optimize SNMP collector documentation, add more example explanations (#1795)
  • Add Pythond collector Kubernetes deployment example, add Git deployment example (#1732)
  • Add InfluxDB, Solr, NSQ, Net collector integration tests (#1758/#1736/#1759/#1760)
  • Add Flink metrics (#1777)
  • Extend Memcached, MySQL metrics collection (#1773/#1742)
  • Update Datakit's own indicator exposure (#1492)
  • Pipeline adds more operator support (#1749)
  • Dial test collector
    • Added built-in dashboard for dial test collector (#1765)
    • Optimized the startup of dial test tasks to avoid concentrated consumption of resources (#1779)
  • Documentation updates (#1769/#1775/#1761/#1642)
  • Other optimizations (#1777/#1794/#1778/#1783/#1775/#1774/#1737)

1.11.0 (2023/07/11)

This release is an iterative release, including the following updates:

New features

  • Added dk collector, removed self collector (#1648)

Bug fixes

  • Fix the problem of timeline redundancy in the Redis collector (#1743), improve the integration test
  • Fix Oracle collector dynamic library security issue (#1730)
  • Fix DCA service startup failure (#1731)
  • Fix MySQL/ElasticSearch collector integration test (#1720)

Function optimization

  • Optimize etcd collector (#1741)
  • StatsD collector supports configuration to distinguish different data sources (#1728)
  • Tomcat collector supports version 10 and above, Jolokia is deprecated (#1703)
  • Container log collection supports configuring files in the container (#1723)
  • SQLServer collector index improvement and integration test function refactoring (#1694)

Compatibility adjustments

The following compatibility modifications may cause problems in data collection. If you use the following functions, please consider whether to upgrade or adopt a new corresponding solution.

  1. Remove deployment tag from container logs
  2. Remove the logic that the source field of the container stdout log is named after short_image_name. Now just use the container name or the label io.kubernetes.container.name in Kubernetes to name 2.
  3. Remove the function of collecting the external file path through the container label (datakit/logs/inside), and change it to container environment variable (DATAKIT_LOGS_CONFIG) way to achieve 3.

1.10.2 (2023/07/04)

  • Fixed prom collector recognition problem in Kubernetes

1.10.1 (2023/06/30)

  • Fix OpenTelemetry HTTP routing support customization
  • Fix the problem that the field of started_duration in the host process object is missing

1.10.0 (2023/06/29)

This release is an iterative release, including the following updates:

Bug fixes

  • Fix profiling data upload problem in Proxy environment (#1710)
  • Fixed the problem that the default collector is enabled during the upgrade process (#1709)
  • Fixed the log truncated problem in SQLServer collection data (#1689)
  • Fix the problem of Metric Server indicator collection in Kubernetes (#1719)

Function optimization

  • KafkaMQ supports multi-line cutting configuration at topic level (#1661)
  • When Kubernetes DaemonSet is installed, it supports modifying the number of Datakit log shards and shard size through ENV (#1711)
  • Added memory_capacity and memory_used_percent two fields for Kubernetes Pod metrics and object collection (#1721)
  • OpenTelemetry HTTP routing supports customization (#1718)
  • Oracle collector optimizes the missing problem of the oracle_system index set, optimizes the collection logic and adds some indexes (#1693)
  • Pipeline adds in operator, adds value_type() and valid_json() functions, adjusts the behavior of load_json() function after deserialization fails (#1712)
  • Added started_duration field for collection in host process object (#1722)
  • Optimize the logic of dial test data sending (#1708)
  • Update more integration tests (#1666/#1667/#1668/#1693/#1599/#1573/#1572/#1563/#1512/#1715)
  • Module refactoring and optimization (#1714/#1680/#1656)

Compatibility adjustments

  • Changed the timestamp unit of Profile data from nanoseconds to microseconds (#1679)

1.9.2 (2023/06/20)

This release is an iterative mid-term release, adding some functions for docking with the center and some bug fixes and optimizations:

New features

Bug fixes

  • Fix some bugs that could cause crashes (!2249)
  • Added Host header support for HTTP network dialing test and fixed random error reporting (#1676)
  • Fix automatic discovery of Pod Monitor and Service Monitor in Kubernetes (#1695)
  • Fixed Monitor issues (#1702/!2258)
  • Fix Pipeline data mishandling bug (#1699)

Function optimization

  • Add more information in Datakit HTTP API return for easier troubleshooting (#1697/#1701)
  • Miscellaneous refactoring (#1681/#1677)
  • RUM collector adds more Prometheus metrics exposure (#1545)
  • Enable Datakit's pprof function by default, which is convenient for troubleshooting (#1698)

Compatibility Adjustments

  • Remove support for logging collection from Kubernetes CRD guance.com/datakits v1bate1 (#1705)

1.9.1 (2023/06/13)

This release is a bug fix, mainly fixing the following issues:

  • Fix DQL query issue (#1688)
  • Fix the crash problem that may be caused by high-frequency writing of the HTTP interface (#1678)
  • Fix datakit monitor command parameter override issue (!2232)
  • Fixed retry error when uploading data via HTTP (#1687)

1.9.0 (2023/06/08)

This release is an iterative release, mainly including the following updates:

New features

Bug fixes

  • Fix the connection leak problem caused by Prometheus indicator collection in K8s (#1662)

Function optimization

  • K8s DaemonSet object adds age field (#1670)
  • Optimize PostgreSQL startup settings (#1658)
  • Added /v3/log/ support for SkyWalking (#1654)
  • Optimize log collection processing (#1652/#1651)
  • Optimize Update Documentation (#1653)
  • Other refactorings and optimizations (#1673/#1650/#1630)
  • Added some integration tests (#1440/#1429)
    • PostgreSQL
    • Network dial test

1.8.1 (2023/06/01)

This release is a bug fix, mainly fixing the following issues:

  • Fix the crash problem when KafkaMQ is multi-opened (#1660)
  • Fixed the problem of incomplete collection of disk devices in DaemonSet mode (#1655)

1.8.0 (2023/05/25)

This release is an iterative release, mainly including the following updates:

New features

  • Datakit adds two debugging commands, which are convenient for users to write glob and regular expressions during configuration (#1635)
  • Added two-way transparent transmission of Trace ID between DDTrace and OpenTelemetry (#1633)

Bug fixes

  • Fix dialing pre-check issue (#1629)
  • Fix two field issues in SNMP collection (#1632)
  • Fixed the default port conflict between the upgrade service and other services (#1646)

Function optimization

  • When eBPF collects Kubernetes network data, it supports converting Cluster IP to Pod IP (need to be opened manually) (#1617)
  • Added a batch of integration tests (#1430/#1574/#1575)
  • Optimize container network related metrics (#1397)
  • Bug report function adds crash information collection (#1625)
  • PostgreSQL collector
    • Add custom SQL metrics collection (#1626)
    • Add DB level tag (#1628)
  • Optimize the host field problem of localhost collection (#1637)
  • Optimize Datakit's own metrics, add Datakit's own metrics document (#1639/#1492)
  • Optimize Prometheus metrics collection on Pod, automatically support all Prometheus metrics types (#1636)
  • Added Performance Test Document collected by Trace class (#1616)
  • Added Kubernetes DaemonSet object collection (#1643)
  • Pinpoint gRPC service supports x-b3-traceid to transparently transmit Trace ID (#1605)
  • Optimize cluster election strategy (#1534)
  • Other optimizations (#1609#1624)

Compatibility adjustments

  • In container collector, remove kube_cluster_role object collection (#1643)

1.7.0 (2023/05/11)

This release is an iterative release, mainly including the following updates:

New features

  • RUM Sourcemap adds applet support (#1608)
  • Added a new collection election strategy to support Cluster-level elections in the K8s environment (#1534)

Bug fixes

  • When Datakit uploads, if the center returns a 5XX status code, the number of Layer 4 connections will increase. This version fixes the problem. At the same time, datakit.conf (K8s can be configured through environment variable configuration ) to expose more connection-related configuration parameters (DK001-15)

Function optimization

  • Optimize the collection of process objects. Some fields that may cause high consumption (such as the number of open files/ports) are closed by default. These fields can be manually enabled through the collector configuration or environment variables. These fields may be important, but we still believe that by default, this should not cause unexpected performance overhead on the host (#1543)
  • Datakit's own indicator optimization:
    • Added Prometheus index exposure for the dial test collector, which is convenient for troubleshooting some potential problems of the dial test collector itself (#1591)
    • Increase the HTTP level indicator exposure when Datakit reports (#1597)
    • Increased indicator exposure during KafkaMQ collection
  • Optimize the collection of PostgreSQL indicators, and add more related indicators (#1596)
  • Optimize JVM-related metrics collection, mainly document updates (#1600) -Pinpoint
    • Add more developer documentation (#1601)
    • Pinpoint fix gRPC Service support (#1605)
  • Optimize the discrepancy of disk index collection on different platforms (#1607)
  • Other engineering optimizations (#1621/#1611/#1610)
  • Added several integration tests (#1438/#1561/#1585/#1435/#1513)

1.6.1 (2023/04/27)

This release is a Hotfix release, which fixes the following issues:

  • The blacklist may not take effect when the old version is upgraded (#1603)
  • Prom collecting info type data problem (#1544)
  • Fix data loss problem caused by Dataway Sinker module (#1606)

1.6.0 (2023/04/20)

This release is an iterative release, mainly including the following updates:

New features

Function optimization

  • Optimize the output method of Windows installation script and upgrade script, so that it is easy to paste and copy directly in the terminal (#1557)
  • Optimize Datakit's own document construction process (#1578)
  • Optimize OpenTelemetry field handling (#1514)
  • Prom Collector supports collecting labels of type info and appending them to all associated indicators (enabled by default) (#1544)
  • In system collector, add CPU and memory usage percentage indicators (#1565)
  • Datakit adds data point markers (X-Points) in the sent data to facilitate the construction of central related indicators (#1410)
    • In addition, the User-Agent tag of Datakit HTTP has been optimized and changed to datakit-<os>-<arch>/<version>.
  • KafkaMQ:
    • Support for processing Jaeger data (#1526)
    • Optimize the processing of SkyWalking data (#1530)
    • Added third-party RUM access function (#1581)
  • SkyWalking added HTTP access function (#1533)
  • Add the following integration tests:
  • Apache(#1553)

Bug fixes

  • Fix JSON format cannot recognize time precision when writing data (#1567)
  • Fix the problem that the dial test collector does not work (#1582)
  • Fix eBPF validator issue on Euler system (#1568)
  • Fix RUM sourcemap segfault (#1458)

Compatibility adjustments

  • Remove the old command line style, for example, the original datakit --version will no longer work and must be replaced by datakit version. For details, see Usage of various commands

1.5.10(2023/04/13)

This release is an emergency release and includes the following updates:

New Features

Feature Optimization

  • Optimized Pipeline execution performance, with approximately 30% performance improvement
  • Optimized logging position handle under container(#1550)

1.5.9 (2023/04/06)

This release is an iteration release and includes the following updates:

New Features

  • Added a remote service to manage Datakit upgrades (#1441)
  • Added a troubleshooting feature (#1377)

Bug Fixes

  • Fixed CPU metrics collection for Datakit to match the CPU usage reported by the monitor and top commands (#1547)
  • Fixed a panic error in the RUM input(#1548)

Feature Optimization

  • Optimized the upgrade function to avoid damaging the datakit.conf file (#1449)
  • Optimized the cgroup configuration and removed the minimum CPU limit (#1538)
  • Optimized the self input to allow users to choose whether or not to enable it, and also improved its performance (#1386)
  • Simplified monitor due to the addition of new troubleshooting methods (#1505)
  • Added the ability to add an instance tag to the Prom input to maintain compatibility with the native Prometheus system (#1517)
  • Added Kubernetes deployment method to DCA (#1522)
  • Improved the disk cache performance of logging input(#1487)
  • Improved the Datakit metrics system to expose more Prometheus metrics (#1492)
  • Optimized the /v1/write API (#1523)
  • Optimized error prompt related to tokens during installation (#1541)
  • The monitor can now automatically retrieve connection addresses from datakit.conf (#1547)
  • Removed eBPF kernel version checks to support more kernel versions (#1542)
  • Added the ability for the Kafka subscription collection to handle multiple lines of JSON (#1549)
  • Added a large batch of integration tests (#1479/#1460/#1436/#1428/#1407)
  • Optimized the configuration of the IO module and added a configuration field for the number of upload workers (#1536)

Breaking Changes

  • Most of the Sinker features have been removed from this release except the Sinker Dataway (#1444). The host installation and Kubernetes installation configurations for Sinker have also been adjusted, and the configuration method is different from before. Please note that when upgrading.
  • Due to performance issues, the previous version of the failed-upload-disk-cache has been replaced with a new implementation. The binary format of the cache is no longer compatible, and if you upgrade, the old data will not be recognized. It is recommended that you manually delete the old cache data (old data may affect the new version of disk cache) before upgrading to the new version of Datakit. Nevertheless, the new version of disk cache is still an experimental feature, so use it with caution.
  • The Datakit metrics system has been updated, which may cause some metrics obtained by DCA to be missing, but this does not affect the core-features of DCA itself.

1.5.8(2023/03/24)

This release is an iterative release, mainly for bug fixes and feature improvements.

Bug Fixes

  • Fix the issue of possible loss of container log collection (#1520)
  • Automatically create the Pythond directory after Datakit startup (#1484)
  • Remove the singleton restriction of the hostdir input(#1498)
  • Fix a problem with the eBPF numeric construction (#1509)
  • Fix the issue of parameter recognition in the Datakit monitor (#1506)

Feature Optimization

  • Add memory-related metrics for the Jenkins input(#1489)
  • Improve support for cgroup v2 (#1494)
  • Add an environment variable (ENV_CLUSTER_K8S_NAME) to configure the cluster name during Kubernetes installation (#1504)
  • Pipeline
    • Add protective measures to the kv_split() function to prevent data inflation (#1510)
    • Optimize the functionality of json() and delete() for processing JSON keys.
  • Other engineering optimizations (#1500)

Documentation Adjustments

  • Add documentation for full offline installation of Kubernetes (#1480)
  • Improve documentation related to statsd and ddtrace-java (#1481/#1507)
  • Supplement documentation related to TDEngine (#1486)
  • Remove outdated field descriptions from the disk input documentation (#1488)
  • Improve documentation for the Oracle input(#1519)

1.5.7(2023/03/09)

This release is an iterative release with the following updates:

New Features

Bug Fixes

  • Fix Pipeline related bugs(#1476/#1469/#1471/#1466)
  • Fix datakit.yaml missing request field, this may cause Datakit pod always pending(#1470)
  • Disable always-retrying on cloud synchronous, this produce a lot of warning logging(#1433)
  • Fix encoding error in logging history cache file(#1474)

Features Optimizations

  • Optimize Point Checker(#1478)
  • Optimize Pipeline function replace() performance (#1477)
  • Optimize Datakit installation under Windows(#1406)
  • Optimize Confd configuration($1402)
  • Add more testing on Filebeat(#1459)
  • Add more testing on Nginx(#1399)
  • Refactor OTEL agent(#1409)
  • Update Datakit Monitor(#1261)

1.5.6(2023/02/23)

This release is an iterative release with the following updates:

New Features

  • Added Parsing Line Protocol in DataKit command line(#1412)
  • Added resource limit in datakit.yaml and Helm (#1416)
  • Add CRD deployment support in datakit.yaml and Helm (#1415)
  • Added SQLServer integration testing (#1406)
  • Add Resource CDN Annotation in RUM(#1384)

Bug Fixes

  • Fixed RUM request return HTTP 5XX issue (#1412)
  • Fixed logging collecting path error (#1447)
  • Fixed K8s Pod's field(restarts) issue (#1446)
  • Fixed DataKit crash in filter module (#1422)
  • Fixed tag-key-naming during Point building(#1413#1408)
  • Fixed Datakit Monitor charset issue (#1405)
  • Fixed OTEL tag override issue (#1396)
  • Fixed public API white list issue (#1467)

Features Optimizations

  • Optimized Dial-Testing on invalid task(#1421)
  • Optimized command-line prompt on Windows (#1404)
  • Optimized Windows Powershell script template (#1403)
  • Optimized Pod/ReplicaSet/Deployment's relationship in K8s (#1368)
  • Partially apply new Point constructor (#1400)
  • Add eBPF support on default installing (#1448)
  • Add CDN support during install downloading (#1457)

Breaking Changes

  • Removed unnecessary datakit install --datakit-ebpf (#1400) due to built-in eBPF input.

  1. Various types of Trace will carry various business fields (called Tag, Annotation or Attribute, etc.) on its data. In order to collect more data, Datakit accepts these fields by default. ↩

  2. In Kubernetes, the value of io.kubernetes.container.name remains unchanged, and in the host container, the container name does not change much, so it is no longer used The original image name as the source for the source field. ↩

  3. It is more convenient to add environment variables to the container than to modify the label of the container (in general, the image needs to be rebuilt) (when starting the container, just inject the environment variable ). ↩

Feedback

Is this page helpful? ×