Skip to content

External Data Sources

By using DataFlux Func, you can connect various types of external data sources such as MySQL and Prometheus to Guance. Direct connections to TiDB Cloud Lake are also supported, enabling unified querying and visualization of data.

Features

  • Native Querying: Use the data source’s native query language directly in charts without any additional transformation.
  • Data Protection: Connection information for Func data sources is stored locally in Func; the Lake DSN for TiDB Cloud Lake is encrypted and stored by the platform. Passwords and full DSNs are not displayed in lists, details, or queries.
  • Custom Management: Easily add and manage various external data sources based on actual needs.
  • Real-time Data: Connect directly to external data sources to obtain data in real time, enabling immediate response and decision-making.

Connection Methods

Connection Method Applicable Data Sources Description
Via DataFlux Func MySQL, Prometheus, etc. Reuse an existing DataFlux Func deployment and Connector. Existing configurations such as the ID must remain unchanged.
TiDB Cloud Lake Direct Connection TiDB Cloud Lake Connected by the Guance server via the TiDB Cloud Lake Driver. Does not depend on DataFlux Func, and no data source ID is required.

For direct connection to TiDB Cloud Lake, refer to Connect and Query TiDB Cloud Lake.

Connecting via DataFlux Func

Add a Data Source in Guance

This means directly adding or viewing the connected DataFlux Func in Integrations and further managing all connected external data sources.

Note

This method is more beginner-friendly than the second path and is recommended.

  1. Select DataFlux Func from the dropdown.
  2. Choose the supported data source type.
  3. Define connection properties, including ID, data source title, associated host, port, database, user, and password.
  4. Test the connection as needed.
  5. Save.

Query External Data Sources Using Func

Note

"External data sources" here have a broad definition, including common external data storage systems (e.g., MySQL, Redis, etc.) as well as third-party systems (e.g., the Guance console).

Prerequisites

You need to download the corresponding installation package and follow the Quick Start guide to deploy the Func platform.

After deployment, wait for initialization to complete and log in to the platform.

Associate Func with Guance

Connectors help developers connect to the Guance system.

Navigate to Development > Connectors > Add Connector:

  1. Select the connector type.
  2. Customize the connector ID.
  3. Add a title. This title will be displayed in the Guance workspace.
  4. Optionally enter a description for the connector.
  5. Select the Guance node.
  6. Add the API Key ID and API Key.
  7. Optionally test connectivity.
  8. Save.

After the association is complete, you can query data sources in the Func platform in two ways:

How to Get an API Key
  1. Go to Guance Workspace > Manage > API Key Management.
  2. Click Create Key on the right side of the page.
  3. Enter a name.
  4. Click Confirm. The system will automatically create an API Key for you, which you can view in the API Key list.

For more details, refer to API Key Management.

Using a Connector

After adding a connector normally, you can use the connector ID in scripts to obtain the corresponding connector operation object.

Using the connector example above, the code to obtain the connector operation object is:

mysql = DFF.CONN('mysql')

Writing Custom Scripts

In addition to using connectors, you can also write functions to query data.

Assume you have correctly created a MySQL connector (with ID mysql), and this MySQL has a table named my_table containing the following data:

id userId username reqMethod reqRoute reqCost createTime
1 u-001 admin POST /api/v1/scripts/:id/do/modify 23 1730840906
2 u-002 admin POST /api/v1/scripts/:id/do/publish 99 1730840906
3 u-003 zhang3 POST /api/v1/scripts/:id/do/publish 3941 1730863223
4 u-004 zhang3 POST /api/v1/scripts/:id/do/publish 159 1730863244
5 u-005 li4 POST /api/v1/scripts/:id/do/publish 44 1730863335
...

Assume you now need to query this table using a data query function, with the following field extraction rules:

Original Field Extracted As
createTime Time time
reqCost Column req_cost
reqMethod Column req_method
reqRoute Column req_route
userId Tag user_id
username Tag username

The complete reference code is as follows:

  • Data Query Function Example
import json

@DFF.API('Query data from my_table', category='dataPlatform.dataQueryFunc')
def query_from_my_table(time_range):
    # Get the connector operation object
    mysql = DFF.CONN('mysql')

    # MySQL query statement
    sql = '''
      SELECT
        createTime, userId, username, reqMethod, reqRoute, reqCost
      FROM
        my_table
      WHERE
        createTime     > ?
        AND createTime < ?
      LIMIT 5
    '''

    # Since the input time_range is in milliseconds
    # but the createTime field in MySQL is in seconds, a conversion is needed
    sql_params = [
      int(time_range[0] / 1000),
      int(time_range[1] / 1000),
    ]

    # Execute the query
    db_res = mysql.query(sql, sql_params)

    # Convert to DQL-like return result

    # Depending on the tags, multiple data series may need to be generated
    # Use the data series tags as keys to create a mapping table
    series_map = {}

    # Iterate through the raw data, transform the structure, and store in the mapping table
    for d in db_res:
        # Collect tags
        tags = {
            'user_id' : d.get('userId'),
            'username': d.get('username'),
        }

        # Serialize the tags (tag keys need to be sorted to ensure consistent output)
        tags_dump = json.dumps(tags, sort_keys=True, ensure_ascii=True)

        # If the data series for this tag has not been created yet, create one
        if tags_dump not in series_map:
            # Basic structure of a data series
            series_map[tags_dump] = {
                'columns': [ 'time', 'req_cost', 'req_method', 'req_route' ], # Columns (first column is always time)
                'tags'   : tags,                                              # Tags
                'values' : [],                                                # Value list
            }

        # Extract time and columns, then append the value
        series = series_map[tags_dump]
        value = [
            d.get('createTime') * 1000, # Time (output unit must be milliseconds; convert as needed)
            d.get('reqCost'),           # Column req_cost
            d.get('reqMethod'),         # Column req_method
            d.get('reqRoute'),          # Column req_route
        ]
        series['values'].append(value)

    # Add DQL outer structure
    dql_like_res = {
        # Data series
        'series': [ list(series_map.values()) ] # Note: wrap in an extra array here
    }
    return dql_like_res

If you only want to understand the data transformation process without concerning yourself with the query (or if you don't have a real database to query), you can refer to the following code:

  • Data Query Function Example (Without the MySQL Query Part)
import json

@DFF.API('Query data from somewhere', category='dataPlatform.dataQueryFunc')
def query_from_somewhere(time_range):
    # Assume the raw data has been obtained through some means
    db_res = [
        {'createTime': 1730840906, 'reqCost': 23,   'reqMethod': 'POST', 'reqRoute': '/api/v1/scripts/:id/do/modify',  'username': 'admin',  'userId': 'u-001'},
        {'createTime': 1730840906, 'reqCost': 99,   'reqMethod': 'POST', 'reqRoute': '/api/v1/scripts/:id/do/publish', 'username': 'admin',  'userId': 'u-001'},
        {'createTime': 1730863223, 'reqCost': 3941, 'reqMethod': 'POST', 'reqRoute': '/api/v1/scripts/:id/do/publish', 'username': 'zhang3', 'userId': 'u-002'},
        {'createTime': 1730863244, 'reqCost': 159,  'reqMethod': 'POST', 'reqRoute': '/api/v1/scripts/:id/do/publish', 'username': 'zhang3', 'userId': 'u-002'},
        {'createTime': 1730863335, 'reqCost': 44,   'reqMethod': 'POST', 'reqRoute': '/api/v1/scripts/:id/do/publish', 'username': 'li4',    'userId': 'u-003'}
    ]

    # Convert to DQL-like return result

    # Depending on the tags, multiple data series may need to be generated
    # Use the data series tags as keys to create a mapping table
    series_map = {}

    # Iterate through the raw data, transform the structure, and store in the mapping table
    for d in db_res:
        # Collect tags
        tags = {
            'user_id' : d.get('userId'),
            'username': d.get('username'),
        }

        # Serialize the tags (tag keys need to be sorted to ensure consistent output)
        tags_dump = json.dumps(tags, sort_keys=True, ensure_ascii=True)

        # If the data series for this tag has not been created yet, create one
        if tags_dump not in series_map:
            # Basic structure of a data series
            series_map[tags_dump] = {
                'columns': [ 'time', 'req_cost', 'req_method', 'req_route' ], # Columns (first column is always time)
                'tags'   : tags,                                              # Tags
                'values' : [],                                                # Value list
            }

        # Extract time and columns, then append the value
        series = series_map[tags_dump]
        value = [
            d.get('createTime') * 1000, # Time (output unit must be milliseconds; convert as needed)
            d.get('reqCost'),           # Column req_cost
            d.get('reqMethod'),         # Column req_method
            d.get('reqRoute'),          # Column req_route
        ]
        series['values'].append(value)

    # Add DQL outer structure
    dql_like_res = {
        # Data series
        'series': [ list(series_map.values()) ] # Note: wrap in an extra array here
    }
    return dql_like_res
  • Return Result Example
{
  "series": [
    [
      {
        "columns": ["time", "req_cost", "req_method", "req_route"],
        "tags": {"user_id": "u-001", "username": "admin"},
        "values": [
          [1730840906000, 23, "POST", "/api/v1/scripts/:id/do/modify" ],
          [1730840906000, 99, "POST", "/api/v1/scripts/:id/do/publish"]
        ]
      },
      {
        "columns": ["time", "req_cost", "req_method", "req_route"],
        "tags": {"user_id": "u-002", "username": "zhang3"},
        "values": [
          [1730863223000, 3941, "POST", "/api/v1/scripts/:id/do/publish"],
          [1730863244000,  159, "POST", "/api/v1/scripts/:id/do/publish"]
        ]
      },
      {
        "columns": ["time", "req_cost", "req_method", "req_route"],
        "tags": {"user_id": "u-003", "username": "li4"},
        "values": [
          [1730863335000, 44, "POST", "/api/v1/scripts/:id/do/publish"]
        ]
      }
    ]
  ]
}

Management List

All connected data sources are visible under Integrations > External Data Sources > Connected Data Sources.

In the list, you can perform the following operations:

  • View the data source type, ID, status, creation information, and update information.
  • Edit a data source to modify configurations other than the DataFlux Func, data source type, and ID.
  • Delete a data source.

Use Cases

A typical scenario for querying external data sources in Guance is Chart > Chart Query.

Returned Data for Different Charts
Line Chart Pie Chart Table Chart

Feedback

Is this page helpful?