Skip to content

External Data Sources

Through DataFlux Func, various data sources such as MySQL can be quickly integrated into Guance, enabling seamless data query and visualization.

Features

  • Native queries: Use the data source's native query syntax directly in charts without any additional transformation.
  • Data protection: Based on data security and privacy considerations, all data source information is stored only in your local Func instance, not on the platform, ensuring data security and preventing leakage.
  • Custom management: Easily add and manage various external data sources according to actual needs.
  • Real-time data: Connect directly to external data sources, obtain data in real time, and respond and make decisions instantly.

Two Integration Paths

Add a Data Source in Guance

That is, directly add or view the connected DataFlux Func in Extensions, and further manage all connected external data sources.

Note

This approach is more beginner-friendly compared to the second path and is recommended.

  1. Select DataFlux Func from the dropdown.
  2. Choose the supported data source type.
  3. Define connection properties, including ID, data source title, associated host, port, database, user, and password.
  4. Test the connection as needed.
  5. Save.

Query External Data Sources Using Func

Note

"External data sources" here have a broad definition, including both common external data storage systems (such as MySQL, Redis, etc.) and third-party systems (e.g., the Guance console).

Prerequisites

You need to download the corresponding installation package and quickly start deploying the Func platform.

After deployment, wait for initialization to complete and log in to the platform.

Link Func with Guance

The connector helps developers connect to the Guance system.

Go to Development > Connector > Add Connector page:

  1. Select the connector type.
  2. Customize the ID of the connector.
  3. Add a title. This title will be displayed in the Guance workspace.
  4. Optionally enter a description for the connector.
  5. Select the Guance node.
  6. Add API Key ID and API Key.
  7. Optionally test the connectivity.
  8. Save.

After linking, you can query data sources in the Func platform in the following two ways:

How to Get an API Key
  1. Go to Guance workspace > Management > API Key Management.
  2. Click Create Key on the right side of the page.
  3. Enter a name.
  4. Click OK. The system will automatically create an API Key for you, which you can view in the API Key list.

For more details, refer to API Key Management.

Using the Connector

After adding the connector normally, you can use the connector ID in scripts to obtain the operation object of the corresponding connector.

Using the connector example above, the code to obtain the operation object of this connector is:

mysql = DFF.CONN('mysql')

Writing Scripts Yourself

In addition to using the connector, you can also write your own functions to query data.

Assume that the user has correctly created a MySQL connector (with the ID defined as mysql), and there is a table named my_table in this MySQL with the following data:

id userId username reqMethod reqRoute reqCost createTime
1 u-001 admin POST /api/v1/scripts/:id/do/modify 23 1730840906
2 u-002 admin POST /api/v1/scripts/:id/do/publish 99 1730840906
3 u-003 zhang3 POST /api/v1/scripts/:id/do/publish 3941 1730863223
4 u-004 zhang3 POST /api/v1/scripts/:id/do/publish 159 1730863244
5 u-005 li4 POST /api/v1/scripts/:id/do/publish 44 1730863335
...

Now suppose you need to query this table data using a data query function, with the following field extraction rules:

Original Field Extracted As
createTime Time time
reqCost Column req_cost
reqMethod Column req_method
reqRoute Column req_route
userId Tag user_id
username Tag username

The complete reference code is as follows:

  • Data Query Function Example
import json

@DFF.API('Query data from my_table', category='dataPlatform.dataQueryFunc')
def query_from_my_table(time_range):
    # Get the connector operation object
    mysql = DFF.CONN('mysql')

    # MySQL query statement
    sql = '''
      SELECT
        createTime, userId, username, reqMethod, reqRoute, reqCost
      FROM
        my_table
      WHERE
        createTime     > ?
        AND createTime < ?
      LIMIT 5
    '''

    # Since the input time_range is in milliseconds
    # but the createTime field in MySQL is in seconds, conversion is needed
    sql_params = [
      int(time_range[0] / 1000),
      int(time_range[1] / 1000),
    ]

    # Execute the query
    db_res = mysql.query(sql, sql_params)

    # Convert to DQL-like result

    # Depending on different tags, multiple data series may need to be generated
    # Use data series tags as keys to create a mapping table
    series_map = {}

    # Iterate through raw data, transform structure, and store in the mapping table
    for d in db_res:
        # Collect tags
        tags = {
            'user_id' : d.get('userId'),
            'username': d.get('username'),
        }

        # Serialize tags (tag keys need to be sorted to ensure consistent output)
        tags_dump = json.dumps(tags, sort_keys=True, ensure_ascii=True)

        # If the data series for this tag hasn't been created yet, create one
        if tags_dump not in series_map:
            # Basic structure of data series
            series_map[tags_dump] = {
                'columns': [ 'time', 'req_cost', 'req_method', 'req_route' ], # Columns (first column is fixed as time)
                'tags'   : tags,                                              # Tags
                'values' : [],                                                # Values list
            }

        # Extract time, columns, and append value
        series = series_map[tags_dump]
        value = [
            d.get('createTime') * 1000, # Time (output unit must be milliseconds, convert as needed)
            d.get('reqCost'),           # Column req_cost
            d.get('reqMethod'),         # Column req_method
            d.get('reqRoute'),          # Column req_route
        ]
        series['values'].append(value)

    # Add DQL outer structure
    dql_like_res = {
        # Data series
        'series': [ list(series_map.values()) ] # Note: wrap in an extra array
    }
    return dql_like_res

If you only want to understand the data transformation process and are not concerned with the query process (or don't have an actual database to query yet), you can refer to the following code:

  • Data Query Function Example (Without MySQL Query Part)
import json

@DFF.API('Query data from somewhere', category='dataPlatform.dataQueryFunc')
def query_from_somewhere(time_range):
    # Assume raw data has been obtained by some means
    db_res = [
        {'createTime': 1730840906, 'reqCost': 23,   'reqMethod': 'POST', 'reqRoute': '/api/v1/scripts/:id/do/modify',  'username': 'admin',  'userId': 'u-001'},
        {'createTime': 1730840906, 'reqCost': 99,   'reqMethod': 'POST', 'reqRoute': '/api/v1/scripts/:id/do/publish', 'username': 'admin',  'userId': 'u-001'},
        {'createTime': 1730863223, 'reqCost': 3941, 'reqMethod': 'POST', 'reqRoute': '/api/v1/scripts/:id/do/publish', 'username': 'zhang3', 'userId': 'u-002'},
        {'createTime': 1730863244, 'reqCost': 159,  'reqMethod': 'POST', 'reqRoute': '/api/v1/scripts/:id/do/publish', 'username': 'zhang3', 'userId': 'u-002'},
        {'createTime': 1730863335, 'reqCost': 44,   'reqMethod': 'POST', 'reqRoute': '/api/v1/scripts/:id/do/publish', 'username': 'li4',    'userId': 'u-003'}
    ]

    # Convert to DQL-like result

    # Depending on different tags, multiple data series may need to be generated
    # Use data series tags as keys to create a mapping table
    series_map = {}

    # Iterate through raw data, transform structure, and store in the mapping table
    for d in db_res:
        # Collect tags
        tags = {
            'user_id' : d.get('userId'),
            'username': d.get('username'),
        }

        # Serialize tags (tag keys need to be sorted to ensure consistent output)
        tags_dump = json.dumps(tags, sort_keys=True, ensure_ascii=True)

        # If the data series for this tag hasn't been created yet, create one
        if tags_dump not in series_map:
            # Basic structure of data series
            series_map[tags_dump] = {
                'columns': [ 'time', 'req_cost', 'req_method', 'req_route' ], # Columns (first column is fixed as time)
                'tags'   : tags,                                              # Tags
                'values' : [],                                                # Values list
            }

        # Extract time, columns, and append value
        series = series_map[tags_dump]
        value = [
            d.get('createTime') * 1000, # Time (output unit must be milliseconds, convert as needed)
            d.get('reqCost'),           # Column req_cost
            d.get('reqMethod'),         # Column req_method
            d.get('reqRoute'),          # Column req_route
        ]
        series['values'].append(value)

    # Add DQL outer structure
    dql_like_res = {
        # Data series
        'series': [ list(series_map.values()) ] # Note: wrap in an extra array
    }
    return dql_like_res
  • Example Return Result
{
  "series": [
    [
      {
        "columns": ["time", "req_cost", "req_method", "req_route"],
        "tags": {"user_id": "u-001", "username": "admin"},
        "values": [
          [1730840906000, 23, "POST", "/api/v1/scripts/:id/do/modify" ],
          [1730840906000, 99, "POST", "/api/v1/scripts/:id/do/publish"]
        ]
      },
      {
        "columns": ["time", "req_cost", "req_method", "req_route"],
        "tags": {"user_id": "u-002", "username": "zhang3"},
        "values": [
          [1730863223000, 3941, "POST", "/api/v1/scripts/:id/do/publish"],
          [1730863244000,  159, "POST", "/api/v1/scripts/:id/do/publish"]
        ]
      },
      {
        "columns": ["time", "req_cost", "req_method", "req_route"],
        "tags": {"user_id": "u-003", "username": "li4"},
        "values": [
          [1730863335000, 44, "POST", "/api/v1/scripts/:id/do/publish"]
        ]
      }
    ]
  ]
}

Management List

All connected data sources are visible under Integrations > External Data Sources > Connected Data Sources.

In the list, you can perform the following operations:

  • View the type, ID, status, creation information, and update information of the data source.
  • Edit a data source, modifying any configuration except for DataFlux Func, data source type, and ID.
  • Delete a data source.

Use Cases

In Guance, a typical scenario for querying external data sources is Charts > Chart Query.

Data Return of Different Charts
Line Chart Pie Chart Table Chart

Feedback

Is this page helpful?