Distributed Tracing (APM) Best Practices¶

Prerequisites¶

Account Registration: Go to the official website https://guance.com/ to register an account, then log in with your registered account/password.

Install Datakit¶

Obtain the Command¶

Click the [Integrations] module, select [DataKit], and choose the appropriate installation command based on your operating system and system type.

Execute the Installation¶

Copy the Datakit installation command and run it directly on the server to be monitored.

• Installation directory: /usr/local/datakit/
• Log directory: /var/log/datakit/
• Main configuration file: /usr/local/datakit/conf.d/datakit.conf
• Plugin configuration directory: /usr/local/datakit/conf.d/

After Datakit is installed, the commonly used Linux host plugins are enabled by default. You can view them in DF → Infrastructure → Built-in Views.

Click the [Infrastructure] module to view the list of all hosts with Datakit installed and their basic information, such as hostname, CPU, memory, etc.

Click a [Hostname] to view detailed system information for that host, integration running status (all installed plugins on that host), and built-in views (host).

Click any plugin name under [Integration Running Status] and then [View Monitoring Dashboard] to see the built-in view for that plugin.

View the APM Monitoring Scenario¶

Log in to Guance, enter the specific workspace, and click APM to view:

Introduction to Distributed Tracing (APM)¶

APM stands for Application Performance Management. It is a monitoring technology that emerged with the development of the internet. From the outset, it was designed to solve application performance bottlenecks caused by high concurrency and large traffic volumes. From a development and operations perspective, it makes the multi-layer call logic of applications transparent, facilitating rapid fault location within an enterprise, reducing MTTR (Mean Time to Repair), and ultimately improving overall user experience. Essentially, APM differs significantly from NPM (Network Performance Monitoring, using bypass listening) and logs in terms of deployment methods and application scenarios. APM deployment is often intrusive (bytecode injection or AOP), while logs and NPM typically do not require code intrusion. APM is mainly used by R&D and operations to view the overall system running status, health, external API and database calls, and even finer-grained code and method call resource consumption or anomalies. It focuses on application-side system performance and stability. NPM is primarily used for network link latency, packet loss, packet capture and analysis, and does not involve in-application code performance analysis. Logs are more oriented towards known issue handling and business logs, and also do not involve specific code quality analysis.

Glossary of Tracing Terms:

DF officially supports all APM monitoring tools that support the OpenTracing protocol, such as the popular Skywalking, Zipkin, Jaeger, Ddtrace, etc. The specific integration method is consistent with the original open-source monitoring tools: add or introduce the relevant monitoring file (e.g., adding a Jar package in Java) in the original code. Then, enable the corresponding inputs in DataKit to send tracing data to the DF platform. This data can be correlated with logs, metrics, and infrastructure data for integrated monitoring, operations, and development, improving overall troubleshooting efficiency.

Distributed Tracing (APM) Configuration¶

Enable Tracing Inputs in datakit.conf¶

###########--------Linux Environment---------##########
$ cd /usr/local/datakit/conf.d/

###########--------Windows Environment-------##########
$ C:\Program Files\datakit\conf.d


## Different monitoring tools require different conf files. If you only use one tool, simply copy and enable the corresponding conf file. This article uses ddtrace.

#############----------Skywalking----------###########
$ cd /traceSkywalking
$ cp traceSkywalking.conf.sample traceSkywalking.conf
#############----------Skywalking----------###########

#############------------Jaeger------------###########
$ cd /traceJaeger
$ cp traceJaeger.conf.sample traceJaeger.conf
#############------------Jaeger------------###########

#############------------Zipkin------------###########
$ cd /traceZipkin
$ cp traceZipkin.conf.sample traceZipkin.conf
#############------------Zipkin------------###########

#############------------Ddtrace------------###########
$ cd /ddtrace
$ cp ddtrace.conf.sample ddtrace.conf
#############------------Zipkin------------###########

## After copying the file, enter vim to edit and uncomment the inputs section.
## Example: ddtrace   You can enable tags related comments as needed to add business or other relevant tags.

$ vim ddtrace.conf

$ [inputs.ddtrace]
$         path = "/v0.4/traces"
$ #       [inputs.ddtrace.tags]
$ #               tag1 = "tomcat"
$ #               tag2 = "tag2"
$ #               tag3 = "tag3"

$ wq!

## Restart Datakit
###########--------Linux Environment---------##########
$ datakit --restart

###########--------Windows Environment--------##########
## Right-click → Task Manager → Services → datakit → Restart

Steps for Application Tracing (APM) Integration¶

Ddtrace agent download links

• Python: https://github.com/DataDog/dd-trace-py
• Golang: https://github.com/DataDog/dd-trace-go
• NodeJS: https://github.com/DataDog/dd-trace-js
• PHP: https://github.com/DataDog/dd-trace-php
• Ruby: https://github.com/DataDog/dd-trace-rb
• C# (dotnet): https://github.com/DataDog/dd-trace-dotnet
• C++: https://github.com/DataDog/dd-opentracing-cpp
• Java: https://github.com/DataDog/dd-trace-java

Ddtrace environment variable (startup parameter) definitions:

Ddd.env: Custom environment type (optional)
Ddd.service.name: Custom application name (required) If data is still not displayed after configuring this, try removing the "name" part.
Ddd.agent.port: Data upload port (default 9529) (required)
Ddd.version: Application version (optional)
Ddd.trace.sample.rate: Sampling rate (default is full sampling) (optional)
Ddd.service.mapping: Alias for redis, mysql, etc. called by the current application, to distinguish them from other applications' calls (optional)
Ddd.host: Data transfer target IP, default is localhost (optional)

Refer to the Datadog official documentation for specific configuration.

Java (ddtrace) Integration:¶

1. Download the [ddtrace-agent], version 0.80.0 is recommended.

Place it in the same directory as the application environment.
(*For Datakit version 1.6 and later, the ddtrace-java-agent is already bundled within Datakit at: /usr/local/datakit/data/)
2. Add the ddtrace.jar startup parameters to the original application startup script. Add the following fields: -javaagent:/xxx/ddtrace.jar -Ddd.env=xxx -Ddd.service.name=xxx -Ddd.agent.port=xxx. Replace all "xxx" with actual values.

Tomcat Integration¶

Add the ddtrace startup parameters to catalina.sh, then restart Tomcat. Replace "xxx" in the code snippet with the absolute path.

$ cd /xxx/tomcat/bin
$ vim catalina.sh

$ CATALINA_OPTS="$CATALINA_OPTS -javaagent:/xxx/ddtrace.jar -Ddd.env=test -Ddd.service.name=demo001 -Ddd.agent.port=9529"; export CATALINA_OPTS

$ wq!

## Restart Tomcat
$ ./bin/startup.sh

Log in to Guance, enter the specific workspace, and click APM to view application data for the service named mall-admin.

Microservice Integration¶

Add the ddtrace startup parameters directly to the startup script, then restart the application. Replace "xxx" in the code snippet with the absolute path.

## Original startup script:
$ nohup java -jar mall-admin.jar &

## Startup script with ddtrace parameters added, execute to restart the application:
$ nohup java -javaagent:/xxx/dd-java-agent-0.72.0.jar -Ddd.service.name=mall-admin -Ddd.agent.port=9529 -jar mall-admin.jar &

Log in to Guance, enter the specific workspace, and click APM to view application data for the service named mall-admin.

Docker Integration¶

There are multiple ways to integrate in a Docker environment. Two examples are shown below:
1. Modify the Dockerfile and rebuild the image. Replace "xxx" in the code snippet with the absolute path.

$ vim Dockerfile

## Add the ddtrace-agent path in the Dockerfile; xxx refers to the absolute path
$ ADD dd-java-agent-0.75.0.jar /xxx/  
$ ENTRYPOINT ["java","-javaagent:/xxx/dd-java-agent-0.75.0.jar","-Ddd.service.name=mall-admin","-Ddd.version=v1","-Ddd.env=product","-Ddd.agent.port=9529","-Ddd.agent.host=172.16.0.198","-jar", "-Dspring.profiles.active=prod","/mall-admin-1.0-SNAPSHOT.jar"]
$ wq!

## Build & Run
$ docker build -t mall/mall-admin:v1 .  [ "." is required]

## docker run
$ docker run -p 8080:8080 --name mall-admin --link mysql:db --link redis:redis -v /etc/localtime:/etc/localtime -v /mydata/app/admin/logs:/var/logs -d mall/mall-admin:v1

1. Without modifying the Dockerfile, override the startup command with startup parameters (may not work in some scenarios).

## Original startup command
$ docker run -p 8080:8080 --name mall-admin --link mysql:db --link redis:redis -v /etc/localtime:/etc/localtime -v /mydata/app/admin/logs:/var/logs -d mall/mall-admin:v1

## Startup command with ddtrace. Check the Dockerfile for the jar startup command.
$ docker run -p 8080:8080 --name mall-admin --link mysql:db --link redis:redis -v /etc/localtime:/etc/localtime -v /mydata/app/admin/logs:/var/logs -d mall/mall-admin:v1 java -javaagent:/wx/dd-java-agent-0.75.0.jar -Ddd.service.name=mall-admin -Ddd.version=v1 -Ddd.env=product -Ddd.agent.port=9529 -Ddd.agent.host=172.16.0.198 -jar -Dspring.profiles.active=prod /mall-admin-1.0-SNAPSHOT.jar

## Note: After adding java -javaagent, you need to add -jar your app name.jar at the end of the startup script.

Log in to Guance, enter the specific workspace, and click APM to view application data for the service corresponding to the service.name.

C# (dotnet-ddtrace) Integration¶

IIS Hosted Environment Integration¶

Note: The [dotnet-agent download link] allows you to download x86, arm64, or other versions as needed.

1. Add server environment variables

## Right-click This PC → Properties → Advanced system settings → Environment Variables
## Create a new system variable and enter the following:

DD_TRACE_AGENT_URL=http://localhost:9529    (required)
DD_ENV=   e.g., test      (optional)
DD_SERVICE=   e.g., myappname     (required)
DD_VERSION=   e.g., 1.0     (optional)
DD_TRACE_SERVICE_MAPPING=   e.g., mysql:main-mysql-db    (optional)

## TRACE_AGENT_URL is the data upload IP and port. It must be set to http://localhost:9529. It is not recommended to change this.
## ENV is the system environment. Set it to pro, test, or other values as needed.
## SERVICE sets the application name displayed on the DF platform. Set it to the specific service name.
## VERSION is the version number. Set it as needed.
## TRACE_SERVICE_MAPPING renames services to distinguish them from components called by other business systems on the DF platform. It accepts a mapping of service name keys to the names to be used, in the format [from-key]:[to-name].
   Note: [from-key] must be standard fields, such as mysql, redis, mongodb, oracle. Do not customize these.
   Example: TRACE_SERVICE_MAPPING=mysql:main-mysql-db
            TRACE_SERVICE_MAPPING=mongodb:offsite-mongodb-service

1. Install the ddtrace-agent

Run the dotnet-agent installer as an administrator. Click Next until the installation is complete.

1. Run the following commands in PowerShell to restart IIS

## Stop IIS service
net stop /y was

## Start IIS service
net start w3svc

Log in to the DF platform and view the application data for the corresponding service name in the APM module.

Python (ddtrace) Integration¶

[Python Integration Guide]

.NET Core (ddtrace) Integration¶

Note: [.NET Core agent download link] .NET Tracer currently supports application tracing on .NET Core 2.1, 3.1, and .NET 5.

1. Run the following commands to install the .NET Core agent based on your environment:

Debian or Ubuntu
sudo dpkg -i ./datadog-dotnet-apm_<TRACER_VERSION>_amd64.deb && /opt/datadog/createLogPath.sh

CentOS or Fedora
sudo rpm -Uvh datadog-dotnet-apm<TRACER_VERSION>-1.x86_64.rpm && /opt/datadog/createLogPath.sh

Alpine or other musl-based distributions
sudo tar -xzf -C /opt/datadog datadog-dotnet-apm<TRACER_VERSION>-musl.tar.gz && sh /opt/datadog/createLogPath.sh

Other distributions
sudo tar -xzf -C /opt/datadog datadog-dotnet-apm<TRACER_VERSION>-tar.gz && /opt/datadog/createLogPath.sh

1. Add application environment variables

Add the following configuration to the application's existing environment variables. This is for reference only; the service name must be changed in the actual configuration.

export CORECLR_ENABLE_PROFILING=1
export CORECLR_PROFILER_PATH=/opt/datadog/Datadog.Trace.ClrProfiler.Native.so
export DD_INTEGRATIONS=/opt/datadog/integrations.json
export DD_DOTNET_TRACER_HOME=/opt/datadog
export DD_TRACE_AGENT_URL=http://localhost:9529
export DD_SERVICE=service_test
export CORECLR_PROFILER={846F5F1C-F9AE-4B07-969E-05C26BC060D8}

1. Restart the application

Q&A¶

Agent and JAR Location Relationship¶

The ddtrace-agent startup parameters must be placed before the -jar in the Java application startup command. The final format is: java -javaagent -jar.

## Original startup script:
$ java -jar mall-admin.jar

## Startup script with ddtrace parameters added, execute to restart the application:
$ java -javaagent:/xxx/dd-java-agent-0.72.0.jar -Ddd.service.name=mall-admin -Ddd.agent.port=9529 -jar mall-admin.jar

tomcat-catalina.sh Parameter Configuration¶

Add the ddtrace startup parameters to catalina.sh, then restart Tomcat. Replace "xxx" in the code snippet with the absolute path.
Ensure that the CATALINA_OPTS configuration is read by the application startup script in catalina.sh; otherwise, the application will start normally but the ddtrace-agent will not be started.

$ cd /xxx/tomcat/bin
$ vim catalina.sh

$ CATALINA_OPTS="$CATALINA_OPTS -javaagent:/xxx/ddtrace.jar -Ddd.env=test -Ddd.service.name=demo001 -Ddd.agent.port=9529"; export CATALINA_OPTS

$ wq!

## Restart Tomcat
$ ./bin/startup.sh

Log in to Guance, enter the specific workspace, and click APM to view application data for the service named mall-admin.