Best Practices for E-Commerce Order Full-Process Observability¶

Introduction¶

To meet the demands of rapid iteration and traffic surges, e-commerce systems are often developed and deployed using microservices. The performance bottleneck of a single microservice can directly impact the customer's shopping experience, especially when payment anomalies or order cancellations occur. We need to observe the entire trace of orders, track real-time metrics such as the number of successfully paid orders, abnormal orders, and canceled orders, and use these metrics to help analyze business bottlenecks. This best practice is based on a Java-based distributed e-commerce platform, using Guance and the order dimension to observe the number of successfully paid orders and analyze the reasons for unsuccessful payments in real time.

Prerequisites¶

Installing DataKit¶

• <Install DataKit>

Data Ingestion¶

Order data is ingested into Guance via logs. Microservices output log files to a cloud server. DataKit is installed on this cloud server, log collection is enabled, and the log file path is specified. To parse fields such as order number, orderer, and order status from the log files, a Pipeline needs to be written to extract these fields.

Enabling Input¶

1. Enable ddtrace

cd /usr/local/datakit/conf.d/ddtrace
cp ddtrace.conf.sample ddtrace.conf  

1. Write the Pipeline

cd /usr/local/datakit/pipeline
vi log_book_order.p

Where %{DATA:username} is the orderer, %{DATA:order_no} is the order number, and %{DATA:order_status} is the order status.

#2021-12-22 10:09:53.443 [http-nio-7001-exec-7] INFO  c.d.s.b.s.i.OrderServiceImpl - [createOrder,164] - ecs009-book-order 7547183777837932733 2227975860088333788 test d6a3337d-ff82-4b00-9b4d-c07fb00c0cfb - 用户:test 已下单,订单号: d6a3337d-ff82-4b00-9b4d-c07fb00c0cfb


grok(_, "%{TIMESTAMP_ISO8601:time} %{NOTSPACE:thread_name} %{LOGLEVEL:status}%{SPACE}%{NOTSPACE:class_name} - \\[%{NOTSPACE:method_name},%{NUMBER:line}\\] - %{DATA:service1} %{DATA:trace_id} %{DATA:span_id} %{DATA:username} %{DATA:order_no} %{DATA:order_status} - %{GREEDYDATA:msg}")

default_time(time)

1. Enable the Logging plugin by copying the sample file

cd /usr/local/datakit/conf.d/log
cp logging.conf.sample log_book_order.conf

[[inputs.logging]]
  ## required
  logfiles = [
    "/usr/local/df-demo/book-shop/logs/order/log.log"    
  ]

  ## glob filteer
  ignore = [""]

  ## your logging source, if it's empty, use 'default'
  source = "log_book_order"

  ## add service tag, if it's empty, use $source.

  service = "book-store"

  ## grok pipeline script path
  pipeline = "log_book_order.p"

  ## optional status:
  ##   "emerg","alert","critical","error","warning","info","debug","OK"
  ignore_status = []

  ## optional encodings:
  ##    "utf-8", "utf-16le", "utf-16le", "gbk", "gb18030" or ""
  character_encoding = ""

  ## The pattern should be a regexp. Note the use of '''this regexp'''
  ## regexp link: https://golang.org/pkg/regexp/syntax/#hdr-Syntax
  multiline_match = '''^\d{4}-\d{2}-\d{2}'''

  ## removes ANSI escape codes from text strings
  remove_ansi_escape_codes = false

  [inputs.logging.tags]
    app = "book-shop"
  # some_tag = "some_value"
  # more_tag = "some_other_value"

1. Restart DataKit

systemctl restart datakit

E-Commerce Data Ingestion¶

Project source code: book-store.

The logs are cut using the Pipeline. These logs are generated by microservices, so the orderer, order number, and order status need to be output to the logs. The logging tool used in this example is Logback. To output business data through Logback, the MDC (Mapped Diagnostic Context) mechanism must be used. That is, before printing the log, put the orderer, order number, and order status into the MDC, and then output them in the PATTERN of logback-spring.xml. In this example, the bookstore-order-service microservice code needs to be modified.

1. Create an Aspect

Create an aspect to add userName, orderNo, and orderStatus to the MDC, and remove them after the request ends.

@Component
@Aspect
public class LogAop {
    private static final String USER_NAME = "userName";
    private static final String ORDER_NO = "orderNo";
    private static final String ORDER_STATUS = "orderStatus";

    @Pointcut("execution(public * com.devd.spring.bookstoreorderservice.controller..*.*(..))")
    public void controllerCall() {
    }

    @Before("controllerCall()")
    public void logInfoBefore(JoinPoint jp) throws UnsupportedEncodingException {
        MDC.put(USER_NAME, "");
        MDC.put(ORDER_NO, "");
        MDC.put(ORDER_STATUS, "");
    }

    @AfterReturning(returning = "req", pointcut = "controllerCall()")
    public void logInfoAfter(JoinPoint jp, Object req) throws Exception {
        MDC.remove(USER_NAME);
        MDC.remove(ORDER_NO);
        MDC.remove(ORDER_STATUS);
    }

}

1. Write Order Data to the Log

Output the log after a successful order placement

1. Configure logback-spring.xml

<property name="CONSOLE_LOG_PATTERN" value="%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{20} - [%method,%line] - %X{dd.service} %X{dd.trace_id} %X{dd.span_id} %X{userName} %X{orderNo} %X{orderStatus} - %msg%n" />

Package and Deploy¶

• Frontend packaging: generates the build directory

cd bookstore-frontend-react-app
yarn  build

• Backend packaging: generates the following JAR files: bookstore-account-service-0.0.1-SNAPSHOT.jar, bookstore-payment-service-0.0.1-SNAPSHOT.jar, bookstore-api-gateway-service-0.0.1-SNAPSHOT.jar, bookstore-billing-service-0.0.1-SNAPSHOT.jar, bookstore-catalog-service-0.0.1-SNAPSHOT.jar, bookstore-eureka-discovery-service-0.0.1-SNAPSHOT.jar, bookstore-order-service-0.0.1-SNAPSHOT.jar

mvn clean install -DskipTests

Enable RUM¶

• Log in to Guance

Click User Access Monitoring → Create Application, enter book-shop, select Web, and copy the JavaScript snippet.

• Copy the build directory to the server

Open index.html, paste the copied JavaScript snippet into the <head>, set datakitOrigin to the address of the DataKit deployed on the current cloud server, and set allowedDDtracingOrigins to the address of the Gateway.

• Install Nginx and deploy the web project

server {
        listen       80;
        location / {            
            proxy_set_header   Host    $host:$server_port;
            proxy_set_header   X-Real-IP   $remote_addr;
            proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;
            root   /usr/local/df-demo/book-shop/build;
            index  index.html index.htm;
        }
}

Enable APM¶

Guance obtains trace data using /usr/local/datakit/data/dd-java-agent.jar.

java -jar bookstore-eureka-discovery-service-0.0.1-SNAPSHOT.jar


java  -javaagent:/usr/local/datakit/data/dd-java-agent.jar \
 -Ddd.service.name=book-gateway \
 -Ddd.env=dev  \
 -Ddd.agent.port=9529   \
 -jar bookstore-api-gateway-service-0.0.1-SNAPSHOT.jar


java  -javaagent:/usr/local/datakit/data/dd-java-agent.jar \
 -Ddd.service.name=book-account \
 -Ddd.env=dev  \
 -Ddd.agent.port=9529   \
 -jar bookstore-account-service-0.0.1-SNAPSHOT.jar  

java  -javaagent:/usr/local/datakit/data/dd-java-agent.jar \
 -Ddd.service.name=book-order \
 -Ddd.env=dev  \
 -Ddd.agent.port=9529   \
 -jar bookstore-order-service-0.0.1-SNAPSHOT.jar


java  -javaagent:/usr/local/datakit/data/dd-java-agent.jar \
 -Ddd.service.name=book-billing \
 -Ddd.env=dev  \
 -Ddd.agent.port=9529   \
 -jar bookstore-billing-service-0.0.1-SNAPSHOT.jar


java  -javaagent:/usr/local/datakit/data/dd-java-agent.jar \
 -Ddd.service.name=book-payment \
 -Ddd.env=dev  \
 -Ddd.agent.port=9529   \
 -jar bookstore-payment-service-0.0.1-SNAPSHOT.jar


java  -javaagent:/usr/local/datakit/data/dd-java-agent.jar \
 -Ddd.service.name=book-catalog \
 -Ddd.env=dev  \
 -Ddd.agent.port=9529   \
 -jar bookstore-catalog-service-0.0.1-SNAPSHOT.jar 

Order Monitoring Dashboard¶

Log in to Guance, go to Scenarios → Create Dashboard → Create Blank Dashboard, enter "Order Monitoring Dashboard", and click OK.

Click the dashboard created above, then click Edit. Drag a Time Series widget. Set the title to "Number of Orders Placed". In the time series widget, select Logs, then select log_book_order (the value of source in log_book_order.conf), then select order_no, and set the aggregation method to Count_distinct_by. Set the filter condition to order_status with value "已下单" (Order Placed). This time series widget counts the number of orders placed. Finally, click + → Transform Function → Cumsum to convert the order count to a cumulative sum over time.

Drag another Time Series widget. Set the title to "Number of Paid Orders". Select Logs, then log_book_order, then order_no with aggregation Count_distinct_by. Set the filter condition to order_status with value "已支付" (Paid). This widget counts the number of paid orders. Click + → Transform Function → Cumsum.

Drag a Time Series widget. Set the title to "Number of Abnormal Orders". Select Logs, then log_book_order, then order_no with aggregation Count_distinct_by. Set the filter condition to order_status with value "支付异常" (Payment Failed). This widget counts the number of orders with payment failures. Click + → Transform Function → Cumsum.