Socket Logging

This article focuses on how the Java Go Python logging framework configures socket output to the DataKit socket log collector.

File collection and socket are mutually exclusive. Please close file collection before opening socket. Please configure logging.conf specific configuration instructions.

Java¶

When configuring log4j, it should be noted that log4j v1 is configured by default using . properties file; log4j v2 is currently configured using the . xml file.

Although there are differences in file names, when log4j looks for configuration files, it always goes to the class path directory. According to the specification, v1 is configured in resources/log4j. properties, and v2 is configured in resources/log4j. xml.

log4j(v2)¶

Import the log4j 2. x jar package in the configuration of maven:

 <dependency>
    <groupId>org.apache.logging.log4j</groupId>
    <artifactId>log4j-api</artifactId>
    <version>2.6.2</version>
 </dependency>

 <dependency>
    <groupId>org.apache.logging.log4j</groupId>
    <artifactId>log4j-core</artifactId>
    <version>2.6.2</version>
  </dependency>

Configure log4j. xml in resources and add Socket Appender:

 <!-- Socket appender でローカルの 9540 ポートへログを送信します。protocol の既定値は tcp -->
 <Socket name="socketname" host="localHost" port="9540" charset="utf8">
     <!-- カスタム出力形式とシリアライズレイアウト-->
     <PatternLayout pattern="%d{yyyy.MM.dd 'at' HH:mm:ss z} %-5level %class{36} %L %M - %msg%xEx%n"/>

     <!--注意：socket コレクターへシリアライズしたデータを送らないでください。DataKit はデシリアライズできないため、プレーンテキストを使用します-->
     <!-- <SerializedLayout/>-->

     <!-- 注意：各ログを 1 行で出力するため、compact と eventEol を true にします-->
     <!-- Guance への送信後に JSON は自動展開されるため、ログは 1 件 1 行で出力してください -->
     <!-- <JsonLayout  properties="true" compact="true" complete="false" eventEol="true"/>-->
 </Socket>

 <!-- 次に logger を定義して appender を参照します。参照した appender のみ有効になります -->
 <loggers>
      <root level="trace">
          <appender-ref ref="Console"/>
          <appender-ref ref="socketname"/>
      </root>
 </loggers>

Java code example:

package com.example;

import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import java.io.PrintWriter;
import java.io.StringWriter;

public class logdemo {
    public static void main(String[] args) throws InterruptedException {
        Logger logger = LogManager.getLogger(logdemo.class);
       for (int i = 0; i < 5; i++) {
            logger.debug("this is log msg to  datakt");
        }

        try {
            int i = 0;
            int a = 5 / i; // ゼロ除算例外
        } catch (Exception e) {
            StringWriter sw = new StringWriter();
            e.printStackTrace(new PrintWriter(sw));
            String exceptionAsString = sw.toString();
            logger.error(exceptionAsString);
        }
    }
}

log4j(v1)¶

Import log4j 1. x jar package in maven configuration

 <dependency>
    <groupId>org.apache.logging.log4j</groupId>
    <artifactId>log4j-api</artifactId>
    <version>1.2.17</version>
 </dependency>

Create the log4j. properties file in the resources directory

log4j.rootLogger=INFO,server
# ... other configurations

log4j.appender.server=org.apache.log4j.net.SocketAppender
log4j.appender.server.Port=<dk socket port>
log4j.appender.server.RemoteHost=<dk socket ip>
log4j.appender.server.ReconnectionDelay=10000

# Configurable to json format
# log4j.appender.server.layout=net.logstash.log4j.JSONEventLayout
...

Logback¶

SocketAppender in Logback cannot send plain text to socket doc

The problem is that the SocketAppender sends serialized Java objects instead of plain text. You can use log4j for input, but I do not recommend replacing the logging component. Rather, I rewrite an Appender that sends log data as plain text, and you use it with JSON formatting.

Golang¶

zap¶

Most commonly used in Golang is Uber's zap open source logging framework, which supports custom output injection

Customize the log exporter and inject it into zap.core

type soceketOutput struct {
    conn net.Conn
}

func (s *soceketOutput) Write(b []byte) (int, error) {
    return s.conn.Write(b)
}

func zapcal() {
    conn, _ := net.DialTCP("tcp", nil, DK_LOG_PORT)
    socket := &soceketOutput{
        conn: conn,
    }

    w := zapcore.AddSync(socket)

    core := zapcore.NewCore(zapcore.NewConsoleEncoder(zapcore.EncoderConfig{
        TimeKey:        "ts",
        LevelKey:       "level",
        NameKey:        "logger",
        CallerKey:      "caller",
        FunctionKey:    zapcore.OmitKey,
        MessageKey:     "msg",
        StacktraceKey:  "stacktrace",
        LineEnding:     zapcore.DefaultLineEnding,
        EncodeLevel:    zapcore.LowercaseLevelEncoder,
        EncodeTime:     zapcore.EpochTimeEncoder,
        EncodeDuration: zapcore.SecondsDurationEncoder,
        EncodeCaller:   zapcore.ShortCallerEncoder,
    }),
        w,
        zapcore.InfoLevel)

    l := zap.New(core, zap.AddCaller())

    l.Info("======= message =======")
}

Python¶

logging.handlers.SocketHandler¶

The native socketHandler sends a log object through the socket, not plain text, so you need to customize the handler and override the makePickle(slef,record) method in the socketHandler.

The code is for reference only:

import logging
import logging.handlers

logger = logging.getLogger("") # Instantiate logging

#Customize the class and override the makePickle method
class PlainTextTcpHandler(logging.handlers.SocketHandler):
    """ Sends plain text log message over TCP channel """

    def makePickle(self, record):
     message = self.formatter.format(record) + "\r\n"
     return message.encode()


def logging_init():
    # Creat file handler
    fh = logging.FileHandler("test.log", encoding="utf-8")
    #Creat custom handler
    plain = PlainTextTcpHandler("10.200.14.226", 9540)

    # Setting logger log levels
    logger.setLevel(logging.INFO)

    # Setting output log format
    formatter = logging.Formatter(
        fmt="%(asctime)s - %(filename)s line:%(lineno)d - %(levelname)s: %(message)s"
    )

    # Specify the output format for the handler, paying attention to the case
    fh.setFormatter(formatter)
    plain.setFormatter(formatter)


    # Log handler added for logger
    logger.addHandler(fh)
    logger.addHandler(plain)

    return True


if __name__ == '__main__':
    logging_init()

    logger.debug(u"debug")
    logger.info(u"info")
    logger.warning(u"warning")
    logger.error(u"error")
    logger.critical(u"critical")

TODO: The log framework of other languages will be supplemented later to use socket to send logs to DataKit.