Grok Pattern¶
Grok Pattern Introduction¶
DataKit Pipeline provides grok() function to implement support for executing Grok patterns (in terms of implementation, the grok() function will translate Grok patterns into regular expressions), And provide add_pattern() function to add custom naming patterns.
The Grok pattern is based on regular expressions. After the pattern is named, it can be used in other patterns in the following three ways. Be careful not to use circular references:
%{pattern_name}%{pattern_name:key_name}%{pattern_name:key_name:type}
The value of type can be in the range of {float, int, str, bool}; more complex Grok patterns can be obtained by combining Grok patterns.
Any regular expression can be regarded as a legal Grok pattern, and supports the mixed use of named Grok patterns and regular expressions to write Grok patterns;
For the pattern notation %{pattern_name:key_name}, it is equivalent to the named capture group in the regular expression:
Grok Pattern Classification in DataKit¶
Grok patterns in DataKit can be divided into two categories:
- Global pattern: The pattern files in the pattern directory are all global patterns, which can be used by all Pipeline scripts
- Partial pattern: The pattern added by the add_pattern() function in the Pipeline script is a partial pattern, which is only valid for the current Pipeline script
The following takes Nginx access-log as an example to explain how to write the corresponding Grok mode. The original nginx access log is as follows:
127.0.0.1 - - [26/May/2022:20:53:52 +0800] "GET /server_status HTTP/1.1" 404 134 "-" "Go-http-client/1.1"
Assuming that we need to obtain client_ip, time (request), http_method, http_url, http_version, status_code from the access log, the grok mode can initially be written as:
grok(_,"%{NOTSPACE:client_ip} %{NOTSPACE} %{NOTSPACE} \\[%{HTTPDATE:time}\\] \"%{DATA:http_method} %{GREEDYDATA:http_url} HTTP/%{NUMBER:http_version}\" %{INT:status_code} %{INT} \"%{NOTSPACE}\" \"%{NOTSPACE}\"")
cast(status_code, "int")
group_between(status_code, [200,299], "OK", status)
group_between(status_code, [300,399], "notice", status)
group_between(status_code, [400,499], "warning", status)
group_between(status_code, [500,599], "error", status)
default_time(time)
Optimize it again and extract the corresponding features respectively:
# client_ip, http_ident, http_auth at the header of the log as a pattern
add_pattern("p1", "%{NOTSPACE:client_ip} %{NOTSPACE} %{NOTSPACE}")
# The middle http_method, http_url, http_version, status_code as a pattern,
# And specify the data type int of status_code in the pattern instead of the cast function used
add_pattern("p3", '"%{DATA:http_method} %{GREEDYDATA:http_url} HTTP/%{NUMBER:http_version}" %{INT:status_code:int}')
grok(_, "%{p1} \\[%{HTTPDATE:time}\\] %{p3} %{INT} \"%{NOTSPACE}\" \"%{NOTSPACE}\"")
group_between(status_code, [200,299], "OK", status)
group_between(status_code, [300,399], "notice", status)
group_between(status_code, [400,499], "warning", status)
group_between(status_code, [500,599], "error", status)
default_time(time)
The optimized cutting is more readable than the preliminary single-line pattern. Since the default data type of the field resolved by grok is string, specifying the data type of the field here avoids the subsequent use of the cast() function for type conversion.
Custom Grok Pattern¶
The essence of grok is to predefine some regular expressions for text matching extraction, and name the predefined regular expressions, which is convenient to use and expand countless new patterns with nested references. For example, DataKit has three built-in modes as follows:
_second (?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?) #Number of seconds to match, _ second is the schema name
_minute (?:[0-5][0-9]) #Match minutes, _ minute is the schema name
_hour (?:2[0123]|[01]?[0-9]) #Match year, _ hour is the schema name
Based on the above three built-in patterns, you can extend your own built-in pattern and name it time:
# Add time to the file under the pattern directory. This mode is global mode, and time can be referenced anywhere.
time ([^0-9]?)%{hour:hour}:%{minute:minute}(?::%{second:second})([^0-9]?)
# It can also be added to the pipeline file via add_pattern (), then the mode becomes local and only the current pipeline script can use time.
add_pattern("time", "(?:[^0-9]?)%{HOUR:hour}:%{MINUTE:minute}(?::%{SECOND:second})(?:[^0-9]?)")
# Extract the time field from the original input through grok. Assuming the input is 12:30:59, the {"hour": 12, "minute": 30, "second": 59}
grok(_, "%{time}")
Attention
- If a pattern with the same name occurs, the local pattern takes precedence (that is, the local pattern overrides the global pattern).
- In the Pipeline script, add_pattern() needs to be called before the grok() function, otherwise the first data fetch will fail.
Built-in Pattern List¶
DataKit has some commonly used Patterns built in, which we can use directly when using Grok cutting:
USERNAME : [a-zA-Z0-9._-]+
USER : %{USERNAME}
EMAILLOCALPART : [a-zA-Z][a-zA-Z0-9_.+-=:]+
EMAILADDRESS : %{EMAILLOCALPART}@%{HOSTNAME}
HTTPDUSER : %{EMAILADDRESS}|%{USER}
INT : (?:[+-]?(?:[0-9]+))
BASE10NUM : (?:[+-]?(?:[0-9]+(?:\.[0-9]+)?)|\.[0-9]+)
NUMBER : (?:%{BASE10NUM})
BASE16NUM : (?:0[xX]?[0-9a-fA-F]+)
POSINT : \b(?:[1-9][0-9]*)\b
NONNEGINT : \b(?:[0-9]+)\b
WORD : \b\w+\b
NOTSPACE : \S+
SPACE : \s*
DATA : .*?
GREEDYDATA : .*
GREEDYLINES : (?s).*
QUOTEDSTRING : "(?:[^"\\]*(?:\\.[^"\\]*)*)"|\'(?:[^\'\\]*(?:\\.[^\'\\]*)*)\'
UUID : [A-Fa-f0-9]{8}-(?:[A-Fa-f0-9]{4}-){3}[A-Fa-f0-9]{12}
MAC : (?:%{CISCOMAC}|%{WINDOWSMAC}|%{COMMONMAC})
CISCOMAC : (?:(?:[A-Fa-f0-9]{4}\.){2}[A-Fa-f0-9]{4})
WINDOWSMAC : (?:(?:[A-Fa-f0-9]{2}-){5}[A-Fa-f0-9]{2})
COMMONMAC : (?:(?:[A-Fa-f0-9]{2}:){5}[A-Fa-f0-9]{2})
IPV6 : (?:(?:(?:[0-9A-Fa-f]{1,4}:){7}(?:[0-9A-Fa-f]{1,4}|:))|(?:(?:[0-9A-Fa-f]{1,4}:){6}(?::[0-9A-Fa-f]{1,4}|(?:(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(?:\.(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(?:(?:[0-9A-Fa-f]{1,4}:){5}(?:(?:(?::[0-9A-Fa-f]{1,4}){1,2})|:(?:(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(?:\.(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(?:(?:[0-9A-Fa-f]{1,4}:){4}(?:(?:(?::[0-9A-Fa-f]{1,4}){1,3})|(?:(?::[0-9A-Fa-f]{1,4})?:(?:(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(?:\.(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(?:(?:[0-9A-Fa-f]{1,4}:){3}(?:(?:(?::[0-9A-Fa-f]{1,4}){1,4})|(?:(?::[0-9A-Fa-f]{1,4}){0,2}:(?:(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(?:\.(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(?:(?:[0-9A-Fa-f]{1,4}:){2}(?:(?:(?::[0-9A-Fa-f]{1,4}){1,5})|(?:(?::[0-9A-Fa-f]{1,4}){0,3}:(?:(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(?:\.(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(?:(?:[0-9A-Fa-f]{1,4}:){1}(?:(?:(?::[0-9A-Fa-f]{1,4}){1,6})|(?:(?::[0-9A-Fa-f]{1,4}){0,4}:(?:(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(?:\.(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(?::(?:(?:(?::[0-9A-Fa-f]{1,4}){1,7})|(?:(?::[0-9A-Fa-f]{1,4}){0,5}:(?:(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(?:\.(?:25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:)))(?:%.+)?
IPV4 : (?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)
IP : (?:%{IPV6}|%{IPV4})
HOSTNAME : \b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:\.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*(?:\.?|\b)
HOST : %{HOSTNAME}
IPORHOST : (?:%{IP}|%{HOSTNAME})
HOSTPORT : %{IPORHOST}:%{POSINT}
PATH : (?:%{UNIXPATH}|%{WINPATH})
UNIXPATH : (?:/[\w_%!$@:.,-]?/?)(?:\S+)?
TTY : (?:/dev/(?:pts|tty(?:[pq])?)(?:\w+)?/?(?:[0-9]+))
WINPATH : (?:[A-Za-z]:|\\)(?:\\[^\\?*]*)+
URIPROTO : [A-Za-z]+(?:\+[A-Za-z+]+)?
URIHOST : %{IPORHOST}(?::%{POSINT:port})?
URIPATH : (?:/[A-Za-z0-9$.+!*'(){},~:;=@#%_\-]*)+
URIPARAM : \?[A-Za-z0-9$.+!*'|(){},~@#%&/=:;_?\-\[\]<>]*
URIPATHPARAM : %{URIPATH}(?:%{URIPARAM})?
URI : %{URIPROTO}://(?:%{USER}(?::[^@]*)?@)?(?:%{URIHOST})?(?:%{URIPATHPARAM})?
MONTH : \b(?:Jan(?:uary|uar)?|Feb(?:ruary|ruar)?|M(?:a|ä)?r(?:ch|z)?|Apr(?:il)?|Ma(?:y|i)?|Jun(?:e|i)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|O(?:c|k)?t(?:ober)?|Nov(?:ember)?|De(?:c|z)(?:ember)?)\b
MONTHNUM : (?:0?[1-9]|1[0-2])
MONTHNUM2 : (?:0[1-9]|1[0-2])
MONTHDAY : (?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])
DAY : (?:Mon(?:day)?|Tue(?:sday)?|Wed(?:nesday)?|Thu(?:rsday)?|Fri(?:day)?|Sat(?:urday)?|Sun(?:day)?)
YEAR : (\d\d){1,2}
HOUR : (?:2[0123]|[01]?[0-9])
MINUTE : (?:[0-5][0-9])
SECOND : (?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?)
TIME : (?:[^0-9]?)%{HOUR}:%{MINUTE}(?::%{SECOND})(?:[^0-9]?)
DATE_US : %{MONTHNUM}[/-]%{MONTHDAY}[/-]%{YEAR}
DATE_EU : %{MONTHDAY}[./-]%{MONTHNUM}[./-]%{YEAR}
ISO8601_TIMEZONE : (?:Z|[+-]%{HOUR}(?::?%{MINUTE}))
ISO8601_SECOND : (?:%{SECOND}|60)
TIMESTAMP_ISO8601 : %{YEAR}-%{MONTHNUM}-%{MONTHDAY}[T ]%{HOUR}:?%{MINUTE}(?::?%{SECOND})?%{ISO8601_TIMEZONE}?
DATE : %{DATE_US}|%{DATE_EU}
DATESTAMP : %{DATE}[- ]%{TIME}
TZ : (?:[PMCE][SD]T|UTC)
DATESTAMP_RFC822 : %{DAY} %{MONTH} %{MONTHDAY} %{YEAR} %{TIME} %{TZ}
DATESTAMP_RFC2822 : %{DAY}, %{MONTHDAY} %{MONTH} %{YEAR} %{TIME} %{ISO8601_TIMEZONE}
DATESTAMP_OTHER : %{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{TZ} %{YEAR}
DATESTAMP_EVENTLOG : %{YEAR}%{MONTHNUM2}%{MONTHDAY}%{HOUR}%{MINUTE}%{SECOND}
HTTPDERROR_DATE : %{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{YEAR}
SYSLOGTIMESTAMP : %{MONTH} +%{MONTHDAY} %{TIME}
PROG : [\x21-\x5a\x5c\x5e-\x7e]+
SYSLOGPROG : %{PROG:program}(?:\[%{POSINT:pid}\])?
SYSLOGHOST : %{IPORHOST}
SYSLOGFACILITY : <%{NONNEGINT:facility}.%{NONNEGINT:priority}>
HTTPDATE : %{MONTHDAY}/%{MONTH}/%{YEAR}:%{TIME} %{INT}
QS : %{QUOTEDSTRING}
SYSLOGBASE : %{SYSLOGTIMESTAMP:timestamp} (?:%{SYSLOGFACILITY} )?%{SYSLOGHOST:logsource} %{SYSLOGPROG}:
COMMONAPACHELOG : %{IPORHOST:clientip} %{HTTPDUSER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-)
COMBINEDAPACHELOG : %{COMMONAPACHELOG} %{QS:referrer} %{QS:agent}
HTTPD20_ERRORLOG : \[%{HTTPDERROR_DATE:timestamp}\] \[%{LOGLEVEL:loglevel}\] (?:\[client %{IPORHOST:clientip}\] ){0,1}%{GREEDYDATA:errormsg}
HTTPD24_ERRORLOG : \[%{HTTPDERROR_DATE:timestamp}\] \[%{WORD:module}:%{LOGLEVEL:loglevel}\] \[pid %{POSINT:pid}:tid %{NUMBER:tid}\]( \(%{POSINT:proxy_errorcode}\)%{DATA:proxy_errormessage}:)?( \[client %{IPORHOST:client}:%{POSINT:clientport}\])? %{DATA:errorcode}: %{GREEDYDATA:message}
HTTPD_ERRORLOG : %{HTTPD20_ERRORLOG}|%{HTTPD24_ERRORLOG}
LOGLEVEL : (?:[Aa]lert|ALERT|[Tt]race|TRACE|[Dd]ebug|DEBUG|[Nn]otice|NOTICE|[Ii]nfo|INFO|[Ww]arn?(?:ing)?|WARN?(?:ING)?|[Ee]rr?(?:or)?|ERR?(?:OR)?|[Cc]rit?(?:ical)?|CRIT?(?:ICAL)?|[Ff]atal|FATAL|[Ss]evere|SEVERE|EMERG(?:ENCY)?|[Ee]merg(?:ency)?)
COMMONENVOYACCESSLOG : \[%{TIMESTAMP_ISO8601:timestamp}\] \"%{DATA:method} (?:%{URIPATH:uri_path}(?:%{URIPARAM:uri_param})?|%{DATA:}) %{DATA:protocol}\" %{NUMBER:status_code} %{DATA:response_flags} %{NUMBER:bytes_received} %{NUMBER:bytes_sent} %{NUMBER:duration} (?:%{NUMBER:upstream_service_time}|%{DATA:tcp_service_time}) \"%{DATA:forwarded_for}\" \"%{DATA:user_agent}\" \"%{DATA:request_id}\" \"%{DATA:authority}\" \"%{DATA:upstream_service}\"