これは、なにをしたくて書いたもの？
Fluent BitのParser Filter Pluginでは、複数のパーサーを設定できるようなので、その挙動を確認してみようかなと。
Parser - Fluent Bit: Official Manual
Parser Filter Plugin？
まず最初に、Parser Filter Pluginとはなにか？を見てみます。
The Parser Filter plugin allows to parse field in event records.
Parsers - Fluent Bit: Official Manual
レコード内のフィールドをパースできるプラグイン、ですと。
パーサーの定義は、Fluent Bitの設定ファイルの「parsers_file」で読み込みます。
[SERVICE]
...
parsers_file parsers.conf
「parsers_file」で指定したファイルには、デフォルトでいくつかのログフォーマットに応じたパーサーが定義してあります。
[PARSER]
Name apache
Format regex
Regex ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
Time_Key time
Time_Format %d/%b/%Y:%H:%M:%S %z
[PARSER]
Name apache2
Format regex
Regex ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>.*)")?$
Time_Key time
Time_Format %d/%b/%Y:%H:%M:%S %z
[PARSER]
Name apache_error
Format regex
Regex ^\[[^ ]* (?<time>[^\]]*)\] \[(?<level>[^\]]*)\](?: \[pid (?<pid>[^\]]*)\])?( \[client (?<client>[^\]]*)\])? (?<message>.*)$
[PARSER]
Name nginx
Format regex
Regex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")
Time_Key time
Time_Format %d/%b/%Y:%H:%M:%S %z
〜省略〜
nginx用のパーサーを見てみましょう。
[PARSER]
Name nginx
Format regex
Regex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")
Time_Key time
Time_Format %d/%b/%Y:%H:%M:%S %z
Nameでパーサー名、Formatでパースに使うフォーマットを指定します。
フォーマットには以下の4つがあり、
さらにJSONがネストしているフィールドであればデコードすることもできます。
Decoders - Fluent Bit: Official Manual
こんなやつですね。
{"log":"{\"status\": \"up and running\"}\r\n","stream":"stdout","time":"2018-03-09T01:01:44.851160855Z"}
今回のnginxの例では、正規表現でのパーサーを使用しています。
Regular Expression - Fluent Bit: Official Manual
こんな感じで、正規表現でグループ化してキャプチャした時の名前を、レコード内のフィールドとしてパースすることができます。
Regex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")
Time_KeyとTime_Formatは、パースした結果のフィールドのうち、Timeフィールドとして扱うものがあればそれを指定します。
Time_Formatは、文字通り時間としてパースするためのフォーマットを指定します。
Time_Key time
Time_Format %d/%b/%Y:%H:%M:%S %z
パーサーを使う時は、Filter Pluginとして「parser」を指定し、Parserで使うパーサーのNameを、Key_Nameでパースする
フィールドを指定します。
[FILTER]
Name parser
Match *
Key_Name log
Parser nginx
このParserを複数指定することができるようなので、その動きを確認してみようというのが今回の話です。
Parser … Specify the parser name to interpret the field. Multiple Parser entries are allowed (one per line).
環境
今回の環境は、こちらです。
$ /opt/td-agent-bit/bin/td-agent-bit --version Fluent Bit v1.5.4
また、取り込むログのお題としては、Apacheを使うことにしましょう。
$ apache2 -v Server version: Apache/2.4.41 (Ubuntu) Server built: 2020-08-12T19:46:17
今回は、Apacheのアクセスログとエラーログが混ざったInputを受け取り、パースする設定を書いてみましょう。
デフォルトの設定
まずは、Fluent Bitのデフォルトの設定を記載しておきます。
$ grep -v '^ *#' /etc/td-agent-bit/td-agent-bit.conf
[SERVICE]
flush 5
daemon Off
log_level info
parsers_file parsers.conf
plugins_file plugins.conf
http_server Off
http_listen 0.0.0.0
http_port 2020
storage.metrics on
[INPUT]
name cpu
tag cpu.local
interval_sec 1
[OUTPUT]
name stdout
match *
パーサーの部分は、使用する最低限の部分だけ載せておきましょう。
/etc/td-agent-bit/parsers.conf
[PARSER]
Name apache
Format regex
Regex ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
Time_Key time
Time_Format %d/%b/%Y:%H:%M:%S %z
[PARSER]
Name apache2
Format regex
Regex ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>.*)")?$
Time_Key time
Time_Format %d/%b/%Y:%H:%M:%S %z
[PARSER]
Name apache_error
Format regex
Regex ^\[[^ ]* (?<time>[^\]]*)\] \[(?<level>[^\]]*)\](?: \[pid (?<pid>[^\]]*)\])?( \[client (?<client>[^\]]*)\])? (?<message>.*)$
[PARSER]
Name nginx
Format regex
Regex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")
Time_Key time
Time_Format %d/%b/%Y:%H:%M:%S %z
〜省略〜
今回は、apache2とapache_errorを使うことにします。
こんな感じで、Apacheのアクセスログとエラーログを用意。
$ cat /var/log/apache2/access.log /var/log/apache2/error.log 127.0.0.1 - - [27/Aug/2020:13:45:16 +0000] "GET / HTTP/1.1" 200 11173 "-" "curl/7.68.0" 127.0.0.1 - - [27/Aug/2020:13:45:18 +0000] "GET /index.html HTTP/1.1" 200 11173 "-" "curl/7.68.0" 127.0.0.1 - - [27/Aug/2020:13:45:30 +0000] "GET /index.html HTTP/1.1" 200 11173 "-" "curl/7.68.0" 127.0.0.1 - - [27/Aug/2020:13:45:32 +0000] "GET / HTTP/1.1" 200 11173 "-" "curl/7.68.0" [Thu Aug 27 13:45:14.957795 2020] [mpm_event:notice] [pid 1863:tid 139821141662784] AH00489: Apache/2.4.41 (Ubuntu) configured -- resuming normal operations [Thu Aug 27 13:45:14.957893 2020] [core:notice] [pid 1863:tid 139821141662784] AH00094: Command line: '/usr/sbin/apache2' [Thu Aug 27 13:45:34.983343 2020] [mpm_event:notice] [pid 1863:tid 139821141662784] AH00491: caught SIGTERM, shutting down [Thu Aug 27 13:45:35.032522 2020] [mpm_event:notice] [pid 1955:tid 139947430849600] AH00489: Apache/2.4.41 (Ubuntu) configured -- resuming normal operations [Thu Aug 27 13:45:35.032606 2020] [core:notice] [pid 1955:tid 139947430849600] AH00094: Command line: '/usr/sbin/apache2'
こちらをTCP Input Pluginで受け取って
[INPUT]
Name tcp
Listen 0.0.0.0
Port 5170
Format none
Tag apache.mixed.log
〜省略〜
[OUTPUT]
Name stdout
Match *
Format json_lines
標準出力で確認する構成としましょう。
$ sudo journalctl -u td-agent-bit -f
Multiple Parser
では、複数のパーサーを使ってみます。
使い方は至って簡単で、Parser自体の定義をこんな感じで複数並べます。
[FILTER]
Name parser
Match apache.*
Key_Name log
Parser apache2
Parser apache_error
確認。
$ cat /var/log/apache2/access.log /var/log/apache2/error.log | nc localhost 5170
結果。
Aug 27 13:55:34 myhost td-agent-bit[2114]: {"date":1598535916,"host":"127.0.0.1","user":"-","method":"GET","path":"/","code":"200","size":"11173","referer":"-","agent":"curl/7.68.0"}
Aug 27 13:55:34 myhost td-agent-bit[2114]: {"date":1598535918,"host":"127.0.0.1","user":"-","method":"GET","path":"/index.html","code":"200","size":"11173","referer":"-","agent":"curl/7.68.0"}
Aug 27 13:55:34 myhost td-agent-bit[2114]: {"date":1598535930,"host":"127.0.0.1","user":"-","method":"GET","path":"/index.html","code":"200","size":"11173","referer":"-","agent":"curl/7.68.0"}
Aug 27 13:55:34 myhost td-agent-bit[2114]: {"date":1598535932,"host":"127.0.0.1","user":"-","method":"GET","path":"/","code":"200","size":"11173","referer":"-","agent":"curl/7.68.0"}
Aug 27 13:55:34 myhost td-agent-bit[2114]: {"date":1598536534.745031,"time":"Aug 27 13:45:14.957795 2020","level":"mpm_event:notice","pid":"1863:tid 139821141662784","message":"AH00489: Apache/2.4.41 (Ubuntu) configured -- resuming normal operations"}
Aug 27 13:55:34 myhost td-agent-bit[2114]: {"date":1598536534.745031,"time":"Aug 27 13:45:14.957893 2020","level":"core:notice","pid":"1863:tid 139821141662784","message":"AH00094: Command line: '/usr/sbin/apache2'"}
Aug 27 13:55:34 myhost td-agent-bit[2114]: {"date":1598536534.745032,"time":"Aug 27 13:45:34.983343 2020","level":"mpm_event:notice","pid":"1863:tid 139821141662784","message":"AH00491: caught SIGTERM, shutting down"}
Aug 27 13:55:34 myhost td-agent-bit[2114]: {"date":1598536534.745032,"time":"Aug 27 13:45:35.032522 2020","level":"mpm_event:notice","pid":"1955:tid 139947430849600","message":"AH00489: Apache/2.4.41 (Ubuntu) configured -- resuming normal operations"}
Aug 27 13:55:34 myhost td-agent-bit[2114]: {"date":1598536534.745032,"time":"Aug 27 13:45:35.032606 2020","level":"core:notice","pid":"1955:tid 139947430849600","message":"AH00094: Command line: '/usr/sbin/apache2'"}
Aug 27 13:55:34 myhost td-agent-bit[2114]: {"date":1598536534.745033,"time":"Aug 27 13:54:54.859070 2020","level":"mpm_event:notice","pid":"1955:tid 139947430849600","message":"AH00491: caught SIGTERM, shutting down"}
Aug 27 13:55:34 myhost td-agent-bit[2114]: {"date":1598536534.745033,"time":"Aug 27 13:54:54.918733 2020","level":"mpm_event:notice","pid":"2048:tid 140052200950848","message":"AH00489: Apache/2.4.41 (Ubuntu) configured -- resuming normal operations"}
Aug 27 13:55:34 myhost td-agent-bit[2114]: {"date":1598536534.745034,"time":"Aug 27 13:54:54.918866 2020","level":"core:notice","pid":"2048:tid 140052200950848","message":"AH00094: Command line: '/usr/sbin/apache2'"}
Aug 27 13:55:34 myhost td-agent-bit[2114]: {"date":1598535916,"host":"127.0.0.1","user":"-","method":"GET","path":"/","code":"200","size":"11173","referer":"-","agent":"curl/7.68.0"}
Aug 27 13:55:34 myhost td-agent-bit[2114]: {"date":1598535918,"host":"127.0.0.1","user":"-","method":"GET","path":"/index.html","code":"200","size":"11173","referer":"-","agent":"curl/7.68.0"}
Aug 27 13:55:34 myhost td-agent-bit[2114]: {"date":1598535930,"host":"127.0.0.1","user":"-","method":"GET","path":"/index.html","code":"200","size":"11173","referer":"-","agent":"curl/7.68.0"}
Aug 27 13:55:34 myhost td-agent-bit[2114]: {"date":1598535932,"host":"127.0.0.1","user":"-","method":"GET","path":"/","code":"200","size":"11173","referer":"-","agent":"curl/7.68.0"}
エラーログがそれぞれパースできていますね。
Aug 27 13:55:34 myhost td-agent-bit[2114]: {"date":1598536534.745031,"time":"Aug 27 13:45:14.957795 2020","level":"mpm_event:notice","pid":"1863:tid 139821141662784","message":"AH00489: Apache/2.4.41 (Ubuntu) configured -- resuming normal operations"}
Aug 27 13:55:34 myhost td-agent-bit[2114]: {"date":1598536534.745031,"time":"Aug 27 13:45:14.957893 2020","level":"core:notice","pid":"1863:tid 139821141662784","message":"AH00094: Command line: '/usr/sbin/apache2'"}
Aug 27 13:55:34 myhost td-agent-bit[2114]: {"date":1598536534.745032,"time":"Aug 27 13:45:34.983343 2020","level":"mpm_event:notice","pid":"1863:tid 139821141662784","message":"AH00491: caught SIGTERM, shutting down"}
Aug 27 13:55:34 myhost td-agent-bit[2114]: {"date":1598536534.745032,"time":"Aug 27 13:45:35.032522 2020","level":"mpm_event:notice","pid":"1955:tid 139947430849600","message":"AH00489: Apache/2.4.41 (Ubuntu) configured -- resuming normal operations"}
Aug 27 13:55:34 myhost td-agent-bit[2114]: {"date":1598536534.745032,"time":"Aug 27 13:45:35.032606 2020","level":"core:notice","pid":"1955:tid 139947430849600","message":"AH00094: Command line: '/usr/sbin/apache2'"}
Aug 27 13:55:34 myhost td-agent-bit[2114]: {"date":1598536534.745033,"time":"Aug 27 13:54:54.859070 2020","level":"mpm_event:notice","pid":"1955:tid 139947430849600","message":"AH00491: caught SIGTERM, shutting down"}
Aug 27 13:55:34 myhost td-agent-bit[2114]: {"date":1598536534.745033,"time":"Aug 27 13:54:54.918733 2020","level":"mpm_event:notice","pid":"2048:tid 140052200950848","message":"AH00489: Apache/2.4.41 (Ubuntu) configured -- resuming normal operations"}
Aug 27 13:55:34 myhost td-agent-bit[2114]: {"date":1598536534.745034,"time":"Aug 27 13:54:54.918866 2020","level":"core:notice","pid":"2048:tid 140052200950848","message":"AH00094: Command line: '/usr/sbin/apache2'"}
パーサーはどうやら前の方から順に適用してみて、パースできた時点で止まるようにソースコード上は見えます。
https://github.com/fluent/fluent-bit/blob/v1.5.4/plugins/filter_parser/filter_parser.c#L266-L301
確認してみましょう。
こんな感じで、なんにでもマッチするパーサーを「/etc/td-agent-bit/parsers.conf」に追加してみます。
[PARSER]
Name blackhole
Format regex
Regex (?<message>.*)
「blackhole」と名付けましょう。
Filterの定義にも反映。
[FILTER]
Name parser
Match apache.*
Key_Name log
Parser blackhole
Parser apache2
Parser apache_error
再起動して
$ sudo systemctl restart td-agent-bit
確認。
$ cat /var/log/apache2/access.log /var/log/apache2/error.log | nc localhost 5170
すると、こんな感じでアクセスログもエラーログも、「blackhole」で定義したパーサーに取られてしまいます。
Aug 27 13:59:33 myhost td-agent-bit[2140]: {"date":1598536770.593083,"message":"127.0.0.1 - - [27/Aug/2020:13:45:16 +0000] \"GET / HTTP/1.1\" 200 11173 \"-\" \"curl/7.68.0\""}
Aug 27 13:59:33 myhost td-agent-bit[2140]: {"date":1598536770.593086,"message":"127.0.0.1 - - [27/Aug/2020:13:45:18 +0000] \"GET /index.html HTTP/1.1\" 200 11173 \"-\" \"curl/7.68.0\""}
Aug 27 13:59:33 myhost td-agent-bit[2140]: {"date":1598536770.593086,"message":"127.0.0.1 - - [27/Aug/2020:13:45:30 +0000] \"GET /index.html HTTP/1.1\" 200 11173 \"-\" \"curl/7.68.0\""}
Aug 27 13:59:33 myhost td-agent-bit[2140]: {"date":1598536770.593086,"message":"127.0.0.1 - - [27/Aug/2020:13:45:32 +0000] \"GET / HTTP/1.1\" 200 11173 \"-\" \"curl/7.68.0\""}
Aug 27 13:59:33 myhost td-agent-bit[2140]: {"date":1598536770.593087,"message":"[Thu Aug 27 13:45:14.957795 2020] [mpm_event:notice] [pid 1863:tid 139821141662784] AH00489: Apache/2.4.41 (Ubuntu) configured -- resuming normal operations"}
Aug 27 13:59:33 myhost td-agent-bit[2140]: {"date":1598536770.593087,"message":"[Thu Aug 27 13:45:14.957893 2020] [core:notice] [pid 1863:tid 139821141662784] AH00094: Command line: '/usr/sbin/apache2'"}
Aug 27 13:59:33 myhost td-agent-bit[2140]: {"date":1598536770.593088,"message":"[Thu Aug 27 13:45:34.983343 2020] [mpm_event:notice] [pid 1863:tid 139821141662784] AH00491: caught SIGTERM, shutting down"}
Aug 27 13:59:33 myhost td-agent-bit[2140]: {"date":1598536770.593088,"message":"[Thu Aug 27 13:45:35.032522 2020] [mpm_event:notice] [pid 1955:tid 139947430849600] AH00489: Apache/2.4.41 (Ubuntu) configured -- resuming normal operations"}
Aug 27 13:59:33 myhost td-agent-bit[2140]: {"date":1598536770.593088,"message":"[Thu Aug 27 13:45:35.032606 2020] [core:notice] [pid 1955:tid 139947430849600] AH00094: Command line: '/usr/sbin/apache2'"}
Aug 27 13:59:33 myhost td-agent-bit[2140]: {"date":1598536770.593088,"message":"[Thu Aug 27 13:54:54.859070 2020] [mpm_event:notice] [pid 1955:tid 139947430849600] AH00491: caught SIGTERM, shutting down"}
Aug 27 13:59:33 myhost td-agent-bit[2140]: {"date":1598536770.593089,"message":"[Thu Aug 27 13:54:54.918733 2020] [mpm_event:notice] [pid 2048:tid 140052200950848] AH00489: Apache/2.4.41 (Ubuntu) configured -- resuming normal operations"}
Aug 27 13:59:33 myhost td-agent-bit[2140]: {"date":1598536770.593089,"message":"[Thu Aug 27 13:54:54.918866 2020] [core:notice] [pid 2048:tid 140052200950848] AH00094: Command line: '/usr/sbin/apache2'"}
というわけで、複数のパーサーを使う時は、その適用順に気をつけましょうね、と。
どのパーサーが適用された？
複数のパーサーを使用した場合、どのパーサーを使ってパースされたのか（＝どのログフォーマットだったのか）知りたくなる
気がします。
パーサーの定義で、マッチしたらタグを追加…みたいなことはできなさそうです。
https://github.com/fluent/fluent-bit/blob/v1.5.4/src/flb_parser.c#L431-L440
フィールドの存在有無で、キーを追加する、とかしたらいいんでしょうか？今回はModify Filterを使い、「pid」フィールドが
あれば「log_type / error」を、「path」フィールドがあれば「log_type / access」を追加してみることにしました。
Modify - Fluent Bit: Official Manual
こんな感じで。
[FILTER]
Name parser
Match apache.*
Key_Name log
Parser apache2
Parser apache_error
[FILTER]
Name modify
Match apache.mixed.log
Condition Key_exists pid
Add log_type error
[FILTER]
Name modify
Match apache.mixed.log
Condition Key_exists path
Add log_type access
確認。
$ cat /var/log/apache2/access.log /var/log/apache2/error.log | nc localhost 5170
OKですね。
Aug 27 14:25:46 myhost td-agent-bit[2162]: {"date":1598535916,"host":"127.0.0.1","user":"-","method":"GET","path":"/","code":"200","size":"11173","referer":"-","agent":"curl/7.68.0","log_type":"access"}
Aug 27 14:25:46 myhost td-agent-bit[2162]: {"date":1598535918,"host":"127.0.0.1","user":"-","method":"GET","path":"/index.html","code":"200","size":"11173","referer":"-","agent":"curl/7.68.0","log_type":"access"}
Aug 27 14:25:46 myhost td-agent-bit[2162]: {"date":1598535930,"host":"127.0.0.1","user":"-","method":"GET","path":"/index.html","code":"200","size":"11173","referer":"-","agent":"curl/7.68.0","log_type":"access"}
Aug 27 14:25:46 myhost td-agent-bit[2162]: {"date":1598535932,"host":"127.0.0.1","user":"-","method":"GET","path":"/","code":"200","size":"11173","referer":"-","agent":"curl/7.68.0","log_type":"access"}
Aug 27 14:25:46 myhost td-agent-bit[2162]: {"date":1598538343.940817,"time":"Aug 27 13:45:14.957795 2020","level":"mpm_event:notice","pid":"1863:tid 139821141662784","message":"AH00489: Apache/2.4.41 (Ubuntu) configured -- resuming normal operations","log_type":"error"}
Aug 27 14:25:46 myhost td-agent-bit[2162]: {"date":1598538343.940817,"time":"Aug 27 13:45:14.957893 2020","level":"core:notice","pid":"1863:tid 139821141662784","message":"AH00094: Command line: '/usr/sbin/apache2'","log_type":"error"}
Aug 27 14:25:46 myhost td-agent-bit[2162]: {"date":1598538343.940817,"time":"Aug 27 13:45:34.983343 2020","level":"mpm_event:notice","pid":"1863:tid 139821141662784","message":"AH00491: caught SIGTERM, shutting down","log_type":"error"}
Aug 27 14:25:46 myhost td-agent-bit[2162]: {"date":1598538343.940818,"time":"Aug 27 13:45:35.032522 2020","level":"mpm_event:notice","pid":"1955:tid 139947430849600","message":"AH00489: Apache/2.4.41 (Ubuntu) configured -- resuming normal operations","log_type":"error"}
Aug 27 14:25:46 myhost td-agent-bit[2162]: {"date":1598538343.940818,"time":"Aug 27 13:45:35.032606 2020","level":"core:notice","pid":"1955:tid 139947430849600","message":"AH00094: Command line: '/usr/sbin/apache2'","log_type":"error"}
Aug 27 14:25:46 myhost td-agent-bit[2162]: {"date":1598538343.940818,"time":"Aug 27 13:54:54.859070 2020","level":"mpm_event:notice","pid":"1955:tid 139947430849600","message":"AH00491: caught SIGTERM, shutting down","log_type":"error"}
Aug 27 14:25:46 myhost td-agent-bit[2162]: {"date":1598538343.940819,"time":"Aug 27 13:54:54.918733 2020","level":"mpm_event:notice","pid":"2048:tid 140052200950848","message":"AH00489: Apache/2.4.41 (Ubuntu) configured -- resuming normal operations","log_type":"error"}
Aug 27 14:25:46 myhost td-agent-bit[2162]: {"date":1598538343.940819,"time":"Aug 27 13:54:54.918866 2020","level":"core:notice","pid":"2048:tid 140052200950848","message":"AH00094: Command line: '/usr/sbin/apache2'","log_type":"error"}
Rewrite Tag Filterを試してみたくもなったのですが、タグに関してはもう少しちゃんと理解した方がよさそうなので、また今度。
