CLOVER🍀

That was when it all began.

Logstashで独自のGrokパターンを定義する際に、ファイルにまとめて読み込む

これは、なにをしたくて書いたもの?

LogstashのGrok filter pluginで使えるGrokパターンは、自分で定義することもできるようなのですが、これをファイルにまとめることが
できるようなので試してみようかなと。

こちらですね。

Grok Filter Configuration Options / patterns_dir

指定のディレクトリ配下にパターンファイルを置くことで、Logstashが認識するようです。
※一応、最後にLogstashのパイプライン定義に直接パターンを書く方法も試しておこうと思います

環境

今回の環境は、こちら。

$ uname -srvmpio
Linux 4.15.0-70-generic #79-Ubuntu SMP Tue Nov 12 10:36:11 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux


$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.3 LTS
Release:    18.04
Codename:   bionic


$ java --version
openjdk 11.0.5 2019-10-15
OpenJDK Runtime Environment (build 11.0.5+10-post-Ubuntu-0ubuntu1.118.04)
OpenJDK 64-Bit Server VM (build 11.0.5+10-post-Ubuntu-0ubuntu1.118.04, mixed mode, sharing)

Filebeat、Logstashのバージョンは、いずれも7.5.1。それぞれ、aptでインストールしたものです。

## Filebeat
$ filebeat version
filebeat version 7.5.1 (amd64), libbeat 7.5.1 [60dd883ca29e1fdd5b8b075bd5f3698948b1d44d built 2019-12-16 21:56:14 +0000 UTC]


## Logstash
Jan 03 15:45:03 ubuntu1804.localdomain logstash[6755]: [2020-01-03T07:45:03,577][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.5.1"}

定義したGrokのパターンが確認できればよいので、Elasticsearchは付けていません。

それぞれのミドルウェアが動作しているサーバーは、以下のとおりとします。

  • Filebeat … 192.168.33.11
  • Logstash … 192.168.33.12

お題とサンプルアプリケーション

アプリケーションが出力した、Multilineなメッセージを含むログを、Filebeatで読み込みLogstashでパースします。

以前やったこちらのエントリで使ったGrok filter pluginの設定を、パターンファイルに切り出して、構成要素からElasticsearchを削ったもの
という感じですね。

Filebeatで読み込んだログを、LogstashでパースしてElasticsearchに放り込む - CLOVER🍀

アプリケーションは、こちらで作成したものと同じものを使います。

Filebeatで、複数行のログをElasticsearchに取り込んでみる - CLOVER🍀

pom.xmlの設定。

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
        <maven.compiler.source>11</maven.compiler.source>
        <maven.compiler.target>11</maven.compiler.target>
    </properties>

    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-dependencies</artifactId>
                <version>2.2.2.RELEASE</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
    </dependencies>

    <build>
        <finalName>${project.artifactId}</finalName>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
                <version>2.2.2.RELEASE</version>
                <executions>
                  <execution>
                    <goals>
                      <goal>repackage</goal>
                    </goals>
                  </execution>
                </executions>
            </plugin>
        </plugins>
    </build>

ソースコード。
src/main/java/org/littlewings/spring/example/App.java

package org.littlewings.spring.example;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

@SpringBootApplication
@RestController
@RequestMapping("app")
public class App {
    Logger logger = LoggerFactory.getLogger(App.class);

    public static void main(String... args) {
        SpringApplication.run(App.class, args);
    }

    @GetMapping("info")
    public String info() {
        logger.info("{}, info logging", "hello");

        return "info logging";
    }

    @GetMapping("warn")
    public String warn() {
        logger.warn("{}, warn logging", "world");

        return "warn logging";
    }

    @GetMapping("error")
    public String error() {
        logger.error("{}, error logging", "oops!!");

        return "error logging";
    }

    @GetMapping("exception")
    public String exception() {
        Exception e = new RuntimeException("Oops!!");

        logger.error("exception occurred, {}", "why?", e);

        return "exception logging";
    }
}

こちらをパッケージングして、以下のようにしてFilebeatが動作しているサーバーで起動させます。

$ mvn package
$ java -Dlogging.file.path=/tmp -jar target/simple-logging-web-app.jar

これで、アプリケーション側の準備は完了です。ログファイルは、「/tmp/spring.log」という名前で出力されます。

Filebeatの設定

Filebeat側は、以下のように設定します。

$ sudo grep -vE '^ *#|^$' /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /tmp/spring.log
  multiline.pattern: '^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\.\d{3}'
  multiline.negate: true
  multiline.match: after
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 1
setup.kibana:
  host: "192.168.33.14:5601"
output.logstash:
  hosts: ["192.168.33.12:5044"]
processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

出力先はLogstashで、Multililne messageを読めるようにしておきます。

設定したら、Filebeatは再起動しておきます。

$ sudo systemctl restart filebeat

Logstashの設定をする

では、Logstashの設定を行っていきます。

自分で作るGrokパターンは、以下のディレクトリに配置することにしました。

$ sudo mkdir /etc/logstash/conf.d/patterns

パターンファイルは、拡張子なしで作るみたいですね。

https://github.com/logstash-plugins/logstash-patterns-core/tree/v4.1.2/patterns

Spring Bootのデフォルトのログフォーマットを使っているので、そんな感じの名前でパターンファイルを作成。
/etc/logstash/conf.d/patterns/spring-app-log

LOG_TIME \d\d\d\d-\d\d-\d\d \d\d:\d\d:\d\d.\d\d\d
PID %{POSINT:pid}
THREAD_NAME \S+
LOGGER_NAME %{NOTSPACE:logger}
LOG_MESSAGE %{GREEDYDATA:logmessage}

SPRING_APP_LOG_MESSAGE (?m)^%{LOG_TIME:logtime} *%{LOGLEVEL:loglevel} %{PID} --- \[%{THREAD_NAME:thread}\] %{LOGGER_NAME} *: *%{LOG_MESSAGE}

※THREAD_NAMEは%{NOTSPACE:logger}をそのまま使ってもいいのですが、今回は定義しなおすサンプルにしています

Grokのパターンは、Onigurumaの正規表現で定義します。

https://github.com/kkos/oniguruma/blob/master/doc/RE

「名前 パターン」な感じですね。

LOG_TIME \d\d\d\d-\d\d-\d\d \d\d:\d\d:\d\d.\d\d\d

既存のパターンに別名を付けたり、その時にパース後の名前を付与したりもできます。

PID %{POSINT:pid}

最後に全部まとめて、こんな感じで定義。

SPRING_APP_LOG_MESSAGE (?m)^%{LOG_TIME:logtime} *%{LOGLEVEL:loglevel} %{PID} --- \[%{THREAD_NAME:thread}\] %{LOGGER_NAME} *: *%{LOG_MESSAGE}

パース後は、マッチした各要素に名前が付けられます。

で、Logstashのパイプラインは、以下のように定義。
/etc/logstash/conf.d/logstash-app-log.conf

input {
  beats {
    port => 5044
  }
}

filter {
  grok {
    patterns_dir => ["/etc/logstash/conf.d/patterns"]
    match => { "message" => "%{SPRING_APP_LOG_MESSAGE}" }
  }

  mutate {
    rename => [ "@timestamp", "event_time" ]
  }

  date {
    match => [ "logtime", "yyyy-MM-dd HH:mm:ss.SSS" ]
    target => "@timestamp"
  }

  mutate {
    remove_field => [ "logtime", "message" ]
  }
}

output {
  stdout { }
}

patterns_dirで自分が定義したパターンファイルを置いたディレクトリを指定し、matchからは自分で定義したパターンが使えるように
なります。

  grok {
    patterns_dir => ["/etc/logstash/conf.d/patterns"]
    match => { "message" => "%{SPRING_APP_LOG_MESSAGE}" }
  }

patterns_dirはリストで指定するので、複数のディレクトリを指定することができます。

ディレクトリ内で読み込むファイルを絞り込む場合は、patterns_files_globで指定するようです。デフォルトは「*」となっています。

patterns_files_glob

ここまでできたらLogstashを再起動。

$ sudo systemctl restart logstash

確認。

Logstash側では、Logstash自身のログを見ておきましょう。

$ sudo journalctl -u logstash -f

では、Filebeat側のサーバーで、アプリケーションログを出力させてみます。

$ curl localhost:8080/app/info
info logging

$ curl localhost:8080/app/exception
exception logging

$ curl localhost:8080/app/warn
warn logging

少し待っていると、Logstashのログ側にパースされた結果が出力されます。

Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]: {
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:          "agent" => {
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:             "hostname" => "ubuntu1804.localdomain",
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:                 "type" => "filebeat",
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:         "ephemeral_id" => "b5ad0ed3-4216-4b6d-b886-77721edb667d",
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:              "version" => "7.5.1",
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:                   "id" => "9610f5f5-8fea-4a42-b8f4-2c8dee0ded55"
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:     },
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:     "logmessage" => "hello, info logging",
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:       "@version" => "1",
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:       "loglevel" => "INFO",
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:     "event_time" => 2020-01-03T16:37:38.985Z,
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:          "input" => {
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:         "type" => "log"
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:     },
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:            "ecs" => {
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:         "version" => "1.1.0"
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:     },
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:           "host" => {
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:              "hostname" => "ubuntu1804.localdomain",
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:                    "os" => {
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:              "version" => "18.04.3 LTS (Bionic Beaver)",
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:             "platform" => "ubuntu",
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:               "family" => "debian",
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:               "kernel" => "4.15.0-70-generic",
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:             "codename" => "bionic",
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:                 "name" => "Ubuntu"
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:         },
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:                    "id" => "7d9a31c8858d4e5a90285fafc05126ba",
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:         "containerized" => false,
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:          "architecture" => "x86_64",
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:                  "name" => "ubuntu1804.localdomain"
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:     },
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:            "log" => {
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:           "file" => {
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:             "path" => "/tmp/spring.log"
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:         },
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:         "offset" => 30735
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:     },
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:         "thread" => "http-nio-8080-exec-7",
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:         "logger" => "org.littlewings.spring.example.App",
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:     "@timestamp" => 2020-01-03T16:37:30.782Z,
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:           "tags" => [
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:         [0] "beats_input_codec_plain_applied"
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:     ],
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:            "pid" => "19574"
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]: }
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]: {
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:          "agent" => {
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:             "hostname" => "ubuntu1804.localdomain",
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:                 "type" => "filebeat",
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:         "ephemeral_id" => "b5ad0ed3-4216-4b6d-b886-77721edb667d",
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:              "version" => "7.5.1",
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:                   "id" => "9610f5f5-8fea-4a42-b8f4-2c8dee0ded55"
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:     },
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:     "logmessage" => "exception occurred, why?\n\njava.lang.RuntimeException: Oops!!\n\tat org.littlewings.spring.example.App.exception(App.java:44) ~[classes!/:na]\n\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]\n\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na]\n\tat java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]\n\tat java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]\n\tat org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190) ~[spring-web-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]\n\tat org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138) ~[spring-web-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]\n\tat org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:106) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]\n\tat org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:888) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]\n\tat org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:793) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]\n\tat org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]\n\tat org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]\n\tat org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]\n\tat org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]\n\tat org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:634) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:741) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) ~[tomcat-embed-websocket-9.0.29.jar!/:9.0.29]\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) ~[spring-web-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) ~[spring-web-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) ~[spring-web-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:526) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:367) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:860) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1591) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na]\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na]\n\tat org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]\n",
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:       "@version" => "1",
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:       "loglevel" => "ERROR",
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:     "event_time" => 2020-01-03T16:37:38.985Z,
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:          "input" => {
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:         "type" => "log"
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:     },
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:            "ecs" => {
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:         "version" => "1.1.0"
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:     },
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:           "host" => {
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:              "hostname" => "ubuntu1804.localdomain",
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:         "containerized" => false,
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:                    "id" => "7d9a31c8858d4e5a90285fafc05126ba",
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:                    "os" => {
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:              "version" => "18.04.3 LTS (Bionic Beaver)",
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:             "platform" => "ubuntu",
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:               "family" => "debian",
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:               "kernel" => "4.15.0-70-generic",
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:             "codename" => "bionic",
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:                 "name" => "Ubuntu"
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:         },
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:          "architecture" => "x86_64",
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:                  "name" => "ubuntu1804.localdomain"
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:     },
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:            "log" => {
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:           "file" => {
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:             "path" => "/tmp/spring.log"
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:         },
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:          "flags" => [
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:             [0] "multiline"
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:         ],
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:         "offset" => 30861
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:     },
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:         "thread" => "http-nio-8080-exec-8",
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:         "logger" => "org.littlewings.spring.example.App",
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:     "@timestamp" => 2020-01-03T16:37:32.265Z,
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:           "tags" => [
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:         [0] "beats_input_codec_plain_applied"
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:     ],
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]:            "pid" => "19574"
Jan 03 16:37:41 ubuntu1804.localdomain logstash[7354]: }
Jan 03 16:37:45 ubuntu1804.localdomain logstash[7354]: {
Jan 03 16:37:45 ubuntu1804.localdomain logstash[7354]:          "agent" => {
Jan 03 16:37:45 ubuntu1804.localdomain logstash[7354]:             "hostname" => "ubuntu1804.localdomain",
Jan 03 16:37:45 ubuntu1804.localdomain logstash[7354]:                 "type" => "filebeat",
Jan 03 16:37:45 ubuntu1804.localdomain logstash[7354]:         "ephemeral_id" => "b5ad0ed3-4216-4b6d-b886-77721edb667d",
Jan 03 16:37:45 ubuntu1804.localdomain logstash[7354]:              "version" => "7.5.1",
Jan 03 16:37:45 ubuntu1804.localdomain logstash[7354]:                   "id" => "9610f5f5-8fea-4a42-b8f4-2c8dee0ded55"
Jan 03 16:37:45 ubuntu1804.localdomain logstash[7354]:     },
Jan 03 16:37:45 ubuntu1804.localdomain logstash[7354]:     "logmessage" => "world, warn logging",
Jan 03 16:37:45 ubuntu1804.localdomain logstash[7354]:       "@version" => "1",
Jan 03 16:37:45 ubuntu1804.localdomain logstash[7354]:       "loglevel" => "WARN",
Jan 03 16:37:45 ubuntu1804.localdomain logstash[7354]:     "event_time" => 2020-01-03T16:37:38.987Z,
Jan 03 16:37:45 ubuntu1804.localdomain logstash[7354]:          "input" => {
Jan 03 16:37:45 ubuntu1804.localdomain logstash[7354]:         "type" => "log"
Jan 03 16:37:45 ubuntu1804.localdomain logstash[7354]:     },
Jan 03 16:37:45 ubuntu1804.localdomain logstash[7354]:           "host" => {
Jan 03 16:37:45 ubuntu1804.localdomain logstash[7354]:              "hostname" => "ubuntu1804.localdomain",
Jan 03 16:37:45 ubuntu1804.localdomain logstash[7354]:         "containerized" => false,
Jan 03 16:37:45 ubuntu1804.localdomain logstash[7354]:                    "os" => {
Jan 03 16:37:45 ubuntu1804.localdomain logstash[7354]:              "version" => "18.04.3 LTS (Bionic Beaver)",
Jan 03 16:37:45 ubuntu1804.localdomain logstash[7354]:             "platform" => "ubuntu",
Jan 03 16:37:45 ubuntu1804.localdomain logstash[7354]:               "family" => "debian",
Jan 03 16:37:45 ubuntu1804.localdomain logstash[7354]:               "kernel" => "4.15.0-70-generic",
Jan 03 16:37:45 ubuntu1804.localdomain logstash[7354]:             "codename" => "bionic",
Jan 03 16:37:45 ubuntu1804.localdomain logstash[7354]:                 "name" => "Ubuntu"
Jan 03 16:37:45 ubuntu1804.localdomain logstash[7354]:         },
Jan 03 16:37:45 ubuntu1804.localdomain logstash[7354]:                    "id" => "7d9a31c8858d4e5a90285fafc05126ba",
Jan 03 16:37:45 ubuntu1804.localdomain logstash[7354]:          "architecture" => "x86_64",
Jan 03 16:37:45 ubuntu1804.localdomain logstash[7354]:                  "name" => "ubuntu1804.localdomain"
Jan 03 16:37:45 ubuntu1804.localdomain logstash[7354]:     },
Jan 03 16:37:45 ubuntu1804.localdomain logstash[7354]:            "ecs" => {
Jan 03 16:37:45 ubuntu1804.localdomain logstash[7354]:         "version" => "1.1.0"
Jan 03 16:37:45 ubuntu1804.localdomain logstash[7354]:     },
Jan 03 16:37:45 ubuntu1804.localdomain logstash[7354]:            "log" => {
Jan 03 16:37:45 ubuntu1804.localdomain logstash[7354]:           "file" => {
Jan 03 16:37:45 ubuntu1804.localdomain logstash[7354]:             "path" => "/tmp/spring.log"
Jan 03 16:37:45 ubuntu1804.localdomain logstash[7354]:         },
Jan 03 16:37:45 ubuntu1804.localdomain logstash[7354]:         "offset" => 37805
Jan 03 16:37:45 ubuntu1804.localdomain logstash[7354]:     },
Jan 03 16:37:45 ubuntu1804.localdomain logstash[7354]:         "thread" => "http-nio-8080-exec-10",
Jan 03 16:37:45 ubuntu1804.localdomain logstash[7354]:         "logger" => "org.littlewings.spring.example.App",
Jan 03 16:37:45 ubuntu1804.localdomain logstash[7354]:     "@timestamp" => 2020-01-03T16:37:33.575Z,
Jan 03 16:37:45 ubuntu1804.localdomain logstash[7354]:           "tags" => [
Jan 03 16:37:45 ubuntu1804.localdomain logstash[7354]:         [0] "beats_input_codec_plain_applied"
Jan 03 16:37:45 ubuntu1804.localdomain logstash[7354]:     ],
Jan 03 16:37:45 ubuntu1804.localdomain logstash[7354]:            "pid" => "19574"
Jan 03 16:37:45 ubuntu1804.localdomain logstash[7354]: }

OKそうですね。

これで、自分で作るGrokパターンをファイルとして用意できることを確認できました、と。

オマケ: Logstashのパイプライン定義の中に、カスタムパターンを用意する

オマケで、Logstashのパイプライン定義の中にカスタムパターンを書く…要するに、インラインで書く例も載せておきます。

この場合、patten_definitionsを使います。

[Grok Filter Configuration Options / pattern_definitions

ひとつのGrok filterの中でのみ有効なので、他の箇所でも使いたい場合はファイルとして切り出した方が良いです。

こんな感じで使います。

  grok {
    pattern_definitions => {
      "SPRING_APP_LOG_MESSAGE" => "(?m)^(?<logtime>\d\d\d\d-\d\d-\d\d \d\d:\d\d:\d\d.\d\d\d) *%{LOGLEVEL:loglevel} %{POSINT:pid} --- \[%{NOTSPACE:thread}\] %{NOTSPACE:logger} *: *%{GREEDYDATA:logmessage}"
    }
    match => { "message" => "%{SPRING_APP_LOG_MESSAGE}" }
  }

pattern_definitionsの引数がハッシュなので、複数パターンを書こうと思ったら全然うまくいきませんでした…。