CLOVER🍀

That was when it all began.

Filebeatで読み込んだログを、LogstashでパースしてElasticsearchに放り込む

Elasticsearch Beats Logstash

これは、なにをしたくて書いたもの？

先日、Filebeatでログファイルを取り込む時に、ElasticsearchのIngest Nodeを使ってパースすることを試してみました。

Filebeatでログファイルを取り込む時に、Ingest Nodeのパイプラインを合わせて使う - CLOVER🍀

今度は、Filebeatが読み込んだログファイルを、1度Logstashに転送してパースし、Elasticsearchに送信するというシナリオをやってみたいと
思います。

Ingest Nodeの役割を、Logstashにさせる感じですね。

環境

今回の環境は、こちらです。

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.3 LTS
Release:    18.04
Codename:   bionic


$ java --version
openjdk 11.0.4 2019-07-16
OpenJDK Runtime Environment (build 11.0.4+11-post-Ubuntu-1ubuntu218.04.3)
OpenJDK 64-Bit Server VM (build 11.0.4+11-post-Ubuntu-1ubuntu218.04.3, mixed mode, sharing)

Filebeat、Logstash、Elasticsearchのバージョンは、いずれも7.5.0。それぞれ、aptでインストールしたものです。

## Filebeat
$ filebeat version
filebeat version 7.5.0 (amd64), libbeat 7.5.0 [6d0d0ae079e5cb1d4f224801ac6df926dfb1594c built 2019-11-26 00:06:12 +0000 UTC]


## Logstash
Dec 15 08:47:38 ubuntu1804.localdomain logstash[13084]: [2019-12-15T00:47:38,191][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.5.0"}


## Elasticsearch
$ curl localhost:9200
{
  "name" : "ubuntu1804.localdomain",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "onE2zA-dSdidCHUHstuHwg",
  "version" : {
    "number" : "7.5.0",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "e9ccaed468e2fac2275a3761849cbee64b39519f",
    "build_date" : "2019-11-26T01:06:52.518245Z",
    "build_snapshot" : false,
    "lucene_version" : "8.3.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

それぞれのミドルウェアが動作しているサーバーは、以下のとおりとします。

  • Filebeat … 192.168.33.11
  • Logstash … 192.168.33.12
  • Elasticsearch … 192.168.33.13

Elasticsearchについては、シングルノードであることと、network.host以外はデフォルトで使用します。

$ sudo grep -vE '^ *#|^$' /etc/elasticsearch/elasticsearch.yml
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 0.0.0.0
discovery.type: "single-node"

お題とサンプルアプリケーション

アプリケーションが出力した、Multilineなメッセージを含むログを、Filebeatで読み込みLogstashでパース、Elasticsearchに取り込みます。

アプリケーションは、こちらで作成したものと同じものを使いましょう。

Filebeatで、複数行のログをElasticsearchに取り込んでみる - CLOVER🍀

pom.xmlの設定。

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
        <maven.compiler.source>11</maven.compiler.source>
        <maven.compiler.target>11</maven.compiler.target>
    </properties>

    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-dependencies</artifactId>
                <version>2.2.2.RELEASE</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
    </dependencies>

    <build>
        <finalName>${project.artifactId}</finalName>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
                <version>2.2.2.RELEASE</version>
                <executions>
                  <execution>
                    <goals>
                      <goal>repackage</goal>
                    </goals>
                  </execution>
                </executions>
            </plugin>
        </plugins>
    </build>

ソースコード。
src/main/java/org/littlewings/spring/example/App.java

package org.littlewings.spring.example;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

@SpringBootApplication
@RestController
@RequestMapping("app")
public class App {
    Logger logger = LoggerFactory.getLogger(App.class);

    public static void main(String... args) {
        SpringApplication.run(App.class, args);
    }

    @GetMapping("info")
    public String info() {
        logger.info("{}, info logging", "hello");

        return "info logging";
    }

    @GetMapping("warn")
    public String warn() {
        logger.warn("{}, warn logging", "world");

        return "warn logging";
    }

    @GetMapping("error")
    public String error() {
        logger.error("{}, error logging", "oops!!");

        return "error logging";
    }

    @GetMapping("exception")
    public String exception() {
        Exception e = new RuntimeException("Oops!!");

        logger.error("exception occurred, {}", "why?", e);

        return "exception logging";
    }
}

こちらをパッケージングして、以下のようにしてFilebeatが動作しているサーバーで起動させます。

$ mvn package
$ java -Dlogging.file.path=/tmp -jar target/simple-logging-web-app.jar

これで、アプリケーション側の準備は完了です。ログファイルは、「/tmp/spring.log」という名前で出力されます。

Filebeatの設定

Filebeat側は、以下のように設定します。

$ sudo grep -vE '^ *#|^$' /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /tmp/spring.log
  multiline.pattern: '^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\.\d{3}'
  multiline.negate: true
  multiline.match: after
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 1
setup.kibana:
  host: "192.168.33.14:5601"
output.logstash:
  hosts: ["192.168.33.12:5044"]
processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

Kibanaの設定も入っていますが、今回はKibanaは使わないので無視…。

Multiline messageを読めるように設定。

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /tmp/spring.log
  multiline.pattern: '^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\.\d{3}'
  multiline.negate: true
  multiline.match: after

Outputは、Logstashとします。

output.logstash:
  hosts: ["192.168.33.12:5044"]

Configure the Logstash output | Filebeat Reference [7.5] | Elastic

デフォルトで設定ファイルに記載されている、ElasticsearchへのOutputはコメントアウトしておきましょう。

#-------------------------- Elasticsearch output ------------------------------
#output.elasticsearch:
  # Array of hosts to connect to.
  #  hosts: ["localhost:9200"]

あとは、Filebeatを再起動すれば設定完了です。

$ sudo systemctl restart filebeat

Logstashの設定をする

では、Logstashの設定をしていきます。

いきなりパースなどに入るのもなんなので、まずは受け取ったログを標準出力してみたいと思います。

Beats input plugin | Logstash Reference [7.5] | Elastic

Stdout output plugin | Logstash Reference [7.5] | Elastic

こんな感じで設定。
/etc/logstash/conf.d/logstash-app-log.conf

input {
  beats {
    port => 5044
  }
}

output {
  stdout { }
}

InputをBeatsにして、Outputを標準出力にしただけのシンプルな構成。

Logstashに含まれている、サンプルをほぼマネした感じです。
/etc/logstash/logstash-sample.conf

# Sample Logstash configuration for creating a simple
# Beats -> Logstash -> Elasticsearch pipeline.

input {
  beats {
    port => 5044
  }
}

output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
    #user => "elastic"
    #password => "changeme"
  }
}

設定を加えたら、Logstashを再起動。

$ sudo systemctl restart logstash

これで、journalctlでログを追ってみます。

$ sudo journalctl -u logstash -f

Filebeat側のサーバーで、アプリケーションログを出力させてみます。

$ curl localhost:8080/app/info
info logging

$ curl localhost:8080/app/exception
exception logging

$ curl localhost:8080/app/warn
warn logging

ログを見てみます。

Dec 15 09:30:49 ubuntu1804.localdomain logstash[13608]: {
Dec 15 09:30:49 ubuntu1804.localdomain logstash[13608]:           "host" => {
Dec 15 09:30:49 ubuntu1804.localdomain logstash[13608]:                    "os" => {
Dec 15 09:30:49 ubuntu1804.localdomain logstash[13608]:             "codename" => "bionic",
Dec 15 09:30:49 ubuntu1804.localdomain logstash[13608]:              "version" => "18.04.3 LTS (Bionic Beaver)",
Dec 15 09:30:49 ubuntu1804.localdomain logstash[13608]:                 "name" => "Ubuntu",
Dec 15 09:30:49 ubuntu1804.localdomain logstash[13608]:               "kernel" => "4.15.0-70-generic",
Dec 15 09:30:49 ubuntu1804.localdomain logstash[13608]:               "family" => "debian",
Dec 15 09:30:49 ubuntu1804.localdomain logstash[13608]:             "platform" => "ubuntu"
Dec 15 09:30:49 ubuntu1804.localdomain logstash[13608]:         },
Dec 15 09:30:49 ubuntu1804.localdomain logstash[13608]:                  "name" => "ubuntu1804.localdomain",
Dec 15 09:30:49 ubuntu1804.localdomain logstash[13608]:         "containerized" => false,
Dec 15 09:30:49 ubuntu1804.localdomain logstash[13608]:          "architecture" => "x86_64",
Dec 15 09:30:49 ubuntu1804.localdomain logstash[13608]:                    "id" => "63059a9a29924ce4a42e15d497b034ca",
Dec 15 09:30:49 ubuntu1804.localdomain logstash[13608]:              "hostname" => "ubuntu1804.localdomain"
Dec 15 09:30:49 ubuntu1804.localdomain logstash[13608]:     },
Dec 15 09:30:49 ubuntu1804.localdomain logstash[13608]:        "message" => "2019-12-15 01:30:42.727  INFO 5756 --- [http-nio-8080-exec-3] org.littlewings.spring.example.App       : hello, info logging",
Dec 15 09:30:49 ubuntu1804.localdomain logstash[13608]:     "@timestamp" => 2019-12-15T09:30:46.488Z,
Dec 15 09:30:49 ubuntu1804.localdomain logstash[13608]:          "input" => {
Dec 15 09:30:49 ubuntu1804.localdomain logstash[13608]:         "type" => "log"
Dec 15 09:30:49 ubuntu1804.localdomain logstash[13608]:     },
Dec 15 09:30:49 ubuntu1804.localdomain logstash[13608]:            "ecs" => {
Dec 15 09:30:49 ubuntu1804.localdomain logstash[13608]:         "version" => "1.1.0"
Dec 15 09:30:49 ubuntu1804.localdomain logstash[13608]:     },
Dec 15 09:30:49 ubuntu1804.localdomain logstash[13608]:       "@version" => "1",
Dec 15 09:30:49 ubuntu1804.localdomain logstash[13608]:            "log" => {
Dec 15 09:30:49 ubuntu1804.localdomain logstash[13608]:         "offset" => 25267,
Dec 15 09:30:49 ubuntu1804.localdomain logstash[13608]:           "file" => {
Dec 15 09:30:49 ubuntu1804.localdomain logstash[13608]:             "path" => "/tmp/spring.log"
Dec 15 09:30:49 ubuntu1804.localdomain logstash[13608]:         }
Dec 15 09:30:49 ubuntu1804.localdomain logstash[13608]:     },
Dec 15 09:30:49 ubuntu1804.localdomain logstash[13608]:          "agent" => {
Dec 15 09:30:49 ubuntu1804.localdomain logstash[13608]:                 "type" => "filebeat",
Dec 15 09:30:49 ubuntu1804.localdomain logstash[13608]:             "hostname" => "ubuntu1804.localdomain",
Dec 15 09:30:49 ubuntu1804.localdomain logstash[13608]:                   "id" => "ac44a220-8185-4915-b6eb-56959dc2c390",
Dec 15 09:30:49 ubuntu1804.localdomain logstash[13608]:              "version" => "7.5.0",
Dec 15 09:30:49 ubuntu1804.localdomain logstash[13608]:         "ephemeral_id" => "65c2eaee-c651-4c29-a1c2-b317dd13066a"
Dec 15 09:30:49 ubuntu1804.localdomain logstash[13608]:     },
Dec 15 09:30:49 ubuntu1804.localdomain logstash[13608]:           "tags" => [
Dec 15 09:30:49 ubuntu1804.localdomain logstash[13608]:         [0] "beats_input_codec_plain_applied"
Dec 15 09:30:49 ubuntu1804.localdomain logstash[13608]:     ]
Dec 15 09:30:49 ubuntu1804.localdomain logstash[13608]: }
Dec 15 09:30:52 ubuntu1804.localdomain logstash[13608]: {
Dec 15 09:30:52 ubuntu1804.localdomain logstash[13608]:           "host" => {
Dec 15 09:30:52 ubuntu1804.localdomain logstash[13608]:                    "os" => {
Dec 15 09:30:52 ubuntu1804.localdomain logstash[13608]:             "codename" => "bionic",
Dec 15 09:30:52 ubuntu1804.localdomain logstash[13608]:              "version" => "18.04.3 LTS (Bionic Beaver)",
Dec 15 09:30:52 ubuntu1804.localdomain logstash[13608]:                 "name" => "Ubuntu",
Dec 15 09:30:52 ubuntu1804.localdomain logstash[13608]:               "kernel" => "4.15.0-70-generic",
Dec 15 09:30:52 ubuntu1804.localdomain logstash[13608]:               "family" => "debian",
Dec 15 09:30:52 ubuntu1804.localdomain logstash[13608]:             "platform" => "ubuntu"
Dec 15 09:30:52 ubuntu1804.localdomain logstash[13608]:         },
Dec 15 09:30:52 ubuntu1804.localdomain logstash[13608]:                  "name" => "ubuntu1804.localdomain",
Dec 15 09:30:52 ubuntu1804.localdomain logstash[13608]:         "containerized" => false,
Dec 15 09:30:52 ubuntu1804.localdomain logstash[13608]:              "hostname" => "ubuntu1804.localdomain",
Dec 15 09:30:52 ubuntu1804.localdomain logstash[13608]:          "architecture" => "x86_64",
Dec 15 09:30:52 ubuntu1804.localdomain logstash[13608]:                    "id" => "63059a9a29924ce4a42e15d497b034ca"
Dec 15 09:30:52 ubuntu1804.localdomain logstash[13608]:     },
Dec 15 09:30:52 ubuntu1804.localdomain logstash[13608]:        "message" => "2019-12-15 01:30:46.449 ERROR 5756 --- [http-nio-8080-exec-5] org.littlewings.spring.example.App       : exception occurred, why?\n\njava.lang.RuntimeException: Oops!!\n\tat org.littlewings.spring.example.App.exception(App.java:44) ~[classes!/:na]\n\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]\n\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na]\n\tat java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]\n\tat java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]\n\tat org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190) ~[spring-web-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]\n\tat org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138) ~[spring-web-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]\n\tat org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:106) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]\n\tat org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:888) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]\n\tat org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:793) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]\n\tat org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]\n\tat org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]\n\tat org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]\n\tat org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]\n\tat org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:634) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:741) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) ~[tomcat-embed-websocket-9.0.29.jar!/:9.0.29]\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) ~[spring-web-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) ~[spring-web-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) ~[spring-web-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:526) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:367) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:860) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1591) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na]\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na]\n\tat org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]\n",
Dec 15 09:30:52 ubuntu1804.localdomain logstash[13608]:     "@timestamp" => 2019-12-15T09:30:46.488Z,
Dec 15 09:30:52 ubuntu1804.localdomain logstash[13608]:          "input" => {
Dec 15 09:30:52 ubuntu1804.localdomain logstash[13608]:         "type" => "log"
Dec 15 09:30:52 ubuntu1804.localdomain logstash[13608]:     },
Dec 15 09:30:52 ubuntu1804.localdomain logstash[13608]:            "ecs" => {
Dec 15 09:30:52 ubuntu1804.localdomain logstash[13608]:         "version" => "1.1.0"
Dec 15 09:30:52 ubuntu1804.localdomain logstash[13608]:     },
Dec 15 09:30:52 ubuntu1804.localdomain logstash[13608]:       "@version" => "1",
Dec 15 09:30:52 ubuntu1804.localdomain logstash[13608]:            "log" => {
Dec 15 09:30:52 ubuntu1804.localdomain logstash[13608]:         "offset" => 25392,
Dec 15 09:30:52 ubuntu1804.localdomain logstash[13608]:          "flags" => [
Dec 15 09:30:52 ubuntu1804.localdomain logstash[13608]:             [0] "multiline"
Dec 15 09:30:52 ubuntu1804.localdomain logstash[13608]:         ],
Dec 15 09:30:52 ubuntu1804.localdomain logstash[13608]:           "file" => {
Dec 15 09:30:52 ubuntu1804.localdomain logstash[13608]:             "path" => "/tmp/spring.log"
Dec 15 09:30:52 ubuntu1804.localdomain logstash[13608]:         }
Dec 15 09:30:52 ubuntu1804.localdomain logstash[13608]:     },
Dec 15 09:30:52 ubuntu1804.localdomain logstash[13608]:          "agent" => {
Dec 15 09:30:52 ubuntu1804.localdomain logstash[13608]:                 "type" => "filebeat",
Dec 15 09:30:52 ubuntu1804.localdomain logstash[13608]:             "hostname" => "ubuntu1804.localdomain",
Dec 15 09:30:52 ubuntu1804.localdomain logstash[13608]:                   "id" => "ac44a220-8185-4915-b6eb-56959dc2c390",
Dec 15 09:30:52 ubuntu1804.localdomain logstash[13608]:              "version" => "7.5.0",
Dec 15 09:30:52 ubuntu1804.localdomain logstash[13608]:         "ephemeral_id" => "65c2eaee-c651-4c29-a1c2-b317dd13066a"
Dec 15 09:30:52 ubuntu1804.localdomain logstash[13608]:     },
Dec 15 09:30:52 ubuntu1804.localdomain logstash[13608]:           "tags" => [
Dec 15 09:30:52 ubuntu1804.localdomain logstash[13608]:         [0] "beats_input_codec_plain_applied"
Dec 15 09:30:52 ubuntu1804.localdomain logstash[13608]:     ]
Dec 15 09:30:52 ubuntu1804.localdomain logstash[13608]: }
Dec 15 09:30:59 ubuntu1804.localdomain logstash[13608]: {
Dec 15 09:30:59 ubuntu1804.localdomain logstash[13608]:           "host" => {
Dec 15 09:30:59 ubuntu1804.localdomain logstash[13608]:                    "os" => {
Dec 15 09:30:59 ubuntu1804.localdomain logstash[13608]:             "codename" => "bionic",
Dec 15 09:30:59 ubuntu1804.localdomain logstash[13608]:              "version" => "18.04.3 LTS (Bionic Beaver)",
Dec 15 09:30:59 ubuntu1804.localdomain logstash[13608]:                 "name" => "Ubuntu",
Dec 15 09:30:59 ubuntu1804.localdomain logstash[13608]:               "kernel" => "4.15.0-70-generic",
Dec 15 09:30:59 ubuntu1804.localdomain logstash[13608]:               "family" => "debian",
Dec 15 09:30:59 ubuntu1804.localdomain logstash[13608]:             "platform" => "ubuntu"
Dec 15 09:30:59 ubuntu1804.localdomain logstash[13608]:         },
Dec 15 09:30:59 ubuntu1804.localdomain logstash[13608]:                  "name" => "ubuntu1804.localdomain",
Dec 15 09:30:59 ubuntu1804.localdomain logstash[13608]:         "containerized" => false,
Dec 15 09:30:59 ubuntu1804.localdomain logstash[13608]:                    "id" => "63059a9a29924ce4a42e15d497b034ca",
Dec 15 09:30:59 ubuntu1804.localdomain logstash[13608]:              "hostname" => "ubuntu1804.localdomain",
Dec 15 09:30:59 ubuntu1804.localdomain logstash[13608]:          "architecture" => "x86_64"
Dec 15 09:30:59 ubuntu1804.localdomain logstash[13608]:     },
Dec 15 09:30:59 ubuntu1804.localdomain logstash[13608]:        "message" => "2019-12-15 01:30:49.499  WARN 5756 --- [http-nio-8080-exec-7] org.littlewings.spring.example.App       : world, warn logging",
Dec 15 09:30:59 ubuntu1804.localdomain logstash[13608]:     "@timestamp" => 2019-12-15T09:30:53.490Z,
Dec 15 09:30:59 ubuntu1804.localdomain logstash[13608]:          "input" => {
Dec 15 09:30:59 ubuntu1804.localdomain logstash[13608]:         "type" => "log"
Dec 15 09:30:59 ubuntu1804.localdomain logstash[13608]:     },
Dec 15 09:30:59 ubuntu1804.localdomain logstash[13608]:            "ecs" => {
Dec 15 09:30:59 ubuntu1804.localdomain logstash[13608]:         "version" => "1.1.0"
Dec 15 09:30:59 ubuntu1804.localdomain logstash[13608]:     },
Dec 15 09:30:59 ubuntu1804.localdomain logstash[13608]:       "@version" => "1",
Dec 15 09:30:59 ubuntu1804.localdomain logstash[13608]:            "log" => {
Dec 15 09:30:59 ubuntu1804.localdomain logstash[13608]:         "offset" => 32335,
Dec 15 09:30:59 ubuntu1804.localdomain logstash[13608]:           "file" => {
Dec 15 09:30:59 ubuntu1804.localdomain logstash[13608]:             "path" => "/tmp/spring.log"
Dec 15 09:30:59 ubuntu1804.localdomain logstash[13608]:         }
Dec 15 09:30:59 ubuntu1804.localdomain logstash[13608]:     },
Dec 15 09:30:59 ubuntu1804.localdomain logstash[13608]:          "agent" => {
Dec 15 09:30:59 ubuntu1804.localdomain logstash[13608]:                 "type" => "filebeat",
Dec 15 09:30:59 ubuntu1804.localdomain logstash[13608]:             "hostname" => "ubuntu1804.localdomain",
Dec 15 09:30:59 ubuntu1804.localdomain logstash[13608]:                   "id" => "ac44a220-8185-4915-b6eb-56959dc2c390",
Dec 15 09:30:59 ubuntu1804.localdomain logstash[13608]:              "version" => "7.5.0",
Dec 15 09:30:59 ubuntu1804.localdomain logstash[13608]:         "ephemeral_id" => "65c2eaee-c651-4c29-a1c2-b317dd13066a"
Dec 15 09:30:59 ubuntu1804.localdomain logstash[13608]:     },
Dec 15 09:30:59 ubuntu1804.localdomain logstash[13608]:           "tags" => [
Dec 15 09:30:59 ubuntu1804.localdomain logstash[13608]:         [0] "beats_input_codec_plain_applied"
Dec 15 09:30:59 ubuntu1804.localdomain logstash[13608]:     ]
Dec 15 09:30:59 ubuntu1804.localdomain logstash[13608]: }

うまく受け取れているようです。

Logstashで、ログのパースや変換などを行ってみる

では、今度はFilter Pluginを使って、パースしたりフィールドを変換したり削除したりしてみましょう。

このあたりのプラグインを使います。

Grok filter plugin | Logstash Reference [7.5] | Elastic

Date filter plugin | Logstash Reference [7.5] | Elastic

Mutate filter plugin | Logstash Reference [7.5] | Elastic

こんな感じに変更。
/etc/logstash/conf.d/logstash-app-log.conf

input {
  beats {
    port => 5044
  }
}

filter {
  grok {
    match => { "message" => "(?m)^(?<logtime>\d\d\d\d-\d\d-\d\d \d\d:\d\d:\d\d.\d\d\d) *%{LOGLEVEL:loglevel} %{POSINT:pid} --- \[%{NOTSPACE:thread}\] %{NOTSPACE:logger} *: *%{GREEDYDATA:logmessage}" }
  }

  mutate {
    rename => [ "@timestamp", "event_time" ]
  }

  date {
    match => [ "logtime", "yyyy-MM-dd HH:mm:ss.SSS" ]
    target => "@timestamp"
  }

  mutate {
    remove_field => [ "logtime", "message" ]
  }
}

output {
  stdout { }
}

Filter Pluginを使い、Grolでパースして

  grok {
    match => { "message" => "(?m)^(?<logtime>\d\d\d\d-\d\d-\d\d \d\d:\d\d:\d\d.\d\d\d) *%{LOGLEVEL:loglevel} %{POSINT:pid} --- \[%{NOTSPACE:thread}\] %{NOTSPACE:logger} *: *%{GREEDYDATA:logmessage}" }
  }

もともとの「@timestamp」フィールドをリネーム、ログ内の時間を「@timestamp」に置き換え、要らなくなった「logtime」と「message」
フィールドを削除。

  mutate {
    rename => [ "@timestamp", "event_time" ]
  }

  date {
    match => [ "logtime", "yyyy-MM-dd HH:mm:ss.SSS" ]
    target => "@timestamp"
  }

  mutate {
    remove_field => [ "logtime", "message" ]
  }

出力先は、Stdout Plugin（標準出力）のままです。

Logstashを再起動。

$ sudo systemctl restart logstash

確認してみます。

$ curl localhost:8080/app/info
info logging

$ curl localhost:8080/app/exception
exception logging

$ curl localhost:8080/app/warn
warn logging
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]: {
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:           "host" => {
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:                  "name" => "ubuntu1804.localdomain",
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:                    "os" => {
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:             "codename" => "bionic",
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:              "version" => "18.04.3 LTS (Bionic Beaver)",
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:               "kernel" => "4.15.0-70-generic",
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:                 "name" => "Ubuntu",
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:             "platform" => "ubuntu",
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:               "family" => "debian"
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:         },
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:          "architecture" => "x86_64",
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:                    "id" => "63059a9a29924ce4a42e15d497b034ca",
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:         "containerized" => false,
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:              "hostname" => "ubuntu1804.localdomain"
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:     },
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:       "@version" => "1",
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:            "pid" => "5756",
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:     "event_time" => 2019-12-15T13:00:07.164Z,
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:           "tags" => [
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:         [0] "beats_input_codec_plain_applied"
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:     ],
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:     "@timestamp" => 2019-12-15T13:00:05.634Z,
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:       "loglevel" => "ERROR",
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:     "logmessage" => "exception occurred, why?\n\njava.lang.RuntimeException: Oops!!\n\tat org.littlewings.spring.example.App.exception(App.java:44) ~[classes!/:na]\n\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]\n\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na]\n\tat java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]\n\tat java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]\n\tat org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190) ~[spring-web-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]\n\tat org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138) ~[spring-web-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]\n\tat org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:106) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]\n\tat org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:888) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]\n\tat org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:793) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]\n\tat org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]\n\tat org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]\n\tat org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]\n\tat org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]\n\tat org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:634) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:741) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) ~[tomcat-embed-websocket-9.0.29.jar!/:9.0.29]\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) ~[spring-web-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) ~[spring-web-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) ~[spring-web-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:526) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:367) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:860) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1591) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na]\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na]\n\tat org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]\n",
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:            "ecs" => {
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:         "version" => "1.1.0"
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:     },
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:         "thread" => "http-nio-8080-exec-1",
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:          "input" => {
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:         "type" => "log"
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:     },
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:          "agent" => {
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:                 "type" => "filebeat",
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:                   "id" => "ac44a220-8185-4915-b6eb-56959dc2c390",
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:              "version" => "7.5.0",
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:         "ephemeral_id" => "65c2eaee-c651-4c29-a1c2-b317dd13066a",
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:             "hostname" => "ubuntu1804.localdomain"
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:     },
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:         "logger" => "org.littlewings.spring.example.App",
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:            "log" => {
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:         "offset" => 32585,
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:          "flags" => [
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:             [0] "multiline"
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:         ],
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:           "file" => {
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:             "path" => "/tmp/spring.log"
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:         }
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:     }
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]: }
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]: {
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:           "host" => {
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:                  "name" => "ubuntu1804.localdomain",
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:                    "os" => {
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:             "codename" => "bionic",
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:               "kernel" => "4.15.0-70-generic",
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:              "version" => "18.04.3 LTS (Bionic Beaver)",
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:                 "name" => "Ubuntu",
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:             "platform" => "ubuntu",
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:               "family" => "debian"
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:         },
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:          "architecture" => "x86_64",
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:                    "id" => "63059a9a29924ce4a42e15d497b034ca",
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:         "containerized" => false,
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:              "hostname" => "ubuntu1804.localdomain"
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:     },
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:       "@version" => "1",
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:            "pid" => "5756",
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:     "event_time" => 2019-12-15T13:00:07.164Z,
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:           "tags" => [
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:         [0] "beats_input_codec_plain_applied"
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:     ],
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:     "@timestamp" => 2019-12-15T13:00:04.185Z,
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:       "loglevel" => "INFO",
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:     "logmessage" => "hello, info logging",
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:            "ecs" => {
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:         "version" => "1.1.0"
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:     },
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:         "thread" => "http-nio-8080-exec-9",
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:          "input" => {
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:         "type" => "log"
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:     },
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:          "agent" => {
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:                 "type" => "filebeat",
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:                   "id" => "ac44a220-8185-4915-b6eb-56959dc2c390",
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:              "version" => "7.5.0",
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:         "ephemeral_id" => "65c2eaee-c651-4c29-a1c2-b317dd13066a",
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:             "hostname" => "ubuntu1804.localdomain"
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:     },
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:         "logger" => "org.littlewings.spring.example.App",
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:            "log" => {
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:         "offset" => 32460,
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:           "file" => {
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:             "path" => "/tmp/spring.log"
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:         }
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]:     }
Dec 15 13:00:10 ubuntu1804.localdomain logstash[13820]: }
Dec 15 13:00:13 ubuntu1804.localdomain logstash[13820]: {
Dec 15 13:00:13 ubuntu1804.localdomain logstash[13820]:           "host" => {
Dec 15 13:00:13 ubuntu1804.localdomain logstash[13820]:                  "name" => "ubuntu1804.localdomain",
Dec 15 13:00:13 ubuntu1804.localdomain logstash[13820]:                    "os" => {
Dec 15 13:00:13 ubuntu1804.localdomain logstash[13820]:             "codename" => "bionic",
Dec 15 13:00:13 ubuntu1804.localdomain logstash[13820]:              "version" => "18.04.3 LTS (Bionic Beaver)",
Dec 15 13:00:13 ubuntu1804.localdomain logstash[13820]:               "kernel" => "4.15.0-70-generic",
Dec 15 13:00:13 ubuntu1804.localdomain logstash[13820]:                 "name" => "Ubuntu",
Dec 15 13:00:13 ubuntu1804.localdomain logstash[13820]:             "platform" => "ubuntu",
Dec 15 13:00:13 ubuntu1804.localdomain logstash[13820]:               "family" => "debian"
Dec 15 13:00:13 ubuntu1804.localdomain logstash[13820]:         },
Dec 15 13:00:13 ubuntu1804.localdomain logstash[13820]:          "architecture" => "x86_64",
Dec 15 13:00:13 ubuntu1804.localdomain logstash[13820]:                    "id" => "63059a9a29924ce4a42e15d497b034ca",
Dec 15 13:00:13 ubuntu1804.localdomain logstash[13820]:         "containerized" => false,
Dec 15 13:00:13 ubuntu1804.localdomain logstash[13820]:              "hostname" => "ubuntu1804.localdomain"
Dec 15 13:00:13 ubuntu1804.localdomain logstash[13820]:     },
Dec 15 13:00:13 ubuntu1804.localdomain logstash[13820]:       "@version" => "1",
Dec 15 13:00:13 ubuntu1804.localdomain logstash[13820]:            "pid" => "5756",
Dec 15 13:00:13 ubuntu1804.localdomain logstash[13820]:     "event_time" => 2019-12-15T13:00:07.165Z,
Dec 15 13:00:13 ubuntu1804.localdomain logstash[13820]:           "tags" => [
Dec 15 13:00:13 ubuntu1804.localdomain logstash[13820]:         [0] "beats_input_codec_plain_applied"
Dec 15 13:00:13 ubuntu1804.localdomain logstash[13820]:     ],
Dec 15 13:00:13 ubuntu1804.localdomain logstash[13820]:     "@timestamp" => 2019-12-15T13:00:07.151Z,
Dec 15 13:00:13 ubuntu1804.localdomain logstash[13820]:       "loglevel" => "WARN",
Dec 15 13:00:13 ubuntu1804.localdomain logstash[13820]:     "logmessage" => "world, warn logging",
Dec 15 13:00:13 ubuntu1804.localdomain logstash[13820]:            "ecs" => {
Dec 15 13:00:13 ubuntu1804.localdomain logstash[13820]:         "version" => "1.1.0"
Dec 15 13:00:13 ubuntu1804.localdomain logstash[13820]:     },
Dec 15 13:00:13 ubuntu1804.localdomain logstash[13820]:         "thread" => "http-nio-8080-exec-3",
Dec 15 13:00:13 ubuntu1804.localdomain logstash[13820]:          "agent" => {
Dec 15 13:00:13 ubuntu1804.localdomain logstash[13820]:                 "type" => "filebeat",
Dec 15 13:00:13 ubuntu1804.localdomain logstash[13820]:         "ephemeral_id" => "65c2eaee-c651-4c29-a1c2-b317dd13066a",
Dec 15 13:00:13 ubuntu1804.localdomain logstash[13820]:                   "id" => "ac44a220-8185-4915-b6eb-56959dc2c390",
Dec 15 13:00:13 ubuntu1804.localdomain logstash[13820]:              "version" => "7.5.0",
Dec 15 13:00:13 ubuntu1804.localdomain logstash[13820]:             "hostname" => "ubuntu1804.localdomain"
Dec 15 13:00:13 ubuntu1804.localdomain logstash[13820]:     },
Dec 15 13:00:13 ubuntu1804.localdomain logstash[13820]:          "input" => {
Dec 15 13:00:13 ubuntu1804.localdomain logstash[13820]:         "type" => "log"
Dec 15 13:00:13 ubuntu1804.localdomain logstash[13820]:     },
Dec 15 13:00:13 ubuntu1804.localdomain logstash[13820]:         "logger" => "org.littlewings.spring.example.App",
Dec 15 13:00:13 ubuntu1804.localdomain logstash[13820]:            "log" => {
Dec 15 13:00:13 ubuntu1804.localdomain logstash[13820]:         "offset" => 39528,
Dec 15 13:00:13 ubuntu1804.localdomain logstash[13820]:           "file" => {
Dec 15 13:00:13 ubuntu1804.localdomain logstash[13820]:             "path" => "/tmp/spring.log"
Dec 15 13:00:13 ubuntu1804.localdomain logstash[13820]:         }
Dec 15 13:00:13 ubuntu1804.localdomain logstash[13820]:     }
Dec 15 13:00:13 ubuntu1804.localdomain logstash[13820]: }

うまくパースなどができているようですね。

では、最後にデータをElasticsearchに保存するように変更します。
/etc/logstash/conf.d/logstash-app-log.conf

input {
  beats {
    port => 5044
  }
}

filter {
  grok {
    match => { "message" => "(?m)^(?<logtime>\d\d\d\d-\d\d-\d\d \d\d:\d\d:\d\d.\d\d\d) *%{LOGLEVEL:loglevel} %{POSINT:pid} --- \[%{NOTSPACE:thread}\] %{NOTSPACE:logger} *: *%{GREEDYDATA:logmessage}" }
  }

  mutate {
    rename => [ "@timestamp", "event_time" ]
  }

  date {
    match => [ "logtime", "yyyy-MM-dd HH:mm:ss.SSS" ]
    target => "@timestamp"
  }

  mutate {
    remove_field => [ "logtime", "message" ]
  }
}

output {
  elasticsearch {
    hosts => ["http://192.168.33.13:9200"]
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
  }
}

変更したのは、Output Pluginだけですね。

Elasticsearch output plugin | Logstash Reference [7.5] | Elastic

output {
  elasticsearch {
    hosts => ["http://192.168.33.13:9200"]
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
  }
}

Logstashを再起動。

$ sudo systemctl restart logstash

ログを出力させて確認。

$ curl localhost:8080/app/info
info logging

$ curl localhost:8080/app/exception
exception logging

$ curl localhost:8080/app/warn
warn logging

Elasticsearch側で確認してみます。

$ curl localhost:9200/filebeat-7.5.0-2019.12.15/_search?pretty
{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 2,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "filebeat-7.5.0-2019.12.15",
        "_type" : "_doc",
        "_id" : "B4ivCW8BHPQzBAWMvx4d",
        "_score" : 1.0,
        "_source" : {
          "input" : {
            "type" : "log"
          },
          "tags" : [
            "beats_input_codec_plain_applied"
          ],
          "@version" : "1",
          "@timestamp" : "2019-12-15T13:11:38.800Z",
          "agent" : {
            "type" : "filebeat",
            "id" : "ac44a220-8185-4915-b6eb-56959dc2c390",
            "version" : "7.5.0",
            "ephemeral_id" : "65c2eaee-c651-4c29-a1c2-b317dd13066a",
            "hostname" : "ubuntu1804.localdomain"
          },
          "thread" : "http-nio-8080-exec-5",
          "ecs" : {
            "version" : "1.1.0"
          },
          "event_time" : "2019-12-15T13:11:47.192Z",
          "host" : {
            "architecture" : "x86_64",
            "name" : "ubuntu1804.localdomain",
            "os" : {
              "kernel" : "4.15.0-70-generic",
              "version" : "18.04.3 LTS (Bionic Beaver)",
              "platform" : "ubuntu",
              "name" : "Ubuntu",
              "codename" : "bionic",
              "family" : "debian"
            },
            "id" : "63059a9a29924ce4a42e15d497b034ca",
            "containerized" : false,
            "hostname" : "ubuntu1804.localdomain"
          },
          "logmessage" : "exception occurred, why?\n\njava.lang.RuntimeException: Oops!!\n\tat org.littlewings.spring.example.App.exception(App.java:44) ~[classes!/:na]\n\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]\n\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na]\n\tat java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]\n\tat java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]\n\tat org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190) ~[spring-web-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]\n\tat org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138) ~[spring-web-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]\n\tat org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:106) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]\n\tat org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:888) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]\n\tat org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:793) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]\n\tat org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]\n\tat org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]\n\tat org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]\n\tat org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]\n\tat org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:634) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:741) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) ~[tomcat-embed-websocket-9.0.29.jar!/:9.0.29]\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) ~[spring-web-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) ~[spring-web-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) ~[spring-web-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:526) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:367) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:860) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1591) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na]\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na]\n\tat org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]\n\tat java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]\n",
          "log" : {
            "flags" : [
              "multiline"
            ],
            "offset" : 39778,
            "file" : {
              "path" : "/tmp/spring.log"
            }
          },
          "logger" : "org.littlewings.spring.example.App",
          "pid" : "5756",
          "loglevel" : "ERROR"
        }
      },
      {
        "_index" : "filebeat-7.5.0-2019.12.15",
        "_type" : "_doc",
        "_id" : "CIivCW8BHPQzBAWMwB4d",
        "_score" : 1.0,
        "_source" : {
          "input" : {
            "type" : "log"
          },
          "tags" : [
            "beats_input_codec_plain_applied"
          ],
          "@version" : "1",
          "@timestamp" : "2019-12-15T13:11:37.534Z",
          "agent" : {
            "type" : "filebeat",
            "id" : "ac44a220-8185-4915-b6eb-56959dc2c390",
            "version" : "7.5.0",
            "ephemeral_id" : "65c2eaee-c651-4c29-a1c2-b317dd13066a",
            "hostname" : "ubuntu1804.localdomain"
          },
          "thread" : "http-nio-8080-exec-4",
          "ecs" : {
            "version" : "1.1.0"
          },
          "event_time" : "2019-12-15T13:11:47.192Z",
          "host" : {
            "architecture" : "x86_64",
            "name" : "ubuntu1804.localdomain",
            "os" : {
              "kernel" : "4.15.0-70-generic",
              "version" : "18.04.3 LTS (Bionic Beaver)",
              "platform" : "ubuntu",
              "name" : "Ubuntu",
              "codename" : "bionic",
              "family" : "debian"
            },
            "id" : "63059a9a29924ce4a42e15d497b034ca",
            "containerized" : false,
            "hostname" : "ubuntu1804.localdomain"
          },
          "logmessage" : "hello, info logging",
          "log" : {
            "offset" : 39653,
            "file" : {
              "path" : "/tmp/spring.log"
            }
          },
          "logger" : "org.littlewings.spring.example.App",
          "pid" : "5756",
          "loglevel" : "INFO"
        }
      }
    ]
  }
}

パース、変換した結果が、うまくElasticsearchに入ったようです。

今回は、こんなところで。