Sending Logs

Quick Start

Now, you get a brand new OpenSearch endpoint, and you want to make sure you can send logs to it. Please follow the steps below.

  • Create an accounts.json file with 2 bank accounts
cat > accounts.json << EOF
{"index":{"_id":"1"}}
{"account_number":1,"balance":39225,"firstname":"Amber","lastname":"Duke","age":32,"gender":"M","address":"880 Holmes Lane","employer":"Pyrami","email":"amberduke@pyrami.com","city":"Brogan","state":"IL"}
{"index":{"_id":"2"}}
{"account_number":2,"balance":5686,"firstname":"Hattie","lastname":"Bond","age":36,"gender":"M","address":"671 Bristol Street","employer":"Netagy","email":"hattiebond@netagy.com","city":"Dante","state":"TN"}
EOF
cat accounts.json
  • Bulk load accounts.json into OpenSearch
curl -u sauron:mypassword \
-X POST \
-H 'Content-Type: application/x-ndjson' --data-binary @accounts.json \
https://elasticsearch.handu-phx.handu.developers.oracledx.com/bank/_bulk?pretty
  • Verify both bank accounts are available for search
curl -u sauron:mypassword https://elasticsearch.handu-phx.handu.developers.oracledx.com/bank/_search?pretty
...
  "hits" : {
    "total" : 2,
  • If you see 2 hits like shown above, then congratulations, your OpenSearch endpoint is ready for prime!!!
Using Beats to send data to OpenSearch

OpenSearch.org publishes a compatibility matrix to help you select a Beats version that will work with OpenSearch.

Please note that older versions of Beats 6.x.x are not supported any more. Also, Beats versions newer than 7.12.x are not supported by OpenSearch directly. If you must update Beats in your environment to a version later than 7.12.x, you can direct traffic from Beats to Logstash first and then use Logstash Output Plugin for OpenSearch to send data.

Filebeat
Run Filebeat as native application

Here is a Filebeat configuration example:

filebeat.inputs:
- type: log
  paths:
  - /opt/my-application/*.log
  - /opt/my-application/*/*.log
setup.template.enabled: false

name: filebeat

output.elasticsearch:
  ##### :443 is required!!!
  hosts: ["https://elasticsearch.handu-phx.handu.developers.oracledx.com:443"]
  username: "sauron"
  password: "password"
  index: my-application-logs-filebeat-%{+yyyy.MM.dd}
  bulk_max_size: 250

##### Explicitly disable ilm is required!!!
setup.ilm.enabled: false

You can launch Filebeat from command line using

./filebeat -e -c filebeat.yml
Run Filebeat on Kubernetes

You can run Filebeat on Kubernetes to retrieve and ship container logs. For more information, please visit running Filebeat on Kubernetes.

Journalbeat
Run Journalbeat as native application

Here is a Journalbeat configuration example:

journalbeat.inputs:
- paths: []
  seek: cursor
  cursor_seek_fallback: tail
setup.template.enabled: false

name: journalbeat

output.elasticsearch:
  ##### :443 is required!!!
  hosts: ["https://elasticsearch.handu-phx.handu.developers.oracledx.com:443"]
  username: "sauron"
  password: "password"
  index: my-system-journals-journalbeat-%{+yyyy.MM.dd}
  bulk_max_size: 250

##### Explicitly disable ilm is required!!!
setup.ilm.enabled: false

You can launch Journalbeat at command line using

./journalbeat -e -c journalbeat.yml
Metricbeat
Run Metricbeat as native application

Here is a Metricbeat configuration example:

metricbeat.modules:
- module: system
  metricsets: ["cpu","memory","network"]
  enabled: true
  period: 15s
  processes: ['.*']
setup.template.enabled: false

name: metricbeat

output.elasticsearch:
  ##### :443 is required!!!
  hosts: ["https://elasticsearch.handu-phx.handu.developers.oracledx.com:443"]
  username: "sauron"
  password: "password"
  index: my-application-metrics-metricbeat-%{+yyyy.MM.dd}

##### Explicitly disable ilm is required!!!
setup.ilm.enabled: false

You can launch Metricbeat at command line using

./metricbeat -e -c metricbeat.yml
Run Metricbeat on Kubernetes

You can run Metricbeat on Kubernetes to retrieve metrics. For more information, please visit running Metricbeat on Kubernetes.