Beats-forwarder:
How to send Elastic Beats through a HTTP output

Beats are an open-source initiative from Elastic and the community. The Elastic Beats goal is to report any metrics or logs from various applications to an Elasticsearch cluster. They are lightweight by design, cross platform (Linux, MacOs and Windows) functional, and written in Go. Once up and running they simply produce, for each “application’s heartbeat”, a Json object: the Beat.

{
    "metadata": {"beat":"metricbeat",  "type":"metricsets" },
    "type":"metricsets",
    "tags": [ "logmatic.io"],
    "timestamp":"2017-01-23T14:06:09.035Z",
    "beat":{  "hostname":"jarvis",  "name":"jarvis",  "version":"5.1.1"  },
    "system": {"process": "..." }
}

What a metric beat looks like

Many developers have contributed and created more data shippers to the original two ones: Packetbeat and Topbeat. So whatever your favorite stack is running on, there are definitely Beats out there for it. Here are some of the stack parts these Beats can take care of:

  • Databases: PostgreSQL, MySQL, Redis
  • Web servers: Nginx, Apache, HaProxy
  • Trending technologies such as Docker, Kafka, Kubernetes, and so on.

But if you just wish to send logs from files or Windows events, FileBeat and WinBeat will do the job 🙂
Have a look at the official repository to discover all options on GitHub.

The need for a HTTP Output

I recently needed to send my beats to a different system than an elastic cluster ( …some say it was into my Logmatic.io account). The default Elasticsearch output uses HTTP transport together with a specific bulk format to improve performance. So hacking the endpoint would make a very ugly hack in handling formatting.

Other community members had come across the same situation, and they started using a Logstash instance to forward all beats to the HTTP endpoint. But I find Logstash definitely not user-friendly, bringing in much complexity for low performance.

Conclusion: there is no HTTP output for beats available for now, but you can have a look at the opened discussion about it on GitHub.

Building my Go proxy: Beats-forwarder

In order to move things forward, I decided to write a very small proxy using Go to gather all metric beats and events via a simple hook. Using the lumberjack protocol and default logstash output in the beats configuration, it would allow all users to send beats to:

  • A Syslog server
  • Any server using a TCP/UDP endpoint
  • An HTTP Endpoint
  • Or a Logmatic.io platform (do I really need to explain why?)

Here is the repository, so feel free to contribute.

Beats-forwarder quickstart

So, let’s look at how you can send beats to your application via a simple HTTP endpoint.

The first thing you need to do is specify how you wish to use the beats-forwarder. You can either centralize all your beats before sending them or just add a proxy to each server or data shippers.

Proxified servers

Solution 1 – Proxified servers

I personally prefer the second solution, as a more manageable (one instance vs multiples ones) and easier to deploy using a tool like Ansible:

Central proxy

Solution 2 – Central proxy

Whatever solution you choose, you’ll have to go through 3 steps:

  1. Installing the beats-forwarder (just download the binary)
  2. Configuring the logstash output for each data shipper
  3. Configuring the output for the beats-forwarder agent to your HTTP endpoint

So, first and depending on your environment (Linux/Windows) get the latest version of the beats-forwarder from the repository.

Then,

  1. Create an installation directory named “beats-forwarder”
  2. Download and drop the binary here
  3. Create a directory named “beats-forwarder/etc”
  4. In this directory, download and drop the default configuration: Configuration link
  5. Go to the directory “beats-forwarder” and that’s all for the moment

By the way, you can just use the ready-to-use Docker image:

docker run -dt \
    -p 5044:5044 \
    -e BFWD_OUTPUT_TYPE=http \
    -e BFWD_HTTP_ENDPOINT=https://<URL_ENDPOINT> \
    --name beats-forwarder \
    logmatic/beats-forwarder

The beats-forwarder uses both the ElasticBeats library and the Lumberjack protocol to make sure it is compatible with future evolutions. You can consider the beat-forwarder like an extremely light logstash instance (without a jruby runtime or plenty of unneeded plugins).

Then, go to the data shippers’ configuration. Here, you just need to add a Logstash output by editing metricbeat-config.yml file and finish it off at the end with:

output:
  logstash:
    # Set the beats-forwarder address
    hosts: [ "localhost:5044"]

The next step is about the beats-forwarder configuration. Create a “config.yml”
file in the “beats-forwarder” directory and drop the following lines:

####
#### beats-forwarder default configuration
####
input:
  # The port to listen on
  port: 5044

output:

  # The wanted output (syslog|udp_tcp|logmatic|http), by default syslog
  type: http

  # Syslog specific settings
  http:
    # The endpoint URL
    endpoint: https://example.com/my_service_collector

All beats are sent to the URL given using the POST method.

Finally, start the beats-forwarder agent and restart each data shipper.

One last word

The beats-forwarder is currently in beta and being tested on Linux and Windows (lots of pain, but …now working. It’s awesome!).

I plan to improve its code and add more outputs. Here is the roadmap as of now.

  • Add a test coverage (shame on me)
  • Add a graylog/gelf output
  • Package it for systemd, Windows services, etc…
  • Add an AWS S3 output (for a long time backup)
  • Improve code and its robustness, if I can …
  • Benchmark it and publish results to the repository

Any help would be really welcome so feel free to contribute and send ideas, or feedbacks to Feedback@logmatic.io and add me on Twitter @gpolaert 😀

Related Posts

Get notified when new content is published