Logstash Testing Tool

At work we do log parsing and shipping using logstash. Logstash has been working great and it’s been stable for us but testing the pipelines has been a bit of a hurdle, especially for people that’s not so well versed with Linux.

To solve this issue I decided to try to develop a tool for testing pipelines. In order for the project to be successful the following criterias had to be met.

The tool had to support any client platform, be it Windows, Linux or Macintosh
It had to be easy to use without knowing Linux commands
The interface should give direct results from the Logstash output section

The result ended up in a combination of a UI built with NodeJs and Logstash using Docker.

Before you start

Please note that this tools is here to test the filter section of a pipeline.

The script natively supports tcp/udp + syslog inputs
If you want to other types of input you need to send the data directly to logstash and also expose the corresponding port
Each output section needs to be defined as stated below

How to start

If needed, install docker on your machine
Clone the repository on GitHub.
Copy your pipeline folder to logstash/logstash-config/pipeline

Replace the output section in each pipeline with the tools predefined output:

output {
    http {
        format => "json"
        http_method => "post"
        url => "http://pipeline-ui:8080/api/v1/receiveLogstashOutput"
    }
}

Enter the repository directory
Run docker-compose build
Run docker-compose up
Open up http://localhost:8080 in your browser to access the interface
It will directly attempt to connect to the backend and logstash. Wait until both status buttons are green before starting to send data.

IMPORTANT – Depending on docker host you might get the question if you want to share your drive with the docker service. If you get this message, share your drive, stop the containers and run docker-compose up again.

Adding pipelines

There are two ways, either modify the existing pipelines or follow thes following steps.

Modify logstash-config/pipelines.yml

# Example covers the creation of a pipeline called mypipeline
- pipeline.id: generic-json
  path.config: "/usr/share/logstash/pipeline/mypipeline"

Then create a directory in logstash-config/pipeline called mypipeline
Copy your logstash config to logstash-config/pipeline/mypipeline
Last but not least, restart the containers using docker-compose up

Modifying pipelines

Since the logstash pipeline directory is mounted in the containers logstash should detect file changes and reload the pipeline. If this does not happen for some reason you need to restart the containers.

Troubleshooting

Check the docker-compose window for any exceptions thrown.

Reporting issues

First, please check if there’s any current issues that matches your problem. If not, please feel free to submit an issue here at Github.

I have very limited time so I won’t be able to act fast on any issue but it’s always good to have it logged and who knows, maybe someone else will pick it up and make a PR.

Application flow diagram

Screenshot from the application

Code available here

https://github.com/epacke/logstash-pipeline-tester