March 2020 – Balanced

At work we do log parsing and shipping using logstash. Logstash has been working great and it’s been stable for us but testing the pipelines has been a bit of a hurdle, especially for people that’s not so well versed with Linux.

To solve this issue I decided to try to develop a tool for testing pipelines. In order for the project to be successful the following criterias had to be met.

The tool had to support any client platform, be it Windows, Linux or Macintosh
It had to be easy to use without knowing Linux commands
The interface should give direct results from the Logstash output section

The result ended up in a combination of a Web Frontent, NodeJs and Logstash.

How to start

If needed, install docker on your machine
Clone the repository on GitHub.
Copy your pipeline folder to logstash/logstash-config/pipeline
Enter the repository directory
Run docker-compose build
Run docker-compose up
Open up http://localhost:8080 in your browser to access the interface

IMPORTANT – Depending on docker host you might get the question if you want to share your drive with the docker service. If you get this message, share your drive, stop the containers and run docker-compose up again.

Adding pipelines

There are two ways, either modify the existing pipelines or follow thes following steps.

Modify logstash-config/pipelines.yml

# Example covers the creation of a pipeline called mypipeline
- pipeline.id: generic-json
  path.config: "/usr/share/logstash/pipeline/mypipeline"

Then create a directory in logstash-config/pipeline called mypipeline
Copy your logstash config to logstash-config/pipeline/mypipeline
Last but not least, restart the containers using docker-compose up

Modifying pipelines

Since the logstash pipeline directory is mounted in the containers logstash should detect file changes and reload the pipeline. If this does not happen for some reason you need to restart the containers.

Troubleshooting

Check the docker-compose window for any exceptions thrown.

Reporting issues

First, please check if there’s any current issues that matches your problem. If not, please feel free to submit an issue here at Github.

I have very limited time so I won’t be able to act fast on any issue but it’s always good to have it logged and who knows, maybe someone else will pick it up and make a PR.