At work we do log parsing and shipping using logstash. Logstash has been working great and it’s been stable for us but testing the pipelines has been a bit of a hurdle, especially for people that’s not so well versed with Linux.
To solve this issue I decided to try to develop a tool for testing pipelines. In order for the project to be successful the following criterias had to be met.
- The tool had to support any client platform, be it Windows, Linux or Macintosh
- It had to be easy to use without knowing Linux commands
- The interface should give direct results from the Logstash output section
The result ended up in a combination of a Web Frontent, NodeJs and Logstash.
How to start
- If needed, install docker on your machine
- Clone the repository on GitHub.
- Copy your pipeline folder to
- Enter the repository directory
- Open up http://localhost:8080 in your browser to access the interface
IMPORTANT – Depending on docker host you might get the question if you want to share your drive with the docker service. If you get this message, share your drive, stop the containers and run
docker-compose up again.
There are two ways, either modify the existing pipelines or follow thes following steps.
# Example covers the creation of a pipeline called mypipeline - pipeline.id: generic-json path.config: "/usr/share/logstash/pipeline/mypipeline"
- Then create a directory in
- Copy your logstash config to
- Last but not least, restart the containers using
Since the logstash pipeline directory is mounted in the containers logstash should detect file changes and reload the pipeline. If this does not happen for some reason you need to restart the containers.
Check the docker-compose window for any exceptions thrown.
First, please check if there’s any current issues that matches your problem. If not, please feel free to submit an issue here at Github.
I have very limited time so I won’t be able to act fast on any issue but it’s always good to have it logged and who knows, maybe someone else will pick it up and make a PR.