At work we do log parsing and shipping using logstash. Logstash has been working great and it’s been stable for us but testing the pipelines has been a bit of a hurdle, especially for people that’s not so well versed with Linux.
To solve this issue I decided to try to develop a tool for testing pipelines. In order for the project to be successful the following criterias had to be met.
- The tool had to support any client platform, be it Windows, Linux or Macintosh
- It had to be easy to use without knowing Linux commands
- The interface should give direct results from the Logstash output section
The result ended up in a combination of a UI built with NodeJs and Logstash using Docker.
Table of Contents
Before you start
Please note that this tools is here to test the filter section of a pipeline.
- The script natively supports tcp/udp + syslog inputs
- If you want to other types of input you need to send the data directly to logstash and also expose the corresponding port
- Each output section needs to be defined as stated below
How to start
- If needed, install docker on your machine
- Clone the repository on GitHub.
- Copy your pipeline folder to
logstash/logstash-config/pipeline - Replace the output section in each pipeline with the tools predefined output:
output { http { format => "json" http_method => "post" url => "http://pipeline-ui:8080/api/v1/receiveLogstashOutput" } } - Enter the repository directory
- Run
docker-compose build - Run
docker-compose up - Open up http://localhost:8080 in your browser to access the interface
- It will directly attempt to connect to the backend and logstash. Wait until both status buttons are green before starting to send data.
IMPORTANT – Depending on docker host you might get the question if you want to share your drive with the docker service. If you get this message, share your drive, stop the containers and run docker-compose up again.
Adding pipelines
There are two ways, either modify the existing pipelines or follow thes following steps.
- Modify
logstash-config/pipelines.yml
# Example covers the creation of a pipeline called mypipeline
- pipeline.id: generic-json
path.config: "/usr/share/logstash/pipeline/mypipeline"- Then create a directory in
logstash-config/pipelinecalledmypipeline - Copy your logstash config to
logstash-config/pipeline/mypipeline - Last but not least, restart the containers using
docker-compose up
Modifying pipelines
Since the logstash pipeline directory is mounted in the containers logstash should detect file changes and reload the pipeline. If this does not happen for some reason you need to restart the containers.
Troubleshooting
Check the docker-compose window for any exceptions thrown.
Reporting issues
First, please check if there’s any current issues that matches your problem. If not, please feel free to submit an issue here at Github.
I have very limited time so I won’t be able to act fast on any issue but it’s always good to have it logged and who knows, maybe someone else will pick it up and make a PR.
Application flow diagram
Screenshot from the application
