Oxidized – Part 1

It’s time to do a refresh of the backup strategy and we since we already used LibreNMS we figured that Oxidized could be a good solution since it can source the device list in LibreNMS and also allows LibreNMS to display the configuration and also sends the configurations to a Git repository.

This first part will only cover setting up Oxidized as a docker container, sourcing the devices via CSV and integrating with GitHub and will try to highlight some of the pitfalls which I encountered. In order to not having to troubleshoot both LibreNMS and Oxidized I recommend starting here too. The CSV source does not take that much time anyway and you’ll probably save time in the long run.

Please note that this is a mind dump by myself after configuring Oxidized for the first time. I am not an expert in any way and what’s written below should not be considered canon. Should you encounter any flaws in this post, please do comment so I can update.

Credits to this Axians blog post. It was very helpful for me.

Preparing a directory

This is up to each and everyone, but this guide will use /opt/oxidized as an example. Start by creating the directory.

mkdir -p /opt/oxidized

This is where all the persistent files will be hosted when running the container.

Create a template config

Since this can change I’d advice against copying this part from the guide. Create a default configuration by running:

docker run --rm -v /opt/oxidized:/root/.config/oxidized -p 8888:8888/tcp -t oxidized/oxidized:latest oxidized

Tip: Lock your dependencies by referring to a hash instead of using :latest to avoid nasty surprises.

Create an SSH Pair

This is one of the steps where I ran into issues and it took a bit too long for me to figure out what the issue was. Turns out the Rugged SSH library used by Oxidized only accepts SSH keys in PEM format.

# Create an SSH directory and enter it
mkdir /opt/oxidized/.ssh
cd /opt/oxidized/.ssh
# Generate a key-pair
ssh-keygen -t rsa -b 2048 -f id_rsa -C 'id_rsa'
# Convert the key to PEM (base64) format
openssl rsa -in id_rsa -outform PEM -out id_rsa.pem
# Protect the keys
chmod 600 

Don’t remove the key passphrase, that’s bad practice.

Create the router.db CSV file

This step highly depends on your needs. In our case the goal is to just get Oxidized up and running so we’ll use a CSV source.

Below is an example with some comments for things I figured was relevant

---
username: xx            # Default username
password: xx            # Default password
resolve_dns: true
interval: 7200
use_syslog: false
debug: false
threads: 30
timeout: 20
retries: 3
log: /root/.config/oxidized/log
prompt: !ruby/regexp /^([\w.@-]+[#>]\s?)$/
rest: 127.0.0.1:8888
vars:
  auth_methods: ["none", "password", "public_key", "keyboard-interactive"] # The ways that oxidized will try when authenticating
  remove_secret: true
  ssh_keys: "/root/.config/oxidized/.ssh/id_rsa.pem" #For Git Auth
models: {}
pid: "/root/.config/oxidized/pid"
crash:
  directory: "/root/.config/oxidized/crashes"
  hostnames: false
stats:
  history_size: 10
input:
  default: ssh, telnet
  debug: false
  ssh:
    secure: false
  utf8_encoded: true
output:
  default: git
  file:
    directory: "/root/.config/oxidized/gitrepo"
  git:
    single_repo: true
    user: Oxidized
    email: oxidized@domain.com
    repo: "/root/.config/oxidized/gitrepo/"
hooks:
  push_to_remote:
    type: githubrepo
    events: [post_store]
    remote_repo: "git@github.com:my-repo/oxidized.git"
    publickey: "/root/.config/oxidized/.ssh/id_rsa.pub"
    privatekey: "/root/.config/oxidized/.ssh/id_rsa.pem"
source:
  default: csv
  csv:
    file: "/root/.config/oxidized/router.db"
    delimiter: !ruby/regexp /:/
    map:
      name: 0
      model: 1
    gpg: false
model_map: # Needed to map model to OS
  juniper: junos
  cisco: ios
  f5: tmos
  arista: eos
  fortigate: fortios
  

Creating the router.db file

The router.db file contains the devices that you want oxidized to take backups of. I’d recommend just adding one or two devices from a brand that you are the most proficient in to make troubleshooting easier if there’s issues.

Personally I picked F5 so here’s an example file with two load balancers:

# /root/.config/oxidized/router.db
# name:model:ip
XX-LB-01:f5:10.0.0.1
XX-LB-02:f5:10.0.0.1

Note how the model in the CSV above is mapped to tmos in the model_map of the oxidized config.

Build and run

docker build -t myoxidized:latest .
# ... building info removed for brevity ...
docker run --restart=always -e 'OXIDIZED_SSH_PASSPHRASE=key_passphrase' -v /opt/oxidized:/root/.config/oxidized -v /opt/oxidized/.ssh:/root/.config/oxidized/.ssh -p 8888:8888/tcp myoxidized:latest

What this does is:

• Expose the the oxidized config folder and .ssh folder to the container
• Set an environment variable named OXIDIZED_SSH_PASSPHRASE which is used by oxidized to decrypt the private deploy key
• Expose port 8888

If all worked as it should (but it almost never does in IT right? :)) you should now see Oxidized fetching the configuration from your devices and push it to the github repository.

Troubleshooting

This was not exactly smooth sailing for me either so I did a bit of troubleshooting and a lot of googling. Dumping a bit of helpful things that might help you

Debug is your friend

Check your config file and turn on the debug flags. Then check the container logs for any solvable errors.

Oxidized user access

1. Does the oxidized user have sufficient access to the device?
2. When logging in, is an enable password needed?
3. Check the model notes in the Oxidized GitHub repo for additional quirks:
   https://github.com/ytti/oxidized/tree/master/docs/Model-Notes

Check the git folder

Does the git folder have content? Oxidized is not showing the repository like you’d see when cloning a public repository instead you’ll see only the content from a .git folder there. Anyway, if you can see content here you should probably verify these things:

1. Is your key really in PEM format?
2. Have you added the public key (ie. id_rsa.pub) to the deploy keys of your repository?
3. Did enable write permissions for the deploy key?
4. Can the oxidized server reach the Github servers?

Runsv problems (unable to start)

At the time of writing the I had some issues with starting the runinit scripts. Some executables used by the Python service wrapper in the container was lacking executable flags so it kept spewing out lines like there:

runsv update-ca-certificates: fatal: unable to start ./run: access denied

After some research on how this worked it was reveled that the service wrapper is entering a folder in which is expects a file named run. This file has to have executable permissions, but it did not have this in the container.

So I created a Docker file in /opt/oxidized/Dockerfile to work-around the problems.

#/opt/oxidized/Dockerfile
FROM oxidized/oxidized@sha256:77225fe645aea44a4ffccbe159100afafb4b3130bbe5eadd05bcadf7b411ba0d
RUN chmod +x /etc/service/auto-reload-config/run
RUN chmod +x /etc/service/oxidized/run
RUN chmod +x /etc/service/update-ca-certificates/run

I tried to reproduce this issue in my lab to make a PR but was not able to reproduce it. Leaving the fix here in case someone else has issues.

Logrotate

Below is a sample logrotate config that will rotate the log file daily and keep 14 days of logs. To use it, create a file called /etc/logrotate.d/oxidized and put the following content in it:

/opt/oxidized/log {
    daily
    rotate 14
    size 10M
    compress
    delaycompress
}

Next step

The next post will cover how to configure LibreNMS to automatically add devices to the backup list of oxidized.

Part 2 is available here