If you’re using the F5 GeoIP database in your environment it’s important to keep it up to date. Public IP networks moves around quite often and the older database your’re using, the higher the likelihoor of you might ending up blocking connections that you did not intend to block, or the other way around.
To solve this I wrote a package around downloading files from F5. This article covers how it was written and how to use it.
Table of Contents
TLDR;
Don’t want to know the inner workings or how to use the script outside just downloading GeoIP databases? Just head on over to Github and read the README.md for the short track.
F5 Downloads library
This one uses requests-html, an extension of the popular requests library which provides ways to select stuff based on CSS selectors. Pretty neat.
First thing we need is to create a requests session in order to store the session cookies that is required to access authenticated resources at downloads.f5.com.
# Create the session
self._session = HTMLSession()
# Simulate a form submission by sending a POST
# to the login api at F5 with the credentials
self._session.post('https://api-u.f5.com/auth/pub/sso/login/user',
headers={'Content-Type': 'application/x-www-form-urlencoded'},
data={
'userid': self.username,
'passwd': self.password,
}
)When we have the session we can just use session.get(<url>) in order to access resources that requires authentication.
Next step was to find a dynamic way to iterate the pages on the downloads portal. To accomplish this is used the fact that python is a functional language together with the power of the dynamic **kwargs.
Take the download_geoipdb function for example:
self.follow_path(
self.get_page('https://downloads.f5.com/esd/productlines.jsp'),
[
{
'f': self.follow_specific_link,
'args': {'pattern': rf'BIG-IP v{version}.x.+'},
}, {
'f': self.follow_specific_link,
'args': {'pattern': r'GeoLocationUpdates', }
},
{
'f': self.download_files,
'args': {'pattern': rf'^ip-geolocation-.+\.zip$',
'download_folder': f'./downloads/GeoIP/v{version}/', 'cb': cb}
}
]
)
What this function is to call follow_path, a function with takes a page object, and, a bunch of steps where each step contains a function and it’s parameters. The follow_path function will then call the function of each step using the parameters in arg together with the current page.
In this case it means that the script will fetch https://downloads.f5.com/esd/productlines.jsp to list the top level products, then call follow_specific_link with the kwargs:
{
'pattern': rf'BIG-IP v{version}.x.+',
'page': <result from https://downloads.f5.com/esd/productlines.jsp>
}The result page from the link matching the pattern will then be fed to the next step, where follow_specific_link will match the pattern GeoLocationUpdates against the result page to find the next link.
Finally, download files will be called where it will find a download link matching ^ip-geolocation-.+\.zip$ and then proceed to download the file from the default destination. First file to download if the md5 file and then, if the geoip database file already exists it will run an MD5 comparison against the local file. If they match it will not download the database again, otherwise it will download the database and run an MD5 comparison again before marking the download job as successful.
To summarise:
- Login to downloads.f5.com.
- For each step in the chain supplied to follow_path, call a function that matches against a pattern and either follows a link or downloads a file.
- Verifies that the md5 file matches the local version.
The REST Client
The REST client has functions to get an auth token from the F5 device, upload files, and run bash commands. The other functions builds on these building blocks in order to test if the latest geoip database is used, upload a file and install if it is not.
The flow used in the script is as follows
- update_geoip_db is called with the path to the latest geoip database file.
- The script gets the version currently running on the device and compares it to the provided file.
- If a legitimate version has been detected both locally and on the device and they do not match the script will proceed with the update functions. Otherwise it will stop processing since either the input file is not legit, or the latest file is already installed on the device.
- Else it will run a bash command to test if the update script that updates the geoip databases exists on the device and upload it if it does not. The script will be saved in /var/tmp/ in order to persist between version upgrades.
- Then it will upload the geoip database and the corresponding md5.
- The update script that was uploaded in the previous step will be executed and it will proceed to validate the md5 against the database, install it and then delete the temporary folder where the database and md5 were.
- Finally, a version check against the latest database and the installed one is executed to ensure that the update was successful. If it was not the device is marked as failed.
Slack integration
If enabling Slack and using main.py for updating your databases you’ll get an update if something has failed, or if something has been updated on the devices.
Using it is easy.
slack = SlackReport('https://slackwebhookurl...')
slack.up_to_date.append(device_name) # Add device that was up-to-date
slack.failed.append(device_name) # Add a failed device
slack.updated.append(device_name) # Add a device that was updated
slack.send_webhook() # Send the reportThe parameter device_name is a string so if you want to add some error message you can just concatenate it to the string.
Potential follow-ups
Boy, that was a long article huh? Congratulations for reaching the end and shame on you if you skipped. 😀
Anyway, potential things to do here that I can think of is to add:
- More report transports
- Remote logging
- Docker manifest
- Kubernetes manifests
With remote logging you could alert if the script has not sent any logs for X hours and then not worry that there’s a crashed server/container somewhere that is not working.