Introducing Tailparse
Tags:
November 1st, 2022

Have you ever wanted to query a log file without mucking with some Frankenstein combination of Unix tools?

Cause I have. So I made Tailparse.

Tailparse is a simple command line tool that lets you query log files as though they were SQL. It's based on an older freeware tool by Microsoft that is limited to IIS logs.

SQL has become the universal language for querying so why not extend it to computer logs? Something below is easy to much easier grok and uses a language you're likely already familiar with.

SELECT COUNT(DISTINCT ip) 
FROM logs 
WHERE date_time_local > datetime("2022-10-01") 
  AND date_time_local <= datetime("2022-10-31")

This lets you get crude metrics in place of something like Google Analytics.

I was using things like AWK for this. However, order isn't always a good guarantee which makes anything time dependent a chore. Dumping it into SQLite and parsing the time columns solves this.

Tools like Spelunk, Logstash, or Kibana made querying logs much easier. They created a slick interface for access. Their application models meant you can collate a bunch of different logs in one place to cross-check issues across multiple servers.

But a lot of people don't need something that overkill. It seemed ridiculous to me that I'd need to set it all up just to do checks against my own projects.

Feel free to check out the GitHub and make suggestions or ideas. In particular, I want to add more log formats, go-to sample queries, and allow for multiple logs as tables at some point.