automatic_log_collector_and_analyzer
Infrastructure
log collection and analysis
SHOW HN2023
Legend
automatic_log_collector_and_analyzer
My Single-File Python Script I Used to Replace Splunk in My Startup
github.com#log analysis#log collection#splunk alternative#sqlite logs#datasette
How HN took it
7 positive, 2 neutral or asking, 1 mixed, 10 critical, out of 20 comments about the project. Thread score -0.06.
Idea: 4 for, 3 againstExecution: 1 for, 3 againstDocs: 0 for, 1 againstDesign: 0 for, 1 againstPerformance: 0 for, 1 against
The thread, comment by comment
Each row is a top-level comment and its replies, in HN's order. Select a cell to open that comment on HN.
positive neutral or mixed critical about another commenter or something else the maker
What HN said
What people liked
Simple, practical tool that suits small-scale needs
From 2 comments
"This simple tool solves X at my org" is probably the most underrated type of project. There's not enough room to overcomplicate something that isn't a core part of the business,…
rollcat on HNIt's not that much code and it has sensible function names. I appreciate that OP took the time to share his tool with us.
facorreia on HN
Using SQLite and Datasette is a good approach
From 2 comments
Your project sounds like something that definitely could come in handy. I forked it as a "bookmark". I particularly like the idea of storing the data in a local SQLite database.…
facorreia on HNI love this! Log analysis isn't one of the core use-cases for Datasette, but I've done my own experiments with it that have worked pretty well - anything up to 10GB or so of data…
simonw on HN
Thanks for sharing
From 2 comments
Thanks for sharing.
rgrieselhuber on HN"Hell hath no fury like a Python dev annoyed" :-) Thank you for sharing!
surfingdino on HN
Objections
Does not scale to large enterprise log volumes
From 2 comments
Volume. 1GB of data per day is rounding error. If you have tens of thousands of servers, each generating hundreds of gigabytes of data per day, tail -f and grep don't scale…
dsXLII on HNI misspoke there-- meant to say: "The application has been tested with log files several gigabytes in size from dozens of machines and can process all of it in minutes." That's…
eigenvalue on HN
Pull-based collection over SSH is fragile compared to shipping logs from each machine
From 1 comment
Fetching logs regularly sounds hard? Wouldn't you need to keep track of the position of all files, with heuristics around file rotations? And if something catastrophic happens,…
xorcist on HN
Code lacks comments and has unused functions and globals
From 2 comments
Quickly skimming some points that would irratate me if I had to maintain this script: * Importing Paramiko but regularly call `ssh` via subprocess * Unused functions like…
oblvious-earth on HN* barely any comments and not a single docstring in the entire kiloline file
msto on HN
Requires Redis despite being a single script
From 1 comment
So it requires redis. Would have loved if it was just a simple script or binary.
prabhatsharma on HN
Questions
How does it handle log file rotation and position tracking?
From 1 comment
Fetching logs regularly sounds hard? Wouldn't you need to keep track of the position of all files, with heuristics around file rotations? And if something catastrophic happens,…
xorcist on HN
Alternatives named
Graylog
From 1 comment
Neat! Definitely a better solution for single source logs. Splunk is ridiculous and Cisco acquiring it isn't going to make that better. For others with a bit more complex needs,…
runjake on HN
Grafana with Graphite
From 1 comment
I used to work at a splunk shop. It was used for alerting, graphing & prediction. It was critical to how the company functioned. There was lots of stuff that relied on splunk,…
KaiserPro on HN
Datadog
From 1 comment
A long long time ago, I used a series of tail -f's and unix pipes to aggregate logs, and grep, less and awk to analyse them. There were about 20 different services written in…
hliyan on HN
Datasette with sqlite-utils
From 1 comment
I love this! Log analysis isn't one of the core use-cases for Datasette, but I've done my own experiments with it that have worked pretty well - anything up to 10GB or so of data…
simonw on HN