git-history

Data

git data analysis

SHOW HN2021

git-history

git-history, for analyzing scraped data collected using Git and SQLite

simonwillison.net

#git#sqlite#scraped data#data analysis

108points
19comments
#490of 2,533 in 2021
Top 20%of 2021 by points

How HN took it

MixedNo clear lean

4 positive, 2 neutral or asking, 1 mixed, 1 critical, out of 8 comments about the project. Thread score +0.32.

Idea: 4 for, 0 againstPerformance: 0 for, 1 against

The thread, comment by comment

Each row is a top-level comment and its replies, in HN's order. Select a cell to open that comment on HN.

positive neutral or mixed critical about another commenter or something else the maker

What HN said

What people liked

  • Useful, interesting technique for scraping data

    From 3 comments
    • Fascinating! Going to use this in one of my projects.
      dasickis on HN
    • I'm likely to use this to generate HTML / CSS for all previous versions of blog posts. Very nice work!
      pdimitar on HN
    • built some hand tools for analyzing the covid data git repos in this way earlier in the pandemic. it was interesting in that it gave visibility into back corrections for old…
      a-dub on HN

Objections

  • Git is slow and bloats with JSON data; delta encoding suffers

    From 2 comments
    • It's unlikely to be the most performant since git is first and foremost designed to version control plain text source code for developing software. This is version controlling…
      cerved on HN
    • Would using Git for this be performant?
      riskneutral on HN

Questions

  • Would using Git for this be performant?

    From 1 comment

Alternatives named

  • Dolt

    From 1 comment
    • People who find this interesting might also like DoltHub: "Dolt marries two familiar concepts, Git and MySQL. The first and only SQL database that supports clone, branch, and…
      harabat on HN
  • Apache Parquet with Arrow

    From 1 comment
    • I also do this, but be weary of situations where you’re using ephemeral containers and you have to download, scrape, then reupload a bare repo.. Sure, you can use GitHub or…
      awill88 on HN
  • Pandas

    From 1 comment

Category, keywords, summaries and labels on this page were made by AI (Thread Reader v2) and not checked by hand. Each point links to the comments it came from.

Add a badge to your READMEShown on HackerShow
[![Shown on HackerShow](https://hackershow.org/badge.svg)](https://hackershow.org/show/29479258)