[{"data":1,"prerenderedAt":216},["ShallowReactive",2],{"project-29479258":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":28,"sentiment":29,"stance_mix":30,"scored_comments":34,"aspects":35,"strip":39,"post_text":42,"what":43,"said":66,"maker_in_thread":122,"shape":133,"grid":136,"attempts":177,"same_week":178,"maker_shows":207,"maker_hidden":208,"reader_version":209,"card":210},"29479258","git-history","git-history, for analyzing scraped data collected using Git and SQLite","title","Show HN: git-history, for analyzing scraped data collected using Git and SQLite","https:\u002F\u002Fsimonwillison.net\u002F2021\u002FDec\u002F7\u002Fgit-history\u002F","simonwillison.net","simonw",1638917844,2021,108,19,490,2533,20,"notable","data","git data analysis","cli",[24,25,26,27],"git","sqlite","scraped data","data analysis","mixed",0.316,{"positive":31,"neutral":32,"mixed":33,"critical":33},4,2,1,8,{"idea":36,"performance":38},{"positive":31,"critical":37},0,{"positive":37,"critical":33},[37,37,37,37,37,37,37,40,41,37,37,37,40,40,40,41,37,41,33],5,3,null,{"problem":44,"audience":42,"pricing":42,"motivation":42,"time_to_build":42,"stack":50,"openness":65,"stage":65,"team_size":65},{"value":45,"source":46,"evidence":47},"Analyzing data scraped over time and stored in Git history","maker",[48],{"id":49},"post",[51,59],{"value":52,"source":53,"evidence":54},"Python","commenter",[55],{"id":56,"by":57,"text":58},"29480222","faizshah","Side note this is a useful pattern I use in general. Use python to get some data into SQLite then I either use fast api, datasette, or Streamlit depending on what I want to do…",{"value":60,"source":46,"evidence":61},"SQLite",[62],{"id":63,"by":11,"text":64},"29481066","That's why I'm using it to build a SQLite database. I've run the script against ~20,000 commits and it took an hour to run... but then on subsequent runs when there are only a…","unknown",{"praise":67,"objections":83,"requests":95,"questions":96,"alternatives":100},[68],{"value":69,"source":53,"evidence":70},"Useful, interesting technique for scraping data",[71,75,79],{"id":72,"by":73,"text":74},"29480024","dasickis","Fascinating! Going to use this in one of my projects.",{"id":76,"by":77,"text":78},"29480090","pdimitar","I'm likely to use this to generate HTML \u002F CSS for all previous versions of blog posts. Very nice work!",{"id":80,"by":81,"text":82},"29480795","a-dub","built some hand tools for analyzing the covid data git repos in this way earlier in the pandemic. it was interesting in that it gave visibility into back corrections for old…",[84],{"value":85,"source":53,"evidence":86},"Git is slow and bloats with JSON data; delta encoding suffers",[87,91],{"id":88,"by":89,"text":90},"29482971","cerved","It's unlikely to be the most performant since git is first and foremost designed to version control plain text source code for developing software. This is version controlling…",{"id":92,"by":93,"text":94},"29480799","riskneutral","Would using Git for this be performant?",[],[97],{"value":94,"source":53,"evidence":98},[99],{"id":92,"by":93,"text":94},[101,108,115],{"value":102,"source":53,"evidence":103},"Dolt",[104],{"id":105,"by":106,"text":107},"29482761","harabat","People who find this interesting might also like DoltHub: \"Dolt marries two familiar concepts, Git and MySQL. The first and only SQL database that supports clone, branch, and…",{"value":109,"source":53,"evidence":110},"Apache Parquet with Arrow",[111],{"id":112,"by":113,"text":114},"29482326","awill88","I also do this, but be weary of situations where you’re using ephemeral containers and you have to download, scrape, then reupload a bare repo.. Sure, you can use GitHub or…",{"value":116,"source":53,"evidence":117},"Pandas",[118],{"id":119,"by":120,"text":121},"29480490","omegalulw","Why not use Pandas?",{"comments":31,"answered":32,"key":123},[124,127,130],{"id":125,"text":126},"29480381","Yeah SQLite is such a powerful intermediary file format, I use it like that all the time.",{"id":128,"text":129},"29480593","You can use Pandas with SQLite - Pandas saves to and loads from SQLite databases out of the box.",{"id":131,"text":132},"29479262","Also summarized in this Twitter thread: https:\u002F\u002Ftwitter.com\u002Fsimonw\u002Fstatus\u002F1468348488379367426",{"commenters":134,"max_depth":41,"maker_replies":31,"first_comment_mins":33,"lifespan_hours":135},14,106,[137,139,140,142,143,146,148,150,154,156,159,161,162,164,165,167,170,172,175],[56,37,37,138,57,37],"other\u002Fneutral\u002Fidea",[125,37,37,138,11,33],[119,37,37,141,120,37],"commenter\u002Fneutral\u002Fnone",[128,37,37,141,11,33],[144,37,37,141,145,37],"29481094","bfung",[147,37,37,138,57,37],"29481526",[131,33,37,149,11,33],"project\u002Fneutral\u002Fdocs",[151,33,40,152,153,37],"29479957","maker\u002Fpositive\u002Fidea","bluefox",[112,32,41,155,113,37],"project\u002Fmixed\u002Fidea",[157,32,37,141,158,37],"29482427","ZeroGravitas",[160,32,37,138,113,37],"29528214",[105,41,37,138,106,37],[76,31,40,163,77,37],"project\u002Fpositive\u002Fidea",[80,40,40,163,81,37],[72,166,40,163,73,37],6,[92,168,41,169,93,37],7,"project\u002Fneutral\u002Fperformance",[63,168,37,171,11,33],"project\u002Fmixed\u002Fperformance",[173,168,41,169,174,37],"29481917","CyberShadow",[88,168,33,176,89,37],"project\u002Fcritical\u002Fperformance",[],[179,184,189,193,198,203],{"id":180,"name":181,"points":182,"category":183},"29461735","Modern Wiki",979,"productivity",{"id":185,"name":186,"points":187,"category":188},"29425735","Emoji to Scale",713,"learning_science",{"id":190,"name":191,"points":192,"category":188},"29488511","Scale of the Universe 2",276,{"id":194,"name":195,"points":196,"category":197},"29428910","Deep Dreams",237,"ai",{"id":199,"name":200,"points":201,"category":202},"29464889","Straw.page",210,"web_saas",{"id":204,"name":205,"points":206,"category":188},"29437527","UI\u002FUX Tips book",198,12,false,"v2",{"id":211,"name":5,"line":6,"ai":208,"cat":20,"year":13,"points":14,"comments":15,"rec":28,"s":212,"top":18,"kw":213,"maker":11,"domain":10,"tier":19,"attempts":37,"strip":214,"url":9,"mix":215,"praise":69,"objection":85,"sub":21},29479258,0.32,[24,25,26,27],"0000000530005553031",[31,32,33,33],1794036614015]