[{"data":1,"prerenderedAt":283},["ShallowReactive",2],{"project-30417411":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":28,"sentiment":29,"stance_mix":30,"scored_comments":34,"aspects":35,"strip":40,"post_text":42,"what":43,"said":79,"maker_in_thread":153,"shape":163,"grid":167,"attempts":244,"same_week":245,"maker_shows":274,"maker_hidden":275,"reader_version":276,"card":277},"30417411","google-drive-to-sqlite","Google Drive to SQLite","title","Show HN: Google Drive to SQLite","https:\u002F\u002Fsimonwillison.net\u002F2022\u002FFeb\u002F20\u002Fgoogle-drive-to-sqlite\u002F","simonwillison.net","simonw",1645459939,2022,305,32,136,2570,6,"legend","data","database export tool","cli",[24,25,26,27],"google drive","sqlite","file metadata","google drive api","loved",0.295,{"positive":18,"neutral":31,"mixed":32,"critical":33},7,0,1,14,{"execution":36,"idea":37,"docs":39},{"positive":32,"critical":33},{"positive":38,"critical":32},3,{"positive":32,"critical":32},[33,32,32,32,32,32,32,38,32,38,41,38,32,41,32,41,38,32,32,38,38,32,32,38,32,41,32,32,32,32,41,41],5,"Here's the repo: https:\u002F\u002Fgithub.com\u002Fsimonw\u002Fgoogle-drive-to-sqlite\n\nThe README is using a trick I'm increasingly leaning on: parts of that document - the --help output and the example database schema - are automatically generated using Cog: https:\u002F\u002Fnedbatchelder.com\u002Fcode\u002Fcog and https:\u002F\u002Ftil.simonwillison.net\u002Fpython\u002Fcog-to-update-help-in-readme",{"problem":44,"audience":52,"pricing":52,"motivation":53,"time_to_build":52,"stack":60,"openness":77,"stage":78,"team_size":78},{"value":45,"source":46,"evidence":47},"Understanding the size and scope of files in Google Drive by querying their metadata.","commenter",[48],{"id":49,"by":50,"text":51},"30419589","NicoJuicy","> Pulling the metadata—file names, sizes, file types, file owners, creation dates—into a SQLite database felt like a great way to start understanding the size and scope of what…",null,{"value":54,"source":55,"evidence":56},"Maker had about half a TB of files in a shared Google Drive folder and wanted to understand it.","maker",[57],{"id":58,"by":11,"text":59},"30419617","In my case the Google Drive folder that was shared with me has about half a TB in it, so going via the API seemed like a better option!",[61,68,73],{"value":62,"source":63,"evidence":64},"Python","inferred",[65],{"id":66,"by":11,"text":67},"30417835","To regenerate the markdown file I run this command in the console: ``` cog -r README.md ``` Presumably most IDEs have q mechanism that can run that command automatically, either…",{"value":69,"source":55,"evidence":70},"SQLite",[71],{"id":72},"post",{"value":74,"source":55,"evidence":75},"Cog",[76],{"id":72},"open_source","unknown",{"praise":80,"objections":103,"requests":121,"questions":129,"alternatives":137},[81,92],{"value":82,"source":46,"evidence":83},"Datasette and related tools are a pleasure to use",[84,88],{"id":85,"by":86,"text":87},"30427622","20after4","Datasette and the various companion tools have all been a pleasure to work with. I've recently been using it to build my personal info archive as well as building a data analysis…",{"id":89,"by":90,"text":91},"30418100","rbosinger","I dived into Datasette a bit last year. Simon has a whole set of tools around SQLite and data analysis with it. I haven't used any of it for anything major but I love to see what…",{"value":93,"source":46,"evidence":94},"Useful SQLite-based design idea",[95,99],{"id":96,"by":97,"text":98},"30418068","anonymouse008","TELL HN: More things like this please -- though I have no real use for this tool directly, it's paradigms are super helpful in thinking through other projects I'm working on.…",{"id":100,"by":101,"text":102},"30419721","mixcocam","Agreed. More uses of SQLite too! I have been working with it to gather ecom sales data across platforms for a couple of months. Such a powerful tool! I want to do more with it!",[104,114],{"value":105,"source":46,"evidence":106},"Google is deprecating the OOB OAuth flow, complicating CLI authentication",[107,111],{"id":108,"by":109,"text":110},"30417735","nickcw","We discovered (in rclone) that Google are deprecating the auth method you are using - the OOB auth here:…",{"id":112,"by":11,"text":113},"30417881","It's pretty hard to understand what I'm meant to do for a CLI app instead here. Are they removing all versions of the \"copy and paste this code\" auth flow? Do I need to spin up a…",{"value":115,"source":46,"evidence":116},"Syncing locally and indexing is slow or rate limited for large Drives",[117],{"id":118,"by":119,"text":120},"30420624","nijave","If you have a lots of files in Google Drive, this can take a significant amount of time or you may not be able to sync it locally. I push incremental backups & disk images…",[122],{"value":123,"source":46,"evidence":124},"Alfred or Spotlight integration for querying the index",[125],{"id":126,"by":127,"text":128},"30417878","tonymet","An Alfred workflow that queries this index would be fantastic. Alternatively feed it into spotlight",[130],{"value":131,"source":46,"evidence":132},"How does the Cog README generation work with IDEs",[133],{"id":134,"by":135,"text":136},"30417790","samhw","> The README is using a trick I'm increasingly leaning on: parts of that document - the --help output and the example database schema - are automatically generated using Cog How…",[138,145,149],{"value":139,"source":46,"evidence":140},"gTakeout (Google Takeout)",[141],{"id":142,"by":143,"text":144},"30424375","smcnally","gTakeout[0] also includes this file meta and allows incremental downloads. Ingesting from a local Takeout archive as an additional source for your tool mitigates some of this…",{"value":146,"source":46,"evidence":147},"voidtools Everything search with local sync",[148],{"id":49,"by":50,"text":51},{"value":150,"source":46,"evidence":151},"rclone",[152],{"id":108,"by":109,"text":110},{"comments":31,"answered":154,"key":155},2,[156,159,160],{"id":157,"text":158},"30417763","Hah, wow thanks I hadn't seen that! Just my luck to implement that the same week they announce they are deprecating it.",{"id":112,"text":113},{"id":161,"text":162},"30418034","I'm collecting notes on how to address this here: https:\u002F\u002Fgithub.com\u002Fsimonw\u002Fgoogle-drive-to-sqlite\u002Fissues\u002F39 It looks like the localhost redirects are being disabled for \"iOS,…",{"commenters":164,"max_depth":41,"maker_replies":31,"first_comment_mins":165,"lifespan_hours":166},22,28,54,[168,170,172,173,175,179,183,186,188,191,193,195,198,201,202,204,206,208,209,212,213,215,217,221,224,227,229,232,235,239,241,243],[108,32,33,169,109,32],"project\u002Fcritical\u002Fexecution",[157,32,32,171,11,33],"maker\u002Fneutral\u002Fnone",[112,32,32,169,11,33],[161,32,32,174,11,33],"maker\u002Fneutral\u002Fexecution",[176,32,32,177,178,32],"30418448","other\u002Fneutral\u002Fnone","mey",[180,32,32,181,182,32],"30421013","other\u002Fcritical\u002Fnone","aaaaaaaaata",[184,32,32,177,185,32],"30433708","sjelkjd",[187,32,38,174,109,32],"30418465",[189,32,32,177,190,32],"30422946","ljm",[142,32,38,192,143,32],"project\u002Fneutral\u002Fidea",[89,33,41,194,90,32],"maker\u002Fpositive\u002Fnone",[196,154,38,192,197,32],"30418571","gwbas1c",[199,154,32,200,86,32],"30427492","commenter\u002Fneutral\u002Fidea",[85,38,41,194,86,32],[203,38,32,194,11,33],"30435808",[205,38,41,194,86,32],"30447119",[49,207,38,192,50,32],4,[58,207,32,192,11,33],[210,207,32,211,50,32],"30419868","commenter\u002Fneutral\u002Fnone",[118,207,38,192,119,32],[134,41,38,214,135,32],"project\u002Fneutral\u002Fdocs",[66,41,32,216,11,33],"maker\u002Fneutral\u002Fdocs",[218,41,32,219,220,32],"30420179","other\u002Fneutral\u002Fdocs","ash",[222,41,38,214,223,32],"30417854","whateveracct",[225,41,32,219,226,32],"30420080","_hl_",[126,18,41,228,127,32],"project\u002Fpositive\u002Fidea",[230,18,32,177,231,32],"30418021","bjtitus",[233,31,32,177,234,32],"30422321","RachelF",[236,237,32,177,238,32],"30424410",8,"soheil",[240,237,32,171,11,33],"30424445",[96,242,41,228,97,32],9,[100,242,41,228,101,32],[],[246,250,255,260,264,269],{"id":247,"name":248,"points":249,"category":20},"30399881","Judy Records",698,{"id":251,"name":252,"points":253,"category":254},"30408434","Rotator",659,"games",{"id":256,"name":257,"points":258,"category":259},"30440828","Simula One",463,"hardware",{"id":261,"name":262,"points":263,"category":259},"30446960","endless-sphere.com",304,{"id":265,"name":266,"points":267,"category":268},"30440275","Supernotes 2",251,"productivity",{"id":270,"name":271,"points":272,"category":273},"30378562","curl-impersonate",249,"dev_tools",12,false,"v2",{"id":278,"name":5,"line":6,"ai":275,"cat":20,"year":13,"points":14,"comments":15,"rec":28,"s":279,"top":18,"kw":280,"maker":11,"domain":10,"tier":19,"attempts":32,"strip":281,"url":9,"mix":282,"praise":82,"objection":105,"sub":21},30417411,0.3,[24,25,26,27],"10000003035305053003300305000055",[18,31,32,33],1794036617240]