[{"data":1,"prerenderedAt":398},["ShallowReactive",2],{"project-38057759":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":28,"sentiment":29,"stance_mix":30,"scored_comments":34,"aspects":35,"strip":42,"post_text":45,"what":46,"said":90,"maker_in_thread":201,"shape":209,"grid":213,"attempts":360,"same_week":361,"maker_shows":269,"maker_hidden":390,"reader_version":391,"card":392},"38057759","Open Australian Legal Corpus","how I built the largest open database of Australian law","title","Show HN: how I built the largest open database of Australian law","https:\u002F\u002Fumarbutler.com\u002Fhow-i-built-the-largest-open-database-of-australian-law\u002F","umarbutler.com","ubutler",1698581218,2023,172,61,349,3109,12,"notable","data","legal dataset","dataset",[24,21,25,26,27],"australian law","llm training data","open data","legal corpus","loved",0.387,{"positive":31,"neutral":31,"mixed":32,"critical":33},15,0,2,32,{"docs":36,"idea":38,"execution":40,"design":41},{"positive":37,"critical":32},1,{"positive":39,"critical":37},7,{"positive":32,"critical":37},{"positive":32,"critical":32},[32,43,32,43,44,44,44,37,32,44,32,43,44,43,43,43,32,44,44,32,32,32,44,44,44,32,43,43,43,32,44,43,32,44,43,44,32,32,32,43],5,3,null,{"problem":47,"audience":57,"pricing":45,"motivation":62,"time_to_build":66,"stack":70,"openness":87,"stage":88,"team_size":89},{"value":48,"source":49,"evidence":50},"No open, freely reusable databases of Australian law existed for training LLMs","maker",[51,54],{"id":52,"by":11,"text":53},"38057760","Hey HN, Over the past year, I’ve been working on building the Open Australian Legal Corpus, the largest open database of Australian law. I started this project when I realised…",{"id":55,"by":11,"text":56},"38064732","So AustLII and Jade are what I was referring to when I say \"While there were certainly a few free-to-access legal databases, none were truly open ...\". They are free but not open…",{"value":58,"source":49,"evidence":59},"People wanting raw legal text for ML models or legal problems",[60,61],{"id":52,"by":11,"text":53},{"id":55,"by":11,"text":56},{"value":63,"source":49,"evidence":64},"Wanted to train an LLM on legal problems and found no open Australian law data",[65],{"id":52,"by":11,"text":53},{"value":67,"source":49,"evidence":68},"Over the past year",[69],{"id":52,"by":11,"text":53},[71,77,83],{"value":72,"source":49,"evidence":73},"Python",[74],{"id":75,"by":11,"text":76},"38064810","> Just one note - the link in your Github readme to https:\u002F\u002Fumarbutler.com\u002Fopen-australian-legal-corpus doesn't seem to go anywhere. Thanks for the heads up! I've fixed that now.…",{"value":78,"source":49,"evidence":79},"pdfplumber",[80],{"id":81,"by":11,"text":82},"38065295","> A particular case I have is where there is a draft bill put out for industry\u002Fcommunity consultation. Quickly diffing the releases is the goal but for now usually relies on one…",{"value":84,"source":49,"evidence":85},"orjsonl",[86],{"id":75,"by":11,"text":76},"open_source","launched","solo",{"praise":91,"objections":123,"requests":142,"questions":167,"alternatives":182},[92,112],{"value":93,"source":94,"evidence":95},"Impressive effort and useful resource for the community","commenter",[96,100,104,108],{"id":97,"by":98,"text":99},"38059460","danielmarkbruce","Insanely great. Amazing work.",{"id":101,"by":102,"text":103},"38060943","thomasfromcdnjs","Incredible effort. These types of projects have the potential to influence a nation.",{"id":105,"by":106,"text":107},"38093924","jfil","I'm floored by what you accomplished in a year. Here in Canada, case law is under an iron grip by Canlii, LexisNexis and Thomson Reuters (Westlaw). As far as I'm aware, there is…",{"id":109,"by":110,"text":111},"38061067","subhashp","Well done!",{"value":113,"source":94,"evidence":114},"Open legal data is valuable for democratising law",[115,119],{"id":116,"by":117,"text":118},"38063696","ENGNR","Love what you're doing! Being able to more easily bring LLMs and other AI in will democratise the law quite a bit. Agreed that even though Austlii exists, it needs to be under a…",{"id":120,"by":121,"text":122},"38096887","Mrirazak1","It would be lovely I think if you used ML to help people ask questions so they can have more accessible law at their hands and understand what lots of things mean. This can be…",[124,131],{"value":125,"source":94,"evidence":126},"Not truly open in the sense of copyright-free, unlike AustLII and Jade",[127],{"id":128,"by":129,"text":130},"38062707","juliangamble","Australia has had free, searchable collections of Australian Law for 25+ years. Austlii is a prime example. There are Federal and State collections as well. The author is…",{"value":132,"source":94,"evidence":133},"Victoria's data is missing from the corpus",[134,138],{"id":135,"by":136,"text":137},"38066726","smcleod","That's really neat. Such a shame VIC couldn't be included though.",{"id":139,"by":140,"text":141},"38062319","cookie_monsta","This is cool and I'm a little surprised to see that Victoria is the one dragging the chain here. Is DataVic just talk, or does that not apply to law for some reason?",[143,153],{"value":144,"source":94,"evidence":145},"Add more metadata such as dates and reference info",[146,150],{"id":147,"by":148,"text":149},"38064971","triggercut","Yes, this. I've been trying to find a general way to automatically semantically chunk various legislation for a while now. Partly so as to diff various versions\u002Famendments, but…",{"id":151,"by":148,"text":152},"38065666","I've used a number of pdf libraries in python and C# over the years, none have worked reliably as needed (that's just pdf I guess), but haven't used pdfplumber, I'll be sure to…",{"value":154,"source":94,"evidence":155},"Replicate corpus to Internet Archive, Zenodo, Kaggle or similar",[156,160,163],{"id":157,"by":158,"text":159},"38058052","DamonHD","Would it we worth getting your corpus replicated into other venues as well, such at the Internet Archive or on GitHub itself?",{"id":161,"by":158,"text":162},"38059710","There are also national and university data repositories that might be interested and for which 5GB is not even noticable!",{"id":164,"by":165,"text":166},"38063565","Shorn","You could also consider one or more of the scientific data repositories like Zenodo, FigShare, DataDryad, etc. 5GB is small potatoes for those folks and they have serious data…",[168,175],{"value":169,"source":94,"evidence":170},"How does it compare with AustLII for completeness?",[171],{"id":172,"by":173,"text":174},"38064642","emmelaich","Nice, but I have to ask how does it compare with https:\u002F\u002Faustlii.edu.au\u002F, especially for completeness? The Australasian Legal Information Institute is a great resource and yet…",{"value":176,"source":94,"evidence":177},"How can the corpus be CC BY 4.0 if source data was not?",[178],{"id":179,"by":180,"text":181},"38085058","Syeposxr","Could you explain how the majority of your corpus is under CC BY 4.0? I realise that's the licence you have picked on HuggingFace, but if the source data was not already CC BY…",[183,188,194],{"value":184,"source":94,"evidence":185},"AustLII",[186,187],{"id":128,"by":129,"text":130},{"id":172,"by":173,"text":174},{"value":189,"source":49,"evidence":190},"Pile of Law",[191],{"id":192,"by":11,"text":193},"38064159","There is! It’s called the [Pile of Law](https:\u002F\u002Fhuggingface.co\u002Fdatasets\u002Fpile-of-law\u002Fpile-of-law), and it’s actually what inspired this project. The only caveat is that it doesn’t…",{"value":195,"source":94,"evidence":196},"CanLII",[197],{"id":198,"by":199,"text":200},"38063424","Obscurity4340","What do you think of the Canadian legal case law website CanLii? What could it do better or do you think its done well? Is it overdue for innovation?",{"comments":31,"answered":202,"key":203},8,[204,205,206],{"id":52,"text":53},{"id":75,"text":76},{"id":207,"text":208},"38063925","> Fantastic work here! I was griping to my team just last night how painful developing a chunking strategy for Australian Legislation is that while there's (generally) layout…",{"commenters":210,"max_depth":211,"maker_replies":31,"first_comment_mins":32,"lifespan_hours":212},36,6,343,[214,216,219,223,225,228,230,232,233,234,238,240,243,246,250,252,254,256,257,260,264,267,271,272,273,275,278,280,282,285,287,289,291,293,295,297,298,299,301,304,307,309,311,315,317,321,322,324,326,330,333,335,337,340,343,347,350,353,354,358],[52,32,32,215,11,37],"maker\u002Fpositive\u002Fnone",[217,32,43,215,218,32],"38060387","nextworddev",[220,32,43,221,222,32],"38063514","maker\u002Fpositive\u002Fdocs","benn0",[75,32,32,224,11,37],"maker\u002Fneutral\u002Fnone",[226,32,43,215,227,32],"38063712","SgtBastard",[207,32,32,229,11,37],"project\u002Fneutral\u002Fexecution",[147,32,44,231,148,32],"project\u002Fneutral\u002Fnone",[81,32,32,224,11,37],[151,32,44,231,148,32],[235,32,44,236,237,32],"38064001","project\u002Fneutral\u002Fidea","duttonw",[239,32,32,231,11,37],"38064846",[241,32,37,242,148,32],"38064775","project\u002Fcritical\u002Fexecution",[244,32,32,245,11,37],"38064794","maker\u002Fneutral\u002Fexecution",[247,32,43,248,249,32],"38115101","maker\u002Fpositive\u002Fidea","kennethologist",[128,37,37,251,129,32],"project\u002Fcritical\u002Fidea",[253,37,32,236,11,37],"38063584",[116,37,43,255,117,32],"project\u002Fpositive\u002Fidea",[139,33,44,231,140,32],[258,33,44,231,259,32],"38063689","mrmincent",[261,33,32,262,263,32],"38064571","other\u002Fneutral\u002Fnone","hugh-avherald",[265,44,43,255,266,32],"38059140","freefaler",[268,269,43,215,270,32],"38066245",4,"anakaine",[105,43,43,255,106,32],[172,211,44,231,173,32],[55,211,32,274,11,37],"maker\u002Fneutral\u002Fidea",[276,39,44,229,277,32],"38064150","RagnarD",[279,39,32,245,11,37],"38064254",[198,202,44,281,199,32],"project\u002Fneutral\u002Fdesign",[283,202,32,284,173,32],"38064691","other\u002Fneutral\u002Fdesign",[286,202,32,262,199,32],"38064724",[288,202,32,224,11,37],"38065131",[290,202,32,284,11,37],"38065124",[292,202,32,284,199,32],"38076731",[157,294,44,236,158,32],9,[296,294,32,224,11,37],"38058133",[161,294,44,231,158,32],[164,294,44,231,165,32],[300,294,32,224,11,37],"38063812",[302,294,32,262,303,32],"38064119","djbusby",[305,294,43,306,158,32],"38072268","project\u002Fpositive\u002Fnone",[308,294,32,262,158,32],"38243108",[120,310,43,255,121,32],10,[312,313,43,255,314,32],"38062647",11,"darcys22",[316,313,32,262,173,32],"38065375",[318,313,32,319,320,32],"38065449","commenter\u002Fneutral\u002Fnone","defrost",[179,18,44,231,180,32],[323,18,32,224,11,37],"38085188",[101,325,43,255,102,32],13,[327,328,32,284,329,32],"38062015",14,"ulrischa",[331,31,44,229,332,32],"38066097","mediumsmart",[135,334,44,231,136,32],16,[97,336,43,306,98,32],17,[338,339,44,236,218,32],"38060388",18,[341,339,32,262,342,32],"38061907","Something1234",[344,339,32,345,346,32],"38062163","other\u002Fneutral\u002Fidea","showerst",[348,339,32,345,349,32],"38062288","yieldcrv",[351,339,32,345,352,32],"38062379","bentley",[192,339,32,274,11,37],[355,356,43,306,357,32],"38067280",19,"tamarlikesdata",[109,359,43,306,110,32],20,[],[362,367,372,376,381,385],{"id":363,"name":364,"points":365,"category":366},"38052344","OpenSign",545,"web_saas",{"id":368,"name":369,"points":370,"category":371},"38044782","WireHole",298,"infrastructure",{"id":373,"name":374,"points":375,"category":20},"38038239","ScratchDB",261,{"id":377,"name":378,"points":379,"category":380},"38084098","postgresql-event-sourcing",231,"dev_tools",{"id":382,"name":383,"points":384,"category":371},"38086598","MicroTCP",204,{"id":386,"name":387,"points":388,"category":389},"38093083","Halloween game (snail terminal demo)",173,"games",false,"v2",{"id":393,"name":5,"line":6,"ai":390,"cat":20,"year":13,"points":14,"comments":15,"rec":28,"s":394,"top":18,"kw":395,"maker":11,"domain":10,"tier":19,"attempts":32,"strip":396,"url":9,"mix":397,"praise":93,"objection":125,"sub":21},38057759,0.39,[24,21,25,26,27],"0505333103053555033000333055503503530005",[31,31,32,33],1793991077090]