[{"data":1,"prerenderedAt":149},["ShallowReactive",2],{"project-37273963":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":29,"sentiment":30,"stance_mix":31,"scored_comments":32,"aspects":34,"strip":38,"post_text":39,"what":40,"said":63,"maker_in_thread":90,"shape":96,"grid":101,"attempts":113,"same_week":114,"maker_shows":141,"maker_hidden":142,"reader_version":143,"card":144},"37273963","StringZilla","Faking SIMD to Search and Sort Strings 5x Faster","title","Show HN: Faking SIMD to Search and Sort Strings 5x Faster","https:\u002F\u002Fashvardanian.com\u002Fposts\u002Fstringzilla\u002F","ashvardanian.com","ashvardanian",1693065426,2023,29,5,1423,3109,46,"shown","languages","string library","library",[24,25,26,27,28],"string search","swar","cpython","simd","string sorting","quiet",null,{"positive":32,"neutral":33,"mixed":33,"critical":33},2,0,{"idea":35,"docs":37},{"positive":36,"critical":33},1,{"positive":36,"critical":33},[15,33,33,15,33],"I want to share a really dumb, but very practical project I have packaged this summer, to perform operations on strings much faster. I was using Python to work with a multi-terabyte newline-delimited file. Reading, splitting, and shuffling it was a nightmare. So, I wrapped a trivial hardware-friendly heuristic I've been using for the last few years into a CPython library.\n\nThe part I enjoyed the most is implementing SIMD behavior without SIMD instructions... Using 64-bit words to work at 8-bit granularity. Unlike conventional SIMD, the code would remain the same for ~~almost~~ any hardware. Le",{"problem":41,"audience":47,"pricing":30,"motivation":51,"time_to_build":30,"stack":55,"openness":60,"stage":61,"team_size":62},{"value":42,"source":43,"evidence":44},"Slow string operations on multi-terabyte newline-delimited text files","maker",[45],{"id":46},"post",{"value":48,"source":43,"evidence":49},"Python users working with large text datasets such as CommonCrawl",[50],{"id":46},{"value":52,"source":43,"evidence":53},"Needed faster operations on a multi-terabyte newline-delimited file in Python",[54],{"id":46},[56],{"value":57,"source":43,"evidence":58},"Python (CPython library)",[59],{"id":46},"open_source","launched","solo",{"praise":64,"objections":80,"requests":81,"questions":88,"alternatives":89},[65,73],{"value":66,"source":67,"evidence":68},"Impressive benchmarks and straightforward API","commenter",[69],{"id":70,"by":71,"text":72},"37277067","itkent","I took a look at Stringzilla (https:\u002F\u002Fgithub.com\u002Fashvardanian\u002Fstringzilla), and in addition to the impressive benchmarks, the API looks pretty straightforward. It's a new star in…",{"value":74,"source":67,"evidence":75},"SWAR bit tricks are useful when SIMD is unavailable",[76],{"id":77,"by":78,"text":79},"37277475","gwittel","Cool library. This sounds like SWAR (simd within a register). I’ve seen these techniques give a nice speed up especially when SIMD isn’t available or a pain (eg in Java…",[],[82],{"value":83,"source":43,"evidence":84},"Add reverse functions such as rsplit and rfind",[85],{"id":86,"by":11,"text":87},"37281096","Thank you too! Once I have time to implement rsplit\u002Frfind and similar reverse functions, it should be a drop-in replacement for the Python string.",[],[],{"comments":32,"answered":36,"key":91},[92,95],{"id":93,"text":94},"37281090","Yes, in the past those tricks were much more common",{"id":86,"text":87},{"commenters":97,"max_depth":98,"maker_replies":32,"first_comment_mins":99,"lifespan_hours":100},4,3,330,18,[102,104,108,109,111],[77,33,15,103,78,33],"project\u002Fpositive\u002Fidea",[105,33,33,106,107,33],"37277613","commenter\u002Fneutral\u002Fnone","djmips",[93,33,33,106,11,36],[70,36,15,110,71,33],"project\u002Fpositive\u002Fdocs",[86,36,33,112,11,36],"maker\u002Fneutral\u002Fexecution",[],[115,120,125,130,134,138],{"id":116,"name":117,"points":118,"category":119},"37326870","compress",815,"dev_tools",{"id":121,"name":122,"points":123,"category":124},"37247767","obsidian-sync",416,"infrastructure",{"id":126,"name":127,"points":128,"category":129},"37252231","Shimmer",229,"consumer_social",{"id":131,"name":132,"points":133,"category":119},"37251969","Continue",187,{"id":135,"name":136,"points":137,"category":124},"37263473","Release AI",143,{"id":139,"name":140,"points":137,"category":119},"37310070","Langfuse",7,false,"v2",{"id":145,"name":5,"line":6,"ai":142,"cat":20,"year":13,"points":14,"comments":15,"rec":29,"s":30,"top":18,"kw":146,"maker":11,"domain":10,"tier":19,"attempts":33,"strip":147,"url":9,"mix":148,"praise":66,"objection":-1,"sub":21},37273963,[24,25,26,27,28],"50050",[32,33,33,33],1794036640782]