[{"data":1,"prerenderedAt":412},["ShallowReactive",2],{"project-32730038":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":28,"sentiment":29,"stance_mix":30,"scored_comments":35,"aspects":36,"strip":45,"post_text":47,"what":48,"said":83,"maker_in_thread":204,"shape":213,"grid":217,"attempts":374,"same_week":375,"maker_shows":34,"maker_hidden":404,"reader_version":405,"card":406},"32730038","WakaQ","a Python distributed task queue","title","Show HN: WakaQ - a Python distributed task queue","https:\u002F\u002Fwakatime.com\u002Fblog\u002F56-building-a-distributed-task-queue-in-python","wakatime.com","welder",1662415419,2022,147,68,409,2570,16,"notable","dev_tools","task queue","library",[24,25,26,27],"python task queue","redis queue","distributed workers","celery alternative","mixed",-0.022,{"positive":31,"neutral":32,"mixed":33,"critical":34},6,12,3,8,29,{"execution":37,"idea":40,"docs":43,"design":44},{"positive":38,"critical":39},0,7,{"positive":41,"critical":42},2,1,{"positive":42,"critical":38},{"positive":42,"critical":38},[33,46,46,38,38,33,38,46,46,42,33,33,33,38,42,42,38,33,46,33,33,38,33,38,33,38,38,38,33,42,38,42,42,42,38,38,38,38,33,33],5,null,{"problem":49,"audience":47,"pricing":47,"motivation":59,"time_to_build":47,"stack":63,"openness":80,"stage":81,"team_size":82},{"value":50,"source":51,"evidence":52},"Running background tasks across workers in Python with less complexity than Celery","maker",[53,56],{"id":54,"by":11,"text":55},"32730727","I was initially excited when Dramatiq was released, but I think the GNU license prohibits using it for commercial projects? Also, I really wanted something simple and easy to…",{"id":57,"by":11,"text":58},"32730773","Maybe I should have used Dramatiq instead of building my own, but it's already done. I'm surprised how well WakaQ is working in production actually. If any issues come up they…",{"value":60,"source":51,"evidence":61},"Wanted something simpler and easier to maintain than Celery or Dramatiq",[62],{"id":54,"by":11,"text":55},[64,71],{"value":65,"source":66,"evidence":67},"Python","inferred",[68],{"id":69,"by":11,"text":70},"32731169","It does have some typehints... but I don't like Python types when they require importing the object only for the purpose of types. I only import objects when they're actually…",{"value":72,"source":51,"evidence":73},"Redis",[74,77],{"id":75,"by":11,"text":76},"32731135","1. That's the part I'm not happy with, but currently I use the `@wakaq.after_worker_started` decorator to setup an error logging handler in each worker. It outputs to a file that…",{"id":78,"by":11,"text":79},"32731180","Distributed means a central broker (Redis) and many distributed workers (multi-process and multi-machine). It's the term used by Celery\u002FRQ\u002FHuey and others.","open_source","launched","unknown",{"praise":84,"objections":108,"requests":138,"questions":157,"alternatives":174},[85,97],{"value":86,"source":87,"evidence":88},"Simple, readable codebase","commenter",[89,93],{"id":90,"by":91,"text":92},"32733214","tatoalo","I really like the simplicity (i.e. readability) of the codebase, nice!",{"id":94,"by":95,"text":96},"32736826","avinassh","Thank you for explaining! I think this should part of the README. I am going through the code and loving the simplicity of it!",{"value":98,"source":87,"evidence":99},"Interesting ambitious project with good design",[100,104],{"id":101,"by":102,"text":103},"32730717","llIIllIIllIIl","Very nice and ambitious project. Did you compare vs Dramatiq? It was created to overcome same shortcomings of Celery.",{"id":105,"by":106,"text":107},"32731010","blondin","modern programming languages should have enough concurrency primitives for building such systems. the python situation is out of control. we have bazillion of libraries and…",[109,124,134],{"value":110,"source":87,"evidence":111},"No tests, type hints or docs",[112,116,120],{"id":113,"by":114,"text":115},"32735315","BiteCode_dev","The project looks nice, and I will certainly follow it. But the author expresses way too much confidence in the description of the project. It has no test, no type hints, no…",{"id":117,"by":118,"text":119},"32731035","hannibalhorn","I really like the overall design and ethos, but a new project without typehints, pydoc, and at least some tests just isn't the kind of thing I'd bring into my stack. Hope it gets…",{"id":121,"by":122,"text":123},"32735407","bosie","You are running this in production? I had a look at the github repo and did not find any tests. How are you QAing the project?",{"value":125,"source":87,"evidence":126},"Redis is a single point of failure, not truly distributed",[127,131],{"id":128,"by":129,"text":130},"32733044","jedberg","I'm talking about if the VM\u002FInstance\u002FDisk goes away. It looks like Redis is a single point of failure unless you run Redis in a redundant way, but skimming the code it looks like…",{"id":132,"by":114,"text":133},"32735229","Then I wouldn't call it distributed, since it's centralized on redis. Otherwised good product.",{"value":135,"source":87,"evidence":136},"Too early to claim production stability",[137],{"id":113,"by":114,"text":115},[139,147],{"value":140,"source":87,"evidence":141},"Document error handling in README",[142,143],{"id":94,"by":95,"text":96},{"id":144,"by":145,"text":146},"32733722","antman","I guess this should be on the FAQ",{"value":148,"source":87,"evidence":149},"Benchmarks against other queues",[150,154],{"id":151,"by":152,"text":153},"32736146","alecr95","Nice project! I'm one of TaskTiger's current maintainers. Just curious if any specific features are missing (or how it compares in benchmarks)?",{"id":155,"by":11,"text":156},"32736422","When workers claim tasks, TaskTiger moves tasks to a new queue instead of popping them like WakaQ. That means if your task fails with WakaQ then it's just gone. > how it compares…",[158,168],{"value":159,"source":87,"evidence":160},"How error handling and failed tasks work",[161,165],{"id":162,"by":163,"text":164},"32730949","osdev","Nice work on this! I’ve looked into task queue libraries for Node and Java in the past. Yours looks straight-forward. A few questions: 1. What’s the error handling strategy for…",{"id":166,"by":95,"text":167},"32736394","Is it at-most once delivery? how do you deal with crashes?",{"value":169,"source":87,"evidence":170},"Whether tasks are lost if Redis goes away",[171],{"id":172,"by":129,"text":173},"32732644","Since Redis is the data store, if your Redis instance goes away, do all of your tasks go away too?",[175,179,186,190,197],{"value":176,"source":87,"evidence":177},"Dramatiq",[178],{"id":101,"by":102,"text":103},{"value":180,"source":87,"evidence":181},"Celery",[182],{"id":183,"by":184,"text":185},"32731008","didip","I love that someone else also thinks that Celery has grown too big and too buggy. It would be awesome if Airflow starts using this.",{"value":187,"source":87,"evidence":188},"TaskTiger",[189],{"id":151,"by":152,"text":153},{"value":191,"source":87,"evidence":192},"Huey",[193],{"id":194,"by":195,"text":196},"32731032","JodieBenitez","In case anyone is after a non-distributed task queue for simple cases, I've used Huey above SQLite with ease.",{"value":198,"source":87,"evidence":199},"SAQ",[200],{"id":201,"by":202,"text":203},"32732658","captaintobs","i made one as well called SAQ! https:\u002F\u002Fgithub.com\u002Ftobymao\u002Fsaq Looks like you use BLPOP, but there could be race conditions or lost jobs if your workers crash, check out BLMOVE…",{"comments":205,"answered":206,"key":207},24,11,[208,211,212],{"id":209,"text":210},"32736385","> Stating \"WakaQ is stable and ready to use in production\" is irresponsible I've changed it to \"WakaQ is still a new project, so use at your own risk.\"",{"id":75,"text":76},{"id":54,"text":55},{"commenters":214,"max_depth":39,"maker_replies":205,"first_comment_mins":215,"lifespan_hours":216},32,85,159,[218,220,222,224,226,228,229,233,234,236,238,240,242,246,249,250,253,255,256,258,260,262,264,266,268,270,274,275,277,278,279,281,283,285,287,289,291,293,295,300,302,305,308,311,313,315,317,320,322,324,327,329,332,336,339,342,344,347,349,351,355,357,359,361,363,367,368,371],[113,38,33,219,114,38],"project\u002Fmixed\u002Fexecution",[209,38,38,221,11,42],"maker\u002Fneutral\u002Fdocs",[162,42,46,223,163,38],"maker\u002Fpositive\u002Fnone",[75,42,38,225,11,42],"maker\u002Fneutral\u002Fnone",[101,41,46,227,102,38],"project\u002Fpositive\u002Fidea",[54,41,38,225,11,42],[230,41,38,231,232,38],"32730737","commenter\u002Fneutral\u002Fnone","Waterluvian",[57,41,38,225,11,42],[235,41,33,225,232,38],"32730901",[121,41,33,237,122,38],"maker\u002Fneutral\u002Fexecution",[239,41,38,237,11,42],"32736609",[241,41,46,223,102,38],"32730955",[243,41,38,244,245,38],"32731781","other\u002Fneutral\u002Fnone","rajasimon",[247,41,38,244,248,38],"32733398","ac130kz",[183,33,46,227,184,38],[201,251,42,252,202,38],4,"project\u002Fcritical\u002Fexecution",[254,251,38,237,11,42],"32733011",[144,251,33,221,145,38],[194,46,33,257,195,38],"project\u002Fneutral\u002Fnone",[172,31,33,259,129,38],"project\u002Fneutral\u002Fexecution",[261,31,38,237,11,42],"32732992",[263,31,33,259,129,38],"32733010",[265,31,38,237,11,42],"32733147",[267,31,42,252,129,38],"32733166",[269,31,38,244,11,42],"32733665",[271,31,38,272,273,38],"32733027","commenter\u002Fneutral\u002Fexecution","belthesar",[128,31,42,252,129,38],[276,31,38,252,11,42],"32733677",[132,31,33,219,114,38],[166,39,33,259,95,38],[280,39,38,237,11,42],"32736537",[94,39,46,282,95,38],"project\u002Fpositive\u002Fdocs",[284,34,33,259,102,38],"32730966",[286,34,38,225,11,42],"32730981",[151,288,33,257,152,38],9,[155,288,38,290,11,42],"maker\u002Fneutral\u002Fperformance",[292,288,38,244,152,38],"32757147",[294,288,38,290,11,42],"32757348",[296,297,33,298,299,38],"32732555",10,"project\u002Fneutral\u002Fdocs","zerop",[301,297,38,221,11,42],"32733052",[303,206,33,259,304,38],"32733008","zamubafoo",[306,206,38,231,307,38],"32733085","dangrie158",[309,206,38,244,310,38],"32733318","blowski",[312,206,38,237,11,42],"32733110",[314,206,38,272,304,38],"32733247",[316,206,38,225,11,42],"32733389",[318,206,38,272,319,38],"32734287","nurettin",[90,32,46,321,91,38],"project\u002Fpositive\u002Fdesign",[117,323,42,252,118,38],13,[325,323,42,252,326,38],"32731096","notacop31337",[328,323,38,237,11,42],"32731243",[330,323,38,244,331,38],"32731454","swyx",[333,323,42,334,335,38],"32731153","project\u002Fcritical\u002Fidea","nerdponx",[337,323,38,244,338,38],"32731809","glaucon",[340,323,38,341,11,42],"32733086","project\u002Fpositive\u002Fperformance",[343,323,42,252,335,38],"32735812",[345,323,42,252,346,38],"32736167","heavenlyblue",[348,323,38,237,11,42],"32736634",[69,323,38,350,11,42],"maker\u002Fneutral\u002Fdesign",[352,323,38,353,354,38],"32731721","commenter\u002Fneutral\u002Fdesign","ttymck",[356,323,38,231,11,42],"32733077",[358,323,38,231,346,38],"32736153",[360,323,38,231,11,42],"32736579",[362,323,38,231,346,38],"32809216",[364,365,33,257,366,38],"32731117",14,"cakoose",[78,365,38,257,11,42],[105,369,33,370,106,38],15,"project\u002Fmixed\u002Fidea",[372,369,38,231,373,38],"32731057","spiffytech",[],[376,381,386,391,395,399],{"id":377,"name":378,"points":379,"category":380},"32690410","Openship",1335,"web_saas",{"id":382,"name":383,"points":384,"category":385},"32747067","quake_watch",692,"games",{"id":387,"name":388,"points":389,"category":390},"32688646","Wavvy",530,"creative",{"id":392,"name":393,"points":394,"category":385},"32717133","Zelda Breath of the Wild Street View",456,{"id":396,"name":397,"points":398,"category":390},"32738654","Bauble",252,{"id":400,"name":401,"points":402,"category":403},"32753471","TinyClock",192,"productivity",false,"v2",{"id":407,"name":5,"line":6,"ai":404,"cat":20,"year":13,"points":14,"comments":15,"rec":28,"s":408,"top":18,"kw":409,"maker":11,"domain":10,"tier":19,"attempts":38,"strip":410,"url":9,"mix":411,"praise":86,"objection":110,"sub":21},32730038,-0.02,[24,25,26,27],"3550030551333011035330303000310111000033",[31,32,33,34],1794036624506]