[{"data":1,"prerenderedAt":275},["ShallowReactive",2],{"project-12842797":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":28,"sentiment":29,"stance_mix":30,"scored_comments":35,"aspects":36,"strip":43,"post_text":44,"what":45,"said":71,"maker_in_thread":154,"shape":160,"grid":164,"attempts":238,"same_week":239,"maker_shows":34,"maker_hidden":268,"reader_version":269,"card":270},"12842797","KQ","Simple Job Queue for Python Using Kafka","title","Show HN: KQ – Simple Job Queue for Python Using Kafka","https:\u002F\u002Fgithub.com\u002Fjoowani\u002Fkq","github.com","joohwan",1477965527,2016,116,30,307,2264,14,"notable","dev_tools","job queue library","library",[24,25,26,27],"python job queue","kafka","task queue","background jobs","mixed",-0.16,{"positive":31,"neutral":32,"mixed":33,"critical":34},2,5,1,4,12,{"docs":37,"idea":39,"performance":41,"execution":42},{"positive":38,"critical":33},0,{"positive":33,"critical":40},3,{"positive":38,"critical":38},{"positive":33,"critical":38},[38,38,38,38,38,38,38,38,40,40,32,40,33,38,38,38,38,40,40,33,38,38,40,32,33,38,38,38,33,38],null,{"problem":46,"audience":44,"pricing":44,"motivation":44,"time_to_build":44,"stack":55,"openness":69,"stage":70,"team_size":70},{"value":47,"source":48,"evidence":49},"Running background jobs using Kafka as the queue","inferred",[50,52],{"id":51},"post",{"id":53,"by":11,"text":54},"12843225","Very true. I indeed found the lack of visibility into per-message information very painful when I was building this. One way I tried to alleviate the issue was providing a…",[56,60],{"value":57,"source":48,"evidence":58},"Python",[59],{"id":51},{"value":61,"source":62,"evidence":63},"Apache Kafka","maker",[64,65],{"id":51},{"id":66,"by":67,"text":68},"12842986","smegel","> It is backed by Apache Kafka and designed primarily for ease of use. That's a bold claim.","open_source","unknown",{"praise":72,"objections":85,"requests":108,"questions":127,"alternatives":141},[73],{"value":74,"source":75,"evidence":76},"Clean, readable code","commenter",[77,81],{"id":78,"by":79,"text":80},"12843364","crucialsnippet","Pretty clean code. Nice job.",{"id":82,"by":83,"text":84},"12843883","jondot","To me, Kafka as a job queue is a painful impedance mismatch. To achieve that, you need to: 1. Figure out which Kafka broker you're using. The concept of a consumer and consumer…",[86,94,101],{"value":87,"source":75,"evidence":88},"Kafka is a poor fit for job queues due to head-of-line blocking and per-message semantics",[89,93],{"id":90,"by":91,"text":92},"12843066","Xorlev","One of the biggest problems with treating Kafka as a job queue is that you suffer from head-of-line blocking. Kafka doesn't expose per-message visibility\u002Facknowledgement…",{"id":82,"by":83,"text":84},{"value":95,"source":75,"evidence":96},"Requires Zookeeper, not mentioned in dependencies",[97],{"id":98,"by":99,"text":100},"12843383","stratospherein","Don't you need zookeeper to use Kafka? Perhaps you should mention that in your dependencies.",{"value":102,"source":75,"evidence":103},"Memory-bound Redis queues like RQ cause production issues, which KQ may avoid",[104],{"id":105,"by":106,"text":107},"12845489","joowani","RQ is certainly useful (with finer control over messages), but we've been having a lot of problems with it lately in production due to it being memory bound (Redis). Improper…",[109,120],{"value":110,"source":75,"evidence":111},"Document when to use KQ versus RQ or Celery and Kafka's limits",[112,113,117],{"id":105,"by":106,"text":107},{"id":114,"by":115,"text":116},"12880084","reubano","Ok, that makes sense. I've never dealt with the scale you are talking about so have yet to run into these issues with redis. It may be best to put this info in the readme so…",{"id":118,"by":106,"text":119},"12845425","Thank you for the excellent feedback and insight. I will definitely give the pdf a read. I agree with all of your points, which admittedly I was not fully aware of when I first…",{"value":121,"source":75,"evidence":122},"Fix the docs link that returned permission denied",[123],{"id":124,"by":125,"text":126},"12842898","azundo","Getting a Permission Denied page for the docs at http:\u002F\u002Fkq.readthedocs.io\u002Fen\u002Fmaster\u002F",[128,134],{"value":129,"source":75,"evidence":130},"Why use this over RQ?",[131],{"id":132,"by":115,"text":133},"12843665","Why should I use this over the simplier RQ?",{"value":135,"source":75,"evidence":136},"What kinds of jobs are put into job queues?",[137],{"id":138,"by":139,"text":140},"12845248","dozzie","What kind of jobs do people put into such queues? Because I've only seen job queues used as a poor man's RPC brokers and (poor?) replacement for crontab. I think I may miss some…",[142,147],{"value":143,"source":75,"evidence":144},"RQ",[145,146],{"id":132,"by":115,"text":133},{"id":105,"by":106,"text":107},{"value":148,"source":75,"evidence":149},"Celery",[150],{"id":151,"by":152,"text":153},"12845121","brianwawok","Or the proven Celery",{"comments":31,"answered":38,"key":155},[156,157],{"id":53,"text":54},{"id":158,"text":159},"12843231","Fixed!",{"commenters":161,"max_depth":32,"maker_replies":31,"first_comment_mins":162,"lifespan_hours":163},20,26,109,[165,167,169,171,175,178,180,183,186,188,190,192,194,196,198,201,204,206,208,210,211,214,216,220,223,225,228,232,234,237],[90,38,38,166,91,38],"commenter\u002Fneutral\u002Fidea",[53,38,38,168,11,33],"maker\u002Fneutral\u002Fidea",[170,38,38,166,91,38],"12849059",[172,33,38,173,174,38],"12843319","other\u002Fneutral\u002Fnone","pfarnsworth",[176,33,38,173,177,38],"12843657","winteriscoming",[179,33,38,173,174,38],"12843824",[181,33,38,173,182,38],"12845810","SEJeff",[184,33,38,173,185,38],"12847797","boredandroid",[98,31,40,187,99,38],"project\u002Fneutral\u002Fdocs",[82,40,40,189,83,38],"project\u002Fmixed\u002Fidea",[118,40,32,191,106,38],"maker\u002Fpositive\u002Fidea",[132,34,40,193,115,38],"project\u002Fneutral\u002Fidea",[151,34,33,195,152,38],"project\u002Fcritical\u002Fidea",[197,34,38,173,139,38],"12845196",[199,34,38,200,152,38],"12845509","commenter\u002Fneutral\u002Fnone",[202,34,38,173,203,38],"12850704","al2o3cr",[205,34,38,173,139,38],"12853066",[105,34,40,207,106,38],"project\u002Fneutral\u002Fperformance",[114,34,40,209,115,38],"maker\u002Fneutral\u002Fdocs",[138,32,33,195,139,38],[212,32,38,200,213,38],"12846207","chrishacken",[215,32,38,200,139,38],"12846399",[217,218,40,187,219,38],"12846620",6,"tobych",[78,221,32,222,79,38],7,"project\u002Fpositive\u002Fexecution",[66,224,33,195,67,38],8,[226,224,38,200,227,38],"12843299","rads",[229,224,38,230,231,38],"12843352","commenter\u002Fcritical\u002Fidea","mperham",[233,224,38,200,67,38],"12843745",[124,235,33,236,125,38],9,"project\u002Fcritical\u002Fdocs",[158,235,38,209,11,33],[],[240,245,250,255,260,264],{"id":241,"name":242,"points":243,"category":244},"12834327","sudo-touchid",347,"security_privacy",{"id":246,"name":247,"points":248,"category":249},"12843466","My first web app",297,"web_saas",{"id":251,"name":252,"points":253,"category":254},"12803860","Zone Messenger",194,"consumer_social",{"id":256,"name":257,"points":258,"category":259},"12838152","Fast Style Transfer",184,"ai",{"id":261,"name":262,"points":263,"category":244},"12802785","Android 'root' phones via dirtyc0w exploit",143,{"id":265,"name":266,"points":267,"category":249},"12848656","Academic Bio",113,false,"v2",{"id":271,"name":5,"line":6,"ai":268,"cat":20,"year":13,"points":14,"comments":15,"rec":28,"s":29,"top":18,"kw":272,"maker":11,"domain":10,"tier":19,"attempts":38,"strip":273,"url":9,"mix":274,"praise":74,"objection":87,"sub":21},12842797,[24,25,26,27],"000000003353100003310035100010",[31,32,33,34],1794036549086]