[{"data":1,"prerenderedAt":405},["ShallowReactive",2],{"project-6393048":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":29,"sentiment":30,"stance_mix":31,"scored_comments":36,"aspects":37,"strip":42,"post_text":44,"what":45,"said":94,"maker_in_thread":228,"shape":239,"grid":242,"attempts":365,"same_week":366,"maker_shows":35,"maker_hidden":397,"reader_version":398,"card":399},"6393048","goworker","a faster, Resque-compatible background worker","title","Show HN: goworker – a faster, Resque-compatible background worker","http:\u002F\u002Fwww.goworker.org\u002F","goworker.org","benmanns",1379337196,2013,179,58,86,1739,5,"notable","infrastructure","background job worker","library",[24,25,26,27,28],"resque","background jobs","go worker","sidekiq","rails","loved",0.176,{"positive":32,"neutral":33,"mixed":34,"critical":35},6,17,0,1,24,{"idea":38,"performance":39,"docs":40,"design":41},{"positive":18,"critical":34},{"positive":35,"critical":35},{"positive":34,"critical":34},{"positive":34,"critical":34},[34,18,43,34,34,34,18,34,34,34,34,34,43,43,34,43,34,43,34,34,34,43,43,43,43,34,43,18,18,43,34,34,43,43,18,43,34,43,43,43],3,null,{"problem":46,"audience":56,"pricing":44,"motivation":65,"time_to_build":44,"stack":69,"openness":91,"stage":92,"team_size":93},{"value":47,"source":48,"evidence":49},"Background jobs queued from Rails are slow to process in Ruby workers","maker",[50,53],{"id":51,"by":11,"text":52},"6393051","This is a weekend project I started on as a way to learn Go that I quickly realized would be something worth sharing. It lets you queue jobs from Rails that are processed in Go,…",{"id":54,"by":11,"text":55},"6393478","They are typically 10x faster in \"raw\" speed and 100x faster for concurrency. However, since Go's goroutines take almost no memory, you can run over 1,000 workers in the same…",{"value":57,"source":58,"evidence":59},"Rails developers using Resque or Sidekiq queues","inferred",[60,61],{"id":51,"by":11,"text":52},{"id":62,"by":63,"text":64},"6393656","kohanz","In my side-project, which is my first foray into Rails, I use a Resque background worker to do processing and it takes a significant amount of time. As I've heard more about Go,…",{"value":66,"source":48,"evidence":67},"Weekend project to learn Go that turned out worth sharing",[68],{"id":51,"by":11,"text":52},[70,74,84],{"value":71,"source":48,"evidence":72},"Go",[73],{"id":51,"by":11,"text":52},{"value":75,"source":58,"evidence":76},"Redis",[77,81],{"id":78,"by":79,"text":80},"6393414","jrochkind1","> For simple dequeuing, deserialization, execution, and updating stats (as with the Hello worker above), goworker is 80x faster than Resque and 23x faster than Sidekiq. In real…",{"id":82,"by":11,"text":83},"6393600","You're right that this is less significant if you are hitting an empty queue (triggering a timeout) between each request. However, even when not comparing (seconds to computer…",{"value":85,"source":48,"evidence":86},"Ruby (Rails queuing side)",[87,90],{"id":88,"by":11,"text":89},"6393337","Yes, workers are written in Go, but can be queued from Ruby. I'll put together an example in the goworker-examples [0] to test web crawling. [0]…",{"id":54,"by":11,"text":55},"open_source","unknown","solo",{"praise":95,"objections":112,"requests":139,"questions":164,"alternatives":193},[96,105],{"value":97,"source":98,"evidence":99},"Useful for speeding up Ruby background jobs with Go workers","commenter",[100,101],{"id":62,"by":63,"text":64},{"id":102,"by":103,"text":104},"6393228","atonse","Very cool - I just interviewed with a company that is using Sidekiq to process up to millions of jobs (at about 2\u002Fsec per worker) and I asked them to look into using Go for…",{"value":106,"source":48,"evidence":107},"Cheap concurrency allows very many workers",[108,111],{"id":109,"by":11,"text":110},"6394171","Even though most of the time is spent on I\u002FO, goworker can get wins from its super cheap concurrency. I was able to run 100,000 workers on my single 4GB RAM laptop.",{"id":54,"by":11,"text":55},[113,124,132],{"value":114,"source":98,"evidence":115},"Workers must be written in Go, so Ruby code can't move over easily",[116,120],{"id":117,"by":118,"text":119},"6393300","AznHisoka","I took a look, and am a little confused. Does the code for the workers need to be written in Go, not Ruby? Have you also tested the performance of this with 40-50 workers doing…",{"id":121,"by":122,"text":123},"6437042","mikeni","interested in your findings regarding goworker, so the worker logic has to be written in go correct? for my rails app if its something simple like sending email, I can use that,…",{"value":125,"source":98,"evidence":126},"Speed gains in the queue overhead matter little when real work dominates",[127,128],{"id":78,"by":79,"text":80},{"id":129,"by":130,"text":131},"6393816","meritt","Yup. I have distributed worker system with all the workers in PHP. The majority of the time spent is waiting on database requests and external HTTP calls. Even if the speed of…",{"value":133,"source":98,"evidence":134},"Rewriting things in many languages creates noise",[135],{"id":136,"by":137,"text":138},"6393574","agentultra","You also forget the competition from Javascript as well. Rewrite all of the things in all of the languages. I'd much rather see neat algorithms and libraries being re-written in…",[140,147,153],{"value":141,"source":98,"evidence":142},"Support RQ message format compatibility",[143],{"id":144,"by":145,"text":146},"6394239","sylvinus","Would be awesome to have RQ compatibility too ;-) http:\u002F\u002Fpython-rq.org",{"value":148,"source":48,"evidence":149},"Full Sidekiq compatibility including scheduling and retries",[150],{"id":151,"by":11,"text":152},"6393361","That's really cool. Where are you using it in production? Full Sidekiq compatibility (scheduling, retries) was my next goal.",{"value":154,"source":98,"evidence":155},"Better graph labels and charts",[156,160],{"id":157,"by":158,"text":159},"6393139","hmottestad","Please label your graphs. X-axis, Y-axis and title :)",{"id":161,"by":162,"text":163},"6393624","g5pw","A nice log graph would be better than the linear ones IMO, especially when the scale stretches a lot (like in the Database Insert tab)",[165,175,185],{"value":166,"source":98,"evidence":167},"How to handle Heroku SIGTERM and lost jobs",[168,171],{"id":169,"by":11,"text":170},"6393380","How do you handle when Heroku sends a SIGTERM to kill your process? I couldn't find a way to preempt running workers, so everything running on Heroku has to finish within 10…",{"id":172,"by":173,"text":174},"6395121","jrallison","Hey Ben, Author of go-workers here. On SIGTERM, I stop accepting new work, and wait for all running workers to finish before halting. If workers take longer than the 10 seconds…",{"value":176,"source":98,"evidence":177},"How to write integration tests across Rails and goworker",[178,182],{"id":179,"by":180,"text":181},"6394521","joevandyk","How would you write integration tests if you had a rails app that pushed jobs out to goworker? Say a user clicks a link that creates an async job that does a API call somewhere.…",{"id":183,"by":180,"text":184},"6394712","The test should use redis though, to test the whole stack. Same question goes for any time of queue system, like quque_classic, which stores the jobs in a postgresql table.",{"value":186,"source":98,"evidence":187},"Whether ActiveRecord-dependent jobs can run in Go",[188,192],{"id":189,"by":190,"text":191},"6398427","oelmekki","That looks great, congrats. I suppose the main use case is working on data loosely coupled with web app, thus ? For example, if we have to do any big processing on active_record…",{"id":121,"by":122,"text":123},[194,199,206,213,220],{"value":195,"source":98,"evidence":196},"Sidekiq",[197,198],{"id":51,"by":11,"text":52},{"id":117,"by":118,"text":119},{"value":200,"source":98,"evidence":201},"go-workers",[202],{"id":203,"by":204,"text":205},"6393290","jemeshsu","Interestingly there is a Sidekiq compatible job queue written in Go that has a similar name: go-workers https:\u002F\u002Fgithub.com\u002Fjrallison\u002Fgo-workers Wondering how these ruby\u002Frails…",{"value":207,"source":98,"evidence":208},"Jesque",[209],{"id":210,"by":211,"text":212},"6393220","adefa","There is Java implementation of Resque as well: https:\u002F\u002Fgithub.com\u002Fgresrun\u002Fjesque I've been using jesque to consume Resque tasks with no problems, and performance is great.",{"value":214,"source":98,"evidence":215},"gokiq",[216],{"id":217,"by":218,"text":219},"6393263","Titanous","I have a similar project that we use in production; a Sidekiq clone in Go: https:\u002F\u002Fgithub.com\u002Fcupcake\u002Fgokiq I should really document it. It has full type safety, stats, scheduled…",{"value":221,"source":98,"evidence":222},"NSQ",[223,224],{"id":203,"by":204,"text":205},{"id":225,"by":226,"text":227},"6399891","rargulati","We're in the midst of switching from quite a large sidekiq deployment to inbound work being done in NSQ. At a high level, NSQ allows for different strategies like fanout (having…",{"comments":229,"answered":230,"key":231},20,4,[232,233,236],{"id":51,"text":52},{"id":234,"text":235},"6393239","Awesome! Can you send me your contact info? Mine is in my HN profile.",{"id":237,"text":238},"6394425","Hmm. I took a look at this and it could be a possibility, but RQ's message format is much different from Resque and Sidekiq's. See https:\u002F\u002Fgist.github.com\u002Fbenmanns\u002F6583712 for a…",{"commenters":240,"max_depth":18,"maker_replies":229,"first_comment_mins":34,"lifespan_hours":241},31,189,[243,245,247,248,250,251,253,254,256,260,262,264,267,268,270,271,273,274,276,278,281,282,283,286,288,291,292,294,297,298,300,301,304,306,309,313,316,318,321,323,325,327,328,330,333,335,336,339,341,342,344,346,349,352,354,356,359,362],[51,34,34,244,11,35],"maker\u002Fneutral\u002Fnone",[102,34,18,246,103,34],"project\u002Fpositive\u002Fidea",[234,34,34,244,11,35],[144,34,43,249,145,34],"project\u002Fneutral\u002Fidea",[237,34,34,249,11,35],[217,35,34,252,218,34],"commenter\u002Fneutral\u002Fnone",[151,35,34,252,11,35],[255,35,34,252,218,34],"6393452",[257,258,18,246,259,34],"6393951",2,"steveklabnik",[261,258,34,252,11,35],"6394155",[203,43,34,263,204,34],"other\u002Fneutral\u002Fnone",[265,43,34,263,266,34],"6393319","jonnii",[169,43,34,252,11,35],[269,43,34,252,266,34],"6394714",[172,43,34,263,173,34],[272,43,34,252,11,35],"6395358",[225,43,34,263,226,34],[117,230,43,275,118,34],"project\u002Fneutral\u002Fperformance",[88,230,34,277,11,35],"project\u002Fneutral\u002Fnone",[279,230,43,275,280,34],"6393368","dkhenry",[54,230,34,275,11,35],[189,230,43,249,190,34],[284,18,34,263,285,34],"6393359","skion",[287,18,34,252,11,35],"6393364",[289,18,43,249,290,34],"6393550","zellyn",[136,18,34,263,137,34],[293,18,34,263,137,34],"6393786",[295,18,34,263,296,34],"6394198","iamwil",[210,32,34,263,211,34],[299,32,34,252,11,35],"6393237",[121,32,43,249,122,34],[302,32,34,263,303,34],"6394634","gresrun",[78,305,43,275,79,34],7,[307,305,43,275,308,34],"6393459","bradleyland",[310,305,35,311,312,34],"6393775","project\u002Fcritical\u002Fperformance","derefr",[314,305,18,315,308,34],"6393941","project\u002Fpositive\u002Fperformance",[317,305,34,252,11,35],"6394187",[319,305,43,249,320,34],"6393500","sanderjd",[322,305,34,275,11,35],"6393580",[324,305,18,246,259,34],"6394263",[326,305,18,246,320,34],"6397601",[82,305,34,275,11,35],[329,305,43,275,79,34],"6393920",[331,305,34,332,11,35],"6396046","commenter\u002Fneutral\u002Fperformance",[129,305,34,334,130,34],"other\u002Fneutral\u002Fperformance",[109,305,34,315,11,35],[179,337,43,338,180,34],8,"project\u002Fneutral\u002Fdocs",[340,337,34,338,11,35],"6394645",[183,337,43,338,180,34],[62,343,18,246,63,34],9,[345,343,34,244,11,35],"6393758",[161,347,43,348,162,34],10,"project\u002Fneutral\u002Fdesign",[350,347,34,351,11,35],"6394510","commenter\u002Fneutral\u002Fdesign",[157,353,43,348,158,34],11,[355,353,34,348,11,35],"6393152",[357,353,43,348,358,34],"6393193","dylz",[360,353,43,348,361,34],"6393250","gary4gar",[363,353,43,348,364,34],"6395859","ylem",[],[367,372,377,382,387,392],{"id":368,"name":369,"points":370,"category":371},"6373645","Wit",321,"ai",{"id":373,"name":374,"points":375,"category":376},"6407794","Signal",151,"productivity",{"id":378,"name":379,"points":380,"category":381},"6375958","bdon.org",96,"learning_science",{"id":383,"name":384,"points":385,"category":386},"6371478","Formbuilder.js",78,"languages",{"id":388,"name":389,"points":390,"category":391},"6390749","Web-controlled Christmas Lights with Node.js, Arduino, and Raspberry Pi",51,"hardware",{"id":393,"name":394,"points":395,"category":396},"6384713","jsbin.com",39,"games",false,"v2",{"id":400,"name":5,"line":6,"ai":397,"cat":20,"year":13,"points":14,"comments":15,"rec":29,"s":401,"top":18,"kw":402,"maker":11,"domain":10,"tier":19,"attempts":34,"strip":403,"url":9,"mix":404,"praise":97,"objection":114,"sub":21},6393048,0.18,[24,25,26,27,28],"0530005000003303030003333035530033530333",[32,33,34,35],1793990897792]