goworker

In my side-project, which is my first foray into Rails, I use a Resque background worker to do processing and it takes a significant amount of time. As I've heard more about Go,…

Very cool - I just interviewed with a company that is using Sidekiq to process up to millions of jobs (at about 2/sec per worker) and I asked them to look into using Go for…

Even though most of the time is spent on I/O, goworker can get wins from its super cheap concurrency. I was able to run 100,000 workers on my single 4GB RAM laptop.

They are typically 10x faster in "raw" speed and 100x faster for concurrency. However, since Go's goroutines take almost no memory, you can run over 1,000 workers in the same…

I took a look, and am a little confused. Does the code for the workers need to be written in Go, not Ruby? Have you also tested the performance of this with 40-50 workers doing…

interested in your findings regarding goworker, so the worker logic has to be written in go correct? for my rails app if its something simple like sending email, I can use that,…

> For simple dequeuing, deserialization, execution, and updating stats (as with the Hello worker above), goworker is 80x faster than Resque and 23x faster than Sidekiq. In real…

Yup. I have distributed worker system with all the workers in PHP. The majority of the time spent is waiting on database requests and external HTTP calls. Even if the speed of…

You also forget the competition from Javascript as well. Rewrite all of the things in all of the languages. I'd much rather see neat algorithms and libraries being re-written in…

Would be awesome to have RQ compatibility too ;-) http://python-rq.org

That's really cool. Where are you using it in production? Full Sidekiq compatibility (scheduling, retries) was my next goal.

Please label your graphs. X-axis, Y-axis and title :)

A nice log graph would be better than the linear ones IMO, especially when the scale stretches a lot (like in the Database Insert tab)

How do you handle when Heroku sends a SIGTERM to kill your process? I couldn't find a way to preempt running workers, so everything running on Heroku has to finish within 10…

Hey Ben, Author of go-workers here. On SIGTERM, I stop accepting new work, and wait for all running workers to finish before halting. If workers take longer than the 10 seconds…

How would you write integration tests if you had a rails app that pushed jobs out to goworker? Say a user clicks a link that creates an async job that does a API call somewhere.…

The test should use redis though, to test the whole stack. Same question goes for any time of queue system, like quque_classic, which stores the jobs in a postgresql table.

That looks great, congrats. I suppose the main use case is working on data loosely coupled with web app, thus ? For example, if we have to do any big processing on active_record…

interested in your findings regarding goworker, so the worker logic has to be written in go correct? for my rails app if its something simple like sending email, I can use that,…

This is a weekend project I started on as a way to learn Go that I quickly realized would be something worth sharing. It lets you queue jobs from Rails that are processed in Go,…

I took a look, and am a little confused. Does the code for the workers need to be written in Go, not Ruby? Have you also tested the performance of this with 40-50 workers doing…

Interestingly there is a Sidekiq compatible job queue written in Go that has a similar name: go-workers https://github.com/jrallison/go-workers Wondering how these ruby/rails…

There is Java implementation of Resque as well: https://github.com/gresrun/jesque I've been using jesque to consume Resque tasks with no problems, and performance is great.

I have a similar project that we use in production; a Sidekiq clone in Go: https://github.com/cupcake/gokiq I should really document it. It has full type safety, stats, scheduled…

Interestingly there is a Sidekiq compatible job queue written in Go that has a similar name: go-workers https://github.com/jrallison/go-workers Wondering how these ruby/rails…

We're in the midst of switching from quite a large sidekiq deployment to inbound work being done in NSQ. At a high level, NSQ allows for different strategies like fanout (having…