PgQueuer
Languages and libraries
job queue library
SHOW HN2024
How HN took it
6 positive, 13 neutral or asking, 0 mixed, 9 critical, out of 28 comments about the project. Thread score -0.07.
Idea: 3 for, 8 againstExecution: 1 for, 0 againstPerformance: 0 for, 1 againstDesign: 1 for, 0 against
The thread, comment by comment
Each row is a top-level comment and its replies, in HN's order. Select a cell to open that comment on HN.
positive neutral or mixed critical about another commenter or something else the maker
What HN said
What people liked
Simplicity and fewer moving parts than a separate broker
From 3 comments
there seems to be a big hype to adapt pg into any infra. I love PG but this seems not be right thing.
redskyluan on HNI use it as a job queue. Yes, it has it's cons, but not dealing with another moving piece in the big picture is totally worth it.
mlnj on HNI really like the emergence of simple queuing tools for robust database management systems. Keep things simple and remove infrastructure complexity. Definitely a +1 from me! For…
mads_quist on HN
Good performance at small to medium scale
From 2 comments
At low-medium scale, this will be fine. Even at higher scale, so long as you monitor autovacuum performance on the queue table. At some point it may become practical to bring a…
sgarland on HNI agree, there is no need for FANG level infrastructure. Imo. in most cases, the simplicity / performance tradeoff for small/medium is worth it. There is also a statistics…
jeeybee on HN
Objections
Postgres used for everything is risky at high throughput (bloat, planner issues)
From 2 comments
Putting low throughput queues in the same DB is great both for simplicity and for getting exactly-once-processing. Putting high throughput queues in Postgres sucks because... No…
jashmatthews on HNBy the time you need append-only job statuses it's better to move to a dedicated queue. Append-only statuses help but they also make the polling query a lot more expensive.…
jashmatthews on HN
LISTEN/NOTIFY does not survive connection pooling cleanly
From 2 comments
Putting low throughput queues in the same DB is great both for simplicity and for getting exactly-once-processing. Putting high throughput queues in Postgres sucks because... No…
jashmatthews on HNasyncpg clears out any listeners you have setup once a connection is returned to pool. This will lead to 'missed' events. I guess its something of the same story with psycopg? If…
jeeybee on HN
Requests
Multi-step pipelines with fan-out and accumulation
From 1 comment
This looks like a great task queue, I'm a massive proponent of "Postgres is all you need" [0] and doubling down on it with my project that takes it to the extreme. What I would…
samwillis on HN
Support for Node and other languages sharing one schema
From 1 comment
I’ve been thinking about the potential for PostgreSQL-backed job queue libraries to share a common schema. For instance, I’m a big fan of Oban in Elixir:…
cklee on HN
Questions
How does it compare to Graphile Worker?
From 2 comments
How does this compare to the popular Graphile Worker library? https://github.com/graphile/worker
bfelbo on HNCool, congrats on releasing. Have you seen graphile worker? Wondering how this compares or if you're building for different use-cases.
rgbrgb on HN
Does it address known Celery problems?
From 1 comment
I've been referring to this post about issues with Celery: https://docs.hatchet.run/blog/problems-with-celery Does PgQueuer address any of them?
odie5533 on HN
Alternatives named
Graphile Worker
From 1 comment
How does this compare to the popular Graphile Worker library? https://github.com/graphile/worker
bfelbo on HN
pgmq
From 1 comment
If you want a generic queue that can be consumed in any runtime, you can just build it directly into postgres via extensions like https://github.com/tembo-io/pgmq.
rileymichael on HN
River
From 1 comment
River ( https://riverqueue.com ) is a Postgres background job engine written in Go, which also has insert only clients in other languages. Currently we have these for Ruby and…
bgentry on HN
Procrastinate
From 1 comment
There is also Procrastinate: https://procrastinate.readthedocs.io/en/stable/index.html Procrastinate also uses PostgreSQL's LISTEN/NOTIFY (but can optionally be turned off and…
_medihack_ on HN
Celery
From 1 comment
I am going to go the other direction on this... to anyone reading this, please consider using a backend-generic queueing system for your Python project. Why? Mainly because those…
rtpg on HN