[{"data":1,"prerenderedAt":601},["ShallowReactive",2],{"project-41284703":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":28,"sentiment":29,"stance_mix":30,"scored_comments":35,"aspects":36,"strip":44,"post_text":46,"what":47,"said":86,"maker_in_thread":194,"shape":203,"grid":208,"attempts":544,"same_week":564,"maker_shows":18,"maker_hidden":593,"reader_version":594,"card":595},"41284703","PgQueuer","Transform PostgreSQL into a Job Queue","title","Show HN: PgQueuer – Transform PostgreSQL into a Job Queue","https:\u002F\u002Fgithub.com\u002Fjanbjorge\u002FPgQueuer","github.com","jeeybee",1724008951,2024,376,122,128,3515,4,"legend","languages","job queue library","library",[24,25,26,27],"postgresql job queue","python task queue","listen notify","skip locked","mixed",-0.071,{"positive":31,"neutral":32,"mixed":33,"critical":34},6,13,0,9,28,{"idea":37,"execution":40,"performance":42,"design":43},{"positive":38,"critical":39},3,8,{"positive":41,"critical":33},1,{"positive":33,"critical":41},{"positive":41,"critical":33},[41,41,38,38,33,41,33,45,38,33,33,41,45,33,45,38,41,33,33,45,33,33,38,33,33,33,33,33,33,38,41,33,33,33,41,45,33,41,33,33],5,"PgQueuer is a minimalist, high-performance job queue library for Python, leveraging the robustness of PostgreSQL. Designed for simplicity and efficiency, PgQueuer uses PostgreSQL's LISTEN\u002FNOTIFY to manage job queues effortlessly.",{"problem":48,"audience":54,"pricing":63,"motivation":63,"time_to_build":63,"stack":64,"openness":85,"stage":85,"team_size":85},{"value":49,"source":50,"evidence":51},"Running background job queues without adding a separate broker, using PostgreSQL","maker",[52],{"id":53},"post",{"value":55,"source":56,"evidence":57},"Python developers already running PostgreSQL","inferred",[58,59],{"id":53},{"id":60,"by":61,"text":62},"41285205","cklee","I’ve been thinking about the potential for PostgreSQL-backed job queue libraries to share a common schema. For instance, I’m a big fan of Oban in Elixir:…",null,[65,69,73,79],{"value":66,"source":50,"evidence":67},"Python",[68],{"id":53},{"value":70,"source":50,"evidence":71},"PostgreSQL",[72],{"id":53},{"value":74,"source":50,"evidence":75},"asyncpg",[76],{"id":77,"by":11,"text":78},"41289729","asyncpg clears out any listeners you have setup once a connection is returned to pool. This will lead to 'missed' events. I guess its something of the same story with psycopg? If…",{"value":80,"source":50,"evidence":81},"anyio",[82],{"id":83,"by":11,"text":84},"41289713","At first glans i i see two thins that PgQueuer can. It has native async support, it you can also have sync functions, they will be offloaded to threads via anyio…","unknown",{"praise":87,"objections":114,"requests":130,"questions":142,"alternatives":161},[88,104],{"value":89,"source":90,"evidence":91},"Simplicity and fewer moving parts than a separate broker","commenter",[92,96,100],{"id":93,"by":94,"text":95},"41285422","redskyluan","there seems to be a big hype to adapt pg into any infra. I love PG but this seems not be right thing.",{"id":97,"by":98,"text":99},"41285572","mlnj","I use it as a job queue. Yes, it has it's cons, but not dealing with another moving piece in the big picture is totally worth it.",{"id":101,"by":102,"text":103},"41291666","mads_quist","I really like the emergence of simple queuing tools for robust database management systems. Keep things simple and remove infrastructure complexity. Definitely a +1 from me! For…",{"value":105,"source":50,"evidence":106},"Good performance at small to medium scale",[107,111],{"id":108,"by":109,"text":110},"41285805","sgarland","At low-medium scale, this will be fine. Even at higher scale, so long as you monitor autovacuum performance on the queue table. At some point it may become practical to bring a…",{"id":112,"by":11,"text":113},"41285901","I agree, there is no need for FANG level infrastructure. Imo. in most cases, the simplicity \u002F performance tradeoff for small\u002Fmedium is worth it. There is also a statistics…",[115,125],{"value":116,"source":90,"evidence":117},"Postgres used for everything is risky at high throughput (bloat, planner issues)",[118,122],{"id":119,"by":120,"text":121},"41287838","jashmatthews","Putting low throughput queues in the same DB is great both for simplicity and for getting exactly-once-processing. Putting high throughput queues in Postgres sucks because... No…",{"id":123,"by":120,"text":124},"41289402","By the time you need append-only job statuses it's better to move to a dedicated queue. Append-only statuses help but they also make the polling query a lot more expensive.…",{"value":126,"source":90,"evidence":127},"LISTEN\u002FNOTIFY does not survive connection pooling cleanly",[128,129],{"id":119,"by":120,"text":121},{"id":77,"by":11,"text":78},[131,138],{"value":132,"source":90,"evidence":133},"Multi-step pipelines with fan-out and accumulation",[134],{"id":135,"by":136,"text":137},"41285037","samwillis","This looks like a great task queue, I'm a massive proponent of \"Postgres is all you need\" [0] and doubling down on it with my project that takes it to the extreme. What I would…",{"value":139,"source":90,"evidence":140},"Support for Node and other languages sharing one schema",[141],{"id":60,"by":61,"text":62},[143,154],{"value":144,"source":90,"evidence":145},"How does it compare to Graphile Worker?",[146,150],{"id":147,"by":148,"text":149},"41289626","bfelbo","How does this compare to the popular Graphile Worker library? https:\u002F\u002Fgithub.com\u002Fgraphile\u002Fworker",{"id":151,"by":152,"text":153},"41286200","rgbrgb","Cool, congrats on releasing. Have you seen graphile worker? Wondering how this compares or if you're building for different use-cases.",{"value":155,"source":90,"evidence":156},"Does it address known Celery problems?",[157],{"id":158,"by":159,"text":160},"41288745","odie5533","I've been referring to this post about issues with Celery: https:\u002F\u002Fdocs.hatchet.run\u002Fblog\u002Fproblems-with-celery Does PgQueuer address any of them?",[162,166,173,180,187],{"value":163,"source":90,"evidence":164},"Graphile Worker",[165],{"id":147,"by":148,"text":149},{"value":167,"source":90,"evidence":168},"pgmq",[169],{"id":170,"by":171,"text":172},"41285821","rileymichael","If you want a generic queue that can be consumed in any runtime, you can just build it directly into postgres via extensions like https:\u002F\u002Fgithub.com\u002Ftembo-io\u002Fpgmq.",{"value":174,"source":90,"evidence":175},"River",[176],{"id":177,"by":178,"text":179},"41288140","bgentry","River ( https:\u002F\u002Friverqueue.com ) is a Postgres background job engine written in Go, which also has insert only clients in other languages. Currently we have these for Ruby and…",{"value":181,"source":90,"evidence":182},"Procrastinate",[183],{"id":184,"by":185,"text":186},"41285787","_medihack_","There is also Procrastinate: https:\u002F\u002Fprocrastinate.readthedocs.io\u002Fen\u002Fstable\u002Findex.html Procrastinate also uses PostgreSQL's LISTEN\u002FNOTIFY (but can optionally be turned off and…",{"value":188,"source":90,"evidence":189},"Celery",[190],{"id":191,"by":192,"text":193},"41286669","rtpg","I am going to go the other direction on this... to anyone reading this, please consider using a backend-generic queueing system for your Python project. Why? Mainly because those…",{"comments":34,"answered":31,"key":195},[196,199,202],{"id":197,"text":198},"41297259","Of my head, you could attach a uuid or sequence number to the emitted event. Then based on the uuid or sequence you can let one or the other event consumer pick? Ex. you have two…",{"id":200,"text":201},"41285278","LISTEN\u002FNOTIFY and SELECT FOR UPDATE SKIP LOCKED serve different purposes in PgQueuer. LISTEN\u002FNOTIFY notifies consumers about changes in the queue table, prompting them to check…",{"id":83,"text":84},{"commenters":204,"max_depth":205,"maker_replies":34,"first_comment_mins":206,"lifespan_hours":207},87,7,47,52,[209,213,217,221,224,226,229,231,234,237,240,242,246,248,251,254,255,258,260,263,266,270,272,274,278,279,280,282,285,288,290,293,296,300,303,306,309,312,316,317,320,323,326,330,332,337,341,345,349,353,355,358,361,363,366,368,371,373,376,378,381,382,386,388,391,394,397,400,403,405,407,409,412,415,417,420,422,426,427,430,432,435,437,440,442,444,447,451,453,457,461,464,467,470,472,475,478,481,484,487,490,493,495,499,501,502,503,505,508,511,513,515,517,520,522,525,529,533,535,538,540,542],[210,33,33,211,212,33],"41285614","other\u002Fneutral\u002Fnone","westurner",[214,41,41,215,216,33],"41286459","project\u002Fcritical\u002Fidea","airocker",[218,41,38,219,220,33],"41295297","project\u002Fneutral\u002Fexecution","m11a",[222,41,41,223,216,33],"41295637","project\u002Fcritical\u002Fperformance",[197,41,33,225,11,41],"project\u002Fneutral\u002Fidea",[227,41,38,219,228,33],"41300356","collingreen",[230,41,38,219,216,33],"41300615",[232,41,33,233,216,33],"41300611","commenter\u002Fneutral\u002Fnone",[235,41,33,233,236,33],"41303023","dopp0",[238,41,38,225,239,33],"41298176","LunaSea",[241,41,38,219,216,33],"41300627",[243,244,38,219,245,33],"41285186",2,"CowOfKrakatoa",[200,244,33,247,11,41],"maker\u002Fneutral\u002Fexecution",[249,244,33,233,250,33],"41285951","severino",[252,244,33,233,253,33],"41286474","halfcat",[191,38,41,215,192,33],[256,38,33,233,257,33],"41286758","nsonha",[259,38,33,233,192,33],"41286805",[261,38,33,233,262,33],"41287510","topspin",[264,38,33,211,265,33],"41291656","miohtama",[267,38,33,268,269,33],"41293674","other\u002Fcritical\u002Fnone","nicktrocado",[271,38,33,233,265,33],"41297301",[273,38,33,211,192,33],"41305196",[275,38,45,276,277,33],"41291794","project\u002Fpositive\u002Fexecution","killerstorm",[158,18,38,225,159,33],[83,18,33,247,11,41],[281,18,38,225,159,33],"41293034",[283,18,33,211,284,33],"41294962","TkTech",[286,45,33,211,287,33],"41286219","ijustlovemath",[289,45,33,211,262,33],"41289678",[291,31,33,211,292,33],"41285896","martinald",[294,31,33,211,295,33],"41285931","eknkc",[297,31,33,298,299,33],"41295319","other\u002Fpositive\u002Fnone","simplyinfinity",[301,31,33,211,302,33],"41286448","hkon",[304,31,41,215,305,33],"41286757","wordofx",[307,205,33,211,308,33],"41285740","fijiaarone",[310,205,33,233,311,33],"41286099","kaoD",[313,39,45,314,315,33],"41289553","project\u002Fpositive\u002Fdesign","nifal_adam",[151,34,38,225,152,33],[318,34,33,233,319,33],"41286735","mind-blight",[321,34,33,211,322,33],"41289403","pestaa",[324,34,33,325,11,41],"41288327","maker\u002Fneutral\u002Fidea",[327,328,33,211,329,33],"41289283",10,"_joel",[147,331,38,225,148,33],11,[333,334,45,335,336,33],"41288868",12,"project\u002Fpositive\u002Fidea","piyushtechsavy",[338,32,45,339,340,33],"41299610","project\u002Fpositive\u002Fnone","mslot",[342,343,41,215,344,33],"41298940",14,"CRConrad",[346,347,33,211,348,33],"41287979",15,"joseferben",[350,351,41,215,352,33],"41287419",16,"worik",[60,354,38,225,61,33],17,[356,354,33,211,357,33],"41285276","memset",[359,354,33,211,360,33],"41285552","GordonS",[362,354,33,211,357,33],"41287030",[364,354,33,211,365,33],"41286936","justinclift",[367,354,33,211,357,33],"41287015",[369,354,33,211,370,33],"41287325","agrothberg",[372,354,33,211,357,33],"41287503",[374,354,45,335,375,33],"41285671","earthnail",[377,354,33,211,319,33],"41286715",[379,354,33,211,380,33],"41287067","jmvoodoo",[177,354,33,211,178,33],[383,354,33,384,385,33],"41285727","commenter\u002Fneutral\u002Fidea","stephenr",[170,354,33,387,171,33],"other\u002Fneutral\u002Fidea",[389,354,33,211,390,33],"41289632","tmountain",[392,354,33,211,393,33],"41290819","vantiro",[395,354,33,211,396,33],"41287831","AlphaSite",[398,354,33,211,399,33],"41289539","kraihx",[401,354,33,384,402,33],"41291711","ukd1",[135,404,38,225,136,33],18,[406,404,33,325,11,41],"41285246",[408,404,33,387,253,33],"41286133",[410,404,33,387,411,33],"41287262","whateveracct",[413,404,33,233,414,33],"41288168","philippta",[416,404,33,211,390,33],"41289684",[418,404,33,211,419,33],"41289649","berkes",[119,404,33,421,120,33],"other\u002Fneutral\u002Fperformance",[423,404,33,424,425,33],"41288079","commenter\u002Fneutral\u002Fperformance","mickeyp",[123,404,33,424,120,33],[428,404,33,424,429,33],"41289485","iTokio",[431,404,33,233,120,33],"41290289",[433,404,33,233,434,33],"41289488","felixyz",[436,404,33,424,120,33],"41290273",[438,404,33,424,439,33],"41289467","ulrikrasmussen",[77,404,33,441,11,41],"maker\u002Fneutral\u002Fperformance",[443,404,33,424,120,33],"41290314",[445,404,33,424,446,33],"41289573","paffdragon",[448,449,33,211,450,33],"41285713",19,"aflukasz",[452,449,33,211,11,41],"41285942",[454,455,33,387,456,33],"41288493",20,"kdunglas",[458,459,38,225,460,33],"41286456",21,"gmag",[462,459,33,211,463,33],"41286598","onionisafruit",[465,459,33,211,466,33],"41286682","bdcravens",[468,459,41,215,469,33],"41288889","rubenvanwyk",[184,471,33,387,185,33],22,[473,471,33,211,474,33],"41289034","joking",[476,471,33,211,477,33],"41289492","lukebuehler",[479,480,33,387,466,33],"41286659",23,[482,480,33,387,483,33],"41287634","strzibny",[485,480,33,211,486,33],"41288234","Lio",[488,480,33,211,489,33],"41288379","sandGorgon",[491,480,33,211,492,33],"41290084","BilalBudhani",[101,494,33,387,102,33],24,[496,497,33,387,498,33],"41292650",25,"jackbravo",[93,500,41,215,94,33],26,[97,500,33,211,98,33],[108,500,33,421,109,33],[112,500,33,504,11,41],"maker\u002Fpositive\u002Fperformance",[506,500,45,335,507,33],"41286428","arp242",[509,500,33,384,510,33],"41287010","nostrebored",[512,500,33,233,120,33],"41287902",[514,500,33,384,120,33],"41287879",[516,500,33,211,295,33],"41285980",[518,500,33,211,519,33],"41286581","jascha_eng",[521,500,33,384,466,33],"41286634",[523,500,41,215,524,33],"41286596","SOLAR_FIELDS",[526,527,33,387,528,33],"41288126",27,"est",[530,527,33,531,532,33],"41288225","commenter\u002Fneutral\u002Fexecution","DemocracyFTW2",[534,527,33,233,528,33],"41288264",[536,527,33,421,537,33],"41288413","hparadiz",[539,527,33,424,528,33],"41288436",[541,527,33,424,537,33],"41288482",[543,527,33,504,11,41],"41289134",[545,548,551,554,557,558,561],{"id":546,"t":547,"points":38,"comments":33},"40187605",1714301514,{"id":549,"t":550,"points":38,"comments":33},"40203680",1714422226,{"id":552,"t":553,"points":343,"comments":41},"40263615",1714903798,{"id":555,"t":556,"points":244,"comments":33},"40553767",1717333409,{"id":4,"t":12,"points":14,"comments":15},{"id":559,"t":560,"points":244,"comments":33},"47210945",1772401086,{"id":562,"t":563,"points":244,"comments":33},"48100177",1778530933,[565,570,575,580,585,588],{"id":566,"name":567,"points":568,"category":569},"41254740","sqlean-time",274,"data",{"id":571,"name":572,"points":573,"category":574},"41291700","Srcbook",267,"dev_tools",{"id":576,"name":577,"points":578,"category":579},"41307815","Handwriter.ttf",191,"creative",{"id":581,"name":582,"points":583,"category":584},"41252731","polyfight.io",171,"games",{"id":586,"name":587,"points":583,"category":579},"41271014","Blender shortcuts",{"id":589,"name":590,"points":591,"category":592},"41311489","Permify 1.0",154,"infrastructure",false,"v2",{"id":596,"name":5,"line":6,"ai":593,"cat":20,"year":13,"points":14,"comments":15,"rec":28,"s":597,"top":18,"kw":598,"maker":11,"domain":10,"tier":19,"attempts":205,"strip":599,"url":9,"mix":600,"praise":89,"objection":116,"sub":21},41284703,-0.07,[24,25,26,27],"1133010530015053100500300000031000150100",[31,32,33,34],1793991098242]