[{"data":1,"prerenderedAt":234},["ShallowReactive",2],{"project-25059440":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":29,"sentiment":30,"stance_mix":31,"scored_comments":36,"aspects":37,"strip":44,"post_text":46,"what":47,"said":65,"maker_in_thread":129,"shape":139,"grid":142,"attempts":197,"same_week":198,"maker_shows":32,"maker_hidden":227,"reader_version":228,"card":229},"25059440","many-requests","Dead easy Python for executing many HTTP requests asynchronously","title","Show HN: Dead easy Python for executing many HTTP requests asynchronously","https:\u002F\u002Fgithub.com\u002Fjoshlk\u002Fmany_requests","github.com","joshlk",1605110616,2020,46,23,1120,3028,37,"shown","languages","python library","library",[24,25,26,27,28],"python","async http","asyncio","trio","concurrent requests","mixed",-0.13,{"positive":32,"neutral":33,"mixed":34,"critical":35},2,7,0,6,15,{"execution":38,"performance":40,"idea":42,"docs":43},{"positive":34,"critical":39},1,{"positive":34,"critical":41},3,{"positive":32,"critical":32},{"positive":34,"critical":34},[34,41,34,39,34,34,34,39,45,41,39,41,34,39,34,39,34,39,45,41,41,41,41],5,null,{"problem":48,"audience":46,"pricing":46,"motivation":46,"time_to_build":46,"stack":55,"openness":64,"stage":64,"team_size":64},{"value":49,"source":50,"evidence":51},"Executing very large numbers of HTTP requests efficiently in Python","maker",[52],{"id":53,"by":11,"text":54},"25059456","Needed when executing 1 million API requests for example. Use the full force of your machine by utilising async IO via the excellent library Trio. It has been tested in the wild…",[56,60],{"value":57,"source":50,"evidence":58},"Python",[59],{"id":53,"by":11,"text":54},{"value":61,"source":50,"evidence":62},"Trio",[63],{"id":53,"by":11,"text":54},"unknown",{"praise":66,"objections":79,"requests":94,"questions":102,"alternatives":117},[67],{"value":68,"source":69,"evidence":70},"Useful for high-volume request workloads","commenter",[71,75],{"id":72,"by":73,"text":74},"25060695","roywiggins","Oh, handy. This would have solved the exact problem I had last month (which I solved with just-good-enough fiddling with async python, which I'm not comfortable in yet)",{"id":76,"by":77,"text":78},"25067373","hansvm","Nice job! Is vanilla Python networking is still dog slow, on the order of 1k requests per second per core on mid-range consumer hardware no matter how you wire it all together?",[80,87],{"value":81,"source":69,"evidence":82},"Python is too slow for heavy load; Go or Java preferred",[83],{"id":84,"by":85,"text":86},"25066419","throwaway189262","Really it's better to use a fast language like Go or Java for huge loads like this. Our typical CRUD Spring backends can handle ~15,000 requests a second with no optimization. I…",{"value":88,"source":69,"evidence":89},"Threads may outperform asyncio for I\u002FO throughput",[90],{"id":91,"by":92,"text":93},"25090802","devxpy","Do you have empirical evidence for this? Even theoretically, threads can perform better because pythom does infact release the Gil for i\u002Fo bound operations and also, not blocking…",[95],{"value":96,"source":69,"evidence":97},"Explain why not use multiprocessing for more cores",[98],{"id":99,"by":100,"text":101},"25061214","nurettin","Why not distribute the concurrent work over multiple processes to get the most out of multiple cores? More event loops, better performance, and maybe more text preprocessing.",[103,110],{"value":104,"source":69,"evidence":105},"Is work distributed across cores or single-threaded",[106],{"id":107,"by":108,"text":109},"25060764","glic3rinu","when you say parallel you mean the work is distributed across multiple cores? e.g. multiple python worker processes. Or is all work running single threaded in a single process…",{"value":111,"source":69,"evidence":112},"Why use this over ThreadPoolExecutor",[113],{"id":114,"by":115,"text":116},"25061048","eatonphil","I'm curious why one would suggest using this over the built-in ThreadPoolExecutor class? Here's a complete, similar example from the Python docs:…",[118,122],{"value":119,"source":69,"evidence":120},"concurrent.futures ThreadPoolExecutor",[121],{"id":114,"by":115,"text":116},{"value":123,"source":69,"evidence":124},"Go with colly",[125],{"id":126,"by":127,"text":128},"25062410","genidoi","Look into Go, specifically the colly pkg. Learnt Go just so I could use this library after wrestling with highly parralelized Python web scraping. The 1k\u002Frequests\u002Fsec\u002Fcore figure…",{"comments":35,"answered":130,"key":131},4,[132,133,136],{"id":53,"text":54},{"id":134,"text":135},"25061184","All running in a single thread - but the work is executed asynchronously. This is preferred for I\u002FO heavy throughput such as many HTTP requests.",{"id":137,"text":138},"25061270","That would be a good idea, but it's non-trivial in Python as multi-processing async support is patchy. If you have to choose multi-processing vs async for HTTP requests, from…",{"commenters":140,"max_depth":33,"maker_replies":35,"first_comment_mins":32,"lifespan_hours":141},14,136,[143,145,147,148,150,152,155,157,159,163,165,167,171,173,175,177,179,182,184,185,188,192,194],[53,34,34,144,11,39],"project\u002Fpositive\u002Fperformance",[107,34,41,146,108,34],"project\u002Fneutral\u002Fexecution",[134,34,34,146,11,39],[99,34,39,149,100,34],"project\u002Fcritical\u002Fexecution",[137,34,34,151,11,39],"project\u002Fpositive\u002Fexecution",[153,34,34,154,100,34],"25061303","commenter\u002Fneutral\u002Fnone",[126,34,34,156,127,34],"other\u002Fneutral\u002Fnone",[84,34,39,158,85,34],"project\u002Fcritical\u002Fperformance",[160,34,45,161,162,34],"25067646","project\u002Fpositive\u002Fidea","rovr138",[76,34,41,164,77,34],"project\u002Fneutral\u002Fperformance",[114,39,39,166,115,34],"project\u002Fcritical\u002Fidea",[168,39,41,169,170,34],"25061153","project\u002Fneutral\u002Fidea","offtop5",[172,39,34,144,11,39],"25061168",[174,39,39,166,115,34],"25061174",[176,39,34,161,11,39],"25061219",[91,39,39,178,92,34],"maker\u002Fcritical\u002Fperformance",[180,39,34,181,11,39],"25109698","maker\u002Fpositive\u002Fperformance",[183,39,39,178,92,34],"25121104",[72,32,45,161,73,34],[186,41,41,169,187,34],"25064263","stevesimmons",[189,130,41,190,191,34],"25061918","project\u002Fneutral\u002Fdocs","texuf",[193,130,41,190,187,34],"25064247",[195,130,41,190,196,34],"25067710","comfydragon",[],[199,204,209,214,218,222],{"id":200,"name":201,"points":202,"category":203},"25059554","22120",537,"infrastructure",{"id":205,"name":206,"points":207,"category":208},"25059133","Etebase",308,"dev_tools",{"id":210,"name":211,"points":212,"category":213},"25013863","Please-unsubscribe.com",218,"productivity",{"id":215,"name":216,"points":217,"category":213},"25023502","Baller ToDo",161,{"id":219,"name":220,"points":221,"category":213},"25058821","The Least Dangerous To-Do List",141,{"id":223,"name":224,"points":225,"category":226},"25004554","Sir",115,"data",false,"v2",{"id":230,"name":5,"line":6,"ai":227,"cat":20,"year":13,"points":14,"comments":15,"rec":29,"s":30,"top":18,"kw":231,"maker":11,"domain":10,"tier":19,"attempts":34,"strip":232,"url":9,"mix":233,"praise":68,"objection":81,"sub":21},25059440,[24,25,26,27,28],"03010001531301010153333",[32,33,34,35],1793991018751]