[{"data":1,"prerenderedAt":410},["ShallowReactive",2],{"project-40913736":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":28,"sentiment":29,"stance_mix":30,"scored_comments":34,"aspects":35,"strip":45,"post_text":47,"what":48,"said":103,"maker_in_thread":206,"shape":217,"grid":222,"attempts":373,"same_week":374,"maker_shows":319,"maker_hidden":402,"reader_version":403,"card":404},"40913736","Crawlee for Python","a web scraping and browser automation library","title","Show HN: Crawlee for Python – a web scraping and browser automation library","https:\u002F\u002Fcrawlee.dev\u002Fpython\u002F","crawlee.dev","jancurn",1720513838,2024,254,57,289,3515,9,"legend","dev_tools","web scraping library","library",[24,25,26,27],"web scraping","browser automation","playwright","python crawler","mixed",0.078,{"positive":18,"neutral":31,"mixed":32,"critical":33},15,1,7,32,{"docs":36,"privacy":38,"idea":40,"performance":41,"design":43,"pricing":44},{"positive":37,"critical":37},2,{"positive":39,"critical":39},0,{"positive":39,"critical":37},{"positive":42,"critical":39},3,{"positive":37,"critical":42},{"positive":37,"critical":39},[32,42,32,39,42,32,46,46,42,42,39,32,32,46,42,42,39,39,39,39,39,39,39,39,39,42,46,46,42,39,46,39,46,42,42,42,32,42,42,39],5,"Hey all,\n\nThis is Jan, the founder of Apify (https:\u002F\u002Fapify.com\u002F) — a full-stack web scraping platform. After the success of Crawlee for JavaScript (https:\u002F\u002Fgithub.com\u002Fapify\u002Fcrawlee\u002F) and the demand from the Python community, we're launching Crawlee for Python today!\n\nThe main features are:\n\n- A unified programming interface for both HTTP (HTTPX with BeautifulSoup) & headless browser crawling (Playwright)\n\n- Automatic parallel crawling based on available system resources\n\n- Written in Python with type hints for enhanced developer experience\n\n- Automatic retries on errors or when you’re getting ",{"problem":49,"audience":58,"pricing":64,"motivation":72,"time_to_build":76,"stack":77,"openness":100,"stage":101,"team_size":102},{"value":50,"source":51,"evidence":52},"Building web scrapers and crawlers requires reinventing crawling plumbing such as retries, proxies, and parallelism","maker",[53,55],{"id":54},"post",{"id":56,"by":11,"text":57},"40914278","The main advantage (for now) is that the library has a single interface for both HTTP and headless browsers, and bundled auto scaling. You can write your crawlers using the same…",{"value":59,"source":60,"evidence":61},"Python developers who build web scrapers and crawlers","inferred",[62,63],{"id":54},{"id":56,"by":11,"text":57},{"value":65,"source":66,"evidence":67},"Free and open source, will always be free to use","commenter",[68],{"id":69,"by":70,"text":71},"40915457","sauain","Crawlee is open source and free to use and we don't have any plans to monetize it in future. It will be always free to use. We provide Apify platform to publish your scrapers as…",{"value":73,"source":51,"evidence":74},"Demand from the Python community after the JavaScript version's success",[75],{"id":54},null,[78,82,86,90,94],{"value":79,"source":51,"evidence":80},"Python",[81],{"id":54},{"value":83,"source":51,"evidence":84},"HTTPX",[85],{"id":54},{"value":87,"source":51,"evidence":88},"BeautifulSoup",[89],{"id":54},{"value":91,"source":51,"evidence":92},"Playwright",[93],{"id":54},{"value":95,"source":66,"evidence":96},"asyncio",[97],{"id":98,"by":70,"text":99},"40914659","hey intev, - Crawlee has out-of-the-box support for headless browser crawling (Playwright). You don't have to install any plugin or set up the middleware. - Crawlee has a…","open_source","launched","team",{"praise":104,"objections":124,"requests":145,"questions":161,"alternatives":180},[105,117],{"value":106,"source":66,"evidence":107},"Clean, minimal API that is easy to start with",[108,109,113],{"id":98,"by":70,"text":99},{"id":110,"by":111,"text":112},"40924155","mnmkng","Crawlee isn’t any less configurable than Scrapy. It just uses different, in my personal opinion more approachable, patterns. It makes it easier to start with, but you can tweak…",{"id":114,"by":115,"text":116},"40916771","ranedk","I found crawlee a few days ago while figuring out a stack for a project. I wanted a python library but found crawlee with typescript so much easier that I ended up coding the…",{"value":118,"source":66,"evidence":119},"Works well for JavaScript-rendered sites via Playwright",[120,121],{"id":114,"by":115,"text":116},{"id":122,"by":111,"text":123},"40919024","It uses Playwright under the hood, so yes, it can do all of that, and more.",[125,136],{"value":126,"source":66,"evidence":127},"Python documentation is incomplete; some features like tiered proxies are undocumented",[128,132],{"id":129,"by":130,"text":131},"40918032","mdaniel","You'll want to prioritize documenting the existing features, since it's no good having a super awesome full stack web scraping platform if only you can use it. I ordinarily would…",{"id":133,"by":134,"text":135},"40919056","bn-l","I literally had to go through the entire codebase the documentation is that lacking. It’s boring to document but imo it’s the lowest hanging fruit to get people moving down that…",{"value":137,"source":66,"evidence":138},"Less flexible than middleware-based frameworks like Scrapy for custom extensions",[139,142],{"id":140,"by":130,"text":141},"40917985","> You don't have to install any plugin or set up the middleware. That cuts both ways, in true 80\u002F20 fashion: it also means that anyone who isn't on the happy path of the way that…",{"id":143,"by":130,"text":144},"40928731","> Crawlee isn’t any less configurable than Scrapy. Oh, then I have obviously overlooked how one would be able to determined if a proxy has been blocked and evict it from the pool…",[146,153],{"value":147,"source":66,"evidence":148},"Robots.txt and opt-out protocol support",[149],{"id":150,"by":151,"text":152},"40917468","Findecanor","Does it have support for web scraping opt-out protocols, such as Robots.txt, HTTP and content tags? These are getting more important now, especially in the EU after the DSM…",{"value":154,"source":66,"evidence":155},"More code examples and documentation",[156,160],{"id":157,"by":158,"text":159},"40919331","fforflo","I'd suggest bringing more code snippets from the test cases to documentation as examples. Nice work though.",{"id":129,"by":130,"text":131},[162,169],{"value":163,"source":66,"evidence":164},"How does it differ from Scrapy",[165],{"id":166,"by":167,"text":168},"40914574","intev","How is this different from Scrapy?",{"value":170,"source":66,"evidence":171},"Whether it supports SSO-protected intranet sites and auto-login",[172,176],{"id":173,"by":174,"text":175},"40918487","manishsharan","Can this work on intranet sites like sharepoint or confluence , which require employee SSO ? I was trying to build a small Langchain based RAG based on internal documents but…",{"id":177,"by":178,"text":179},"40919350","bmitc","Can you use this to auto-logon to systems?",[181,185,192,199],{"value":182,"source":66,"evidence":183},"Scrapy",[184],{"id":166,"by":167,"text":168},{"value":186,"source":66,"evidence":187},"Selenium",[188],{"id":189,"by":190,"text":191},"40922193","3abiton","How does it compare to selenium for python?",{"value":193,"source":66,"evidence":194},"PhantomJS",[195],{"id":196,"by":197,"text":198},"40918740","holoduke","Does it have event listeners to wait for specific elements based on certain pattern matches. One reason i am still using phantomjs is because it simulates the entire browser and…",{"value":200,"source":66,"evidence":201},"Golang colly",[202],{"id":203,"by":204,"text":205},"40930346","8organicbits","Cache management is also what I had in mind. Ive been using golang+colly and the default caching behavior is just different enough from what I need. I haven't written a custom…",{"comments":207,"answered":208,"key":209},6,4,[210,213,216],{"id":211,"text":212},"40917919","Not yet, but it’s on the roadmap",{"id":214,"text":215},"40917423","Thank you!",{"id":56,"text":57},{"commenters":218,"max_depth":219,"maker_replies":207,"first_comment_mins":220,"lifespan_hours":221},34,10,84,55,[223,225,228,229,233,236,238,240,244,248,251,253,255,257,259,261,263,264,265,267,270,272,275,277,281,284,287,291,295,298,300,303,307,309,312,315,317,322,325,327,328,330,332,334,335,339,344,346,349,351,352,354,358,361,363,365,369],[129,39,32,224,130,39],"project\u002Fcritical\u002Fdocs",[226,39,42,227,111,39],"40918791","maker\u002Fneutral\u002Fdocs",[133,39,32,224,134,39],[230,39,39,231,232,39],"40933936","commenter\u002Fneutral\u002Fdesign","NoThisIsMe",[234,39,39,231,235,39],"40934333","ttymck",[150,32,42,237,151,39],"project\u002Fneutral\u002Fprivacy",[211,32,39,239,11,32],"maker\u002Fneutral\u002Fprivacy",[241,37,32,242,243,39],"40925560","project\u002Fcritical\u002Fidea","nobodywillobsrv",[245,42,46,246,247,39],"40937540","project\u002Fpositive\u002Fperformance","renegat0x0",[249,208,46,246,250,39],"40915166","c0brac0bra",[214,208,39,252,11,32],"maker\u002Fpositive\u002Fnone",[166,46,42,254,167,39],"project\u002Fneutral\u002Fidea",[98,46,46,256,70,39],"maker\u002Fpositive\u002Fdesign",[140,46,32,258,130,39],"project\u002Fcritical\u002Fdesign",[260,46,39,231,204,39],"40922731",[110,46,39,262,111,39],"commenter\u002Fmixed\u002Fdesign",[143,46,32,258,130,39],[203,46,32,258,204,39],[114,207,46,266,115,39],"project\u002Fpositive\u002Fdesign",[268,207,42,269,70,39],"40917088","maker\u002Fneutral\u002Fnone",[189,207,42,271,190,39],"project\u002Fneutral\u002Fnone",[273,33,42,254,274,39],"40914219","marban",[56,33,39,276,11,32],"project\u002Fpositive\u002Fidea",[278,33,39,279,280,39],"40915309","other\u002Fneutral\u002Fnone","robertlagrant",[282,33,39,279,283,39],"40915345","fullspectrumdev",[285,33,39,286,280,39],"40915828","commenter\u002Fneutral\u002Fprivacy",[288,33,39,289,290,39],"40916077","commenter\u002Fneutral\u002Fidea","nlh",[292,33,39,293,294,39],"40916690","commenter\u002Fcritical\u002Fidea","thebytefairy",[296,33,39,289,297,39],"40916909","amarcheschi",[299,33,39,293,280,39],"40917683",[301,33,39,302,11,32],"40917106","maker\u002Fpositive\u002Fidea",[304,33,39,305,306,39],"40918218","other\u002Fneutral\u002Fidea","beeboobaa3",[308,33,39,305,111,39],"40918866",[310,33,39,289,311,39],"40917936","BiteCode_dev",[313,33,39,289,314,39],"40919760","nurettin",[316,33,39,289,280,39],"40919810",[318,319,42,320,321,39],"40915767",8,"project\u002Fneutral\u002Fdocs","VagabundoP",[323,319,46,324,111,39],"40915880","maker\u002Fpositive\u002Fdocs",[157,18,46,326,158,39],"project\u002Fpositive\u002Fdocs",[173,219,42,271,174,39],[329,219,42,271,111,39],"40920508",[331,219,39,271,11,32],"40920588",[196,333,42,271,197,39],11,[122,333,46,246,111,39],[336,337,39,305,338,39],"40918419",12,"barrenko",[340,341,46,342,343,39],"40915318",13,"project\u002Fpositive\u002Fpricing","ijustlovemath",[69,341,46,345,70,39],"maker\u002Fpositive\u002Fpricing",[347,348,42,271,247,39],"40916919",14,[350,348,42,271,70,39],"40917022",[177,31,42,271,178,39],[353,31,39,269,11,32],"40920578",[355,356,32,242,357,39],"40918880",16,"localfirst",[359,356,42,360,111,39],"40919198","project\u002Fmixed\u002Fidea",[362,356,42,271,357,39],"40919489",[364,356,42,271,111,39],"40920448",[366,367,39,305,368,39],"40917174",17,"thelastgallon",[370,367,39,371,372,39],"40917226","commenter\u002Fneutral\u002Fnone","lifesaverluke",[],[375,379,384,388,393,397],{"id":376,"name":377,"points":378,"category":20},"40899242","Satyrn",611,{"id":380,"name":381,"points":382,"category":383},"40902012","Sabby",561,"creative",{"id":385,"name":386,"points":387,"category":20},"40932492","Dut",396,{"id":389,"name":390,"points":391,"category":392},"40916326","BitWHIP",353,"infrastructure",{"id":394,"name":395,"points":396,"category":20},"40904862","Onlook",336,{"id":398,"name":399,"points":400,"category":401},"40881836","Telescope",273,"web_saas",false,"v2",{"id":405,"name":5,"line":6,"ai":402,"cat":20,"year":13,"points":14,"comments":15,"rec":28,"s":406,"top":18,"kw":407,"maker":11,"domain":10,"tier":19,"attempts":39,"strip":408,"url":9,"mix":409,"praise":106,"objection":126,"sub":21},40913736,0.08,[24,25,26,27],"1310315533011533000000000355305053331330",[18,31,32,33],1793991095391]