[{"data":1,"prerenderedAt":168},["ShallowReactive",2],{"project-40856544":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":28,"sentiment":29,"stance_mix":30,"scored_comments":32,"aspects":33,"strip":35,"post_text":36,"what":37,"said":82,"maker_in_thread":107,"shape":117,"grid":120,"attempts":131,"same_week":132,"maker_shows":14,"maker_hidden":161,"reader_version":162,"card":163},"40856544","Copy from URL","A tiny tool to bypass robots.txt for AI chatbot","title","Show HN: Copy from URL – A tiny tool to bypass robots.txt for AI chatbot","https:\u002F\u002FCopyFromURL.com\u002Fen","copyfromurl.com","ngkw",1719927572,2024,2,6,3379,3515,97,"shown","ai","ai chatbot web content extraction","web_app",[24,25,26,27],"robots.txt bypass","chatbot web content","copy webpage text","extract text from url","quiet",null,{"positive":31,"neutral":31,"mixed":31,"critical":32},0,1,{"idea":34},{"positive":31,"critical":32},[31,31,32,31,31,31],"Hello HN! I'd like to share a very small project I've built called \"Copy from URL\" (https:\u002F\u002Fcopyurl.io). It's a minimal tool designed to solve a common frustration when working with AI chatbots like ChatGPT or Claude.\n\nThe Problem: Many of us need to copy website content for AI queries. However, popular sites often use robots.txt to block AI crawlers, resulting in errors when trying to use these sites with AI. The Solution: Copy from URL is a simple, no-frills web tool that lets you extract text content from any webpage, bypassing robots.txt restrictions. Here's all it does:\n\n- You enter a URL",{"problem":38,"audience":44,"pricing":51,"motivation":55,"time_to_build":29,"stack":62,"openness":79,"stage":80,"team_size":81},{"value":39,"source":40,"evidence":41},"Websites block AI crawlers via robots.txt, so chatbots like ChatGPT or Claude cannot read their content","maker",[42],{"id":43},"post",{"value":45,"source":40,"evidence":46},"People who copy website content into AI chatbots",[47,48],{"id":43},{"id":49,"by":11,"text":50},"40861708","Thanks for your comment, JSDevOps! I appreciate your input. I understand that robots.txt can be bypassed by some. The idea behind \"Copy from URL\" is to address a specific need…",{"value":52,"source":40,"evidence":53},"Completely free",[54],{"id":43},{"value":56,"source":40,"evidence":57},"Personal necessity: frustration with AI chatbots being unable to open certain links",[58,59],{"id":43},{"id":60,"by":11,"text":61},"40861718","To be honest, I'm really frustrated that OpenAI and Anthropic don't open certain links. It's not just because of robots.txt restrictions, but also due to copyright issues and…",[63,67,71,75],{"value":64,"source":40,"evidence":65},"Next.js",[66],{"id":43},{"value":68,"source":40,"evidence":69},"React",[70],{"id":43},{"value":72,"source":40,"evidence":73},"Tailwind CSS",[74],{"id":43},{"value":76,"source":40,"evidence":77},"Cheerio",[78],{"id":43},"open_source","launched","unknown",{"praise":83,"objections":84,"requests":100,"questions":101,"alternatives":106},[],[85,93],{"value":86,"source":87,"evidence":88},"Robots.txt is easily ignored anyway, so the tool adds little","commenter",[89],{"id":90,"by":91,"text":92},"40856761","JSDevOps","Or you know you can just ignore robots.txt like everyone else. What do you think robots.txt does exactly?",{"value":94,"source":87,"evidence":95},"Bypassing restrictions is seen as disrespecting the rule of law",[96],{"id":97,"by":98,"text":99},"40867227","Juliate","Aka \"isn't it annoying to respect the rule of law?\" Reminded of this post by https:\u002F\u002Ftwitter.com\u002FGergelyOrosz\u002Fstatus\u002F1808515221817885017",[],[102],{"value":103,"source":87,"evidence":104},"What does robots.txt actually do, given it can be ignored",[105],{"id":90,"by":91,"text":92},[],{"comments":108,"answered":14,"key":109},4,[110,113,116],{"id":111,"text":112},"40856553","Github codes: https:\u002F\u002Fgithub.com\u002FAmunzen\u002FCopyFromURL Any comment is welcome! Thanks.",{"id":114,"text":115},"40862269","haha I have mistaken the URL in the comments. Here's the right one: https:\u002F\u002Fcopyfromurl.com",{"id":49,"text":50},{"commenters":118,"max_depth":118,"maker_replies":108,"first_comment_mins":32,"lifespan_hours":119},3,26,[121,123,124,126,128,129],[111,31,31,122,11,32],"project\u002Fneutral\u002Fdocs",[114,32,31,122,11,32],[90,14,32,125,91,31],"project\u002Fcritical\u002Fidea",[49,14,31,127,11,32],"project\u002Fneutral\u002Fidea",[60,14,31,127,11,32],[97,14,31,130,98,31],"commenter\u002Fcritical\u002Fidea",[],[133,138,143,148,153,157],{"id":134,"name":135,"points":136,"category":137},"40843867","Pikimov",1150,"creative",{"id":139,"name":140,"points":141,"category":142},"40837610","SmoothMQ",656,"infrastructure",{"id":144,"name":145,"points":146,"category":147},"40832146","Docmost",551,"productivity",{"id":149,"name":150,"points":151,"category":152},"40869877","Xcapture-BPF",447,"dev_tools",{"id":154,"name":155,"points":156,"category":152},"40847699","Doggo",400,{"id":158,"name":159,"points":160,"category":147},"40846242","Edna",331,false,"v2",{"id":164,"name":5,"line":6,"ai":161,"cat":20,"year":13,"points":14,"comments":15,"rec":28,"s":29,"top":18,"kw":165,"maker":11,"domain":10,"tier":19,"attempts":31,"strip":166,"url":9,"mix":167,"praise":-1,"objection":86,"sub":21},40856544,[24,25,26,27],"001000",[31,31,31,32],1793991094845]