[{"data":1,"prerenderedAt":655},["ShallowReactive",2],{"project-35191303":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":28,"sentiment":29,"stance_mix":30,"scored_comments":35,"aspects":36,"strip":47,"post_text":49,"what":50,"said":78,"maker_in_thread":202,"shape":208,"grid":212,"attempts":618,"same_week":619,"maker_shows":42,"maker_hidden":647,"reader_version":648,"card":649},"35191303","GPT Repo Loader","load entire code repos into GPT prompts","title","Show HN: GPT Repo Loader – load entire code repos into GPT prompts","https:\u002F\u002Fgithub.com\u002Fmpoon\u002Fgpt-repository-loader","github.com","mpoon",1679014794,2023,373,149,93,3109,3,"legend","dev_tools","ai coding context","cli",[24,25,26,27],"gpt-4","code context","repository prompt","llm coding","curious",-0.177,{"positive":31,"neutral":32,"mixed":33,"critical":34},10,28,0,17,55,{"idea":37,"performance":39,"pricing":41,"execution":43,"privacy":45,"docs":46},{"positive":18,"critical":38},7,{"positive":33,"critical":40},2,{"positive":33,"critical":42},1,{"positive":18,"critical":44},4,{"positive":33,"critical":40},{"positive":33,"critical":33},[18,42,33,33,33,33,18,18,42,44,48,33,33,48,33,33,33,33,33,18,18,40,18,18,48,33,42,18,18,18,42,18,18,18,42,42,33,33,40,18],5,"I was getting tired of copy\u002Fpasting reams of code into GPT-4 to give it context before I asked it to help me, so I started this small tool. In a nutshell, gpt-repository-loader will spit out file paths and file contents in a prompt-friendly format. You can also use .gptignore to ignore files\u002Ffolders that are irrelevant to your prompt.\n\ngpt-repository-loader as-is works pretty well in helping me achieve better responses. Eventually, I thought it would be cute to load itself into GPT-4 and have GPT-4 improve it. I was honestly surprised by PR#17. GPT-4 was able to write a valid an example repo a",{"problem":51,"audience":57,"pricing":57,"motivation":58,"time_to_build":57,"stack":62,"openness":75,"stage":76,"team_size":77},{"value":52,"source":53,"evidence":54},"Copy-pasting large codebases into GPT-4 to give context before asking for help is tedious.","maker",[55],{"id":56},"post",null,{"value":59,"source":53,"evidence":60},"Maker was tired of manually copy-pasting code into GPT-4 for context.",[61],{"id":56},[63],{"value":64,"source":65,"evidence":66},"Python","inferred",[67,71],{"id":68,"by":69,"text":70},"35193965","kanyethegreat","nice. general question: how many lines of code (at 120 char col len) could you send in one prompt? also, the entire thing is literally 60 lines of python. sometimes i don't get…",{"id":72,"by":73,"text":74},"35193131","swyx","61 LOC for implementation, 42 LOC for tests. this repo currently has more HN upvotes than LOC. must be great code!!","open_source","prototype","unknown",{"praise":79,"objections":92,"requests":136,"questions":159,"alternatives":175},[80],{"value":81,"source":82,"evidence":83},"Useful tool and nice idea","commenter",[84,88],{"id":85,"by":86,"text":87},"35192583","yangjunyu","Seems to be very useful! Thanks for making it!",{"id":89,"by":90,"text":91},"35191700","fire","now that's a name I haven't seen in years, hi mpoon! also this is slick as hell",[93,107,118,129],{"value":94,"source":82,"evidence":95},"Token limits make whole-repo loading impractical for real projects",[96,100,104],{"id":97,"by":98,"text":99},"35199419","joenot443","So I just tried it out. The output.txt which was generated was... 375mb of mostly binary junk. I'm a lazy lark and have lots of nonsense in this repo which I shouldn't, but I was…",{"id":101,"by":102,"text":103},"35191544","textninja","I’m curious to see how this turns out as well, though you’ll probably have to devise a workaround for the token limit for this to be effective for all but the smallest projects.",{"id":105,"by":11,"text":106},"35191545","I'm waiting on my GPT-4 API access so I can use gpt-4-32k which maybe can soak up 10k LOC? Clearly this will break eventually, but I am playing around with some ideas to extend…",{"value":108,"source":82,"evidence":109},"Sending code to OpenAI raises privacy and commercial IP concerns",[110,114],{"id":111,"by":112,"text":113},"35192194","tflinton","Before anyone working on commercial code bases thinks to use this, stop. Uploaded code becomes part of OpenAI.",{"id":115,"by":116,"text":117},"35192112","hackernewds","isn't this a massive privacy violation? any employer would most likely not be okay with this.",{"value":119,"source":82,"evidence":120},"API cost per call is very high",[121,125],{"id":122,"by":123,"text":124},"35192397","wokwokwok","Just remember the API charge is 6c per input token [1]. If you push 32k input tokens in, you're looking at $2000 per API call just as input. You... might wanna consider a self…",{"id":126,"by":127,"text":128},"35192414","dangond","6c per thousand tokens, so $2 per maxed out API call",{"value":130,"source":82,"evidence":131},"Slow for interactive use",[132],{"id":133,"by":134,"text":135},"35192180","alooPotato","It's just so slow for the autocompletion use case to do it like that. Ideally, you're never chaining serial requests to the LLM. Even if you do stuff in all the data into a…",[137,152],{"value":138,"source":82,"evidence":139},"Embeddings or retrieval instead of dumping the whole repo into the prompt",[140,144,148],{"id":141,"by":142,"text":143},"35229720","underlines","Im no expert, but wouldn't it make more sense to give the repo-context (structure, source code, PRs, Issues, ...) as embeddings? You could use langchain to generate an embedding…",{"id":145,"by":146,"text":147},"35194717","amrb","This will fail to get the commit history over time, as it's just reads files in the directory. If this was using the git library and…",{"id":149,"by":150,"text":151},"35191468","bebrws","From what I understand this seems useful if you have a model that will accept a large or unlimited number of tokens. I was looking into doing the same thing with ChatGPT and went…",{"value":153,"source":82,"evidence":154},"Let the model pick relevant files first",[155],{"id":156,"by":157,"text":158},"35191663","ftufek","This is awesome, can't wait to get api access to the 32k token model. Rather than this approach of just converting the whole repo to a text file, what I'm thinking is, you can…",[160,167],{"value":161,"source":82,"evidence":162},"How does the output file get used with ChatGPT?",[163],{"id":164,"by":165,"text":166},"35212889","printvoid","Maybe I didn't understand the usage of this but how is this output.txt file that this repo generates to be used and provided as an input to chatgpt. Can someone eloborate this…",{"value":168,"source":82,"evidence":169},"How much text can GPT-4 accept?",[170,174],{"id":171,"by":172,"text":173},"35191470","EGreg","How much text can you feed GPT-4? Our codebase is 1 million lines of code. Can we feed the documentation to it? What are the limits? Is it possible to train it on our data…",{"id":68,"by":69,"text":70},[176,183,188,195],{"value":177,"source":82,"evidence":178},"GitHub Copilot",[179],{"id":180,"by":181,"text":182},"35207312","darshanps","Wouldn't GitHub copilot not solve the repository load problem?",{"value":184,"source":82,"evidence":185},"Embeddings with langchain or ada",[186,187],{"id":141,"by":142,"text":143},{"id":149,"by":150,"text":151},{"value":189,"source":82,"evidence":190},"AutoPR GitHub Action",[191],{"id":192,"by":193,"text":194},"35209882","irgolic","So I have been toying with an AutoPR GitHub Action for a bit but seeing this spurred me to actually put it into a format for y'all to try. It uses GPT-4 to automatically generate…",{"value":196,"source":82,"evidence":197},"Fine-tuning a model on business data",[198],{"id":199,"by":200,"text":201},"35194703","bob1029","Seeing a lot of comments in here about the token limits. Another path you can take is to fine tune a model on your business. Each training item has to fit within the token limit,…",{"comments":40,"answered":42,"key":203},[204,205],{"id":105,"text":106},{"id":206,"text":207},"35191731","Oh hi fire",{"commenters":209,"max_depth":31,"maker_replies":40,"first_comment_mins":210,"lifespan_hours":211},100,18,138,[213,215,217,220,222,224,226,230,233,237,239,242,244,248,252,255,259,261,263,265,267,269,271,274,278,281,283,285,287,290,293,297,300,303,306,309,312,315,319,323,325,327,330,332,335,337,340,343,346,348,351,354,356,360,364,366,369,371,374,377,380,383,386,389,392,394,397,399,401,404,406,409,412,414,416,418,420,423,425,428,430,433,435,437,439,441,444,447,450,453,456,458,461,465,467,469,472,475,478,480,482,485,488,491,495,498,500,502,504,506,509,512,513,515,517,520,522,524,528,530,534,535,537,538,540,543,545,548,550,553,557,560,563,568,572,574,577,580,582,585,589,592,594,596,601,604,606,610,611,615],[156,33,48,214,157,33],"project\u002Fpositive\u002Fidea",[133,33,42,216,134,33],"project\u002Fcritical\u002Fperformance",[218,33,33,219,157,33],"35192342","commenter\u002Fneutral\u002Fnone",[122,33,42,221,123,33],"project\u002Fcritical\u002Fpricing",[126,33,33,223,127,33],"commenter\u002Fneutral\u002Fpricing",[225,33,33,223,123,33],"35192428",[227,33,33,228,229,33],"35193786","other\u002Fneutral\u002Fpricing","holoduke",[231,33,33,223,232,33],"35194022","anonzzzies",[234,33,33,235,236,33],"35194277","commenter\u002Fneutral\u002Fexecution","cornholio",[238,33,33,235,232,33],"35194307",[240,33,33,219,241,33],"35197643","sharemywin",[243,33,33,219,241,33],"35197508",[245,33,33,246,247,33],"35198102","other\u002Fneutral\u002Fnone","samstave",[249,33,33,250,251,33],"35193533","other\u002Fneutral\u002Fidea","visarga",[253,33,33,246,254,33],"35194924","marginalia_nu",[256,33,33,257,258,33],"35195091","commenter\u002Fneutral\u002Fidea","dilap",[260,33,33,257,254,33],"35195112",[262,33,33,257,258,33],"35196134",[264,33,33,257,254,33],"35196432",[266,33,33,257,258,33],"35198501",[268,33,33,257,254,33],"35198633",[270,33,33,257,258,33],"35199455",[272,33,33,246,273,33],"35196567","victorbjorklund",[275,33,18,276,277,33],"35194395","project\u002Fneutral\u002Fidea","groestl",[279,33,18,276,280,33],"35195931","michaelmior",[282,33,33,246,247,33],"35198066",[284,33,33,246,277,33],"35199083",[286,33,33,246,247,33],"35204286",[288,33,18,276,289,33],"35194907","wongarsu",[291,33,18,276,292,33],"35195289","capableweb",[294,33,42,295,296,33],"35196717","project\u002Fcritical\u002Fidea","wantsanagent",[298,42,33,246,299,33],"35191830","xwdv",[301,42,33,246,302,33],"35191889","tobiasSoftware",[304,42,33,246,305,33],"35192105","antibasilisk",[307,42,42,308,299,33],"35192338","project\u002Fcritical\u002Fexecution",[310,42,48,214,311,33],"35192635","UncleEntity",[313,42,48,214,314,33],"35193391","ozfive",[316,42,48,317,318,33],"35193061","project\u002Fpositive\u002Fnone","osigurdson",[320,42,48,321,322,33],"35193571","project\u002Fpositive\u002Fexecution","zenlikethat",[324,42,33,246,318,33],"35193050",[326,42,33,219,322,33],"35193547",[328,42,33,246,329,33],"35194625","tablatom",[331,42,33,246,318,33],"35195790",[333,42,33,246,334,33],"35193705","tlrobinson",[336,42,33,219,318,33],"35195658",[338,42,33,246,339,33],"35193242","hackerlight",[341,42,33,246,342,33],"35191941","Bjartr",[344,42,33,246,345,33],"35192632","greesil",[347,42,48,321,318,33],"35193017",[349,42,48,321,350,33],"35193216","SparkyMcUnicorn",[352,42,33,246,353,33],"35194364","serpix",[355,42,33,219,345,33],"35221421",[357,42,33,358,359,33],"35192394","other\u002Fcritical\u002Fnone","layer8",[361,42,33,362,363,33],"35193387","other\u002Fcritical\u002Fidea","ChatGTP",[365,42,33,250,322,33],"35193585",[367,42,33,368,363,33],"35195531","other\u002Fcritical\u002Fprivacy",[370,42,33,358,299,33],"35199486",[372,42,33,250,373,33],"35195119","amq",[375,42,33,250,376,33],"35199979","thingification",[378,40,33,246,379,33],"35192603","jerpint",[381,40,33,219,382,33],"35192758","teaearlgraycold",[384,40,33,219,385,33],"35192836","EMIRELADERO",[387,40,33,219,388,33],"35192982","zarzavat",[390,40,33,219,391,33],"35193287","yangff",[393,40,33,219,251,33],"35193524",[395,40,33,219,396,33],"35194016","thomasahle",[398,40,33,219,236,33],"35194335",[400,40,33,219,241,33],"35197687",[402,40,33,219,403,33],"35195536","PartiallyTyped",[405,40,33,219,385,33],"35197195",[407,40,33,246,408,33],"35192931","itsnotlupus",[410,18,18,276,411,33],"35191526","DaiPlusPlus",[413,18,33,246,157,33],"35191607",[415,18,33,246,69,33],"35194005",[417,18,33,219,241,33],"35197702",[419,18,33,246,102,33],"35191620",[421,18,33,246,422,33],"35192543","abecedarius",[424,18,18,276,241,33],"35197729",[426,44,42,308,427,33],"35193687","adltereturn",[171,48,18,429,172,33],"project\u002Fneutral\u002Fperformance",[431,48,18,429,432,33],"35191515","macawfish",[105,48,33,434,11,42],"maker\u002Fneutral\u002Fperformance",[436,48,18,429,342,33],"35191958",[438,48,33,246,314,33],"35193438",[440,48,33,246,329,33],"35194596",[442,48,33,246,443,33],"35195848","Olphs",[445,48,18,429,446,33],"35192159","dc-programmer",[448,48,18,276,449,33],"35192388","WXLCKNO",[451,48,18,276,452,33],"35192185","lukasb",[454,48,18,276,455,33],"35192751","nico",[457,48,18,276,379,33],"35192788",[459,48,48,317,460,33],"35193835","freezed88",[462,463,33,246,464,33],"35192011",6,"waynenilsen",[466,463,33,246,359,33],"35192327",[468,463,33,246,449,33],"35192366",[470,463,33,246,471,33],"35193198","exit",[473,463,33,246,474,33],"35193412","stubybubs",[476,463,33,246,477,33],"35193415","ajmurmann",[479,463,33,246,464,33],"35196483",[72,38,42,481,73,33],"project\u002Fcritical\u002Fnone",[483,38,42,295,484,33],"35193148","wahnfrieden",[486,38,18,276,487,33],"35193241","jeremy_k",[489,38,18,276,490,33],"35195769","VadimPR",[492,493,33,246,494,33],"35193373",8,"m3kw9",[496,493,33,246,497,33],"35193622","srcreigh",[499,493,33,246,232,33],"35194055",[501,493,33,246,247,33],"35198183",[149,503,18,276,150,33],9,[505,503,18,429,379,33],"35192767",[507,503,18,429,508,33],"35194733","inciampati",[510,503,33,219,511,33],"35196980","sp332",[192,31,33,246,193,33],[68,514,42,295,69,33],11,[516,514,42,295,232,33],"35194062",[518,514,42,308,519,33],"35194308","cjbprime",[97,521,42,216,98,33],12,[141,523,18,276,142,33],13,[525,526,18,276,527,33],"35265386",14,"j0hannes",[101,529,18,429,102,33],15,[531,532,33,246,533,33],"35229823",16,"andreyvit",[145,34,42,308,146,33],[89,210,48,536,90,33],"maker\u002Fpositive\u002Fnone",[206,210,33,219,11,42],[539,210,18,276,90,33],"35202036",[541,210,33,246,542,33],"35192174","DANmode",[199,544,33,250,200,33],19,[546,544,33,219,547,33],"35194743","mungoman2",[549,544,33,219,200,33],"35197741",[115,551,42,552,116,33],20,"project\u002Fcritical\u002Fprivacy",[554,551,33,555,556,33],"35194600","commenter\u002Fneutral\u002Fprivacy","umanwizard",[558,551,33,555,559,33],"35194613","kolinko",[164,561,18,562,165,33],21,"project\u002Fneutral\u002Fdocs",[564,565,18,566,567,33],"35192222",22,"project\u002Fneutral\u002Fnone","eddsh1994",[569,570,42,295,571,33],"35197931",23,"somid3",[111,573,42,552,112,33],24,[575,573,33,555,576,33],"35192269","hunter2_",[578,573,33,555,579,33],"35192805","TedDoesntTalk",[581,573,33,219,576,33],"35212079",[583,573,33,555,584,33],"35192354","yed",[586,573,33,587,588,33],"35192673","commenter\u002Fcritical\u002Fprivacy","kadoban",[590,573,33,246,591,33],"35194049","Espressosaurus",[593,573,33,555,559,33],"35194622",[595,573,33,555,588,33],"35198747",[597,598,18,599,600,33],"35192391",25,"maker\u002Fneutral\u002Fnone","megablast",[602,598,18,566,603,33],"35192454","xyzzy123",[180,605,42,295,181,33],26,[607,608,42,295,609,33],"35194646",27,"yaantc",[85,32,48,317,86,33],[612,613,33,246,614,33],"35200280",29,"bharathvajg",[616,617,18,566,247,33],"35198006",30,[],[620,625,629,633,637,642],{"id":621,"name":622,"points":623,"category":624},"35184985","Alpaca.cpp",673,"ai",{"id":626,"name":627,"points":628,"category":624},"35256769","simple-llama-finetuner",449,{"id":630,"name":631,"points":632,"category":624},"35258553","ChatLLaMA",402,{"id":634,"name":635,"points":636,"category":20},"35223759","Chatblade",315,{"id":638,"name":639,"points":640,"category":641},"35263542","Zapier Natural Language Actions",235,"infrastructure",{"id":643,"name":644,"points":645,"category":646},"35179871","Dadagrams",232,"games",false,"v2",{"id":650,"name":5,"line":6,"ai":647,"cat":20,"year":13,"points":14,"comments":15,"rec":28,"s":651,"top":18,"kw":652,"maker":11,"domain":10,"tier":19,"attempts":33,"strip":653,"url":9,"mix":654,"praise":81,"objection":94,"sub":21},35191303,-0.18,[24,25,26,27],"3100003314500500000332335013331333110023",[31,32,33,34],1794036632703]