[{"data":1,"prerenderedAt":290},["ShallowReactive",2],{"project-47008163":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":27,"sentiment":28,"stance_mix":29,"scored_comments":33,"aspects":34,"strip":38,"post_text":39,"what":40,"said":63,"maker_in_thread":155,"shape":165,"grid":169,"attempts":252,"same_week":253,"maker_shows":32,"maker_hidden":283,"reader_version":284,"card":285},"47008163","Data Engineering Book","An open source, community-driven guide","title","Show HN: Data Engineering Book – An open source, community-driven guide","https:\u002F\u002Fgithub.com\u002Fdatascale-ai\u002Fdata_engineering_book\u002Fblob\u002Fmain\u002FREADME_en.md","github.com","xx123122",1771018552,2026,251,30,194,4366,5,"legend","learning_science","open source book","other",[24,25,26,21],"data engineering","llm data pipelines","rag pipeline","curious",0,{"positive":30,"neutral":31,"mixed":32,"critical":30},3,7,1,14,{"docs":35,"idea":37},{"positive":36,"critical":30},2,{"positive":28,"critical":28},[18,30,28,28,28,28,28,18,30,28,30,28,28,28,28,28,28,18,32,28,30,30,30,28,30,30,28,32,28,28,32,28,28,28,28],null,{"problem":41,"audience":48,"pricing":39,"motivation":52,"time_to_build":39,"stack":59,"openness":60,"stage":61,"team_size":62},{"value":42,"source":43,"evidence":44},"Learning resources for modern data engineering are fragmented across articles and tutorials","maker",[45],{"id":46,"by":11,"text":47},"47010982","Hi HN, OP here! I'm currently a Master's student at USTC (University of Science and Technology of China). I've been diving deep into Data Engineering, especially in the context…",{"value":49,"source":43,"evidence":50},"Developers learning data engineering, especially for LLMs",[51],{"id":46,"by":11,"text":47},{"value":53,"source":43,"evidence":54},"Open-sourcing learning notes from a semester of study",[55,58],{"id":56,"by":11,"text":57},"47023439","Let me clarify the team structure to avoid any misunderstanding. We are actually three first-year Master's students. This project is indeed a summary of our learning from this…",{"id":46,"by":11,"text":47},[],"open_source","beta","team",{"praise":64,"objections":81,"requests":111,"questions":130,"alternatives":142},[65,77],{"value":66,"source":67,"evidence":68},"High-quality translation and useful later chapters","commenter",[69,73],{"id":70,"by":71,"text":72},"47016588","fudged71","Thank you so much for this book! I'm finding the translation is very high quality. I am a complete novice in training LLMs, and have been trying to train a novel architecture for…",{"id":74,"by":75,"text":76},"47012011","hliyan","I'm not sure whether this is an artefact of translation, but things like this don't inspire confidence: > The \"Modern Data Stack\" (MDS) is a hot concept in data engineering in…",{"value":78,"source":67,"evidence":79},"Good RAG sections",[80],{"id":74,"by":75,"text":76},[82,93,97,104],{"value":83,"source":67,"evidence":84},"Text reads as LLM-generated with fake warmth",[85,89],{"id":86,"by":87,"text":88},"47012848","nimonian","I think it was. It's a wall of information, lots of summary tables, fake warmth, and has that LLM smell to it. I'd be very surprised if this wasn't generated text. Whether it's…",{"id":90,"by":91,"text":92},"47011809","osamabinladen","this is great and i bookmarked it so i can read it later. i’m just curious though, was the readme written by chatgpt? i can’t tell if im paranoid thinking everything is written…",{"value":94,"source":67,"evidence":95},"Modern Data Stack intro is unclear from translation",[96],{"id":74,"by":75,"text":76},{"value":98,"source":67,"evidence":99},"Missing table formats like Delta and Iceberg",[100],{"id":101,"by":102,"text":103},"47012654","alexott","Parquet alone is not for modern data engineering. Delta, Iceberg should be in the list",{"value":105,"source":67,"evidence":106},"Figures were in Chinese in some chapters",[107],{"id":108,"by":109,"text":110},"47010087","guillem_lefait","The figures in the different chapters are in english (it's not the case for the image in README_en.md).",[112,119,126],{"value":113,"source":67,"evidence":114},"Cover table and storage formats like Lance and Vortex",[115],{"id":116,"by":117,"text":118},"47021233","cpard","It's important in a book treating an emerging field (data eng for LLMs) to mention emerging categories related to it such as storage formats purpose built for the full ML…",{"value":120,"source":67,"evidence":121},"Cover hybrid search and reranking",[122],{"id":123,"by":124,"text":125},"47015082","13pixels","The 'Vector DB vs Keyword Search' section caught my eye. In your testing for RAG pipelines, where do you draw the line? We've found keyword search (BM25) often beats semantic…",{"value":127,"source":67,"evidence":128},"Add code-focused datasets and tooling",[129],{"id":70,"by":71,"text":72},[131,135],{"value":132,"source":67,"evidence":133},"Was the readme written by ChatGPT?",[134],{"id":90,"by":91,"text":92},{"value":136,"source":67,"evidence":137},"Why is a Chinese publication on the HN front page?",[138],{"id":139,"by":140,"text":141},"47009154","rafavargascom","谢谢 How is possible a Chinese publication gets to the top in HN?",[143,147,151],{"value":144,"source":67,"evidence":145},"Lance",[146],{"id":116,"by":117,"text":118},{"value":148,"source":67,"evidence":149},"Vortex",[150],{"id":116,"by":117,"text":118},{"value":152,"source":67,"evidence":153},"Delta Lake and Iceberg",[154],{"id":101,"by":102,"text":103},{"comments":33,"answered":156,"key":157},4,[158,161,164],{"id":159,"text":160},"47013061","That's a great point. I completely agree—'Data Engineering for LLMs' is much more accurate given the content.I'll pass this feedback on to the project lead immediately. Thanks…",{"id":162,"text":163},"47023454","I promise there is a human behind the keyboard!QAQ My English is not good, so I use GPT to help translate and polish my replies to be polite. Maybe it made them sound too…",{"id":56,"text":57},{"commenters":166,"max_depth":18,"maker_replies":33,"first_comment_mins":167,"lifespan_hours":168},20,76,40,[170,172,176,178,182,185,186,187,190,192,194,198,202,203,205,207,209,211,212,214,216,217,221,222,224,227,229,231,233,235,239,243,245,247,249],[70,28,18,171,71,28],"project\u002Fpositive\u002Fdocs",[173,32,30,174,175,28],"47011441","project\u002Fneutral\u002Fidea","esafak",[159,32,28,177,11,32],"maker\u002Fneutral\u002Fnone",[179,32,28,180,181,28],"47014510","commenter\u002Fneutral\u002Fnone","dotancohen",[183,32,28,180,184,28],"47016170","gtowey",[162,32,28,177,11,32],[56,32,28,177,11,32],[188,32,18,189,181,28],"47023527","maker\u002Fpositive\u002Fnone",[74,36,30,191,75,28],"project\u002Fmixed\u002Fdocs",[193,36,28,177,11,32],"47013022",[195,30,30,196,197,28],"47008993","project\u002Fneutral\u002Fdocs","joshuaissac",[199,30,28,200,201,28],"47009129","other\u002Fneutral\u002Fnone","dang",[46,30,28,174,11,32],[204,30,28,200,11,32],"47013127",[206,30,28,174,11,32],"47010977",[208,30,28,174,11,32],"47011485",[210,30,28,200,11,32],"47013135",[90,156,18,171,91,28],[86,156,32,213,87,28],"project\u002Fcritical\u002Fdocs",[215,156,28,177,11,32],"47013045",[116,18,30,174,117,28],[218,219,30,196,220,28],"47013678",6,"baalimago",[123,31,30,174,124,28],[223,31,28,177,11,32],"47015276",[225,31,30,174,226,28],"47017273","eshaham78",[108,228,30,196,109,28],8,[230,228,28,177,11,32],"47011659",[101,232,32,213,102,28],9,[234,232,28,177,11,32],"47013116",[236,237,28,200,238,28],"47009886",10,"dvrp",[240,241,32,213,242,28],"47008919",11,"mixmastamyk",[139,244,28,200,140,28],12,[246,244,28,200,140,28],"47009219",[248,244,28,177,11,32],"47011671",[250,244,28,200,251,28],"47014109","heliumtera",[],[254,259,264,268,273,278],{"id":255,"name":256,"points":257,"category":258},"47031334","Jemini",488,"ai",{"id":260,"name":261,"points":262,"category":263},"47051852","AsteroidOS 2.0",470,"hardware",{"id":265,"name":266,"points":267,"category":20},"47026186","Microgpt",282,{"id":269,"name":270,"points":271,"category":272},"47040375","FreeFlow",277,"productivity",{"id":274,"name":275,"points":276,"category":277},"47011567","SQL-tap",233,"dev_tools",{"id":279,"name":280,"points":281,"category":282},"47014500","Sameshi",230,"games",false,"v2",{"id":286,"name":5,"line":6,"ai":283,"cat":20,"year":13,"points":14,"comments":15,"rec":27,"s":28,"top":18,"kw":287,"maker":11,"domain":10,"tier":19,"attempts":28,"strip":288,"url":9,"mix":289,"praise":66,"objection":83,"sub":21},47008163,[24,25,26,21],"53000005303000000510333033010010000",[30,31,32,30],1794036693183]