[{"data":1,"prerenderedAt":300},["ShallowReactive",2],{"project-16161673":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":29,"sentiment":30,"stance_mix":31,"scored_comments":35,"aspects":36,"strip":44,"post_text":46,"what":47,"said":75,"maker_in_thread":187,"shape":193,"grid":197,"attempts":262,"same_week":263,"maker_shows":45,"maker_hidden":292,"reader_version":293,"card":294},"16161673","Bistro","A light-weight column-oriented data processing engine","title","Show HN: Bistro – A light-weight column-oriented data processing engine","https:\u002F\u002Fgithub.com\u002Fasavinov\u002Fbistro","github.com","asavinov",1516132339,2018,95,29,456,2155,22,"notable","data","data processing engine","library",[24,25,26,27,28],"columnar data processing","dataframe","java data engine","column store","linking tables","mixed",-0.126,{"positive":32,"neutral":33,"mixed":34,"critical":33},4,6,1,17,{"idea":37,"performance":39,"execution":41,"docs":42},{"positive":32,"critical":38},2,{"positive":40,"critical":40},0,{"positive":40,"critical":34},{"positive":40,"critical":43},3,[34,40,43,40,43,45,43,40,40,40,43,45,34,40,40,40,34,34,43,40,34,40,45,40,34,43,43,40,45],5,null,{"problem":48,"audience":46,"pricing":46,"motivation":58,"time_to_build":46,"stack":63,"openness":72,"stage":73,"team_size":74},{"value":49,"source":50,"evidence":51},"Set-based joins and group-bys are hard to reason about; defining data processing via column operations is proposed as simpler","maker",[52,55],{"id":53,"by":11,"text":54},"16163056","At the physical (storage) level it is in-memory and organized as a column store, that is, internally it stores a list of tables (with no data) and a list of columns (each being a…",{"id":56,"by":11,"text":57},"16166250","Bistro is not about a physical model and columnar (physical) representation although it relies on it. It is about logical level of representation and processing. There are at…",{"value":59,"source":50,"evidence":60},"Maker believes column-based logical data modeling is easier and less error-prone than set operations",[61,62],{"id":53,"by":11,"text":54},{"id":56,"by":11,"text":57},[64],{"value":65,"source":50,"evidence":66},"Java",[67,68],{"id":53,"by":11,"text":54},{"id":69,"by":70,"text":71},"16166430","agibsonccc","Ok so you're defining new operations on top of existing primitives. Makes sense! The concepts look more interesting than the library right now (it doesn't move the needle for me…","open_source","prototype","unknown",{"praise":76,"objections":90,"requests":120,"questions":139,"alternatives":163},[77],{"value":78,"source":79,"evidence":80},"Ideas around linking and column operations are interesting","commenter",[81,82,86],{"id":69,"by":70,"text":71},{"id":83,"by":84,"text":85},"16170771","munro","I'm working on a columnar Python DSL right now, I think of it like SQLAlchemy for Pandas\u002FSpark\u002FFlink. My goal is to create a language that makes the cumbersome parts of the…",{"id":87,"by":88,"text":89},"16163851","KasianFranks","This is neat. Vectorspace based AI calculations will benefit from this approach. Great work!",[91,105,113],{"value":92,"source":79,"evidence":93},"Not clear what problem it solves or why use it over SQL or pandas",[94,98,101],{"id":95,"by":96,"text":97},"16162740","jitl","An example would be great. Can you show how to do a given task with SQL, map\u002Freduce, and your framework? Because right now I have no idea why I’d choose to learn this new stuff…",{"id":99,"by":70,"text":100},"16165482","The core looks close enough to dataframes that I'd be curious to know how you compare to tablesaw: https:\u002F\u002Fgithub.com\u002Fjtablesaw\u002Ftablesaw This looks neat but I'm not sure why I…",{"id":102,"by":103,"text":104},"16163041","jnordwick","Might new a cool idea, but not nearly fleshed out enough. I think a larger example instead of just individuals lines of code would be useful. Show a toy widget sales spreadsheet.…",{"value":106,"source":79,"evidence":107},"Lacks documentation and examples",[108,112],{"id":109,"by":110,"text":111},"16162802","hoprocker","Agreed -- a motivating example in the Readme would really help people totally new to this concept grasp the how\u002Fwhy of the project.",{"id":102,"by":103,"text":104},{"value":114,"source":79,"evidence":115},"Code style issues",[116],{"id":117,"by":118,"text":119},"16162749","krat0sprakhar","Sorry for being that guy, but I just clicked into a random file in src to read the code, and found the code style (indentation etc.) to be quite weird…",[121,128,135],{"value":122,"source":79,"evidence":123},"Persistence and connectors such as Arrow, Parquet or JDBC",[124,127],{"id":125,"by":70,"text":126},"16166723","The arrow integration and persistence and connecting this to jdbc would be an MVP to me that would at least allow folks to imitate pandas. While you do have new ideas here if…",{"id":53,"by":11,"text":54},{"value":129,"source":79,"evidence":130},"Python DSL or Python API",[131,132],{"id":83,"by":84,"text":85},{"id":133,"by":11,"text":134},"16166694","You are right - it is an MVP, and the goal is to choose a direction. In fact, I am still not sure in which direction to go: * JavaScript data processing framework for in-browser…",{"value":136,"source":79,"evidence":137},"Backend agnostic execution such as Spark or Flink",[138],{"id":69,"by":70,"text":71},[140,147,155],{"value":141,"source":79,"evidence":142},"How does it differ from views or derived columns",[143],{"id":144,"by":145,"text":146},"16164472","nickpeterson","I skimmed the readme but didn't see the answer to what I regard as a basic question. How is this different from a view? I can easily make derived columns based on functions and…",{"value":148,"source":79,"evidence":149},"How does it compare to tablesaw, SAS DATA steps, or Spark",[150,151],{"id":99,"by":70,"text":100},{"id":152,"by":153,"text":154},"16163207","julienfr112","How do that compare to SAS software (https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FSAS_(software)) ? Particularly the \"DATA\" steps.",{"value":156,"source":79,"evidence":157},"Does it support in-memory only, sharding, or time series",[158,162],{"id":159,"by":160,"text":161},"16162886","buremba","Is it in-memory? Does it support replication or sharding? What's the main use-case? How does it differ from ORC, Parquet or Arrow? The repository doesn't have any information.",{"id":102,"by":103,"text":104},[164,168,172,176,180],{"value":165,"source":79,"evidence":166},"tablesaw",[167],{"id":99,"by":70,"text":100},{"value":169,"source":79,"evidence":170},"pandas",[171],{"id":56,"by":11,"text":57},{"value":173,"source":79,"evidence":174},"Spark",[175],{"id":99,"by":70,"text":100},{"value":177,"source":79,"evidence":178},"SAS",[179],{"id":152,"by":153,"text":154},{"value":181,"source":79,"evidence":182},"Vertica",[183],{"id":184,"by":185,"text":186},"16168350","dgudkov","Interesting idea. Columnar ETL can be quite efficient in some scenarios because frequently an ETL transformation (e.g. calculating a new column) effectively modifies an existing…",{"comments":188,"answered":38,"key":189},7,[190,191,192],{"id":56,"text":57},{"id":133,"text":134},{"id":53,"text":54},{"commenters":194,"max_depth":45,"maker_replies":188,"first_comment_mins":195,"lifespan_hours":196},19,90,36,[198,200,202,204,206,207,209,211,212,215,219,222,223,225,229,233,236,238,239,243,246,247,249,251,253,254,255,258,260],[99,40,34,199,70,40],"project\u002Fcritical\u002Fidea",[56,40,40,201,11,34],"project\u002Fneutral\u002Fidea",[69,40,43,203,70,40],"project\u002Fmixed\u002Fperformance",[133,40,40,205,11,34],"maker\u002Fneutral\u002Fidea",[125,40,43,201,70,40],[83,40,45,208,84,40],"project\u002Fpositive\u002Fidea",[159,34,43,210,160,40],"project\u002Fneutral\u002Fperformance",[53,34,40,201,11,34],[213,34,40,214,103,40],"16163541","other\u002Fneutral\u002Fidea",[216,34,40,217,218,40],"16165940","other\u002Fneutral\u002Fdocs","PeCaN",[220,34,43,201,221,40],"16163812","heavenlyblue",[184,38,45,208,185,40],[117,43,34,224,118,40],"project\u002Fcritical\u002Fexecution",[226,43,40,227,228,40],"16163754","other\u002Fneutral\u002Fexecution","jahewson",[230,43,40,231,232,40],"16163944","other\u002Fneutral\u002Fnone","tomcam",[234,43,40,231,235,40],"16165105","jasonkostempski",[95,32,34,237,96,40],"project\u002Fcritical\u002Fdocs",[109,32,34,237,110,40],[240,32,43,241,242,40],"16162870","project\u002Fneutral\u002Fdocs","sgolestane",[244,32,40,245,11,34],"16166355","maker\u002Fneutral\u002Fdocs",[102,45,34,237,103,40],[248,45,40,205,11,34],"16163178",[250,45,45,208,103,40],"16163498",[252,45,40,201,11,34],"16171479",[144,33,34,199,145,40],[152,188,43,201,153,40],[256,188,43,201,257,40],"16172495","philkrylov",[259,188,40,205,11,34],"16175343",[87,261,45,208,88,40],8,[],[264,269,274,279,283,287],{"id":265,"name":266,"points":267,"category":268},"16132502","Bato",159,"languages",{"id":270,"name":271,"points":272,"category":273},"16144746","Languagecrunch",152,"ai",{"id":275,"name":276,"points":277,"category":278},"16166973","Genact",131,"creative",{"id":280,"name":281,"points":282,"category":268},"16138990","SwiftPascalInterpreter",103,{"id":284,"name":285,"points":286,"category":20},"16170892","Datasette Publish",101,{"id":288,"name":289,"points":290,"category":291},"16156723","HardyPress",86,"web_saas",false,"v2",{"id":295,"name":5,"line":6,"ai":292,"cat":20,"year":13,"points":14,"comments":15,"rec":29,"s":296,"top":18,"kw":297,"maker":11,"domain":10,"tier":19,"attempts":40,"strip":298,"url":9,"mix":299,"praise":78,"objection":92,"sub":21},16161673,-0.13,[24,25,26,27,28],"10303530003510001130105013305",[32,33,34,33],1794036562612]