[{"data":1,"prerenderedAt":655},["ShallowReactive",2],{"project-40456236":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":28,"sentiment":29,"stance_mix":30,"scored_comments":35,"aspects":36,"strip":47,"post_text":50,"what":51,"said":102,"maker_in_thread":214,"shape":225,"grid":230,"attempts":609,"same_week":616,"maker_shows":43,"maker_hidden":647,"reader_version":648,"card":649},"40456236","Dataherald","We open sourced our entire text-to-SQL product","title","Show HN: We open sourced our entire text-to-SQL product","https:\u002F\u002Fgithub.com\u002FDataherald\u002Fdataherald","github.com","aazo11",1716479425,2024,464,144,77,3515,3,"legend","data","text-to-sql engine","library",[24,25,26,27],"text-to-sql","natural language database","llm sql agent","open source","curious",0.058,{"positive":31,"neutral":32,"mixed":33,"critical":34},15,52,1,10,78,{"idea":37,"privacy":40,"performance":41,"execution":42,"docs":44,"design":46},{"positive":38,"critical":39},9,7,{"positive":33,"critical":33},{"positive":33,"critical":33},{"positive":43,"critical":33},2,{"positive":33,"critical":45},0,{"positive":45,"critical":45},[18,45,18,18,48,48,18,43,18,18,45,49,33,48,33,48,18,45,45,49,18,18,49,18,18,48,49,18,43,33,45,33,18,49,33,18,49,18,18,18],5,4,"Long story short: We (Dataherald) just open-sourced our entire codebase, including the core engine, the clients that interact with it and the backend application layer for authentication and RBAC. You can now use the full solution to build text-to-SQL into your product.\n\nThe Problem: modern LLMs write syntactically correct SQL, but they struggle with real-world relational data. This is because real world data and schema is messy, natural language can often be ambiguous and LLMs are not trained on your specific dataset.\n\nSolution: The core NL-to-SQL engine in Dataherald is an LLM based agent wh",{"problem":52,"audience":58,"pricing":66,"motivation":67,"time_to_build":66,"stack":71,"openness":99,"stage":100,"team_size":101},{"value":53,"source":54,"evidence":55},"LLMs write valid SQL but struggle with messy real-world relational schemas and ambiguous questions","maker",[56],{"id":57},"post",{"value":59,"source":60,"evidence":61},"Developers who want to embed text-to-SQL into their own products","commenter",[62],{"id":63,"by":64,"text":65},"40470303","saigal","the target audience is developers who wish to embed text to SQL functionality into their own products. the target audience is less the 'internal use case' (i.e. a data analyst)…",null,{"value":68,"source":54,"evidence":69},"Open sourced the entire codebase so others can build text-to-SQL into their products",[70],{"id":57},[72,79,83,89,93],{"value":73,"source":74,"evidence":75},"Python","inferred",[76],{"id":77,"by":11,"text":78},"40472162","Tables, columns and views are scanned at configuration time (or based on an API trigger) and stored in the data store and a vector store, not on every run. They are then…",{"value":80,"source":54,"evidence":81},"NextJS",[82],{"id":57},{"value":84,"source":54,"evidence":85},"OpenAI LLMs",[86],{"id":87,"by":11,"text":88},"40470143","The agent is LLM agnostic and you can use it with OpenAI or self-hosted LLMs. For self hosted LLM we have benchmarked performance with Mixtral for tool selection and CodeLlama…",{"value":90,"source":54,"evidence":91},"Mixtral and CodeLlama for self-hosted models",[92],{"id":87,"by":11,"text":88},{"value":94,"source":54,"evidence":95},"Chroma, Pinecone and Astra vector stores",[96],{"id":97,"by":11,"text":98},"40476142","Right now the supported Vector stores are Chroma (which you can self-host), Pinecone and Astra. Adding a new vector store is quite easy: you just need to extend the VectorStore…","open_source","launched","company",{"praise":103,"objections":125,"requests":154,"questions":169,"alternatives":192},[104,114],{"value":105,"source":60,"evidence":106},"Useful for non-technical users and analysts who want first-draft SQL",[107,111],{"id":108,"by":109,"text":110},"40475366","edmundsauto","I think a lot of people want something like this. Especially as more non technical people are adding business analysts to their jd. I’ve tried to teach SQL to PMs, bug triage…",{"id":112,"by":64,"text":113},"40475693","we've encountered a lot of instances when people know SQL but just want a first draft of SQL to expedite the process. we see this a lot from data analysts too.",{"value":115,"source":60,"evidence":116},"Open sourcing is a valuable contribution",[117,121],{"id":118,"by":119,"text":120},"40473852","evan_ry","This is a historical contribution. Thank you for doing it! Basically all the enterprises with a lot of data need to \"chat with their data\" right now. I can't imagine how many…",{"id":122,"by":123,"text":124},"40473633","zurfer","That's one of the more feature rich AI analytics assistants. (1) Kudos for open sourcing. I think it's really difficult to build a business around that, but there are some…",[126,137,143],{"value":127,"source":60,"evidence":128},"Accuracy and correctness are not guaranteed, risking wrong business numbers",[129,133],{"id":130,"by":131,"text":132},"40470673","TheRealPomax","With what level of accuracy? And what guarantee of correctness? Because a report that happens to get the joins wrong once every 1000 reports is going to lead to fun legal…",{"id":134,"by":135,"text":136},"40471765","nicoburns","But who wants text-to-sql in products that they use? You wouldn't be able to trust the results. So what is it useful for? Of course you could learn to check the output. But then…",{"value":138,"source":54,"evidence":139},"Latency of about 20-30 seconds",[140],{"id":141,"by":11,"text":142},"40476099","There are organizations using Dataherald in production right now. The latency is ~20-30s and it takes some set up, so as long as those are not blockers it can be used in prod.",{"value":144,"source":60,"evidence":145},"Security risk when end users run generated queries",[146,150],{"id":147,"by":148,"text":149},"40471165","throwaway115","What guarantees do you offer with query security if I turn this over to an end user? How do I keep them only accessing their own data?",{"id":151,"by":152,"text":153},"40490424","tirumaraiselvan","Why do you think that's more reasonable :) To me it seems a more risky use-case since you don't have control\u002Fobservability over what an untrusted user is asking for?",[155,162],{"value":156,"source":60,"evidence":157},"Support for GraphQL",[158],{"id":159,"by":160,"text":161},"40470969","kwerk","Will it work with GraphQL?",{"value":163,"source":60,"evidence":164},"Better handling of complex joins and outer joins",[165],{"id":166,"by":167,"text":168},"40470345","thedynamicduo","This looks really cool, can't wait to check it out. The problem I've seen with other tools I've tinkered with is that they do well with simple stuff like: \"what are my latest…",[170,174,181],{"value":171,"source":60,"evidence":172},"How does it guarantee query security for end users",[173],{"id":147,"by":148,"text":149},{"value":175,"source":60,"evidence":176},"Whether it can handle complex joins",[177],{"id":178,"by":179,"text":180},"40459291","instabart","Interesting! I am assuming it can do complex joins. Are there any examples of text -> sql it produces? I looked on the website but only saw \"coming soon\"",{"value":182,"source":60,"evidence":183},"Why it was open sourced",[184,188],{"id":185,"by":186,"text":187},"40470506","mholubowski","Hey! Why did you open source it? Genuinely curious.",{"id":189,"by":190,"text":191},"40469790","vlovich123","Are there strategic parts of the stack you haven’t open-sourced?",[193,200,207],{"value":194,"source":60,"evidence":195},"Vanna",[196],{"id":197,"by":198,"text":199},"40475387","zainhoda","Would you be interested in merging with Vanna in some way? You’re ahead of us in terms of interface but we’re ahead of you in terms of adoption (because of specific choices we’ve…",{"value":201,"source":60,"evidence":202},"SQLAI",[203],{"id":204,"by":205,"text":206},"40474717","Kiro","The demand is huge. Accuracy is less important because the alternative is being completely in the dark or wait for a developer to get the data for you. In my experience people…",{"value":208,"source":60,"evidence":209},"Defog sql-eval based tools",[210],{"id":211,"by":212,"text":213},"40478539","westurner","\u002F? awesome \"sql\" llm site:github.com https:\u002F\u002Fwww.google.com\u002Fsearch?q=awesome+%22sql%22+llm+site%3Agithub.com : - awesome-Text2SQL:…",{"comments":215,"answered":216,"key":217},17,13,[218,219,222],{"id":141,"text":142},{"id":220,"text":221},"40476072","Yes when you connect Dataherald to a DB it scans it and you can do exploratory queries.",{"id":223,"text":224},"40472305","While the engine response is not accurate all the time, the engine returns a confidence score. We have never encountered cases where a deployment with necessary training data…",{"commenters":226,"max_depth":227,"maker_replies":215,"first_comment_mins":228,"lifespan_hours":229},80,11,256,175,[231,235,239,242,245,248,251,254,257,260,263,266,269,271,274,277,279,282,284,287,291,293,297,300,303,305,309,310,312,314,317,320,321,324,327,331,334,337,340,343,345,349,351,354,356,357,358,359,361,362,363,365,367,370,373,376,378,381,383,385,388,391,393,395,398,401,403,405,407,408,411,414,418,420,423,428,429,432,435,436,438,442,445,448,450,451,453,455,458,462,465,466,468,471,473,475,479,481,483,487,489,492,494,496,498,501,504,506,509,511,513,515,517,520,524,526,530,532,536,540,542,544,546,550,554,557,559,562,564,568,570,574,577,580,582,585,589,591,595,597,599,601,603,605],[232,45,18,233,234,45],"40469824","maker\u002Fneutral\u002Fidea","bobismyuncle",[236,45,45,237,238,45],"40470346","commenter\u002Fneutral\u002Fnone","robertlagrant",[240,45,45,237,241,45],"40471873","e1g",[243,45,45,237,244,45],"40472179","hackernewds",[246,45,45,237,247,45],"40472254","learnedbytes",[249,45,45,237,250,45],"40474751","tomhallett",[252,45,45,237,253,45],"40474935","throwup238",[255,45,45,237,256,45],"40475338","re-thc",[258,45,45,237,259,45],"40474119","cess11",[261,45,18,233,262,45],"40473110","krawczstef",[264,45,18,265,64,45],"40474667","project\u002Fneutral\u002Fprivacy",[267,45,18,268,109,45],"40475323","maker\u002Fneutral\u002Fprivacy",[270,45,18,265,64,45],"40475335",[272,45,18,273,109,45],"40475701","maker\u002Fneutral\u002Fperformance",[275,45,45,237,276,45],"40476064","threecheese",[141,45,45,278,11,33],"project\u002Fneutral\u002Fperformance",[280,45,48,281,64,45],"40476598","project\u002Fpositive\u002Fprivacy",[166,33,48,283,167,45],"project\u002Fpositive\u002Fperformance",[285,33,48,286,64,45],"40470432","project\u002Fpositive\u002Fexecution",[288,33,18,289,290,45],"40473748","project\u002Fneutral\u002Fidea","BossingAround",[220,33,45,292,11,33],"project\u002Fneutral\u002Fexecution",[294,33,18,295,296,45],"40478010","project\u002Fneutral\u002Fdocs","momothereal",[298,33,18,289,299,45],"40477778","roughly",[301,43,45,237,302,45],"40473919","iloveitaly",[304,43,45,237,64,45],"40474677",[306,18,33,307,308,45],"40470230","project\u002Fcritical\u002Fidea","arrosenberg",[63,18,18,289,64,45],[311,18,18,289,308,45],"40470369",[313,18,18,289,64,45],"40470779",[315,18,18,292,316,45],"40473072","rkuodys",[318,18,18,319,64,45],"40474691","project\u002Fneutral\u002Fnone",[151,18,45,237,152,45],[322,18,18,289,323,45],"40470523","greenavocado",[325,18,45,237,326,45],"40470716","boredemployee",[328,18,45,329,330,45],"40473193","other\u002Fneutral\u002Fnone","threeseed",[332,18,45,329,333,45],"40474268","sfn42",[335,18,45,329,336,45],"40475169","lelanthran",[338,18,45,329,339,45],"40475317","meekaaku",[341,18,45,329,342,45],"40478380","cerved",[344,18,45,237,64,45],"40470815",[346,18,48,347,348,45],"40470994","project\u002Fpositive\u002Fidea","twojacobtwo",[350,18,48,347,64,45],"40471121",[352,18,33,307,353,45],"40471160","skydhash",[355,18,18,289,64,45],"40471198",[134,18,33,307,135,45],[223,18,45,278,11,33],[204,18,48,347,205,45],[360,18,45,237,64,45],"40477207",[108,18,48,347,109,45],[112,18,48,286,64,45],[130,18,33,364,131,45],"project\u002Fcritical\u002Fperformance",[366,18,18,289,64,45],"40470829",[368,18,33,307,369,45],"40471134","chx",[371,18,45,237,372,45],"40471219","warkdarrior",[374,18,45,329,375,45],"40471456","EasyMark",[377,18,48,347,64,45],"40471587",[379,18,18,380,369,45],"40473602","maker\u002Fneutral\u002Fnone",[382,18,18,380,259,45],"40474143",[384,18,18,380,64,45],"40474679",[386,18,45,329,387,45],"40471272","Terr_",[389,18,45,237,390,45],"40471684","altdataseller",[392,18,45,237,64,45],"40472403",[394,18,45,237,64,45],"40472410",[396,18,45,329,397,45],"40476021","ecjhdnc2025",[399,18,45,329,400,45],"40476941","panarky",[402,18,45,237,64,45],"40478256",[404,18,45,329,244,45],"40472199",[406,18,45,329,64,45],"40472436",[178,49,18,319,179,45],[409,49,45,410,11,33],"40470042","maker\u002Fneutral\u002Fexecution",[412,48,48,347,413,45],"40476248","npsimons",[415,416,18,265,417,45],"40470067",6,"fpater",[419,416,45,265,11,33],"40470876",[421,39,18,319,422,45],"40477097","dhanushreddy29",[424,425,18,426,427,45],"40475489",8,"project\u002Fneutral\u002Fdesign","pamelafox",[97,425,45,292,11,33],[430,38,45,329,431,45],"40470022","alchemist1e9",[433,34,18,278,434,45],"40472135","iandanforth",[77,34,45,278,11,33],[122,227,48,437,123,45],"project\u002Fpositive\u002Fnone",[439,440,18,292,441,45],"40473537",12,"DeathArrow",[443,216,18,292,444,45],"40469769","coder543",[446,216,18,292,447,45],"40470014","kasmura",[449,216,45,237,444,45],"40470044",[87,216,45,292,11,33],[452,216,45,292,11,33],"40470135",[454,216,45,237,205,45],"40474789",[456,216,45,329,457,45],"40475737","lmeyerov",[459,460,18,319,461,45],"40474401",14,"jaynpatel",[463,460,18,426,464,45],"40474776","PartiallyTyped",[197,31,18,319,198,45],[118,467,48,347,119,45],16,[469,215,45,329,470,45],"40470827","winphone1974",[472,215,18,289,64,45],"40470838",[474,215,18,289,64,45],"40470874",[476,477,48,347,478,45],"40472373",18,"chenster",[480,477,18,380,64,45],"40472426",[482,477,18,380,64,45],"40472448",[484,485,18,319,486,45],"40471401",19,"RyanHamilton",[488,485,18,380,64,45],"40471413",[490,491,33,307,330,45],"40473177",20,[493,491,45,289,11,33],"40473284",[185,495,18,380,186,45],21,[147,497,18,265,148,45],22,[499,497,18,265,500,45],"40471176","freeone3000",[502,497,45,503,148,45],"40471230","commenter\u002Fneutral\u002Fprivacy",[505,497,45,265,11,33],"40471247",[507,497,33,508,390,45],"40471689","project\u002Fcritical\u002Fexecution",[510,497,45,237,64,45],"40472421",[512,497,45,237,390,45],"40472486",[514,497,45,503,64,45],"40474705",[516,497,45,265,11,33],"40471231",[518,497,33,519,148,45],"40471261","project\u002Fcritical\u002Fprivacy",[521,522,18,289,523,45],"40470794",23,"totalhack",[525,522,45,292,11,33],"40470861",[527,528,18,380,529,45],"40521904",24,"mariarmestre",[531,528,18,380,64,45],"40529543",[533,534,45,329,535,45],"40475786",25,"emmender2",[537,538,48,347,539,45],"40471742",26,"fkm0r0ns",[189,541,18,380,190,45],27,[543,541,45,319,11,33],"40470020",[545,541,18,233,190,45],"40470772",[547,541,45,548,549,45],"40474025","commenter\u002Fneutral\u002Fidea","fhd2",[551,552,33,307,553,45],"40474363",28,"devd00d",[555,552,45,237,556,45],"40474372","badgersnake",[558,552,33,307,64,45],"40474699",[560,552,45,237,561,45],"40474880","yard2010",[563,552,45,237,556,45],"40475573",[565,566,18,295,567,45],"40469899",29,"akch",[569,566,45,295,11,33],"40470013",[571,566,18,572,573,45],"40470482","project\u002Fmixed\u002Fdocs","numlocked",[575,566,18,576,64,45],"40470507","maker\u002Fneutral\u002Fdocs",[578,566,48,579,573,45],"40470540","maker\u002Fpositive\u002Fdocs",[581,566,45,295,11,33],"40471559",[583,566,18,295,584,45],"40470023","threesevenths",[586,566,45,587,588,45],"40470528","commenter\u002Fneutral\u002Fdocs","munk-a",[590,566,45,587,64,45],"40470534",[592,593,18,319,594,45],"40470621",30,"tootie",[596,593,18,319,64,45],"40470734",[159,598,18,319,160,45],31,[600,598,45,319,11,33],"40471013",[602,598,18,319,64,45],"40471217",[211,604,45,329,212,45],32,[606,607,45,329,608,45],"40470279",33,"_hzw",[610,615],{"id":611,"t":612,"points":613,"comments":614},"37240363",1692815936,215,103,{"id":4,"t":12,"points":14,"comments":15},[617,622,627,632,637,642],{"id":618,"name":619,"points":620,"category":621},"40501021","Grooved",452,"hardware",{"id":623,"name":624,"points":625,"category":626},"40455478","PaperTalk",319,"productivity",{"id":628,"name":629,"points":630,"category":631},"40502956","Openkoda",318,"web_saas",{"id":633,"name":634,"points":635,"category":636},"40469592","Spot",309,"dev_tools",{"id":638,"name":639,"points":640,"category":641},"40456959","Porter Cloud",258,"infrastructure",{"id":643,"name":644,"points":645,"category":646},"40481578","DiscoGrad",232,"languages",false,"v2",{"id":650,"name":5,"line":6,"ai":647,"cat":20,"year":13,"points":14,"comments":15,"rec":28,"s":651,"top":18,"kw":652,"maker":11,"domain":10,"tier":19,"attempts":43,"strip":653,"url":9,"mix":654,"praise":105,"objection":127,"sub":21},40456236,0.06,[24,25,26,27],"3033553233041515300433433543210134134333",[31,32,33,34],1794036655206]