[{"data":1,"prerenderedAt":329},["ShallowReactive",2],{"project-40345696":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":28,"sentiment":29,"stance_mix":30,"scored_comments":35,"aspects":36,"strip":41,"post_text":42,"what":43,"said":73,"maker_in_thread":181,"shape":190,"grid":194,"attempts":291,"same_week":292,"maker_shows":31,"maker_hidden":321,"reader_version":322,"card":323},"40345696","Pipecat","An open source framework for voice assistants","title","Show HN: An open source framework for voice assistants","https:\u002F\u002Fgithub.com\u002Fpipecat-ai\u002Fpipecat","github.com","kwindla",1715620874,2024,346,39,156,3515,5,"legend","ai","voice assistant framework","library",[24,25,26,27],"voice assistant","real-time ai","speech pipeline","llm voice agent","curious",0.167,{"positive":31,"neutral":32,"mixed":33,"critical":34},3,6,0,1,10,{"idea":37,"docs":38,"execution":39,"performance":40},{"positive":34,"critical":34},{"positive":34,"critical":33},{"positive":34,"critical":33},{"positive":33,"critical":33},[33,33,33,33,31,31,31,18,18,33,33,33,33,33,33,33,33,33,33,33,33,33,33,33,33,33,33,33,31,18,31,33,31,33,33,34,33,33,33],"I've been obsessed for the past ~year with the possibilities of talking to LLMs. I built a bunch of one-off prototypes, shared code on X, started a Meetup group in SF, and co-hosted a big hackathon. It turns out that there are a few low-level problems that everybody building conversational\u002Freal-time AI needs to solve on the way to building\u002Fshipping something that works well: low-latency media transport, echo cancellation, voice activity detection, phrase endpointing, pipelining data between models\u002Fservices, handling voice interruptions, swapping out different models\u002Fservices.\n\nOn the theory th",{"problem":44,"audience":50,"pricing":54,"motivation":55,"time_to_build":54,"stack":59,"openness":70,"stage":71,"team_size":72},{"value":45,"source":46,"evidence":47},"Low-level problems in real-time voice AI: latency, echo cancellation, VAD, endpointing, interruptions, swapping models.","maker",[48],{"id":49},"post",{"value":51,"source":46,"evidence":52},"Developers building conversational and real-time AI assistants or agents.",[53],{"id":49},null,{"value":56,"source":46,"evidence":57},"Maker was obsessed with talking to LLMs and wanted a LangChain-like framework for real-time voice AI.",[58],{"id":49},[60,64],{"value":61,"source":46,"evidence":62},"Python",[63],{"id":49},{"value":65,"source":46,"evidence":66},"GPT-4 Turbo",[67],{"id":68,"by":11,"text":69},"40346654","Here's a translation demo in Pipecat using the now ancient and arthritic GPT-4 Turbo model. :-) https:\u002F\u002Fgithub.com\u002Fpipecat-ai\u002Fpipecat\u002Ftree\u002Fmain\u002Fexamples\u002Ftranslation-chatbot As…","open_source","beta","team",{"praise":74,"objections":95,"requests":114,"questions":132,"alternatives":148},[75,87],{"value":76,"source":77,"evidence":78},"Open source implementation of a needed voice AI layer","commenter",[79,83],{"id":80,"by":81,"text":82},"40346934","awenix","Nice to see an open source implementation, i have been seeing many startups get into this space like https:\u002F\u002Fwww.retellai.com\u002F, https:\u002F\u002Ffixie.ai\u002F etc. They always end up needing…",{"id":84,"by":85,"text":86},"40351306","orliesaurus","The whole VAD thing is very interesting, keen to learn more about how it works and especially with multiple speakers!",{"value":88,"source":77,"evidence":89},"Examples like speech-to-speech and translation demos look impressive",[90,94],{"id":91,"by":92,"text":93},"40459312","TomTom101101","That's absolutely amazing, both visually and technically! Do you share any insights of the development process, perhaps some code?",{"id":68,"by":11,"text":69},[96,107],{"value":97,"source":77,"evidence":98},"Pipelines of STT, LLM and TTS seem inferior to native speech-to-speech models like GPT-4o",[99,103],{"id":100,"by":101,"text":102},"40346599","avarun","Yeah, same question here. Building pipelines for bridging LLMs and TTS and STT models with lower latency is fine and all, but when you compare to a natively multimodal model like…",{"id":104,"by":105,"text":106},"40346373","bamazizi","I wonder how the just announced \"GPT-4o\" with real-time voice impacts projects like this? The demo on real-time multi language translation conversation blew me away!",{"value":108,"source":77,"evidence":109},"Speech-to-speech support is not visible in the code",[110],{"id":111,"by":112,"text":113},"40346529","ilaksh","Your project is amazing and I'm not trying to take away from what you have accomplished. But..I looked at the code but didn't see any audio-to-audio service or model. Can you…",[115,122],{"value":116,"source":77,"evidence":117},"Add speech-to-speech or audio-to-audio model support",[118,121],{"id":119,"by":112,"text":120},"40346398","This is great but we really need an audio-to-audio model like they demoed in the open source world. Does anyone know of anything like that? Edit: someone found one:…",{"id":111,"by":112,"text":113},{"value":123,"source":77,"evidence":124},"Live translation of phone calls",[125,129],{"id":126,"by":127,"text":128},"40355594","35mm","How would I go about using this to live translate phone calls?",{"id":130,"by":127,"text":131},"40363716","To talk to people in languages I am not fluent in. Whisper is too slow and doesn’t allow interruption.",[133,140,144],{"value":134,"source":77,"evidence":135},"How speech-to-speech models work and how they differ from pipelines",[136],{"id":137,"by":138,"text":139},"40350643","j-bos","How do speech to speech models work? Do they just that many more tokens to capture nuances of spoken language?",{"value":141,"source":77,"evidence":142},"How the VAD works with multiple speakers",[143],{"id":84,"by":85,"text":86},{"value":145,"source":77,"evidence":146},"Impact of GPT-4o on projects like this",[147],{"id":104,"by":105,"text":106},[149,156,162,166,174],{"value":150,"source":77,"evidence":151},"LiveKit Agents",[152],{"id":153,"by":154,"text":155},"40347530","russ","LiveKit Agents, which OpenAI uses in voice mode is also open source: https:\u002F\u002Fgithub.com\u002Flivekit\u002Fagents",{"value":157,"source":77,"evidence":158},"Vapi",[159],{"id":160,"by":11,"text":161},"40347672","Adding to your list: https:\u002F\u002Fvapi.ai -- really nice tools. (I try to keep up with all the different layers\u002Fplayers in this space.)",{"value":163,"source":77,"evidence":164},"Retell AI",[165],{"id":80,"by":81,"text":82},{"value":167,"source":77,"evidence":168},"Fixie.ai",[169,170],{"id":80,"by":81,"text":82},{"id":171,"by":172,"text":173},"40348900","zkoch","We're (fixie.ai) working on on our SLM (speech language model). We'll release something soon to play with :)",{"value":175,"source":77,"evidence":176},"Bolna",[177],{"id":178,"by":179,"text":180},"40347247","xan_ps007","We're also building bolna an open source voice orchestration: https:\u002F\u002Fgithub.com\u002Fbolna-ai\u002Fbolna",{"comments":182,"answered":183,"key":184},4,2,[185,186,189],{"id":160,"text":161},{"id":187,"text":188},"40346532","An audio-to-audio model is definitely a step forward. And I do think that's where things are going to go, generally speaking. For context relating to real-time voice AI: once…",{"id":68,"text":69},{"commenters":191,"max_depth":182,"maker_replies":182,"first_comment_mins":192,"lifespan_hours":193},29,41,243,[195,197,198,199,201,203,207,209,211,214,216,218,222,225,228,231,234,236,238,241,244,246,250,253,256,259,262,263,264,265,270,272,276,278,281,283,285,286,289],[80,33,33,196,81,33],"other\u002Fneutral\u002Fnone",[160,33,33,196,11,34],[171,33,33,196,172,33],[137,33,33,200,138,33],"other\u002Fneutral\u002Fidea",[119,34,31,202,112,33],"project\u002Fneutral\u002Fidea",[204,34,31,205,206,33],"40346494","maker\u002Fneutral\u002Fdocs","makeitmore",[111,34,31,208,112,33],"maker\u002Fneutral\u002Fidea",[91,34,18,210,92,33],"maker\u002Fpositive\u002Fexecution",[212,34,18,213,92,33],"40459353","project\u002Fpositive\u002Fdocs",[187,34,33,215,11,34],"other\u002Fneutral\u002Fperformance",[217,34,33,215,112,33],"40346555",[219,34,33,220,221,33],"40349455","commenter\u002Fcritical\u002Fperformance","tempusalaria",[223,34,33,215,224,33],"40350509","huac",[226,183,33,196,227,33],"40346465","johnmaguire",[229,183,33,196,230,33],"40346576","petemir",[232,183,33,196,233,33],"40346996","sroussey",[235,183,33,196,227,33],"40347059",[237,183,33,196,233,33],"40351001",[239,183,33,196,240,33],"40347515","michaelmior",[242,183,33,196,243,33],"40347576","magicalhippo",[245,183,33,196,227,33],"40348095",[247,183,33,248,249,33],"40349345","other\u002Fcritical\u002Fnone","toastercat",[251,183,33,196,252,33],"40352312","kybernetikos",[254,183,33,196,255,33],"40355386","regularfry",[257,183,33,196,258,33],"40357768","T-A",[260,31,33,196,261,33],"40351499","userhacker",[178,182,33,196,179,33],[153,18,33,196,154,33],[84,32,31,202,85,33],[266,267,18,268,269,33],"40346273",7,"project\u002Fpositive\u002Fidea","canadiantim",[126,271,31,202,127,33],8,[273,271,33,274,275,33],"40356012","commenter\u002Fneutral\u002Fnone","generalizations",[130,271,31,277,127,33],"project\u002Fneutral\u002Fperformance",[279,271,33,196,280,33],"40368487","vr000m",[104,282,33,200,105,33],9,[100,282,34,284,101,33],"project\u002Fcritical\u002Fidea",[68,282,33,208,11,34],[287,282,33,274,288,33],"40346771","jshreder",[290,282,33,205,11,34],"40346821",[],[293,298,303,308,313,317],{"id":294,"name":295,"points":296,"category":297},"40318542","Jam",886,"dev_tools",{"id":299,"name":300,"points":301,"category":302},"40331968","Omniclip",588,"creative",{"id":304,"name":305,"points":306,"category":307},"40307519","Hackerverse",520,"data",{"id":309,"name":310,"points":311,"category":312},"40338275","Cap",351,"productivity",{"id":314,"name":315,"points":316,"category":20},"40346995","Pi-C.A.R.D",344,{"id":318,"name":319,"points":320,"category":307},"40367090","Quary",293,false,"v2",{"id":324,"name":5,"line":6,"ai":321,"cat":20,"year":13,"points":14,"comments":15,"rec":28,"s":325,"top":18,"kw":326,"maker":11,"domain":10,"tier":19,"attempts":33,"strip":327,"url":9,"mix":328,"praise":76,"objection":97,"sub":21},40345696,0.17,[24,25,26,27],"000033355000000000000000000035303001000",[31,32,33,34],1794036654787]