[{"data":1,"prerenderedAt":560},["ShallowReactive",2],{"project-41203475":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":28,"sentiment":29,"stance_mix":30,"scored_comments":35,"aspects":36,"strip":49,"post_text":51,"what":52,"said":99,"maker_in_thread":239,"shape":249,"grid":253,"attempts":521,"same_week":522,"maker_shows":38,"maker_hidden":552,"reader_version":553,"card":554},"41203475","Thunder Compute","Attaching to a virtual GPU over TCP","title","Show HN: Attaching to a virtual GPU over TCP","https:\u002F\u002Fwww.thundercompute.com\u002F","thundercompute.com","bmodel",1723222241,2024,344,108,158,3515,5,"legend","infrastructure","remote gpu access","cli",[24,25,26,27],"remote gpu","gpu over network","cloud gpu","cuda","curious",-0.079,{"positive":31,"neutral":32,"mixed":33,"critical":34},16,38,0,19,73,{"execution":37,"pricing":39,"performance":41,"idea":43,"privacy":46,"design":47,"docs":48},{"positive":38,"critical":38},1,{"positive":40,"critical":40},2,{"positive":40,"critical":42},4,{"positive":44,"critical":45},7,10,{"positive":33,"critical":33},{"positive":38,"critical":33},{"positive":33,"critical":40},[18,40,18,38,33,33,18,38,50,50,40,38,38,50,38,38,18,18,50,18,50,40,50,42,50,33,18,50,50,50,50,50,38,38,18,50,38,50,18,18],3,"We developed a tool to trick your computer into thinking it’s attached to a GPU which actually sits across a network. This allows you to switch the number or type of GPUs you’re using with a single command.",{"problem":53,"audience":61,"pricing":69,"motivation":75,"time_to_build":79,"stack":80,"openness":97,"stage":98,"team_size":97},{"value":54,"source":55,"evidence":56},"Developers need GPUs only some of the time and want to switch GPU type or count without changing instances","maker",[57,60],{"id":58,"by":11,"text":59},"41203971","Great point, there are a few benefits: 1. If you're actively developing and need a GPU then you typically would be paying the entire time the instance is running. Using Thunder…",{"id":58,"by":11,"text":59},{"value":62,"source":63,"evidence":64},"Developers running CPU-heavy code who occasionally need GPUs, and ML training\u002Finference users","inferred",[65,66],{"id":58,"by":11,"text":59},{"id":67,"by":11,"text":68},"41203806","Great questions! To clarify the demo, we were ssh'd into a linux machine with no GPU. Going into more details for how this works, we intercept communication between the CPU and…",{"value":70,"source":55,"evidence":71},"Free during beta; pay-as-you-go planned",[72],{"id":73,"by":11,"text":74},"41204101","We're still in our beta so it's entirely free for now (we can't promise a bug-free experience)! You have to make an account but it won't require payment details. Down the line we…",{"value":76,"source":55,"evidence":77},"Pay for GPU only while actively using it",[78],{"id":58,"by":11,"text":59},null,[81,87,93],{"value":82,"source":55,"evidence":83},"Linux",[84],{"id":85,"by":11,"text":86},"41204248","At the moment out tech is linux-only so it would not work with Blender. Down the line, we could see this being used for batched render jobs (i.e. to replace a render farm).",{"value":88,"source":55,"evidence":89},"PyTorch",[90],{"id":91,"by":11,"text":92},"41204179","We have tested this with pytorch and huggingface and it is mostly stable (we know there are issues with pycuda and jax). In theory this should work with any libraries, however…",{"value":94,"source":55,"evidence":95},"HuggingFace",[96],{"id":91,"by":11,"text":92},"unknown","beta",{"praise":100,"objections":124,"requests":166,"questions":198,"alternatives":213},[101,113],{"value":102,"source":103,"evidence":104},"Useful for GPU-heavy dev work while keeping CPU instances cheap","commenter",[105,109],{"id":106,"by":107,"text":108},"41203970","billconan","it's more transparent to your system, for example, if you have a gui application that needs gpu acceleration on a thin client (Matlab, solidworks, blender), you can do so without…",{"id":110,"by":111,"text":112},"41206291","teaearlgraycold","I develop GPU accelerated web apps in an EC2 instance with a remote VSCode session. A lot of the time I’m just doing web dev and don’t need a GPU. I can save thousands per month…",{"value":114,"source":103,"evidence":115},"Interesting idea for remote GPU use",[116,120],{"id":117,"by":118,"text":119},"41204281","bkitano19","this is nuts",{"id":121,"by":122,"text":123},"41211975","ellis0n","In 2008, I had a powerful server with XEON CPU, but the motherboard had no slots for a graphics card. I also had a computer with a powerful graphics card but a weak Core 2 Duo. I…",[125,140,151],{"value":126,"source":103,"evidence":127},"Network latency and bandwidth make training impractical",[128,132,136],{"id":129,"by":130,"text":131},"41205229","doctorpangloss","Okay, but your GPUs are in ECS. Don't I just want this feature from Amazon, not you, and natively via Nitro? Or even Google has TPU attachments. > 1. If you're actively…",{"id":133,"by":134,"text":135},"41208685","delijati","Even a directly attached eGPU via thunderbold 4 was after some time too slow for machine learning aka training. As i work now fully remote i just have a beefy midi tower. Some…",{"id":137,"by":138,"text":139},"41206233","m3kw9","So won’t that make the network the prohibitive bottle neck? Your memory bandwidth is 1gbps max",{"value":141,"source":103,"evidence":142},"Cheaper to use own GPU or existing cloud GPU instances",[143,146,147],{"id":144,"by":130,"text":145},"41203886","I don't get it. Why would I start an instance in ECS, to use your GPUs in ECS, when I could start an instance for the GPUs I want in ECS? Separately, why would I want half of…",{"id":129,"by":130,"text":131},{"id":148,"by":149,"text":150},"41208626","amelius","Sounds like you can save thousands by just buying a simple GPU card.",{"value":152,"source":103,"evidence":153},"Subscription cloud model is unwelcome",[154,158,162],{"id":155,"by":156,"text":157},"41208424","johnisgood","I am increasingly growing tired of these \"cloud\" services, paid or not. :\u002F",{"id":159,"by":160,"text":161},"41209045","rowanG077","There is a vast gap between walled garden cloud service rent seeking and giving away software as open source. In the olden days you could buy software licenses to run it wherever…",{"id":163,"by":164,"text":165},"41212373","mhuffman","I agree. Paying by the month for the rest of your life or they cut you off is not something I am a fan of. I feel sorry for people too young to remember that you could actually…",[167,182,191],{"value":168,"source":103,"evidence":169},"Self-hosting option",[170,174,178],{"id":171,"by":172,"text":173},"41204095","Cieric","This is interesting, but I'm more interested in self-hosting. I already have a lot of GPUs (some running some not.) Does this have a self-hosting option so I can use the GPUs I…",{"id":175,"by":176,"text":177},"41211751","xyst","Exciting. But would definitely like to see a self hosted option.",{"id":179,"by":180,"text":181},"41204740","winecamera","I saw that in the tnr CLI, there are hints of an option to self-host a GPU. Is this going to be a released feature?",{"value":183,"source":103,"evidence":184},"Vulkan and OpenGL support",[185,188],{"id":186,"by":107,"text":187},"41203783","is this a remote nvapi? this is awesome. can it do 3d rendering (vulkan\u002Fopengl)",{"id":189,"by":11,"text":190},"41203844","Thank you! > is this a remote nvapi Essentially yes! Just to be clear, this covers the entire GPU not just the NVAPI (i.e. all of cuda). This functions like you have the physical…",{"value":192,"source":103,"evidence":193},"Hashcat support",[194],{"id":195,"by":196,"text":197},"41208233","tamimio","I’m more interested in using tools like hashcat, any benchmark on these? As the docs link returns error.",[199,206],{"value":200,"source":103,"evidence":201},"How is latency and I\u002FO handled for datasets larger than VRAM",[202],{"id":203,"by":204,"text":205},"41204617","radarsat1","I'm confused, if this operates at the CPU\u002FGPU boundary doesn't it create a massive I\u002FO bottleneck for any dataset that doesn't fit into VRAM? I'm probably misunderstanding how it…",{"value":207,"source":103,"evidence":208},"Does it work with AMD GPUs and non-CUDA workloads",[209],{"id":210,"by":211,"text":212},"41204154","rubatuga","What ML packages do you support? In the comments below it says you do not support Vulkan or OpenGL. Does this support AMD GPUs as well?",[214,225,232],{"value":215,"source":103,"evidence":216},"Juice Labs",[217,221],{"id":218,"by":219,"text":220},"41206401","tommsy64","There is a GPU-over-network software called Juice [1]. I've used it on AWS for running CPU-intensive workloads that also happen to need some GPU without needing to use a huge GPU…",{"id":222,"by":223,"text":224},"41204871","goku-goku","www.juicelabs.co does all this today, including the GPU sharing and fractionalization.",{"value":226,"source":103,"evidence":227},"ffmpeg-over-ip",[228],{"id":229,"by":230,"text":231},"41203918","steelbrain","Ah this is quite interesting! I had a usecase where I needed a GPU-over-IP but only for transcoding videos. I had a not-so-powerful AMD GPU in my homelab server that somehow kept…",{"value":233,"source":103,"evidence":234},"pocl remote backend and qCUDA",[235],{"id":236,"by":237,"text":238},"41205188","test20240809","pocl (Portable Computing Language) [1] provides a remote backend [2] that allows for serialization and forwarding of OpenCL commands over a network. Another solution is qCUDA [3]…",{"comments":31,"answered":240,"key":241},13,[242,245,248],{"id":243,"text":244},"41204835","That understanding of the system is correct. To make it practical we've implemented a bunch of optimizations to minimize I\u002FO cost. You can see how it performs on inference with…",{"id":246,"text":247},"41204030","Exactly! The finer grain sharing is one of the key things on our radar right now",{"id":58,"text":59},{"commenters":250,"max_depth":44,"maker_replies":31,"first_comment_mins":251,"lifespan_hours":252},61,24,99,[254,256,260,263,267,271,273,277,280,283,286,288,291,293,295,298,299,301,304,308,311,314,317,320,322,325,328,331,333,335,339,342,344,347,348,349,351,352,354,356,357,360,362,364,366,368,370,374,377,381,383,386,388,391,394,397,399,403,408,411,412,414,416,420,423,427,428,431,433,436,439,441,443,444,449,450,452,455,458,460,463,465,467,468,469,471,473,475,477,480,482,485,487,488,490,494,496,499,500,503,507,510,512,514,516,518],[229,33,18,255,230,33],"project\u002Fpositive\u002Fexecution",[257,33,50,258,259,33],"41204329","project\u002Fneutral\u002Fnone","crishoj",[261,33,50,262,230,33],"41208463","project\u002Fneutral\u002Fexecution",[264,33,38,265,266,33],"41204586","project\u002Fcritical\u002Fpricing","bhaney",[268,33,50,269,270,33],"41205686","project\u002Fneutral\u002Fperformance","trws",[218,33,18,272,219,33],"project\u002Fpositive\u002Fperformance",[274,33,18,275,276,33],"41207361","project\u002Fpositive\u002Fidea","Fnoord",[278,33,50,269,279,33],"41207533","lostmsu",[281,33,38,282,266,33],"41208181","project\u002Fcritical\u002Fidea",[284,33,33,285,279,33],"41211299","commenter\u002Fneutral\u002Fnone",[155,33,33,287,156,33],"other\u002Fneutral\u002Fnone",[289,33,33,285,290,33],"41208620","adwn",[159,33,33,292,160,33],"commenter\u002Fneutral\u002Fpricing",[163,33,33,294,164,33],"other\u002Fneutral\u002Fpricing",[296,33,33,287,297,33],"41205253","toomuchtodo",[203,38,50,269,204,33],[243,38,33,300,11,38],"maker\u002Fneutral\u002Fperformance",[302,38,18,303,204,33],"41205006","maker\u002Fpositive\u002Fperformance",[305,38,38,306,307,33],"41206394","project\u002Fcritical\u002Fperformance","semitones",[309,38,50,269,310,33],"41208475","the8472",[312,38,33,287,313,33],"41211183","ruined",[315,38,50,258,316,33],"41206729","ranger_danger",[318,38,33,285,319,33],"41209453","zozbot234",[321,38,33,285,316,33],"41210976",[323,40,50,262,324,33],"41204517","dishsoap",[326,40,50,262,327,33],"41206526","almostgotcaught",[329,40,50,262,330,33],"41206878","yarpen_z",[332,40,38,282,310,33],"41208527",[171,50,50,334,172,33],"project\u002Fneutral\u002Fidea",[336,50,50,337,338,33],"41204191","maker\u002Fneutral\u002Fnone","cpeterson42",[340,50,38,282,341,33],"41205228","covi",[343,50,38,282,122,33],"41212022",[345,50,38,282,346,33],"41238808","donnygreenberg",[144,42,38,282,130,33],[106,42,18,275,107,33],[246,42,33,350,11,38],"maker\u002Fpositive\u002Fidea",[222,42,38,282,223,33],[353,42,38,265,316,33],"41211020",[58,42,33,355,11,38],"maker\u002Fpositive\u002Fpricing",[129,42,38,282,130,33],[358,42,33,359,11,38],"41205381","maker\u002Fneutral\u002Fidea",[361,42,38,282,130,33],"41205403",[110,42,18,363,111,33],"project\u002Fpositive\u002Fpricing",[148,42,33,365,149,33],"commenter\u002Fcritical\u002Fpricing",[367,42,33,285,111,33],"41208736",[369,18,18,355,338,33],"41204005",[371,18,50,372,373,33],"41206860","project\u002Fneutral\u002Fpricing","dheera",[375,18,50,376,338,33],"41210561","maker\u002Fneutral\u002Fpricing",[378,379,50,269,380,33],"41204021",6,"tptacek",[382,379,33,269,11,38],"41204054",[384,379,50,385,380,33],"41204074","project\u002Fneutral\u002Fprivacy",[387,379,33,385,11,38],"41204227",[389,379,18,390,380,33],"41204618","maker\u002Fpositive\u002Fnone",[392,379,50,385,393,33],"41207485","azinman2",[395,44,50,334,396,33],"41205453","mmsc",[398,44,50,359,338,33],"41205624",[400,401,50,334,402,33],"41212171",8,"somat",[404,405,38,406,407,33],"41207625",9,"project\u002Fcritical\u002Fexecution","userbinator",[409,45,50,372,410,33],"41204066","orsorna",[73,45,33,376,11,38],[133,413,38,306,134,33],11,[121,415,18,275,122,33],12,[417,240,18,418,419,33],"41204057","project\u002Fpositive\u002Fdesign","kawsper",[421,240,50,258,422,33],"41204106","kevmo314",[424,425,50,258,426,33],"41204231",14,"the_reader",[85,425,33,258,11,38],[429,425,33,285,430,33],"41204299","comex",[432,425,33,337,11,38],"41204402",[434,425,50,334,435,33],"41206982","chmod775",[437,438,18,275,111,33],"41204234",15,[440,438,33,337,11,38],"41204326",[442,438,33,287,223,33],"41204983",[175,31,18,275,176,33],[445,446,38,447,448,33],"41203681",17,"project\u002Fcritical\u002Fdocs","talldayo",[67,446,33,337,11,38],[451,446,50,262,107,33],"41203829",[453,446,33,454,11,38],"41203853","maker\u002Fneutral\u002Fexecution",[456,446,50,269,457,33],"41204067","ZeroCool2u",[459,446,33,269,11,38],"41204168",[461,446,50,269,462,33],"41204979","samstave",[464,446,50,269,457,33],"41206345",[210,466,50,258,211,33],18,[91,466,33,337,11,38],[179,34,50,258,180,33],[470,34,50,337,338,33],"41204801",[195,472,38,447,196,33],20,[474,472,33,337,11,38],"41213115",[137,476,38,306,138,33],21,[478,476,33,479,111,33],"41206273","commenter\u002Fneutral\u002Fperformance",[186,481,50,258,107,33],22,[483,481,18,275,484,33],"41203838","czbond",[486,481,18,390,338,33],"41204232",[189,481,33,337,11,38],[489,481,50,258,107,33],"41203879",[491,492,50,258,493,33],"41204145",23,"throwaway888abc",[495,492,38,306,338,33],"41204161",[497,492,50,258,498,33],"41207383","boxerbk",[236,251,38,282,237,33],[501,251,50,334,502,33],"41205269","fpoling",[504,505,50,334,506,33],"41204596",25,"Zambyte",[508,505,33,285,509,33],"41205097","K0IN",[511,505,33,287,506,33],"41206223",[117,513,18,275,118,33],26,[515,513,18,390,338,33],"41204330",[517,513,33,287,223,33],"41204872",[519,520,50,337,338,33],"41204109",27,[],[523,528,533,538,543,548],{"id":524,"name":525,"points":526,"category":527},"41247023","ytch",2741,"consumer_social",{"id":529,"name":530,"points":531,"category":532},"41224689","PGlite",509,"data",{"id":534,"name":535,"points":536,"category":537},"41203306","LLM-aided OCR",479,"productivity",{"id":539,"name":540,"points":541,"category":542},"41217109","Grandma's Word",454,"games",{"id":544,"name":545,"points":546,"category":547},"41213711","flutter_rust_bridge",301,"languages",{"id":549,"name":550,"points":551,"category":537},"41227350","bookshelf",263,false,"v2",{"id":555,"name":5,"line":6,"ai":552,"cat":20,"year":13,"points":14,"comments":15,"rec":28,"s":556,"top":18,"kw":557,"maker":11,"domain":10,"tier":19,"attempts":33,"strip":558,"url":9,"mix":559,"praise":102,"objection":126,"sub":21},41203475,-0.08,[24,25,26,27],"5251005133211311553532343053333311531355",[31,32,33,34],1794036659089]