[{"data":1,"prerenderedAt":382},["ShallowReactive",2],{"project-40481578":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":28,"sentiment":29,"stance_mix":30,"scored_comments":35,"aspects":36,"strip":41,"post_text":45,"what":46,"said":83,"maker_in_thread":161,"shape":174,"grid":178,"attempts":338,"same_week":343,"maker_shows":44,"maker_hidden":374,"reader_version":375,"card":376},"40481578","DiscoGrad","Boldly go where Gradient Descent has never gone before with DiscoGrad","title","Show HN: Boldly go where Gradient Descent has never gone before with DiscoGrad","https:\u002F\u002Fgithub.com\u002FDiscoGrad\u002FDiscoGrad","github.com","frankling_",1716725659,2024,232,66,321,3515,10,"notable","languages","automatic differentiation library","library",[24,25,26,27],"automatic differentiation","gradient descent","c++","source-to-source transformation","curious",0.132,{"positive":31,"neutral":32,"mixed":33,"critical":34},7,25,0,2,34,{"idea":37,"execution":39,"performance":40},{"positive":38,"critical":34},4,{"positive":33,"critical":33},{"positive":33,"critical":33},[42,33,33,42,43,42,43,43,43,33,33,44,43,42,33,33,33,42,42,42,33,43,44,42,33,33,42,42,33,43,42,42,42,42,33,42,42,33,42,42],3,5,1,"Trying to do gradient descent using automatic differentiation over branchy programs? Or to combine them with neural networks for end-to-end training? Then this might be interesting to you.\n\nWe develped DiscoGrad, a tool for automatic differentiation through C++ programs involving input-dependent control flow (e.g., \"if (f(x) \u003C c) { ... }\", differentiating wrt. x) and randomness. Our initial motivation was to enable the use of gradient descent with simulations, which often rely heavily on such discrete branching. The latter makes plain autodiff mostly useless, since it can only account for the ",{"problem":47,"audience":56,"pricing":60,"motivation":61,"time_to_build":60,"stack":65,"openness":80,"stage":81,"team_size":82},{"value":48,"source":49,"evidence":50},"Plain autodiff ignores alternative branches, so gradients are useless for programs with input-dependent control flow","maker",[51,53],{"id":52},"post",{"id":54,"by":11,"text":55},"40484620","That's right, plain autodiff just ignores branches. Our canonical \"why is this even needed\" example is a program like \"if (x >= 0) return 1; else return 0\", x being the input.…",{"value":57,"source":49,"evidence":58},"Researchers and developers optimizing simulations or combining imperative programs with neural networks",[59],{"id":52},null,{"value":62,"source":49,"evidence":63},"Enable gradient descent with simulations that rely on discrete branching",[64],{"id":52},[66,70,74],{"value":67,"source":49,"evidence":68},"C++",[69],{"id":52},{"value":71,"source":49,"evidence":72},"LLVM",[73],{"id":52},{"value":75,"source":49,"evidence":76},"Torch",[77],{"id":78,"by":11,"text":79},"40484236","We're doing something less expensive: essentially, the overall gradient is computed based on certain statistics based on the branch condition and its derivatives when a branch is…","open_source","unknown","team",{"praise":84,"objections":97,"requests":108,"questions":116,"alternatives":132},[85],{"value":86,"source":87,"evidence":88},"Interesting approach to differentiating through branches","commenter",[89,93],{"id":90,"by":91,"text":92},"40483761","vessenes","When all you have is a hammer, ... you split up your house into wood and nails, and reduce it to a previously solved problem! In all seriousness, this is super interesting. I…",{"id":94,"by":95,"text":96},"40483993","dwrodri","This is the sort of thing I expected to see when Chris Lattner moved to Google and started working on the Swift for Tensorflow project. I am so grateful that someone is making it…",[98],{"value":99,"source":87,"evidence":100},"Existing autodiff tools like Enzyme and CppAD already work well",[101,105],{"id":102,"by":103,"text":104},"40485826","zaitanz","So this confuses me slightly and I am keen to know the advantage of using this. I work on projects that heavily use auto-differentiation for complex models. The models are…",{"id":106,"by":11,"text":107},"40483986","Enzyme is traditional, but super duper optimized, autodiff, that is, it returns the partial derivatives for one path taken through the program, ignoring other branches. DiscoGrad…",[109],{"value":110,"source":87,"evidence":111},"Comparison with MCMC approaches",[112],{"id":113,"by":114,"text":115},"40488547","usgroup","Does that imply that every possible branch is optimised separately and then convoluted thereafter? If so, does it scale for very branchy programs? Do you have any comparisons to…",[117,125],{"value":118,"source":87,"evidence":119},"How does it compare to Enzyme and other autodiff tools",[120,124],{"id":121,"by":122,"text":123},"40483912","s_tim","How does this compare to Enzyme (https:\u002F\u002Fenzyme.mit.edu\u002F)?",{"id":102,"by":103,"text":104},{"value":126,"source":87,"evidence":127},"Whether gradients are finite differences",[128],{"id":129,"by":130,"text":131},"40485651","epr","Obviously, some of this branching, etc. is not differentiable. Is it doing finite differences?",[133,137,144,148,155],{"value":134,"source":87,"evidence":135},"Enzyme",[136],{"id":121,"by":122,"text":123},{"value":138,"source":87,"evidence":139},"Ceres Solver",[140],{"id":141,"by":142,"text":143},"40483747","ipunchghosts","Talk about the differences between this and ceres http:\u002F\u002Fceres-solver.org\u002F.",{"value":145,"source":87,"evidence":146},"CppAD, ADOL-C, TMB",[147],{"id":102,"by":103,"text":104},{"value":149,"source":87,"evidence":150},"Tapenade",[151],{"id":152,"by":153,"text":154},"40489577","Loic","You should take a look at the Tapenade[0] automatic differentiation engine, more than 20 years of active development for Fortran and C. [0]:…",{"value":156,"source":87,"evidence":157},"Stan \u002F MCMC",[158],{"id":159,"by":114,"text":160},"40484708","Not really. The world of Bayesian modelling has much fancier tools: Hamiltonian MC. See MC Stan. There’s also been Gibbs samplers and other techniques which support discrete…",{"comments":162,"answered":163,"key":164},14,12,[165,168,171],{"id":166,"text":167},"40483448","Yep, that's exactly it. The smoothness can either come from randomness in the program itself (then the objective function is asymptotically smooth and DiscoGrad estimates the…",{"id":169,"text":170},"40483687","I agree with that intuition. In our experience, it's easiest to see gains over other optimization techniques when the program is \"branch-wise smooth and non-constant\". Then, we…",{"id":172,"text":173},"40483898","Thanks for the kind words! We'd be super happy if this work gets picked up, whether in a commercial context or not. We were thinking of some disco ball-based logo (among some…",{"commenters":175,"max_depth":43,"maker_replies":162,"first_comment_mins":176,"lifespan_hours":177},33,252,107,[179,183,187,189,190,192,193,196,198,201,204,206,208,209,210,211,214,216,219,223,225,228,232,234,237,239,242,245,247,249,251,253,255,256,259,261,263,264,267,271,273,275,278,281,285,286,288,290,293,296,298,302,305,307,310,312,315,317,321,324,325,329,331,333,335,337],[180,33,42,181,182,33],"40483342","project\u002Fneutral\u002Fidea","eranation",[184,33,33,185,186,33],"40483408","commenter\u002Fneutral\u002Fnone","PeterHolzwarth",[188,33,33,185,182,33],"40483424",[166,33,33,181,11,44],[191,33,42,181,182,33],"40483514",[169,33,33,181,11,44],[194,33,43,195,182,33],"40483862","maker\u002Fpositive\u002Fnone",[129,33,42,197,130,33],"project\u002Fneutral\u002Fexecution",[199,33,42,181,200,33],"40488477","pthr",[202,33,42,181,203,33],"40488772","justinnk",[205,33,43,195,200,33],"40489133",[90,44,43,207,91,33],"project\u002Fpositive\u002Fidea",[172,44,33,185,11,44],[94,34,43,207,95,33],[159,34,33,185,114,33],[212,34,33,185,213,33],"40485644","cma",[215,34,33,185,95,33],"40529937",[217,34,33,218,203,33],"40484793","other\u002Fneutral\u002Fnone",[220,34,44,221,222,33],"40488893","project\u002Fcritical\u002Fidea","YeGoblynQueenne",[224,34,33,185,95,33],"40530002",[226,42,43,207,227,33],"40483987","szvsw",[229,42,42,230,231,33],"40484166","maker\u002Fneutral\u002Fnone","avibryant",[233,42,33,218,227,33],"40484758",[235,42,33,236,11,44],"40484186","project\u002Fneutral\u002Fnone",[238,42,33,218,227,33],"40484711",[240,38,33,218,241,33],"40483823","pavlov",[243,38,33,218,244,33],"40483900","elpocko",[246,38,33,218,241,33],"40483955",[248,43,42,181,114,33],"40484673",[250,43,33,181,11,44],"40484728",[252,43,42,181,227,33],"40484837",[254,43,33,181,11,44],"40485014",[113,43,42,197,114,33],[257,43,33,258,11,44],"40488957","project\u002Fneutral\u002Fperformance",[260,43,43,195,114,33],"40489211",[152,262,42,236,153,33],6,[102,31,44,221,103,33],[265,31,42,181,266,33],"40490206","big-chungus4",[268,269,42,181,270,33],"40484787",8,"sundalia",[272,269,33,181,11,44],"40484863",[274,269,33,218,222,33],"40488987",[276,269,33,185,277,33],"40488332","radarsat1",[279,269,33,185,280,33],"40488345","tomsmeding",[282,283,42,181,284,33],"40484535",9,"HarHarVeryFunny",[54,283,33,181,11,44],[287,283,42,197,284,33],"40485069",[289,283,33,181,11,44],"40485165",[291,283,42,181,292,33],"40485925","Y_Y",[294,18,43,207,295,33],"40487822","andyferris",[297,18,42,181,266,33],"40490137",[299,300,42,181,301,33],"40484296",11,"casualscience",[303,300,42,181,304,33],"40484364","jey",[306,300,42,181,203,33],"40484510",[308,300,42,181,309,33],"40487980","nextaccountic",[311,300,42,181,203,33],"40484456",[313,163,33,218,314,33],"40484435","brap",[316,163,33,218,227,33],"40484808",[318,319,33,218,320,33],"40489281",13,"memming",[322,162,42,258,323,33],"40484167","esafak",[78,162,33,258,11,44],[326,327,42,181,328,33],"40484199",15,"boywitharupee",[330,327,33,181,11,44],"40484324",[141,332,42,181,142,33],16,[334,332,33,181,11,44],"40483870",[121,336,42,181,122,33],17,[106,336,33,181,11,44],[339,342],{"id":340,"t":341,"points":34,"comments":33},"39980554",1712676866,{"id":4,"t":12,"points":14,"comments":15},[344,349,354,359,364,369],{"id":345,"name":346,"points":347,"category":348},"40456236","Dataherald",464,"data",{"id":350,"name":351,"points":352,"category":353},"40501021","Grooved",452,"hardware",{"id":355,"name":356,"points":357,"category":358},"40455478","PaperTalk",319,"productivity",{"id":360,"name":361,"points":362,"category":363},"40502956","Openkoda",318,"web_saas",{"id":365,"name":366,"points":367,"category":368},"40469592","Spot",309,"dev_tools",{"id":370,"name":371,"points":372,"category":373},"40456959","Porter Cloud",258,"infrastructure",false,"v2",{"id":377,"name":5,"line":6,"ai":374,"cat":20,"year":13,"points":14,"comments":15,"rec":28,"s":378,"top":18,"kw":379,"maker":11,"domain":10,"tier":19,"attempts":34,"strip":380,"url":9,"mix":381,"praise":86,"objection":99,"sub":21},40481578,0.13,[24,25,26,27],"3003535550015300033305130033053333033033",[31,32,33,34],1794036655346]