[{"data":1,"prerenderedAt":174},["ShallowReactive",2],{"project-32545235":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":28,"sentiment":29,"stance_mix":30,"scored_comments":34,"aspects":35,"strip":39,"post_text":29,"what":40,"said":65,"maker_in_thread":99,"shape":108,"grid":112,"attempts":136,"same_week":137,"maker_shows":166,"maker_hidden":167,"reader_version":168,"card":169},"32545235","Nevermind XOR","Deep Learning Has an Issue with Sin","title","Show HN: Nevermind XOR – Deep Learning Has an Issue with Sin","https:\u002F\u002Flostmsu.github.io\u002FNevermind-XOR-Deep-Learning-has-an-isssue-with-Sin\u002F","lostmsu.github.io","lostmsu",1661121755,2022,34,10,1190,2570,47,"shown","ai","deep learning experiment","other",[24,25,26,27],"deep learning","sine activation","modulo","neural networks","quiet",null,{"positive":31,"neutral":32,"mixed":31,"critical":33},0,1,3,4,{"idea":36,"docs":38},{"positive":31,"critical":37},2,{"positive":31,"critical":32},[32,31,31,31,33,31,32,31,31,32],{"problem":41,"audience":29,"pricing":29,"motivation":51,"time_to_build":29,"stack":55,"openness":64,"stage":64,"team_size":64},{"value":42,"source":43,"evidence":44},"Neural networks struggle to learn sine-wave and modulo-style periodic functions over large inputs.","maker",[45,48],{"id":46,"by":11,"text":47},"32548257","I am sure a LSTM or even a small 2 layer network can learn to predict the next numbers on the clock from the numbers on the clock exactly one second ago, but that is not the…",{"id":49,"by":11,"text":50},"32548204","Well, the point is that neither large models could be trained, nor could a trivial model which is known to have a set of parameter values, that satisfy the given task.",{"value":52,"source":43,"evidence":53},"Show that deep learning fails at a task that modulo division solves trivially.",[54],{"id":46,"by":11,"text":47},[56],{"value":57,"source":58,"evidence":59},"PyTorch","inferred",[60],{"id":61,"by":62,"text":63},"32548015","shreyshnaccount","> def forward(self, x): return torch.sin(self.freq * x) I don't think this qualifies as deep learning anyway","unknown",{"praise":66,"objections":67,"requests":86,"questions":87,"alternatives":94},[],[68,76,80],{"value":69,"source":70,"evidence":71},"Pointwise loss treats timepoints as independent; an autoregressive model would fit better.","commenter",[72],{"id":73,"by":74,"text":75},"32547270","rryan","You're using a pointwise loss, which means you are treating each timepoint as conditionally independent. That's a deeply flawed assumption, probabilistically. Try an…",{"value":77,"source":70,"evidence":78},"A sin-based forward pass does not qualify as deep learning.",[79],{"id":61,"by":62,"text":63},{"value":81,"source":70,"evidence":82},"The problem is framed in a flawed way.",[83],{"id":84,"by":62,"text":85},"32552339","yeah, and the way the problem is being treated is flawed, but that's another thing",[],[88],{"value":89,"source":70,"evidence":90},"How the maker concluded the x + sin(x) activation only worked when cycles were fewer than parameters.",[91],{"id":92,"by":62,"text":93},"32547965",">Their x + sin(x) * 2 activation only really worked when the number of full cycles in the dataset was less than the number of paramters of the network, which completely misses…",[95],{"value":96,"source":70,"evidence":97},"WaveNet autoregressive model",[98],{"id":73,"by":74,"text":75},{"comments":34,"answered":32,"key":100},[101,102,105],{"id":46,"text":47},{"id":103,"text":104},"32556773","The remark about the Chebyshev polynomials just sent me down another rabbit hole.",{"id":106,"text":107},"32548266","It is just a guess, that tries to explain the observed behavior.",{"commenters":109,"max_depth":34,"maker_replies":34,"first_comment_mins":110,"lifespan_hours":111},5,332,22,[113,115,117,121,123,125,126,128,130,132],[73,31,32,114,74,31],"maker\u002Fcritical\u002Fidea",[46,31,31,116,11,32],"commenter\u002Fneutral\u002Fperformance",[118,31,31,119,120,31],"32548780","commenter\u002Fneutral\u002Fnone","azalemeth",[103,31,31,122,11,32],"other\u002Fneutral\u002Fnone",[92,32,33,124,62,31],"maker\u002Fneutral\u002Fidea",[106,32,31,119,11,32],[61,37,32,127,62,31],"project\u002Fcritical\u002Fidea",[49,37,31,129,11,32],"project\u002Fneutral\u002Fidea",[84,37,31,131,62,31],"commenter\u002Fcritical\u002Fidea",[133,33,32,134,135,31],"32551101","project\u002Fcritical\u002Fdocs","natch",[],[138,142,147,152,157,161],{"id":139,"name":140,"points":141,"category":20},"32572770","Pornpen.ai",711,{"id":143,"name":144,"points":145,"category":146},"32561127","Crawlee",282,"dev_tools",{"id":148,"name":149,"points":150,"category":151},"32568423","react-design-editor",232,"creative",{"id":153,"name":154,"points":155,"category":156},"32521395","SineRider",220,"games",{"id":158,"name":159,"points":160,"category":146},"32578311","Fuzz Map",205,{"id":162,"name":163,"points":164,"category":165},"32551862","Digs.fm",131,"consumer_social",6,false,"v2",{"id":170,"name":5,"line":6,"ai":167,"cat":20,"year":13,"points":14,"comments":15,"rec":28,"s":29,"top":18,"kw":171,"maker":11,"domain":10,"tier":19,"attempts":31,"strip":172,"url":9,"mix":173,"praise":-1,"objection":69,"sub":21},32545235,[24,25,26,27],"1000301001",[31,32,31,33],1794036623889]