[{"data":1,"prerenderedAt":390},["ShallowReactive",2],{"project-21157463":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":29,"sentiment":30,"stance_mix":31,"scored_comments":35,"aspects":36,"strip":42,"post_text":44,"what":45,"said":97,"maker_in_thread":204,"shape":214,"grid":218,"attempts":352,"same_week":353,"maker_shows":34,"maker_hidden":382,"reader_version":383,"card":384},"21157463","Imagededup","Finding duplicate images made easy","title","Show HN: Imagededup – Finding duplicate images made easy","https:\u002F\u002Fgithub.com\u002Fidealo\u002Fimagededup","github.com","datitran",1570195977,2019,290,57,108,2195,5,"legend","dev_tools","image deduplication library","library",[24,25,26,27,28],"duplicate images","perceptual hash","image deduplication","python","cnn embeddings","loved",0.236,{"positive":32,"neutral":33,"mixed":34,"critical":34},8,27,1,37,{"idea":37,"performance":39,"execution":40,"docs":41},{"positive":34,"critical":38},0,{"positive":34,"critical":38},{"positive":34,"critical":34},{"positive":38,"critical":38},[38,43,38,38,18,43,43,43,43,43,43,43,43,43,18,43,18,38,43,18,43,43,34,38,38,43,38,38,43,38,38,38,18,18,18,43,43,43,43,18],3,null,{"problem":46,"audience":53,"pricing":44,"motivation":62,"time_to_build":44,"stack":66,"openness":94,"stage":95,"team_size":96},{"value":47,"source":48,"evidence":49},"Finding exact and near duplicate images in an image collection.","maker",[50],{"id":51,"by":11,"text":52},"21157492","We've just open-sourced our library imagededup, a Python package that simplifies the task of finding exact and near duplicates in an image collection. It includes several hashing…",{"value":54,"source":55,"evidence":56},"Developers and ML practitioners working with image collections.","inferred",[57,58],{"id":51,"by":11,"text":52},{"id":59,"by":60,"text":61},"21177742","Roritharr","Thanks! This is a very important problemspace for us, as we do document processing and double\u002Ftriple uploads are hard to detect and create unnecessary effort. Can you elaborate a…",{"value":63,"source":48,"evidence":64},"Finding image duplication is important in computer vision and ML, since duplicates bias model evaluation.",[65],{"id":51,"by":11,"text":52},[67,71,79,83,87],{"value":68,"source":48,"evidence":69},"Python",[70],{"id":51,"by":11,"text":52},{"value":72,"source":73,"evidence":74},"TensorFlow","commenter",[75],{"id":76,"by":77,"text":78},"21178370","albertzeyer","This looks nice, and seems to support the most common methods for fingerprinting\u002Fhashing. This comes with some heavy dependencies, though (which is reasonable): ```…",{"value":80,"source":48,"evidence":81},"PHash, DHash hashing",[82],{"id":51,"by":11,"text":52},{"value":84,"source":48,"evidence":85},"Convolutional neural networks",[86],{"id":51,"by":11,"text":52},{"value":88,"source":73,"evidence":89},"bktree search",[90],{"id":91,"by":92,"text":93},"21196224","tanujjain","Only realized now that the 100-200ms time you refer to is for a single search and not for 400,000 searches. The package already achieves this brute-force speed. In fact, the…","open_source","launched","unknown",{"praise":98,"objections":118,"requests":136,"questions":162,"alternatives":182},[99,110],{"value":100,"source":73,"evidence":101},"Useful tool for removing duplicate photos and memes.",[102,106],{"id":103,"by":104,"text":105},"21178030","rcarmo","This is really nice. I have a semi-abandoned project that uses PHash to canvas my photo library to flag duplicates, and am quite likely to use this instead. Now if only Apple…",{"id":107,"by":108,"text":109},"21181153","dandigangi","Sweet. Just what I needed to remove duplicate memes from my phone.",{"value":111,"source":73,"evidence":112},"Nice, well-scoped project.",[113,114],{"id":103,"by":104,"text":105},{"id":115,"by":116,"text":117},"21179576","cosmic_quanta","Nice project! I wonder how much of it could be adapted to finding duplicate documents, e.g. homeworks, CVs, etc. Presumably, the hashing would have to be adapted slightly. But…",[119,126,130],{"value":120,"source":73,"evidence":121},"Pinned dependency versions are too strict for a library.",[122],{"id":123,"by":124,"text":125},"21179015","orf","If this is a library then locking those dependencies down is not great. Does it really need Pillow 6.0.0, and not Pillow 6.0.1? As this is being consumed by larger applications…",{"value":127,"source":73,"evidence":128},"Heavy dependencies such as TensorFlow.",[129],{"id":76,"by":77,"text":78},{"value":131,"source":73,"evidence":132},"Does not scale beyond a couple thousand images locally.",[133],{"id":134,"by":92,"text":135},"21178879","Scale is unfortunately not the focus of the current implementation. We would address this aspect in the future releases however. Considering the speed and memory requirements,…",[137,144,155],{"value":138,"source":73,"evidence":139},"Support for animated GIFs.",[140],{"id":141,"by":142,"text":143},"21177378","innagadadavida","Any idea if there is something that supports animated gifs?",{"value":145,"source":73,"evidence":146},"Use approximate nearest neighbor search such as FAISS.",[147,151],{"id":148,"by":149,"text":150},"21179438","dgacmu","Use approximate nearest neighbor. The FAISS library is good.",{"id":152,"by":153,"text":154},"21181827","gravypod","What way is the hash represented? Have you looked at NN libraries like faiss [0] and NGT [1]? Those can quite easily handle a nearest neighbor search of 10 million vectors and,…",{"value":156,"source":73,"evidence":157},"Ordering images by similarity.",[158],{"id":159,"by":160,"text":161},"21178249","JonathanFly","Could this be used to order a large set of images by similarity?",[163,174],{"value":164,"source":73,"evidence":165},"How does performance scale to millions of images?",[166,170],{"id":167,"by":168,"text":169},"21177944","mkl","This looks really interesting. Can you give us an idea of the performance? E.g. roughly how long would it take to process 1 million 1920×1080 JPEGS, without GPU and with GPU?…",{"id":171,"by":172,"text":173},"21179288","jayalpha","\"The tricky part is the retrieval of duplicates, which on the same 10K dataset takes a few minutes\" \"Doesn't sound too bad.\" I doesn't? As a word of caution: If a 10k dataset…",{"value":175,"source":73,"evidence":176},"Whether it works for text documents.",[177,178],{"id":59,"by":60,"text":61},{"id":179,"by":180,"text":181},"21181845","herohamp","For documents I wouldnt do anything listed here. I would just compare the contents on a line by line or word by word basis",[183,190,197],{"value":184,"source":73,"evidence":185},"imagehash library",[186],{"id":187,"by":188,"text":189},"21181013","Dowwie","did you evaluate the 'imagehash' [1] library prior to working on this-- any limitations\u002Fconcerns? the additional CNN seems to be the difference between the two libraries [1]…",{"value":191,"source":73,"evidence":192},"PhotoDNA hash",[193],{"id":194,"by":195,"text":196},"21179768","jsjohnst","Did you consider using the PhotoDNA hash algo for finding duplicates? If you’ve heard of it and ruled it out, love to know why. While designed for a very different (and dark)…",{"value":198,"source":73,"evidence":199},"XnViewMP",[200],{"id":201,"by":202,"text":203},"21178742","wruza","If anyone is interested in gui, XnViewMP does that too. https:\u002F\u002Fwww.journeybytes.com\u002F2018\u002F03\u002Fhow-to-find-duplicate-photos-for-free-using-xnview.html",{"comments":43,"answered":205,"key":206},2,[207,208,211],{"id":51,"text":52},{"id":209,"text":210},"21181492","Yes you're right thanks for pointing this out!",{"id":212,"text":213},"21181450","Yes, before developing the package, we were also using this great library for hash generation. There are a bunch of differences we have compared to imagehash: 1. Added CNN as you…",{"commenters":35,"max_depth":215,"maker_replies":43,"first_comment_mins":216,"lifespan_hours":217},7,4,165,[219,221,225,229,232,236,238,240,243,245,246,249,251,252,254,257,259,261,262,264,266,268,270,272,275,277,278,280,283,284,286,288,290,292,296,297,300,303,306,309,312,314,317,321,324,326,329,330,332,335,338,339,342,345,347,349,350],[51,38,38,220,11,34],"maker\u002Fpositive\u002Fnone",[222,38,43,223,224,38],"21177044","project\u002Fneutral\u002Fidea","darkmighty",[226,38,38,227,228,38],"21177867","commenter\u002Fneutral\u002Fnone","heipei",[230,38,38,227,231,38],"21179325","bonoboTP",[233,38,38,234,235,38],"21178310","commenter\u002Fneutral\u002Fidea","jeeceebees",[141,38,43,237,142,38],"project\u002Fneutral\u002Fnone",[59,38,18,239,60,38],"project\u002Fpositive\u002Fperformance",[241,38,43,242,172,38],"21179290","maker\u002Fneutral\u002Fnone",[167,38,43,244,168,38],"project\u002Fneutral\u002Fperformance",[134,38,43,244,92,38],[247,38,43,244,248,38],"21178920","flud",[250,38,43,244,92,38],"21179081",[148,38,43,244,149,38],[253,38,38,227,92,38],"21179975",[255,38,43,244,256,38],"21179668","jonatron",[258,38,43,244,92,38],"21179933",[171,38,38,260,172,38],"commenter\u002Fneutral\u002Fperformance",[152,38,43,244,153,38],[263,38,43,244,92,38],"21196161",[265,38,18,220,153,38],"21197437",[267,38,43,244,256,38],"21178950",[269,38,43,244,168,38],"21179034",[271,38,43,244,256,38],"21179149",[273,38,18,274,92,38],"21179940","project\u002Fpositive\u002Fexecution",[276,38,38,260,256,38],"21180174",[91,38,43,244,92,38],[279,38,43,244,256,38],"21223070",[281,38,18,220,282,38],"21178069","amrrs",[194,38,43,223,195,38],[285,38,43,223,92,38],"21180207",[76,34,43,287,77,38],"project\u002Fmixed\u002Fexecution",[123,34,34,289,124,38],"project\u002Fcritical\u002Fexecution",[209,34,38,291,11,34],"maker\u002Fneutral\u002Fexecution",[293,205,38,294,295,38],"21179511","other\u002Fneutral\u002Fnone","donatj",[201,43,38,294,202,38],[298,216,38,227,299,38],"21179106","shifto",[301,216,43,223,302,38],"21180461","ptaffs",[304,18,38,294,305,38],"21177843","sekasi",[307,18,38,234,308,38],"21177879","chrischen",[310,18,38,294,311,38],"21178020","throwaway_bad",[313,18,43,244,92,38],"21196283",[315,18,38,294,316,38],"21178206","mosselman",[318,18,38,319,320,38],"21178269","other\u002Fneutral\u002Fperformance","cheschire",[322,18,38,260,323,38],"21178280","tdhoot",[325,18,38,294,195,38],"21179821",[115,327,18,328,116,38],6,"project\u002Fpositive\u002Fidea",[179,327,38,227,180,38],[103,215,18,331,104,38],"project\u002Fpositive\u002Fnone",[333,215,18,331,334,38],"21178152","wertenshap",[336,215,43,237,337,38],"21178254","ggm",[159,32,43,223,160,38],[340,32,43,223,341,38],"21180327","jlg23",[343,32,43,344,92,38],"21196247","project\u002Fneutral\u002Fdocs",[346,32,43,223,92,38],"21180352",[187,348,43,223,188,38],9,[212,348,38,223,11,34],[107,351,18,331,108,38],10,[],[354,359,364,369,374,378],{"id":355,"name":356,"points":357,"category":358},"21175492","FeastGenius",422,"productivity",{"id":360,"name":361,"points":362,"category":363},"21207132","uPlot.js",260,"data",{"id":365,"name":366,"points":367,"category":368},"21157066","Able",248,"web_saas",{"id":370,"name":371,"points":372,"category":373},"21147058","Scale Heaven",231,"creative",{"id":375,"name":376,"points":377,"category":358},"21179190","spotify-tui",212,{"id":379,"name":380,"points":381,"category":20},"21179037","Small Clojure Interpreter",143,false,"v2",{"id":385,"name":5,"line":6,"ai":382,"cat":20,"year":13,"points":14,"comments":15,"rec":29,"s":386,"top":18,"kw":387,"maker":11,"domain":10,"tier":19,"attempts":38,"strip":388,"url":9,"mix":389,"praise":100,"objection":120,"sub":21},21157463,0.24,[24,25,26,27,28],"0300533333333353503533100300300055533335",[32,33,34,34],1794036581559]