[{"data":1,"prerenderedAt":426},["ShallowReactive",2],{"project-38902042":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":28,"sentiment":29,"stance_mix":30,"scored_comments":35,"aspects":36,"strip":46,"post_text":49,"what":50,"said":113,"maker_in_thread":256,"shape":263,"grid":267,"attempts":380,"same_week":391,"maker_shows":42,"maker_hidden":419,"reader_version":420,"card":421},"38902042","Quickwit","OSS Alternative to Elasticsearch, Splunk, Datadog","title","Show HN: Quickwit – OSS Alternative to Elasticsearch, Splunk, Datadog","https:\u002F\u002Fgithub.com\u002Fquickwit-oss\u002Fquickwit","github.com","francoismassot",1704641739,2024,145,51,584,3515,17,"notable","data","search engine for logs","other",[24,25,26,27],"elasticsearch alternative","log search","open source search engine","observability","curious",-0.04,{"positive":31,"neutral":32,"mixed":33,"critical":34},8,16,0,6,30,{"performance":37,"execution":39,"docs":40,"design":41,"idea":43,"privacy":44,"pricing":45},{"positive":38,"critical":33},1,{"positive":38,"critical":38},{"positive":33,"critical":33},{"positive":33,"critical":42},2,{"positive":38,"critical":38},{"positive":33,"critical":33},{"positive":42,"critical":42},[47,33,38,47,33,48,33,47,48,47,48,33,38,33,33,47,33,33,48,47,47,48,47,33,38,47,38,33,47,33,48,47,47,38,33,38,48,48,47,47],3,5,"Hi folks, Quickwit cofounder here.\n\nWe started Quickwit 3 years ago with a POC, \"Searching the web for under $1000\u002Fmonth\" (see HN discussions [0]), with the goal of making a robust OSS alternative to Elasticsearch \u002F Splunk \u002F Datadog.\n\nWe have reached a significant milestone with our latest release (0.7) [1], as we have witnessed users of the nightly version of Quickwit deploy clusters with hundreds of nodes, ingest hundreds of terabytes of data daily, and enjoy considerable cost savings.\n\nTo give you a concrete example, one company is ingesting hundreds of terabytes of logs daily and migrating",{"problem":51,"audience":57,"pricing":66,"motivation":75,"time_to_build":79,"stack":80,"openness":110,"stage":111,"team_size":112},{"value":52,"source":53,"evidence":54},"Expensive log and observability search that scales poorly and costs a lot in storage and compute","maker",[55],{"id":56},"post",{"value":58,"source":59,"evidence":60},"Teams running large-scale logs and observability workloads","inferred",[61,62],{"id":56},{"id":63,"by":64,"text":65},"38923388","fulmicoton","Most users do not use Kafka\u002FZookeeper. The only external service for them is a S3 bucket. They then use the PushAPI. It is perfectly fine if you have only a couple of TB a day.…",{"value":67,"source":53,"evidence":68},"Licensed under AGPLv3; commercial license available for companies that need it",[69,72],{"id":70,"by":11,"text":71},"38925576","Quickwit is under AGPLv3. Are you saying that AGPLv3 is not FOSS?",{"id":73,"by":11,"text":74},"38904018","Well, I would say it depends. We have many companies using the AGPL version without buying a license. We also know that some companies have strict policies and will forbid using…",{"value":76,"source":53,"evidence":77},"Build a robust open source alternative to Elasticsearch, Splunk and Datadog",[78],{"id":56},null,[81,89,95,104],{"value":82,"source":83,"evidence":84},"Rust","commenter",[85],{"id":86,"by":87,"text":88},"38921634","evanjrowley","I'm surprised no one has mentioned that it's mostly programmed in Rust.",{"value":90,"source":53,"evidence":91},"Tantivy search library",[92],{"id":93,"by":11,"text":94},"38902751","So Quickwit is primarily a search engine and thus relies on an inverted index. We also implemented our schemaless columnar storage optimized for object storage. The inverted…",{"value":96,"source":53,"evidence":97},"S3-compatible object storage",[98,101],{"id":99,"by":11,"text":100},"38902558","We store the indexes on the object storage. We worked on the index data structure to optimize the query path. For use cases where you have a lot of QPS, we recommend using Garage…",{"id":102,"by":64,"text":103},"38939943","Quickwit is compatible with S3-compatible object storages yes. I don't remember the feedback for R2 specifically, but we have users running Quickwit on S3, GFS, Azure, MinIO,…",{"value":105,"source":53,"evidence":106},"PostgreSQL metastore (optional)",[107],{"id":108,"by":11,"text":109},"38923401","Quickwit supports different data sources: Kafka, Pulsar, Google pubsub, ... and we have our own ingest API (not HA right now, but it will be the case in the next release in 1…","open_source","launched","team",{"praise":114,"objections":144,"requests":168,"questions":203,"alternatives":228},[115,126,133],{"value":116,"source":83,"evidence":117},"Cost savings and performance compared with Elasticsearch",[118,122],{"id":119,"by":120,"text":121},"38922545","dikei","> To give you a concrete example, one company is ingesting hundreds of terabytes of logs daily and migrating from Elasticsearch to Quickwit. They divided their compute costs by…",{"id":123,"by":124,"text":125},"38922954","marsupialtail_2","Glad this is getting some love. This is seriously good software. Have you guys supported generic substring search yet? I recall it was not supported as of a few months ago.",{"value":127,"source":83,"evidence":128},"Object-storage design allows horizontal scaling with S3-compatible storage",[129],{"id":130,"by":131,"text":132},"38904208","jszymborski","The most interesting part of Quickwit for me has been that because it stores the index in object stores, you can theoretically abstract away a lot of the horizontal scaling with…",{"value":134,"source":83,"evidence":135},"AGPL seen as a fitting free license for internal use",[136,140],{"id":137,"by":138,"text":139},"38904878","dvfjsdhgfv","I find AGPL perfect for this use case, and my org (>100k hosts) can use it without any problem as we are using it for internal purposes, and not, say, rebranding it and offering…",{"id":141,"by":142,"text":143},"38904551","dwb","The GNU AGPL is considered a Free Software licence by the OSI, FSF, and the Debian project. That's good enough for me.",[145,152,163],{"value":146,"source":83,"evidence":147},"Deployment seems complex with Kafka and other dependencies",[148],{"id":149,"by":150,"text":151},"38921801","figmert","Looking at the docker compose, this seems like a very complicated tool to run. You'd need Kafka, zookeeper, and Jaeger. All would need to be HA. Then also this service. Not…",{"value":153,"source":83,"evidence":154},"Licence criticised as not truly open source or commercially restrictive",[155,159],{"id":156,"by":157,"text":158},"38925393","besus","That license is not FOSS. Users beware of the impending subscription costs to come.",{"id":160,"by":161,"text":162},"38903106","jcwayne","I find presenting this as an open source alternative to commercial solutions a little disingenuous when any commercial use of it also requires a paid license. Like many other…",{"value":164,"source":83,"evidence":165},"Lacks substring search and cardinality aggregation",[166,167],{"id":123,"by":124,"text":125},{"id":102,"by":64,"text":103},[169,179,186,195],{"value":170,"source":83,"evidence":171},"Kibana\u002FOpenDashboard compatibility",[172,176],{"id":173,"by":174,"text":175},"38905110","BrandoElFollito","What I am really missing in these (really nice) alternatives to ELK is Kibana and its Lens. I tried to tapping this with Grafana but never learned to have graphs as easily as…",{"id":177,"by":11,"text":178},"38905212","Good point. Several users are asking us the OpenDashboard\u002FKibana compatibility, and this is on the 2024 roadmap. That being said, we also hear users complaining about…",{"value":180,"source":83,"evidence":181},"Elasticsearch Query DSL compatibility",[182],{"id":183,"by":184,"text":185},"38924532","m1sta_","How far away is the ES Query DSL compatibility?",{"value":187,"source":83,"evidence":188},"Support for updates and deletes",[189,192],{"id":190,"by":11,"text":191},"38904305","So yes, currently we only support infrequent deletes for GDPR reasons mainly. It's possible to add updates\u002Fdeletes to Quickwit, but this is a lot of work, and for now, we have…",{"id":193,"by":131,"text":194},"38908908","Oh, just run-of-the-mill full-text search of web crawls. Some web pages update frequently, so I would need something that can handle that. I do understand that you seem to be…",{"value":196,"source":83,"evidence":197},"Analytics and retention analysis",[198,202],{"id":199,"by":200,"text":201},"38938971","matesz","I am impressed with this work, especially given our current use of Loki with Prometheus and Grafana. Few questions come to mind. Firstly, is Quickwit compatible with any S3…",{"id":102,"by":64,"text":103},[204,212,223],{"value":205,"source":83,"evidence":206},"How easy is it to run and maintain compared with Kafka-based setups",[207,208],{"id":149,"by":150,"text":151},{"id":209,"by":210,"text":211},"38923367","PaoloBarbolini","According to the documentation[1] Kafka is just one of the supported inputs for ingestion, so it should be possible to run Quickwit without it if you're not intending to write…",{"value":213,"source":83,"evidence":214},"How it compares with SigNoz",[215,219],{"id":216,"by":217,"text":218},"38923628","wiradikusuma","How is it compared to Signoz, which is also open source?",{"id":220,"by":221,"text":222},"38925195","pranay01","SigNoz maintainer here. We also have traces, metrics and logs in a single application which makes correlation across them much easier. From what I can understand from Quickwit…",{"value":224,"source":83,"evidence":225},"Whether it works with S3-compatible storage such as R2",[226,227],{"id":199,"by":200,"text":201},{"id":102,"by":64,"text":103},[229,237,245,252],{"value":230,"source":83,"evidence":231},"Elasticsearch",[232,233],{"id":119,"by":120,"text":121},{"id":234,"by":235,"text":236},"38922648","laserbeam","Elasticsearch was written in Java. No one cared. We still used it because it was good software.",{"value":238,"source":83,"evidence":239},"SigNoz",[240,244],{"id":241,"by":242,"text":243},"38923654","sidcool","When open source providers say 'alternative' to a commercial solution, do they consider the serious engineering needed to scale such systems? I mean I used SigNoz, and it's…",{"id":216,"by":217,"text":218},{"value":246,"source":83,"evidence":247},"ClickHouse",[248],{"id":249,"by":250,"text":251},"38902620","debarshri","Why don't you use clickhouse or some columnar storage for the logs? You get indexing or querying for free.",{"value":253,"source":83,"evidence":254},"Grafana",[255],{"id":173,"by":174,"text":175},{"comments":31,"answered":48,"key":257},[258,259,260],{"id":108,"text":109},{"id":177,"text":178},{"id":261,"text":262},"38923798","I must admit that 'alternative' is always a tricky word... Datadog, Elasticsearch, and Splunk are giant beasts, and the alternative makes sense only on a subset of features (and…",{"commenters":264,"max_depth":34,"maker_replies":31,"first_comment_mins":265,"lifespan_hours":266},28,54,376,[268,270,273,275,277,279,283,285,287,289,291,293,294,297,299,301,304,305,308,312,314,317,318,321,322,324,327,332,333,336,337,339,340,342,343,344,346,350,352,355,357,360,361,364,366,368,369,371,372,375,377],[119,33,47,269,120,33],"project\u002Fneutral\u002Fperformance",[271,33,33,272,64,33],"38923182","commenter\u002Fneutral\u002Fnone",[149,38,38,274,150,33],"project\u002Fcritical\u002Fexecution",[209,38,47,276,210,33],"project\u002Fneutral\u002Fdocs",[278,38,33,272,64,33],"38925130",[280,38,33,281,282,33],"38931279","other\u002Fneutral\u002Fnone","mdaniel",[63,38,48,284,64,33],"project\u002Fpositive\u002Fexecution",[108,38,33,286,11,38],"maker\u002Fneutral\u002Fnone",[173,42,47,288,174,33],"project\u002Fneutral\u002Fdesign",[177,42,33,290,11,38],"maker\u002Fneutral\u002Fdesign",[199,47,48,292,200,33],"project\u002Fpositive\u002Fnone",[102,47,47,286,64,33],[295,47,48,296,200,33],"38942358","maker\u002Fpositive\u002Fnone",[298,47,33,281,200,33],"38991488",[300,47,33,281,64,33],"39100519",[241,302,38,303,242,33],4,"project\u002Fcritical\u002Fidea",[261,302,33,286,11,38],[306,302,33,307,221,33],"38925102","other\u002Fneutral\u002Fperformance",[309,302,33,310,311,33],"38925220","commenter\u002Fneutral\u002Fperformance","ankitnayan",[86,48,47,313,87,33],"project\u002Fneutral\u002Fnone",[315,48,33,281,316,33],"38921989","esafak",[234,48,33,281,235,33],[319,48,33,281,320,33],"38923880","datadeft",[123,34,48,292,124,33],[323,34,47,286,64,33],"38923732",[325,34,47,326,124,33],"38929519","project\u002Fneutral\u002Fidea",[328,329,48,330,331,33],"38922726",7,"project\u002Fpositive\u002Fidea","jjtang1",[183,31,47,313,184,33],[334,335,47,288,250,33],"38902524",9,[99,335,33,290,11,38],[249,335,38,338,250,33],"project\u002Fcritical\u002Fdesign",[93,335,33,290,11,38],[341,335,47,269,64,33],"38907770",[130,335,38,338,131,33],[190,335,33,290,11,38],[193,335,47,345,131,33],"project\u002Fneutral\u002Fprivacy",[347,335,33,348,349,33],"38922723","commenter\u002Fneutral\u002Fprivacy","piperswe",[351,335,33,348,131,33],"38923192",[353,335,48,354,64,33],"38923402","project\u002Fpositive\u002Fperformance",[216,356,47,313,217,33],10,[358,356,47,313,359,33],"38923747","jakozaur",[220,356,33,281,221,33],[156,362,38,363,157,33],11,"project\u002Fcritical\u002Fpricing",[70,362,33,365,11,38],"maker\u002Fneutral\u002Fpricing",[160,367,38,363,161,33],12,[73,367,33,365,11,38],[141,367,48,370,142,33],"project\u002Fpositive\u002Fpricing",[137,367,48,370,138,33],[373,367,47,374,64,33],"38907797","project\u002Fneutral\u002Fpricing",[376,367,47,374,320,33],"38923893",[378,367,47,374,379,33],"38926424","hippich",[381,384,387,390],{"id":382,"t":383,"points":335,"comments":38},"35196082",1679055561,{"id":385,"t":386,"points":38,"comments":33},"37976084",1697987764,{"id":388,"t":389,"points":31,"comments":42},"38877728",1704451911,{"id":4,"t":12,"points":14,"comments":15},[392,397,401,406,410,415],{"id":393,"name":394,"points":395,"category":396},"38901504","Nino",810,"productivity",{"id":398,"name":399,"points":400,"category":396},"38876707","time",758,{"id":402,"name":403,"points":404,"category":405},"38906989","HTMX Playground",482,"dev_tools",{"id":407,"name":408,"points":409,"category":405},"38915999","Auto Wiki",183,{"id":411,"name":412,"points":413,"category":414},"38905590","scientific.place",165,"learning_science",{"id":416,"name":417,"points":418,"category":414},"38869672","sip-to-webrtc",154,false,"v2",{"id":422,"name":5,"line":6,"ai":419,"cat":20,"year":13,"points":14,"comments":15,"rec":28,"s":29,"top":18,"kw":423,"maker":11,"domain":10,"tier":19,"attempts":302,"strip":424,"url":9,"mix":425,"praise":116,"objection":146,"sub":21},38902042,[24,25,26,27],"3013050353501003005335301310305331015533",[31,32,33,34],1794036647837]