[{"data":1,"prerenderedAt":270},["ShallowReactive",2],{"project-32823589":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":28,"sentiment":29,"stance_mix":30,"scored_comments":35,"aspects":36,"strip":40,"post_text":42,"what":43,"said":81,"maker_in_thread":162,"shape":171,"grid":175,"attempts":230,"same_week":231,"maker_shows":38,"maker_hidden":262,"reader_version":263,"card":264},"32823589","pg_netstat","A Postgres extension to monitor database network traffic","title","Show HN: pg_netstat, a Postgres extension to monitor database network traffic","https:\u002F\u002Fgithub.com\u002Fsupabase\u002Fpg_netstat","github.com","burmecia",1663073478,2022,158,25,383,2570,15,"notable","data","database monitoring extension","library",[24,25,26,27],"postgres extension","network traffic monitoring","libpcap","database egress","curious",-0.263,{"positive":31,"neutral":32,"mixed":33,"critical":34},1,4,0,3,8,{"idea":37,"performance":39},{"positive":31,"critical":38},2,{"positive":33,"critical":31},[34,33,34,33,34,33,33,33,41,31,33,33,33,34,33,33,31,31,33,33,33,33,33,33,33],5,"pg_netstat is a Postgres extension to monitor database network traffic. It uses libpcap to capture packets and aggregates at user-specified interval.",{"problem":44,"audience":51,"pricing":60,"motivation":61,"time_to_build":60,"stack":65,"openness":78,"stage":79,"team_size":80},{"value":45,"source":46,"evidence":47},"Ingesting real-time database network traffic data into Postgres for analysis.","maker",[48],{"id":49,"by":11,"text":50},"32823980","Hey HN, I have spent some time to search for a tool that can ingest realtime network traffic data to Postgres but have no luck, so I developed this extension and used it…",{"value":52,"source":53,"evidence":54},"Postgres users monitoring database network traffic.","inferred",[55,56],{"id":49,"by":11,"text":50},{"id":57,"by":58,"text":59},"32828772","adamcharnock","I've written a tool to do this not once but twice now! The first was for ingesting netflow data, the second from ingesting AWS VPC flow logs. In both cases this was storing…",null,{"value":62,"source":46,"evidence":63},"Could not find a tool to ingest real-time network traffic into Postgres, so built one for internal team use.",[64],{"id":49,"by":11,"text":50},[66,70,74],{"value":67,"source":46,"evidence":68},"Rust",[69],{"id":49,"by":11,"text":50},{"value":71,"source":46,"evidence":72},"pgx",[73],{"id":49,"by":11,"text":50},{"value":26,"source":46,"evidence":75},[76],{"id":77},"post","unknown","launched","team",{"praise":82,"objections":88,"requests":103,"questions":108,"alternatives":116},[83],{"value":84,"source":85,"evidence":86},"Useful approach for getting network data into Postgres.","commenter",[87],{"id":57,"by":58,"text":59},[89,96],{"value":90,"source":85,"evidence":91},"Requiring packet capture permission for the Postgres binary is a security concern.",[92],{"id":93,"by":94,"text":95},"32823977","xani_","> Before install this extension, you need to give network packet capture permission to Postgres binary. At that point you have root and can use any of the dozen estabilished ways…",{"value":97,"source":85,"evidence":98},"Postgres is a poor time series database compared with dedicated TSDBs.",[99],{"id":100,"by":101,"text":102},"32824803","londons_explore","Nearly all usecases for this seem to be recreating a time series database... Postgres isn't a great TSDB - the indexes don't understand columns that will only have updates at one…",[104],{"value":105,"source":46,"evidence":106},"Feature requests and bug reports welcomed.",[107],{"id":49,"by":11,"text":50},[109],{"value":110,"source":85,"evidence":111},"How are the stats used for monitoring?",[112],{"id":113,"by":114,"text":115},"32824493","Dowwie","I'm curious as to how you're using these stats for monitoring. What kind of insights are you gaining with this, and in what context is it used?",[117,124,131,138,142,149,156],{"value":118,"source":85,"evidence":119},"nftables counters with jq and Zabbix.",[120],{"id":121,"by":122,"text":123},"32824314","merpkz","could achieve something similar a bit different way - by using Linux nftables ( new iptables ) netfilter interface. I have setup like this for measuring used traffic by certain…",{"value":125,"source":85,"evidence":126},"TimescaleDB.",[127],{"id":128,"by":129,"text":130},"32825055","jpgvm","Or just use TimescaleDB which is competitive on most axes but already PG.",{"value":132,"source":85,"evidence":133},"ClickHouse.",[134],{"id":135,"by":136,"text":137},"32825124","mritchie712","or clickhouse!",{"value":139,"source":85,"evidence":140},"InfluxDB.",[141],{"id":100,"by":101,"text":102},{"value":143,"source":85,"evidence":144},"VictoriaMetrics.",[145],{"id":146,"by":147,"text":148},"32833941","dengolius","I recommend to use VictoriaMetrics instead of using InfluxDB for many reasons. PostgreSQL is good SQL database for different purposes but as TSDB on a big installations it works…",{"value":150,"source":85,"evidence":151},"file_fdw reading \u002Fproc files.",[152],{"id":153,"by":154,"text":155},"32826692","hyperman1","For another alternative: I added some monitoring, by using file_fdw. This is a standard pg extension that presents files or program stdout as tables. So I file_fdw'd some files…",{"value":157,"source":46,"evidence":158},"tcpdump.",[159],{"id":160,"by":11,"text":161},"32824039","Yes, you need root to grant capture permission to Postgres, but Postgres doesn't need to run as root. The main purpose is to easily ingest network traffic data into Postgres for…",{"comments":41,"answered":34,"key":163},[164,167,168],{"id":165,"text":166},"32824367","That's a nice approach, thanks for sharing with us.",{"id":49,"text":50},{"id":169,"text":170},"32824755","Our price model based on database usage which includes db egress, so we use it to monitor db egress traffic just for now. We are all db fans and we like put things into db, so…",{"commenters":172,"max_depth":41,"maker_replies":41,"first_comment_mins":173,"lifespan_hours":174},19,35,42,[176,178,180,183,185,186,188,192,195,197,199,200,203,205,206,209,211,215,216,217,220,223,226,227,229],[121,33,34,177,122,33],"project\u002Fneutral\u002Fidea",[165,33,33,179,11,31],"commenter\u002Fneutral\u002Fnone",[181,33,34,177,182,33],"32830702","Self-Perfection",[49,31,33,184,11,31],"maker\u002Fneutral\u002Fnone",[113,31,34,184,114,33],[169,31,33,187,11,31],"maker\u002Fneutral\u002Fpricing",[189,31,33,190,191,33],"32825047","other\u002Fneutral\u002Fnone","nathancahill",[193,31,33,190,194,33],"32827050","jrockway",[57,31,41,196,58,33],"project\u002Fpositive\u002Fidea",[93,38,31,198,94,33],"project\u002Fcritical\u002Fidea",[160,38,33,177,11,31],[201,38,33,179,202,33],"32824236","mrits",[204,38,33,184,11,31],"32824409",[153,34,34,177,154,33],[207,34,33,179,208,33],"32828870","djbusby",[210,34,33,179,154,33],"32848207",[212,32,31,213,214,33],"32826589","project\u002Fcritical\u002Fperformance","zoobab",[100,41,31,198,101,33],[128,41,33,190,129,33],[218,41,33,190,219,33],"32825459","_joel",[221,41,33,190,222,33],"32826481","GordonS",[224,41,33,190,225,33],"32836561","LoriP",[135,41,33,190,136,33],[228,41,33,190,147,33],"32833950",[146,41,33,190,147,33],[],[232,237,242,247,252,257],{"id":233,"name":234,"points":235,"category":236},"32768590","Modern for HN",308,"productivity",{"id":238,"name":239,"points":240,"category":241},"32798891","HiFiScan",260,"creative",{"id":243,"name":244,"points":245,"category":246},"32828309","Infernal Sigil",238,"games",{"id":248,"name":249,"points":250,"category":251},"32837829","StackAid",226,"infrastructure",{"id":253,"name":254,"points":255,"category":256},"32828799","s3sqlite",160,"dev_tools",{"id":258,"name":259,"points":260,"category":261},"32833907","dato",134,"ai",false,"v2",{"id":265,"name":5,"line":6,"ai":262,"cat":20,"year":13,"points":14,"comments":15,"rec":28,"s":266,"top":18,"kw":267,"maker":11,"domain":10,"tier":19,"attempts":33,"strip":268,"url":9,"mix":269,"praise":84,"objection":90,"sub":21},32823589,-0.26,[24,25,26,27],"3030300051000300110000000",[31,32,33,34],1794036624952]