Difference between revisions of "Clickhouse"

Revision as of 21:53, 18 October 2019

You'll find my collection of scripts and things related to Clickhouse here.

pmacct

This is a collection of configurations and scripts which effectively enable netflows/IPFIX records to be collected and imported into a clickhouse database.

Importing netflows

pmacct uses nfacct.conf to collect netflows/IPFIX records and output them to /tmp/nfacct.json in JSONEachRow format and then triggers the nfacct-clickhouse-import script.
The nfacct-clickhouse-import script pre-filters the /tmp/nfacct.json file using grep and PCRE patterns stored in nfacct_drop_patterns and imports what is left into clickhouse into the netflow.nfacct table.

Visualisation

I use grafana to visualise the data using this dashboard, you'll need the [1] plugin, never mind about the don't let friends abuse pie-charts

@@ Line 7: / Line 7: @@
 # [[pmacct]] uses [[pmacct|nfacct.conf]] to collect netflows/IPFIX records and output them to /tmp/nfacct.json in JSONEachRow format and then triggers the [[nfacct-clickhouse-import]] script.
 # The [[nfacct-clickhouse-import]] script pre-filters the /tmp/nfacct.json file using grep and PCRE patterns stored in [[nfacct_drop_patterns]] and imports what is left into [[clickhouse]] into the [[netflow.nfacct]] table.
+=== Visualisation ===
+I use grafana to visualise the data using this [[clickhouse-netflow-dashboard|dashboard]], you'll need the [https://grafana.com/grafana/plugins/grafana-piechart-panel|piechart] plugin, never mind about the [https://grafana.com/blog/2015/12/04/friends-dont-let-friends-abuse-pie-charts/|friends don't let friends abuse pie-charts]