Difference between revisions of "Clickhouse"
From Initech Technical Wiki
| Line 1: | Line 1: | ||
You'll find my collection of scripts and things related to [[Clickhouse]] here. | You'll find my collection of scripts and things related to [[Clickhouse]] here. | ||
| − | + | == [[pmacct]] == | |
| + | This is a collection of configurations and scripts which effectively enable netflows/IPFIX records to be collected and imported into a [[clickhouse]] database. | ||
*# [[pmacct]] uses [[pmacct|nfacct.conf]] to collect netflows/IPFIX records and output them to /tmp/nfacct.json in JSONEachRow format and then triggers the [[nfacct-clickhouse-import]] script. | *# [[pmacct]] uses [[pmacct|nfacct.conf]] to collect netflows/IPFIX records and output them to /tmp/nfacct.json in JSONEachRow format and then triggers the [[nfacct-clickhouse-import]] script. | ||
*# The [[nfacct-clickhouse-import]] script pre-filters the /tmp/nfacct.json file using grep and PCRE patterns stored in [[nfacct_drop_patterns]] and imports what is left into [[clickhouse]] into the [[netflow.nfacct]] table. | *# The [[nfacct-clickhouse-import]] script pre-filters the /tmp/nfacct.json file using grep and PCRE patterns stored in [[nfacct_drop_patterns]] and imports what is left into [[clickhouse]] into the [[netflow.nfacct]] table. | ||
Revision as of 21:44, 18 October 2019
You'll find my collection of scripts and things related to Clickhouse here.
pmacct
This is a collection of configurations and scripts which effectively enable netflows/IPFIX records to be collected and imported into a clickhouse database.
- pmacct uses nfacct.conf to collect netflows/IPFIX records and output them to /tmp/nfacct.json in JSONEachRow format and then triggers the nfacct-clickhouse-import script.
- The nfacct-clickhouse-import script pre-filters the /tmp/nfacct.json file using grep and PCRE patterns stored in nfacct_drop_patterns and imports what is left into clickhouse into the netflow.nfacct table.