Difference between revisions of "Nfacct-clickhouse-import"
From Initech Technical Wiki
| (One intermediate revision by the same user not shown) | |||
| Line 1: | Line 1: | ||
| − | This script expects a json formatted (JSONEachRow in [[clickhouse]] terminology) input file at /tmp/nfacctd.json. It uses /usr/local/bin/[[nfacct_drop_patterns]] to pre-filter out some flows that we don't care about and then imports the remainder into [[clickhouse]]. | + | This script expects a json formatted (JSONEachRow in [[clickhouse]] terminology) input file comprising netflow / IPFIX records exported from [[pmacct]] at /tmp/nfacctd.json. It uses /usr/local/bin/[[nfacct_drop_patterns]] to pre-filter out some flows that we don't care about and then imports the remainder into [[clickhouse]] into the [[netflow.nfacct]] table. |
It also imports the whole file without filters into a seperate table so that post-import DOS analysis can be done by a seperate script. (work in progress) | It also imports the whole file without filters into a seperate table so that post-import DOS analysis can be done by a seperate script. (work in progress) | ||
Latest revision as of 21:23, 18 October 2019
This script expects a json formatted (JSONEachRow in clickhouse terminology) input file comprising netflow / IPFIX records exported from pmacct at /tmp/nfacctd.json. It uses /usr/local/bin/nfacct_drop_patterns to pre-filter out some flows that we don't care about and then imports the remainder into clickhouse into the netflow.nfacct table.
It also imports the whole file without filters into a seperate table so that post-import DOS analysis can be done by a seperate script. (work in progress)
#!/bin/bash cat /tmp/nfacctd.json | grep -v -f /usr/local/bin/nfacct_drop_patterns | clickhouse-client --input_format_skip_unknown_fields=1 --query="INSERT INTO netflow.nfacct FORMAT JSONEachRow" clickhouse-client --query="truncate table netflow.nfacct_dos" cat /tmp/nfacctd.json | clickhouse-client --input_format_skip_unknown_fields=1 --query="INSERT INTO netflow.nfacct_dos FORMAT JSONEachRow"