Difference between revisions of "Clickhouse"
From Initech Technical Wiki
| (3 intermediate revisions by the same user not shown) | |||
| Line 9: | Line 9: | ||
=== Visualisation === | === Visualisation === | ||
| − | I use grafana to visualise the data using this [[clickhouse-netflow-dashboard|dashboard]], you'll need the [https://grafana.com/grafana/plugins/grafana-piechart-panel/ piechart] plugin, never mind about [https://grafana.com/blog/2015/12/04/friends-dont-let-friends-abuse-pie-charts/ friends don't let friends abuse pie-charts] | + | I use [[grafana]] to visualise the data using this [[clickhouse-netflow-dashboard|dashboard]], you'll need the [https://grafana.com/grafana/plugins/grafana-piechart-panel/ piechart] plugin, never mind about [https://grafana.com/blog/2015/12/04/friends-dont-let-friends-abuse-pie-charts/ friends don't let friends abuse pie-charts] |
| − | == [[XC5 | + | == [[XC5]] Softswitch == |
This is a collection of scripts which export CDR records from an XC5 softswitch, import them into [[clickhouse]] and visualise them using a [[grafana]] dashboard. | This is a collection of scripts which export CDR records from an XC5 softswitch, import them into [[clickhouse]] and visualise them using a [[grafana]] dashboard. | ||
=== Importing CDRs === | === Importing CDRs === | ||
# This step is triggered by cron (i run mine every minute) and calls a php script called [[clickhouse-cdr-import.php]] which logs into the XC5 JSON api, exports the CDRs and imports them into [[clickhouse]] into the [[softswitch.cdr]] table | # This step is triggered by cron (i run mine every minute) and calls a php script called [[clickhouse-cdr-import.php]] which logs into the XC5 JSON api, exports the CDRs and imports them into [[clickhouse]] into the [[softswitch.cdr]] table | ||
| + | |||
| + | === Visualisation === | ||
| + | Again I use [[grafana]] to visualise the data using this [[clickhouse-softswitch-dashboard|dashboard]], and again you'll need the [https://grafana.com/grafana/plugins/grafana-piechart-panel/ piechart] plugin. | ||
Latest revision as of 04:07, 19 October 2019
You'll find my collection of scripts and things related to Clickhouse here.
Contents
pmacct
This is a collection of configurations and scripts which effectively enable netflows/IPFIX records to be collected and imported into a clickhouse database.
Importing netflows
- pmacct uses nfacct.conf to collect netflows/IPFIX records and output them to /tmp/nfacct.json in JSONEachRow format and then triggers the nfacct-clickhouse-import script.
- The nfacct-clickhouse-import script pre-filters the /tmp/nfacct.json file using grep and PCRE patterns stored in nfacct_drop_patterns and imports what is left into clickhouse into the netflow.nfacct table.
Visualisation
I use grafana to visualise the data using this dashboard, you'll need the piechart plugin, never mind about friends don't let friends abuse pie-charts
XC5 Softswitch
This is a collection of scripts which export CDR records from an XC5 softswitch, import them into clickhouse and visualise them using a grafana dashboard.
Importing CDRs
- This step is triggered by cron (i run mine every minute) and calls a php script called clickhouse-cdr-import.php which logs into the XC5 JSON api, exports the CDRs and imports them into clickhouse into the softswitch.cdr table
Visualisation
Again I use grafana to visualise the data using this dashboard, and again you'll need the piechart plugin.