NetoFlow User Guide
Overview
NetoFlow Connector is software based flow collector that customers can run on premises. It currently supports Netflow versions 5, 9 and 10(IPFIX) and Sflow version 5. The NetoFlow Connector accepts all versions of flow on a single configurable UDP port and delivers the parsed flow data to the Netography cloud via HTTPS and optionally tees it off to an array of nodes.
Teeing Flows
The NetoFlow Connector has the ability to tee flow packets to an array of targets. The NetoFlow Connector will preserve the source IP of the device sending it flows. This is useful because often times you want to send flow to an array of tools within the infrastructure. Teeing in the netoflow accomplishes this while preserving this visibility to the other tools of what device the flows originated from. Refer to the "tees" section of the configuration sample for an example.
Flow Delivery Performance Tuning
In the general section of the configuration there are several options to modify the behavior flow delivery to the cloud. By default flows will be delivered to the cloud when either of 2 conditions are hit, a 5 second flushinterval timer or 2000 flow records. Depending on the amount of traffic the NetoFlow Connector is handling, one might wish to modify this behavior.
Flow Filters
Users can define flow filters to exclude flow data for that traffic from being sent into the cloud. This is useful in the case where specific data governance or particular sensitive applications can not be sent to the cloud. This might also be useful if you would like to reduce the data in the Netography portal. Flow filters behave very much like firewall rules where you have a default behavior and then set an action in individual rules. Flow filters can contain a combination of the following array fields: srcnets, dstnets, srcports, dstports, protocols, srcasns, dstasns. The fields within a flow filter are processed as an “and”. Within a field, if any array member matches that field is a match. All fields must match for a flow filter to match. Dropped records are logged at debug level. You can have many flow filters as they are currently not capped. This is primarily CPU bound but impact should be minimal. Refer to the "flowfilter" example in the configuration sample in this document and note the filter default setting which defaults to "allow" as the normal case is to send Netography all flows.
A flow filter must contain an action and at least 1 set of criteria to match on.
Client Side Downsampling
Users may define an additional downsample to their flow as needed. This can be done one of two ways:
- Apply a global downsample to any device netoflow sees flow from. (general section of configuration file "downsample")
- Apply a device specific downsample by adding a device block to the configuration and specifying the flowsrcip as well as the downsample you would like to apply. See example configuration below.
Specific device downsample configurations will override the global downsample. The global downsample will only be used if there is no device specific configuration. Your chosen downsample must be between 2 and 10000.
Note: If you are downsampling on a device that doesn't send a sample rate (IE sample rate zero). Set the sample rate for the device in portal to what you are downsampling to or metrics will be inaccurate.
Flows Rate Testing
It is possible to use netoflow simply to acquire flows/sec number per device without delivering that flow to the Netography cloud. The easiest way to achieve this is to set a default filter of "deny" like such:
[filter]
default = "deny"
logdeny = false
Then use the stats api end point detailed below to obtain flows/sec numbers. This will give you per device statistics.
Configuration Sample
API Key Authentication
[general]
udpbuffer = 1048576 #Socket buffer for the listen port
statsinterval = 10 #Interval at which to calculate NetoFlow Connector statistics in seconds
flushinterval = 5 #Interval at which to flush records if flush count has not been hit
flushcount = 2000 #Number of records to flush at
retries = 5 #Number of times to retry Netography API in case of failure
retryinterval = 5 #Number of seconds to wait between retries
downsample = 10 #Apply a global downsample of a factor of 10 to every device
[flow]
listen = "0.0.0.0:2055" #Netflow/Sflow listen address/port
workers = 4 #Flow processing threads
queue = 100000 #In memory queue size for flow records
[syslog]
level = "info" #Syslog level of logging
server = "localhost:514" #Syslog server:port
[api]
enabled = true #Whether to enable netoflow API or not
port = 8080 #Listen port for the internal API
[sflow]
payload = false #Override to drop sflow payload in the NetoFlow Connector
[tees]
enabled = true #Whether to enable tees
updateinterval = 300 #Interval in seconds to update routing table for tees
targets = ["192.168.2.31:3333"]
[endpoint]
customer = “PortalShortName” #Account shortname from portal
appname = “APINameFromPortal” #API key name from portal
appkey = “APIKeyFromPortal” #API key from portal
sharedsecret = “xxxxYourSecretxxxx” #API secret from portal
[proxy]
enabled = false #Whether to use proxy or not
url = "http://191.96.42.86:3121" #HTTP Proxy url
[filter]
default = "allow"
[flowflilter]
action = "deny"
srcnets = [“10.0.0.0/24”, “192.168.0.0/16”]
dstnets = [“10.0.0.0/24”, “192.168.0.0/16”]
srcports = [40000, 40001]
dstports = [40000, 40001]
protocols = [“esp”, “icmp”]
dstasns = [65001, 65002]
srcasns = [65001, 65002]
[[device]
flowsrcip = "192.168.2.254"
downsample = 100
Username Authentication
[general]
udpbuffer = 1048576 #Socket buffer for the listen port
statsinterval = 10 #Interval at which to calculate NetoFlow Connector statistics in seconds
flushinterval = 5 #Interval at which to flush records if flush count has not been hit
flushcount = 2000 #Number of records to flush at
retries = 5 #Number of times to retry Netography API in case of failure
retryinterval = 5 #Number of seconds to wait between retries
downsample = 10 #Apply a global downsample of a factor of 10 to every device
[flow]
listen = "0.0.0.0:2055" #Netflow/Sflow listen address/port
workers = 4 #Flow processing threads
queue = 100000 #In memory queue size for flow records
[syslog]
level = "info" #Syslog level of logging
server = "localhost:514" #Syslog server:port
[api]
enabled = true #Whether to enable netoflow API or not
port = 8080 #Listen port for the internal API
[sflow]
payload = false #Override to drop sflow payload in the NetoFlow Connector
[tees]
enabled = true #Whether to enable tees
updateinterval = 300 #Interval in seconds to update routing table for tees
targets = ["192.168.2.31:3333"]
[endpoint]
user = "[email protected]" #Netography API User
pass = "XXX-secret-XXX" #Netography API Secret
[proxy]
enabled = false #Whether to use proxy or not
url = "http://191.96.42.86:3121" #HTTP Proxy url
[filter]
default = "allow"
[flowflilter]
action = "deny"
srcnets = [“10.0.0.0/24”, “192.168.0.0/16”]
dstnets = [“10.0.0.0/24”, “192.168.0.0/16”]
srcports = [40000, 40001]
dstports = [40000, 40001]
protocols = [“esp”, “icmp”]
dstasns = [65001, 65002]
srcasns = [65001, 65002]
[device]
flowsrcip = "192.168.2.254"
downsample = 100
Statistics via API
Netoflow provides a simple api to provide client side statistics of what the application is processing from a flow perspective. The per device statistics are calculated pre filter and pre downsample. By default the API listens on tcp port 8080. An example curl request to fetch statistics would look like such:
curl localhost:8080/api/v1/stats/rate
{
"code": 200,
"meta": {
"count": 1,
"timems": 0
},
"data": {
"type": "rate stats 1m",
"instance": null,
"aggregates": {
"netflow-filtered": 0,
"netflow-in": 1034.8078748433154,
"netflow-out": 1034.8078748433154,
"nfpktdecerrs": 0,
"notemplate-all": 0,
"notemplate-v10": 0,
"notemplate-v9": 0,
"sflow-filtered": 0.09559831561527027,
"sflow-in": 0.18078967640840649,
"sflow-out": 0.08519136079313624,
"sfpktdecerrs": 0
},
"devices-inflow": [
{
"device": "10.0.10.253",
"flows": 1082.9726620649064
},
{
"device": "192.168.2.253",
"flows": 0.18944462469620377
}
],
"devices-outflow": [
{
"device": "10.0.10.253",
"flows": 1082.9726620649064
},
{
"device": "192.168.2.253",
"flows": 0.08951883493703489
}
]
}
}
Governance
All flow data and user metadata in the Netography cloud is stored encrypted at rest. The NetoFlow Connector only sends the flow records into the cloud which contains the 5 tuple it parses out of the flow records and the associated network interfaces and statistical information. An example of a record is shown below:
{
"timestamp": 1574220661,
"srcip": "10.0.0.1",
"srcport": 34765,
"srcowneras": {
"asnumber": 3333,
"asorg": ""
},
"srcas": {
"asnumber": null,
"asorg": ""
},
"dstip": "192.168.2.2",
"dstport": 179,
"dstowneras": {
"asnumber": 14618,
"asorg": ""
},
"dstas": {
"asnumber": null,
"asorg": ""
},
"nexthop": null,
"nexthopowneras": {
"asnumber": null,
"asorg": ""
},
"bgpnexthop": null,
"bgpnexthopowneras": {
"asnumber": null,
"asorg": ""
},
"bits": 480,
"packets": 1,
"pbratio": 0.0020833333333333333,
"flowbrate": 480,
"flowprate": 1,
"protocol": "tcp",
"protocolint": 6,
"tos": 192,
"srcmask": null,
"dstmask": null,
"start": 3056874,
"end": 3056874,
"duration": null,
"input": null,
"output": 4,
"samplerate": 10,
"uptime": null,
"flowsrcip": "172.16.10.10",
"tcpflags": {
"urg": false,
"ack": true,
"psh": false,
"rst": false,
"syn": false,
"fin": false,
"ece": false,
"cwr": false,
"ns": false
},
"tcpflagsint": 16,
"flowtype": "netflow",
"flowversion": 10,
"srcvlan": null,
"dstvlan": null,
"payload": "",
"ipversion": 4,
"endreason": null,
"direction": 1
}
Updated 7 months ago