Docs

Configure NetoFuse

Suggest Edits

Using the default configuration

The Getting Started > Launch section provides the basic configuration steps to run a NetoFuse module.

Where configuration is set

NetoFuse reads configurations in the following order:

  1. netofuse.yml configuration file (use --config option to set a different filename/path)
  2. .netofuse.env environment settings (use --env option to set a different filename/path)
  3. Local environment settings

Environment variables override environment settings in a fille. Environment settings in a file override the YAML configuration file settings.

⚖️

Choosing between YAML configuration and environment variables

If you want to follow a simple best practice, use environment variables for credentials and the YAML configuration for all other settings.

Environment settings

All configuration settings can be specified as environment variables that can be read from the local environment or an environment settings file.

Start with the correct environment variable names

If you used the Docker host setup script, the correct environment variable names and default values for every setting are in the file: /etc/netofuse/netofuse.env.template

You can create this file at any time by running:

netofuse init --envfile /app/netofuse/config/netofuse.env.template

Remember, if you are using the container, file paths for netofuse command line arguments and settings refer to the location inside the container. Using the default setup script, /app/netofuse/config inside the container binds to /etc/netofuse on the Docker host.

Copying environment variables for all secrets to an environment file

This command will append all the sub-keys of all credentials keys to the .netofuse.env file.

grep __CREDENTIALS__ /etc/netofuse/netofuse.env.template >> /etc/netofuse/.netofuse.env

NETO__API__APP_NAME=value   # this is the correct naming convention
NETO__API__APP__NAME=value  # this is incorrect, name is a subkey of app here
NETO_API_APP_NAME=value     # this is also incorrect, the whole string is 1 key

neto:
  api:
    app_name: value        # NETO__API__APP_NAME - correct
    app:
      name:value		       # NETO__API__APP__NAME - __ between app and name adds a subkey 

neto_api_app_name: value   # NETO_API_APP_NAME - all _ makes whole string 1 key

neto.api.app_name					 # NETO__API__APP_NAME
neto.api.app.name					 # NETO__API__APP__NAME
neto_api_app_name					 # NETO_API_APP_NAME

Modifying netofuse.yml and specifying its location

NetoFuse looks in its current directory to find netofuse.yml unless its location is specified with one of:

  • --dir command line option
  • --config or -c command line option
  • NETOFUSE_CONF_PATH environment variable

This is a YAML format configuration file.

📘

What is YAML?

"YAML is a human-readable data serialization language that is often used for writing configuration files. Depending on whom you ask, YAML stands for yet another markup language or YAML ain’t markup language (a recursive acronym), which emphasizes that YAML is for data, not documents."

Source: https://www.redhat.com/en/topics/automation/what-is-yaml

If you are running a container

Using the provided docker run scripts

Default location for configuration files are /etc/netofuse/netofuse.ymland /etc/netofuse/.netofuse.env

By default, .netofuse.env does not exist - rather a template file netofuse.env.template is written to the directory to select the settings you want to use.

The run script will also pass all environment variables that start with the prefix NETO__ on the Docker host to the container.

The docker run script provided in the Install page places a default copy of the configuration in /etc/netofuse/ unless you modified the environment variables in that script before running it.

Advanced container configuration

If you are using your own docker run command and not using the docker run script provided, netofuse will look in its local directory inside the container for the default configuration file. Just placing the file in the local directory on the Docker host is not accessible within the container.

If you saved a copy of netofuse.yml on your docker host to $PWD/config/netofuse.yml, then you can run netofuse with this configuration by mapping that directory with docker as follows:

docker run -v $PWD/config/:/app/netofuse/config public.ecr.aws/netography/netofuse:latest --config /app/netofuse/config/netofuse.yml

If you’re using a different container orchestrator system like Kubernetes, supply a configmap of your configuration file, mount it inside the container, and supply arguments as you normally would.

If you installed the Python package

Save a default configuration file (netofuse init)

Run netofuse init to save a default configuration file to netofuse.yml in your current directory.

Run netofuse with a specified configuration file

The netofuse.yml in the current directory when netofuse is run will be used as the configuration. You can use the -c or --config command line option to specify an alternate file path.

For example: netofuse --config /etc/netofuse/netofuse.yml run axonius

netofuse.yml

The default netofuse.yml contains common settings and a default instance of each NetoFuse module. Each section of the file is documented in more detail below.

neto:

The top-level section of a netofuse YAML file is neto:

neto:

syslog:

This section defines the level of logging output and where that output is sent. The level field can be overridden with the -l LEVEL or --level LEVEL command line options (eg netofuse -l DEBUG). 

neto: 
  syslog:                 # Default is log at INFO level to /dev/log
    level: VERBOSE        # Sets logging level (INFO,VERBOSE,WARNING,ERROR,DEBUG,TRACE)
    server: localhost     # Sends syslog to specified IP address if set (on port 514)
    address: /dev/log     # Sends syslog to specified LOG_DAEMON address if set (server has precedence) 
    console: VERBOSE      # Sets a different logging level for console from syslog if set

common:

This section is applied to NetoFuse and the modules that execute:

neto: 
  common:
    retries: 3						# Number of retries when a request fails 
    retry_interval: 10    # Seconds to wait after first retry (exponential backoff is used)
    per_page: 9900        # Context labels to write per Netography API request
    upload: true          # Upload results to Netography 
    cert: true            # Verify SSL certificates or path to cert
    container: true			  # Running within a container

save:

Output from each module can be saved to files. This is useful for testing a module to see the output before uploading the results to the Netography Fusion API. It is also useful if you need to do any more advanced post-processing of data from a module before uploading it as an intermediary step, as you can save the output, edit it, and then use the netofuse upload command to upload the results.

Files will be saved to modulename-output.ext, modulename-transform.ext, and modulename-labels.ext where ext is the setting value (csv or json). If a path or filename is specified instead of just the extension, that wil be used instead of the default filenames.

neto: 
  save:
    output: csv     # if set to csv or json, saves pre-transform output from modules to local file
    transform: csv  # if set to csv or json, saves post-transform output from modules to local file
    labels: csv     # if set to csv or json, saves Netography context label format output from modules to local file

cache:

NetoFuse has 2 types of caching that can be enabled:

Context label caching

Caching context labels will create a file that stores a hash of the asset information that is being uploaded to Netography, and then only send changes in subsequent executions of the same module. It is recommended this is NOT enabled unless you have discussed the specific use case with Netography Support, as it is still in development.

Authentication token caching

If you make frequent successive calls to the Netography Fusion API, the authentication tokens used should be cached to improve performance and reduce load. With the token caching enabled, authentication can happen one time until the token expires, no matter how many different times NetoFuse is run in that period. If it is disabled, however, each time NetoFuse is run, it will request a new JWT request token, potentially creating many concurrent API sessions.

Adding a cache: section within a module configuration will override the global values.

neto: 
  cache:
    enabled: false  # caching of context labels for a module 
    token: true     # caching of netography JWT tokens
    dir: cache/     # directory to store cached files in

api:

Defines the Netography API configuration parameters. If you are configuring netofuse for a single Netography API account, you can directly specify the values under the API key. If you are configuring multiple API instances (when you have multiple accounts within Netography Fusion and want to use a single configuration to manage and integrate them), give each instance a name and add values under that key.

Note: If you have multiple sub-accounts in Netography Fusion, you can generate an API key in the parent account and then change the shortname key to any sub-account that you want to use.

Example for a single account:

neto: 
  api:
    credentials:                                 
      netosecret: 					   				    # Generated when creating API key

Example when using multiple Netography API accounts:

neto: 
  api:
    api1:
      shortname: 												    # Account name to use with API
      credentials:                                 
        netosecret: 					   				    # Generated when creating API key       
		api2:    
      shortname: 												    # Account name to use with API
      credentials:                                 
        netosecret: 					   				    # Generated when creating API key

Note: Instead of setting the netosecret value, you can instead set url, app_name, shortname,credentials.app_key, and credentials.shared_secretfields individually. This may be useful if you are using a legacy-style API key that predates the netosecret encoding format.

Module Configuration

Each NetoFuse module is configured by the module name under the neto key .

The unique configuration parameters associated with each NetoFuse module are defined in its section of the NetoFuse Modules documentation.

neto: 
  localfile:
    files: import/*   # Configuration parameter specific to the localfile module

transform:

The transforms section for a module defines how keys from a module are mapped to context names in Fusion and how values are modified when they are written to context values in Fusion. Details on writing and editing transforms are documented in the Context Transforms section of the documentation.

Each module instance has a transform key set to a value of one of the transforms in this section (although individual modules may bypass this by setting skip_transform: true in the module configuration).

neto: 
  localfile:
    transform:
      ip:
        context: ip
      id:
        context: my_id
      name:
        context: name
      mac:
        context: mac_addr

Advanced Configuration Details

Removing a configuration setting with UNSET

An environment variable that is set to the value UNSET will remove this setting entirely from the active configuration. This can be useful if you want to override a default configuration set in the YAML by removing it rather than changing the value.

Complex configurations in environment variables

You can directly set any transform key in an environment variable to YAML. This allows you to store a complete transform, consisting of many sub-keys and values in a single environment variable.

You can also set any environment variables to other types (TOML, JSON) and have that automatically parsed when the variable is read at runtime.

DynaConf library used for configuration

NetoFuse uses the DynaConf library for reading configuration. You can find detailed documentation on all the options for setting complex configuration values and how they are interpreted at:

https://www.dynaconf.com/envvars/?h=environment

Mapping configuration settings to environment variable names manually

The environment variable name is the key to the configuration setting mapped to the YAML configuration file. Each level of YAML is delineated in the environment name by using 2 underscores __ in the name.

Some configuration settings have a single underscore in the setting name. A single underscore is part of the key name. A double underscore splits the string that follows into a subkey of what precedes it.

For example, the setting neto.api.urlmaps to:

NETO__API__URL=value

neto:
  api:
    url: value

❗️

Environment variables can mix single and double underscores

Some configuration settings may have a single underscore _ in the setting name. A single underscore is part of the name of the configuration key, whereas a double underscore indicates what folllows is a sub-key (child, or new level) of the preceding key.

It can be easy when writing these names by hand to use _ instead of __ or vice-versa. For example, this is how 3 environment variable names with different use of underscores would map to different values in the YAML configuratio

NETO__API__APP_NAME=value. # this is the correct naming convention
NETO__API__APP__NAME=value # this is incorrect, name is a subkey of app here
NETO_API_APP_NAME=value    # this is also incorrect, the whole string is 1 key

neto:
  api:
    app_name: value        # NETO__API__APP_NAME - correct
    app:
      name:value		       # NETO__API__APP__NAME - __ between app and name adds a subkey  

neto_api_app_name: value   # NETO_API_APP_NAME - all _ makes whole string 1 key

neto.api.app_name					 # NETO__API__APP_NAME
neto.api.app.name					 # NETO__API__APP__NAME
neto_api_app_name					 # NETO_API_APP_NAME

Updated 2 months ago