Docs

NetoFuse Context Transforms

Suggest Edits

About

Context transform configurations define how the field names and values from a NetoFuse module are modified before being sent to Netography Fusion as context label names and values. Default values are provided for context transforms for each module. These can be modified in the configuration file.

There must be a context label name ip with at least 1 valid IPv4 and/or IPv6 address as the value.

A default transform configuration ships with each module and is part of the default NetoFuse configuration file.

📘

The structure of a Netography context label

A context label consists of an IP address, a context name, and one or more context values.

For example, here is a simple set of context labels for an IP address in the standard Netography label CSV format

10.0.0.1,name,MYHOSTNAME
10.0.0.1,criticality,low
10.0.0.1,mac_addr,00:01:02:03:04:05,00:01:02:03:04:06
10.0.0.1,osver,windows

Example of how context transforms work

You made an API call to an asset management system, and it returned 1 asset (the same one as in the Netography context label example above). It might return that asset in a format that looks something like this:

mac,asset_type,criticality__,addresses,os,asset_class,assetid,name
[00:01:02:03:04:05,00:01:02:03:04:05:06],ePrinter,low,10.0.0.1,Windows unknown version,IT,872310323181,MYHOSTNAME[

In this example, you can see that the field names in the header are not the context label names that I want to use, that some field values have extra characters I want to strip out, and there are extra fields irrelevant to be used as context label names.

By creating a transform configuration for this module, I can have it handle the conversion with minimal effort:

{
    "transforms": {
        "ipv4": {
            "context": "ip"
        },
        "mac": {
            "context": "mac_addr"
        },
        "criticality__": {
            "context": "criticality",
            "replace": {
                "eLow": "low",
                "eMedium": "medium",
                "eHigh": "high"
            }
        },
        "os": [
            {
                "context": "osver"
            },
            {
                "context": "os",
                "function": {
                    "function": "transform_os"
                }
            }
        ]
    }
}

Renaming a field with context:

To change the name of a field, add a context value. For example, this will rename a data source field named asset_type__ to asset_type:

{
    "transforms": {
        "asset_type__": {
            "context": "asset_type"
        }
    }
}

Replacing the value of a field with replace:

The value of a field can also be transformed using a list of matching values to substitute or regular expressions. For example, this will rename the field criticality__ to criticality and change eLow to Low, eMedium to Medium, and eHigh to High:

{
    "transforms": {
        "criticality__": {
            "context": "criticality",
            "replace": {
                "eLow": "low",
                "eMedium": "medium",
                "eHigh": "high"
            }
        }
    }
}

Replacing the value of a field with regex:

This is the same transform done with a regex using match and replace fields:

{
    "transforms": {
        "criticality__": {
            "context": "criticality",
            "regex": {
              "match": "^e(.*)$",
              "replace": "$1"
            }
        }
    }
}

This is a more complicated regex that replaces an ISO timestamp with a mm/dd/yy value:

{
    "transforms": {
        "timestamp": {
            "context": "last_seen_date",
            "regex": {
              "match": "(\\d{4})-(\\d{2})-(\\d{2})T.*",
              "replace": "\\2/\\3/\\1"
            }
        }
    }
}

Create a new context transform with buildtransform

Use the buildtransform command) to build a context transform from an existing module based on the actual results of the module executing or from a CSV file with a header row.

For example, to build a new transform configuration file from mycsvfile.csv and write it to myfile.transform, run netofuse buildtransform mycsvfile.csv myfile.transform --csv

Some common edits you may need to make after using the buildtransform command include:

  1. Removing extra fields that are not relevant asset information to convert to context labels.
  2. Rename the field containing the IP address to ip as that is required context name for a IP context label.
  3. Change an operating system field by renaming it to os for the context label and using a transform to make it normalized to a Netography Fusion OS field. The transform_osfunction will convert many operating system strings to a normalized format.
  4. Change the name of a mac address to mac_addr as that is the standard Netography field name for that.
  5. Rename other context values to make them more readable and meaningful to a Netography Fusion user.

Although the fields you will return from any API you integrate to will differ, these basic steps of reviewing each field, removing unnecessary ones, renaming them to standard Netography field names, and adding transforms to convert values where needed is the same.

Refer back to the default tranform configuration for the module when editing the content created by a buildtransform as that may have specific field mappings already defined and transforms written that may be useful for you if you are using some of the same or similar fields.

Writing custom transform code in Python

You can enable custom transforms if a more advanced transform is needed to convert a field value. To enable, add the command line argument --allow-custom-transforms <filename>. This can not be enabled from a configuration setting. The filename points to a Python file that defines functions used for a string transformation.

🖐️

Using custom transforms in SaaS

Custom transforms can only be used when deploying netofuse as a container or Python code locally. They are not permitted via the Fusion Portal. However, if you have written a custom transform function that you would like to use in SaaS (eg you developed the transform locally and now want to deploy it via SaaS for production use), submit your function in a request to Netography Support, and after a quick review Netography Engineering can make it available as a standard transform function that can be used in SaaS. Standard transform functions are available to all customers, so this option is not suitable if your transform function contains any proprietary code or data.

To use a function defined in a custom Python transform, define the context transform in the transform file as:

{
    "transforms": {
        "officename": [
            {
                "context": "office",
                "function": {
                    "function": "transform_upper"
                }
            }
        ]
    }
}

That would then look for the function transform_upper in the file specified. This is a sample custom_transforms.pyfile you can use as a template to create your transformations and a handy debug transform for development:

# custom_transforms.py  
#
# This is an example of a custom transforms Python file that can be called by function name 
# in a transform file.  This is useful for transforms that can not be handled by a simple
# simple replace or regex.

def transform_lower(key:str, value:str, asset:dict):
    return value.lower()

def transform_upper(key:str, value:str, asset:dict):
    return value.upper()

def transform_debug(key:str, value:str, asset:dict):
    print(f"custom transform debug:")
    print(f"    key: {key}")
    print(f"  value: {value}")
    print(f"  asset: {asset}")
    return "TRANSFORM_DEBUG"

Updated 2 months ago