Custom Modules

Suggest Edits

If you can get a file into a CSV or JSON format to disk from a data source, then using the Local File Module is the easiest way to integrate it with NetoFuse.

To connect directly to APIs and more advanced use cases, you can build your own NetoFuse modules. New modules that connect to a 3rd party API can be built in less than 50 lines of Python. These instructions expect you to have basic familiarity with Python development.

🖐️

Need help building a custom module?

Netography's engineers are experts at building NetoFuse modules. While we are happy to help point you in the right direction if you are a developer and want to do it yourself, we also can develop custom modules for you as part of the services we offer. Reach out to your Netography sales rep to discuss options for custom development.

NetoFuse Module Development Tips

NetoFuse is written entirely in Python. Although none of the following steps are required, the following are some tips to help make writing a new NetoFuse module easier:

  1. Use Visual Studio Code as your code editor. It is free.
  2. Use GitHub Copilot to help you write your Python code. It is $10/mo after a free trial.
    1. Install the GitHub Copilot extension for VS Code
    2. Install the GitHub Copilot Chat extension for VS Code
  3. Use the NetoFuse Python package on your local development system.
  4. Use a Mac or a Linux VM for your development system.

Making a copy of the NetoFuse source code to work from

The following shell script will make a copy of the Python source code from the site-packages directory it is installed in when using the RPM or Python package installation into your home directory in the netofuse-custom directory, and then make that copy debuggable in Visual Studio Code.

#!/bin/sh
SITE_PACKAGES=`python3 -m site --user-site`
mkdir -p netofuse-custom/.vscode
cd netofuse-custom
cp -R $SITE_PACKAGES/netofuse .
echo '{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Python: Current File",
            "type": "python",
            "request": "launch",
            "module": "netofuse.main",
            "console": "integratedTerminal",
            "justMyCode": true,
            "cwd": "${workspaceFolder}",
            "env": {
                "PYTHONPATH": "${workspaceFolder}"
            },
            "args": ["-i"]
         }
    ]
}' > .vscode/launch.json

If the script produces an error because it does not find the right location for your Python packages (because python3 -m site --user-sitedid not return the correct result), it may be installed into a different directory in your Python path. Try this command to see the correct directory, and then edit the script to use that directory for SITE_PACKAGES:
python3 -m site|grep site-packages

Step by step instructions for creating a new NetoFuse module

  1. In the modules/ subdirectory, make a copy of the template.py file and give it the name of your new module (eg acme.py). If an existing module is similar to the one you are building, it may be easier to start from that existing module rather than the template.
  2. Update the __description__variable to identify the new module.
  3. Rename TemplateModuleClass to YourModuleNameModuleClass (eg AcmeModuleClass)
  4. Write the code in the authenticate method. It is often helpful to copy and modify the code from other modules as they contain various authentication methods already implemented for standard REST APIs.
  5. Write the code in the get_assetsmethod to make the correct API call to the product and parse the results. The template, or other existing modules, has standard code to handle retries, back-off timing, and errors. In this method, you can define additional configuration parameters when the module is run.
  6. Update the netofuse.ymlconfiguration file to add a section under neto for an instance of this module to execute. For example: 
    neto:
  acme1:
    credentials:
      key: 
      secret: 
    transform:
    The filename of your new module without the extension is used as the module name (eg if you named the file acme.pythe module is acme).
  7. Update the netofuse.yml configuration file to add transforms for the context transform to use for this module.

Custom modules are not supported and could cause unexpected behavior if not built correctly. But let us know if you run into problems, as we can probably point you in the right direction with your code.

Developer documentation for creating a new NetoFuse module

Research the 3rd Party API

Before you start coding the module, it is helpful to determine how the integration to the external system will work. To do this, you will want to determine three key things:

  1. How to authenticate: There is likely an API key, bearer authentication, username/password or similar type of authentication required for the API. If presented options for username and password or an API key, choose the API key as it is more appropriate for this type of use case.
  2. How to retrieve asset information: The product may call an asset different words - IPs, Devices, Assets, Systems. Determine the correct name of what you are trying to retrieve, and then identify the correct API endpoint that needs to be called to gather asset information from the system. Some APIs may require you to make multiple API calls to gather all the data you want.
  3. What parameters to use: APIs work in many different ways. Some will limit how many assets can be retrieved at once, implement paging, provide filters, or let you specify the specific fields you want to gather. Determine what is required to gather the information you want. If you are integrating 1 specific instance of this product for only your own company's use, you could hardcode the required parameters in the code. Still, it would be good practice to make as many of the parameters configurable in the module as may be useful. If you are developing a module that you may want to let others use in the future or are in a large, complex environment, then providing lots of configuration options will make the module more valuable.

How NetoFuse modules are loaded

A module in NetoFuse loads dynamically at runtime in Python by placing the module python file in the modules directory where NetoFuse is installed. That module python file must contain a class that inherits from the NetofuseModulesClass. The module name is the name of the python file that contains that class excluding the extension (eg the module class contained inmodules/acme.py is the acme module.

The name of the actual class in Python is not used in how the module is named in configuring and running it.

NetofuseModulesClass

__init__(self, name, config, log)

This method initializes a session and loads module configuration.

All modules should call: super().__init__(self,name,config,log

This loads module-specific configuration fields before executing the module.

authenticate(self)

This method authenticates to the API using the credentials provided. If a token is retrieved, this may also be where it is saved to a token cache if caching is supported for the module. authenticate() is called in 3 different circumstances:

  1. By the auth command, to test if authentication for the module is working with the provided credentials.
  2. By the verify command, to test if the module is able to authenticate and gather asset information with the provided credentials and configuration.
  3. By the run and runonce commands, during the get_assets() call that retrieves asset data. Note that authentication does not occur when the module is initialized, only when it is run.

get_assets(self)

This is the main method of the module that gathers the asset information and passes it to the main NetoFuse code to be processed. For each page of asset information retrieved by an API call, a list of assets is created. Each asset in that list is a dictionarycontain key:value pairs. Once that set of assets is created, yield is called with the asset list as a parameter (eg yield assets). `

📘

What does yield do in Python?

yield is used in functions to turn the function into a generator. A generator is a special type of iterator that allows for lazy evaluation, meaning it generates items on the fly and does not store them in memory. Rather than returning a value and exiting like a normal function, each time the yield statement executes it outputs a value and the function pauses until it is called again, and then it resumes where it left off.

NetoFuse modules uses this to handle large amounts of data. If all the asset information from a 3rd party product had to be gathered up into a list before it was processed, a NetoFuse module could take tens of gigabytes of data to run. By using lazy evaluation, no more memory is ever used than is returned by a single API call. If even a single API call could return a large amount of data, the function can implement a yield for each asset that is returned.

Building a transform configuration

The Context Transforms documentation describes the available options for transforming the values returned by an API and mapping fields returned by an API to context names used by Netography Fusion. Although you could do all of the transformations and mapping directly in the module, letting NetoFuse's transform capability handle this makes it easier to build the module, and more flexibility to modify how it is used when it is run.

However, if the 3rd party product itself can already handle modifying any values and returning results with specific field names that you specify, adding a context transform may be redundant.

Brag about it!

Let us know if you built a custom module; we'd love to see what you've done!

Updated 5 months ago