Custom Test Cases

Test Case Framework

Test cases must follow the framework guidelines to be discovered, executed, and reported back to VSEC Test correctly.

AuditRunner Interface

The AuditRunner is the base interface for any test case and the relevant parts are shown here:

class AuditRunner:
    # list [ class Parameter ]
    parameters = []
    # list of pip package names required to run this test
    dependencies = []
    # multi select [ "CAN", "ETH", "WiFi", "USB", "CELL", "BT", "BLE" ]
    relevant_interfaces = []

    friendly_name = "Audit Name"
    description = "audit description..."
    fail_condition = "The test will fail if ..."

    def __init__(self, params: dict[str, str], logger=Logger()):
        self.params = params # param name -> param value
        self.log = logger

    @abstractmethod
    def run_audit(self) -> dict[str, Any]:
        return {
            "test_data": { ... },
            "pass_rules": { "vulnerable": True }
        }

    @abstractmethod
    def format_results(self, results):
        return [ { "title": "", "segments": [] } ]

The return value of run_audit determines whether the test will be marked as passed, failed, or neither. If the key pass_rules is not present this test will be considered as not providing any result. This can happen in the case a server is unreachable, a parameter is mistyped or an error occured during test execution. If pass_rules are present, the test will be marked as failed if pass_rules.vulnerable evaluates to true and passed otherwise.

The format_results method is used to generate a more formal output of the results. The results parameter is the return value from run_audit and this format_results method returns a list of sections to add to a report, where each section contains a title and a list of segments. Segments can be created using methods in the Report class, of which the following exist:

class Report:
    @staticmethod
    def create_text_segment(text: str):
        pass

    @staticmethod
    def create_heading_segment(text: str):
        pass

    @staticmethod
    def create_list_segment(lst: [str]):
        pass

These reports are generated by VSEC and can be downloaded from the Test Run tab.

Parameter Interface

Each parameter on the AuditRunner class implements the following parameter interface. It also provides methods to interpret parameters of integer and boolean type in a standardized way.

class Parameter:
    name = ""
    friendly_name = ""
    default_value = None
    description = ""
    required = True

    def __init__(self, name, friendly_name, default_value, description="", required=True):
        self.name = name
        self.friendly_name = friendly_name
        self.default_value = default_value
        self.description = description
        self.required = required

    @staticmethod
    def number(str_value):
        return int(str_value, 0)

    @staticmethod
    def boolean(str_value):
        return str_value in ['True', 'true', '1']

Logger Interface

The Logger interface is available Audit class at self.log and provides the following public methods used to log with different severity levels:

class Logger:
    def debug(self, *args):
        pass
    def info(self, *args):
        pass
    def warn(self, *args):
        pass
    def error(self, *args):
        pass

Logs from a test will be saved on disk after execution and can be accessed via the VSEC web interface.

Example Custom Audit

We will start with a full example custom test case, then explain each piece. This example performs a DNS query for the domain name specified in parameter lookup_name to the DNS server specified by parameter dns_server. The test passes if an A record record is resolved, and fails if an A record cannot be resolved.

from framework import *

@register_audit("dns_query_audit")
class DNSQueryAudit(AuditRunner):
    dependencies = ["dnspython"]
    relevant_interfaces = ["ETH"]
    parameters = [
        Parameter("dns_server", "DNS Server IP", "8.8.8.8", "The IP address of the DNS server to query"),
        Parameter("lookup_name", "Lookup Name", "google.com", "The domain name to resolve")
    ]

    friendly_name = "DNS Resolution Audit"
    description = "Performs a DNS query against a specific server to verify network reachability and resolution."
    fail_condition = "The test will fail if the DNS server is unreachable or the name cannot be resolved."

    def run_audit(self):
        import dns.resolver
        dns_server = self.params["dns_server"]
        lookup_name = self.params["lookup_name"]
       
        resolver = dns.resolver.Resolver(configure=False)
        resolver.nameservers = [dns_server]

        self.log.debug(f'Starting DNS Query for {lookup_name} via {dns_server}')
        answers = resolver.resolve(lookup_name, 'A')
        response_ip = answers[0] if len(answers) > 0 else None
        self.log.debug(f"DNS Response: {response_ip}")
        
        failed = response_ip == None
        return {
            "dns_server": dns_server,
            "lookup_name": lookup_name,
            "response": response_ip,
            "pass_rules": {"vulnerable": failed},
        }

    def format_results(self, results):
        title = f"DNS Query"
        
        segments = [
            Report.create_text_segment(f"Target Server: {results['dns_server']}"),
            Report.create_text_segment(f"Query Name: {results['lookup_name']}"),
            Report.create_text_segment(f"Response: {results['response']}")
        ]

        return [{"title": title, "segments": segments}]

Import the Framework

from framework import *

Importing the framework provides the types necessary to create an audit class. This includes the registration decorator, the AuditRunner and Parameter base classes, and Reporting helpers.

Importing Dependencies

While the framework is imported at the top of the custom script, it is important that third party dependencies are imported within the run_audit method so that the framework has a chance to install those depencies before the imported is attempted. You can use the dependencies array to install any pip package and use it in your script within the run_audit method.

Register an Audit

@register_audit("dns_query_audit")
class DNSQueryAudit(AuditRunner):

Whenever a new test case is created, it must be registered with the class decorator ‘@register_audit(“new_audit_name”)’

Define Parameters

parameters = [
    Parameter("dns_server", "DNS Server IP", "8.8.8.8", "The IP address of the DNS server to query"),
    Parameter("lookup_name", "Lookup Name", "google.com", "The domain name to resolve")
]

Parameters are how dynamic input is provided to an audit. You must provide a parameter ID, name, default value and description for each.

Fill Necessary Properties

dependencies = ["dnspython"]
relevant_interfaces = ["ETH"]
friendly_name = "DNS Resolution Audit"
description = "Performs a DNS query against a specific server to verify network reachability and resolution."
fail_condition = "The test will fail if the DNS server is unreachable or the name cannot be resolved."

Many properties such as friendly_name, description and fail_condition are intuitive to fill in.

Design Test Logic

def run_audit(self):
    import dns.resolver
    dns_server = self.params["dns_server"]
    lookup_name = self.params["lookup_name"]
    
    resolver = dns.resolver.Resolver(configure=False)
    resolver.nameservers = [dns_server]

    self.log.debug(f'Starting DNS Query for {lookup_name} via {dns_server}')
    answers = resolver.resolve(lookup_name, 'A')
    response_ip = answers[0] if len(answers) > 0 else None
    self.log.debug(f"DNS Response: {response_ip}")
    
    failed = response_ip == None
    return {
        "dns_server": dns_server,
        "lookup_name": lookup_name,
        "response": response_ip,
        "pass_rules": {"vulnerable": failed},
    }

Within the body of the run_audit method is where your test logic belongs. If the test ran successfully the dictionary should contain the boolean pass_rules.vulnerable to determine whether a test has passed or failed. The rest of the returned context is used by the format_results method.

Format Test Results

def format_results(self, results):
    title = f"DNS Query"

    segments = [
        Report.create_text_segment(f"Target Server: {results['dns_server']}"),
        Report.create_text_segment(f"Query Name: {results['lookup_name']}"),
        Report.create_text_segment(f"Response: {results['response']}")
    ]

    return [{"title": title, "segments": segments}]

Within the body of the format_results method you can access the results passed from run_audit. Here you must return a dictionary with the title and segments properties which will be used for PDF reporting. It is nice if the title indicates the test result and the segments should provide some explaination of the result or show any available evidence.

Usage

To better understand the usage of the AuditRunner methods and how their output is consumed we provide pseudocode to show how the framework runs a test case via an AuditRunner:

instance = AuditRunner(params) # your subclass

results = instance.run_audit()
formatted_results = instance.format_results(results)

if "pass_rules" in results:
    if results["pass_rules"]["vulnerable"]:
        result_code = 1 # FAILED
    else:
        result_code = 0 # PASSED
else:
    result_code = 2 # NO RESULT

Both the raw and formatted results are saved by VSEC to be displayed on the web and built into reports.

Running a Custom Test Case

Sign-in to vsec.blockharbor.io
Navigate to Test on the sidebar menu to enter the VSEC Test application.
Select the Test Cases tab to begin.
Click the Create Test Case button to open the prompt for creating a New Test Case.
Fill out the basic details of the Test Case.

Field	Requirement	Description
Name	Required	The string passed to `@register_audit`
Friendly Name	Required	The string `friendly_name` property on the audit
Description	Required	The string `description` property on the audit
Interface	Required	The relevant interface the test will run on
Default Parameters	Optional	Override the default parameter values from the test

Upload the Python file for your test case by clicking the File button.
Click the Create button to create the Test Case.
Follow the Test Plans guide to add this test case to a test plan and execute it!

Test Cases

Explore our library of Pre-Built Test Cases

DOCS

Supported Hardware

List of supported hardware adapters

DOCS

Last updated on May 13, 2026