Custom Test Cases

Test Case Framework

Test cases must follow the framework to be properly picked up, executed and finally have their results collected and returned to the user.

Example Custom Audit

We will start with a full example custom test case, then explain each piece.

from framework import *

@register_audit("example_cutom_audit")
class ExampleCustomAudit(AuditRunner):
    parameters = [
        Parameter("parameter_id", "Parameter Name", "DEFAULT VALUE", "Parameter Description")
    ]
    friendly_name = "Example Custom Audit"
    description = "This is an example of a custom audit to be run through VSEC Test"
    fail_condition = "This test will fail if ..."
    relevant_mitigations = [] # Relevant Mitigation IDs for Compliance Tracking
    relevant_interfaces = [ "CAN", "WiFi", "ETH" ]
    bw_audit_id = "EXAMPLE_CUSTOM_001"
    estimated_time_in_minutes = "1" # extimate runtime for test planning

    def run_audit(self):
        param_val = self.params["parameter_id"]

        self.log.log('Inside Example Custom Audit')
        self.log.warn(f'Example Custom Audit Parameter: {param_val}')

        return { "param": param_val, "pass_rules": { "vulnerable": False }}

    def format_results(self, results):
        title = 'Example Custom Audit ' + 'Passed' if results['passed'] else 'Failed'
        segments = []
        if results['pass_rules']['vulnerable']:
            segments.append(Report.create_text_segment('Parameter: ' + results['param']))
        else:
            segments.append(Report.create_text_segment('Parameter: ' + results['param']))

        return [{ "title": title, "segments": results["segments"] }]

Import the Framework

from framework import *

Importing the framework provides the types necessary to create an audit class. This includes the registration decorator, the AuditRunner and Parameter base classes, and Reporting helpers.

Register an Audit

@register_audit("example_cutom_audit")
class ExampleCustomAudit(AuditRunner):

Whenever a new test case is created, it must be registered with the class decorator ‘@register_audit(“new_audit_name”)’

Define Parameters

parameters = [
    Parameter("parameter_id", "Parameter Name", "DEFAULT VALUE", "Parameter Description")
]

Parameters are how dynamic input is provided to an audit. You must provide a parameter ID, name, default value and description for each.

Fill Necessary Properties

friendly_name = "Example Custom Audit"
description = "This is an example of a custom audit to be run through VSEC Test"
fail_condition = "This test will fail if ..."
relevant_mitigations = []
relevant_interfaces = [ "CAN", "WiFi", "ETH" ]
bw_audit_id = "EXAMPLE_CUSTOM_001"
estimated_time_in_minutes = "1"

Many properties such as friendly_name and description are intuitive to fill in.

relevant_mitigations is a property provided to support compliance tracking. If your test case evaluates an issue with an associated mitigation ID, you may provide that in this field.

bw_audit_id is an external ID field you may use to link audits to other systems.

estimated_time_in_minutes is a property provided to support test planning. The value may be displayed to the user when designing a test set.

Design Test Logic

def run_audit(self):
    param_val = self.params["parameter_id"]

    self.log.log('Inside Example Custom Audit')
    self.log.warn(f'Example Custom Audit Parameter: {param_val}')

    return { "param": param_val, "pass_rules": { "vulnerable": False }}

Within the body of the run_audit method is where your test logic belongs. This method must return a dictionary with at least a pass_rules dictionary containing the vulnerable boolean property. You may include other context as well.

Format Test Results

def format_results(self, results):
    title = 'Example Custom Audit ' + 'Passed' if results['passed'] else 'Failed'
    segments = []
    if results['pass_rules']['vulnerable']:
        segments.append(Report.create_text_segment('Parameter: ' + results['param']))
    else:
        segments.append(Report.create_text_segment('Parameter: ' + results['param']))

    return [{ "title": title, "segments": results["segments"] }]

Within the body of the format_results method you can access the results passed from run_audit. Here you must return a dictionary with the title and segments properties which will be used for reporting. It is nice if the title indicates the test result and the segments should provide some explaination of the result or show any available evidence.

Running a Custom Test Case

Upload a custom test case to VSEC Test by clicking the Create Test Case button under the Custom Test Cases tab and be sure to fill out the relevant fields.

You can now create a Test Spec including your new custom test case and add its parameters, and finally run the test by creating a Test Run.