Create a Test Case

Understanding the components in a test case

Breakwater test cases have a framework and the code for the framework can be found under common/audit/framework.py. The function ‘register_audit’ is responsible for registring a new audit to the breakwater environment, it uses a dictionary to store all the audit test cases.

Registering the audit in the framework

So whenever a new test case is created, it must be registered first calling the ‘@register_audit(new_audit_name)’

The test case also should have ‘from common.audit.framework import *’ as an import function so the framework get’s imported. All the necessary import functions should also be mentioned for that specific test case. For example these are the import functions for the ‘uds_server_scan’ test case.

snippet from uds_server_scan.py test case

from common.audit.framework import *
import can
import time
from common.audit.udsutil import *
import threading
import subprocess

Format of a test case

The test case should be created as a class with the same name as the one used for registering the test case through ‘register_audit’.

‘AuditRunner’ is a class defined in the framework.py which initializes the required variables for the test case to function properly. AuditRunner is necessary for creating the report as well. AuditRunner class is defined below

snippet from framework.py

class AuditRunner:
    severity_list = ['n/a', 'information', 'low', 'medium', 'high', 'critical']
    log = Logger()
    params = {}
    parameters = []
    relevant_interfaces = []
    relevant_mitigations = []
    bw_audit_id = "N/A"

    def __init__(self, params: dict, pass_rules: dict = None):
        self.params = params
        pass

    @abstractmethod
    def run_audit(self):
        pass

    @abstractmethod
    def format_results(self, results):
        return results

The test case must be defined in the function ‘run_audit’ which needs to have ‘self’ instance of the class. All the variables and required code must be within this function. The return value would be ‘results’ which gets passed onto the ‘format_results’ function which is to be mentioned in the section below.

Variables required

Input parameters can be defined in the ‘parameters’ array. The parameters should be defined in the array as ‘Parameter(“name_of_the_input_variable”, “name_of_the_interface”, “default_value_of_the_variable”, “short_description_of_the_variable”)

Example snippet from telnet_check.py

 parameters = [
        Parameter("iface", "Network Interface", "eth0", "The network interface on which to interact"),
        Parameter("target_ip", "Target IP", "10.1.0.1", "The IP to use when attempting to connect to"),
        Parameter("port", "Telnet Port", "23", "The port number to attempt to connect to telnet on"),
        Parameter("temp_ip", "Temporary IP", "10.1.0.12", "A temporary IP address to set on the network interface, which will be reset after the test completes"),
        Parameter("temp_mask", "Temporary Subnet Mask", "255.255.255.0", "A temporary subnet mask to set on the network interface, which will be reset after the test completes")
    ]

The test case also requires these variables to be defined according to the test cases functionality as well

Example snippet from telnet_check.py

friendly_name = "Telnet Check"
    description = "Attempts to connect to a telnet server at the specified IP and port."
    fail_condition = "If a telnet server responds, the test will be marked failed."
    relevant_mitigations = [ "M23" ]
    relevant_interfaces = [ "Automotive Ethernet", "WiFi", "Cellular" ]
    bw_audit_id = "TS_NET_0014"
    estimated_time_in_minutes = "1"

Additional variables:

segments is an array which stores the information required to be printed in the report. To add an info to this array we must call the segments.append function, here’s an example

segments.append(Report.create_text_segment(f'Telnet is available on {target}:{port}, test FAILED'))

Generating the report/results

‘format_results’ is the function which needs to be defined to pass the results onto the report to be generated. The return variables would be

 return [{"title": title, "segments": segments }]

Example of a test case

Mentioned below is the code written for a test case which prints out text which is defined in the variable ‘custom_text’

from common.audit.framework import *

@register_audit("print_text")
class PrintText(AuditRunner):
    parameters = [
        Parameter("custom_text", "", " ", "Print any custom text "),
    ]
    friendly_name = "Prints custom texts and exits"
    description = "Print custom texts and exists"
    fail_condition = "Nil"
    relevant_mitigations = [ "Nil" ]
    relevant_interfaces = [ "Nil", "Nil", "Nil" ]
    bw_audit_id = "PRINT_CUSTOM"
    estimated_time_in_minutes = "1"

    def run_audit(self):
        custom_text = self.params["custom_text"]

        passed = True
        segments = []
        severity = "n/a"

        print(custom_text)
        segments.append(Report.create_text_segment(f'Test passed. Printed {custom_text}'))

        return { "passed": passed, "pass_rules": { "severity": severity } }

    def format_results(self, results):
        title = "Test Complete"
        segments = []

        if results["passed"]:
            segments.append(Report.create_text_segment("Test passed"))

        return [{ "title": title, "segments": segments }]

Final touches

  1. Once the test case is generated, move the test case onto the proper folder under ’tests’ in the breakwater2 directory.
  2. If a new folder is created to store the test case a. Add the folder onto the import function in both ‘bw.py’ and ‘bw_api.py’ scripts b. Continue to step 2 below.
  3. Edit the ‘init.py’ in the corresponding folder and add the import for the new test case