Analysis Tools for Testing

The TestTools class is a set of unit tests for validating the consistency of baseline results across different project directories. The tests ensure that the columns in the CSV files generated by the buildstockbatch and run_analysis processes match and that the values in these columns are comparable within an acceptable tolerance.

These tests are designed to ensure that the output of the baseline analysis process is consistent both in terms of the structure of the CSV files (columns) and the numerical results (values). By performing these checks, we can ensure that the analysis workflow is functioning correctly and that the data is reliable for further analysis.

The test_baseline_columns and test_baseline_results tests are located in the TestTools class. This class is run as part of the standard testing process for baseline result validation. It is designed to ensure that no discrepancies occur between the expected and actual output during the baseline analysis phase of the project.

The tests are designed to check two main aspects of the baseline analysis output:

  1. baseline columns consistency,

  2. baseline results consistency.

Each test is executed for two project directories, national and testing, and ensures that the results produced by both the buildstockbatch and run_analysis are consistent in terms of both the structure and numerical values.

Dependencies

  • Minitest : This test framework is used to define and run the tests.

  • CSV : The CSV library is used to read and manipulate the result files.

  • HPXMLtoOpenStudio : The helper files for the test setup and any necessary utilities are included via the minitest_helper in the HPXMLtoOpenStudio/resources directory.

test_baseline_columns Function

This test verifies that the CSV files generated by the buildstockbatch and run_analysis processes have the same column structure.

Test Description:

  • Input:

    • buildstockbatch results: Located at buildstockbatch/project_<project>/project_baseline/results_csvs/results_up00.csv

    • run_analysis results: Located at run_analysis/project_<project>/project_baseline/results-Baseline.csv

  • Process:

    1. For each project (national and testing), the CSV files from both the buildstockbatch and run_analysis directories are read into memory.

    2. The headers (columns) of each CSV file are compared.

    3. Any extra columns in buildstockbatch that are not present in run_analysis (excluding specific columns such as apply_upgrade columns) are reported.

    4. Similarly, any extra columns in run_analysis that are not present in buildstockbatch are reported.

    5. An assertion is made to check that there are no additional columns in either of the files after excluding the specific columns related to apply_upgrade.

  • Expected Outcome:

    • The test passes if the column headers in the two files match, except for the known differences related to apply_upgrade columns.

  • Assertion:

    • The test asserts that there should be no extra columns in either the buildstockbatch or run_analysis files, after excluding the apply_upgrade columns.

test_baseline_results Function

This test verifies that the numerical results in the baseline CSV files from buildstockbatch and run_analysis are consistent, within a specified tolerance.

Test Description:

  • Input:

    • buildstockbatch results: Located at buildstockbatch/project_<project>/project_baseline/results_csvs/results_up00.csv

    • run_analysis results: Located at run_analysis/project_<project>/project_baseline/results-Baseline.csv

  • Process:

    1. For each project (national and testing), the specified column (report_simulation_output.energy_use_total_m_btu) from both the buildstockbatch and run_analysis CSV files is extracted.

    2. The numerical values in the column are summed for both files.

    3. An assertion is made to check that the number of rows in the specified column matches between the two files.

    4. The sums of the values are compared using assert_in_epsilon, which allows for small differences (up to 0.01).

  • Expected Outcome:

    • The test passes if the sums of the values in the specified column are nearly identical (within 0.01 tolerance).

  • Assertion:

    • The test asserts that the sums of the report_simulation_output.energy_use_total_m_btu values from both files are within a tolerance of 0.01, ensuring that there are no significant discrepancies between the results.