.. _testing_index: ====================== Testing Infrastructure ====================== This page describes the ResStock testing infrastructure, which ensures that code, configuration, and documentation meet quality and functional standards throughout development. Overview ======== ResStock's testing infrastructure is built on `GitHub Actions `_ and supports: - **Continuous Integration (CI)** to ensure code stability - **Automated Testing** for model integrity and software logic - **Documentation Generation** - **Results Validation** CI is triggered automatically on pull requests, merges to the ``develop`` and ``main`` branches, or via manual workflow dispatch. The test suite includes jobs for running different types of tests and tasks, including file formatting, unit and integration tests, as well as comparisons of simulation results. .. note:: To merge pull requests, it is highly recommended that CI passes (the green check mark on the latest branch commit) before merging a new feature or bug fix. There are some situations where a CI failure is okay (example: synching between BuildStock-Batch repository changes) temporarily while repository management across the software stack is being changed. Functions --------- The CI infrastructure performs the following core functions: - Ensures input files are well-formatted and sorted - Validates HPXML models and measures - Compares outputs from different simulation tools (`run_analysis.rb` and BuildStockBatch) - Builds HTML and PDF documentation from source - Uploads test artifacts to aid debugging and verification - Validates and publishes SDR upgrade results Test Scenarios -------------- The test suite covers: - Integrity of input files and simulation workflows - Functionality of measures and upgrade logic - Consistency of annual and timeseries simulation outputs - Differences between feature and base branches - Documentation generation and structure - Specialized SDR simulations and data formatting CI Triggering Conditions ------------------------ CI is triggered on: - Pushes to `main` or `develop` - Pull request events (`opened`, `synchronize`, `reopened`) - Manual invocation via the GitHub Actions tab (`workflow_dispatch`) - Completion of dependent workflow runs Concurrency Control ------------------- To avoid test queue congestion, CI jobs from the same branch or PR are automatically canceled if a new run is queued. This helps to avoid situations where resources are used for meaningless tests against out-of-date branch versions. Environment Variables --------------------- Shared environment variables (in both the GitHub Actions configuration and project YML files) define: - OpenStudio installer package (e.g., version, SHA, and platform) - Branches for BuildStock and other dependencies (e.g., BuildStock-Batch and BuildStock-Query) - Simulation batch settings (e.g., account, n_jobs, etc.) CI Jobs ======= This section documents the testing infrastructure and continuous integration (CI) process for ResStock to ensure the software stack works when running testing or production runs. Each CI job is defined in the main workflow (i.e., ``config.yml``) and other supporting workflows. They are executed in a structured order (see image below), with dependencies enforced through the `needs` keyword. .. image:: ../../images/ci_workflow.png :width: 1200 :alt: The GitHub Actions workflow for ResStock's CI tests A brief overview will be provided for each job. Some of the more complicated jobs and testing suites have their own page discussing the contents of the tests in more detail. Main CI Workflow ---------------- format-files ~~~~~~~~~~~~ Cleans and sorts the ``resources/options_lookup.tsv`` file: - Removes trailing whitespace - Sorts by the first two columns - Uploads the cleaned file as an artifact This job helps readability of ``options_lookup.tsv`` changes and organizes the parameters and options for easy navigation. The formatting also helps with differences between development on MacOS vs Windows machines and developers editing the file in Microsoft Excel. Artifacts uploaded: - options_lookup unit-tests ~~~~~~~~~~ Runs test suites in the OpenStudio container. Executes: - Project integrity checks - Measure unit tests - Integrity check tests Artifacts uploaded: - precomputed_buildstocks - feature_samples - coverage .. note:: If the unit tests fail, this will likely cause almost all of the simulations to fail during a test run or production run. postprocessing-tests ~~~~~~~~~~~~~~~~~~~~ Adds a test to ensure the columns used in the SDR dictionary is valid by matching against the ResStock data dictionary. build-documentation ~~~~~~~~~~~~~~~~~~~ Builds and commits: - **Technical Development Guide** (Sphinx, HTML) - **Technical Reference Guide** (LaTeX, PDF) Artifacts uploaded: - documentation After a successful merge into develop, this job creates the documentation ResStock Technical Development Guide, which is hosted on Read-the-Docs. To view the Technical Development Guide locally, use the `README instructions `_. .. note:: The documentation artifact is where a developer can find the latest ResStock Technical Reference Guide.pdf. analysis-tests ~~~~~~~~~~~~~~ Runs the project_testing and project_national baseline YML files using ``run_analysis.rb``. Then processes the results to be compared to the BuildStockBatch run. Uses ``project_testing/testing_baseline.yml`` and ``project_national/national_baseline.yml`` for the simulation. Artifacts uploaded: - run_analysis_results_csvs .. note:: Using ``run_analysis.rb`` is the easiest way to simulate a single building model for debugging purposes. Running ``run_analysis.rb`` is best when modifying the models, not the input characteristics. integration-tests ~~~~~~~~~~~~~~~~~ Runs full simulations of project_testing and project_national using BuildStockBatch. The data is then processed to be compared with the results from ``run_analysis.rb`` run in the analysis-tests job. Uses ``project_testing/testing_baseline.yml`` and ``project_national/national_baseline.yml`` for the simulation. Commits validated result sets to version control: - Base branch samples - Base branch annual and timeseries results - Precomputed buildstocks - Formatted `options_lookup.tsv` Runs (only on pull requests) to compare the base branch results with the feature branch results. This job is useful for comparing impacts of model changes. A model change happens when something is modified/added/deleted (e.g., piece of code) in ResStock or OpenStudio-HPXML measures. Compares: - Baseline results (annual + timeseries) - Differences in samples and outputs Artifacts uploaded: - base_results - buildstockbatch_results_csvs - raw_simulation_output - comparisons .. note:: Using ``buildstock_local`` is the easiest way to locally run a small stock simulation for debugging purposes. This is especially useful when modifying the input characteristics or inspecting results changes accross models. .. note:: This job is *not* useful for TSV changes as the samples are shuffled between the base and feature branches. compare-tools ~~~~~~~~~~~~~ This job compares outputs from ``run_analysis.rb`` (run in the analysis-tests job) vs. BuildStockBatch (run in the integration-tests job). It does this by comparing the columns and ensuring there are no missing or extra columns in either of the results. The job also tests to ensures the total annual energy is the same between the two simulation tools. sdr-options-analysis ~~~~~~~~~~~~~~~~~~~~ Analysis to determine upgrade applicability and generation of the 550,000 ResStock sample and minimal buildstock for SDR integration tests. - Generates 550K sample - Generates minimal builstock for SDR integration tests - Analyzes with `buildstock_query` - Tests for apply logic not applying to any samples Artifacts uploaded: - national_550ksamples - sdr_options_analysis Commits: - ``project_national/resources/sdr_minimal_buildstock.csv`` - ``project_national/resources/sdr_option_application_detailed_report.txt`` - ``project_national/resources/sdr_option_application_report.csv`` sdr-integration-tests ~~~~~~~~~~~~~~~~~~~~~ Full simulation test of SDR baseline and upgrades on the minimal buildstock from the SDR Options Analysis test: - Executes BuildStockBatch ``buildstock_local`` using ``project_national/sdr_upgrades_tmy3.yml`` - Validates successful simulations (no failures in the simulations) Artifacts uploaded: - buildstockbatch_results_sdr_raw_csvs - buildstockbatch_results_sdr_published_csvs - sdr_comparisons Commits: - Annual results from the SDR run on the minimal buildstock - Uploads raw and published results - Commits validated outputs .. note:: Check the ``buildstockbatch_results_sdr_published_csvs`` artifact to see how results will show up on OEDI. If you do not see a column or new output, then the data dictionaries have not been updated. Supporting CI Jobs ------------------ These CI Jobs are outside the main GitHub actions workflow for new feature development and bug fixes. They are used to help with the organization of the software development or assist with the new features and bug fixes. Add Pull Request or Issue to Project ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Adds newly opened or reopened Pull requests or Issues to a GitHub project board using `actions/add-to-project@v1.0.0`. **Location of the Project Board:** https://github.com/orgs/NREL/projects/38/views/1 This project board contains issues and pull requests from `ResStock `_, `ResStock-Estimation `_, `BuildStock-Batch `_, and `BuildStock-Query `_ repositories. Post Run: SDR Diff and Final Status ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Runs after the main CI completes successfully: - **SDR Diff**: - Generates and attaches CSV diffs as PR annotations - Uses a custom Python script for rich diff reporting - **Pass Checks**: - Manually marks required CI checks as passing - Updates GitHub commit status using the CLI Artifacts Produced ------------------ Artifacts produced if tests are successful: - options_lookup: A sorted and formatted version of ``options_lookup.tsv`` that allows diffs to easily be seen. - precomputed_buildstocks: Precomputed `buildstock.csv` for ``test/test_yml_files``. - feature_samples: The samples simulated in the feature branch for the analysis and integration tests. - coverage: Coverage logs. - documentation: A current version of the ResStock documentation in Read-the-Docs and the Technical Reference Guide PDF. - run_analysis_results_csvs: Annual results from the analysis tests. - base_results: Baseline and upgrades results stored on the base branch. - buildstockbatch_results_csvs: Annual results from the integration tests. - raw_simulation_output: Unprocessed simulation output files from the integration tests. - comparisons: Plots that show the difference in the results between the base branch and feature branch for integration-tests. Helpful during model changes, not characteristics changes. - national_550ksamples: The current 550,000 sample in ResStock that will be simulated. Used in the SDR integration tests. - sdr_options_analysis: Analysis of the upgrade options and their percent applicability based on the current national 550,000 sample. The minimal buildstock for SDR integration tests. - buildstockbatch_results_sdr_raw_csvs: The results.csv files from the ResStock SDR upgrades project file being run with BuildStock-Batch. - buildstockbatch_results_sdr_published_csvs: The results.csv files from the ResStock SDR upgrades project file being run with Buildstock-Batch transformed into what is being published on OEDI. - sdr_comparisons: Plots that show the difference in the results between the base branch and feature branch for sdr-integration-tests. Helpful during model changes, not characteristics changes. Test Suites and Files ===================== .. toctree:: :maxdepth: 2 integrity_checks running_task_commands run_analysis_tests bsb_analysis_tests test_analysis_tools