Integrity Checks and Testing
============================

Overview
--------

This section documents both the **integrity checking logic** and the **corresponding test suite** that ensures correctness of ResStock input configurations. These two components work in tandem:

- The `integrity_checks.rb` file provides programmatic validation of `.tsv` input files, ensuring they are well-formed, consistent, and complete.
- The `test_integrity_check.rb` file contains unit tests that exercise this logic under various failure scenarios, ensuring that errors are correctly caught and reported.

Together, they form a robust validation pipeline to prevent invalid configurations from entering the simulation workflow.

This documentation applies to the following files:

- `test/integrity_checks.rb`
- `test/test_integrity_check.rb`

The purpose of the `integrity_checks.rb` file provides logic to validate input `.tsv` files under `housing_characteristics/` and the associated `options_lookup.tsv`. These checks are essential to prevent simulation errors due to malformed or logically inconsistent configuration data. 
The `integrity_checks.rb` file **implements** the core validation logic for `.tsv` parameter files, `options_lookup.tsv`, and `buildstock.csv`. The `test_integrity_check.rb` file **tests** this logic by executing a wide range of expected failure and success conditions.
Each test case corresponds directly to a branch or condition within the `integrity_check`, `check_buildstock`, or related functions.
This separation allows robust unit testing of the validation logic before it is used in full simulation runs.

**Dependencies**

- `buildstock.rb`
- `run_sampling_lib.rb`
- Ruby standard `CSV` library

These dependencies provide utilities for parsing files and performing sampling.

Functions
---------

``integrity_check``

This is the main validation method for parameter `.tsv` files. It verifies:

- File structure and formatting (e.g., proper newlines).
- That all referenced parameters and their dependencies exist.
- That dependencies are resolved before use.
- That sampling probabilities sum to 1.0 for each dependency set.
- That each option defined is valid and present in `options_lookup.tsv`.

``integrity_check_options_lookup_tsv``

This function validates the `options_lookup.tsv` file itself. It checks that:

- No illegal characters appear in parameter/option names (`(`, `)`, `|`, `&`).
- Measure arguments are properly defined and not duplicated.
- All options listed are tied to valid measures and arguments.

``check_buildstock``

This function validates a `buildstock.csv` file generated by sampling. It ensures that:

- Every parameter and option in the CSV is valid per `options_lookup.tsv`.
- There are no missing or extraneous arguments or assignments.
- Measure argument duplication across parameters is not present.

**Helper Functions**

- `check_for_illegal_chars`: Verifies parameter and option names do not contain restricted characters.
- `check_parameter_file_format`: Ensures consistent newline characters (`\r\n`) across files.

Test Suite for Integrity Checks
-------------------------------

This section describes the test suite defined in `test_integrity_check.rb`.

**Purpose:** The test suite systematically tests invalid input scenarios to confirm that the integrity check functions correctly catch and report errors. It also validates successful scenarios to confirm proper integration between sampling, configuration, and CLI invocation.

Test Environment
~~~~~~~~~~~~~~~~

- Test lookup file: `resources/test_options_lookup.tsv`
- Invalid test folders: `test/tests_housing_characteristics/*`
- CSV test inputs: `test/tests_buildstock_csvs/*`

Test Class: `TestResStockErrors`
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The `TestResStockErrors` class extends `Minitest::Test` and includes:

- Setup for defining project directories and loading configuration.
- 18 test cases covering different failure and success modes.

Test Scenarios
~~~~~~~~~~~~~~

1. **Newline Characters:** Detects incorrect newline formatting in `.tsv` files.

2. **Sum Not Equal to One:** Validates probability weights for sampling sum to 1.0.

3. **Duplicate Rows:** Identifies redundant parameter rows with identical dependencies.

4. **Missing Sampling Row:** Checks that every possible sampling value maps to an option.

5. **Non-Numeric Values:** Ensures values meant to be numeric are not text strings.

6. **Missing Parent Parameter File:** Fails if a parameter’s dependency is not defined via its own `.tsv`.

7. **Unused TSV File:** Detects files not referenced in `options_lookup.tsv`.

8. **Missing Measure Argument:** Validates that required measure arguments are not omitted.

9. **Extra Measure Argument:** Flags undefined arguments passed to measures.

10. **Invalid Argument Value:** Ensures measure arguments have allowed enumerated values.

11. **Missing Measure Definition:** Ensures each referenced measure exists and is accessible.

12. **Invalid Dependency Option:** Checks for undefined values in a parameter’s dependency.

13. **Duplicate Measure Argument Assignment:** Detects repeated assignments of the same argument across parameters.

14. **Missing Option in Lookup File:**  Catches mismatches between `.tsv` parameter files and the lookup file.

15. **Option Spelling Errors:** Fails if option names do not match due to typos or case errors.

16. **Valid CSV Input:** Confirms that a well-formed `buildstock.csv` file passes validation both via Ruby method and CLI.

17. **Invalid CSV Parameters:** Tests presence of parameters not listed in `options_lookup.tsv`.

18. **Invalid CSV Options:** Flags values in `buildstock.csv` that do not map to any known options.