fwTest — Smart Test Runner

What It Is

fwTest is not a C library — it's a bash + Python test runner that serves as the standard test harness for every fw-lib and every FiWorks application. Tests are defined as simple text files with NAME, RUN, and EXPECT directives. The runner executes each test, captures stdout/stderr, and compares against expected output using intelligent matching.

• REGEX patterns — match dynamic content like UUIDs, timestamps, and version strings in expected output
• SORT blocks — order-independent comparison for non-deterministic output (e.g., hash table iteration order)
• GUI diff viewer — side-by-side visual comparison with color-coded match/fail highlighting
• Flexible selection — run by name, index, range (--ixList 1-5,10,25-30), or retest only failures (-re)
• Color output — green for pass, red for fail, yellow for skip

Test File Format

Each test is a .test file with simple directives. Paths are relative to the project root where fwTest.sh lives.

# Comments start with #
NAME:        Human-readable test description
RUN:         command to execute
EXPECT:      path/to/expected-stdout.txt
ERR-EXPECT:  path/to/expected-stderr.txt

Example Test

# test/funcTests/cases/parse-json.test
NAME:        Parse simple JSON object
RUN:         ./myjson test/data/simple.json
EXPECT:      test/expect/simple.expect

For error cases:

# test/funcTests/cases/invalid-json.test
NAME:        Reject invalid JSON
RUN:         ./myjson test/data/invalid.json
ERR-EXPECT:  test/expect/invalid.err-expect

REGEX and SORT

REGEX Patterns

Match dynamic content in expected output files. Wrap any regex in REGEX(...):

{
  "id": REGEX([a-f0-9-]{36}),
  "timestamp": REGEX([0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9:.]+Z),
  "version": REGEX([0-9]+\.[0-9]+\.[0-9]+)
}

SORT Blocks

Mark sections where line order doesn't matter. Lines between #SORT_START and #SORT_END are sorted before comparison:

{
  "items": [
#SORT_START
    {"name": "apple"},
    {"name": "banana"},
    {"name": "cherry"}
#SORT_END
  ]
}

Combined Usage

REGEX and SORT work together — regex matching applies within sorted blocks:

#SORT_START
id: REGEX([0-9]+), user: alice
id: REGEX([0-9]+), user: bob
id: REGEX([0-9]+), user: charlie
#SORT_END

Command-Line Usage

# Run all tests (default: test/funcTests/cases/)
fwTest.sh

# Run single test by name
fwTest.sh demo
fwTest.sh demo.test

# Run all tests in subdirectory
fwTest.sh subdir/

# Run by index range
fwTest.sh --fromIx 10 --toIx 20
fwTest.sh --ixList 1-5,10,25-30
fwTest.sh --skip 3,7-10

# Rerun failed tests
fwTest.sh -re

# Visual diff on failure
fwTest.sh -g              # GUI viewer
fwTest.sh -d meld         # Use meld

# Control output
fwTest.sh -v              # Verbose (show commands)
fwTest.sh -q              # Quiet (only failures)
fwTest.sh -s              # Stop on first failure

Option	Description
-d, --diff TOOL	Diff tool: diff, tkdiff, meld
-g, --gui	Open graphical diff viewer on failure
-re, --retest	Only run previously failed tests
--fromIx N	Start from test index N
--toIx N	End at test index N
--ixList LIST	Run specific tests: 1-5,10,20-30
--skip LIST	Skip specific tests
-v, --verbose	Show commands being executed
-s, --stop	Stop on first failure
-q, --quiet	Only show failures

Test Output

0001/50/0: Parse JSON object....................................... OK   (0 s)
0002/50/0: Handle invalid input.................................... FAIL (0 s) - stdout mismatch
0003/50/1: Process large file...................................... OK   (2 s)

Format: TEST_NUM/TOTAL/FAILURES: Name... RESULT (time) [- reason]

GUI Diff Viewer

Use -g for a graphical side-by-side comparison with color-coded output:

• White — match (exact, REGEX, or SORT)
• Red — failure (mismatch, missing, or extra line)

The viewer supports click-to-navigate failures, Previous/Next buttons, and synchronized scrolling.

Failed Test Artifacts

On failure, output files are kept for inspection:

test/funcTests/cases/mytest.out       # Actual stdout
test/funcTests/cases/mytest.err       # Actual stderr
test/funcTests/cases/.fwTest-failed    # List of failed tests (for --retest)

Dependencies

• bash 4.0+ — the test runner itself
• Python 3.6+ — for the smart diff engine (REGEX/SORT matching)
• tkinter (optional) — for the GUI diff viewer (-g flag)