HatariWii/tests/tosboot

Files
-----

Tools and data files:
Makefile      -- builds the test program and floppy & HD images
                 containing it
disk/*        -- sources, binaries and input files for test program,
                 directory is also used for GEMDOS HD emu testing
floppy/*      -- files to autostart test program from floppy
tos_tester.py -- test driver, described below

Generated files:
output/*       -- Test report and screenshots, temporary output files
dummy.cfg      -- Hatari config file generated by test driver
blank-a.st.gz  -- blank floppy image to avoid TOS disk dialogs
bootauto.st.gz -- floppy image with the test files and test program
                  run from auto/-folder for TOS <1.04
bootdesk.st.gz -- floppy image with the test files and test program
                  run from *.INF desktop file
hd.img         -- HD image with the the test program / files
                  and DOS partition table

Other files and directories:
readme.txt -- this text file
tos/*      -- TOSDIR in Makefile points here for your TOS images

There's also "screenshot-report.sh" script to generate a HTML report
out of the screenshots saved by TOS tester which will list missing
tests and any differences in the produced screenshots.  For that, you
need "reference" directory to contain screenshots from an earlier,
succesful run of TOS tester.


Usage
-----

If you want to test Hatari version that isn't in your PATH,
you need to give PATH for the Hatari binary you want to test,
like this:
	PATH=../../build/src:$PATH make

Before running that, tos/ subdirectory should have (symbolic links to)
TOS images you want to test, at least EmuTOS etos512k.img image.
Or add this before above command:
	TOSDIR=<path to TOS images dir>

Alternatively, you can call the TOS tester directly and specify
the TOS images it should test:
	PATH=../../build/src:$PATH ./tos_tester.py <TOS images>

To view the produced screenshots, either use ImageMagick:
	display output/*.png

Or use the script that creates a HTML page with them and opens
browser to view it.


What TOS tester tests
---------------------

These are the HW configuration combinations that TOS tester currently
supports:

* ST, STE, TT and Falcon machine types

  EmuTOS 512k is tested for all the machine types.  EmuTOS 192k, 256k
  and TOS v2.x are tested with all machine types except Falcon. Rest
  of TOSes are tested only with a single machine type.

* TV, VGA, RGB and monochrome monitors and 1, 2 & 4 plane VDI modes

* Different amounts of ST-RAM from 0 (0.5MB) to 14MB

* Different amounts of TT-RAM from 0 to 256MB

* With and without GEMDOS harddisk directory emulation

  Test program is started either from a floppy or an emulated GEMDOS
  HD (directory) using *.INF file, or in case of TOS v1.00 - 1.02,
  from floppy auto/-folder.  GEMDOS HD testing is done with more
  extensive gemdos.prg test program, floppy testing with minimal.prg
  test program which doesn't change the floppy content (to avoid its
  repository file update).

* ACSI and IDE interface testing with EmuTOS

* Arbitrary boolean Hatari command line options specified
  with the "--bool" option

You can use the command line options to specify which set of these
is used and TOS tester will go through all combinations of them.

See "tos_tester.py -h" output for examples.


What to test
------------

For each Hatari release it would be good to test e.g. the following
TOS versions:
  v1.00 de, v1.02 de, v1.04 de, v1.04 us, v1.62 de, v1.62 us,
  v2.06 de, v3.06 us, v4.04, etos192k, etos512k[1]

[1] Just the latest release of EmuTOS.

And following monitor configurations:
  ST:     tv,  mono, vdi-1, vdi-4
  STE:    rgb, mono, vdi-1, vdi-4
  TT:     vga, mono, vdi-1, vdi-4
  Falcon: rgb, mono, vga

Memory configurations:
  ST:   0.5 &  2 MB
  STE:    1 &  4 MB
  TT:     2 & 10 MB ST-RAM, 0 & 32 MB TT-RAM
  Falcon: 4 & 14 MB ST-RAM, 0 & 32 MB TT-RAM

And both with GEMDOS HD and just floppy.  For EmuTOS, also IDE & ACSI.

Testing should be done both for old UAE CPU core and the new WinUAE
one.  This should give good enough coverage of all the possible bootup
issues.

Note that it's enought to give the whole set of HW configurations to
TOS tester, it will automatically select a suitable subset of HW
combinations to test, for each given TOS versions.


Potential TODOs
---------------

Extend GEMDOS test program to test also:
* starting another program
* file redirection

But only if all TOS versions support that properly.

-

Add testing of ASCI and IDE drives with normal TOS in addition to the
GEMDOS HD and floppy tests.

This isn't very straightforward because both need different drivers
and therefore different disk images and the drivers either have issues
with e.g. EmuTOS, or don't support all machines.  Formatting and
installing the drivers requires using interactive Atari programs,
so these images cannot be automatically (re-)generated.

(EmuTOS supports IDE and ACSI directly, without any need for drivers.
Both HDs with DOS (not Atari) partition table, and ones without
partition table at all.  That's why it can be already tested.)

-

Testing a HD disk having also MiNT (= WinUAE / MMU testing at the same time).

-

Machine type specific test programs e.g. for:
* ST color resolution overscan
* STE blitter and overscan
* TT FPU operations, could output e.g. speed
* Falcon DSP operations

If pre-existing demo programs are used for these, it's better
if they're small.

Such programs also needs to have some static screen which doesn't
automatically advance so that a screenshot can be taken of it.
Alternatively, test program could be accompanied with debugger
script(s) that stop the program at suitable point and take a
screenshot.

-

Current screenshot-report.sh script assumes that Hatari will always
create identical screenshots for the same screen content.  This might
not be true if underlying libpng gets updated, so it would be better
to have some e.g. SDL binary that loads two images, compares their
uncompressed content and either reports that, or shows the difference.

-

Tester for DMA sound output and comparison for the produced sound.