Testing methods

by MSX Resource Center on 21-04-2005, 00:00

To see how good an MSX emulator actually is we calculate its MRC EmuRank, using various methods of testing. We don't only test the accuracy of the emulation, but we also have a look at features, usability and music quality. The testing methods used to calculate the MRC EmuRank are described below. Every emulator review ends with a list of scores and features the emulator achieved during these tests. To give an indication of how much system-resources the emulator needs, we also added the average CPU load the emulator needed in MSX2 and MSXturboR mode while being idle (in BASIC), using the indication of the Windows Task Manager. All tests were run on the same PC (Intel Pentium 4 2.55GHz, 1024MB RAM, NVIDIA GeForce 4 Ti46000, Terratec DMX 6fire 24/96) running Microsoft Windows XP SP2. Throughout the tests we tried to run the emulators in their default settings for as much as possible, as we assume most end-users would do so as well.  

 

Benchmarks

 

The first tests we created for testing the accuracy of the emulators are two very simple benchmarking tools in BASIC, that run on any MSX emulator. The first tool runs 10 different algorithmic tests while the second tool runs 10 different tests in text and graphical modes. A normal MSX2 (Philips NMS-8250) scores exactly 1000 points on both benchmarks. Higher scores indicate a faster computer (emulation). A small indication: An MSX turboR scores 5857 points in the CPU benchmark and 2208 points in the VDP benchmark.  

 

Of course running a benchmark which is based on the timer of the (emulated) computer seems kind of ridiculous. Different scores can have all sorts of reasons, especially since the benchmarking tool was coded in MSX-BASIC. If, for instance, the timer itself was emulated incorrectly, the whole score would mean totally nothing. No one can argue, however, that a perfect MSX-emulator should score (very close to) 1000 points on both benchmarks, just like a real MSX. Although the results of these benchmarks are not used in the calculation of the MRC EmuRank they do give a nice first impression on what an emulator has got to offer.  

 

MSX compatibility

 

Because most emulators run almost all common MSX1 and MSX2 games without any problems, we decided to give the emulators a real hard time with games and demos that are known to be hard to emulate. As expected, none of the emulators passed all tests. The differences between the emulators show quite obvious, however, and that was the objective of this comparison. To make these difference even clearer, we used a rating system. Every test would result in a score from 0 to 5 points according to these rules:  

 

0 points - Does not boot at all  

1 point - Hangs in the beginning, black screens  

2 points - Excessive slowdowns, hangs after some tests, totally messed-up effects, not possible to play the game  

3 points - Annoying errors, hangs during a very hard test, difficult effects displayed incorrectly  

4 points - Runs almost perfect. The software is playable. Slowdowns. Some glitches  

5 points - Runs as on MSX  

 

The following products were used to test the emulators:  

 

  • 20th anniversary demo (Maggoo)  
  • Aleste 2 (Compile)  
  • Almost Real (Mayhem)  
  • Avaakkus (Lieves!Tuore)  
  • Bombaman (Team Bomba)  
  • FAC Demo V (FAC/FCS)  
  • Illusion City (Microcabin)  
  • Maggoo's Vertical Sync Scroller (Maggoo)  
  • Metal Limit (IOD)  
  • Moonlight Saga Promo (Mapleyard)  
  • Multiplex (Maina)  
  • No Waste (Xelasoft)  
  • Paradream (RAC House)  
  • Relax (Anma)  
  • SD Snatcher (Konami)  
  • Seed of Dragon (Riverhill Software)  
  • Sphere (Infinite)  
  • Stone (Mountainsoft)  
  • SWISS Demo (Mi-Chi and A.T.)  
  • TNI's Bounce Demo (TNI)  
  • Turnix (FCS)  
  • Unknown Reality (NOP)  
  • Vscreen (Maggoo)  
  • Woei Demo (Team Bomba)  
  • Zone Terra (Quadrivium)  

Music quality

 

Because music has always played an important role on MSX the emulation of the various music chips of the MSX are being tested as well and rated in a similar way compared to the MSX Compatibility rating:  

 

0 points - Not available  

1 point - Poor  

2 points - Average  

3 points - Good  

4 points - Almost perfect  

5 points - Exactly like the real thing  

 

Usability and Features

 

After performing 16 MSX2 compatibility tests, 10 turboR compatibility tests and rating the music quality of the PSG, SCC, MSX-MUSIC, MSX-AUDIO and OPL4 chips one final score is needed before the MRC EmuRank can be calculated: usability and features. This score is based on how easy (or hard) the emulator can be operated and whether or not features like real disk emulation, image enhancement (e.g. allowing you to make images look better by using a Scale2x filter) are implemented. Each review ends with a list of these features. This list will give a quick overview of the capabilities of the emulator.  

 

MRC EmuRank

 

The ideal MSX emulator is capable of running all MSX software without any problems, sounds exactly like a real MSX and has an extremely intuitive User Interface with all the features one could wish for. To see how close current Emulators get to this ideal emulator we have introduced the MRC EmuRank. This Rank is based on the MSX2 and turboR compatibility, music quality, usability and features of each emulator. The MRC EmuRank is calculated as follows:  

 

MSX Accuracy Score = (8*MSX2 compatibility score + 2*turboR compatibility score)/10  

MRC EmuRank = (4*MSX Accuracy Score + 2*Music Score + usability and Features score)/7  

 

Over here you can find the top 12 MSX emulators based on EmuRank scores.  

 

Future testing

 

And that's about all you need to know about this comparison. We are always happy to hear comments and suggestions. So if you spotted an error, found some software that is hard-to-emulate, or think something is missing from the comparison, please let us know by sending an e-mail to emucomp@msx.org. We're especially interested in interesting software titles which are suited for testing MSX compatibility and accuracy.