Emulator Comparison: The Saga continues

Note that not only 'hard' things should be tested. There should at least be a few very popular software titles tested, since they will be run by many users of the emulators. (Like Metal Gear (2), SD-Snatcher, etc.)
You could also pay attention to the completeness of the emulator. Emulating a basic MSX2 is one thing, but there's more devices for MSX On the other hand: many people (the kind that I just mentioned) won't care aboutg that...
How about testing how suitable the emulator is for MSX software developers? Or portability to other platforms? There are so many aspects one might consider... Good luck!

Oh and yes: please wait until we release openMSX 0.4.0.... (note that 0 is the major version part, 4 is minor.... But it's a relatively major release indeed, since this release will come with a full version of the Command Interface and a(n optional) GUI which will make heavily use of it!)

Dunno... MSX-Player is very good aswell, and BlueMSX... never tried that one, but I heard good things about it, at least user-interface-wise.

I don't think taking stuff manuel mentioned into account will instantly put openMSX on the top... Compared to good emulation e.d. it doesn't weigh heavily ofcourse. And where it does, perhaps other emulators their possibly better UI's (haven't tried the new Catapult yet) or other features will counter it.

And besides, it's not like, a bad thing to have openMSX on the top . Also, there doesn't nessecarily have to be one 'ultimate' winner... Some others might be reasoned secondary winners like, 'if you're a common windows user and just want to play a few of the popular games with one click drag n drop support blah blah, use this' and 'if you want good emulation with still an ok interface, use that', etc.

~Grauw

If something is 'the best' doesn't that automatically mean it's 'superior' to the rest?
I'm looking forward to your definitions of 'the best' and 'superior'.

It's nice to have enthusiastic users, but it's true that openMSX is still a long way from being perfect.

What makes openMSX different from the other MSX emulators is the EmuTime model, which allows synchronisation between different devices at an arbitrary precision. This makes it possible to do near-perfect emulation (meaning there is no theoretical limit to accuracy), but the complexity of the code also makes bugs more likely. Emulators which use a simpler timing model (such as those based on fMSX) have a limit to the accuracy they can achieve, but because the code is less complex, they are less likely to have bugs.

About test cases:

A demo we had lots of trouble with is Relax from ANMA. In fact, the mouse emulation still doesn't work correctly with Relax. The "Boring Scroll" part does work now, but the fix for that is a bit of a hack.

Asking for test cases on this forum will create a bias of course, because emulator authors will list software they have encountered problems with before and in many cases those problems have been fixed. So it might be a good idea to also take some random samples from your software collection and if you find a program which is not emulated correctly on all emulators, include it in the comparison.

About the format:

It would be useful to have a kind of "features matrix": a table which lists features on one axis and emulators on another. So you can easily see which emulators have MSX Audio, turbo R, video filters, debugging support etc. It could be as simple as yes/no, but a quality indicator (0-5 penguins?) would be nicer, if you have the time to make that.

Also I think it would be useful to make the comparison less of a contest and more of a summary of what is available. The previous comparison focused a lot on the score system, which lead to the strange result that openMSX was in the lead on the score board (mainly because of its ability to run demos well) but was not one of the recommended emulators (because of slowness and the lack of a GUI).

You could try to change the score system, but no matter what system you come up with, your weighing of different aspects of an emulator will not match that of a user, because no two users are alike. For example, if you use the emulator for gaming, support for joysticks, save states, speed control and auto fire would be important. If you use the emulator for watching demos, accurate emulation of obscure features of various chips (especially the VDP) is important. If you use the emulator for MSX cross-development, debugging features are important and accuracy is important as well, because it's annoying if your developed program runs fine on the emulator but shows glitches when run on a real MSX. Also the ability to emulate many different MSX models and extensions would be useful for development.

Another problem with a score system is that some features are very subjective. For example video filters (scalers, scanlines, blur): some people love them, others turn them off immediately. Or the emulation of the Kanji ROM: important if you can read Japanese, but otherwise more of a nice-to-have. Or the inclusion of system ROMs in the emulator distribution: it's convenient for the users, but it violates copyrights in many countries (except for MSX Player which does have licensed ROMs, but only in Japan).

I think it would be useful to make the emulator comparison not a snapshot of the state at a certain moment in time, but something that is continuously updated. So when a new version or even new emulator comes out, (re)test it and update the comparison. Maybe not retest for every single release, but try to make sure the test results are not more than a couple of months old. I understand this is a lot of work, so make sure you get some help.

I'll anxiously be awaiting new versions of all the major MSX emulators