MSX Basic Tokenizer. Should I?

Страница 3/4
1 | 2 | | 4

By Manuel

Ascended (15620)

Аватар пользователя Manuel

03-08-2019, 09:57

By the way, headless, or renderer none, is default when using the -control option.

By farique

Rookie (23)

Аватар пользователя farique

06-08-2019, 00:54

So,
That Python script back there to use openMSX to tokenize the programs was horrendous and doesn't even work, mea culpa. So I fixed it and made it into a proper (almost) respectful program; it even goes both ways now.

It also made me realize that when openMSX is headless it loads the ASCII file MUCH faster, which made that whole Python tokenizer kind of obsolete, by the way.

Testing with a ~500 line ASCII program I got these load times:
>30s normal, ~8s throttled, <2s headless!

I think it is now ready for you, @Louthrax.

Here are the script and the readme.

By farique

Rookie (23)

Аватар пользователя farique

06-08-2019, 06:00

Be careful when using the script above!
I've found a bug when using an MSX 2 with a built in disk drive (Sony HB-F500P) that is corrupting the files on the folder inserted as a disk.
I noticed that after a few conversions with the script, the files on the folder are corrupted and become "stuck" on the machine, a "files" command no longer show the contents of the real folder anymore, showing the files that were there when the corruption occurred. Removing the disk has no effect.
Deleting the .cmos file on the "persistent" folder helps when booting normally but the savestate is irreparable.
I tried to "set save_settings_on_exit off" and to remove the disk before quitting with no avail.
My Sharp HotBit MSX 1 with A Microsol Disk controller works without a problem.
Any ideias on what might be happening?
Thanks.

By Manuel

Ascended (15620)

Аватар пользователя Manuel

06-08-2019, 08:26

Why are you using a savestate?

By Grauw

Ascended (8317)

Аватар пользователя Grauw

06-08-2019, 10:07

farique wrote:

It also made me realize that when openMSX is headless it loads the ASCII file MUCH faster, which made that whole Python tokenizer kind of obsolete, by the way.

I don’t know, it’s neat that it works but I think I would still rather invoke a simple tokenizer script than boot up an entire emulator for it... 2 seconds is still a pretty long time in a build chain for tokenizing just one file if you ask me, my current builds fire up almost instantly.

By farique

Rookie (23)

Аватар пользователя farique

06-08-2019, 15:22

@Manuel, I am using a savestate because of @Grauw's argument, speed. A savestate avoids the boot time and, as he said, 2 seconds IS a long time in this context. The need for speed came from the Dignified Build and Run process, in fact.

@Grauw, It was really just an exploration of the possibilities, just to see where it went. I had already made a tokenizer from scratch (not perfect yet but not far from it and not relying on external tools). Turns out I found the openMSX approach useful to quickly DEtokenize stuff.
The "obsolete" referred to the Build process of the Dignified Basic (the motivation to write the tokenizer), ~3s in that context is not horrible. Opening and closing a bunch of emulators to batch convert stuff is definitely NOT nice Smile (not that batch converting stuff cannot be made on a single instance of the emulator)

By Manuel

Ascended (15620)

Аватар пользователя Manuel

06-08-2019, 17:59

At full throttle the boot up should be really short.

By Manuel

Ascended (15620)

Аватар пользователя Manuel

07-08-2019, 00:02

Anyway, try without the savestate.

Also, take care that the contents of the dir doesn't exceed the maximum size which the disk drive can handle (360kB for the Sony HB-F500P, if I recall correctly. Otherwise 720kB.)

Reasoning of not using the savestate: f the content of the 'disk' isn't the same as when the state was saved, the 'disk' will be in read only mode to avoid corruption.
I don't think it will really help if you use full throttle anyway.

By farique

Rookie (23)

Аватар пользователя farique

09-08-2019, 02:30

Yeah, I only realised how fast openMSX is at full throttle after testing the tokenizer script.
I'm really busy right now and haven't had a chance to look further into it.
I need to complete the custom tokenizer to have an all Python solution to Basic Dignified and for that I need the openMSX script to follow your idea of mass tokenize and detokenize random code on a proven platform to compare with my results.
I am aware of the disk lock due to the changing folder content, the script was ejecting and inserting the folder again to avoid problems, was working fine on an MSX 1. I wonder if creating a blank disk file and messing with disk management to copy files to and from it wold work, maybe a bit convoluted. Anyway, the full throttle speed should be enough to get rid of the savestate.
Well, I will give it a try and post the results here for posterity Tongue
Cheers.

By Manuel

Ascended (15620)

Аватар пользователя Manuel

09-08-2019, 22:00

Please do, as I'd really like to know whether that corruption can be explained somehow because of the savestate usage. So if it doesn't happen without it, it would be nice. I'm also interested to know how much time it takes at full throttle headless Smile You could also mute all sound, may save some more time.

Страница 3/4
1 | 2 | | 4