My plans for Nextor v3 (warning: broken drivers ahead!)

Страница 1/3
| 2 | 3

By konamiman

Paragon (1083)

Аватар пользователя konamiman

01-08-2020, 20:30

Sorry for the clickbait-ish title but I really wanted to grab your attention. Smile

This post is a dump of what’s going on my head regarding the future of the Nextor project, and will be of interest mostly for driver developers. Nothing of what follows is set in stone, and feedback is highly welcome.

The Nextor project has been quite a long and interesting ride so far, and I’m happy to see how well it has been accepted by the MSX community. At this point it’s got most of the features I had planned to implement when I started the project, being the remaining ones:

  • Support for loading drivers in RAM, additionally to store them in ROM. To see the usefulness of this keep in mind that a Nextor driver can not only handle storage devices, but also provide interesting capabilities like implementing BASIC CALL commands, providing a timer interrupt hook, or hooking into the extended BIOS; these things require currently manual work like allocating RAM in page 3 and adjusting various system pointers, imagine if Nextor could be able to handle all of this gracefully.
  • Support for filesystem drivers. That’s a huge one. So additionally to low-level device drivers having basically a “read and write sectors” function, a filesystem driver would instead have functions like “find files”, “read from file”, “rename file”, etc; of course directly mapped to the equivalent system function calls. That would allow amazing stuff, from leaving advanced hardware to take the burden of dealing with filesystem management, to map Nextor drive letters directly to FTP servers.
  • One that I hadn’t considered initially but needs some love is the support for floppy disk drives. There’s no way to implement basic floppy-related features like formatting a disk or providing a phantom drive (the last one can be done, although in an overly complicated way, in Nextor 2.1), and this needs to be fixed - so that you can e.g. use a USB FDD as it’s meant to be used, natively from a Nextor driver.

However, before plunging into those I would like to take a step back and make a couple of changes to the driver structure. These changes are intended to somewhat simplify the code of Nextor (and hopefully the process of writing new drivers too), and are driven by the observation of how existing drivers have been written:

  • Get rid of drive-based drivers

Nextor allows to use two kind of drivers: “drive-based”, where the driver itself handles a number of drives much like MSX-DOS does; and “device-based”, where the driver controls a number of devices and Nextor maps drives to partitions on these devices.

To the best of my knowledge, nobody has written a drive-based driver for Nextor; all the existing drivers are device-based.

Removing support for drive-based drivers would allow me to remove some code from the Nextor kernel, and would make the process of writing a new driver (and the associated documentation) less confusing.

  • Get rid of LUNs

Nextor drivers can control up to seven devices, each having up to seven logical units. I designed it that way because that’s how SCSI and (more relevantly nowadays) the USB storage class work.

However this has proven to be confusing. I have been asked questions like “My device has two card slots, is that two devices or one device with two LUNs?”; and the truth is that there’s no right answer, both methods would work. I always end up recommending using two devices with a single LUN for simplicity.

Therefore, I think it’s safe to change the “7 devices with 7 LUNs” design to just “255 devices”. If a driver still needs to internally use LUNs, it can do something like supporting 15 devices and mapping device numbers as “main device id + 16 * LUN id” or something like that.

  • Unify the info/status routines

Right now Nextor drivers need to implement three routines to get information about devices: DEV_INFO, DEV_STATUS, and LUN_INFO. The idea is to deprecate those and replace them with a single DEV_QUERY routine, where the index of the information to be queried would be passed in a register; this mechanism is less confusing and easily extensible without having to create new routine entry points.

As for getting information about the driver itself, Nextor has three mechanisms: hardcoded information (driver flags and name), a DRV_VERSION routine, and an optional DRV_CONFIG routine. The plan would be to leave only DRV_CONFIG (renamed to DRV_QUERY) making it mandatory; and, similarly to DEV_QUERY, use an input register to indicate the index of the information to be queried.

Although implementing both routines would be mandatory for all drivers, not all the queries would be mandatory to implement; in fact the idea is to have as much of them as possible being optional, with Nextor being able to resort to sensible defaults as appropriate.

“Wait a second”, you’ll say. “But that’s incompatible with the current driver design. Does that mean that drivers would have to be rewritten?”

Well… yes and no.

Please be so kind to drop your torches and pitchforks and allow me to finish the post. Thanks.

If Nextor was a Windows-like sized project, with thousands (millions?) of supported pieces of hardware out there, I wouldn’t be able to afford introducing such breaking changes. But we are the MSX community, which is comparatively very small; and only a handful of drivers have been developed. From the top of my mind:

  • Sunrise IDE (and compatibles like Carnivore2)
  • MegaFlashROM SCC+ SD
  • Flashjacks
  • Rookie Drive
  • SD Mapper
  • GR8NET

...and maybe a few others, but you get the idea: we are in “small committee” and I’m sure we could somehow coordinate to get any required changes done.

“But why don’t you support both types of drivers at the same time?” I had considered that, but that approach would add too much complexity to the Nextor code, especially if LUNs are indeed removed from the equation. And given the constraints I have on the time I can afford to dedicate to the project, I need to simplify stuff as much as I can.

“Then… will I have to rewrite my driver from scratch??” No. Not from scratch. Apart from the mentioned changes, the rest of the driver structure would remain unchanged (most notably the DEV_RW routine, which would only require changes in case of supporting LUNs). The changes would mostly consist of moving some code around (notably, from DEV_INFO/DEV_STATUS/LUN_INFO to DEV_QUERY); I would publish a detailed driver migration guide, and I would migrate the Sunrise IDE driver myself to guide by example.

Going back to the title of the post, I think that such a breaking change well deserves a main version bump, so these changes would be implemented in a future Nextor 3.0.

“And what about Nextor 2? Will you abandon it?” No. Or better said, not yet. Nextor 2 would enter into “maintenance mode”, this means that I would continue fixing any bugs found but I wouldn’t implement any new feature on it (in other words: but fixes would go to Nextor 2 and 3, anything new would go to Nextor 3 only). Of course I can be flexible here: if I get a request for a feature that’s fairly easy to implement I might add it to Nextor 2 as well.

Thus, Nextor 2 would die only after Nextor 3 has a stable release and has drivers for all devices (a device explicitly or blatantly abandoned by its creator might be an exception here, if that ever happens).

So, what would be the next steps if I decide to go ahead with this plan?

  1. Publish a temptative detailed design of the new driver structure and the driver migration guide, get feedback, adjust if needed.
  2. Develop and publish a first alpha of Nextor 3.0, having the exact same functionality of Nextor 2.1 but with the new driver system, and two drivers migrated by myself: standalone and Sunrise IDE. That’s when driver developers could start the migrations.
  3. Start working in the new features of Nextor 3, while helping anyone struggling with driver migration, while fixing bugs in Nextor 2.1.

Uh… now that I re-read it, 3 sounds like too many things at once; but well, you get the idea, that would roughly be the plan.

“And what schedule/time estimation would this plan have?”

Ha ha ha!! That was a very good one! LOL!

So… that’s basically what’s in my mind regarding the future of Nextor. Now, what’s in yours?

Для того, чтобы оставить комментарий, необходимо регистрация или !login

By S0urceror

Master (137)

Аватар пользователя S0urceror

01-08-2020, 20:48

I agree. With the limited amount of drivers around we can do this. I don’t mind to update my MSXUSB driver. It looks like it would become much simpler. And simpler means less bugs.

For my Nextor wish list:
* dynamic memory allocation, in page 3, after DRV_INIT
* mechanism to tell volume has been changed
* telnet console via serial,tcpip

By geijoenr

Master (180)

Аватар пользователя geijoenr

01-08-2020, 21:09

I wish:
* long filenames (64 bytes?)

By Grauw

Ascended (9338)

Аватар пользователя Grauw

01-08-2020, 21:57

It's good to hear plans for the future of Nextor!

I'm not a driver developer, however it seems reasonable to ask for a driver update when a new major version is released. However, I guess this means that if I plug in two Nextor devices, one on version 3 and one on version 2, that the drives of the version 2 device can't be used?

As a developer the feature I would be most interested in is long file name support. In VGMPlay I would love to show the full file name of the file you're playing, and in the future also have them in a playlist / file browser. Also I would like to have it for m3u playlist support, they are often bundled with the VGMs and typically contain long filenames. I also think e.g. SofaRun would instantly adopt it the second it becomes available.

I think most DOS2 BDOS calls would be compatible with long filenames, since they already accept an asciiz string or a FIB (which needs no awareness of LFN). Only a new version of FFIRST and FNEXT would be needed where in addition to a FIB, the matching long filename is placed in a buffer passed by the developer. This is also the mechanism for applications to know the long filename given a short one and vice versa. I think a first version could be released without updating command2, to limit the scope.

As a user I would also like FAT32 support. FAT16 is generally big enough (although I can fill up 4 GB with VGMs no problem), but many storage media come preformatted as FAT32, and finding the FAT16 formatting options on modern PCs can sometimes be a bit of a challenge. So it would add convenience and lower the barrier of entry. Especially for novice MSX users who come to ask for help here on the forum. Being able to use media larger than 4 GB without creating multiple partitions would also be nice, because mapping partitions is a hassle and drive letters are limited to 8.

By ducasp

Champion (366)

Аватар пользователя ducasp

01-08-2020, 22:26

File System Drivers is a great call, I personally thought a bit on how to improve the slow response to FAT16 on SM-X due to the calculations being too heavy for a z80, but FPGA can handle it, FAt32, and other file systems quite easily, and that would also translate to an easier path to long file name support. Also, I know it wouldn't be standard, but I would like to see new Asyncrhonous functions extension (and that is a desire for UNAPI as well). We can't do DMA on MSX, but nowadays most hardware either has a microcontroller, fpga, or something powerfull enough to do IO operations on its own, and probably the driver itself can handle storing/retrieving the data to a buffer on page 3 (highest performance) or any mapper block (lower performance) and then calling a callback when done (either by success or error). That way applications coded to take advantage of the new functionality can offload a buffer loading while it proccess text on screen, audio, etc... Not DMA, but I believe it is a quite good step on what can be done on MSX architecture, would love to see asynchronous calls where the driver (be it Nextor or UNAPI) call back when done. Cool

By Grauw

Ascended (9338)

Аватар пользователя Grauw

01-08-2020, 22:13

Then there are a few more things that I have come across;

Firstly, writing to a file becomes increasingly slower as the file gets larger and the disk gets fuller, because new clusters need to be allocated and this search becomes slower and slower. It is possible to speed things up massively by pre-allocating the entire file in one go by writing a single byte to the last location of the file, which I use in gunzip and I know Louthrax uses in SofaUnzip. However it would be nice if a zero-length write would actually also allocate the file, but not write any contents. A zero-length write currently acts as a no-op.

Secondly, when exiting to DOS I always need to be careful to restore all mapper segments and slots to their initial values. Even though the DOS2 environment reference says:

Quote:

A program may do any slot switching and paging which it likes while it is running and need not restore either the slot selections or the RAM paging before it exits, since COMMAND2.COM will handle this.

That’s a big lie Smile. The fact that DOS2 does not do this by itself always seemed strange and risky to me, especially in the case of abnormal program termination. As a programmer it’s also a hassle because I need to store and remember these, and define abort handlers to ensure they are restored. To be honest I also kinda don’t trust all programmers to take sufficient care to do this Smile.

By konamiman

Paragon (1083)

Аватар пользователя konamiman

01-08-2020, 23:12

S0urceror wrote:

* dynamic memory allocation, in page 3, after DRV_INIT

That's complicated. It would need NEXTOR.SYS to reload (since it's relocatable) and that's why any application allocating page 3 code does the "jump to BASIC, then _SYSTEM" trick.

S0urceror wrote:

* mechanism to tell volume has been changed

That's what DEV_STATUS currently does, or do you mean something different?

S0urceror wrote:

* telnet console via serial,tcpip

Isn't this kind of already possible by using the console related hooks?

By konamiman

Paragon (1083)

Аватар пользователя konamiman

01-08-2020, 23:20

Grauw wrote:

I guess this means that if I plug in two Nextor devices, one on version 3 and one on version 2, that the drives of the version 2 device can't be used?

I'm afraid that would most likely be the case. Trying to have simultaneous compatibility for v2 and v3 sounds like a big headache.

Grauw wrote:

As a developer the feature I would be most interested in is long file name support.

That might be doable with the understanding that new function calls would be needed and thus COMMAND2.COM wouldn't support it out of the box. Note however that LFN involves Unicode which is something that MSX doesn't even remotely support so we would need to figure out how to handle that (support only ASCII maybe?)

Grauw wrote:

As a user I would also like FAT32 support.

That one seems definitely too big for a MSX to handle, not only for the data structures needed but also for the processing speed (e.g. the allocation of new clusters that you mention later). However with filesystem drivers implemented that could at least be delegated to external hardware.

Grauw wrote:

and finding the FAT16 formatting options on modern PCs can sometimes be a bit of a challenge

And that's precisely why Nextor has a built-in FDISK. Smile

By Pencioner

Scribe (1172)

Аватар пользователя Pencioner

01-08-2020, 23:24

Grauw wrote:

Firstly, writing to a file becomes increasingly slower as the file gets larger and the disk gets fuller, because new clusters need to be allocated and this search becomes slower and slower. It is possible to speed things up massively by pre-allocating the entire file in one go by writing a single byte to the last location of the file, which I use in gunzip and I know Louthrax uses in SofaUnzip. However it would be nice if a zero-length write would actually also allocate the file, but not write any contents. A zero-length write currently acts as a no-op.

I have noticed this problem before, but pre-allocating implementation might be tricky. What i see as possible solution for FAT drivers - when searching for the free cluster, don't start from he beginning each call, but store the last succesfully found free cluster index in RAM and start from there. If end of drive is reached the start over from first one. That should be easier i guess...
This problem exists not only in Nextor, but Fat16 driver for DOS2 by Okei has exactly the same issue

(BTW on large devices it becomes hilariously slow, i once tested with 4Mb partition filled 75% and copying 16Mb file into it - after one hour i just turned computer off - it was still copying stuff. Computer was not freeze, because the leds were blinking normally)

By Grauw

Ascended (9338)

Аватар пользователя Grauw

02-08-2020, 01:39

konamiman wrote:
Grauw wrote:

As a developer the feature I would be most interested in is long file name support.

That might be doable with the understanding that new function calls would be needed and thus COMMAND2.COM wouldn't support it out of the box. Note however that LFN involves Unicode which is something that MSX doesn't even remotely support so we would need to figure out how to handle that (support only ASCII maybe?)

Hmm, you could either translate to UTF-8, or pass it through verbatim. Regardless the application will need to handle it in the end. The former is a little more convenient since it’s ASCII compatible.

Pencioner wrote:

I have noticed this problem before, but pre-allocating implementation might be tricky.

Pre-allocation already works, just I need to write 1 byte unnecessarily at the last position of the file.

By sd_snatcher

Prophet (3348)

Аватар пользователя sd_snatcher

02-08-2020, 01:02

konamiman wrote:
Grauw wrote:

As a user I would also like FAT32 support.

That one seems definitely too big for a MSX to handle, not only for the data structures needed but also for the processing speed (e.g. the allocation of new clusters that you mention later). However with filesystem drivers implemented that could at least be delegated to external hardware.

Is it? I have a Master Everdrive, and it can handle SDHC cards formatted in FAT32 on the Sega Master System without hassle. And the SMS has only 8KB of RAM. Much less than an MSX running Nextor.

And it's not hw assisted in any way. This everdrive contains just a very simple CPLD, very similar to the one inside the SD-Mapper on the MSX. Everything else is done entirely by software.

AFAIK, EXSDOS for the ZX-Spectrum DivIDE also has support for FAT32 read/write without any extra hardware.

If I'm not wrong, even SymbOS has FAT32 support without external hw help.

Страница 1/3
| 2 | 3