Idea/Proposal: ROM Headers for emulation

صفحة 4/6
1 | 2 | 3 | | 5 | 6

بواسطة Vampier

Prophet (2402)

صورة Vampier

11-01-2021, 20:59

She signed up to add 2+ cents to the conversation.

بواسطة Grauw

Ascended (10638)

صورة Grauw

11-01-2021, 21:43

Additionally, I’m not sure what is now claimed that is hard to implement.

I can understand the difficulty of zip, even on MSX loaders this would be a barrier of entry. A zip-strategy answer to that could be to for example unzip it on the machine that you downloaded it on before copying it to your SD card, and match the manifest with the ROM file based on filename without extension. While emulators and loaders like SofaRun could directly access the zip.

I could even understand the processing simplicity of a static header, although then I think general purpose format is out of the window and you’re treading well into the territory that the format must match the internal representation, iow a config register dump followed by a ROM dump, but ok.

However when a RIFF format is brought up as feasible, I lose understanding of what the problem is with an editable text format (no, not xml). Then it just seems unwillingness to think along.

Basically if you want to define a format for everybody, and some person puts all kinds of restrictions on it specific to one platform (MiSTer), quickly dismissing anything that does not match the preconceived idea, then basically the format is simply better defined by this person alone, and you’re not defining a general format but a MiSTer format.

بواسطة mcolom

Champion (277)

صورة mcolom

11-01-2021, 21:17

I think it's an interesting idea to have a new format with useful metadata. I wouldn't call that a "ROM" not to confuse with a plain ROM dump, but a different format which might include a ROM dump. BTW, I'm not an expert in FPGAs or the last detail of MSX, just another user giving a humble opinion Smile
I don't see it's that important that the format can't be read directly by an FPGA. As far as I know, these are not reading ROMs, but rely on 3rd party tools (like SofaROM and others) under DOS to patch them and run them within a mapper. And even if the ROM type is well identified, patching might not work, or it might not be enough.
To me, it's a question of having a format with complete information (anything that might be useful), and hardware (like MFR or C2) that can be configured to mimic a mapper. These are FPGAs, but I don't think they should be able to read the format directly, but instead to be configured by an external tool (like c2man and others). This tool could help the hardware to simulate a right behavior. The more complete the information in the format, the better. How much information you add to the metadata format should be independent from tools that read them, in the sense that these configuration tools might read the information they need to configure the megaROM. And emulators can also benefit from complete information.
Well, just an opinion from a user point of view.

بواسطة ren

Paragon (1930)

صورة ren

11-01-2021, 21:34

@tfh: haha, okay so I was mistaken, my molestake.
Know that I reacted slightly out of emotion, but definitely out of surprise of this topic showing up, just some days after I posted a very similar 'idea/proposal'. So with my assumption in mind (which I had my reasons for), I reacted to seeing your reaction here, but not in 'my' topic.

Anyway, I'm not sure if Vampier was aware of my topic, if he is/was I find his ways remarkable to say at least. I can imagine he felt like stepping up after seeing that topic (I do realize he does the romdb stuff). But I should be careful making assumptions (it could be a big coincidence of course.. Wink)

And I think @Kitrinx could have done with an introduction, I kinda suspected/assumed from the outset already she & Vampier spoke on this topic already / were aligned.

Anyway, we should strive for the best (feasible) solution, and perhaps both ideas don't necessarily bite. I might react more on the subject later.

بواسطة tfh

Prophet (3258)

صورة tfh

11-01-2021, 21:30

@ren: It still leaves me a bit puzzled on what you were trying to achieve with that remark. Some would call it "passive aggrasive". I suggest the next time you think there might be some issue, you just send me an e-mail instead.

For the rest I also think it might be a good idea to merge these ideas and get to a single solution which can be used on all software for MSX. Examine what has already been done, learn from their mistakes, add in MSX-specifics and make something nice, universal & future-proof.


Enlighted (5976)


11-01-2021, 22:55

I think here is exactly where this discussion started to go south:

Vampier wrote:

FPGA can't do pre-processing as it doesn't work like an emulator where you load a blob of data - dissect it and have nice small chunks. FPGA is byte by byte processing and making decisions tight there.

The thing is that this has nothing to do with FPGA or MSX. Even if people would try to solve this problem by using factory generated CMOS or NMOS dices instead the problem would be exactly same as the hardware is anything you make out of it.

What they mean is that on their Minster-project they have generated some basic framework that they try to keep as simple as possible so that it does not take much space from the FPGA, but is still capable of loading stuff for multiple platforms... and this makes sense, because they want to run dozens of platforms other than MSX and this would mean that each core would need to have their own hardware, filesystem, network etc. drivers just for loading and that would be impossible to maintain... Technically their common loading platform COULD be ie. Amiga, PC or MSX but in this case it is not.

Part of the problem is that when we talk about FPGA MSX most of us think about OCM and there the loading platform is indeed MSX... and that makes sense as adding new core just for loading BIOS and stuff from SD would be pretty stupid... But these are different starting points that aim to different outcome...

Is it stupid that Minster has this simple loader core? I don't think so... I think it is exactly as complex as it needs to be to handle the given task that is loading data. It is anyway just a design choice and not technical limitation.

Should we design a format based on this particular design choice is yet another question. Yes there are ways around the problem... The loader core can ie. just dump the zip file to RAM as is and then kick the MSX core up to handle the rest, but the performance would be similar to real MSX.

I think it is good to keep simplicity in mind because that will help performance also on MSX, but some particular FPGA design can't be the reason to start making things super hard way, but let's try to find a solution that would make most of the people happy. Maybe ie. TAR could be considered better that ZIP?

بواسطة Grauw

Ascended (10638)

صورة Grauw

11-01-2021, 23:31

For MiSTer you could write loaders that run before execution of the BIOS. The number of processors is limited, so if you have a loader for Z80, 6502 and 68000 you got a lot of systems covered already. Each platform probably needs adjustments for their respective file formats, but it would give a lot of flexibility and seems simple to implement. The gate cost would be near zero.

The 1chipMSX does this, it is called the “initial program loader” (IPL).

You could even pre-load the raw data into memory via the existing method (for improved boot time) and then run a post-processor on the CPU which configures the hardware mappers etc.


Enlighted (5976)


11-01-2021, 23:32

Hmm... thinking about it again, TAR might not be good as the header is quite UNIX specific and the files may end up to wrong order that can be a bit troublesome especially if IPS-patch is stored before ROM... I'm not totally against custom binary blob with new extension, but I hope someone could come up with better and more backwards compatible idea.

بواسطة Grauw

Ascended (10638)

صورة Grauw

11-01-2021, 23:50

I’ve thought about tar… My preference for zip is that it’s universal in modern PCs, so they’re easy to create, and if your target can’t accept it, just as easy to decompress before transferring over (or after, with SofaUnzip). However that does not apply to tar, particularly on Windows and mobile devices.

And that applies even less to techniques like prefixing the ROM with the manifest, which will require custom tools to combine and split (pretty trivial ones, but nevertheless). Users lose the benefit of easily viewing the manifest to e.g. see the license or mapper type, or editing it to correct a mistake.

But I think in principle the system should not depend on being combined in a single file. The manifest and binary could just live next to each other on the file system. Similar to m3u or srt files.

p.s. Zip can store files without compression if needed to e.g. reduce decompression time.


Enlighted (5976)


15-01-2021, 12:32

Although we currently may not agree, if we want manifest file or embedded binary header the discussion should not end here. Once we have the information in some standard and well thought out format the conversion can be fully automatic. Maybe we should be thinking the manifest as "source" from where the information can be compiled to binary?

صفحة 4/6
1 | 2 | 3 | | 5 | 6