DMK file format and tools

Page 1/2
| 2

By Louthrax

Prophet (2076)

Louthrax's picture

30-06-2017, 14:52

Here's a little report on my recent experience playing with the DMK file format and tools (I'm currently working on a tool to write-back DMK files to real floppies using a SuperCard Pro device).

So I first tried the official openMSX READ-DMK.COM tool, and it was not working on any of my MSX machines here: the drive was spinning but its LED stayed off. I checked the code and discovered that the phil_select routine was selecting drive #2 instead of drive #0. That can be fixed by replacing 0xC2 by 0xC0 here:

; 1) select
phil_select	ld	a,(0xF348)
	ld	h,0x40
	call	0x0024	; select FDC slot in page 1
	ld	a,(side)
	ld	(phil_control1),a
	;ld	a,0xc2	; motor on, led on, drive A
	ld	a,0xc0	; motor on, led on, drive A
	ld	(phil_control2),a
	ret

My guess is that on some machines, bit 1 is ignored, so when you use 0xC2 it activates drive #0. On some other machines, that bit is used, and drive #0 is not activated.

I also did some changes in READ-DMK.COM to write the .DAT files to the current drive and specify the slot of the disk drive to use, allowing use of mass storage devices.

So I finally had a working READ-DMK.COM tool. Before that I tried to insert a 2nd drive on a Philips 8250 I had in the attic, breaking some screws when removing the drive plate holder (dismantling a 8250 is a real super-mess!), just to finally discover that my 8250 was not working anymore (black screen... was pissed off and did not investigate further).

I then tried to make DMK images of both SD Snatcher (disk A) and Ys II (Program disk), which I know were copy-protected.

For SD Snatcher, READ-DMK stopped with a "Read track returned too much data" error. Maybe that could be fixed by increasing a buffer size? DMK Creator 5.1 created a DMK file without reporting any error, but when launching that image in openMSX, I got a "This disk is not original error".

For Ys II, READ-DMK worked, and the image was totally working in openMSX. With the image created with DMK Creator 5.1, game was stuck on a black screen. I did some checks with my beta DMK-write program, and it seems that DMK Creator does not handle the sector size field or CRC correctly:

READ-DMK.COM report (track 7 has a sector size of 0 for sector #5 and in consequence bad CRCs for both id and data):

Track 7:
	Sector #1: track 3 side 1 number 1 size 2 idcrc 6683 (6683) datacrc b15a (b15a)
	Sector #2: track 3 side 1 number 2 size 2 idcrc 33d0 (33d0) datacrc bc42 (bc42)
	Sector #3: track 3 side 1 number 3 size 2 idcrc 00e1 (00e1) datacrc 0564 (0564)
	Sector #4: track 3 side 1 number 4 size 2 idcrc 9976 (9976) datacrc 787d (787d)
	Sector #5: track 3 side 1 number 5 size 0 idcrc 8098 (8a05) datacrc 01c0 (5c11) *IIC *IDC
	Sector #6: track 3 side 1 number 6 size 2 idcrc ff14 (ff14) datacrc c1dd (c1dd)
	Sector #7: track 3 side 1 number 7 size 2 idcrc cc25 (cc25) datacrc 57d9 (57d9)
	Sector #8: track 3 side 1 number 8 size 2 idcrc dc1b (dc1b) datacrc 1fb0 (1fb0)
	Sector #9: track 3 side 1 number 9 size 2 idcrc ef2a (ef2a) datacrc 796c (796c)

DMK Creator 5.1 report (everything seems normal here, even if track 7 is protected):

Track 7:
	Sector #1: track 3 side 1 number 1 size 2 idcrc 6683 (6683) datacrc b15a (b15a)
	Sector #2: track 3 side 1 number 2 size 2 idcrc 33d0 (33d0) datacrc bc42 (bc42)
	Sector #3: track 3 side 1 number 3 size 2 idcrc 00e1 (00e1) datacrc 0564 (0564)
	Sector #4: track 3 side 1 number 4 size 2 idcrc 9976 (9976) datacrc 787d (787d)
	Sector #5: track 3 side 1 number 5 size 2 idcrc aa47 (aa47) datacrc 90ee (90ee)
	Sector #6: track 3 side 1 number 6 size 2 idcrc ff14 (ff14) datacrc c1dd (c1dd)
	Sector #7: track 3 side 1 number 7 size 2 idcrc cc25 (cc25) datacrc 57d9 (57d9)
	Sector #8: track 3 side 1 number 8 size 2 idcrc dc1b (dc1b) datacrc 1fb0 (1fb0)
	Sector #9: track 3 side 1 number 9 size 2 idcrc ef2a (ef2a) datacrc 796c (796c)

I also noticed that READ-DMK.COM sometimes failed when trying to identify the number of sectors on a track (displaying 9, 18, 27... and some weird number after that). That's more likely to happen on my HB-F1XDJ here, and might be related to too small delays after moving the heads (just a supposition here).

A final note on the DMK format: from what I've seen, it does not allow to store all "markers" for a track, it's just keeping an index on the sector's first IDAM marker. The 2nd DAM marker is not stored, and has to be guessed from the DMK file bytes when re-creating a real disk. Maybe that could compromise the backup integrity of some disks with sophisticated protection mechanisms involving weird marker patterns.

Login or register to post comments

By Manuel

Ascended (15686)

Manuel's picture

30-06-2017, 21:16

I'd like to know on which machines you tried the original READ-DMK.COM. It did work fine on my 8255.
Also, it would be cool if you could also share the other improvements.

With which program did you make these 'reports'? Do you also know analyze-dmk?

By Louthrax

Prophet (2076)

Louthrax's picture

30-06-2017, 23:59

Hi Manuel,

I tried READ-DMK on my VG-8235, Sony HB-F1XDJ and turboR, with the same results (disk drive spinning but no LED activated on that drive and program stuck). My attempts on my VG-8250 failed because that machine just died (yet another thing to fix here Smile).

Maybe I'll make a new version of READ-DMK with an improved interface in C code (scanning available disk drivers and offering a choice between them, allowing a direct DMK generation without the need for an external tool for concatenation, possibility for bigger tracks size when needed...). My changes so far are a bit hacky and not super user-friendly (need to specify a specific slot manually for the disk-driver)...

The program I used to make these reports is my new DMK to SCP tool (name is still to be defined), which should allow to write back DMK images to real floppies.I'll make it public with sources when (if?) it works. That should be generic C code portable to Linux & Mac. You'll still need a SuperCard Pro device to write that image back to a real disk... The ultimate goal would be to have DMK files supported in SofaRun / SofaRunIt Smile

I tried other tools before that (DMK2IMD and DISK-ANALYSE), but they all failed for some reasons (DMK2IMD generated non-working images, DISK-ANALYSE is not handling "bad" tracks with copy-protections...). I did not try ANALYSE-DMK as I had already started working on my new tool.

By Manuel

Ascended (15686)

Manuel's picture

01-07-2017, 00:07

You did use the latest READ-DMK and latest openMSX to test? There have been some fixes in openMSX regarding this lately.

By Louthrax

Prophet (2076)

Louthrax's picture

01-07-2017, 00:51

I used version 0.38 from the sources I got... somewhere by googling, do not really remember. I had to re-compile that in order to patch the changes I mentionned. I'd be glad if you could point me to that latest release with sources (maybe that's inlcuded in the openMSX sources repository ?).

By Manuel

Ascended (15686)

Manuel's picture

01-07-2017, 09:21

Well that sounds good. The latest is in the openMSX source repository under Contrib/DMK. But when testing the resulting DMK in openMSX, better use the latest development version.

By wouter_

Champion (417)

wouter_'s picture

01-07-2017, 10:24

Louthrax wrote:

... That can be fixed by replacing 0xC2 by 0xC0 here: ...
...I also did some changes in READ-DMK.COM to write the .DAT files to the current drive and specify the slot of the disk drive to use, allowing use of mass storage devices...

Nice. And as Manuel said, it would be great if you could share those improvements.

I'm well aware that READ-DMK is not a very user-friendly tool. That's why I'm happy to see the development of new tools like Dmk Creator (unfortunately that tool still has various limitations).

I see you're also interested in creating an improved READ-DMK tool. I'd be very happy to help you with that.

Louthrax wrote:

...For SD Snatcher, READ-DMK stopped with a "Read track returned too much data" error. Maybe that could be fixed by increasing a buffer size?

The ideal data-rate and rotation speed for a 3.5" DD floppy drive are 250kbps and 300rpm. That results in an ideal track-size of 6250 bytes. In reality this number can be slightly lower or higher. READ-DMK will print the message "Read track returned too much data" when the read-track command already returned more than 14000 bytes. That's way out of spec! I've seen that happening on drives where the index pulse isn't detected reliably anymore (so the FDC can't detect the end of the track and continues reading for a 2nd or 3rd revolution). So READ-DMK stops with an error (and retries) instead of continuing to read and overwrite all memory (and crash). Sometimes ejecting/reinserting the disk helps to fix this problem. If not you may need to clean the drive.

Louthrax wrote:

...I also noticed that READ-DMK.COM sometimes failed when trying to identify the number of sectors on a track (displaying 9, 18, 27... and some weird number after that). That's more likely to happen on my HB-F1XDJ here ...

In this step READ-DMK is executing WD2793 read-address commands to read all sector headers in the track. It does this for (approximately) 3 revolutions of the disk. Normally the 2nd and 3rd revolution should be a repetition of the 1st revolution. If not there was a read error somewhere. READ-DMK tries to detect these read errors by checking for a repeating pattern. In case of an error the length of this pattern might be 18 or even 27 sectors long instead of 9 sectors (length=27 means there's no repetition at all). Next READ-DMK checks the length (in bytes) of this period, that should be around 6250 bytes. If not it will retry.

I found that the data returned from a read-address command generally has a much higher error-rate than the data returned from normal read-sector commands. So read errors in this step, especially on older/lower quality drives, are not uncommon.

Louthrax wrote:

...A final note on the DMK format: from what I've seen, it does not allow to store all "markers" for a track, it's just keeping an index on the sector's first IDAM marker. The 2nd DAM marker is not stored, and has to be guessed from the DMK file bytes when re-creating a real disk. Maybe that could compromise the backup integrity of some disks with sophisticated protection mechanisms involving weird marker patterns.

That's right. The DMK format does have some limitations. This is one of them. Though, so far, I've not seen an actual disk where these limitations are a problem.

Over 5 years ago, when I had to pick a protected file format for openMSX, I explicitly wanted an existing file format instead of trying to invent my own. DMK seemed like a good choice, and I'm still happy with that choice today. (It's a very simple format and still powerful enough for all known disks). Also at that time I didn't know enough yet about this low-level disk stuff to be able to fully understand the limitations.

If these limitations ever do become a problem we can switch to an even lower level disk format like 'MESS floppy image' (.mfi).

By Manuel

Ascended (15686)

Manuel's picture

01-07-2017, 13:21

At least I already put that drive select fix in READ-DMK, creating version 0.39 Smile

By Louthrax

Prophet (2076)

Louthrax's picture

01-07-2017, 13:56

Thanks for the details Wouter, I'll keep you updated! Just an extra question: do you think that writing back a DMK file would be possible on an MSX machine? Any challenge you see with that? I'll first go for a PC version, but that will be restricted to people owning a SuperCard Pro.

By wouter_

Champion (417)

wouter_'s picture

01-07-2017, 15:11

For sure not all DMK files can be written back on a MSX using a WD2793 FDC. Probably less can be written back on MSX machines with a TC8566AF. The game 'Pixess' by 'Compjoetania is an example of a disk that cannot be created on a MSX machine.

I think DSK-PRO does a reasonable job of copying images that can be read/written by a WD2793. Although I don't think it can handle the important subset of Sunrise and Micro Cabin copy protections. And I know those can be written by a WD2793.

So if you want to be able to write all possible DMK images you should stick to your SuperCard pro approach. If you're happy with a subset you can try to make a MSX program, but it can get rather complex.

By Louthrax

Prophet (2076)

Louthrax's picture

01-07-2017, 22:48

wouter_ wrote:

For sure not all DMK files can be written back on a MSX using a WD2793 FDC. Probably less can be written back on MSX machines with a TC8566AF. The game 'Pixess' by 'Compjoetania is an example of a disk that cannot be created on a MSX machine.

Ok, maybe that will be a later project then. The important thing would be to notify the user that the disk has not be successfully written.

Another question for you Wouter: I was planning to generate a whole DMK file directly, but my guess is that it might be oversized as the maximum track length for the disk is not known in advance. My understanding is that currently READ-DMK writes all tracks in individual .DAT files, and that the PC tool that does the concatenation computes the biggest track size to generate a smaller DMK file (as the track size is global for the DMK file). Is that correct?

Page 1/2
| 2