".TSX" --> ".TZX" Format in MSX

Page 8/11
1 | 2 | 3 | 4 | 5 | 6 | 7 | | 9 | 10 | 11

By TomH

Champion (327)

TomH's picture

12-02-2018, 15:47

Oh, no, sorry! I just didn't want my comments getting shuffled onto the wrong pile, as my understanding of the discussion to date is:

  • TZX is something already well-supported, albeit that we're extending the specification for TSX;
  • but it's an ugly file format, in the sense of imposing a substantial development burden;
  • so let's discuss whether there's something 'better', so we'll at least have discussed it.

If the issue is implementation burden then obviously the verdict of a broad range of tool and emulator authors is the only way to evaluate a suggestion.

My thoughts run the gamut:

  1. taking the advice that deflate is too onerous for an Arduino on account of its sliding window size, the simplest thing to adopt is the subset of CSW that permits only RLE compression. No Z-RLE;
  2. if we were planning to define an Arudino-compatible compression standard, the next step along the CSW chain of thought is adding a sufficiently-trivial predictor or set of predictors, following the proven example of PNG;
  3. conversely, if it were more desirable to come at this from a CAS angle, then keep blocks of bytes but make the encoding explicit (I still think microcoding does this much with a much lower burden than templating, but either way), and add an extra block for things that don't fit the block of byte (which in the case of microcoding is just another sequence of instructions and a repeat count; if you were templating you'd flip the way around you were doing it — the template would specify segments in the encoding of the raw pulses block).

Which of those is the 'best' choice, and whether any is better than TZX-as-extended, is a judgement call for the community to make.

Whatever we pick, I will implement it. I've been vocal against TSX but have already (mostly) implemented it.

EDIT: oh, sorry, forgot the additional argument point: TZX chunk 0x18 is the CSW chunk. So a complete TSX implementation requires a full CSW implementation, unless the intention is to modify TZX both by adding and by taking away. Therefore TSX is unambiguously (i) a greater implementation burden than CSW (because it's CSW plus a bunch of stuff); and (ii) unsuitable for Arduino implementation (because CSW with Z-RLE compression can require the full deflate 32kb sliding window).

By nataliapc

Resident (53)

nataliapc's picture

16-02-2018, 11:09

Hi TomH, thanks for your time to write that answer.

TomH wrote:
  • TZX is something already well-supported, albeit that we're extending the specification for TSX;
  • but it's an ugly file format, in the sense of imposing a substantial development burden;
  • so let's discuss whether there's something 'better', so we'll at least have discussed it.

I agree with the 3 points.
I guess TZX was selected like base format for TSX because the often use of Spectrum data blocks in MSX commertial games in order to preserve them more accurately.
And, TSX is technically a superset of TZX format, but in practice (for MSX, and maybe the rest of KCS systems) you can implement it in your solution like a subset implementing the main used blocks (10, 11, 12, 13, 20, 4B) and how to skip the rest.

TomH wrote:

If the issue is implementation burden then obviously the verdict of a broad range of tool and emulator authors is the only way to evaluate a suggestion.

That's will be the better.
I also developed a MSX1 emulator some years ago for Android devices (unreleased right now).
http://www.natygames.com/lang/es/projects/msx-emulator/

TomH wrote:

My thoughts run the gamut:

    [ ... ]
  1. conversely, if it were more desirable to come at this from a CAS angle, then keep blocks of bytes but make the encoding explicit (I still think microcoding does this much with a much lower burden than templating, but either way), and add an extra block for things that don't fit the block of byte (which in the case of microcoding is just another sequence of instructions and a repeat count; if you were templating you'd flip the way around you were doing it — the template would specify segments in the encoding of the raw pulses block).

Which of those is the 'best' choice, and whether any is better than TZX-as-extended, is a judgement call for the community to make.

I'm for a new byte based format like CAS or TSX but simple and fully scalable (using pulse based bit/byte templates), and with some special block allowing pulse sequences for more complex exceptions. Also can be added a block to define atomic pulse operations (microcode?), and tone pulses, and for pauses.

Some like this for required implemented blocks:

  • Block to define bit template (here we are defining also bauds freq)
  • Block to define byte template (1/0, start, stop, parity...)
  • Bytes block
  • Block for atomic operations: pulse state change/reset, periodic tone pulses, pauses, ...
  • Block for general pulse sequence (allowing RLE?)

And for optional/skipped blocks:

  • Metadata/Archive block
  • Group start/end block (maybe to distinguish tape side A/B and other light uses using the same file)

Use of RLE could be discussed. I'm for a fully uncompressed format to be lighter for implementations, and having a raw format in order to edit/change inner data easily.

TomH wrote:

Whatever we pick, I will implement it. I've been vocal against TSX but have already (mostly) implemented it.

LOL, +1
As I said simply a subset of full TSX implementation could be enough for concrete purposes :)

TomH wrote:

EDIT: oh, sorry, forgot the additional argument point: TZX chunk 0x18 is the CSW chunk. So a complete TSX implementation requires a full CSW implementation, unless the intention is to modify TZX both by adding and by taking away. Therefore TSX is unambiguously (i) a greater implementation burden than CSW (because it's CSW plus a bunch of stuff); and (ii) unsuitable for Arduino implementation (because CSW with Z-RLE compression can require the full deflate 32kb sliding window).

Yes, I agree that full TSX superset have a high implementation burden.
But the subset of most used blocks is enough for 99% of use cases and have a better low burden.

By nataliapc

Resident (53)

nataliapc's picture

16-02-2018, 13:27

Another thing to discuss could be if bit/byte templates must have his own global block (like UEF baudrate chunk) or if must be inserted innerly in the bytes block definition.

The reason is for possible issues and/or increase in complexity, when rewind or go forward blocks in the tape stream.

By TomH

Champion (327)

TomH's picture

16-02-2018, 19:36

Baud rate as a modal was one of my decisions, so I have to take responsibility for that; my feeling at the time was that random access is always problematic in a chunk-based file format — e.g. there's no easy way to go back one chunk without some sort of record of accumulated knowledge, so why not accumulate all state instead of partial state?

I think I was also leaning into the idea that it was something some emulators might prefer to ignore, and the default semantics for that are to put it into a chunk.

The latter isn't as compelling an argument, but I still have a lot of time for the former.

I think if you wanted to resolve the issue entirely in favour of no-knowledge then probably the smartest thing would be a table of contents? That's what PDF does. In PDF's case it's always at the end of the file but, ignoring that, you'd open with a list of chunks that just went (ID, pointer within file, length in time) or some other fixed-size structure, and then throw chunk contents onto the end.

Even with something like CSW that isn't explicitly chunked you still can't seek cheaply because the current offset is the sum of all lengths encountered to date.

By CASDuino

Expert (90)

CASDuino's picture

18-05-2018, 15:54

UEF is becoming a more viable possibility for TZXDuino as it can now handle unzipped UEF files, it would just need some work into adding the MSX custom and standard recognition.

By eimaster

Master (220)

eimaster's picture

19-05-2018, 04:05

Hi friends.
All of you are scientests developing new things for humanity. Discussion between you will certainly lead to better solutions. I am really thankfull for your efforts.
I only have a small question...
If a WAV file is really a close -if not equal- recording of this or that computer tape sounds/audio, and if that WAV file become smaller when compressed to ZIP or 7z file, then why inventing new formats?! We can record our own tapes to WAV files, then compress them -for easy preserving/uploading/downloading purposes. What is left is that emulatorists implement ZIP or 7z decompression along with their own emulators in order to read the WAV file and deal with it!
Convence me of the benefits of inventing formats other than a compressed WAV files.
NOTE: All of you forgot about SVI-3x8 tapes format which openMSX emulator understands only as WAV.

By CASDuino

Expert (90)

CASDuino's picture

19-05-2018, 11:27

eimaster wrote:

If a WAV file is really a close -if not equal- recording of this or that computer tape sounds/audio, and if that WAV file become smaller when compressed to ZIP or 7z file, then why inventing new formats?!

A zipped WAV file is still relatively large when compared to a dedicated file format. Admittedly with modern storage and internet download speeds it is less important than it was in the earlier days of emulation but it still an important factor.

By nataliapc

Resident (53)

nataliapc's picture

19-05-2018, 15:58

Quote:

Convence me of the benefits of inventing formats other than a compressed WAV files.

Sorry, I don't want convince no one, but if you want to help, you are welcome, ofcoz...

Have you any SVI tape like that to see how the pulses are encoded?

Thanks ^^

By CASDuino

Expert (90)

CASDuino's picture

19-05-2018, 21:15

nataliapc][quote wrote:

Have you any SVI tape like that to see how the pulses are encoded?

Thanks ^^

You can download the files in a .CAS format from here but it may not be the same .CAS format as the MSX. In fact if OpenMSX wopn't load them as .CAS then it isn't the same format.

I've had a quick look at the CAS files and they are different. I will try to see if I can make a .WAV for you.

By Tolvatar

Paladin (935)

Tolvatar's picture

19-05-2018, 23:42

eimaster wrote:

Hi friends.
Convence me of the benefits of inventing formats other than a compressed WAV files.

Every time that you create a WAV file it's different. Even if you use the same tape and the same reader and program
This makes impossible to know if is a good dump or not.
Cas format it's very good, but it's not perfect, specially por games that use Spectrum loading methods.
If you want to play then you can use the format you want if works on your MSX
If you want preservation you need to get 1:1 images of the tape.
Thats how i think

Page 8/11
1 | 2 | 3 | 4 | 5 | 6 | 7 | | 9 | 10 | 11