".TSX" --> ".TZX" Format in MSX

Page 7/11
1 | 2 | 3 | 4 | 5 | 6 | | 8 | 9 | 10 | 11

By rderooy

Champion (486)

rderooy's picture

31-01-2018, 13:44

Were there also not some games on tape that had a mix of data and audio? How would you capture this, other then WAV obviously?

By Tolvatar

Paladin (929)

Tolvatar's picture

31-01-2018, 14:03

rderooy wrote:

Were there also not some games on tape that had a mix of data and audio? How would you capture this, other then WAV obviously?

Never hear of that. Could you put some examples?

By montagut

Master (196)

montagut's picture

31-01-2018, 14:31

Tolvatar wrote:
rderooy wrote:

Were there also not some games on tape that had a mix of data and audio? How would you capture this, other then WAV obviously?

Never hear of that. Could you put some examples?

Deus ex machina:
https://www.youtube.com/watch?v=AJCS8TDhyro

By TomH

Champion (322)

TomH's picture

31-01-2018, 16:12

nataliapc wrote:

Ok... byte based blocks needs a format template assigned to be encoded in a MSX way.
In this case something like "0"b0b1b2b3b4b5b6b7"11" for byte level.
But is needed a bit level template too.

In each byte based block we need to include:
- Bit level template (using pulses)
- Byte level template (using bits)
- Byte stream

Defining templates in a flexible way, we could reproduce every data block of every platform, not only for MSX systems (remember that some european games use Spectrum data blocks like copy protection).

The pilot tone could be externalized from byte stream block (using other block) to not increase complexity.

That's exactly what I said, and the microcode version does; it provides the template for producing a 0, the template for producing a 1, then the template for producing bits from a byte. Except that I also included parity, because some micros use it.

Simplifying everything to nibbles, pseudo code for my proposal is something like:

Program {
    Program() {
    }

    Program(Program zero_program, Program one_program) : zero_program(zero_program), one_program(one_program) {
    }

    void step(byte) {
        void output_bit(value) {
            parity ^= value;
            if(value) one_program.perform(); else zero_program.perform();
        }

        switch(next_nibble()) {
            case 0: // output 0
                zero_program.perform();
            break;
            case 1: // output 1
                one_program.perform();
            break;
            case 2: // output and reset parity
                output_bit(parity); // the reset comes for free!
            break;
            case 3: // output bit n
                bit = next_nibble();
                output_bit((byte >> bit)&1);
            break;
            case 4: // wait
                triple = next_triplet(); // i.e. next three nibbles, compiled in little-endian format
                add_wait(triple);
            break;
            case 5: // toggle parity
                parity ^= 1;
            break;
            case 6: // toggle output
                add_crossing();
            break;
        }
    }

    void perform(byte) {
        pointer = 0;
        while(pointer < end) step(byte);
    }
};

void perform_data_block() {
    zero_program = read_program();
    one_program = read_program();
    byte_program = read_program(zero_program, one_program);

    while(bytes) {
        byte_program.perform(read_byte());
    }
}

void perform_repetition_block() {
    repeats = read_int();
    program = read_program();

    while(repeats--) program();
}

void perform_tape() {
    while(not end of file) {
        block_header = get_block_header();
        parity = 0;
        if(is data block)
            perform_data_block();
        else
            perform_repetition_block();
    }
}

I don't see how making the byte and bit formats declarative rather than programmatic would actually simplify in this case.

Off the top of my head, I am aware of the tape encodings and tape hardware of the:

  • Spectrum (/Amstrad/Enterprise/etc);
  • ZX80/81;
  • Electron/BBC;
  • Commodores;
  • Oric; and
  • MSX

That would cover everything that any of those machines can read with perfect fidelity.

EDIT2: to be absolutely explicit:

  • a data block would cover most if not all byte-oriented streams;
  • a series of changes that could not be encoded into a data block could be stored as a repetition block with a repetition count of 1;
  • lead-in tone could be encoded into a repetition block with a repetition count of several thousand;
  • limited range on WAIT isn't problematic: just put multiple of them in a row. If you're encoding a long gap between files, use a repetition block to avoid having to spell each out individually.

You're all smart people. I'm sure you realised that already. The main additional block I might consider would be sequence-of-bits, where a 0 program and a 1 program are specified, then n bits. No byte program, no requirement for a multiple-of-eight number of bits.

rderooy wrote:

Were there also not some games on tape that had a mix of data and audio? How would you capture this, other then WAV obviously?

The audio is intended for human ears, so MP3 would be smarter. But as a corollary, it's not intended for the computer to process or control, so one might argue it is out of scope — we're trying to preserve only the digital signal.

As well as your example, there are at least two albums off the top of my head that include ZX Spectrum software on them (including last year's OK Computer reissue), and some games that just had music on side two (Confuzion springs to mind, but it's not the only one).

By TomH

Champion (322)

TomH's picture

31-01-2018, 15:44

For fairness, pseudocode of the predictor route, assuming a Commodore TAP-esque byte-or-int sizing of errors:

int length_history[8];

void mode_predictor() {
    return mode_of(length_history);
}

void lookup_predictor(int distance) {
    return [] () {
        return length_history[distance];
    }
}

void output(int length) {
    length_history <<= 1; // pseudocode for: shuffle all array entries one place away from 0
    length_history[0] = length;
}

void output_segment(void (* predictor)) {
    length_history = {all 0s};
    while(data_remaining) {
        next_length = predictor();

        stream_value = get_byte();
        if(stream_value == 0x80) stream_value = get_int();

        // or possibly:
        /*
         stream_value = get_byte();
         if(stream_value & 0x80) stream_value = ((stream_value & 0x40) << 9) | ((stream_value & 0x7f) << 8) | get_byte();
         else stream_value |= (stream_value << 1) & 0x80; 
        */
        // if it makes more sense to offer 7-or-15-bit signed encoding, rather than either the range +-127 or else a full 32-bit integer.

        output(next_length + stream_value);
    }
}

void output_block() {
    switch(v = get_byte()) {
        case 0...7: predictor = lookup_predictor(v); break; // likely to be 0, 1 or 3 on an MSX
        case 8: predictor = mode_predictor(); break;
    }
    output_segment(predictor)
}

With other predictors being possible. I think we'd have to try a few to decide what makes sense, but then be very controlled about not expanding on them. Otherwise the risk is that we end up with dozens and all elegance is lost.

Probably specifying the sampling rate is also desirable, probably per block, as that reduces number size and therefore frequency with which a byte will fit the whole value. It'd likely be 44100 for something created from a WAV, 4800 for something created from a CAS, or any other value.

Then the only question for both formats is: what makes sense for back-end compression? Presumably gzip but subject to a limited sliding window?

(EDIT: and the mode predictor might need to be fuzzy, assuming we're not forcing exactly regular durations — e.g. any two values within n microseconds of each other are assigned to the same bin, the mode is the bin with the most things in it, that bin's value is the average of its contents)

By rderooy

Champion (486)

rderooy's picture

31-01-2018, 16:19

TomH wrote:
rderooy wrote:

Were there also not some games on tape that had a mix of data and audio? How would you capture this, other then WAV obviously?

The audio is intended for human ears, so MP3 would be smarter. But as a corollary, it's not intended for the computer to process or control, so one might argue it is out of scope — we're trying to preserve only the digital signal.

As well as your example, there are at least two albums off the top of my head that include ZX Spectrum software on them (including last year's OK Computer reissue), and some games that just had music on side two (Confuzion springs to mind, but it's not the only one).

Would be nice if this could be part of the image format, embedded in effect. That way emulators can also emulate that part of it. On the other hand, it would bloat the files significantly, and be too large for a real MSX still to process. So you would need some kind of tool to strip such data if you want to load the file on a real MSX.
Another way would be to add some metadata to the file to indicate there should be an external MP3 file (or files?), and when it should be used. This could be ignored when loading the file on a real MSX. The problem with this is that such external files are easily lost, but that is probably no different then giving users the ability to strip the MP3 from the file.

By TomH

Champion (322)

TomH's picture

31-01-2018, 17:16

Possibly it might be acceptable to allow a general outward "additional resource" link? Assuming a chunked format is uncontroversial (?), it would be a chunk and therefore positioned within the stream — an emulator that was sufficiently smart would therefore be able to present the additional resource at the proper moment.

If resource weren't typed then that would keep options open: start companion audio at the proper respective moment, offer a scan of the tape and/or inlay when the thing loads up, offer the text or a scan of the code sheets at the point that it prompts you to enter a code, etc.

Having them be typed within the cassette image itself isn't too terrible I think, if the chunk format is so regular that an emulator can skip them without knowing about them, but if it's a reference to an external file then doing so is duplicative and possibly reductive.

By nataliapc

Resident (53)

nataliapc's picture

05-02-2018, 09:54

I think that is important maintain the format in a raw and uncompressed way for easiest read/manipulation in old and/or low performance platforms.

The prediction method is cool but IMHO add an unnecessary complexity layer to the format. File size is not a critical factor due low program lengths and high mass storage sizes, even in old 8 bits computers.

Anyway the prediction+compression could be a format optional feature. But maybe this will force us to to create conversion tools to use the raw/no compressed format in the devices that not support it and leave the prediction+compression for storage/distribution purposes.

Right now I'm interested to discuss about microcode and how to represent the definitions of bits (using pulses) and bytes (using bits, start/stop bits, parity, etc...) in a flexible and configurable way.

I would like to hear too the opinion about all this (compression/raw, microcode, ...) of emulator owners/maintainers.

By TomH

Champion (322)

TomH's picture

05-02-2018, 15:07

I think my opinions as above don't need repetition but I just thought I'd add: I am also an emulator owner/maintainer. So add the things I've said above into that bracket.

Otherwise: being lifted entirely from the PNG file format, I don't think prediction is inherently a huge burden but that's very dependent on the predictors used. So it would warrant empirical exploration, I think.

If the decision were: raw encoding, no compression then a straight adaptation of CSW would be appropriate I think, just require that files be version 1.0. That's the one that supports only RLE compression, i.e. count of times between transitions, no further compression.

By nataliapc

Resident (53)

nataliapc's picture

10-02-2018, 00:53

Hi TomH, my intention wasn't undervalue your proposal, just to known other owner/maintainers opinions...

So, for a uncompressed file format I understand that you propose the use of a pulse based format (like CSW as you say), right?

Page 7/11
1 | 2 | 3 | 4 | 5 | 6 | | 8 | 9 | 10 | 11
My MSX profile