Gunzip for MSX

페이지 3/11
1 | 2 | | 4 | 5 | 6 | 7 | 8

By Louthrax

Prophet (2083)

Louthrax의 아바타

20-10-2015, 23:54

Grauw, have you tried 7zip with compression set to "Ultra" ? You can also set the "word size" to max value (258) for a small extra compression gain. Looks like 7zip beats the other compression tools you mentionned before. And all compression settings for .gz in 7zip are supported by gunzip so far :-)

By Grauw

Ascended (8457)

Grauw의 아바타

21-10-2015, 00:09

7zip will create smaller files for sure, it’s the major selling point of 7zip as compared to older compression formats like gzip and zip. I just tried with the Aleste 2 disk I used for my earlier tests, the resulting file is 206016 bytes.

However I don’t think 7zip is suitable for MSX for the reasons mentioned here:

Grauw wrote:

The problem with rar and 7zip is that the dictionary sizes are too big (megabytes / gigabytes). Effectively on MSX it means you have to decompress the entire file into memory before writing to disk. Whereas the dictionary size for gzip and zip is just 32K which will fit in main memory just fine without even dealing with memory mapping.

So although they achieve a little higher compression ratios, I don’t think rar and 7z are suitable for MSX due to the memory requirements. There’s no need for them either, as gzip and zip are still extremely popular formats with built-in support in nearly every modern desktop OS.

In other words, a 7zip decompressor for MSX would not be able to decompress files when your MSX does not have enough memory to contain the extracted file.

By Louthrax

Prophet (2083)

Louthrax의 아바타

21-10-2015, 00:10

I've been a bit too quick in my explanations: I'm talking about using 7zip to to compress in .gz (or .ZIP) format, not in.7z format.Have you tried that ? All options (compression level, word size), seems to work well with gunzip !

By Grauw

Ascended (8457)

Grauw의 아바타

21-10-2015, 00:17

Ah, that’s what you mean. I haven’t got 7zip installed (I don’t think it’s available for Mac), but I use Zopfli for compression which approaches the theoretical maximum of gzip, I doubt 7zip can beat that. Of course you can try it yourself :), you know which disk I used…

By Louthrax

Prophet (2083)

Louthrax의 아바타

21-10-2015, 00:17

Grauw, some dev news: I'm porting Gunzip from Glass assembler to Hitech C "ZAS" assembler (in order to integrate that in my C framework that handles zip headers, directories, file date, etc...).

Tricky things were to adapt the "assembly object oriented" macros you have to Zas, which is not as flexible from this point of view. Also, all "heap" functions are re-directed to malloc / free. What I really like with your code is that you managed to combine a modern "object oriented" approach with high-performance.

Some other notes:
.gz and .zip format are using exactly the same compression format (checked with a binary diff tool). Only the headers / footers surrounding the compressed data are different.
.gz does not support "STORE" compression method, but .zip does. This might be a nice solution if you just want to use .zip files to store long file names. I will support that (decompression for that format should be fast... I think Wink).

By Louthrax

Prophet (2083)

Louthrax의 아바타

21-10-2015, 00:19

Grauw wrote:

Ah, that’s what you mean. I haven’t got 7zip installed (I don’t think it’s available for Mac), but I use Zopfli for compression which approaches the theoretical maximum of gzip, I doubt 7zip can beat that. Of course you can try it yourself :), you know which disk I used…

Mmmm... if I remember my tests it's better by 3 or 4 KB for that disk (wow!).

By Grauw

Ascended (8457)

Grauw의 아바타

21-10-2015, 00:20

Hey Louthrax,

Louthrax wrote:

.gz does not support "STORE" compression method, but .zip does. This might be a nice solution if you just want to use .zip files to store long file names. I will support that (decompression for that format should be fast... I think Wink).

Deflate has 3 compression formats, formats 1 and 2 are Huffman-LZ with predefined or custom alphabet, but format 0 is pure uncompressed data.

By Louthrax

Prophet (2083)

Louthrax의 아바타

21-10-2015, 00:26

Ah, that option does not appear in 7zip GUI. Does gunzip already support that (uncompressed) format ?

By Grauw

Ascended (8457)

Grauw의 아바타

21-10-2015, 00:52

Louthrax wrote:

Ah, that option does not appear in 7zip GUI. Does gunzip already support that (uncompressed) format ?

Yeah, it’s the first format I added support for. It’s tested by the test file in res/test00.gz.

Louthrax wrote:

Tricky things were to adapt the "assembly object oriented" macros you have to Zas, which is not as flexible from this point of view. Also, all "heap" functions are re-directed to malloc / free.

Before I made Glass I was using Pasmo, and I used to write class definitions like this:

Alphabet:
Alphabet_root: equ $ - Alphabet
	dw 0
Alphabet_codeLengthCounts: equ $ - Alphabet
	ds Alphabet_MAX_CODELENGTH, 0
Alphabet_nextCodes: equ $ - Alphabet
	ds Alphabet_MAX_CODELENGTH * 2, 0
Alphabet__size: equ $ - Alphabet

Btw, I would not worry about performance around object instantiation, it’s not on the critical code path at all, so be as inefficient as you like. My Heap_Allocate is quite slow as well as it does lots of integrity checking.

Louthrax wrote:

What I really like with your code is that you managed to combine a modern "object oriented" approach with high-performance.

Thanks, I like it too Smile.

By Grauw

Ascended (8457)

Grauw의 아바타

21-10-2015, 00:38

By the way, I’m running out of ideas for further performance improvements so the current code base should be relatively safe to pick up, it probably won’t undergo any further major changes that affect performance.

So, SofaZip is coming? Smile

페이지 3/11
1 | 2 | | 4 | 5 | 6 | 7 | 8