Assembler Optimizer

Enlighted (6935)

19-07-2020, 22:50

BTW I find this error (parsing SJASM 0.42c)

J:\Google Drive\hobby\mystuff\scrll8way>java -jar mdl.jar src\main128.asm -I src -po -asm+ mainopt.asm
ERROR: Cannot parse line src\main128.asm#6: macro _setbank4 value

where the first lines of the file main128.asm are


DEBUG	equ	0
DEVEL	equ	1


	macro _setbank4 value
	ld (_bank4),value
	ld (rbank4),value
	endmacro
	macro _resetbank4 
	ld a,(rbank4)
	ld (_bank4),a
	endmacro

    macro _setVdp register       	; without DI/EI
    out (0x99),a
    ld  a,register + 0x80
    out (0x99),a
    endmacro

Van santiontanon

Paragon (1805)

19-07-2020, 23:00

OH, interesting. I'll investigate! I don't see anything weird in that line at all. Must be a bug somewhere in the parser! Wil fix asap!

Van santiontanon

Paragon (1805)

23-07-2020, 07:49

I haven't posted an update for a few days, but a lot of progress is happening behind the scenes

Artrag shared some code with me that helped me improve support for sjasm. So, I'm quite confident more or less most of the functionality of sjasm is supported by MDL, which is nice!

What I am currently working on before releasing v0.7 is on debugging/robustness. I am taking each of the projects I use internally for testing (VGMPlay, Metal Gear, XSpelunker and Uridium), and making sure that the code automatically generated by MDL with all the optimizations already applied is compilable by a basic Z80 compiler and produces a working ROM (MDL's output removes all dialect specific syntax, and produces almost plain just official z80 syntax, with all macros resolved, etc.).

I already have this working for Metal Gear and XSpelunker. VGMPlay and Uridium use features of their respective assemblers (Glass and sjasm) that relocate assembler code ("sections" and "code" blocks respectively) rather than having it sequentially in the same order as in the source files. So, I need to mimmic that code relocation in MDL's parser (while making sure I keep annotations of where each source line originally came from (source file, line #, macro expansion, etc.) for when MDL is used in linter mode). Should be done by the end of the weekend I think.

Van santiontanon

Paragon (1805)

26-07-2020, 05:25

I just pushed a new release (v0.7), with LOTS of improvements! https://github.com/santiontanon/mdlz80optimizer/releases/tag...

Of course, more optimizations (and safer) are found. For example, in the projects I use as benchmarks (bytes / cycles saved):

MDL version        |   vgmplay  |  metal gear  |  xspelunker  |  uridium
-------------------------------------------------------------------------
alpha v0.5         |  43 / 188  |   53 / 260   |   21 / 116   |   - /  -
alpha v0.6         |  38 / 198  |   89 / 383   |   38 / 178   |  80 / 631
alpha v0.6 (size)  | 127 /  -   |  196 /  -    |  530 /  -    | 208 /  -
-------------------------------------------------------------------------
alpha v0.7         |  53 / 221  |   97 / 415   |   44 / 202   |  51 / 477
alpha v0.7 (size)  | 186 /  -   |  205 /  -    |  537 /  -    | 179 /  -

As you can see many more optimizations are generated (except in Uridium). This is because I found a few missing constraints, which resulted in unsafe optimizations (and that appeared in Uridium repeatedly). However, the extended set of patterns compensates.

The most exciting thing is that it's now robust enough as for applying optimizations automatically in many projects! For example, if you do:

java -jar mdl.jar MetalGear.asm -dialect sjasm -po -asm MetalGear-opt.asm
java -jar glass.jar MetalGear-opt.asm MetalGear-opt.rom

This produces a working MetalGear ROM, with all the optimizations already applied! If you want to produce size-oriented optimizations (that might make things slower), this can be done like this:

java -jar mdl.jar MetalGear.asm -dialect sjasm -po -asm MetalGear-opt.asm -popatterns data/pbo-patterns-size.txt

I still think MDL is more useful as a linter, but being able to generate .asm directly and verifying this results in a working ROM is very useful to me to verify optimizations are safe! (I had to fix many, many patterns before this worked :)). For now, out of the 4 projects above, this works for MetalGear, XSpelunker and Uridium. I still do not have this working in VGMPlay, as the "section" keyword of Glass is not yet fully supported (next version :) )

Also, notice that MetalGear was disassembled using sjasm syntax, however in the example above, it's compiled with Glass. That's because the code generated by MDL resolves all macros, dialect-specific syntax, etc. replaces unofficial Z80 ops by the official versions (e.g., "sub a,1" -> "sub 1") and produces a fairly plain .asm file (you could even add "-asm-expand-inbcin", if you want to replace "incbin" statements by "db" statements with the data directly added there, so the .asm file is self-contained). This works for any other dialect supported by MDL, so, it can also be used as a way to translate some code base to plain z80 syntax :) (this is done because the optimizer only knows basic z80 syntax, so, I convert the input to plain z80 syntax before passing it to the optimizer).

I think I might already jump to 1.0 in the next version and skip 0.8, and 0.9, as my goal is to add basic support for Pasmo and SDCC assembler, and with that I'd have covered the most common assembler dialects already, and have a fairly robust pattern-based optimizer! After that, I might spend some time testing it with more and more projects, and then it might be time to start thinking beyond the pattern-based optimizer, and see what other type of optimization technique could be interesting!

Van ARTRAG

Enlighted (6935)

26-07-2020, 07:17

This is amazing! Your MDL is becoming a sort of Rosetta stone among assembly dialects
I could use MDL directly as assembler

Van Giangiacomo Zaffini 2

Champion (286)

26-07-2020, 11:28

@ santiontanon - You're on fire! Please consider supporting GNU gas assembler syntax as well!

Van santiontanon

Paragon (1805)

26-07-2020, 18:29

Thanks! Yes, the "translation from one dialect to standard Z80 syntax" can be a handy feature!

And oh, I wasn't aware of "GNU gas", I'll make a note to check it out! Do you know if there are any open source MSX projects using GNU gas that I could check out for testing?

Van Giangiacomo Zaffini 2

Champion (286)

27-07-2020, 22:57

Unfortunately I'm not aware of fresh project using GNU assembler gas and GNU ld. So I'm trying a quick porting if I can succeed. Uhm.

Van Metalion

Paragon (1625)

28-07-2020, 18:55

Santianon, is there some reference somewhere regarding the inline commands ???
I just tried to use MDL for the first time and I've got lots and lots of errors ...

EDIT : Found the command arguments, OK

So, it does not produce any optimizations, because the parser fails before even going there :
1. MDL does not recognize 'jp s,xxxx' as a correct opcode, but it's official sjasm dialect
2. the parser fails on 'variable[multiple_spaces]=[space]value'
... and lots of other errors.

java -jar main.asm -asm+ annotated.txt

line:
VDP=9938

errors:
ERROR: expression failed to parse with token list: [=, 9938]
ERROR: parseMacroCall: Cannot parse line main.asm#7: VDP=9938

It fails also on this line in another one of my sources :

line:
macro	pset_x			; (30)

errors:
ERROR: parseMacroDefinition: Cannot parse line main.asm#15: macro	pset_x			; (30)

Basically, it does not work with any of my sources, because the parser fails on each of them.

Van santiontanon

Paragon (1805)

28-07-2020, 18:55

Hi Metalion, thanks for trying MDL! Indeed it is currently very rough (but getting more and more stable in each version hehe)

Quick question, did you add "-dialect sjasm" to the command line arguments? Otherwise, it's not going to recognize all the sjasm syntax. For a reference on the command line arguments, the readme in GitHub explains them, you you can just acll MDL without any arguments, like:

java -jar mdl.jar

And you will get the help screen.

But in any case, if with that it still doesn't work, is any of the projects you are trying it on online so I can try it? That would be great for me to help debug!