UNAPI Tools - An update on some tools to allow better performance

Door ducasp

Master (151)

afbeelding van ducasp

26-07-2019, 01:39

Click here to see a video presentation about those tools update and more details
Hi,

Some of you might have met me before on the topics about MSX-SM, and for those that never met me, well, I'm the guy who worked on implementing the interface between ESP8266 and MSX-SM in FPGA as well as implementing an UNAPI driver for it and a customized firmware for the ESP8266 so it "talks" UNAPI making the driver more like a tunnel and very lightweight.

Some friends that were evaluating prototype versions of it or watching some videos I've showed wished the performance for some applications woul be better, so I've made an investigation on the bottlenecks. I won't bother with details here as the video itself and its description explains and shows it, but the end result is that I found that through code changes and some different approaches to writting transfer progress on the screen, I could improve performance of a few tools. The performance difference goes from good (TFTP) to very good (HGET) to really amazing (MSX HUB).

I really don't have Obsonet/Denyonet/GR8NET, but I believe those will benefit and get better performance as well from those tools, so, I've made them available at my github for download:

https://github.com/ducasp/MSX-Development/tree/master/UNAPI

Hope you enjoy it!

P.s.: for sure I've made pull requests for Konamiman (HGET/TFTP) and fr3nd (MSX HUB) of my changes. I've been bothering Konamiman a lot with tons of questions and my lack of github skills as well, so I would like to say a big THANK YOU to him, he is a remarkable person not only for his huge knowledge and contribution to the MSX scene, but also for being a fantastic person. B-)

Aangemeld of registreer om reacties te plaatsen

Van DamnedAngel

Master (143)

afbeelding van DamnedAngel

26-07-2019, 05:04

Hi ducasp,

I own a obsonet2 and am willing to to make any testing you may need during your development. Fell free to reach me. I will be very happy to help.

Best,

Van fr3nd

Expert (89)

afbeelding van fr3nd

26-07-2019, 05:37

Thank you for your contribution! I can't wait to test it. I'll check it as soon as possible Smile

Van ducasp

Master (151)

afbeelding van ducasp

26-07-2019, 05:38

Hi DamnedAngel, tools are available at the github link in my first post, so just go ahead and let me know if it improves performance for you, or, at least, if it doesn't get worse... :-)

Van ducasp

Master (151)

afbeelding van ducasp

26-07-2019, 10:28

fr3nd wrote:

Thank you for your contribution! I can't wait to test it. I'll check it as soon as possible Smile

No rush Smile I'm just hoping that the changes are as positive to other UNAPI adapters as they are for the ESP8266 built in the MSX SM. Wink

Van DamnedAngel

Master (143)

afbeelding van DamnedAngel

28-07-2019, 12:32

Hi,

Just tested in my setups with Obsonet2 and here are the results:

HGET:
--------
34% speed improvement.
Tested with http://www.httpvshttps.com/. Its index.html file is 37KB, and the download time dropped from 59s to 39s.

MSXHUB:
----------
39% speed improvement.
Tested with ianna.rom file from ianna package. It is 704KB, and the download time dropped from 422s to 256s.

TFTP and SNTP:
-------------------
Could not get them working, neither the originals nor the updated ones. I don't know exactly what is happening, but I suspect it might be some stupid configuration in my network. As soon as I get them running I will report.

Thanks for you work, ducasp!!!

Van ducasp

Master (151)

afbeelding van ducasp

28-07-2019, 18:39

DamnedAngel wrote:

Hi,

Just tested in my setups with Obsonet2 and here are the results:

HGET:
--------
34% speed improvement.
Tested with http://www.httpvshttps.com/. Its index.html file is 37KB, and the download time dropped from 59s to 39s.

MSXHUB:
----------
39% speed improvement.
Tested with ianna.rom file from ianna package. It is 704KB, and the download time dropped from 422s to 256s.

TFTP and SNTP:
-------------------
Could not get them working, neither the originals nor the updated ones. I don't know exactly what is happening, but I suspect it might be some stupid configuration in my network. As soon as I get them running I will report.

Thanks for you work, ducasp!!!

Thanks for testing DamnedAngel, glad to know that Obsonet is also getting a boost by using those tools.

P. s.: it is curious that both tools that didn't work are the ones that use UDP protocol...

Van Grauw

Ascended (8454)

afbeelding van Grauw

28-07-2019, 21:49

Nice video, nice improvements! So what if you update the text of TFTP not every kB, but every 16 kB? Does it not shave off another second or so, with no worse user experience?

In VGMPlay I would like to show some information during playback (like song position), however the DOS text I/O is just too slow, it would impact the timing of the music. So this feature needs to wait until I make a GUI, when I access the VRAM directly myself.

For not such timing sensitive tools like these of course showing progress is better than not, but even if a smoothly incrementing counter looks nice, DOS isn’t well suited to it and the user doesn’t really need such frequent status updates.

Van ducasp

Master (151)

afbeelding van ducasp

28-07-2019, 22:50

Grauw wrote:

Nice video, nice improvements! So what if you update the text of TFTP not every kB, but every 16 kB? Does it not shave off another second or so, with no worse user experience?

In VGMPlay I would like to show some information during playback (like song position), however the DOS text I/O is just too slow, it would impact the timing of the music. So this feature needs to wait until I make a GUI, when I access the VRAM directly myself.

For not such timing sensitive tools like these of course showing progress is better than not, but even if a smoothly incrementing counter looks nice, DOS isn’t well suited to it and the user doesn’t really need such frequent status updates.

Hi Grauw,

TFTP before: a full line of text and update every block the whole line, after receiving a packet it will send ACK back, call TCPIP_WAIT, when it returns wait until one vdp interrupt occur and JIFFY value changes before checking if there is any response

TFPT now: changed the routine that convert the received length to text to one that is not generic (so it is shorter, faster, do not use IX), just print the numbers (go back the last # of printed digits and print just the digits) and after sending ack and calling TCPIP_WAIT it won't wait JIFFY to change...

It could have changed to update the screen even less? Yes,it could, but I've compared results of this approach to not printing progress at all, and had negligible performance differences related to how it is now. My guess is that this is mostly due to TFPT not being a streaming protocol, needing to receive ACKs before sending the next packet, and the packets being 512 bytes long, so there is no buffering like HGET that uses TCP and streams data non stop. (pretty much the same thing with XMODEM, XMODEM 1K, Ymodem-G)

Perhaps adding the possibility to negotiate packet size and using larger blocks (my UNAPI can work with up to 2KB datagrams, not sure about obsonet, Denyonet and gr8net) would then show the need for more efficient progress update (and then I would choose something along the lines of what I've done in HGET, a progress bar that you will print a single character just every 4% received, and the algorithm to determine whether to print or not use just add operation and comparison).

Food for thought... oO