IDE HDD via SMB very slow [solved]

Página 2/4
1 | | 3 | 4

Por ro

Scribe (4422)

Imagen del ro

04-01-2021, 13:43

ah, so disable reverse will shut down the hash creation?

set power off
set auto_enable_reverse off
reverse stop
set save_settings_on_exit on
set hda f:\msxharddisk2.dsk
set power on

that did it Smile

Okay, I know why and how. brill!
solved. thanx all!

Por Manuel

Ascended (17947)

Imagen del Manuel

04-01-2021, 22:35

During the 2 minutes, you do get updates on the calculation? It should print updates regularly....

Por Grauw

Ascended (9917)

Imagen del Grauw

05-01-2021, 01:42

Maybe openMSX does random access, or uses a small block size?

Por ro

Scribe (4422)

Imagen del ro

05-01-2021, 08:18

Manuel wrote:

During the 2 minutes, you do get updates on the calculation? It should print updates regularly....

Yes, looks like this

C:\Program Files\openMSX>openmsx.exe -machine Panasonic_FS-A1ST -ext ide -hda "U:/msx/emulated/MSXHardDisk2.dsk"
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 0%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 1%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 2%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 4%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 5%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 6%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 7%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 8%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 10%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 11%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 12%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 13%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 14%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 15%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 17%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 18%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 19%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 20%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 21%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 22%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 23%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 25%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 26%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 27%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 28%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 29%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 30%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 30%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 31%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 32%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 33%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 35%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 36%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 37%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 38%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 39%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 40%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 41%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 43%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 44%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 45%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 46%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 47%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 48%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 50%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 51%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 52%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 53%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 54%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 56%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 57%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 58%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 59%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 60%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 61%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 63%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 64%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 65%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 66%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 67%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 68%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 69%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 69%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 70%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 72%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 73%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 74%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 75%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 76%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 78%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 79%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 80%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 81%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 83%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 84%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 85%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 86%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 87%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 88%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 89%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 90%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 91%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 93%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 94%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 95%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 96%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 97%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 99%
progress: Calculating hash for U:/msx/emulated/MSXHardDisk2.dsk... 100%

Por Manuel

Ascended (17947)

Imagen del Manuel

05-01-2021, 09:27

OK, so why did you think it was hanging/crashing if you got progress updates? Just wondering...

Por ro

Scribe (4422)

Imagen del ro

05-01-2021, 11:18

Cuz I wasn't looking at the CLI screen Smile
I missed that "wait up, we are calculating stuff for ye. hang on bro!" message
meanwhile openMSX "waits" (Windows says "not responding"). So, that's why.

Por DrWh0

Paladin (820)

Imagen del DrWh0

05-01-2021, 12:57

Manuel, Cuz I just already have added the issue in github (I am affected too because I use big HDD/SD images too):

https://github.com/openMSX/openMSX/issues/1315#issue-778922332

Por Manuel

Ascended (17947)

Imagen del Manuel

05-01-2021, 19:11

That looks like a bug, thanks for reporting! You could try with the latest development build, see if that's better.

Por wouter_

Champion (456)

Imagen del wouter_

05-01-2021, 19:25

It was always the intention that these progress messages show up in the main openMSX screen. But unfortunately that got broken about a year ago (messages were still printed, but screen wasn't redrawn anymore). I fixed that bug, thanks for reporting.

Some more background information:
For harddisk images we don't calculate a "sha1" but a "Tiger Tree Hash" (TTH). The main advantage is that the latter can be recalculated incrementally. That is if you change a few sectors on the disk, the new hash value can be calculated nearly instantly instead of having to rehash the full file again.

To explain the difference of 20 seconds vs 2 minutes for the hash calculation:
Maybe calculating a TTH is slower than calculating sha1? I don't know.
Though more likely it's because we read the harddisk image in chunks of 512 bytes (the emulated sector size). I'll see if I can optimize this. Other optimization hints are also welcome of course.

Por ro

Scribe (4422)

Imagen del ro

05-01-2021, 19:30

Hai Wouter, thanx for chiming in.

Reading a file in 512bytes, on a modern file system with 4k of blocks sure slows down. Let alone using SMB of CIFS on that. You'd rather read a big chunk to memory and create a hash on 512bytes from there.

Hope you can optimize some Smile

Página 2/4
1 | | 3 | 4