I took a stab at installing the GNU tools referenced in the "Compiling openMSX" document. All was well until I got to MSYS, which does not run correctly on Vista x64. It errors during installation, and when launches it promptly spawns about 100 desktop windows and then closes them.
This mailinglist post about installing MSYS on W64 Vista describes that problem as well, including a possible solution.
I'm currently attempting to install MSYS + MinGW too on WinXP Pro 64bit, but so far without much luck.
So far I've tried:
MinGW-w64, which unfortunately lacks an installer..
Automated MinGW Installer - MinGW 5.1.4
MSYS Base System - MSYS-1.0.10.exe
However, when msys.bat tries to start rxvt.exe, it fails (doesn't open anything and just exists)
renaming rxvt so it will run sh.exe instead does work, however I can't run commands from that, as it generates a fork error:
Administrator@FIXATO /m/Applications/SVN/openmsx $ make F:\msys\1.0\bin\sh.exe: *** fork: can't reserve memory for stack 0x480000 - 0x68 0000, Win32 error 0 25143822 [main] sh 1948 sync_with_child: child 1244(0x2D0) died before initializ ation with status code 0x1 25143983 [main] sh 1948 sync_with_child: *** child state waiting for longjmp sh: fork: Resource temporarily unavailable
Which is probably due to the MSYS being a Win32 app I guess.
Update:
when trying to run rxvt manually from command prompt, I get the same kind of error:
F:\msys\1.0\bin>rxvt -sl 2500 -fg Black -bg White -sr -fn Courier-12 -tn msys -g
eometry 80x25 -e /bin/sh --login -i
F:\msys\1.0\bin\rxvt.exe: *** fork: can't reserve memory for stack 0x480000 - 0x
680000, Win32 error 0
0 [main] rxvt 1772 sync_with_child: child 1232(0x260) died before initiali
zation with status code 0x1
261 [main] rxvt 1772 sync_with_child: *** child state waiting for longjmp
rxvt: can't fork
rxvt: aborting