MSX demo: Utopia

By viejo_archivero

Paragon (1295)

29-04-2007, 10:24

Another IMPRESSIVE demo! you guys really know how to stretch MSX1 limits! congratulations for this awesome release

By Google

By Ivan

Scribe (8942)

29-04-2007, 11:01

Another impressive demo from dvik and Vincent van Dam!

Btw, dvik sent us a note saying that some screenshots included in utopia_v1.0.zip are not from the final release.

By hap

Paragon (1711)

29-04-2007, 11:09

Nice =)
Does it run as it should on the current blueMSX?

By KNM

Master (165)

29-04-2007, 12:21

WooooowwwWWW!!!!!INCREDIBLE DEMO!Dvik & Joyrex RULEZ!!

By ARTRAG

Prophet (3866)

29-04-2007, 15:40

Great demo!! greetings!
Strangely bluemsx does not show the last effect with the 3 girls, where open works fine.
On the contrary openmsx has some sync problems when displaying the 105 color images (that works much better on bluemsx)

By Maggoo

Paladin (834)

29-04-2007, 16:05

Great demo and great effects ! MSX1 still can surprise me
Congratulation !

By dvik

Prophet (2173)

29-04-2007, 18:40

The demo does indeed not run 100% on neither blueMSX or openMSX. The last part requires pixel accuracy in screen 0 and last release did not have it. since then I've added support for it and it will work fine in next version.
The images does not show that well in openMSX. It looks like some minor timing issue but I haven't investigated it that much. Also interlaced images or images that change from frame to frame usually show better in blueMSX than in openMSX. The reason is that blueMSX has a frequency conversion between the MSX frequency and the PC frequency. The improvment is minor but noticable in for example the interlaced images.

By dvik

Prophet (2173)

29-04-2007, 19:19

@Ivan: The screenshots are from the release but they shouldn't be part of the actual download if you want to host the demo. I just added them so you didn't have to do your own screenshots.

By hap

Paragon (1711)

29-04-2007, 19:26

When can users expect to see this if they don't have the necessary MSX hardware? <-- polite way of asking when the next version of blueMSX is due

Also interlaced images or images that change from frame to frame usually show better in blueMSX than in openMSX.That's an external issue. Setting the PC screen refresh rate at that of the emulated MSX (or a multiple of it), will fix it as I've just witnessed on my PC with openMSX.

By dvik

Prophet (2173)

29-04-2007, 19:34

polite way of asking when the next version of blueMSX is due

We are in the final stages preparing the release and it will most likely come in a week or two (unless something unexpected shows up).

Apart from the two 105 color images, openMSX shows the demo quite well. There is also a minor glitch in the end part where the top and bottom border is one pixel off sometimes.

By [WYZ]

Master (213)

29-04-2007, 19:59

What a good demo!. I enjoy a lot with this stuffs, VDP is up to limits and some PSG FX are great! (Fade out with envelopes...O_o)

it's a pitty, I couldn't watch it at Barcelona.

Congratulations Dvik & Joyrex.

By Ivan

Scribe (8942)

29-04-2007, 20:46

@Ivan: The screenshots are from the release but they shouldn't be part of the actual download if you want to host the demo. I just added them so you didn't have to do your own screenshots.

I don't remember seeing the picture of 'utopia_multicolor.png' in the demo... I will run it again.

By dvik

Prophet (2173)

29-04-2007, 23:19

I don't remember seeing the picture of 'utopia_multicolor.png' in the demo... I will run it again.

Its only sown if you run it in an MSX1 machine (or rather one with a MSX1 VDP)

By Latok

msx guru (3022)

29-04-2007, 23:23

I tried running the ROM-file on my turboR with loadrom 1.8. Didn't work

By dvik

Prophet (2173)

29-04-2007, 23:25

The demo doesn't work on TR or some MSX2+ machines, mainly because the TR is too slow. If you run the disk version I think you'll get a warning. Did it not run at all or did it stop after a while. Some parts runs ok on TR and some don't. I suppose I could make a TR version with only the parts that runs ok on TR.

By Latok

msx guru (3022)

29-04-2007, 23:34

I just used PUTDISK and loaded the DSK-file onto a diskette. Booting that diskette immediately gave a 'Direct Statement in File'

By dvik

Prophet (2173)

29-04-2007, 23:37

I can look at it but since the demo isn't intended to run on TR anyways I don't think its a big issue. You need to buy a real MSX to run the demo

By Manuel

Enlighted (7229)

29-04-2007, 23:37

Heh, the 105 colour pictures are not shown on an openMSX MSX2, but they were visible on my real MSX2 (with wobbly effect we talked about). Weird?

By Latok

msx guru (3022)

29-04-2007, 23:39

It also gives a direct statement in file on my philips nms8280

By dvik

Prophet (2173)

29-04-2007, 23:41

@manuel: That is actually a bit weird. I suppose the demo thinks its an MSX1 then. The reason why we wanted to disable that part is that its not running properly on MSX2.

@latok: Then its probably something wrong with the actual disk. We've tried on quite a few different MSXes and not seen this problem.

@latok (again): The disk is single sided. That may be the problem.

By Latok

msx guru (3022)

29-04-2007, 23:43

But why is everybody else able to watch the demo then? Do they all use emulators?

By wolf_

Ambassador (9115)

29-04-2007, 23:46

yes

By wolf_

Ambassador (9115)

29-04-2007, 23:47

scared?

By Manuel

Enlighted (7229)

29-04-2007, 23:48

I actually used the MegaFlashROM SCC (And openMSX as well, of course.)

By Latok

msx guru (3022)

29-04-2007, 23:50

Alright, now I tried booting the DSK-file with FDE.COM on my turboR and that works. At least....I get this warning you told me about dvik. That the demo won't run properly on my turboR. But that indicates the DSK-file isn't corrupt or something.

Maybe you're right the problem is the fact it's a single sided DSK-image. Probably PUTDISK.COM (I don't have another utility like that) can't handle single sided diskimages?

I also can't put the DSK-file on a diskette on a PC, I don't have a PC with a diskdrive anymore ^_^

Guess I'll just have to run the demo on an emulator then

By dvik

Prophet (2173)

29-04-2007, 23:53

Maybe someone can help you get a double sided disk. And if MRC hosts the demo they usually do an lzh packing as well with the individual files on the disk. But until then I guess you're stuck with the emulator

By Latok

msx guru (3022)

30-04-2007, 00:46

Alright, extracted all files from the DSK-image with Diskmanager 0.11. Then copied the files onto a CompactFlash Card and from there back onto a diskette. That worked

The demo is great to watch! And of course....I especially like the sample-part. Crystal clear sounds with nice visual effects! Great stuff!!!

By dvik

Prophet (2173)

30-04-2007, 07:36

@Ivan: I suppose its a good idea to make a 720kB dsk image for the download database to avoid the issue Latok had.

By spl

Paragon (1230)

30-04-2007, 12:44

I saw this demo at last 31 BCN RU and I was very impressed! I like these kind of demos! Also... if they do this on MSX 1.... what they will do on a MSX 2? o_O

By wolf_

Ambassador (9115)

30-04-2007, 13:41

As long as things stay tiles (e.g. screen 2 or 4) then: not much more, other than palette and more sprites. Well, scrolling perhaps, but that's an aged -thus boring -demo-technique.

By Manuel

Enlighted (7229)

30-04-2007, 20:40

Daniel and Vincent: you two need name for your cooperation. What about DaVinDa? (Daniel Vik & Vincent van Dam. Or ViDaDaVi, ViDaVi, DaViDa, DaViViDa, whatever. Or something that sounds a lot cooler and more original than this crap

By wolf_

Ambassador (9115)

30-04-2007, 21:15

Davilex ^_^

or just "D&V" .. like T&E

By Vincent van Dam

Champion (512)

30-04-2007, 22:04

Davirex

I kinda like the 'dvik & joyrex' logo

By poke-1,170

Paragon (1605)

30-04-2007, 22:09

Impressive stuff indeed ! I suppose it would look even better on a real tv (instead of on en emulator), with the flickering extra colour effects.
The stuff with the eyes and the way it's displayed looks very charming, is it made with characters or something ?
nice music as well, including the filtered tone in the loading parts

By ARTRAG

Prophet (3866)

01-05-2007, 09:09

D&V:
Is it possible to have a short text that explains the key point of all the parts of the demo?
It would be very interesting for many of us and would allow to maintain and preserve
your knowhow in the time.

By dvik

Prophet (2173)

02-05-2007, 01:10

Here is a draft of the spoler: www.bluemsx.com/demos/utopia.txt

If you miss some info, let us know and we'll try to add it.

By Huey

Prophet (2131)

02-05-2007, 09:01

Grrrrrrreat!

By ARTRAG

Prophet (3866)

02-05-2007, 12:24

Great!!! Thanks!! The Utopia spoiler could get as much accesses as the demo

By ARTRAG

Prophet (3866)

02-05-2007, 12:31

Big Eyes
========

The graphics in this part is basically a simple 2-D transformation effect doing
offset, rotation and scaling. The part uses screen 2 characters with dithered
patterns to simulate different color tones. The source image is a 256x128 pixel
image with 8 bit color depth.

The transormation matrix is


  | x' |     | x0 |   | dx  -dy | | x |
  |    |  =  |    | + |         | |   |
  | y' |     | y0 |   | dy   dx | | y |

To make the math easy on the Z80, fix point arithmetics is used. 16 bit
registers are used in the calculation and the high 8 bits are the integer part.

Each source pixel (x, y) represents one character on the MSX screen. The
character shown depends on the pixel in the source image closest to (x', y').

Each frame in the animation is defined by four values. x0, y0 is the top left
corner and dx, dy which are the transformation parameters. So each frame, the
engine reads the transformation data from a table and renders the output image.

Do you use vector quantization for defining the characters you display?
Do you define tables to associate pixel (x, y) to each character ? Which metric do you use ?
I see you use the same character set for all the scales:
Do you use different tables to associate pixel (x, y) to each character depending on the scale ?

By Prodatron

Paragon (1214)

02-05-2007, 19:38

Really impressive, what an MSX1 can do! Great work!

By dvik

Prophet (2173)

02-05-2007, 19:55

Do you define tables to associate pixel (x, y) to each character ?

I use a 256x128 bitmap with 8bit pixels, so yes its a table.

Each pixel represent an MSX character. The character is made up by two colors and a pattern to get as close
as possible to the color of the source bitmap.

Do you use different tables to associate pixel (x, y) to each character depending on the scale ?
No its only one table and its quite big, 32kB.

By ARTRAG

Prophet (3866)

02-05-2007, 23:43

I see, each point in the 256x128 map is a char.

The character is made up by two colors and a pattern to get as close
as possible to the color of the source bitmap

how do you define the characters? do you use some specific algorithm ?

By dvik

Prophet (2173)

03-05-2007, 00:45

how do you define the characters? do you use some specific algorithm ?

Its a quite simple algoritm. From a given RGB value, it picks the two closest colors and calculates the percentage of each color. Then it generates the pattern by starting at the top left pixel with the bg color, and to set the pixels in the pattern it does sortof linear interpolation of a step signal that is 0 for time 0-x and 1 for x-1 where x is the percentage of the background color.

By ARTRAG

Prophet (3866)

03-05-2007, 23:22

Why do you blend only 2 colors?
in theory you could blend up to 16 colors in the same tile...
this could lead to better results without almost any effort

By dvik

Prophet (2173)

04-05-2007, 00:24

Thats true. You'll get quite a few colors with the current method as well. You can combine 15 colors in 105 ways. Then you have 256 ways of mixing each pair of colors. So you'll get 105*256 different colors which is quite a lot, 26880 to be exact. And as a bonus you get interesting patterns (at least some times).

By poke-1,170

Paragon (1605)

04-05-2007, 00:59

would you be able to convert videos in such a way too ? I must say the patterns add a nice touch to it.

By dvik

Prophet (2173)

04-05-2007, 01:32

Its definately possible. The question is how often you need to update the color table though. The part in this demo uses a 256 color palette (the number of possible characters) and if you would do a video you probably need to update the palette every now and then since a 256 color palette may not be sufficient for an entire video.

An option would be do do the part in screen 1 and create the most compatible palette based on the color combinations you can get in screen 1. The benefit is that you could update the palette every frame without much cost.

I think that wolf_'s patterns he use in Sphere may work better for video though, mainly because I think they give higher resolution (It looks like sphere resolution is 64x48 or 64x24, correct me if I'm wrong).

By ARTRAG

Prophet (3866)

04-05-2007, 13:47

About reproducing videos (or even the rotation of pre-defined objects - it can be considered a video as well)
many well known methods exist in order to find the best definition of the tile set for a given video sequence.

Assume you have a video sequence of 100 frames 256x192, 8 bit per pixel.
Assume we use the s.c. hybrid mode, i.e. screen 2 with only one tileset (or the same tile set replicated 3 times, it is the same for now)

Split the frames in 8x8 blocks: you get 76800 vectors of 64 bytes (i.e. 8x8 = 64 )
This is usually called "test set".

You need to find the best 256 tile set to represent the video.

This is the set of 256 64 bytes vectors that, accordingly some criteria, best approximate the 76800 test set.
This is usually called "code set".

The additional complexity is that the code set is chosen among the tiles that can be represented by the VDP (“valid” tiles).

Without this last constrain, the easiest way to define the code set is to use an algorithm like
K-means http://en.wikipedia.org/wiki/K-means_algorithm

The steps are:
1) start with a random code set, (256 random vectors of 64 bytes)
2) for each code vector, find what are the test vectors that should be represented by that code vector: call it cluster (you get 256 clusters in our case)
3) define the centroid of each cluster you have found above (it is the mean of the cluster)
4) replace the code set with the centroids of the clusters (you get 256 new code vectors)
5) repeat from 2) till convergence (i.e. when centroids does not move any more)

Adding the constrain on the fact that not all the vectors can be centroids (i.e. not all the tiles can be displayed by the VDP due to color clash), the K-means could be restated like this:

1) start with a random code set, among those that can be displayed by the VDP
2) for each code vector, find what are the test vectors that should be represented by that code vector: call it cluster (you get 256 clusters in our case);
3) define the centroid of each cluster you have found above (mean of the cluster);
4) convert to screen 2 the centroids, i.e. find the best valid tile that approximate each centroid (any existing conversion strategy can be used here)
5) replace the code set with the converted centroids of the clusters (you get 256 new code vectors)
6) repeat from 2) till convergence (i.e. when centroids does not move any more)

Actually MSX1 could do even better:

Split vertically the video in 3 separate videos accordingly the 3 tile sets available on the MSX1 and do the process above for each part of it. You get 3 different tile sets each one fitted on the part of video that should be displayed in its area.

IMHO all this is doable and is the best that can be done on msx1

Naturally quality improves when the code vectors are fitted on sequences of fewer frames. This implies that instead of changing the tileset each 100 frames you need to change it each 50 frames or less. As always it is a trade off between video quality and z80 resources.

By wolf_

Ambassador (9115)

04-05-2007, 14:37

64x48 for Sphere and Wings. However, the diff with for example screen 3 is that our chars are not just setup like:
00001111
00001111
00001111
00001111
22223333
22223333
22223333
22223333

but more like

00001111
00110011
00001111
22112211
11331133
22332233
22223333
22332233

or sumfin', dunno exactly. Anyway, point is that it'll look a little less blocky that way, more like mpeg blocks.

By ARTRAG

Prophet (3866)

04-05-2007, 16:04

A small update about the feasibility of my proposal.

I did some experiments in matlab rigth now,

Standard K-means on a test set of 7680 vectors (10 frames of 256x192 at 8 bit per pixel) - random noise at the moment - needs about 51 minutes to extract the 256 optimal tiles (on a AMD 3400+).

I know, having 10 frames per sec, with my 7680 vectors I optimized the tileset of 1 sec of video only, never the less everithing seems doable...

I'm sure taht the k-means I proposed is more complex than the standard, but you need to take into account that C can be much more efficent than matlab (as in matlab all the data are represented as double, while here we deal with bytes and ints)

DVIK what do you think ?

By ARTRAG

Prophet (3866)

04-05-2007, 20:24

There are only 4,73304E+35 possible values for a 8x8 screen 2 tile.

Someone proposing exhaustive search?

By dvik

Prophet (2173)

04-05-2007, 23:22

I've used vector optimizations in the past, mainly to do compression of images. Never really though of applying it on MSX though.

I cheated and had Photoshop do the color optimization. Just convert the 24bit color image to 256 indexed colors in PS and you're almost there. Just need to create the optimal partern and color match for the 256 indexed colors. Would this method be significantly less good than doing an MSX optimization from the 24 bit color source image?

By ARTRAG

Prophet (3866)

05-05-2007, 01:18

We do not speak of the same thing. You associate a pint in the image to a char on the screen.

I speak of a general method to reproduce videos.
In my proposal the source video has frames of 256x192 points (and 8 bit per pixel) and the proposed vector quantization does a true compression, as at each block of 8x8 bytes is associate only one byte (the number of the tile in screen 2)

The algorithm I proposed is able to define the codebook finding the best 256 tiles of screen2 in order to reproduce the video.
This is tile set, or the 3 tile sets, to be used in screen 2.

Once you have the codebook (i.e. the tile sets), you store only the char number instead of the 8x8 blocks.

This saves a lot of room in rom.
The final video has a compression rate of 0.1250 bits per pixel - very poor for color images - I know, but still something.

In theory the process is able to reproduce any video with resolution 256x192.
In practice I guess that reproducing videos with lower resolution - but expanded at 256x192 - should lead to good results in terms of codebook generation.
(I mean that if you start from a video that has few bit per pixel, the compressed version at 0.125 bit per pixel will be very close to the original)

1 min of 256x192 video at 10fps could fit in about 450Kbytes of ROM (tiles apart)

By ARTRAG

Prophet (3866)

05-05-2007, 15:43

I'll try some quick and dirty development in matlab...
ready for VDPENC ?

By poke-1,170

Paragon (1605)

06-05-2007, 22:44

hmmm, what if you stick to indexed colour, and have a fixed palette ?
when you use monochrome colours 16 is often enough.
it's graphically an appealing effect I must say, done with the patterns
used in the eye rotation part.
so it that all calculated realtime ? Would be an interesting video coded
sort of thing hehe

By poke-1,170

Paragon (1605)

06-05-2007, 22:45

now that I think of it, didn't bandwagon do some video in monochrome (or 2 colours even)
video in some of their demos ?

By NYYRIKKI

Prophet (2771)

06-05-2007, 23:17

Dvik & ARTRAG, check out this idea that I wrote with you in mind:
http://www.msx.org/forumtopicl7311.html
:D

By dvik

Prophet (2173)

06-05-2007, 23:41

bandwagon did indeed a short video in one of their demos. Its a 64x48 pixel video made in screen 1 or screen 2. The colors are black and blue and there are four different intensities per pixel, so two bits per pixel. Similar patterns to the big scroller in the utopia demo. The tunnel effect also has two bits per pixel color depth but instead of using different patterns, its using different colors.

By ARTRAG

Prophet (3866)

07-05-2007, 00:07

Some preliminary results at
http://ragozini.googlepages.com/vdpenc

The VDP now is only in cascade to kmeans, but WIP to bring it into the loop!
stay tuned...

By dvik

Prophet (2173)

08-05-2007, 01:43

@Ivan, wolf_ or snout: I know you may be busy with other things but I just wanted to ask if you still want to host the demo?

By wolf_

Ambassador (9115)

08-05-2007, 02:12

From those 3, only snout can access the MRC-FTP, and he's far from motivated recently. Bear with him/us.

By dvik

Prophet (2173)

08-05-2007, 17:58

I saw you mentioned something similar in another thread. I hope he is ok and its nothing serious. I'll keep the temporary location as long as needed, no problem.

By dvik

Prophet (2173)

13-05-2007, 01:18

Emu users: Now you can download blueMSX2.7 and it runs the entire demo pretty well. Use a European MSX1 machine for best result (or rather to show all parts and effects).

By wolf_

Ambassador (9115)

13-05-2007, 01:37

I hope he is ok and its nothing serious.

Euhm, well, serious enough to have been out of the msx-running for months, and prolly for a while to come..

By Yukio

Paragon (1514)

23-10-2007, 23:37

Used BlueMSX version 2.4 for this.
Later will download a newer version of BlueMSX.
The demonstration is very nice ... Sounding astonishing!
Very cool text scrolls.

By dvik

Prophet (2173)

24-10-2007, 21:25

use bluemsx 2.7 for best result and run the demo in a european MSX1 machine (e.g. the default MSX1 machine).

By Yukio

Paragon (1514)

24-10-2007, 23:40

I completed the download of BlueMSX version 2.7 generic.
The connection and services over here are troubleshot and the server does not support GetRight. Because of it I (at this time) can't get the SS2 version 2.7. Am I missing something for not use the SS2 extensions ?

By dvik

Prophet (2173)

26-10-2007, 09:02

earlier versions of bluemsx had different builds for non-SSE, SSE1 and SSE2 but 2.7 is the same build regardless of what you try to download. There are some hand made SSE optimizations in the build but these are automatically detected.

Share

Related content

Comments (68)