MSX Basic Tokenizer. Should I?

Страница 1/4
| 2 | 3 | 4

By farique

Rookie (23)

Аватар пользователя farique

30-07-2019, 01:51

So,
The newest version of Basic Dignified imposed certain constrains that maybe, perhaps, who knows, could make it easier for me to try to create a tokenizer in Python. It wold certainly speed up the build and run process.
I have Two questions:
1. Is there already a cross platform, command line tool for that?
2. Is there any interest on the community for something like that?
It is not something I can do right now (moving again) but I certainly can begin the research.
So?

Для того, чтобы оставить комментарий, необходимо регистрация или !login

By Latok

msx guru (3684)

Аватар пользователя Latok

30-07-2019, 08:03

I'd say if you like doing it, do it! Don't do it for others...

By Grauw

Ascended (8442)

Аватар пользователя Grauw

30-07-2019, 09:24

What Latok says is really the most important thing in my opinion Smile.

Although as an assembly programmer it's not something I would use, I can see how it would be interesting to research and implement, and I think it's definitely good to have a more structured and less restricted version of Basic which compiles down to minimized MSX-Basic. I would make sure it integrates well with Basic-kun as well.

By farique

Rookie (23)

Аватар пользователя farique

30-07-2019, 17:25

Oh yeah, sure, the challenge is the motivation; even I will not use it beyond some toying. If it already existed, however, the allure would be much smaller.
And the question about the community is to mesure the scales on letting it go or enduring when the problems arrive.
Thank for the feedback.

By brnomade

Supporter (2)

Аватар пользователя brnomade

30-07-2019, 20:45

This project implements a tokenizer but not in python. Maybe you can write a wrapper to reuse the existing implementation? Just as a suggestion.
https://launchpad.net/msxbas2asm

By PingPong

Prophet (3447)

Аватар пользователя PingPong

31-07-2019, 10:05

brnomade wrote:

This project implements a tokenizer but not in python. Maybe you can write a wrapper to reuse the existing implementation? Just as a suggestion.
https://launchpad.net/msxbas2asm

Wow! This is practically the core of a basic compiler. Can serve as a basis to create a good msxbasic compiler

By zPasi

Champion (471)

Аватар пользователя zPasi

31-07-2019, 19:37

That certainly looks promising.

I managed to compile that on my Mac. It required some tweaks:

On Makefile:
CFLAGS = -Wall -fexceptions -std=c++11
On main.cpp:
#include <sys/malloc.h>

By farique

Rookie (23)

Аватар пользователя farique

01-08-2019, 04:15

That looks amazing!
Shame it is way beyond my pay grade for any meaningful repurpose (not that I haven't put it on my reference list).
And anyway, where is the fun in making a wrapper :P

On a happier note, I managed to find some time and a quick proof of concept is already churning workable little pieces of tokenized code.

By farique

Rookie (23)

Аватар пользователя farique

02-08-2019, 20:18

Well, it was easier than I thought.
Of course there should be a ton of fringe (and not so fringe) cases not covered but it passed some tests.
No Basic-Kun or even Dignified integration yet, it is still very preliminary.

Here:
MSX-Basic-Tokenizer

By Manuel

Ascended (15744)

Аватар пользователя Manuel

02-08-2019, 20:31

Could be massively tested using a script that saves a basic program to ASCII and then tokenize it back and compare the results.

By farique

Rookie (23)

Аватар пользователя farique

02-08-2019, 20:49

Great idea.
Is there any easy source of varied and tested basic programs in good quantity I could download?

Страница 1/4
| 2 | 3 | 4