Tokenize
Tokenize is a cross-platform tool for turning text files containing untokenised BBC BASIC into tokenized ARM BASIC V files. It was written to be used in a cross-platform environment, but the version here is a standard RISC OS executable suitable for using when developing software natively.
Over and above the basic TEXTLOAD command, Tokenize offers a number of advantages:
- It offers more flexible crunching options than the
CRUNCHcommand. - It can bring files referenced by
LIBRARYcommands in-line, removing the references in the process. - It is able to convert SWI names from
SYScommands into numeric constant format. - It can replace ‘constant’ variables with buld-time values, so as to ‘bake in’ build dates and version numbers.
- It can perform a basic level of variable, function and procedure name checking, identifying potential typos in the code.
The version of Tokenize that is provided here is a standard RISC OS executable which can be used when developing software natively. If you’re looking for a version to use on Linux, you should see the Build Tools section.
In addition, the current trunk in the RISC OS.info repository can be built automatically for Linux using the native-tokenize package in the GCCSDK Autobuilder. This is the recommended way of getting a version suitable for use on the Linux platform.
In order to use Tokenize, you will require:
- a minimum of RISC OS 3.10, and
- the 32-bit Shared C Library (version 5.43 or later), which is a standard part of RISC OS 5 and can be obtained for other systems as part of the System Resources download from RISC OS Open Ltd.
