The BBC and Master Computer Public Domain Library

[&8E3E] The character is between 'A' and 'W', so check whether it is the start of a BASIC Keyword, as follows:

* Set pointer (&39, &3A) to point to &8456, which is the address of the beginning of the BASIC Keyword
table within the BASIC rom.

* [&8E46] Compare the character (in A) with the first charater of the next BASIC Keyword at (&39, &3A).

* If the character is less than that of the start character of the next BASIC keyword then no tokenisation is required
as the character belongs to a variable name and not a BASIC Keyword, so jump to &8E2A to skip the rest of the
characters in the variable name, store #&FF in location &3B (meaning that we are not at the start
of a statement), zero location &3C (as a line number is not expected) and jump back to &8DB2 to check the new
character (the next character after the variable name) for tokenisation.

* If the character (in A) is not equal to the first character of the next Keyword then check the next character in the
Keyword table the jump to &8E5D to advance to the next Keyword in the Keyword table.

* [&8E4E] Otherwise, increment the character index (Y - the position of the current character within the Keyword).

* If the next character is negative (>= #&80 - i.e. a Token value) then all the characters from the current
word on the command line (pointed to by (&37, &38)) match with the Keyword, so goto &8E84 to tokenise the keyword.
Otherwise, compare the next character of the Keyword with that of the next character on the Command Line. If the next character
matches then jump to &8E4E to check the next character.

* If the characters do not match and the character is a dot '.', then jump to &8E68 to advance (&39,&3A) to the next
token value and jump to &8E84 to tokenise (replace the Keyword on the command line with that token).

* [&8E5D] Advance (&39,&3A) to point to the token value (the byte that is >= #&80 and located
directly after the keyword).

* If the token value is &FE (WIDTH, the last token in the BASIC Keyword table) then we have reached the end of
the token table, so jump to &8E2A to skip the rest of the characters in the variable name, store #&FF in
location &3B (meaning that we are not at the start of a statement), zero location &3C (as a line number is not
expected) and jump back to &8DB2 to check the new character (the next character after the variable name) for
tokenisation.

* Otherwise, jump to &8E75 to advance to the next Keyword in the Keyword table and then
jump to &8E46 to compare the character with the next BASIC Keyword.

If bit 0 of the flag (meaning 'Don't tokenise if Keyword is followed by an alphabetic character') is set then:

Load the next character from the command prompt location (&37, &38) call &8D84 to check whether the character
is valid for a variable name (i.e. it's a digit, a letter or '_'). If it is a valid variable name character then do not
tokenise the Keyword (as, in this context, it is not a Keyword, but a variable name) and jump to &8E2A to skip the rest
of the characters in the variable name, store #&FF in location &3B (meaning that we are not at the start
of a statement), zero location &3C (as a line number is not expected) and jump back to &8DB2 to check the new
character (the next character after the variable name) for tokenisation.

If bit 6 of the &3D flag (meaning 'Pseudo Variable - where the keyword can be on either side of an assignment,
i.e. PAGE= and =PAGE') is set then:

If location &3B is 0 (we are at the start of a statement, i.e. 'PAGE=') add #&40 to the token value, as Pseudo
variable Keywords at the start of a statement are being assigned to, and so have a different token value.

If bit 1 of the Keyword flag (meaning 'Go into middle of statement mode' - i.e. Keywords IF & LET) is set then:

Set location &3B to value #&FF and zero byte &3C. This sets the tokenise routine to middle of statement
mode, and clears the 'Line Number expected' byte.

If bit 2 of the Keyword flag (meaning 'Go into Start of Statement mode' - i.e. Keywords THEN & FOR) is set then:

Clear location &3B (to tell the tokenise routine to go into start of statement mode) and clear location &3C
(as a line number is no longer expected).

If bit 3 of the Keyword flag (meaning 'The Keyword is FN or PROC' - so don't tokenise the subroutine name) is set then:

Push A (Flag) to the stack. Skip any alphabetic characters (including digits and '_' characters) on the program line
after the 'FN' or 'PROC' token and then, after the name has been skipped (a non-variable name character is found),
retrieve A back (the keyword flag) from the stack again.

If bit 4 of the Keyword flag (meaning 'Tokenise a Line Number next' - i.e. Keywords GOTO, GOSUB, ELSE, THEN) is set then:

Set location &3C to #&FF (i.e. tell the tokenise routine to expect a Line Number next - however, if no line
number is found next then this flag is ignored).

If bit 5 of the Keyword flag (meaning 'Don't tokenise the rest of the line' - i.e. Keywords REM and DATA) is set then:

exit from the tokenise line routine (so that the rest of the line is not tokenised) - and should instead be ignored
as no more keywords are valid on this line after a REM or DATA keyword.

Disassembly for the Tokenise Command Line routine

If character is not "." then check for digit (Line Number) (carry set if found)