3. Lexical syntax#

This page defines the characters of a source, and the names, numbers, string literals and other pieces of text that the parser reads whole. The next page, Grammar, defines how the parser puts these pieces together into expressions, forms and declarations. The definitions of this page and the rules of the next page are one grammar, written in the notation of the Lark parser and shown in pieces on the two pages. The whole grammar is also one file, mtp.lark.

This page specifies the language. The parser accepts exactly the text that this page allows, and rejects everything else.

3.1. How the grammar is written#

The grammar is a list of definitions, name: definition. A definition whose name is in capitals gives a piece of text that the parser reads whole, such as a name, a number or a string literal, with no whitespace or comment inside it. A definition whose name is in small letters is a rule of Grammar. In a definition:

  • "…" is the text between the quotes.

  • /…/ is a regular expression, in the notation of Python’s re module. [\s\S] is any character.

  • a b is a followed by b, a | b is either one, a? is a or nothing, a* is any number of a, a+ is one a or more, and ( ) groups.

  • (?!…) matches no text. It fails when the text that follows matches …. A definition that is only such a check, such as NOT_PART, is used only inside other definitions.

  • INT.2 gives the definition INT the priority 2. A definition with no number has the priority 0.

  • The tree of forms does not keep the text of a definition whose name starts with _.

Rule 3.1

At each place in a source, the parser reads only the text that the grammar allows at that place. Where several definitions in capitals could match there, the parser reads the one of the highest priority, and among those of the same priority, the one that matches the longest text.

This is how Lark reads the grammar with parser="lalr" and lexer="contextual". So truex is a mtp:NAME, and not mtp:TRUE followed by x. In the same way, circle_ matches mtp:HEAD only where a form may have its head (Rule 3.8).

3.2. Characters#

_BOM ::= /\ufeff/

Rule 3.2

A source is a sequence of Unicode characters. A file is read as UTF-8, and each \r\n and each \r alone becomes \n. A byte order mark (U+FEFF) at the very start of a source is ignored. The text has no other change.

A byte order mark at any other place, outside string literals and comments, is an error, because it is not whitespace (Rule 3.4) and not a character of a word (Rule 3.5).

Rule 3.3

Each expression that the parser produces has a position: the line and the column of its first character, counting from 1. A byte order mark at the start is not counted. A newline (line feed, U+000A) starts a new line. Every other character, a tab included, adds 1 to the column. An expression that the parser builds from a shorter spelling has the position given in Rule 4.8.

3.3. Whitespace and comments#

_whitespace_or_comment ::= (_WHITESPACE | _COMMENT)+
_WHITESPACE            ::= /[ \t\n]+/
_COMMENT               ::= ";" /[^\n]/*

Rule 3.4

Whitespace is the three characters space (U+0020), tab (U+0009) and newline (line feed, U+000A). A comment starts with ; and runs up to the next newline, or to the end of the source. Whitespace and comments have no meaning. Inside a string literal, ; and whitespace are text.

A source holds no carriage return, since Rule 3.2 turns each one into a newline. So U+00A0 (no-break space), U+2028 (line separator), a form feed, a vertical tab and the other characters whose Unicode general category is Zs, or whose bidirectional class is WS, B or S, are errors outside string literals and comments. Grammar gives the places where whitespace and comments may be written, and Rule 3.16 the places inside an interpolated string.

3.4. Where words end#

Rule 3.5

A word is a name, a number, an operator, a bool literal or none. From the first character of a word up to the next whitespace, the end of the source, or the next of the characters ( ) [ ] { } " | ; :, the text is exactly one word. Text that starts where a word may start and is not one word up to there is an error.

The characters ', $, #, @, ., / and ^ do not end a word, so a$'x', 1x and a'b are errors: none of them is one word. The grammar rejects them, as the parser does, because by Rule 4.3 only a form may follow a word with no whitespace between them.

Rule 3.6

The ) that closes a form is never followed directly by . or [: (sort xs).x and (sort xs)[0] are errors.

A part of the value of a form is read with the form (. X Y) (Rule 5.8), as (. (sort xs) 0).

RPAR     ::= ")" NOT_PART
NOT_PART ::= /(?![.\[])/

3.5. Names#

A name is a word that refers to something declared: theta, p.pos, show-tan. It is one or more name parts joined by dots. p.pos.x is the name of the three name parts p, pos and x.

NAME        ::= PARTS
PARTS       ::= PART ("." PART)*
PART        ::= NOT_LITERAL LETTER (LETTER | DIGIT | "-")* "?"?
NOT_LITERAL ::= /(?!(?:true|false|none)(?![A-Za-z0-9?-]))/
LETTER      ::= /[A-Za-z]/
DIGIT       ::= /[0-9]/

Rule 3.7

Each name part starts with a letter, one of the ASCII letters a to z and A to Z. Then come any number of letters, digits and hyphens (-). Last comes ?, or nothing. A name part is never exactly true, false or none.

So theta, show-tan, x2, a-, none?, the-true-value, truex, p.empty?, found?.x and ball.on-floor? are names. 2x, -x, a?b, x', a!b, a*b, a=b, a:b, θ, π and x² are not, and neither is any word that holds a character that is not ASCII. true.x, p.true, none.x, xs[0].none and none[0] are errors, since one of their name parts is true or none. Since no name holds ' or @, text that starts with tex' is a TeX string, and a name followed by @ starts a prefixed form, at the start of an expression (Rule 3.14, Rule 3.15). Both are errors in every other place, such as after | in a broadened type or after a . that follows brackets: a|tex'q', xs[0].tex'q' and a|r@1 are errors.

HEAD.1 ::= (PARTS | "none") "_"

Rule 3.8

An underscore may be written only at the very end of the head of a form, right after its last name part: (circle_ (r 1)), (marks.grid_ …), (line|none_ …) and (none_ a). Such a head matches mtp:HEAD.

So an underscore in any other place is an error: x_ as an argument, a_b, _x, _ alone, a_.b, a._, _.a and _[0]. An underscore at the end of the head of a form means that the form has no name (Rule 4.9).

3.6. Bools, none and the operators#

TRUE  ::= "true"
FALSE ::= "false"
NONE  ::= "none"

Rule 3.9

The word true or false is a bool literal. The word none is the literal none. Each of the words + - * / ^ < <= > >= = != . is an operator: a name with one name part, which is never followed directly by [.

So +[0] is an error. true[0], none[0] and 3[0] are errors too, since by Rule 4.3 only a form may follow a bool literal, none or a number directly. !=x is not a word.

3.7. Keywords#

Rule 3.10

A keyword is one of the words of Table 3.1, and no other word.

Each keyword is the head of a form of the language.

Table 3.1 The keywords, from the inventory of heads in Core and derived forms.#

Heads of

Keywords

Declarations of objects

scene, formula, namespace, palette, transition, animation

Declarations of shapes and containers

circle, rect, ellipse, line, triangle, polyline, polygon, curve, arc, ring, text, group, row, column

Declarations of components

canvas, readout, display, slider, button, play, stepper, toggle

Declarations of listeners

on-drag, on-click, on-release, on-dragstart, on-dragend, on-enter, on-leave, on-pointermove, on-show, on-hide, on-view, on-ready, on-live, on-quiet

Declarations of functions, types and choreos

func, colormap, type, interface, motif, variant, choreo

Other items of bodies

import, let, foreach, exists-when, term, doc

Expression forms

., name, if, when, cond, else, and, or, fn, type-of, as, free

Steps of an animation

move, wait, roll-dice, at-once, in-order

The heads of optional arguments, such as x, r, size, gap, step, fill, duration, like, defaults and loop, are not keywords, and neither are the heads of constraints, such as const, length and wrap. A word that is the head of an optional argument and also a keyword, such as animation or move, is a keyword.

Which names no declaration and no binding may take is defined in Rule 6.7.

Rule 3.11

Outside a string literal, {, } and " are errors, and so is a $ or a # that is not followed by '. Outside the format spec of an interpolated string (Rule 3.16), : is an error. String literals are written in single quotes only.

3.8. Numbers#

INT.2     ::= SIGN? DIGITS
REAL.3    ::= SIGN? ((DIGITS "." DIGITS? | "." DIGITS) EXPONENT? | DIGITS EXPONENT)
RATIO.4   ::= SIGN? DIGITS "/" NONZERO
IMAG.5    ::= SIGN? UREAL "i"
COMPLEX.6 ::= SIGN? UREAL SIGN UREAL "i"
UREAL     ::= (DIGITS "." DIGITS? | "." DIGITS | DIGITS) EXPONENT?
EXPONENT  ::= /[eE]/ SIGN? DIGITS
NONZERO   ::= /[0-9]*[1-9][0-9]*/
DIGITS    ::= /[0-9]+/
SIGN      ::= /[+-]/

Rule 3.12

A digit is one of the ASCII digits 0 to 9. A word that matches one of the number definitions is a number, of the kind of the definition:

  • mtp:INT is an int literal, of any size. 007 is 7 and +4 is 4.

  • mtp:RATIO is a ratio literal: the exact fraction of two ints, reduced, so -2/4 is -1/2. Only the numerator has a sign. A denominator of 0 is an error.

  • mtp:REAL is a real literal: the double-precision float nearest to the decimal number. A real literal has a decimal point or an exponent: 2.5, 1., .5, 1e3, 1.5E-3.

  • mtp:IMAG is a complex literal whose real part is 0, such as 2i and -1.5e3i.

  • mtp:COMPLEX is a complex literal with both parts, the real part first, such as 1+2i and -1e+2-3.5i. Each number in it is read as a real, so 1+2i is 1.0 + 2.0i.

The priorities of the grammar make a word the first of COMPLEX, IMAG, RATIO, REAL and INT that matches it whole.

So ٣ (ARABIC-INDIC DIGIT THREE) and ² are not digits. 1-2 is an error, since it is no number, and the difference of 2 and 1 is written (- 2 1).

3.9. Strings#

STRING ::= "'" (/[^'\\]/ | ESCAPE)* "'"
ESCAPE ::= /\\['\\n]/
TEX.1  ::= "tex'" (/[^'\\]/ | /\\[\s\S]/)* "'"

Rule 3.13

A string literal is written in single quotes. Inside it, \' is a quote, \\ is a backslash and \n is a line feed. A backslash followed by any other character is an error. Every other character, a line feed included, is itself.

Rule 3.14

A TeX string is written tex'…'. Inside it, \' is a quote, and every other character is itself: \frac, \\ and \q stay as written. A backslash and the character after it are read together, so in tex'a\\' the string is a\\ and the last quote closes it.

3.10. Prefixes#

PREFIX.1 ::= PARTS "@"

Rule 3.15

A name, @ and an item, written with nothing between them, are short for the form of that name and that item: H@X is (H X). H is a name that is not an operator and does not end with _. X is any expression that is not itself a prefixed form. @ may not be written in any other place.

So doc@'The dot.' is the docstring (doc 'The dot.'), free@green is the free starting value (free green), r@1 is (r 1), and (circle c r@1 fill@'red') is (circle c (r 1) (fill 'red')). free@(mix a b) is (free (mix a b)), r@(* 2 k) is (r (* 2 k)) and pos@[1 2] is (pos [1 2]). fill@free@green, +@1, circle_@c, r@ 1 and r @1 are errors. The form (H X) must be allowed where it is written, by the same rules as when it is written with parentheses: if@c is (if c), which is an error.

3.11. Interpolated strings and name strings#

DOLLAR_QUOTE ::= "$'"
HASH_QUOTE   ::= "#'"
QUOTE        ::= "'"
TEXT         ::= (/[^'\\{}]/ | ESCAPE | "{{" | "}}")+
FORMAT       ::= ":" /[^{}']*/

Rule 3.16

An interpolated string is written $'…', and a name string is written #'…'. Their text has the escapes of Rule 3.13, and in it {{ is { and }} is }. A { that is not doubled opens braces, which hold one expression, with whitespace or nothing before and after it, then a format spec or nothing, and last }. No comment may be written in the braces, around the expression. A format spec is : followed by any characters but {, } and '. A } that is not doubled and does not close braces is an error. A name string holds braces at least once.

The parser keeps the format spec as written. This page does not give its meaning. Since : ends a word (Rule 3.5), {x:.2f} holds the name x and the format spec .2f. A : in an expression nested in the expression in braces is an error, so {(f a:b)} is an error. A comment inside a form in the braces is allowed, since it is inside the form and not around the expression: $'{(f ; c\n x)}' holds the form (f x). $'{x ; c\n}' is an error.

The grammar has no rule that depends on how deep braces are nested, so braces are read in the same way at every depth. When a string literal nested in the expression in braces ends, the expression and the rest of the braces go on by the same rules as before the nested string literal. So $'{$'{a}'}' is the interpolated string whose braces hold the interpolated string $'{a}', and in $'{#'n-{k}'.x:y}' the braces hold #'n-{k}'.x and the format spec y.