Module

lexie

src/lexie.pooka

Teach a computer to read your language, one word at a time.

Lexie turns a token specification, written in TOML, into plain Pooka modules: a Token family, a lexer, source positions and, if you ask, a syntax highlighter. The generated code needs nothing but the standard library, so lexie is only ever a development dependency.

Most of the time you will not call this module yourself. Add lexie to a project and run it with pooka run -m lexie; the guide shows how. The functions here are for tools that want the files without the disk: generate checks a specification and returns the files to write, or every problem at once.

Pooka
import lexie
import lexie/config
import pooka/list

let assert Ok(files) = lexie.generate(config.default_config())
assert list.map(files, fn(f: lexie.GeneratedFile) { f.path }) == [
  "src/location.pooka",
  "src/token.pooka",
  "src/lexer.pooka"
]

API reference

class

GeneratedFile

src/lexie.pooka:43
class GeneratedFile(path: String, content: String)

A file to write: where it goes, relative to the project, and what it says.

field

path

src/lexie.pooka:43
path: String
INFERREDString
field

content

src/lexie.pooka:43
content: String
INFERREDString
constructor

new

src/lexie.pooka:43
INFERREDfn(String, String) -> lexie.GeneratedFile
function

generate

src/lexie.pooka:59
generate(c: Config) -> Result(List(GeneratedFile), List(ConfigError))

Check a specification and generate its files: location.pooka, token.pooka and lexer.pooka, plus highlighter.pooka when the highlighter is enabled. Nothing is written; that is up to the caller.

A specification with problems produces no files at all, and every problem comes back at once, so they can be fixed in one sitting.

Pooka
import lexie
import lexie/config

let result = lexie.generate(config.default_config())
// Ok(files), ready to write; no files have been touched yet.
function

read

src/lexie.pooka:102
read(text: String) -> Result(#(Config, List(ConfigError)), String)
INFERREDfn(String) -> Result(#(lexie/config.Config, List(lexie/config.ConfigError)), String)

Read standalone TOML text, retaining every configuration problem.

Pooka
import lexie

let text = "[keywords]\nitems = [\"let\"]\n"
let result = lexie.read(text)
// Ok(#(specification, problems)); generate validates before emitting code.
function

is_generated

src/lexie.pooka:118
is_generated(content: String) -> Bool
INFERREDfn(String) -> Bool

Is this file one lexie generated? Only those are ever overwritten or removed: lexie looks for its marker on the very first line.

Pooka
import lexie

assert lexie.is_generated("//// Code generated by lexie. DO NOT EDIT.\n")
assert !lexie.is_generated("module lexer\n")