TokenMatcher

Lexer-style longest-match tokenizer over a priority-ordered list of patterns.

Backed by a DFA obtained via Brzozowski-derivative subset construction over all patterns in parallel (a single automaton, not one-per-pattern): building it walks the derivative state space once (fromRegexes/fromSubsets), so matchAt itself is just array lookups - a binary search to classify the current code point into an alphabet partition, then an Int transition-table read - with no Regex/Subset allocation or derivation on the hot path. Ties broken by lowest priority index (i.e., the first pattern in the input list wins).

boundaries(i) is the first code point of partition i (boundaries(0) == 0); the partition itself spans [boundaries(i), boundaries(i + 1)), with boundaries.length acting as the exclusive upper partition for the last one. All derivative states share this same partition: deriving only ever discards Chars leaves (via smart-constructor normalization), never introduces new ones, so the alphabet fixed at the initial patterns already bounds every reachable state.

transitions is numStates * boundaries.length entries, row-major by state; -1 marks the (unrepresented) dead state, i.e. every pattern's derivative going empty.

accept(state) is the lowest pattern index nullable in that state at a position followed by more input, or -1 if none is; acceptAtEnd(state) is the same at the end of the input. They differ only for $/\Z/\z (see Regex.nullableBeforeMoreInput): a token ends wherever the next one starts, so those hold only when it's the last one. Any other lookahead would have to see the input after the token, so the builders reject it - see UnsupportedTokenConstruct.

State 0 starts a match past the start of the input, inputStartState one at offset 0. They differ only when a pattern uses ^/\A, which holds only at offset 0 (state 0 starts from the patterns with every ^/\A already failed, like any derivative). fastAscii is built from state 0, so it's skipped for a match at offset 0 whenever the two differ.

A pattern containing Regex.Group nodes (capturing/named groups) compiles fine - they're erased the same way Subset.of erases them for containment checks (fromRegexes/ fromSubsets go through Subset.of too) - but matchAt/findFirst never return capture spans; this DFA only ever tracks priority/end position. Wiring captures into the compiled automaton is separate, not-yet-done work.

Constructor kept private - the only place that ever builds one directly is compile below, in the companion object; @publicInBinary exists solely so given ToExpr[TokenMatcher]'s quoted call to it still resolves once that quote is spliced into a different compilation unit's generated bytecode (by tokenMatcher(...) below, or a user's own macro), despite being private at the source level everywhere else.