Struct ketos::lexer::Lexer

pub struct Lexer<'lex> { /* fields omitted */ }

Produces Tokens from an input string.

Methods

impl<'lex> Lexer<'lex>

fn new(input: &str, offset: BytePos) -> Lexer

Creates a new Lexer to read tokens from the input string.

fn next_token(&mut self) -> Result<(Span, Token<'lex>), ParseError>

Scans the input stream for the next token, returning the token and the span of input text from which it was scanned.

fn skip_shebang(&mut self)

Skips over a shebang line at the start of input. This is used when parsing files which, on Unix systems, may use a line consisting of #! followed by a path to the interpreter.

Panics

Panics if any tokens have already been scanned from the input stream.

fn current_position(&self) -> BytePos

Returns the current position within the input string.

The next call to next_token will begin searching at this point.