Lexer Class

Inheritance diagram of Lexer

class Lexer(environment)

Class that implements a lexer for a given environment. Automatically created by the environment class, usually you don’t have to do that.

Note that the lexer is not automatically bound to an environment. Multiple environments can share the same lexer.

Methods

__init__(environment)
tokeniter(source, name[, filename, state]) This method tokenizes the text and returns the tokens in a generator.
tokenize(source[, name, filename, state]) Calls tokeniter + tokenize and wraps it in a token stream.
wrap(stream[, name, filename]) This is called with the stream as returned by tokenize and wraps

Descriptions

class Lexer

Method details

__init__(environment)
tokeniter(source, name, filename=None, state=None)

This method tokenizes the text and returns the tokens in a generator. Use this method if you just want to tokenize a template.

tokenize(source, name=None, filename=None, state=None)

Calls tokeniter + tokenize and wraps it in a token stream.

wrap(stream, name=None, filename=None)

This is called with the stream as returned by tokenize and wraps every token in a Token and converts the value.

Table Of Contents

Previous topic

Failure Class

Next topic

Token Class