TokenStream Class

Inheritance diagram of TokenStream

class TokenStream(generator, name, filename)

A token stream is an iterable that yields Tokens. The parser however does not iterate over it but calls next() to go one token ahead. The current active token is stored as current.

Methods

__init__(generator, name, filename)
close() Close the stream.
expect(expr) Expect a given token type and return it.
look() Look at the next token.
next() Go one token ahead and return the old one
next_if(expr) Perform the token test and return the token if it matched.
push(token) Push a token back to the stream.
skip([n]) Got n tokens ahead.
skip_if(expr) Like next_if() but only returns True or False.

Attributes

eos Are we at the end of the stream?

Descriptions

class TokenStream

Method details

__init__(generator, name, filename)
close()

Close the stream.

expect(expr)

Expect a given token type and return it. This accepts the same argument as jinja2.lexer.Token.test().

look()

Look at the next token.

next()

Go one token ahead and return the old one

next_if(expr)

Perform the token test and return the token if it matched. Otherwise the return value is None.

push(token)

Push a token back to the stream.

skip(n=1)

Got n tokens ahead.

skip_if(expr)

Like next_if() but only returns True or False.

Attribute details

eos

Are we at the end of the stream?

Table Of Contents

Previous topic

Token Class

Next topic

TokenStreamIterator Class