lex2.lexer

Components of lexer implementations.

Classes

BaseLexer	Abstract base class partially implementing ILexer.
GenericLexer	An generic implementation of ILexer.
ProfilerLexer	A wrapper around a lexer implementation to provide profiling functionality.

class lex2.lexer.BaseLexer

Bases: TextIO, ILexer, ABC

Abstract base class partially implementing ILexer.

abstract __init__()

AbstractLexer object instance initializer.

push_ruleset(ruleset)

Pushes a ruleset to the ruleset-stack.

Parameters

ruleset (RulesetType) –

pop_ruleset()

Pops a ruleset from the ruleset-stack.

clear_rulesets()

Clears all rulesets from the ruleset-stack.

get_options()

Gets the lexer options to define processing options of the lexer.

Return type

LexerOptions

set_options(options)

Sets the lexer options to define processing options of the lexer.

Parameters

options (LexerOptions) –

class lex2.lexer.ProfilerLexer

Bases: GenericLexer

A wrapper around a lexer implementation to provide profiling functionality.

abstract __init__()

AbstractLexer object instance initializer.

get_next_token()

Finds the next token in the textstream using the currently active ruleset.

Return type

Token

Raises

• UnknownTokenError – If an unknown token type has been encountered.

• EOF – If the lexer has reached the end of input data from a textstream.

close()

Closes and deletes textstream resources.

show_report(value_occurrence_threshold=10)

Prints a report of which rules (identifiers) occur the most.

This method is automatically called when it hasn’t been called manually yet and the textstream is closed (also done when the lexer is destroyed by the garbage collector).

After the report is printed, the profile will be cleared.

Parameters

value_occurrence_threshold (int, optional) –

Threshold to display the top most occurring values of a rule. A value lower than 1 disables the display of values entirely.

By default 10

class lex2.lexer.GenericLexer

Bases: BaseLexer

An generic implementation of ILexer.

abstract __init__()

AbstractLexer object instance initializer.

get_next_token()

Finds the next token in the textstream using the currently active ruleset.

Return type

Token

Raises

• UnknownTokenError – If an unknown token type has been encountered.

• EOF – If the lexer has reached the end of input data from a textstream.