Miscellaneous
This section contains some information on miscellaneous functionality of the library.
Profiling Order of Rules (Improving Performance)
Due to the design of the library, the lexer will try to match rules sequentially one at a time. In practice, this would mean a common rule that is put as last in a ruleset will cause slowdown compared to putting such a rule first.
The library includes another lexer built on top of the default lexer to provide some basic profiling on the input data, in order to more easily find an optimal order of rules.
The profiling lexer (ProfilerLexer) can be used by simply defining it as class to the make_lexer() factory function. The profile information is shown either by manually calling the show_report() method or by closing the lexer’s textstream (also done automatically when the lexer is destroyed by garbage collection).
lexer: lex2.ILexer = lex2.make_lexer(LEXER_T=lex2.lexer.ProfilerLexer)(ruleset, options)
...
# Manual call; you can set the occurrence threshold on showing values of a rule
lexer: lex2.lexer.ProfilerLexer
lexer.show_report(value_occurrence_threshold=10)
# Automatically calls .show_report() if it wasn't already called manually
lexer.close()
Note
If you wish to write the output of show_report() to a file, you can do so using redirection operators. See also this page on how to redirect output to a file.