Documentation
¶
Overview ¶
Package parser defines generic LL(*) parser.
Index ¶
- Constants
- func RegisterHookLayer(name string, tpl HookLayerTemplate) error
- type HookLayer
- type HookLayerTemplate
- type Hooks
- type NodeContext
- type NodeHook
- type NodeHookInstance
- type NodeHooks
- type Option
- type ParseContext
- type ParseOption
- type Parser
- func (p *Parser) IsSideType(tt int) bool
- func (p *Parser) IsSpecialType(tt int) bool
- func (p *Parser) IsValidType(tt int) bool
- func (p *Parser) MakeToken(typeName string, content []byte) (*Token, error)
- func (p *Parser) Parse(ctx context.Context, q *source.Queue, hs Hooks, opts ...ParseOption) (result any, e error)
- func (p *Parser) ParseString(ctx context.Context, name, content string, hs Hooks, opts ...ParseOption) (result any, e error)
- func (p *Parser) TokenType(typeName string) (tokenType int, valid bool)
- func (p *Parser) Tokens() []grammar.Token
- type Token
- type TokenContext
- type TokenHook
- type TokenHooks
Constants ¶
const ( // expecting token, got end of input UnexpectedEoiError = llx.SyntaxErrors + iota // cannot match grammar rule for incoming token UnexpectedTokenError )
Syntax error codes used by parser:
const ( // trying to emit token of unknown type, a literal, or an error token EmitWrongTokenError = llx.ParserErrors + iota // token hook for unknown token type name UnknownTokenTypeError // node hook for unknown node UnknownNodeError // trying to register layer template again LayerRegisteredError // trying to use unknown layer template UnknownLayerError // non-side tokens left in source file(s) after parsing is done RemainingSourceError )
Other error codes used by parser:
const ( // AnyToken means any token type defined in grammar, not counting EoF nor EoI special tokens. AnyToken = "" // EofToken means special end-of-file token emitted after the end of source file. EofToken = lexer.EofTokenName // EoiToken means special end-of-input token emitted right after the EoF token of the last source file. EoiToken = lexer.EoiTokenName // end-of-input token )
Special token type names used by token hooks.
const AnyNode = ""
AnyNode denotes any node, used by node hooks.
Variables ¶
This section is empty.
Functions ¶
func RegisterHookLayer ¶
func RegisterHookLayer(name string, tpl HookLayerTemplate) error
RegisterHookLayer registers named hook layer template that can be referred by grammar. Returns ErrLayerRegistered if layer template with this name is already registered.
Types ¶
type HookLayer ¶
type HookLayer interface {
// Init is called for each layer when parsing context is created but before parsing is started.
Init(ctx context.Context, pc *ParseContext) Hooks
}
HookLayer is a token/node hook layer configured for specific grammar.
type HookLayerTemplate ¶
type HookLayerTemplate interface {
// Setup is called for every hook layer used by specific grammar.
Setup(commands []grammar.LayerCommand, p *Parser) (HookLayer, error)
}
HookLayerTemplate is used to create hook layers for different grammars.
type Hooks ¶
type Hooks struct {
// Tokens contains hooks for different token types. Key is either token type name or AnyToken constant.
// AnyToken hook is a fallback.
Tokens TokenHooks
// Literals contains hooks for tokens with specific content. Key is token content.
// These hooks have top priority (if token type allows matching against literals).
Literals TokenHooks
// Nodes contains hooks for nodes. Key is either node name or AnyNode constant.
// AnyNode hook is a fallback.
Nodes NodeHooks
}
Hooks contains all token and node hooks used in parsing process. Default action when no suitable token hook found is to use token as is.
type NodeContext ¶
type NodeContext struct {
// contains filtered or unexported fields
}
NodeContext contains methods that may be useful for node hooks.
func (*NodeContext) NodeStack ¶
func (nc *NodeContext) NodeStack() []string
NodeStack returns list of stacked node names. The first element is the current node being processed, and the last one is the root.
func (*NodeContext) ParseContext ¶
func (nc *NodeContext) ParseContext() *ParseContext
type NodeHook ¶
type NodeHook = func(ctx context.Context, node string, token *Token, nc *NodeContext) (NodeHookInstance, error)
NodeHook allows to perform actions on nodes emitted by parser. Called before new node is pushed. Receives node name and initial token.
type NodeHookInstance ¶
type NodeHookInstance interface {
// NewNode is called before a child node is pushed on stack.
// Receives child node name and its initial token.
NewNode(node string, token *Token) error
// HandleNode is called after nested node is finalized and dropped.
// Receives child node name.
// Receives result of closest nested hook EndNode() call or nil if none of nested nodes was hooked.
HandleNode(node string, result any) error
// HandleToken is called when a token belonging to current node is received.
HandleToken(token *Token) error
// EndNode is called before current node is finalized and dropped.
// result is passed to parent node hook or returned as a parse result if current node is the root.
EndNode() (result any, e error)
}
NodeHookInstance receives notifications for node being processed by parser.
type Option ¶
type Option func(*parserSettings)
Option is an optional argument for New
func WithLayerTemplates ¶
func WithLayerTemplates(templates map[string]HookLayerTemplate) Option
WithLayerTemplates defines hook layer templates that override registered ones.
type ParseContext ¶
type ParseContext struct {
// contains filtered or unexported fields
}
ParseContext contains data used in parsing process.
func (*ParseContext) Parser ¶
func (pc *ParseContext) Parser() *Parser
Parser returns parser used by this context.
func (*ParseContext) Sources ¶
func (pc *ParseContext) Sources() *source.Queue
Sources returns source queue used by this context.
type ParseOption ¶
type ParseOption func(*ParseContext)
ParseOption is an optional argument for Parser.Parse*
func WithFullSource ¶
func WithFullSource() ParseOption
WithFullSource instructs parser to check whether there are any non-side tokens left in source file after parsing is done. By default parser stops and returns result as soon as root node is finalized, no matter if end of source is reached or not. With this option parser returns error if there are any non-side tokens left.
func WithSides ¶
func WithSides() ParseOption
WithSides instructs parser to pass all tokens (include side ones) to node hooks. By default only non-side tokens are passed to node hooks.
type Parser ¶
type Parser struct {
// contains filtered or unexported fields
}
Parser holds prepared data for some grammar. Parser is immutable and reusable.
func New ¶
New constructs new parser for specific grammar. Grammar must not be changed after this function is called.
func (*Parser) IsSideType ¶
IsSideType returns true if given argument is a valid side token type.
func (*Parser) IsSpecialType ¶
IsSpecialType returns true if given argument is a valid special token type (EoF, EoI).
func (*Parser) IsValidType ¶
IsValidType returns true if given argument is a valid token type that can be used by parser.
func (*Parser) MakeToken ¶
MakeToken generates artificial token. Returns error if given typeName is not defined in grammar.
func (*Parser) Parse ¶
func (p *Parser) Parse(ctx context.Context, q *source.Queue, hs Hooks, opts ...ParseOption) (result any, e error)
Parse launches new parsing process with new ParseContext. result is the value returned by root node hook or nil if no node hooks used.
func (*Parser) ParseString ¶
func (p *Parser) ParseString(ctx context.Context, name, content string, hs Hooks, opts ...ParseOption) (result any, e error)
ParseString is same as Parse, except it creates source queue containing single source having provided content with provided name (name may be empty).
type TokenContext ¶
type TokenContext struct {
// contains filtered or unexported fields
}
TokenContext contains methods that may be useful for token hooks.
func (*TokenContext) ParseContext ¶
func (tc *TokenContext) ParseContext() *ParseContext
func (*TokenContext) PeekToken ¶
func (tc *TokenContext) PeekToken() (*Token, error)
PeekToken allows token hook to look ahead one or more tokens without affecting parser. Each call returns different token, starting with extra tokens generated by upper hook layers, followed by tokens fetched from sources. E.g. if upper layers generated extra tokens "foo" and "bar", and source contains tokens "baz" and "qux", then sequential calls of PeekToken() will return "foo", then "bar", then "baz", then "qux". Parse context is restored when token hook returns result, as if PeekToken was never called. Returns nil token and error when lexer returns error. NB: this feature interferes with ambiguous token fetching and token hooks: this method tries to fetch any token from any group, and fetched tokens are not handled by hooks.
type TokenHook ¶
type TokenHook = func(ctx context.Context, token *Token, tc *TokenContext) (emit bool, extra []*Token, e error)
TokenHook allows to perform additional actions when token is fetched from lexer, but before it is fed to parser, e.g. emit external $indent/$dedent tokens when text indentation changes, fetch complex lexemes (e.g. heredoc strings). emit flag set to true means that incoming token must be fed to parser, false means that it must be dropped. extra contains additional tokens that must be fed to parser after or instead of hooked one.
type TokenHooks ¶
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
Package layers contains subpackages defining standard hook layer templates.
|
Package layers contains subpackages defining standard hook layer templates. |
|
common
Package common contains definitions used by all standard hook layers.
|
Package common contains definitions used by all standard hook layers. |
|
convert
Package convert contains layer template to convert specific tokens to other tokens with different type and/or content.
|
Package convert contains layer template to convert specific tokens to other tokens with different type and/or content. |
|
indent
Package indent contains layer template to emit specific external tokens when source indentation level changes.
|
Package indent contains layer template to emit specific external tokens when source indentation level changes. |
|
restrict
Package restrict contains layer template to restrict specific nodes based on ancestor node types.
|
Package restrict contains layer template to restrict specific nodes based on ancestor node types. |