write out C code files which can be compiled to make a parser.
"2D support" means that indentation and line breaks can be significant.
+Indent tokens (IN and OUT) and end-of-line (EOL) tokens can be used to
+describe the grammar and the parser can selectively ignore these where
+they aren't relevant.
There are several distinct sections.
before the beginning. As we need to store some value, `TK_eof` is used
to mark the beginning of the file as well as the end.
+Indents (IN) are sometimes shifted and sometimes only accounted.
+Whatever decision is made must apply equally to the matching OUT. To
+ensure this we keep a stack of bits in `ignored_indents` and
+`indent_depth`. When we process an IN, we record if it was ignored.
+When we see an out, we pop of the relavant bit and use it to decide how
+to handle the OUT.
+
###### parser functions
struct parser {
void **asn_stack;
int stack_size;
int tos;
+
+ ## parser state
};
#### Shift and pop
So `shift` finds the next state. If that succeeds it extends the
allocations if needed and pushes all the information onto the stacks.
+An extra complication is added to `shift` by the `EOL` token. This
+token must be generated when a `NEWLINE` is seen, but an `EOL` is
+expected. When this happens, the `NEWLINE` is NOT consumed, so multiple
+EOL can appear before a NEWLINE. To indicate that the token was shifted
+by not consumed, `shift` can return the special value `2`. The token
+number for `EOL` cannot be statically declared, so when the parser
+starts we need to look through the array of non-terminals to find the
+EOL.
+
+###### parser state
+ int tk_eol;
+
+###### find eol
+ p.tk_eol = 0;
+ while (strcmp(non_term[p.tk_eol], "EOL") != 0)
+ p.tk_eol += 1;
+ p.tk_eol += TK_reserved + config->known_count;
+
+
###### parser functions
static int shift(struct parser *p,
const struct state states[])
{
struct frame next = {0};
+ int ret;
int newstate = p->tos
? search(&states[p->stack[p->tos-1].state],
sym)
: 0;
- if (newstate < 0)
+ if (newstate >= 0)
+ ret = 1;
+ else if (sym != TK_newline)
return 0;
+ else {
+ // have a NEWLINE, might need an EOL
+ sym = p->tk_eol;
+ newstate = p->tos
+ ? search(&states[p->stack[p->tos-1].state],
+ sym)
+ : 0;
+ if (newstate < 0)
+ return 0;
+ ret = 2;
+ asn = tok_copy(*(struct token*)asn);
+ }
+
if (p->tos >= p->stack_size) {
p->stack_size += 10;
p->stack = realloc(p->stack, p->stack_size
p->stack[p->tos] = next;
p->asn_stack[p->tos] = asn;
p->tos++;
- return 1;
+ return ret;
}
`pop` primarily moves the top of stack (`tos`) back down the required
### The heart of the parser.
Now we have the parser. For each token we might shift it, trigger a
-reduction, or start error handling. 2D tokens (IN, OUT, EOL) also need
-to be handled.
+reduction, or start error handling. 2D tokens (IN, OUT, NEWLINE, EOL)
+might also be ignored. Ignoring tokens is combined with shifting.
-We return whatever `asn` was returned by reducing production zero.
-
-When we find `TK_in` and `TK_out` tokens which report indents we need
-to handle them directly as the grammar cannot express what we want to
-do with them.
-
-`TK_in` tokens are easy: we simply update indent count in the top stack frame to
-record how many indents there are following the previous token.
-
-`TK_out` tokens must be canceled against an indent count
-within the stack. If we can reduce some symbols that are all since
-the most recent indent, then we do that first. If the minimum prefix
-of the current state then extends back before the most recent indent,
-that indent can be cancelled. If the minimum prefix is shorter then
-the indent had ended prematurely and we must start error handling, which
-is still a work-in-progress.
+###### parser vars
-`TK_newline` tokens are ignored unless the top stack frame records
-that they are permitted. In that case they will not be considered for
-shifting if it is possible to reduce some symbols that are all since
-the most recent start of line. This is how a newline forcibly
-terminates any line-like structure - we try to reduce down to at most
-one symbol for each line where newlines are allowed.
-A consequence of this is that a rule like
+ struct parser p = { 0 };
+ struct token *tk = NULL;
+ int accepted = 0;
-###### Example: newlines - broken
+###### heart of parser
- Newlines ->
- | NEWLINE Newlines
- IfStatement -> Newlines if ....
+ shift(&p, TK_eof, NULL, states);
+ while (!accepted && p.tos > 0) {
+ struct frame *tos = &p.stack[p.tos-1];
+ if (!tk)
+ tk = tok_copy(token_next(tokens));
+ parser_trace(trace, &p,
+ tk, states, non_term, config->known_count);
-cannot work, as the NEWLINE will never be shifted as the empty string
-will be reduced first. Optional sets of newlines need to be include
-in the thing that preceed:
+ ## try shift or ignore
+ ## try reduce
+ ## handle error
+ }
-###### Example: newlines - works
+Indents are ignored unless they can be shifted onto the stack
+immediately. The end of an indented section - the OUT token - is
+ignored precisely when the indent was ignored. To keep track of this we
+need a small stack of flags, which is easily stored as bits in an
+`unsigned long`. This will never overflow and the scanner only allows
+20 levels of indentation.
- If -> if
- | NEWLINE If
- IfStatement -> If ....
+###### parser state
+ unsigned long ignored_indents;
+ int indent_depth;
-Here the NEWLINE will be shifted because nothing can be reduced until
-the `if` is seen.
+NEWLINE/EOL is ignored when in an indented section of text which was not
+explicitly expected by the grammar. So if the most recent indent is
+ignored, so is any EOL token.
For other tokens, we shift the next token if that is possible, otherwise
we try to reduce a production.
###### try shift or ignore
- if (tk->num == TK_in) {
+ if ((tk->num == TK_newline || tk->num == TK_out) &&
+ (p.ignored_indents & (1 << p.indent_depth))) {
+ /* indented, so ignore OUT and NEWLINE */
+ if (tk->num == TK_out)
+ p.indent_depth -= 1;
free(tk);
tk = NULL;
- parser_trace_action(trace, "Record");
+ parser_trace_action(trace, "Ignore");
continue;
}
- if (tk->num == TK_out) {
- if (1) {
- // OK to cancel
- free(tk);
- tk = NULL;
- parser_trace_action(trace, "Cancel");
- continue;
- }
- // fall through to error handling as both SHIFT and REDUCE
- // will fail.
- }
- if (tk->num == TK_newline) {
- if (1) {
- free(tk);
- tk = NULL;
- parser_trace_action(trace, "Discard");
- continue;
+
+ switch (shift(&p, tk->num, tk, states)) {
+ case 1:
+ if (tk->num == TK_out)
+ p.indent_depth -= 1;
+ if (tk->num == TK_in) {
+ p.indent_depth += 1;
+ p.ignored_indents &= ~(1 << p.indent_depth);
}
- }
- if (shift(&p, tk->num, tk, states)) {
tk = NULL;
- parser_trace_action(trace, "Shift");
+ /* fallthrough */
+ case 2:
+ parser_trace_action(trace, tk ? "ShiftEOL" : "Shift");
## did shift
continue;
}
+ if (tk->num == TK_in) {
+ /* No indent expected here, so ignore IN */
+ free(tk);
+ tk = NULL;
+ p.indent_depth += 1;
+ p.ignored_indents |= (1 << p.indent_depth);
+ parser_trace_action(trace, "Ignore");
+ continue;
+ }
+
We have already discussed the bulk of the handling of a "reduce" action,
with the `pop()` and `shift()` functions doing much of the work. There
is a little more complexity needed to manage storage for the asn (Abstract
FILE *trace, const char *non_term[],
struct token_config *config)
{
- struct parser p = { 0 };
- struct token *tk = NULL;
- int accepted = 0;
## parser vars
- shift(&p, TK_eof, NULL, states);
- while (!accepted && p.tos > 0) {
- struct frame *tos = &p.stack[p.tos-1];
- if (!tk)
- tk = tok_copy(token_next(tokens));
- parser_trace(trace, &p,
- tk, states, non_term, config->known_count);
+ ## find eol
+
+ ## heart of parser
- ## try shift or ignore
- ## try reduce
- ## handle error
- }
free(tk);
pop(&p, p.tos, do_free);
free(p.asn_stack);
code / ocean / blobdiff