NeilBrown [Sun, 4 May 2014 10:31:05 +0000 (20:31 +1000)]
parsergen: track when newline is permitted, and discard if not.
A newline is only permitted (as a recognised symbol) if we are
parsing a non-indented line-like segment.
If we have seen an internal indent since the last line-like start,
newline tokens should be ignored.
NeilBrown [Sat, 3 May 2014 21:55:03 +0000 (07:55 +1000)]
parsergen: compute "can_eol" for each symbol.
A symbol is "can_eol" if it can derive a phrase which ends with a
newlike token.
This will allow us to recognise line-like sections of code and
thus know when to ignore newlines and when not to.
NeilBrown [Sun, 24 Nov 2013 07:14:11 +0000 (18:14 +1100)]
parsergen: centralise (some of) the collecting of next token.
A future patch will introduce next sites where we want to
discard the current token.
Rather than calling "token_next" at each site, make it possible
to just set "tk = NULL", and the next token will automatically
be collected when needed.
NeilBrown [Sun, 24 Nov 2013 06:50:09 +0000 (17:50 +1100)]
parsergen: recorded a prefered shift-symbol for error recovery.
When we find there is no valid parse step, one option that we don't
currently try is to synthesize a symbol and shift it. i.e. insert a
missing symbol.
A future patch will provide a circumstance where this is the ideal
response, so here we choose a symbol which could usefully be shifted.
We choose the one that will get us closest to the end of a production.
NeilBrown [Sun, 24 Nov 2013 06:36:41 +0000 (17:36 +1100)]
parsegen: unify the "next" frame to go onto stack.
We current have a current 'state' in the parser and a 'sym'
which is a local variable passed around in different ways.
Both of these get pushed onto the stack at the next 'shift'.
We will shortly add some more fields to the stack frame, so unify
'state' and 'sym' in to a 'next' struct in the parser struct which can
easily be extended.
These names work better for me.
There is an indent on every line, so the place where the
indent increases shouldn't be called "indent".
And "undent" isn't a word.
Change the tracing to print the full stack after every step.
This can be cross-referenced with the report that parsergen
can generate to get a full picture of how the parse is progressing.
parsergen: make sure we continue making states until all done.
Whenever we add a state, we need to check again, as it might have been
added early in the list.
There is probably a much more efficient way to do this....
I want items with larger indexes to always preceed smaller
indexes. This makes the report easier to follow.
So negate the index number (and add an offset) when sorting.
code / ocean / log