-Also you need to know how the scanner can report indent information to the parser. I discuss that in some detail when introducing my lexical scanner but you probably don't need to read that now. It is sufficient to know that the token stream will have paired "INDENT" and "UNDENT" tokens, and that the "NEWLINE token that you might expect to come before an "INDENT" is delayed until after the matching "UNDENT".
+Also you need to know how the scanner can report indent information to the parser. I discuss that in some detail when introducing my lexical scanner but you probably don't need to read that now. It is sufficient to know that the token stream will have paired "INDENT" and "UNDENT" tokens, and that the "NEWLINE" token that you might expect to come before an "INDENT" is delayed until after the matching "UNDENT".
With this in place, it seems natural to allow the parser to absorb INDENT and UNDENT, but to require that when it reduces and production, the net indent absorbed while shifting the tokens which make up that production is zero. i.e for every INDENT there must be exactly one UNDENT. So:
After trying to describe this I need to refine it again.
- A symbol is "can_eol" if any symbol in any production from this symbol
is followed only by nullable symbols.
- - A state is "starts_line" if the previous symbol in any item "can_eol",
+ - A state is "starts_line" if the previous symbol in any item "can_eol",)
or is nullable and previous to that "can_eol", etc.
- - The start state is "starts_line" if the start symbol "can_eol".
\ No newline at end of file
+ - The start state is "starts_line" if the start symbol "can_eol".
+
+19may2019 - in Texas!
+I hit a problem.
+
+Because anything that ends at a NEWLINE is considered LineLike, an
+Expression is LineLike. I didn't want that.
+So if I have
+
+ print
+ a + b
+ + c
+
+The NEWLINE after b is not Ignored in the expression,
+ only once it has reduced the simplestatement because....
+
+ A NEWLINE is discarded when the tos is NOT newline_permitted.
+ A frame gets newline_permitted when the state "starts_line"
+ which happens if a starting sym is "line_like", meaning it
+ can usefully be followed by a newline.
+
+
+25may2019
+ I've re-read most of my notes about NEWLINE parsing and I'm not sure....
+ I originally wanted "linelike" to be anything containing a newline. I apparently
+ found problems and now it is anything followed by a new line.
+ But that seems weird, because in
+ assignment -> name = expression
+ statement -> assignment NEWLINE
+ expression is followed by a newline, so is linelike. I think.
+
+ Can I return to my original idea?
+ Any symbol containing a newline is lineline, and newlines shouldn't be ignored.
+ If the symbol started since an indent. If it started before an indent, then it
+ doesn't affect newlines.
+
+ During parsing, we don't know which symbol we are working on (yet),
+ only which states we have, which can be working on multiple symbols.
+ The decision we need to make when we see a newline is which of these to choose:
+ - ignore - if states since an indent don't start a line
+ - reduce - if there is start-of-line state since start of line.
+ - shift - if we can, and not ignored
+ - reduce - if we can
+ - error - if nothing else
+
+ Maybe every item in an itemset must be followed by a newline if any are.
+ That cannot work as we always add prefixed of 'next' symbol.
+
+
+ if expr { NEWLINE
+ if expr NEWLINE {
+
+ Should the NEWLINE be ignored? No! It isn't indented.
+ So if I want NEWLINEs to be allowed, they need to be explicit.
+ This makes the state startsline.
+
+ So: we mark can_eol on any symbol that can derive NEWLINE
+ we mark a state as starts_line if any item with dot-at-start can derive a can_eol.
+
+ Then if the most recent starts_line is further back than an indent, we ignore newlines.
+ So I don't need the lineline flag.
+
+03jun2019
+ I'm making progress with newlines...
+ Newlines *are* shifted if there is only one symbol since the start of the line..
+ The current conundrum is what happens to several blank lines after:
+ if cond:
+ action
+
+ The next thing could be
+ - an else:
+ - a new complex statement
+ - the end of the block
+
+ The first two are currently handle with newlines before things, so we stack up
+ several newlines, then when we see a thing, we reduce the newlines out from under it.
+ But that doesn't work for the end of the block.
+ A Newlines symbol cannot insert an optional linebreak, but can absorb blank lines.
+ So places where a line break is optional, we have
+ Open -> {
+ | Newline {
+ Where we want to allow blank lines, we have
+ Newlines ->
+ NEWLINE Newlines
+
+ Then if we see a newline, not at start of line, we shift if it is optional
+ When we see a newline at start of line, we shift if blank lines are allowed.
+ So if grammar allows
+ Newlines Open
+ Then .... that doesn't work. Newlines won't get any lines.
+ Maybe we want:
+ Open -> {
+ | NEWLINE Newlines {
code / ocean-D / blobdiff