parsegen: Add brief explanation about optional newlines.

Optional newlines need care with a parsergen parse and the special
rules around them mean you cannot have a symbol that just absorbs
newlines.  Rather, you need to put the newline absorbtion in front of
whatever is allowed to follow newlines.

Signed-off-by: NeilBrown <neil@brown.name>

diff --git a/csrc/parsergen.mdc b/csrc/parsergen.mdc
index 94c29df1d2c7665f61903d413918d0cbee44e720..a7330351d767b52f52ba7b14a802e1b2556dd842 100644 (file)
--- a/csrc/parsergen.mdc
+++ b/csrc/parsergen.mdc
@@ -2692,9 +2692,29 @@ is still a work-in-progress.
 `TK_newline` tokens are ignored unless the top stack frame records
 that they are permitted.  In that case they will not be considered for
 shifting if it is possible to reduce some symbols that are all since
-the most recent start of line.  This is how a newline forcible
+the most recent start of line.  This is how a newline forcibly
 terminates any line-like structure - we try to reduce down to at most
 one symbol for each line where newlines are allowed.
+A consequence of this is that a rule like
+
+###### Example: newlines - broken
+
+       Newlines ->
+               | NEWLINE Newlines
+       IfStatement -> Newlines if ....
+
+cannot work, as the NEWLINE will never be shifted as the empty string
+will be reduced first.  Optional sets of newlines need to be include
+in the thing that preceed:
+
+###### Example: newlines - works
+
+       If -> if
+               | NEWLINE If
+       IfStatement -> If ....
+
+Here the NEWLINE will be shifted because nothing can be reduced until
+the `if` is seen.
 
 When, during error handling, we discard token read in, we want to keep
 discarding until we see one that is recognised.  If we had a full set