From: NeilBrown <neil@brown.name>
Date: Fri, 17 May 2019 13:31:48 +0000 (+1000)
Subject: scanner: handle missing newline at EOF
X-Git-Tag: JamisonCreek-3~9
X-Git-Url: https://ocean-lang.org/code/?p=ocean;a=commitdiff_plain;h=83e9fd29cdd342f5324dae5573211b9d311059cf

scanner: handle missing newline at EOF

If there is no newline at EOF, we can see EOF immediately after
a valid symbol.  This can lead to calling close_token() when
state->node is NULL, which crashes.
The code in close_token() only makes sense if state->node is still the
same as token->node.  If it isn't, the token must be at the very end of
its code-node, so a different calculation is needed.

This avoids the NULL deref.

Signed-off-by: NeilBrown <neil@brown.name>
---

diff --git a/csrc/scanner.mdc b/csrc/scanner.mdc
index 0d7a62d..15306dd 100644
--- a/csrc/scanner.mdc
+++ b/csrc/scanner.mdc
@@ -1068,8 +1068,11 @@ parsed too much already.  For that there is `reset_token`.
 	static void close_token(struct token_state *state,
 	                        struct token *tk)
 	{
-		tk->txt.len = (state->node->code.txt + state->offset)
-		              - tk->txt.txt;
+		if (state->node != tk->node)
+			tk->txt.len = tk->node->code.len - (tk->txt.txt - tk->node->code.txt);
+		else
+			tk->txt.len = (state->node->code.txt + state->offset)
+			              - tk->txt.txt;
 	}
 
 	static void reset_token(struct token_state *state, struct token *tok)