scanner: handle completely unrecognized characters.

If we are ignoring numbers, then a digit looks like
nothing at all, not even an unknown mark.
Make sure to handle that properly.

Signed-off-by: NeilBrown <neil@brown.name>

diff --git a/csrc/scanner.mdc b/csrc/scanner.mdc
index 6eee0892deb07e78fb3f6de7497a810515a313c7..705e02e096939e4b20e3e3ad4da5a91264d107a3 100644 (file)
--- a/csrc/scanner.mdc
+++ b/csrc/scanner.mdc
@@ -867,12 +867,23 @@ If the token we have is not empty and `TK_mark` is allowed,
 we have an unknown mark, otherwise this must be an error.
 
 ###### unknown mark
-       /* one unknown character */
+
+       /* one unknown mark character */
+       if (tk.txt.len) {
+               close_token(state, &tk);
+               if (ignored & (1<<TK_mark))
+                       tk.num = TK_error;
+               else
+                       tk.num = TK_mark;
+               return tk;
+       }
+       /* Completely unrecognised character is next, possibly
+        * a digit and we are ignoring numbers.
+        * What ever it is, make it an error.
+        */
+       get_char(state);
        close_token(state, &tk);
-       if (ignored & (1<<TK_mark))
-               tk.num = TK_error;
-       else
-               tk.num = TK_mark;
+       tk.num = TK_error;
        return tk;
 
 ## Tools For The Task