From: NeilBrown Date: Thu, 25 Jul 2013 10:13:22 +0000 (+1000) Subject: Rename Indent and Undent to IN and OUT X-Git-Tag: indentparser~7 X-Git-Url: https://ocean-lang.org/code/?p=ocean;a=commitdiff_plain;h=f66288249841261c6580cc03a192bc5efb1dbdce Rename Indent and Undent to IN and OUT These names work better for me. There is an indent on every line, so the place where the indent increases shouldn't be called "indent". And "undent" isn't a word. Signed-off-by: NeilBrown --- diff --git a/csrc/parsergen.mdc b/csrc/parsergen.mdc index e5c0dec..df217dd 100644 --- a/csrc/parsergen.mdc +++ b/csrc/parsergen.mdc @@ -143,8 +143,8 @@ different token types that `scanner` can report. [TK_mark] = "MARK", [TK_string] = "STRING", [TK_multi_string] = "MULTI_STRING", - [TK_indent] = "INDENT", - [TK_undent] = "UNDENT", + [TK_in] = "IN", + [TK_out] = "OUT", [TK_newline] = "NEWLINE", [TK_eof] = "$eof", }; @@ -539,8 +539,8 @@ Now we are ready to read in the grammar. | (0 << TK_number) | (1 << TK_string) | (1 << TK_multi_string) - | (1 << TK_indent) - | (1 << TK_undent), + | (1 << TK_in) + | (1 << TK_out), }; struct token_state *state = token_open(code, &conf); @@ -2550,8 +2550,8 @@ something like this. struct token_config config = { .ignored = (1 << TK_line_comment) | (1 << TK_block_comment) - | (1 << TK_indent) - | (1 << TK_undent), + | (1 << TK_in) + | (1 << TK_out), .number_chars = ".,_+-", .word_start = "", .word_cont = "", diff --git a/csrc/scanner.mdc b/csrc/scanner.mdc index 7e33d0c..0e15d8e 100644 --- a/csrc/scanner.mdc +++ b/csrc/scanner.mdc @@ -618,22 +618,22 @@ node (detected by `at_son()`); If a line starts with more white-space than the previous non-blank line - or if the first non-blank line in the document starts with any -white-space - then an Indent is reported at the start of the line. +white-space - then an "IN" is reported at the start of the line. Before the next non-blank line which starts with less white space, or -at the latest at the end of the document, a matching Undent token -is reported. There will always be an exact match between Indent and -Undent tokens. +at the latest at the end of the document, a matching "OUT" token +is reported. There will always be an exact match between "IN" and +"OUT" tokens. -It is possible for Undent to be followed (almost) immediately by an -Indent. This happens if, for example, the indent of three consecutive +It is possible for "OUT" to be followed (almost) immediately by an +"IN". This happens if, for example, the indent of three consecutive lines are 0, 8, 4 spaces. Before the second line we report an -Indent. Before the third line we must report an Undent, as 4 is less +"IN". Before the third line we must report an "OUT", as 4 is less than 8, then also an Ident as 4 is greater than 0. ###### token types - TK_indent, - TK_undent, + TK_in, + TK_out, For the purpose of measuring the length of white space, a tab adds at least one space, and rounds up to a multiple of 8. @@ -661,9 +661,9 @@ or a multi-line string are not reported separately, but each of these must be followed immediately by a newline so these constructs cannot hide the fact that a newline was present. -When Indents are being reported, the Newline which would normally be -reported immediately before the Indent is delayed until after the -matching undent. This makes an indented section act like a +When indents are being reported, the Newline which would normally be +reported immediately before the "IN" is delayed until after the +matching "OUT". This makes an indented section act like a continuation of the previous line to some extent. A blank line would normally be reported simply as two consecutive Newline @@ -672,7 +672,7 @@ reported) then the right thing to do is less obvious as Newlines should be delayed - but how many Newlines? The approach we will take is to report the extra Newlines immediately after -the Indent token, so the blank line is treated as though it were an indented +the IN token, so the blank line is treated as though it were an indented blank line. ###### token types @@ -681,49 +681,49 @@ blank line. If we find a newline or white space at the start of a block, we keep collecting spaces, tabs, and newlines until we find some real text. Then depending on the indent we generate some number of tokens. These -will be a sequence of "Newline Undent" pairs representing a decrease -in indent, then either a Newline or an Indent depending on whether the +will be a sequence of "Newline OUT" pairs representing a decrease +in indent, then either a Newline or an IN depending on whether the next line is indented, then zero or more Newlines representing all the blank lines that have been skipped. When a Newline leads to the next block of code there is a question of -whether the various Newline and Undent/Indent tokens should appear to +whether the various Newline and OUT/IN tokens should appear to pbelong to the earlier or later block. This is addressed by processing the tokens in two stages based on the relative indent levels of the two blocks (each block has a base indent to which the actual indents are added). -Any "Newline Undent" pairs needed to reduce the current indent to the +Any "Newline OUT" pairs needed to reduce the current indent to the maximum of the base indents of the old and new blocks are generated against the old block. Then if the next block does not have an increased indent, one more "Newline" is generated. -If further "Newline Undent" pairs are needed to get to the indent +If further "Newline OUT" pairs are needed to get to the indent level of the 'next' block, they are generated against that block, though the first Newline is suppressed (it having already been generated). -Finally the Newline or Indent for the first line of the new block is +Finally the Newline or IN for the first line of the new block is generated, unless the Newline needs to be suppressed because it appeared at the end of the previous block. -This means that a block may start with an Undent or an Indent, but +This means that a block may start with an OUT or an IN, but will only start with a Newline if it actually starts with a blank line. We will need to represent in the `token_state` where in this sequence of delayed tokens we are. As `state.col` records the target indent we -don't need to record how many undents or indents are needed. We do +don't need to record how many OUTs or INs are needed. We do need to record the number of blank lines, and which of Newline and -Undent is needed next in the initial sequence of pairs. +OUT is needed next in the initial sequence of pairs. For this we store one more than the number of blank lines as -`delayed_lines` and a flag for `undent_next`. +`delayed_lines` and a flag for `out_next`. ###### state fields int check_indent; int delayed_lines; - int undent_next; + int out_next; Generating these tokens involve two separate pieces of code. @@ -738,7 +738,7 @@ information and return one token. if (is_newline(ch) || (at_son(state) && ch <= ' ')) { int newlines = 0; int was_son = at_son(state); - if (ignored & (1<delayed_lines = newlines; - state->undent_next = was_son; + state->out_next = was_son; state->check_indent = 1; continue; } @@ -773,15 +773,15 @@ information and return one token. if (state->check_indent || state->delayed_lines) { if (state->col < state->indent_sizes[state->indent_level]) { - if (!state->undent_next && + if (!state->out_next && !(ignored & (1<undent_next = 1; + state->out_next = 1; tk.num = TK_newline; return tk; } state->indent_level -= 1; - state->undent_next = 0; - tk.num = TK_undent; + state->out_next = 0; + tk.num = TK_out; return tk; } if (state->col > state->indent_sizes[state->indent_level] && @@ -789,7 +789,7 @@ information and return one token. state->indent_level += 1; state->indent_sizes[state->indent_level] = state->col; state->delayed_lines -= 1; - tk.num = TK_indent; + tk.num = TK_in; return tk; } state->check_indent = 0; @@ -1090,7 +1090,7 @@ searching for. Now we have all the bits there is just one section missing: combining all the token parsing code into one block. -The handling of delayed tokens (newlines, indents, undents) must come +The handling of delayed tokens (Newlines, INs, OUTs) must come first before we try getting another character. Then we parse all the test, making sure that we check for known marks @@ -1194,8 +1194,8 @@ so that it can be used to tracing processed strings too. [TK_multi_string] = "mstring", [TK_line_comment] = "lcomment", [TK_block_comment] = "bcomment", - [TK_indent] = "indent", - [TK_undent] = "undent", + [TK_in] = "in", + [TK_out] = "out", [TK_newline] = "newline", [TK_eof] = "eof", [TK_error] = "ERROR", @@ -1205,8 +1205,8 @@ so that it can be used to tracing processed strings too. default: /* known word or mark */ fprintf(f, "%.*s", tok.txt.len, tok.txt.txt); break; - case TK_indent: - case TK_undent: + case TK_in: + case TK_out: case TK_newline: case TK_eof: /* No token text included */