]> ocean-lang.org Git - ocean/log
ocean
3 years agoparsegen: detect left-recursive symbols in non-final position.
NeilBrown [Tue, 6 Oct 2020 06:02:22 +0000 (17:02 +1100)]
parsegen: detect left-recursive symbols in non-final position.

A left-recursive symbol that appear other than at the end of a
production causes problem for indent-based parsing, as describe in the
document.  So teach parsergen to be able to report them.

Ocean currently has several of these, which I'll need to look into at a
later date.

Signed-off-by: NeilBrown <neil@brown.name>
3 years agoscanner: change the meaning of ignoring comment tokens.
NeilBrown [Tue, 6 Oct 2020 04:44:46 +0000 (15:44 +1100)]
scanner: change the meaning of ignoring comment tokens.

Previously ignoring comment tokens meant they were still parsed, but not
returned.  The only way to stop them being parsed was to declare
known marks for the start symbols.

This made is not possible for parsergen to define a language that had
a known mark that would otherwise start a comment.

So change the ignoring of comment tokens to mean they aren't parsed.  If
you want to parse comments but not return them, leave the new
"return_comments" field as so.  In the unusual case that you want to
return comments set return_comments to 1.

Confirm that this has the desired effect by added in "//" as an
integer-division operator to the sample calculator.

Signed-off-by: NeilBrown <neil@brown.name>
3 years agoindent_test: fix makefile
NeilBrown [Mon, 5 Oct 2020 23:00:31 +0000 (10:00 +1100)]
indent_test: fix makefile

Maybe 'make' has changed a little to be less forgiving, but 'make itest'
isn't working now.  All of LDLIBS are included in the 'cc' line, but
there are no dependencies to make sure they have been built.

The problem is that I'm using LDLIBS for different programs which need
different libs.  This isn't such a good idea.
So change indent_test to use itestLDLIBS and itestCFLAGS.

Signed-off-by: NeilBrown <neil@brown.name>
3 years agoRename string.h and number.h headers
NeilBrown [Fri, 11 Sep 2020 07:53:49 +0000 (17:53 +1000)]
Rename string.h and number.h headers

Having a "string.h" is potentially confusing as there
is a standard <string.h>
So rename it and for consistency, rename number.h too.

Signed-off-by: NeilBrown <neil@brown.name>
4 years agoparsergen: add token location to tracing
NeilBrown [Fri, 28 Jun 2019 09:40:11 +0000 (19:40 +1000)]
parsergen: add token location to tracing

When we print the look-ahead token for tracing, add the line/column
location so it can be easily cross-references to the source code.

Signed-off-by: NeilBrown <neil@brown.name>
4 years agoparsergen: only non-terminals should make a state "starts_line"
NeilBrown [Fri, 28 Jun 2019 09:36:49 +0000 (19:36 +1000)]
parsergen: only non-terminals should make a state "starts_line"

If a state is followed by NEWLINE, then it isn't starts_line - more like
ends_line.
It is only non-terminals containing NEWLINEs that cause a state
to be starts_line.
So move the test to after we stop looking at terminals.

Signed-off-by: NeilBrown <neil@brown.name>
4 years agooceani: allow 'then' in simple if statements.
NeilBrown [Sun, 23 Jun 2019 05:37:50 +0000 (15:37 +1000)]
oceani: allow 'then' in simple if statements.

Allow 'then' after "if expression", and don't require a ':' if
it is followed by simple statements.
Similarly "else" doesn't need a colon for simple statements

Signed-off-by: NeilBrown <neil@brown.name>
4 years agooceani: change parsing for ; at end
NeilBrown [Sun, 23 Jun 2019 04:41:47 +0000 (14:41 +1000)]
oceani: change parsing for ; at end

When we have 'for' and 'then' on the same line, I want to
require a ';' for the 'for' (and 'while').
So change SimpleStatemnts to never end with ';', and require
a ; or Newline after each instance of SimpleStatements.

Hmm... I guess that could be abstracted.

Signed-off-by: NeilBrown <neil@brown.name>
4 years agooceani: modify grammar to not waste stack on newlines
NeilBrown [Sun, 23 Jun 2019 04:29:13 +0000 (14:29 +1000)]
oceani: modify grammar to not waste stack on newlines

Current grammar uses one stack frame per newline for leading
newlines as these productions are right-recursive.  This is
unnecessary and inelegant. Change to use a left-recursive Newlines
production.

Signed-off-by: NeilBrown <neil@brown.name>
4 years agoindent_test: reduce stack usage for preceding NEWLINEs
NeilBrown [Sun, 23 Jun 2019 03:51:46 +0000 (13:51 +1000)]
indent_test: reduce stack usage for preceding NEWLINEs

In the cases where we allow preceding newlines (Statementlist Open Close)
we current use one parse-stack from for each newline.  While there are
unlikely to be many, this is inelegant.
Change the right-recursive form to use a left-recursive Newlines rule
that absorbs one or more NEWLINEs using at most 2 stack frames.

Signed-off-by: NeilBrown <neil@brown.name>
4 years agoparsergen: allow $$OUT to be satisfied are start-of-line.
NeilBrown [Sun, 23 Jun 2019 00:21:14 +0000 (10:21 +1000)]
parsergen: allow $$OUT to be satisfied are start-of-line.

If a $$OUT (or $$NEWLINE) production is being reduced at
start-of-line (with no indents), then that is satisfactory,
we don't need NEWLINE etc as look-ahead.

This means that in cases where this is relevant, the computed
lookahead is wrong - we shouldn't have striped it.
I don't think this matters as it only affects conflict warnings,
and I think these will be reported at a higher level if relevant.
If essense, the $$OUT marking is like a precendence marking which
suppresses shift/reduce warnings as it say that decision is being made
on some basis other than look-ahead.

Signed-off-by: NeilBrown <neil@brown.name>
4 years agoparsergen: allow $$OUT as well as $$NEWLINE
NeilBrown [Sat, 22 Jun 2019 23:00:06 +0000 (09:00 +1000)]
parsergen: allow $$OUT as well as $$NEWLINE

Both of these mean the same thing: the product must end
in a 2-d visible way

Signed-off-by: NeilBrown <neil@brown.name>
4 years agoparsergen: don't assume TK_newline in look-ahead for lineline
NeilBrown [Sat, 22 Jun 2019 22:56:15 +0000 (08:56 +1000)]
parsergen: don't assume TK_newline in look-ahead for lineline

A linelike production might not have TK_newline in the look-ahead,
so don't assume it.  If it isn't there, make the look-ahead
set empty.

Signed-off-by: NeilBrown <neil@brown.name>
4 years agoparsergen: revert the allowance for "non-critical" conflicts.
NeilBrown [Sun, 16 Jun 2019 02:37:23 +0000 (12:37 +1000)]
parsergen: revert the allowance for "non-critical" conflicts.

Having improved the parsing approach, I don't get any
"non-critical" conflicts any more, and I no longer think
they aren't critical.

Conflicts can often be resolved with precendence if they cannot
be resolved any other way.

So treat all conflict, except shift/reduce on NEWLINE, as genuine
conflicts.

Signed-off-by: NeilBrown <neil@brown.name>
4 years agooceani-tests: add tests with lots of blank lines.
NeilBrown [Sun, 16 Jun 2019 02:34:40 +0000 (12:34 +1000)]
oceani-tests: add tests with lots of blank lines.

This found some problems with the grammer, which I fixed.

Signed-off-by: NeilBrown <neil@brown.name>
4 years agooceani: convert declarations to new approach to newlines
NeilBrown [Sun, 16 Jun 2019 02:16:44 +0000 (12:16 +1000)]
oceani: convert declarations to new approach to newlines

There are now no conflicts.  Yay!!

Signed-off-by: NeilBrown <neil@brown.name>
4 years agooceani: re-indent DeclareStruct
NeilBrown [Sun, 16 Jun 2019 01:37:03 +0000 (11:37 +1000)]
oceani: re-indent DeclareStruct

It wasn't indented to easily allow alternate productions.

Signed-off-by: NeilBrown <neil@brown.name>
4 years agogitignore - add coverage_scanner
NeilBrown [Sun, 16 Jun 2019 01:35:00 +0000 (11:35 +1000)]
gitignore - add coverage_scanner

This is a tmp file - should have been ignored before.

Signed-off-by: NeilBrown <neil@brown.name>
4 years agoparsegen: fix up look-ahead for $$NEWLINE items.
NeilBrown [Sun, 16 Jun 2019 01:31:54 +0000 (11:31 +1000)]
parsegen: fix up look-ahead for $$NEWLINE items.

I was discarding all non-newlines from the lookahead
in the wrong place.
I need to do it based on the productions added, not
the item the are generated by.

Signed-off-by: NeilBrown <neil@brown.name>
4 years agooceani: change NEWLINE parsing in statements to new model.
NeilBrown [Sun, 16 Jun 2019 00:16:07 +0000 (10:16 +1000)]
oceani: change NEWLINE parsing in statements to new model.

The new module is:

 A list of line-like things must access a newline first:
  Statementlist -> Statements
                | NEWLINE Statementlist

 Any line-like thing must reduce to a single symbol:
   SimpleStatementLine IfPart WhilePart CasePart etc
 An individual line-like thing must allow following newlines
 e.g.
   IfHead -> if Expression Block
           | IfHead NEWLINE

 A block that can be multi-line or single-line should be marked with
   $$NEWLINE
 This will require a NEWLINE to reduce it, but won't swollow the newline.

Signed-off-by: NeilBrown <neil@brown.name>
4 years agoindent_test: adjust grammer to handle blank lines better.
NeilBrown [Sat, 15 Jun 2019 23:47:58 +0000 (09:47 +1000)]
indent_test: adjust grammer to handle blank lines better.

This uses the new $$NEWLINE are other techniques to ensure
blank lines are handled well.
We also test that adding blank lines everywhere doesn't break
anything.

Signed-off-by: NeilBrown <neil@brown.name>
4 years agoident_test: declare precedence for 'else'
NeilBrown [Sat, 15 Jun 2019 23:20:30 +0000 (09:20 +1000)]
ident_test: declare precedence for 'else'

By declaring precedence for 'else', we suppress conflict warnings
Normally newlines and indents will resolve any conflict, but
if not, else associates to the right - it should be shifted, not cause
a reduce (which is the default anyway)

Signed-off-by: NeilBrown <neil@brown.name>
4 years agoindent_test: make testing easier - and include expected output
NeilBrown [Sat, 15 Jun 2019 23:17:58 +0000 (09:17 +1000)]
indent_test: make testing easier - and include expected output

Signed-off-by: NeilBrown <neil@brown.name>
4 years agoparsergen: allow non-virtual symbol to carry precedence.
NeilBrown [Sat, 15 Jun 2019 23:14:03 +0000 (09:14 +1000)]
parsergen: allow non-virtual symbol to carry precedence.

When a production has $$foo to explicity set precedence, the 'foo'
doesn't really need to be virtual, it just needs to actually
carry a precedence.

Signed-off-by: NeilBrown <neil@brown.name>
4 years agoparsergen: introuce $$NEWINE pseudo-precedence.
NeilBrown [Sat, 15 Jun 2019 22:29:16 +0000 (08:29 +1000)]
parsergen: introuce $$NEWINE pseudo-precedence.

Sometimes we need a produce to be terminated by a newline, but we
don't want to consume the newline with a "shift".
Case in point is:
   Block -> : StatementList

Which can be used with
   Statement -> if Expression Block
   StatementList -> Statement

I want this to parse:

   if something: if otherthing: action

which might seem a little odd, but is syntactically sensible.
The NEWLINE at the end is requred, and must close both nested Statements.
The NEWLINE will already cause a REDUCE, but if we don't have
   Block -> : Statementlist NEWLINE
then something else could force a reduce, and we don't want that.
So introduce a marking "$$NEWLINE" which is similar to imposing a precedence
on a production.  Now

   Block -> : StatementList $$NEWLINE

means that a NEWLINE is required to end a Block, but it isn't
shifted.  If anything else if found here, it is an error.

We also allow $eof and OUT to reduce this production.

Signed-off-by: NeilBrown <neil@brown.name>
4 years agoparsergen: allow $<1 to be used for structs as well as pointer.
NeilBrown [Sat, 15 Jun 2019 22:25:23 +0000 (08:25 +1000)]
parsergen: allow $<1 to be used for structs as well as pointer.

A future patch will want to use this for non-pointers, so make it
work - it is more consistent this way anyway.

Also: update to parsers to use $< where possible.

Signed-off-by: NeilBrown <neil@brown.name>
4 years agoOceani: be sure to set precedence on required terminals.
NeilBrown [Mon, 10 Jun 2019 06:51:31 +0000 (16:51 +1000)]
Oceani: be sure to set precedence on required terminals.

In some cases I was only setting on nonterminals which derived the
terminals.  That set it for the reduction, but not the terminal.

Also remove an ERROR production that duplicates one from Declaration

Signed-off-by: NeilBrown <neil@brown.name>
4 years agoparsergen: don't report conflicts resolved by precedence.
NeilBrown [Mon, 10 Jun 2019 06:34:55 +0000 (16:34 +1000)]
parsergen: don't report conflicts resolved by precedence.

If precedence information has been given which can resolve
a conflict, then don't report it.

Signed-off-by: NeilBrown <neil@brown.name>
4 years agooceani: convert expression parsing to use precedences.
NeilBrown [Sun, 9 Jun 2019 23:11:05 +0000 (09:11 +1000)]
oceani: convert expression parsing to use precedences.

As precedences are now working (and largely have been for a long time).
it is time to start using them for oceani.

Signed-off-by: NeilBrown <neil@brown.name>
4 years agoparsergen: flip ordering of precedence declarations.
NeilBrown [Sun, 9 Jun 2019 23:00:43 +0000 (09:00 +1000)]
parsergen: flip ordering of precedence declarations.

Change so first precedence declaration is the lowest precedence.
This is consistent with bison, and will make converting
'oceani' easer.
When not using precedence, it is easier to do the lowest
precedence first - so keep for approach.

Signed-off-by: NeilBrown <neil@brown.name>
4 years agooceani: use 'bracket' printing for expressions.
NeilBrown [Sun, 9 Jun 2019 22:49:08 +0000 (08:49 +1000)]
oceani: use 'bracket' printing for expressions.

Adding brackets to expression printing removes and ambiguity.
As I'm about to change expression parsing, I want to be able to
see that the result is correct.

Signed-off-by: NeilBrown <neil@brown.name>
4 years agoparsergen: include virtual symbols in table of non-terminals
NeilBrown [Sun, 9 Jun 2019 22:35:36 +0000 (08:35 +1000)]
parsergen: include virtual symbols in table of non-terminals

Symbol numbers assigned in grammar_analyse are in three groups:
 - predefined (NUMBER, STRING, etc)
 - Terminals
 - everything else: non-terminals and virtual.

When creating the non_term[] list of names, we need to include
virtual symbols in there, otherwise lookup by symbol-number
might find the wrong value - or might reach beyond end of array.

Signed-off-by: NeilBrown <neil@brown.name>
4 years agooceani: labels only in 'use' statement.
NeilBrown [Sun, 9 Jun 2019 05:55:21 +0000 (15:55 +1000)]
oceani: labels only in 'use' statement.

A variable is only set as a label if it appear at top
level of "use" statement - otherwise it is "unknown".

This makes error messages more meaningful.

Signed-off-by: NeilBrown <neil@brown.name>
4 years agooceani: add more syntax error handling.
NeilBrown [Sun, 9 Jun 2019 00:15:39 +0000 (10:15 +1000)]
oceani: add more syntax error handling.

We catch some errors earlier so more errors can be
reported in a single run.
The indent structure make recovery easy!

As we might abort out of a var scope, we only insist the scope
is balanced in there are no parser errors.

Signed-off-by: NeilBrown <neil@brown.name>
4 years agoscanner: hide impossible code from coverage testing.
NeilBrown [Sun, 9 Jun 2019 00:03:44 +0000 (10:03 +1000)]
scanner: hide impossible code from coverage testing.

The code in "white space" section is truly impossible, as
we don't return WEOF untile ->node is NULL, and when that
happens, ->col is set to 0.

Other code is only "mostly dead".

This brings coverage to 95%

Signed-off-by: NeilBrown <neil@brown.name>
4 years agoscanner: test for errors with multi-line objects crossing node.
NeilBrown [Sat, 8 Jun 2019 23:54:39 +0000 (09:54 +1000)]
scanner: test for errors with multi-line objects crossing node.

multi-line strings and comments musn't cross nodes.
Testing this increases coverage above 94%

Signed-off-by: NeilBrown <neil@brown.name>
4 years agoscanner: improve transition from node to node.
NeilBrown [Sat, 8 Jun 2019 23:42:05 +0000 (09:42 +1000)]
scanner: improve transition from node to node.

When we are at the end of a node, it is wrong to use do_strip() as
that looks beyond the end of the node.
It is better, once we have determined to accept the newline at the
end of a node (i.e. once no unget is possible), to move to the
start of the next node, and assess column position and indents from
that perspective.
Do this removes some tests on at_son/at_eon, and make some code a
bit more transparent - for example the flag that say whether an "out"
is next now depends on where a newline was recently seen, which makes
more sense than whether we were at the start of a node (out and newline
alternate in some contexts).

Also: add the test which found this problem.   This requires a
new set of tests - tests which can scan tokens from multiple nodes.

Now that we are testing node transitions, the coverage has jumped
over 92%

Signed-off-by: NeilBrown <neil@brown.name>
4 years agoscanner: allow a section to be specified.
NeilBrown [Sat, 8 Jun 2019 10:27:15 +0000 (20:27 +1000)]
scanner: allow a section to be specified.

If --section arg is given, only report on that section.

Signed-off-by: NeilBrown <neil@brown.name>
4 years agoscanner: fix bug with indent at start of node.
NeilBrown [Sat, 8 Jun 2019 10:18:35 +0000 (20:18 +1000)]
scanner: fix bug with indent at start of node.

If we find an indent, we assume there are delayed newlines
to comsume.
This is often true, but not at the start of a node.
So don't decrement delayed_lines if it is already zero.

Add the test case that found this bug for me.

Signed-off-by: NeilBrown <neil@brown.name>
4 years agoscanner: fix handling of indents in sub-nodes
NeilBrown [Sat, 8 Jun 2019 04:35:34 +0000 (14:35 +1000)]
scanner: fix handling of indents in sub-nodes

I seem to have confused ->indent_sizes[] and ->col

->col is used for the reported location of a token so
must be the actual column in the file, with no adjustment.

->indent_sizes[] is indents, which must include any inherited from
parent nodes.  So this is a completely different value.

So change mdcode to store the local node indent in ->needs_strip -
this is the number of text columns that are stripped off.
This, subtracted from ->indent is the text offset of the physical
start-of-line.  Adding the measured ->col then gives us
the indent in the composed file, the indent that must be used
for detecting TK_in and TK_out.

Introduce a new function state_indent() which determines that indent,
and use it instead of ->col.

Signed-off-by: NeilBrown <neil@brown.name>
4 years agoscanner: fix at_son()
NeilBrown [Sat, 8 Jun 2019 04:26:15 +0000 (14:26 +1000)]
scanner: fix at_son()

Current test for "at start of node" is broken for 2 reasons.

1/ it doesn't account for the node-indent chars that are stripped
   off by do_strip()
2/ it check the ->offset *after* a character has been extracted,
   it needs to check the offset from before, which is in ->prev_offset

Signed-off-by: NeilBrown <neil@brown.name>
4 years agoscanner: fix some typos in text
NeilBrown [Sat, 8 Jun 2019 04:23:05 +0000 (14:23 +1000)]
scanner: fix some typos in text

Signed-off-by: NeilBrown <neil@brown.name>
4 years agoparsergen: allow prefix code for do_reduce to be provided.
NeilBrown [Thu, 6 Jun 2019 22:51:23 +0000 (08:51 +1000)]
parsergen: allow prefix code for do_reduce to be provided.

This allow common variables to be declared and initialised
for all reduce fragments.

Use this in oceani to declare 'c' holding the parser context,
so we don't need to call config2context in so many places.

Signed-off-by: NeilBrown <neil@brown.name>
4 years agooceani: setting ->parse_error now aborts type analysis.
NeilBrown [Thu, 6 Jun 2019 22:31:07 +0000 (08:31 +1000)]
oceani: setting ->parse_error now aborts type analysis.

When ever we call type_err() or tok_err(), in type analysis,
we currently need to also set *ok=0;  This is uhly and error-prone.

So instead, we can detect ->parse_error being set, and assume
something is no OK.
This is most easily done in a wrapper for propagate_types.

Signed-off-by: NeilBrown <neil@brown.name>
4 years agooceani: redo parsing of blank lines.
NeilBrown [Wed, 5 Jun 2019 08:21:18 +0000 (18:21 +1000)]
oceani: redo parsing of blank lines.

I've been puzzling how best to write a grammar to
handle blank lines and option line-breaks well.
The "OptNL" approach didn't work, and "Newlines"
only sometimes works.

I won't try to explain all the logic here, but I do plan to write a
blog post about it soon.

Signed-off-by: NeilBrown <neil@brown.name>
4 years agooceani-tests: use LALR to check for conflicts.
NeilBrown [Wed, 5 Jun 2019 08:12:20 +0000 (18:12 +1000)]
oceani-tests: use LALR to check for conflicts.

LR1 analysis takes a lot longer than LALR and will never find
more conflicts, only ever fewer.
So with to that when checking for conflicts.

Also report that the check is happening, and fix a typo.

Signed-off-by: NeilBrown <neil@brown.name>
4 years agoparsergen - don't completely hide non-critical conflicts.
NeilBrown [Wed, 5 Jun 2019 08:09:59 +0000 (18:09 +1000)]
parsergen - don't completely hide non-critical conflicts.

Shift/Reduce conflicts that are likely to be resolved by
a line break are currently hidden.  This is probably a good idea,
but sometimes it can be useful to see them anyway.
So report them as "non-critical" conflicts, and don't
count them - so if no other conflicts are found, the
"no conflicts" message is still generated.

Signed-off-by: NeilBrown <neil@brown.name>
4 years agoparsegen: Add brief explanation about optional newlines.
NeilBrown [Sun, 2 Jun 2019 06:45:56 +0000 (16:45 +1000)]
parsegen: Add brief explanation about optional newlines.

Optional newlines need care with a parsergen parse and the special
rules around them mean you cannot have a symbol that just absorbs
newlines.  Rather, you need to put the newline absorbtion in front of
whatever is allowed to follow newlines.

Signed-off-by: NeilBrown <neil@brown.name>
4 years agooceani: allow spaces in numbers.
NeilBrown [Wed, 29 May 2019 11:54:53 +0000 (21:54 +1000)]
oceani: allow spaces in numbers.

Spaces are now allowed in (decimal) numbers to highlight
grouping.
Tests are added.

Also ensure unknown marks are reported as errors.

Signed-off-by: NeilBrown <neil@brown.name>
4 years agoscanner: improve number parsing.
NeilBrown [Wed, 29 May 2019 11:51:22 +0000 (21:51 +1000)]
scanner: improve number parsing.

In particular, space must be preceeded and followed by a digit
(not a letter).
Also '_' must be preceded and followed by a hex digit, but this
wasn't enforced.

Add tests to check on numbers more thoroughly.

Signed-off-by: NeilBrown <neil@brown.name>
4 years agoRemove excess blank lines
NeilBrown [Wed, 29 May 2019 08:31:55 +0000 (18:31 +1000)]
Remove excess blank lines

We never need 2 blank lines in a row.

Signed-off-by: NeilBrown <neil@brown.name>
4 years agoscanner tests: handle errors and more
NeilBrown [Wed, 29 May 2019 08:27:47 +0000 (18:27 +1000)]
scanner tests: handle errors and more

Handle assorted more cases, particularly errors.

Now coverage has passed 90%

Signed-off-by: NeilBrown <neil@brown.name>
4 years agoscanner: handle completely unrecognized characters.
NeilBrown [Wed, 29 May 2019 08:25:50 +0000 (18:25 +1000)]
scanner: handle completely unrecognized characters.

If we are ignoring numbers, then a digit looks like
nothing at all, not even an unknown mark.
Make sure to handle that properly.

Signed-off-by: NeilBrown <neil@brown.name>
4 years agoscanner: handle unknown marks once.
NeilBrown [Wed, 29 May 2019 08:19:54 +0000 (18:19 +1000)]
scanner: handle unknown marks once.

There were two places in the code that tried to handle
unknown marks.  We only need one.

Signed-off-by: NeilBrown <neil@brown.name>
4 years agoSeparate demos from tests.
NeilBrown [Wed, 29 May 2019 00:38:39 +0000 (10:38 +1000)]
Separate demos from tests.

'tests' check that the code is working, and fail if the results
aren't what was expected.
'demos' simply run the code and show what it can do.  The don't provide
any immediate assurance that it is doing the right thing.

Separate these into "make tests" and "make demos"

Signed-off-by: NeilBrown <neil@brown.name>
4 years agoAdd test code for the scanner
NeilBrown [Tue, 28 May 2019 05:57:09 +0000 (15:57 +1000)]
Add test code for the scanner

Currently only 85% coverage.  I haven't tested error yet,
or various other things.  I have found bugs though!

Signed-off-by: NeilBrown <neil@brown.name>
4 years agoscanner: fix typo in testing ignored flagged
NeilBrown [Tue, 28 May 2019 05:49:36 +0000 (15:49 +1000)]
scanner: fix typo in testing ignored flagged

Use '&' to test a bit, not '&&' !!!

Signed-off-by: NeilBrown <neil@brown.name>
4 years agoscanner: fix multi-line strings
NeilBrown [Tue, 28 May 2019 05:04:35 +0000 (15:04 +1000)]
scanner: fix multi-line strings

I broke multi-line strings.  A quote cannot be part of an
unknown mark, but it can be part of '"""' or "'''".

Signed-off-by: NeilBrown <neil@brown.name>
4 years agoscanner: add options to allow more complete testing.
NeilBrown [Sun, 26 May 2019 22:48:35 +0000 (08:48 +1000)]
scanner: add options to allow more complete testing.

Allow all facets of scanner to be controlled by command line
options.  This will make it easier to do exhaustive testing.

Also make sure to free things that are allocated.

Signed-off-by: NeilBrown <neil@brown.name>
4 years agoparsergen - fix newline parsing (again)
NeilBrown [Sun, 26 May 2019 05:04:43 +0000 (15:04 +1000)]
parsergen - fix newline parsing (again)

Add a test-case to oceani-tests.mdc which fails, but shouldn't.
It fails because expressions are treated as line-like, so newlines
aren't ignored.

I realize that having linelike symbols being those that are followed by
a newline really doesn't work.
So go back to the original idea that "linelike symbols are those which
contain a newline".

Then a state starts a line if it is at the start of a linelike symbol.

This simplifies the code, seems to work correctly for existing tests,
and allows the new test to pass.

Signed-off-by: NeilBrown <neil@brown.name>
4 years agoparsergen: make it easier to test the simple 'calc' code.
NeilBrown [Sun, 26 May 2019 05:03:13 +0000 (15:03 +1000)]
parsergen: make it easier to test the simple 'calc' code.

Allow 'calc' to read a test code from parsergen.mdc.

Signed-off-by: NeilBrown <neil@brown.name>
4 years agoOceani - Jamison Creek Version JamisonCreek-3
NeilBrown [Sat, 18 May 2019 20:46:13 +0000 (06:46 +1000)]
Oceani - Jamison Creek Version

Clean up text and provide new version name.
Additions for this version include:
 - type identifiers
 - arrays and structs
 - global const
 - "and then" "or else" "if .. else"
 - test suite
 - valgrind testing, coverage testing

Signed-off-by: NeilBrown <neil@brown.name>
4 years agooceani-tests: add tests for str/bool command line args
NeilBrown [Sat, 18 May 2019 15:39:23 +0000 (01:39 +1000)]
oceani-tests: add tests for str/bool command line args

This brings test coverage over 94%

Signed-off-by: NeilBrown <neil@brown.name>
4 years agooceani: cleanup "sayhello" output
NeilBrown [Sat, 18 May 2019 15:06:34 +0000 (01:06 +1000)]
oceani: cleanup "sayhello" output

This had become unreadable due to the long arrays.
A few other little cleanups and comments.

Signed-off-by: NeilBrown <neil@brown.name>
4 years agooceani: add structs
NeilBrown [Sat, 18 May 2019 14:16:01 +0000 (00:16 +1000)]
oceani: add structs

This is just basic struct support, showing how they
are declared and used.

There is more sophistication to be added such as anonymous
fields and per-field attributes.

Signed-off-by: NeilBrown <neil@brown.name>
4 years agooceani: fix merging of conditionally-scoped variables.
NeilBrown [Sat, 18 May 2019 13:51:24 +0000 (23:51 +1000)]
oceani: fix merging of conditionally-scoped variables.

The problem here was that the list seen in ->in_scope
includes more than just what is currently in-scope.
It also contains things that have been replaced by new instances
of the name.
These can be detected a they aren't the first variable listed
under their name any more.

Signed-off-by: NeilBrown <neil@brown.name>
4 years agooceani-tests: delay valgrind tests
NeilBrown [Sat, 18 May 2019 02:21:55 +0000 (12:21 +1000)]
oceani-tests: delay valgrind tests

As valgrind can be slow, separate the valgrind tests out to
after the main tests.  That was failing that can be found quickly
are found quickly.

Signed-off-by: NeilBrown <neil@brown.name>
4 years agooceani: fix indent for Xval propagate
NeilBrown [Sat, 18 May 2019 00:49:19 +0000 (10:49 +1000)]
oceani: fix indent for Xval propagate

this is double-indented.  I wonder why.

Signed-off-by: NeilBrown <neil@brown.name>
4 years agooceani-tests: test code that has been printed
NeilBrown [Sat, 18 May 2019 00:11:43 +0000 (10:11 +1000)]
oceani-tests: test code that has been printed

Test that the printed code actually works, as well as being re-printable.
Also simplify the messages so they don't use as much space
and fix a typo "exit1" -> "exit 1"

Signed-off-by: NeilBrown <neil@brown.name>
4 years agooceani: print pre-declared constants when printing program.
NeilBrown [Sat, 18 May 2019 00:01:30 +0000 (10:01 +1000)]
oceani: print pre-declared constants when printing program.

The program is no more than just an 'exec'.  We need to print
the declarations too. As yet, there are only constants.

Signed-off-by: NeilBrown <neil@brown.name>
4 years agoscanner: handle missing newline at EOF
NeilBrown [Fri, 17 May 2019 13:31:48 +0000 (23:31 +1000)]
scanner: handle missing newline at EOF

If there is no newline at EOF, we can see EOF immediately after
a valid symbol.  This can lead to calling close_token() when
state->node is NULL, which crashes.
The code in close_token() only makes sense if state->node is still the
same as token->node.  If it isn't, the token must be at the very end of
its code-node, so a different calculation is needed.

This avoids the NULL deref.

Signed-off-by: NeilBrown <neil@brown.name>
4 years agooceani: add the option for "const" sections
NeilBrown [Mon, 13 May 2019 06:17:12 +0000 (16:17 +1000)]
oceani: add the option for "const" sections

These are introduced with "const" and can define one or
more constants - on one line with ';' or multiple lines
with {} or :

Signed-off-by: NeilBrown <neil@brown.name>
4 years agooceani: allow list of declarations as top level structure
NeilBrown [Sun, 12 May 2019 05:03:27 +0000 (15:03 +1000)]
oceani: allow list of declarations as top level structure

We can have constants, types, etc at top level, as well
a program

Signed-off-by: NeilBrown <neil@brown.name>
4 years agooceani: mark code that doesn't need testing.
NeilBrown [Sat, 11 May 2019 01:41:59 +0000 (11:41 +1000)]
oceani: mark code that doesn't need testing.

Some code is included to check and report impossible
conditions.  Failure to exercise this code shouldn't
be seen as a failure of test coverage.
So mark such code as //NOTEST and exclude it
from statistics.

This brings test coverage up to 93%

Signed-off-by: NeilBrown <neil@brown.name>
4 years agooceani: move "complex types" earlier.
NeilBrown [Sat, 11 May 2019 01:30:30 +0000 (11:30 +1000)]
oceani: move "complex types" earlier.

I think this fits better before "language elements",
as part of "data structures".  Nearly everything it needs
is defined by there.

Signed-off-by: NeilBrown <neil@brown.name>
4 years agooceani: fix valgrind-reported errors.
NeilBrown [Fri, 10 May 2019 10:25:01 +0000 (20:25 +1000)]
oceani: fix valgrind-reported errors.

The test suite wasn't running valgrind in a useful way, and wasn't
actually reporting errors.
So fix it to report leaks and unfreed memory, and then fix the
bugs that it finds.

Signed-off-by: NeilBrown <neil@brown.name>
4 years agooceani tests: check for conflicts.
NeilBrown [Thu, 9 May 2019 12:05:23 +0000 (22:05 +1000)]
oceani tests: check for conflicts.

If the grammar, when analysed as LR1, contains conflicts, then
don't proceed until they are resolved.

Signed-off-by: NeilBrown <neil@brown.name>
4 years agoparsergen: don't report expected shift/reduce conflicts.
NeilBrown [Thu, 9 May 2019 11:59:07 +0000 (21:59 +1000)]
parsergen: don't report expected shift/reduce conflicts.

If a NEWLINE is being treated as a grammar symbol which will sometimes
force a reduction, then an apparent SHIFT/REDUCE conflict that could be
resolved to REDUCE by a newline and would default to shift, is expected
and not really a conflict.  So don't report those.

Signed-off-by: NeilBrown <neil@brown.name>
4 years agooceani: add conditional expression
NeilBrown [Thu, 9 May 2019 11:13:16 +0000 (21:13 +1000)]
oceani: add conditional expression

[value] if [condition] else [alternate value]

is a conditional expression which will only evaluate and return one of
the two values, depending on the condition.
This has lowest precedence of all expressions.

Signed-off-by: NeilBrown <neil@brown.name>
4 years agooceani: add "and then" and "or else"
NeilBrown [Wed, 8 May 2019 10:50:30 +0000 (20:50 +1000)]
oceani: add "and then" and "or else"

"and then" and "or else" are short-circuit versions
of "and" and "or" - they only evaluate second arg if it
is needed.

Signed-off-by: NeilBrown <neil@brown.name>
4 years agooceani-tests: assorted more tests.
NeilBrown [Wed, 8 May 2019 07:22:53 +0000 (17:22 +1000)]
oceani-tests: assorted more tests.

Assorted tests chosen after examining lines of
code that were not being run.
This pushes the coverage over 90% !!!

Also a bugfix - test suites are good !!!

Signed-off-by: NeilBrown <neil@brown.name>
4 years agooceani-tests: add tests for various type errors.
NeilBrown [Mon, 6 May 2019 08:54:40 +0000 (18:54 +1000)]
oceani-tests: add tests for various type errors.

Now we have test coverage for all tok_err and type_err
calls, and have exceeded 85% total.

Signed-off-by: NeilBrown <neil@brown.name>
4 years agooceani-tests: add tests for various token-errors
NeilBrown [Mon, 6 May 2019 07:57:05 +0000 (17:57 +1000)]
oceani-tests: add tests for various token-errors

This pushes our coverage over 80%

Signed-off-by: NeilBrown <neil@brown.name>
4 years agooceani-tests: add tests for error cases.
NeilBrown [Mon, 6 May 2019 07:38:17 +0000 (17:38 +1000)]
oceani-tests: add tests for error cases.

Only one test so far, but this infrastructure will be useful
for others.

Signed-off-by: NeilBrown <neil@brown.name>
4 years agooceani-tests: add tests for bad command line args.
NeilBrown [Sun, 5 May 2019 03:25:39 +0000 (13:25 +1000)]
oceani-tests: add tests for bad command line args.

Ensuring the correct behaviour with bad command line args
helps increase coverage.

Signed-off-by: NeilBrown <neil@brown.name>
4 years agomdcode: indent must start with a TAB
NeilBrown [Sun, 5 May 2019 03:18:46 +0000 (13:18 +1000)]
mdcode: indent must start with a TAB

For Makefiles, indent must start with a TAB.  So
insert a TAB if there are 8 or more spaces of indent.

Signed-off-by: NeilBrown <neil@brown.name>
4 years agooceani-tests: add test suite.
NeilBrown [Sun, 5 May 2019 01:56:59 +0000 (11:56 +1000)]
oceani-tests: add test suite.

This is the beginning of a test suite.  It guards
against errors, memory leaks, etc.

Signed-off-by: NeilBrown <neil@brown.name>
4 years agomdcode: don't include blank lines at end of section.
NeilBrown [Sun, 5 May 2019 01:51:08 +0000 (11:51 +1000)]
mdcode: don't include blank lines at end of section.

completely blank lines (no indent) at the end of a section
should not be considered part of that section, but rather
separation between this and the next section.
So don't include them in the code.

When a section is used, for example, as sample output to
test against actual output in a test suit, having the
stray blank at the end can be problematic.

Signed-off-by: NeilBrown <neil@brown.name>
4 years agomdcode: allow a specific section to be extracted.
NeilBrown [Sun, 5 May 2019 01:38:22 +0000 (11:38 +1000)]
mdcode: allow a specific section to be extracted.

Previously, the md2c tool only extracted "File:" sections.
Sometimes we need more control, so allow a second argument
to identify a single section to be extracted.  This will
be written to stdout.

Signed-off-by: NeilBrown <neil@brown.name>
4 years agooceani: add array type.
NeilBrown [Wed, 1 May 2019 09:01:48 +0000 (19:01 +1000)]
oceani: add array type.

I can now declare arrays and access or assign
members there-of.

Signed-off-by: NeilBrown <neil@brown.name>
4 years agooceani: prepare for adding new types with new syntax.
NeilBrown [Wed, 1 May 2019 08:22:38 +0000 (18:22 +1000)]
oceani: prepare for adding new types with new syntax.

1/ 'Variable' now returns an 'exec', so other things returning
   and 'exec' can be the LHS of an assignment.
2/ provide forward-decls of generic functions so that
   new type functions can use them to work with
   components of the types.

Signed-off-by: NeilBrown <neil@brown.name>
4 years agooceani: add "remainder" operator.
NeilBrown [Wed, 1 May 2019 08:13:21 +0000 (18:13 +1000)]
oceani: add "remainder" operator.

a % b
treats a and b as integers, does a division, and reports
the remainder.

Signed-off-by: NeilBrown <neil@brown.name>
4 years agooceani: add some support for lvalues.
NeilBrown [Tue, 30 Apr 2019 06:24:09 +0000 (16:24 +1000)]
oceani: add some support for lvalues.

New linterp_exec() returns an lvalue, (pointer to value)
and individual cases can provide either an lvalue or an rvalue.

This means the RHS of assignment might not be a simple variable,
so we leave the "v = v->merged" part to the "interp exec" for Xvar.
Also, linterp_exec() might return NULL, so be careful.
This might happen, e.g., when an array index is out of bounds.

Signed-off-by: NeilBrown <neil@brown.name>
4 years agooceani: fix up test program.
NeilBrown [Tue, 30 Apr 2019 06:23:21 +0000 (16:23 +1000)]
oceani: fix up test program.

The messages being printed didn't make sense - weird.

Signed-off-by: NeilBrown <neil@brown.name>
4 years agooceani: delay constant assignment test to type analysis.
NeilBrown [Tue, 30 Apr 2019 04:00:03 +0000 (14:00 +1000)]
oceani: delay constant assignment test to type analysis.

When we get more complex types, it will be easier to
guard against assigning to a constant during type analysis.

Signed-off-by: NeilBrown <neil@brown.name>
4 years agooceani: allow a type to control how it is printed.
NeilBrown [Sun, 28 Apr 2019 04:08:23 +0000 (14:08 +1000)]
oceani: allow a type to control how it is printed.

this is needed for e.g. printing the program when there
are declared types.

Signed-off-by: NeilBrown <neil@brown.name>
4 years agooceani: delegate type compatability to type.
NeilBrown [Sun, 28 Apr 2019 03:45:26 +0000 (13:45 +1000)]
oceani: delegate type compatability to type.

Rename vtype_compat to type_compat, and have it
call the ->compat function if the required type
provides it.
Otherwise require type equivalence.

Signed-off-by: NeilBrown <neil@brown.name>
4 years agooceani: introduce type->prepare()
NeilBrown [Sun, 28 Apr 2019 03:10:15 +0000 (13:10 +1000)]
oceani: introduce type->prepare()

During parsing and analysis we don't want to "init" a
variable as that can allocate memory - when we add arrays,
we might not know yet how much memory.

So introduce 'prepare' to prepare a value - such that calling
free on it will work - without allocating.

'init' is then called when a variable is declared - unless something
is assigned to it instead.

Signed-off-by: NeilBrown <neil@brown.name>
4 years agooceani: disallow assignment if no 'dup' operation.
NeilBrown [Sun, 28 Apr 2019 01:10:03 +0000 (11:10 +1000)]
oceani: disallow assignment if no 'dup' operation.

'dup' is needed for assignment, so if it isn't present
it must be forbidden.

Signed-off-by: NeilBrown <neil@brown.name>
4 years agooceani: allow a variable to be declared with no value
NeilBrown [Sun, 28 Apr 2019 00:40:12 +0000 (10:40 +1000)]
oceani: allow a variable to be declared with no value

 var:type

is now allowed.  "var:" is not - there must be either a
type or an initial value.

If only the type is given, the type's 'init' function
is called to provide an initial value.

Signed-off-by: NeilBrown <neil@brown.name>