parsergen: move EOL handling out of shift()

shift() is called in several places, and in only one of those is there a
need to make NEWLINE to EOL.

Move the code out of shift() and instead make a second call to shift()
if shifting NEWLINE failed.  I think this make the code clearer.

Signed-off-by: NeilBrown <neil@brown.name>