The Preprocessor and tokens

[Henry Spencer]

In article <3137 at wyse.wyse.com> cmills at wyse.wyse.com () writes:
>...I was under the impression
>that the preprocessor and the compiler agreed on what constituted a
>token... 

Unfortunately, no.  There is a separate notion of a "preprocessing token",
which is like a real token in a lot of ways but has a much more generous
syntax for numbers.  This is what you're running afoul of.

The problem is that there are demented people who want to use vile and
disgusting perversions like token concatenation to actually put numbers
together out of pieces at preprocessing time.  The lexical syntax of C
numbers is really ugly, and trying to produce a lexical definition of a
valid *piece* of a number is agonizing.  X3J11 therefore came up with
the notion of a "preprocessing number" which covers anything vaguely
number-like, and stipulated full lexical validation of numbers only
after preprocessing.
-- 
"[Some people] positively *wish* to     | Henry Spencer @ U of Toronto Zoology
believe ill of the modern world."-R.Peto|  henry at zoo.toronto.edu  utzoo!henry