ANSI C token set (including $ and @)

[Leendert van Doorn]

In article <11262 at haddock.ima.isc.com> karl at haddock.ima.isc.com writes:

>>anything using @ or $ (since they're not part of the C character set)
>
>That's what I thought, too, until I noticed that 3.1.7 in the May88 dpANS
>contains an example wherein @ and $ are scanned as separate preprocessing
>tokens.  The accompanying text does not mention whether or not this behavior
>is required of a conforming implementation.

This example is valid. Hence the behaviour is required of a conforming
implementation. This has to do with section 2.1.1.2 (phases of translation).
In phase 3 the source file is decomposed into preprocessing tokens, and
in phase 7 the preprocessor tokens are converted into (normal) tokens.
This allows @ and $ character to be part of the preprocessor token set,
but not to be part of the (normal) token set.

However, nowhere in the standard is the conversion of preprocessor tokens
to (normal) tokens described. This is an issue that should be clarified.

In the compiler I wrote the lexical analyser simply breaks up the input into
preprocessor tokens and these go (without any conversion) into the compilation
process. The later one will filter out illegal things like $ and @. (the
parser chokes and starts up an error recovery routine).

-- 
Leendert P. van Doorn 			   		 <leendert at cs.vu.nl>
Vrije Universiteit / Dept. of Maths. & Comp. Sc.
De Boelelaan 1081
1081 HV Amsterdam / The Netherlands			tel. +31 20 548 5302