The Cxx Lexer tokenizes an input stream such that
a Parser can build a parse tree out of it. The
appearence of comments in the input stream has to
be transparent as far as this analysis is concerned.
However, comments should be retained for a number
of reasons:
* they may contain metainfo (documentation, say)
that is to be used in later AST processing
* they may be used in source-to-source generation
Right now the lexer uses some heuristics to
'tokenize' comments and retain them in a way that
allows the parser to attach them to a declaration
if it finds one. While this works reasonably well
for the typical use case of document generation,
it should be enhanced for the general case.
The first step may be to run the lexer in a mode
that returns *only* comment tokens. That would
actually be enough to correctly deal with cross-
referenced code a la SXR. |