Skip to content

Commit

Permalink
[lex.pptoken] Separate out placeholder keywords
Browse files Browse the repository at this point in the history
The _preprocessing-token_ term is defined as one of another set of
grammar productions, with the three placeholder tokens standing out
as specific single tokens.

Moving the placeholder tokens into their own production simplifies
both the _preprocessing-token_ and _keyword_ grammar productions,
plus it guarantees a consistency for the set of placeholder between
both places.  IT also makes it easier for [cpp] to directly
cross-reference where needed.

Pedantically, the term placeholder token itself is problematic as
tokens are a phase 7 concern, where they become keywords.  Within
the earlier phases where these placeholders are used the correct
term is a preprocessing token, so strictly these should be
placeholder preprocessing tokens, which is a mouthful.  So renaming
the term to placeholder keyword, which is hopefully less of a
concern as there are no keywords in phase 3--6 to confuse with
phase 7 keywords.

Finally, clean up the informative list in text of the various kinds
of preprocessing tokens, rendering as a list in the source (one kind
per line) and adding a cross-reference to each such item.
  • Loading branch information
AlisdairM committed Nov 9, 2024
1 parent 13ba96c commit 9746dfc
Show file tree
Hide file tree
Showing 2 changed files with 40 additions and 26 deletions.
62 changes: 38 additions & 24 deletions source/lex.tex
Original file line number Diff line number Diff line change
Expand Up @@ -495,12 +495,10 @@
\indextext{token!preprocessing|(}%
\begin{bnf}
\nontermdef{preprocessing-token}\br
placeholder-keyword\br
header-name\br
import-keyword\br
module-keyword\br
export-keyword\br
identifier\br
pp-number\br
identifier\br
character-literal\br
user-defined-character-literal\br
string-literal\br
Expand All @@ -515,12 +513,15 @@
In this document,
glyphs are used to identify
elements of the basic character set\iref{lex.charset}.
The categories of preprocessing token are: header names,
placeholder tokens produced by preprocessing \tcode{import} and \tcode{module} directives
(\grammarterm{import-keyword}, \grammarterm{module-keyword}, and \grammarterm{export-keyword}),
identifiers, preprocessing numbers, character literals (including user-defined character
literals), string literals (including user-defined string literals), preprocessing
operators and punctuators, and single non-whitespace characters that do not lexically
The categories of preprocessing token are:
placeholder keywords\iref{lex.placeholder},
header names\iref{lex.header},
preprocessing numbers\iref{lex.ppnumber},
identifiers\iref{lex.name},
character literals\iref{lex.ccon} including user-defined character literals,
string literals\iref{lex.string} including user-defined string literals,
preprocessing operators and punctuators\iref{lex.operators}, and
single non-whitespace characters that do not lexically
match the other preprocessing token categories.
If a \unicode{0027}{apostrophe} or a \unicode{0022}{quotation mark} character
matches the last category, the program is ill-formed.
Expand Down Expand Up @@ -548,17 +549,6 @@
shall have the lexical form of a keyword, an identifier, a literal,
or an operator or punctuator.

\pnum
The \grammarterm{import-keyword} is produced
by processing an \keyword{import} directive\iref{cpp.import},
the \grammarterm{module-keyword} is produced
by preprocessing a \keyword{module} directive\iref{cpp.module}, and
the \grammarterm{export-keyword} is produced
by preprocessing either of the previous two directives.
\begin{note}
None has any observable spelling.
\end{note}

\pnum
If the input stream has been parsed into preprocessing tokens up to a
given character:
Expand Down Expand Up @@ -628,6 +618,32 @@
\end{example}
\indextext{token!preprocessing|)}

\rSec1[lex.placeholder]{Placeholder keywords}
\indextext{keyword!placeholder|(}%

\begin{bnf}
\nontermdef{placeholder-keyword}\br
import-keyword\br
module-keyword\br
export-keyword
\end{bnf}

\pnum
Placeholder keywords are produced by preprocessing \grammarterm{import}
and \grammarterm{module} directives.
The \grammarterm{import-keyword} is produced
by preprocessing an \keyword{import} directive\iref{cpp.import},
the \grammarterm{module-keyword} is produced
by preprocessing a \keyword{module} directive\iref{cpp.module}, and
the \grammarterm{export-keyword} is produced
by preprocessing either of the previous two directives.

\pnum
\begin{note}
Placeholder keywords do not have any observable spelling.
\end{note}
\indextext{keyword!placeholder|)}%

\rSec1[lex.header]{Header names}

\indextext{header!name|(}%
Expand Down Expand Up @@ -960,9 +976,7 @@
\begin{bnf}
\nontermdef{keyword}\br
\textnormal{any identifier listed in \tref{lex.key}}\br
\grammarterm{import-keyword}\br
\grammarterm{module-keyword}\br
\grammarterm{export-keyword}
\grammarterm{placeholder-keyword}
\end{bnf}

\pnum
Expand Down
4 changes: 2 additions & 2 deletions source/preprocessor.tex
Original file line number Diff line number Diff line change
Expand Up @@ -796,7 +796,7 @@
\pnum
The \tcode{module} and \tcode{export} (if it exists) preprocessing tokens
are replaced by the \grammarterm{module-keyword} and
\grammarterm{export-keyword} preprocessing tokens respectively.
\grammarterm{export-keyword} placeholder keywords respectively\iref{lex.placeholder}.
\begin{note}
This makes the line no longer a directive
so it is not removed at the end of phase 4.
Expand Down Expand Up @@ -854,7 +854,7 @@
In all three forms of \grammarterm{pp-import},
the \tcode{import} and \tcode{export} (if it exists) preprocessing tokens
are replaced by the \grammarterm{import-keyword} and
\grammarterm{export-keyword} preprocessing tokens respectively.
\grammarterm{export-keyword} placeholder keywords respectively\iref{lex.placeholder}.
\begin{note}
This makes the line no longer a directive
so it is not removed at the end of phase 4.
Expand Down

0 comments on commit 9746dfc

Please sign in to comment.