[clang] Only propagate AtStartOfLine when expanding token stream and hit the end of TokenLexer

[yronglin]

This patch fix a bug introduced by #107168.

Consider the following code:

#define str(s) # s 
#define xstr(s) str(s) 
#define INCFILE(n) vers ## n 

include xstr(INCFILE(2).h) 

The period has an unexpected AtStartOfLine flag when it's lexed from TokenLexer, after phase 4, we expected include "vers2.h", but got include "vers2 .h", an unexpected whitespace occurred before ..

We only need to propagate AtStartOfLine flag when current TokenLexer expanding an token stream.

No new test added because this issue was covered by clang/test/Preprocessor/c99-6_10_3_4_p6.c.

[llvmbot]

@llvm/pr-subscribers-clang

Author: None (yronglin)

Changes

This patch fix a bug introduced by #107168.

Consider the following code:

#define str(s) # s 
#define xstr(s) str(s) 
#define INCFILE(n) vers ## n 

include xstr(INCFILE(2).h) 

The period has an unexpected AtStartOfLine flag when it's lexed from TokenLexer, after phase 4, we expected include "vers2.h", but got include "vers2 .h", an unexpected whitespace occurred before ..

We only need to propagate AtStartOfLine flag when current TokenLexer expanding an token stream.

---

Full diff: https://github.com/llvm/llvm-project/pull/173052.diff

1 Files Affected:

• (modified) clang/lib/Lex/TokenLexer.cpp (+1-1)

diff --git a/clang/lib/Lex/TokenLexer.cpp b/clang/lib/Lex/TokenLexer.cpp
index e9531ee9794d2..ce072de79b6af 100644
--- a/clang/lib/Lex/TokenLexer.cpp
+++ b/clang/lib/Lex/TokenLexer.cpp
@@ -632,7 +632,7 @@ bool TokenLexer::Lex(Token &Tok) {
     //
     // The 'extern' token should has 'StartOfLine' flag when current TokenLexer
     // exits and propagate line start/leading space info.
-    if (isLexingCXXModuleDirective()) {
+    if (!Macro && isLexingCXXModuleDirective()) {
       AtStartOfLine = true;
       setLexingCXXModuleDirective(false);
     }

[yronglin]

Hmm, it's weird to me why this issue was not catched in premerge CI. I have verified this fix in my local.

[dyung]

Hmm, it's weird to me why this issue was not catched in premerge CI. I have verified this fix in my local.

The failure seems to be intermittently happening for some reason, at least on one of my bots:
https://lab.llvm.org/buildbot/#/builders/144

[ilovepi]

I tested this PR on our CI. it still seems to fail. https://ci.chromium.org/ui/p/fuchsia/builders/toolchain.ci.shadow/clang-mac-arm64/b8695027857614028065/overview

the input properties on https://ci.chromium.org/ui/p/fuchsia/builders/toolchain.ci.shadow/clang-mac-arm64/b8695027857614028065/infra show that its using your PR branch.

[yronglin]

https://lab.llvm.org/buildbot/#/builders/144

The failure and current PR is not the same issue. This PR fixes extra whitespace before token. I think I have found the root cause of the crash.