Skip to content

Scanner(unification) #782

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
AdamSpeight2008 wants to merge 12 commits into from
Closed

Scanner(unification) #782

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
3595ec9
Unifying two methods.
AdamSpeight2008 Feb 22, 2015
a09f808
Revert "Unifying two methods."
AdamSpeight2008 Feb 22, 2015
18dbd45
Revert "Revert "Unifying two methods.""
AdamSpeight2008 Feb 22, 2015
c66e09c
ignore
AdamSpeight2008 Feb 22, 2015
955eb58
Difference in internals
AdamSpeight2008 Feb 22, 2015
ad65bf6
Fix my spelling error in string value
AdamSpeight2008 Feb 22, 2015
dd49734
Typo
AdamSpeight2008 Feb 22, 2015
a903875
Reverting to a previous state
AdamSpeight2008 Feb 23, 2015
6c321f6
Revert "ignore"
AdamSpeight2008 Feb 23, 2015
6242893
Reunify
AdamSpeight2008 Feb 23, 2015
f9696a0
Refactoring code
AdamSpeight2008 Feb 26, 2015
a28bdcd
Refactoring
AdamSpeight2008 Feb 27, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 8 additions & 7 deletions src/Compilers/VisualBasic/Portable/Analysis/MissingRuntimeMemberDiagnosticHelper.vb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,13 +10,14 @@ Namespace Microsoft.CodeAnalysis

' Details on these types and the feature name which is displayed in the diagnostic
' for those items missing in VB Core compilation.
Private ReadOnly metadataNames As New Dictionary(Of String, String) From {
{"Microsoft.VisualBasic.CompilerServices.Operators", "Late binding"},
{"Microsoft.VisualBasic.CompilerServices.NewLateBinding", "Late binding"},
{"Microsoft.VisualBasic.CompilerServices.LikeOperator", "Like operator"},
{"Microsoft.VisualBasic.CompilerServices.ProjectData", "Unstructured exception handling"},
{"Microsoft.VisualBasic.CompilerServices.ProjectData.CreateProjectError", "Unstructured exception handling"}
}
Private ReadOnly metadataNames As New Dictionary(Of String, String) From
{
{"Microsoft.VisualBasic.CompilerServices.Operators", "Late binding"},
{"Microsoft.VisualBasic.CompilerServices.NewLateBinding", "Late binding"},
{"Microsoft.VisualBasic.CompilerServices.LikeOperator", "Like operator"},
{"Microsoft.VisualBasic.CompilerServices.ProjectData", "Unstructured exception handling"},
{"Microsoft.VisualBasic.CompilerServices.ProjectData.CreateProjectError", "Unstructured exception handling"}
}

Friend Function GetDiagnosticForMissingRuntimeHelper(typename As String, membername As String, embedVBCoreRuntime As Boolean) As DiagnosticInfo
Dim diag As DiagnosticInfo
Expand Down
1 change: 1 addition & 0 deletions src/Compilers/VisualBasic/Portable/BasicCodeAnalysis.vbproj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -945,6 +945,7 @@
<Compile Include="VisualBasicCompilationOptions.vb" />
<Compile Include="VisualBasicExtensions.vb" />
<Compile Include="VisualBasicParseOptions.vb" />
<Compile Include="_Exts\Char.vb" />

@AnthonyDGreen AnthonyDGreen Feb 27, 2015

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is _Exts? Did this folder exist beforehand?

@AdamSpeight2008 AdamSpeight2008 Feb 27, 2015

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nope, I added it, it's name such to make it obvious is not a blessed namespace.
_Exts\Char

<EmbeddedResource Include="Symbols\EmbeddedSymbols\Embedded.vb" />
<EmbeddedResource Include="Symbols\EmbeddedSymbols\InternalXmlHelper.vb" />
<EmbeddedResource Include="Symbols\EmbeddedSymbols\VbCoreSourceText.vb" />
Expand Down
17 changes: 3 additions & 14 deletions src/Compilers/VisualBasic/Portable/Parser/ParseReportError.vb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -113,14 +113,8 @@ Namespace Microsoft.CodeAnalysis.VisualBasic.Syntax.InternalSyntax
End Function

Private Function ReportModifiersOnStatementError(errorId As ERRID, attributes As SyntaxList(Of AttributeListSyntax), modifiers As SyntaxList(Of KeywordSyntax), keyword As KeywordSyntax) As KeywordSyntax
If modifiers.Any Then
keyword = keyword.AddLeadingSyntax(modifiers.Node, errorId)
End If

If attributes.Any Then
keyword = keyword.AddLeadingSyntax(attributes.Node, errorId)
End If

If modifiers.Any Then keyword = keyword.AddLeadingSyntax(modifiers.Node, errorId)
If attributes.Any Then keyword = keyword.AddLeadingSyntax(attributes.Node, errorId)
Return keyword
End Function

Expand All @@ -142,12 +136,7 @@ Namespace Microsoft.CodeAnalysis.VisualBasic.Syntax.InternalSyntax
' // A FEATUREID_* constant defined in errors.inc
' // the string for the version that /LangVersion is targeting
' .Parser::ReportSyntaxErrorForLanguageFeature( [ unsigned Errid ] [ _In_ Token* Start ] [ unsigned Feature ] [ _In_opt_z_ const WCHAR* wszVersion ] )
Private Sub ReportSyntaxErrorForLanguageFeature(
Errid As ERRID,
Start As SyntaxToken,
Feature As UInteger,
wszVersion As String
)
Private Sub ReportSyntaxErrorForLanguageFeature( Errid As ERRID, Start As SyntaxToken, Feature As UInteger, wszVersion As String)
#If UNDONE Then 'davidsch
m_ErrorCount += 1

Expand Down
87 changes: 32 additions & 55 deletions src/Compilers/VisualBasic/Portable/Parser/ParseScan.vb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,13 +21,7 @@ Namespace Microsoft.CodeAnalysis.VisualBasic.Syntax.InternalSyntax
' The scanner scans the fist token of a statement differently with regard to trivia. It we peek past the EOL and then get the token.
' The token may not have the correct trivia attached to it when we get it. The solution is to attach the state to the token so we
' know if we peeked the token in the first token of new statement state or the next token of a statement state.

If PeekToken(i).Kind = SyntaxKind.StatementTerminatorToken Then
If PeekToken(i + 1).Kind <> SyntaxKind.EmptyToken Then
Return True
End If
End If
Return False
Return (PeekToken(i).Kind = SyntaxKind.StatementTerminatorToken) AndAlso (PeekToken(i + 1).Kind <> SyntaxKind.EmptyToken)
End Function

'TODO - This is really peekToken skipping optional statementterminator
Expand Down Expand Up @@ -128,38 +122,31 @@ Namespace Microsoft.CodeAnalysis.VisualBasic.Syntax.InternalSyntax
'Parser::BeginsGeneric // A generic is signified by '(' [tkStatementTerminator] tkOF

Private Function BeginsGeneric(Optional nonArrayName As Boolean = False, Optional allowGenericsWithoutOf As Boolean = False) As Boolean

If CurrentToken.Kind = SyntaxKind.OpenParenToken Then

If nonArrayName Then
Return True
End If

Dim t = PeekPastStatementTerminator()

If t.Kind = SyntaxKind.OfKeyword Then
Return True
ElseIf allowGenericsWithoutOf Then
' // To enable a better user experience in some common generics'
' // error scenarios, we special case foo(Integer) and
' // foo(Integer, garbage).
' //
' // "(Integer" indicates possibly type parameters with missing "of",
' // but not "(Integer." and "Integer!" because they could possibly
' // imply qualified names or expressions. Also note that "Integer :="
' // could imply named arguments. Here "Integer" is just an example,
' // it could be any intrinsic type.
' //
If SyntaxFacts.IsPredefinedTypeOrVariant(t.Kind) Then
Select Case PeekToken(2).Kind
Case SyntaxKind.CloseParenToken, SyntaxKind.CommaToken
Return True
End Select
End If
If CurrentToken.Kind <> SyntaxKind.OpenParenToken Then Return False
If nonArrayName Then Return True
Dim t = PeekPastStatementTerminator()

If t.Kind = SyntaxKind.OfKeyword Then Return True
If allowGenericsWithoutOf Then
' // To enable a better user experience in some common generics'
' // error scenarios, we special case foo(Integer) and
' // foo(Integer, garbage).
' //
' // "(Integer" indicates possibly type parameters with missing "of",
' // but not "(Integer." and "Integer!" because they could possibly
' // imply qualified names or expressions. Also note that "Integer :="
' // could imply named arguments. Here "Integer" is just an example,
' // it could be any intrinsic type.
' //
If SyntaxFacts.IsPredefinedTypeOrVariant(t.Kind) Then
Select Case PeekToken(2).Kind
Case SyntaxKind.CloseParenToken, SyntaxKind.CommaToken
Return True
End Select
End If

End If


Return False
End Function

Expand Down Expand Up @@ -259,9 +246,8 @@ Namespace Microsoft.CodeAnalysis.VisualBasic.Syntax.InternalSyntax
Exit While
End If

If IsTokenOrKeyword(CurrentToken, resyncTokens) Then
Exit While
End If
If IsTokenOrKeyword(CurrentToken, resyncTokens) Then Exit While


skippedTokens.Add(CurrentToken)
GetNextToken(state)
Expand All @@ -285,18 +271,14 @@ Namespace Microsoft.CodeAnalysis.VisualBasic.Syntax.InternalSyntax
Private Function ResyncAndConsumeStatementTerminator() As SyntaxList(Of SyntaxToken)
Dim skippedTokens = Me._pool.Allocate(Of SyntaxToken)()

While CurrentToken.Kind <> SyntaxKind.EndOfFileToken AndAlso
CurrentToken.Kind <> SyntaxKind.StatementTerminatorToken
While (CurrentToken.Kind <> SyntaxKind.EndOfFileToken) AndAlso (CurrentToken.Kind <> SyntaxKind.StatementTerminatorToken)

skippedTokens.Add(CurrentToken)
GetNextToken(ScannerState.VB)
End While

If CurrentToken.Kind = SyntaxKind.StatementTerminatorToken Then
If CurrentToken.HasLeadingTrivia Then
skippedTokens.Add(CurrentToken)
End If

If CurrentToken.HasLeadingTrivia Then skippedTokens.Add(CurrentToken)
GetNextToken(ScannerState.VB)
End If

Expand Down Expand Up @@ -368,10 +350,8 @@ Namespace Microsoft.CodeAnalysis.VisualBasic.Syntax.InternalSyntax
Private Function TryEatNewLineIfFollowedBy(kind As SyntaxKind) As Boolean
Debug.Assert(CanUseInTryGetToken(kind))

If NextLineStartsWith(kind) Then
'Add trivia to the token that has been peeked on next line
Return TryEatNewLine()
End If
If NextLineStartsWith(kind) Then Return TryEatNewLine() 'Add trivia to the token that has been peeked on next line

Return False
End Function

Expand All @@ -387,9 +367,7 @@ Namespace Microsoft.CodeAnalysis.VisualBasic.Syntax.InternalSyntax
Private Function TryEatNewLineIfNotFollowedBy(kind As SyntaxKind) As Boolean
Debug.Assert(CanUseInTryGetToken(kind))

If Not NextLineStartsWith(kind) Then
Return TryEatNewLine()
End If
If Not NextLineStartsWith(kind) Then Return TryEatNewLine()
Return False
End Function

Expand All @@ -408,10 +386,9 @@ Namespace Microsoft.CodeAnalysis.VisualBasic.Syntax.InternalSyntax
If CurrentToken.IsEndOfLine Then
Dim nextToken = PeekToken(1)

If nextToken.Kind = kind Then
Return True
If nextToken.Kind = kind Then Return True

ElseIf nextToken.Kind = SyntaxKind.IdentifierToken Then
if nextToken.Kind = SyntaxKind.IdentifierToken Then
Dim contextualKind As SyntaxKind = Nothing
If TryIdentifierAsContextualKeyword(nextToken, contextualKind) AndAlso contextualKind = kind Then
Return True
Expand Down
36 changes: 12 additions & 24 deletions src/Compilers/VisualBasic/Portable/Parser/ParserFeature.vb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,30 +51,18 @@ Namespace Microsoft.CodeAnalysis.VisualBasic.Syntax.InternalSyntax
<Extension>
Friend Function GetResourceId(feature As Feature) As ERRID
Select Case feature
Case Feature.AutoProperties
Return ERRID.FEATURE_AutoProperties
Case Feature.LineContinuation
Return ERRID.FEATURE_LineContinuation
Case Feature.StatementLambdas
Return ERRID.FEATURE_StatementLambdas
Case Feature.CoContraVariance
Return ERRID.FEATURE_CoContraVariance
Case Feature.CollectionInitializers
Return ERRID.FEATURE_CollectionInitializers
Case Feature.SubLambdas
Return ERRID.FEATURE_SubLambdas
Case Feature.ArrayLiterals
Return ERRID.FEATURE_ArrayLiterals
Case Feature.AsyncExpressions
Return ERRID.FEATURE_AsyncExpressions
Case Feature.Iterators
Return ERRID.FEATURE_Iterators
Case Feature.GlobalNamespace
Return ERRID.FEATURE_GlobalNamespace
Case Feature.NullPropagatingOperator
Return ERRID.FEATURE_NullPropagatingOperator
Case Feature.NameOfExpressions
Return ERRID.FEATURE_NameOfExpressions
Case Feature.AutoProperties : Return ERRID.FEATURE_AutoProperties
Case Feature.LineContinuation : Return ERRID.FEATURE_LineContinuation
Case Feature.StatementLambdas : Return ERRID.FEATURE_StatementLambdas
Case Feature.CoContraVariance : Return ERRID.FEATURE_CoContraVariance
Case Feature.CollectionInitializers : Return ERRID.FEATURE_CollectionInitializers
Case Feature.SubLambdas : Return ERRID.FEATURE_SubLambdas
Case Feature.ArrayLiterals : Return ERRID.FEATURE_ArrayLiterals
Case Feature.AsyncExpressions : Return ERRID.FEATURE_AsyncExpressions
Case Feature.Iterators : Return ERRID.FEATURE_Iterators
Case Feature.GlobalNamespace : Return ERRID.FEATURE_GlobalNamespace
Case Feature.NullPropagatingOperator : Return ERRID.FEATURE_NullPropagatingOperator
Case Feature.NameOfExpressions : Return ERRID.FEATURE_NameOfExpressions
Case Else
Throw ExceptionUtilities.UnexpectedValue(feature)
End Select
Expand Down
32 changes: 8 additions & 24 deletions src/Compilers/VisualBasic/Portable/PredefinedPreprocessorSymbols.vb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,16 +37,12 @@ Namespace Microsoft.CodeAnalysis.VisualBasic
''' <param name="symbols">An ImmutableArray of KeyValue pairs representing existing symbols.</param>
''' <returns>Array of symbols that include VBC_VER and TARGET.</returns>
Public Function AddPredefinedPreprocessorSymbols(kind As OutputKind, symbols As ImmutableArray(Of KeyValuePair(Of String, Object))) As ImmutableArray(Of KeyValuePair(Of String, Object))
If Not kind.IsValid Then
Throw New ArgumentOutOfRangeException("kind")
End If
If Not kind.IsValid Then Throw New ArgumentOutOfRangeException("kind")

Const CompilerVersionSymbol = "VBC_VER"
Const TargetSymbol = "TARGET"

If symbols.IsDefault Then
symbols = ImmutableArray(Of KeyValuePair(Of String, Object)).Empty
End If
If symbols.IsDefault Then symbols = ImmutableArray(Of KeyValuePair(Of String, Object)).Empty

If symbols.FirstOrDefault(Function(entry) IdentifierComparison.Equals(entry.Key, CompilerVersionSymbol)).Key Is Nothing Then
' This number should always line up with the current version of the compilerString
Expand All @@ -62,24 +58,12 @@ Namespace Microsoft.CodeAnalysis.VisualBasic

Friend Function GetTargetString(kind As OutputKind) As String
Select Case kind
Case OutputKind.ConsoleApplication
Return "exe"

Case OutputKind.DynamicallyLinkedLibrary
Return "library"

Case OutputKind.NetModule
Return "module"

Case OutputKind.WindowsApplication
Return "winexe"

Case OutputKind.WindowsRuntimeApplication
Return "appcontainerexe"

Case OutputKind.WindowsRuntimeMetadata
Return "winmdobj"

Case OutputKind.ConsoleApplication : Return "exe"
Case OutputKind.DynamicallyLinkedLibrary : Return "library"
Case OutputKind.NetModule : Return "module"
Case OutputKind.WindowsApplication : Return "winexe"
Case OutputKind.WindowsRuntimeApplication : Return "appcontainerexe"
Case OutputKind.WindowsRuntimeMetadata : Return "winmdobj"
Case Else
Throw ExceptionUtilities.UnexpectedValue(kind)
End Select
Expand Down
Loading