Skip to content
12 changes: 6 additions & 6 deletions src/Compilers/VisualBasic/Portable/Scanner/Blender.vb
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
' Copyright (c) Microsoft. All Rights Reserved. Licensed under the Apache License, Version 2.0. See License.txt in the project root for license information.

'-----------------------------------------------------------------------------
' Contains the definition of the Scanner, which produces tokens from text
' Contains the definition of the Scanner, which produces tokens from text
'-----------------------------------------------------------------------------
Option Compare Binary
Option Strict On
Expand Down Expand Up @@ -139,7 +139,7 @@ Namespace Microsoft.CodeAnalysis.VisualBasic.Syntax.InternalSyntax
Return node.GetTrailingTrivia().Any(SyntaxKind.EndOfLineTrivia)
Case SyntaxKind.SingleLineIfStatement,
SyntaxKind.SingleLineElseClause
' Steer clear of single-line if's because they they have custom handling of statement
' Steer clear of single-line if's because they they have custom handling of statement
' terminators that may make it difficult to reuse sub-statements.
Return False
Case Else
Expand Down Expand Up @@ -301,7 +301,7 @@ Namespace Microsoft.CodeAnalysis.VisualBasic.Syntax.InternalSyntax
Private Shared Function ShouldCrumble(node As VisualBasicSyntaxNode) As Boolean
If TypeOf node Is StructuredTriviaSyntax Then
' Do not crumble into structured trivia content.
' we will not use any of the parts anyways and
' we will not use any of the parts anyways and
' evaluation of directives may go out of sync.
Return False
End If
Expand Down Expand Up @@ -516,14 +516,14 @@ Namespace Microsoft.CodeAnalysis.VisualBasic.Syntax.InternalSyntax
_currentToken = _currentToken.With(nextPreprocessorState)
End If

' this will discard any prefetched tokens, including current.
' this will discard any prefetched tokens, including current.
' We do not need them since we moved to completely new node.
MyBase.MoveToNextSyntaxNode()

TryPopNode()

' at this point, all three pointers (position in old tree, position in text, position in parser)
' and all preprocessor state (_currentPreprocessorState, _scannerPreprocessorState, _currentToken.PreprocessorState)
' and all preprocessor state (_currentPreprocessorState, _scannerPreprocessorState, _currentToken.PreprocessorState)
' should point to same position (in sync)
End Sub

Expand All @@ -537,7 +537,7 @@ Namespace Microsoft.CodeAnalysis.VisualBasic.Syntax.InternalSyntax
' just move forward
_lineBufferOffset = _lineBufferOffset + _curNodeLength

' this will just verify that we do not have any prefetched tokens, including current.
' this will just verify that we do not have any prefetched tokens, including current.
' otherwise advancing linebuffer offeset could go out of sync with token stream.
MyBase.MoveToNextSyntaxNodeInTrivia()

Expand Down
38 changes: 36 additions & 2 deletions src/Compilers/VisualBasic/Portable/Scanner/CharacterInfo.vb
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
' Copyright (c) Microsoft. All Rights Reserved. Licensed under the Apache License, Version 2.0. See License.txt in the project root for license information.

'-----------------------------------------------------------------------------
' Contains the definition of the Scanner, which produces tokens from text
' Contains the definition of the Scanner, which produces tokens from text
'-----------------------------------------------------------------------------
Option Strict On
Option Explicit On
Expand Down Expand Up @@ -339,7 +339,7 @@ Namespace Microsoft.CodeAnalysis.VisualBasic
End Function

''' <summary>
''' Creates a half width form Unicode character string.
''' Creates a half width form Unicode character string.
''' </summary>
''' <param name="text">The text representing the original identifier. This can be in full width or half width Unicode form. </param>
''' <returns>A string representing the text in a half width Unicode form.</returns>
Expand Down Expand Up @@ -488,4 +488,38 @@ Namespace Microsoft.CodeAnalysis.VisualBasic
End Function
End Class

Friend Module CharExts

<Runtime.CompilerServices.Extension>
Friend Function IsFrom(c As Char, c0 As Char, c1 As Char) As Boolean
Return (c = c0) OrElse (c = c1)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OrElse seems unnecessary here. OR with unconditional extra char compare is likely to be cheaper than deciding whether to do it or not.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@VSadov
The use of the OrElse adds only 2 IL bytes per a usage. So don't think that cost isn't too much.

End Function

<Runtime.CompilerServices.Extension>
Friend Function IsFrom(c As Char, c0 As Char, c1 As Char, c2 As Char) As Boolean
Return (c = c0) OrElse (c = c1) OrElse (c = c2)
End Function

<Runtime.CompilerServices.Extension>
Friend Function IsFrom(c As Char, c0 As Char, c1 As Char, c2 As Char, c3 As Char) As Boolean
Return (c = c0) OrElse (c = c1) OrElse (c = c2) OrElse (c = c3)
End Function

<Runtime.CompilerServices.Extension>
Friend Function IsntFrom(c As Char, c0 As Char, c1 As Char) As Boolean

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Prefer IsNotFrom over IsntFrom. In general we don't use contractions in the APIs.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So you are saying that there is occasions that you use contractions. The APIs isn't public.
Using the contraction Isnt clearly separates it from IsNot

Return (c <> c0) AndAlso (c <> c1)
End Function

<Runtime.CompilerServices.Extension>
Friend Function IsntFrom(c As Char, c0 As Char, c1 As Char, c2 As Char) As Boolean

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same

Return (c <> c0) AndAlso (c <> c1) AndAlso (c <> c2)
End Function

<Runtime.CompilerServices.Extension>
Friend Function IsntFrom(c As Char, c0 As Char, c1 As Char, c2 As Char, c3 As Char) As Boolean

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same

Return (c <> c0) AndAlso (c <> c1) AndAlso (c <> c2) AndAlso (c <> c3)
End Function

End Module

End Namespace
12 changes: 6 additions & 6 deletions src/Compilers/VisualBasic/Portable/Scanner/Directives.vb
Original file line number Diff line number Diff line change
Expand Up @@ -31,8 +31,8 @@ Namespace Microsoft.CodeAnalysis.VisualBasic.Syntax.InternalSyntax
Dim restorePoint = CreateRestorePoint()
Me._isScanningDirective = True

' since we do not have lookahead tokens, this just
' resets current token to _lineBufferOffset
' since we do not have lookahead tokens, this just
' resets current token to _lineBufferOffset
Me.GetNextTokenInState(ScannerState.VB)

Dim currentNonterminal = Me.GetCurrentSyntaxNode()
Expand All @@ -43,7 +43,7 @@ Namespace Microsoft.CodeAnalysis.VisualBasic.Syntax.InternalSyntax
Me.MoveToNextSyntaxNodeInTrivia()

' adjust current token to just after the node
' we need that in case we need to skip some disabled text
' we need that in case we need to skip some disabled text
'(yes we do tokenize disabled text for compatibility reasons)
Me.GetNextTokenInState(ScannerState.VB)

Expand Down Expand Up @@ -91,10 +91,10 @@ Namespace Microsoft.CodeAnalysis.VisualBasic.Syntax.InternalSyntax
disabledCode = SkipConditionalCompilationSection()
End If

' Here we add the directive and disabled text that follows it
' Here we add the directive and disabled text that follows it
' (if there is any) to the trivia list

' processing statement could add an error to it,
' processing statement could add an error to it,
' so we may need to rebuild the trivia node
If statement IsNot directiveTrivia Then
directiveTrivia = statement
Expand Down Expand Up @@ -507,7 +507,7 @@ Namespace Microsoft.CodeAnalysis.VisualBasic.Syntax.InternalSyntax
'//
'// Skip all text until the end of the current conditional section. This will also return the
'// span of text that was skipped
'//
'//
'//-------------------------------------------------------------------------------------------------

Private Function SkipConditionalCompilationSection() As SyntaxList(Of VisualBasicSyntaxNode)
Expand Down
5 changes: 2 additions & 3 deletions src/Compilers/VisualBasic/Portable/Scanner/KeywordTable.vb
Original file line number Diff line number Diff line change
Expand Up @@ -269,7 +269,7 @@ Namespace Microsoft.CodeAnalysis.VisualBasic.Syntax
Public kdIsQueryClause As Boolean

'// Note: "CanFollowExpr" says whether this token can come after an expression.
'// e.g. "Dim x = From i In <expression> Select i" is valid, therefore "Select" can come follow an expression.
'// e.g. "Dim x = From i In <expression> Select i" is valid, therefore "Select" can come follow an expression.
'// The complete list was discovered through the tool in vb\Language\Tools\VBGrammarAnalyzer\vbgrammar.html
'// If you add keywords, then make sure that they're added to the official language grammar, and re-run the tool.
Public kdCanFollowExpr As Boolean
Expand Down Expand Up @@ -304,9 +304,8 @@ Namespace Microsoft.CodeAnalysis.VisualBasic.Syntax

Private Shared Function EnsureHalfWidth(s As String) As String
Dim result As Char() = Nothing

For i As Integer = 0 To s.Length - 1
Dim ch = s(i)
Dim ch As Char = s(i)

If SyntaxFacts.IsFullWidth(ch) Then
ch = SyntaxFacts.MakeHalfWidth(ch)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,16 +16,16 @@ Imports Microsoft.CodeAnalysis.VisualBasic.Syntax
Namespace Microsoft.CodeAnalysis.VisualBasic.Syntax.InternalSyntax
'' The QuickTokenAccumulator is a small mini-tokenizer that may fail. It consumes characters and
'' eventually either decides that either it found a complete token (including the trivia on either
'' side), or it gives up and says that the full scanner should take an attempt. It also accumulates the
'' side), or it gives up and says that the full scanner should take an attempt. It also accumulates the
'' token into a character buffer and computes a hash code. The entire tokenization is done by a single
'' routine without any memory allocations to keep it very fast.
''
''
'' Currently it only handles two cases:
'' optional-whitespace keyword-or-identifier optional-whitespace
'' optional-whitespace single-char-punctuation optional-whitespace
''
''
'' where the whitespace does not include newlines.
'
''
'' The VB tokenization rules are complex, and care needs to be taken in constructing the quick
'' tokenization. For example "REM" begins a comment, so it can't be tokenized as a keyword or identifier.
'' Similar problems arise with multi-character punctuation tokens, which can have embedded spaces.
Expand Down Expand Up @@ -63,7 +63,7 @@ Namespace Microsoft.CodeAnalysis.VisualBasic.Syntax.InternalSyntax

'TODO: why : and ; are complex? (8th row, 3 and 4)

' The following table classifies the first &H180 Unicode characters.
' The following table classifies the first &H180 Unicode characters.
' R and r are marked as COMPLEX so that quick-scanning doesn't stop after "REM".
' # is marked complex as it may start directives.
' PERF: Use UShort instead of CharFlags so the compiler can use array literal initialization.
Expand Down Expand Up @@ -163,7 +163,7 @@ Namespace Microsoft.CodeAnalysis.VisualBasic.Syntax.InternalSyntax

' Attempt to scan a single token.
' If it succeeds, return True, and the characters, length, and hashcode of the token
' can be retrieved by other functions.
' can be retrieved by other functions.
' If it fails (the token is too complex), return False.
Public Function QuickScanToken(allowLeadingMultilineTrivia As Boolean) As QuickScanResult
Dim state As AccumulatorState = If(allowLeadingMultilineTrivia, AccumulatorState.InitialAllowLeadingMultilineTrivia, AccumulatorState.Initial)
Expand Down Expand Up @@ -330,7 +330,7 @@ Namespace Microsoft.CodeAnalysis.VisualBasic.Syntax.InternalSyntax

index += 1

'FNV-like hash should work here
'FNV-like hash should work here
'since these strings are short and mostly ASCII
hashCode = (hashCode Xor unicodeValue) * Hash.FnvPrime
End While
Expand Down
Loading