CXTranslationUnit.Tokenize always crashes

[f1nalspace]

The Tokenize() function in CXTranslationUnit does not work and always crashes with:

System.ArgumentException: 'Cannot use type 'ClangSharp.Interop.CXToken'. Only value types without pointers or references are supported.'

Even parsing one simple function wont work:

using CXIndex index = CXIndex.Create();

// We only parse single C files
CXTranslationUnit_Flags flags = CXTranslationUnit_Flags.CXTranslationUnit_SingleFileParse;

// Parse all comments
ReadOnlySpan<string> arguments = new ReadOnlySpan<string>(new[] { "-fparse-all-comments" });

// Very simple C code for testing
string testContent = "/*! @brief The main entry point */\nint main(int argc, char **argv) {\n return 0;\n}";

// We load the content ourself, so we use CXUnsavedFile for parsing the actual content
// Filename is just a "identifier" for GetFile() later
ReadOnlySpan<CXUnsavedFile> unsavedFiles = new ReadOnlySpan<CXUnsavedFile>(new[] { CXUnsavedFile.Create(filename, testContent) });

using CXTranslationUnit translationUnit = CXTranslationUnit.Parse(index, filename, arguments, unsavedFiles, flags);

// Create the full range for the entire file
CXFile firstFile = translationUnit.GetFile(filename);
CXSourceLocation firstLoc = translationUnit.GetLocation(firstFile, 1, 1);
CXSourceLocation lastLoc = translationUnit.GetLocationForOffset(firstFile, (uint)content.Length);
CXSourceRange range = CXSourceRange.Create(firstLoc, lastLoc);

 // Always crashes with System.ArgumentException: 'Cannot use type 'ClangSharp.Interop.CXToken'. Only value types without pointers or references are supported.'
Span<CXToken> tokens = translationUnit.Tokenize(range);

[tannergooding]

Can you provide details about:

• Target Framework
• Targeted version of ClangSharp
• Target Operating System
• Installed version of libClang

[f1nalspace]

Can you provide details about:

* Target Framework

* Targeted version of ClangSharp

* Target Operating System

* Installed version of libClang

Sure:

Target FrameWork -> NET 4.8
Target OS -> Windows 10 x64
No clang installed, I use the NuGet package which automatically installs a libclang.dll (https://www.nuget.org/packages/ClangSharp/11.0.0-beta3)

[f1nalspace]

Any updates? I really need a C/C++ parser as soon as possible, so I can replace my crappy and slow one inside (https://github.com/f1nalspace/doxygeneditor/tree/develop).

Unfortunately, I cannot use the results from CXTranslationUnit because I need all tokens, for syntax highlighting and for extracting every single and multiline comments.

[tannergooding]

Looking at this now.

[tannergooding]

This is a "bug" in the portable Span implementation in that it doesn't account for/allow pointer types.

This would require a fix in full framework, which I'd expect is unlikely to happen.

The "fix" here is to allocate an array and then copy the data over.

That is, the current implementation is:

public Span<CXToken> Tokenize(CXSourceRange sourceRange)
{
    CXToken* pTokens; uint numTokens;
    clang.tokenize(this, sourceRange, &pTokens, &numTokens);
    return new Span<CXToken>(pTokens, (int)numTokens);
}

You'd need to define your own method that does something like:

public CXToken[] Tokenize(CXSourceRange sourceRange)
{
    CXToken* pTokens;
    uint numTokens;

    clang.tokenize(this, sourceRange, &pTokens, &numTokens);
    var result = new CXToken[checked((int)numTokens)];

    fixed (CXToken* pResult = result)
    {
        var size = sizeof(CXToken) * numTokens;
        Buffer.MemoryCopy(pTokens, pResult, size, size);
    }
    return result;
}

Taking this in ClangSharp would be a "breaking" change and would make the default path for .NET Core/.NET 5+ less efficient.

Switching to .NET Core is also possible and would solve this issue.

[tannergooding]

Logged dotnet/runtime#53821

[tannergooding]

Looks like this is only a problem for the constructor that takes a pointer, and so the following (or similar, didn't actually check the correctness here) is also possible as a workaround and can be implemented in ClangSharp (happy to take a PR here, or I'll try to get to it soonish):

public Span<CXToken> Tokenize(CXSourceRange sourceRange)
{
    CXToken* pTokens;
    uint numTokens;
    clang.tokenize(this, sourceRange, &pTokens, &numTokens);

#if NETSTANDARD
    var result = new CXToken[checked((int)numTokens)];

    fixed (CXToken* pResult = result)
    {
        var size = sizeof(CXToken) * numTokens;
        Buffer.MemoryCopy(pTokens, pResult, size, size);
    }

    return result;
#else
    return new Span<CXToken>(pTokens, (int)numTokens);
#endif
}

[f1nalspace]

This is a "bug" in the portable Span implementation in that it doesn't account for/allow pointer types.

This would require a fix in full framework, which I'd expect is unlikely to happen.

The "fix" here is to allocate an array and then copy the data over.

That is, the current implementation is:

public Span<CXToken> Tokenize(CXSourceRange sourceRange)
{
    CXToken* pTokens; uint numTokens;
    clang.tokenize(this, sourceRange, &pTokens, &numTokens);
    return new Span<CXToken>(pTokens, (int)numTokens);
}

You'd need to define your own method that does something like:

public CXToken[] Tokenize(CXSourceRange sourceRange)
{
    CXToken* pTokens;
    uint numTokens;

    clang.tokenize(this, sourceRange, &pTokens, &numTokens);
    var result = new CXToken[checked((int)numTokens)];

    fixed (CXToken* pResult = result)
    {
        var size = sizeof(CXToken) * numTokens;
        Buffer.MemoryCopy(pTokens, pResult, size, size);
    }
    return result;
}

Taking this in ClangSharp would be a "breaking" change and would make the default path for .NET Core/.NET 5+ less efficient.

Switching to .NET Core is also possible and would solve this issue.

That fixes it, thank you. For now i use the fixed version, but in the future may upgrade to .NET 5 or something.