[flang] [flang-rt] Addition of the Fortran 2023 TOKENIZE intrinsic. - #181030
Merged
Merged
Conversation
kwyatt-ext
marked this pull request as ready for review
February 12, 2026 19:29
kwyatt-ext
temporarily deployed
to
main-branch-only
February 12, 2026 19:29 — with 
GitHub Actions
Inactive
GitHub Actions
Inactive
llvmbot
added
flang
llvmbot
temporarily deployed
to
main-branch-only
February 12, 2026 19:29 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
llvmbot
temporarily deployed
to
main-branch-only
February 12, 2026 19:29 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
llvmbot
temporarily deployed
to
main-branch-only
February 12, 2026 19:29 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
llvmbot
temporarily deployed
to
main-branch-only
February 12, 2026 19:29 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
Member
|
@llvm/pr-subscribers-flang-semantics @llvm/pr-subscribers-flang-fir-hlfir Author: None (kwyatt-ext) ChangesThis implements the TOKENIZE intrinsic per the Fortran 2023 Standard. TOKENIZE is a more complicated addition to the flang intrinsics, as it is the first subroutine that has multiple unique footprints. Intrinsic functions have already addressed this challenge, however subroutines and functions are processed slightly differently and the function code was not a good 1:1 solution for the subroutines. To solve this the function code was used as an example to create error buffering within the intrinsics Process and select the most appropriate error message for a given subroutine footprint. A simple FIR compile test was added to show the proper compilation of each case. A thorough negative path test has also been added, ensuring that all possible errors are reported as expected. Testing prior to commit: = check-flang ========================================== = check-flang-rt ========================================== = llvm-test-suite ========================================== Additionally, (FYI) an executable test has been written and will be added to the llvm-test-suite under a separate PR. Patch is 45.36 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/181030.diff 9 Files Affected:
diff --git a/flang-rt/lib/runtime/character.cpp b/flang-rt/lib/runtime/character.cpp
index c9ac55736d427..fd312fb8fe037 100644
--- a/flang-rt/lib/runtime/character.cpp
+++ b/flang-rt/lib/runtime/character.cpp
@@ -17,6 +17,7 @@
#include "flang/Runtime/freestanding-tools.h"
#include <algorithm>
#include <cstring>
+#include <iostream>
namespace Fortran::runtime {
@@ -571,6 +572,419 @@ static RT_API_ATTRS void MaxMin(Descriptor &accumulator, const Descriptor &x,
}
}
+template <typename CHAR>
+static inline RT_API_ATTRS bool TokenizeIsInSet(
+ CHAR ch, const CHAR *set, std::size_t setChars) {
+ for (std::size_t j{0}; j < setChars; ++j) {
+ if (set[j] == ch) {
+ return true;
+ }
+ }
+ return false;
+}
+
+// Pad the token with spaces.
+template <typename CHAR>
+static inline RT_API_ATTRS void TokenizeFillBlanks(
+ CHAR *to, std::size_t chars) {
+ if (chars == 0) {
+ return;
+ }
+ if constexpr (std::is_same_v<CHAR, char>) {
+ runtime::memset(to, ' ', chars);
+ } else {
+ for (std::size_t j{0}; j < chars; ++j) {
+ to[j] = static_cast<CHAR>(' ');
+ }
+ }
+}
+
+struct TokenizeAnalysis {
+ std::size_t tokenCount{0};
+ std::size_t maxTokenLen{0}; // in characters
+};
+
+template <typename CHAR>
+static RT_API_ATTRS TokenizeAnalysis AnalyzeTokenize(const CHAR *str,
+ std::size_t strChars, const CHAR *set, std::size_t setChars) {
+ TokenizeAnalysis analysis;
+ if (strChars == 0) {
+ return analysis;
+ }
+ if (setChars == 0) {
+ analysis.tokenCount = 1;
+ analysis.maxTokenLen = strChars;
+ return analysis;
+ }
+
+ // Split STRING at each delimiter character. This produces empty tokens
+ // when delimiters are consecutive or when STRING starts/ends with a
+ // delimiter.
+ std::size_t tokenStart{0};
+ for (std::size_t pos{0}; pos < strChars; ++pos) {
+ if (TokenizeIsInSet(str[pos], set, setChars)) {
+ analysis.maxTokenLen = std::max(analysis.maxTokenLen, pos - tokenStart);
+ analysis.tokenCount++;
+ tokenStart = pos + 1;
+ }
+ }
+ analysis.maxTokenLen = std::max(analysis.maxTokenLen, strChars - tokenStart);
+ analysis.tokenCount++;
+
+ return analysis;
+}
+// Allocates and populates the result arrays for TOKENIZE Form 1.
+template <typename CHAR>
+static RT_API_ATTRS void TokenizeFillForm1(Descriptor &tokens,
+ Descriptor *separator, const Descriptor &string, const CHAR *str,
+ std::size_t strChars, const CHAR *set, std::size_t setChars,
+ const TokenizeAnalysis &analysis, Terminator &terminator) {
+
+ // (Re)allocate TOKENS.
+ if (tokens.IsAllocated()) {
+ tokens.Deallocate();
+ }
+ SubscriptValue tokensExtent[1]{
+ static_cast<SubscriptValue>(analysis.tokenCount)};
+ std::size_t tokenElemBytes{
+ analysis.tokenCount == 0 ? 0 : analysis.maxTokenLen * sizeof(CHAR)};
+ tokens.Establish(string.type(), tokenElemBytes, nullptr, 1, tokensExtent,
+ CFI_attribute_allocatable);
+ tokens.GetDimension(0).SetBounds(1, tokensExtent[0]);
+ if (tokens.Allocate(kNoAsyncObject) != CFI_SUCCESS) {
+ terminator.Crash("TOKENIZE: could not allocate TOKENS array");
+ }
+
+ // (Re)allocate SEPARATOR if present.
+ std::size_t sepCount{analysis.tokenCount > 0 ? analysis.tokenCount - 1 : 0};
+ std::size_t sepElemBytes{sizeof(CHAR)};
+ if (separator) {
+ if (separator->IsAllocated()) {
+ separator->Deallocate();
+ }
+ SubscriptValue sepExtent[1]{static_cast<SubscriptValue>(sepCount)};
+ separator->Establish(string.type(), sepElemBytes, nullptr, 1, sepExtent,
+ CFI_attribute_allocatable);
+ separator->GetDimension(0).SetBounds(1, sepExtent[0]);
+ if (separator->Allocate(kNoAsyncObject) != CFI_SUCCESS) {
+ terminator.Crash("TOKENIZE: could not allocate SEPARATOR array");
+ }
+ }
+
+ if (analysis.tokenCount == 0) {
+ return;
+ }
+
+ // Populate tokens and separators.
+ if (setChars == 0) {
+ // One token (possibly empty) equal to STRING.
+ if (tokenElemBytes > 0) {
+ CHAR *tokDest{tokens.OffsetElement<CHAR>(0)};
+ TokenizeFillBlanks(tokDest, analysis.maxTokenLen);
+ if (strChars > 0) {
+ runtime::memcpy(tokDest, str, strChars * sizeof(CHAR));
+ }
+ }
+ return;
+ }
+
+ std::size_t tokenIndex{0};
+ std::size_t sepIndex{0};
+
+ auto storeToken = [&](std::size_t tokenStart, std::size_t tokenEnd) {
+ std::size_t tokenLen{tokenEnd - tokenStart};
+ if (tokenElemBytes > 0) {
+ // Each element is stored in a fixed-size slot of `tokenElemBytes`.
+ CHAR *tokDest{tokens.OffsetElement<CHAR>(tokenIndex * tokenElemBytes)};
+ TokenizeFillBlanks(tokDest, analysis.maxTokenLen);
+ if (tokenLen > 0) {
+ runtime::memcpy(tokDest, str + tokenStart, tokenLen * sizeof(CHAR));
+ }
+ }
+ ++tokenIndex;
+ };
+
+ // Split at each delimiter character, producing empty tokens at boundaries
+ // and between consecutive delimiters.
+ std::size_t tokenStart{0};
+ for (std::size_t pos{0}; pos < strChars; ++pos) {
+ if (TokenizeIsInSet(str[pos], set, setChars)) {
+ storeToken(tokenStart, pos);
+ if (separator) {
+ CHAR *sepDest{separator->OffsetElement<CHAR>(sepIndex * sepElemBytes)};
+ sepDest[0] = str[pos];
+ ++sepIndex;
+ }
+ tokenStart = pos + 1;
+ }
+ }
+ storeToken(tokenStart, strChars);
+}
+
+template <int KIND>
+static RT_API_ATTRS void TokenizeStoreIntAt(
+ const Descriptor &result, std::size_t at, std::int64_t value) {
+ StoreIntegerAt<KIND>{}(result, at, value);
+}
+
+using TokenizeStoreIntFn = void (*)(
+ const Descriptor &, std::size_t, std::int64_t);
+
+static RT_API_ATTRS TokenizeStoreIntFn GetTokenizeStoreIntFn(
+ int kind, Terminator &terminator, const char *which) {
+ switch (kind) {
+ case 1:
+ return &TokenizeStoreIntAt<1>;
+ case 2:
+ return &TokenizeStoreIntAt<2>;
+ case 4:
+ return &TokenizeStoreIntAt<4>;
+ case 8:
+ return &TokenizeStoreIntAt<8>;
+ case 16:
+ return &TokenizeStoreIntAt<16>;
+ default:
+ terminator.Crash(
+ "TOKENIZE: unsupported INTEGER kind=%d for %s", kind, which);
+ }
+}
+
+template <typename CHAR>
+static RT_API_ATTRS void TokenizeFillPositions(Descriptor &first,
+ Descriptor &last, const CHAR *str, std::size_t strChars, const CHAR *set,
+ std::size_t setChars, TokenizeStoreIntFn storeFirst,
+ TokenizeStoreIntFn storeLast, Terminator &terminator) {
+ if (strChars == 0) {
+ return;
+ }
+ if (setChars == 0) {
+ storeFirst(first, 0, 1);
+ storeLast(last, 0, static_cast<std::int64_t>(strChars));
+ return;
+ }
+
+ std::size_t tokenIndex{0};
+ std::size_t tokenStart{0};
+ for (std::size_t pos{0}; pos < strChars; ++pos) {
+ if (TokenizeIsInSet(str[pos], set, setChars)) {
+ storeFirst(first, tokenIndex, static_cast<std::int64_t>(tokenStart + 1));
+ storeLast(last, tokenIndex, static_cast<std::int64_t>(pos));
+ ++tokenIndex;
+ tokenStart = pos + 1;
+ }
+ }
+ storeFirst(first, tokenIndex, static_cast<std::int64_t>(tokenStart + 1));
+ storeLast(last, tokenIndex, static_cast<std::int64_t>(strChars));
+ ++tokenIndex;
+
+ // Sanity check: we should have filled exactly the allocated extent.
+ if (tokenIndex != static_cast<std::size_t>(first.GetDimension(0).Extent())) {
+ terminator.Crash("TOKENIZE: internal error populating FIRST/LAST");
+ }
+}
+
+// Tokenize Form 1 implementation.
+static RT_API_ATTRS void TokenizeImpl(Descriptor &tokens, Descriptor *separator,
+ const Descriptor &string, const Descriptor &set, Terminator &terminator) {
+ RUNTIME_CHECK(terminator, string.rank() == 0);
+ RUNTIME_CHECK(terminator, set.rank() == 0);
+ RUNTIME_CHECK(terminator, string.raw().type == set.raw().type);
+ RUNTIME_CHECK(terminator, tokens.rank() == 1);
+ RUNTIME_CHECK(terminator, tokens.IsAllocatable());
+ if (separator) {
+ RUNTIME_CHECK(terminator, separator->rank() == 1);
+ RUNTIME_CHECK(terminator, separator->IsAllocatable());
+ }
+
+ switch (string.raw().type) {
+ case CFI_type_char: {
+ std::size_t strBytes{string.ElementBytes()};
+ std::size_t setBytes{set.ElementBytes()};
+ std::size_t strChars{strBytes};
+ std::size_t setChars{setBytes};
+ const char *str{
+ strBytes == 0 ? nullptr : string.OffsetElement<const char>()};
+ const char *setPtr{
+ setBytes == 0 ? nullptr : set.OffsetElement<const char>()};
+ auto analysis{AnalyzeTokenize(str, strChars, setPtr, setChars)};
+ TokenizeFillForm1(tokens, separator, string, str, strChars, setPtr,
+ setChars, analysis, terminator);
+ break;
+ }
+ case CFI_type_char16_t: {
+ std::size_t strBytes{string.ElementBytes()};
+ std::size_t setBytes{set.ElementBytes()};
+ std::size_t strChars{strBytes >> 1};
+ std::size_t setChars{setBytes >> 1};
+ const char16_t *str{
+ strBytes == 0 ? nullptr : string.OffsetElement<const char16_t>()};
+ const char16_t *setPtr{
+ setBytes == 0 ? nullptr : set.OffsetElement<const char16_t>()};
+ auto analysis{AnalyzeTokenize(str, strChars, setPtr, setChars)};
+ TokenizeFillForm1(tokens, separator, string, str, strChars, setPtr,
+ setChars, analysis, terminator);
+ break;
+ }
+ case CFI_type_char32_t: {
+ std::size_t strBytes{string.ElementBytes()};
+ std::size_t setBytes{set.ElementBytes()};
+ std::size_t strChars{strBytes >> 2};
+ std::size_t setChars{setBytes >> 2};
+ const char32_t *str{
+ strBytes == 0 ? nullptr : string.OffsetElement<const char32_t>()};
+ const char32_t *setPtr{
+ setBytes == 0 ? nullptr : set.OffsetElement<const char32_t>()};
+ auto analysis{AnalyzeTokenize(str, strChars, setPtr, setChars)};
+ TokenizeFillForm1(tokens, separator, string, str, strChars, setPtr,
+ setChars, analysis, terminator);
+ break;
+ }
+ default:
+ terminator.Crash("TOKENIZE: bad string type code %d",
+ static_cast<int>(string.raw().type));
+ }
+}
+
+// Tokenize Form 2 implementation.
+static RT_API_ATTRS void TokenizePositionsImpl(Descriptor &first,
+ Descriptor &last, const Descriptor &string, const Descriptor &set,
+ Terminator &terminator) {
+ RUNTIME_CHECK(terminator, string.rank() == 0);
+ RUNTIME_CHECK(terminator, set.rank() == 0);
+ RUNTIME_CHECK(terminator, string.raw().type == set.raw().type);
+ RUNTIME_CHECK(terminator, first.rank() == 1);
+ RUNTIME_CHECK(terminator, last.rank() == 1);
+ RUNTIME_CHECK(terminator, first.IsAllocatable());
+ RUNTIME_CHECK(terminator, last.IsAllocatable());
+
+ auto firstCK{first.type().GetCategoryAndKind()};
+ auto lastCK{last.type().GetCategoryAndKind()};
+ if (!firstCK || firstCK->first != TypeCategory::Integer) {
+ terminator.Crash("TOKENIZE: FIRST is not an INTEGER array");
+ }
+ if (!lastCK || lastCK->first != TypeCategory::Integer) {
+ terminator.Crash("TOKENIZE: LAST is not an INTEGER array");
+ }
+ int firstKind{firstCK->second};
+ int lastKind{lastCK->second};
+ auto storeFirst{GetTokenizeStoreIntFn(firstKind, terminator, "FIRST")};
+ auto storeLast{GetTokenizeStoreIntFn(lastKind, terminator, "LAST")};
+
+ // Count tokens.
+ std::size_t tokenCount{0};
+ switch (string.raw().type) {
+ case CFI_type_char: {
+ std::size_t strBytes{string.ElementBytes()};
+ std::size_t setBytes{set.ElementBytes()};
+ std::size_t strChars{strBytes};
+ std::size_t setChars{setBytes};
+ const char *str{
+ strBytes == 0 ? nullptr : string.OffsetElement<const char>()};
+ const char *setPtr{
+ setBytes == 0 ? nullptr : set.OffsetElement<const char>()};
+ tokenCount = AnalyzeTokenize(str, strChars, setPtr, setChars).tokenCount;
+ break;
+ }
+ case CFI_type_char16_t: {
+ std::size_t strBytes{string.ElementBytes()};
+ std::size_t setBytes{set.ElementBytes()};
+ std::size_t strChars{strBytes >> 1};
+ std::size_t setChars{setBytes >> 1};
+ const char16_t *str{
+ strBytes == 0 ? nullptr : string.OffsetElement<const char16_t>()};
+ const char16_t *setPtr{
+ setBytes == 0 ? nullptr : set.OffsetElement<const char16_t>()};
+ tokenCount = AnalyzeTokenize(str, strChars, setPtr, setChars).tokenCount;
+ break;
+ }
+ case CFI_type_char32_t: {
+ std::size_t strBytes{string.ElementBytes()};
+ std::size_t setBytes{set.ElementBytes()};
+ std::size_t strChars{strBytes >> 2};
+ std::size_t setChars{setBytes >> 2};
+ const char32_t *str{
+ strBytes == 0 ? nullptr : string.OffsetElement<const char32_t>()};
+ const char32_t *setPtr{
+ setBytes == 0 ? nullptr : set.OffsetElement<const char32_t>()};
+ tokenCount = AnalyzeTokenize(str, strChars, setPtr, setChars).tokenCount;
+ break;
+ }
+ default:
+ terminator.Crash("TOKENIZE: bad string type code %d",
+ static_cast<int>(string.raw().type));
+ }
+
+ // (Re)allocate FIRST/LAST.
+ if (first.IsAllocated()) {
+ first.Deallocate();
+ }
+ if (last.IsAllocated()) {
+ last.Deallocate();
+ }
+ SubscriptValue extent[1]{static_cast<SubscriptValue>(tokenCount)};
+ first.Establish(TypeCategory::Integer, firstKind, nullptr, 1, extent,
+ CFI_attribute_allocatable);
+ first.GetDimension(0).SetBounds(1, extent[0]);
+ last.Establish(TypeCategory::Integer, lastKind, nullptr, 1, extent,
+ CFI_attribute_allocatable);
+ last.GetDimension(0).SetBounds(1, extent[0]);
+ if (first.Allocate(kNoAsyncObject) != CFI_SUCCESS) {
+ terminator.Crash("TOKENIZE: could not allocate FIRST array");
+ }
+ if (last.Allocate(kNoAsyncObject) != CFI_SUCCESS) {
+ terminator.Crash("TOKENIZE: could not allocate LAST array");
+ }
+
+ if (tokenCount == 0) {
+ return;
+ }
+
+ // Populate FIRST/LAST.
+ switch (string.raw().type) {
+ case CFI_type_char: {
+ std::size_t strBytes{string.ElementBytes()};
+ std::size_t setBytes{set.ElementBytes()};
+ std::size_t strChars{strBytes};
+ std::size_t setChars{setBytes};
+ const char *str{
+ strBytes == 0 ? nullptr : string.OffsetElement<const char>()};
+ const char *setPtr{
+ setBytes == 0 ? nullptr : set.OffsetElement<const char>()};
+ TokenizeFillPositions(first, last, str, strChars, setPtr, setChars,
+ storeFirst, storeLast, terminator);
+ break;
+ }
+ case CFI_type_char16_t: {
+ std::size_t strBytes{string.ElementBytes()};
+ std::size_t setBytes{set.ElementBytes()};
+ std::size_t strChars{strBytes >> 1};
+ std::size_t setChars{setBytes >> 1};
+ const char16_t *str{
+ strBytes == 0 ? nullptr : string.OffsetElement<const char16_t>()};
+ const char16_t *setPtr{
+ setBytes == 0 ? nullptr : set.OffsetElement<const char16_t>()};
+ TokenizeFillPositions(first, last, str, strChars, setPtr, setChars,
+ storeFirst, storeLast, terminator);
+ break;
+ }
+ case CFI_type_char32_t: {
+ std::size_t strBytes{string.ElementBytes()};
+ std::size_t setBytes{set.ElementBytes()};
+ std::size_t strChars{strBytes >> 2};
+ std::size_t setChars{setBytes >> 2};
+ const char32_t *str{
+ strBytes == 0 ? nullptr : string.OffsetElement<const char32_t>()};
+ const char32_t *setPtr{
+ setBytes == 0 ? nullptr : set.OffsetElement<const char32_t>()};
+ TokenizeFillPositions(first, last, str, strChars, setPtr, setChars,
+ storeFirst, storeLast, terminator);
+ break;
+ }
+ default:
+ break;
+ }
+}
+
extern "C" {
RT_EXT_API_GROUP_BEGIN
@@ -918,6 +1332,22 @@ void RTDEF(CharacterMin)(Descriptor &accumulator, const Descriptor &x,
MaxMin<true>(accumulator, x, sourceFile, sourceLine);
}
+// TOKENIZE Form 1 entry point
+void RTDEF(Tokenize)(Descriptor &tokens, Descriptor *separator,
+ const Descriptor &string, const Descriptor &set, const char *sourceFile,
+ int sourceLine) {
+ Terminator terminator{sourceFile, sourceLine};
+ TokenizeImpl(tokens, separator, string, set, terminator);
+}
+
+// TOKENIZE Form 2 entry point
+void RTDEF(TokenizePositions)(Descriptor &first, Descriptor &last,
+ const Descriptor &string, const Descriptor &set, const char *sourceFile,
+ int sourceLine) {
+ Terminator terminator{sourceFile, sourceLine};
+ TokenizePositionsImpl(first, last, string, set, terminator);
+}
+
RT_EXT_API_GROUP_END
}
} // namespace Fortran::runtime
diff --git a/flang/include/flang/Optimizer/Builder/IntrinsicCall.h b/flang/include/flang/Optimizer/Builder/IntrinsicCall.h
index b248106b51101..8cf856fd36c61 100644
--- a/flang/include/flang/Optimizer/Builder/IntrinsicCall.h
+++ b/flang/include/flang/Optimizer/Builder/IntrinsicCall.h
@@ -435,6 +435,7 @@ struct IntrinsicLibrary {
fir::ExtendedValue genTeamNumber(mlir::Type,
llvm::ArrayRef<fir::ExtendedValue>);
mlir::Value genTime(mlir::Type, llvm::ArrayRef<mlir::Value>);
+ void genTokenize(llvm::ArrayRef<fir::ExtendedValue>);
mlir::Value genTrailz(mlir::Type, llvm::ArrayRef<mlir::Value>);
fir::ExtendedValue genTransfer(mlir::Type,
llvm::ArrayRef<fir::ExtendedValue>);
diff --git a/flang/include/flang/Optimizer/Builder/Runtime/Character.h b/flang/include/flang/Optimizer/Builder/Runtime/Character.h
index 261ac348a4024..86991e4d1d02c 100644
--- a/flang/include/flang/Optimizer/Builder/Runtime/Character.h
+++ b/flang/include/flang/Optimizer/Builder/Runtime/Character.h
@@ -128,6 +128,22 @@ mlir::Value genVerify(fir::FirOpBuilder &builder, mlir::Location loc, int kind,
mlir::Value setBase, mlir::Value setLen,
mlir::Value back);
+/// Generate call to TOKENIZE runtime (Form 1).
+/// Splits \p stringBox into tokens based on separator characters in \p setBox.
+/// \p tokensBox must be an unallocated allocatable array that receives the
+/// token substrings. \p separatorBox is optional and receives separator chars.
+void genTokenize(fir::FirOpBuilder &builder, mlir::Location loc,
+ mlir::Value tokensBox, mlir::Value separatorBox,
+ mlir::Value stringBox, mlir::Value setBox);
+
+/// Generate call to TOKENIZE runtime (Form 2).
+/// Returns token positions rather than substrings.
+/// \p firstBox and \p lastBox must be unallocated allocatable integer arrays
+/// that receive the starting and ending positions of each token.
+void genTokenizePositions(fir::FirOpBuilder &builder, mlir::Location loc,
+ mlir::Value firstBox, mlir::Value lastBox,
+ mlir::Value stringBox, mlir::Value setBox);
+
} // namespace fir::runtime
#endif // FORTRAN_OPTIMIZER_BUILDER_RUNTIME_CHARACTER_H
diff --git a/flang/include/flang/Runtime/character.h b/flang/include/flang/Runtime/character.h
index dd47686fe858f..c2b709beda898 100644
--- a/flang/include/flang/Runtime/character.h
+++ b/flang/include/flang/Runtime/character.h
@@ -127,6 +127,13 @@ std::size_t RTDECL(Verify4)(const char32_t *, std::size_t, const char32_t *set,
void RTDECL(Verify)(Descriptor &result, const Descriptor &string,
const Descriptor &set, const Descriptor *back /*can be null*/, int kind,
const char *sourceFile = nullptr, int sourceLine = 0);
+
+void RTDECL(Tokenize)(Descriptor &tokens, Descriptor *separator,
+ const Descriptor &string, const Descriptor &set,
+ const char *sourceFile = nullptr, int sourceLine = 0);
+void RTDECL(TokenizePositions)(Descriptor &first, Descriptor &last,
+ const Descriptor &string, const Descriptor &set,
+ const char *sourceFile = nullptr, int sourceLine = 0);
}
} // namespace Fortran::runtime
#endif // FORTRAN_RUNTIME_CHARACTER_H_
diff --git a/flang/lib/Evaluate/intrinsics.cpp b/flang/lib/Evaluate/intrinsics.cpp
index 72ac9e2f68758..a94a1088d2205 100644
--- a/flang/lib/Evaluate/intrinsics.cpp
+++ b/flang/lib/Evaluate/intrinsics.cpp
@@ -22,6 +22,7 @@
#include "flang/Support/Fortran.h"
#include "llvm/Support/raw_ostream.h"
#include <algorithm>
+#include <climits>
#include <cmath>
#include <map>
#include <string>
@@ -1743,6 +1744,26 @@ static const IntrinsicInterface intrinsicSubroutine[]{
{{"seconds", AnyInt, Rank::scalar, Optionality::required,
common::Intent::In}},
{}, Rank::elemental, IntrinsicClass::impureSubroutine},
+ {"tokenize",
+ {{"string", SameCharNoLen, Rank::scalar, Optionality::required,
+ common::Intent::In},
+ {"set", SameCharNoLen, Rank::scalar, Optionality::required,
+ common::Intent::In},
+ {"tokens", SameCharNoLen, Rank::vector, Optionality::requir...
[truncated]
|
tblah
reviewed
Feb 23, 2026
I'm not sure how this file ended up in the test only update commit.
This reverts commit 861ed09.
tblah
reviewed
Feb 24, 2026
tblah
left a comment
tblah
left a comment
Contributor
There was a problem hiding this comment.
Thank you for updating the lowering test. A few more comments from me
- Please could you add unittests in flang-rt/unittests/Runtime/CharacterTest.cpp.
- On my reading of the standard, an empty string should return one zero-length token, whereas I think this implementation gives zero tokens.
| CHAR *tokDest{tokens.OffsetElement<CHAR>(0)}; | ||
| TokenizeFillBlanks(tokDest, analysis.maxTokenLen); | ||
| if (strChars > 0) { | ||
| runtime::memcpy(tokDest, str, strChars * sizeof(CHAR)); |
Contributor
There was a problem hiding this comment.
TokenizeFillBlanks could do the same as this to always use memcpy
Contributor
Author
There was a problem hiding this comment.
I'm not quite sure what you are suggesting with this. TokenizeFillBlanks is there to blank (space) pad the token array and then memcpy to copy over just the actual token.
Contributor
There was a problem hiding this comment.
Sorry I wasn't clear.
TokenizeFillBlanks has a specialized implementation for char and falls back to a for loop. I think you could just do a runtime::memcpy adjusted for sizeof(CHAR) as you did here. I think that would make the code slightly less effort to understand.
Contributor
Author
I will add unit tests. You are correct about the zero-length token. I wrestled with that before implementing it, but went the wrong way. I've corrected it in my local copy and will include that fix in my next merge. |
Contributor
Author
|
I made all requested changes. I was still not clear on 1 or 2 of the questions. Perhaps the change addressed those. Also, a Fortran runtime test has been added to llvm-test-suite that more thoroughly tests the tokenize function. It has been reviewed and is awaiting this change before it gets merged. It can be found here: |
tblah
approved these changes
Feb 27, 2026
tblah
left a comment
tblah
left a comment
Contributor
There was a problem hiding this comment.
Thank you for the updates and for adding the tests!
| CHAR *tokDest{tokens.OffsetElement<CHAR>(0)}; | ||
| TokenizeFillBlanks(tokDest, analysis.maxTokenLen); | ||
| if (strChars > 0) { | ||
| runtime::memcpy(tokDest, str, strChars * sizeof(CHAR)); |
Contributor
There was a problem hiding this comment.
Sorry I wasn't clear.
TokenizeFillBlanks has a specialized implementation for char and falls back to a for loop. I think you could just do a runtime::memcpy adjusted for sizeof(CHAR) as you did here. I think that would make the code slightly less effort to understand.
Contributor
Author
|
@tblah Thank you for clarifying. Your idea is clearer, but I am filling with spaces. The corresponding call would be runtime::memset. The problem with memset is that it fills memory on a per byte basis. Using a larger character type will result in an invalid fill (i.e. 0x2020 instead of 0x0020). The TokenizeFillBlanks is to set each typeless character to a space. I could be missing a better method, but that is why it is done this way. |
Contributor
Author
|
Also, I'm not sure if this is also waiting on a review from @eugeneepshteyn, but I am unable to merge. When this is ready, I would appreciate it if you wouldn't mind taking care of that for me. Thank you. |
Contributor
|
Thanks for the explanation. I'll merge. Once you have a few patches in LLVM you can request commit access. https://llvm.org/docs/DeveloperPolicy.html#obtaining-commit-access |
Contributor
|
@kwyatt-ext , did you run tests from llvm-test-suite? Also, if you want to add relevant new execution tests, they could go to llvm-test-suite. |
Contributor
Author
|
Thank you for the merge. @eugeneepshteyn Yes, I did run llvm-test-suite against the change. There is also a new test added within the the llvm-test-suite: Fortran/cray/f2023/16/09/tokenize01.f90 |
Contributor
|
I saw a buildbot problem that specifically failed with the new test in llvm-test-suite: https://lab.llvm.org/buildbot/#/builders/121/builds/2128 No other buildbot problems, so perhaps it was a timing issue between submitting a test and submitting this PR? |
Contributor
Author
|
Hmm. I must have missed re-running against the last review update. There is an issue with the expected results where the STRING and SET are both empty. I'll patch it right away. ... Nevermind. It is passing in my environment, but doesn't when I just build the test source manually. I need to figure out what my environment is doing. However, the expected results DO need to be updated for the STRING empty cases. |
Contributor
|
Just adding the fix PR link here: llvm/llvm-test-suite#359 |
sujianIBM
pushed a commit
to sujianIBM/llvm-project
that referenced
this pull request
Mar 5, 2026
…lvm#181030) This implements the TOKENIZE intrinsic per the Fortran 2023 Standard. TOKENIZE is a more complicated addition to the flang intrinsics, as it is the first subroutine that has multiple unique footprints. Intrinsic functions have already addressed this challenge, however subroutines and functions are processed slightly differently and the function code was not a good 1:1 solution for the subroutines. To solve this the function code was used as an example to create error buffering within the intrinsics Process and select the most appropriate error message for a given subroutine footprint. A simple FIR compile test was added to show the proper compilation of each case. A thorough negative path test has also been added, ensuring that all possible errors are reported as expected. Testing prior to commit: = check-flang ========================================== ``` Testing Time: 139.51s Total Discovered Tests: 4153 Unsupported : 77 (1.85%) Passed : 4065 (97.88%) Expectedly Failed: 11 (0.26%) FLANG Container Test completed 2 minutes (160 s). Total Time: 2 minutes (160 s) Completed : Wed Feb 11 04:05:50 PM CST 2026 ``` = check-flang-rt ========================================== ``` Testing Time: 1.55s Total Discovered Tests: 258 Passed: 258 (100.00%) FLANG Container Test completed 0 minutes (55 s). Total Time: 0 minutes (56 s) Completed : Wed Feb 11 04:08:32 PM CST 2026 ``` = llvm-test-suite ========================================== ``` Testing Time: 1886.64s Total Discovered Tests: 6926 Passed: 6926 (100.00%) CCE SLES Container debug compile completed 31 minutes (1895 s). CCE SLES Container debug install completed in 0 minutes (0 s). Total Time: 31 minutes (1895 s) Completed : Wed Feb 11 05:46:52 PM CST 2026 ``` Additionally, (FYI) an executable test has been written and will be added to the llvm-test-suite under a separate PR. --------- Co-authored-by: Kevin Wyatt <kwyatt@hpe.com>
This was referenced Apr 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This implements the TOKENIZE intrinsic per the Fortran 2023 Standard.
TOKENIZE is a more complicated addition to the flang intrinsics, as it is the first subroutine that has multiple unique footprints. Intrinsic functions have already addressed this challenge, however subroutines and functions are processed slightly differently and the function code was not a good 1:1 solution for the subroutines. To solve this the function code was used as an example to create error buffering within the intrinsics Process and select the most appropriate error message for a given subroutine footprint.
A simple FIR compile test was added to show the proper compilation of each case. A thorough negative path test has also been added, ensuring that all possible errors are reported as expected.
Testing prior to commit:
= check-flang ==========================================
= check-flang-rt ==========================================
= llvm-test-suite ==========================================
Additionally, (FYI) an executable test has been written and will be added to the llvm-test-suite under a separate PR.