[z/OS] Add z/OS archive reading support

[uyoyo0]

Add support for reading z/OS archives, which use EBCDIC-encoded header fields and an EBCDIC magic string. The z/OS archive format shares the same structural layout as traditional Unix archives but all text fields (member names, timestamps, permissions, and symbol names) are in EBCDIC. This patch adds:

• K_ZOS archive kind
• ZOSArchiveMemberHeader: converts EBCDIC header fields to ASCII on read
• ZOSArchive: parses the __.SYMDEF symbol table, converting EBCDIC symbol names to ASCII
• Updates to symbol table traversal for K_ZOS, which uses big-endian 4-byte offsets paired with 4-byte attribute words per symbol

This is part 2 of a patch series adding z/OS archive support to LLVM.

Part 1: #186854

[llvmbot]

@llvm/pr-subscribers-llvm-binary-utilities

Author: Uyiosa Iyekekpolor (uyoyo0)

Changes

Add support for reading z/OS archives, which use EBCDIC-encoded header fields and an EBCDIC magic string. The z/OS archive format shares the same structural layout as traditional Unix archives but all text fields (member names, timestamps, permissions, and symbol names) are in EBCDIC. This patch adds:

• K_ZOS archive kind
• ZOSArchiveMemberHeader: converts EBCDIC header fields to ASCII on read
• ZOSArchive: parses the __.SYMDEF symbol table, converting EBCDIC symbol names to ASCII
• Updates to symbol table traversal for K_ZOS, which uses big-endian 4-byte offsets paired with 4-byte attribute words per symbol

This is part 2 of a patch series adding z/OS archive support to LLVM.

Part 1: #186854

---

Full diff: https://github.com/llvm/llvm-project/pull/187110.diff

4 Files Affected:

• (modified) llvm/include/llvm/Object/Archive.h (+43-1)
• (modified) llvm/lib/Object/Archive.cpp (+209-2)
• (added) llvm/test/Object/Inputs/zos-archive-test.a ()
• (added) llvm/test/Object/zos-archive-read.test (+9)

diff --git a/llvm/include/llvm/Object/Archive.h b/llvm/include/llvm/Object/Archive.h
index c97018d3231d5..0e53be307904e 100644
--- a/llvm/include/llvm/Object/Archive.h
+++ b/llvm/include/llvm/Object/Archive.h
@@ -158,6 +158,35 @@ class LLVM_ABI BigArchiveMemberHeader
   Expected<bool> isThin() const override { return false; }
 };
 
+// Define file member header of z/OS archive.
+class ZOSArchiveMemberHeader : public ArchiveMemberHeader {
+public:
+    ZOSArchiveMemberHeader(Archive const *Parent, const char *RawHeaderPtr,
+                           uint64_t Size, Error *Err);
+  std::unique_ptr<AbstractArchiveMemberHeader> clone() const override {
+    return std::make_unique<ZOSArchiveMemberHeader>(*this);
+  }
+
+  // Converted EBCDIC to ASCII header string fields.
+  std::string RawMemberName;
+  std::string MemberName;
+  std::string LastModified;
+  std::string UID;
+  std::string GID;
+  std::string AccessMode;
+
+  void setMemberHeaderStrings(Error *Err, uint64_t Size);
+
+  Expected<StringRef> getRawName() const override;
+  Expected<StringRef> getName(uint64_t Size) const override;
+  StringRef getRawAccessMode() const override;
+  StringRef getRawLastModified() const override;
+  StringRef getRawUID() const override;
+  StringRef getRawGID() const override;
+  Expected<uint64_t> getSize() const override;
+  Expected<bool> isThin() const override { return false; }
+};
+
 class LLVM_ABI Archive : public Binary {
   virtual void anchor();
 
@@ -343,7 +372,7 @@ class LLVM_ABI Archive : public Binary {
   /// Size field is 10 decimal digits long
   static const uint64_t MaxMemberSize = 9999999999;
 
-  enum Kind { K_GNU, K_GNU64, K_BSD, K_DARWIN, K_DARWIN64, K_COFF, K_AIXBIG };
+  enum Kind { K_GNU, K_GNU64, K_BSD, K_DARWIN, K_DARWIN64, K_COFF, K_AIXBIG, K_ZOS };
 
   Kind kind() const { return (Kind)Format; }
   bool isThin() const { return IsThin; }
@@ -434,6 +463,19 @@ class BigArchive : public Archive {
   bool has64BitGlobalSymtab() { return Has64BitGlobalSymtab; }
 };
 
+class ZOSArchive : public Archive {
+public:
+  // Fixed-Length header.
+  struct FixLenHdr {
+    char Magic[sizeof(ZOSArchiveMagic) - 1]; ///< ZOS archive magic string.
+  };
+
+public:
+  ZOSArchive(MemoryBufferRef Source, Error &Err);
+
+private:
+  std::string SymbolTableBuf; // __.SYMDEF strings converted to ASCII.
+};
 } // end namespace object
 } // end namespace llvm
 
diff --git a/llvm/lib/Object/Archive.cpp b/llvm/lib/Object/Archive.cpp
index 17c926e621f36..44ef12c24d379 100644
--- a/llvm/lib/Object/Archive.cpp
+++ b/llvm/lib/Object/Archive.cpp
@@ -17,6 +17,7 @@
 #include "llvm/Object/Binary.h"
 #include "llvm/Object/Error.h"
 #include "llvm/Support/Chrono.h"
+#include "llvm/Support/ConvertEBCDIC.h"
 #include "llvm/Support/Endian.h"
 #include "llvm/Support/EndianStream.h"
 #include "llvm/Support/Error.h"
@@ -104,7 +105,11 @@ ArchiveMemberHeader::ArchiveMemberHeader(const Archive *Parent,
     *Err = createMemberHeaderParseError(this, RawHeaderPtr, Size);
     return;
   }
-  if (ArMemHdr->Terminator[0] != '`' || ArMemHdr->Terminator[1] != '\n') {
+  if ((ArMemHdr->Terminator[0] != '`' || ArMemHdr->Terminator[1] != '\n') &&
+      (ArMemHdr->Terminator[0] != '\x79' ||
+       ArMemHdr->Terminator[1] !=
+           '\x15') // '\x79\x15' is '`\n' in EBCDIC for z/OS archive terminator.
+   ){
     if (Err) {
       std::string Buf;
       raw_string_ostream OS(Buf);
@@ -368,6 +373,121 @@ Expected<uint64_t> BigArchiveMemberHeader::getSize() const {
   return *SizeOrErr + alignTo(*NameLenOrErr, 2);
 }
 
+template <class T, std::size_t N>
+StringRef getFieldRawStringE2A(const T (&Field)[N], SmallString<64> &Dst) {
+  StringRef Src = StringRef(Field, N);
+  ConverterEBCDIC::convertToUTF8(Src, Dst);
+  return Dst.str().rtrim(" ");
+}
+
+ZOSArchiveMemberHeader::ZOSArchiveMemberHeader(const Archive *Parent,
+                                               const char *RawHeaderPtr,
+                                               uint64_t Size, Error *Err)
+    : ArchiveMemberHeader(Parent, RawHeaderPtr, Size, Err) {
+  ErrorAsOutParameter ErrAsOutParam(Err);
+  setMemberHeaderStrings(Err, Size);
+}
+
+Expected<uint64_t> ZOSArchiveMemberHeader::getSize() const {
+  SmallString<64> Dst;
+  return getArchiveMemberDecField(
+      "size", getFieldRawStringE2A(ArMemHdr->Size, Dst), Parent, this);
+}
+
+Expected<StringRef> ZOSArchiveMemberHeader::getRawName() const {
+  return StringRef(RawMemberName);
+}
+
+Expected<StringRef> ZOSArchiveMemberHeader::getName(uint64_t Size) const {
+  return StringRef(MemberName);
+}
+
+StringRef ZOSArchiveMemberHeader::getRawAccessMode() const {
+  return StringRef(AccessMode);
+}
+
+StringRef ZOSArchiveMemberHeader::getRawLastModified() const {
+  return StringRef(LastModified);
+}
+
+StringRef ZOSArchiveMemberHeader::getRawUID() const { return StringRef(UID); }
+
+StringRef ZOSArchiveMemberHeader::getRawGID() const { return StringRef(GID); }
+
+void ZOSArchiveMemberHeader::setMemberHeaderStrings(Error *Err, uint64_t Size) {
+  SmallString<64> Dst;
+  uint64_t Offset =
+      reinterpret_cast<const char *>(ArMemHdr) - Parent->getData().data();
+
+  // Set RawMemberName.
+  StringRef RawNameSR = getFieldRawStringE2A(ArMemHdr->Name, Dst);
+  if (RawNameSR.empty() || RawNameSR[0] == ' ') {
+    *Err = malformedError("name contains a leading space for archive member "
+                          "header at offset " +
+                          Twine(Offset));
+    return;
+  }
+  RawMemberName.append(RawNameSR);
+
+  // Set MemberName.
+  if (RawNameSR.starts_with("#1/")) {
+    Expected<StringRef> NameOrErr = ArchiveMemberHeader::getName(Size);
+    if (!NameOrErr) {
+      *Err = NameOrErr.takeError();
+      return;
+    }
+    StringRef Name = NameOrErr.get();
+    Dst.clear();
+    ConverterEBCDIC::convertToUTF8(Name, Dst);
+    MemberName.append(Dst.str());
+  } else
+    MemberName = RawMemberName;
+
+  // LastModified
+  Dst.clear();
+  StringRef LastModifiedSR = getFieldRawStringE2A(ArMemHdr->LastModified, Dst);
+  if (LastModifiedSR.empty()) {
+    *Err = malformedError("problem converting LastModified field in "
+                          "header at offset " +
+                          Twine(Offset));
+    return;
+  }
+  LastModified.append(LastModifiedSR);
+
+  // UID
+  Dst.clear();
+  StringRef UIDSR = getFieldRawStringE2A(ArMemHdr->UID, Dst);
+  if (UIDSR.empty()) {
+    *Err = malformedError("problem converting UID field in "
+                          "header at offset " +
+                          Twine(Offset));
+    return;
+  }
+  UID.append(UIDSR);
+
+  // GID
+  Dst.clear();
+  StringRef GIDSR = getFieldRawStringE2A(ArMemHdr->GID, Dst);
+  if (GIDSR.empty()) {
+    *Err = malformedError("problem converting GID field in "
+                          "header at offset " +
+                          Twine(Offset));
+    return;
+  }
+  GID.append(GIDSR);
+
+  // AccessMode
+  Dst.clear();
+  StringRef AccessModeSR = getFieldRawStringE2A(ArMemHdr->AccessMode, Dst);
+  if (AccessModeSR.empty()) {
+    *Err = malformedError("problem converting AccessMode field in "
+                          "header at offset " +
+                          Twine(Offset));
+    return;
+  }
+  AccessMode.append(AccessModeSR);
+}
+
 Expected<uint64_t> BigArchiveMemberHeader::getRawNameSize() const {
   return getArchiveMemberDecField(
       "NameLen", getFieldRawString(ArMemHdr->NameLen), Parent, this);
@@ -668,6 +788,8 @@ Expected<std::unique_ptr<Archive>> Archive::create(MemoryBufferRef Source) {
 
   if (Buffer.starts_with(BigArchiveMagic))
     Ret = std::make_unique<BigArchive>(Source, Err);
+  else if (Buffer.starts_with(ZOSArchiveMagic))
+    Ret = std::make_unique<ZOSArchive>(Source, Err);
   else
     Ret = std::make_unique<Archive>(Source, Err);
 
@@ -680,6 +802,10 @@ std::unique_ptr<AbstractArchiveMemberHeader>
 Archive::createArchiveMemberHeader(const char *RawHeaderPtr, uint64_t Size,
                                    Error *Err) const {
   ErrorAsOutParameter ErrAsOutParam(Err);
+
+  if (kind() == K_ZOS)
+    return std::make_unique<ZOSArchiveMemberHeader>(this, RawHeaderPtr, Size,
+      Err);
   if (kind() != K_AIXBIG)
     return std::make_unique<ArchiveMemberHeader>(this, RawHeaderPtr, Size, Err);
   return std::make_unique<BigArchiveMemberHeader>(this, RawHeaderPtr, Size,
@@ -695,7 +821,6 @@ uint64_t Archive::getArchiveMagicLen() const {
 
   return sizeof(ArchiveMagic) - 1;
 }
-
 void Archive::setFirstRegular(const Child &C) {
   FirstRegularData = C.Data;
   FirstRegularStartOfFile = C.StartOfFile;
@@ -714,6 +839,10 @@ Archive::Archive(MemoryBufferRef Source, Error &Err)
     Format = K_AIXBIG;
     IsThin = false;
     return;
+  } else if (Buffer.starts_with(ZOSArchiveMagic)) {
+    Format = K_ZOS;
+    IsThin = false;
+    return;
   } else {
     Err = make_error<GenericBinaryError>("file too small to be an archive",
                                          object_error::invalid_file_type);
@@ -971,6 +1100,8 @@ object::Archive::Kind Archive::getDefaultKindForTriple(const Triple &T) {
     return object::Archive::K_AIXBIG;
   if (T.isOSWindows())
     return object::Archive::K_COFF;
+  if (T.isOSzOS())
+    return object::Archive::K_ZOS;
   return object::Archive::K_GNU;
 }
 
@@ -1042,6 +1173,8 @@ Expected<Archive::Child> Archive::Symbol::getMember() const {
     // the archive of the member that defines the symbol.  Which is what
     // is needed here.
     Offset = read64le(Offsets + SymbolIndex * 16 + 8);
+  } else if (Parent->kind() == K_ZOS) {
+    Offset = read32be(Offsets + SymbolIndex * 8);
   } else {
     // Skip offsets.
     uint32_t MemberCount = read32le(Buf);
@@ -1171,6 +1304,9 @@ Archive::symbol_iterator Archive::symbol_begin() const {
     buf += ran_strx;
   } else if (kind() == K_AIXBIG) {
     buf = getStringTable().begin();
+  } else if (kind() == K_ZOS) {
+    uint32_t symbol_count = read32be(buf);
+    buf += sizeof(uint32_t) + (symbol_count * (sizeof(uint64_t)));
   } else {
     uint32_t member_count = 0;
     uint32_t symbol_count = 0;
@@ -1244,6 +1380,9 @@ uint32_t Archive::getNumberOfSymbols() const {
     return read32le(buf) / 8;
   if (kind() == K_DARWIN64)
     return read64le(buf) / 16;
+  if (kind() == K_ZOS) {
+    return read32be(buf);
+  }
   uint32_t member_count = 0;
   member_count = read32le(buf);
   buf += 4 + (member_count * 4); // Skip offsets.
@@ -1448,3 +1587,71 @@ BigArchive::BigArchive(MemoryBufferRef Source, Error &Err)
   setFirstRegular(*I);
   Err = Error::success();
 }
+
+ZOSArchive::ZOSArchive(MemoryBufferRef Source, Error &Err)
+    : Archive(Source, Err) {
+  ErrorAsOutParameter ErrAsOutParam(&Err);
+
+  // Get the special members.
+  child_iterator I = child_begin(Err, false);
+  if (Err)
+    return;
+  child_iterator E = child_end();
+
+  // See if this is a valid empty archive and if so return.
+  if (I == E) {
+    Err = Error::success();
+    return;
+  }
+  const Child *C = &*I;
+
+  auto Increment = [&]() {
+    ++I;
+    if (Err)
+      return true;
+    C = &*I;
+    return false;
+  };
+
+  Expected<StringRef> NameOrErr = C->getRawName();
+  if (!NameOrErr) {
+    Err = NameOrErr.takeError();
+    return;
+  }
+  StringRef Name = NameOrErr.get();
+
+  if (Name == "__.SYMDEF") {
+    // We know that the symbol table is not an external file, but we still must
+    // check any Expected<> return value.
+    Expected<StringRef> BufOrErr = C->getBuffer();
+    if (!BufOrErr) {
+      Err = BufOrErr.takeError();
+      return;
+    }
+
+    // Copy symbol table converting embedded EBCDIC names to ASCII.
+    StringRef ESymbolTable = BufOrErr.get();
+    uint32_t ESymbolCount = read32be(ESymbolTable.data());
+    uint32_t OffsetToENames =
+        sizeof(uint32_t) + (ESymbolCount * (sizeof(uint64_t)));
+    uint32_t ENamesSize = (uint32_t)ESymbolTable.size() - OffsetToENames;
+    const char *ENamesPtr = (const char *)ESymbolTable.data() + OffsetToENames;
+    StringRef ENames(ENamesPtr, ENamesSize);
+
+    SmallString<64> Dst;
+    ConverterEBCDIC::convertToUTF8(ENames, Dst);
+    SymbolTableBuf.append(ESymbolTable.data(), OffsetToENames);
+    SymbolTableBuf.append(Dst.str());
+    SymbolTable = StringRef(SymbolTableBuf.data(), SymbolTableBuf.size());
+    if (Increment())
+      return;
+    setFirstRegular(*C);
+
+    Err = Error::success();
+    return;
+  }
+
+  setFirstRegular(*C);
+  Err = Error::success();
+  return;
+}
\ No newline at end of file
diff --git a/llvm/test/Object/Inputs/zos-archive-test.a b/llvm/test/Object/Inputs/zos-archive-test.a
new file mode 100644
index 0000000000000..0244f176c6447
Binary files /dev/null and b/llvm/test/Object/Inputs/zos-archive-test.a differ
diff --git a/llvm/test/Object/zos-archive-read.test b/llvm/test/Object/zos-archive-read.test
new file mode 100644
index 0000000000000..7adf2a5256697
--- /dev/null
+++ b/llvm/test/Object/zos-archive-read.test
@@ -0,0 +1,9 @@
+## Test reading a z/OS archive.
+
+# RUN: llvm-ar t %p/Inputs/zos-archive-test.a | FileCheck %s --check-prefix=LIST
+# RUN: llvm-nm --print-armap %p/Inputs/zos-archive-test.a | FileCheck %s --check-prefix=SYMS
+
+# LIST: foo.o
+
+# SYMS: Archive map
+# SYMS-NEXT: foo in foo.o
\ No newline at end of file

[github-actions]

✅ With the latest revision this PR passed the C/C++ code formatter.

[github-actions]

🐧 Linux x64 Test Results

• 195200 tests passed
• 5197 tests skipped

✅ The build succeeded and all tests passed.

[jh7370]

I'll look into this in the next few days (I've got a big stack on my plate to catch up on first), from the point of view of a maintainer of the generic code. It might be worth getting other z/OS experts to review too.

Please remember to avoid force pushes: use fixup commits and push those (see https://llvm.org/docs/GitHub.html#rebasing-pull-requests-and-force-pushes). The squash and merge process will fold these into a single commit with the description based on the PR description, so they are the preferred method for updates. If you need to integrate changes from main, use a merge commit from main (again, this will disappear during squash and merge).

[diggerlin]

I suggest adding as much information as possible about the z/OS archive format in the source code comments, so external reviewers can better understand and review the code.

[diggerlin]

The patch look good to me generally. after you address the comment , I will approve it.

[jh7370]

I've not reviewed the tests yet (I'm out of time today). I am a little concerned by a couple of things:

1. I don't like the use of canned binaries and we generally try to avoid them for newer tests (they're unreviewable and unmaintainable, plus permanently bloat the git repository size). Can we achieve the same goal via some other mechanism, e.g. generating them on the fly?

2. There seems to be inadequate test coverage of all the new code paths. I'd expect everything introduced to be tested.

[uyoyo0]

I don't like the use of canned binaries and we generally try to avoid them for newer tests (they're unreviewable and unmaintainable, plus permanently bloat the git repository size). Can we achieve the same goal via some other mechanism, e.g. generating them on the fly?

Yea this is something that I was a bit concerned about myself. The challenge is that z/OS archives use EBCDIC-encoded headers, magic numbers and symbol table names, so I don't believe there is an existing LLVM tool that can generate a z/OS archive from scratch on a non-z/OS platform. I believe yaml2obj produces ASCII archives and using llvm-ar --format=zos to create the test input would require testing the reader with the writer which has not been reviewed yet.

The binary used for the testing was created on an actual z/OS system to make sure that we're testing against a archive that we know is correct and I've kept it fairly small (single member, single symbol). That said, I'm open to exploring alternatives if you have a preferred approach. Would something like a python script that assembles the binary from hex be acceptable, or could we consider keeping the canned binary given the unique EBCDIC constraints @jh7370 ?

[diggerlin]

I took a look on the python script, I am not a expertise on python, I am generally OK with the script to generate the Z/OS archive.

[diggerlin]

I don't have any further comments on the patch, but please address James's comments and make sure he's happy with it before merging.

[github-actions]

🪟 Windows x64 Test Results

• 134574 tests passed
• 3264 tests skipped

✅ The build succeeded and all tests passed.

[jh7370]

LGTM too!

[dyung]

@uyoyo0 the test added with this change seems to be failing on MacOS bots:

https://lab.llvm.org/buildbot/#/builders/190/builds/42544
https://lab.llvm.org/buildbot/#/builders/23/builds/19989

Can you take a look and revert if you need time to investigate?

[ilovepi]

We're seeing a similar set of failures in our mac aarrch64 bots: https://luci-milo.appspot.com/ui/p/fuchsia/builders/toolchain.ci/clang-mac-arm64/b8682002400978909393/overview

Logs: https://logs.chromium.org/logs/fuchsia/buildbucket/cr-buildbucket/8682002400978909393/+/u/clang/test/stdout

[uyoyo0]

Thanks for catching this @dyung and @ilovepi. The issue seems to be that one of the tests produced a file too small to pass the initial archive size check on macOS. The fix would be to replace the final test

# RUN: printf '\x5A\x4C\x81\x99\x83\x88\x6E\x15' > %t.badhdr.a
# RUN: printf '\x00\x00\x00' >> %t.badhdr.a
# RUN: not llvm-ar t %t.badhdr.a 2>&1 | FileCheck %s --check-prefix=ERR-CHILD
# ERR-CHILD: truncated or malformed archive

with

## Test that a truncated archive with incomplete member header is rejected.
# RUN: %python %p/Inputs/generate_zos_archive.py --output %t.full.a \
# RUN:   --member foo.txt
# RUN: head -c 28 %t.full.a > %t.badhdr.a
# RUN: not llvm-ar t %t.badhdr.a 2>&1 | FileCheck %s --check-prefix=ERR-CHILD
# ERR-CHILD: truncated or malformed archive

This generates a valid archive then truncates it to 28 bytes so that its large enough to pass the initial size check but too small for the 60-byte member header, so it correctly hits the expected failure on all platforms. I don't have commit access. Could one of you push the fix, or should I open a follow-up PR?

[cachemeifyoucan]

Yes, this break macOS: https://green.lab.llvm.org/job/llvm.org/job/clang-stage1-RA-cmake-incremental/job/main/923/testReport/junit/LLVM/Object/zos_archive_read_test/

Do you have a followup PR ready?

[uyoyo0]

Yes, here's the PR: #197290. Sorry for the delay.