This is a Java library for processing bibliographic data in the following formats:
- ISO 2709/Z39.2
- MARC (USMARC, MARC 21, MARC XML)
- MarcXchange (ISO 25577:2013)
- UNIMARC
- MAB (MAB2, MAB XML)
- dialects of MARC (Aleph Sequential, Pica, SISIS format)
The motivation of this library is to transport bibliographic data into XML or JSON based formats, with the focus on european/german application environment.
The most known and widespread bibliographic data format is MARC, which stands for "machine readable cataloging" and was developed by the Library of Congress 1968. Inspired by the success of MARC, several other formats, mostly based on MARC, were developed in the 1970s, some very similar, some with significant differences. Most notable is the UNIMARC format, developed by IFLA.
MARC does not offer the features of XML or JSON, it is not a document format or a format for the Web. MARC is stream-based "structured data", composed of fields in sequential order, and was targeted to write records on magnetic tape. Today, magnetic tape data distribution service is history. Also, file distribution via FTP, common in the 1990s, does not fit well into a highly linked and sophisticated information infrastructure like the Semantic Web.
This library offers the first step in the complex procedure to move MARC data into computer applications of today, by writing MARC fields to XML or JSON formats. More steps would include the generation of graph structures (RDF triples) by processing MARC records in context, but that is not part of this package.
The library provides a fluent interface and a rich set of input streams, content handlers and listeners. Provided are writers for XML, stylesheet transformations (MODS), and a JSON writer for key/value-oriented JSON, suitable for indexing into Elasticsearch. Indexing into Elasticsearch is not part of this package.
Here is a code example for reading from an ISO 2709 stream and writing into a MarcXchange collection.
try (MarcXchangeWriter writer = new MarcXchangeWriter(out)) {
Marc.builder()
.setInputStream(in)
.setCharset(Charset.forName("ANSEL"))
.setMarcListener(writer)
.build()
.writeCollection();
}
Here is an example to create MODS from an ISO 2709 stream
Marc marc = Marc.builder()
.setInputStream(marcInputStream)
.setCharset(Charset.forName("ANSEL"))
.setSchema(MARC21_FORMAT)
.build();
StringWriter sw = new StringWriter();
Result result = new StreamResult(sw);
System.setProperty("http.agent", "Java Agent");
marc.transform(new URL("http://www.loc.gov/standards/mods/v3/MARC21slim2MODS3.xsl"), result);
And here is an example showing how records in "Aleph Sequential") can be parsed and written into a MarcXchange collection:
try (MarcXchangeWriter writer = new MarcXchangeWriter(out, true)
.setFormat(MarcXchangeConstants.MARCXCHANGE_FORMAT)) {
Marc marc = Marc.builder()
.setInputStream(in)
.setCharset(StandardCharsets.UTF_8)
.setMarcListener(writer)
.build();
marc.wrapIntoCollection(marc.aleph());
}
Another example, writing compressed Elasticsearch bulk format JSON from an ANSEL MARC input stream:
MarcValueTransformers marcValueTransformers = new MarcValueTransformers();
// normalize ANSEL diacritics
marcValueTransformers.setMarcValueTransformer(value -> Normalizer.normalize(value, Normalizer.Form.NFC));
// split at 10000 records, select Elasticsearch bulk format, set buffer size 65536, gzip compress = true
try (MarcJsonWriter writer = new MarcJsonWriter("bulk%d.jsonl.gz", 10000,
MarcJsonWriter.Style.ELASTICSEARCH_BULK, 65536, true)
.setIndex("testindex", "testtype")) {
writer.setMarcValueTransformers(marcValueTransformers);
Marc.builder()
.setFormat(MarcXchangeConstants.MARCXCHANGE_FORMAT)
.setType(MarcXchangeConstants.BIBLIOGRAPHIC_TYPE)
.setInputStream(in)
.setCharset(Charset.forName("ANSEL"))
.setMarcListener(writer)
.build()
.writeCollection();
}
where the result can be indexed by a simple bash script using curl, because our JSON
format is compatible to Elasticsearch JSON (which is a key/value format serializable JSON).
#!/usr/bin/env bash
# This example file sends compressed JSON lines formatted files to Elasticsearch bulk endpoint
# It assumes the index settings and the mappings are already created and configured.
for f in bulk*.jsonl.gz; do
curl -XPOST -H "Accept-Encoding: gzip" -H "Content-Encoding: gzip" \
--data-binary @$f --compressed localhost:9200/_bulk
done
The result is a very basic MARC field based index, which is cumbersome to configure, search and analyze. In upcoming projects, I will show how to turn MARC into semantic data with context, and indexing such data makes much more sense and is also more fun.
By executing curl localhost:9200/_search?pretty the result can be examined.
This Java program scans through a MARC file, checks for ISSN values, and collects them in
JSON format (the library org.xbib:content-core:1.0.7 is used for JSON formatting)
public void findISSNs() throws IOException {
Map<String, List<Map<String, String>>> result = new TreeMap<>();
// set up MARC listener
MarcListener marcListener = new MarcFieldAdapter() {
@Override
public void field(MarcField field) {
Collection<Map<String, String>> values = field.getSubfields().stream()
.filter(f -> matchISSNField(field, f))
.map(f -> Collections.singletonMap(f.getId(), f.getValue()))
.collect(Collectors.toList());
if (!values.isEmpty()) {
result.putIfAbsent(field.getTag(), new ArrayList<>());
List<Map<String, String>> list = result.get(field.getTag());
list.addAll(values);
result.put(field.getTag(), list);
}
}
};
// read MARC file
Marc.builder()
.setInputStream(getClass().getResource("issns.mrc").openStream())
.setMarcListener(marcListener)
.build()
.writeCollection();
// collect ISSNs
List<String> issns = result.values().stream()
.map(l -> l.stream()
.map(m -> m.values().iterator().next())
.collect(Collectors.toList()))
.flatMap(List::stream)
.distinct()
.collect(Collectors.toList());
// JSON output
XContentBuilder builder = contentBuilder().prettyPrint()
.startObject();
for (Map.Entry<String, List<Map<String, String>>> entry : result.entrySet()) {
builder.field(entry.getKey(), entry.getValue());
}
builder.array("issns", issns);
builder.endObject();
logger.log(Level.INFO, builder.string());
}
private static boolean matchISSNField(MarcField field, MarcField.Subfield subfield) {
switch (field.getTag()) {
case "011": {
return "a".equals(subfield.getId()) || "f".equals(subfield.getId());
}
case "421":
case "451":
case "452":
case "488":
return "x".equals(subfield.getId());
}
return false;
}
Bibliographic character sets predate the era of Unicode. Before Unicode, characters sets were scattered into several standards. Bibliographic standards were defined on several bibliographic characters sets. Since Unicode, UTF-8 encoding has been accepted as the de-facto standard, which fit into XML and JSON, but processing input data that was created by using bibliographic standards still requires handling of ancient and exotic encodings.
Because Java JDK does not provide bibliographic character sets from before the Unicode era, it must be extended by a a bibliographic character set library. it is recommended to use https://github.com/xbib/bibliographic-character-sets if the input data is encoded in ANSEL/Z39.47 or ISO 5426.
The library can be used as a Gradle dependency
"org.xbib:marc:2.8.0"
or as a Maven dependency
<dependency>
<groupId>org.xbib</groupId>
<artifactId>marc</artifactId>
<version>1.0.11</version>
</dependency>
First, install OpenJDK 8. If in doubt, I recommend SDKMan http://sdkman.io/ for easy installation.
Then clone the github repository
git clone https://github.com/xbib/marc
Then change directory into marc folder and enter
./gradlew test -Dtest.single=MarcFieldFilterTest
for executing the ISSN demo.
Gradle takes care of all the setup in the background.
There is also a Java program called MarcTool which is thought to run without Gradle
It could be extended to include a command for finding ISSNs (essentially, by copying the junit test code into the
MarcTool class, and wiring some suitable arguments into the code).
After
./gradlew assemble
there will find a file called marc-{version}.jar in the build/libs folder. To run this Java program, the command would be something like
java -cp build/libs/marc-1.0.11.jar org.xbib.marc.tools.MarcTool
MarcTool is not perfect yet (it expects some arguments, if not present,
it will merely exit with an unfriendly Exception in thread "main" java.lang.NullPointerException).
To run the Java program as standalone program, including the JSON format as output, some more jar dependency files
must be on the runtime class path (e.g. org.xbib:content-core:1.0.7, com.fasterxml.jackson.core:jackson-core:2.8.4)
In Gradle, the exact dependencies for the JSON format in the junit test class MarcFieldFilterTest
can be found by executing the command
./gradlew dependencies
Then, see section testRuntime.
The XSLT transformation is broken in Java 8u102. Please use Java 8u92 if there are problems, or use Xerces/Xalan.
All contributions are welcome. Any bug reports, comments, or pull requests are welcome, just open an issue at https://github.com/xbib/marc/issues
This project was inspired by MARC4J, but is not related to MARC4J or makes reuse of the source code. It is a completeley new implementation.
There is a MARC4J fork at https://github.com/ksclarke/freelib-marc4j where Kevin S. Clarke implements modern Java features into the MARC4J code base.
For the curious, I tried to compile a feature comparison table to highlight some differences. I am not very familiar with MARC4J, so I appreciate any hints, comments, or corrections.
Feature comparison of MARC4J to xbib MARC
| MARC4J | xbib MARC | |
|---|---|---|
| started by | Bas Peters | Jörg Prante |
| Project start | 2001 | 2016 |
| Java | Java 5 | Java 17+ |
| Build | Ant | Gradle |
| Supported formats | ISO 2709/Z39.2, MARC (USMARC, MARC 21, MARC XML), tries to parse MARC-like formats with a "permissive" parser | ISO 2709/Z39.2, MARC (USMARC, MARC 21, MARC XML), MarcXchange (ISO 25577:2013), UNIMARC, MAB (MAB2, MAB XML), dialects of MARC (Aleph Sequential, Pica, SISIS format) |
| Bibliographic character set support | builtin, auto-detectable | dynamically, via Java Charset API, no autodetection |
| Processing | iterator-based | iterator-based, iterable-based, Java 8 streams for fields, records |
| Transformations | on-the-fly, pattern-based filtering for tags/values, field key mapping, field value transformations | |
| Cleaning | substitute invalid characters with a pattern replacement input stream | |
| Statistics | can count tag/indicator/subfield combination occurences | |
| Concurrency support | can write to handlers record by record, provides a MarcRecordAdapter to turn MARC field events into record events |
|
| JUnit test coverage | extensive testing over all MARC dialects, >80% code coverage | |
| Source Quality Profile | ||
| Jar size | 447 KB (2.7.0) | 150 KB (1.0.11), 194 KB (2.8.0) |
| License | LGPL | Apache |
Copyright (C) 2016-2022 Jörg Prante
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. you may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.