A library to easily scrape metadata from an article on the web using Open Graph, JSON+LD, regular HTML metadata, and series of fallbacks.
- Table of Contents
- Getting Started
- Installation
- Usage
- Metadata
- How it works
- Importing Rules
- Rules bundles
- API
- Benchmark
- License
metascraper is library to easily scrape metadata from an article on the web using Open Graph metadata, regular HTML metadata, and series of fallbacks.
It follows a few principles:
- Have a high accuracy for online articles by default.
- Make it simple to add new rules or override existing ones.
- Don't restrict rules to CSS selectors or text accessors.
$ npm install metascraper --saveLet's extract accurate information from the following article:
Then call metascraper with the rules bundle you want to apply for extracting content:
const metascraper = require('metascraper')([
require('metascraper-author')(),
require('metascraper-date')(),
require('metascraper-description')(),
require('metascraper-image')(),
require('metascraper-logo')(),
require('metascraper-clearbit')(),
require('metascraper-publisher')(),
require('metascraper-title')(),
require('metascraper-url')()
])
const got = require('got')
const targetUrl = 'http://www.bloomberg.com/news/articles/2016-05-24/as-zenefits-stumbles-gusto-goes-head-on-by-selling-insurance'
;(async () => {
const { body: html, url } = await got(targetUrl)
const metadata = await metascraper({ html, url })
console.log(metadata)
})()The output will be something like:
{
"author": "Ellen Huet",
"date": "2016-05-24T18:00:03.894Z",
"description": "The HR startups go to war.",
"image": "https://assets.bwbx.io/images/users/iqjWHBFdfxIU/ioh_yWEn8gHo/v1/-1x-1.jpg",
"publisher": "Bloomberg.com",
"title": "As Zenefits Stumbles, Gusto Goes Head-On by Selling Insurance",
"url": "http://www.bloomberg.com/news/articles/2016-05-24/as-zenefits-stumbles-gusto-goes-head-on-by-selling-insurance"
}?> Other metadata can be defined using a custom rule bundle.
Here is an example of the metadata that metascraper can collect:
-
audio— eg. https://cf-media.sndcdn.com/U78RIfDPV6ok.128.mp3
A audio URL that best represents the article. -
author— eg. Noah Kulwin
A human-readable representation of the author's name. -
date— eg. 2016-05-27T00:00:00.000Z
An ISO 8601 representation of the date the article was published. -
description— eg. Venture capitalists are raising money at the fastest rate...
The publisher's chosen description of the article. -
video— eg. https://assets.entrepreneur.com/content/preview.mp4
A video URL that best represents the article. -
image— eg. https://assets.entrepreneur.com/content/3x2/1300/20160504155601-GettyImages-174457162.jpeg
An image URL that best represents the article. -
lang— eg. en
An ISO 639-1 representation of the url content language. -
logo— eg. https://entrepreneur.com/favicon180x180.png
An image URL that best represents the publisher brand. -
publisher— eg. Fast Company
A human-readable representation of the publisher's name. -
title— eg. Meet Wall Street's New A.I. Sheriffs
The publisher's chosen title of the article. -
url— eg. http://motherboard.vice.com/read/google-wins-trial-against-oracle-saves-9-billion
The URL of the article.
metascraper is built out of rules bundles.
It was designed to be easy to adapt. You can compose your own transformation pipeline using existing rules or write your own.
Rules bundles are a collection of HTML selectors around a determinate property. When you load the library, implicitly it is loading core rules.
Each set of rules load a set of selectors in order to get a determinate value.
These rules are sorted with priority: The first rule that resolve the value successfully, stop the rest of rules for get the property. Rules are sorted intentionally from specific to more generic.
Rules work as fallback between them:
- If the first rule fails, then it fallback in the second rule.
- If the second rule fails, time to third rule.
- etc
metascraper do that until finish all the rule or find the first rule that resolves the value.
metascraper exports a constructor that need to be initialized providing a collection of rules to load:
const metascraper = require('metascraper')([
require('metascraper-author')(),
require('metascraper-date')(),
require('metascraper-description')(),
require('metascraper-image')(),
require('metascraper-logo')(),
require('metascraper-clearbit')(),
require('metascraper-publisher')(),
require('metascraper-title')(),
require('metascraper-url')()
])Again, the order of rules are loaded are important: Just the first rule that resolve the value will be applied.
Use the first parameter to pass custom options specific per each rules bundle:
const metascraper = require('metascraper')([
require('metascraper-clearbit')({
size: 256,
format: 'jpg'
})
])?> Can't find the rules bundle that you want? Let's open an issue to create it.
See CONTRIBUTING.
Create a new metascraper instance declaring the rules bundle to be used explicitly.
Type: Array
The collection of rules bundle to be loaded.
Call the instance for extracting content based on rules bundle provided at the constructor.
Required
Type: String
The URL associated with the HTML markup.
It is used for resolve relative links that can be present in the HTML markup.
it can be used as fallback field for different rules as well.
Type: String
The HTML markup for extracting the content.
Type: Array
You can pass additional rules to add on execution time.
These rules will be merged with your loaded rules at the beginning.
To give you an idea of how accurate metascraper is, here is a comparison of similar libraries:
| Library | metascraper |
html-metadata |
node-metainspector |
open-graph-scraper |
unfluff |
|---|---|---|---|---|---|
| Correct | 95.54% | 74.56% | 61.16% | 66.52% | 70.90% |
| Incorrect | 1.79% | 1.79% | 0.89% | 6.70% | 10.27% |
| Missed | 2.68% | 23.67% | 37.95% | 26.34% | 8.95% |
A big part of the reason for metascraper's higher accuracy is that it relies on a series of fallbacks for each piece of metadata, instead of just looking for the most commonly-used, spec-compliant pieces of metadata, like Open Graph.
metascraper's default settings are targetted specifically at parsing online articles, which is why it's able to be more highly-tuned than the other libraries for that purpose.
If you're interested in the breakdown by individual pieces of metadata, check out the full comparison summary, or dive into the raw result data for each library.
metascraper © Ian Storm Taylor, Released under the MIT License.
Maintained by Kiko Beats with help from contributors.