Skip to content
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
1a915a1
Datawrapper loader example
palewire Aug 17, 2024
51b4209
Delete examples/loader-datawrapper/Pipfile
palewire Aug 17, 2024
f4318ac
add the resulting asset in the repo, so we can build it without havin…
Fil Aug 19, 2024
269bd86
link in the examples list
Fil Aug 19, 2024
30919d4
data/chart.html
Fil Aug 19, 2024
0903ddd
Merge branch 'main' into datawrapper-example
palewire Aug 19, 2024
e98f242
Update examples/loader-datawrapper/requirements.txt
palewire Aug 19, 2024
c2c0bf8
Fixes
palewire Aug 19, 2024
4257f14
Update examples/loader-datawrapper/src/data/chart.html.py
palewire Aug 19, 2024
d8d0c17
Change chart
palewire Aug 19, 2024
0754b1c
Trim
palewire Aug 19, 2024
cda1154
Update yarn.lock
palewire Aug 19, 2024
d756ada
a helper for responsive dark mode and height
Fil Aug 21, 2024
6e6ce4b
show the code
Fil Aug 21, 2024
04ab679
override the dark=auto parameter if we pass dark=false
Fil Aug 21, 2024
32a1691
Add dark mode config
palewire Aug 21, 2024
56a06ff
Merge branch 'main' into datawrapper-example
palewire Aug 21, 2024
05d8ddd
Web component
palewire Aug 23, 2024
7c2b1e3
get or create chart
palewire Aug 23, 2024
aa01fed
Merge branch 'main' into datawrapper-example
palewire Aug 23, 2024
dfb3b5d
Add chart.txt to cache this for the example site
palewire Aug 23, 2024
c501f96
Merge branch 'datawrapper-example' of https://github.com/palewire/fra…
palewire Aug 23, 2024
2d559bb
Rename as datawrapper-api
palewire Aug 23, 2024
5c44d33
Rename as datawrapper-api
palewire Aug 23, 2024
34f6f2f
Rename as datawrapper-api
palewire Aug 23, 2024
0c9d940
Start with a simple example
palewire Aug 23, 2024
c8c56b3
dark mode edits, helper function
Fil Aug 23, 2024
20a75ab
Merge branch 'main' into datawrapper-example
palewire Aug 26, 2024
23ea365
Merge branch 'main' into datawrapper-example
Fil Aug 26, 2024
b07b608
wording
Fil Aug 26, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions examples/loader-datawrapper/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
*.pyc
.DS_Store
.venv
/dist/
node_modules/
yarn-error.log
9 changes: 9 additions & 0 deletions examples/loader-datawrapper/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
[Framework examples →](../)

# Datawrapper data loader

View live: <https://observablehq.observablehq.cloud/framework-example-loader-datawrapper/>

This Observable Framework example demonstrates how to write a data loader in Python that generates an HTML [iframe](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/iframe) tag that embeds a [Datawrapper](https://www.datawrapper.de/) chart.

The data loader lives in [`src/data/chart.html.py`](./src/data/chart.html.py).
3 changes: 3 additions & 0 deletions examples/loader-datawrapper/observablehq.config.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
export default {
root: "src"
};
20 changes: 20 additions & 0 deletions examples/loader-datawrapper/package.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
{
"type": "module",
"private": true,
"scripts": {
"clean": "rimraf src/.observablehq/cache",
"build": "rimraf dist && observable build",
"dev": "observable preview",
"deploy": "observable deploy",
"observable": "observable"
},
"dependencies": {
"@observablehq/framework": "^1.7.0"
},
"devDependencies": {
"rimraf": "^5.0.5"
},
"engines": {
"node": ">=18"
}
}
2 changes: 2 additions & 0 deletions examples/loader-datawrapper/requirements.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
pandas
datawrapper
1 change: 1 addition & 0 deletions examples/loader-datawrapper/src/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
/.observablehq/cache/
47 changes: 47 additions & 0 deletions examples/loader-datawrapper/src/data/chart.html.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
import sys

import pandas as pd
from datawrapper import Datawrapper

# Get an extract of arrests made by the Baltimore Police Department
df = pd.read_csv(
"https://raw.githubusercontent.com/palewire/first-automated-chart/main/_notebooks/arrests.csv",
parse_dates=["ArrestDateTime"]
)

# Create a year column
df['year'] = df.ArrestDateTime.dt.year

# Calculate the total number by year
totals_by_year = df.year.value_counts().sort_index().reset_index()

# Connect to the datawrapper api
dw = Datawrapper()

# Create a chart
chart_config = dw.create_chart(
# Headline the chart
title="Baltimore Arrests",
# Set the type
chart_type="column-chart",
# Give the data
data=totals_by_year,
# Configure other bits
metadata={
# Set the description
"describe": {
"source-name": "OpenBaltimore",
"source-url": "https://data.baltimorecity.gov/datasets/baltimore::bpd-arrests/about",
}
}
)

# Pull out the chart's unique identifier
chart_id = chart_config["id"]

# Publish the chart
dw.publish_chart(chart_id)

# Write the chart's embed code to stdout
html = dw.get_iframe_code(chart_id, responsive=True)
sys.stdout.write(html)
83 changes: 83 additions & 0 deletions examples/loader-datawrapper/src/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
# Datawrapper data loader

Here’s a Python data loader that generates an HTML [iframe](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/iframe) tag that embeds a [Datawrapper](https://www.datawrapper.de/) chart.

```python
import sys

import pandas as pd
from datawrapper import Datawrapper

# Get an extract of arrests made by the Baltimore Police Department
df = pd.read_csv(
"https://raw.githubusercontent.com/palewire/first-automated-chart/main/_notebooks/arrests.csv",
parse_dates=["ArrestDateTime"]
)

# Create a year column
df['year'] = df.ArrestDateTime.dt.year

# Calculate the total number by year
totals_by_year = df.year.value_counts().sort_index().reset_index()

# Connect to the datawrapper api
dw = Datawrapper()

# Create a chart
chart_config = dw.create_chart(
# Headline the chart
title="Baltimore Arrests",
# Set the type
chart_type="column-chart",
# Give the data
data=totals_by_year,
# Configure other bits
metadata={
# Set the description
"describe": {
"source-name": "OpenBaltimore",
"source-url": "https://data.baltimorecity.gov/datasets/baltimore::bpd-arrests/about",
}
}
)

# Pull out the chart's unique identifier
chart_id = chart_config["id"]

# Publish the chart
dw.publish_chart(chart_id)

# Write the chart's embed code to stdout
html = dw.get_iframe_code(chart_id, responsive=True)
sys.stdout.write(html)
```

<div class="note">

To run this data loader, you will need to create an API token in Datawrapper and set it as an environment variable named `DATAWRAPPER_ACCESS_TOKEN`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be nice to have detailed instructions to create the venv. In particular since this repo expects the venv to live in .venv (judging from the .gitignore file).

I tried this:

python -m venv .venv
source .venv/bin/activate
pip3 install -r requirements.txt

but it failed with:

error: externally-managed-environment

× This environment is externally managed

To make this work I had to run the last command with the --break-system-packages option 😅:

python -m venv .venv
source .venv/bin/activate
pip3 install -r requirements.txt --break-system-packages

(Maybe it's just me—I appear to be doomed when I need to install python stuff.)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm agnostic on Python package managers, so I'm open to handling this however you want.

My routine is to use pipenv. So, in my world, I run something like this:

pipenv install datawrapper pandas
pipenv run npm run dev

I put the requirements file in my proposals because I saw it that way in other examples and presumed it was your preference. It's not the canonical way to run pipenv, but it's supported, so here's how I did things in this case.

pipenv shell
pipenv install -r requirements.txt

I don't have any experience with venv but there must be a way to make it work without resorting to hacks or unusual options. Let me know how you'd like to proceed.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wow, I didn't know about pipenv; maybe it can save me from my personal python hell!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the DATAWRAPPER_ACCESS_TOKEN can we specify where to create it (https://app.datawrapper.de/account/api-tokens) and the scopes we have to create? I initially tried with a narrow scope (Chart | read  write) and the data loader created the chart, but couldn't read it back. It worked when I enabled more reading capabilities (Folder | read  write; Theme | read; User | read; Visualization | read), but I'm not sure which of those were strictly necessary.

It could also be useful to explain how to set it inside the venv.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm open to whatever environment management technique you want to recommend. I typically use a .env file with the standard pipenv conventions that is excluded from the git repository. In this case, for simplicity's sake I just did something like:

DATAWRAPPER_ACCESS_TOKEN=xxxyyzzz pipenv run npm run dev


You’ll also need python3 and the pandas and datawrapper modules installed and available on your `$PATH`.

</div>

<div class="tip">

We recommend using a [Python virtual environment](https://observablehq.com/framework/loaders#venv), such as with venv or uv, and managing required packages via `requirements.txt` rather than installing them globally.

</div>

The above data loader lives in `data/chart.html.py`, so we can load the data using `data/chart.html` with `FileAttachment`:

```js echo
const iframe = FileAttachment("data/chart.html").text();
```

We can then insert the HTML into the page by creating an element and setting its `innerHTML` attribute:

```html echo
<div id="chart"></div>
```

```js echo
document.getElementById("chart").innerHTML = iframe;
```