Skip to content

Commit

Permalink
add impersonate-browser extra: uses curl_cffi for browser impersonation
Browse files Browse the repository at this point in the history 
This commit tries to more closely mimic web browser behavior in an attempt to
circumvent Cloudflare's bot protection.

If built with the 'impersonate-browser' extra, garminexport will use curl_cffi
and a patched libcurl to produce TLS fingerprints that are identical to those of
a real web browsers.

The prior 'cloudflare' extra is dropped in favor of the new
'impersonate-browser' extra.
  • Loading branch information
@petergardfjall
petergardfjall committed Nov 11, 2023
1 parent e03d125 commit e30897d
Show file tree
Hide file tree
Showing 6 changed files with 179 additions and 176 deletions.
2 changes: 1 addition & 1 deletion Makefile
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ venv: $(VENV_DIR)

# install pinned dependencies and package itself in editable mode
dep-sync: $(VENV_DIR)
(source $(VENV_ACTIVATE); pip install -r requirements-dev.txt ; pip install -e . --no-deps)
(source $(VENV_ACTIVATE); pip install -r requirements-dev.txt ; pip install -e .[impersonate_browser] --no-deps)

# creates a virtualenv with development dependencies installed
dev-init: dep-sync
Expand Down
127 changes: 71 additions & 56 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,50 +5,69 @@

# About

`garminexport` is both a library and a tool for downloading/backing up [Garmin
Connect](http://connect.garmin.com/) activities to a local disk.
`garminexport` is both a library and a tool for downloading/backing up
[Garmin Connect](http://connect.garmin.com/) activities to a local disk.

The main utility script is called `garmin-backup` and performs incremental
backups of your Garmin account to a local directory. The first time
`garmin-backup` is run, it will download *all* activities. After that, it will
`garmin-backup` is run, it will download _all_ activities. After that, it will
do incremental backups of your account. That is, the script will only download
activities that haven't already been downloaded to the backup directory.


# Installation

`garminexport` is available on [PyPi](https://pypi.org/) and can be installed
with [pip](http://pip.readthedocs.org):
with [pip](http://pip.readthedocs.org).

> **WARNING**
>
> GarminConnect employs Cloudflare's bot protection to prevent scripted access
> to their services. Therefore a [vanilla-installation](#vanilla-installation)
> is no longer likely to work. Instead, try the
> [browser-impersonating installation](#browser-impersonating-installation).
## Vanilla installation

To only install `garminexport` and required dependencies run:

``` bash
```bash
pip install garminexport
```

## Browser-impersonating installation

To install `garminexport` with support to circumvent Cloudflare's bot protection
(which has been known to impact some users) you can install the module with the
`cloudflare`
[extra](https://setuptools.pypa.io/en/latest/userguide/dependency_management.html#optional-dependencies) like so:
you should install the module with the `impersonate_browser`
[extra](https://setuptools.pypa.io/en/latest/userguide/dependency_management.html#optional-dependencies)
like so:

``` bash
pip install garminexport[cloudflare]
```bash
pip install garminexport[impersonate_browser]
```

This replaces the default [requests](https://github.com/psf/requests) library
with [cloudscraper](https://github.com/VeNoMouS/cloudscraper) for HTTP request.
with [curl_cffi](https://github.com/yifeikong/curl_cffi) for HTTP session
handling.

When `curl_cffi` is used, the `GARMINEXPORT_IMPERSONATE_BROWSER` environment
variable can be used to control which browser is impersonated (default is
`chrome110`, see
[full list](https://github.com/lwthiker/curl-impersonate#supported-browsers)).

# Usage

## Prerequisites
To be of any use you need to register an account at [Garmin
Connect](http://connect.garmin.com/) and populate it with some activities.

To be of any use you need to register an account at
[Garmin Connect](http://connect.garmin.com/) and populate it with some
activities.

## As a command-line tool (garmin-backup)

The backup program is run as follows (use the `--help` flag for a full list of
available options):

``` bash
```bash
garmin-backup --backup-dir=activities <username or email>
```

Expand All @@ -57,54 +76,52 @@ in to your Garmin Connect account to download activities to the specified backup
directory on your machine. The program will only download activities that aren't
already in the backup directory.

Activities can be exported in any of the formats outlined below. Note that
by default, the program downloads all formats for every activity. Use the
Activities can be exported in any of the formats outlined below. Note that by
default, the program downloads all formats for every activity. Use the
`--format` option to narrow the selection.

Supported export formats:

- `gpx`: activity GPX file (XML).

- `gpx`: activity GPX file (XML).
<sub>[GPX](https://en.wikipedia.org/wiki/GPS_Exchange_Format) is an open
format, mainly for storing GPS routes/tracks. It does support extensions and
Garmin appears to annotate the GPS data with, for example, heart-rate and
cadence, when available on your device.</sub>

<sub>[GPX](https://en.wikipedia.org/wiki/GPS_Exchange_Format) is an open
format, mainly for storing GPS routes/tracks. It does support extensions
and Garmin appears to annotate the GPS data with, for example, heart-rate
and cadence, when available on your device.</sub>
- `tcx`: an activity TCX file (XML). _Note: a `.tcx` file may not always be
possible to export, for example if an activity was uploaded in gpx format. In
that case, Garmin won't try to synthesize a tcx file._

- `tcx`: an activity TCX file (XML).
*Note: a `.tcx` file may not always be possible to export, for example
if an activity was uploaded in gpx format. In that case, Garmin won't try
to synthesize a tcx file.*
<sub>[TCX](https://en.wikipedia.org/wiki/Training_Center_XML) (Training Center
XML) is Garmin's own XML format. It is, essentially, an extension of GPX which
includes more metrics and divides the GPS track into "laps" as recorded by
your device (with "lap summaries" for each metric).</sub>

<sub>[TCX](https://en.wikipedia.org/wiki/Training_Center_XML) (Training
Center XML) is Garmin's own XML format. It is, essentially, an extension
of GPX which includes more metrics and divides the GPS track into "laps"
as recorded by your device (with "lap summaries" for each metric).</sub>
- `fit`: activity FIT file (binary format). _Note: a `.fit` file may not always
be possible to export, for example if an activity was entered manually rather
than imported from a Garmin device._

- `fit`: activity FIT file (binary format).
*Note: a `.fit` file may not always be possible to export, for example
if an activity was entered manually rather than imported from a Garmin device.*
<sub>The [FIT](https://www.thisisant.com/resources/fit/) format is the "raw
data type" stored in your Garmin device and should contain all metrics your
device is capable of tracking (GPS, heart rate, cadence, etc). It's a binary
format, so tools are needed to read its content.</sub>

<sub>The [FIT](https://www.thisisant.com/resources/fit/) format is the
"raw data type" stored in your Garmin device and should contain all
metrics your device is capable of tracking (GPS, heart rate, cadence,
etc). It's a binary format, so tools are needed to read its content.</sub>
- `json_summary`: activity summary file (JSON).

- `json_summary`: activity summary file (JSON).
<sub>Provides summary data for an activity. Seems to lack a formal schema and
should not be counted on as a stable data format (it may change at any time).
Only included since it _may_ contain additional data that could be useful for
developers of analysis tools.</sub>

<sub>Provides summary data for an activity. Seems to lack a formal schema
and should not be counted on as a stable data format (it may change at any
time). Only included since it *may* contain additional data that could be
useful for developers of analysis tools.</sub>
- `json_details`: activity details file (JSON).

- `json_details`: activity details file (JSON).
<sub>Provides detailed activity data in a JSON format. Seems to lack a formal
schema and should not be counted on as a stable data format (it may change at
any time). Only included since it _may_ contain additional data that could be
useful for developers of analysis tools.</sub>

<sub>Provides detailed activity data in a JSON format. Seems to lack a
formal schema and should not be counted on as a stable data format (it may
change at any time). Only included since it *may* contain additional data
that could be useful for developers of analysis tools.</sub>

All files are written to the same directory (`activities/` by default). Each
All files are written to the same directory (`activities/` by default). Each
activity file is prefixed by its upload timestamp and its activity id.

`garminexport` also contains a few smaller utility programs:
Expand All @@ -121,14 +138,14 @@ To build your own tools around the Garmin Connect API you can import the
with Garmin Connect. For example use, have a look at the command-line tools
under [garminexport/cli](garminexport/cli).

For example, in your `setup.py`, `setup.cfg`, `pyproject.toml` ([PEP
631](https://peps.python.org/pep-0631/)) add something like:
For example, in your `setup.py`, `setup.cfg`, `pyproject.toml`
([PEP 631](https://peps.python.org/pep-0631/)) add something like:

``` python
```python
install_requires=[
'garminexport',
# also installs 'cloudscraper' as a dependency
# 'garminexport[cloudflare]',
# also installs 'impersonate_browser as a dependency
# 'garminexport[impersonate_browser]',
...
]
```
Expand All @@ -148,7 +165,5 @@ development environment) and install the required dependencies like so:
# activate virtualenv
source .venv/bin/activate

# code ...

# test
make test
9 changes: 0 additions & 9 deletions garminexport/cli/backup.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -18,11 +18,6 @@
DEFAULT_MAX_RETRIES = 7
"""The default maximum number of retries to make when fetching a single activity."""

DEFAULT_USER_AGENT = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36'
"""The default `User-Agent` to use for HTTP requests when none is supplied by
the user.
"""

def parse_args() -> argparse.Namespace:
"""Parse CLI arguments.
Expand Down Expand Up @@ -63,9 +58,6 @@ def parse_args() -> argparse.Namespace:
help=("The maximum number of retries to make on failed attempts to fetch an activity. "
"Exponential backoff will be used, meaning that the delay between successive attempts "
"will double with every retry, starting at one second. DEFAULT: {}").format(DEFAULT_MAX_RETRIES))
parser.add_argument(
"--user-agent", type=str, default=DEFAULT_USER_AGENT,
help="A value to use for the `User-Agent` request header. Use an authentic browser agent string to prevent being blocked by Garmin. A tool such as `user_agent` (`ua`) can be used to generate such values.")

return parser.parse_args()

Expand All @@ -77,7 +69,6 @@ def main():
try:
incremental_backup(username=args.username,
password=args.password,
user_agent_fn=lambda:DEFAULT_USER_AGENT,
backup_dir=args.backup_dir,
export_formats=args.format,
ignore_errors=args.ignore_errors,
Expand Down
Loading

0 comments on commit e30897d

Please sign in to comment.