Shimmer making large number of requests to FitBit API?

[nosh]

I'm making a single request to shimmer for my physical activity data from FitBit. e.g. for the last 6-12 months:
http://omh-shimmer.datanosh.com:8083/data/withings/[email protected]&dateStart=2015-01-01&dateEnd=2015-12-15&normalize=true

This should only result in a few (maybe < 100 records). I seem to be frequently getting messages saying that my API limits have been expected:
There was an unexpected error (type=Internal Server Error, status=500).
org.springframework.web.util.NestedServletException: Request processing failed; nested exception is org.openmhealth.shim.common.mapper.MissingJsonNodeMappingException: The required field 'activities' wasn't found in node '{"errors":[{"errorType":"request","fieldName":"n/a","message":"Rate limit exceeded for this user. Please try again at the start of the hour. More information about rate limiting is at https://dev.fitbit.com/."}],"success":false}'.

Given that the published API limits are 150 requests/hour per user, and I don't think my code is making a crazy number of requests, is it possible Shimmer is making a large number of requests to FitBit for each call to Shimmer, consequently causing it to exceed the API limits?

[emersonf]

Shimmer has to make a large number of requests. The Fitbit activity endpoints only allow the retrieval of one day's worth of data at a time. Shimmer issues one request per day and composes the result.

We have plans to be aware of rate limit information and fail fast if we determine the number of requests required will exceed a rate limit, but besides that there's not much that can be done. Fitbit is the only supported API that does this, IIRC, at least for summary data.

[nosh]

Ok - so it seems like that is answer - for physical activity, this is a limitation of the daily activity summary end point, which is required to get the details of the physical activity.

Is it correct to assume that this limitation doesn't impact the step count and other endpoints, since you can use Fitbit's time series and body endpoints which seem to allow data retrieval by larger time ranges?

[emersonf]

The status quo is more complicated, because of two factors. The first is that Shimmer needed significant refactoring to be able to properly support pagination, dynamic request building, and request composition. That work is well underway, and we should release it in January. The second is that the Fitbit API has changed since the Fitbit implementation, adding new data types and moving to OAuth 2.0. We've been holding off making additions until we finish the refactoring, both to retain focus and because the refactoring will simplify the additions.

This table shows the status quo. The days per Fitbit query column shows how many days of data the Fitbit endpoint can return in a single request. The days per Shimmer request column shows how many days of data Shimmer requests in one call. The only inconsistencies are BMI and step count, which will be fixed. There are some endpoints that Shimmer doesn't support for a given data type, indicated by n/a, which should be added post refactoring, except for physical activity which doesn't return enough data.

data type	endpoint	days per Fitbit query	days per Shimmer request	note
BMI	body/log/weight	multiple	single	to be fixed (#34)
weight	body/log/weight	multiple	multiple
physical activity	activities/date	single	single
physical activity	compose activities/*	multiple	n/a	missing required activity name
step count	activities/steps	multiple	single	to be fixed (#33)
step count (intraday)	activities/date	single	single
sleep duration	sleep/date	single	single
sleep duration	compose sleep/*	multiple	n/a	required refactoring
heart rate	activities/heart	multiple	n/a	new addition and requires refactoring to OAuth 2.0
heart rate (intraday)	activities/heart	single	n/a	new addition and requires refactoring to OAuth 2.0

[nosh]

Thanks for the detailed explanation. It's much clearer now!
Glad to see #33 (and #34) on the enhancement list. This is pretty important as it's easy to blow through the fitbit default rate limits with just a few calls to shimmer when looking back through a few months of data, consequently rendering an app effectively useless till the rate limits reset.

Let me know how I can help

[hisham]

@emersonf is the refactoring you mention done?

[emersonf]

Not yet. We have work in progress, but haven't completed it. We'll be
planning some new sprints to carve off pieces of that work, test and
release them in iterations, and I'll make it a point to keep this thread
updated when we have that.
On 1 Feb 2016 12:28 p.m., "hisham" [email protected] wrote:

@emersonf https://github.com/emersonf is the refactoring you mention
done?

—
Reply to this email directly or view it on GitHub
#32 (comment).

[hisham]

Ok thanks @emersonf!