Metadata and Full text GET for many items through API and limits

sdspieg · September 14, 2018

Can somebody please advise what the best way is to get large amounts of metadata (and then later also (cached) full-texts) through API URL-requests and without overloading the server or causing any other problems? Thanks!

adamsmith · September 14, 2018

Are you running into specific problems? The API has built-in meachanisms for dealing with this (https://www.zotero.org/support/dev/web_api/v3/basics#rate_limiting ) that you should follow, but since you're mainly just downloading full library content, i.e. essentially the equivalence of a full sync, I don't think there should be any problems even for large libraries.

sdspieg · September 14, 2018

Yes we are. We have one group library that has 8797 items. But the following request - https://api.zotero.org/groups/2225272/items?format=json&key=xxx yields only 25 items...

adamsmith · September 14, 2018

See https://www.zotero.org/support/dev/web_api/v3/basics#link_header
the default item limit is 25, you can use the link header to get the next 25 etc.

sdspieg · September 14, 2018

Update (just documenting in case somebody has the same problem later and finds this thread) - I guess we figured it out. limit integer 1-100 (default - 50). The maximum number of results to return with a single request. Need to do slicing to export api like this: start=0&end=100, start=101&end=200 etc, then combine. I wish we could go start=0&end8797 but I guess that's the mechanism Sebastian was talking about

adamsmith · September 14, 2018

(it's standard for APIs to not let you download single multi-MB-sized data files; that's why using API libraries such as https://pyzotero.readthedocs.io/en/latest/ is so convenient)