Massive download reported by ASCE

grolimur · March 6, 2017

Hi,

A publisher (ascelibrary.org) blocked 2-3 IP addresses and onctacted the library because of text and data mining (TDM) or massive download activities (about 300 downloads in less than a minute).

The researchers were contacted and say they were using Zotero. There was no TDM or massive download.
But as far as I know, Zotero can only download the references and PDFs displayed on the screen (max. 100 on ascelibrary.org), not more.

Does anyone else face the same problem?
And has anyone an answer to the question 'how does this happen'?

adamsmith · March 6, 2017

Yeah, Zotero isn't doing anything unusual, as you say.
It's possible, though, to have multiple search pages up at the same time and then scrape all search results at once and ASCE may balk at that (though I think 300 is rather stingy).

dstillman · March 6, 2017

I haven't looked at how our translator for ASCE works, but worth noting that saving items from search results to Zotero often involves one or more additional requests per item, including at least a PDF download for every item. If you save all items from a search results page — and particularly if you increase the result limit — sites very well may block you, because what you're doing is essentially automated bulk downloading.

People tend to run into this when they try to use Zotero for systematic reviews, saving lots of data from sites into Zotero and analyzing it after the fact. That's not really an approach we recommend, largely because of these issues.

A possible workaround is to use the site's native functionality for exporting a BibTeX/RIS file with data for a large set of items and import that into Zotero, though that wouldn't include attached files. (Eventually we may add the ability to retrieve a PDF (if you have access) for a given item, which would help with that workflow.)