Zotero causing continuous article-download request loop?

rbrekhus

I received the following email from our electronic resources librarian this morning. Is Zotero aware of this issue or the setting that could be causing it?

...

Good morning.
Since late last week, I’ve received messages from some users and some vendors regarding unusual activity when attempting to access or download articles from some of the libraries’ online resources.

I am hearing from my colleagues at other institutions across the country that they are receiving similar reports. We are all still troubleshooting, but it appears that there may be some issue with Zotero.

For some users, Zotero is requesting permission to access via the proxy. If the user gets this message, they can accept it and it should permit access.

For some users, they are receiving blocks when trying to download articles. As best we can tell at the moment, it seems that Zotero sets off a loop of continuous attempts to download the article that then triggers an automatic block from the publishers’ site. Disabling Zotero or using a different browser should allow the user to access.

My colleagues and I, as well as some publishers, are investigating. I wanted to alert you in case a user contacts you with an access issue. Please send those reports to me so that I can assist them and gather information to help with the troubleshooting.

rbrekhus

(I recommend Zotero all the time and teach workshops on how to use it...so I don't like it when "disable Zotero" is the solution offered. Is it just a matter of unchecking one or two of the File Handling preferences checkboxes? Those don't *seem* like they would cause this problem, but maybe they do??)

adamsmith

The only function that could even possibly cause this is the "Find Available PDF" function when triggered manually. That has, in the past, had some looping errors, but a) those were for inaccessible resources and b) have been fixed.

I don't believe I have seen any reports of this here (though it's possible, if rare, that Zotero has received something privately), so any details you can find out would be helpful. The link to the proxy (is this a web or a system proxy?) seems very odd.

rbrekhus

I've asked my colleague to comment on this thread, as I don't know the difference between a web proxy and a system proxy.

For clarification, when you say "those were for inaccessible resources," do you mean, when users tried to use the function to access something their university doesn't subscribe to/provide?

adamsmith

For clarification, when you say "those were for inaccessible resources," do you mean, when users tried to use the function to access something their university doesn't subscribe to/provide?

Yes, either that or when unpaywall (which Zotero uses to look up OA PDFs where available) gave a PDF URL that returned an error.

dstillman

We haven't heard anything about this, so we'd need more info.

I'm pretty sure this would be about the Zotero Connector's proxy support, not "Find Available PDF" in the desktop app or automatic OA PDF downloading. The latest Zotero Connector release made some improvements to automatic proxy detection and redirection (separating the Login URL Scheme and the Proxied URL Scheme), and it's possible there's some bug or incompatibility there. But we'd need a more specific report, ideally with Debug IDs from the Zotero Connector and Zotero for reproducing it, to say more.

rbrekhus

As far as I understand this problem, it's not a problem that occurs when someone is explicitly trying to use Zotero; it is a problem that occurs when someone has a Zotero connector installed, and Zotero does something in the background that cuts off the person's access to the full-text document altogether - something that causes repeated requests for the PDF, which the database vendor reads as "unusual activity" by that user, triggering the lack of access. So, I don't think this is exactly generating debug ID's in Zotero, because it's the mere presence of the plugin, not its intentional use, that appears to be causing the problem.

dstillman

A Debug ID is a log of Zotero activity that you generate while performing a specific activity so that we can see what exactly the software is doing. The problem described here would likely be related to Zotero's automatic proxy redirection feature, which would only run when loading a site using a previously detected and user-accepted proxy URL scheme, so the relevant activity would be whatever browsing activity results in an error, timeout, etc. It's always possible to provide a Debug ID, and we can't help further without one.

adamsmith

You'd want that debug from the browser connector or from Zotero?

dstillman

If it's triggered just by browsing, the Connector would be sufficient. If it's triggered by saving, we'd need from both.

rwschwab

Our institution has been seeing this for about 10 days now. It seems to be specifically affecting Chrome browser on MacOS. We've had breach reports mainly from Taylor & Francis, but also Oxford English Dictionary, Science.org, U of Chicago Press, and Web of Science.

I'm beginning to ask patrons and colleagues to try and get a debug report of the issue. In the meantime, wanted to share the specifics of what we've seen so far.

dstillman

@rbrekhus, @rwschwab, in addition to Debug IDs, if you have contacts at the publishers that have reported this, please ask them to send any technical details (e.g., server logs) for what they're seeing to support@zotero.org.

rbrekhus

Latest email from our e-resources librarian:

"We spent last week working with various users and providers, as well as our colleagues at other institutions on this issue.

Apparently, recent web browser updates for Chrome and Firefox caused some problems with third-party plug-ins. We were seeing the issue with Zotero as that was the plug-in our users had installed but other institutions were seeing the same issue with other plug-ins as well.

The reports started tapering off at the end of last week and no new reports were received over the weekend. We will continue monitoring but the suspicious activity reports we received from vendors have been deemed resolved and the result of these browser issues rather than user action.

Please send users to me if they are experiencing any problems with access. While we must work with the vendors to clear the IP blocks, other errors this causes can be handled by the user clearing their cache and/or disabling the plug-in."

rkwlib

Thank you for following up on this. Working with vendors, they have identified recent browser changes affecting third-party plug-ins like Zotero. While our institution was seeing the issue with our Zotero users, it seems others were reporting similar issues with other plug-ins.

dstillman

We still would want actual technical information on this from vendors or users in order to investigate further. "Recent web browser updates" did not cause problems with Zotero, and obviously no one should need to disable the Zotero Connector.

mdinatale

At AAAS this is resulting in a number of libraries having IP addresses banned temporarily for violating out abuse monitoring policies which trigger if someone attempts to access the same article 200 times in 60 seconds. Our platform vendor Atypon is aware that there is an issue stemming from the Zotero Connector. They've had trouble reaching out. Is there a contact I can put them in touch with to discuss further?

dstillman

Yes, they can contact us at support@zotero.org. Please do have them reach out, as we're not able to debug this without additional technical details of what they're seeing on their side.

mdinatale

Thanks - I've shared that email with our technical contact. Hopfully they'll reach out soon.

dstillman

While we wait for any vendors to reach out, a question for the folks who have seen this (@rbrekhus, @rwschwab, @mdinatale, and any others):

At your institution, with the Zotero Connector disabled, if you're on campus and you try to access a gated resource via your institution's proxy (https://www-example-com.proxy.school.edu), are you redirected back to an unproxied domain (https://example.com/), or do you remain on the proxied domain even though the proxy isn't necessary for on-campus access?

One theory we're pursuing is that this is a combination of 1) greatly improved proxy detection in the latest version of the Zotero Connector, causing automatic proxy redirection to be enabled for many more people and 2) a failure to detect a redirect loop if a proxy server redirects on-campus clients back to an unproxied domain. That could result in repeated requests to a vendor site for someone who previously used the proxy off-campus and then returned to campus. (It likely wouldn't result in repeated PDF downloads, but we suspect that's just imprecision in some of the reports and this is actually about webpage requests.)

We're working to add redirect-loop detection to the Zotero Connector (which it should have regardless, and used to many years ago), so if that is the cause, we'll have a fix out soon, but it'd be good to know if this is a plausible source of the current problem.

dstillman

Actually, the current Connector already should be detecting a redirect loop if a proxy redirects to an unproxied domain for on-campus access. We'd still be curious about the answer to my question above about the proxy behavior at the affected institutions, since it's possible there was some regression in that code.

rwschwab

@dstillman, we use EZProxy, and do have it set up to redirect off the proxy while on a campus IP address.

As an example, while off-campus I can access this ebook through our proxy: https://ebookcentral-proquest-com.oca.ucsc.edu/lib/ucsc/detail.action?pq-origsite=primo&docID=4940559

Then, if I sign into VPN and gain an on-campus IP address, visiting that link again ultimately sends me here:
https://ebookcentral.proquest.com/lib/ucsc/detail.action?pq-origsite=primo&docID=4940559

I also have Zotero connecter installed but not configured at all. After arriving on the 2nd page I get this status message:
Zotero detected that you are accessing ebookcentral.proquest.com through a proxy. Would you like to automatically redirect future requests to ebookcentral.proquest.com through %h/%p?

dstillman

OK, we believe we've identified the problem.

As we suspected, the latest version of the Zotero Connector was failing to detect a redirect loop if someone who previously used a proxy later came onto campus (or used a VPN) and the proxy redirected back to an unproxied URL. We had code to deal with that previously, but the latest Connector version changed the proxy redirection process in a way that broke the loop detection.

We've now updated the Connector to properly detect these redirect loops. When a loop is detected, the Connector will suspend automatic proxy redirection for an hour. You can force it to continue redirecting by right-clicking on the save button and choosing Reload via Proxy or by clicking "Reenable proxy redirection" in the Proxies pane of the preferences.

The fix is included in Zotero Connector 5.0.104 for Firefox, available now, and it will be in version 5.0.104 for Chrome and 5.0.105 for Edge, which should be available later today, as soon as they're approved by Google and Microsoft.

Sorry for the trouble this caused, and thanks for helping us troubleshoot it.

And the good news is that, once these fixes are out, automatic proxy detection should work better in general at many more institutions, with many more people benefiting from automatic proxy redirection.

jhharris63

@dstillman Great news! Thank you for diagnosing the issue and making the necessary changes to the Zotero Connector. Will users need to manually update their connectors, or can you push the updates to them?

dstillman

@jhharris63: People will automatically get updated to the latest version within a few hours of it being available for their browser. (We're still waiting for the Chrome and Edge versions to be approved, though.)

dstillman

5.0.105 for Edge is now available with this fix.

dstillman

5.0.104 for Chrome is now available with this fix.