Zotero causing continuous article-download request loop?

rbrekhus

I received the following email from our electronic resources librarian this morning. Is Zotero aware of this issue or the setting that could be causing it?

...

Good morning.
Since late last week, I’ve received messages from some users and some vendors regarding unusual activity when attempting to access or download articles from some of the libraries’ online resources.

I am hearing from my colleagues at other institutions across the country that they are receiving similar reports. We are all still troubleshooting, but it appears that there may be some issue with Zotero.

For some users, Zotero is requesting permission to access via the proxy. If the user gets this message, they can accept it and it should permit access.

For some users, they are receiving blocks when trying to download articles. As best we can tell at the moment, it seems that Zotero sets off a loop of continuous attempts to download the article that then triggers an automatic block from the publishers’ site. Disabling Zotero or using a different browser should allow the user to access.

My colleagues and I, as well as some publishers, are investigating. I wanted to alert you in case a user contacts you with an access issue. Please send those reports to me so that I can assist them and gather information to help with the troubleshooting.

rbrekhus

(I recommend Zotero all the time and teach workshops on how to use it...so I don't like it when "disable Zotero" is the solution offered. Is it just a matter of unchecking one or two of the File Handling preferences checkboxes? Those don't *seem* like they would cause this problem, but maybe they do??)

adamsmith

The only function that could even possibly cause this is the "Find Available PDF" function when triggered manually. That has, in the past, had some looping errors, but a) those were for inaccessible resources and b) have been fixed.

I don't believe I have seen any reports of this here (though it's possible, if rare, that Zotero has received something privately), so any details you can find out would be helpful. The link to the proxy (is this a web or a system proxy?) seems very odd.

rbrekhus

I've asked my colleague to comment on this thread, as I don't know the difference between a web proxy and a system proxy.

For clarification, when you say "those were for inaccessible resources," do you mean, when users tried to use the function to access something their university doesn't subscribe to/provide?

adamsmith

For clarification, when you say "those were for inaccessible resources," do you mean, when users tried to use the function to access something their university doesn't subscribe to/provide?

Yes, either that or when unpaywall (which Zotero uses to look up OA PDFs where available) gave a PDF URL that returned an error.

dstillman

We haven't heard anything about this, so we'd need more info.

I'm pretty sure this would be about the Zotero Connector's proxy support, not "Find Available PDF" in the desktop app or automatic OA PDF downloading. The latest Zotero Connector release made some improvements to automatic proxy detection and redirection (separating the Login URL Scheme and the Proxied URL Scheme), and it's possible there's some bug or incompatibility there. But we'd need a more specific report, ideally with Debug IDs from the Zotero Connector and Zotero for reproducing it, to say more.

rbrekhus

As far as I understand this problem, it's not a problem that occurs when someone is explicitly trying to use Zotero; it is a problem that occurs when someone has a Zotero connector installed, and Zotero does something in the background that cuts off the person's access to the full-text document altogether - something that causes repeated requests for the PDF, which the database vendor reads as "unusual activity" by that user, triggering the lack of access. So, I don't think this is exactly generating debug ID's in Zotero, because it's the mere presence of the plugin, not its intentional use, that appears to be causing the problem.

dstillman

A Debug ID is a log of Zotero activity that you generate while performing a specific activity so that we can see what exactly the software is doing. The problem described here would likely be related to Zotero's automatic proxy redirection feature, which would only run when loading a site using a previously detected and user-accepted proxy URL scheme, so the relevant activity would be whatever browsing activity results in an error, timeout, etc. It's always possible to provide a Debug ID, and we can't help further without one.

adamsmith

You'd want that debug from the browser connector or from Zotero?

dstillman

If it's triggered just by browsing, the Connector would be sufficient. If it's triggered by saving, we'd need from both.

rwschwab

Our institution has been seeing this for about 10 days now. It seems to be specifically affecting Chrome browser on MacOS. We've had breach reports mainly from Taylor & Francis, but also Oxford English Dictionary, Science.org, U of Chicago Press, and Web of Science.

I'm beginning to ask patrons and colleagues to try and get a debug report of the issue. In the meantime, wanted to share the specifics of what we've seen so far.

dstillman

@rbrekhus, @rwschwab, in addition to Debug IDs, if you have contacts at the publishers that have reported this, please ask them to send any technical details (e.g., server logs) for what they're seeing to support@zotero.org.

rbrekhus

Latest email from our e-resources librarian:

"We spent last week working with various users and providers, as well as our colleagues at other institutions on this issue.

Apparently, recent web browser updates for Chrome and Firefox caused some problems with third-party plug-ins. We were seeing the issue with Zotero as that was the plug-in our users had installed but other institutions were seeing the same issue with other plug-ins as well.

The reports started tapering off at the end of last week and no new reports were received over the weekend. We will continue monitoring but the suspicious activity reports we received from vendors have been deemed resolved and the result of these browser issues rather than user action.

Please send users to me if they are experiencing any problems with access. While we must work with the vendors to clear the IP blocks, other errors this causes can be handled by the user clearing their cache and/or disabling the plug-in."

rkwlib

Thank you for following up on this. Working with vendors, they have identified recent browser changes affecting third-party plug-ins like Zotero. While our institution was seeing the issue with our Zotero users, it seems others were reporting similar issues with other plug-ins.

dstillman

We still would want actual technical information on this from vendors or users in order to investigate further. "Recent web browser updates" did not cause problems with Zotero, and obviously no one should need to disable the Zotero Connector.

mdinatale

At AAAS this is resulting in a number of libraries having IP addresses banned temporarily for violating out abuse monitoring policies which trigger if someone attempts to access the same article 200 times in 60 seconds. Our platform vendor Atypon is aware that there is an issue stemming from the Zotero Connector. They've had trouble reaching out. Is there a contact I can put them in touch with to discuss further?

dstillman

Yes, they can contact us at support@zotero.org. Please do have them reach out, as we're not able to debug this without additional technical details of what they're seeing on their side.

mdinatale

Thanks - I've shared that email with our technical contact. Hopfully they'll reach out soon.

dstillman

While we wait for any vendors to reach out, a question for the folks who have seen this (@rbrekhus, @rwschwab, @mdinatale, and any others):

At your institution, with the Zotero Connector disabled, if you're on campus and you try to access a gated resource via your institution's proxy (https://www-example-com.proxy.school.edu), are you redirected back to an unproxied domain (https://example.com/), or do you remain on the proxied domain even though the proxy isn't necessary for on-campus access?

One theory we're pursuing is that this is a combination of 1) greatly improved proxy detection in the latest version of the Zotero Connector, causing automatic proxy redirection to be enabled for many more people and 2) a failure to detect a redirect loop if a proxy server redirects on-campus clients back to an unproxied domain. That could result in repeated requests to a vendor site for someone who previously used the proxy off-campus and then returned to campus. (It likely wouldn't result in repeated PDF downloads, but we suspect that's just imprecision in some of the reports and this is actually about webpage requests.)

We're working to add redirect-loop detection to the Zotero Connector (which it should have regardless, and used to many years ago), so if that is the cause, we'll have a fix out soon, but it'd be good to know if this is a plausible source of the current problem.

dstillman

Actually, the current Connector already should be detecting a redirect loop if a proxy redirects to an unproxied domain for on-campus access. We'd still be curious about the answer to my question above about the proxy behavior at the affected institutions, since it's possible there was some regression in that code.

rwschwab

@dstillman, we use EZProxy, and do have it set up to redirect off the proxy while on a campus IP address.

As an example, while off-campus I can access this ebook through our proxy: https://ebookcentral-proquest-com.oca.ucsc.edu/lib/ucsc/detail.action?pq-origsite=primo&docID=4940559

Then, if I sign into VPN and gain an on-campus IP address, visiting that link again ultimately sends me here:
https://ebookcentral.proquest.com/lib/ucsc/detail.action?pq-origsite=primo&docID=4940559

I also have Zotero connecter installed but not configured at all. After arriving on the 2nd page I get this status message:
Zotero detected that you are accessing ebookcentral.proquest.com through a proxy. Would you like to automatically redirect future requests to ebookcentral.proquest.com through %h/%p?

dstillman

OK, we believe we've identified the problem.

As we suspected, the latest version of the Zotero Connector was failing to detect a redirect loop if someone who previously used a proxy later came onto campus (or used a VPN) and the proxy redirected back to an unproxied URL. We had code to deal with that previously, but the latest Connector version changed the proxy redirection process in a way that broke the loop detection.

We've now updated the Connector to properly detect these redirect loops. When a loop is detected, the Connector will suspend automatic proxy redirection for an hour. You can force it to continue redirecting by right-clicking on the save button and choosing Reload via Proxy or by clicking "Reenable proxy redirection" in the Proxies pane of the preferences.

The fix is included in Zotero Connector 5.0.104 for Firefox, available now, and it will be in version 5.0.104 for Chrome and 5.0.105 for Edge, which should be available later today, as soon as they're approved by Google and Microsoft. [The fix is available in Zotero Connector 5.0.106 for Chrome and Edge. See this update.]

Sorry for the trouble this caused, and thanks for helping us troubleshoot it.

And the good news is that, once these fixes are out, automatic proxy detection should work better in general at many more institutions, with many more people benefiting from automatic proxy redirection.

jhharris63

@dstillman Great news! Thank you for diagnosing the issue and making the necessary changes to the Zotero Connector. Will users need to manually update their connectors, or can you push the updates to them?

dstillman

@jhharris63: People will automatically get updated to the latest version within a few hours of it being available for their browser. (We're still waiting for the Chrome and Edge versions to be approved, though.)

dstillman

5.0.105 for Edge is now available with this fix.

dstillman

5.0.104 for Chrome is now available with this fix.

jhharris63

@dstillman: I received no blocked IP reports over the weekend, but I received 4 yesterday and 5 so far today. The audit logs show the same items accessed repeatedly. Could some users have failed to receive updated connectors? You mentioned Firefox, Edge, and Chrome. What about Safari for Apple devices? Thanks!

dstillman

@jhharris63: The vast majority of people will have received automatic updates by now, but ~6% of Zotero Connector installs are still on older versions. Some people might have disabled automatic updates, some computers may be locked down and not allow automatic extension updates, or there might just be some glitch in their browser. People can check which version of the Zotero Connector they have from their browser extensions pane and compare to the versions listed above.

If you send audit logs to support@zotero.org, we can check whether we can determine anything from our end, but most likely these are just older versions that haven't yet been updated.

Automatic proxy redirection isn't possible in Safari, so it's not a feature there at all.

berku101

Zotero Connector has a default setting detecting used proxy addresses, and asks the user to proxy the site in the future. After confirming, the publisher url will be listed as a site to proxy by the Connector.
You all probably have EzProxy configured for on campus users to pass the proxy server (ExcludeIP [IP range] in the configuration file). The loop starts when you visit a proxy url while on campus, where the url is stripped from the prefix, now Zotero starts automatically redirect through the proxy, where the url is stripped, and Zotero redirects again.
Publisher IP blocks might happen when an off campus user has an HTML article open on the laptop browser screen, with the url having the proxy prefix, the user closes the laptop, goes on campus, and opens the laptop again. The browser would refresh, and might repeatedly refresh causing the publisher to block the IP. Almost all of the blocked IP addresses at our university (30+) are Eduroam addresses, the first network the user connects with when arriving on campus.
It is sad this setting is set automatically after installing the connector. We will not change our proxy config for this, and in the worst case, will advice our users to not use Zotero.
You can find and reset the list of url's in the Zotero Connector advanced options menu, click config editor, and reset the proxies:proxies line.
Can anyone confirm my findings?

dstillman

@berku101: Please read the thread you're posting to before commenting. I explained exactly what the issue is above, explained that the Zotero Connector has for years automatically detected these redirect loops but that there was a regression in a recent version that stopped that from happening (hence these reports), and said that we've now released new versions that again detect redirect loops when moving on-campus.

If you're still seeing a problem with the current version of the Connector, we'd want to see a Debug ID from the Zotero Connector for triggering this, but as far as we know this is fixed.

(And you don't need to go into the Config Editor in any case — the current settings are all listed in the Proxies tab of the Connector preferences.)

berku101

Sorry for my ignorance, thank you for fixing this bug.