I used the "Find Available PDFs" on about 700 items and Zotero downloaded about 200 PDFs. The remaining PDFs are from journals not subscribed by my work organization. I can get to these PDFs if I login to my university account and have .prox part in the URL.
Can I setup the find PDF feature to use the .prox version of the URL?
If something automatic is not possible for the 500 odd remaining items, how can I do this manually for each item? Currently, the DOI and URL for these items takes me to the non-prox website. I have "enable proxy redirection" checked in the browser connector preferences.
This will work if you have direct or VPN-based access to the PDF. If you use a web-based proxy, only open-access PDFs will be automatically retrieved using this new functionality, but you can continue to save items with gated PDFs from the browser using the Zotero Connector.
We're hoping to support web-based proxies for Find Available PDF in a future version.
I was able to install and connect using Cisco AnyConnect VPN client software. However, I still cannot get Zotero to download the PDF/s. For an example item, where the find PDF does not work, if I click on the DOI or URL field in Zotero, it opens the web page and auto-redirects using the proxy, and the web proxied page allows me to download the PDF.
Do I need to manually change the proxy settings in Firefox (General > Network Proxy) or IP addresses in my internet connection? Can't find anything along these lines on the NCSU website.
Part 2 I also have a ScienceDirect login through my work that lets me access many but not all papers. At home, when I go through this login, about half of the pending ScienceDirect papers are available for download. Note that with this login, the URL does not get any "prox" suffix and stays exactly how it appears in the item metadata in Zotero. But Zotero cannot pull these PDFs.
Part 1 is clearly not a Zotero problem. And likely part 2 is also not a Zotero issue, but for a moment I thought that would work and then was disappointed it did not.
When Zotero downloads files directly, that's totally separate from your browser, so anything that involves a web-based login won't help with Find Available PDF right now.
A direct or VPN-based connection that gives you system-wide, non-login-based access to PDFs will work.
Thanks, Dan. For now, I am working my way manually through these pending PDFs. No clue why the VPN login is not making the PDFs available to me - I'll look in to that (again) later.
I setup a VPN to another university and re-ran the "Find Available PDFs". No split tunneling here, so it worked but I found another issue. There are two "types" of journals/articles for which the PDF is available via VPN (without any web based login).
Type 1: the link to download the PDF on the article webpage actually downloads the PDF file. No issues here. Mostly at ACS, SAGE, etc.
I think the way Elsevier handles DOI resolution (which goes through an intermediary page) might be breaking Zotero's ability to automatically get the PDF from them. Not sure about IEEE.
When connected via the VPN and no web-based proxy, does Zotero save the PDF for this one using the Save to Zotero button in your browser? If so, but it doesn't work for Find Available PDF, could I see a Debug ID from Zotero for a Find Available PDF attempt? If it doesn't work via Save to Zotero either, I'd want to see a Debug ID from the Zotero Connector for a save attempt. There is an intermediate page here for the PDF download, but the translator should be able to extract the URL from it, and that should work the same in the Zotero Connector and in Zotero.
For this one, the page itself is being generated via JavaScript, and Find Available PDF doesn't currently work for such pages. That one is a bit tricky, but I've created an issue to track it.
OK, @adamsmith was right — the intermediate page Elsevier uses for DOI resolution was breaking this. But there's then another problem with translating the ScienceDirect page in Zotero (though not the Connector) that we'll need to resolve. We have a fix for that, but we're still evaluating it to make sure it's OK to roll out.
OK, thanks. I have a few hundred items for which I can download PDFs but not through the "Find Available PDF" function. For each of these: 1. I could create a new item via the connector and merge with the old one. Or, 2. Download the PDF and then attach it using Zotfile
Both methods will have to be one item at a time end to end. I believe it's not possible that I first download all hundreds of PDFs (manually) in to an empty folder and then in just a couple clicks have Zotero (or Zotero+Zotfile) attach them to the correct items in a dedicated collection or selected by me? Or, any other suggestion to minimize the number of clicks/time?
@dstillman I am having a similar problem. I would like to build a library based on the references in a publication. "Find available pdf" successfully found 50 of the ~136 references. I have manually checked access to 15 of the references that were not retrived using "find available pdf". I had access to 9/15: five were on Elsevier. The remaining four were on U Chicago Press, Jstor, NRC research press, and Unicamp. I am using a VPN to access articles through my institution. Are there any settings I can change to improve retrieval?
Same here. Using https://www.sciencedirect.com/science/article/pii/S0006320720307497?via=ihub as an example, - not using the VPN, if I click on Zotero's browser importer, I will get the paper on my library but no pdf. - If I now connect my university's VPN and click on "Find available PDF" for the entry I just created, no PDF is found. - However, keeping VPN connected and clicking Zotero's importer will add a new (duplicated) entry that includes the PDF.
Therefore, the VPN gives access to the paper, and Zotero is able to directly import it from the browser. However, using the "Find available PDF" does not work under the same conditions.
Even if it's not yet smooth always, thanks for working on all these cool features that make life easier! Javier
@dstillman Yes, thanks for looking at it! Here's the ID: D344850053 Notice that I did two things, both with the VPN active. First, unsuccessfully retrieve PDF via the button "Find Available PDF", second, successfully getting the PDF from the browser using "save to zotero" - this includes the PDF
Previous ID was with Zotero 6.0.26 on Windows. I was able to reproduce the issue with Zotero 7.0.0-beta.32+5e27a7501 (64-bit) on the same machine; ID: D1662378605.
I see a CloudFlare "are you a robot?" page when opening the link using Firefox on a private window. I wonder if that could be an issue.
I am able to get the PDF directly by using Microsoft Edge (no login, but a CloudFlare captcha appears: "are you a robot?"). I am only able to get the PDF by logging in to Elsevier when I use Firefox (like Zotero does, right?).
Can I setup the find PDF feature to use the .prox version of the URL?
Example before and after URL:
https://www.sciencedirect.com/science/article/pii/S2352340918303950
https://www-sciencedirect-com.prox.lib.ncsu.edu/science/article/pii/S2352340918303950
If something automatic is not possible for the 500 odd remaining items, how can I do this manually for each item? Currently, the DOI and URL for these items takes me to the non-prox website. I have "enable proxy redirection" checked in the browser connector preferences.
@danstillman Just FYI, overall this feature worked smooth and fast, so thanks a bunch for getting this implemented.
Do I need to manually change the proxy settings in Firefox (General > Network Proxy) or IP addresses in my internet connection? Can't find anything along these lines on the NCSU website.
With proxy redirection disabled but Cisco VPN connected, I do not have access to the PDF.
The automatic installation did not work for me, so I went the manual route.
https://oit.ncsu.edu/campus-it/campus-data-network/vpn/vpn-installation-instructions-for-windows/
Part 2
I also have a ScienceDirect login through my work that lets me access many but not all papers. At home, when I go through this login, about half of the pending ScienceDirect papers are available for download. Note that with this login, the URL does not get any "prox" suffix and stays exactly how it appears in the item metadata in Zotero. But Zotero cannot pull these PDFs.
Part 1 is clearly not a Zotero problem. And likely part 2 is also not a Zotero issue, but for a moment I thought that would work and then was disappointed it did not.
A direct or VPN-based connection that gives you system-wide, non-login-based access to PDFs will work.
*Update* I think this is why the VPN is not working:
https://www.lib.ncsu.edu/faq/when-i-use-the-vpn-why-do-i-have-to-authenticate-to-get-access-to-library-resources-like-e
Type 1: the link to download the PDF on the article webpage actually downloads the PDF file. No issues here. Mostly at ACS, SAGE, etc.
Type 2: the "Download PDF" link opens the PDF in a separate browser window. Zotero was unable to auto-download the PDFs for these. Mostly at ScienceDirect:
https://www.sciencedirect.com/science/article/pii/S0021850218301642
*Update* I found articles that are Type 1, but Zotero cannot get PDF. So maybe the issue is something different than my initial guess.
https://www.sciencedirect.com/science/article/pii/S1361920916308197
https://ieeexplore.ieee.org/document/6896636
Any help or solutions?
1. https://www.sciencedirect.com/science/article/pii/S0021850218301642
2. https://www.sciencedirect.com/science/article/pii/S1361920916308197
3. https://ieeexplore.ieee.org/document/6896636
The debug output for #2 is: D2016402217
1. I could create a new item via the connector and merge with the old one. Or,
2. Download the PDF and then attach it using Zotfile
Both methods will have to be one item at a time end to end. I believe it's not possible that I first download all hundreds of PDFs (manually) in to an empty folder and then in just a couple clicks have Zotero (or Zotero+Zotfile) attach them to the correct items in a dedicated collection or selected by me? Or, any other suggestion to minimize the number of clicks/time?
- not using the VPN, if I click on Zotero's browser importer, I will get the paper on my library but no pdf.
- If I now connect my university's VPN and click on "Find available PDF" for the entry I just created, no PDF is found.
- However, keeping VPN connected and clicking Zotero's importer will add a new (duplicated) entry that includes the PDF.
Therefore, the VPN gives access to the paper, and Zotero is able to directly import it from the browser. However, using the "Find available PDF" does not work under the same conditions.
Even if it's not yet smooth always, thanks for working on all these cool features that make life easier!
Javier
Here's the ID: D344850053
Notice that I did two things, both with the VPN active. First, unsuccessfully retrieve PDF via the button "Find Available PDF", second, successfully getting the PDF from the browser using "save to zotero" - this includes the PDF
I am connecting from my institution local network.
I was able to reproduce the issue with Zotero 7.0.0-beta.32+5e27a7501 (64-bit) on the same machine; ID: D1662378605.
I see a CloudFlare "are you a robot?" page when opening the link using Firefox on a private window. I wonder if that could be an issue.