Find Available PDFs doesn't find some available PDFs

dstillman Zotero Team
This discussion was created from comments split from: Available for beta testing: improved PDF retrieval with Unpaywall integration.
  • Gurdas_Sandhu
    edited November 16, 2018
    I used the "Find Available PDFs" on about 700 items and Zotero downloaded about 200 PDFs. The remaining PDFs are from journals not subscribed by my work organization. I can get to these PDFs if I login to my university account and have .prox part in the URL.

    Can I setup the find PDF feature to use the .prox version of the URL?

    Example before and after URL:
    https://www.sciencedirect.com/science/article/pii/S2352340918303950
    https://www-sciencedirect-com.prox.lib.ncsu.edu/science/article/pii/S2352340918303950

    If something automatic is not possible for the 500 odd remaining items, how can I do this manually for each item? Currently, the DOI and URL for these items takes me to the non-prox website. I have "enable proxy redirection" checked in the browser connector preferences.
  • bwiernik
    Can you connect to your universiry’s VPN? If so, do that and the Find Available PDFs option should retrieve them.
  • dstillman Zotero Team
    Yeah, we mentioned this in the announcement:
    This will work if you have direct or VPN-based access to the PDF. If you use a web-based proxy, only open-access PDFs will be automatically retrieved using this new functionality, but you can continue to save items with gated PDFs from the browser using the Zotero Connector.
    We're hoping to support web-based proxies for Find Available PDF in a future version.
  • adamsmith
    NCSU does seem to offer VPN for all affiliates: https://oit.ncsu.edu/campus-it/campus-data-network/vpn/
  • Gurdas_Sandhu
    Thanks, all. I'll try the VPN and report back if I have issues.

    @danstillman Just FYI, overall this feature worked smooth and fast, so thanks a bunch for getting this implemented.

  • Gurdas_Sandhu
    I was able to install and connect using Cisco AnyConnect VPN client software. However, I still cannot get Zotero to download the PDF/s. For an example item, where the find PDF does not work, if I click on the DOI or URL field in Zotero, it opens the web page and auto-redirects using the proxy, and the web proxied page allows me to download the PDF.

    Do I need to manually change the proxy settings in Firefox (General > Network Proxy) or IP addresses in my internet connection? Can't find anything along these lines on the NCSU website.
  • bwiernik
    Do you have access to the PDF if you disable proxy redirection, then click the DOI link?
  • Gurdas_Sandhu
    edited November 17, 2018
    Part 1
    With proxy redirection disabled but Cisco VPN connected, I do not have access to the PDF.

    The automatic installation did not work for me, so I went the manual route.
    https://oit.ncsu.edu/campus-it/campus-data-network/vpn/vpn-installation-instructions-for-windows/

    Part 2
    I also have a ScienceDirect login through my work that lets me access many but not all papers. At home, when I go through this login, about half of the pending ScienceDirect papers are available for download. Note that with this login, the URL does not get any "prox" suffix and stays exactly how it appears in the item metadata in Zotero. But Zotero cannot pull these PDFs.

    Part 1 is clearly not a Zotero problem. And likely part 2 is also not a Zotero issue, but for a moment I thought that would work and then was disappointed it did not.
  • dstillman Zotero Team
    When Zotero downloads files directly, that's totally separate from your browser, so anything that involves a web-based login won't help with Find Available PDF right now.

    A direct or VPN-based connection that gives you system-wide, non-login-based access to PDFs will work.
  • Gurdas_Sandhu
    edited November 17, 2018
    Thanks, Dan. For now, I am working my way manually through these pending PDFs. No clue why the VPN login is not making the PDFs available to me - I'll look in to that (again) later.

    *Update* I think this is why the VPN is not working:
    https://www.lib.ncsu.edu/faq/when-i-use-the-vpn-why-do-i-have-to-authenticate-to-get-access-to-library-resources-like-e
  • adamsmith
    Yes, that would be it
  • Gurdas_Sandhu
    edited November 18, 2018
    I setup a VPN to another university and re-ran the "Find Available PDFs". No split tunneling here, so it worked but I found another issue. There are two "types" of journals/articles for which the PDF is available via VPN (without any web based login).

    Type 1: the link to download the PDF on the article webpage actually downloads the PDF file. No issues here. Mostly at ACS, SAGE, etc.

    Type 2: the "Download PDF" link opens the PDF in a separate browser window. Zotero was unable to auto-download the PDFs for these. Mostly at ScienceDirect:
    https://www.sciencedirect.com/science/article/pii/S0021850218301642

    *Update* I found articles that are Type 1, but Zotero cannot get PDF. So maybe the issue is something different than my initial guess.
    https://www.sciencedirect.com/science/article/pii/S1361920916308197
    https://ieeexplore.ieee.org/document/6896636

    Any help or solutions?
  • Gurdas_Sandhu
    @dstillman What might be causing that last issue mentioned above?
  • adamsmith
    I think the way Elsevier handles DOI resolution (which goes through an intermediary page) might be breaking Zotero's ability to automatically get the PDF from them. Not sure about IEEE.
  • Gurdas_Sandhu
    Anything I can do at my end to improve/fix this?
  • dstillman Zotero Team
    When connected via the VPN and no web-based proxy, does Zotero save the PDF for this one using the Save to Zotero button in your browser? If so, but it doesn't work for Find Available PDF, could I see a Debug ID from Zotero for a Find Available PDF attempt? If it doesn't work via Save to Zotero either, I'd want to see a Debug ID from the Zotero Connector for a save attempt. There is an intermediate page here for the PDF download, but the translator should be able to extract the URL from it, and that should work the same in the Zotero Connector and in Zotero.
    For this one, the page itself is being generated via JavaScript, and Find Available PDF doesn't currently work for such pages. That one is a bit tricky, but I've created an issue to track it.
  • Gurdas_Sandhu
    edited November 26, 2018
    With only VPN and no web-based proxy or login, Zotero was able to download PDF for each of these three when creating the item using the Firefox browser connector:
    1. https://www.sciencedirect.com/science/article/pii/S0021850218301642
    2. https://www.sciencedirect.com/science/article/pii/S1361920916308197
    3. https://ieeexplore.ieee.org/document/6896636

    The debug output for #2 is: D2016402217
  • dstillman Zotero Team
    edited November 26, 2018
    OK, @adamsmith was right — the intermediate page Elsevier uses for DOI resolution was breaking this. But there's then another problem with translating the ScienceDirect page in Zotero (though not the Connector) that we'll need to resolve. We have a fix for that, but we're still evaluating it to make sure it's OK to roll out.
  • Gurdas_Sandhu
    OK, thanks. I have a few hundred items for which I can download PDFs but not through the "Find Available PDF" function. For each of these:
    1. I could create a new item via the connector and merge with the old one. Or,
    2. Download the PDF and then attach it using Zotfile

    Both methods will have to be one item at a time end to end. I believe it's not possible that I first download all hundreds of PDFs (manually) in to an empty folder and then in just a couple clicks have Zotero (or Zotero+Zotfile) attach them to the correct items in a dedicated collection or selected by me? Or, any other suggestion to minimize the number of clicks/time?
  • lmallenjacobson
    @dstillman I am having a similar problem. I would like to build a library based on the references in a publication. "Find available pdf" successfully found 50 of the ~136 references. I have manually checked access to 15 of the references that were not retrived using "find available pdf". I had access to 9/15: five were on Elsevier. The remaining four were on U Chicago Press, Jstor, NRC research press, and Unicamp. I am using a VPN to access articles through my institution. Are there any settings I can change to improve retrieval?
  • dstillman Zotero Team
    @lmallenjacobson: Can you provide Debug IDs for attempts that don't work when connected via the VPN?
  • lmallenjacobson
    edited March 4, 2019
    @dstillman D1607903447 ...Does anyone else have this problem? Or has anyone had a chance to investigate? Thanks for your help!
  • javierfajnolla
    Same here. Using https://www.sciencedirect.com/science/article/pii/S0006320720307497?via=ihub as an example,
    - not using the VPN, if I click on Zotero's browser importer, I will get the paper on my library but no pdf.
    - If I now connect my university's VPN and click on "Find available PDF" for the entry I just created, no PDF is found.
    - However, keeping VPN connected and clicking Zotero's importer will add a new (duplicated) entry that includes the PDF.

    Therefore, the VPN gives access to the paper, and Zotero is able to directly import it from the browser. However, using the "Find available PDF" does not work under the same conditions.

    Even if it's not yet smooth always, thanks for working on all these cool features that make life easier!
    Javier
  • dstillman Zotero Team
    @javierfajnolla: Can you provide a Debug ID from Zotero for a "Find Available PDF" that doesn't work while connected to your VPN?
  • javierfajnolla
    @dstillman Yes, thanks for looking at it!
    Here's the ID: D344850053
    Notice that I did two things, both with the VPN active. First, unsuccessfully retrieve PDF via the button "Find Available PDF", second, successfully getting the PDF from the browser using "save to zotero" - this includes the PDF
  • fraph24
    Here is my ID: D1287547738.
    I am connecting from my institution local network.
  • fraph24
    Previous ID was with Zotero 6.0.26 on Windows.
    I was able to reproduce the issue with Zotero 7.0.0-beta.32+5e27a7501 (64-bit) on the same machine; ID: D1662378605.

    I see a CloudFlare "are you a robot?" page when opening the link using Firefox on a private window. I wonder if that could be an issue.
  • fraph24
    edited August 10, 2023
    I am able to get the PDF directly by using Microsoft Edge (no login, but a CloudFlare captcha appears: "are you a robot?"). I am only able to get the PDF by logging in to Elsevier when I use Firefox (like Zotero does, right?).
Sign In or Register to comment.