Zotero is not saving PDFs from ScienceDirect

dstillman Zotero Team
This discussion was created from comments split from: Zotero does not save anymore PDF when it should.
«13
  • Lori Tschirhart
    another Zotero user just reported to me and I replicated:Zotero is not saving PDFs from certain Elsevier titles on the ScienceDirect platform such as this one: https://doi.org/10.1016/j.precamres.2016.03.004

    If I try to save to My Library, which allows for PDF saving, I get the usual :Saving to [My Library] message, and below that, I see the "ScienceDirect Snapshot" icon, and below that, I see a red "X" and the words, "ScienceDirect Full Text PDF" in red.
  • dstillman Zotero Team
    edited December 26, 2023
    ScienceDirect employs some pretty zealous anti-bot protections that can break PDF saving in a way that doesn't happen on most other sites. We're going to try to fix this, but it'll take some work. For now, your best bet is to download the PDF to disk and drag it onto the new item in Zotero.
  • jiaojiaodubai23
    I think CNKI also has the same problem. Users often report that they have download permission but cannot download PDF through the translator. Through remote assistance, I found that the attachment link obtained by the translator is the same as the actual download link, but the translator really cannot download PDF, and it is always effective for users to click to download manually.
  • dstillman Zotero Team
    edited September 3, 2024
    In the Zotero 7 beta Zotero 7, we've rolled out a change that should show the ScienceDirect CAPTCHA page if you get one while trying to save an article. If you complete the CAPTCHA, Zotero should continue downloading the current file and be better able to download additional PDFs from ScienceDirect in that Zotero session.

    The CAPTCHA window appears in the Zotero app, and there's currently no indication that it was opened when saving from the browser, so if the PDF download seems to be stuck, switch to the Zotero app to see if it's waiting for input. We'll work on better feedback there.

    I would recommend not trying to download many files at once via ScienceDirect, as that's very likely to still get you blocked even with the CAPTCHA.

    Unfortunately we're not able to fix this in Zotero 6. If you don't want to run beta software, your best bet for now is to download the PDF separately and drag the PDF onto the parent item you saved.

    Update: Zotero 7 is now available.
  • anand4k
    Thank You for listening to even smaller concerns :)
  • SBolz
    I'm using Zotero 6.0.30 right now and experience the same problem. Thanks for clarification on that issue so far!

    I assume I'm "blocked" as I tried to download many papers via Zotero. Do you have an idea how to get "unblocked"?
    Also, as there is this likelihood of being blocked, it could be useful for further Zotero version to be able to set a time limit for using the automatic PDF download (ignore this comment if it's already planned).
  • Tannenfreund 87
    @SBolz
    > I assume I'm "blocked" as I tried to download many papers via Zotero. Do you have an idea how to get "unblocked"?

    I had this problem with Elsevier and Taylor and Francis. To get unblocked, I had to call the library, as they blocked the whole IP-range of my institute. Elsevier got unblocked after a few hours, to unblock Taylor and Francis, the library had to file a ticket with them.

    Bureaucracy maxing. For my review, I now disabled all automatic downloading in Zotero except for sci-hub. The few papers that are not on sci-hub, I download by hand. Again, piracy is more user friendly than the legal path, sigh.
  • SBolz
    Thanks for the hint! I think I'll just deal with manually adding the papers now (seems less time consuming than delving into such a process).
  • grandoldmen
    I am also experiencing the same problem after downloading the latest version and have read the comments in this forum. however, I am still confused regarding the problem. so the final conclusion is it because this latest version of zotero has a bug? or what is the final conclusion? I hope someone can answer my confusion. thank you very much.
  • dstillman Zotero Team
    @grandoldmen: It's explained above:

    https://forums.zotero.org/discussion/comment/450008/#Comment_450008
    https://forums.zotero.org/discussion/comment/451390/#Comment_451390
  • leighalice
    I have gotten an error on every PDF I have tried to download starting today (I was successfully pulling PDFs in last week). I restarted my computer and now I am able to download the citation metadata, but it will still not pull the PDF across automatically (I get a message saying there's an 'error downloading metadata. if this persists get help'). Occasionally a CAPTCHA pops up- but only when I am trying to separately download the PDF- it's not popping up during the Zotero autoload. This is a sudden change from last week, definitely applied to Elsevier.

    Any help would be appreciated.
  • dstillman Zotero Team
    edited February 27, 2024
    @leighalice: As noted above, you need to be running the Zotero 7 beta, which will show a CAPTCHA in Zotero itself, for this to work. There's nothing new here — this has been an issue for months. If you weren't being blocked by Elsevier earlier, you were just lucky.
  • leighalice
    Ok, thank you. It was just a little shocking how sudden it was. I will see if I can download the beta 7 version. In the meantime- I saw that the initial announcement of the beta rollout was last May- is there an estimate on when the full rollout/integration will be? (apologies if it was in the thread and I missed it).
  • fmuro
    I find this problem in iOS too.
  • gothamgreen
    My workaround for this issue that's easier (for me) than manual download:
    - Click on the pdf link on the publisher's page, as if to view it.
    - The publisher presents a robot check. Do it.
    - Click back and reload the page, and now Zotero can download with the pdf.

    You might have some issues with re-downloading (not captcha-related, historical): sometimes when an item downloaded without a snapshot or pdf, and you want to re-do the download, either zotero or the plugin (not sure) notices the existing item and refuses to replace it. It can take trashing the item and then reloading Zotero and the webpage to make it work. But the ScienceDirect aggressive captcha problem is solvable without manual downloads.

    I did see the new Zotero version captcha popup, and it has worked for me at least once!
  • yaoliyun
    I tried downloading the pdf to the disk then drag it onto the zotero, but I couldn't even download the pdf to the disk. After clicking the download button, nothing displayed.
  • dstillman Zotero Team
    @yaoliyun: We obviously wouldn't have anything to do with downloading PDFs from ScienceDirect outside of Zotero.
  • bmveremis
    It looks like Zotero 7 beta is capturing many articles I was missing using Find Available PDFs in Zotero 6 (which appear to be from the ScienceDirect Captcha)--thank you for all your hard work!
  • jiaojiaodubai23
    @dstillman: For websites other than Science Direct, using Connector to save PDFs may also fail due to CAPTCHA. If possible, I really hope Zotero 7's special handling for Science Direct can be applied to other websites. In that case, translator developers may declare URL pattern or key element's selector of the CAPTCHA page in the translator (just like detectWeb did).
  • elsinore00
    Any solutions or suggestions for this problem? Using Zotero on mac OS 14.5 and Chrome.
  • poettli
    It's all explained above: https://forums.zotero.org/discussion/comment/451390/#Comment_451390
  • elsinore00
    @poettli, Thank you.
    I downloaded the beta version and completed the human test, and now it works.
    It is rather slow but it works!
  • jamesnicholas
    I'm having an issue where the captcha page appears within Zotero, but then it asks me to search for my institution, and sign in with my credentials. Once I have done this, the webpage for the article appears within this new Zotero window, but after some time, the windows closes and the PDF has failed to download. Is this a bug, or just due to how my institution and Elsevier handle authentication specifically?

    Connector debug: D715631940 and Zotero debug: D85041265 from trying to save a Science Direct PDF in case.
  • mjthoraval
    @jamesnicholas: See the reply here:
    https://forums.zotero.org/discussion/comment/463841/#Comment_463841
  • jamesnicholas
    @mjthoraval ah, I hadn't come across that discussion, thanks
  • xiaoqq123
    I updated my version to the latest, but I still can't store the PDF in Zotero. It always shows "An error occurred while saving this item. Try again, and if the issue persists see Getting Help for more information." How do I address this issue?
  • mmoole
    to give this problem another take, since Zotero also doesn't save PDF files from some other hosts, would it be possible to use Zotero as a 'destination' via the 'open in browser' extension (https://github.com/Rob--W/open-in-browser)?
    I am using that extension since long times, and it's very convenient to open for example PDFs directly in browser even if they arrive as a file to save (set server sides).
  • nrbond
    I'm using the latest beta version (7.0.1-beta.1+d879c507a) and am connected to my University through a VPN. I am finding that Zotero is unable to download papers from Elsevier sciencedirect even though when I select view online from the menu the ViewPDF button allows me to immediately download the PDF to my hard drive. Thanks for a great set of updates overall.... would be fantastic if this issue could be resolved somehow.
  • dstillman Zotero Team
    edited August 14, 2024
    @nrbond: Generally in Zotero 7 you should get a CAPTCHA within Zotero, but we're making changes to the way Zotero downloads files to try to address this more reliably and for all sites that use anti-bot protections like this. We should have an update on that soon.
  • nrbond
    thanks @dstillman. Yes, the CAPTCHA pops up - albeit a bit random in its timing - but even then PDFs are not downloading, nor are they being found when searching for full text, but they are available for manual download via the Elsevier website. Thanks for continuing to work on this!!
Sign In or Register to comment.