Imported library from Mendeley, "Find Available PDF" fails for over half the library

arejaygraham

Hello,

Just switched over from Mendeley, and I'm trying to get my library sorted out. I successfully moved all the information from Mendeley over into Zotero, but now i'm trying to use "Find Available PDF" on the ~1300 or so papers I've moved over, and it's only working about half the time. Then, when I double click a paper that it has failed to find a pdf for, it sends me to a url where there is clearly access to the pdf. I'm on a university network, and many of the papers are open access anyway. Why is the pdf downloader failing? I don't want to manually add pdfs for hundreds of papers, and if I need to do that I'll probably just stick with Mendeley. What can I do to make this work better?

Thanks in advance!

dstillman

Can you provide a Debug ID for an attempt that fails where you have access to the PDF from the article page?

arejaygraham

D1672279481

dstillman

(3)(+0000001): HTTP GET https://doi.org/10.[…] succeeded with 302

[…]

(3)(+0000001): HTTP GET http://www.nature.com/doifinder/10.[…] succeeded with 301

[…]

(3)(+0000000): HTTP GET http://www.nature.com/articles/doi:10.[…] succeeded with 301

[…]

(3)(+0000001): HTTP GET https://www.nature.com/articles/doi:10.[…] succeeded with 302

[…]

(3)(+0000001): HTTP GET https://idp.nature.com/authorize?response_type=cookie&client_id=grover&redirect_uri=https://www.nature.com/articles/doi:10./[…] succeeded with 302

[…]

(3)(+0000001): HTTP GET https://idp.nature.com/transit?redirect_uri=https://www.nature.com/articles/doi:10./[…]&code=[…] succeeded with 302

[…]

(3)(+0000001): Redirect URL has already been tried -- skipping

Can you try the latest Zotero beta and see if that helps? It looks like Nature uses a complicated scheme for on-campus authentication that involves loading the same URL more than once (after redirecting through intermediate, cookie-setting authentication URLs). Zotero was skipping URLs that it had already tried, including URLs that redirected, which meant that it didn't try the main article page the second time. I disabled that behavior for redirected URLs, which hopefully will fix this.

dstillman

(Out of curiosity, though, how did you do the transfer? If you had the PDFs in Mendeley, you should still have them in Zotero if you followed the recommended import steps. You shouldn't actually need to re-fetch all the files.)

arejaygraham

I'll try the beta.

The pdfs didn't get associated with the data when I imported my Mendeley library for some reason.

I followed these steps since I had a post-1.18 Mendeley version:

"Make sure you've synced all data and — if you want to avoid reassociating files manually — files to Elsevier's servers.
Close Mendeley.
Move your database (the .sqlite file) out of the Mendeley data directory to a temporary location.
Download and install Mendeley 1.18.
Open Mendeley and perform a fresh sync to pull down your Mendeley data from the Elsevier servers.
Start the import in Zotero by going to File → “Import…” and choosing the “Mendeley” option."

arejaygraham

Ok, with the beta version, it now successfully finds Nature pdfs! Thank you :)

However, there's still some goofery afoot:

I tried to find the pdf associated with this paper (https://pubs.geoscienceworld.org/msa/elements/article-abstract/1/5/293/137594/The-Link-between-Large-Igneous-Province-Eruptions?redirectedFrom=fulltext, which I do have access to through my institution), and zotero failed again.

Debug code:

D1215593084

dstillman

Zotero is correctly finding the PDF URL on that page (https://pubs.geoscienceworld.org/elements/article-pdf/1/5/293/3109617/293_v1n5.pdf), but it's receiving an HTML page rather than a PDF when it loads it.

The site may be doing something weird here for authentication. If you click on that URL in your browser, does it load the PDF without changing the URL?

arejaygraham

No, it takes me to the HTML page, then I have to click a pdf button, which takes me to the following url:

https://watermark.silverchair.com/293_v1n5.pdf?token=AQECAHi208BE49Ooan9kkhW_Ercy7Dm3ZL_9Cf3qfKAc485ysgAAArEwggKtBgkqhkiG9w0BBwagggKeMIICmgIBADCCApMGCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQMu5Btm2CaOgigaZoBAgEQgIICZPrMZY9bqL4BGL5CjU1Os2zvKcdjlDveJHYvwR3ahOtYL2SBv5CeGbOpknN8WFrg3d4ttHAYqK2D71D4cjdf-jr_pT2N80JdEXvZ_acAKyJCXt77BH9786TmTBNhp7ysclmAdAlSXArmWB-d_pcn9mjA66PadNcKC1uW7hjn9D6ueUPAXF-fHPH95VXGomRXdvJtHsiBNjNvJP6V-9cIcSPdvvr5A4Juei5BN0o79VZA8-CoTBD6wmEul65dlj9-RdKY6IrblCUjcyXgXqrIhJ3en22a6O52nyMVHlau588MDtuSWjqoiNDS4mO9kVq6Q1f3DwyS4wbHdo2rN8nJNM-tx9v082QrJYOSI9-6vSUgU3roDsJW_6nSg6JjKgxM8rzoyghwq8ghrs74We8rB2mTTRPy5hg18C-csyDlxeYpC64akNDItnIHQ3HDnaG1sK4VTvsFioihmuvkaDsDRFfO0fJrFgZ4JnQcABdqKQv3ycVQH-oLROYCVIakO34PFtGtwwrW_TUp9oBNnat2JBHhiaQFVplPtyc7L5_Qw2zGHPPrK-9sY_wl68OGntp524J24EHjIextbIdrTpwJYB-pp28TRE8RgoW6CgCzQmHSPpiNi6jHFEZNAC6JNLzkaZzi6DPFWpEO5duFjPhKWVR4ON2M9wUwE92N03NK6MAXgNIIzLMjzIi7urL47HwsNJlCnamaFok7vuIIpTyjVAQqBlLToLRW68-T65BJqeeqiErdaryHCz6p28kKpcRoXfg0xjRclXQhp8n_pqGYkh1Ktx8nz0NxoPzkQ1aEtLQPbZCq5w

dstillman

Ah, well that would explain it. That's not something that would work. Zotero gets the PDF URL from the metadata embedded in the document (citation_pdf_url), and it expects that to be an actual PDF URL. If it's just some random HTML page dependent on the third-party authentication system being used by the site, that's not something Zotero would have any way of knowing how to deal with.

If you View Source for the HTML page and email it to support@zotero.org with a link to this thread, we can look at it to see if the PDF is provided in some straightforward way, but if you have to click a PDF button that doesn't sound promising.

arejaygraham

Ok, thank you.

Have you considered having zotero check sci-hub for pdfs that it fails to acquire due to goofy site design like this?