Upgrade StorageFind Available PDFs power use case; obstacles and ways forward?
I'm planning to use "Find Available PDFs" for thousands of entries. I know this is not the typical use case. This is for a large computer-aided lit review I'm doing in which Zotero does some of the heavy lifting of managing bib entries and retrieving and storing the full text. I plan to connect to my university's VPN to ensure I can access the most paywalled PDFs possible. Many of these items will not have PDFs available and will be culled afterward. (Also, I'm not using Sci-hub for this project.)
I've tried to dig through the forums a bit to understand what difficulties may lie ahead. I know that a couple years ago Zotero started using Unpaywall, but I'm not sure what those implications are, if any. I did read in the Zotero blog that 'When you use "Add Item by Identifier" or "Find Available PDF", Zotero will load the page associated with the item’s DOI or URL and try to find a PDF to download before looking for OA copies. This will work if you have direct or VPN-based access to the PDF.' I'm not sure what my university VPN will do with that, but there's one way to find out.
I've found through trial and error that using the Find Available PDF option on only 300 entries seems to trigger some sort of temporary lock-out for whatever service Zotero is calling. Does Unpaywall block requests of this size? The progress window seems to hang indefinitely after doing only a few items in the 300 queue. Doing 5 at a time seems to work fine. This makes me wonder if there's some non-obvious issue with the Find Available PDF option and how it queues items when making that many requests. Doing it 5 at a time for thousands of papers is a lot of unproductive manual work. I'm fine with limiting my rate, if there's a way to do that. I can take a custom code approach, but I'd like some guidance on how and where to do that.
What difficulties or possible issues should I be aware of?
What approach should I take for retrieving this many PDFs?
I've tried to dig through the forums a bit to understand what difficulties may lie ahead. I know that a couple years ago Zotero started using Unpaywall, but I'm not sure what those implications are, if any. I did read in the Zotero blog that 'When you use "Add Item by Identifier" or "Find Available PDF", Zotero will load the page associated with the item’s DOI or URL and try to find a PDF to download before looking for OA copies. This will work if you have direct or VPN-based access to the PDF.' I'm not sure what my university VPN will do with that, but there's one way to find out.
I've found through trial and error that using the Find Available PDF option on only 300 entries seems to trigger some sort of temporary lock-out for whatever service Zotero is calling. Does Unpaywall block requests of this size? The progress window seems to hang indefinitely after doing only a few items in the 300 queue. Doing 5 at a time seems to work fine. This makes me wonder if there's some non-obvious issue with the Find Available PDF option and how it queues items when making that many requests. Doing it 5 at a time for thousands of papers is a lot of unproductive manual work. I'm fine with limiting my rate, if there's a way to do that. I can take a custom code approach, but I'd like some guidance on how and where to do that.
What difficulties or possible issues should I be aware of?
What approach should I take for retrieving this many PDFs?
What I would expect is that several of the commercial resources you'll be querying this way will start locking you out of PDF downloads if you do this for large amounts of items at once (because you look like a bot downloading their PDFs) -- that shouldn't affect Unpaywall, though.
But as to the hanging. It lasts 25+ minutes for a single entry. The trouble seems to be items with proquest URLs. (I wish I could sort by URL but I can't list it as a column.) But the trouble doesn't quite stop there.
Even stranger, if I cancel one and start the entry below it, both being single-selections, sometimes the title of the previous item will appear in the processing window, as if I had selected them together (which I did not). For example, say I start a proquest URL Find PDF but then cancel and do one below it instead. It won't progress either. If I close Zotero, and reopen, "Find ... PDF" again, that second (not-proquest) item will progress with Find PDF.
I should mention that I have a number of add-ons, although I don't know if any of them make any difference. E.g., Zotfile, Zutillo, DOI Manager. I think all are fairly standard add-ons.
And now, right as I preview this post, the items with proquest URLs are back to working for Find PDF!
It sounds like the way forward is just trial and error. Somewhere between a rapid and slow rate--running all items as one big batch and running a few items every few minutes--is a sweet spot where I am unlikely to be locked out, but who knows where it is. And it probably varies by publisher. I will mention that no one publisher seems favored in my search results that have been imported into Zotero, so maybe that randomness will help.
I think Zotero has huge potential to be used in these kinds of reviews, and these types of reviews have huge potential for science itself, but I realize it's not standard use. I do very much appreciate that Zotero has the functionality to help me do this, because it would be a lot more work otherwise.
Related to the temporary hanging issue, I just saw there was another thread yesterday mentioning proquest having issues. Good to know it wasn't just me. Re: dstillman's request, there's no way I could submit a bug report or debug information if the issue requires replicating proquest having whatever issue it had. Hopefully if a site has that issue in the future and it affects someone trying Find Available PDFs they'll find this post in a search and realize the issue. The solution is to cancel the Find... PDFs and restart Zotero, then select items with URLs that do not point to a site/publisher having an issue.
It's indeed that broken ProQuest import freezes the find PDF function, but that would likely be a bug, too: translators will fail occasionally, and the tool should handle that "gracefully," i.e. by trying Unpaywall and then moving on to the next item.
(And another clarification: Unpaywall servers aren’t involved here. We don’t send any data to them. So there’s no “testing their good graces”.)
I went and reread the blog post about Unpaywall: "if you save an item from a webpage where Zotero can’t find or access a PDF, Zotero will automatically search for an open-access PDF using data from Unpaywall." Very subtle: you're using their data, but that doesn't necessarily mean you're hitting their server. I didn't catch that the first time, but maybe you can see why I assumed you were. Great, the situation is even better than I hoped! Thanks for the clarification.
It was apparently trying to access this URL: https://stud.epsilon.slu.se/16926/
When I go to that URL, I receive the page:
EPrints System Error
Error connecting to MySQL server: Too many connections. To fix this increase max_connections in my.cnf:
[mysqld]
max_connections=300
Let me know if I can do anything further to help.