Upgrade StorageFeature Request: Add Configuration Options for Bulk Find Available PDF
Hi!
I have been using Zotero for a bit now and was using it to help me in doing bulk downloading of the papers I selected for a systematic review. However, when doing so, I got myself IP banned from the ACM digital library. I'm not sure exactly what part of it got me banned but I've reached out to them and hopefully I hear back soon.
I will say though, I had some comments about the process of using "Find Available PDFs".
First, it would be nice if it had a warning saying that if this hits too quickly that this could get you IP banned.
Second, it would be nice if you could set up an override on how long to wait between each request.
Third, when I was looking at the code, it seems like the onBeforeRequest will automatically return if the resolver is doi.org but that does not account for actually hitting the domain where it's stored so with my limited understanding, is it possible that it's hitting the same domain repeatedly even though the goal is to not do that?
Fourth, it would be nice to have a general configuration modal pop up when you do "Find Available PDFs" with multiple entries so you can do things like set the minimum amount of time to wait between each consecutive request, or set up the maximum number of requests to have going at one time. And so on. I don't know what all the relevant possible options are but I thought it seemed like a decent top-level feature to have.
If there's enough guidance, I could possibly try to spin up a PR but would need to balance that between my research and my other project for creating an RSS superfeed of new papers.
I have been using Zotero for a bit now and was using it to help me in doing bulk downloading of the papers I selected for a systematic review. However, when doing so, I got myself IP banned from the ACM digital library. I'm not sure exactly what part of it got me banned but I've reached out to them and hopefully I hear back soon.
I will say though, I had some comments about the process of using "Find Available PDFs".
First, it would be nice if it had a warning saying that if this hits too quickly that this could get you IP banned.
Second, it would be nice if you could set up an override on how long to wait between each request.
Third, when I was looking at the code, it seems like the onBeforeRequest will automatically return if the resolver is doi.org but that does not account for actually hitting the domain where it's stored so with my limited understanding, is it possible that it's hitting the same domain repeatedly even though the goal is to not do that?
Fourth, it would be nice to have a general configuration modal pop up when you do "Find Available PDFs" with multiple entries so you can do things like set the minimum amount of time to wait between each consecutive request, or set up the maximum number of requests to have going at one time. And so on. I don't know what all the relevant possible options are but I thought it seemed like a decent top-level feature to have.
If there's enough guidance, I could possibly try to spin up a PR but would need to balance that between my research and my other project for creating an RSS superfeed of new papers.
I'd be down to spin up a PR but need a little bit of guidance on general things I should look for.
I had the same problem while doing a systematic review. How long did it take to ACM unblock your IP? Did you find any solution to use "Find Available PDFs" and don`t get blocked again?
Thanks