Upgrade StorageSupercharged my Zotero
Hey all,
I just finished a big project to totally automate my Zotero experience using the API and a few scripts... and I feel like there's literally no one else in the world who would understand the simple joy of my accomplishment and my new level of appreciation for Zotero and the API, except you people here, so please excuse my need to share!
Problem: I had a back catalog of thousands of PDFs I have accumulated over the years. I already had a consistent naming scheme for the files, but I needed to get these all into Zotero, and I wanted to automate the process of importing the files. Additionally, I wanted to bypass Zotero's automatic imports from just dragging a bunch of files into Zotero, because I need to control the attachments and control some other aspects of data quality.
I wrote two scripts, one is a shell script running every night as a cron job, which scans my Library folder, looks for new files, generates a list of DOIs from any new PDFs, and then uses the doi2bib API service to download a concatenated bibtex file, which is then automatically imported into Zotero. (Automating this was difficult, because of the way Zotero handles imports - requiring the user to confirm the imports in a little dialog. I use MacOS and wrote a tiny little AppleScript application that runs as part of the shell script, after the file is imported, to automate the selecting of the "Unfiled Items" folder, and then the confirmation of the import in the little "confirm import" dialog.
The second script is a Node.js script that automatically adds 2 attachments to every item in my Zotero library, a relative link to the PDF file itself, and a link to the file on Dropbox (my library is on Dropbox, and this lets me view my files remotely), that I configured to run as a global command line program, with a pretty nice suite of arguments/options. It runs regularly as a cron job, where it creates these two attachments for any new items added from the previous script, but I can also run it with customized options from the command line if I need to do anything manually for any reason. It runs in 3 modes: make attachments by key(s), where it is fed a list of Zotero item keys, and then there are two variations of make attachments by chunk of files, ordered by most recent, where you either 1. specify the starting item, number of items, how many times to repeat the chunk (eg 30 items at a time, starting at item 0, the most recent, 3 times, would make 3 requests of 30 each, for items 0-30, 30-60, and 60-90, respectively, or 2.) same as #1, but where the program repeats the specified chunk as many times as are needed to completely process the whole library.
I feel like I would like to share these various utilities (I also have made a ton of little JS snippets that can be used in the "Run Javascript" too within Zotero for all kind of different situations), but I am not sure how much interest there would be, and also the system is fairly well tailored to my own specific use-case. I looked at work that was already out there such as pyzotero, but found them to have simultaneously too many features and not enough features that I needed for my specific use. On the other hand, these tools, while probably difficult to understand to non-programmers or people who are looking for an out of the box solution, might prove to be a useful starting point for someone else who is looking to jump start their own custom solution. If anyone is interested I would be happy to share any of the code. I did spend a lot of time fool-proofing the argument set, annotating the code, and writing a help file. It's as easy as npm installing as a package, setting up a .env file with your Zotero userID and API key and your PDF directory. Since I didn't originally set out to make this for anyone by myself though, there are constraints based on my setup, such as my naming convention and file structure. But it would be trivial for anyone with some JS experience to adapt it to their own specific situation.
Below is an example output of just 4 items (of thousands in my library, all automatically added, and attachments automatically generated). I didn't add a single thing manually, neither top level items, nor attachments. All I have to do is add new PDFs to a specific Dropbox folder, with a specific naming convention (and I have a MacOS service/shortcut for that too), and my library stays up-to-date, fully remotely accessible, and complete. No more rummaging through hundreds of filenames.
screenshot showing just a few items, with attachments automatically added: https://i.imgur.com/FCaBIBH.png
I guess overall I just wanted to say, I love what you guys have done here with Zotero, and how easy it is to build for.
I just finished a big project to totally automate my Zotero experience using the API and a few scripts... and I feel like there's literally no one else in the world who would understand the simple joy of my accomplishment and my new level of appreciation for Zotero and the API, except you people here, so please excuse my need to share!
Problem: I had a back catalog of thousands of PDFs I have accumulated over the years. I already had a consistent naming scheme for the files, but I needed to get these all into Zotero, and I wanted to automate the process of importing the files. Additionally, I wanted to bypass Zotero's automatic imports from just dragging a bunch of files into Zotero, because I need to control the attachments and control some other aspects of data quality.
I wrote two scripts, one is a shell script running every night as a cron job, which scans my Library folder, looks for new files, generates a list of DOIs from any new PDFs, and then uses the doi2bib API service to download a concatenated bibtex file, which is then automatically imported into Zotero. (Automating this was difficult, because of the way Zotero handles imports - requiring the user to confirm the imports in a little dialog. I use MacOS and wrote a tiny little AppleScript application that runs as part of the shell script, after the file is imported, to automate the selecting of the "Unfiled Items" folder, and then the confirmation of the import in the little "confirm import" dialog.
The second script is a Node.js script that automatically adds 2 attachments to every item in my Zotero library, a relative link to the PDF file itself, and a link to the file on Dropbox (my library is on Dropbox, and this lets me view my files remotely), that I configured to run as a global command line program, with a pretty nice suite of arguments/options. It runs regularly as a cron job, where it creates these two attachments for any new items added from the previous script, but I can also run it with customized options from the command line if I need to do anything manually for any reason. It runs in 3 modes: make attachments by key(s), where it is fed a list of Zotero item keys, and then there are two variations of make attachments by chunk of files, ordered by most recent, where you either 1. specify the starting item, number of items, how many times to repeat the chunk (eg 30 items at a time, starting at item 0, the most recent, 3 times, would make 3 requests of 30 each, for items 0-30, 30-60, and 60-90, respectively, or 2.) same as #1, but where the program repeats the specified chunk as many times as are needed to completely process the whole library.
I feel like I would like to share these various utilities (I also have made a ton of little JS snippets that can be used in the "Run Javascript" too within Zotero for all kind of different situations), but I am not sure how much interest there would be, and also the system is fairly well tailored to my own specific use-case. I looked at work that was already out there such as pyzotero, but found them to have simultaneously too many features and not enough features that I needed for my specific use. On the other hand, these tools, while probably difficult to understand to non-programmers or people who are looking for an out of the box solution, might prove to be a useful starting point for someone else who is looking to jump start their own custom solution. If anyone is interested I would be happy to share any of the code. I did spend a lot of time fool-proofing the argument set, annotating the code, and writing a help file. It's as easy as npm installing as a package, setting up a .env file with your Zotero userID and API key and your PDF directory. Since I didn't originally set out to make this for anyone by myself though, there are constraints based on my setup, such as my naming convention and file structure. But it would be trivial for anyone with some JS experience to adapt it to their own specific situation.
Below is an example output of just 4 items (of thousands in my library, all automatically added, and attachments automatically generated). I didn't add a single thing manually, neither top level items, nor attachments. All I have to do is add new PDFs to a specific Dropbox folder, with a specific naming convention (and I have a MacOS service/shortcut for that too), and my library stays up-to-date, fully remotely accessible, and complete. No more rummaging through hundreds of filenames.
screenshot showing just a few items, with attachments automatically added: https://i.imgur.com/FCaBIBH.png
I guess overall I just wanted to say, I love what you guys have done here with Zotero, and how easy it is to build for.