Upgrade StorageMigrating from Papers 2 or 3 to Zotero (macOS)
I recently had to move a >9k library from Papers 2 to Zotero. I ended up using Papers 3 as an intermediary, so this applies to Papers 3 to Zotero as well. Here are some notes on how I did this, with the hope that can be useful for somebody else. I should note that my main goal was to preserve a large number of manual collections in the Papers 2 library. Other than the entries and files themselves, I didn't care about highlights, tags, or other things that might make this process more complex.
After doing some research, the best option seemed to be to use the Papers 2 to Zotero python code here (https://github.com/jdidion/papers2). However, when I loaded the library, it only listed around 6k of the papers, and was unable to deal with the very large number of collections in the library. So that option was out.
The strategy was to export each collection to a .bib file, import each as a new collection in Zotero, and recreate the folder structure I had in Papers.I ended up moving the Papers 2 library to Papers 3, because the scripting is better. Steps were:
1. I wrote a bit of AppleScript (you can find it here: https://github.com/moffat/applescript, https://raw.githubusercontent.com/moffat/applescript/main/Export Papers Collections.applescript) to export each collection in Papers 3 to a given folder (you need to change the folder to something that exists in your computer). My folder structure in Papers 3 only had two levels (so Collection/SubCollection) so if you have a deeper structure, you can either modify the script to keep going down more levels, or 'flatten' your folder structure in Papers so that you don't have more than 2 levels of nested collections.
2. The code will export one collection per folder (smart or manual) in Papers 3. I was trying to avoid ending up with too many duplicates in Zotero as as I could, so I first created a "smart collection" in papers that contains all the unfiled papers. That is, all items are either in an existing collection, or in the "undefiled paper" smart collection. That way, I could import that smart collection and all the individual collections and capture the entire papers 3 library. If in doubt, you could also export the entire papers 3 library as a .bib file, but I'm fairly sure the above works.
3. The code will recreate the structure in Papers 2. For example, if you have a collection structure that looks like:
- Collection 1 (contains entries)
- Collection 2
-- SubCollection 1 (contains entries)
-- SubCollection 2 (contains entries)
The applescript will create a .bib file for both collections, but also a folder called Collection 2 with .bib files for each sub collection. Obviously, you don't need to import both Collection 2.bib and SubCollection 1.bib and SubCollection 2.bib to Zotero, as the first file just contains everything in the SubCollections.
4. Import each individual "collection" file to Zotero. This is a pain (I had many collections) but at least can be done more or less in large groups. Make sure you have the "add as a new collection" option selected.
5. Recreate the folder structure in Zotero. In the example above, I've imported the two SubCollections, so I had to create "Collection 2" as a folder in Zotero, and move the two SubCollections to the folder. This is a *massive pain* in Zotero because you can only select one folder at a time to move them around. Very 1986!
6. Now, if you had items in several separate collections in Papers, they will show up as duplicates in Zotero. The right way to deal with this is merging duplicates. If you have a few, you're lucky. If you have many hundreds like me, please read: https://forums.zotero.org/discussion/85991/automatic-merging-of-duplicates-using-keyboard-maestro-macos-only#latest
I'm sure the import process can be done in a smarter way using some of the Zotero APIs, but the above, while slow, allowed me to do this transfer at my own computer without much trouble.
After doing some research, the best option seemed to be to use the Papers 2 to Zotero python code here (https://github.com/jdidion/papers2). However, when I loaded the library, it only listed around 6k of the papers, and was unable to deal with the very large number of collections in the library. So that option was out.
The strategy was to export each collection to a .bib file, import each as a new collection in Zotero, and recreate the folder structure I had in Papers.I ended up moving the Papers 2 library to Papers 3, because the scripting is better. Steps were:
1. I wrote a bit of AppleScript (you can find it here: https://github.com/moffat/applescript, https://raw.githubusercontent.com/moffat/applescript/main/Export Papers Collections.applescript) to export each collection in Papers 3 to a given folder (you need to change the folder to something that exists in your computer). My folder structure in Papers 3 only had two levels (so Collection/SubCollection) so if you have a deeper structure, you can either modify the script to keep going down more levels, or 'flatten' your folder structure in Papers so that you don't have more than 2 levels of nested collections.
2. The code will export one collection per folder (smart or manual) in Papers 3. I was trying to avoid ending up with too many duplicates in Zotero as as I could, so I first created a "smart collection" in papers that contains all the unfiled papers. That is, all items are either in an existing collection, or in the "undefiled paper" smart collection. That way, I could import that smart collection and all the individual collections and capture the entire papers 3 library. If in doubt, you could also export the entire papers 3 library as a .bib file, but I'm fairly sure the above works.
3. The code will recreate the structure in Papers 2. For example, if you have a collection structure that looks like:
- Collection 1 (contains entries)
- Collection 2
-- SubCollection 1 (contains entries)
-- SubCollection 2 (contains entries)
The applescript will create a .bib file for both collections, but also a folder called Collection 2 with .bib files for each sub collection. Obviously, you don't need to import both Collection 2.bib and SubCollection 1.bib and SubCollection 2.bib to Zotero, as the first file just contains everything in the SubCollections.
4. Import each individual "collection" file to Zotero. This is a pain (I had many collections) but at least can be done more or less in large groups. Make sure you have the "add as a new collection" option selected.
5. Recreate the folder structure in Zotero. In the example above, I've imported the two SubCollections, so I had to create "Collection 2" as a folder in Zotero, and move the two SubCollections to the folder. This is a *massive pain* in Zotero because you can only select one folder at a time to move them around. Very 1986!
6. Now, if you had items in several separate collections in Papers, they will show up as duplicates in Zotero. The right way to deal with this is merging duplicates. If you have a few, you're lucky. If you have many hundreds like me, please read: https://forums.zotero.org/discussion/85991/automatic-merging-of-duplicates-using-keyboard-maestro-macos-only#latest
I'm sure the import process can be done in a smarter way using some of the Zotero APIs, but the above, while slow, allowed me to do this transfer at my own computer without much trouble.
-
pcatarsiYou’ve done a great job. I’d like to ask the developers, to integrate your scripts into the next version of Zotero. How can you do that?
-
johnostrowickHi. Thanks very much for this but I can't get it to work. When I run the applescript it completes without any errors but does not seem to generate any files. My library is about 3400 papers in a single collection called "ALL". If you can advise further it would be very helpful. MacOS X 10.13. (Can't upgrade, hardware limit).