Advice on import of large Papers3 (formerly mekentosj papers) library

pcavdwel
Hi,

After years of gradual degradation, it looks like the Papers app is finally coming to a hard stop. So I am looking to move on with minimal disruption, but am not hopeful that can be done.

I did quite a bit of searching for alternatives that allow for some sort of import of the information (notes, PDFs, collections, etc), without much luck. I was hoping that maybe others have some experience that could be helpful. It seems to me that some sort of scripting is in principle possible, but I have not found any evidence that this has been done (at least not recently). I suppose most sensible people have abandoned Papers at v2...

Any advice would be appreciated.

Thanks!
Patrick

PS. I have a huge array of collections and sub-collections (many levels), and >8K entries, so any overly manual solutions are not appealing (although possibly unavoidable).
«1
  • bwiernik
    Unfortunately, I don't think that anyone has developed a tool to automatically export all collections from Papers. You can export your whole library and import that, but to maintain the collection structure, you would need to export and import them one at a time.

    Some previous discussions on this issue:
    https://forums.zotero.org/discussion/34363/how-to-migrate-from-papers-3-0-to-zotero
    https://forums.zotero.org/discussion/35693/importing-from-papers3-into-zotero4
  • RintzeRintze
    After years of gradual degradation, it looks like the Papers app is finally coming to a hard stop.
    Just curious, but what makes you say that Papers has come to a stop?
  • emilianoeheyns
    @pcavdwel the papers3 database format isn't too complicated it seems; if you can get me a copy of your database I can have a stab at a conversion script.
  • Nachury
    Likewise, papers is no longer useable for me. Citation tool no longer recognizes {inline references} and the pdf organizer constantly gets stuck.

    @emilianoeheyns did you write a script to migrate the collections into Zotero? That would be pretty slick.
  • Fedster
    I'd be pretty interested too -- my Papers3 library is basically a bunch of PDF and 10 collections (which I can recreate really). If I had a reasonably fast way of moving the lot to Zotero I'd abandon Papers3 without ever looking back.
  • adamsmith
    If you don't need the collections exported, just exporting via BibTeX or RIS should work.
  • emilianoeheyns
    @Nachury never got a DB to toy with, but if you're not looking to convert the collection info, follow @adamsmith 's recommendation.
  • Matthias
    Papers 3 is pretty scriptable and offers access to its manual & smart collections (and to its contained publications) via its scripting API. IMO, this would be much better & easier than messing with the database.

    Here's some basic examples: 
    tell application id "com.mekentosj.papers3"
    	-- returns all (manual & smart) collections as scriptable objects
    	set a to every collection item

    	-- returns list of all manual collection names
    	set m to name of every manual collection item

    	-- returns list of all smart collection names
    	set s to name of every smart collection item

    	-- returns all publications for the first manual collection as scriptable objects
    	set p to publication items of manual collection item 1
    end tell
    I.e., you could write a script that iterates over each manual collection, gets all of its contained publications and exports them as a group.

    Similarly, Papers also offers scripting access to the PDF files and the individual annotations.

    For Bookends, I've written a "Papers to Bookends" script which exports all publications selected in your Papers 3 library to Bookends. This script could serve as an example how to get access to Papers elements via the scripting API:

    https://github.com/extracts/mac-scripting/tree/master/Papers3/Papers_To_Bookends

    The script will also transfer the publication's primary PDF (if there's any) and can also transfer the publication's rating, color label, flagged status, language, edition, citekey, and "papers://..." link. The PDF file's annotations can be also transferred (see the README file).

    An intro to scripting Papers 3 can be found here: https://extracts.github.io/mac-scripting/Papers3/Getting_Started

    I could help developing a "Papers to Zotero" export script, but I would need firm advice how to automate the import on the Zotero side.
  • catherineseo
    Matthias, Did you ever develop the script to migrate collections from Papers3 to Zotero? I am looking to do it.
    thanks,
    Catherine
  • Matthias
    edited August 8, 2018
    Hi Catherine, great if you can tackle it! And no, I haven't done this so far, since I lack the knowledge on the Zotero side. I could help with the Papers part. But the "Papers_To_Bookends" script may give you some hints already.
    Matthias
  • Sekator
    Catherine, Matthias, any news on the script? It would help many people, me included.
  • Matthias
    Hi Sekator, as I mentioned above, I would probably be able to contribute the Papers part but I don't know how to automate Zotero in order to assign collections or transfer Papers-specific metadata.
  • emilianoeheyns
    Zotero has an API you can use to get data in (but that requires storage quota for any attached files). The other way would be to prep the data into a format that can be imported; if you want to import collections, that's going to be BibTeX or Zotero RDF.
  • Matthias
    Thanks @emilianoeheyns, I‘ll have a look!
  • solesedeno
    Another one looking for the sript Matthias... It would be very useful!
  • idewitt
    A note for anyone who might have the wherewithal to take this on, there is a Papers2 to Zotero migration script on github (https://github.com/jdidion/papers2) that utilizes the Zotero API. From my own testing (with Papers2), it needs some updating.
  • daeh
    I got frustrated with the indefinite delays in ReadCube's desktop app and the overall instability of their software and decided to transition to Zotero. Here's the python script I wrote to facilitate importing my library while preserving the Papers citekeys and retaining the supplementary materials associated with the publications. It's definitely not foolproof but seemed to do the trick for me. https://gist.github.com/daeh/abc6d46d897b58a657699fa1a408573e

  • nrepina
    THANK YOU @daeh your script works perfectly!!!!!
  • emilianoeheyns
    @daeh is there anything I can do to have BBT import those paths by default?
  • daeh
    @emilianoeheyns can you describe the desired behavior in more detail? Which paths?
  • ChrisWaloszek
    @daeh
    Do you have any advice how to transfer ReadCube's "flags", "color labels" and "reading status" to Zotero? They are not included in the BibTex or RIS export from ReadCube. See also my post here:
    https://forums.zotero.org/discussion/comment/336191#Comment_336191
  • emilianoeheyns
    @daeh I see in your script you're doing something with file lines, I assume to change the input such that supplementary attachments are properly imported? I might be able to just add that to BBT so these files would always import without further action.
  • daeh
    @emilianoeheyns the issue is that the paths to supplementary files are not exported by Papers3, so there's not much BBT can do about that. Part of what the python script does is search the Papers3 library for supplementary files and add those to the *.bib file so that they're properly imported by Zotero/BBT.

    @ChrisWaloszek does Zotero have things equivalent to colors and flags?
    One dumb manual solution might include making a collection of the relevant category in Papers (e.g. a Collection of all the items that have a red color label), export that Collection like I describe in the instructions for the script (i.e. with the "BibTex Record" set to "Standard"), and then import it to Zotero.

    That will give you a new Zotero Group of all the items from your Zotero library that had the color label 'red' in Papers. Then you can easily batch-change the metadata for all the items in that Group in one go. But again, I'm not sure what the Zotero equivalent to color labels is (I imagine there's an add-on but I'm new to the Zotero ecosystem).
  • emilianoeheyns
    Ah I see. I can in principle just hand Zotero attachment paths to try, but I cannot know papers_library.
  • adamsmith
    @daeh see https://www.zotero.org/support/collections_and_tags#colored_tags for color labels
  • daeh
    @adamsmith do you know if there's a bibtex key that will cause Zotero to assign a color tag on import? Otherwise the strategy I described above might still be the best I can think without comitting to the next level of depth with Zotero and Papers scripting.
  • adamsmith
    Definitely nothing in bibtex. I'm actually not even sure they can be preserved on export/import at all.
  • dstillman
    They can't.
  • Matthias
    edited July 18, 2019
    @daeh @ChrisWaloszek If it's the classic Papers 3 desktop app for Mac from which you're trying to migrate to Zotero, it's worth noting again that Papers 3 is fully scriptable and as such provides programmatic access to many of its data model objects and properties.

    Using the scripting access in Papers 3, you can easily get programmatic access to say, primary attachments or supplementary files for a publication, or get the publication's color label, flagged status, read status, rating, or the names of the manual collections the paper is contained in, etc. I.e., for a given publication, you can programmatically access many properties which aren't readily available in one of the standard export formats.

    Note that you can also run AppleScript code from the command line. See e.g. https://alvinalexander.com/source-code/mac-os-x/how-run-multiline-applescript-script-unix-shell-script-osascript. So if you prefer other scripting languages, it's still possible to benefit from the scriptability in Papers 3.

    As mentioned in a post above, I had plans to write a fully-featured "Papers To Zotero" migration script, but I still lack the knowledge on the Zotero side, and unfortunately haven't found time to learn its sync API. Still, the above mentioned "Papers To Bookends" script can give clues how to script Papers 3.
  • jasperkenter
    Hi all, I am also looking to transition. I have a large number of collections, so manually exporting each collection and then importing is pretty unappealing... Happy to share my library if that's of help.
Sign In or Register to comment.