Available for beta testing: Import Mendeley annotations

dstillman
The latest Zotero beta adds the ability to import Mendeley annotations directly into Zotero for use in the new PDF reader.

Zotero's Mendeley importer will now import highlight and note annotations along with attachments. Zotero will extract the text for text highlights, allowing you to view the text in the annotation sidebar and drag the text into your notes. Rectangle highlight annotations will be automatically converted to Zotero's image annotations, allowing you to view thumbnails in the annotation sidebar and drag the images directly into your notes.

If you've previously imported a Mendeley database into Zotero, you should be able to simply repeat the process to add annotations to your existing imported attachments. The importer previously created notes from note annotations, and those existing notes won't be touched.

As before, you'll need to be running Mendeley 1.18 for the importer to work, due to database encryption in later versions of Mendeley.

For this initial version, I would recommend making a backup of your Zotero data directory and temporarily disabling auto-sync before the import. You can re-enable auto-sync once you've confirmed that everything was imported correctly.

Let us know if you run into any trouble.
«1
  • basilevh
    It works perfectly, thank you!
  • gschrock
    Absolutely brilliant! Only difficulty I ran into was downgrading Mendeley to 1.18 - it wouldn't automatically download all my attachments from the server so I had to click on hundreds of entries to make sure they downloaded. I'm sure that was no accident...

    One question: are there any potential issues if I run the beta version (with the PDF reader) on one machine and sync the database and attachments with another machine running the release version?
  • dstillman
    are there any potential issues if I run the beta version (with the PDF reader) on one machine and sync the database and attachments with another machine running the release version?
    If you have the option to upgrade, I'd recommend doing so, but this should theoretically be OK. You'll receive a warning on the non-beta computer that some data couldn't be downloaded, and that it may have been saved with a newer version of Zotero — that'll be referring to the annotations. Notes created in the beta may appear a bit differently in the release version, and the newer features will be missing (live annotations, embedded images), but the old note editor shouldn't damage the notes. Possible there's something we haven't thought of, though we've been testing with both for a while. If you do this and run into problems, let us know.
  • gschrock
    @dstillman Okay, I've gone ahead and upgraded then. Do you know whether this preview feature is going to be in an upcoming 5.x release version or not until 6.0 is released?
  • dstillman
    @gschrock: The PDF reader and new note editor will be part of Zotero 6, if that's what you're referring to.
  • gschrock
    @dstillman: I assumed that, but was more asking whether it would be part of any 5.x releases before 6 comes out. Part of the assumption here is that the release of 6 isn't imminent - but maybe/hopefully so?
  • glanciano
    edited March 17, 2021
    Hi @dstillman, many thanks for your work on this new feature.

    I followed the instructions provided at https://www.zotero.org/support/kb/mendeley_import to import data from Mendeley, but didn't manage to have my highlights and pdf notes imported. Only Mendeley's general notes are correctly imported.

    I am using version 5.0.97-beta.5+b3e46392b of Zotero. Is there any known issue with this version? Are there extra steps to be performed to have everything imported?
  • dstillman
    @glanciano: Have you enabled Zotero's PDF reader in the General pane of the preferences? If you don't turn that on and just open a PDF, it will open in an external PDF reader on your computer, where you won't see the annotations.
  • glanciano
    edited March 17, 2021
    @dstillman: I'm using Zotero's PDF reader, but I still can't see the annotations I was expecting (i.e., highlights and "sticky" notes).

    The annotated papers come from a Mendeley group library that, as suggested in the import instructions, has been copied to Mendeley's "My Library", so that Zotero could retrieve them.
    In Mendeley Desktop, I can see my annotations both in the originals and in the copies. Indeed, they seem "linked" in some way (e.g., after creating a new highlight in one, I can see it also in the other).

    Could it be that the annotations are not actually copied from the original group to "My Library"? Unfortunately, I cannot even check how a sample pair of documents is rendered in Mendeley's web library (or in the Mendeley Reference Manager client), as it appears to be down since (at least) yesterday. :/

    Thanks for your support!
  • dstillman
    @glanciano: Hmm, that's bizarre, and I'm not sure it's not a Mendeley bug. If you annotate a PDF in a group library in Mendeley, and then drag the item to your library, the annotations show up when you open the PDF from your library, but in the database they only exist on the group item, so Zotero doesn't import them. I guess we can follow this behavior and people can just delete annotations copied from groups that they don't want on their personal files, but I have no idea why they're doing it this way.
  • glanciano
    @dstillman: thanks for your reply.

    So, would it be feasible for you to detect whether an item is a copy of another one belonging to a group and, in that case, fetch the existing annotations during import?
    I really hope the answer is "yes", I can't wait to ditch Mendeley once and for all. :)
  • MACorreia
    Hello,

    So, I ran into the same problem as @gschrock of having to individually download hundreds of pdf (about 2/3 of my database) in Mendeley after downgrading to 1.18, which equals to a frustrating morning. It also deleted around ~100 files for no reason I can understand. But like it was said, this seems to be Mendeley problem. Mentioning just so other people making the move are aware.

    But also, I'm having the same issue as @glanciano , in which none of my highlights or notes within papers have been imported. Zotero's pdf reader is enabled and these are ALL the documents on My Library in Mendeley. I'm not sure I entirely followed the exchange above on what the problem might be. But, when I used the Importer on Zotero, there was an option for File Handling, where I chose "Link to files in original location" as opposed to "Copy files to the Zotero storage folder". Could this be the problem or am I way off? I chose that option because it felt like the other option would just fill up the 300MB Zotero storage very quickly.

    I've also downloaded Zotfile, not sure if that's relevant in any way.
  • dstillman
    @MACorreia: Nothing to do with "Link…" vs. "Copy…" — see Stored Files and Linked Files for more an explanation of that.

    The problem discussed above is about annotations on files copied from (or possibly to) group libraries, which Mendeley seems to treat in a very bizarre way in its database. We're looking into addressing that.
  • cberrigan
    I also tried to downgrade Mendeley Desktop to 1.18 on a Mac in order to export my annotations, but the Mendeley database would not recognize my annotations from 1.19. Total panic and frustration. 1.19 will still recognize my annotations (thankfully!!) but it unfortunately doesn't look like this will work for me to import everything into Zotero.
  • dstillman
    @cberrigan: Not sure what you mean there. Are you saying that Mendeley Desktop 1.18 doesn't show you your annotations, or that your annotations aren't being imported into Zotero?
  • cberrigan
    @dstillman I mean that after downgrading to Mendeley Desktop 1.18, Mendeley does no longer shows me the annotations I made in Mendeley. When going back to 1.19, they are there. When exporting from Mendeley Desktop 1.19, as you said, Zotero does not import the Mendeley annotations. The only way for me to import annotations into Zotero is to export a PDF with annotations from Mendeley, but at that point they are embedded within the file.

    Also: a super useful feature of Mendeley is that you can click one tab called "Notes" to see general notes + a color-coded list of all highlights and annotations. Is there a way to view annotations this way in the Zotero beta? The only way I could see to do it was to make a new note from annotations, which requires an extra step, and also removes the color-coding of the highlights.

    Thanks!
  • dstillman
    @cberrigan: Can you view the annotations online on the Mendeley site? This is fundamentally a question for Mendeley, but if they're visible online it should be possible to retrieve them.
    a super useful feature of Mendeley is that you can click one tab called "Notes" to see general notes + a color-coded list of all highlights and annotations. Is there a way to view annotations this way in the Zotero beta?
    Zotero has much more functionality. In the left sidebar in the Zotero PDF reader, you can see all annotations, including extracted highlight text, image thumbnails, and tags. In the right-hand sidebar, you can view all child notes for the parent item. Creating a note from annotations is for when you want to pull those annotations into a note or draft, which you can later insert into your word processor with active Zotero citations. See the announcement page for details, and start a new thread if you have other questions on this.
  • bailh
    Many thanks to the team !

    I successfully imported my Mendeley collection to Zotero beta with my highlights and pdf stored through a cloud storage facility.
    However, when I open the pdf reader, a red banner says :
    "An error occured during 'setState'. What does it mean ? How to fix it ?

    PDF.js v2.8.146 (build: 7dd64325d)
    Message:
    Stack: Unix error 2 during operation open on file /home/pauline/Zotero/storage/9QDTZTPP/.zotero-pdf-state (No such file or directory)
    (No stack trace)
    From previous event:
    captureStackTrace@resource://zotero/loader.jsm -> resource://zotero/bluebird/debuggability.js:915:23
    CapturedTrace@resource://zotero/loader.jsm -> resource://zotero/bluebird/debuggability.js:807:5
    longStackTracesCaptureStackTrace@resource://zotero/loader.jsm -> resource://zotero/bluebird/debuggability.js:482:19
    doThenable@resource://zotero/loader.jsm -> resource://zotero/bluebird/thenables.js:60:5
    tryConvertToPromise@resource://zotero/loader.jsm -> resource://zotero/bluebird/thenables.js:28:20
    module.exports/Promise.cast@resource://zotero/loader.jsm -> resource://zotero/bluebird/promise.js:222:15
    Zotero.File</this.putContentsAsync@chrome://zotero/content/xpcom/file.js:410:14
    _setState@chrome://zotero/content/xpcom/reader.js:495:2723
    ReaderInstance/<@chrome://zotero/content/xpcom/reader.js:415:21
  • dstillman
    @bailh: That should be fixed in the latest Zotero beta, available now via Help → "Check for Updates…". Thanks for reporting.
  • bailh
    edited March 26, 2021
    Thanks for your reply! That fixed the problem of the red banner.
    However I noticed that many (some are still present) of my highlights were lost during the migration + update. Do you see anything that can explain that loss ? The update ? The sync ? I use a cloud storage service for my pdfs. In this case, where are stored the highlights and annotations ? Within Zotero server ?
  • dstillman
    edited March 26, 2021
    @bailh: It may be the issue discussed above — if the PDFs were copied from/to a group library in Mendeley, the annotations may be (bizarrely/incorrectly) stored on the group items in the Mendeley database and won't import. We're looking into addressing that, and if we do you'll be able to import again to pull them in.

    If you're not sure if that's the issue for you and are willing to share a copy of your Mendeley database privately, you can ZIP it and email it (or a link to it, if it's large) to support@zotero.org and we can see what's happening in your case.
  • nerimarsili
    @gschrock @MACorreia I had the same issue you had (Mendeley would not re-download the pdf attachment from the cloud after the downgrade). But I noticed that it downloaded some, namely all the PDFs that were included in my folders. The solution was to create a new folder and drag and drop all my library into the folder. When I restarted Mendeley, it downloaded every single pdf in my library!
  • dstillman
    @nerimarsili: OK, we'll add that fun little detail to the import documentation — thanks.
  • jarombae
    Hi,
    After a couple of panic attacks with downgrading the mendeley version, as I did not see all my notes, references and attachments; restoring from a backup from mendeley, etc.
    I can say, the import from 1.18 works smoothly.

    Question: how would a re-import look like? Would it do a delta compare and reimport only missing docs in zotero or would it reimport everything?

    Thanks.
  • dstillman
    @jarombae: Yes, it will import only items that haven’t already been imported.
  • MelWeiß
    Hi,
    I only understood what was happening to my library last sunday. I am about to submit my PhD thesis by the end of this month. You guys have really saved me. Thank you so much, really. This was a quick and uncomplicated solution in comparison to everything that the elsevier company offered as solution in their blurry "we improve your experience". Thank you, thank you, thank you for this smooth solution.
  • gretamik
    It worked! Some highlights are out of place though, but it is not a big deal for me. Thanks to the new PDF editor I am now definitely switching to Zotero once and for all. The beta version is fantastic and can't wait for the new official upgrade. Thanks!!
  • e2henrik
    Hi,

    I have a strange issue where some highlights are not imported for me. For example, I have a document where only 4 out of 5 highlights are imported. I checked the PDF on Mendeley online, and all highlights are there. Any ideas why this is happening?
  • dstillman
    @e2henrik: It may be what's explained above — Mendeley stores annotations for files that exist in both your personal library and a group in a bizarre, possibly unintentional way that depends on which library you open the file from. We're working on a fix.
  • e2henrik
    @dstillman Ok, great, thanks for your quick reply!
This discussion has been closed.