Mendeley import - Zotfile not moving all files

edited September 28, 2019
Hello everyone,

I'm importing my Mendelay library into Zotero since I'd like to try to leave Mendelay completely.

The import is slow but everything goes smoothly. Now, since I don't want to sync all the PDFs (> 1.5gb for 1100 entries) with Zotero cloud, I've setup Zotfile to move files and create attachements.

The issue is that when I import the Mendelay library, the PDF files get split: some files are moved and renamed correctly by Zotfile and some others are not. At the end of the import process I basically have only roughly 500 entries (800Mb) in the Zotfile destination and the rest are still being imported and saved in the Zotero metadata storage folder.

I cannot find a way to move them all during the import and I don't even know how to do it after them import, once the library is already built.

Any ideas?
  • edited September 27, 2019
    One of the developers might be able to help diagnose during import. But, after import, you can right-click an item and choose Manage Attachments -> Rename Attachments. That will move and rename the files.

    What cloud system are you using?
  • I'm not using cloud right now. All the folders are local with no syncs but I plan to setup manual or semi-automatic syncing on a GDrive account.

    Can I actually do what you suggest on all the entry in one time? Because I cannot afford to that one at a time for more than 500 entries.

    If yes, I will re-import (right now I've deleted all the imported entries) mendeley data and just don't bother about the import behavior.
  • So, I tried a little more and no matter what I do. Overall I'm getting hundreds of duplicates.

    The import always splits the files and does not rename them all. Some of them remain inside the Zotero metadata folder and some of them are moved in the Zotfile attachment folder.

    If I then try to use the "manage attachments -> rename attachments" approach on the whole library I get a ton of duplicates. However I'm quite sure I have just a few duplicates in the starting mendeley library.
  • We can't really help with ZotFile, but I believe there's a bug that prevents it from moving multiple files at once — or perhaps that causes it to fail on some files, which interrupts the process when moving multiple. If you can figure out how to consistently reproduce that on a specific set of files, or if you can provide a Debug ID for it happening, someone might be able to submit a patch to ZotFile to fix that.

    Note, too, that Zotero provides an option during the import to add links to files rather than storing them, so if your Mendeley files are already where you want them, you might consider wiping your Zotero data directory, disabling ZotFile for the import, and just using that setting instead, at least for the initial transfer.
  • edited September 29, 2019
    The bug is constantly there so I guess it should be fairly easy to reproduce it.

    Ok, I'll try to disable ZotFile and let Zotero attach them without copying.

    Just a couple of question on this:

    1) What if I do it and then want to move files manually from the original folder to another folder? Is it possible without losing the links?

    2) Can Zotero still rename files while importing and linking? ...I guess not since I just tried.

    So, the final question is:
    is there a way to at least rename all files during or after the import? (Enabling Zotfile after the import and force rename still leads to duplicates or missing renames)
  • @dstillman Does Zotero do its native renaming during Mendeley import?
  • It doesn't. Or at least it didn't during my various attempts.
  • It does for stored files. It looks like it doesn't currently for linked files, even if "Rename linked files" is checked in the General pane of the preferences, but you can rename all the linked files after the fact by selecting them in Zotero, right-clicking, and choosing "Rename File from Parent Metadata". (To select multiple files, you can create a saved search for [Attachment File Type] [is] [PDF] and then press Ctrl-A/Cmd-A to select all the matching items.)

    To be clear, this is about Zotero's renaming, not ZotFile's. Zotero's is currently much more basic, with just title, author, and year.

    In theory it should follow the "Rename linked files" preference while importing, but I'm a little uncomfortable doing that for imports. Even though that preference is off by default, it seems very easy for someone new to Zotero to do a large import and inadvertently rename all their external files, whereas the rest of the time they might be more likely to realize what's going on as they link to individual files. I suppose an alternative would be to just add that as an option during import and have it override the current setting in the preferences.
  • ah, didn't think about the saved search trick to select all pdfs.

    I'll just dump Zotfile and go with the Zotero rename then. I would have chosen author title and year only anyway.
  • Note that the main advantage of ZotFile going forward, once you have everything imported, is to automatically convert new attachments into linked files. By default Zotero will save files you add, including all the files you add via the web (which is the main way to use Zotero), as stored files. So if you want linked files, you can re-enable ZotFile once you're satisfied with the import.
  • yeah, that's exactly what I was thinking.

    Although I still have one doubt, is there a way to move all the linked PDF files after the import without losing the link?

    Because right now every PDF is in the old Mendeley folder and I'd like to change the path for the new Zotero PDF container. If I turn on Zotfile to move all the PDF I think I will be stuck again with the bug (even if the files are already named correctly).
  • Yep — you want to use the Linked Attachment Base Directory feature, which will convert all the file paths to relative paths so that you can still access them after you move the directory or access them from another computer where they're synced externally.
  • edited September 30, 2019
    ok.

    I guess I'll give it a try then:

    1) disable Zotfile
    2) import from mendeley
    3) set to import and attach files (no copy & store)
    4) save an advanced search for PDF only
    5) set up the Zotero renaming rule which I prefer and activate it
    6) select all the results from the saved search (point 4) and "Rename File from Parent Metadata"
    7) disable the autorename of Zotero and close Zotero
    8) move files where I want
    9) open Zotero
    10) set the correct "linked attachment base directory" (point 8)
    11) setup Zotfile with the same renaming rule of Zotero
    12) setup Zotfile and make it point to the same "linked attachment base directory" that I setup for Zotero
    13) enable Zotfile

    Now everything should be working properly like I've always used Zotero and Zotfile.


    EDIT:
    nevermind, forget this. Zotero does not allow renaming with multiple elements selected.
  • Yes it does — you just need to have only file attachments selected. That's the whole point of the search part.

    (Also, just to clarify terminology, for (3) it's "link" vs. "store", not "attach" — all attachments are attached.)
  • I think having an option to rename linked files in the Import dialog would be useful and transparent to the user (versus just following the preference or not renaming at all).
  • edited October 1, 2019
    Actually, I saved the advanced search with the [Attachment File Type] [is] [PDF] setting but when I select all with ctrl+A and I right click I do not have the option of "Rename File from Parent Metadata".

    On the other hand if I click on one PDF only or just a few (manual selection) I can actually get the "Rename File from Parent Metadata".

    I guess there might be a limit on the selection number for some of the commands.

    EDIT: I think I found the culprit. The [Attachment File Type] [is] [PDF] also finds attachments of online PDFs (direct links like www.blablabla.com/asdasdasd.pdf). Fortunately I only had a couple of items with attached online PDFs and by manually deselecting them I could get back the "Rename File from Parent Metadata" command. I have now started the renaming process.

    EDIT2: ok, everything went fine. Now I'm struggling with the moving part? ...How do I avoid using Zotfile to just move the PDFs?
  • For what it's worth, the failures and duplicates from ZotFile are probably now fixed in ZotFile 5.0.13.
  • nice to know. I think I will check it out since I would still have trouble moving files otherwise.

    Right now should work directly from the mendeley database import then?
  • Seems like the bug is still there. If I import from Mendeley while Zotfile is active, some files are correctly renamed and moved in the zotfile path, while some others are simply copied in the zotero metadata folder. I will try forcing a move and rename but I guess duplicates will be created again due to the files being split between the two folders.
  • Can you provide a Report ID after reproducing the problem?
  • edited 9 days ago
    Ok, seems like the import bug is not fixed but the "move and rename" is instead good to go:

    - the import still does not move and rename all the files. Some will end up in the Zotfile folder correctly renamed and some will end up in the metadata folder. The number of files ending in one folder or the other appears to be random (I tried 3 times)

    - however, the bruteforce "move and rename" on the entries with an attachment now correctly moves the remaining files from the metadata folder to the Zotfile folder without creating duplicates even if the user selects ALL the entries. Actually, this renamed all the files but 5. I then did a second time the "move and rename" and also the last 5 files were correctly moved.


    I guess we can consider it ok since this makes the import a little slower because it becomes a multiple-step process but in the end everything looks fine now. One has just to remember to check if some of the attachments get stuck into the metadata folder and, if that's the case, one has to use the "move and rename" a sufficient number of times.

    Do you still need a Report ID? If yes, do I reimport again and create it after the import but before the forced "move and rename"? Or what do you want me to do?
  • Do you still need a Report ID?
    Up to you. If you're just getting started and can just wipe your Zotero data directory and reimport, then a Report ID would help us get this fixed along with the ZotFile developer. If your database is already set up, it's not worth worrying about.
    f yes, do I reimport again and create it after the import but before the forced "move and rename"?
    Yes.
  • Ok then,

    I can wipe and reimport. I should have some spare time to do it today.
  • Report ID: 1826849175

    When importing Mendeley library with Zotfile enabled some files are correctly moved to Zotfile folder while some other are copied to the metadata folder. The number of files stuck in the metadata folder is random. Now I have 436 in the metadata folder and 542 in the Zotfile folder for example.

    Steps to reproduce:

    1. Happens everytime when importing the library from Mendeley with Zotfile enabled.
    2. Always tried with a wiped up Zotero library.

    Steps to "fix" with a bruteforce "move and rename":

    1. Select all the entries of the library
    2. Right click
    3. Manage attachments
    4. Rename attachments
    5. Repeat all the above steps until no more files are stuck in the metadata folder and everything is moved in the Zotfile folder.


Sign In or Register to comment.