Moving the linked files folders to a different drive

Hi, I use Zotfile. I moved the folder where I have my PDFs (it is called "Zotfile"), which are only linked in Juris-M (a Zotero fork), from D:/ to C:/.

I changed the linked attachment base directory in the preferences.

I also changed the Zotfile directory in the Zotfile preferences.

I suspect that maybe my Juris-M didn't treat the directory as a relative path, but as an absolute path.

Yet, Juris-M can't find the linked PDFs when I double-click them.

It would take a long time to manually localize every linked PDF in Juris-M.

Any ideas how I can redirect the links? Should I use system links in Windows? BTW, as of now I don't have a fixed drive that I assign to D:/ all the time.
  • This certainly should have worked using the base directory, but since it didn't, you can batch-move links using the Zutilo add-on:
    https://github.com/willsALMANJ/Zutilo
  • Thanks!

    I replaced the links using Zutilo.

    I replaced "D:\username\Dokumente\ZotFile" with "C:\Users\username\Documents\ZotFile".

    Strangely, the link targets now usually lack the first letter of the filename. I don't know if this was the case already before executing the Zutilo action.

    Example:

    The Juris-M item is titled: 2012 - »There is no rule of law in China – not yet« – 4th.pdf

    The link target in Juris-M is named: 012 - »There is no rule of law in China – not yet« – 4th.pdf

    The real file in the Zotfile folder is titled: 2012 - »There is no rule of law in China – not yet« – 4th.pdf
  • > Strangely, the link targets now usually lack the first letter of the filename.

    Does someone have an idea what I could do?
  • I'd guess that it's something to do with the backslash and how Zutilo handles that (or doesn't handle that, particularly if it wasn't developed on Windows). A backslash is an escape character in JavaScript, but only Windows uses backslashes in paths.

    Unless that data — meaning the first letter — exists somewhere, there wouldn't be any sort of automated way of recovering those beyond relinking the attachments from the file-not-found dialog.
  • Thanks for your answer. It's a pity… I will do it through said dialog.

    I want to tell every other user to whom this happens: Consider DocFetcher as an alternative tool for indexing and searching.
  • @moebio: I couldn't reproduce your error with the missing first character. Were you working with files in the Linked Attachment Base Directory? Do you remember your old and new partial paths?

    It might be possible to grab the first characters or the full filenames from your attachment titles. Then it could be possible to adapt Zutilo's code (here) and use it with Zotero's JavaScript API, but this could be quite involved.

    There could be an easy partial solution for filenames that are starting with a year. You could, e.g., modify the partial paths from "C:\test1\0" to "C:\test1\20".

    There is also this add-on that might help with removing broken attachments:
    https://github.com/retorquere/zotero-storage-scanner

    I'll provide some general advice on the Zutilo function below.
  • Zutilo's "Modify attachment paths" function is very flexible, so it should be used carefully. Before using this function, remember to back up your Zotero data. Here, your main concern should be the zotero.sqlite database file. (Zotero also keeps automatic backups of recent versions the zotero.sqlite database that might help in some cases.) Note that this Zutilo function is not touching paths on your filesystem. It's only changing the pointers to linked files within Zotero's database.

    Normally, you wouldn't need to use Zutilo's "Modify attachment paths" function. Working with Zotero's Linked Attachment Base Directory setting should generally be your preferred option.

    If you decide to use Zutilo's "Modify attachment paths" function:

    1. Test the function on a small number of attachments first. In case you experience a problem, this allows you to manually correct the paths. If everything works as intended, you can modify all linked files.

    2. Zutilo's "Modify attachment paths" function should always be used in combination with its "Show attachment paths" function. It will replace one character sequence in the output of the "Show attachment paths" function with another character sequence. By default, it will only replace the beginning of attachment paths. If you click the "replace all instances" check box, it can also replace something in the middle of the paths, e.g., replacing \ with /.

    3. Zutilo's "Show attachment paths" function shows you a relative path of a linked file in the Linked Attachment Base Directory as "attachments:references/coolPaper.pdf". For a linked file that is not stored with a relative path, Zutilo shows its absolute path, e.g., "C:\userData\references\coolPaper.pdf". A file in Zotero's storage is shown as "storage:coolPaper.pdf". Zutilo won't touch stored files.
  • Here are some tests on Windows for using the "Modify attachment paths" function on linked files stored with absolute paths. You can see that you can mess up your paths. But it's working as expected, given the flexibility of the tool.

    modify partial paths:
    old C:\test1\
    new C:\test2\
    filename change:
    old C:\test1\123.pdf
    new C:\test2\123.pdf

    modify partial paths:
    old C:\test1
    new C:\test2
    filename change:
    old C:\test1\123.pdf
    new C:\test2\123.pdf

    modify partial paths:
    old C:\test1\
    new C:\test2
    filename change:
    old C:\test1\123.pdf
    new C:\test2123.pdf

    modify partial paths:
    old C:\test1
    new C:\test2\
    filename change:
    old C:\test1\123.pdf
    new C:\test2\\123.pdf
  • edited August 28, 2020
    There's an earlier discussion about the same issue. I'm wondering whether the use of the substr function could be problematic (here) in combination with backslashes, but I'm not sure.

    @moebio: Could you provide as much information as possible about how this happened? If there is an issue with the Zutilo code, this might help finding it. Did it happen on a Windows-only setup?
  • Sorry, I tried to describe what I did in as much detail as possible in the entry post. I made the post not immediately after making the changes, but some hours or days later. I hope that I can answer with more information if specific questions come up.

    About the setup, yes, I have been using Windows only.
  • Here are some more specific questions.

    Information on the issue:

    1. Can you provide the Juris-M and Zutilo versions that you were using?

    2. Do you remember whether you clicked the "replace all instances" check box?

    3. You wrote that the links "usually lack the first letter". Does this mean that not all attachments were affected in the same way? If some attachments were fine, could this be related to a difference in the first character of the filename?

    4. Are all your linked files saved to Zotfile's "Custom Location" without subfolders? (Did you leave "Use subfolder defined by" unchecked?)

    5. It would be great if you could try to reproduce the issue on a test item. First, create a new linked attachment. Select this attachment and use Zutilo's "Show attachment paths" and "Modify attachment paths" functions to set the path to the original path of an attachment that ended up with a missing character. If this works, see if you can reproduce the issue.

    Fixing your problem:

    6. Did you check whether you have a backup as described above?

    7. If you haven't fixed your paths yet, I could try to write a script that uses the attachment titles for fixing the filenames.
  • Thank you for your well-structured aid! I must admit that I didn't fully understand Zotero, Zotfile and Zotilo

    1. I'd been using Juris-M 5.0.85m1 (Portable, from https://github.com/pedrom34/JurisMPortable/releases/tag/dev01)
    2. I am afraid I don't remember.
    3. There are some that have functioning attachment links. I think (only and all of) the ones that are _not_ *linked*, but "stored" (i.e. copied) were not affected.
    Furthermore, I clarify that when I write "first letter" this includes numbers. It's the same for files that start with a number and files that start with A-z.
    4. I'm 90% sure that I left it unchecked.
    5. I have linked a PDF from my downloads folder. Then, I created a parent item for it. Zutilo now shows the attachment path "C:\Users\myusername\Downloads\example.pdf". Next, I used the "Modify attachment paths" function, replacing the path component "C:\Users\myusername\Downloads" with "D:\Zotfile". I have reiterated this for the other three combinations that you already went through in your comment 27 Aug. The Filename property in Juris-M does *not* lack the first letter in my tests.
    6. I wasn't able to restore backups from before the mess-up.
    7. I would surely be thankful. There is no important need from my side, because I have found a kind of solution. First of all, I now use DocFetcher to search my files – thus, I don't need to access the files from Juris-M in most cases. And whenever I do need to, I use Zutilo to bulk-change the attachment paths. This is easy, because I just need to add one character. Eventually, I will have gradually corrected the attachment paths.

    PS: Perhaps it's helpful to know that currently my base directory is set to "C:\Users\myusername\Documents\ZotFile".
  • edited December 24, 2020
    I found the tool [ZotFile Doctor](https://github.com/giorginolab/zotfile_doctor).

    It enabled me to create a list of the files that are in the DB but not in the Zotfile directory (401 out of 648).

    Now I'm trying to find a way to edit the sqlite database efficiently.
  • Now I'm trying to find a way to edit the sqlite database efficiently.
    See dstillman's comment here:
    You absolutely shouldn't modify the database directly, though. If you're going to read the database directly, you'd want to just rename the files based on the DB, not the other way around.
  • edited December 24, 2020
    Oh… well, I did modify it directly using "DB Browser". Before opening it in the editor, I closed Zotero/Juris-M. The main reason why I did this because it is advised when using ZotFile Doctor (and therefore I thought it is a good idea when editing the SQlite). Note that I use Juris-M for the PortableApps.com platform (https://github.com/pedrom34/JurisMPortable).

    Does this entail an unexpected danger? Does the risk come from the users (in-)competency?

    I spent a couple of hours adjusting the database. The easiest fixes were adding a "9" to the beginning of the filenames that were named after their respective ISBN.

    Now Zotfile Doctor says:

    > There were 392/1122 files in DB but not in zotfile directory

    In Zotero/Juris-M, I can open the fixed attachments again.
  • edited December 24, 2020
    https://www.zotero.org/support/dev/client_coding/direct_sqlite_database_access

    I see what makes it dangerous. So far I am not noticed indications that the DB is corrupted.
  • I'm going to use Zutilo to fix remaining paths. It's not much more cumbersome.
  • I wrote a script that might help with fixing some of your filenames. Thanks for your tests above. There's another test you could do in order to see whether you can reproduce the original issue. But that can also wait until you've fixed your filenames.

    Do you remember what version of Zutilo you were using when the issue occurred? Did you update Zutilo in the meantime?

    Another tool that should be helpful is this add-on:
    https://github.com/retorquere/zotero-storage-scanner
  • edited May 28, 2021
    This script for Zotero's JavaScript API tries to fix filenames of selected linked attachments using the attachment titles. For the new path, it takes the old directory and the attachment title as the new filename. It checks for the existence of files. The filenames will only be updated if the files exist at the new paths. Before using the script, make sure that the directories are already correct. You can adjust them with Zutilo, as you know. Selecting parent items etc. is ok.

    Close Zotero and make a backup of the 'zotero.sqlite' file in your Zotero data directory. Then reopen Zotero, go to "Tools" -> "Developer" -> "Run JavaScript", and paste in and run this code:

    // https://regex101.com/r/ZdzzFC/1
    const re = /^(.*?)([^\/\\]+)$/;
    const mode = Zotero.Attachments.LINK_MODE_LINKED_FILE;
    var attArray = ZutiloChrome.zoteroOverlay.getSelectedAttachments(mode);
    var attItem = new Zotero.Item("attachment");
    attItem.attachmentLinkMode = mode;
    var m, n, attTitle, oldFullPath, newFullPath;
    m = n = 0;
    attTitle = oldFullPath = newFullPath = '';

    for (let i = 0; i < attArray.length; i++) {
    if (await attArray[i].fileExists()) continue
    m++;
    oldFullPath = attArray[i].attachmentPath;
    attTitle = attArray[i].getField('title');
    newFullPath = '' + oldFullPath.replace(re, '$1') + attTitle;
    if (newFullPath === oldFullPath) continue
    attItem.attachmentPath = newFullPath;
    if (!await attItem.fileExists()) continue
    attArray[i].attachmentPath = newFullPath;
    await attArray[i].saveTx();
    n++;
    }

    return `Summary of selected linked attachment items:\n
    ${attArray.length} item(s) selected
    ${m} item(s) broken
    ${n} item(s) fixed\n

    Values for last broken item:\n
    attachment title:\n${attTitle}\n
    old full path:\n${oldFullPath}\n
    new full path:\n${newFullPath}
    `;


    It should help to use this together with the storage scanner. (I think selecting "Tools" -> "Storage Scanner" will update the '#broken_attachments' tags.)

    Edited (December 26, 2020): The code has been corrected, taking into account dstillman's comments below.

    Edited (May 28, 2021): The script now returns more detailed text.
  • @qqbb:

    1) The second argument to getField() should be a boolean.

    2) You shouldn't modify the item (attachmentPath) without knowing if you're going to save it — if you just skip saving, you've created an inconsistency between the internal state of the item and the database.
  • @dstillman: Thanks for reviewing the code!

    1) I guess the second and third arguments can be omitted here.

    2) I had thought about this issue, but I was hoping the inconsistencies would be gone after restarting Zotero. My initial draft was creating a dummy attachment variable for 'fileExists()'. I then used the more compact version as it was working as intended and I wasn't sure whether creating a temporary attachment item was a good thing. I've now edited the code above, using my original approach. I hope the code is ok now.
Sign In or Register to comment.