Upgrade StorageMoving the linked files folders to a different drive
Hi, I use Zotfile. I moved the folder where I have my PDFs (it is called "Zotfile"), which are only linked in Juris-M (a Zotero fork), from D:/ to C:/.
I changed the linked attachment base directory in the preferences.
I also changed the Zotfile directory in the Zotfile preferences.
I suspect that maybe my Juris-M didn't treat the directory as a relative path, but as an absolute path.
Yet, Juris-M can't find the linked PDFs when I double-click them.
It would take a long time to manually localize every linked PDF in Juris-M.
Any ideas how I can redirect the links? Should I use system links in Windows? BTW, as of now I don't have a fixed drive that I assign to D:/ all the time.
I changed the linked attachment base directory in the preferences.
I also changed the Zotfile directory in the Zotfile preferences.
I suspect that maybe my Juris-M didn't treat the directory as a relative path, but as an absolute path.
Yet, Juris-M can't find the linked PDFs when I double-click them.
It would take a long time to manually localize every linked PDF in Juris-M.
Any ideas how I can redirect the links? Should I use system links in Windows? BTW, as of now I don't have a fixed drive that I assign to D:/ all the time.
https://github.com/willsALMANJ/Zutilo
I replaced the links using Zutilo.
I replaced "D:\username\Dokumente\ZotFile" with "C:\Users\username\Documents\ZotFile".
Strangely, the link targets now usually lack the first letter of the filename. I don't know if this was the case already before executing the Zutilo action.
Example:
The Juris-M item is titled: 2012 - »There is no rule of law in China – not yet« – 4th.pdf
The link target in Juris-M is named: 012 - »There is no rule of law in China – not yet« – 4th.pdf
The real file in the Zotfile folder is titled: 2012 - »There is no rule of law in China – not yet« – 4th.pdf
Does someone have an idea what I could do?
Unless that data — meaning the first letter — exists somewhere, there wouldn't be any sort of automated way of recovering those beyond relinking the attachments from the file-not-found dialog.
I want to tell every other user to whom this happens: Consider DocFetcher as an alternative tool for indexing and searching.
It might be possible to grab the first characters or the full filenames from your attachment titles. Then it could be possible to adapt Zutilo's code (here) and use it with Zotero's JavaScript API, but this could be quite involved.
There could be an easy partial solution for filenames that are starting with a year. You could, e.g., modify the partial paths from "C:\test1\0" to "C:\test1\20".
There is also this add-on that might help with removing broken attachments:
https://github.com/retorquere/zotero-storage-scanner
I'll provide some general advice on the Zutilo function below.
Normally, you wouldn't need to use Zutilo's "Modify attachment paths" function. Working with Zotero's Linked Attachment Base Directory setting should generally be your preferred option.
If you decide to use Zutilo's "Modify attachment paths" function:
1. Test the function on a small number of attachments first. In case you experience a problem, this allows you to manually correct the paths. If everything works as intended, you can modify all linked files.
2. Zutilo's "Modify attachment paths" function should always be used in combination with its "Show attachment paths" function. It will replace one character sequence in the output of the "Show attachment paths" function with another character sequence. By default, it will only replace the beginning of attachment paths. If you click the "replace all instances" check box, it can also replace something in the middle of the paths, e.g., replacing
\with/.3. Zutilo's "Show attachment paths" function shows you a relative path of a linked file in the Linked Attachment Base Directory as "attachments:references/coolPaper.pdf". For a linked file that is not stored with a relative path, Zutilo shows its absolute path, e.g., "C:\userData\references\coolPaper.pdf". A file in Zotero's storage is shown as "storage:coolPaper.pdf". Zutilo won't touch stored files.
modify partial paths:
old C:\test1\
new C:\test2\
filename change:
old C:\test1\123.pdf
new C:\test2\123.pdf
modify partial paths:
old C:\test1
new C:\test2
filename change:
old C:\test1\123.pdf
new C:\test2\123.pdf
modify partial paths:
old C:\test1\
new C:\test2
filename change:
old C:\test1\123.pdf
new C:\test2123.pdf
modify partial paths:
old C:\test1
new C:\test2\
filename change:
old C:\test1\123.pdf
new C:\test2\\123.pdf
@moebio: Could you provide as much information as possible about how this happened? If there is an issue with the Zutilo code, this might help finding it. Did it happen on a Windows-only setup?
About the setup, yes, I have been using Windows only.
Information on the issue:
1. Can you provide the Juris-M and Zutilo versions that you were using?
2. Do you remember whether you clicked the "replace all instances" check box?
3. You wrote that the links "usually lack the first letter". Does this mean that not all attachments were affected in the same way? If some attachments were fine, could this be related to a difference in the first character of the filename?
4. Are all your linked files saved to Zotfile's "Custom Location" without subfolders? (Did you leave "Use subfolder defined by" unchecked?)
5. It would be great if you could try to reproduce the issue on a test item. First, create a new linked attachment. Select this attachment and use Zutilo's "Show attachment paths" and "Modify attachment paths" functions to set the path to the original path of an attachment that ended up with a missing character. If this works, see if you can reproduce the issue.
Fixing your problem:
6. Did you check whether you have a backup as described above?
7. If you haven't fixed your paths yet, I could try to write a script that uses the attachment titles for fixing the filenames.
1. I'd been using Juris-M 5.0.85m1 (Portable, from https://github.com/pedrom34/JurisMPortable/releases/tag/dev01)
2. I am afraid I don't remember.
3. There are some that have functioning attachment links. I think (only and all of) the ones that are _not_ *linked*, but "stored" (i.e. copied) were not affected.
Furthermore, I clarify that when I write "first letter" this includes numbers. It's the same for files that start with a number and files that start with A-z.
4. I'm 90% sure that I left it unchecked.
5. I have linked a PDF from my downloads folder. Then, I created a parent item for it. Zutilo now shows the attachment path "C:\Users\myusername\Downloads\example.pdf". Next, I used the "Modify attachment paths" function, replacing the path component "C:\Users\myusername\Downloads" with "D:\Zotfile". I have reiterated this for the other three combinations that you already went through in your comment 27 Aug. The Filename property in Juris-M does *not* lack the first letter in my tests.
6. I wasn't able to restore backups from before the mess-up.
7. I would surely be thankful. There is no important need from my side, because I have found a kind of solution. First of all, I now use DocFetcher to search my files – thus, I don't need to access the files from Juris-M in most cases. And whenever I do need to, I use Zutilo to bulk-change the attachment paths. This is easy, because I just need to add one character. Eventually, I will have gradually corrected the attachment paths.
PS: Perhaps it's helpful to know that currently my base directory is set to "C:\Users\myusername\Documents\ZotFile".
It enabled me to create a list of the files that are in the DB but not in the Zotfile directory (401 out of 648).
Now I'm trying to find a way to edit the sqlite database efficiently.
Does this entail an unexpected danger? Does the risk come from the users (in-)competency?
I spent a couple of hours adjusting the database. The easiest fixes were adding a "9" to the beginning of the filenames that were named after their respective ISBN.
Now Zotfile Doctor says:
> There were 392/1122 files in DB but not in zotfile directory
In Zotero/Juris-M, I can open the fixed attachments again.
I see what makes it dangerous. So far I am not noticed indications that the DB is corrupted.
Do you remember what version of Zutilo you were using when the issue occurred? Did you update Zutilo in the meantime?
Another tool that should be helpful is this add-on:
https://github.com/retorquere/zotero-storage-scanner
Close Zotero and make a backup of the 'zotero.sqlite' file in your Zotero data directory. Then reopen Zotero, go to "Tools" -> "Developer" -> "Run JavaScript", and paste in and run this code:
// https://regex101.com/r/ZdzzFC/1
const re = /^(.*?)([^\/\\]+)$/;
const mode = Zotero.Attachments.LINK_MODE_LINKED_FILE;
var attArray = ZutiloChrome.zoteroOverlay.getSelectedAttachments(mode);
var attItem = new Zotero.Item("attachment");
attItem.attachmentLinkMode = mode;
var m, n, attTitle, oldFullPath, newFullPath;
m = n = 0;
attTitle = oldFullPath = newFullPath = '';
for (let i = 0; i < attArray.length; i++) {
if (await attArray[i].fileExists()) continue
m++;
oldFullPath = attArray[i].attachmentPath;
attTitle = attArray[i].getField('title');
newFullPath = '' + oldFullPath.replace(re, '$1') + attTitle;
if (newFullPath === oldFullPath) continue
attItem.attachmentPath = newFullPath;
if (!await attItem.fileExists()) continue
attArray[i].attachmentPath = newFullPath;
await attArray[i].saveTx();
n++;
}
return `Summary of selected linked attachment items:\n
${attArray.length} item(s) selected
${m} item(s) broken
${n} item(s) fixed\n
Values for last broken item:\n
attachment title:\n${attTitle}\n
old full path:\n${oldFullPath}\n
new full path:\n${newFullPath}
`;
It should help to use this together with the storage scanner. (I think selecting "Tools" -> "Storage Scanner" will update the '#broken_attachments' tags.)
Edited (December 26, 2020): The code has been corrected, taking into account dstillman's comments below.
Edited (May 28, 2021): The script now returns more detailed text.
1) The second argument to
getField()should be a boolean.2) You shouldn't modify the item (
attachmentPath) without knowing if you're going to save it — if you just skip saving, you've created an inconsistency between the internal state of the item and the database.1) I guess the second and third arguments can be omitted here.
2) I had thought about this issue, but I was hoping the inconsistencies would be gone after restarting Zotero. My initial draft was creating a dummy attachment variable for 'fileExists()'. I then used the more compact version as it was working as intended and I wasn't sure whether creating a temporary attachment item was a good thing. I've now edited the code above, using my original approach. I hope the code is ok now.