Upgrade StorageLost a bunch of PDFs. Safe to combine old & new Zotero Storage folders to reattach?
Hi, a bunch (maybe half) of my PDFs are no longer attached in Zotero, and although I know where at least some versions of them are, I don't know the best way to get them all back in the same place. My first instinct is to copy all of the subfolders in the old Zotero>Storage folder (which is apparently not the one linked to Zotero at the moment) and paste them into the current Storage folder. I assume/hope that this would pull everything (including any notes, metadata, etc.) while still keeping any items I've subsequently added. However, not being an expert in file syncing/Zotero, I'm unsure if this is a dangerous move (and/or how it might be affected by sync).
As background, I purchased a storage plan last May specifically so that I could more easily transfer my library when needed, which ended up being important because I had a hard drive failure within months of transferring everything to a new computer. I'd followed the instructions for moving to a new computer; option A is simply to set up Zotero sync, and there were no warnings that this wouldn't allow for a complete transfer, so that's what I did. I have the 2G plan and am nowhere near capacity, so that's not the problem. It does appear that only some of my library is synced right now (~ the same MBs I'm seeing in the "current" Zotero storage folder), but I don't know which happened first - if syncing somehow removed half of my files along the way w/o me knowing, or if only half of my files were synced in the first place. The latter seems highly unlikely because I think I'd have checked that everything synced, and I *definitely* was not having issues accessing these specific files during the first few months after purchasing the storage plan (I was very actively working on my paper and accessing tons of refs at the time, through August). So, if what's showing up now as synced is all that's ever been synced, then there was a difference between what I had stored on my computer and what synced. (If you can see my sync history and confirm that a bunch of stuff disappeared somehow, I'd appreciate it, but I guess that's less of a concern right now than just getting the missing half of my files re-linked in Zotero.)
The old version of my Zotero storage (the full "Zotero" folder which I manually copied over from an external hard drive backup) - is over twice the size of the current storage folder, so it's definitely the comparatively more complete one. However, I don't want to fully replace my active Storage folder and SQLITE file with the old ones because the old version is missing things I've added in the last couple months, before I realized anything was missing. I don't think I added a ton, but still...(esp. because one of them I had to add every piece of metadata manually.) I also might have removed some duplicates at some point, although I can't really recall when exactly I did that cleanup, as it's been piecemeal (& is nowhere near done).
So is it safe for me to simply copy over the subfolders in Storage from the old to the new Storage folder, skip the prompt to "replace" any of the files, and just hope that everything, including all of the metadata and notes, if applicable, copies over successfully without overwriting new refs? I'm not sure if any of the references themselves or notes are actually missing, as opposed to just the PDFs (and it might be a while before I find out), but I suspect it's only the PDFs because the number of folders in the old Storage fodler is similar to (in fact, ~30 folders less than) the current Storage, despite being twice as many MB in size. Thanks in advance for the help.
As background, I purchased a storage plan last May specifically so that I could more easily transfer my library when needed, which ended up being important because I had a hard drive failure within months of transferring everything to a new computer. I'd followed the instructions for moving to a new computer; option A is simply to set up Zotero sync, and there were no warnings that this wouldn't allow for a complete transfer, so that's what I did. I have the 2G plan and am nowhere near capacity, so that's not the problem. It does appear that only some of my library is synced right now (~ the same MBs I'm seeing in the "current" Zotero storage folder), but I don't know which happened first - if syncing somehow removed half of my files along the way w/o me knowing, or if only half of my files were synced in the first place. The latter seems highly unlikely because I think I'd have checked that everything synced, and I *definitely* was not having issues accessing these specific files during the first few months after purchasing the storage plan (I was very actively working on my paper and accessing tons of refs at the time, through August). So, if what's showing up now as synced is all that's ever been synced, then there was a difference between what I had stored on my computer and what synced. (If you can see my sync history and confirm that a bunch of stuff disappeared somehow, I'd appreciate it, but I guess that's less of a concern right now than just getting the missing half of my files re-linked in Zotero.)
The old version of my Zotero storage (the full "Zotero" folder which I manually copied over from an external hard drive backup) - is over twice the size of the current storage folder, so it's definitely the comparatively more complete one. However, I don't want to fully replace my active Storage folder and SQLITE file with the old ones because the old version is missing things I've added in the last couple months, before I realized anything was missing. I don't think I added a ton, but still...(esp. because one of them I had to add every piece of metadata manually.) I also might have removed some duplicates at some point, although I can't really recall when exactly I did that cleanup, as it's been piecemeal (& is nowhere near done).
So is it safe for me to simply copy over the subfolders in Storage from the old to the new Storage folder, skip the prompt to "replace" any of the files, and just hope that everything, including all of the metadata and notes, if applicable, copies over successfully without overwriting new refs? I'm not sure if any of the references themselves or notes are actually missing, as opposed to just the PDFs (and it might be a while before I find out), but I suspect it's only the PDFs because the number of folders in the old Storage fodler is similar to (in fact, ~30 folders less than) the current Storage, despite being twice as many MB in size. Thanks in advance for the help.
All files added since you upgraded your storage were uploaded, but that would've had to have been on a different computer that never had these files, since old files would've immediately uploaded along with any new files once you had the storage space.
These are the 35 missing files:
3J7ZMREM4R58DLFS
7PQM3CZY
7W8JLLFE
97J2NH5G
97Z5AWS4
ACZFJ5LW
C3LDJN2B
C6GCGIGC
EYCKBTU9
FTSXDH99
GD7VYASK
GZ439W4J
IXS7C6XA
J67L8SZQ
JHC9DTQ2
KQZCGZI4
KV6ZNJVC
L4QGYP5W
LAUNIRX9
LYBI56MF
NXFJF3N7
PYBHPQVW
QFBRF6ME
QPK836EM
RYCNPJBI
T4TTSDEB
TM4JEVI4
TMDJZMBK
UJ6VPIFA
UWTIXP2D
VHBNFL53
WYZPPHBG
YJKYJR3J
YT25CWKM
If you have these folders, I'd recommend just copying these instead of overwriting anything.
I appreciate the response. I did try copying over those folders, although some of them didn't have the PDFs in them, and I'm still running into missing PDFs after having done that. I still don't understand how this happened because I definitely never turned off file syncing. But I've had a lot of computer problems and a couple of new OS installations/new hard drives over the last year, so it might just have to be chalked up to that. I just worry that I'm not reinstalling /copying over Zotero properly, which is why I upgraded my storage in the first place - to ensure it would go smoothly. :( And now even though I'm still dealing with missing files on my current installation, I'm about to have to go through it all AGAIN because my laptop started freaking out on me, and I'm told all I can do is a fresh Windows reinstall, which means losing all of my programs again. (I'm told my data is safe, but I'm guessing they're referring to things like my Desktop, not data within Program Files.) Any guidance on the smoothest way to do this? I followed instructions on the website on how to do it the last couple of times and clearly still managed to screw something up, so any explicit guidance/help would be greatly appreciated. Thank you.
To fix the missing PDFs issue in the desktop app/storage subfolders, I used a Python script to see how many files were affected (hundreds) and then adapted it to copy/paste the relevant files from my older backup Zotero storage folder to the matching subfolder in the active Zotero storage. With the script, I was able to tell it to ignore any subfolders that already had a PDF file (and also to not copy over a PDF if the md5 checksums matched ANY others overall in the active zotero storage folder). Not sure how well the latter part worked, but I seem to have my several hundred missing PDFs reattached in Zotero now. (Previously, they were all showing as still attached but could not be found once I tried to open them & were clearly missing from the folders when using "Locate."
To clarify, I know that I synced all my files including attachments on my computer after purchasing the 2 GB storage plan, and I can even still see the original PDF attachments on the web version, but they are grayed out (similarly, I could see them in the desktop app but just couldn't open them). And I was successfully opening those PDF attachments before this mess happened, so they did somehow get unlinked despite my carefully following the instructions for syncing Zotero to a new computer.
Again, my newly restored PDF attachments on my computer are still not syncing in Zotero web, so I'll need to figure out how to address that.
rsynccommand to copy any files from the source folder that didn't exist in the target folder. There was no need to script this. This part wasn't correct. If the files existed in other folders, those were different attachment items (e.g., in different libraries). I know you still think that the files were somehow unlinked and given new keys here, but I promise you that's not a thing that can happen. The files just weren't uploaded from the computer where they were added. No, this is a misunderstanding. You did not sync them — if you did, they would've still existed online, any other Zotero client would've been able to download them automatically when you tried to open them, and you never would've gotten the file-not-found error. Attachment items always show up online, but that has nothing to do with whether the files themselves are available.The only exception to this — the only way for synced files online to disappear — is if you manually ran "Purge Storage in My Library" from your storage settings page, but I assume that's not something you did. See Files Not Syncing, and if you're still having trouble provide the requested info.
What I want to focus on now is preventing this from happening again (if needed); i,e, to get files synced that still aren't syncing. I at least have most of my PDFs back in my local zotero library due to my prior step of copying things over. Per the Files Not Syncing help page, I have file syncing set up (as always) but files are still not all being synced, which takes me to step 5 on that page (i.e., on a computer, not getting sync errors, and do have file syncing activated as always, but only some of my files appear to be actually attached in my web library, now that I know what to look for to verify this). The suggestion at that point is a sync reset, but it sounds like that is usually unnecessary, so, if there is any safer diagnostic to try first, please let me know. (Perhaps the sync reset is perfectly safe, and at this point I've already lost a lot of highlighting/notes I previously had when originally transferring from Mendeley that it probably doesn't matter so long as the files upload. But anything called "reset" always makes me pause.)
This is less important, but I also have a bunch of PDFs already open in the Zotero web application right now - would the "reset" cause those to all close? Again, I am guessing not, but just wanted to be sure! I do really want to get these PDFs synced fully, finally, so the latter issue would be more just an inconvenience as opposed to the former. I just don't want to screw anything up, and there seem to be some cautions associated with the instructions for it. Thank you.