Lost entire library with sync

Chiffon73

Hello,

I have used my zotero offline for years. I apparently had an account that I created years ago, but synchronization was disabled.
Today I needed to share a file with a colleague and as I understood, I needed to connect using sync. I forgot that I had an account, and created a new one. When logging to this new account, Zotero told me that everything related to the former account would be deleted from my laptop. Since, in my mind, I had never been connected - and probably because I did not understand what that meant- I accepted.
As you can imagine, I have lost about all my library, years of references. The zotero file is brand new, I can't restore an older version of the sqlite file.
Using a recovery program, I managed to find a 4-day old sqlite.1.bak file. I followed this procedure: https://forums.zotero.org/discussion/3134/zotero-sqlite-bak (basically, renaming my sqlite.bak as sqlite and putting it into the zotero file) but zotero can't open and the sqlite file is now "sqlite.damaged". I have tried to fix it using the DB repair tool, but it tells me "this is not a sqlite file".

Do you know if there is anything else I can try?
A huge thank in advance, I am desperate!

dstillman

I have tried to fix it using the DB repair tool, but it tells me "this is not a sqlite file".

Was it named zotero.sqlite when you tried?

If you can't fix it with the repair tool, you don't have a backup of your Zotero data directory, and you can't find any other copies of zotero.sqlite, I'm afraid there's nothing to be done.

For what it's worth, this was the dialog:

This Zotero database was last synced with a different account (‘[old username]’) from the current one (‘[new username]’). If you continue, data associated with the ‘[old username]’ account will be removed from this computer.

You would've then had to check a "Remove existing data" checkbox to proceed, at which point it would indeed have deleted your existing local data.

You're not the first to do this by mistake, and we can try to make this message more dire-sounding or maybe add some sort of backup option, but there's only so much we can do. People switch accounts on shared computers where the old data needs to be deleted.

Chiffon73

Dear dstillman,

thank you for your answer.

"Was it named zotero.sqlite when you tried?"
Yes, I edited the name and extension of the bak file myself.

"You would've then had to check a "Remove existing data" checkbox to proceed, at which point it would indeed have deleted your existing local data."
Yes, exactly, I checked this box to proceed -I am not sure that I can proceed without checking the box-. It is true that I did not understand that all my library would be deleted. I thought that only data about the account (name, password etc) would be removed. Indeed, I think that a backup or undo button would be great. Also, yes please, make this message (way) more explicit: this is a huge step, and it should be clear that everything from Zotero (files, structure, pdfs, annotations, references) will be lost and that there is not stepping back...

Is there any directory were these deleted files are stored until, say, the next time the laptop turns on?

Thank you again for your answer,

dstillman

Is there any directory were these deleted files are stored until, say, the next time the laptop turns on?

No.

Just to be clear, you don’t have a backup of your computer from any point?

qqbb

@dstillman: I don't know how the function is currently working. But if it's not already the case, maybe moving the data and files to the operating system's trash by default would be an improvement, in addition to possible other changes discussed above.

@Chiffon73: Did you also try to restore your Zotero Data Directory using your recovery program? Maybe you could at least get back some files from the storage subdirectory.

Chiffon73

Hi all,


@dstillman I have one backup but can't access it during the covid lockdown. So not everything is lost, but I will have to wait several months to recover an old version. There is still hope!

@qqbb when I use the recovery programm, I can access a large number of zotero-info and zotero-cache items. Can it be of any use?

Thanks!

qqbb

@Chiffon73: Those files are probably not very useful, they are related to Zotero's PDF full-text indexing. Note that each stored file is saved to an individual folder: /Zotero/Data/storage/[itemKey]/[filename]. So it would be best if you could try to recover a full directory. If that's not the case, search for .pdf files.

Here are some tutorials that might be helpful for data recovery. I don't know some of those programs, part of the information could be outdated. But the general procedure should hold:
https://superuser.com/a/790566
https://www.redhat.com/sysadmin/find-lost-files-scalpel
https://www.pendrivelinux.com/yumi-multiboot-usb-creator/

Chiffon73

Hi @qqbb,

thanks again for your message. I haven't been able to recover the full folder with my software. It looks like it has vanished!
I had a look at the links you sent, they are for Linus and I use a Windows laptop, but they mostly recommand to use a software similar to the one I have just used. That is too bad...

I hope that the zotero team will find a way to prevent this kind of damage in the future, this is a silly mistake by the user (just pressing one button) which has very unpleasant consequences...

Thank you for your help anyway!

qqbb

You could try to recover pdf's with PhotoRec, which should also work in Windows. Save the recovery program and restored files to an external hard drive. If possible, boot from a CD or USB device, e.g., using a Live USB system.

Some backup function that can be launched from within Zotero should be helpful, especially for less technically minded users. Moving the data directory to the operating system's trash is probably not a good option. It seems there're generally no system tools to do that.

Chiffon73

@qqbb many thanks, this works!! I have been able to retreive most of my pdfs! This is a very long and tedious process since it recovers literally every. single. pdf. that has been on my laptop since I have it, but still better than having to retrieve all articles one by one from the internet.
Best answer so far!

dstillman

Just to note, there's a chance some of the recovered files may be corrupted — when files are deleted, the system basically just considers the space as free, so any other write activity runs the risk of overwriting the files or parts of the files. But if the disk is big enough and usage was limited since the deletions, hopefully most of the files will be fine.

I thought that only data about the account (name, password etc) would be removed.

Thanks for that explanation — I can see how you could interpret it that way.

For what it's worth, in most cases switching accounts wouldn’t result in data loss, because the whole point is that it appears when you already had a Zotero account that you had been syncing this library with previously. In your case, that syncing just hadn't happened in a long time.

We'll clarify the dialog — and possibly check for unsynced data — for the next version.

qqbb

@Chiffon73: I'm curious whether you were able to recover files from a HDD or SSD. The latter can be more problematic for data recovery, see here.

If PhotoRec worked well for PDF files, you could also try to recover SQLITE files with it. It's in the list of supported file formats. Maybe the zotero.sqlite database can be recovered as well.

While PhotoRec is a program for recovering individual files, TestDisk might be able to recover a directory.

Chiffon73

@dstillman and @qqbb: The photorec solution is not perfect indeed. I realized that it only finds pdfs that were registered at some point on my laptop before being uploaded onto zotero. The pdfs that I directly uploaded from internet to zotero (with the mozilla extension for example) can't be found by photorec or are damaged. So, I have access to part of my library, but not all.

@qqbb: This is a great software, but it does not find older versions of the zotero file. As @dstillman suggested, these files were overwritten when sync the account, and it can't find older versions of the zotero files (all the ones that are found are 2 days old max).

dstillman

The pdfs that I directly uploaded from internet to zotero (with the mozilla extension for example) can't be found by photorec or are damaged.

Just to clarify, files saved via the Zotero Connector exist on your laptop the same as any others. Unless you don't have Zotero open, they're not "uploaded" — Zotero saves the PDFs to its data directory. But there certainly may be differences in recoverability of files in different directories on your computer.

Chiffon73

@dstillman: exactly, I think that everything that was directly saved into the zotero file (i.e., that was not saved somewhere else before) can't be found by the recovery software, probably because it was not deleted per se, it was overwritten (I think).

qqbb

@Chiffon73: If your laptop uses an SSD, it's possible that your files in Zotero's storage folder are not recoverable due to TRIM commands. You can find a very detailed explanation here: https://superuser.com/a/407659. That's probably more than you want to know. So just remember to keep backing up your Zotero data.

If accidential file deletion should happen to anyone on Windows on an SSD system, I would recommend to follow https://superuser.com/a/709755 to disable TRIM immediately. It's possible this would already be too late, but if the TRIM command is not issued right away, it could increase the chances that data can be recovered. According to Microsoft's fsutil documentation here, enabling or disabling DisableDeleteNotify does not require a restart.