Sudden loss of entire Library in Zotero Standalone

Hello,

I bought a new Windows laptop on Monday, restored my Zotero library from a backup and installed Zotero Standalone. I set up my usual custom data directory, so that my references could be regularly backed up to the cloud.

Everything worked well for three days, and I reorganised plenty of references yesterday afternoon, followed by a sync and cloud backup.

I opened Zotero Standalone this afternoon, and my library is empty. The zotero.sqlite file in my custom data directory has reduced to 1,000 KB.

Thankfully, I can still access my references by logging in to zotero.org, but I am wary of syncing, in case the empty data directory will be copied to my Zotero account on the web, losing all my references.

I look forward to your help.

Thanks and best wishes,
Elizabeth
  • are any of the sqlite.bak files larger than ~1MB?
    do you sync that folder to any other computer or just up to whichever cloud service you're using?
  • Thanks Adam.
    All sqlite.bak files are 1MB.
    I sync the folders to a cloud backup and then to a smaller portable device.
  • Something swapped out your Zotero data folder then. It almost certainly wasn't Zotero, and my guess would be the sync solution you have, but hard to say. Whenever I'm seeing cloud sync with multiple devices used on a Zotero database, anything going wrong is by far most likely explained by that. That's why we keep harping on the "Don't place your Zotero folder into Dropbox".

    You can do reset from server in the sync/Reset tab of the Zotero preferences (make sure you pick the right direction of reset) to be 100% safe, though in all likelihood you're just looking at a fresh database and regular sync would work just fine.

    If you're using a well-set-up cloud service, you may also be able to look at file version history online to do some detective work of where the database was changed and how.
  • Thanks Adam. There seem to be three options.

    1) I have some recent backups of the Zotero directory, from 6 December when my old laptop backed up to the cloud (the SQLite files are 92.88MB on this version) and an earlier version from mid-November on an external drive. If I downloaded those files or synced them to my new laptop, I could restore a recent version, and then hopefully sync with the new data on the server. Would that work?

    2) If I do a regular sync, could the empty Zotero library be synced to the server, deleting all my references?

    3) The preferences menu mentions that resetting data from the server can cause difficulties. Is that also a high risk approach?

    I don't use Dropbox, and have a number of backup strategies, but they do not seem sufficient at the moment!

    Thanks for your help.
  • Your backup strategies seem to be working just fine. You have multiple ways of getting your library back.

    1) actually seems like the easiest option here and I don't see any risks at all, so I'd go with that.

    I'd still try to figure out what happened. I realize you're not using Dropbox, but for the purpose of syncing a database, Dropbox and pretty much any file sync service are roughly functionally equivalent (most use about the same technology, some variety of rsync, under the hood). Using it to sync your Zotero database in any setup that involves more than one device and the cloud is a bad idea.
  • Thanks Adam. I will try option 1), and will modify my daily backup schedule so it includes a daily backup of my Zotero directory to an external drive instead of a cloud backup.
  • and will modify my daily backup schedule so it includes a daily backup of my Zotero directory to an external drive instead of a cloud backup.
    Yeah, that's what I do. If you want to be hardcore, you can then have that backup synced to the cloud (that's what I did during the last month of my dissertation). But with Zotero sync and external HD you already have two copies that will protect you against different types of disaster, so that seems like a very solid strategy.
  • Thank you Adam. Finally, a simple sync from the server to Zotero Standalone restored my library. I have now created a dated backup of zotero.sqlite and the storage folders elsewhere on my hard drive, backing up to an external drive at this writing.

    I will try to remove my Zotero data directory from the cloud sync, but I need advice from my cloud backup provider on how to do this. The Changelog from the backup software indicates that the 96.55MB versions of zotero.sqlite and zotero.sqlite.bak were both deleted this afternoon during a sync, so it does seem that the cloud backup/sync software caused the problem.

    Thank you again for your help.
  • This just happened to me, and I only see 80 (of hundreds of) references in my Zotero Standalone and my online profile at zotero.org. I hadn't used Zotero in a few months, and when I started it, an error message appeared. I have to admit I hadn't read it properly and clicked by mistake on "Quit Zotero". I was using the default data directory and the sqlite.bak and sqlite file are both around 1.5MB. How can I get my library back? :(
  • Did the message say that your database was corrupted? If so, check if there's a zotero.sqlite.damaged file in your Zotero data directory, and try running it through the DB Repair Tool. First make a backup of your entire data directory.

    Most likely your database got corrupted, Zotero automatically made a backup of the damaged DB and reverted to a clean database, and then Zotero synced down what was on the server, but you hadn't successfully synced in a couple months. (The last upload I see from you was at the end of October.)
  • Hi Dan, thank you! I cannot recall the message, but it did mention Firefox. I am currently not using a Firefox profile (not signed in) but don't think I was before either because I cannot sign in with any of my email addresses.

    I tried repairing the sqlite file (ID 567d4e686fae3), but nothing changed -- I only see some 80 references in my library. Is there any way you can see the content of the October upload?
    Should I try updating the Zotero Standalone?
  • was there an sqlite.damaged file that you tried to repair or was that the regular .sqlite?
    I am currently not using a Firefox profile (not signed in) but don't think I was before either because I cannot sign in with any of my email addresses.
    not sure what you mean by that. Firefox profiles are local, you don't need to sign into them (signing into Firefox is only for syncing profiles between computers).

    When you previously were using Zotero, was that on the same computer? Also with Standalone? Could the message have said something about sharing the data directory with Firefox?
  • There was no damaged file, just the regular sqlite and the sqlite.bak, both of the same size.
    Yes, I was using the Standalone on the same computer. Nothing has changed except that Firefox was probably updated a few times. Yes, the error message might have said something like this, but I unfortunately did not get to read it fully as I clicked by mistake and then it was gone :(
  • yeah, if you don't have .sqlite.damaged then what Dan suggested likely wasn't the case.

    Another possibility might be that you somehow started a new data profile. Could you check what it says in the Advanced Tab of the Zotero Preferences --> Files and Folders under "Data directory location?" Is default checked, or does it specifcy a custom location (which)?
  • The default is checked.
    Is there any way to retrieve (on your end or mine) the library which was synced in October? I am not interested in the files, as I had not uploaded many, just the hundreds of references.
  • There isn't really any way to just lose data, but it's hard to say exactly what happened without knowing what the error message was. But if you don't see any files within your current Zotero data directory older than the other day, it's likely that there's another data directory elsewhere on your computer, so you could try searching for zotero.sqlite:

    https://www.zotero.org/support/zotero_data#troubleshooting_zotero_data_issues

    It doesn't look like you've uploaded anything to the server in over two years, so you'd have to find the data locally.
  • It seems that the most probable cause of my library loss is that the local database in Zotero Standalone was overwritten with the online version of the database that dates back to 2013 and whose file size is only about 1.55MB. This database contains only a small fraction of the previous local database that was stored on my laptop.

    Unfortunately, despite the automatic sync option checked in Zotero, my data were not synced with the server, but the sync was done from the server to my laptop, overwriting everything I had locally.

    Is there any way to see the sync history or revert to an older online backup prior to Dec 24, 2015? A good feature request would be sync versioning with rollback options both locally and online.
  • I've done extensive troubleshooting that consisted of the following:
    - checked the Event Viewer for any issues regarding Zotero to see the error message from failed application start on the evening of Dec 24, but I didn't find any references to that event in the Windows Event Log. After selecting "Quit Zotero" in the error message and finally starting Zotero Standalone that evening, it loaded an old database with 80 records (where there should be hundreds more).
    - checked Zotero sync options -- they were enabled the whole time.
    - there were no .crashed files in the Zotero directory, but there is a store.json.mozlz4 file in the /crashed directory; however, the Zotero extension for Firefox was not used for months and there is a possibility that an automatic Firefox update from a few weeks ago disabled the Zotero extension.
    - there are no multiple zotero.sqlite files on my computer. The only .sqlite and .sqlite.bak were in the defined (default) data directory so the issue is not coming from fiddling with the wrong database.
    - I did not use Zotero on any other devices except for this laptop, so we can also exclude the possibility that the library was overwritten from another device.
    - the Zotero Standalone data folder was not set up to any 3rd party cloud backup such as Dropbox, GDrive, Box, etc. It was only expected to sync with Zotero's online database.
    - I backed up all of Zotero Standalone's profile files and the database was made before recovery attempts. I tried to restore the zotero.sqlite database and the zotero.sqlite.bak file, after the crash, but it seems that both files are actually the same, synced from Zotero's server.
    - I also used my friend's computer (Zotero MacOSX) to try to restore the .sqlite and .sqlite.bak databases from the date of crash -- without sync options enabled -- and it seems that the database corresponds to what I have on my laptop and what is on Zotero's server, leading to the conclusion that my laptop synced with the server and downloaded what was in the library while overwriting my local data.

    Literally years of research just vanished after an unwanted sync push. So, it appears that uploading from Standalone to Zotero online works (so we can rule out that a firewall or antivirus rule might block it the upload), however, the question is, with the sync enabled the whole time, why the local copy didn't update all the changes to the server's online database in over two years?
  • there are no multiple zotero.sqlite files on my computer
    Did you try searching your entire local hard drive(s) for just "zotero", to see if there is more than one "zotero" directory on your computer?

    Also, could you verify that both Zotero for Firefox and Zotero Standalone share the same data directory? (follow Dan's link above for instructions; for Zotero for Firefox, make sure Zotero Standalone is closed, then start Firefox and open Zotero from within Firefox, and look up the data directory via the Zotero preferences; for Zotero Standalone, just open Zotero Standalone and do the same)
  • What Rintze says would have been my next step too.

    One other possibility that occurred to me: Any chance you have a 2nd Zotero account and you're logged in (and synce with) the wrong one? We do see this surprisingly often.

    What you suspect as having happend is incredibly unlikely to be the case. Zotero sync doesn't just overwrite local data -- it only ever deletes items when either
    a) you have explicitly deleted the items or
    b) you explicitly use the reset options in the sync preferences.
    Neither of these sounds like is the case here.


    Also, for the worst case, don't you have a backup of your harddisk for prior to the 24th? That should make restoring pretty easy.
  • there were no .crashed files in the Zotero directory, but there is a store.json.mozlz4 file in the /crashed directory
    Not .crashed — .damaged. It's probably best if you can provide a list or screenshot of the files in your current data directory and 'storage' directory, including file sizes and modification times. We should be able to tell you if this is in fact the original data directory you were using or a different one.
    It seems that the most probable cause of my library loss is that the local database in Zotero Standalone was overwritten with the online version of the database that dates back to 2013 and whose file size is only about 1.55MB.
    No, just to be clear, this didn't happen. As adamsmith says, this only happens if you use the Restore from Zotero Server reset option. If your database became corrupted, Zotero would store a copy of your corrupted database and sync down to a new clean database, but it would never overwrite your existing database.
Sign In or Register to comment.