Last 10 days of zotero data gone missing!

shane.lindsay

This is pretty frustrating. I was using zotero today and realised that everything I saved over the last 10 days has disappeared, which has shaken my confidence in using zotero.
This isn't the first time. A week ago, I saved references over a day. Firefox crashed, and I lost everything in zotero I had stored about a day before the crash.
Doesn't zotero store items as it goes along? How can it suddenly lose all recent data? Is there any way to get back what has been lost?

dstillman

Shane: What version of Zotero are you using? Go to the Actions (gear) menu, "About Zotero" to check.

This is obviously a serious problem, but it shouldn't have been possible since Beta 2, the initial public beta released last October.

shane.lindsay

I have the most recent version, b4.r2.r1365, which I updated as soon as it came out.
The last ref in zotero was the 3rd April, However, checking the storage location in the extensions folder I find that all my snapshots are saved after the 3rd April.

As I mentioned originally, it has happened more than once (the previous time was with the newest version also), in the first time it tied in with a firefox crash. The second time, I can't pinpoint when the data was lost because it might have taken a while for me to notice. Zotero did crash firefox recently (when I tried dragging some references to the tag selecter), this might have had something to with it but I can't say for sure. Also to note is that I tend to keep my computer and browser running for many days at a time without switching off.

Looking at the zotero folder, there is a zotero.sqlite and a zotero.sqllite.tmp file. Correct me if I am wrong, but I think what happens is that when firefox closes it updates the temporary file (which is the backup) with new changes. My guess is that I lost the data because there was some castrophic firefox/zotero crash, which meant that the tmp file didn't get updated, and somehow the main zotero stlite database got corrupted, so it ended up using the backup tmp version which hadn't been updated for a week or so. So if I am right, this would suggest that the zotero tmp file should get updated on a regular basis I.e. at least daily,rather than than just when firefox closes?

As I am writing a paper right now so I have lost a heap of new references. If am to keep using zotero I will have to try closing down and reopening firefox more often, or keep a daily backup of the zotero database.

dstillman

It wouldn't have been related to the .tmp file—the zotero.sqlite.tmp is just a temporary name for the zotero.sqlite.bak file, but there was a bug in Beta 4 and earlier that prevented the .tmp file from being renamed to .bak on Windows. Zotero only uses .bak, not .tmp, for auto-repairs, and if it does so it alerts the user that the main database was damaged and creates a .damaged file in the Zotero directory.

So I have to say I'm not sure what happened. Beta 2 had a problem in which, if an error occurred during a nesting SQL transaction, the transaction might never finish, and changes would be rolled back on Firefox close. That certainly would've explained your problem, but in Beta 3 and up, the Notifier (which triggers GUI updates on item modifications) doesn't even get called if a transaction is open, so if you were interacting with Zotero at all you would've noticed that new items weren't appearing.

There are a couple things I could suggest to try to debug this:

1) Making sure showInConsole is enabled, as explained in the Advanced Instructions section of the Reporting Bugs page, and keep the Error Console open. If you see an error that looks like it might be related to Zotero, let us know.

2) Start Firefox from the command line with the -console flag:

Start, Run, "C:\Program Files\Mozilla Firefox\firefox.exe" -console

The console on Windows is far from ideal, but you'll want to go to the window's menu (upper-left corner), Properties, Layout, and change the Scrollback Buffer Size height to 9999 (and you might want to say "Save properties for future windows with same title"). The console is pretty verbose, but every now and then you can just look at the end to make sure that there's a "Committing transaction" line. If there's a "Decreasing transaction level to x" line (where x is 0 or higher) without a "Committing transaction" below it, that's a problem, and it means there probably is an error in the Error Console too. If that happens, first note if changes in Zotero are actually being reflected in the GUI (since, if they are, the Notifier is somehow still getting updating in the middle of a transaction), and then make a copy of the text of the error console and send it to us at support@zo.....org along with any Error Console messages. Then, finally, close Firefox (you could export your library to Zotero RDF first if you want to be safe), which should close out the transaction and save any unwritten data.


Again, as far as I know what you're describing shouldn't be possible, but if it's happened to you twice, perhaps it will happen again, and having the Error Console and debug console open would be the only way of figuring out what's going on.

Sorry for the trouble, and thanks for your help.

dstillman

Sorry, forget to mention that extensions.zotero.debug.log needs to be enabled for debug logging (#2) to work. Go to about:config in the URL bar, type debug.log into the Filter box, and make sure extensions.zotero.debug.log is enabled.

shane.lindsay

Thanks for the comments. Unfortunately I dropped my laptop and it died, so I am using a Zotero on a different machine now.
I am going to keep my firefox profile and zotero constantly synced with Foldershare from now on.
I know you said that this can't happen (in theory) but I see another thread (i.e."firefox crashed - data now gone") which appears to the same problem I had.

dstillman

Shane — upon further reflection, I think I may know what happened here. When you were using Zotero during the sessions in which Firefox crashed, was it possible that you were just adding items via the URL bar without interacting with the Zotero pane much? If an error occurred within an SQL transaction, UI updates in Zotero would be mostly broken, with new items not appearing in the current view until clicking to another collection and back and various other oddities, but if you were just adding items from web pages via the URL bar icon, you might not notice that the items weren't appearing immediately, the transaction would remain open, and a Firefox crash would cause it to lose data.

Does that seem plausible?

shane.lindsay

I have been problems quite a bit recently with new items not appearing in the current view until clicking to another collection and back - but this didn't prompt me to restart - it prompted me to click to another collection and back to get the items to appear! I will know better now, and hope an updated beta comes out soon.

But I don't recall this happening when I added much of data which got lost. I only noticed data was missing when I tried to move a reference to a folder that I had created some time ago and put items in, and couldn't find the folder.

dstillman

OK, this is quite likely the result of a previous error leaving a transaction open, then. I imagine you've read it, but I'll link anyway to the other thread in which this issue was discussed.

We're trying to get a new beta out very soon. In the meantime, if you do experience problems of items not refreshing properly while the Zotero pane is open, before restarting, please follow the steps in the Advanced Instructions section of the Reporting Bugs page and let us know what errors show up in the Error Console.

grub

I've experienced similar problems with lost data after a Firefox crash. I rarely use the URL bar and would have noticed if things weren't updating properly. The FF crash doesn't appear to be related to Zotero. I lose all data that has been entered since the FF process was restarted. I've resorted to restarting FF whenever I've entered several entries but this is pretty inconvenient.

I also experience a bug when exporting Zotero RTF where FF CPU usage shoots up to 100% if I leave things for 10 minutes or so everything seems to work. But it does mean that I have no easy way of backing up the Zotero DB. I think it'd be great to have a 'save' button until these bugs are resolved.

I'll start using the console as suggested and give some more informed feedback.

dstillman

grub: It'd mainly be new items that wouldn't show up. Edited items would likely appear to change, and switching between collections would cause new items to appear. However, all of this would require there to be errors in the Error Console (with showInConsole on)...

For the RTF/CPU issue, does this happen with any size RTF? As in, just a single item? Extensions run in a single thread in Firefox, so a large operation will hang the UI and quite possibly peg the CPU at 100%. You should be able to see what's happening in the debug console, though.

dstillman

This should no longer be an issue in 1.0.0b4.r3, available now via the home page. This latest release will disable the UI completely and prompt a restart if a transaction is left open due to an error.

grub

Thanks Dan, I remember losing a few hours after importing an Endnote library and fixing the entries, FF crashed and I lost the lot. I guess this falls into the category of items that would appear to update even though there was an open error.

I tried the RTF with a single item and it was fine. So it must be the single thread issue.

Thanks for the update.