You could try the database repair tool:
http://www.zotero.org/utils/dbfix/

If I remember correctly from the one time I used the repair tool, which was a while ago, eventually a link to download the repaired database file should appear on the page. I don't recall it taking that long.

If something like that doesn't happen any time soon, I'd think your best bet would be to send an email to Zotero support as per the instructions on the repair tool page.

I can't get to upload my database either for repair. I'm using Firefox 2.0.0.9. The error I get is:

ERROR
The requested URL could not be retrieved
While trying to retrieve the URL: http://www.zotero.org/utils/dbfix/?
The following error was encountered:
* Write Error
The system returned:
(110) Connection timed out
An error condition occurred while writing to the network. Please retry your request.

Hey there,

WinXP SP2. FF 2.0.0.4, Z 1.0.1

I just had the same thing happen to me. What was I doing? Mostly deleting some web page snapshots and replacing them with "Create new item from current page." Also dragging items from the library list into a sub-collection. Often I would get the "do not enter" cursor when placed atop subcollections that I know the item was not present in (a hint as to the problem?). FF/Zotero crashed, Zotero said it recovered using an autosave from 1.5 hours ago and now when I click the "check database integrity" button, it tells me there are errors.

My database is >500MB (which apparently makes the utility referred to a non-starter) with a bunch of pdfs . Bunch of notes I've added. I decided to store the pdfs in Zotero DB as the functionality permits. Now... how to recover?

Aside: Can one reconfigure the auto-save interval or uncover an explanation of what, exactly, is auto-saved?

Thanks!

I had the same problem as Frit - database got corrupted when I was deleting (duplicate) entries. I lost a week's worth of data, as I don't shut down firefox very often, so the backup was a week old. Wouldn't it be better for Zotero to do a backup on a daily (or user specified interval) rather than just when zotero is shut down? I have raised this issue before as I had a similar problem of losing data due to an old backup.

Thanks for the ideas, Dan.

I tend to disagree, however, by your characterizing it as a Firefox crash as FF hasn't crashed on me before (although because of, i suppose, its process/thread structure it has successfully tied up my machine - waiting on a large download on my dial-up conx - until I have killed it off). Unfortunately I forget the actual sequence of error messages/disappearing windows at the time that one or the other or both FF and/or Z crashed (and I don't know if Z has its own thread or is run off a FF thread/process. If it's the latter than of course Z can't crash but could only cause it's host to jump out the window).

Is your description of pdfs not being really stored in the DB one of semantics? The actual pdf file is stored by Z within the Z storage file structure which, to me, means that it is, effectively, stored in the Z DB. This doesn't appear any different to any of thousands of gif images associated with web pages that are also "stored" in the db. The significance here either way is that I need to choose whether to store artifacts in Z directly or in some other simple file system - such as folders organized by subject, author, etc. - and then simply referenced (linked) in Z. If Z's db is going to get corrupted even irregularly then I must regard it as unreliable and therefore not a wise choice to use. I really like the tool that you folks have created and how it works but computers already give me enough grief and I don't need yet another piece of non-robust software to cause me more.

I also recognize that it is still in early going (version 1) but, please, nail down the robustness - i.e. zero chance of db corruption - then work on the features. I'm looking at this tool to store my references from here on, not just for a project lasting a month or a year. Whether that's wise in the best case is another matter.

I'll have a look at the tools you mentioned.

Frit: I didn't characterize it as a Firefox crash—you did, above: "FF/Zotero crashed". Did you mean something else?

You are correct, however, that Zotero runs on the main FF thread, so no, Zotero itself cannot crash—only Firefox can crash. A Firefox crash indicates a Firefox bug, though such bugs might be exposed by code in Zotero, and we can sometimes work around them if given sufficient details (such as the CrashReporter log output on OS X and steps to reproduce).

My description of PDFs not being stored in the DB is not one of semantics. The Zotero database is the zotero.sqlite file. The contents of the storage folder, where PDFs and other files are stored, are just files in the filesystem. Zotero doesn't store files in the database. As I said above, it does store the full-text search word index in the database, however, which is why the size of the SQLite file will increase after adding a file to Zotero.

We haven't seen any widespread data reliability problems in Zotero. Generally, database corruption we've seen has been due either to Firefox crashes or instability of the underlying Firefox 2.0 storage system during very large transactions (which has generally only manifested itself during database upgrades, and we're not doing any more of those in Firefox 2).

Obviously, neither of these problems should happen, but there's not really anything we can do about them from our end. From what we've seen, however, the Firefox 3 storage mechanism should be much more robust, and the next major version of Zotero will require Firefox 3. In the meantime, we recommend you back up your Zotero data directory daily (as you would do with any critical data).

It's the very first question in the FAQ.

Sorry, for Step 3 I meant this:

sqlite3 zotero.sqlite "PRAGMA auto_vacuum=1"

Fixed above.

As for the Step 4 errors, those indicate problems in your full-text word index. Did you try just rebuilding that from the Zotero prefs?

Thanks for the help and the quick turn-around as well. My db now reports no errors.

Can I offer some observations on this whole db integrity issue in hopes of enhancing robustness and reliability? Now that my blood pressure has dropped since the db seems OK again, they are framed in hopefully helpful language.

First, I'm not suggesting prescriptions but rather what result is useful to me and, I think, to others.

The wording of the FAQ on backups is what I would refer to as jocularly catastrophic - i.e. "hard drive melts, computer is stolen, etc." - suggesting that a user maybe really doesn't need to be concerned; who would want to steal this old desktop and, anyway, I've got RAID 1 against disk crash. It doesn't mention other equally catastrophic but somewhat less Fox-new-ish events such as an otherwise innocuous crash of Firefox that might irrecoverably corrupt the database which might have a day's or 10 year's worth of data entered.

Could I suggest that the FAQ answer include, therefore, the possibility of non-fixable database corruption from a non-catastrophic computer or s/w failure? For example, by Firefox crashing, if you want to phrase it that way. Also, it should likely include a note to check the database integrity at the time one backs up the directory. If the db has become corrupted without the user knowing about it - as happened to me when I was working this issue today - who knows what will happen days, weeks, years down the road when they realize the db is corrupted.

Even the above is a second choice. So, what is the first choice?

Let me include a little of my high blood pressure rant which I penned when I thought my db was irretrievably corrupted (though it still appeared to work):

As it is, I am not pleased for the following reasons (this is not a personal attack on the developers - who I think have the makings of a truly fine piece of s/w - I don't even know them):

1. I was thrilled to come across Zotero and its functionality, immediately passed the word around to friends and had high expectations,
2. the db was corrupted last night after an entire day of entering files, web sites, and other data,
3. I went back to an earlier copy of the Zotero directory/database and it, too, has db errors which I didn't know existed,

Now I'm trying to figure out what to do. I've got many webpages and many notes against different items to recover (I'm working on a masters thesis). Because I chose the option to copy the pdfs into the Z directory, I also have to go through each of the old Z folders and find those pdfs, move them somewhere else, and never again choose to save the file in the Z directory, but simply link to them."

[end of high pressure]

As mentioned, using the techniques Dan and MC provided/tweaked, the db seems to have returned to an even keel.

Nonetheless, the strategy for recovering from db corruptions and, equally important, avoiding them, seems to need work. The "you should always back up your data daily" line might be true - to an extent - but how many of the PC user community actually do that? And, if FF is inclined to crash from time to time, why isn't Z safe-guarded against that eventuality? Even as a plug-in, it's just makes sense to have a stand-alone strategy for maintaining integrity. For example (not as presciption but simply to make a point), is there anything preventing a FF add-in from writing a file (not necessarily THE file) to disk whenever it likes? I can't imagine there would be much pleasure amongst users who find they have to go back to the last Z directory save every time FF crashes or is killed (frequent or infrequent), not to mention spend time each time FF crashes to determine if they, in fact, need to go back to a previous version. this seems to me a bit of a show-stopper which, once people realize exists, can only prevent further acceptance and use.

Let me again say how useful Zotero is to me and how much I appreciate the work that has been done on it. I just know how quickly that appreciation can turn around though if a lack of reliability causes me grief (i.e. wasted and lost time and effort).

Dan,

Last thing first. Yes, it was my entire directory (my def'n of the db) that was >500mb. The sqlite file (your def'n of the db) is only about 15mb so that was simply a difference in understanding which is now corrected. I can see I'll need a lot more references in my db before I get to 70mb.

For the rest...

I am a huge fan of learning through "work flows". Sadly for me, the tech writer community in general is apparently not. Consequently, s/w documentation (in my experience) describes (hopefully) features but not in a way that describes how they were intended to be used (i.e. "what were the designers thinking"). This is a blanket statement which doesn't apply in every case.

In the current discussion context re db integrity and recovery, although I have seen the zotero.sqlite.bak file in my Zotero data folder I could not find a description of when it is created and how it is or should be used. I'm too lazy to observe changing file creation times as I do things like exit FF.


I found through Google a couple of mentions of the zotero.sqlite.bak file in Zotero.org from which could gleen a little info:

http://dev.zotero.org/overview_of_zotero_and_the_technologies_it_uses

http://forums.zotero.org/discussion/1410/recovery-of-a-document/

http://www.zotero.org/utils/dbfix/

There is nothing in the above or the documentation that I could find that gives a clear description of the db files and how to work with them. Then again, I am a guy and consequently half-blind.

For example, documentation might say:

1. the db file is zotero.sqlite. It is created automatically and placed in the directory specified in the "Storage Location" field under the "Advanced" tab in Zotero Preferences which can be found [blah blah] (default directory is [blah blah]). (We might even venture to call the storage location ZOTERO_DATA_HOME). In here are stored [blah blah]. It is created when [blah blah] and updated when [blah blah].

2. Actual computer file artifacts such as MS Word, pdf documents and other application files, captured webpages, [blah blah] are not located within the zotero.sqlite file but are, instead, placed in the directory ZOTERO_DATA_HOME\storage (in some structure which makes sense to Zotero but don't try and figure it out!)

3. It is possible that through various system failures - from a problem with Firefox to a hard drive crash to a stolen computer - the zotero.sqlite file (and possibly the ZOTERO_DATA_HOME\storage directory) can be lost. To aid in recovery from various forms of system failure, Zotero automatically creates a snapshot backup of the zotero.sqlite file in ZOTER_DATA_HOME called zotero.sqlite.bak. This is created when [blah blah]. The section "Zotero Database Recovery" provides a clear description of:
2.0 Day-to-day procedures to follow to ensure a successful recovery in the event of subsequent failure,
2.1 how to tell if your Zotero database has a problem, and
2.2 work flows that provide step-by-step instruction on how to use the zotero.sqlite.bak file and other tools to recover your Zotero database when a problem is encountered and recovery limitations.


Does this make sense?

A further suggestion: how about an option for more than one backup file (number user-selectable to some max). The problem of course, even with a single .bak, is that the db file and the storage directory will likely be out of sync. Maybe down the road some sort of audit function might be needed to either identify items missing from \storage (i.e. need to restore from .bak after an item was deleted since .bak taken) or get rid of artifacts no longer referenced (i.e. need to restore from .bak after an item was added since .bak taken). Which is why I suggested that the entire \zotero directory is the database.

I've kicked this horse enough. Hopefuly something proves useful.