Freeze then crash large sqlite file

Dasaan

I cannot currently access my zotero database (2.1.10, Firefox 7.0.1, Windows XP).

When I click on the Zotero button firefox freezes and then will eventually crash with no error reported.

I initially disabled syncing through Tools->add-ons, but this did not fix the problem. I have clicked and left to wait it out several times, but firefox will eventually crash before loading zotero. I checked the database integrity, and it passed.

At one point, I imported a large reference set into this database (1000s of entries). Zotero still functioned at this point, but loading was slow (because of initial caching, I understand), so I deleted these references from the database.

However, looking at the zotero.sqlite file, I notice that it is 47.4 MB, which seems large for an otherwise small database of ~200 entries. I decided to check one of the oldest file, zotero.sqlite.73.bak, and zotero opens with this as the database file.

So, my issue is in my current zotero.sqlite file. All of my most recent .bak files are large as well and will not open. The .73.bak from a previous version will open, but is missing a lot of research I have done since March.

I would like to get my current database to open without crashing firefox? Is this possible?

Cheers,
John

dstillman

Did you empty the trash after deleting the imported items? Firefox may be running out of memory, but that shouldn't happen unless you have a huge number of items.

Try uploading your database to the DB Repair Tool. If you still get the crash with a repaired database, e-mail the Upload ID from the repair tool to support@zot....org with a link to this thread. Don't post the Upload ID publicly.

dstillman

You might also want to try disabling all your other Firefox extensions and closing any other programs you have running (and/or restarting your computer).

Dasaan

Back in the office.

I do not know if I emptied the trash. It has been a while since I accessed the database. I do not think that I did.

I uploaded the database to the repair tool (ID sent seperately), and after replacing the old database with the repaired database, I still experienced the freeze then crash.

It does seem like it may be memory issue, since I can open an older, smaller database. There must be some way to clean out the database so that it opens again...

Hopefully, there is something in the uploaded database that can be identified to fix this issue.

Dasaan

Any further technical advice?

Is there a way to open this sqlite file? Maybe someway I could clean it out or get firefox to handle the largeness of it?

dstillman

Does the crash still happen if you disable auto-sync in the Sync pane of the Zotero preferences (which you can access via Tools -> Add-ons).

How much RAM does your computer have?

Do you see any relevant crashes if you go to about:crashes in the Firefox address bar? If so, generate a report for one and post the Mozilla URL here.

Dasaan

Dan,

Thank you for your reply.

I disabled auto-sync early on in the troubleshooting process.

RAM is 1.96 GB

Here is the Mozilla crash report URL.

https://crash-stats.mozilla.com/report/index/bp-5b4a5cd4-30b1-4351-bb8b-0fcba2111118

I am not sure why this has Firefox version 4.01 listed, it was 7.01 this morning and just updated to 8.0.

I appreciate any further advice you may have on this error.

John

dstillman

That's an old crash, from Zotero 2.1.5. Are there no recent crashes listed? There might not be if Windows is just killing Firefox for memory usage. You can watch the memory usage in Task Manager to see if that's the case.

Can you upload again to the DB Repair Tool and send another Upload ID via e-mail? You can ignore the download links the tool provides.

Dasaan

Dan,

I believe I found a current crash

https://crash-stats.mozilla.com/report/index/bp-cb434810-098c-4bfd-9a1f-6725d2111118

I have watched the memory usage after clicking on Zotero, and it slowly ramps up the page file usage until it maxes out and crashes.

I have uploaded again to the DB Repair tool and sent the ID and link to this thread via email.

Hopefully, we can hash this out.

Cheers,
John

Dasaan

After looking at the crash report, it looks like this was generated from a quick time problem (something else I was working on today)and not Zotero.

All of the crash reports that would link up to this issue report "Crash Not Found" even though there is a report listed. For example,

https://crash-stats.mozilla.com/report/index/bp-b4a54e8f-2290-4d00-bfee-a267c2111118

dstillman

Well, you do have 12,400 items in that database, 6,000 of which are in the trash. That certainly shouldn't cause Firefox to crash, but it's a fairly large database.

Things to try:

1) Delete localstore.rdf in your Firefox profile and restart Firefox. (That file stores window settings and will be recreated automatically.)

2) Disable all your other add-ons and plugins. (You don't say if you've done this.)

3) You can install Zotero Standalone, close Firefox before opening it, and when it asks tell it to use your existing Zotero database. Assuming that starts without crashing, empty your trash there. Then close that and restart Firefox, and see if Zotero works again.

Dasaan

OK.

I tried 1) and 2) together first. After clicking on Zotero it ran out and increased page file usage like usual, but just before the limit, the Zotero pane opened stating that an error had occurred.

The error number was 1354104199.

I attempted to empty the trash (right-click, empty trash), and let Firefox run for quite a while while it worked on this. Eventually, the page file usage dropped to ~14K and stayed there, so I assumed this was complete.

I restarted Firefox, but after clicking on Zotero, same freeze then crash with maxed out page file size. No Zotero pane this time.

Then I tried 3) next. The first time, Zotero Standalone exhibited the same symptoms as the Firefox version. I tried again out of curiosity, and it did another almost maxed page file but opened with an error. However, the "Report Errors" menu item was greyed out.

I attempted to empty the recycling bin here (note: as before, I could see me folders on the left pane, but they were not viewable in the center pane.), but after right-clicking on the trash and selecting empty trash, Zotero Standalone promptly crashed.

I double-checked the sqlite file, but it appears to be the same size as before. I do not believe I can empty the trash with the error present.

Maybe the error number above will help. More suggestions are appreciated.

dstillman

Zotero's RAM usage can surely be optimized, but this really shouldn't happen with 2GB of RAM. Using a 10-year-old OS probably isn't helping, but have you tried this immediately after a computer restart, with no other programs running?

Are you low on disk space?

dstillman

Also, it looks to me that, at least as of yesterday, a client of yours was repeatedly trying to pull down hundreds of items from the server (and failing, given all the identical requests). Are you sure auto-sync is disabled?

Dasaan

Thought to try the restart and no other programs running after my last post. Same result with both Firefox and Standalone.

I have 199 GB of free space.

I double-checked the sync option pane, all fields are blank and unchecked. It should be disabled.

Is there some way Zotero might be trying to complete a sync it never completed even though I don't want it to sync anymore?

Otherwise, I suppose RAM optimization?

dstillman

Is there some way Zotero might be trying to complete a sync it never completed even though I don't want it to sync anymore?

No. Are you syncing from another computer?

In a pinch, you could get your library onto a non-XP computer by copying your Zotero data directory over manually, open it in Zotero on there, empty your trash, and then copy it back.

Dasaan

No, I am not syncing from another computer.

I don't really have the option of moving this file to another computer because of office protocols. If I can do so, I will report back.

However, I had a thought over the weekend wondering if Mendeley could be used to read the file and then port it back into Zotero.

I enabled Mendeley's sync feature, and the Zotero database loaded directly into Mendeley with all of my updated work. It did not include any of the entries from the large database I imported and then deleted. Perhaps it does not sync the trash folder?

In an effort to access my database in Zotero, I copied the zotero.sqlite file, reset the Zotero database, and exported the synced Mendeley database as a .bib. This worked; however, it did not include the folder structure of the database and (obviously) reset the dates for each entry (which I use to sort workflow, periodically). So this method is not quite satisfactory.

So, I am wondering if there is a way to either use Mendeley for reconstructing my Zotero database or figure out why Zotero has such an issue opening my original zotero.sqlite? It seems like there would be a method that could clean out the database prior to opening it and crashing Zotero.

dstillman

Revert to your original zotero.sqlite. Mendeley won't help here, and the sync reset could cause you more trouble.

I ran some more tests on your uploaded database, and it looks like your tag situation is somewhat out of control, and that's what's causing Zotero to crash at startup. 6,096 items have a total of 186,461 tag associations, an average of over 30 tags per item. Eighty items have 100 or more tags assigned. 6,080 of those items with tags are in the trash, so I assume something went very wrong in the import.

I'm going to see if there's any way I can optimize this and get you a build to test. If that fails, I can help you clear the deleted items manually. I'll have something for you later today. Stay tuned.

Dasaan

Good news. I was able to use a much more powerful computer (Windows 7, 16 GB RAM) to open my database file.

It took a little over 2.4 GB RAM to load up Zotero Standalone with my sqlite file, but when it opened, there were no errors reported, the trash was empty, and the file size had decreased to 2.3 MB from 36 MB.

I have successfully replaced the sqlite on my computer and Zotero is functioning normally.

Apparently, Zotero was running up against my memory limitations as it was clearing out the trash on startup (with all of those tags). This issue did not occur until after I deleted the ~6,000 multi-tagged entries.

In speaking with the co-worker who exported the database I imported, it was apparently derived from ReferenceWorks, which automatically builds a collection of keywords (tags) for each of the references in the database. Exported files from these type of ReferenceWorks databases will be inherently heavy on tags, and apparently become an issue in Zotero if they are deleted en masse.

If there is a way to optimize the handling of a large number of heavily tagged entries as they are cleared from the database on startup, this may prevent related crashes in the future.

Otherwise, I suppose it's like my father says, "Just use a bigger hammer" (i.e. more RAM).

Dan, thank you for helping me troubleshoot this issue. Let me know if I can clarify anything following up the thread.

dstillman

Apparently, Zotero was running up against my memory limitations as it was clearing out the trash on startup (with all of those tags).

Yeah, that was it exactly. Zotero automatically removes items from the trash after 30 days. (A workaround would've been to increase that delay in the hidden prefs and then selectively delete smaller sets of items.) We'll see what we can do to improve this. Glad to hear you got it working.

dstillman

Actually, an easy (and good) fix would just be to limit the number of trashed items purged at once. I'll make that change.