Large database fails to upgrade to 5.0

ChristianK
Greetings - heavy/happy/thankful Zotero user for ~10y now. My database grew to ~45.000 records, nearly all of them with pdf file attached and most of these indexed. Size of zotero.sqlite is at ~4.3 GB, never experienced database failure since ~2010. Last working Zotero version is 4.0.29.16 running within Firefox 53.0b9, OS is Windows 10 Pro v1703.

After Zotero Standalone 5.0.17 install, the software starts, then apparently hangs while displaying "Upgrading database" message, Windows Task Manager shows constant CPU/memory/file activity from Zotero process. After 3-4 hours, Zotero 5.0 displays error message cited below and exits after confirmation. Zotero background process remains active for another few minutes, then exits too. Checking reveals that database file remained unchanged and accessible by Zotero 4.0/FF (which was not running during the events described). This sequence repeated 5 times with no changes.

Searching the forum, I found only a few discussions of similar problems, but none that could help me in achieving the conversion. E.g., an earlier timeout issue seemed resolved (https://forums.zotero.org/discussion/63727/zotero-5-0-beta-repeated-failure-to-upgrade-database-with-standalone-beta).

How can I upgrade my Zotero database to version 5.0?
Please advise. Many thanks in advance.

Christian


Error message displayed by Zotero 5.0:

Datenbank-Upgrade-Fehler
Error Transaction timeout, most likely caused by unresolved pending work.
ConnectionData.prototype<.executeTransaction/promise</timeoutPromise</< ©resource:
//gre/modules/Sqlite.jsm:641:33
setTimeout_timer©resource://gre/modules/Timer.jsm:30:5
From previous event:
ZoteroService©file:///C:/Program%20Files%20(x86)/Zotero/components/zotero-service.js:347:4
XPCOMUtils_getFactory/factory.createlnstance@resource://gre/modules/XPCOMUtils.jsm:387:19
@chrome://zotero/content/include.js:4:14
  • dstillman
    3-4 hours is obviously an unreasonable amount of time to begin with. Can you try to generate real-time debug output for the upgrade, and submit it after about 10 minutes so that we can see what it's actually doing? If you can generate a few Debug IDs during the process, and after the error occurs if possible, that might tell us more about why it's taking so long and why it's failing.
  • ChristianK
    Thanks for the prompt reply.

    I just did as advised, submitting 10-15mins of initial output under debug ID D46735849 and another after about 35mins, ID D524697273.

    An earlier, full debug run up to the error message produced an impressively long debug output but apparently did not complete the submit procedure, as no debug ID was given. I was able to save that output as an HTML file though, from which I insert the final statements below.

    (4)(+0000094): CREATE TABLE fulltextItemWords ( wordID INT, itemID INT, PRIMARY KEY (wordID, itemID), FOREIGN KEY (wordID) REFERENCES fulltextWords(wordID), FOREIGN KEY (itemID) REFERENCES items(itemID) ON DELETE CASCADE )

    (4)(+0000107): INSERT OR IGNORE INTO fulltextItemWords SELECT * FROM fulltextItemWordsOld

    (1)(+0673278): Rolled back DB transaction 0FEyP3aq

    (1)(+0000065): Transaction timeout, most likely caused by unresolved pending work.

    (4)(+0000066): PRAGMA foreign_keys = true

    (4)(+2166536): DROP INDEX IF EXISTS fulltextItemWords_itemID

    (1)(+0000078): Error: Transaction timeout, most likely caused by unresolved pending work. Error: Transaction timeout, most likely caused by unresolved pending work. ConnectionData.prototype<.executeTransaction/promise</timeoutPromise</<@resource://gre/modules/Sqlite.jsm:641:33 setTimeout_timer@resource://gre/modules/Timer.jsm:30:5 From previous event: ZoteroService@file:///C:/Program%20Files%20(x86)/Zotero/components/zotero-service.js:347:4 XPCOMUtils__getFactory/factory.createInstance@resource://gre/modules/XPCOMUtils.jsm:387:19 @chrome://zotero/content/include.js:4:14

    (1)(+0000065): Error: Transaction timeout, most likely caused by unresolved pending work. Error: Transaction timeout, most likely caused by unresolved pending work. ConnectionData.prototype<.executeTransaction/promise</timeoutPromise</<@resource://gre/modules/Sqlite.jsm:641:33 setTimeout_timer@resource://gre/modules/Timer.jsm:30:5 From previous event: ZoteroService@file:///C:/Program%20Files%20(x86)/Zotero/components/zotero-service.js:347:4 XPCOMUtils__getFactory/factory.createInstance@resource://gre/modules/XPCOMUtils.jsm:387:19 @chrome://zotero/content/include.js:4:14

    (3)(+8227874): HTTP POST "1504110399496 addons.update-ch…" (8965181 chars) to https://repo.zotero.org/repo/report?debug=1
  • bwiernik
    You can paste the full debug output log to a secret gist at http://gist.github.com and post a link here or alternatively save it to a .txt file and email to support@zotero.org with a link to this thread.
  • dstillman
    An email (with a zipped .txt file) would be better in this case, but we probably don't need it. I think I have an idea what's going on here. I'll have a new beta build you can try in a bit.
  • ChristianK
    Cool, many thanks for your consideration and advice. I am looking forward to the revised version.
  • dstillman
    All right, try installing the 5.0 Beta (5.0.18-beta.2) and seeing if the upgrade goes through with that. Basically, there was a one-hour time limit on database operations, and the upgrade was taking longer than that for you. That's really quite a long time, but the upgrade has to rewrite almost the entire database, and you have the data directory on an external (or network?) drive, which likely makes this much slower than it would be on an internal drive. We might be able to do more to speed up the upgrade, but the beta removes that time limit, which hopefully will allow this to go through for you. If so, you can then switch back to the release version after that.
  • ChristianK
    Worked like a charm, thanks a million. Zotero 5.0 started, converted, and now gives me full access to the new database, collections and attachments.

    Conversion still took ~8 hours though, and startup time is 10-15 mins. Current configuration: OS and applications starting from an SSD, the database sitting on an internal hard disk, in a four years old machine. I am going to transfer the database now to an all-SSD environment and see how it works from there. I will update you on startup times there.
  • dstillman
    Starting Zotero takes 10-15 minutes?

    Can you provide a Debug ID for Zotero startup using the "Restart with Logging Enabled…" option?
  • ChristianK
    I am sorry, but as I am travelling a lot these days, I'll only get back again by the end of the month to the PC where Zotero took so long to start (and to upgrade the database). Since yesterday however, I installed Zotero 5.0 and the converted database on an all-SSD machine, where startup takes no more than 1-2 minutes. If this is of any help, please refer to debug ID D162777755 for a log of the latest launch there.

    Besides that, I would like to thank all of the Zotero team again for providing this wonderful piece of software, and in particular for the help in making the upgrade to 5.0.

    Christian
Sign In or Register to comment.