Database upgrade error when migrating from version 4.0 to 6.0 directly (skipping Zotero 5)

edited April 29, 2022
This is a follow-up of https://forums.zotero.org/discussion/95892/upgrading-zotero-sqlite-from-version-4-0-to-6-0-directly-skipping-zotero-5.

@dstillman , when following your reply in the above link to upgrade directly from 4.0 to 6.0, I got the following error:

Database upgrade error

TypeError: target is undefined
From previous event:
_migrateUserData_80_relations@chrome://zotero/content/xpcom/schema.js:3660:34
![screenshot](https://user-images.githubusercontent.com/104580395/165862888-15a17be6-6a08-4f38-aba3-61bdce3de172.png)

If I click the "Check for Update", Zotero will search for any available update, but since I'm on the latest version, it won't find anything and it will simply quit after I click the OK at the software upgrade window. So I get stuck at this point.

A similar error/bug can be found at https://forums.zotero.org/discussion/95598/bug

So what should I do now? Should I upgrade to some sub-version of 5.0? If so, which version 5.x will work? Many thanks.
  • Same answer from the other report. We need a Debug ID using "Restart with Logging Enabled". If you can't access the menu, see Reporting Startup Errors.
  • When I get stuck, here is the screenshot: https://user-images.githubusercontent.com/104580395/165865662-aa8056b7-cfe2-420b-b56d-85af9ffb67dc.png, so there is no "Debug Output Logging" available as in https://www.zotero.org/support/debug_output#debug_output_logging.

    Then I tried https://www.zotero.org/support/reporting_problems#reporting_startup_errors and added '-ZoteroDebug' to the of command line arguments. But then I get stuck after clicking the "Submit" button: https://user-images.githubusercontent.com/104580395/165866299-f6f81154-7394-48de-8370-64728f26664d.png, and there is no way to "click the clipboard icon to copy the Debug ID to the clipboard" as in "3b." of https://www.zotero.org/support/reporting_problems#reporting_startup_errors
  • Can you copy and paste the full output and email it to support@zotero.org with a link to this thread? (We'll respond here.)
  • edited April 29, 2022
    I uploaded it to GitHub (actual link is temporarily removed for privacy reason; if @dstillman you haven't downloaded it yet please let me know and I'll update this reply to include the download link). After extraction, it contains "Debug Output.htm" and the associated folder "Debug Output_files", and you can open the "Debug Output.htm" in a browser. Please let me know if I still need to email it to support@zotero.org.
  • (2)(+0000002): Subject not found: http://zotero.org/users/local/R6TvOA40/items/Z834P7NM - dc:replaces - http://zotero.org/users/local/R6TvOA40/items/6V2RWEMI

    (4)(+0000003): INSERT INTO relationPredicates (predicateID, predicate) VALUES (NULL, ?) ['dc:replaces']

    (4)(+0000012): SELECT predicateID FROM relationPredicates WHERE predicate=? ['dc:replaces']

    (1)(+0000063): Rolled back DB transaction M83NgtMF

    (1)(+0000005): target is undefined
    Weird.

    If you're willing to share the DB with us, either by uploading it somewhere and emailing us a link or, if it's not too big, uploading it to the DB Repair Tool and posting the Upload ID, we can try to reproduce this and see what's going on. (If you use the DB Repair Tool, you can ignore the download link the tool provides, which won't help.)

    Also, just to be clear, if you're not running the official Zotero build, you should try that first. You can also try holding down Shift at startup to disable third-party plugins. This error shouldn't really be possible.
  • edited April 29, 2022
    My zotero.sqlite is about 160MB, which is too big to upload to anywhere... and I'm sure that it has no integrity issues when used with Zotero 4 with Firefox. This error happens only after I copy all the contents within \PortableApps\FirefoxPortableESR\Data\profile\zotero\ to Zotero 6's data directory (e.g., \PortableApps\ZoteroPortable\Data\profile\).

    So now what should I do? I'm migrating from Zotero 4 with Firefox portable to Zotero 6 portable, which is now an official portable release at https://portableapps.com/apps/office/zotero-portable. Should I try to upgrade to some sub-version of 5.0 first (if so, which version of 5.x)? Or should I try the official non-portable Zotero 6 (which is strongly not preferred for my use case and should be the last resort)?
  • edited April 29, 2022
    Update: I tried holding down Shift at startup to disable third-party plugins, but still get the same error.

    Is this error related to the
    Subject not found: http://zotero.org/users/local/R6TvOA40/items/Z834P7NM - dc:replaces - http://zotero.org/users/local/R6TvOA40/items/6V2RWEMI ? If so, how can I reversely locate the zotero item within Zotero 4 for Firefox?
  • edited April 29, 2022
    My zotero.sqlite is about 160MB, which is too big to upload to anywhere... and I'm sure that it has no integrity issues when used with Zotero 4 with Firefox.
    Try to upload it to the DB Repair Tool. That's just to let us look at the file — it has nothing to do with the integrity of the file, which is why I say to ignore the download link

    Again, this error theoretically shouldn't be possible, so we can't comment on it without reproducing it ourselves and seeing what's actually going on.

    You're also not using an official version of Zotero, so at this point we don't even know if it happens in the official version.
  • For privacy concern, I can't share my zotero.sqlite...

    I'll try using the official non-portable version of Zotero soon, but before that, here is what I get after opening the zotero.sqlite (after copying from \PortableApps\FirefoxPortableESR\Data\profile\zotero\ to \PortableApps\ZoteroPortable\Data\profile\ and launching Zotero 6 to produce the error):
    https://user-images.githubusercontent.com/104580395/165879309-5e476818-cb5a-4baf-ac42-a4aa7b1a6175.png

    There seems to have several non-unique item keys such that some key is replaced by the more than one key (or different keys are replaced by a same key). Is this the reason of this error?
  • We can't help you further without being able to reproduce it. If you don't want to share the database as is, you can empty all the tables containing private data first and share just that version as long as the error still occurs. But if you're inclined to do that, you should test this in an official version of Zotero first before going to the trouble.

    Otherwise, you can export to Zotero RDF from Zotero 4 and import into Zotero 6, though it will be a little lossy.
  • edited April 29, 2022
    OK, I'll try that later, as it will take some time to test removing tables while still being able to produce the error.

    As a last resort as you suggest, if I export the entire library in Zotero 4 with Firefox to Zotero RDF, and then import it into Zotero 6, I will "lose the link to those items in word processor documents" according to https://forums.zotero.org/discussion/comment/386622/#Comment_386622. But fortunately, I didn't really use this particular database with MS Word. However, I did use it heavily with Better BibTeX. So what else will I lose other than links to those items in word processor documents? And will that affect Better BibTeX's automatic export of collections?
  • We can't help with BBT, but you'll lose Date Added/Modified values, a few fields might be changed in unexpected ways, and there could potentially be other bugs — we can't vouch for the RDF output from many years ago.
  • I've installed the official non-portable version of Zotero, and change the datadir to where the zotero.sqlite is located (U:\PortableApps\ZoteroPortable\Data\profile) in order to open that database, and I still get the same error.

    I'm trying to empty the tables of zotero.sqlite containing private data now, but it doesn't allow me to do so: https://user-images.githubusercontent.com/104580395/165884800-7be4a3f8-547d-4ab2-bc5c-403d4e8857d3.png. Is there any method to force deletion?
  • And will it help anyway if upgrading to some sub-version of 5.0 first?
  • PRAGMA foreign_keys = OFF might do it. Otherwise you'd have to delete in the order of the foreign key constraints.
  • I tried both executing the SQL command (as in https://user-images.githubusercontent.com/104580395/165887217-48403851-5ecc-4f76-9fd3-562958321e03.png) and unchecking the "Foreign Keys" (as in https://user-images.githubusercontent.com/104580395/165887174-bdc79abd-2339-4d6c-b5bb-1409cf5912af.png), but I still get the same error of "violates foreign key constraint".

    So how can I "delete in the order of the foreign key constraints" as you mentioned?
  • Sorry, but we can't help you further without the database.

    We have no interest in your data and will delete the database immediately after trying to reproduce this. I completely understand if you still don't want to share it, but then I'm afraid you're on your own for debugging this, replacing/removing private values from the database so you can share a stripped down version, or finding another workaround to upgrade to a current version.
  • I can try replacing/removing private values from the database on my own and share it, but could you please let me know how I can do this?

    You mentioned "order of the foreign key constraints" in https://github.com/zotero/zotero/blob/master/resource/schema/userdata.sql, which has 62 occurrences of FOREIGN KEY (...). The first one is FOREIGN KEY (fieldFormatID) in CREATE TABLE fields (...), and the last one is FOREIGN KEY (customFieldID) in CREATE TABLE customBaseFieldMappings (...). So which order should I delete or replace values in tables?

    And as I asked before, will it help if I upgrade to some sub-version of Zotero 5.0 first?
  • edited April 30, 2022
    OK, I was able to anonymize my zotero.sqlite by using SQL queries like (after removing the UNIQUE constraints on the columns to be anonymized):
    UPDATE collections
    SET collectionName = collectionID

    These queries set the private and sensitive names/values to the IDs which are integers and unique within each table. And I've also set NULL to some columns where this is allowed.

    And I can reproduce the same error message of _migrateUserData_80_relations@chrome://zotero/content/xpcom/schema.js:3660:34, using BOTH the official non-portable version of Zotero and the portable version of Zotero.

    Now I've uploaded my zotero.sqlite to https://www.zotero.org/utils/dbfix/, and the Upload ID is 626d37e414e4e

    So @dstillman , when can I expect a fix of this issue? If it takes days (which I can't wait for, as I need to use it now or soon, even though I'll have to use it temporarily with Zotero 4 for Firefox and upgrade the database using Zotero 6 later), will it help if I try upgrading to some sub-version of Zotero 5.0 first?
  • Reproduced, thanks. Fixed in the latest Zotero beta. You can switch back to the release version after upgrading.
  • edited May 6, 2022
    Thanks a lot! I can confirm that the beta version works. But extra steps of switching to a beta version for database upgrade and switching back to the release version is needed, and I've been using the old database these days with Zotero 4 for Firefox and many changes have been made to the old database, so another upgrade will be needed. Therefore, I'm wondering when will the next release version become generally available for download. I hope it will be very soon, and if not, could you please provide an estimated date of release? Many thanks again!
  • I'm not clear on what the problem is? You can install the beta, perform the upgrade, and then install the production version. It's an extra few seconds, and there's no risk to it — there are almost no other changes in the beta.

    6.0.7 will probably be out next week.
Sign In or Register to comment.