Please, I need to downgrade back to version 8.0-beta.21

warguelles

Is it possible to downgrade back to version 8.0-beta.21 (for Windows 11)? Yesterday, after updating to beta.23 and BBT to version 7.0.71, some (few) inconsistencies appeared between Zotero and Obsidian. I need to finish an urgent job within the next few days, and by that time I will be able to take care of the transition.

Note: before updating, I backed up all the databases (including *.bak, ha, ha!), so I think I can restore the app to its previous state. I haven't done anything since then so as not to complicate things.

By the way, the update took quite a long time... more than 40-45 minutes! Is it normal?

Thank you very much!

dstillman

Windows x64: https://download.zotero.org/client/beta/8.0-beta.21+e942278a9/Zotero-8.0-beta.21+e942278a9_x64_setup.exe

By the way, the update took quite a long time... more than 40-45 minutes! Is it normal?

No, it shouldn't take that long. This is for the popup in the bottom right, saying it's migrating fields from Extra? Is your data directory on a hard drive or an SSD?

warguelles

Thanks for your help, I successfully downgraded to beta.21 without any problems. I had previously restored the databases that I had backed up before updating to beta.23.

Is your data directory on a hard drive or an SSD?

SSD

This is for the popup in the bottom right, saying it's migrating fields from Extra?

Since I was sure everything would go well, I didn't really pay attention to many details. The idea was to open Zotero without updating it, so I could update BBT first. BUT, at the last minute, I got confused and updated beta.23 first, and then the add-on. Let me explain how I did it:

1. Yes, first a window appeared at the bottom right saying that it was updating around 3,200 records (I have a total of about 6,100 items), and it showed the progress of the operation. That took about 10 or 15 minutes.

2. After it finished with that, the window icon changed, but it still said it was updating (or something like that). That may be that took also around 10-15 minutes, not sure!

3. When it was finished, the app was still working in the background, and it was very slow. I think it wasn't until the end that I was able to update BBT.

The strange thing is that since I've been using BBT for years to communicate with Obsidian, I thought that all items had their citekey (?). That's why I assume that it changed some citekeys and that's why some inconsistencies with Obsidian appeared...

Now that I've restored beta.21, I see that references added since 2016 have their citekeys. I find it strange that the rest of the information in the Extra fields takes so long to be relocated (such as the DOIs of chapter items, etc.).

Hope it may help you... thanks!

dstillman

a window appeared at the bottom right saying that it was updating around 3,200 records

This doesn't sound like us. We show a popup in the bottom right, but it doesn't list the number of items:

https://s3.amazonaws.com/zotero.org/images/forums/u6/nltug7mlckg0vzzm8avk.png

If you have time, could you test how long the update takes with all plugins disabled? (Auto-sync disabled, a copy of the database, etc.) For me, in a test library, it migrates fields from 2,400 items in <10 seconds.

dstillman

a window appeared at the bottom right saying that it was updating around 3,200 records

@emilianoeheyns: Does this sound like something you're doing? If you're modifying many items at once, are you doing so in a DB transaction or in a Notifier queue?

emilianoeheyns

I never modify items in bulk (unless a user initiates a bulk pin manually). One thing that BBT can pop up is first time start when I assign keys in bulk, but those go to the separate BBT database.

dstillman

So the popup that @warguelles describes (not the one in my screenshot) isn't from you?

warguelles

If you have time, could you test how long the update takes with all plugins disabled? (Auto-sync disabled, a copy of the database, etc.) For me, in a test library, it migrates fields from 2,400 items in

Yes, I could test now...

That popup is definitively different!
In my case, the first popup in the bottom right said something like "updating X/3200 items". The "X" number was obviously growing during the update. This is because I know how many items were updated.

dstillman

OK, so that's not from us.

warguelles

Yes, that was the problem. All plugins disabled, the upgrade took around 10-15 seconds (database with 6110 items), and it looks like no problems with citekeys now.

https://s3.amazonaws.com/zotero.org/images/forums/u2580262/x5buucrc0m1n969yyu67.png

I already enabled BBT and MarkDB-connect plugins. The latter has recognized all notes from Obsidian...

I will gradually enable the rest of the plugins, but it looks like everything is perfect now! Thank you!

dstillman

If you have the time to do a little experimentation, it would be nice to know which plugin was slowing things down. (If you're working with a backup of the database, it's fine to force-quit it once you determine it's slow.) We're going to push these updates to Z7 shortly, but we'd like to avoid a situation where people are seeing 45 minutes of slowness due to some plugin.

warguelles

I will gradually enable the rest of the plugins, but it looks like everything is perfect now!

Update: I enabled all my plugins (over the updated db) and added a couple of references and it worked fine.
Then enabled sync, it took around 5 minutes. Is it ok?

If you have the time to do a little experimentation, it would be nice to know which plugin was slowing things down.

Yes, I can.
Just to be sure:
I will overwrite the security databases I have from beta.21, and activate the plugins little by little. Correct?
Is it okay that I have already synced (I have it manually)?

dstillman

I will overwrite the security databases I have from beta.21, and activate the plugins little by little. Correct?

Yes. Reverting to the pre-migration database each time.

Is it okay that I have already synced (I have it manually)?

Yes. As long as auto-sync is off, it will just perform the same migration again. You can keep a copy of your current active database and switch back to that after testing. (If you didn't, it should just resolve the identical local and remote migrated changes automatically, but no need for that.)

warguelles

It seems BBT (7.0.71). It is the first plugin I enabled.

The first popup was yours, and took around 50 seconds. BUT, after this first popup it came the other popup I mentioned:

https://s3.amazonaws.com/zotero.org/images/forums/u2580262/ju3wykgmcz7vxoquyek5.png

It is working for 15 minutes, and continues...

I will stop it, ok?

warguelles

Just stop! It took less that 20 minutes

warguelles

It stopped, but the app continues to run in the background. I'm going to force it to close.

emilianoeheyns

So the popup that @warguelles describes (not the one in my screenshot) isn't from you?

No. I assumed that was Zotero itself.

emilianoeheyns

But it is very possible that BBT was slowing things down. At the very least key generation, which really shouldn't have too big of an impact, but keys are persisted to better-bibtex.sqlite, and if auto-exports are present they will be triggered many times over. This is why I wanted to see if I could detect it running, so that I can postpone and accumulate auto-exports.

The key persistance (and thus better-bibtex.sqlite) is going away as soon as Z7 has the broader native fields support, because I am just going to use the citation key field from that point on.

dstillman

The first popup was yours, and took around 50 seconds. BUT, after this first popup it came the other popup I mentioned

That’s Zotero’s popup that I showed above. You said there was another one that had a count.

emilianoeheyns

Right, that would have said "assigning keys". I've just tried in on a large database, for 24k items, all without keys; that took about 2 minutes. It will only trigger for as many items are hit by an extra-migration of course. But this does not modify the zotero.sqlite database.

dstillman

@emilianoeheyns: I explained on the dev list how to check whether this had completed, but that certainly shouldn't be necessary for auto-exports, which should just be debounced. Are they not? BBT shouldn't be running the same export process over and over if there are thousands of item updates (for whatever reason) within a few seconds.

To be clear, modifying zotero.sqlite isn't the concern here — it's slowing down the update process (or any other large set of item updates).

warguelles

Ok... let me give more details:

BBT (7.0.71) enabled. The whole process took about 20 minutes.

1. First, the circular bar popup took about 30 seconds.
2. Then the other popup appeared, and the process continued for about 20 minutes... Maybe updating the BBT database?
3. The app continued to work in the background

NOTE: When the migration took about 40 minutes, the previous version of BBT (7.0.70?) was active.

The Mark-DB plugin started to give errors.

warguelles

If you need I can try the migration with only Mark-DB enabled.

warguelles

You said there was another one that had a count.

Yes, It was three days ago. I suppose it was because, as I explained before:

The idea was to open Zotero without updating it, so I could update BBT first. BUT, at the last minute, I got confused and updated beta.23 first, and then the add-on.

I have no seen that popup with the count today.

emilianoeheyns

@emilianoeheyns: I explained on the dev list how to check whether this had completed, but that certainly shouldn't be necessary for auto-exports, which should just be debounced. Are they not?

Good point -- they are.

BBT shouldn't be running the same export process over and over if there are thousands of item updates (for whatever reason) within a few seconds.

The default debounce delay is 5 seconds, every change resets the timer. So if this hits a single auto-export, it is very unlikely that more than one actually ran.

Generated citation keys however are persisted to better-bibtex.sqlite one by one. That is probably slow.

The BBT "assigning" popup has a circular progress meter, and it gives an ETA to completion.

dstillman

Generated citation keys however are persisted to better-bibtex.sqlite one by one. That is probably slow.

Can you debounce those changes and make them within transactions in batches?

dstillman

@warguelles: Can you set extensions.zotero.debug.store to try in the Config Editor from the Advanced pane of the Zotero settings, reproduce this on the next restart (i.e., by updating with plugins enabled), and then submit the debug output from Help → Debug Output Logging after a few minutes of the processing? It would help to see exactly what’s happening when it’s taking so long for you.

warguelles

I already made the configuration setting.
I am ready to start, but, I don't understand if before to update to beta.23 I should enable the Debug Output Logging or not

dstillman

You just set the setting and then it logs on the next restart.

warguelles

Do you need the output during migration or when it already finish?

The first popup with the circular progress bar took 2 minutes

It is running with the other popup for other 8 minutes now... and continue
NOTE: I am at home, and this PC is slower

warguelles

It has just finished... 10 minutes