Startup is currently significantly slower in 5.0 — we're working on that, but for now with a large library you'd want to leave Zotero open when possible instead of closing and reopening frequently.
Once open, the UI should generally be much more responsive, with various operations that hung the UI no longer do so. Some operations that affected many items are now batched instead of performed all at once, which should avoid out-of-memory problems. And some operations (e.g., moving many items to the trash) should just be dramatically faster.
And, of course, the new sync architecture is designed to handle very large databases that couldn't be synced under the old architecture.
We still have a lot more optimization to do, but one of the benefits of 5.0 is a very large test suite that should allow us to work on performance issues quickly and safely. If there are specific operations in 5.0 (beyond startup) that people are finding slow, we can take a look.
Electron will bring certain architectural advantages, but that's still quite a ways off, and we should be able to speed up parts of 5.0 well before then.
"Startup is currently significantly slower in 5.0 — we're working on that, but for now with a large library you'd want to leave Zotero open when possible instead of closing and reopening frequently." @dstillman Can you give a sense of database sizes and slowness. For example, what might be the startup times for databases with 500; 2000; 20,000; 100,000 items? Just ballpark or best estimate numbers to get a sense of what you meant in that quoted part.
Just deleted the comment I had posted because I found my answer by reading more carefully through the above posts. Thanks for all the work folks are doing to test drive 5.0. Excited about the release.
The default data dir for Zotero 5 (at least on my Mac) is "[home]/Zotero", but a) that's a very unusual choice on a Mac, where application data usually lives under ~/Library/Application Support/[App], and b) I already had a ~/zotero, which Zotero didn't check was empty and started writing files in. Nothing lost fortunately.
"[home]/Zotero", but a) that's a very unusual choice on a Mac
It's really not. It's the same as Dropbox, Google Drive, and Adobe Creative Cloud, among others — all apps that, like Zotero, sync data to the cloud on their own but also need those files to be user-accessible.
As I'm sure you know, Zotero 4.0 stored data in Application Support. That meant that 1) PDFs and other files weren't indexed by Spotlight and 2) starting in Sierra it wasn't possible to annotate stored PDFs, because Preview would refuse to modify files within Application Support, since it's not meant for files that need to be directly user accessible.
And since Zotero includes a database, it can't use ~/Documents, which in Sierra can be synced via iCloud.
So ~/Zotero is really the only option. (In future versions, we'll also likely try to make the contents of ~/Zotero a bit more user-approachable.)
b) I already had a ~/zotero, which Zotero didn't check was empty and started writing files in
It checks before doing an automatic migration. Do you mean for a new installation? I don't think we have a check there, but it should also be extremely rare for someone to have a ~/Zotero that's not a data directory and not have an existing installation with another data directory. But I suppose it could pop up a dialog that gave the option of proceeding (which you'd want to do if you deleted zotero.sqlite on purpose) or quitting.
Stranger and stranger -- I deleted the profile that 5.0 upgraded, moved ~/zotero out of the way, deleted ~/Library/Application Support/Zotero, and re-started Z5. It looks like it randomly picked one of my existing 4.0 profiles, copied the data into a newly-created ~/Zotero, and went from there. I'd have expected Z5 to ask which, if any, profile should be copied. The previous profile that got the same treatment seemed to have been upgraded in-place, as when I opened it with 4.0 it complained about DB incompatibilities. I'm sorry to say I deleted that profile now so I can't do a post-mortem on it, but the upgrade procedure is (at least on my system) a little odd.
This was a new installation of Z5 (no Z5 or ZSA4 on my system before I downloaded and installed Z5), but I can't say for sure whether there wasn't a ~/Library/Application Support/Zotero directory -- I've worked with Z5 before after all.
~/zotero is my zotero-related development directory. It felt a little strange to have it claimed without consulting me TBH. I understand the reasons behind the choice for ~/Zotero better now, but I still wonder whether it wouldn't have been better to split the dirs in such a case -- ~/L/AS/Zotero for the DB, and ~/Documents/Zotero for the attachments. But that's really not a big deal I guess.
This is all by design. If there's no Zotero profile directory, it will look for a default Firefox profile, transfer any extensions.zotero.* prefs, and use either a custom data directory location if set or the default 'zotero' directory within that profile. If the latter, and Zotero 4.0 for Firefox isn't currently open, it will move (not copy) the data directory contents to ~/Zotero. It also updates the prefs in the Firefox profile directory to point to the new location, which ensures that 4.0 gives an appropriate upgrade warning — if it didn't, when you then opened Zotero 4.0 it wouldn't find a data directory, would create a new one in the profile directory, and then quite possibly sync down all of your data from the server, leaving you with a giant duplicated data directory that would never be migrated. Also, when this rolls out, people in Firefox will either have already been upgraded to the Zotero Connector for Firefox or will get it when they click the button in that upgrade dialog.
For the vast majority of people, this will automatically do the safe and appropriate thing. Since all current users of Zotero for Firefox will need to install the standalone version for Zotero 5.0, it doesn't make sense to prompt whether to use the Zotero for Firefox data as it did in 4.0.
I still wonder whether it wouldn't have been better to split the dirs in such a case -- ~/L/AS/Zotero for the DB, and ~/Documents/Zotero for the attachments
We considered keeping the DB in AS, but 1) that would absolutely lead to data loss from people doing incorrect manual backups and 2) it doesn't really make sense as long as the storage directory is so tightly coupled to the database (DB keys, etc.). And given that Zotero has its own syncing functionality, putting files into another potentially cloud-synced location is somewhat fraught. (People who symlink 'storage' to Dropbox also turn off Zotero syncing.)
OK, but in my case I have multiple profiles so which was picked was rather arbitrary from my POV, and I had *not* expected Z5 to take ownership of my profile, making changes that are not easily revertible, without warning. I'm glad it wasn't my production profile or I'd have had a real problem.
I do occasionally clear out the profile as I work (easiest way to sandblast a profile filled with test data) but this would mean if I do that, Z5 may accidentally claim my production Z4 profile as it starts up? That's scary TBH. I do understand the vast majority argument, but in this case it works out rather inelegantly for the this particular non-vast-majority user. It may mean that I'd have to find a separate system to do my dev work on, and that's wholly new compared to Z4; ZSA 4 never did things like this.
I mean, if you install a major new version of software, it upgrades the old database. That's essentially all that happened here. Any other choice would be pretty terrible for almost everyone else.
I'm glad it wasn't my production profile or I'd have had a real problem.
(Zotero makes a backup before upgrading the database, so it wouldn't really be a problem. )
I do occasionally clear out the profile as I work […] but this would mean if I do that, Z5 may accidentally claim my production Z4 profile
Z4 is being discontinued, so this isn't a consideration. And as I said, it will only "claim" the default Firefox profile, which you can check in profiles.ini.
It may mean that I'd have to find a separate system to do my dev work on
I'm not sure why that would be. First, after Zotero 5.0, there's no reason to have Zotero data — prefs or data — in any Firefox profile, so this wouldn't even come into play. And for 5.0, you can still create separate profiles and have those pointing to different data directories. (It's true that if you delete prefs.js in one of those, it will then default to ~/Zotero, but then don't use ~/Zotero for any of the profiles and you won't accidentally interfere with an existing data directory with a fresh install.)
But that's the thing. I did not install a new version of the same software. I had previously had Zotero for Firefox and Zotero Standalone side by side without issue. I'd sort of have understood if Z5 had affected my ZSA4 profile (even though ZSA4 used to ask whether it could use it). But now Zotero Standalone (5) affected my Zotero for Firefox profile. I do understand the reasoning but it scared the living crap out of me.
And while my use-case is of course a-typical, I have what I consider to be a very compelling reason to want to have both Z5 and Z4 running. One is my work environment that I need to manage references. The other is where I'm trying to get a version of BBT working. BBT won't magically appear, I need to be able to work on it under 5.0 while I still use 4.0. If I understand your explanation correctly this is doable if I'm cautious, but I'm not well known for my cautiousness and I'd rather not have to rely on it under stress. So I think I'll opt for something that strictly separates these two environments. Maybe two systems (but that'd be a pain TBH), maybe something that'd allow me to isolate Z5 in some way. Or perhaps a new user profile.
If someone has experience with app isolation on MacOS, color me interested.
I'm not looking to have anything changed. I thought this behavior was a bug, you explained it is this way by design. It scared me, I don't think I like it very much, but I am clear on the reasons why it's there.
Even with Z4 profiles, I wouldn't expect this to be a problem — easiest would be to just ensure that there's no Zotero data in the Firefox profile marked as Default=1. Alternatively, you could just avoid nuking Standalone profile directories (perhaps with a script that filters out relevant prefs and localstore.rdf when if you need to reset things) and have multiple profiles pointed at different data directories.
But a separate user profile would certainly be the simplest and most error-proof. With Fast User Switching that should be pretty painless to use.
(Only thing you might run into with multiple users is the connector server, which could conflict between users if multiple copies are open. You can change that with extensions.zotero.httpServer.port, but that might be hard-coded in the connectors right now.)
It's great that you're looking for a way to make PDFs visible to Spotlight. But please reconsider the choice of ~/Zotero. The user expectation is that anything directly visible in the Home folder will be readily comprehensible and editable. Keep in mind that even the relatively understandable Library folder is hidden by default. I would suggest that the move should not be made until the structure is more user-friendly.
No, I didn't miss that; Preview is simply very messed up right now, but surely it will be cleaned up soon. I would expect that anyone who really cares about annotation is doing it on a tablet or with another program. Moving a folder of what looks like gibberish to the Home folder is going to confuse a lot of people. I can see some people thinking the folder a product of malware and deleting it.
Preview is simply very messed up right now, but surely it will be cleaned up soon.
I'm not sure where you got that idea — this is almost certainly a decision on Apple's part to protect files within Application Support, which aren't really meant to be directly user-editable. And as the thread shows, this is a major existing for Zotero 4.0 users that 5.0 fixes. We're not going to change this. (We do hope to improve the folder structure in the not-too-distant future, though. In the meantime, I don't think a Zotero user is going to delete a folder in their home directory named "Zotero".)
I knew several people who broke their Office installation after moving or deleting the Microsoft User Data folder, back when it used to be created in the Documents folder by default.
You can't protect paranoid Users from deleting folders for programs they just installed. This will happen when you just mistrust the programmer of the software you just installed...
Funny to read when they break their installations on trying to be wise... do they stilll know that they are connected to the Internet when they retrieve some articles and even download them?
As I read it, the folder is not directly user-editable. But any user will find ways to do so.
[en] Hello. Just to be sure : the update from version 4 of zotero standalone to version 5 (final release) will perform automatically ? We organize trainings these days and must be clear on this point. (I could'nt find the answer in that theard or searching the forum). Thanks
[fr] Bonjour. Juste pour être certain : la mise à jour de la version 4 de zotero standalone vers la version 5 se fera automatiquement ? Nous organisons des formations ces jours-ci et devons être clair sur ce point. (désolé je n'ai pas trouvé l'information plus haut). Merci
Zotero Standalone Version: 5.0-beta.157+82b789e On: Windows 7 Pro, 64-bit
1. Zotero Preferences > General: Clicking on "Update now" for translators and styles either does nothing or changes to "Error". Could this be related to the Amazon AWS outage?
2. Would be nice to be able to double-click or right-click on "My Publications" and go to the profile page where the publications are displayed.
@emilianoeheyns My linux machine is still running Zotero 4.0.28 and has downloaded the server data which was uploaded by my mac running Zotero 5 beta. AFAIK the database can be moved between these systems using a USB stick, so one could revert back to Zotero 4 on the mac if one needed to - I think. Looking forward to your update of BetterBibLatex for Zotero 5.
Upgrade Storage
Startup is currently significantly slower in 5.0 — we're working on that, but for now with a large library you'd want to leave Zotero open when possible instead of closing and reopening frequently.
Once open, the UI should generally be much more responsive, with various operations that hung the UI no longer do so. Some operations that affected many items are now batched instead of performed all at once, which should avoid out-of-memory problems. And some operations (e.g., moving many items to the trash) should just be dramatically faster.
And, of course, the new sync architecture is designed to handle very large databases that couldn't be synced under the old architecture.
We still have a lot more optimization to do, but one of the benefits of 5.0 is a very large test suite that should allow us to work on performance issues quickly and safely. If there are specific operations in 5.0 (beyond startup) that people are finding slow, we can take a look.
@dstillman Can you give a sense of database sizes and slowness. For example, what might be the startup times for databases with 500; 2000; 20,000; 100,000 items? Just ballpark or best estimate numbers to get a sense of what you meant in that quoted part.
As I'm sure you know, Zotero 4.0 stored data in Application Support. That meant that 1) PDFs and other files weren't indexed by Spotlight and 2) starting in Sierra it wasn't possible to annotate stored PDFs, because Preview would refuse to modify files within Application Support, since it's not meant for files that need to be directly user accessible.
And since Zotero includes a database, it can't use ~/Documents, which in Sierra can be synced via iCloud.
So ~/Zotero is really the only option. (In future versions, we'll also likely try to make the contents of ~/Zotero a bit more user-approachable.) It checks before doing an automatic migration. Do you mean for a new installation? I don't think we have a check there, but it should also be extremely rare for someone to have a ~/Zotero that's not a data directory and not have an existing installation with another data directory. But I suppose it could pop up a dialog that gave the option of proceeding (which you'd want to do if you deleted zotero.sqlite on purpose) or quitting.
~/zotero is my zotero-related development directory. It felt a little strange to have it claimed without consulting me TBH. I understand the reasons behind the choice for ~/Zotero better now, but I still wonder whether it wouldn't have been better to split the dirs in such a case -- ~/L/AS/Zotero for the DB, and ~/Documents/Zotero for the attachments. But that's really not a big deal I guess.
For the vast majority of people, this will automatically do the safe and appropriate thing. Since all current users of Zotero for Firefox will need to install the standalone version for Zotero 5.0, it doesn't make sense to prompt whether to use the Zotero for Firefox data as it did in 4.0.
I do occasionally clear out the profile as I work (easiest way to sandblast a profile filled with test data) but this would mean if I do that, Z5 may accidentally claim my production Z4 profile as it starts up? That's scary TBH. I do understand the vast majority argument, but in this case it works out rather inelegantly for the this particular non-vast-majority user. It may mean that I'd have to find a separate system to do my dev work on, and that's wholly new compared to Z4; ZSA 4 never did things like this.
And while my use-case is of course a-typical, I have what I consider to be a very compelling reason to want to have both Z5 and Z4 running. One is my work environment that I need to manage references. The other is where I'm trying to get a version of BBT working. BBT won't magically appear, I need to be able to work on it under 5.0 while I still use 4.0. If I understand your explanation correctly this is doable if I'm cautious, but I'm not well known for my cautiousness and I'd rather not have to rely on it under stress. So I think I'll opt for something that strictly separates these two environments. Maybe two systems (but that'd be a pain TBH), maybe something that'd allow me to isolate Z5 in some way. Or perhaps a new user profile.
If someone has experience with app isolation on MacOS, color me interested.
Even with Z4 profiles, I wouldn't expect this to be a problem — easiest would be to just ensure that there's no Zotero data in the Firefox profile marked as Default=1. Alternatively, you could just avoid nuking Standalone profile directories (perhaps with a script that filters out relevant prefs and localstore.rdf when if you need to reset things) and have multiple profiles pointed at different data directories.
But a separate user profile would certainly be the simplest and most error-proof. With Fast User Switching that should be pretty painless to use.
You can't protect paranoid Users from deleting folders for programs they just installed. This will happen when you just mistrust the programmer of the software you just installed...
Funny to read when they break their installations on trying to be wise... do they stilll know that they are connected to the Internet when they retrieve some articles and even download them?
As I read it, the folder is not directly user-editable. But any user will find ways to do so.
Just to be sure : the update from version 4 of zotero standalone to version 5 (final release) will perform automatically ?
We organize trainings these days and must be clear on this point.
(I could'nt find the answer in that theard or searching the forum).
Thanks
[fr] Bonjour.
Juste pour être certain : la mise à jour de la version 4 de zotero standalone vers la version 5 se fera automatiquement ?
Nous organisons des formations ces jours-ci et devons être clair sur ce point.
(désolé je n'ai pas trouvé l'information plus haut).
Merci
On: Windows 7 Pro, 64-bit
1. Zotero Preferences > General: Clicking on "Update now" for translators and styles either does nothing or changes to "Error". Could this be related to the Amazon AWS outage?
2. Would be nice to be able to double-click or right-click on "My Publications" and go to the profile page where the publications are displayed.