While folks respond to my larger query, let me explain what my immediate problem is and ask how to use "Multiple instances" to solve it:
I recently became part of a 'global assessment' project that involves about 70 researchers working over 2 years to synthesize the literature on a particular set of topics. The team created a group library for its use. At last count, the number of references in the database has reached 42,000+! Since I use zotero for all my 'regular' research, in which I a member of another 15 group libraries (each a much more modest number of refs--say 100 to 1000), the sudden addition of 42k references is really slowing zotero down, both when it launches, and when I do a search.
So I was wondering whether I can use the "multiple profiles" approach, create a new profile with its own data folder, and use this profile only to deal with this 'global project' group library. But when I created the new profile, it is a) still using the old data folder, and b) even if it had created a new folder, I don't know how I can copy the 42k group library into this folder. Else I have to set sync to only sync this library in the new profile, and then switch off syncing for this library in the old profile. Right? But that still leaves the 42k files in the old profile, so when I use the old profile, it will be as slow as before.
So firstly, how would I remove the 42k group library from the older profile to speed up searches there, and if that is possible, why is creating a new profile with the -p flag not resulting in a new data directory?
Sorry for the long (and possibly confused or misplaced) post.
When you create a new profile, Zotero should automatically create a new data directory named after the new profile, as noted on the multiple profiles page. If that didn't happen for some reason, you can specify a new directory from the Advanced → Files and Folders pane of the Zotero preferences.
You'd then want set up syncing with a new account, invite that account to the group library, and leave the group from the old account.
But if you plan to use these libraries at the same time, you should note the caveats about using multiple instances.
@sharachchandra: Also note that Zotero doesn't load most of the data in the group library until you first click on it in a Zotero session, and searches are limited to specific libraries, so having a large library shouldn't significantly affect startup speed or search if it wasn't selected when you last closed Zotero. If you think it is, I'd be curious to see a Debug ID for Zotero startup, using the Restart with Logging Enabled option, and a Debug ID for a search that's slow.
Using two instances of Zotero at the same time would use more memory, so depending on how much RAM you have in your computer that could easily make things slower.
@dstillman : thanks for the suggestion. I went to My Library and then did Restart-with-logging, and sent D220407654 debug report. My observations: 0. Launching zotero was still a bit slow, don't you think? (system specs are pretty good: i7 8th gen CPU, SSD drive, 16GB RAM, Windows10.) 1. Although My Library contains only 591 records, the starting search was still pretty slow. And it goes through 2 steps: an arbitrary subset of records show up first, and then finally it zooms into the one ref that had the keyword. Subsequent searches were okay. 2. Then going to the 42k library and searching was not so bad. Your thoughts from the debug report will be useful here. 3. Use of the classic menu has solved the problem of cite-while-write for the Global Assessment work that only uses the 42k library. But will not help in my other writing. So might still look into the separate profile idea.
Launching zotero was still a bit slow, don't you think?
While Zotero doesn't load all item data until you click on libraries, it does need to do some basic setup at startup, and you have both a large number of items and also a very large number of groups. But 10 seconds is still a bit slow for your database, and there are a couple things I notice in there that I'm going to see if we can improve. Stay tuned.
an arbitrary subset of records show up first, and then finally it zooms into the one ref that had the keyword
There's currently a bug where if you search too soon during startup, it can get reset. That needs to be fixed, but I believe it only happens if you search before Zotero is fully loaded.
@dstillman Is there a way to move the files (cites and pdfs) of my 42k ref library from my original profile to the new profile, without having to sync via web?
If I'm understanding the situation, you don't need to — and wouldn't want to — move anything. The items are already in a group library. You would just need to set up a new account, set up syncing with it in a new profile and data directory, and join the group with that account in the new profile, before leaving the group in your original account.
I want to clarify, though, that we don't really recommend this. If there are performance issues, we can look into them you provide Debug IDs for specific operations that are slow, but 42K items shouldn't really pose much of a problem, and there are significant drawbacks to using multiple profiles or instances.
Upgrade Storage
I recently became part of a 'global assessment' project that involves about 70 researchers working over 2 years to synthesize the literature on a particular set of topics. The team created a group library for its use. At last count, the number of references in the database has reached 42,000+! Since I use zotero for all my 'regular' research, in which I a member of another 15 group libraries (each a much more modest number of refs--say 100 to 1000), the sudden addition of 42k references is really slowing zotero down, both when it launches, and when I do a search.
So I was wondering whether I can use the "multiple profiles" approach, create a new profile with its own data folder, and use this profile only to deal with this 'global project' group library. But when I created the new profile, it is a) still using the old data folder, and b) even if it had created a new folder, I don't know how I can copy the 42k group library into this folder. Else I have to set sync to only sync this library in the new profile, and then switch off syncing for this library in the old profile. Right? But that still leaves the 42k files in the old profile, so when I use the old profile, it will be as slow as before.
So firstly, how would I remove the 42k group library from the older profile to speed up searches there, and if that is possible, why is creating a new profile with the -p flag not resulting in a new data directory?
Sorry for the long (and possibly confused or misplaced) post.
You'd then want set up syncing with a new account, invite that account to the group library, and leave the group from the old account.
But if you plan to use these libraries at the same time, you should note the caveats about using multiple instances.
Using two instances of Zotero at the same time would use more memory, so depending on how much RAM you have in your computer that could easily make things slower.
0. Launching zotero was still a bit slow, don't you think? (system specs are pretty good: i7 8th gen CPU, SSD drive, 16GB RAM, Windows10.)
1. Although My Library contains only 591 records, the starting search was still pretty slow. And it goes through 2 steps: an arbitrary subset of records show up first, and then finally it zooms into the one ref that had the keyword. Subsequent searches were okay.
2. Then going to the 42k library and searching was not so bad.
Your thoughts from the debug report will be useful here.
3. Use of the classic menu has solved the problem of cite-while-write for the Global Assessment work that only uses the 42k library. But will not help in my other writing. So might still look into the separate profile idea.
I want to clarify, though, that we don't really recommend this. If there are performance issues, we can look into them you provide Debug IDs for specific operations that are slow, but 42K items shouldn't really pose much of a problem, and there are significant drawbacks to using multiple profiles or instances.