Upgrade StorageSpeeding up Zotero (including search)
Hello,
I'm struggling a bit with the Zotero responsiveness. For example, when I right-click on an item with a few attachments, the menu takes perhaps 2-3 seconds to come up. Similarly, just clicking on a collection can take up to 10 seconds before items appear. Searching is also quite slow (again, easily 10 seconds).
My laptop performs other tasks reasonably quickly (including Chrome, which is notoriously resource-hungry). I guess the issue with Zotero is the MySQL performance, and my MySQL file is quite large (2.7G). That is for around 55k entries across my libraries, so about 50kB per entry - that seems reasonable (though perhaps a little high)?
Any ideas or suggestions?
I'm struggling a bit with the Zotero responsiveness. For example, when I right-click on an item with a few attachments, the menu takes perhaps 2-3 seconds to come up. Similarly, just clicking on a collection can take up to 10 seconds before items appear. Searching is also quite slow (again, easily 10 seconds).
My laptop performs other tasks reasonably quickly (including Chrome, which is notoriously resource-hungry). I guess the issue with Zotero is the MySQL performance, and my MySQL file is quite large (2.7G). That is for around 55k entries across my libraries, so about 50kB per entry - that seems reasonable (though perhaps a little high)?
Any ideas or suggestions?
Going forward, you could rebuild the index with a reduced number of characters and pages per file. An alternative would be to rely on external search tools for full-text searches. If you want to disable full-text indexing, set the value for "Maximum characters to index per file" to 0. The most relevant data might already be available in the abstracts, notes, tags, etc., which will remain in your Zotero database if you delete the full-text index.
I had experienced similar performance issues a year ago and was able to solve them that way. By getting rid of the Full-Text Cache, my zotero.sqlite file had shrunk from 1 GB to 100 MB. I'm using Zotero's 32-bit Windows version, where performance problems might arise ealier as compared to 64-bit editions. Add-ons didn't seem to be the cause of the issue.
Back then, I was following recommendations given in these discussions:
https://forums.zotero.org/discussion/73238/how-to-clear-index-in-a-large-database
https://forums.zotero.org/discussion/66637/disable-pdf-full-text-indexing
https://forums.zotero.org/discussion/31395/huge-decrease-of-zotero-sqlite-file-size-something-wrong
Anyone reading this who is experiencing performance issues, please follow dstillman's advice.
(Adding to the steps above, one might want to switch off syncing during the procedure.)
thanks!
@dstillman - there are two types of issues, depending on whether a sync is going on or not.
(1) With sync, the right-click takes time. I also get the selected item changing erratically. E.g., I click on an item in a collection, but then the selection jumps back to the first item in that collection.
(2) With no sync going on, the right-click is fairly responsive. However, selecting a library or a collection takes a few secs to load. Also, searching is very slow.
I'll try turning the plugins off and see what happens.
Q: Are my settings adding to the sqlite DB size without any real search benefits?
To give some more feedback on this. I have disabled full-text indexing, and disabled all plugins, but was still getting the two issues above (1 and 2). My db is 2.9 GB. As Dan said above, that in itself should not slow down the right-click, but somehow it does (as described above).
In Ubuntu, I can start zotero with the profile manager (/opt/zotero/zotero --ProfileManager) and then set up a new profile. I've done this for a set of libraries I use daily. I first included one library (~ 3000 items, sqlite size ~ 40MB) and the performance is super-fast/instant.
I've then included a few more libraries (~ 10,000 items, ~ 130MB), the performance is still very fast, but interestingly it's slower than with just one library. There's no problem at all - it's very fast. However, I wouldn't have expected the performance difference to be noticeable. One thing that happens with this larger db size is that the search is 'two stepped': There's an initial delay, then one set appears, that then flips into a second set. With the smaller db, there was only a minimal/unnoticable delay, then the set of results would just appear.
I've now re-added another library, which has 20,000 items. The sql storage is now 600MB, and there's now a significiant lag. Not as bad as before, but operations that were pretty quick before, now take ~1-2 seconds (without sync).
When a sync is going on, the libraries (and right click) are very slow to respond.
Any thoughts?
Thanks!