Speeding up Zotero (including search)

Hello,

I'm struggling a bit with the Zotero responsiveness. For example, when I right-click on an item with a few attachments, the menu takes perhaps 2-3 seconds to come up. Similarly, just clicking on a collection can take up to 10 seconds before items appear. Searching is also quite slow (again, easily 10 seconds).

My laptop performs other tasks reasonably quickly (including Chrome, which is notoriously resource-hungry). I guess the issue with Zotero is the MySQL performance, and my MySQL file is quite large (2.7G). That is for around 55k entries across my libraries, so about 50kB per entry - that seems reasonable (though perhaps a little high)?

Any ideas or suggestions?
  • edited October 23, 2020
    I would recommend clearing your Full-Text Cache once in order to reduce the size of your Zotero database. In the Search preferences, press the "Clear Index…" button, then select "Clear All Except Web Links", since those can't be rebuilt. This could take some time to finish, so do it overnight. You might be able to decrease the size of your zotero.sqlite file roughly by a factor of ten. Of course, this will depend on the amount of notes, etc., in your database. Just in case, remember to back up your Zotero data.

    Going forward, you could rebuild the index with a reduced number of characters and pages per file. An alternative would be to rely on external search tools for full-text searches. If you want to disable full-text indexing, set the value for "Maximum characters to index per file" to 0. The most relevant data might already be available in the abstracts, notes, tags, etc., which will remain in your Zotero database if you delete the full-text index.
  • @qqbb, please try to avoid giving advice like this as if it's authoritative. If something is slow, we need to investigate first before giving advice. Our defaults are chosen for a reason, and clearing or rebuilding the search index isn't something we generally recommend. There shouldn't be any need to touch these settings for the vast majority of people, and they'll likely be removed from the visible prefs in a future version.
    For example, when I right-click on an item with a few attachments, the menu takes perhaps 2-3 seconds to come up.
    That's not normal. If you have any third-party extensions installed, try with them disabled. If you're still seeing this, we'd want to see a Debug ID for this, and others for other operations that are slow.
  • (To be clear, 2.7 GB is obviously very large, and the full-text index is likely the main contributor to that size, but we need to rule out other problems first. A delay when right-clicking isn't normal and shouldn't be in any way related to the full-text index. Databases are designed to remain fast even when large, as long as they're indexed properly.)
  • @dstillman: Sorry, this was meant as a personal recommendation to a fellow user. It's certainly not something I would give as general advice.

    I had experienced similar performance issues a year ago and was able to solve them that way. By getting rid of the Full-Text Cache, my zotero.sqlite file had shrunk from 1 GB to 100 MB. I'm using Zotero's 32-bit Windows version, where performance problems might arise ealier as compared to 64-bit editions. Add-ons didn't seem to be the cause of the issue.

    Back then, I was following recommendations given in these discussions:
    https://forums.zotero.org/discussion/73238/how-to-clear-index-in-a-large-database
    https://forums.zotero.org/discussion/66637/disable-pdf-full-text-indexing
    https://forums.zotero.org/discussion/31395/huge-decrease-of-zotero-sqlite-file-size-something-wrong

    Anyone reading this who is experiencing performance issues, please follow dstillman's advice.

    (Adding to the steps above, one might want to switch off syncing during the procedure.)
  • Hi both,

    thanks!

    @dstillman - there are two types of issues, depending on whether a sync is going on or not.

    (1) With sync, the right-click takes time. I also get the selected item changing erratically. E.g., I click on an item in a collection, but then the selection jumps back to the first item in that collection.

    (2) With no sync going on, the right-click is fairly responsive. However, selecting a library or a collection takes a few secs to load. Also, searching is very slow.

    I'll try turning the plugins off and see what happens.
  • @dstillman regarding your comment about the indexing settings, I have mine bumped up so entire file gets indexed. My settings are 50000000 characters and 9999 pages per file (default is 500000 and 100). When I am adding a lot of items quickly, I change the pages to 0 (zero) to speed up addition and then go back and change the pages to 9999 and rebuild the index. With 3430 indexed items, my sqlite is about 380 MB. I have *not* seen any slowing down, but still:

    Q: Are my settings adding to the sqlite DB size without any real search benefits?
  • Hi all,

    To give some more feedback on this. I have disabled full-text indexing, and disabled all plugins, but was still getting the two issues above (1 and 2). My db is 2.9 GB. As Dan said above, that in itself should not slow down the right-click, but somehow it does (as described above).

    In Ubuntu, I can start zotero with the profile manager (/opt/zotero/zotero --ProfileManager) and then set up a new profile. I've done this for a set of libraries I use daily. I first included one library (~ 3000 items, sqlite size ~ 40MB) and the performance is super-fast/instant.

    I've then included a few more libraries (~ 10,000 items, ~ 130MB), the performance is still very fast, but interestingly it's slower than with just one library. There's no problem at all - it's very fast. However, I wouldn't have expected the performance difference to be noticeable. One thing that happens with this larger db size is that the search is 'two stepped': There's an initial delay, then one set appears, that then flips into a second set. With the smaller db, there was only a minimal/unnoticable delay, then the set of results would just appear.





  • Hi all,

    I've now re-added another library, which has 20,000 items. The sql storage is now 600MB, and there's now a significiant lag. Not as bad as before, but operations that were pretty quick before, now take ~1-2 seconds (without sync).

    When a sync is going on, the libraries (and right click) are very slow to respond.

    Any thoughts?

    Thanks!
  • So I have been working with a 20k+ entries library which has been painfully slow and very annoying to work with. After finding this thread I've cleared my index and updated the indexing settings and it suddenly only takes me a few seconds to add or edit citations rather than over a minute. Thanks for sharing your experiences @qqbb, this is very helpful, I sure hope no one removes these settings!
  • Again, if a specific operation is slow, we need a Debug ID for it so that we can investigate. E.g., the full-text index shouldn't have any effect on adding or editing citations (and I'm not even sure what that means).
Sign In or Register to comment.