Zotero has been becoming slower and slower over the last weeks. Also hangs appear.

gerald
In our research group (~20 people) we use a Zotero group biblipgraphic library with many items. About three weeks ago, I received the first complaints of team members that Zotero is becoming slower when using it (the start-up loading of the library always takes some time, but we got used to it). We realized the slow-down when searching the library, adding full-texts, but particularly with citations from within Word or Writer. Now additionally to these slow-downs, people (including myself) also get hangs of Zotero (when adding full-texts and citing from within the word processor).

Most of us use Windows 7 with Firefox/Zotero, some with Zotero standalone. A few use MacOS X with the Standalone. Some use Ubuntu 13.04 with Firefox/Zotero. Word processors in use are Word 2010 and LibreOffice 4.0.x

What can we do to improve the speed again? Is it a bug? Or is there any problem with our Zotero group library in general?
  • sara_cologne
    Hei from cologne,

    I have read gerald`s post with big interest caus we have the same problem. We also use a group library and zotero works very, very slow or shuts down. Me and my colleagues need a lot of patience when working with zotero.
    I am looking forward for your advice.
    Are there some special settings in gerenal when working with group librarys?

    Thank U
    Sara
  • adamsmith
    what size libraries are we talking? http://www.zotero.org/support/kb/item_count
    (both top-level and total items)
  • gerald
    edited July 20, 2013
    The size doesn't seem to be the problem. However, maybe a developer could have a brief look at the library, whether there is something not working correctly in the library-interna.

    I will try to report the slow-downs and temporary freezes as detailled as possible and - if possible - ad a debug ID http://www.zotero.org/support/debug_output (Thanks, adamsmith, for the link). However, in many cases the slow-downs seem to come random. At least, it will be difficult to reproduce them. Anyhow, I will try.

    Here is the first Debug ID: D1117503505. Attaching a document (300kb) to an entry froze Zotero for ~30 seconds and Firefox asked me whether I want to kill the process. The computer is an Intel core i5 with 4GB, so this is not the limiting factor.

    I will add more debug IDs soon.
  • gerald
    Second Debug ID: D789664589. Start-up of Zotero slow. Mark all entries results in temporary hang.

    After cold start of computer and Firefox, I started Zotero (clicking on the Zotero tab). This took 28 seconds until Firefox responded again and Zotero finally was active and entries were selectable.
    Then I selected one of the entries and hit Control+A to highlight all entries. This temporarily froze Zotero for 50 seconds. I even got an error message: "Firefox is not responding, do you want to wait?".
  • dstillman
    Why do you say the size isn't the problem? It almost certainly is. Note that it's the size of the database that matters, not the particular library that you're using at any given moment. So anyone in that large group might see slowdowns when using Zotero generally.
    Mark all entries results in temporary hang.
    What do you mean by "mark all entries"? Or are you just referring to the Select All that you describe later?

    For the start-up hang, I suspect you'll find that that goes away if you temporarily disable file syncing for groups in the Sync pane of the Zotero preferences. We're currently debugging freezing at sync time for people with libraries with many files, as Zotero checks for locally modified files. I don't believe this is a new issue, but as people's libraries grow it becomes more apparent. We're determining the best way to address this.

    For the hang while saving an item, I'd try disabling full-text indexing by setting the two max values to 0 in the Search pane of the Zotero preferences. I think the issue there is a very large full-text index. This would likely be more noticeable on the computers of the people who've been adding more documents, since currently the full-text index doesn't sync and files aren't indexed automatically on other computers.

    Let me know if those two changes make a difference.
  • adamsmith
    (Gerald sent me an e-mail with the number of items in the library - around 5k - which I told him wasn't the issue, but obviously that didn't take into account other libraries).
  • gerald
    Thank you very much for your help, Dan.

    Yes, by "mark all entries" I mean "Select all" (Ctrl-A)
  • gerald
    Sorry for my late response, but I needed a bit time to test your ideas over the weekend. I tried out what you described and Zotero feels usable again, but still not perfect. E.g. the temporary hangs are still there, but at least not as long as before.

    Do I understand you right that someone in the group who has a very large database (several groups and my library) can have an impact on slowdowns of other people in the group?

    Is it also correct that you do not mean by "size of the database" the size of the zotero.sqlite, but rather the total number of items in all groups and my library on the particular system?

    Is my assumption right that a large full-text index in the zotero.sqlite database does not impact speed much?

    Here the results of some testing:

    * Start situation: Cold start avg. 45 seconds, after restart of Firefox avg. 9 seconds. Temporary hangs after a few seconds of usage long a described in my trouble description.
    * Disabling file syncing did not help much: Avg. 43 seconds, 8 seconds. Still temporary hangs after a few seconds of usage, I think a bit better.
    * Setting the two max entries of full text indexing did not really change a lot when attaching files.

    What helped quite well:
    * I deleted one old group library with roughly 2k toplevel items and emptied all trashbins deleting another 1.5k toplevel items. This lead to a reduction of the zotero.sqlite from 129MB to 113 MB. Zotero was somewhat quicker, but not good yet.
    * I deleted the entire full-text index. The result was a reduction of zotero.sqlite from 113 to 24MB. The system was much quicker now: Cold start avg. 28 seconds and after restarting Firefox avg. 6sec. I did not encounter temporarily hangs.
    * Now it is getting *really* funny: I let Zotero do a complete full-text re-indexing and this increased the zotero.sqlite to 399 MB. Average cold start time has decreased(!) down to 16 seconds, and after restarting firefox it stays at 6 seconds. How can it be explained that the existence of a large full-text index increases start-up speed? I did a couple of tests. This seems to be consistent. However, now again I encounter a few temporary hangs.


    * P.S. I did a couple of OS reboots between the tests. Also, no other software was running besides a file manager and Firefox.
  • adamsmith
    Do I understand you right that someone in the group who has a very large database (several groups and my library) can have an impact on slowdowns of other people in the group?
    no. The speed of your Zotero is only affected by the size of your database - i.e. the total of your library and all groups you belong to.
    Is it also correct that you do not mean by "size of the database" the size of the zotero.sqlite, but rather the total number of items in all groups and my library on the particular system?
    correct.

    Dan probably has more on the hangs.
  • dstillman
    Is my assumption right that a large full-text index in the zotero.sqlite database does not impact speed much?
    No, it does, if the index is being accessed. That's why I suggested setting the full-text settings to 0, because with a large index it takes longer for Zotero to index new files. An overhaul of the full-text search functionality is planned for Zotero 4.1 or 4.2, and that will hopefully improve performance with large full-text indexes.
    Temporary hangs after a few seconds of usage long a described in my trouble description.
    You should determine whether that's related to syncing (file or otherwise). Disable auto-syncing while you're troubleshooting this.
    How can it be explained that the existence of a large full-text index increases start-up speed?
    Was a saved search selected in Zotero? The full-text index shouldn't have any effect on startup speed unless you're in a saved search that uses it. Zotero doesn't touch the full-text index for a normal library or collection load.

    The database can become fragmented over time, and the full-text index might be particularly susceptible to that, but just clearing and recreating the full-text index wouldn't defragment other parts of the database, so it shouldn't have any effect on non-full-text-search-related Zotero performance. Are you positive that you restarted between all of the tests? It seems more likely to me that regenerating the index just caused the system to cache more of the database in memory. I suppose it also could have caused the system to relocate the database file on disk in a way that increased performance. In any case, I wouldn't read too much into those results.

    I'm happy to look at Debug IDs for other slow operations.
  • gerald
    Dear Dan, thanks for your answer. I did some more testing and I think it narrows down the problem a bit.

    Answers to your short questions:
    * There is no saved search
    * I always restarted the computer between the test results (not if I indicated "just restarted Firefox")
    * There was not relocation of files on my system. Everything, including the database file sits exactly there /home/user/.mozilla/firefox/profilexyz/zotero where it was before.

    Here some more test results:
    * There is absolutely no difference if I activate or disable automatic syncing in the Zotero preferences (for both, group libraries and mylibrary):
    ** The cold start time of Zotero varies between 16 sec and 34 sec, independently of manual or automatic syncing setting.
    ** In both cases, I encounter a long time lag (Firefox hangs) of about 51 seconds. This hang *always* begins after start-up of Zotero and the possibility to click on one of the entries. The duration of the hang is the same for automatic and manual syncing.

    I did the same test when the computer was offline (and set to manual syncing). The start-up time is not different from the cases when being online. But, when offline I do not encounter any of these hangs that immediately start after successful start-up of Zotero.
  • gerald
    Here the related debug ID: D1481946191

    Settings: computer online, manual file sync.
    Results: Zotero cold start 16 sec. Hang after successful start-up 48 seconds. After that, I submitted the debug ID.
  • gerald
    I need to correct one piece of information in my reports above:
    When I refer to manual syncing, I always referred to the settings "attached files in [mylibrary or group library] syncing with" under the heading File-syncing. I had disabled these and set to manual/on demand.

    In my testing above, I had never changed the setting automatic-synching under the heading "Zotero sync server". I just did that now and in fact there is no hang, just like when being offline.
    (cold start time in this situation (automatic-synching disabled and computer online) was 17 sec).

    To sum up:
    The long after-start-up hangs disapear when being offline or when automatic syncing (under the heading ZoteroSyncServer) is disabled.

    Sorry for the confusion.
  • dstillman
    OK, yeah, I was referring to the main auto-sync, not the file syncing settings. In Zotero 4.0.10 we're addressing temporary freezing during file sync for libraries with many attachments. If you want, you can try the latest 4.0 Branch dev XPI. I'd be curious to know whether that helps (though some of the improvements will require Firefox 23). If you do try it, you should switch back to 4.0.9 afterward.

    So, then, with auto-sync disabled, are there other slowdowns that you're still experiencing?
  • gerald
    A side-question: How long can I disable auto-sync (for testing) without risking syncing conflicts later on? I remember having had disabled auto-sync back in the 2.x days and it resulted in a couple of conflicts. Is it still like that?
  • dstillman
    You can turn it back on, or just sync manually as frequently as you want. Disabling it was just a troubleshooting step to determine the problem. I'm just saying to make sure it's off while trying to reproduce other slowdowns so that it's clear what's causing them.
  • gerald
    Dear Dan, I did some more testing and the Branch version 4.0.9.r11352 (downloaded today) looks very promising:

    Whereas the stable version 4.0.9 has a cold start time of (first test/second test) 18sec/32sec, it always has the temporary hang immediately after the successful Zotero start of 52sec/57sec.

    With version 4.0.9.r11352, the start-up performance has not changed (22sec/30sec), but the after-Zotero-startup-hangs are completely gone. I can immediately work after Zotero shows up. This is already a very great relief and improvement, because these hangs have not only appeared immediately after the Zotero start-up but also from time to time. The latter I haven't tried out yet. I will use the Branch version for half day to test this, and will then switch back to the stable 4.0.9 for safety reasons.

    A big thank to you!
  • dstillman
    4.0.10 is now out with these changes.
  • gerald
    Dear Dan, the version 4.0.10/11/12 indeed helped a lot. There are much less hangs and working with Zotero became actually possible again.

    Since a few days I added/changed tags in the group library and I encounter (again) very long delays that appear frequently after changing/adding some tags. Here is a bug ID that I let run while tagging for a while (during this time Firefox asked three times whether I want to kill Firefox or to wait), BugID: D1981219566
  • dstillman
    Can you say exactly when the hangs happened, though? At what point during the process did they occur? And can you reproduce them with auto-sync off?
  • dstillman
    during this time Firefox asked three times whether I want to kill Firefox or to wait
    Also, what was the text of this message, exactly?
  • gerald
    The hangs appear often and they are not really associated with a particular kind of action. Sometimes, it hangs after renaming an attachment and file, it often hangs during/after changing or adding a tag. The temporary hangs are usually around 30-40 seconds and the CPU usage goes up very much. The error message is the kind of Ubuntu error message that tells me that Firefox hangs and whether I want to kill it.

    I switched the auto-sync off and there was no hang anymore. So, it seems to be a problem with auto-syncing.
  • dstillman
    What about file sync? Can you figure out if this happens with file sync disabled, and then generate a Debug ID for a single sync that reproduces it?
  • gerald
    Thanks Dan, I will try it. Unfortunately, syncing is broken since this morning (https://forums.zotero.org/discussion/31620/). Maybe someone made changes while I had disabled the auto-sync on my computer. Do you have an idea what I can do about this syncing problem?
  • strath-mts
    Try to hide the tag selection window in the bottom left corner. This helped me to resolve the same issue.
  • Yichao
    strath-mts. Your approach works on my zotero! Thank you.
Sign In or Register to comment.