Zotero is slow

Hi,

Zotero is my favorite application. The developers have demonstrated a tremendous vision. I have been using it as database for all of references as well as my web browsing.

However, Zotero has one huge problem: it is painfully slow. I typically have to wait 15-20 seconds for Zotero to add or delete an entry. (If I accidentally enter anything into the auto search box, Firefox hangs for 30 seconds.) My database is only of modest size ( ~3000 entries).

My problem is not unique. I have installed it on several machines, and started with fresh databases. There are numerous other posts complaining of the same problem. Here are just a few:

http://forums.zotero.org/discussion/13822/slowtero/
http://forums.zotero.org/discussion/2218/
http://forums.zotero.org/discussion/2314/
http://forums.zotero.org/discussion/14080/too-slow/
http://forums.zotero.org/discussion/7728/way-too-slow/

The developers have posted some responses that the 3.0 version of firefox would improve the speed issue. Firefox is now at 3.6 and the issue remains. Even the new improvements to the Javascript engine for firefox 4.0 would not improve the speed sufficiently to reduce a 30 second delay to something acceptable.

Simply put, Zotero has the potential to be a superb aggregator of information, but the current implementation is crippling.

It is frustrating that the developers will not engage the community in a frank discussion of the speed issue. Is the underlying database too slow? (Sqlite?) Dan Stillman mentioned something a couple of years ago about dom and javascript being the fundamental problem? Does the entire application need to rewritten from scratch? Smallish database of several thousand entries should be much faster.
«1
  • I typically have to wait 15-20 seconds for Zotero to add or delete an entry.
    Then, clearly, your experience differs from most users of Zotero, because Zotero wouldn't have very many users if that were the case for everyone. (In general, it's more helpful to just focus on debugging an issue you're experiencing rather than assuming that it's something that the developers have ignored and the entire userbase has somehow learned to cope with.)
    There are numerous other posts complaining of the same problem. Here are just a few:
    Actually, many of the issues in those threads have nothing to do with yours, or with each other. (The first one is about a Zotero plugin that we didn't write. A couple are about opening the Zotero pane after a Firefox restart, and we've explained design decisions there. The fourth one is about the Word plugin and explains that Microsoft hasn't given us access to the interfaces we need to make the plugin as fast as EndNote's.)

    We're always trying to speed up Zotero, and we hope to make significant performance improvements in future versions, but there can be completely different and unrelated reasons for performance issues in separate components. And beyond that, different people have different experiences based on the particularities of their databases, systems, and usage patterns.
    It is frustrating that the developers will not engage the community in a frank discussion of the speed issue.
    Actually, as those threads show, we've responded in some depth on quite a few performance issues, but they mostly don't have anything to do with one another. So there's no single "speed issue" to discuss.

    But if you'd like us to look into problems you're experiencing personally, we'd be happy to help.

    So:

    1) What are your full-text indexing stats in the Search pane of the Zotero prefs?

    2) Is there a speed difference for adding items between the tag selector in the bottom left being open or closed?

    3) Provide a Debug ID for adding an item, and another (after clearing the output) for deleting.
  • "Then, clearly, your experience differs from most users of Zotero, because Zotero wouldn't have very many users if that were the case for everyone."

    Fair enough. It may be particular to my usage.

    Here are the indexing stats:
    Indexed: 5707
    Partial: 15
    Unindexed: 965
    Words: 345240

    The speed problem existed (at least) < 3000 indexed entries.

    Anecdotally, it seems slower when the tag selector box is shown. (There is no longer a tag selector box icon. I use the grippy to show or hide the tag selector box.)

    I'll upload debug info shortly.
  • Hi,
    unfortunately I have the same problem. It now takes me about 20-30 seconds to add an entry.
    Might it have something to do with the size of our libraries? My library now contains about 1.5 GB of pdf-files (around 900), and the zotero.sqlite file is 61 MB. And if I understand the programm right, this is the index and has to be rewritten everytime I add a new file.
    Nevertheless, I still love the programm and keep advertizing it to everyone I meet since I just successfully finished my PhD thesis with Zotero without any major problems and it has saved me soooo much time, so thanks to evereyone who made this possible.
  • See above. We need Debug IDs to debug this at all.
  • Here is the debug ID: D110973561.

    Thanks.
  • ragan: The delay you're seeing looks to be due entirely to full-text indexing—there's a 20-second delay adding words from that item to the existing index. If you disable full-text indexing by setting the max values in the Search pane of the Zotero prefs to 0, saving should be much faster. (That's also why the quick search is slow for you, though there are various workarounds for that discussed in other threads.)

    As I've noted in other threads related to this, we're planning to switch to SQLite's full-text indexing system (which wasn't available in Firefox when we first added full-text indexing to Zotero), which should alleviate this problem. That will happen for either 2.1 or the following major release.
  • edited September 12, 2010
    Dan-

    Ok - that seems to work. To test, I added three or four references with and without indexing. The import time drops from ~20 seconds to ~2 seconds. Is the underlying problem the full text index size (~345,000)? I have several large PDFs in the library.

    It's great to hear about the new SQLite full text indexing. For now, can I just disable full text indexing and manually rebuild the index every week or so? (It might be useful to include a caveat about speed repercussions on the indexing page until the new indexing is incorporated. I searched the forums several times but somehow missed this.)

    Thanks very much. I had been dealing with this the last couple years and had tried installing Zotero on several computers, disabling other extensions and so forth. Frustratingly, it kept reappearing.

    Zotero is a great piece of software. I'll keep singing its praises.
  • Is the underlying problem the full text index size (~345,000)?
    Yes, most likely. It could depend a lot on what you've indexed—people with the same number of items might have a much smaller index depending on the kind of content they were indexing. And the current index implementation is suboptimal in a number of ways.
    For now, can I just disable full text indexing and manually rebuild the index every week or so?
    Yes.
  • edited January 2, 2011
    Dan,

    I have similar experiences. When it works as intended Zotero is easily the best reference manager I've used. I've found it slow on two kinds of occasions. When I start Zotero the dialogue box offering to stop the script or continue it appears many times. I can send you a debug file if you want to give me an address to send it to but it's rather big to paste here. Then, once Zotero is started, when I try to add an item to the data-base the GUI appears to hang although the debug konsole shows lots of activity taking place behind it. I can send this debug file too but in brief it contains hundreds of lines like this:

    [unrelated output removed — D.S.]

    The start-up delay is only once per session but the add-new-record delay makes a big difference to the usability of Zotero.
  • Solved?

    Preferences > Sync
    Uncheck the buttons 'Sync automatically' and 'Sync attachment files in group libraries using Zotero storage'.

    Normal service is then restored (at least, for the local PC). The problem seems to have been the size of data - too much to synchronise on-line. For the time being I'll have to synchronise using files on usb-sticks.
  • I too am having a slowness issue. I tried the fix posted above about setting search numbers to 0, but it's still having issues
    this is particular when, for example, I do a search, find the reference i want, and then drag it into a folder. at that point I get a long hangup - well, long or indefinite.

    any suggestions? or do I need to start a debug number?
  • I have just imported my Endnote database (~9000 entries, but no PDFs) and it took > 1h! Since I have Vista, I have replicated the import on a XP machine (both run Firefox 3.6) and the same thing happened: the import lasted over 1h. Unfortunately, that's not where it ends: whenever I access Zotero, it uses 50% CPU on my Vista machine, it is REALLY slow, and closing Zotero while "it is busy" doesn't work without Firefox crushing! And, syncing does not work.
    By comparison, Mendeley handles my database with a breeze and syncing works just fine. I am in the process of looking for alternatives to Endnote, but Zotero does not work for me right now.
    Please let me know whether I can provide you with info that can help you addressing these problems.
  • Foellmer: You'll need to be more specific about how exactly it is "slow". Plenty of people have databases with 9,000 items.

    Disable auto-sync in the Sync pane of the Zotero preferences to make sure you're not just seeing repeated failed syncs (though "syncing does not work" is not a valid bug report).
  • Hi Dan,

    Opening up Zotero takes 30 secs during which Firefox is frozen and the rest of the computer slowed down due to excessive CPU usage. The same happens when I switch between the Trash and my database.

    Btw, I just tried to insert a citation into a doc in Libre Office (3.3.0) and I got first the error message "Action not supported - Invalid procedure call", and then after repeated tries "Open Office could not communicate with Zotero - please check whether Firefox is open". Of course Firefox and Zotero were open.
  • If you're referring to the first opening of the Zotero pane after restarting Firefox, that will be slow with large libraries, and the reasons for that have been covered extensively in other threads. In short: Zotero is optimized for the common usage pattern where at least one Firefox window is left open. (For example, on Macs, applications are almost never fully closed.) Subsequent openings of the Zotero pane should be instantaneous.

    But opening the Zotero pane will also trigger an auto-sync. Have you disabled auto-sync in the Sync pane of the Zotero preferences to make sure the slowness you're experiencing isn't related to syncing? If it is, start a new thread.

    Switching between collections within Zotero should not be slow unless you have either a massive library (are you including child items in your item count?) or a slow computer. But as stated above, we'd need a Debug ID for a particular operation (clearing output immediately before) to debug it.
    I just tried to insert a citation into a doc in Libre Office
    Start new threads for unrelated issues.
  • Dan, the Debug ID is D1069341117. I just switched back and forth between trash and my collection. I will generate another debug ID for using the search function.

    The 9000 items count refers to articles/book chapters and hence does not include child items.

    I always have Firefox open, so the way Zotero is intended to work should be perfect for me.

    And yes, I had already disabled the sync function.
  • The Debug ID is D2116443313 for using the search (I looked up an author). Took about 10 secs and lead to a temporary freeze.
  • I forgot: my laptop is a Dell Latitude E6400 ATG with a 2.94Ghz core duo processor and 4GB RAM. But I do run Vista, as I mentioned, and this - I suspect - is often a cause for trouble.
  • The 9000 items count refers to articles/book chapters and hence does not include child items.
    OK, and what's your full item count, then?
  • edited January 2, 2011
    Foellmer: Try the first one with the tag selector in the bottom-left closed.
  • Dan, closing the tag selector did it: switching between collections is quick now.

    My full item count is about 18000 (I don't now whether the exact number can be displayed).
  • The search function is also quick now :)
  • I was having similar problems with slooooow adds to the library and slow searches. It seemed much worse when I moved to firefox 4.

    Anyway I cleared the index and rebuilt it to include fewer pdf pages (15 instead of the default 100). It seems dramatically improved.

    Is it possible for the index to become corrupted?

    This time I cleared the index before I rebuilt it, which took a while but worked. Last time I was problem-solving re: slow searches but I couldn't get the index to rebuild. I ended up clicking the 'reindex item' button on a bunch of the individual files by hand, which helped. When I initialized a full rebuild, Firefox would go off on its own and the process just wouldn't finish. I left it overnight a couple of times to no avail.

    What's the latest report on whether an improved indexing subsystem will come out soon?
  • edited July 5, 2011

    Hi guys,

    I've got some similar speed issues. I'm running Zotero 2.1.8 in Firefox 4 on Windows 7. My database only has around 2000 entries, many of them with one or more attachments.

    I already turned of sync and auto-index, as someone suggested in some previous discussion. That solved part of the problem, although it's a shame that I had to turn off all those nifty advanced features. Three problems remain:


    • The quick search still bothers me. I know, you can stop the instant search by starting each query with ", but if you're in the middle of an urgent paper, you don't always remember that. Wouldn't it be possible to create a toggle for instant search so that people who want to can just search on enter?

    • The trick with the double quotes does not appear to work in the OpenOffice plugin, so when you try to insert a quotation, you still get a delay. Is there anything I'm overlooking?

    • Sometimes, especially when adding new entries or attachments or when changing the value of certain fields (often the author fields, it seems), Zotero just hangs for 5~20 seconds. If your going through 20 or 30 citations and you're on a tight schedule, that can get a bit annoying after a while.

    I agree with Ragan and some of the others that these speed issues start to work on your nerves after a while, especially if you use Zotero for heavy-duty tasks (writing a thesis, doing intensive research). And frankly, for some people it might actually be a dealbreaker.

    Dan, I know it must be annoying for you to get all these complaints while you're working your ass of to give everyone free software, so I understand you react a bit cranky once in a while. However, I think it would be a bad idea to dismiss them as coming from a whining minority (as you did above). Lots of newbies might bump into these speed issues when they first try Z and then just give up.

  • Your missing Dan's point. Dan was answering a specific point, in which a user experienced delays that just are not part of a regular user experience - many people have databases with a couple of thousand items and long delays creating or deleting items just aren't general issues and it doesn't make sense to write as if these are general Zotero issues - instead.

    As for your specific points - the first two are being worked on - I'd recommend completely ignoring the quick search fields in both Zotero and Ooo plugin until then - I assume you usually know the author or title of the reference you want to cite, so you can just sort by that column and type the first couple of letters.

    Your third point is too unspecific to be useful - attaching files can take a little longer because of indexing, but changing field content really shouldn't, if you can provide steps to reproduce that reliably that would be helpful.
  • I'm having an identical problem to rickus' third point - i.e.

    Sometimes, especially when adding new entries or attachments or when changing the value of certain fields (often the author fields, it seems), Zotero just hangs for 5~20 seconds. If your going through 20 or 30 citations and you're on a tight schedule, that can get a bit annoying after a while.

    In my case though it's often longer - up to 2 minutes or so in some cases.

    I've reproduced an example at debug ID D2115085388.

    Running - Zotero 2.18 in Firefox 5, Windows 7 Home Premium SP1 fully patched, integrated with MS Word 2007 (fully patched). Intel Core 2 Quad Q6600 @2.4ghz, 3GB RAM.

    1248 parent items, 2619 total items in the library - many large PDFs.

    I'd very much appreciate any thoughts / comments.

    Thanks in advance.
  • what exactly are you doing that results in the delay?
  • tj: There's definitely something wrong in that output. Have you tried disabling all your other Firefox extensions?

    If this happens with all extensions disabled, upload your database to the DB Repair Tool and e-mail the Upload ID to support@zot....org with a link to this thread. Don't post the Upload ID publicly. Also don't worry about the download link the tool provides, at least until we take a look to see if it would help.
  • @adam - that delay resulted from editing either the title or the author of the item.

    @Dan I'll try that and revert - many thanks.
  • @Dan - thanks for the help. Apologies for the delay. I've uploaded the database and emailed the details to the support email.
Sign In or Register to comment.