How big can a library get?

sciwonder

Realistically, how big can a library get before noticeable issues (e.g., slower performance in search) begin to emerge simply due to the library's size? Here, I am just assuming a library solely populated by references; no PDFs or notes—those are stored elsewhere. My library is currently at 5428 items and I've noticed that Zotero is a bit sluggish on start-up and search seems slower, although still much faster than other reference managers I've used in the past.

adamsmith

Depends on lots of factors including your computer as well as your definition of noticeable. As you say, you certainly _notice_ the size of the library at 5k, but it's still pretty quick. My sense is that for most people things get noticeably slower once you go over ~20k, you hit serious performance issues in the 50-100k range, and it becomes difficult to work with Zotero much beyond 100k items.

Zotero is designed to frontload a lot of its database work on startup, so if possible it's recommended to just have it running.

sciwonder

Well, I am running it off spinning rust, so I imagine that even the small dip in performance would become less noticeable if it were launching on an SSD instead. I'm glad to hear that there's plenty of headroom for large libraries though. I can't imagine reaching 20k, but then again I didn't foresee reaching 5k either!

ejzejz

Is the best way to deal with large libraries without having to delete things to move some of the older/less relevant/specialized files into another library? Or do you have another suggestion?

adamsmith

It really depends. I'd say for most researchers, that size of library that actually requires worrying about this is unusual, so these are mostly going to be special cases with particular needs.

Moving projects to groups will work to some extent (currently that still affects the word processor add-on, which searches through all groups) and in some situation different accounts & profiles could be an option.

ejzejz

Thanks. My library is about 40K. And things are slow, particularly inserting citations with the Word add-on.
I can cull some of it, but would prefer to keep many of the older items stored somehow in Zotero (actually Jurism) even if it's somewhere else and more like an archive.
In that case, do you have a suggestion?

dstillman

And things are slow, particularly inserting citations with the Word add-on.

What do you mean, exactly? Which part of that?

ejzejz

The desktop app is slow to launch, which I don't mind that much. And it takes a while for various collections to load once I click on them. But when I want to insert a citation into a paper, it takes a long time for Zotero to find whatever I have typed in, unless it's something I've already used. Can I restrict what it searches through? Or do you have any other ideas?

adamsmith

But when I want to insert a citation into a paper, it takes a long time for Zotero to find whatever I have typed in, unless it's something I've already used.

I'm seeing that even with a smaller library (~10k parents spread across multiple groups), particularly on slower machines and worse on google docs.

I've occasionally moved to using the classic add citation dialog because that does allow you to restrict where you search (might also be useful for you ejzjez), but obviously that's otherwise much clumsier so quite unsatisfactory.

dstillman

Could we see a Debug ID from Zotero for a search in the citation dialog that's slow?

ejzejz

Thanks that should help with the citation speed. I'll get that to you in the next couple of days.
In terms of managing the size of my library, the sense I'm getting is that I could set aside some older items in a group for my own organizational purposes, but this won't impact the performance/speed and that my alternative is to create a separate library of older items I don't need to access regularly.

dstillman

We don't recommend creating separate libraries for performance reasons. If there's a performance issue, we'll try to address it. But we'd have to see what's happening first.

As adamsmith says, the classic citation dialog can limit the search to specific libraries or collections, and we're planning an updated main citation dialog that will allow you to restrict searches in a similar way.

b0c5

Related: https://forums.zotero.org/discussion/79760/how-many-items-can-i-safely-use-in-a-zotero-library#latest