Large database?

ariek

apologies if this topic has been discussed already.


My library contains around 3500 items (not including sub-items) and the size of the database is around 1.8 GB.

What is the maximal recommended size of the database?
Should I split the library

Thanks

adamsmith

even on a slow-ish computer 15-20k items (including child items) should be not problem. On a reasonably fast machine, you should be able to go to at least 40k without problems.

ariek

Thanks for the prompt reply!
A.

ct4

Dear All,

Would it make sense then to work with Zotero with a library which has around 170'000 items?

Have there been any attempts to try an alternative db-engine to sqlite in case this should be the bottleneck (e.g. open source Wakanda DB http://wakandadb.org)?

Do server- and client-side DB have the same "size restrictions" or could the server-DB store more entries than the client-DB? How would synchronization work in that case?

Regards,
Adrian

adamsmith

I'd advise against having a db with 170k items in Zotero. It will bog down about any computer.
Currently sync will also crumble under such databases, although that might get better.
Can't speak to the other questions, though I believe it's not just sqlite, but also xulrunner's (i.e. mozilla's) sqlite implementation that causes problem and switching database systems is a huge undertaking, likely beyond what's possible to do.

dstillman

There's no reason to think that SQLite itself is a limitation, but a 170K library would certainly be too large for Zotero at the current time. Zotero hasn't been developed or tested with anything approaching that size, and many parts of the code base or feature set would likely need to be optimized or redesigned to support libraries of that size.

dstillman

Re: the Mozilla framework, the main issue there is that Zotero predates the availability of asynchronous DB access—which keeps DB access off the main thread so that it doesn't block the UI—and large parts of Zotero will need to be rewritten to use that. We're planning to start doing more asynchronous DB access in Zotero 4.1.

pojeda

Hi,

my library has 1161 items most of them (~95%) have an attached
pdf journal article. Sometimes Zotero takes about 9 seconds to
display the items. Is it normal? I remember when I had less than
50 items in the library Zotero was really much more faster. The
version I am using is 4.0.8.

regards.

gusv2007

Hi,
I have similar concerns than those of previous posters. I have a 250Mb (3500 items) db running on a W7, Firefox 32.0.3, Zotero 4.0.22. The records have almost all pdf attached and the time to load Zotero for the first time is about 15 sec. During the loading time Firefox doesn't respond. Is it because of the size of the db? The number of pdfs attached? Laptop memory/CPU or any other reason? Or all of the above?
Many thanks

dstillman

gusv2007: 15 seconds sounds rather slow for a DB of that size, though Zotero is designed to cache data at startup so that later use is quicker (with the idea that most people won't open and close Zotero very often). We'd need to see a Debug ID for the opening of the Zotero window to tell you more. (You can use the "Enable after restart" option to do this more easily.)

dstillman

(Though it also looks like you're at your file sync quota, so if you still have file sync enabled that might be slowing things down further as Zotero keeps trying to upload files. If you're not interested in upgrading your storage plan or using WebDAV, you should disable file syncing in the Sync pane of the Zotero preferences.)