long delays, slow response

toms-library

I've just tried installing zotero for mac, on a macbook running OSX 10.9.1. I imported a large database of ~6000 entries from endnote. It took a good hour or so to install (which seems slow to me). I have only ~30 pdfs attached, max. I find that zotero seems to work as advertised, but often only after delays (often with the spinning wheel of death but sometimes it just does't respond at all) of several seconds, sometimes up to several minutes. It really doesn't seem useable, compared to endnote. As an example. When I type "Sherwood" into the search bar, it hangs after "Sher", giving me the spinning wheel for ~20 seconds, then, fills out to "Sherwoo", then more spinning wheel for another ~20 seconds, then finally "Sherwood", followed by more delay. Then, when clearing the search field, I'll get no response at all for 5 seconds, then the spinning wheel for another few.

I've tried uninstalling and trying again, but without any improvement. So, either:
1) zotero has problems with large libraries
2) I have some setting wrong
3) zotero doesn't play well with 10.9.1
4) something else is getting in the way.

Any thoughts? It seems like a great program otherwise.

dstillman

How many total items do you have, including child items? What you describe does not sound normal for a 6,000-item library.

When I type "Sherwood" into the search bar, it hangs after "Sher"

Which mode are you using for the quick search? In a large library, you're usually better off using Title/Creator/Year mode.

You can also preface the search string with " (quotation marks) to stop Zotero from searching until you press Enter.

adamsmith

what's the total number of items, including attachments?
http://www.zotero.org/support/kb/item_count

Zotero is slower than Endnote with large libraries, but that should kick in at 20k+, certainly not under 10k and even then it shouldn't take 20 secs let alone minutes for any normal operations (large imports can take a while).

toms-library

OK, I have 15589 for all items including attachments. I have 6410 entries. Looks like most of the entries have notes attached (which is making the total higher than the entries). But they are just notes, most with only a few lines of text (a lot of them are "the following have no corresponding Zotero field..."). Could this really be causing the performance issues?

toms-library

I get similar delays using the "title, creator, year" search option.

adamsmith

the notes could cause at least part of the issue, yes. If you don't need/want the "no corresponding Zotero field" notes, you can easily delete them in batch. See the paragraph starting with "Tip" here: http://www.zotero.org/support/kb/importing_records_from_endnote#importing_into_zotero
The delete itself will take some time, though.

dstillman

(And if you do delete them, remember to empty the Zotero trash afterward.)

dstillman

adamsmith: We can discuss elsewhere, but we may want to revisit those notes, since I'm guessing they effectively double the library size. Unless there's a compelling reason not to use Extra for those, we may want to do that instead.

toms-library

I see this:

"Tip: You can display all of the notes generated automatically during the import using the “_RIS import” tag. To select and delete these notes after clicking the “_RIS import” tag..."

however I don't understand these instructions. Where is the "_RIS import" tag? I don't see it anywhere on my standalone zotero.

I also noticed this:

"There are a few Zotero hidden preferences that affect RIS import and, if desired, should be set before importing the RIS file into Zotero."

however, again, I don't see how to set these. I went to the advanced section of the preferences, but there is no obvious place to set/change these hidden preferences... Any hints would be appreciated!

adamsmith

You should see the "_RIS import" tag in the tag selector at the bottom left:
http://www.zotero.org/support/collections_and_tags?&#the_tag_selector

The preferences involve changing the RIS.import.ignoreUnknown hidden preference: http://www.zotero.org/support/preferences/hidden_preferences
but since you've imported your library already I wouldn't re-do the import and stay away from that.

(Btw. - I'd also suggest turning off auto-sync while you deal with all this if you have that turned on - sync tab of the preferences.)


@Dan, Aurimas - yes, I think Dan is right that this is creating too much clutter/slowdown. We should default the "RIS.import.ignoreUnknown" preference to "true" imo. That would solve this and if someone really wants that data we can tell them how to import it.

toms-library

OK, I see "_RIS import", but there is no way to "click" the "_RIS import" tag... I can enter "_RIS import" into the search box below, and all the other tags disappear. But now what? I can't select the remaining "_RIS import" tag. How is this done?

(By the way, when I typed "_RIS import", it stopped after "RI" and I got the spinning wheel again, for about 20 seconds, before it completed a bit more)

toms-library

also, I notice that a number of my entries have notes that don't have any tags, but simply have parts of the entry, e.g.:

Butti, Camilla
Sherwood, Chet C
Hakeem, Atiya Y
Allman, John M
Hof, Patrick R
Comparative Study
Research Support, Non-U.S. Gov't
United States
The Journal of comparative neurology
J Comp Neurol. 2009 Jul 10;515(2):243-59.

How do I remove all these, given they have no tags associated (apparently)? Do I have to do this manually?

adamsmith

also, I notice that a number of my entries have notes that don't have any tags, but simply have parts of the entry, e.g.:

those would have been in a note field in Endnote, that's why they got turned into Zotero notes.

You almost certainly _can_ click the RIS import tag. Make sure you are at the "My Library" level (don't have a collection selected) and don't have any other tag selected. Also make sure your Zotero trash is empty

toms-library

OK, got the RIS import tag to work, and I trashed (and emptied the trash) of the RIS import notes. Things have gotten noticeably faster (though it still takes ~30 seconds to load after startup).

I now have 11080 total items including attachments, with 6410 total items. Is there an easy way to remove all the (tag-less) notes that are left, which all seem to have completely redundant info with their main entries. I most certainly did not create these in endnote, so this is some feature of exporting from endnote and/or importing to zotero. Any ideas I might try (including exporting from endnote again)?

Thanks for all your suggestions and clarifications. This has been very helpful

adamsmith

do an advanced search (magnifying glass) for
Item Type --> is --> Note
(leave all checkboxes unchecked)
Click on "Saved Search" to create a saved search for that. It will show up in your left-hand panel.
In that saved search, select all (Ctrl+a or on a Mac Cmd+a). Note how that only selects the notes, not the items they're attached to.
Right-click --> Move items to trash.

I'd also be curious what happened during import/export.
Could you open the RIS file with a text editor and find one such entry - e.g. the Butti one above. It will start with TY - and go to ER-
Post that entry here so we can see what's going on.

toms-library

Also: it takes ~40 seconds for zotero to startup (even after removing so many notes). Is this normal?

toms-library

Here is the text from the RIS export file from endnote for that Butti entry above:

TY - JOUR
AB - Von Economo neurons (VENs) are a type of large, layer V spindle-shaped neurons that were previously described in humans, great apes, elephants, and some large-brained cetaceans. Here we report the presence of Von Economo neurons in the anterior cingulate (ACC), anterior insular (AI), and frontopolar (FP) cortices of small odontocetes, including the bottlenose dolphin (Tursiops truncatus), the Risso's dolphin (Grampus griseus), and the beluga whale (Delphinapterus leucas). The total number and volume of VENs and the volume of neighboring layer V pyramidal neurons and layer VI fusiform neurons were obtained by using a design-based stereologic approach. Two humpback whale (Megaptera novaeangliae) brains were investigated for comparative purposes as representatives of the suborder Mysticeti. Our results show that the distribution of VENs in these cetacean species is comparable to that reported in humans, great apes, and elephants. The number of VENs in these cetaceans is also comparable to data available from great apes, and stereologic estimates indicate that VEN volume follows in these cetacean species a pattern similar to that in hominids, the VENs being larger than neighboring layer V pyramidal cells and conspicuously larger than fusiform neurons of layer VI. The fact that VENs are found in species representative of both cetacean suborders in addition to hominids and elephants suggests that these particular neurons have appeared convergently in phylogenetically unrelated groups of mammals possibly under the influence of comparable selective pressures that influenced specifically the evolution of cortical domains involved in complex cognitive and social/emotional processes.
AD - Department of Neuroscience, Mount Sinai School of Medicine, New York, New York 10029, USA.
AN - 19412956
AU - Butti, C.
AU - Sherwood, C. C.
AU - Hakeem, A. Y.
AU - Allman, J. M.
AU - Hof, P. R.
DA - Jul 10
DO - 10.1002/cne.22055
ET - 2009/05/05
KW - Animals
Body Weight/physiology
Brain Mapping
Cell Count
Cerebral Cortex/cytology/*physiology
Dolphins/*physiology
Neurons/*physiology
Organ Size/physiology
Pyramidal Cells/physiology
Species Specificity
Whales/*physiology
LA - eng
M1 - 2
N1 - Butti, Camilla
Sherwood, Chet C
Hakeem, Atiya Y
Allman, John M
Hof, Patrick R
Comparative Study
Research Support, Non-U.S. Gov't
United States
The Journal of comparative neurology
J Comp Neurol. 2009 Jul 10;515(2):243-59.
PY - 2009
SN - 1096-9861 (Electronic)
SP - 243-59
ST - Total number and volume of Von Economo neurons in the cerebral cortex of cetaceans
T2 - Journal of Comparative Neurology
TI - Total number and volume of Von Economo neurons in the cerebral cortex of cetaceans
UR - http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Citation&list_uids=19412956
VL - 515
ID - 3464
ER -

toms-library

Looking at the endnote entry for that, I do see a field for "notes" that has that info in it. That particular entry I know was an auto import from an online database. So this looks like something Endnote is doing when importing? And zotero is just doing its job?

adamsmith

Initial startup loads the entire database so that other operations should be pretty instantaneous, so that may take some time, though still this seems odd. You have a pretty new computer? My 6k library takes <5secs to startup with Standalone.

Exactly what you say on the N1: That's supposed to be a note, so Zotero is doing that correctly on import.

toms-library

Thanks for all your help. After removing the extra notes (both “_RIS import” tagged ones and the ones that Endnote mysteriously decided to add), the library works very fast (minor delay when entering some name to search for, but not bad). It still takes a long time to load, for some reason, but once it is loaded, things are good.

megan.reif

I imported my EndNote files many versions ago, and so I don't have the _RIS import tag, but still have hundreds, maybe thousands of notes with "no corresponding Zotero field: ID ####". Is there another way to delete them en masse? Also, I'm using the 4.0.18 Beta which worked wonders for retrieving all the metadata, but I don't seen an option for the recursive collections and all of my tags have disappeared. I am going to revert back to the current version, but I wonder if I my tags do not return, how I can import the tags from Standalone before syncing.

adamsmith

What do you mean you don't see an option for recursive collections? Where/how are you looking? Nothing about that has changed.
Same with tags. I'm suspecting this may be a settings issue. Have you checked for items you know to be tagged in the right-hand panel?

You can just search for the notes in advanced search (note --> contains --> "no corresponding Zotero field" ), create a saved search, select all (only the notes will be selected) --> right-click --> Move items to trash.

aurimas

I imported my EndNote files many versions ago, and so I don't have the _RIS import tag, but still have hundreds, maybe thousands of notes with "no corresponding Zotero field: ID ####".

FWIW, I'm pretty sure that the _RIS tag and the "no corresponding Zotero field" notes were released at the same time. So it looks like you either deleted the tag or you're having issues with tasg.

adamsmith

FWIW, I'm pretty sure that the _RIS tag and the "no corresponding Zotero field" notes were released at the same time.

you're right. There was a brief time where we put each field in a separate note, but those didn't have the "no corresponding Zotero field" label. So if you have that label you should have the tag.
Tags are working fine for me in the beta and there are no known problems, so if there _is_ a problem with the beta it will likely not get fixed unless you help us figure out what it is.

megan.reif

I have tried the search many times but I always get the parent items even if I uncheck the box "include parent and child items". Sadly I think I must have inadvertently deleted the _RIS tag years ago when deleting auto-tags that were slowing things down.

dstillman

I suspect you're misinterpreting what you're seeing. If items are gray, they're not matches. Press Ctrl-A/Cmd-A to select just the matched notes.

toms-library

I still have long delays opening (~30-40 seconds), which sounds much longer than others (with about the same number of items: ~6500). Also (and more annoyingly), when I select an item to put it in on of my collections, I get the spinning wheel of death for ~10 seconds. This seems inordinately long for simply copying a reference into a collection. I realize something may be wrong with my system (external to Zotero), but I don't have delays like this in other programs, so any thoughts about what I might try to ferret out the problem would be very much appreciated.

aurimas

Sorry if I missed this in the discussion above, but have you tried (or are you) using Zotero Standalone? This could help eliminate some variables

toms-library

Update: I think my delays correspond to the loading of the tags.
I have My Library, a collection (call this "my collection") under this, and a single group library (under "group libraries"). When I click between these (in the lefthand column), I notice there are no tags form my group library, but there are many for My Library and "my collection". Once the tags are loaded (displayed) for My Library or "my collection", clicking between them is almost instantaneous. However, clicking from the group library (without tags) to either My Library or "my collection" results in a ~10 second delay (during which no tags are displayed). So my problem appears to be correlated with the loading of the tags.

One thought: is there a way to remove all tags (or the database that holds them), and retag things? I'm wondering if they are odd or corrupted in some way.

Not sure if this is helpful at all, but during the delay, the spinning wheel is displayed for ~4 seconds, then goes away for a second or two, then is redisplayed for another ~4 seconds. I'm assuming this pattern relates in some way to the processing it is doing (loading the tags?). Might not be a helpful clue, but just in case...

toms-library

Yes, I'm using Zotero standalone

aurimas

Try using https://www.zotero.org/utils/dbfix/ At the very least it will give Dan a chance to look at your database and maybe try to reproduce the lag. Remember to post the ID here