Importing EndNote Library: import time, “_EndnoteXML import” tag and PDF availability

manook

Dear Zotero-Team and fellow Zotero users,

it took about 53 hours to import an EndNote library to an empty Zotero library fully following your recommendations (https://www.zotero.org/support/kb/endnote_import).

EndNote had created an XML-export file with 7MB and the EndNote PDF folder had a size of 1.75GB.
This EndNote library is a comparably small and rather recent one originally created within EndNote X8 about five years ago and now running with EndNote 20.1. Still, I followed your recommendations and checked for figure attachements, database integrity, etc., beforehand.

Zotero reported 1264 imported items (corresponding to available entries in Endnote). According to your kb-remarks, the tag check with ​“_EndnoteXML import” revealed 1191 tagged items from those 1264 imported ones and it seems, this would already explain why Zotero took so long for the import. 1191 is too many to go through individually as you describe in your knowledgebase.
For your info, besides using the field "research notes" within EndNote, no other fields of those reference entries where modified after importing them. Therefore, all these tagged fields were created by EndNote automatically (it appears, often author addresses).

From those 1264 imported items (that mostly originated from PubMed), 1186 had PDF-attachments within EndNote. Unfortunately the majority of these attachements could not be opened in Zotero ("The attached file could not be found in the following path..." along this kb-article of yours: https://www.zotero.org/support/kb/files_not_syncing). In Zotero's data storage folder, filtering the entries according to their creation time revealed that only 40 folders were created during the first 43 hours and these were containing PDFs in Zotero. The remaining creation of 1348 folders happened within less than 20 minutes at the end of import and contained no PDFs.

Both databases are only running locally with data stored on a local network drive and for now, syncing was not enabled. Therefore, it appears unlikely that security software was blocking the process while the work was only happening within the local system.

Further system info would be:
Windows 10, 20H2 (19042)
Kaspersky Endpoint Security 11.6.0.394
EndNote 20.1
Zotero 5.0.96.3


To summarize the three major issues after import from Endnote:
- extremely long import time
- 94% of entries with ​“_EndnoteXML import” tag
- only 3% of PDFs transferred


Combining this error constellation with the fact that the EndNote XML export does not contain any grouping information alludes that this rather standard EndNote library would need to be transported manually into Zotero entry by entry and group by group. Or would there be something that you suggest to try before that? In any way, I hope this info is at least helping your development process.

Kind regards
André

adamsmith

I think you are misunderstanding the _EndnoteXML tag. If you don't want that information, you can simply delete all notes with that tag at once.

manook

Hi Adam,

Thank you for your fast reply.

Probably, I misunderstood the link between the automatically attached notes and their related tag within each reference entry. And, I was biased by the statement in your kb-article: "You should review each of these notes to determine if the data need to be retained".

Now, I followed your suggestion to mark all notes of that tag which revealed an overall count of 6612 for those 1264 items. The deleting operation now seems to be called "Move Items to Trash..." which I did, followed by emptying the trash.

The most important aspect of my description above probably was the lacking transfer of PDFs. Would you have any suggestions for that and possibly also for the very slow import in regard to the transfer/fusioning of further libraries?

adamsmith

The PDF issue is mysterious. What you're seeing does seem like Zotero being unable to copy the files. (FWIW, import also sounds way slow. I'd have expected 1h for a library that size.

You have plenty of space on your hard disk?

Could you try exporting 2-3 items for which PDFs didn't transfer and import those again with sync disabled and debug output enabled. If the PDFs dot make it over again, post.the debug output ID here.
https://www.zotero.org/support/debug_output

manook

Yes, there should be enough disk space. The network drive is sized to 50GB and has 8GB available. According to my settings, the local disk should not be needed but has more than 300GB available, as well.

An isolated export of three items that did not have their PDFs imported before, worked as one would expect and within less than a minute.

adamsmith

Ah I missed the network drive part earlier. Almost certainly that's the problem. Running the Zotero db in general, and import in particular, are unusually disk-intensive operations and using a Network drive can deft cause problems. At a minimum, I'd try having the data folder on the local disk during import. You can try moving it to the network folder for regular operations (why? Don't share a DB over multiple computers with a network drive), but I would not be surprised if that caused some issues, too

manook

Hi Adam,

yes, thank you - so that solved the import speed part.

It seems it would be worth to clearly warn about that in your kb-article on importing, don't you think? (I wasn"t using some network drive over the internet but a local one)

I even copied the EndNote PDF folder to local disk before importing from there and let Zotero set up new on the same disk. The import, then, took half an hour.

BUT: the majority of PDFs still were not imported, only 70 of them.

Therefore, I also tried moving the .xml file into the "PDF" folder, i.e. one level below and on the same level as all folders that each contain a PDF. This, however, caused Zotero to import zero files.

Would it make any sense to run this import of 1264 items with debugging mode enabled? The output would be huge...

adamsmith

Ideal would be debug for the smallest import that produces at least one missing PDF

manook

Hi Adam,

while trying to find the smallest possible import for you, I found the problem myself when I was down to importing 60 items:
when I wrote the system info above, I unfortunately forgot to mention that I had installed and enabled Zotfile (immediately after installing Zotero). And this seems to cause quite a number of issues during an import like this.

After disabling Zotfile, the import worked nicely within less than 20 minutes and seemed to include all PDFs.

Concluding:
- it be helpful if your kb-article on importing from Endnote clearly addressed these two issues, i.e. (A) not using network drives (even when they are on your local network) and probably even more important (B) disable Zotfile during the import.
- Zotfile appears to be a central and one of Zotero's most important plugins and most Zotero users seem to have it installed for their file handling (at least the ones I know and heard from). Therefore, and even though it is "third-party" from your perspective, it would probably be helpful for all sides to include appropriate kb-articles on the use of Zotfile as the Zotfile page provides good information but lacks the use case perspective of your knowledge base.

Adam, Thank You for your fast and precise replies.