Upgrade StorageImporting EndNote Library: import time, “_EndnoteXML import” tag and PDF availability
Dear Zotero-Team and fellow Zotero users,
it took about 53 hours to import an EndNote library to an empty Zotero library fully following your recommendations (https://www.zotero.org/support/kb/endnote_import).
EndNote had created an XML-export file with 7MB and the EndNote PDF folder had a size of 1.75GB.
This EndNote library is a comparably small and rather recent one originally created within EndNote X8 about five years ago and now running with EndNote 20.1. Still, I followed your recommendations and checked for figure attachements, database integrity, etc., beforehand.
Zotero reported 1264 imported items (corresponding to available entries in Endnote). According to your kb-remarks, the tag check with “_EndnoteXML import” revealed 1191 tagged items from those 1264 imported ones and it seems, this would already explain why Zotero took so long for the import. 1191 is too many to go through individually as you describe in your knowledgebase.
For your info, besides using the field "research notes" within EndNote, no other fields of those reference entries where modified after importing them. Therefore, all these tagged fields were created by EndNote automatically (it appears, often author addresses).
From those 1264 imported items (that mostly originated from PubMed), 1186 had PDF-attachments within EndNote. Unfortunately the majority of these attachements could not be opened in Zotero ("The attached file could not be found in the following path..." along this kb-article of yours: https://www.zotero.org/support/kb/files_not_syncing). In Zotero's data storage folder, filtering the entries according to their creation time revealed that only 40 folders were created during the first 43 hours and these were containing PDFs in Zotero. The remaining creation of 1348 folders happened within less than 20 minutes at the end of import and contained no PDFs.
Both databases are only running locally with data stored on a local network drive and for now, syncing was not enabled. Therefore, it appears unlikely that security software was blocking the process while the work was only happening within the local system.
Further system info would be:
Windows 10, 20H2 (19042)
Kaspersky Endpoint Security 11.6.0.394
EndNote 20.1
Zotero 5.0.96.3
To summarize the three major issues after import from Endnote:
- extremely long import time
- 94% of entries with “_EndnoteXML import” tag
- only 3% of PDFs transferred
Combining this error constellation with the fact that the EndNote XML export does not contain any grouping information alludes that this rather standard EndNote library would need to be transported manually into Zotero entry by entry and group by group. Or would there be something that you suggest to try before that? In any way, I hope this info is at least helping your development process.
Kind regards
André
it took about 53 hours to import an EndNote library to an empty Zotero library fully following your recommendations (https://www.zotero.org/support/kb/endnote_import).
EndNote had created an XML-export file with 7MB and the EndNote PDF folder had a size of 1.75GB.
This EndNote library is a comparably small and rather recent one originally created within EndNote X8 about five years ago and now running with EndNote 20.1. Still, I followed your recommendations and checked for figure attachements, database integrity, etc., beforehand.
Zotero reported 1264 imported items (corresponding to available entries in Endnote). According to your kb-remarks, the tag check with “_EndnoteXML import” revealed 1191 tagged items from those 1264 imported ones and it seems, this would already explain why Zotero took so long for the import. 1191 is too many to go through individually as you describe in your knowledgebase.
For your info, besides using the field "research notes" within EndNote, no other fields of those reference entries where modified after importing them. Therefore, all these tagged fields were created by EndNote automatically (it appears, often author addresses).
From those 1264 imported items (that mostly originated from PubMed), 1186 had PDF-attachments within EndNote. Unfortunately the majority of these attachements could not be opened in Zotero ("The attached file could not be found in the following path..." along this kb-article of yours: https://www.zotero.org/support/kb/files_not_syncing). In Zotero's data storage folder, filtering the entries according to their creation time revealed that only 40 folders were created during the first 43 hours and these were containing PDFs in Zotero. The remaining creation of 1348 folders happened within less than 20 minutes at the end of import and contained no PDFs.
Both databases are only running locally with data stored on a local network drive and for now, syncing was not enabled. Therefore, it appears unlikely that security software was blocking the process while the work was only happening within the local system.
Further system info would be:
Windows 10, 20H2 (19042)
Kaspersky Endpoint Security 11.6.0.394
EndNote 20.1
Zotero 5.0.96.3
To summarize the three major issues after import from Endnote:
- extremely long import time
- 94% of entries with “_EndnoteXML import” tag
- only 3% of PDFs transferred
Combining this error constellation with the fact that the EndNote XML export does not contain any grouping information alludes that this rather standard EndNote library would need to be transported manually into Zotero entry by entry and group by group. Or would there be something that you suggest to try before that? In any way, I hope this info is at least helping your development process.
Kind regards
André
Thank you for your fast reply.
Probably, I misunderstood the link between the automatically attached notes and their related tag within each reference entry. And, I was biased by the statement in your kb-article: "You should review each of these notes to determine if the data need to be retained".
Now, I followed your suggestion to mark all notes of that tag which revealed an overall count of 6612 for those 1264 items. The deleting operation now seems to be called "Move Items to Trash..." which I did, followed by emptying the trash.
The most important aspect of my description above probably was the lacking transfer of PDFs. Would you have any suggestions for that and possibly also for the very slow import in regard to the transfer/fusioning of further libraries?
You have plenty of space on your hard disk?
Could you try exporting 2-3 items for which PDFs didn't transfer and import those again with sync disabled and debug output enabled. If the PDFs dot make it over again, post.the debug output ID here.
https://www.zotero.org/support/debug_output
An isolated export of three items that did not have their PDFs imported before, worked as one would expect and within less than a minute.
yes, thank you - so that solved the import speed part.
It seems it would be worth to clearly warn about that in your kb-article on importing, don't you think? (I wasn"t using some network drive over the internet but a local one)
I even copied the EndNote PDF folder to local disk before importing from there and let Zotero set up new on the same disk. The import, then, took half an hour.
BUT: the majority of PDFs still were not imported, only 70 of them.
Therefore, I also tried moving the .xml file into the "PDF" folder, i.e. one level below and on the same level as all folders that each contain a PDF. This, however, caused Zotero to import zero files.
Would it make any sense to run this import of 1264 items with debugging mode enabled? The output would be huge...
while trying to find the smallest possible import for you, I found the problem myself when I was down to importing 60 items:
when I wrote the system info above, I unfortunately forgot to mention that I had installed and enabled Zotfile (immediately after installing Zotero). And this seems to cause quite a number of issues during an import like this.
After disabling Zotfile, the import worked nicely within less than 20 minutes and seemed to include all PDFs.
Concluding:
- it be helpful if your kb-article on importing from Endnote clearly addressed these two issues, i.e. (A) not using network drives (even when they are on your local network) and probably even more important (B) disable Zotfile during the import.
- Zotfile appears to be a central and one of Zotero's most important plugins and most Zotero users seem to have it installed for their file handling (at least the ones I know and heard from). Therefore, and even though it is "third-party" from your perspective, it would probably be helpful for all sides to include appropriate kb-articles on the use of Zotfile as the Zotfile page provides good information but lacks the use case perspective of your knowledge base.
Adam, Thank You for your fast and precise replies.