Zotero vs. EndNote

Happy new year (it's got to be better than the last one...)

I have long wanted to leave EndNote, for various reasons. But, I am very heavily invested in that program: My library is about 65K records, accumulated over 30+ years (starting in 1989 with EndNote Version 2), with many records customized for various purposes, many customize styles and import and export scripts, etc. Since then, I’ve used EndNote with MS Word for hundreds of research papers and other documents, most with collaborators who use EndNote. I lead a research group that shares my master database, using devices that vary in OS, age, etc.

I expect to encounter lots of obstacles when/if I make the switch. So far, I’ve concluded it would not be worth the trouble. But, while setting up my (fully maxed out) new MacBook Pro, I was shocked at the interfaces changes in the latest version of EndNote (v20.2). I think it went from the best interface among the competitors to really awful. (Specific example: the rows in the main window listing the records are effectively “double spaced” and there appears to be no option to condense this.)

So, before I settle into making do with EndNote 20, I’m giving the alternatives another look. I quickly narrowed the search to Zotero, but I have not been able to convince myself that Zotero is (yet) ready to replace EndNote in my (group’s) workflow. I’ve been browsing various sources for insight into this, and this list seemed promising. But, I haven’t found the detailed advanced-user level comparison of pros and cons that I am looking for.

Hopefully, some of you with extensive experience using both programs (and porting between them) will be motivated to document some of the specific similarities and differences you’ve encountered. A few examples of the issues I am worried about:

(1) Is Zotero ready for very large databases? (I followed some instructions for migrating my EN database to Zotero, which mostly worked OK, but now it takes 3-4 minutes for Zotero to open the file and finish loading “tags”. That’s not promising.)
(2) Done right, can the EN database be transferred with 99.9% accuracy? Including a pretty wide range of reference types, data fields, math symbols, diacriticals, etc.
(3) Assuming there are errors to correct, does Zotero have sufficiently powerful global search-replace tools?
(4) When it comes time to work with old MS Word files formatted with EndNote, can the citation markers be converted from EndNote to Zotero with minimal errors?
(5) How well can Zotero use the record numbers that EN used to rely on?
(6) Assuming there’s a way to import custom styles from EndNote, what advanced features get lost (like linking annotations to fields so that an empty field does not leave an orphan annotation)

I could list many more. But, I am most interested in issues that I am not anticipating that some of you might know about. I am sure there are plenty. Thanks.
  • (1) I use Zotero with a database with over 40k items. It works fine. You can also import some of your libraries to Zotero Group libraries and keep them online and not synced to the local program.

    (2) Yes, Zotero has very good Endnote import. User-custom fields cannot be directly imported by default, but you could customize the import script if needed. Unmapped fields are stored in a note to preserve the data.

    (3) Zotero has a full JavaScript api that can be used for search and replace. There are example scripts for batch editing in the support documentation, and we can help you out if you have questions. There is not currently a graphical search and replacement menu.

    (4) Endnote citations in Word cannot be read or converted by Zotero. They would need to be replaced. When I moved from Endnote, I used Zotero with new documents and a few in progress documents with few citations, but did not bother converting old archival documents. Due to past aggressive legal action by Endnote, I don’t expect Zotero to add additional Endnote conversion features.

    (5) I don’t know what you mean. Zotero does not use any Endote internal database identifiers. Items imported to Zotero from Endnote are not connected to the Endnote library in any way.

    (6) Zotero uses CSL citation styles. This system is in my view the most robust and accurate system for writing citation styles. Ensuring that you don’t get spurious punctuation with empty fields, for example, is much easier in CSL than with Endnote’s syntax. CSL supports over 9000 journals and style guides out of the box, and you can write custom styles as well or request new styles here on the forums. There is a visual CSL editor if you aren’t comfortable working with XML code. Endnote styles cannot be read by Zotero or converted to CSL (again due to the past aggressive legal actions by Endnote).

    Beyond these concerns, let me also say from my experience that Zotero’s web import is vastly superior and easier than Endnote’s and Zotero’s Group library sharing is much better than Endnote’s (Groups are why I switched myself years ago).

    If you switch and have questions during your transition, please feel free to ask here. The lead Zotero developers read every thread and there is a large community of active users to provide help. If you have a very large library you want to import, I suggest doing a few folders/collections at a time just to ensure you don’t run into any computer memory issues. The best export format to use is Endnote XML. See here for details https://www.zotero.org/support/kb/endnote_import
  • Re: 2 -- I don't think 99.9% is realistic migrating between any two tools that don't share a standardized database structure. Yes, Zotero's Endnote XML import is good, but I don't even think Endnote puts 99.9% of data into the export. I'd say 95% or so is realistic
  • edited January 3, 2022
    I started using bibliography management software when Reference Manager was CP/M software (pre-DOS, in the early 1980s) and was using EndNote in the later 1980s when it was a Niles Mac-only package (before it was ported to DOS/Windows). I converted to Zotero while in the middle of writing my doctoral thesis almost 15 years ago.

    That was before Zotero had an EndNote import utility. Even this very early version of Zotero seemed to me better than the EndNote (Windows) user interface. I was able to get all of my Reference Manager and EndNotes records into Zotero to my satisfaction.
  • Thanks to bwiernik for their very helpful reply. That level of detail resolved a few key questions for me [e.g., for (2), what happens to imported data when fields don't match].

    As I had hoped, the details also uncovered a few new issues. For example, from (4), "replacing" EndNote reference markers with Zotero reference markers presumably is effective and straight-forward (just tedious), but can Zotero markers be edited directly? I resolved a lot of issues by directly editing the hidden part of EndNote markers over the years.

    FWIW, the lack of further input on this thread leaves me reluctant to commit to full migration. Reluctantly, I will focus on making do with EndNote awhile longer.
  • Unclear what you mean with 4: Zotero fields are connected to the items that are cited and are updated when you update those items in Zotero. While at a very technical level it's possible to update the underlying hidden data in the Word field, it's quite technical and very fragile, so no one ever does that.

    You can manually edit references inserted with Zotero, but they will no longer update -- that's a bigger issue with some citation styles (esp. numeric ones) than others, but generally not ideal, of course -- but the only way this will work since Zotero has no way of telling with edits to overwrite and which to keep.
  • FWIW, the lack of further input on this thread leaves me reluctant to commit to full migration. Reluctantly, I will focus on making do with EndNote awhile longer.
    Also a bit puzzled by this -- what did you expect? Dozens of people writing long answers about this? You got one detailed answer and a couple of additional pointers -- that seems quite generous in terms of responses to a very general question, no?
  • If you have any other specific questions, we’re happy to answer further
  • I am a long-time Zotero user who has converted several research groups over from Endnote, and would never go back. I think Zotero is superior in many ways, including a) collaboration, b) managing your actual files, c) reduced risks of corruption - I have experienced Endnote's unreliable database structure and corruption firsthand multiple times, d) support - even though you pay for Endnote support, my experience is that the Zotero community (including those who have posted replies) is faster and more helpful than the paid employees at Endnote. Oh and e) entering data (especially grey literature on the web). The new Zotero beta version is also far superior in managing the note-taking process.

    Having said all that, I think migrating from Endnote to Zotero does cause pain points and the benefits/disadvantages really depend upon each case, and also on how much pain you are willing to go through for the advantages you perceive. While Zotero has no difficulty with large libraries and has a much better database structure than Endnote, from your description I think the project of converting the whole database for you may not be worth it.

    The main reasons for this is that you sounds like you have a fair degree of customisation in your Endnote database - custom reference types and custom styles - which are not easy for non-coders to replicate. Zotero won't convert Endnote custom styles (see your point 6) and the process of creating custom styles or adding custom fields is not as straightforward as it is in Endnote, so if you rely on these a lot then that will cause some additional pain.

    The other reason is that it doesn't sound like the benefits of Zotero are worth it for you. If your main issue is the interface, rather than (say) easier collaboration with others or a better research workflow, then I expect the advantages of Zotero are not going to be worth the disadvantages for you. If you collaborate with other Endnote users, there's also a pain point in getting them on board with Zotero - my experience has been that they need support in that transition.

    I would agree with @bwiernik that if you want to try it out, just start out with a small number of references rather than migrate the entire library. As it's free, you can certainly just play with a small dataset and see how that goes - you might find that there are other features of Zotero that might make a fuller transition work for you, and there are definitely ways to overcome the pain points if you decide to go that way.
  • Thanks, some of this is promising. (I really want to be convinced.)

    With respect to the database structure, I agree that EndNote's is fragile. I've had to rebuild mine many times, for various reasons. But, my 65K record EndNote database opens immediately. The version of it that I have imported into Zotero takes about 5 seconds to launch and another 15 seconds to load items and tags. Is this normal, say because Zotero compiles the keywords/tags every time from scratch? I might be able to live with that, but I can't help be get the impression that EndNote handles large databases better than Zotero. Also, is there any absolute size limit to the database?

    Note that my database contains many abstracts and lots of key wording, but the records are not linked to external PDF or image files. If it did, the database size/performance/mobility issues would be even more challenging. It would be nice to link the database records to my (completely separate) archive of PDF files, but I would only consider doing that if it could use my existing collection and not make it's own copy of everything like EndNote does. Can Zotero do that?

    Adamsmith confirmed that you can do after that fact editing of Zotero formatted Word documents. I assumed that would be the case, and that reformatting would wipe out those edits. Regarding editing the underlying/hidden citation markers within Word, he said "While at a very technical level it's possible to update the underlying hidden data in the Word field, it's quite technical and very fragile, so no one ever does that." Maybe, but this isn't hard and can be useful with EndNote markers. The syntax for unformatted citation markers is pretty simple, for example: {Nurmi, 2006 #25614;Tratnyek, 2006 #25034}. Is it really more fragile with Zotero markers?

    This leads back to the one of my original specific examples: the issue of EndNote record numbers (25614 and 25034 above). ENs reliance on these has created all sorts of problems. Recent versions of EN allow formatting with markers from multiple libraries, and inspecting the unformatted markers (as above) shows record numbers corresponding to the source database, but no indication of how it knows which database the record number refers to. This is a detail I haven't figured out, and has made me avoid mixing databases in manuscripts. Is Zotero's handling of these issues really more transparent and robust?

    Almost all of what I do is with internal and external collaborators, so the details of how EN and Zotero compare is another aspect that I am interested in. Clearly, the folks who have replied think this is a strength of Zotero, but I can't yet think of any specific reasons for this (assuming everyone already has access to a licensed copy of EndNote). Maybe a specific example would be this: suppose you have a manuscript where the primary author is using EndNote and a secondary author using Zotero, or you have the reverse, which would allow the more efficient workflow?

    BTW, in their first reply, bwiernik mentioned the litigiousness of EndNote's recent owners. I am aware of that, in fact that was a big part of way I've been following Zotero's progress since it began to look like a viable alternative.
  • It would be nice to link the database records to my (completely separate) archive of PDF files, but I would only consider doing that if it could use my existing collection and not make it's own copy of everything like EndNote does. Can Zotero do that?
    I’m not sure what you mean by this. Zotero can either attach links to files stored on your computer or can import the files and manage them itself.
  • edited January 10, 2022
    Zotero items are identified with a combination of library and item ID numbers. These aren’t generally shown to users. They are robust and cannot be confused or crossed. The Zotero word processor plugin has a button in the Add Citation interface to take a user to the item in the appropriate library.
  • With respect to sharing, there are several specific strengths of Zotero. Zotero groups have much more functionality than Endnote (at least when I last checked)—Endnote groups could not be synced to the desktop app and were difficult to use in Word documents. Zotero groups are fully integrated into the desktop app.

    Additionally, because of the fragile reference numbers in Endnote you mentioned, I had a lot of trouble writing documents collaboratively with Endnote. If two people tried to insert items from their libraries, I had frequent errors. In Zotero, the full item citation data is embedded in the document, so others can add citations from their own libraries and Zotero can smoothly integrate all of them. A group library isn’t even required. The only limits there are (1) if an item needs to be updated (eg, to fix a typo in the data) and it’s from a user’s private library, that person would need to update, and (2) occasionally two people might cite their own copies of the same item, which Zotero would treat as separate items and add things like 2020a, 2020b, etc.—Zotero helps to prevent that by putting already-cited items to the top of search results, and it’s easy to consolidate if it does happen.
  • (1) Loading time of Zotero: I suspect what you're seeing might be caused by the syncing. By default, Zotero is set up to automatically sync (which is great if you are collaborating, and I generally leave this on), but you can choose to switch this default off you can sync only when you want to (of course, you have to remember to do so, but if you're not actively collaborating with anyone this won't be an issue).

    Personally, I have set up my computer so Zotero boots up after my other applications in the background - I'm rarely in a position to cite immediately, so to be honest I don't really notice the loading time at all.

    Regarding syncing of files, this does slow things down, but if you make sure all collaborators change the sync so you 'download files as needed', this doesn't have much impact at all on the speed.

    I'm not aware of an absolute size limit (other than a file storage limit if you're using that). I certainly have 20+ years of data in there without an issue.

    (2) Linking your archive to your PDFs - for me this is a real pain point of Endnote and massive advantage of Zotero. Once you get used to having immediate access to your PDFs, it's hard to go back! Zotero's notes also link you directly to the page of your annotated PDF, which is another thing that Endnote doesn't do - if you have to do a lot of footnote checking (as I have had the misfortune to need to do), this is a huge timesaver.

    Many years ago, I did have a Dropbox archive for all my PDFs and used a super-useful extension for Zotero called Zotfile (http://zotfile.com/#features) to manage this, so it is certainly possible to store it this way. However, it comes with risk - if you change where those files are stored, you'll break the link. I haven't done this for years, as in the end I found it easier to use Zotero to store the files (Zotero does a much better job of managing the files than I do), but it certainly is possible. See https://www.zotero.org/support/sync#alternative_syncing_solutions.

    (3) In general, the problem with editing Word markers is with Word. I have lost count of the times that people have accidentally edited Endnote unformatted markers and corrupted the document accidentally. It's just more trouble than it's worth. In my experience if there's anything too customised, I just highlight the footnote marker and edit it after I've broken the link to Zotero. Word is a fragile beast and it just reduces a further risk of corruption.

    (4) Absolutely agree that Endnote record numbers are the source of all kinds of evils. As @bwiernik says, Zotero gets it right in that it is very difficult for an end-user to ever see the identifiers, which massively reduces the risk of corruption. It's a pretty basic feature of all databases that Endnote just gets completely wrong in my opinion.

    (5) Collaboration - Zotero was built with collaboration in mind. Endnote is bearable if you have just a few collaborators, and ideally on the same network with access to IT support. But there's no doubt that Zotero's syncing is far superior than Endnote's - I've never had any group libraries corrupt on Zotero (Word documents, sure, but also less than with Endnote).

    If you're into more collaboration than just citing footnotes, Zotero is also better because you can both annotate on the same files or write notes (and create notes from your annotations) within Zotero. The latest version of Zotero (in beta) will make that even better by allowing you to dump all your Zotero notes into a Word document, with citations,

    Permissions is often a problem - the assumption that everyone has a licensed copy of Endnote is, in this day of a casualised academic workforce, is less and less likely. I've come in and out of universities over my career, as have many of my collaborators, and with Zotero my library just stays with me. As it's free, it's also usually easier to get others on board to change to Zotero (but in those cases, I've usually been the champion for change, with training etc).

    I wouldn't suggest ever using both Endnote or Zotero to collaborate - I can't imagine that would end well. In general, we've picked one or the other. Usually, if we're collaborating on a specific project, the database is small enough for me just to import it into the one we've agreed to use. In my experience, if your collaborators already use Endnote, then it's hard for them to move because they're used to it.

    In terms of 'getting people on board', if your users have never used either Endnote or Zotero, universally I've found it easier (and more pleasant) for people to start with Zotero. There's a bit of a learning curve, but for the basic functionality its user interface is just a bit more standard and the 'one-click' functionality of getting data in and then accessing the PDFs means it becomes part of the research workflow, rather than just that annoying bit of software when you want to cite.

    What I would say about Zotero versus Endnote is that Zotero is no longer just a citation tool. It's a research tool and 'brain' - and the new version of Zotero that is coming soon completes the research circle, by allowing me to annotate and extract annotations directly from Zotero and then import those notes into Word. It's when I've emphasised that aspect of Zotero that I've 'converted' many Endnote users away from Endnote (and it's why I would never go back unless forced to!)
  • I'm still watching this thread, and experimenting with Zotero. Some further thoughts:

    The load time issue is still serious for me. I am not synching the database online. Simply opening the app takes almost 10 seconds, then almost 1 minute to finish "loading items" and over 3 minutes to finish "loading tags" (with spinning beach-ball most of that time). I've closed and reopened the app multiple times, and there's been no improvement. Yes, my database is very large, but my computer is very fast (brand new MacBook Pro M1 Max) and EndNote opens its version of my database instantly.

    I followed some instructions and believe I imported the database correctly, but I haven't tried alternative ways to get that data into Zotero. Don't want to spend time on that unless there's a promising hypothesis to test. It seems like a big part of the load time issue is that Zotero recompiles the tags from scratch each time. This is normal?

    (Once they are fully loaded) I've spent some time experimenting with tags. This aspect is very different than EndNote, and I see how they could be very useful. But, (i) the tags Zotero compiles are quite messy and would take a ton of effort to clean and optimize. I haven't investigated yet what the options for this might be. I presume they are coming mainly from the keywords field in my EndNote database, but maybe also abstracts? What exactly is going on? (ii) Before I got a clue how Tags worked, I tried shift-clicking combinations, editing them, etc. and managed to make Zotero hang (not crash, I had to force the app to quit.)

    Regarding linking to PDFs, joycekwc clearly likes a workflow that relies heavily on Zotero for managing and sharing these file. I can see the benefits of that. But, I'm pretty sure that I will continue to the opposite approach of maintaining PDFs of papers completely independently. The main reason is that my workflow relies very heavily on the built in Spotlight search on my Mac. It provides reliable, flexible, and very quick full text searching any/all drives linked to my computer. If I start using Zotero's (or EndNote's) cloud to store PDFs, I would loose that integrated search capability. This is why I asked earlier about whether one could point Zotero to a local folder for linked PDFs. I think the answer was a qualified yes, so I plan to investigate this further.

    In addition of sharing PDFs, joycekwc advocates using Zotero as a collaboration tool for sharing notes, groups, etc. Again, I see some of the advantages of this workflow, but (i) I suspect the differences between current versions of EndNote and Zotero are not that great, and (ii) I haven't adopted this in my workflow so far. The main reason is that I find there is too much variation in how students, postdocs, other colleague/collaborators want to work. For example, some really want to annotate PDFs using Mendeley (long ago, I tried using Papers for that), others are invested in GoodNotes, etc. The variations are endless. So, I've given up on the idea of convincing others to do what I do and aim to be flexible enough to accommodate what they do. This relates back to the overall question of EndNote vs. Zotero: just about every collaborator I've worked with recently is using EndNote, so it's hard to see how switching to Zotero wouldn't make me less able to adapt to their workflows.

    Finally, back to the citation markers in Word. The replies above on this topic are very interesting to me. I think the overall conclusion is: (i) EndNote uses a marker mechanism that is partly visible (and editable) as "unformatted" citation markers, where you can see the EndNote record numbers, but also has hidden other formation more deeply, but (ii) Zotero hides all that information more deeply, and provides no mechanism for reviewing or direct editing any of it. Just now, I see that using Zotero in Word does produce Field Codes that are easy to toggle to unformatted. For example, I now see {"citationID":"ziOIg7rd" ...., but that ID isn't viewable in EndNote?
  • Just picking out two things I noticed:
    If I start using Zotero's (or EndNote's) cloud to store PDFs, I would loose that integrated search capability.
    That's not the case. Zotero still stores files locally (they are synced to the cloud, not stored exclusively there) and you can still search them with spotlight.

    Also for both tags & performance after import from Endnote, see this part from the importing from Endnote page:
    Note, if Zotero encounters any fields in the EndNote XML data that it does not support (e.g., custom fields, author address, author affiliation), it will add these data to a note attached to the imported item. These notes will be tagged with “_EndnoteXML import”. If the import adds many of these notes, Zotero's performance can be negatively impacted. You should review each of these notes to determine if the data need to be retained (you can also move these data to the Zotero “Extra” field). Any unnecessary notes can be deleted. Additionally, you should check these notes to determine if any data could be migrated to proper Zotero fields (this is particularly important if you were using EndNote fields in non-standard ways).
    If you added an extra note with every imported item, and have a tag associated with 65k items, that would have a massive performance impact.

  • Just getting around to thanking adamsmith for his 1/19/22 reply. It was right on the mark: (1) exactly what I wanted to hear about storing local PDF files, and (2) yes, too many tags is almost certainly the root of the performance issue I have been stuck on.

    The Zotero database that I imported from exported EndNote data does in fact contain a note on almost every record, due to mismatching of fields. Most of the notes say "The following values have no corresponding Zotero field:"

    So, the solution to (2) is clear, but the amount of time it will take me to develop an export-import workflow that results in few or no mismatched field is daunting. I may get to this eventually, but can't afford that much time for it right now.
  • edited February 28, 2022
    An easy workaround for the performance problem would be to just delete the _EndnoteXML import tag, and set up a saved search for "Item Type is Note" + "Note contains The following values have no corresponding Zotero field".

    How would you want to get to a workflow that results in few or no mismatched fields? By fixing things in Endnote before export? If you have some idea of where the mismatched data should show up, a javascript runnable in Zotero to make it so given the data in these notes wouldn't be too hard to create.
Sign In or Register to comment.