Business plan / competitiveness

tzucker
So I'm relatively new to Zotero from EndNote and I'm not completely convinced it's a better alternative. I got to thinking about it last night and this morning.

Here I am someone who is willing to try this new software out, someone that is very comfortable with the way EndNote works, yet in the end Zotero makes it difficult for me to switch. Let me explain...

1. Even if I truly want to switch, I can't unless I can export my current database (that means ALL fields) and then import those records into Zotero with at least 99.9% confidence (preferably 100%) that all information will be both exported and imported. Yet not only is the export convoluted at best, the import is just as bad. Given that EndNote has an extremely large market share (~90% according to this article - which is mostly about Zotero - http://www.bsos.umd.edu/gvpt/syoder/research/muldrowyoderps.pdf), wouldn't it make sense to either create a single application that does this import/export or to modify one of the export filters that EndNote already supports and call it Zotero, making sure it exports all fields in all records and change the import in Zotero to import through this filter?

2. Yes there is more. The ability to group saved searches is not something minor, at least for me. EndNote allows me to group searches in logical sets. For instance I have a group of searches that search my database for each separate author. I have over 150 authors that I am directly concerned with and need to track. So the ability to expand this search group and collapse it is essential. I also have a group of searches just for research programs of our center. There are 5 programs, but I need to search the references for each program 3 times (e.g., program 1 inter programmatic, program 1 intra programmatic, program 1 inter and intra programmatic), so that is 15 searches. Again the ability to expand and collapse this group is essential. There are more groups, but you get the idea. And yes I realize I can do some of these things with tags, but not all.

3. I'm glad to hear that adding a scrollbar to the advanced search window is at least in the queue to work on. This feature is not merely a prettying up of the GUI, it speaks to one's ability to work with the database. If I need to download and even buy a secondary program in order to get Zotero to work, that not only doesn't make sense, but it makes me not want to use the software. The GiMeSpace software paid version or trial of the paid version makes windows users able to do large searches (read definite benefit over EndNote's 10 item limit) practical, but the software interferes with other software I use. And, dare I mention again, since EndNote seems to have so great a market share ANYTHING Zotero does to make it easier to use it's software is a better business plan.

4. I'm not sure about this one. Ever time I click on a search in Zotero it seems like it is redoing the search, while in EndNote it doesn't seem that way at all. Let me give an example. First EndNote, which I might add displays the number of references found by the search next to the search name (very handy), if I click on a already saved search or Smart Group in EndNote it shows the results almost immediately. On the other hand, if I click on a saved search in Zotero, it takes a little time before the results show up.

If Zotero truly wants to make inroads into the bibliographic world, these and more need to be taken into account. Yes the search feature alone makes Zotero a superior piece of software, but that won't get it off the ground if you can't import your data into the program to use it. If the creators of Zotero really want to win over a portion of EndNote users these points (especially the import/export) need to be addressed sooner rather than later.

Finally, I'd like to make it clear that I love the idea of Zotero, but some of the execution leaves way for improvement. After I finish with this 1000 page grant due next month, I will look more closely at this product.
  • adamsmith
    While these thoughts are appreciated, I don't think you realize the degree to which Zotero has "made inroads" in the bibliographic world.

    Look at the websites of major University libraries. Talk to graduate students, among whom Zotero and Mendeley are almost certainly more popular than Endnote. As that trend continues, more and more libraries will decide that it may not be worth it to spend the ~10kUS$/year for an Endnote site license (for a medium/large University) and then huge effort they have to make in training users for software with a good ol' 1980s feel. Consider why Endnote sued Zotero in the first place - I'm pretty sure they're hugely concerned about the growing competition from Zotero and others (Mendeley, Papers2, etc.)

    All of which is not to say that better export/import wouldn't be nice (though Endnote really does do everything in its power to make that hard - e.g. your idea with a Zotero Endnote filter would almost certainly be considered a violation of their EULA by Thomson Reuter - another reason why it may be a good idea to reconsider the software you're using) and that additional features and improvements to Zotero are not desirable.

    But do give the people behind Zotero some credit. When Zotero started, there was a virtual monopoly on ref management with Endnote and Refworks, with a small faction of hard scientist using bibtex/jabref etc. Of course people developing Zotero, writing grants, etc. have spend time - lots of time - thinking about ways to position Zotero in that marketplace and if you look at both the growth of Zotero and its impact on the industry at large (e.g. by shaping expectations on functionality, by providing open source citation styles for software like Mendeley and Papers2) the impact has been dramatic.
  • tzucker
    edited August 30, 2012
    Points taken. I know here at my uni they are just starting to develop a program to help Zotero users. But it is in its' infancy. It turns out I know more about Zotero and its' workings than the reference librarian in charge of our Zotero project (and I've been looking at this for only a few days).

    That said, I do give a lot of credit to the developers and the development team, I may seem more critical because I would really like this to succeed and these few items I mentioned seem like minor things (except the import/export) to do (I'm not a programmer, so excuse me if that is incorrect) for the benefit they would provide.

    Not to belabor the point, but it seems to me without a good way to export and import the data, any plan to win over EndNote users will eventually be relatively ineffective. That is just my opinion.
  • maras
    edited August 30, 2012
    I totally agree with importance of import from EndNote to Zotero. That should be one of the main tasks of Zotero I think.

    And even documentation of it is quite poor. In example, there is persistent problem with keywords import, as all the keywords of one EndNote reference are imported as one tag to Zotero. That is nonsense. Yes, this is fault of Endnote, but who cares? And translator of RIS (and other translators) can't overcome that.
    But why this simple solution: http://forums.zotero.org/discussion/8062/endnote-9-keyword-import/#Item_10 isn't published there: http://www.zotero.org/support/kb/importing_records_from_endnote Why I should spend my time in search with possibility to find nothing? And this solution works, is simple and understandable to all users without knowledge of scripting, coding etc.

    Other frequent problem is that not all fields are exported from EndNote by default. But that can be overcomed by replacing all the reference types with generic in EndNote RIS, ReferBiblX or whatever other export style. And that should be included in documentation of importing to Zotero too, I think.

    And yes, import. There is no fully working translator, RIS or ReferBiblX. Latter is abandoned, as I can understand. It doesn't import even language field that is common to all reference types. And no reaction to my proposal include it to translator. OK, RIS is the default format. Why it isn't working properly: http://forums.zotero.org/discussion/22844/ris-import-problem/ And who cares about specifications? Not ordinar user. And not EndNote especially. Why they should create format that is easily imported to Zotero? Yes, they can change mapping of fields, but we can too ;). If there are some fields that can't be mapped to fields of Zotero (because of abesence of appropriate fields) lets map all them to extra field. Main thing is that all exported fields MUST appear in Zotero (and 7 custom too).

    That is how ordinary user see the problem. He don't care about formats, specifications etc. He wants a result. And I too love idea of Zotero and I think, that it is better than EndNote. But as ordinary user I should spent too much time and energy only to export/import from EN to Zotero (and there are other problems, as style creation etc...). And majority of potential users consider such costs as too big for moving to Zotero, I think.
  • tzucker
    edited August 30, 2012
    I believe maras has put it very well, in the end it is what the average user perceives as the difficulty. If something is extremely difficult to code, and it adds minor functionality then the end user generally sees it as minor.

    The more time one has to spend in order to make the Zotero library work like the EndNote library the less likely it is that the user will use Zotero.
  • adamsmith
    I totally agree with importance of import from EndNote to Zotero. That should be one of the main tasks of Zotero I think.
    Important, yes, main task no. The main task of Zotero is to provide users with a better reference manager. While I think it's great to get people to switch, convincing new users that Zotero is a better product is as important in the short run and more important in the long run. And there is a trade-off to developer time.

    But if you want to improve
    http://www.zotero.org/support/kb/importing_records_from_endnote
    you're more than welcome to - it's a wiki and that page is entirely user generated, see here for how:
    http://www.zotero.org/support/dev/documentation
    providing custom scripts or translators is certainly welcome
    OK, RIS is the default format. Why it isn't working properly
    That thread is from earlier this spring and we agree that RIS fields should be added. Coding doesn't happen on its own, though, and again, is made harder because there are no well documented standards.
    ReferBiblX. Latter is abandoned, as I can understand. It doesn't import even language field that is common to all reference types.
    remind me where that is? Wasn't that a standards issue, again? I realize you're going to blast me on these standards, but the whole reason we're in this mess is that Endnote never cared about standards. I think Zotero has a responsibility to not follow down that same path.
    And not EndNote especially. Why they should create format that is easily imported to Zotero?
    because that's what responsible software does. That's e.g. what google does: http://www.dataliberation.org/ . That's what users should expect from their software.

    Standards/specifications matter
    a) because Endnote isn't the only software exporting RIS - there are also RefManager, Procite, multiple online databases etc.
    and
    b) because principles matter. It's just not viable for Zotero to reverse engineer whatever EndNote decides to do with its export formats whenever they do it.
    If there are some fields that can't be mapped to fields of Zotero (because of abesence of appropriate fields) lets map all them to extra field. Main thing is that all exported fields MUST appear in Zotero (and 7 custom too).
    I don't see how importing custom Endnote fields into Zotero can be done in a useful way (the impossibility to exchange custom fields is one of the reason Zotero doesn't have them).
    What would you do with them? Pack them all into Extra? That hardly seems useful, it certainly wouldn't be for tzucker's case above. Turn them all into notes? Maybe, but those can't be cited and they're not displayed easily. etc.
  • maras
    Am I blind, or there isn't mapping of language field in RIS translator? There: https://github.com/aurimasv/translators/wiki/RIS-Tag-Map I can see that field LA is mapped to language in ALL EndNote referece types. So why there isn't mapping in translator?
  • adamsmith
    @maras - please don't shout.
  • adamsmith
    edited August 30, 2012
    So why there isn't mapping in translator?
    because the page you link to is part of Aurimas' larger project to improve the RIS translator which isn't finished.
  • tzucker
    As an aside, although this doesn't sound too useful to this discussion, mapping all the custom fields to tags would work for me. It is what I will have to do anyway when I switch (which I am more and more convinced I will do after our site visit - Jan 2013).

    This works prety well for me in my tests. I will have to investigate the "Related" tab more. It seems that this may be a way to categorize my citations as well.
  • adamsmith
    @tzucker - right, and I think that's typical: People will need custom fields for different purposes and that will affect how they should import. Short of a very substantial import app I don't see how that could be done easily.
    Obviously that'd be a nice tool to have, but just to give you a sense of dimensions:
    Mendeley and Columbia university are co-developing a GUI graphical CSL style editor (which will obviously also work for Zotero, because CSL is an open standard ;-)). If anything, automating the XML output required for styles is easier than doing that for the javascript that import translators require. The grant amount for that project is 1million US$.
  • maras
    edited August 30, 2012
    "Important, yes, main task no."

    One of the main tasks, not main ;). This is big difference.

    OK, I'll contribute to wiki :).

    "Coding doesn't happen on its own, though, and again, is made harder because there are no well documented standards."

    But not a language field ;).

    "remind me where that is? Wasn't that a standards issue, again?"

    There: http://forums.zotero.org/discussion/24747/import-from-endnote-all-the-fields-even-custom-via-referbiblx-my-experience/
    "I added several rules of conversion in "var fieldMap = {":

    6:"numberOfVolumes",
    G:"language",
    "!":"shortTitle"

    and some other.

    Such configuration suited for me, everyone can do own. Maybe language variable can be included to official Zotero conversion file? Is it not common to all reference types?"

    I don't think that is standarts issue. And that's OK - comment was very long etc. :).

    "the whole reason we're in this mess is that Endnote never cared about standards. I think Zotero has a responsibility to not follow down that same path."

    Yes, I really understand your position. It is, how to say, honourable. And yes, EN isn't responsible software. But why users should suffer for that? Users who want transit to Zotero? So I didn't agree with:

    "It's just not viable for Zotero to reverse engineer whatever EndNote decides to do with its export formats whenever they do it."

    Because users are the goal, I think, not software. And that doesn't change Zotero design principles, only the way of interaction with EN. It's self-defence ;).

    "I don't see how importing custom Endnote fields into Zotero can be done in a useful way (the impossibility to exchange custom fields is one of the reason Zotero doesn't have them).
    What would you do with them? Pack them all into Extra? That hardly seems useful, it certainly wouldn't be for tzucker's case above. Turn them all into notes? Maybe, but those can't be cited and they're not displayed easily. etc."

    Well, my position is such: if user used custom fields, they were needful to him. So, that data must be transferred. BUT there is no way to display it properly as you said, so let's pack it to extra field (not to notes, but that is my oppinion and I do that in my version of translator). User can see that all the data is transferred. If he needs some specific mapping, it can be done by modification of translator (tzuker's case, because there is no other way). Main point is that all the data MUST be transferred.

    I agree with that:
    "convincing new users that Zotero is a better product is as important in the short run and more important in the long run."

    But there is Mendelay...

    Am I citing in a proper way? I don't see frames in preview.
  • maras
    Yes, I am citing inproperly, but after that: http://www.w3schools.com/tags/tag_phrase_elements.asp

    And I don't shouting.

    because the page you link to is part of Aurimas' larger project to improve the RIS translator which isn't finished.

    Sorry, but I don't understand There are some really complicated fields and maybe a lot of time to decide how to map them in translator. And all easy to map fields will stay unmapped until then, as I can understand. No, I don't understand such a policy.
  • adamsmith
    wrt shouting: http://email.about.com/od/netiquettetips/qt/Writing-In-All-Caps-Is-Like-Shouting.htm (cf. your MUST and ALL above)

    using italics is perfectly fine for citations - if you want to use the boxes I use you can use <blockquote> tags.
    Sorry, but I don't understand There are some really complicated fields and maybe a lot of time to decide how to map them in translator. And all easy to map fields will stay unmapped until then, as I can understand. No, I don't understand such a policy.
    there is no "policy," but it often makes sense to not spend time on small incremental updates when someone is working on a larger re-write of the code (in this case:
    https://github.com/zotero/translators/pull/431 )
    since all those incremental changes would be superseded once the new translator code goes live.
  • maras
    edited August 30, 2012
    I see. For me shouting is when several words, sentence or more is in uppercase. How emphasize word without shouting?

    Thank you for that tag.

    I understand. But language field isn't imported for more than 3 years. And that was one of the reasons why I abandoned Zotero till now after testing Zotero and registering to this forum in 2009. And language field was one of the reasons, why I chose Zotero, not Mendelay (it didn't use it).
    I think that's typical: People will need custom fields for different purposes and that will affect how they should import. Short of a very substantial import app I don't see how that could be done easily.
    But that isn't reason for not importing custom fields at all. And I think, that there is no way for proper importing of them at all, because they are used in very strange and even weird manners. Developers only must decide where they should be imported and there is only two alternatives: extra field and notes (as tags are very specific mapping, I think). I prefer extra field as there I can see all thing imported at a glance, without going to notes.

    (Edit. Maybe I am looking very angry, but I'm not. I really wanted to say those things prior to tzuker's post. That is my humble opinion and it is such. Maybe it will contribute to better Zotero :). Truth cam out of discussions and it lies in the middle.)
  • adamsmith
    How emphasize word without shouting?
    italics in html mode or _underscores_ (for underline) in plain text mode.

    The initial reason for not having the language tag (and some others like call number) was that it wasn't documented anywhere. Now that we have that information there is no principled reason against including it - for me it's just a question of how far along Aurimas is - as you can see if you look at his code, there is some rather elaborate mapping in there (including, of course, language) and it's just not worth it to provide an intermediate solutions - obviously if he's too caught up in other work and will need a long time to finish that up we should implement a quick stop-gap solution in the meantime.
  • maras
    or _underscores_ (for underline)
    Only now I understand what means underscores in your comments :). There are different internet etiquette in different countries.

    Well, EN can easily obstruct importing to Zotero by changing documentation. And their lawsuit tells me, that they can do such a things.
    Short of a very substantial import app I don't see how that could be done easily.
    Obviously that'd be a nice tool to have, but just to give you a sense of dimensions:
    Mendeley and Columbia university are co-developing a GUI graphical CSL style editor [...] The grant amount for that project is 1million US$.
    Such a tool can be rather uncomplicated (I am not a programmer, so my knowledge is limited). There are not so many EN fields that haven't equivalent in Zotero. As a starter it can let choose only how to map EN custom (+ undocumented?) fields. And that's enough if there will be good RIS translator in near future, I think.
  • adamsmith
    yeah, maybe you're right - I guess it depends on how user friendly such a tool would need to be - but a simple text-guided tool that just inserts some code into one place in a js file could be feasible.

    That could be done by pretty much anyone, though
  • noksagt
    edited August 30, 2012
    But language field isn't imported for more than 3 years.
    The initial reason for not having the language tag (and some others like call number) was that it wasn't documented anywhere.
    Indeed, it has been less than a year & I think less than about six months since the language field was documented for RIS.

    (EDIT: replaced broken internet archive link with bit.ly link)
  • maras
    Is there any board "Volunteers needed to create/help..." in Zotero site? Because good ideas can be lost in such a mess of everything in forum.
    This can be the way of attracting volunteers to do not essential but wanted things.
  • maras
    edited August 30, 2012
    @noksagt That was from a viewpoint of ordinary user ;). No offence to developers.
  • adamsmith
    There is this. It's not complete, but some starts. Beyond that, a lot of volunteers just grab the stuff they feel needs to be done.
    https://github.com/zotero/zotero/issues?direction=desc&labels=Help+Welcome&page=1&sort=created&state=open
  • noksagt
    There: http://forums.zotero.org/discussion/24747/import-from-endnote-all-the-fields-even-custom-via-referbiblx-my-experience/
    "I added several rules of conversion...

    Such configuration suited for me, everyone can do own. Maybe language variable can be included to official Zotero conversion file? Is it not common to all reference types?"

    I don't think that is standarts issue. And that's OK - comment was very long etc. :).
    So a few things here.
    • this thread is less than a month old
    • the first post doesn't fully document your changes
    • There are standards issues to worry about.
    This format is described at http://sti15.com/bib/formats/refer.html and doesn't include the fields you enumerate. Do other programs besides Endnote use these (and, for that matter, how long have they been in the Endnote translator?)? I don't know of any other programs that import these...Also: there are other programs that emit REFER/BIBIX. Are there potential tag collisions for undocumented tags?
  • daniel_l_sweden
    edited August 30, 2012
    As an Endnote user having tested Zotero for a few days I must say I completely agree with tzucker and maras; making the transition from Endnote to Zotero as smooth as possible is crucial if established time EndNote users are to make the switch. Actually, I was just about to post on this matter as I found the migration process rather frustrating, especially regarding PDFs (the 'internal-pdf' and relative path problem).

    Even so, today I suggested Zotero to a colleague of mine that was very frustrated with Endnote as it crashed several times a day on her computer. However, after I showed her the instructions regarding getting endnote pdfs into Zotero she said it would have to wait, she'd rather just continue restarting her computer two or three times a day...

    While I as a former software engineer had no problem following the instructions once I found them she just plain gave up when she saw there was some code on the wiki page: http://www.zotero.org/support/kb/exporting_from_endnote_with_pdfs

    I think having to look up a wiki, let alone running scripts to edit the data before importing data exported from the software dominating the market is _way_ too much trouble for most people. And this 'internal-pdf'-problem has been around for years from what I gather reading this forum.

    Not being able to automatically get PDFs from EndNote in a reliable manner into Zotero is a deal breaker if you got an extensive EndNote library, built up during years of research.

    Anyway, while I have never worked with XUL I have worked extensively with Javascript. So tonight I made a few small changes to the RIS translator and also a very small change to the client core. I can now import EndNote-produced RIS _with PDFs_ without first having to modify the exported data in any way.

    I will post my changes to the dev list or maybe this forum to see if it is anything that could be of interest.
  • adamsmith
    (I really hope people spend as much time telling Endnote about this. The fact that a free and open software project with a tiny staff and a bunch of active volunteers gets told that it _has_ to do do certain things whereas the corporate, proprietary, and expensive Endnote somehow gets a pass at being terrible is really a rather sad state of affairs).
    I will post my changes to the dev list or maybe this forum to see if it is anything that could be of interest.
    that'd be great. We do support a bunch of different pdf import formats -including software specific ones e.g. for Sente - in the bibtex translator, but IIRC correctly the [internal-pdf] path in endnote RIS output doesn't contain the filepath to those PDFs and I don't quite see how we can get that - but if you have a good way to do that that'd be great.
  • maras
    edited August 30, 2012
    @adamsmith I see. But my proposal would be to include link from first page of Zotero (and/or forum?) site to that page although. I searched quite a lot, but never saw this page.

    @noksagt
    this thread is less than a month old
    Well, fbennet spoiled me, I think :).
    the first post doesn't fully document your changes
    I didn't include all the translator because it is very, how to say, particular, only for my needs and didn't suit for general purpose. I demonstrated only that parts, which could be useful for other and for general needs (such as proposals to improve translator).
    But I can post the translator if there is need for it.
    There are standards issues to worry about.

    This format is described at http://sti15.com/bib/formats/refer.html and doesn't include the fields you enumerate. Do other programs besides Endnote use these (and, for that matter, how long have they been in the Endnote translator?)? I don't know of any other programs that import these...Also: there are other programs that emit REFER/BIBIX. Are there potential tag collisions for undocumented tags?
    I don't care and I won't care about standards. And I don't have time (and interest) to read all the documentation. And that is why I can't answer to your questions.
    I found purpose of fields by empirical way: exported test references via that format. Voila, I know which symbols are mapped to which fields. So to say, practical documentation :).
    What regards the last question. Why could not exist translator "ReferBibX(EndNote)" that imports only EN references? Besides, EN style have such a name - "EndNote export". So they can do what they want, I think ;).
    And this is proposal for Aurimas too. As I understand, his RIS translator encompass EN, ProCite etc. and that is one of obstacles to write it. But why it should encompass all if there can be two, three or more separate translators for each software? Especially if EN and other TR software doesn't observe their own standards and different software behaves differently.

    And that is point of view of average user without knowledge of documentation etc. ;) With best regards and full respect for developers of such a good piece of software.
  • daniel_l_sweden
    edited August 30, 2012
    @adamsmith:
    Done, see here:
    http://forums.zotero.org/discussion/25117/endnote-ris-with-attached-pdfs-code-suggestion/

    It has its weaknesses being dependent on RIS file name and placement, but it works.
  • maras
    @adamsmith
    I know several stories about people complaining to EndNote concerning some of EN features or lack of them and that they didn't got any answer not to say improvements.
    The fact that a free and open software project with a tiny staff and a bunch of active volunteers gets told that it _has_ to do do certain things whereas the corporate, proprietary, and expensive Endnote somehow gets a pass at being terrible is really a rather sad state of affairs
    You named the reason why ;). Because it is corporate, propietary, expensive and till recent years was nearly monopoly in citation managers market. TR is too big and too fat.
  • adamsmith
    edited August 30, 2012
    see. But my proposal would be to include link from first page of Zotero (and/or forum?) site to that page although. I searched quite a lot, but never saw this page.
    There is a "Getting Involved" link right at the top of every page:
    http://www.zotero.org/getinvolved/
    That doesn't lead to the help-wanted tickets, but people with the abilities to help with Zotero client development are experienced enough to find the github issue tracker by themselves.
    For people with some technical abilities, who aren't experienced coders, people like yourself (and mostly me, for that matter), that page probably contains the more salient suggestions for useful places to volunteer - translator development and maintenance e.g. is a good place because you can go quite far with pretty basic javascript.
Sign In or Register to comment.