Interoperability with other reference managers

hardronrtap
Zotero is a wonderful tool. It has active development and a community. However, I personally feel that Zotero should be made interoperable with the other reference managers and vice versa.

1) Zotero's browser based front end is unparalleled. It blows out everyone else. None of the other reference managers come even close to it. I prefer the browser based interface and so do many users around here.

2) The beauty of desktop based managers like Bookends is their deep integration with the writing tools. While Zotero has a workable solution to citations in Word; any serious academic prefers Scrivener/Mellel than Word. Herein, it gets complicated. Zotero, at best offers a RTF Scan; not a very elegant solution.

If there is interoperability with the reference managers, it can avoid duplication of efforts on many counts.

a) Bookends can further refine and polish the existing citations (it's tag cloud is far better than Zotero's implementation)
b) Deeper integration with Scrivener and Mellel with the existing solutions in Bookends- frees up the resources in Zotero to pursue to make it even better !

I am happy to support both the reference managers (I pay for Zotero Storage, even though it doesn't sync well with Dropbox knowing fully well that the high costs subsidize the cost of development) and Bookends- an independent developer including other paid software like Scrivener and Mellel.

This is a happy scenario though; I have had a discussion with the Bookends Developer and he's willing to support the idea- I am not sure how it would work out in Zotero.

Please sound off in the comments!
  • adamsmith
    while
    any serious academic prefers Scrivener/Mellel than Word.
    is obviously nonsense, better integration for a wider range of writing tools is certainly on the radar of Zotero devs.
    There is already http://zoteromusings.wordpress.com/2013/05/06/announcing-rtfodf-scan-for-zotero/ and there's likely to be further progress in that area.
    Zotero neither will nor should ever give up on that end of its functionality (which I already consider far superior to Bookends - e.g. more citation styles, one-click switching between footnotes and in-text citations, automated journal abbreviation support, not to speak of the fact that you can co-author with people who don't happen to use a Mac...), so better interoperability wouldn't free up any resources — which doesn't mean it wouldn't be a good thing.

    That said, I'm not sure how interoperability would look like - what ideas have you discussed with the Bookends developer? One thing we'll likely do is implement Endnote XML export/import into Zotero, so that should improve the quality of data transfer between the two. But in terms of syncing the database or so - I don't really see that happening from the Zotero, even if it were possible (which I don't know). Obviously Zotero is open source and has both a (not terribly well documented) local and a (well documented) server API, so anyone is free to hack away at this.
  • dstillman
    Yeah, interoperability in terms of import/export is certainly a high priority, but interoperability in terms of being able to share a database with another tool in an ongoing fashion (if that's what you're getting at) just isn't realistic, at least as something implemented by us (though as adamsmith says, an interested party could work on it and maintain it).

    Our goal is to make Zotero the best tool possible, not to force people to use multiple similar tools, so if there's a particular area that can be improved — and compatibility with other text editors is one such area — that's what we'd devote resources to.

    That said, one facet of interoperability that I do think could possibly be improved is the ability to export something from Zotero and import it back without generating a duplicate — that is, to include an identifier in the export and handle updates (and possibly conflicts/merging) on import. Our ability to do that would probably vary by format (or at least we'd need to add custom, non-standard tags), and to do it properly would require the long-planned support for unlimited arbitrary identifiers in Zotero, not to mention a lot of complicated import logic/UI, but it could potentially open the door to the sort of ongoing data exchange you have in mind, at least on our end. That's not really anywhere on our to-do list right now, but it's something to consider long-term.
  • hardronrtap
    Better writing tools- well I became awar of Zotero way back in 2009; I think the writing tools became all too familiar around much later than that. Its a slow gradual filtering process.

    There is bound to be a difference of opinion; however, please note that I am not knocking off a product but speaking of as an invested user because my workflow (and many others) depends on Zotero critically. However, we all know that development takes time, bugs to be quashed, QA to be done and yada yada.

    I envision a product say, with a common library on the desktop that can be shared with multitude of reference managers with a common language that every product understands. Zotero remains central to the collection of the repository.

    It does not end with that. Each one addresses a core concern for their own set of users but remain common in the backend. For example, Papers has a nice UI but I find it very sedate. Zotero has a plain UI but it's highly functional. Further, for example, EndNote has some kind of an arrangement with Apple Pages. So, they would go ahead to address that core market for the users who don't rely on MS Word-where Zotero has it's own option for citation. Of course, mentioning End Note is probably heresy here because they had unsuccessfully sued Zotero :) (Not to mention that EndNote is pretty lame and useless for me since I get more and better functionality elsewhere). This is just to highlight the point.
  • dstillman
    Each one addresses a core concern for their own set of users but remain common in the backend.
    That's a nice ideal, but it just doesn't work that way. The database is intricately linked to the functionality offered by the tool — or even particular versions of the tool — as well as the specific technology used to implement it (generic SQLite, Core Data, something else...). The best you could hope for is the sort of thing I describe above: an export/import that maintained tool-specific identifiers, such that data in the tool-specific databases remained intact but could be updated via imports.
  • hardronrtap
    My concerns are more practical and philosophical. I agree with your contention that whatever you suggest works pretty much out of the box. However, I strongly feel that Zotero, by itself can take the lead and show how data inoperability can function well.

    I also feel that this is a missed opportunity; while the market for reference managers remains a niche product, this niche also consists of people who can mould opinion and have the ability to influence the opinion. At the same time, Zotero can be central to everything. Both as the backend+ Front End (as it evolves for citations etc) and the storage for the sync across the devices. I have no clue what it entails for the existing code or it how it can be implemented.

    The end result- a common interface for the users who would have a real choice to choose how each one of them fits for their usage pattern, user interface and how it integrates with the workflow patterns. It's a win win for the true open source ideals where data interoperability is the key.
  • adamsmith
    Unfortunately, philosophy doesn't write computer code ;) - you're dramatically underestimating how tightly the choice of database and database structure is tied into software.

    Just for one example - Zotero is set up for synching with a central server, across devices, allow people to collaborate in groups etc. - those things are reflected in the way its database is set up - none of this is true for Bookends, which means the databases are incompatible at a quite fundamental level, even though they use the same database system (sqlite) other reference managers don't even share that.

    We don't even have a good data exchange standard that covers things like collections or is able to accommodate the type of data required for legal or historical citations - so if we're looking for interoperability that would be something to strive for - and even on that rather moderate goal I'm not terribly optimistic.
  • jda
    I'm posting here at the request of hardronrtap. I think what he really wants is an easy way to move references between reference managers (he'd like it to be transparent, but that's unrealistic). We've developed simple protocols with other apps (e.g. Papers, Delicious Library, DevonThink, etc.) where reference information is moved to/from Bookends via AppleEvents, ideally as EndNote XML to preserve as much styled text formatting as possible as well as attachment information. If that's something the Zotero team would be interested in pursuing, we can discuss it off-forum.

    Jon
    Sonny Software
  • adamsmith
    yeah, that makes sense. Right now one problem is that Zotero doesn't support Endnote XML and you don't support Bibliontology or MODS afaik, so all of the richest exchange formats are out. We've been wanting to add that for a while, but no one has ever gotten around to it.

    Once that's done we can revisit that, though one problem is that Zotero is a cross-platform app, so Mac-only stuff would likely have happen as an external script rather than built into the software.
  • jda
    We could use something simpler, like RIS, and stash the attachment names either in a field we create or in one of the more esoteric RIS fields (the beauty of writing your own APIs is that you can adapt the available tools to fit your needs). For the vast majority of users, styled text in the reference metadata itself is not important. As for AppleScript, it could be created in code using the AppleEvent APIs (hm, perhaps you can't do that) or a script could be included as a resource (again, since I don't know how Zotero is coded, perhaps that's not an option). Having an external script is OK, too.
  • dstillman
    Just to clarify, it's not styled text that's the problem. Both RIS and BibTeX are lossy with regard to Zotero's data model (though adamsmith, aurimas, and others have done heroic work trying to improve compatibility with the semi-arbitrary RIS output of other tools). RDF and maybe MODS are the only non-lossy formats, which is why adamsmith is saying those (or perhaps EndNote XML) are a prerequisite for lossless data transfer (and again, that's only lossless in terms of bibliographic data — you still wouldn't be able to bring that data back into Zotero without creating duplicates, as things stand now). While we could use custom fields, I don't think we want to contribute to the abomination that is RIS any further — it should just go away as a format.
  • hardronrtap
    I don't have a domain expertise in the forementioned ideas. However, I believe that it is a doable option. Each one plays to the strength of the others making a uniform whole. The end users benefit in terms of cross platform access plus the spirit of open source!

    When do we see a timeline for such a project? What needs to be done for implementation? I believe that this is a long term solution. What can be done in the meantime? Just export my Zotero Library to Bookends? (I need that option with Mellel though).
  • dstillman
    What can be done in the meantime? Just export my Zotero Library to Bookends?
    To be clear, that's all we're really talking about. "Interoperability" of the sort you seem to want — automatic, continuous sharing of a library — is simply not going to happen, for the reasons adamsmith and I provided above.

    Zotero obviously already supports many export formats. With jda we're simply discussing additional export formats from Zotero (e.g., EndNote XML) or additional import formats into Bookends (RDF, MODS) that would allow one-time, one-way transfers of data without (or with less) data loss.

    If someone wants to automate a one-way push or pull from Zotero into Bookends using an existing export format (or a new one if it's implemented), it would be fairly easy to do so, given that Zotero is open-source and extensible. (There are a few Zotero plugins already that export new items automatically. Those might be a good place to start. Using the Zotero server API would also be an option, as adamsmith says above.) But that's not something we can provide. If someone wants to work on such a thing, they can post to zotero-dev with any specific technical questions.

    The thing I mentioned above about adding support for round-tripping export formats without creating duplicates could open the door for a form of two-way sync — still by using export and import, but with unique identifiers kept intact, such that existing items in Zotero could be updated — but it's not going to happen anytime soon. (It'd also be pretty complicated and prone to data loss. If you exported an item and then imported a record with the same ID that was missing some fields (because, say, the other tool didn't include them, or used different fields), you'd lose data in Zotero if it simply updated the existing item.)
  • adamsmith
    In the short to medium term yes, export from Zotero to RIS, import into Bookends - should work reasonably well. I'd like to get Endnote XML running within 6 months, but no promises.

    btw.
    ...the spirit of open source!
    doesn't mean what you think it means. Data portability, which this is really about, is only tangentially related to free and open source software and its various spirits. Zotero is open source. Bookends, Mellel, and Scrivener aren't.
  • hardronrtap
    No issues. I honestly don't want to debate the philosophical nitty gritty of the definitions It is an honorable goal to work towards data portability.

    I understand that there are "no promises" but atleast some form of commitment is welcome that it is being considered on the roadmap. Let's look forward for that.

    I am aware of the closed source applications; however, I am happy to support independent software developers who have envisioned the same aspect in a different way, just that I do that to Zotero.

    Cheers!
  • adamsmith
    Endnote XML export/import is now available. This will only marginally improve this, but it's a step. (It's still a pretty bad data format, btw., with a lot of the decision based on whatever Endnote decides to put into one field or the other, but oh well).
Sign In or Register to comment.