Upgrade StorageWhat do app developers need to do to "keep the door open" for full Zotero integration?
MS Word, LibreOffice, and Google Docs enjoy first-class integration with Zotero.
Some other text editors, like Zettlr, have a second-class type of integration. Still good, but not as fantastic as what I'm calling first-class integration.
Every other text editor can work with Zotero via the RTF/ODF Scan plugin, which is wonderfully enabling because of the freedom of choice it allows, but still isn't on the same level as first or second-class integration.
In the past, I recall seeing a discussion about how Scrivener is unlikely to get Zotero integration because it was built in a way that simply isn't compatible. Scrivener or Zotero would need to be rebuilt from scratch, which isn't going to happen. It's a shame because it seems that a bit of foresight could've made sure that impasse was avoided, but oh well. Too late now. But it isn't too late for all other projects!
If an app developer is interested in keeping the door open for first-class integration with Zotero, what do they need to know? How can they set themselves up to make that as easy as possible later?
I know that this question might require a fairly comprehensive answer, which is more effort than the typical forum answer. But my hope is that this post might become a resource to which interested projects can refer. With a sufficiently clear and thorough answer, they might even be able to do 100% of the work involved in achieving first-class integration. So, writing adequate guidance is an investment that would pay huge dividends as more and more text editors achieve first-class Zotero integration without addition work from Zotero devs. (But however much guidance you can give, even if it's not very much, could still yield impressive dividends.)
Thank you for considering!
Some other text editors, like Zettlr, have a second-class type of integration. Still good, but not as fantastic as what I'm calling first-class integration.
Every other text editor can work with Zotero via the RTF/ODF Scan plugin, which is wonderfully enabling because of the freedom of choice it allows, but still isn't on the same level as first or second-class integration.
In the past, I recall seeing a discussion about how Scrivener is unlikely to get Zotero integration because it was built in a way that simply isn't compatible. Scrivener or Zotero would need to be rebuilt from scratch, which isn't going to happen. It's a shame because it seems that a bit of foresight could've made sure that impasse was avoided, but oh well. Too late now. But it isn't too late for all other projects!
If an app developer is interested in keeping the door open for first-class integration with Zotero, what do they need to know? How can they set themselves up to make that as easy as possible later?
I know that this question might require a fairly comprehensive answer, which is more effort than the typical forum answer. But my hope is that this post might become a resource to which interested projects can refer. With a sufficiently clear and thorough answer, they might even be able to do 100% of the work involved in achieving first-class integration. So, writing adequate guidance is an investment that would pay huge dividends as more and more text editors achieve first-class Zotero integration without addition work from Zotero devs. (But however much guidance you can give, even if it's not very much, could still yield impressive dividends.)
Thank you for considering!
If the app developer wants someone else to do the work, all this app developer has to do is give us extension points into the app, documentation on how to use them, and explain how it would benefit enough of the Zotero users that it would fit the Zotero mission to spend their resources on it. That's all. This explains the MS Word, LibreOffice, and Google Docs integrations -- Google, MS and the LO devs made extension points & docs available, and Zotero built the world-class integration on top of this. If you have trouble convincing on the userbase part -- plenty of people on these forums will be happy to get you started on a Zotero plugin that gets you the desired behavior. Zotero is *crazy* extendible.
There is absolutely nothing in place that would prevent absolutely any app developer to build a world-class integration with Zotero. You can, in principle, get the *exact* Zotero behavior in a markdown editor, without any changes or plugin to Zotero, since the protocol that the Word-addin and others speak to Zotero is available to any and all integration, but you'd have to do the work.
WRT Scrivener, Literature & Latte show no interest in doing the work, and they do not offer any extension points, so the people who can do it won't, and the people who would do it can't. And if you ask on their Stockholm-Syndrome userboards, that's exactly the way the most vocal Scrivener users like it. Scrivener is perfect, and therefore anything that isn't already in Scrivener must therefore not be there by design, and they'll have no adulteration of their heavenly editor.
Markdown is a markup language—it’s intended that things like headings, text formatting, etc. have coded placeholders until the final document is rendered. Inserting citation keys in the pandoc syntax is how referencing is intended to be used with Markdown, where final formatting occurs when the document is processed by pandoc.
If an app wants to have buttons to call up the Zotero window, BBT provides all of the hooks they would need (e.g., see the rbbt package for R/RStudio). With that in mind,
Word processors like Word, LibreOffice, Google Docs are a different beast. There, the expectation is that the document is always in a publishable final rendered state. That requires a more active plugin, which requires an API in the word processor program for Zotero to be able to hook into. Currently, only Word, LibreOffice, and Google Docs have such an API. Other somewhat popular word processor programs, such as Apple Pages and WPS don’t have any way for Zotero to be able to read the document, insert and edit text, or embed citation data. So there isn’t a way for Zotero to make any integration that doesn’t involve going through another program like pandoc, Word, or LibreOffice.
The state of Scrivener isn’t as dire as you describe. Nothing would need to be rewritten in any program. What would need to change is for Scrivener to have an API like Word or Google Docs. They have no such API. The Scrivener developers have in the past been very hostile to the idea of supporting citations—they seem to regard their app as more for creative writing and aren’t interested in supporting scholarly citation. In general, the Scrivener developers have been opposed to adding an API to their program to permit third party plugins. If there were such an API, given Scrivener’s enthusiastic fan base, I suspect someone would write a plugin to connect Zotero to it. But without a programming API in Scrivener, that’s just not possible.
Okay, maybe I need to back up a few steps in order to figure out a better way to frame my question.
1. Do you agree that there are basically three tiers of integration with Zotero? Specifically, do you agree that MS Word, LibreOffice, and Google Docs have the highest tier of integration; apps like Zettlr, Sublime Text, and Atom are on the next tier; and all other writing software is on the third tier thanks to the OTF/RDF-Scan?
2. Do you agree that there are meaningful differences between the UX provided at these different tiers?
3. Even if it's already technically possible for app developers to do the work to achieve the functionality that defines the first tier (as opposed to expecting someone else, like the Zotero project, to do that work for them), do you think there are early decisions could make achieving that functionality easier or harder? To put this question another way, is there any advice you can offer that would be helpful for developers who want to make apps that provides the UX of the first tier as early as possible?
--------------------------
@bwiernik
I'm not implying that the UX of second-tier integration is bad. I think it's good! It's one of Zettlr's best selling points! I wish Obsidian had this functionality! (Hopefully it will in the future, but that's beside the point.)
However, I think there are important differences between the UX of first-tier and second-tier integration. Calling up the Zotero box isn't the only difference. Having the citation right in the text as if it had manually been entered (instead of any kind of code that will be scanned and replaced when the file is exported) and having the citation's style easily and automatically changeable throughout the document are the other two features of first-tier integration that I think of when I consider the differences between the UX of using Zotero with MS Word, LibreOffice, and Google Docs vs the UX of using Zotero with Zettlr, Sublime Text, and Atom.
I'm grateful for the ones that at least offer second-tier integration, but I still wish more programs offered first-tier integration. My belief is that if it was easier to implement, then more programs would have it. One of the ways to make things like this easier is to provide guidance on how to do it. That's the purpose of my post.
1. Yes, but that's because a lot more effort went into the MS Word, LibreOffice, and Google Docs side. There's nothing on the Zotero side that makes something possible there that is not equally possible in Zettlr, Sublime Text, and Atom. They don't have to rely on OTF/RDF-Scan. It's just that nobody has put in the work to implement an integration with the same polish as the Zotero crew has done for Word, LO, GDocs.
2. Yep, but not because anything in Zotero only makes it possible for Word et al. It's not that it can't be done for Zettlr et al, it's just not been done.
3. "do you think there are early decisions could make achieving that functionality easier or harder?" Nope. Everything that's needed to make a first-tier integration is present. It's just a non-trivial amount of work. As to advice: get cracking, and ask questions here, or on zotero-dev.
edit: and anywhere that someone would say "it can't be done" that means the editor in question doesn't allow it. Zotero is ready and has always been ready for new first-tier integrations.
I'm not arguing with you. I'm just saying your question "what would a developer need to create a first-tier integration" is fully answered with "time".
So is literally impossible with a plaintext (i.e. markdown, RST, LaTeX) editor if you also want the citations to live update. The whole point of those is that all information is in the plain text, so you can't (in Emiliano's sense of "the editor doesn't allow it") store stuff in hidden fields like Word/LO/Gdocs allow.
FWIW, there could be live rendering into a preview window (Overleaf, e.g., does that nicely), and to the extent that counts, it's completely independent of Zotero.
Ditto for selecting the citation style. Since Zotero isn't actually doing the citation formatting in markdown-based apps (they mostly rely on pandoc for very good reasons), this is entirely up to the app developer but would be quite easy to put into a UX -- it's just that most people writing in markdown don't mind putting the name of a citation style into a file or command so no one has bothered.
Thank you for your answers. I'm not trying to argue either. My goal is only to make it easy as possible for developers to add Zotero integration to their apps. It's not that it isn't possible. It's that the current level of difficulty is preventing it from being widely done. When I look at the forums for different projects, I see that there's pretty clear interest. There's no avoiding that it'll take work, but I think the economics are fairly simple: if it was easier, then it would be done more. So, the easier it can become, the better.
For most tasks, some ways to make it easier are to plan ahead for the task and get advice from people who have knowledge and experience. If there is no particular way to plan ahead for Zotero integration and no advice to give, then that's a bit surprising, but I don't know enough about programming to question it. It's guess it's just difficult for me to accept that nothing at all can be done to make it easier. No "Start here" guide, no checklist of things to consider, no recommendations about how to format code, no survey of developers to find out what's holding them up…
Bah! That's just the way my brain works. I'm an interaction designer, so "How might X be made easier?" is a question that my brain is hardwired to ask. Apologies if the insistence is frustrating. It's just that one of the last tools I can use in this situation is approaching the goal from different angles. But I recognize that I'm not actually on the Zotero team, so I'll peel myself away.
----------
@adamsmith
It might be a red herring from a programmer's perspective, but it isn't a red herring from a user's perspective. The UX is very different.
Also, I think that it's worth pointing out that there's a whole spectrum between WYSIWYG editors like LibreOffice and plaintext code editors like Atom. Take Typora as one example. It's a markdown editor that "stores text in hidden fields" in exactly the way I imagine a first-tier integration could work.
Zettlr is another example of a markdown editor that hides some of the markup. It explicitly aims to be somewhere between WYSIWYG and plaintext. "Zettlr strikes a balance between the distracting layout of WYSIWYG editors and raw Markdown syntax. Even more: If you don't like some of these elements, you can toggle which elements Zettlr will render in the settings!"
No, not all markdown editors would be interested in this functionality, but I'm not talking about forcing Zotero integration onto any project. I'm talking about all the projects that do think it'd be nice to have Zotero integration, preferably the first-tier kind.
It's also worth pointing out that my goal wasn't only to make it easier for markdown editor projects. Manuscripts is an open source Scrivener-like WYSIWYG editor that has second-tier Zotero integration. That's awesome, but it has the equivalent of first-tier integration with other a few reference managers. I think I saw an old post where emilianoeheyns said he was in contact with their project, so maybe it's on the way. Since Manuscripts is made by the same people who made Papers, I completely get why it'd have that kind of first-tier integration with Papers. But it also has that kind of integration with a few others. I'd think that Zotero would be at the top of their list. A cynical interpretation might be that they're intentionally holding back in order to give Papers a competitive advantage, but if that were the case, then why offer that level of integration with other reference managers? Isn't it more likely that they did the other integrations first because they were easier (and thus less costly)?
I don't know. I think maybe I've taken this conversation as far as it can go. My goal was to make it a little easier for other projects to achieve first-tier Zotero integration. I've tried approaching this from a few different directions, but I think it's time for me to let this go.
Thanks for reading this, and for all the other work you do. Sorry this conversation didn't bear fruit.
Anybody who wants Zotero integration for their project can have it. It can work like it does in MS Word, or it can work like it does in Zettlr. There's no need to do anything weird or special to get it working. There are ample extension points in Zotero, and the devs are happy to answer questions.
The zotero crew maintains 3 separate "first tier" integrations. If there was anything that made integrations easier to create just generally, wouldn't they already have done this if only for their own sake? Ergo, the current state of the zotero side is what is general about integrations, and they've made that as easy to use as they could. This infrastructure is available to anyone to use, no barriers in place. Manuscripts can do absolutely anything that the zotero-maintained integrations do, if they choose to spend the time building it.
Nobody is forcing anything on anyone. At this point the discussion is starting to get actually ridiculous. Projects who want this integration will not only find no barriers but plenty of help, but they're not exactly beating down the door. You'd have to ask the manuscripts people why they're not interested, not us. We can't read their minds. Yeah, I was in contact with manuscripts, long ago, but they never got started building their end, even when I was going to do most of the work of the integration.
If you want to move anything forward, pick an app you think that will want this, design an UX for it, and we'll gladly provide the technical know-how needed to build it. I don't know what else we could do. I don't currently understand the shape an answer from my end would have to be for you to feel you've been taken seriously.
Edit: our messages crossed. Yes, that is exactly what I was trying to explain.
Anyone who's seen my work knows that if I had to live off my UX skills, I'd have starved ages ago; I've worked with UXers a fair bit though and it always catches me off-guard how different the thought processes are from the "my
Xtool of choice? vi" crowd that I'm part of. Coop can be rewarding but it's hard to find a shared vocabulary.Secondly, most of the work in providing Zotero integration plugin is not in talking to Zotero, but in implementing the actual logic. Zotero requires 2 specific functionalities from the document format:
1. You must be able to associate metadata with parts of the text that are Zotero citations. Zotero needs to know which parts of the document are its citations and be able to store metadata required to update them. In Word it's Fields, in LibreOffice it's ReferenceMarks and in Google Docs it's madness-fueled NamedRanges and links.
2. You must be able to store text location non-specific, i.e. global metadata in the document. This is where Zotero stores things like the citation style used by the document and other prefs.
Depending on how close to the metal of the original doc editor a plugin developer works, accessing these 2 functionalities of a doc editor can be super easy and convenient, which makes implementing the plugin also relatively easy, or enragingly frustrating and difficult. We would love to provide first-class integration for other popular document editing options, but we simply don't have enough time to do that given how complicated it is and how much work it takes to simply maintain the 3 (actually 4, since mac word and win word do not share code) existing integrations.
Yeah, I hear you. I've become acutely aware of how differently us design folk think. I've been convinced for a long, long time that the differences in our thinking and culture are the main reasons why designers have historically been so rare in the engineer-dominated open source community. We might always be a bit enigmatic to each other, but I hope we all get better at communicating. I want to believe that I'm slowly getting better at talking to programmers, at least. (I'm not quickly getting better at it, that's for sure!)
Anyway, thanks for your patience, and sorry again for the frustration.
-------------
@adomasven
That is very helpful! The next time I see a developer talk about how they like the idea of Zotero integration, I'll be sure to refer them to your post. And it's good to know that TextMaker is proving that "first-class" integration can be done entirely without needing support from Zotero devs. Thank you for sharing your insights!