Zotero 5.0 - what will remain same, what will change, availability

Gurdas_Sandhu

We've been waiting for Zotero 5.0 and it is said to be a major update, specially for management of local database and online sync system. Beyond that, very little is known.

I think Zotero dev team has a responsibility to their ardent users to provide more details on this upcoming major update. I have been waiting to launch a group, but have held back because it will be somewhat disconcerting to new users to have a major update shortly after I convince them to move to Zotero.

So, my requests:

- What will be the major updates in Zotero 5.0?
- Will anything change for end-users, specifically, for groups?
- Give a tentative timeframe when 5.0 will be available.

Please keep us informed. We love the work you are doing, but we also want to be prepared and be able to plan things at our end.

References:
https://juris-m.github.io/about/
https://groups.google.com/forum/#!topic/zotero-dev/efFuMEqLarY

adamsmith

For a start, 5.0 will start as a beta / preview, so anyone interested will be able to take a look before it is widely distributed (and anyone _really_ interested can already run it checked out from github).

Beyond that, the biggest changes are under the hood and shouldn't affect you much beyond better performance, particular for large syncs. The most visible changes, I believe, are the ability to add RSS feeds and a "My Publications" folder that integrates with institutional repositories (I know it does with Sufia 7 currently, not sure if anything else).

Longer term, Zotero will almost certainly switch away from the Firefox platform (i.e. just provide a connector as it does for Chrome now) and switch to a different platform for Standalone, which will entail more visible changes, but that's still a good bit out.

Gurdas_Sandhu

Thanks, adamsmith.

I am most interested in these issues wrt Zotero 5.0 (Z5):

1. Will we have batch editing of fields such as author name or journal name?

2. Will it be significantly faster when inserting citations in large Word documents?

3. Will it allow in-line citations in Powerpoint?

4. Any major updates to Tags (I love them!)?

5. Will add-ins - such as Zotfile, Zutilo, Prevent Duplicates - continue to work?

6. Will Groups allow "attach link to file" functionality?

7. Can we store Advanced Searches in a nested way, like we do with Collections?

8. Any improvements to Zotero Reports?

9. Ability to preview PDFs?

10. DOI field for more item types?

11. In-built functionality to prevent duplicates (refer to 5)

12. Easier methods to share/transfer items between "My Library" and group libraries?

adamsmith

Likely in the first instance of 5.0 no to all of these except 5: Zotfile and Zutilo will almost certainly work (they require modifications, but the developers have signalled they're working on them). Prevent Duplicates will likely stop working, though.

Gurdas_Sandhu

No to all except 5? That's disappointing.

May be the first release is all about laying the foundation for enabling faster and better updates in future. But really, its been a while since Zotero has had major new features added and I was hoping 5.0 will get us some of these most wanted features. I get a feeling that Zotero is falling behind or at least losing its lead .... Am I wrong?

Whats the good news going forward?

Rintze

Maybe the first release is all about laying the foundation for enabling faster and better updates in future.

That's a large part of it, yes:

"Overall, the changes in 5.0 should provide for a much smoother user
experience and lay the groundwork for some major, long-awaited changes,
such as an improved data model."
https://groups.google.com/d/msg/zotero-dev/efFuMEqLarY/jauqguFSEwAJ

In particular, 5.0 should feel much snappier and perform better with syncing large libraries.

bwiernik

As far as I have seen, almost every part of the Zotero code was changed in a minor to major way as part of 5.0, so this is really a major update, even if most of the changes are to the back end.

dstillman

5.0 certainly has taken longer than we had hoped, but I'd add that a big part of the work — in addition to the architectural changes to allow for future data model changes, improve responsiveness, and better support syncing of very large libraries — has been building up a large test suite covering most of the code. Zotero has gotten so large and complex that it was becoming difficult to make changes without breaking things. The test suite should greatly improve Zotero's stability and allow us to make major changes without breaking existing functionality. Between the architectural work and the test suite, I expect the pace of new features to pick up quite a bit shortly after 5.0 comes out.

Gurdas_Sandhu

Rintze, bwiernik, and Dan: Thank you for the responses.

I understand the need to rethink the architecture and Dan commented elsewhere that the test suite has over 500 (unit?) tests. That's impressive. But please also understand where users are coming from. Some of the features I mentioned have been requested for over five years, so it just feels like a big let down that these are going to take even longer.

Zotero is a gifted horse and as the saying goes you don't look at one in the mouth. But, many of us are heavily invested in Zotero and some, including me, champion for it among peers in our scientific community.

Sync speed has been mentioned multiple times, but how many users need that above batch editing or speed of adding citations in large documents? May be some of these are not possible without the architectural change, but surely some were possible. If a poll of most requested new features has not been done, I would suggest doing one. Give us the wanted features first.

I truly hope we will see new features soon.

adamsmith

no. prioritizing desirable frontend feature over infrastructure and backend stuff may provide instant gratification (and many commercial products like doing it for just those reasons), but it's a recipe for creating bad software that's unsustainable, outdated, and unstable in the long term.

If we did a user survey, of course "create a comprehensive test suite" wouldn't make the top ten. But that makes it no less necessary. Same for changing the sync infrastructure.

Gurdas_Sandhu

A major update not having most (if any) of the top features that have been requested for five or more years isn't really what I'd define as sacrificing "instant gratification."

There is always a balance between end-user features and backend updates. I think Zotero 5.0 got the balance wrong. And we can agree to disagree on that.

LiborA

@gurdas: if I see another threads I hope upgrade to 5.0 is infrastructural and upgrade to 5.1 take new functionality. Yes I agree with you that development period of Zotero is extremly long, but I also must agree with Adam - stability is priority! I will have stable software without some features rather than software able to do everythink but loosing data or be able to damage my files etc. (a lot of people can talking long stories about functionality and compatibility of MS Word)

adamsmith

Sure we can agree to disagree, but I think, while your disappointment is understanding -- and shared by many of us (recall that pretty much anyone but Dan here is primarily an academic in some capacity and only works on Zotero because we find it useful) -- it is based on a poor understanding of what developing software involves.

Take the test suite. Obviously not something that end users would appreciate much per se. But don't you think that a lot of people would assign a high priority to "Should Zotero be able to make updates that work reliably and without breaking the software temporarily?"

Or the sync infrastructure. No, I don't think faster syncing is a terribly high priority, nor, I think has it been for Zotero devs. But if we phrase it like "Should Zotero work to make sync more reliable and also make it possible to add new item types and add fields to existing item types in the process" I think we'd get a lot more votes.

The other thing is, as bwiernik notes, a lot of the backend changes affected nearly every aspect of the code. Which of course means that any new features based on 4.0 would also have to be re-written for 5.0. I could come up with a question for that, too, but you get the idea.

(Also, I'd invite you to look at the Changelog's since Zotero 3.0, which isn't even 5 years old, to get a sense of the amount of features actually added over the last 5 years; this idea that Zotero hasn't added new features for 5 years is a bit puzzling to me:
https://www.zotero.org/support/3.0_changelog
https://www.zotero.org/support/4.0_changelog

Of course there are many features that have been requested for years. And I'd love for them to have happened yesterday. But features being unfulfilled for years is hardly unique to Zotero. The top requested features on Mendeley's Uservoice are between 3 and 8 years old.

dstillman

Or the sync infrastructure. No, I don't think faster syncing is a terribly high priority, nor, I think has it been for Zotero devs. But if we phrase it like "Should Zotero work to make sync more reliable and also make it possible to add new item types and add fields to existing item types in the process" I think we'd get a lot more votes.

Or, "As people build up very large Zotero libraries, should they still be able to sync them at all, or should we abandon them to focus on new features?"

Or, "Do you want Zotero developers to work on improving Zotero or spend their time trying to keep an aging sync architecture from crashing as usage increases?"

A key goal of the 5.0 changes is to actually give us the time to work on new features without spending so much time on maintenance of client and server architectures that are 7-10 years old. Put another way, if 5.0 existed, 5.0 wouldn't have taken so long.

If we had known how long the new architecture would take, sure, we probably would've implemented some new features on the old architecture, but 1) as adamsmith says, that would've required rewriting them later, 2) they would've been buggier, and 3) that would've pushed back changes that some people desperately need (e.g., better large-library syncing, new item types and fields).

I'd also suggest not overthinking version numbers. 5.0 will be called 5.0 because it includes major architectural changes. There could be a 5.0.5, or a 5.1, or a 6.0 with lots of new features a few months after that. (Recall that 4.0.27 totally overhauled how you save from webpages in Zotero for Firefox, and we made similar changes to the Chrome connector a couple weeks ago.)

Gurdas_Sandhu

adamsmith and Dan: Thanks, that's more helpful.

"There could be a 5.0.5, or a 5.1, or a 6.0 with lots of new features a FEW MONTHS after that." (caps mine)

You could have just said that earlier in the discussion :)

I am a Zotero evangelist and so am forever hungry for improvements that make my mission easier.

adamsmith

in fairness, Dan did say "Between the architectural work and the test suite, I expect the pace of new features to pick up quite a bit shortly after 5.0 comes out. "

Gurdas_Sandhu

I am an engineer, what Dan said earlier is not quantitative enough. Fuzzy words don't excite me as much ....

Nodragem

I just wanted to add to that conversation that, perhaps, developers should keep the users updated on the work done on Zotero a bit more often.

I personally shifted to Mendeley thinking that Zotero was abandoned as there was no major update for years and no roadmap anywhere :/ :/

But now, I am happy to see that I am wrong :) Zotero is alive! I'm going to uninstall Mendeley now :D

Gurdas_Sandhu

Dan's post about Zotero 5.0 alpha:
https://groups.google.com/forum/#!topic/zotero-dev/dX2gMoOSwRs


Dan: Any ETA for beta and public release?

Javier.Ormeno

Justo wanted to add my two cent's on this topic:

hurray! Love the work done until now, and I trust developers are making the right decision on this update. Kudos

Best regards
Javier

erazlogo

Many thanks for the syncing improvements for large databases in Z 5.0.

As part of 6.0 or later, is there any plan to create a mobile Zotero version? Currently on iPad, PaperShip can't handle large databases and the online library display is limited. With the new iPad Pro w/keyboard, and the iOS version of Scrivener coming this summer or fall, it will be possible to do all of the manuscript researching & writing on a tablet and iPhone--it would be so convenient to have the Zotero database easily accessible on the mobile as well.

adamsmith

Based on their dev hires, my sense of what they'll do (and certainly what I'd advocate) is a significantly improved web app (with a good mobile site, obviously). There's really very little that an actual app can do that you can't just do in html5, and that way Zotero works across platforms.
But I completely agree that better mobile integration of some kind should be high up the list of priorities.

Gurdas_Sandhu

We know Zotero 5.0 beta is available (yeah!): https://forums.zotero.org/discussion/59829/zotero-50-beta/

The feature I am most eagerly waiting for is "6. Will Groups allow "attach link to file" functionality?"

@Dan or someone, can you provide some update when this feature will be available?

adamsmith

My sense has been that this isn't at the top of the agenda for Zotero's core dev, so I wouldn't necessarily expect it super soon unless someone else does this.

Gurdas_Sandhu

Thanks, @adamsmith. I’ve held back on launching my Zotero group for the lack of this feature. I’ll explain my use case and maybe there a partial workaround.

I have a Zotero library that should be useful to researchers in my field. And yet, it is only the tip of the iceberg. So, I want to use my library as a starting point to create the most elaborate curated bibliography for my field of work. That means hundreds of group members, but may be only a half dozen Admins with edit rights. Let’s call this group “Main”. I also have a second group called “Main_beta” and every member has edit rights. Main_beta is simply used to let members feed/update items in to Main. Members add items to Main_beta and these are copied to Main (and then deleted from Main_beta) by Admins of Main.

This brings me to two issues:

1. If Main is going to become the kind of library I hope it will, then I do not wish to maintain “My Library” separately. Why have two copies of the same item? Plus, syncing items between My Library and Main is going to be a challenge. But I cannot drop My Library because all my attachments are using 'link to file'.

2. If Groups allowed and synced 'link to file', then, interested members can have the same base folder, name their file same, and they are done. No need to even go and attach the link. Right?

Of the two issues, number 1 is a blocker for me to launch my group.

adamsmith

Just to be clear: there are no objections against the feature, nor I think does anyone doubt its usefulness. When I was talking with someone potentially interested in paying for its implementation (didn't work out), Dan was pretty clear that they'd accept it (provided good-quality code, of course).

All I'm saying is that I don't get the sense that among the many things that would be great to have in Zotero this one is high priority for the core devs.

Gurdas_Sandhu

I understood that. I was hoping you'd hear my pain and come up with a stop-gap measure :)

I want to launch that group, harvest the benefits of a global network of researchers, and in the process establish Zotero as the go to reference management tool in my trade.

Gurdas_Sandhu

How much would coding this feature cost? If I can start a fund page and arrange the money, do we have a person who is willing and capable of writing the code?

adamsmith

I had asked WillS, who maintains Zutilo and wrote most of the code for relative links for folders and he was generally interested, but you'd have to check with him if that's still the case.

Gurdas_Sandhu

Ok, will do. Thanks.

@Dan. If WillS writes the code, will you be open to adding the feature to Zotero?

adamsmith

(Dan has indicated that they're open to this on multiple occasions, including at least once -- probably more often -- on these forums)