I had the impression that Zotero was moving away from its original design as a "web-first tool", through the standalone app and the very nice recent pdf reader with annotations and notes. I am trying to point some of the limitations I experience in my workflow with the current tools available in Zotero to understand the underlying reasons of these choices, and therefore understand whether Zotero is suitable for me or not.

I use Zotero for reference management in scientific research, where the core information is the scientific publication itself (mostly pdf files), while its metadata is the information needed to organize them and cite them.

I really do not see the point of having two strictly identical files stored twice on my computer. In the example you provide, making annotations in an external PDF reader, the annotations process actually creates a different file, with a different identifier. I completely agree that it is perfectly fine in that case to have another copy of the file after editions added to a separate item. But until these changes are actually made, you are just storing the file twice without any other added value that I can see.
With the new PDF reader, Zotero now lets you always "keep one pristine copy of the file" and still add annotations in its viewer. If you prefer making annotations outside of Zotero, you can always go back to that original copy whenever you want. No need to store it twice.

Even if for some reasons you still want to duplicate the exact same attachment file in your library, that may still be added through some options. But my guess is that this is not necessary for the large majority of cases and users. At least I would find the ability to prevent adding strictly identical files twice, and correct for it later if it still happened, extremely valuable.

I understand that Zotero probably has very good reasons to select one of the files to keep in the item when dealing with duplicates. But however good these arguments are, that choice may not be what the user actually wants. The main issue I am pointing at is that in the current duplicates folder, the user does not have any way of judging the decision made by Zotero. I think that making an informed decision is important when deleting files.
The only case scenario which should be treated differently, in my view, is where the files are actually strictly identical, based on some unique identifier. But even in that case, it would still be useful to receive clearly this information from Zotero before deleting the duplicate file.

In your example of keeping one clean copy and one annotated copy of a pdf file in your library, I guess that Zotero will consider these two files "identical" when searching for duplicates. It is critical at that point to keep the control over the merging process, and keep the ability to disagree with the decision made by Zotero.

I have listed a few cases in which Zotero has considered that the "identical" files in my library should be deleted, while I actually wanted to keep the two files within the same item. I can therefore say from what I have experienced that I disagree with what Zotero considers "identical" files. I guess other users may have similar worries when pressing the merge button.

I have a lot of respect for the work achieved in the development of Zotero. Hopefully, the feedback I have provided above can still be useful to consider future development decisions.
I am perfectly fine if the conclusion of the discussion is still that Zotero is not for me.
Having some feedback from other people on their workflow may also help me reconsider my approach and redesign it in a way taking advantage of the current design choices made in Zotero.

To reply to your last question:
If an item has multiple attachments, Mendeley will show a different icon, so that I have clear visual information about it directly from the main panel of my library.
In Zotero, items with one or multiple attachments will be displayed in exactly the same way.

Here are two simple examples of the problems I describe:

1) I usually download a list of PDF files on my computer to have a look at them first and eventually edit them or remove the irrelevant ones before adding the selected ones to my library. This initial scanning through the files lets me control what goes into my library and keep it cleaner.
After that, I simply directly import the files to Zotero, following the instructions in "Adding PDFs and Other Files", either through drag and drop in the main library, or “Store Copy of File…”.
Here is the result of adding the exact same file multiple times, after creating manually the correct item from the DOI for the first time it was added: Zotero_Duplicates_01.png
It is easy to spot the duplicates here, because they are added successively, with the same file name containing clear information. But I could also add the same file again after several weeks or months, with another file name. In that case, I do not see any way in Zotero to identify these duplicates.
What I would like is:
- If I add the same exact file again in my library, I would like to have the option in Zotero that it takes me to the existing entry in my library, rather than copy the exact same file multiple times.
- For cases in my library where this has already happened, I would like Zotero to be able to tell me that I have multiple copies of the same file in my library, from file hash, and give me the option to merge them.
I understand that strictly preventing identical files is not necessarily suitable for everyone. But having the ability to receive the information of duplicate files (same file hash) from Zotero is definitely something that should be useful to everyone, however they decide to deal with these duplicates.

2) The second example is from the folder "Duplicate Items": Zotero_Duplicates_02.png
As you can see, the merging tool shows that there are 2 different items, but I can see only one PDF file in the main panel. This suggests that Zotero has decided to keep only that PDF file in the merged item, even though the two PDF files are clearly different. I can navigate between the two items in the right panel, but it does not show me anything about the attachments. So I have no idea of what will happen to the attachments after merging.
In Mendeley Desktop (the excellent software that was "replaced" by the useless Mendeley Reference Manager) the Check for Duplicates tool would bring the two items identified as duplicates, give me that ability to inspect the content of each of them easily, show the content of the proposed merged item, and finally give me the option to remove some of the attachments if I think they are not needed in the merged item: Zotero_Duplicates_03.png.

My argument here is that file duplicates (from file hash) are fundamentally different from metadata duplicates (items in Zotero). I want to be able to keep different files under the same item. But I do not want to create a new item when a file already in my library is added another time as a top-level item.
Having the ability to identify file duplicates and decide how to deal with them should be useful to everyone. As I understand, Zotero does not do anything at the moment about file duplicates, and does not even search for file hash duplicates.
Deciding the right interface to deal with file duplicates in Zotero may need some thoughts to accommodate everyone's workflow, but the ability to identify file duplicates from file hash should be fairly easy I guess?

Ok, I see it now.

If you sort by almost any other field they'll be sorted together.

This statement is based on the assumption that the metadata is already "almost" identical, probably related to the way that these duplicates are identified. But if one of them has different content in the field selected to sort the entries, the only way I would have to see it is to scroll down the hundreds of potential duplicate items. Grouping together the potential duplicates would work better for me.
In the case where identical files (but different names) are also suggested as potential duplicates, files without metadata would not be possible to group by ordering.

Again, Duplicates Items shows you the exact same items in the exact same way as the other views.

I would prefer to see the proposed content of the merged item directly as a new item in the items list, followed by the original items that will be merged. This will still fulfil your requirements, but still make it easier to navigate the duplicates.
This will likely require more work to implement, but I do not feel confident in merging duplicates with the current interface.

And I still do not know what is going to happen to the PDF files (or other child items) if I decide to click on merge.

I have tested to merge the items in this example: Zotero_Duplicates_02.png.
The merging process did not take me to the resulting merged item. So I had to search for it manually in the main library, through ordering by "Date Modified" for example. The result was one item with two attached pdf files. Depending on the user, one could argue that this is good... or not... Arguing on what should be done is missing the point.

The main thing that would be beneficial for me is not really to let Zotero make any kind of smart decision, either to remove the duplicate file or not, but simply to give me the control on which files will ends up in the merged item. In exactly the same way as you give the ability to decide which metadata to keep in the merged item.
I guess that my feedback is very similar to people asking for the ability to deduplicate files. The underlying feature request is exactly the same, which is to be able to control the merging process. You can never produce a satisfactory automatic deduplication process, because every user will have a different expectation.

It is easy to choose in Mendeley Desktop which file is kept in the merged item, as you can see all attached files at the bottom in the right side panel. It may be more difficult to do in Zotero with its approach to show child items only inside the central panel. But hopefully there is a nice way to do it also in Zotero, giving the user the control over the merging process.

Our approach adds the flexibility of sorting by any column — including Date Added — and seeing results from all versions, which is incompatible with always grouping together.

I don't think grouping by items to merge is incompatible with sorting. I appreciate the ability to sort by any field, but you are sorting on the fields of the items to be merged, which is probably not what you really want to do. You could keep this ability to sort by any field rather on the proposed merged item.

Thank you for your replies. I now understand better the reasoning behind the decisions for the design, although my workflow does not completely agree with them. Hopefully this discussion can help bring a different light on other related comments from other users.