I'm just getting started with Zotero. So far it is great, but dups are already an issue. From reading this discussion, there are two issues. The first is finding exact duplicates. The second is merging entries about the same item that are not exact. Finding and removing exact duplicates is not hard right now. Detection on import would be nice.

Merging entries about the same item that differ is the important feature. A simple approach that would work for me would be: 1) Information (and links) that exist in only one entry is combined, 2) In the case of a conflict, let me pick a ‘master’ entry. A more complex solution would be to let me choose on a per-tag basis, but I could live without something that complex.

I'm sure that other users have different needs and the issue is more complex than I understand. But, I would prefer to have an imperfect solution over none.

Hi. This thread was started by CB on October 10, 2006. Do we have pdf duplicate detection yet?

plus it would lower my total number of citation downloads from Google Scholar, and thus lower my risk of being locked out for adding unnecessary numbers of items

I don't know how likely this is: Zotero would need to retrieve bibliographic data before it could tell you whether the record was an exact duplicate, which google would count as a query.

@glientsc This is what I do, but it's a real pain to spot them when the list is several hundred citations long. I've gone back to using EndNote and and Mendeley. I find them more functional in combination, though EndNote is painfully slow.

Agree that this is an important item. I had a similar idea with the approach I suggested under the zotero-dev post that I linked above: the risk of acquiring duplicates is greatest when items enter Zotero through a translator; translators are limited to a single site, and typically work on a limited set of item types; given those constraints, checking for duplicates in the translator after the item fields are collected, but before submission, is simple to do reliably -- which is maybe not the case for a one-size-fits-all duplicates bloodhound function.

If a simple internal interface leveraging Zotero search were exposed in the translators, and tied to a popup warning mechanism (use existing item/download duplicate item/reject), contributors could slot it in to fix the translators they use often.

A first step would be to build an author management tool which allows for merging multiple versions of the same author name.

Without experimenting with heuristics, an author merge feature could be implemented simply as a manual select-from-search. (Similar to one very popular webmail client's contact merging feature.)

A similar function for other potentially duplicated items such as publisher, journal or book title, location, etc. could eventually give some human logic help to the heuristic check for full-duplicates.

This could help with the first-name-disambiguation confusion which has been reported (incorrectly) as a bug in a number of styles. Sometimes, the disambiguation is triggered when the same author is entered twice with slight differences such as with/without period following intial, etc.
see: http://forums.zotero.org/discussion/7457/should-there-be-a-no-givenname-disambiguation-default-style/

I suggest a list of very simple algorithms for duplication management that can be easily implemented. Although this would not be perfect, I think most users will be *far happier* having this, than having none.

1. Notifying when the user tries to add an article, and ask whether to continue to add that article or not, if there already exists an article in the library with either of the same ...
1) title (case-insensitive, ignoring whitespaces),
2) DOI, or
3) publication, volume, issue, and page at once

This would be satisfying, at least for most scholars.
Regarding the timing,
it should obviously be done after retrieving the article's meta data, and
it would be best to be done before the article is actually added to the library.

If there are many articles added at once, you can just show up the dialog box many times.

2. Duplicate management in the existing library
1) Detect the duplicates with the above algorithm when the user asks to do so (i.e. press a specific button)
2) Show a list of duplicates
3) Let the users do the rest!

I know that these algorithms will be far from perfection, but again, what is definitely and desperately needed for many existing users, is this simple improvement. I think there has been strong and persistent need for this functionality, as is proved by the many replies on this discussion, and this functionality might be given priority over others.

If there remain any possible complications in implementing the algorithms I suggested, I'm eager to hear about that.

I wonder if a couple of these proposed algorithms could be implemented using Kieren's zotero-browser approach to accessing Zotero data (http://github.com/singingfish/zotero-browser), to see how effective they are.

Kieren: Would you want to try some of these out? I imagine duplicate detection and handling would be a good killer app for zotero-browser.

ajlyon

Well I have a working installation of my browser here, and the (non-gui parts of the) code should be identical in the browser to what would end up in zotero, so in principle I agree. My time is a bit limited for the next month or so, but in principle I can have a look later in June. Developing duplicate detection algorithms was the third use I came up with for the thing, after enhanced reports, and playing with my naive text mining environment.

If you fork on github, just notify me of changes through pull requests.

IMP: As mentioned in the post below this will break things if you are using wordprocessor plugin. If not but you are using stuff like lyx/latex and normal .bib file genrated from zotero export you can follow

One temporary solution.
Step:
Occasionally...
1) Export your references from zotero as .bib
2) Use jabref to find duplicates and correct the database and save it.
3) Delete all data from zotero.
4) Import the corrected .bib file into zotero.

I dont know if Jabref people are willing to share the code/algorithm and if so how portable it is to zotero coding.

@fbennet Thanks for pointing this out. I was not aware of that as I was not using the plugin.

There is an urgent need to add the duplication detection function, or else it cannot compete with Endnote.

In my experience it doesn't work that well. Takes a long time to load in my 5000 item library and generates loads of false positives, including different chapters from books and articles with relatively similar titles but different authors. A very strict algorithm (author lastnames, titles, and dates exactly identical) would be an indispensable option especially for larger libraries.

Could you please show the steps? thx

@phlustik
Instructions by Frank Bennett here.