Merge all duplicates
Hello, first time Zotero user here, bit baffled that I cant ask the program to merge all duplicates, instead of having to click "merge" on each individual case.
As I have just combined some old and new reference libraries of mine, I have several hundred (if not thousands) of duplicates from the different ref. libraries, and I'm not in the mood of clicking "merge" then waiting 5 seconds and clicking "merge" again.. will take me hours..
Any solution to this?
/thanks
As I have just combined some old and new reference libraries of mine, I have several hundred (if not thousands) of duplicates from the different ref. libraries, and I'm not in the mood of clicking "merge" then waiting 5 seconds and clicking "merge" again.. will take me hours..
Any solution to this?
/thanks
Thanks for you reply.
Yes, next to identical at least - and since I have so many of them anyway I'm not going to bother choosing one over the other, a simple merge using the newest item as "master" would suffice.
If there isnt a way to get a solution to this I will basically have to choose one of my libraries and discard all of the others, since there is a big overlap. It also prevents me from searching on ISI for a topic/organism/method/etc I'm currently interested in and writing about, downloading all the citations and then putting them into my library. Because that would in almost all cases lead to a huge amount of duplicates...
Actually, simply deleting the oldest duplicate would be perfectly fine too, if merging is intensive computationally.
1. Select one of the items that is duplicated and tag it with "duplicate" tag. The tag should now appear in the tag selector in the bottom left.
2. Open the duplicates collection. Select all item with Ctrl/Cmd + A and drag-drop them onto the "duplicate" tag in the tag selector. This will tag all duplicate items.
3. Go to My Library. In the middle pane add the Date Added column. Use the column to sort all of your items. Now select the "duplicate" tag in the tag selector. This will display all duplicate items in your library sorted by Date Added. (e.g. A2, B2, C2, D2, A1, B1, C1, D1).
4. Now, using Shift + click you can select all items up to some date (e.g. the timestamp when you imported the first time) and delete those items.
I found a "solution" by installing endnote in trial mode, exporting the zotero library to RIS, importing the RIS file into endnote and chose the "remove duplicates" when prompted in the import sequence. But then I'm left wondering if I should just stick with endnote instead... although I dont particularly like it.
In order to use this you will need to have your zotero library open in firefox and have a set of duplicate items selected in the duplicate pane.
Open a browser tab and enter: chrome://zotero/content/include.js
Then open the "Web Console" from the developer tools(there are other options like Execute JS too)
Then paste the following lines into the console:
var Zot_Dup_Pane = Components.classes["@mozilla.org/appshell/window-mediator;1"] .getService(Components.interfaces.nsIWindowMediator).getMostRecentWindow("navigator:browser").Zotero_Duplicates_Pane;
//note this will fail on duplicates that are not the same item type or it may crash if you have a large library - in which case I export subsections of the
//library and merge them in smaller sets - importing them into a new library at the end.
//You may want to change the while loop into a for loop with smaller iterations so you can control how many records are merged in each go
while (true){
Zot_Dup_Pane.merge()
}
https://github.com/escaped-echidna/zotero_merge_all_duplicates
Also I noted that there was an 'AutoHotKey' solution posted earlier by a user called 'Graham_MTM.' I haven't used it but it could work:
https://forums.zotero.org/discussion/30816/semi-automate-merging-duplicates
The issue has been raised a few times already:
https://forums.zotero.org/discussion/68501/merge-auto-function
https://forums.zotero.org/discussion/40457/merge-all-duplicates
https://forums.zotero.org/discussion/29381/merging-many-duplicates-in-zotero-standalone
https://forums.zotero.org/discussion/50221/accept-all-duplicates
https://forums.zotero.org/discussion/31565/how-do-i-merge-12-000-duplicates
https://forums.zotero.org/discussion/36391/tips-for-speeding-up-duplicate-merging
Open up the Duplicate Items folder. Go to Tools > Developer > Run JavaScript
and use this Snippet:
var DupPane = Zotero.getZoteroPanes();
for(var i = 0; i < 100; i++) {
await new Promise(r => setTimeout(r, 1000));
DupPane[0].mergeSelectedItems();
Zotero_Duplicates_Pane.merge();
}
Basically, this click 100 times the "Merge X items" button with a second waiting time in between. This worked well for me, as I had to merge some thousands of items after importing multiple files. Maybe that helps someone. As I am not really aware of any implications / side-effect this function may have, use it with caution!
`TypeError: oldestItem is undefined`
One can fix this by fixing dates on any items with stuff like "In press".
('date accessed' also does not reflect the date items are added to zotero)
I imported references on several occassions, each with its own collection.
Any ideas on mass de-deduplications in this case?
Thank you
I had to import several times as the import process failed several times.
It creates duplicates.
Files do not have distinct date added values.
Your trick seems to be working for me. Running it now. No problems so far.
It worked. Thanks.
"...Open the duplicates collection. Select all item with Ctrl/Cmd + A and drag-drop them onto the "duplicate" tag in the tag selector. This will tag all duplicate items..."
I notice the 'Select All' shifts back to only one set of duplicates (A1+A2 (+A3 and so on where applicable)) based on the item where the cursor was last located to begin the drag action. i.e. it will not drag ALL items in the 'duplicate items' collection to the duplicate tag.
It seems 'selection' in the 'duplicate items' collection does not behave like it does in other collections.
Does anyone else experience this? Is there a way to drag ALL duplicate items to the tag as suggested in this solution?
You can use Zutilo to copy a tag to all selected items.
It works for a few items, then I get the message: TypeError: oldestItem is undefined
Any idea why?