Bibtex (bibdesk) import preserve 'Date-Modified' and 'Date-Added'

kidwellj

I've just begun the process of importing a large set of libraries I've been managing with bibdesk to Zotero. The only missing piece seems to be the lack of integration on the fields for 'Date-Modified' and 'Date-Added'. These are really pretty important for my workflow, as I use date to parse out which stage of research I've added items to a library. Has anyone seen work done towards this before? I've trawled github, forums and elsewhere and come up blank. Seems like this could be achieved by modifying a translation file...

emilianoeheyns

Actually no. Import translators cannot set the date-added or date-modified. BBT will on import write these to the extra field so they'll be re-exported when you export again, but they won't (since they can't) affect the date added/modified you see in Zotero.

kidwellj

Ok, that's good to know. So will need a bit more of a plugin approach. I'm assuming it's just a matter of modifying dateModified and dateAdded. Perhaps this could be grafted into Better Bibtex (http://retorque.re/zotero-better-bibtex/)?

emilianoeheyns

*grin* I'm the author of BBT.

Yeah, it'd be possible, but it'd be hella ugly. I'd have to either patch the save methods for translator items, or remember which items I imported, and find and patch them after the import has finished. The latter seems way too fragile, and the former... I'm trying to minimize the number of runtime patches I install these days, as it's way too easy to destabilize Zotero this way. Zotero actively removes these dates from the import items. Preferably that would just be either removed or made optional.

emilianoeheyns

*actually*... I could pick these up from the extra field where I already write them to... but then there's still the problem that it's conceivable that changing these timestamps would mess with syncing. I really don't want to have that on my conscience.

timmb

I'm in the same position. If I were to move from Bibdesk to Zotero I would lose the Date Added on 12 years of references, which is a dealbreaker.

I appreciate the syncing issue but I don't yet have a database to sync. Perhaps the solution would be a separate tool to update the added and modified dates to match the tex.date-added and tex.date-modified extra fields left by the BBT importer?

emilianoeheyns

@danstillman back-dating `dateAdded` shouldn't be problematic, right? And for sync, `clientDateModified` is leading IIRC, so maybe modifying `dateModified` wouldn't be problematic either?

dstillman

@emilianoeheyns: The server used to reject dateAdded changes for existing items, but that's currently disabled because it was happening unexpectedly (I'm not sure I determined why). Realistically that probably won't be enforced again, but technically those dates aren't supposed to change.

But now that we have an options window for imports, this should really just be an option there. I've created an issue to track this.

emilianoeheyns

Is this option something I could detect in the translator, so I can choose between leaving it in the `extra` field and just setting the Zotero fields?

Can I detect in the plugin whether an item has been synced? Back-dating for not-yet-synced items would be safe then I take it?

dstillman

No, it wouldn't be exposed to translators. But if people choose not to keep the field given the option, I'm not sure why you'd worry about it.

An item that has been synced before would have a .version > 0.

emilianoeheyns

Imports can also be done from the clipboard, which doesn't have an UI.

emilianoeheyns

@timmb the option that dstillman proposes is going to be a lot cleaner than what I'd be able to do. Given that it's coming to Zotero, I don't see much value in adding it in BBT; it'd be fairly hacky. If you're in a hurry, I can probably get you something for the Javascript Runner that's in Zotero, as long as you have not synced.

dstillman

Those wouldn't use the dates.

crowding

@timmb @kidwellj I found myself in the same situation in wanting to preserve my Bibdesk library's date-added. I've hacked together a [Python script](https://gist.github.com/crowding/54ddca96229441a0c7bb3a087631321b) that will pull those values out of the 'extra' field. It should be used after importing your .bib file but before syncing.

emilianoeheyns

https://retorque.re/zotero-better-bibtex/installation/preferences/hidden-preferences/#patchdates

emilianoeheyns

@crowding Zotero advises against direct updates of the Zotero DB.

cl.bu

Dear Zotero users,

currently, I am trying to import a Papers 3 library to Zotero for Mac, and I would like to retain the original "date-added" date from my old library.

As recommended by @emailianoeheyns (many thanks!) in this discussion, I have downloaded BBT, then imported my old Papers 3 library and attempted to use the "patchDates" hidden preference to change the "Date Added" info in Zotero to the original date of my old Papers 3 library.

As described at the patchDates hidden preferences info (https://retorque.re/zotero-better-bibtex/installation/preferences/hidden-preferences/#patchdates), I have entered the command "tex.dateadded=dateadded, tex.datemodified=datemodified" (alternatively, I also tried "tex.date-added=dateadded, tex.date-modified=datemodified") at "extensions.zotero.translators.better-bibtex.patchDates" in the Config Editor.

However, when I then select the BetterBibTeX command "Copy date-added/date-modified from extra field" from the BBT menu, I only get the error message "Better BibTeX: could not parse field mapping tex.date-added=dateadded, tex.date-modified=datemodified".

In the extra field from which I want to retrieve the original information, the date-added resp. date-modified info reads as: "tex.date-added: 2020-07-20T15:15:36GMT, tex.date-modified: 2020-07-20T15:34:24GMT", however even if I change the date and time formats (e.g. DD/MM/YYYY HH:MM:SS, etc.), the error message "could not parse field mapping" remains.

Any help would be greatly appreciated! I have been trying in vain for hours to solve this problem.

emilianoeheyns

For now you have to remove all spaces from patchDates. An upcoming releases eases that requirement. You can also just leave it empty, because tex.date-added=dateadded,tex.date-modified=datemodified is always active.

cl.bu

@emilianoeheyns Thank you! As a new Zotero user, I appreciate your indispensable support and help.
The date-added/modified" command in the BBT menu worked like a charm after changing "tex.date-added" resp. "tex.date-modified" in the extra field to "tex.dateadded" resp. tex.datemodified" and getting rid of the exact time info.

emilianoeheyns

Wait - you mean you had to edit the input before it worked? That sounds like a bug, could you report that on github?

cl.bu

@emilianoeheyns
Yes, I made these edits in the "extra" field before running the "Copy date-added/date-modified from extra field" command.
I now have reported my problems on github. When I hear back from them, I will post it here.

emilianoeheyns

Them is me ;)

cl.bu

@emilianoeheyns
In this case allow me to thank you here for your fast help on github (cf. https://github.com/retorquere/zotero-better-bibtex/issues/2275)!

I briefly tested the test build you provided there, and the "date-added" info from the extra field can be copied without having to make any prior changes.

druartl

@emilianoeheyns
Sorry to reopen this discussion (again).
I'm having trouble with the solutions described above. I've tried changing the hidden preferences patchDates as stated both above and on your website (https://retorque.re/zotero-better-bibtex/installation/preferences/hidden-preferences/#patchdates). Yet upon import it's still adding with today's date instead of the old one.

Here are the settings I've tried last : tex.date-added=dateadded,tex.date-modified=datemodified

After the import here is an example of the tex infos that goes in extra :
Citation Key: Eysenbach2011
PMID: 22209829
tex.date-added: 2019-12-05 18:50:15 +0100
tex.date-modified: 2019-12-05 18:50:24 +0100
tex.howpublished: epublish
tex.journal: Journal of medical Internet research
tex.pmc: PMC3278112

I'm not sure what I'm doing wrong here ! May I ask for your assistance ?

PS : out of curiosity, I don't understand the pin symbol next to the generated citation keys. I don't see them when importing from the connector. What do they mean ?

emilianoeheyns

Import translators are not allowed to set the dateAdded and dateModified fields of the entries being imported, so after the import, you can right-click the imported entries and choose "Better BibTeX -> Copy date-added....". That will set the dateAdded/dateModified of the imported item, and will remove those lines from the extra field.

emilianoeheyns

Wrt the PS: The pin symbol means the citekey is set to a fixed value that does not derive (anymore) from the fields of the item. When you import a bibtex file, those entries will have a citekey, and the assumption is you want that key preserved, so it is pinned to the value present in the entry.

When importing from the connector, no such citation key is present to preserve, so those entries get a dynamic key, which does derive from the item fields. You can pin their key by right-clicking the item and choosing "Better BibTeX -> Pin citation key", or by adding a "citation key: [something]" in the extra field (which "pin citation key" automates for you, but it's the same thing).

druartl

Thanks a lot for these explanations ! They helped me migrate from Bibdesk to Zotero.
Also thanks for all the plugins you've contributed, being almost done with my setup I can see I'm relying on several of yours !

emilianoeheyns

You're welcome.

jlcarney

Hello, I'm trying to shift to Zotero after 16 years on Endnote with a library of nearly 5k references, and, like those in this thread, retaining the original "Date added" information is crucial to me. I'm wondering whether using Better BibTeX as an intermediary might be an option and, if so, whether folks out there have any advice for someone considering this route. I am trying to shift for a number of reasons, but the fact that Endnote doesn't work well with iterations across multiple machines (different item identifiers on each machine) is a major one. The other option I considered was using the "Extra" field directly from Endnote, but I don't have a good idea about how to go about that. Thanks for any thoughts.

adamsmith

jlcarney's question continued here: https://forums.zotero.org/discussion/105303/import-from-endnote-preserving-original-added-to-library-date#latest