Custom import options

sdflewrit783 · August 14, 2022

I think I am at a stage where I want to transfer all references from Biblioscape to Zotero.

Many of these references have custom fields that will not be supported by standard formats such as RIS.

I can write custom export fields from Biblioscape but would appreciate any direction on how to create custom import procedures for Zotero.

Challenges I see:

1. Remapping nonstandard fields
2. Importing attachments
3. Avoiding duplicates

Programming-based solutions are fine, we are talking about some thousands of references so it is not feasible by hand...

Thanks...

adamsmith · August 14, 2022

I don't quite understand 2. and 3. (please provide more details on those if you need help), but generally speaking, rewriting parts of the RIS translator to account for any custom tags you have, including file links, isn't particularly hard. I'd start by just looking at the code -- you can modify the copy in the translator folder in the Zotero data directory and those changes will take effect immediately.
You could also do this in the Translator Editor (under Developer Tools) for easy editing and save to the folder using the save icon with the Z.

We can help with specific questions, but this mainly requires finding the right place to add the mappings and only very limited javascript or understanding of Zoteor translator development.

sdflewrit783 · August 15, 2022

Thanks for the advice. Shall look.

I shall try to write a recipe to see if I am missing something...

But briefly:

2. The sources have PDF attachements. Each reference stores a string with a relative path. it woudl be good to have those added to the imported references in Zotero on import.

3. There are multiple duplicates between the source database and the relevant Zotero group. It would be good to have a strategy to avoid importing them or to have a way to recognise and remove them quickly.

adamsmith · August 15, 2022

OK, so 2. is going to work just like the rest, but I don't see how 3. is going to be reasonably possible, certainly not within the translator alone. I think importing and then merging duplicates in the group is going to be the fastest, even with 100s or even 1000s of duplicates. (You can find scripts that auto-merge all duplicates in the forums if that'd be of help)

sdflewrit783 · August 15, 2022

Thanks again.

I see that RIS translator code is 7000 lines. Wow!

Shall definitely need guidance here. :)

Is RIS the best intermediate format?

Here, they have presets for EndNote (not xml), RIS and latex. All minimal but I can extend.

If it is RIS, where in the translator do I see the fields I shall need to match?

---
2.

The path to the attachment looks like this:

L1 - attachments\creativity\Boden - Dimensions of Creativity (1994).pdf

This does not get imported into Zotero even if the import option is 'copy files to the Zotero storage folder'

Can this be wrangled to work?

---
3.

Yes, I guess I can import all into a separate collection, find all duplicates and delete/merge more recent ones. I would prefer to delete, not sure if possible.

Is it possible to search for duplicates and then sort/filter the results by collection? If I import in the new collection, I could just delete all duplicates there.

adamsmith · August 15, 2022

The field map goes from L. 175 to 465 and has 3 parts: fields that are always getting mapped the same way, fields that are getting mapped differently depending on item type, and less common fallback options. I think you should just need to work with the first two.

For the file import, that should almost work. Since this uses a relative link, make sure that you open the RIS file from the directory that contains the attachments folder. The L1 link may need an additional slash at the beginning. The relevant code to change attachment import starts in line 1389. (Note that the translator only has 2000 lines of code; the rest are tests)

sdflewrit783 · August 15, 2022

If I modify a local translator, should I be saving a new translator with the new ID, name, whatever it needs?

Or should I just keep the original as a backup and restore after I am done?

sdflewrit783 · August 15, 2022

Ah, yes, PDF gets imported if I run from the relevant folder, thanks!

Shall look at mapping.

Any further tips about those duplicates?

adamsmith · August 15, 2022

For duplicates, I'm sure there are several ways, but what I'd do is sort the Duplicate view by date added, select all items with the current date (i.e. the date when you imported the large RIS) -- at that point it might be possible to delete them, otherwise assign to a tag or a collection and delete them.'

For the translator, you can either create a new one with a lower priority number (lower = higher priority) or temporarily overwrite the existing one and then reset translators. In the latter case, make sure to keep a copy of your custom translator outside the translator folder so it doesn't get overwritten during updates before you're done.