large Procite bibliography: how to start?

fvg

I have quite a large Procite bibliography ( < 50.000 ) that needs to be imported into zotero. I gather, from discussion on this list, that the first step is to get those records into a RIS format, and I've guessed that to do that I need to use the Procite conversion utility "Biblio-Link II", which I've got.
But at that point I'm stuck. I can't seem to get it to work, nor do I really understand how to take it from there. (With zotero I'm a novice.) I'd be really grateful for a bit of advice.
Best,
Dr. Frederik van Gelder

adamsmith

I don't know much about Procite, but in general, once you have a RIS file, you click on the Gears menu in Zotero, select import, and select the file and it should import. It'll likely take a good while and Zotero may appear to freeze in the process. Just let it run. If you should get an "unresponsive script" message, click return.

fvg

Thank you. The problem is, I'm not quite sure how to get that Procite into RIS format in the first place, using that Biblio-Link.
best,
fvg

adamsmith

you shouldn't need a biblio-link tool.
The general idea is the same as here:
http://www.refworks.com/rwsingle/help/Exporting_from_Bibliographic_Programs_and_Importing_into_RefWorks.htm#ProCite

You'd need to find RIS.pos, which may be called Endnote-RIS.pos or something along those lines. There may be several of those around on the internet, so try them out. Unfortunately procite itself appears entirely dead, so you'll have to find it from somewhere else, maybe the one attached at the bottom here will do: http://www.adeptscience.co.uk/kb/article/223A

aurimas

Here's one from my Dropbox https://dl.dropboxusercontent.com/u/24145369/RIS-EndNote.pos i think there are some changes that need to be made for a smooth export. The details are buried somewhere here https://forums.zotero.org/discussion/22615/1/procite-to-zotero-conversion-translator-ris-and-testing/ I'll go thorough it and make changes to the translator linked above (maybe today).

I would definitely experiment importing a small subset of your data (like 50-100 items of various types) to make sure everything goes smoothly. 50,000 items is a lot and there are other suggestions I would make for such a large import.

fvg

Thank you adamsmith and aurimas - that is already a huge step forward. I've got some reading up and experimenting to to, but that has already pointed me in the right direction.
best,
fvg

aurimas

If you're reading through the thread linked above, then note that most of the changes on Zotero _import_ side have been integrated and you don't need to install any custom translators or modify anything. If you feel that you do need to modify something, let us know and we'll advise.

aurimas

I integrated some fixes into the ProCite export filter, so if you had already downloaded it, you should download it again (from the dropbox link).

I also added some tweaks to the RIS import translator in Zotero. Update your translator via Preferences -> General -> Update Now.

Before you even begin testing, make sure that your library is synced with zotero.org (unless you're not using sync at all, or if you are working with a fresh library). Then, make a backup of your Zotero library (unless you're starting with an empty one). Then, I would disable automatic syncing (Preferences -> Sync -> uncheck Sync automatically). Then, after your whole library is in the state that you want it to be in, I would use Preferences -> Sync -> Reset -> Restore to Server (Dan, can you confirm that this would provide performance improvements) and re-enable automatic sync.

Finally, as I said above, you should do some test runs before performing the full import. For the test runs, you should set the following hidden preferences:
extensions.zotero.translators.RIS.import.ignoreUnknown = false
extensions.zotero.translators.RIS.import.ignoreUnknown = true Definitely flip those off for the actual import (unless you really don't want to lose any data, but this will slow things down)

Oh, few more things. Currently, note attachments and tags (i.e. keywords) significantly impact Zotero's performance. If you do not need/want to keep notes/tags, there are tweaks you can make to Zotero that would prevent them from being imported. This may offer significant speed improvements for a huge import like that.

Are you also importing file attachments? (I don't think I've ever tested file attachment import from ProCite)

fvg

Thank you, that is all very useful.

1) That procite export filter you kindly placed on dropbox: if I click on it it gets opened in my browser, and I can't download it. (I must be doing something wrong.)

2) Umlauts, special characters in the RIS input file - how does zotero handle this? (The usual UTF8/unicode problem. Is there a switch somewhere to say this RIS input file is ANSI or whatever?)

3) Yes, notes, tags, it's all in that RIS file.

4) File attachments. Procite has the data-field "Location/URL". Using that first procite export filter this parses correctly, to give me on the zotero side a field called "URL". This links correctly to the file that's needed, whether locally or on the web, opening it in the browser.
best,
fvg

aurimas

1) That procite export filter you kindly placed on dropbox: if I click on it it gets opened in my browser, and I can't download it. (I must be doing something wrong.)

Right-click on the link -> Save Link As.. (or something similar depending on your browser). File extensions should be .pos

2) Umlauts, special characters in the RIS input file - how does zotero handle this? (The usual UTF8/unicode problem. Is there a switch somewhere to say this RIS input file is ANSI or whatever?)

You can set this manually under Preferences -> Export (look at the bottom of that panel)

3) Yes, notes, tags, it's all in that RIS file.

Well, the question was whether you care about keeping them.

4) File attachments. Procite has the data-field "Location/URL". Using that first procite export filter this parses correctly, to give me on the zotero side a field called "URL". This links correctly to the file that's needed, whether locally or on the web, opening it in the browser.

I'm not sure what's the mechanism of linking to/attaching files in ProCite, but could you post an example of a RIS entry where you have a local file entered? You don't really want links to local files in the URL field, since that field is sometimes included when generating citations. Generally, the URL field should indicate where the original was retrieved from.

fvg

I've placed a RIS test file here: http://fvg.fvg.f-m.fm/RIS_test.txt (Like I said, I have < 50,000 of these.)

1) I'm beginning to realize - with these rather limited regular expressions skills of mine - that the format of choice for doing some intermediary editing on these bibliographies is going to be right here: in this RIS format. (Rather than the Procite or the Zotero side of the conversion stream..)

2) But once I've got this sorted out, the next step is going to be finding the best way of integrating this bibliography into my website. (Which at the moment resides here: http://www.amsterdam-adorno.net/) With my limited knowledge I see two alternatives at the moment:
a) place the bibliography on zotero.org (or perhaps on my own website?) and then write some php-type scripts to access it. (I've been looking at the Web API documentation.)
b) As a second, much less attractive alternative, re-do this whole website of mine in WordPress, trying out Zotpress.

Any ideas on this would be most appreciated. (I'm a philosopher, with modest programming skills, but those API's look to me vaguely manageble..) I've been wondering where I should start looking for some code that has already implemented all this, that even a beginner like me could learn from. (Or alternately: that I could point to if I try to find someone who would do it for me.)
best,
fvg

adamsmith

2) doing this yourself shouldn't be hard. See e.g. here:
http://librarian.newjackalmanac.ca/2012/09/how-to-load-zotero-into-webpage-with-3.html

aurimas

1) I'm beginning to realize - with these rather limited regular expressions skills of mine - that the format of choice for doing some intermediary editing on these bibliographies is going to be right here: in this RIS format. (Rather than the Procite or the Zotero side of the conversion stream..)

Depends on the modifications you need to make. Regex may be sufficient, but you also have the option of customizing the import translator to make any tweaks that you need. If you have any specifics we can tell you what would be the best way to go.

fvg

That sounds very interesting. Would there be a documentation of the import translator anywhere?

adamsmith

https://www.zotero.org/support/dev/translators/coding

but don't be deterred by the technical aspects. Once you look at the translator (I would recommend installing Scaffold for that purpose, at least if you're using Firefox), changing specific mappings is, for the most part, quite easy as they're explicitly listed at the top of the translator.