Endnote 'note' field stripped of formatting

mcsugarfree

hi,

i'm trying to make the big switch and encountering a glitch: in Endnote the text that i've typed into my 'note' fields is carefully formatted with line and paragraph breaks and sometimes italics and other text attributes--the line and paragraph breaks are preserved in the .txt file that Endnote exports, but zotero does _not_ seem to preserve them, so i end up with a massive block of undifferentiated text. i can let go of the attributes, i think, 'cause i know that the .txt format doesn't preserve them, but i need to line and paragraph breaks!

my version of Endnote allows me to use the .RIS export format, but it only allows me to _save_ in .txt, .rtf, .htm, or .xml. [when the "export window" comes up, what i'm offered diverges from the instructions on the zotero instructions document:

Once RefMan (RIS) is set as the format, select “Export” from the File menu. In the Export window that pops up, choose “Text Only”, select the RIS output style immediately below “Text Only” in the dialog, and hit “Save.”

".RIS" is not offered as an output style, only the four formats i listed above.

any suggestions would be welcome!

ajlyon

The italics and whatnot won't be preserved, for the simple reason that Endnote doesn't export them, so Zotero doesn't know about them.

Linebreaks should be possible-- hopefully one of our Endnote migration gurus can chip in here.

mcsugarfree

This problem is still outstanding (see above). Could someone in the know give a couple minutes to it, please? I'm stiking out, here.

thanks.

Loren

ajlyon

As I've said, you won't be able to maintain the formatting when exporting as RIS, since Endnote.

It should be possible to copy and paste the note contents into Zotero and maintain formatting. I know that that sounds horribly slow, but I don't know of any other way you can be sure that the formatting will be preserved. The formatting of notes is not specified in any of the main standards for exchanging bibliographic data, so no software does it very well.

adamsmith

well, the main question is about line breaks and you said it should be possible -

actually, I wonder if this isn't a bug/deficiency: Zotero doesn't import line or paragraph breaks into notes. That's not a RIS limitation - line breaks in abstracts are imported fine.

This is even true for Zotero's own RIS export, which uses <p> </p> for hard returns in notes - and has it promptly stripped out on import. There has got to be a better solution for this.

mcsugarfree

yeah. line breaks are the main thing. i can live without the italics and whatnot, but i _gotta_ have the line breaks (which endnote _does_ export).

is this fixable?

thanks.

ajlyon

I suppose so. Time to go diving into RIS.js again. See you on the other side.

ajlyon

Can you post an example record with such a note so I can see exactly how Endnote formats them? I don't have a copy to test with.

dstillman

This is a longstanding issue since the switch to HTML notes. The data layer now expects note input to be in HTML, and plaintext newlines will be lost. So RIS.js should probably covert newlines to appropriate HTML.

Losing the paragraphs and line breaks that Zotero puts in itself is bad.

dstillman

The alternative is for the data layer to be smart about this and, if it gets plaintext, to convert it automatically. Then translators wouldn't have to be modified to pass in HTML.

But I'm not sure what test we'd use to determine if the input was plaintext.

ajlyon

We could provide a utility function that handles conversion, so that translator authors can just call it when they know it'll be needed.

adamsmith

but html doesn't seem to work either - this:

<p>Not all actors in policy coalition have same interest. Sometimes ambiguity is helpful.</p>

<p>In French Pensions: Establishment of a secondary </br> save as you earn tier based on mutliple support, first from the right, but then also from savors and finally from leftists in an anti-American turn, because anglo-investment funds were becoming too influential.</p>

imports as one paragraph for me - the Ps were added by Zotero RIS export and I tried to insert the </br> to see if it did anything, all to no avail. (the html of the imported note just has the paragraph tags at the beginning and the end).

ajlyon

But in the meantime, I'd still like to see what Endnote's notes look like in RIS, so that I can tweak the translator correctly.

dstillman

but html doesn't seem to work either

I understand. If you can confirm that the HTML is being passed to the data layer (and not somehow stripped in the translator itself), I'll look into it.

ajlyon

This isn't happening in the translator. The only HTML-related code is:if (Zotero.Utilities.unescapeHTML) {
value = Zotero.Utilities.unescapeHTML(value.replace("\n", "<br>", "g"));
}(at the beginning of processTag(..). No cleaning of HTML tags anywhere in RIS.js.

dstillman

unescapeHTML() converts HTML paragraphs and line breaks to plaintext equivalents.

ajlyon

Oh. Well, that does it! I thought it just unescaped HTML entities.

ajlyon

Please go to http://github.com/ajlyon/zotero-bits/raw/master/RIS.js and save the file to the translators directory of your Zotero data directory (http://www.zotero.org/support/zotero_data).

Try importing again. I may have been too aggressive in maintaining spacing-- let me know.

adamsmith

that one killed RIS export and import for me entirely - doesn't even show up as an export option anymore (I've restarted and made sure RIS.js is in the translator directory).

ajlyon

Yes-- I should note that it's a 2.1-formatted translator. Hold on and I'll get you one for 2.0.9.

ajlyon

For people using 2.0.9:
http://github.com/ajlyon/zotero-bits/raw/master/RIS-2.0.9.js

Save it as "RIS.js" to the translators directory.

adamsmith

works great for me, both with and without html tags in the RIS - thanks!

ajlyon

And the existing logic in RIS.js makes even less sense if unescapeHTML strips HTML-- so the translator first replaces newlines with HTML line breaks, then strips out the HTML line breaks and replaces them with newlines? Oh, well.

@mcsugarfree: Please try the new translator and let us know how it goes.

dstillman

I should have mentioned: there's a text2html() function in Zotero.Utilities that would be better to use for notes. It attempts to add proper paragraphs, line breaks, and non-breaking tab spaces where appropriate.

ajlyon

Versions at Github now updated to use text2html. Please test.

mcsugarfree

thanks for working on this! you guys have lost me... :)

@ajlyon: how do i test the new translator (do i need to install a different version of zotero, or am i just installing a new little "module" in the version i already have?)? if there is a standard explanation in the documentation of how to implement (and test) changes like these, would you point me to it, please?

@Dan Stillman: please explain where in the process of exporting records from Endnote as .txt files in RIS format and then importing them into Zotero i would use the "text2html() function" you mention. is this part of the same process that ajlyon is talking about, or is it some alternative strategy.

sorry for my (relative) helplessness!

adamsmith

mcsugar - for your purposes ignore everything that Dan is saying - it's under the hood stuff, mainly intended for ajlyon.

"how do i test the new translator"
see ajlyon's post above - download RIS.js (or RIS-2.0.9.js if you're using ZOtero 2.0.9 and rename it to RIS.js) and use it to replace the file of the same name in Zotero's translator directory (see again ajlyon's post on how to find that).

mcsugarfree

@ajlyon: pretty good! the line breaks are preserved when i import. the only remaining problem is that some characters (", ', &, and a few others) get replaced with """, "&apos;" etc...that's not a big deal for some records, but makes other records very labor intensive to use (for example the ones that have bits of elizabethan dialogue in them...).

mcsugarfree

crap. in my last post the computer turned my tags [notated here with artificial spaces between each character for obvious reasons] into the characters I wish they were the "notes" field in my records: what looks like this """ in my last post was supposed to look like: "& q u o t ;" etc. etc.

what i was trying to say was that for some reason zotero is not showing certain characters but is instead inserting some sort of, like, html or ASCII tag for those characters into my notes...kapeesh?

ajlyon

What did those characters look like in the RIS file you exported from Endnote? Were they normal " characters, or were they character entities like & quot;?

mcsugarfree

in the .txt file exported from Endnote they are normal characters. once imported into zotero they became "entities."