Upgrade StorageIssues with RTF Scan and with export
The immediate problem I face
----------------------------
I am attempting to use the RTF scan facility documented at https://www.zotero.org/support/rtf_scan.
The input file contains 1801 lines, the first few of which look like this:
{Buckingham Shum, 2003}
{Kirschner, Buckingham-Shum and Carr, 2003}
{Okada, Buckingham Shum and Sherborne, 2008}
{Okada, Buckingham Shum and Sherborne, 2014}
{Pollitt et al., 1993}
{Stapleton et al., 2012}
When I attempt to run the RTF scan, after specifying the input and output files, I click on next. The verifying cited items screen appears immediately and is completely blank.
When I click on next, I specify a citation style – in this case, Harvard – Anglia Ruskin University.
I immediately get the next screen, which tells me that:
“Your document has now been scanned and processed. Please ensure that it is formatted correctly.”
I find this very suspicious; I would expect a slow response and then at least some ambiguity, for example concerning documents with the same authors in the same year.
When I open the output document in Microsoft Word 2010, it appears to be – to all intents and purposes – an exact copy of the input file. There are no Microsoft Word fields – each citation is simply text. In addition, no bibliography is appended to the document.
Please, what am I doing wrong? Perhaps my expectation / hope is unreasonable. What I am expecting is that a true Zotero citation be created, as a Word field code.
The background to this immediate problem: what I am trying to achieve
---------------------------------------------------------------------
I wish to create a Microsoft Word document which contains all the 1801 references in my Zotero database, formatted correctly as citations – that is, in Microsoft Word terms as the fields which Zotero creates when you insert a citation; together with the date created and the date last modified. The reason I am doing this is that over the last three or four years, I have maintained in Microsoft Word 2010 a research diary which consists of one row for each significant dated event. Each row contains a date and a comment. Sometimes, the comments include Zotero citations; often they do not. What I need to do now is retrospectively to add a column to this table which gives a citation for each document which I first encountered on that date. It will look something like this:
Date Comment Citations used on this date
20/08/2014 Wednesday Fascinating text, which may also include citations such as (Codd, 1970).
(Gregory and Macgilchrist, 2014)
(Popper, 1979)
In Zotero in Firefox, I have displayed the date added column and ordered the entries by ascending date added. I used the right click ‘create bibliography from items’ drop-down, specifying output mode citations and save to clipboard. Generating this file took a few hours on this low powered computer. I opened the output file in Word, and converted left-parenthesis to left brace, right parenthesis to right brace throughout. I used the resultant file as the input to the RTF scan.
I was expecting – probably unreasonably - that the results of the RTF scan would be true citations complete with field codes. But instead, the output looks like the input – including braces {}.
The workaround that I am employing
----------------------------------
I have created an Excel spreadsheet in which the first worksheet is the results of an export in CSV format from Zotero, and the second worksheet is a list of citations as it has been generated by ‘create bibliography from items’ drop-down, specifying output mode citations and save to clipboard. This second worksheet was generated with Zotero displaying citations in descending order of date modified.
In a thid worksheet I put suitably sorted data from the first and second worksheets side-by-side and carry out a semi-automated (VLOOKUP), semi-manual reconciliation whose outcome is that I know the exact Zotero ID for each item in my Zotero database. I use the reconciled list as I go back through my journal, adding Zotero citations for each date on which I first encountered and stored a reference in Zotero.
Some suggestions, mine – and yours?
-----------------------------------
1. It would have been useful for me in this context to have been able to specify that the results of an export from Zotero should include a list of citations in a particular format, in my case Harvard Anglia Ruskin. Perhaps there is a generic use case underlying my specific application? Perhaps it has already been met?
2. My task has been considerably complicated because many of the authors that I cite have Nordic or Teutonic surnames. In my Windows environment, Zotero does a poor job of exporting items including Nordic or Teutonic characters to CSV format. I appreciate that that this may have something to do with the age and informality of the CSV format, but wonder whether improvements might not be possible.Examples: author Björner becomes Bjørner; Ågerfalk, Pår becomes Ã…gerfalk, Pär. NB: I am using Windows 7 with French keyboard and French system software.
3. I am very open to suggestions as to a better way in which I could have tackled my problem in the first place!
Thank you for reading this far!
----------------------------
I am attempting to use the RTF scan facility documented at https://www.zotero.org/support/rtf_scan.
The input file contains 1801 lines, the first few of which look like this:
{Buckingham Shum, 2003}
{Kirschner, Buckingham-Shum and Carr, 2003}
{Okada, Buckingham Shum and Sherborne, 2008}
{Okada, Buckingham Shum and Sherborne, 2014}
{Pollitt et al., 1993}
{Stapleton et al., 2012}
When I attempt to run the RTF scan, after specifying the input and output files, I click on next. The verifying cited items screen appears immediately and is completely blank.
When I click on next, I specify a citation style – in this case, Harvard – Anglia Ruskin University.
I immediately get the next screen, which tells me that:
“Your document has now been scanned and processed. Please ensure that it is formatted correctly.”
I find this very suspicious; I would expect a slow response and then at least some ambiguity, for example concerning documents with the same authors in the same year.
When I open the output document in Microsoft Word 2010, it appears to be – to all intents and purposes – an exact copy of the input file. There are no Microsoft Word fields – each citation is simply text. In addition, no bibliography is appended to the document.
Please, what am I doing wrong? Perhaps my expectation / hope is unreasonable. What I am expecting is that a true Zotero citation be created, as a Word field code.
The background to this immediate problem: what I am trying to achieve
---------------------------------------------------------------------
I wish to create a Microsoft Word document which contains all the 1801 references in my Zotero database, formatted correctly as citations – that is, in Microsoft Word terms as the fields which Zotero creates when you insert a citation; together with the date created and the date last modified. The reason I am doing this is that over the last three or four years, I have maintained in Microsoft Word 2010 a research diary which consists of one row for each significant dated event. Each row contains a date and a comment. Sometimes, the comments include Zotero citations; often they do not. What I need to do now is retrospectively to add a column to this table which gives a citation for each document which I first encountered on that date. It will look something like this:
Date Comment Citations used on this date
20/08/2014 Wednesday Fascinating text, which may also include citations such as (Codd, 1970).
(Gregory and Macgilchrist, 2014)
(Popper, 1979)
In Zotero in Firefox, I have displayed the date added column and ordered the entries by ascending date added. I used the right click ‘create bibliography from items’ drop-down, specifying output mode citations and save to clipboard. Generating this file took a few hours on this low powered computer. I opened the output file in Word, and converted left-parenthesis to left brace, right parenthesis to right brace throughout. I used the resultant file as the input to the RTF scan.
I was expecting – probably unreasonably - that the results of the RTF scan would be true citations complete with field codes. But instead, the output looks like the input – including braces {}.
The workaround that I am employing
----------------------------------
I have created an Excel spreadsheet in which the first worksheet is the results of an export in CSV format from Zotero, and the second worksheet is a list of citations as it has been generated by ‘create bibliography from items’ drop-down, specifying output mode citations and save to clipboard. This second worksheet was generated with Zotero displaying citations in descending order of date modified.
In a thid worksheet I put suitably sorted data from the first and second worksheets side-by-side and carry out a semi-automated (VLOOKUP), semi-manual reconciliation whose outcome is that I know the exact Zotero ID for each item in my Zotero database. I use the reconciled list as I go back through my journal, adding Zotero citations for each date on which I first encountered and stored a reference in Zotero.
Some suggestions, mine – and yours?
-----------------------------------
1. It would have been useful for me in this context to have been able to specify that the results of an export from Zotero should include a list of citations in a particular format, in my case Harvard Anglia Ruskin. Perhaps there is a generic use case underlying my specific application? Perhaps it has already been met?
2. My task has been considerably complicated because many of the authors that I cite have Nordic or Teutonic surnames. In my Windows environment, Zotero does a poor job of exporting items including Nordic or Teutonic characters to CSV format. I appreciate that that this may have something to do with the age and informality of the CSV format, but wonder whether improvements might not be possible.Examples: author Björner becomes Bjørner; Ågerfalk, Pår becomes Ã…gerfalk, Pär. NB: I am using Windows 7 with French keyboard and French system software.
3. I am very open to suggestions as to a better way in which I could have tackled my problem in the first place!
Thank you for reading this far!
My best guess on why it didn't even do that (and your expectation on how that should have looked--take some time, list all recognized citations, ask you to clarify ambiguous ones--are basically correct) is that you saved your file in a Word format (.doc or .docx) not as .rtf. For the scan to work it requires .rtf.
As for your questions
1. Not possible and not going to happen: CSV export didn't use to be possible at all and is a very recent addition in the first place. Export formats and citation styles work entirely separately and combining the two is basically impossible.
2. CSV is a plain text format--it's just text and commas and quotation marks, essentially--so it can handle any character encoding. IIRC you can specify character encoding on importing a CSV into Excel. Selecting utf-8 should get this right.
3. I don't have a good enough sense on what you need--why, for example, a specific Harvard citation style matters--to say much. The easiest would likely be to modify the CSV export translator, which can be done with minimal javascript.
Concerning Nordic and Teutonic characters: the temporary workaround which you have implemented as revised translators works perfectly. There is no need in my case to change character encoding; in any event I use UTF-8 normally.
Concerning RTF scan: I have checked and can confirm that my input document has been saved by Word as a .RTF file and has that extension. Therefore my original problem, that the RTF scan does not appear to be effective, remains. In any event, because as you have made clear it does not insert true citations, it is of no value to me in this context.
The suggested use of the RTF/ODF scan plug-in is something I will certainly take into account when I trial Scrivener in the relatively near future. However, because in the current context it requires specific markup for each citation, it is just as quick for me to insert each citation in Word in the usual way; and that is what I shall do.
The suggestion to modify the CSV export translator, because it involves JavaScript programming, is something I will not allow myself to try before I have completed my current Ph.D. My first supervisor (adviser) banned me from any programming during my doctorate shortly before he died…!
Thanks again.
So I have just spent a frustrating 36 hours or so getting a good working Java runtime environment. I have at last succeeded in reaching a situation where I can create a document in LibreOffice, create citation markers, and use the RTF/OTF scan facility to convert the citation markers to citations. But I still have the following problems and issues:
When I look at the file output by the RTF/OTF scan, I have a large block of citation items, each with a reasonable-looking Zotero ID. However, in some way they are incomplete. If I use insert bibliography, no bibliography is created. If by contrast, I insert a single citation using the menu facility in LibreOffice Writer: that does appear in the inserted bibliography. I have set the document preferences in LibreOffice Writer such that bookmarks are created, rather than reference marks; and that references are stored in the document. I have done the former, because what I really want is to do a once and for all creation of a set of citations in LibreOffice Writer and then carry across that set of citations to Microsoft Word. I insist on the use of Microsoft Word because inter alia it has a very good outlining facility for which no equivalent exists in LibreOffice; and because I will integrate these references into an existing 170,000 word, 350 page research journal which I have created and maintained using Microsoft Word.
So I now want to save the citations in a way in which they are accessible in Microsoft Word. Should I use LibreOffice to create a .docx file? Or should I carry across the .odt file created by the LibreOffice, opening it in Word? Neither of these approaches seems to work, in the sense that I do not end up with true Microsoft Word citations properly using the Microsoft field functionality.
These are the steps which I have followed to get all my Zotero references exported as full Zotero citations, including field codes:
• Set the Default Output Format to Scannable Cite in the Export tab of the Zotero Preferences.
• Create a new Word document.
• Insert citation markers by using ctrl+alt+c to copy them to your clipboard and then paste them using ctrl+v or the right-click context menu, or by drag-and-drop. A citation marker will look like this
{ | Smith, (2012) | | |zu:2433:WQVBH98K}
• Save your document as .odt (Open Document Format-ODF).
• In Zotero, click on RTF/ODF Scan in the gears/action menu. Select ODF (to citations) as file type and select your saved .odt file as the input and your desired output file. Click Next and the plugin will convert your document.
• Open the converted document — by default it will have (citation) in its file name — in LibreOffice.
• In Zotero's LibreOffice plugin toolbar, click on Set Document Preferences and choose the citation style.
• If you want a bibliography, move the cursor to the location where it should be inserted and click Insert Bibliography. This step does not work for me.
• In order to be able to create citations in LibreOffice and reuse them in Microsoft Word, it is necessary to create the citations as bookmarks; Bookmarks is the only option that allows Zotero users of Word and LibreOffice to collaborate. If you don't need this compatibility it is recommended to use the default, i.e. “Fields” (Word) or “Reference Marks” (LibreOffice). I have set the option for bookmarks.
• Check the "Restore References in Document" before sending the document. *** I have no idea what this step is for or how to undertake it. ***
• Change to Fields/Reference Marks in Word. *** I have no idea what this step is for or how to undertake it. ***
I'm not sure where you got that list of instructions--so it's hard to comment e.g. on the last two there.
Summarising so far: first, the good news. I have successfully created significant numbers of markers and successfully converted them to citations, which I can use in a Microsoft Word document.
The bad news is that the conversion route seems to have associated limits.
What works
----------
By a process of reading multiple forum entries, and by extensive experimentation, I have established these steps as those required to get all my Zotero references exported as full Zotero citations, including Microsoft Word field codes:
• If you have not already done so, install LibreOffice. Note that certain functionality in and associated with LibreOffice requires a Java Runtime Environment JRE. Ensure that the JRE is correctly installed and is working. Only then will the LibreOffice Zotero toolbar be visible and be functional.
• Install the RTF/ODF scan feature. The best documentation available for this feature appears to be in the forum entry at https://forums.zotero.org/discussion/29308/5/rtfodf-scan-for-zotero/.
• Set the Default Output Format to Scannable Cite in the Export tab of the Zotero Preferences in your browser.
• Create a new Word document. In order to insert dated citations, create a Word table with one column for the date, one for the citations.
• Insert citation markers by ctrl + shift + c or by drag-and-drop from the Zotero pane in Firefox. A citation marker will look something like this:
{ | Smith, (2012) | | |zu:2433:WQVBH98K}
• Save your Word document as .odt (Open Document Format-ODF.
• In Zotero, click on RTF/ODF Scan in the gears/action menu. Select ODF (to citations) as file type. Select your saved file as the input and give a name to your desired output file — by default it will have (citation) in its file name. Click Next and the plugin will convert your document.
• Open the converted document in LibreOffice.
• In Zotero's LibreOffice plugin toolbar, click on Set Document Preferences and choose the citation style as Reference Marks.
• If you want a bibliography, move the cursor to the location where it should be inserted and click Insert Bibliography. This is a useful test that the citations have been correctly created and are live.
• In order to be able to create citations in LibreOffice and reuse them in Microsoft Word, it is necessary to create the citations as bookmarks; Bookmarks is the only option that allows Zotero users of Word and LibreOffice to collaborate. [In other contexts where you don't need this compatibility it is recommended to use the default, i.e. “Fields” (Word) or “Reference Marks” (LibreOffice).]
• In Zotero's LibreOffice plugin toolbar, click on Set Document Preferences and choose the citation style as Bookmarks.
• Save the file for use as a Word file in .doc format (NOT .docx).
• Open the .doc file in Word. Click the "Zotero Set Doc Prefs" button on the Zotero toolbar. Then choose Format Using: Fields.
• Save the file in .doc or .docx format.
• The Word document so created contains only the table of dates and citations. The content can then of course be merged into more interesting documents – in my case, into my research journal.
Limits associated with the RTF/ODF scan facility
------------------------------------------------
So yesterday I worked hard to create a table of Zotero markers in a .odt LibreOffice file. This morning, I am experiencing difficulty as I try to convert the markers into live citations in a Word document. This is what happens:
1. I start Zotero in Firefox.
2. I can successfully insert citations into a blank Word document: e.g. (Mulinski and Sachs, 2009) ; (Fendt, Kaminska-Labbé and Sachs, 2007).
3. I open a large table of Zotero citations in LibreOffice; I can still successfully insert citations in Word.
4. I switch to LibreOffice Writer. When I click on set document preferences – nothing happens, and I can no longer insert citations into the Microsoft Word document. It seems as though Zotero has ceased to function.
5. Closing and reloading Word is not sufficient to restore the Zotero function.
6. Closing and reopening the Zotero pane in Firefox is not sufficient to restore the Zotero function.
7. Closing and restarting Firefox does restore the Zotero function.
8. If I switch to LibreOffice, an attempt to insert a citation causes Zotero to lock up as it seeks to insert the citation, with a green busy bar stuck on the screen. This is true whether the citation is to be inserted in the document which contains many references, or subsequently in a blank document. It is also independent of whether the attempt is made to insert the citation in Microsoft Word or in LibreOffice – both lock up in the same way.
9. I can clear the green busy bar using Alt + F4, but I then need to close and reload Firefox to restore the Zotero functionality.
The problems appear to be related to the number of citations in the document to be converted. If I copy just a portion of the table, I can successfully convert the markers to bookmarks, open the resultant file in Word, convert to fields and everything works fine.
The workaround is obvious: split the table of Zotero citations into smaller portions. There are currently about 400 rows in the table, one row for each date on which I have inserted Zotero markers. There are 1800 markers in the table.
Unfortunately, when I experiment with reducing the number of rows in the table, the process appears only to be reliable for the first 10 or so rows in the table. These together contain about 10% of the markers, less than three pages in a 27 page document.
Are there any published or unpublished limits associated with this component? Or can you suggest a better way of tackling this problem?
Thanks - I am very encouraged that at least the process is possible.
Reading through the thread, I'm not sure what the final target of the work is, so I should confirm before pitching in. If I understand correctly, you have a single document with perhaps 1,500 entries (as table lines, one per day covering four years), and you have a Zotero library containing 1,800 entries, each of which pertains to one or more of the entries in the document; and you want to merge cites based on the Zotero library into the document. Each Zotero item may pertain to one or more diary entries, and single diary entries may relate to multiple Zotero library items.
If that's a correct description, I don't see how the document and the library content can be merged automatically. It sounds like you would need to add the cites manually, and that ordinary drag-and-drop of individual (non-dynamic) citations from the Zotero library into the document (after sorting the library by date added or modified) would be the way to do it by hand. It would take time, but it would give you control over detail.
But maybe I'm not understanding the objective.
In fact, my requirement is not as onerous as you suppose. With the help of your colleagues @adamsmith and @aurimas, and by using RTF/ODF scan, I have already succeeded in creating citations in a Word document which correspond to markers which I have dragged into a predecessor document. This did involve going via an intermediate LibreOffice document. My residual problem concerns the fact that I need to convert a lot of bookmarks in a Word document to fields; this appears not to work if there are too many bookmarks in the Word document.