Errors when I attempt synchronisation using Zotero 2.0b6.1

I have just tried uploading my library to the online Zotero server, using Zotero 2.0b6.1 and have failed (Report ID: 498962374). There was no repeat of the earlier problems with Script stack overflow errors, but the error report shows a large number of Javascript errors. Could the fact that my library is a 'hybrid' i.e. a majority of the references in it have been captured by Zotero in the normal way, but there is a significant minority of references have been imported from Biblioscape 6, be a factor in the generation of these errors, and is there anything I can do about it.

Many thanks,

Al
  • There are no Zotero errors in there.

    Upgrade to 2.0b6.2 and try to sync. If you get an error, tell us what error you see when hovering over the sync icon, and also send in another error report before restarting and post the Report ID here.
  • I have upgraded my desktop computer to 2.0b6.2 and tried a new sync. This again failed, so I have sent in the error report (Report ID: 242358952). I am unable to tell you what the error is that is reported by making the cursor hover over the sync icon, as no message ever appears when I do that. It may be worth mentioning that no message appears if I hover over any of the icons, so it is not just the sync icon which has this particular problem. It might also be relevant to say that this particular computer is running very low on memory, and occasionally messages appear about having to factor in more virtual memory, though I have not actually witnessed that when running Zotero.

    As a cross-check, I upgraded to 2.0b6.2 on my laptop, which is greatly more powerful than the desktop machine, and tried to replicate the experiment on that. I did indeed get a sync failure when I tried to sync that, but with a very different set of errors. I have submitted that second error report to you (Report ID: 79071089) to see if there are any indications as to what the problem might be. Both of the computers are running on a network which requires a proxy in order to use it.

    Al Gemmell
  • Both error reports show the same "script stack space quota is exhausted" error you were getting previously. Looking at the line number, however, it appears that this is happening while trying to process a particular note in your library, though I couldn't tell you which one. Is it possible that you have a note with a massive amount of text? If so, you may need to split it into multiple notes. I'd also be very interested in knowing how many characters long it is.

    It's actually entirely possible that you'd get the same error from just trying to load the note in the interface, though possibly not.

    If you're having trouble finding the note, you could upload your DB to the DB Repair Tool and e-mail the Upload ID to support@zot...org, and we can run a query for the largest note length. If you do this, you can ignore the download links the tool provides.

    We may be able to work around this in the code.
  • I have looked through a number of notes (there are far too many to scrutinise manually), but the largest text note I have found to date has 15,649 characters, including spaces. However, i have come across some notes which are huge - the largest i have is 929,263 characters, including spaces! I suspect that there may be a number of this order of magnitude, as I have found a pattern. Those enormous notes are actually sequences of figures generated when I have pasted a diagram copied from a pdf file into a note. I have used this technique in some cases where there is a key diagram in a paper (especially, though not exclusively from Nature) which i want to keep a copy of without downloading an entire pdf file. Although the graphic is pasted into the Note, it does not seem to be stored as a graphic but as a huge stream of characters.

    As a check, i will upload the DB to the DB Repair Tool, and email the upload ID to the Support site.
  • I have tried twice to upload the database to the DB Repair Tool, and that has failed both times. The error message which appears is:
    Fatal error: Allowed memory size of 314572800 bytes exhausted (tried to allocate 289267336 bytes) in /usr/share/pear/Archive/Zip.php on line 2190

    This surprises me, as according to the computer here, the size of the zotero.sqlite.zip file is only 35.312kb. Is there an alternative way of sending the file to you for inspection?
  • Although the graphic is pasted into the Note, it does not seem to be stored as a graphic but as a huge stream of characters.
    I found your database on the server. You indeed have RTF character sequences embedded in your notes, and those are causing the errors. A couple questions:

    1) Is there a reason you've created a number of these? As far as I know, there's no way these would appear correctly as pictures within Zotero notes, at least after a reload—though you seem to suggest that they appeared as graphics immediately after pasting? (That would be surprising but not impossible.)

    2) What PDF reader did you copy these from? I think there's probably a bug in either it or in Firefox, since data on the clipboard is typed, and Firefox doesn't support RTF data, so either the PDF reader is identifying the data incorrectly or Firefox is grabbing data of the wrong type. The former seems more likely (though I'm not sure how or why a PDF reader would generate RTF data in the first place). At any rate, you shouldn't really be able to paste in raw RTF data.
    This surprises me, as according to the computer here, the size of the zotero.sqlite.zip file is only 35.312kb.
    The database itself is 181MB. The notes table itself is about 100MB.
  • 1) Is there a reason you've created a number of these? As far as I know, there's no way these would appear correctly as pictures within Zotero notes, at least after a reload—though you seem to suggest that they appeared as graphics immediately after pasting? (That would be surprising but not impossible.)

    There are a couple of reasons why I tried to copy images - most commonly maps from papers relating to my research interests in glacial history (Ice Ages etc.). The first was an attempt to save space, to keep the size of the database to a minimum, by taking copies of only those elements of some papers which I might want to use for teaching purposes, rather than have the entire paper in the database. The more important reason for 'extracting' diagrams and pasting them into notes was that I could then title the note appropriately from the content of the diagram, and that would allow me to search the database for suitable diagrams if i wanted to put some together for a class. I thought this might be rather faster than searching the database for figure captions contained within complete PDF files.

    Could the upgrading of the Zotero database from 1.0.10 format to 2.0b6 format possibly have transformed the captured figures into RTF character sequences?

    2) What PDF reader did you copy these from?

    I have been using Adobe Reader 9, though I would think some of the diagrams could have been copied from PDF files when I was using Adobe Reader 7 or 8.

    The size of the database you recorded (181Mb) as receiving from me is the same as that I have here, so i think the entire database must have got through to you intact, despite the error message I received.
  • There are a couple of reasons why I tried to copy images
    Sorry, I don't mean conceptually. I just mean...was this actually working for you, technologically speaking? Did you, immediately upon pasting and later upon returning to the notes, actually see the images rather than the RTF sequences? Because we're struggling to think of any way that this could've actually worked, since Firefox doesn't support RTF rendering (and notes in 1.0.10 were just plaintext).
    The size of the database you recorded (181Mb) as receiving from me is the same as that I have here
    Yes, I just meant that 1) that was the reason it was failing and 2) that's pretty big for a notes table. It may simply be due to having an awful lot of notes, but it may also be from having a number of these large pasted images embedded (in some form or another) in the database. For what it's worth, when you attach PDFs or other documents, you're not actually increasing the size of the database itself (other than for full-text indexing)—the files are stored separately from the database.
  • As an experiment, I have just tried to replicate my actions to copy an image to a note. I am using 2.0b6.2, and I had not tried to do this before. I took a figure from a pdf file for a paper I have in my database (Ackert et al), opened it using Adobe Reader 9.1.2, and then used the Edit, Copy tool to take a copy of a coloured map in the paper. I then went back to Zotero, selected to Add a note to my Ackert et al record, the window opened, I right clicked in the Notes window, selected Paste, and the diagram appeared in the Notes window.
    As a further check, I then closed Zotero, shut down Firefox, restarted the browser, reopened Zotero, scrolled to the relevant reference, and the map is still there in the Note!

    One thing I did note this time is that all that has been captured by the Copy is the actual artwork. Place names etc. on the original map have not been copied across, though I presume this is to do with the way Adobe Reader functions rather than anything to do with Zotero.

    Your comment about not being sure how it might have worked in 1.0.10 as the notes there were plain text makes me realise that I do not recollect going back into any of the diagrams I had saved as notes - I have been presuming that because I could see the image when I pasted it into a note, that was it secure for the future. Today's experiment is the first time I can remember consciously going back to a note later to see whether or not the image was still there.
  • OK, this was actually discussed in a recent thread.

    In your new note, can you click the HTML button and look at the source code? Based on posts in that thread, I'm guessing that you'll find an <img> tag with a file:// link to the image in your temp directory. This behavior is a fluke that should be prevented if possible, since those temp files could be deleted at any time. But it's at least dealing with HTML, and so seemingly not related to the RTF issue.

    At this point, I'd recommend going through those large RTF notes (which you can probably find with a search for 'rtf'), copying them into text files in Notepad, and saving them with .rtf extensions. (You may need to adjust the extension outside of Notepad after saving, and possibly adjust your Windows extension display settings as well.) That should give you RTF files that you can open in WordPad or Word, which you can then drag back into Zotero as file attachments. Once you've confirmed that that works, you should delete the RTF notes and empty your Zotero Trash folder. After that, with any luck, syncing will work.
  • I have done what you suggested - clicking the HTML button and looking at the source code. As you surmised, there is an <img> tag linking to a .png file in the temp directory.
    I will now start locating all the large RTF notes and following your suggestions re converting them to file attachments.
    Thanks for the assistance with this great product.
  • I have now located all the large RTF Note files, and have deleted each of them by right-clicking on the Note symbol and selecting 'Delete this item....'. Once that was complete, I went to the Trash folder, right-clicked that and selected 'Empty Trash'. This ran for a few moments, and then an error message appeared, which i have sent to you (Report ID: 706139067). At the same time as the error message appears, the central Zotero panel went blank and then a message saying:

    An error has occurred. Please restart Firefox. You can report this error by selecting "Report Errors" from the Actions (gear) menu.

    This is what I have done. No matter which folder or sublibrary I select, the same message appears in all of them. Restarting Firefox brought back the reference library, but when I tried to empty the trash again, the same thing happened as before. I am not sure what to do now to get rid of the trash.
  • You'll need to wait for a fix.
  • Well, one (completely unsupported) possibility would be to use an SQLite client such as SQLite Manager to open a copy of your Zotero database, find the notes in the itemNotes table, shorten them, and then close Firefox and copy the modified database into your Zotero data directory (after moving your original database to a safe location elsewhere, obviously). If you got them all you should then be able to empty the trash.

    It's possible that SQLite Manager, being within Firefox, would exhibit the same problem, though, so a different SQLite client might be necessary.
  • I have now used SQLite to remove all notes which include large rtf strings from the database, and successfully cleared the trash out. I then attempted a synchronization with the Zotero server, but got an error message (Report ID: 28548903), suggesting that there is a problem at the server end with some uploaded data. Is there something else in the data that is problematic? I have a feeling that the clearing out of large rtf strings may have left a number of empty notes. Would that be enough to cause an error, and if so, should I locate these notes and delete the actual notes rather than just the content?

    Al
  • On loading Zotero this morning, I discovered that apparently a sync was still in progress from yesterday. That sync continued to run for a couple of hours, but has now encountered an error (Report ID: 977043876), again the message referring to problems processing uploaded data. Are there any indications as to the nature of the problems, and what the resolution of the problem might be?
  • It's an issue we're having on the server with very large libraries. (Yours has over 15,000 items, including notes and attachments.) There's not really a workaround, but we're working to fix the problem. In the meantime, you'll want to turn off automatic syncing.
  • I have now installed Zotero 2.0b6.4, and tried another synchronization. It again failed (Report ID : 1544895612). Hovering the mouse cursor above the synchronization icon with the yellow triangle gave the message that 'Error processing uploaded data (Report ID: 55c37f82)'. Am I correct in thinking that this is again the problem referred to earlier in this thread regarding very large libraries still being an issue as far as synchronization is concerned?
  • I have been trying to sync at various dates since I last posted to this forum, but always got a message that the database was too large. Today, I tried twice. The first time the error message was that the link to the server had timed out, and the second time (Report ID: 575859474) the message was that there was an Invalid Response from the Server.
    Is it possible that I have got the sync settings wrong (I have no WEBDAV server set up at present).
  • I have also been getting the "script stack space quota is exhausted" error (Report ID: 1766999967) when I try to sync (Firefox 3.5.6; Zotero 2.0b7.6). I have read this thread, as well as the related one (titled "script stack space quota is exhausted").

    How can I go about determining which note is the offending one (i.e., the one that's too long)? I have tried to search from the date of my last sync, and I have tried deleting some lengthy notes, but that still hasn't resolved it.

    Many thanks!
  • oxfordhebrew: Did you empty the trash after deleting the notes? We'll see if we can add a check for this, but, in the meantime, you could find the problematic notes by opening a copy of your Zotero database in an SQLite client, as I suggested to Al above. You'd want to run a command like this:

    SELECT title, LENGTH(note) FROM itemNotes ORDER BY LENGTH(note) DESC LIMIT 5;

    The top note should be the offending one.
  • Many thanks for the reply; this sounds helpful. I have not emptied the trash after deleting the notes because I cannot. I have found at least one other thread related to the same issue, so for the sake of organization, I'll post my query about emptying the trash there.
  • Your issue has nothing to do with the issue on that other thread, which, as I noted there, was fixed in 2.0b7.5.

    The error you're getting trying to empty the trash is the same one you're getting here, so you'll need to wait for a fix.
  • Thanks so much, Dan. I did a little surgery to remove that one large note using SQLite Manager. As it happens, I had the text saved in a separate document, so no loss of data for me. Anyway, syncing and trash emptying work like a charm now. Many thanks again.
  • Great—glad to hear it.
  • The "script stack space quota is exhausted" error is now fixed in the latest dev build, and the fix will be included in the next official release.
This discussion has been closed.