What is wrong with my account?

dale_huffman
This is getting ridiculous, something has to be wrong with my account.

I have and EMPTY 'My Library' except for the quick start guide, sync'd with server, EMPTY.
I created a new group and did a sync to get the empty group into firefox.
I imported the refs, copied them to the group, deleted them from my library. Now only the group has any refs in it. (except for quick start guide)

Attempting to sync, first time failed, with empty response from server.
2nd time, I get a conflict resolution, 2113 of them.

WTF? How is there a conflict when nothing existed, this is my initial sync?

What am I supposed to do here, I've wasted countless hours, days now just trying to get a group library setup for our lab. Is Zotero just not ready for this? Is something wrong with my account, PLEASE HELP and lets get to the bottom of this. I still have the conflict screen up if you need to see it.
  • dstillman
    Well, clearly this is something particular to your usage pattern, because other people aren't reporting problems like this.

    First, you're likely making things worse by doing things in such large batches. Leave auto-sync on, and don't perform operations on thousands of items at once between syncs.

    For some reason Zotero is performing a full sync of your library. I'm not sure why that's happening, but you also shouldn't get erroneous conflicts. What are the conflicts you're seeing?

    Also, are you running Zotero 2.1.6? If not, upgrade.
  • dale_huffman
    How can I do that, you don't have an option for importing part of a library? Are you saying I need to edit the library that I'm importing by hand and import a few at a time. That's ridiculous.

    Are you saying there is NOTHING about my account that is broken. Is it carrying around thousands of deletions like the group was, that I had to delete.

    I spelled out exactly what I did, step by step - what is abnormal about that pattern? I can't believe I'm the only person who has ever imported a library moved it to the group library and tried to sync it. Seems to me that is how most people would start using Zotero for a group.

    QUOTE:
    >For some reason Zotero is performing a full sync of your library. I'm not sure why that's >happening, but you also shouldn't get erroneous conflicts. What are the conflicts you're >>seeing?
    Did you not even read my post, it is the initial sync of a newly imported library.

    Is autosync going to sync while importing the library? I will not use the personal library, only the group library, so there is no reason for me to sync 7800 articles in the personal, library, to just then turn around and delete them all and resync the delete, after moving them to the group library. Seems like silly waste of resources. And if there is some transaction history being stored, like "carrying around thousands of deletions", I would prefer to avoid that.

    CONFLICTS: There are 2113 of them. 1st on is titled "Effect of luminal angiotensin II and ANP on early ....". Local Object, Remote Object and Merged Object have exact same Date Added and Modified Wed 04 May 2011 01:59:20 PM EDT, all 6 entries have same date/time. There is a plus minus sign by the Authors, but they appear exactly matching.
  • dstillman
    I am, in fact, trying to help you here. You can be civil, or you can go away. It really makes no difference to me.
    Did you not even read my post, it is the initial sync of a newly imported library.
    I'm not referring to the initial sync. Zotero performed a full download sync an hour and a half after your large upload, and that's what triggered the conflicts. I suspect what happened here is that your initial ~11,000-item sync took so long that it resulted in a rare condition in which the client doesn't register the upload, so even though the data was (eventually) written to the server the client doesn't know about it and thinks it hasn't synced, causing it to pull all data back down on the next sync. That should occur only with very large uploads. We'll be fixing that, but you still shouldn't get erroneous conflicts.

    So that we can look into the conflicts, generate a Debug ID for a sync attempt through the the conflict resolution dialog. (You can cancel after the first conflict.) If you get an error trying to submit the debug output due to its size, click View Output, File->Save As it to a text file, zip the text file, and e-mail it to support@zot....org with a link to this thread.

    As for the sync size, that's just general advice that may not apply to you. It's reasonable to not sync the changes in your personal library if you're going to be deleting them right away, and if you don't want to split up your import into batches, that's fine. Zotero should still work, and if it doesn't then those are things to fix. I'm just pointing out that the more operations you do at once, the longer the sync will take and the greater the likelihood you'll run into trouble that others haven't faced.
  • dale_huffman
    The Debug ID is D1042186009.

    The first thing worrying me is you say ~11,000 - there are only 7800 references in the database??? Does other stuff get counted, or is that just the refs? If so then something is wrong. Also I'm noticing that as soon as I open the Zotero pane in Firefox, there is a major delay, similar to when one clicks on a large collection, but it is on MyLibrary which is empty. ???

    As far as the "be civil" comment, I try to be complete in my posts, to give all the relevant information so that I do not waste others time. Your response, in my opinion was not complete, I will explain.
    I layed out the complete set of steps that I performed. Your reply about the usage pattern didn't state "what" about my usage pattern was abnormal, if you had I would gladly change that pattern. Your second paragraph about large sizes, was very helpful, I will avoid that, as I didn't know it was an issue, except for the delays it causes. The 3rd comment, made it sound like you didn't read my post because you didn't state it was a 2nd download into the client, which you then did on the next post, which helped immensly in my understanding. If you reread your comment you should see what I mean, and would then hopefully understand the frustration on my side, makeing me think that you are wasting my time with a generic post. Thank you for the clarification post, with the complete imformation, I appreciate that. And I am trying to help, the time I've spent debugging, I hope, will help you make the product better. I want Zotero to succeed, as I am a large proponent of open source, non corporate projects. I also guessing that many other people that may have had the problems I'm having, probably just walked away and bought some piece of junk from Thompson, they wouldn't have spent the time to debug anything, therefore you will never hear about them, again I'm guessing, but I'm often correct.

    An additional question - do you have stats somewhere on the sizes of others libraries, imports etc. Just curious to see where I stand with our library.
  • ajlyon
    11,000 is a good deal smaller than the largest collections I know of, which are closer to 40-60K.

    The number includes child items, like notes and attachments.
  • dstillman
    So all the conflicts you're getting appear to be due to the many superscript/subscript characters in your items, which are being represented differently in the client and the server.

    Can you send an excerpt of the file you imported, containing at least some references with subscript/superscript, to support@zot....org, with a link to this thread? If you don't mind and the file is <15MB zipped, you can send the whole thing, and then we can be sure the encoding didn't change at all.
  • dale_huffman
    edited May 6, 2011
    Done - it's gzipped

    This file is exported from Reference Manage Network 10 in .RIS format.

    Let me know if you need any other info.
  • dale_huffman
    I forgot to tell you which refs:

    ID - 4751 on line 116987
    and
    ID - 208 on line 4297

    are good examples.

    And I forgot to mention that we were expecting to lose the super/subscripts and be reinserting them as we use each reference in the new Zotero lib, unless the importer can salvage them somehow, now they show up as blocks.
  • dstillman
    edited May 8, 2011
    Yeah, so the original file you imported was in the rather ancient IBM850 character set, which was the original character set for RIS files that nobody uses anymore. (Reference Manager Network 10 came out in 2002, which is why it still uses it.) Zotero isn't able to auto-detect this character set, and so all extended characters in your file were imported as invalid characters. The database software we're using on the server automatically replaces invalid Unicode characters with a literal dotted square, which is why you're getting many conflicts when Zotero attempts to compare your whole library against the server—the invalid characters don't match the dotted squares.

    We should probably have Zotero perform the same conversion of invalid characters to literal dotted squares, but in your case you're probably going to want to clear your library and reimport that file. You can manually choose IBM850 as the import format in the Export pane of the Zotero preferences before importing the file, and then switch it back to auto-detection after. Alternatively, if Reference Manager lets you select an export character set, choose UTF-8.
  • dale_huffman
    edited May 9, 2011
    OK - I've broken my lib into 8 chunks of about 1000 refs each. I've converted them from IBM850 to UTF-8 with GNU iconv command:

    Example:
    > iconv -f IBM850 -t UTF-8 bl_1000.txt > cbl_1000.txt

    Will attemtp to get this imported, I'll post back with success/errors.
  • dale_huffman
    Can you tell if my sync is running, it's a "restore to server" after deleting everything. Hope that was the right thing to do. It's been going ~5 hours.
  • dstillman
    Why did you do a Restore to Server? That has no effect on group libraries, as I believe I've mentioned to you before—all that did was cause you to re-download your entire group library. That process appears to have hung on the server.

    At this point you should probably just delete the group again, and not invite anyone to the new one until it actually has valid data that's syncing properly with your own account.
  • dstillman
    The alternative would be to just restart Firefox, but even after the download process finished you'd have to delete all the group items manually, and those deletes would remain in the group's history. We may be able to come up with a better solution in the future for dealing with large delete histories in online libraries, but at the moment they'll cause slower initial syncs.
  • dstillman
    And finally, the sync icon continuing to spin for a large download that failed is a bug that we'll fix.
  • dale_huffman
    Oops - yeah I forgot, that button is just so tempting when you have the local side setup perfectly, especially when just deleting everything.

    So - I'm currently importing and syncing? And everything seems a lot snappier, so that IBM850 encoding must have been doing something nasty. My copies from my lib to group lib are faster, the group shows up on the group website, it's syncing...

    I'll let you know if anything goes wrong...
  • dale_huffman
    edited May 10, 2011
    Red Exclamation Dialog:
    Uploaded data not well-formed (Report ID: 73f7019ec5) in dialog box - reported

    Report ID 1437779699

    All was going quite well until this.

    When syncing the 6000-7000th ish items I received this error.
    This file has been converted to UTF-8. Contains a little less than 1000 references.
    I can email the file if you wish.
  • dstillman
    Every now and then we see some weird errors like this in the uploaded XML, due, I believe, to Firefox XML bugs. Restart Firefox and try again.
  • dstillman
    Also, for what it's worth, it looks to me like you converted to ASCII, not UTF-8. Some of the extended characters (e.g. 'æ') have been converted to ASCII equivalents ('ae'), and there are a lot of literal dotted squares.
  • dale_huffman
    More issues:

    I was able to upload the entire library, 1000 at a time, all is working on my main machine.

    SideNote: Had a problem with part of 1 file, if you are interested I can send it. It would not import the whole file, but would import it if I broke it up at a specific point. Said unsupported file format or something similar - don't have exact error any more??? Let me know if you want the details.

    Anyhow:
    I am now trying to sync my account on another computer Windows 7, FF 4.0, Z 2.1.6 and it is constantly failing. The sync starts, runs for ~30 seconds, then memory usage jumps to a range of many hundreds of MBs, seems to be fine. Then the zotero.sqllite file starts growing from 852KB up to 27,392KB, then the file stops growing, the memory usage spikes very quickly s to 1.8 GB and Firefox pukes with a crash, and a dialog saying they are sorry.

    ASCII/UTF-8 Issue: So I used iconv as stated above to convert from IBM850 to UTF-8. My tools state the original file as "binary", not IBM850, how did you conclude IBM850 - maybe that wasn't correct either, maybe its some bastard that Refman put out. This is way out of my realm, would you like to see the original file, (or rather forget that refman ever existed, which is what I'd like to do). I can do it all over again if you have any suggestions on file conversion. Is the ASCII OK or do you think it's causing a problem?
  • dstillman
    Had a problem with part of 1 file, if you are interested I can send it. It would not import the whole file, but would import it if I broke it up at a specific point.
    Yes, please send to support@zot....org with a link to this thread.
    the memory usage spikes very quickly s to 1.8 GB and Firefox pukes with a crash, and a dialog saying they are sorry.
    How much memory is there on this system? Have you tried immediately after a computer restart, with nothing else running?

    We can try to optimize memory usage during large downloads in future versions, but I probably can't tell you more than that.
    My tools state the original file as "binary", not IBM850, how did you conclude IBM850
    I forced my editor to IBM850 (knowing that that was an old encoding used in RIS files) and all the characters became correct.
    Is the ASCII OK or do you think it's causing a problem?
    It's certainly not causing a problem—it's just a loss of fidelity. But if you want to fix as you go, that's fine.
  • dale_huffman
    edited May 12, 2011
    This machine is 32-bit OS with 4GB. And only Firefox, Windows Explorer and the tools to tell me how much memory has been used and Chrome so I can post this with FF locked up.

    I will send file in few minutes.
    Details:
    I tried to import whole file containing Ref #s 7000-7999 -failed
    I broke in half, tried 7000-7499 failed.
    Tried single ref 7000 - it was ok so I continued.
    Tried 7001-7019 - imported OK.
    Got to 7020-7029 - failed import.
    7020-7024 imported OK
    7025-7029 imported OK.
    Went and tried 7020-7049 again and it worked ???
    It then imported everything above 7029 OK.
    So it seems the error was somewhere between 7020 and 7029, but it vanished on the split and was gone after they were imported???
  • dstillman
    All right. As long as it's reproducible we should be able to fix it easily.
  • dale_huffman
    edited May 16, 2011
    the memory usage spikes very quickly s to 1.8 GB and Firefox pukes with a crash, and a dialog saying they are sorry.
    How much memory is there on this system? Have you tried immediately after a computer restart, with nothing else running?

    We can try to optimize memory usage during large downloads in future versions, but I probably can't tell you more than that.
    So everything seems to be working on our high-end computers, 16GB RAM Windows and Linux. Yeah!!!

    However syncing will not work on ANY standard desktop machines, 4GB RAM, Windows, less than 6 month old i7 processors; it always fails with high RAM usage > 1.8GB and a Firefox crash. So I can blame Windows for not providing enough RAM or FF for not handling it properly, BUT is there a plan for you to make the Zotero memory footprint/requirements smaller so we don't run into this limitation, this is going to keep us from using it for group library. If so, any idea when or where on a priority list it would fall. (Crossing my fingers hoping this effects a lot of people...)
  • adamsmith
    How do you do the box around quoted items?
    put it between
    <blockquote>and </blockquote> in Html mode.
  • dstillman
    I'd be curious if this happened on a 4GB OS X or Linux machine—I kind of suspect not—but we'll take a look at memory usage for large downloads and see what we can do.
  • dale_huffman
    Linux 32-bit 4GB
    1st try - crashed firefox
    2nd try - spinning ??? (restarted)
    3rd try after reboot - will let run overnight.


    Mac OSX version took a while - but finally worked.
  • dale_huffman
    Different bug under Linux:
    Report ID: 532842366
    Under account bl_refs
  • dstillman
    I explained that bug to you before, with a workaround:

    http://forums.zotero.org/discussion/17668/report-id-1324158656/#Item_36
  • dale_huffman
    working under Linux -> therefore Windows/Firefox issue.
Sign In or Register to comment.