large amount of files conflicting between local and server db

dhercot

Hi,
I tried to go through the help files and forums on the question.
I have a issue with my database when syncing between the server and my local copy.
I did a debug log of the issue. here : The Debug ID is D1645874769.


Can you help restore my database ? I'd love to have my local copy to overwrite the server version.

The problem:
I tried to import a large xml file (circa 18000 lines) that I had previously exported from REFman to xml.
I used XML because RIS format could not export special characters which I use often as my references are mainly in French.
A first import attempt bugged after 625 lines for an unknown reason, I supect it was because some field in the initial databank contained markers < & > which are used by xml to mark the beginnning and the end of a field. Those have been removed from the databank using xml notepad 2007.
I removed from zotero the references which had been imported imported and started over again importing the 18000 lines.
The programm failed again at around 650 references.
There the syncing started to become impossible, XUL runner crashing.
When I tried to delete the 650 references I realised that around 18000 references where in my trash. No clue how this could happen.
I have reduced storage time in the trash at 1 day because zotero would stop working when I tried to erase them all at once.
And they are now removed from my local copy.
Now syncing will pop a window saying there are above 1600 conflicting references, deleted locally and still available on the server. I have to validate 1 reference conflict at a time.
At the same time, the 17000 references are still present on the server (not in the trash but) in unfiled items which folder I cannot find online. Furthermore i can see only 25 references at a time which is a no go to start deleting them 25 at a time, be it the 17000 unfilled or the 1600 conflicting references.

Can you help ?

Thanks

adamsmith

I don't have a great solution of what you should do, but as for large imports: While doing large imports, it probably makes sense to turn off auto sync and Zotero will _freeze_ which is not the same as crash. You should simply let it run when that happens.

As for solutions - would it be OK to just go back to a completely empty database, or do you have items from before the RefMan import that you need?

dhercot

Hi adamsmith,
Thanks for your reply.

Hmm, I would prefer to keep my references I have added in the separate folders in the database.

Maybe I can do a back up of my local files before you erase my data on the server then let it sync again as my local version seems ok now - on the front stage at least.
We'll have to put it on hold till Monday anyway as I am out of the office till Monday European time by now. You can tell me what I should do on Monday morning before you wake up.

Second. Any chance that someone can help us transfer our refman database to a private zotero group ?
Regards.
David

adamsmith

@Dan - I'm always a bit unsure what to suggest with large libraries on the server. Thoughts?

@dhercot - not sure what type of help you're looking for? Do you want advice (it sounds like you're doing everything right, I'd just suggest letting it run longer)? You want to hire someone?

dhercot

We are considering to hire someone yes. At first to optimise the configuration of our zotero applications (about 10 people working on a shared database - the one we are talking about in this thread). Though the forum and help file are quite clear on the topic. But also because I really struggle with the refman export - zotero import exercise. This means cleaning the XML code after the refman export and renaming fields before importing to zotero to make sure as many fields as possible are recognised by zotero.

I'll give it another try next week if you think what I do is right. Any suggestion on the best time of the day to do it for the server to be available ? Maybe early morning European time as you are sleeping at that time ?

Thanks.

adamsmith

Servers load is generally not the issue - if it is though then yes, European mornings are probably the best time.

Is this all happening with a group library or with a personal library? Solutions/options vary slightly between the two (e.g. you can overwrite your personal library online with the local version, but not a group library).

I could work with you on the setting up the import, yes - contact me (Sebastian) at the e-mail the the bottom of this post: https://www.zotero.org/blog/summer-zotero-workshops/ .

dhercot

Hi, Back in town,...
I tried to stop syncing automatically then empty my dustbin locally but after 14hours Zotero reported a crash :"out of memory when calling [nslObserver ::observe]" nsresult 8057001e NS_error__XPC_JS_THREW_STRING)"Location native frame :: :: :: line 0 » data : no] »]
then a message about JAVA unable to check status.

But even the crash report freezed for more than 30 minutes. I managed to save a print screen about it. if you need to look at it ask me.


I backed up any article I wanted to keep from my local library using zotero RDF export. (I know it's not the best).

So can you do as you proposed to reset/empty my own personal library.
The group observatbru.be has no problem I think.

After this, I'll contact you for the import.

thanks for your help.

Best.

adamsmith

I can't clear your server database - Dan would have to say whether that's even possible on their side - I don't think so.
I was asking about your situation to understand what would be feasible solutions.
If your local library now looks like you want it to - including with the trash emptied - I'd try to use "Reset to Server" from the "Reset" tab of the Zotero preferences and see if that goes through.

dstillman

If you're saying it crashed trying to empty the trash, you can delete in batches by selecting items from the trash and pressing Delete to remove them permanently.

Also, though, I see repeated large downloads from your account, the same every time. Are you force-quitting Zotero? You shouldn't — that would cause it to just try to sync the same amount the next time.

dhercot

Thanks for your support.
The final word for now :
I exported my references using Zotero RDF
Then I restored an old backup of zotero files I add with no references at all in it. And imported back my references.
After this I did a "reset to server" as suggested. Now it seems to be all right. I'll do a new test of importing the refman references to zotero in a few days. We can close this discussion for now. All the best.