2.0b7 WebDAV sync error

Bagstone

I've seen several WebDAV topics in this forum, but no posts regarding my problem in beta7.

Error ID: 1464402346

Or just here the error messages: http://nopaste.org/p/aqMXh2olB
Component returned failure code: 0x80004005 (NS_ERROR_FAILURE) [nsIFile.remove] ...

Happened to anyone else? I've looked on this forum, but didn't find anything about this particular error in 2.0b7.

dstillman

There've been a few other reports of that. Could you generate debug output for the sync attempt from the Advanced pane of the Zotero prefs, submit it to the server, and post the Debug ID here? Thanks.

Bagstone

The Debug ID is D496861405.

Thanks for your efforts :-)

Bagstone

Any news on that? Can't get sync to work at all, at the moment...

dstillman

We're looking into it, but, just to be clear, metadata sync should continue to work for you. This is only a file sync issue (though obviously it's still a problem if you're trying to access the same files on multiple machines).

dstillman

I haven't tried to reproduce this yet, but you might try removing 2HGHQVF2.zip and 2HGHQVF2.prop from your WebDAV server to see if that fixes the problem. You'd need to do a Reset File Sync History on the other computer to sync that file back up.

Bagstone

Yes, metadata sync works. But I need the file sync to convince my colleagues to use Zotero, too ;-)

Btw, I know it's off-topic but I don't wanna open a new thread again: As you can see in the debug log, my file storage is something like bla.com/zotero/ for login. As the beta7 changelog says, it's fixed now in terms that you can verify those URLs; but you can't put them into the box, it always gets cut to the base URL. Seems like an overeager form input check. I have to put the WebDAV URL into the preferences via about:config on every machine I install Zotero on... is this on the list to fix for next release?

And for your hint: I tried that (just removed the file and put it back later manually), but it didn't work. In the temp folder there was the file again, with something like the 404 html error content in it (obviously). What exactly led you to the assumption that this file is corrupt (or causing this error for some other reason)? I browsed my debug log but couldn't find a hint.

Do you have an idea what's causing this whole problem?

dstillman

What exactly led you to the assumption that this file is corrupt (or causing this error for some other reason)?

(3)(+0000000): Finished download of D:\My Documents\zotero\tmp\2HGHQVF2.zip.tmp

(2)(+0000000): 2HGHQVF2.zip.tmp is not a valid ZIP file

Can you provide another Debug ID?

As the beta7 changelog says, it's fixed now in terms that you can verify those URLs; but you can't put them into the box, it always gets cut to the base URL.

What's the actual problem you're having? The /zotero is hard-coded into the form and the code. You should only need to use about:config if the remote path is actually /zotero/zotero.

Bagstone

To the folder thing: please forget everything about that. The problem vanished as I turned on my brain...


Here two new debug IDs. I cleared the tmp folder before clicking the sync button. The second debug ID was with the first 5 zip+prop files on my WebDAV folder deleted.

1st: The Debug ID is D59688648.
2nd: The Debug ID is D1230005291.

Edit: Just to add. I randomly downloaded a few files and tried to unzip them - didn't work. Every zip file seems to be corrupt. The upload sync was done on a Mac OS X Snow Leopard machine, I don't know what zip library you're using... but maybe it has got something to do with that?

dstillman

Can you send one or two of the ZIP files to support@zot....org, with a link to this thread? Thanks.

Also, in case you used Firefox to download them, it's worth trying a separate browser/program as well.

Bagstone

Mail sent.
I tried your pre-edit suggestion (renaming the temp files and extracting them), didn't work.

Sync fails on both Mac OS X and Windows XP machines by the way.
And I'm accessing WebDAV with TotalCommander with the WebDAV plugin from their website. Tried downloading with IE8 as well, same problem, zip file corrupt...

dstillman

Well, I'm not sure how this happened—possibly some sort of network glitch—but you might as well delete all the files on the WebDAV server, do a Reset File Sync History from the Sync->Reset pane of the Zotero prefs, and try uploading again.

Bagstone

I tried, didn't work.

Seems as if every zip file my Zotero creates is corrupt... so any suggestion?

dstillman

Well that's a new one.

To see if this is a local problem or a network/WebDAV problem, delete a few of the larger files from the WebDAV server, reset file sync history again, open up the 'tmp' directory within the data directory, and, after starting a sync, copy (don't move) any zip files that appear in the directory to another folder. You'll want to wait until the file size stops increasing. (The reason to pick larger files is that it should give you more time to copy them before uploading is complete and they're deleted.)

Then see if those copied files are also corrupt.

Bagstone

Ok... I have to admit... I don't get it :-(
I don't get what you're trying to achieve.

"Copying during upload" is complicated, since that Mac is at university and the upload takes like a millisecond ;-)

What I did was:
1st) Did a complete history reset and uploaded again from the Mac. Zip files again corrupt, sync didn't work.
2nd) I went home to my Windows XP machine, created a new subfolder on the WebDAV space and sync'ed to a second account (this one). The zip file on WebDAV is not corrupt. So it seems to be a problem only happening on the university's Mac.

Any ideas? Which zip routine do you use in Zotero that it makes a difference on which system the upload sync is happening?

dstillman

Ok... I have to admit... I don't get it :-(
I don't get what you're trying to achieve.

Well, I said above: to see if this is a local problem (e.g., a problem with zipping or writing to the disk) or a network/WebDAV problem (e.g., connections being closed prematurely, data being corrupted in transit). If the files aren't corrupted before uploading, it's the latter (which seems much more likely).

If uploading is too quick, try this:

1) Do the steps you did above to delete a few large files and clear file sync history. Don't sync yet.

2) Create a directory on your desktop called 'zoterozips'.

3) Open your Zotero data directory in a Finder window. If the 'tmp' folder doesn't yet exist because you've recently restarted Zotero, create it.

4) Open a Terminal window. Type bash and hit Return.

5) Drag the 'tmp' folder from the Finder to the Terminal window. It should paste the full filesystem path of the folder to the command line with spaces preceded by backslashes. Hit Ctrl-c.

6) Type/paste the following all on a single new Terminal line:

while true; do cp -v /path/to/tmp/folder/* ~/Desktop/zoterozips/ 2>/dev/null; sleep .1s; done

The /path/to/tmp/folder/* should be the path that was automatically pasted in above followed by a slash (if not there already) and an asterisk. If you're using the default data directory location, it should be something like /Users/dan/Library/Application\ Support/Firefox/Profiles/a5f2h5a6.default/zotero/tmp/*

7) Hit Return, and then begin a sync. If everything worked properly, you should see some ZIP files appear in the zoterozips file on your desktop.

8) When the sync is done, press Ctrl-c in the Terminal.

Then test the ZIP files and see if they're corrupt.

Bagstone

The zip files are corrupt :(
So it's definitely no WebDAV problem...

dstillman

Can you provide a Report ID from Report Errors from the Mac, after trying a sync that involves uploading a ZIP? Also be sure to try disabling all other Firefox extensions.

Anything else unique about that Mac's setup? Is the Zotero data directory on a local, external, or network drive?

Bagstone

Not at university today, so report ID will have to wait until Thursday, at last.

I can't tell you what's "unique" about the Mac, I don't own a Mac myself, it's just the PC at the university lab. There are a bunch of Macs, I tried on different ones, it's the same error everywhere.

It's a Snow Leopard Mac with basic user rights, quite limited as far as I can tell. The Zotero data are on a network storage, otherwise it would be very complicated if you get into the lab and your PC is occupied ;-) I guess it's a basic network setup as for every company/university...

dstillman

My best guess is that it's something with the network storage. Would you be able to temporarily switch the data directory to somewhere on the local hard drive, copy your files over, and see if the problem goes away?

Bagstone

No, can't do that. No write access to the local hard disk. :(

dstillman

USB flash drive?

Bagstone

Great idea. It worked!

Strange. So, uploading files from a network home folder on a WebDAV space corrupts the zip files on Mac OS X.
Funny thing: The hard disk might even be the same (at least it's the same network address for both WebDAV and the network home folder)...

Ok, the USB flash drive was a nice try but it's no permanent solution. Any idea what might cause this? I'm not really in to Mac OS X.

Bagstone

Am I the only one who has encountered this problem?
Any idea how to fix this?

Zotero is great, I love it, but since everyone at our chair is using the networked home folder, no one can upload any new files. So I can only add new files from my Windows PC (or an USB stick as workaround which is akward and annoying since Firefox restart and moving the DB takes like forever).

dstillman

Just as a reminder (regarding your Oct. 27th message), this has nothing to do with WebDAV—we determined that the files are being corrupted before upload. Presumably you can reproduce the same behavior using the "Zotero" file sync option, e.g. with group file syncing.

I don't recall seeing anything else like this, but this may not be something that we can fix—it could be a problem with the network setup or a bug in Firefox.

What's the exact network environment? What kind of server? OS X Server, Windows Server, a dedicated NAS device? If NAS, what protocol? Have you spoken to your network administrator about the problem?

Bagstone

The network administrator is kinda busy, I haven't spoken to him, no.

It's an OS X server, as far as I know. I've tried different OS versions (Tiger, Snow Leopard) and different Firefox versions (well, Zotero doesn't work with very very old versions).

I'm just curious of how the Zotero zip routine works. I don't get why zipping a file from my local disk (USB) works, but zipping a file from the remote server doesn't. If it would be an OS or browser problem, I don't see why this happens only in one of those setups. So, how does Zotero zip the file? Anything I could check there...

dstillman

If it would be an OS or browser problem, I don't see why this happens only in one of those setups.

Are you actually accessing the same network share from Windows? Not WebDAV—the network share. If not, then Windows is irrelevant. (A network share and WebDAV are entirely different protocols and can't be compared, even if they're being served from the same server.)

If you can find another Firefox extension that uses Firefox's built-in ZIP writing routines, you can test if the problem occurs there as well. I don't know of any offhand, but that'd be a good test of whether this is something we'd be able to fix ourselves.

dstillman

I don't know if this uses the built-in ZIP-writing routines, but you could try it:

https://addons.mozilla.org/en-US/firefox/addon/212

Edit #1: Actually, this extension predated ZIP writing in Mozilla, so I suspect it uses its own component.

Edit #2: Yeah, it looks like it uses its own platform-specific code. You could still try it for good measure, but it wouldn't be particularly enlightening if it worked.

dstillman

This one looks like it uses native components:

https://addons.mozilla.org/en-US/firefox/addon/2109

Bagstone

Thanks for all your efforts. I'll try this out tomorrow at university. Btw, with "different OS" I meant different OS X versions. There is no Windows at our chair, just Apple everywhere ;-) I know that WebDAV is something completely different, but as you also already said, we eliminated the WebDAV as error reason.

dstillman

Well, if you were referring to different OS X versions, then I'm confused, because you said it happened in only one setup. So are you saying that it worked in other OS X versions?

If by "setup" you just meant network share (on any OS X version and any Firefox version) vs. non-network drive, well, it just means that something about Zotero/Firefox/OS X's interaction with the network drive is causing the problem.