Problems with pywebdav

dan.s

I configured a WebDAV server using pywebdav:
https://code.google.com/p/pywebdav/

I was able to connect to this server using cadaver and everything seemed fine.

I then configured zotero to use this server. It connected fine and uploaded all my files. Along with uploading files (.prop and .zip), it created a file called lastsync.txt.

However, on the next sync, an error was reported from Zotero: invalid last sync id

This error was reported by hovering over a red exclamation icon near the sync icon.

Then it appeared that the file lastsync.txt was actually deleted in this process, as it no longer appeared on the server. Future Zotero sync requests failed with a more generic failure (as there was no longer any lastsync.txt, as opposed to one that it originally thought was malformed).

I tried this process multiple times (starting from scratch so that the files would be re-uploaded). I tried where the second sync was from the same box w/ Zotero where the files were originally uploaded from, and then I also tried using Zotero on other computers for the second sync. Always the same problem: The second sync attempt reports an "invalid last sync id", even though that file exists on the server with an ID, and then that file would be deleted, and subsequent syncs don't work.

I then tried setting up an nginx WebDAV server. It took a bit of tweaking to get working (required a module that enables PROPFIND and OPTIONS). Anyhow, I got that all set up and everything works. Zotero can sync fine, and there is no issue with "invalid last sync id".

I thought it would be worthwhile to report that there are issues with pywebdav.

Any ideas why that particular server has problems, but nginx does not?

Thanks.

dstillman

We'd need to see a Debug ID for a pywebdav sync that returned the "Invalid last sync id" error.

dan.s

Hi Dan,

Debug ID D175435639 was captured under the following procedure:
- verify WebDAV server. Since the server was a fresh set up of pywebdav, it detects the server and creates a zotero folder.
- sync content (this seemingly goes well, as all files are pushed onto server).
- try again to sync content and get an error: invalid last sync id

dstillman

Your server is corrupting the lastsync.txt file somehow.

(3)(+0000000): HTTP PUT jeyY4OC9MLtS9eqhW4NaRT10ZD4dqV to http://desktop:8008/zotero/lastsync.txt

[...]

(3)(+0000053): HTTP GET http://daniel:********@desktop:8008/zotero/lastsync.txt succeeded with 200

(3)(+0000000): 1e
jeyY4OC9MLtS9eqhW4NaRT10ZD

[...]

(3)(+0000000): 'message' => "Invalid last sync id '1e
jeyY4OC9MLtS9eqhW4NaRT10ZD'"

No idea what '1e' is, why there's a newline, or where the rest of the string is, but that won't work.

dan.s

Thanks for the quick reply.

It appears that the problem is that pywebdav is sending "Transfer-Encoding: chunked". Since Zotero's request is HTTP/1.1, this seems acceptable, but pywebdav returns an HTTP/1.0 *with* chunked encoding, which is possibly against the specs.

The server is not corrupting lastsync.txt, since the file is fine on the server, and transfers fine through cadaver. The issue seems to be a consequence of the chunked transfer encoding (nginx does not use chunked encoding in my set up, which could explain why it works).

Does Zotero support chunked encoding for HTTP/1.1 responses? If so, is this issue a result of pywebdav saying it's giving an HTTP/1.0 response and using chunked encoding?

Thanks.

dan.s

I just changed the pywebdav code locally to never use transfer-encoding: chunked, and Zotero worked fine, so that is the issue.

In doing so, I noticed that there is a config flag (that must be entered with a config file, as opposed to a command line switch) to turn off chunked encoding, so I will just use that as opposed to forcing it off in their source.

Like I mentioned, I don't think pywebdav is conforming to HTTP specs by combining an HTTP/1.0 response with chunked encoding. It appears that cadaver handles that case. I'm not sure if Zotero is having a problem due to HTTP/1.0 combined with chunked encoding.

aurimas

Zotero itself doesn't handle the underlying HTTP protocol, Firefox/XULRunner handle it instead. If Zotero is having issues, so would Firefox.

anderslundstedt

I had the same problem. The command line switch -T can be used to turn off chucked encoding:

-T, --noiter Deactivate iterator. Use this if you encounter file corruption during
download. Also disables chunked body response.

dan.s

anderslundstedt, thanks for the update. I had not seen the -T option until you mentioned it. I often rely on the man pages for checking options, and it appears that on my Ubuntu system, it the man page is from 2011-07-10 and corresponds to version 0.9.4 of the software, even though the version installed is 0.9.8. I also missed that option when I skimmed through the source code a few weeks ago. Thanks again.

dan.s

(I will likely start using --help more often now for other software, or checking the version of the man page)