syncing files does not work (Debug ID D1715543768)

tom7320
Hi!

I tried to sync all my attached files with my webdav server (apache with mod_webdav (Apache/2.2.8 (Ubuntu) DAV/2)) but it does not work. The first sync works fine but newly created attachments do not show up on another computer. I followed all the steps on
http://www.zotero.org/support/kb/files_not_syncing
but still it does not work. Thus I created a debug id and send it to you. The file definitely exists on the webdav server...

TIA!

Thorsten
  • dstillman
    Are you sure you generated that Debug ID from the right computer? It simply shows lots of skipped uploads because the files already exist on the server. And it shows the attachment you tried to open at the beginning being downloaded successfully.
  • tom7320
    Hi!

    The skipped uploads are probably a result from the reset of sync history. The attachment was _not_ downloaded successfully, but I will try again in a minute....
  • dstillman
    The file you tried to open at the beginning—in the '9FDEQXTX' directory of the 'storage' directory within your Zotero data directory—was downloaded and extracted, according to the debug output. Your debug output does not show an attempt to open the file after the sync, as the instructions request.
  • tom7320
    Here is what I did:

    1. Added a library item with pdf attachment on my work computer (Win XP). The upload took forever ("waiting for sync server"), but finally the upload finished (I had to restart firefox for this to work). The file is definitely on the webdav server:
    http://billroth.ath.cx:8888/tmp/01.png

    2. At home on my Mac I hit the sync button. As expected the library item appeared in the library but I was not able to download the attached pdf ("Attached file could not be found..."):
    http://billroth.ath.cx:8888/tmp/02.png

    3. I did the same with debug log enabled (D132013932). No success.

    4. Then I did a reset of the sync history (D807527617). Again "waiting for the sync server" took forever but it finally finished. After that the attachment was there.

    Thx for your efford! Please tell me if I can send you any further details!

    Regards,

    Thorsten
  • dstillman
    At home on my Mac I hit the sync button. As expected the library item appeared in the library but I was not able to download the attached pdf ("Attached file could not be found...")
    Then I'll need a Debug ID for that first sync when you expect it to be pulled down.
  • tom7320
    Since it starts syncing automatically this is a log from the beginning. I hit the sync button a couple times and tried to download the pdf:

    D1610967115
  • tom7320
    PS: Why does the "upload accepted - waiting for sync server" takes so long? It is a really fast webdav server (since it's located almost next to me... ;-)?
  • dstillman
    edited July 20, 2010
    (3)(+0000000): HTTP OPTIONS for http://********:********@billroth.../zotero/

    (3)(+0000067): HTTP HEAD http://********:********@billroth.../zotero/lastsync

    (3)(+0000020): ===>200
    The 'lastsync' file on your WebDAV server isn't being properly updated when you upload a file. This is most likely a problem with your WebDAV server.
    Why does the "upload accepted - waiting for sync server" takes so long?
    That's unrelated to your WebDAV server—that's the Zotero data server, the speed of which varies based on how much data you're uploading, how busy it is, and various adjustments we make constantly.
  • tom7320
    Hm, I agree:
    http://billroth.ath.cx:8888/tmp/03.png
    It's sorted by date/time.
    But I don't think that this is a webdav server problem. It's an up-to-date installation of mod_webdav on apache 2.2. If zotero correctly wrote the file each time the time/date stamp of the file would be correct... When I open a text file on the webdave share, change it on my deskop machine and save it again, the time/date stamp is updated correctly. Maybe you should save the last-sync-information within the file?
  • dstillman
    If you provide a Debug ID for an upload I can look at it, but when people have problems with WebDAV, it's almost always an issue with the server. If this was a problem with a stock Apache install we'd have many reports of it.
  • tom7320
    Hm, but it is a stock Apache (Apache/2.2.8 (Ubuntu) DAV/2 mod_python/3.3.1 Python/2.5.2 PHP/5.2.4-2ubuntu5.10 with Suhosin-Patch mod_ssl/2.2.8 OpenSSL/0.9.8g) on Ubuntu 8.04....
    I tried an upload again. This time I had an "Error processing uploaded data (Report ID 61071927)". Bugreport 1576664224. I had debug log enabled: D507521579.

    THX again!
  • tom7320
    PS: After I restarted firefox the upload worked...
  • dstillman
    edited July 20, 2010
    Are you saying the problem is solved? If not you still need to provide a Debug ID for the upload file sync.
  • tom7320
    No unfortunately it's not. I encountered another problem.... I was not even able to upload the pdf. It worked after I restarted firefox. I'll try to provide a Debug ID for the upload soon...
  • tom7320
    This is the upload:
    D1293891148
  • dstillman
    (3)(+0000000): HTTP PUT '' to http://********:********@billroth.../zotero/lastsync

    (3)(+0000016):

    (3)(+0000000): ===>204<===(number)

    (3)(+0000001): HTTP HEAD http://********:********@billroth.../zotero/lastsync

    (3)(+0000019): ===>200<===(number)

    (3)(+0000000): Last successful storage sync was Tue Jul 20 2010 13:51:42 GMT+0200</blockquote>
    It's your WebDAV server, not Zotero. The server is returning a 204, and the timestamp isn't changing. I'm afraid I can't really help you more with this—we can't support individual WebDAV servers. (And again, if this were a problem with stock Apache installations, we'd have many reports of it, so it's almost certainly something particular about your configuration.)
  • dstillman
    Maybe you should save the last-sync-information within the file?
    We could do this, but there's no particular reason to unless there's actually a problem with the current method, and we don't (yet) have any reason to think there is.

    You should be able to test this with your server with curl, at any rate.
  • tom7320
    Hm, ok, again thanks a lot for your very patient help!! And again, please believe me: it definitely is a stock Apache installation (Apache/2.2.8 (Ubuntu) DAV/2 mod_python/3.3.1 Python/2.5.2 PHP/5.2.4-2ubuntu5.10 with Suhosin-Patch mod_ssl/2.2.8 OpenSSL/0.9.8g)! Really! I'll try to fix the problem on the webdav server side and post the results here if I can solve the problem....

    THX!
  • dstillman
    I think you misunderstand what I mean by "stock"—I'm not saying you made changes to the Apache code and recompiled it, simply that there's something about your configuration that's non-standard, since Zotero obviously works with most Apache servers people use it with.
  • tom7320
    I probably misunderstood. Let me state more precisely: It is a clean install from my ISP of Ubuntu 8.04 without any modifications made by me. I do not know whether or not my provider changed certain settings. Since there is almost nothing to configure on mod_webdav I have unfortunately no idea where I can start looking for the problem...
  • tom7320
    edited July 23, 2010
    Sorry, but I must bother again. My WebDAV server _does_ update the timestamps:

    Client:

    apfel:~ ts$ echo "abc" > test
    apfel:~ ts$ touch test
    apfel:~ ts$ ls -l test
    -rw-r--r-- 1 ts staff 4 23 Jul 15:54 test
    apfel:~ ts$ curl -T test -u zotero:secret http://abc.de/zotero/test
    (...waiting for 2 minutes...)
    apfel:~ ts$ touch test
    apfel:~ ts$ ls -l test
    -rw-r--r-- 1 ts staff 4 23 Jul 15:56 test
    apfel:~ ts$ curl -T test -u zotero:secret http://abc.de/zotero/test

    Server:

    root@abc:/etc/apache2# ls -l /var/zotero/test
    -rw-r--r-- 1 www-data www-data 4 2010-07-23 15:54 test
    root@abc:/etc/apache2# ls -l /var/zotero/test
    -rw-r--r-- 1 www-data www-data 4 2010-07-23 15:56 test

    However, it does _not_ work if the file is empty:

    Client:

    apfel:~ ts$ touch empty
    apfel:~ ts$ ls -l empty
    -rw-r--r-- 1 ts staff 0 23 Jul 16:03 empty
    apfel:~ ts$ curl -T empty -u zotero:secret http://abc.de/zotero/empty
    (...waiting for 2 minutes...)
    apfel:~ ts$ touch empty
    apfel:~ ts$ ls -l empty
    -rw-r--r-- 1 ts staff 0 23 Jul 16:05 empty
    apfel:~ ts$ curl -T empty -u zotero:secret http://abc.de/zotero/empty

    Server:

    root@abc:/etc/apache2# ls -l /var/zotero/empty
    -rw-r--r-- 1 www-data www-data 0 2010-07-23 16:01 /var/zotero/empty
    root@abc:/etc/apache2# ls -l /var/zotero/empty
    -rw-r--r-- 1 www-data www-data 0 2010-07-23 16:01 /var/zotero/empty

    My question is: is it possible to make "lastsync" not an empty file?

    THX again!

    Thorsten
  • dstillman
    Does it update if there's a single space in the file?
  • dstillman
    (And what I said above wasn't true, actually—we can't store the timestamp in the file, because the whole point of the file is that it's a server-based timestamp.)
  • tom7320
    Yes, it updates if there is a single space in the file. If Zotero uses libcurl it might be possible that curl is causing the problem?!

    Empty file:

    apfel:~ ts$ curl -T empty -u zotero:secret --verbose http://abc.d/zotero/empty
    * About to connect() to abc.de port 80 (#0)
    * Trying 12.34.56.78... connected
    * Connected to abc.de (12.34.56.78) port 80 (#0)
    * Server auth using Basic with user 'zotero'
    > PUT /zotero/empty HTTP/1.1
    > Authorization: Basic xyz==
    > User-Agent: curl/7.19.7 (universal-apple-darwin10.0) libcurl/7.19.7
    OpenSSL/0.9.8l zlib/1.2.3
    > Host: abc.de
    > Accept: */*
    > Content-Length: 0
    > Expect: 100-continue
    >
    < HTTP/1.1 204 No Content
    < Date: Fri, 23 Jul 2010 17:14:29 GMT
    < Server: Apache/2.2.8 (Ubuntu) DAV/2 mod_python/3.3.1 Python/2.5.2 PHP/5.2.4-2ubuntu5.10 with Suhosin-Patch mod_ssl/2.2.8 OpenSSL/0.9.8g
    < Content-Length: 0
    < Content-Type: text/plain
    <
    * Connection #0 to host billroth.de left intact
    * Closing connection #0

    With a singe space in it:

    apfel:~ ts$ curl -T empty -u zotero:secret --verbose http://abc.de/zotero/empty
    * About to connect() to abc.de port 80 (#0)
    * Trying 12.34.56.78... connected
    * Connected to abc.de (12.34.56.78) port 80 (#0)
    * Server auth using Basic with user 'zotero'
    > PUT /zotero/empty HTTP/1.1
    > Authorization: Basic xyz==
    > User-Agent: curl/7.19.7 (universal-apple-darwin10.0) libcurl/7.19.7 OpenSSL/0.9.8l zlib/1.2.3
    > Host: abc.de
    > Accept: */*
    > Content-Length: 2
    > Expect: 100-continue
    >
    < HTTP/1.1 100 Continue
    < HTTP/1.1 204 No Content
    < Date: Fri, 23 Jul 2010 17:42:25 GMT
    < Server: Apache/2.2.8 (Ubuntu) DAV/2 mod_python/3.3.1 Python/2.5.2 PHP/5.2.4-2ubuntu5.10 with Suhosin-Patch mod_ssl/2.2.8 OpenSSL/0.9.8g
    < Content-Length: 0
    < Content-Type: text/plain
    <
    * Connection #0 to host billroth.de left intact
    * Closing connection #0

    The only difference is the "< HTTP/1.1 100 Continue".

    Regards,

    Thorsten
  • dstillman
    No, that's a server response (and Zotero uses Mozilla's networking functionality, which doesn't use libcurl).

    Try the latest 2.0 branch XPI (not the trunk), which stores a space in the lastsync file.
  • tom7320
    Hurray, this seems to work now!!!! I'll test more detailed tomorrow, but I think it is really working now!!

    In summery the reason for the trouble was that Apache/WebDAV did not update the timestamp of the file "lastsync" as long as this file was empty. If it is not empty the server seems to update the timestamp correctly.

    Just out of curiosity I still would like to know what exactly the problem was and whether or not it is a bug or a feature of Apache/WebDAV. But I first have to read (and understand) RFC2616 (HTTP)....

    Regards,

    Thorsten
  • dstillman
    edited July 23, 2010
    Well, it's WebDAV more than HTTP, but I don't know that there's anything in the WebDAV spec that touches on this.

    Note, though, that a 204 isn't a "rejection" (as the person in your httpd-apache thread suggested)—it's a success code. So if the server is returning a success code, I would argue that the timestamp should be updated—and your server does exactly that for non-empty files, which suggests that this is simply a bug. Bear in mind that Apache 2.2.8 was released in January 2008, so it's quite possible this isn't current behavior (and given that we've never seen it, I'm guessing it isn't).

    Also, since there seems to be some confusion in your other thread, this has nothing to do with curl. I suggested using curl simply because it's the best way to test HTTP interactions.
  • tom7320
    Well, I'm not an expert with servers, RFCs, Apache, ... I just tried to solve a problem... Are you talking about the thread in the Apache Users Mailing List?
  • tom7320
    There is a long discussion on the topic in the Apache Users Mailing List. It might not be a good idea to depend on the timestamp of a file to store the time and date of the last sync. There are many MAYs and SHOULDs in the RFC. Here is a statement of one of the helpful people on the mentioned mailing list:

    ---------- snip ----------
    My interpretation of this is that

    1) the server cannot be trusted to update properties such as time stamps
    when modifying the content of a resource. I might be a very good idea,
    but not a requirement.

    2) A put of a zero size file to an existing zero size file might not be
    considered a change int the body of the resource by all servers.


    The proper (untested by me) way to do this in DAV probably is to use the
    PROPPATCH method to set the time stamps (creationdate and
    getlastmodified properties), and use PROPFIND to check them.

    Using PROPPATCH would be the equivalent of using the "touch" command
    line utility. If it works.


    > apfel:~ ts$ curl -T empty -u zotero:secret --verbose

    [output snipped]

    Assuming that <http://abc.d/zotero/empty>; allready existed as that
    command was executed I see nothing wrong in either the request or the
    answer in that transaction.

    I can't know wether the Date header in the answer is correct or not, but
    note how RFC2616 uses MAY and SHOULD with relation to the 204 answer.
    This means that the client can't really trust that the Date header in
    the answer contains the new time stamp of the uploaded file.
    ---------- snip ----------

    The whole thread can be found here:
    http://thread.gmane.org/gmane.comp.apache.user/92345

    Regards,

    Thorsten
  • dstillman
    I know these aren't your responses, but:
    I can't know wether the Date header in the answer is correct or not, but
    note how RFC2616 uses MAY and SHOULD with relation to the 204 answer.
    This means that the client can't really trust that the Date header in
    the answer contains the new time stamp of the uploaded file.
    Zotero isn't using the Date header of the 204 answer, so this (and the referenced section of the RFC) isn't relevant. The uploaded file doesn't have a new timestamp.
    The proper (untested by me) way to do this in DAV probably is to use the
    PROPPATCH method to set the time stamps (creationdate and
    getlastmodified properties), and use PROPFIND to check them.
    The point is that we need a server-based timestamp, not a client-based timestamp, hence the current method.

    For what it's worth, I couldn't reproduce your empty-file behavior on Apache versions older (2.2.3) or newer (2.2.14) than yours, and we've never seen it with any other servers. There's not much point in diving into RFCs and debating the validity of what's clearly a fluke of upload processing (rather than an intentional design choice) of one particular version/build/configuration.

    In any case, a whole lot of people's time has now been taken up by an issue that only one person experienced and that I fixed in five seconds, so I'm closing this thread. Glad to hear it's now working for you.
This discussion has been closed.