Cleaning up sync problems

marsh · November 14, 2009

Hi,

I started having sync problems some time last spring. Because of both a lack of time on my part and the fact that the Zotero folks were fixing some related problems, I continued to use Zotero without attending to the sync problem. I use four different computers, so right now their libraries are way out of sync, but I'm starting to fix them. As I do, I want to make sure I fix them correctly. Hence this post.

The first machine I'm working on runs Windows XP Pro (SP3), FF 3.5.5, and Zotero 2.0b7.4. Its Zotero sync icon has had an exclamation point since last spring. Today, I checked the integrity of the Zotero database, found problems, and rebuilt it using the Zotero database repair tool at https://www.zotero.org/utils/dbfix/. Zotero no longer reports integrity problems for this database.

Nonetheless, the database still doesn't sync correctly, and the exclamation point is still there. I reported the error (386926754) and submitted a debug report (D572033858). What's my next step in fixing this database?

Additionally, should i start working on fixing the databases on the other computers or wait until this one if fixed first? I have a project due and want to get this done as quickly as possible, but I'm afraid one database will erase entries from another if I'm not careful.

Thanks for your help.

dstillman · November 14, 2009

This is due to some incorrect old records in your database. I've updated the DB Repair Tool to fix the problem, so you should run your database through it again. I'll also add correction steps for 2.0b7.5 for others running into this problem.

Thanks.

marsh · November 19, 2009

Hi Dan,

Your suggestion seemed to work with Zotero on one computer but not another. I reported the error from the second computer (594353875) and submitted a log (D392752936).

In the meanwhile, I have a few suggestions and one question.

1. On the computer that seemed to clear up the problem, before I fixed the file choosing Actions gear > Preferences > Advanced > Check Database Integrity resulted in a message, "No errors were found in the database." If the incorrect records cause sync problems as you say, then this message is wrong. Zotero is reporting a false negative.

2. When the database started to sync, I got up to 98 messages about versions of files with similar content and that had been merged into a new record. Since there was no way to check the accuracy of the original information and I needed to fix the database, all I could do for the most part was accept the merged records. It would be helpful to have more options during this process, such as accept the merged version for all 98 matches (Accept All) or edit the matched versions.

3. Similarly, after approving all the matched records, I got messages about new collections and tags. Again, I felt I had no alternative but to accept all the new things that Zotero detected. Pressing "OK" was the only choice, and again I'd have liked an "Accept All" button.

Now my question. With these files not having been synced for over half a year, I expect to have a good number of duplicate records. In another thread I've seen that Zotero already has a "Find Duplicates" tool but it is not turned on yet in the standard installation. Will the next release will have the tool turned on? If not, when can we expect it?

As usual, thanks for all your help.

dstillman · November 19, 2009

Your suggestion seemed to work with Zotero on one computer but not another. I reported the error from the second computer (594353875) and submitted a log (D392752936).

There are no Zotero errors in the error report. The only errors I see in the debug output are from your not having entered a WebDAV password. It should tell you that if you click on the sync error icon.

Check Database Integrity resulted in a message, "No errors were found in the database." If the incorrect records cause sync problems as you say, then this message is wrong. Zotero is reporting a false negative.

It's not. Check Database Integrity is just checking for SQLite database corruption, which you don't have (and, really, no one without serious disk problems should have had since Firefox 2, which had database corruption issues).

Your problem was just due to some incorrect Zotero-created records. It's no longer possible to create those, and I added a database upgrade step to fix them, so there's no need to test for such things in Check Database Integrity. I put the fix into the repair tool just to help you and others in the meantime.

When the database started to sync, I got up to 98 messages about versions of files with similar content and that had been merged into a new record.

Not sure exactly what you're referring to. This was the three-pane conflict resolution window? Items or files? If items, there should generally be differences in the left two panes (though Zotero could be more helpful and highlight those differences). There might be a few cases where there are no visible differences, but those are bugs and should be fixed (and possibly have in the latest dev builds).

Similarly, after approving all the matched records, I got messages about new collections and tags. Again, I felt I had no alternative but to accept all the new things that Zotero detected. Pressing "OK" was the only choice, and again I'd have liked an "Accept All" button.

Yes, "OK" is the only option. These are notifications only and will be moved to a less obtrusive notification system eventually. With automatic syncing (and a few remaining bugs fixed), these messages should be pretty rare.

I've seen that Zotero already has a "Find Duplicates" tool but it is not turned on yet in the standard installation. Will the next release will have the tool turned on? If not, when can we expect it?

No—it won't be in 2.0. It may be in 2.1. Finding duplicates is relatively trivial, but dealing with them—particularly when they might be referenced in word processor documents, linked to group items, etc.—is not.

marsh · November 24, 2009

Thanks Dan. Sorry not to get back to you sooner. I'm not always in the office where the system is. BTW, how do you quote things the way you do? Here, I'll use [quote][/quote] tags until I learn a better way.

[quote]
The only errors I see in the debug output are from your not having entered a WebDAV password. It should tell you that if you click on the sync error icon.
[/quote]

This fixed things. Actually I had entered a WebDAV password, and it passed the verification. Apparently this did not last. My keyboard sometimes double types, and I suspect something like this happened to corrupt what was typed in. It's working OK now.

[quote]
[quote]
When the database started to sync, I got up to 98 messages about versions of files with similar content and that had been merged into a new record.
[/quote]
Not sure exactly what you're referring to. This was the three-pane conflict resolution window? Items or files? If items, there should generally be differences in the left two panes (though Zotero could be more helpful and highlight those differences). There might be a few cases where there are no visible differences, but those are bugs and should be fixed (and possibly have in the latest dev builds).
[/quote]

Yes, it was the 3-pane conflict resolution window comparing items. A large majority of items had no discernible differences. Highlighting differences would be very helpful. In addition, comparing records in the window is very difficult. The three panes do not render with equal width, so the narrowest pane renders in many more lines than the widest, and the text to compare does not appear side-by-side. Additionally, I've been unable to resize the width of the panes. Neither have I been able to resize the window itself, either by dragging its borders or by using the Maximize icon for the window (because on Windows XP the window does not have the resizing icons).

In some cases the two items do seem very different, and I wonder why they're even considered to be the same item. Since I hadn't been able to sync the databases on the two systems for quite some time, I suspect the comparison is based on identical index keys to the database but the two items refer to totally different bibliographic entities. If so, it would be very helpful to have a way to save BOTH versions of the item, choosing one for the present item and adding a new item for the other.

[quote]
[quote]
I've seen that Zotero already has a "Find Duplicates" tool but it is not turned on yet in the standard installation. Will the next release will have the tool turned on? If not, when can we expect it?
[/quote]

No—it won't be in 2.0. It may be in 2.1. Finding duplicates is relatively trivial, but dealing with them—particularly when they might be referenced in word processor documents, linked to group items, etc.—is not.
[/quote]

Too bad. Because of not being able to sync for so long, I'm sure I have numerous duplicates. In addition, when I first started using Zotero I accidentally imported the same set of bibliography records repeatedly for about 3-4 times resulting in numerous duplicates. So far, my usage of the database has been minimal (this is about to change), so I don't worry much about problems in word processor documents, groups, etc. I want to clean the database before starting to use it in earnest.

Thanks for your help!

Marsh Feldman

P.S. Next week I will be synchronizing the database on my two Macs. Hopefully, I won't have to bother you again.

marsh · December 1, 2009

Well, it's next week, and I tried syncing on one of the Macs. Initially I got an error, #55257679. Then, following the instructions, I ran a second sync manually. It seemed to work, but the 3-pane conflict-resolution window said there were 98 problems. In the first 20 or so conflicts, the problem was that the item had been deleted from the remote machine but was still here on the local one. Since I'd just spent about 4-5 hrs going through the database on another machine and cleaning up duplicate entries, I was afraid that accepting the local changes would reinstate duplicate entries and therefore canceled the sync.

This brings up a few questions.

1. If we actually want the item deleted, how can we best do this from the local machine? The resolution window does not give the choice of accepting the deletion, and it's too time consuming to search out 98 remotely deleted items in the local database. Besides, the resolution pane has already identified the problematic entry. Why do the work twice?

2. Ideally, I'd want to search the remote database to see if there's an entry like the current one. For instance, with 3 duplicate records, I would have deleted two, and one of them may be the one being reported as having been deleted. The third, remaining, undeleted record is the one I want to use. However, I see no easy way to confirm that there is a third, remaining, undeleted record on the remote database. Is there one?

3. For a situation like mine, just searching out and locating the duplicates would be a terrific help. Since Zotero does not warn the user when a duplicate entry has been made in the database, such duplicates are bound to appear. An easy way to locate and eliminate them would be terrific help. At this point, I don't need to fix references in word processors, etc. Please don't let perfect be the enemy of feasible.

Marsh