Importing a collection results in subcollection contents ending up in collection root

HaroldTheWombat
Dear Zotero people,
Firstly, nice app and I've managed to get a few other people to use it. Sorry for the length of this, I just wanted to get everything down.

I'm in the process of sorting out a decent sized collection of pdf books, notes, papers, patents, etc etc and managed to get myself in a bit of a tangle between my desktop machine and laptop. I've bit the bullet and decided to have the desktop as the authoritative and get sync working for Lappy but I need everything tidied up first.

This is where the problem is. I have the data on Lappy broken down into multiple collections and subcollections. I now want to move the collections, and their subcollections, one at a time over to the desktop. I select one of the collections and export it, right click on collection and "Export Collection...". I provide a path and an export name and off it goes to a memory stick.

The memory stick is then taken to the desktop and I do File -> Import Collection, navigate to the memory stick and select the previously exported collection. Zotero thinks for a bit, and imports the collection.

I now go to the newly imported collection and to my dismay see that all of the entries that were in subcollections are now also in the root of the collection. They still exist in their respective subcollections but are now also in the root.

As you can imagine, this now needs a fair bit of post-processing, deleting all the dupes. I can't just select everything in the root and hit "remove from Collection" as there are entries that should be in the root.

This is very easy to replicate. Just select a collection with subcollections, export into a temp dir giving it a new name then reimport into Zotero.

I don't know if I've missed something in my understanding of the import/export process or if this is a bug/feature. Any thoughts greatly appreciated.

Andrew
  • adamsmith
    That's Zotero RDF?
  • HaroldTheWombat
    edited yesterday at 3:46pm
    Are you asking if I'm using RDF? IF so, my steps are as follows, just as I outlined to recreate the problem:

    Select Collection to export
    Right click on it in the LHS panel
    Select "Export Collection..."
    Dialogue pops up:
    Zotero RDF
    Export Notes
    Export Files
    selected
    Hit OK
    File requester dialogue pops up. Name of exported data -> "trout". Drop into temp dir
    In the same instance of Zotero select File -> Import...
    I want to import from "A file (BibTeX........etc.)"
    Hit Next, a File Import dialogue pops up.
    Navigate to the trout dir, enter the dir and select trout.rdf. Hit Open
    Options dialogue
    Default ticked "Place imported..."
    File Handling is "Copy files...."
    Next
    Import dialogue pops up with "Importing" and a progress bar. Eventually the import completes with an item count which looks about right.
    Hit finish to dismiss the dialogue.
    Now within Zotero navigate to the trout collection.
    Items that were in the subcollections of the Source collection are now, within trout, both in their correct subcollection but ALSO in the root of trout.

    Forgot to mention this before, 9.0.6, 64 bit, up to date Gentoo Linux. Whilst getting the version, I noticed the "Report Errors..." option. Following is the data:

    ***********************************

    [removed — D.S.]
  • HaroldTheWombat
    I've just tested this on Win 10 and the problem persists. Next, I'm going to set up a much smaller instance of Zotero and see if it can provide some more insight. Will also do the export & import with debug logging enabled.
  • adamsmith
    I think this is likely a genuine bug, no need to do a ton of testing; -- what's your use case? For most purposes, there are better options than large import/export operations. For one offs, there are workarounds.
  • HaroldTheWombat
    So, some simple testing.
    - Create 6 simple pdf's, problem01.pdf, problem02.pdf......., each file with simply a single line of random text.

    - Create a new base dir for Zotero and fire up Zotero pointing to this dir.

    - Create a new collection -> "firstCollection"

    - Create two sub collections, "sub01", "sub02" under "firstCollection"

    - Drag and drop problem01.pdf and problem02.pdf into "firstCollection"

    - Drag and drop problem03.pdf and problem04.pdf into "firstCollection -> sub01"

    - Drag and drop problem05.pdf and problem06.pdf into "firstCollection -> sub02"

    - Right click on "firstCollection" and select "Export Collection". Navigate to a temp dir and provide a name, "exportTest", for the exported data. Do the export.

    - Import the previously exported data, File -> Import, resulting in a new collection called "exportTest"

    - Review the contents of the imported collection, "exportTest". The original collection, "firstCollection" contained only "problem01.pdf" and "problem02.pdf" in it's root. The recently imported collection, "exportTest" now contains all of the files, "problem01.pdf" -> "problem06.pdf" in it's root. "sub01" and "sub02" in both "firstCollection" and "exportTest" contain the same files as expected.

    This test was done on the latest Zotero and an up to date Gentoo Linux machine.

    Can someone, Adam?, please have a quick try of the above and see if the same results are achieved. It took me about 10 minutes to create this.

    The problem has now shown itself for an Import using both 6 entries and about 180.
    It looks like something is flawed, no idea if it is the original Export or the subsequent Import. I'll have a look at the exported XML.
  • HaroldTheWombat
    edited 8 days ago
    Looking at the RDF file that is exported, and without knowing the actual schema, and also being an XML novice, it appears well formed and correct. The way it's laid out also looks like how I'd[1] do it. Looks like the problem is in the "Import".

    I suppose I'll file a bug......

    Just read the Issue Tracking stuff on the "source_code" page. Do I create the bug or will someone higher up in the Zotero world do that?

    [1] 30 odd years of technical code monkey experience
  • HaroldTheWombat
    Just downloaded and installed V10. I did the steps as I outlined 2 entries above and, sorry to say, it's still broken. The "Export" appears to be working correctly, all the exported data is there and the *.rdf file looks correct, whilst the "Import" still creates the subcollection hierarchy, places the data in the correct subcollection but also drops a copy of said data into the root of the collection.

    I might have to have a look at the source code, is there a doc that explains what is where in the repo, and see if I can work out the problem
  • aborel
    Are you absolutely sure the item in the root collection is an actual copy, or is it just the same item seen in different places? Right-click + "Show in Finder" (on my Mac... it might be "Show Files" for you) to check.

    Maybe we can take a step back before you dive into that rabbit hole :-)
  • HaroldTheWombat
    Good point. I hadn't thought of that.

    Create a new instance of Zotero

    Create "Collection 1"

    Create two sub collections, "Sub 1", "Sub 2"

    Drag and drop problem01.pdf, problem02.pdf into "Collection 1"
    problem03.pdf, problem04.pdf into "Sub 1"
    problem05.pdf, problem06.pdf into "Sub 2"

    Shutdown Zotero & fire up DBeaver and check db

    Viewing "collections" & "collectionItems" tables shows what I would expect to describe the collections hierarchy and contents

    Back into Zotero and export "Collection 1" - give it a different name, "exporterdData"

    Import "exportedData"

    Check root of "exportedData"

    Problem is there. Files are as per "Collection 1" but also the contents of the two sub's have been copied into the root.

    Doing "Show Files" on all of the imported entries shows that the copies are not actual/multiple copies but more like sym links in Linux. The "storage" dir now contains 12 entries rather than the 16 it would it it had been actual copies.

    Closing Zotero and firing up DBeaver again shows "collections" to contain what I would expect for two collections and two subcollections each

    Looking in "collectionItems" shows, potentially, the problem. The "itemID" for "Collection 1" are unique whereas for "exportedData" they are duplicated, indicating an entry for both the collection and the sub-collection.

    Anyone got any thoughts on this?
  • aborel
    I don't think they are duplicated. The fields just describe that the element appears at several places in the collection structure. This is the normal functionality of Zotero collections. See https://www.zotero.org/support/collections_and_tags#collections
  • HaroldTheWombat
    OK, lets simplify this down as much as possible.

    1) On Lappy, create 6 simple pdf files, all different, file01.pdf -> file06.pdf, say a single line of gibberish in each file

    2) On Lappy create a new instance of Zotero. In turn create a collection, "Collection 1" and under this create two sub-collections, "sub1", "sub2"

    3) Drop "file01.pdf" & "file02.pdf" into "Collection 1"
    "file03.pdf" & "file04.pdf" into "sub1"
    "file05.pdf" & "file06.pdf" into "sub2"
    https://s3.amazonaws.com/zotero.org/images/forums/u21244052/bf0xg6p8z6du39124jlg.png
    https://s3.amazonaws.com/zotero.org/images/forums/u21244052/j4hjifhea09l6vq0vm77.png
    https://s3.amazonaws.com/zotero.org/images/forums/u21244052/iu4escoe67vvr1om9jmw.png

    4) Export this and transfer over to Bluey, the desktop machine.

    5) Look at the exported *.rdf file - does it make sense? Yes, it has simple relationships that put the files where it would expect.

    6) Create a new instance of Zotero and import the data from Lappy

    7) Review the imported data. "Collection 1" contains ALL of the files, whether they be copies/duplicates/sym links at this stage I have NFI. "sub1" & "sub2" contain the entries as per on Lappy, ie they appear correct.

    https://s3.amazonaws.com/zotero.org/images/forums/u21244052/bl0y4r44z8zkq9i87vpo.png
    https://s3.amazonaws.com/zotero.org/images/forums/u21244052/lhxuskq2pv8dx7amhavx.png
    https://s3.amazonaws.com/zotero.org/images/forums/u21244052/pr7w801dgptge7wmdmiz.png

    8) Now check the file system to see if it can shed any light. There are, as expected, 6 entries under "storage", one for each file so it looks like the actual files aren't duplicated. Doing "Show file" shows pdf03 -> pdf06 to have the same dir whether they are in the root or their respective subcollections.

    9) Fire up an SQL thingy, dbeaver, and check the db. "items" has only 6 entries, file01 -> file06. "collections" has three entries, the collection & two subcollections. "collectionItems" is dodgy. It has a "many to one" of all the entries pointing to the collection, the root. It also has entry 3 & 4 pointing to sub1 and 5 & 6 pointing to sub2. The root should not have the itemIDs for entries 3, 4, 5 & 6, only entriesw 1 & 2.

    https://s3.amazonaws.com/zotero.org/images/forums/u21244052/oryh7jpdy24kriyzp9nf.png

    In conclusion, I can't explain it any simpler. This is obviously a bug. It should not have all the ref's in the root, only for 1 & 2. Am I fundamentally misunderstanding how the Export & Import work? It's a simple concept, drop data out of one instance, with the attendant structure and import it into another instance, using the contained structure.
  • aborel
    Thanks for the details!
    One last check: do you have the same view of subcollections on both machines? https://www.zotero.org/support/collections_and_tags#show_items_from_subcollections
  • HaroldTheWombat
    "...do you have the same view of subcollections on both machines..." - what do you mean by this? What I see is what is in the screen grabs above. It's simple, just follow my steps above and you will see the problem. It will take you, literally, 5 minutes. As to the link you provided, that was turned off.
  • aborel
    edited 3 days ago
    I see what you mean, that's not my point. I just want to get one thing out of the way that has absolutely nothing to do with export and possible bugs therein. Which takes significantly less than 1 minute, if you want to compare :-)

    The behaviour of Zotero collections is often misunderstood, so I think it is always useful to investigate that aspect before anything else. If the option described in the link was somehow different on Lappy and Bluey, that could have explained what you observe without any database-level debugging. If it is indeed the same, then I agree that a more technical study feels necessary, or a convenient workaround as suggested earlier in the thread. But I might not be the right person for that in-depth analysis, and unfortunately I do not really see which workaround(s) adamsmith had in mind.
  • HaroldTheWombat
    edited 3 days ago
    The install on the two machines is "out of the box". I'm not an academic who is using this to drive their research, I'm just a Civil/Chemical enginer, and code monkey, who is using Zotero basically as a filing cabinet. To this end, I use it as it comes. I don't even know what that option does so I haven't used it. With respect to Bluey, where I'm seeing the the problem right now, I had to turn it on to see that nothing happened. I'm assuming the default install is turned off.

    With respect to workarounds, that is a zero sum game. There is, an easy to understand concept, export some stuff, import some stuff. Fixing the functionality would be the way to go and from what I've seen it appears to just be a few misformed/mispopulated SQL insert statements.

    It's currently 4:51 Saturday morning here in Perth, Australia, so I'm now off to bed.
  • DoomedBengal
    Hi Harold! Another sandgroper from Perth here….
  • dstillman Zotero Team
    edited yesterday at 4:01pm
    We'll fix (and, as adamsmith said, we didn't need all this debugging). It's not quite as simple as it seems — exports can be for entire libraries, collections, or selected items, and the import has to create a top-level collection to place the items in before it knows whether there are collections, and because of that, the top-level collection itself has never actually been included in collection exports, since it would duplicate the initially created one, and all exported items have instead been placed in the initial top-level collection in addition to their subcollections. But we can make this work properly.

    Just be clear, though, if you export individual top-level collections, and any items beneath those are in other top-level collections, you'll end up with duplicates on the target computer. You would want to export the whole library instead.

    And this is all assuming you actually have different data on each computer. If you have any of the same data, you'll end up with duplicates from an export and import regardless. If you have the same items and this is just an organization issue, you would want to just fix one computer and sync.
Sign In or Register to comment.