Library building pains

deandavidgeorge
Recently started using Zotero plugin for Firefox on Mac at home and PC at work. Trying to get started working on a systematic review paper with two other colleagues and need get my library into our group library.

Used PubMed, Web of Science, and Embase to find articles matching our disease entity. Got just over 14K hits from PubMed, exported to Medline.txt and tried to import but my Mac froze after a few hours of rainbow pinwheel, so I rebooted and reset from the server as the pinwheel had returned as soon as I opened Zotero. Embase only let me export 200 RIS format records at a time from its 10K hits on my entity so I painstakingly uploaded all of them, waited for sync to finish and merged the pieces into one Embase folder. Went through a similar process for Web of Science BibTex files of 500 records which Zotero seemed to process much faster. Rather than try the PubMed file whole again, I manually divided it into 20 chunks and imported those. After a few days of this, all the records from the 3 searches (32,888 to be exact) are finally on the server.

On my Mac, I highlighted my entire library and dragged it into the group folder. Two days of pinwheel and racing internal fans later, all records made it into my local group folder. My challenge now is syncing the group folder to the server. I have lots of pinwheel and fans and the sync icon only changes position every 15 minutes or so, this has been going on for three days.

Should I let this continue for a few more days? Is there possibly a way for me to move my library into the group folder on the server? Sorry for the long message but I've been at this for a couple weeks now and am worried it won't finish before I go home for the holidays. Thanks.
  • adamsmith
    what's your total number of items: https://www.zotero.org/support/kb/item_count including notes & attachments (I assume the 32k you say above are just for top level items).

    It sounds like you may be hitting a limit of what type of library Zotero can reasonably handle at this time, especially with sync. 32k shouldn't be a problem, but if that somehow became much larger because of notes ore attachments, it might be.
  • deandavidgeorge
    Thanks for getting back so quickly! There are 32,888 in my library and 32,889 in the group library. There are no PDFs or other attachments except for 1 citation in the group library we had used as a test. Our plan is to whittle down the group library to ~200 citations based on abstracts and then attach PDFs.
  • adamsmith
    Dan should be around later & be able to look at your group's sync and see if it has stalled or what's going on. If you hover over the spinning sync icon, what does it say?
  • deandavidgeorge
    The sync icon remains in one position while the pinwheel spins, the pointer returns briefly and the sync icon advances to another position, then the pinwheel returns. No additional information appears when I hover over the sync icon with the pinwheel. I'll leave it over the sync icon and see if any text appears if/when the pointer returns.
  • deandavidgeorge
    Just caught it, it says:
    Sync with Zotero Server
    Uploading data to sync server
    Last sync: 4 days ago
  • adamsmith
    OK, let it run for now, wait what Dan says.
  • dstillman
    You have 50,000 items in your personal library on the server (including child items), 3 items in your group library, and nothing queued at the moment.

    If you have two libraries with the same items, that's 100,000 items in your database, which would be really pushing Zotero's current abilities.

    If you look at Activity Monitor, does Firefox show CPU usage? Memory usage?
  • deandavidgeorge
    Sorry, I wasn't aware that collections in my personal library are duplicates. My intention was to keep a separate collection for references from Embase, PubMed, and Web of Science, but I can certainly delete them and just use the main library as I have the original small files on my hard drive. I have 4GB of RAM on my Mac.

    From Activity Monitor:
    Firefox (Not Responding) % CPU is ranging from 22-59%, memory 2.15 GB
  • adamsmith
    Collections shouldn't cause duplicates, no.
    The only duplication is my librar/group -- Dan may have suggestions for how to best proceed if, as I think I understand, you only need these in one place (the group).

    (@Dan/aurimas -- this btw. is exactly the reason we need to be able to import into a group library other than from clipboard)
  • deandavidgeorge
    Ideally, I would like to keep the combined 32,888 in my library in case a group member deletes a citation accidentally and we need to reference the original list. If all I can get are 32,888 in the group folder and nothing in my library so be it. I'll print a hard copy of the original list.
  • deandavidgeorge
    I left sync going as it was overnight and my computer is frozen this morning. After a reboot and preference change to disable automatic syncing I'm back where I started prior to the group library upload attempt that began four days ago.

    As a next step, I was going to delete my collections on the server, reset my local database from the server, then click and drag alphabetical chunks of my library into the group folder (after each chunk has synced with the server). Does this sound like the best course of action?
  • deandavidgeorge
    I tried the above plan yesterday and was able to delete the server collections and reset my local library. The sync icon was spinning normally this time. It spun all day and was still going when I left but my Mac was frozen again this morning. Looks like Zotero isn't the right tool for this kind of job.
  • dstillman
    You did this in the account you're currently signed into? I don't see any sign of syncing from you in the last few days. Also not sure what you mean by "delete the server collections" — you still have 50,700 items in your personal library online (which is fine, because you don't want to actually delete anything that's already synced, since that would likely make things worse).

    But anyhow, I'd agree that syncing a database with 100,000 items more or less at once — as opposed to building it up gradually — is not something Zotero handles particularly well at the moment. The next major version of Zotero will have a completely new sync architecture — with per-library syncing and more granular data transfer — that should be much better at this sort of thing.
Sign In or Register to comment.