Bulk Upload Guide

Why use Bulk Upload?

Uploading large numbers of files, such as sequencing read sets, one at time through the Data Panel in the Narrative Interface can be time consuming. To address this issue, KBase has expanded its data import functionality to support bulk upload of data, meaning that users can import multiple files simultaneously. This feature is especially beneficial for large-scale analysis of data sets. Currently, KBase supports bulk upload of sequencing reads, genomes, and FASTA files through its bulk upload interface.

Getting Started–What You Need

  • KBase user account – if you don’t have one you can sign up for free here: https://kbase.us/sign-up-for-a-kbase-account/
  • Globus Connect Personal – data management and file transfer system that facilitates bulk upload of data into the Narrative interface. Installation instructions for Mac OX, Windows and Linux systems can be found here: https://docs.globus.org/how-to/
  • Data in your local file system that needs to be uploaded into your Narratives. Currently, KBase supports bulk upload of reads sets (single- and paired-end read libraries) including SRA reads, genomes, and FASTA files.

Uploading Data to Globus

You can bulk upload data into KBase by uploading files from your local computer into Globus, then transferring those files into one of your Narratives. After you have created a KBase account and successfully downloaded and installed Globus Connect Personal, please follow these instructions for uploading your files into the Globus file system and transferring them into KBase.

  1. Create a Globus Connect Personal Endpoint. This Endpoint allow Globus to access files on your local computer. Instructions to create an Endpoint as part of the Globus Connect Personal client installation process and can be found here: https://docs.globus.org/how-to/
  2. Start the Globus Connect Personal client software you installed on your local computer
  3. Login to KBase. After signing in with your KBase account credentials, you will be taken to your Dashboard.
  4. Navigate to the KBase Bulk Importer Interface using the menu in the top left corner of the Dashboard.
    Bulk-Import-Menu

There are two ways that you can bulk upload files into KBase using the Globus file system:

Drag and Drop Data Files into the Bulk Importer

  1. From the Bulk Importer page, click “Browse” button on the top of the page.Bulk-Import-Browse
  2. Using the file manager on your computer, select the files you want to import from your local computer, then drag and drop the files into the bulk upload interface.
    Bulk-Import-DragDrop
  3. You will see a progress bar to the side of each file, showing the upload status for each file. The progress bar will disappear once the data files have been successfully uploaded.Bulk-Import-Loading

Using the Globus File Transfer Tool

  1. Open your file browser and the Bulk Importer page. Click “Browse” button on the top of the page.Bulk-Import-Browse
  2. Click “Upload with Globus Online” button in the top left corner of the page.Bulk-Import-Globus1
  3. This will take you to the Transfer Files page on Globus. Login to the Globus Web App by clicking the “Globus ID” link. Once prompted use your KBase username and the password to log in.
  4. Once logged in, you should be able to see two Endpoint fields.
  5. Next, select your Endpoints – the places where your data will be transferred to and from using Globus. Click the Endpoint field, which will open up a searchable list of available endpoints. You should have already set up an endpoint for your local file system when you installed Globus Connect Personal (see “What do you need?” section).Select Endpoints for:
      1. Local file directory on your computer
      2. KBase Bulk Share – where your files will be uploaded. Please note KBase Bulk Share Endpoint is already created for you by KBase admin and available for you to select at this point.

    Bulk-Import-Globus2

  6. Once you have set your Endpoints, you should able to see your local directory appear in the left data pane and the KBase Bulk Share directory on the right. Now you can transfer the files by selecting them and clicking the transfer button, represented by an arrow for the direction of the transfer.Bulk-Import-Globus3
  7. After clicking the file transfer button, you will see a message that your request was successfully submitted. Once this process finishes, your files will have transferred from your local computer to the Globus file system.Bulk-Import-Globus4
  8. Once the files have transferred successfully, go back to the KBase Bulk Importer page (https://narrative.kbase.us/bulk-ui/) to see your transferred files.Bulk-Import-Globus5

Transfer Data from Globus into KBase

At this point, you have successfully transferred the selected data files from your local computer to Globus. The next step is to transfer your data from Globus into KBase. With all of your files in Globus, you are ready to transfer your data into your Narratives. Currently, KBase supports bulk upload of reads sets (single- and paired-end read libraries), including SRA reads and genomes. More data types will be supported in the near future. In this example we show how to upload a set of genomes into a Narrative.

  1. First, select the type of data from the drop down menu at the top of the screen. Select “Genomes” from the menu. Then, select the reads files that you want to transfer to KBase by selecting the check boxes next to each file. Once you have finished, click Continue.Bulk-Import-Genomes
  2. Clicking Continue opens a page with a list of available Narratives. Select the Narrative you want to transfer your data into. You can only transfer your data into an existing Narrative, so be sure that you have created the Narrative to hold your data. To learn how to create a Narrative, see the Narrative User Guide. Click the Start Import button at the top right corner of the page to initiate transferring your data files into the selected Narrative. Your data will be converted into a KBase-specific data type to ensure compatibility throughout the system, but you can still download the files in their original format.
  3. After starting the upload process, a page will appear showing you the progress of the each upload task and the status of the upload job. You may refresh the upload status by clicking the refresh button in the top right corner.Bulk-Import-Status
  4. Once the transfer process successfully completes, you will see “completed” message under the Status column for each file. If the transfer process failed for some reason, this column will display “suspended.” Clicking “view details” for each file under the Options column opens a detailed log of the transfer process. If the transfer fails, these logs can be very helpful for user support to determine and troubleshoot the cause of the failure.
  5. After your data has successfully transferred, open your Narrative from your Dashboard. You should be able to see the newly uploaded data appear in the Data Panel for your, meaning your data is ready to analyze with KBase apps!

Bulk-Import-Final

 

Bulk Import of Reads

Bulk import of sets of reads offers some additional functionality for users to be aware of when transferring files from Globus into KBase. These instructions refer to the steps after you have imported some paired-end reads into Globus and are ready to transfer them into KBase.

  1. After successfully transferring short reads from your local computer to Globus, select “Paired-end Reads” from the menu. Then, select the reads files that you want to transfer to KBase by selecting the check boxes next to each file.
  2. Notice that the Add set button near the top right corner is now activated and ready to add the your reads files as a “set”. Click the Add set This process ensures that the two reads files selected are a set in paired-end reads type. Repeat this process if you want to add more sets of paired-end reads. Once you have finished, click Continue.
  3. Clicking Continue opens a page with a list of available Narratives. Select the Narrative you want to transfer your data into. You can only transfer your data into an existing Narrative, so be sure that you have created the Narrative to hold your data. To learn how to create a Narrative, see the Narrative User Guide.
  4. Selecting a Narrative from the drop down list gives you options for providing metadata for this transfer. You can provide a name for the import and information about the insert size. If the read sets are sequence read archives (SRA), you may check the SRA check box.

Additional Information

For additional help and examples, consult the following resources:

If you have any questions, please contact us through this page: https://kbase.us/contact-us/