NIH RADx Data Hub File Upload SOP

File Upload Standard Operating Procedures (SOP) In the NIH RADx Data Hub, users with a Submitter role can submit documents (study documentation and README files), data files (non-harmonized and harmonized), and supporting files—metadata (“META”) files and data dictionary (“DICT”) files—for use by other research community members. After upload, the files must be categorized, validated, and reviewed prior to final submission and a subsequent RADx Data Hub curator review. Important: The RADx Data Hub introduces the concept of “bundles,” which consists of a harmonized or non-harmonized data file, a data dictionary file, and a metadata file. It is impossible to upload a data file without including these two required supporting files in the bundle. Study documents (e.g., README files) create standalone bundles and do not require supporting files. Figure 1: File Submission Process Step 1: Navigate to and Upload Files There are three methods for uploading files as described in Table 1 below. Table 1: The Three Upload Methods and Their Use Cases Method Use Case Via the User Interface (UI) Best for file packages under 250 MB and/or when uploading files to one study Via Single Study Secure File Transfer Protocol (sFTP) Best for file packages over 250 MB Via Multiple Studies sFTP Best for uploading files to multiple studies simultaneously File Upload Standard Operating Procedures (SOP) Note: When uploading a new version of an existing file, you should upload the new file with the exact same file name as the existing file. This will replace the existing file with the new file and update the version of the file within the study. Older versions will be available by request. Following the naming conventions found in Data File Naming Convention Guidance enables efficient file submission. Uploading Files via the RADx Data Hub User Interface (UI) This option allows you to upload files directly, via the RADx Data Hub UI. To use this option: 1. Click “Data Submission” under “Data Submitter” in the navigation bar. 2. Click “+ New Submission” to start a new files submission (Figure 2). Figure 2: Submitter Dashboard 3. In the dropdown menu, select the study. The list contains studies aligned to your program (e.g., RADx-UP, RADx-rad, RADx Tech). 4. Click “Create Submission,” which creates an ID number for the submission. Figure 3: Study Dropdown and Create Submission Button 5. Click “Select File(s) to Upload,” which opens the File Explorer. Figure 4: Select File(s) to Upload Button 6. Select the file(s) you wish to submit and click “Open”. File Upload Standard Operating Procedures (SOP) Note: To select multiple files, hold the CTRL (For Windows)/Command (For Mac) button, and click the desired files. Figure 5: The File Explorer 7. Once you are done uploading the files, click the “Next Page” button. The system will navigate you to the Categorization step. Proceed to “Step 2: Categorize Files” in this document. Note: The “Next Page” button will be disabled until at least one file is uploaded. Uploading Files via Single Study sFTP – Example Using FileZilla Note: Before using the sFTP function, you must have an sFTP account. This is separate from your RADx Data Hub account. Use the contact form in the navigation bar or footer to request an account, and the Support Team will assist you. The team typically responds to inquiries within 24 hours. The sFTP function uses an external application to upload files. The sFTP function is preferred for larger file package uploads (> 250 MB). To use this option, follow the steps outlined below. Note: The following steps show the sFTP process using FileZilla. Depending on your sFTP client, steps may differ. 1. Open the File Explorer. Create and name a new folder. 2. Add all files to be submitted to the new folder. 3. Download this blank metadata.yaml file, and fill out the following: File Upload Standard Operating Procedures (SOP) -study_id (Required): This unique study identifier can be located through the navigation bar. Under “Data Submitter,” click “Data Submission.” This will bring you to the Submitter Dashboard, where you can download all existing study keys by pressing “Download Study Key.” -files (Optional): Enter the names of the files in the upload. Figure 6: Blank metadata.yaml File 4. Store the metadata.yaml file in the folder created in Step 1. 5. Add the study folder to a parent folder and compress the parent folder into a zip file. 6. Open the sFTP client and fill in the following fields: a. Host: Enter “https://radxdatahub.nih.gov/” b. User Name: Assigned by the RADx Data Hub Support Team c. Password: Assigned by the RADx Data Hub Support Team d. Port: Enter “22” Note: FileZilla may have a different UI and processes than other sFTP clients. Figure 7: sFTP Window in FileZilla 7. Connect to the RADx Data Hub Server. The status will indicate if the connection was successful. If the connection failed, contact the RADx Data Hub Support Team. File Upload Standard Operating Procedures (SOP) 8. Find and select the zip folder, created in the earlier step, and drag it into the RADx Data Hub window within FileZilla to securely transfer and upload to the RADx Data Hub. Figure 8: File Transfer Pane in FileZilla 9. Depending on the upload size, files may not be visible in the RADx Data Hub for a few minutes. You will receive an email if the upload was successful, and you can return to the Data Hub to complete the submission. If you have not received an email, check the RADx Data Hub to see if the files were submitted. If not, you will need to upload the files again. Verify that you followed the steps for uploading files correctly and contact the RADx Data Hub Support Team if problems continue. 10. Return to the RADx Data Hub, go to the navigation menu, and select “Data Submitter.” Then click “Data Submission.” 11. Locate the study in the dashboard and click the link in the ID column. This will bring you to the Categorize Files step, where you can proceed with the submission. Uploading Files via Multiple Studies sFTP – Example Using FileZilla The sFTP function allows you to upload multiple studies’ files simultaneously. This process is similar to uploading files for one study. To upload multiple studies via sFTP, follow the steps outlined below. Note: The following steps show the sFTP process using FileZilla. Depending on your sFTP client, steps may differ. 1. Open the “File Explorer” and create a new folder for each study in the upload. 2. Add all the files for the submission packages into the appropriate new folders. 3. Download this blank metadata.yaml file and enter the following details for each study: a. study_id (Required): To find the unique study identifier, go to the navigation bar, find “Data Submitter,” and click “Data Submission.” This will bring you to the Submitter Dashboard where you can download all existing study keys by pressing “Download Study Key.” b. files (Optional): Enter the name of each file in the upload. 4. Store the metadata.yaml files in the appropriate study folders. 5. Compress all study folders into a single zip file. File Upload Standard Operating Procedures (SOP) 6. Open the sFTP client, and fill in the following fields: a. User Name: Assigned by the RADx Data Hub Support Team b. Password: Assigned by the RADx Data Hub Support Team c. Host: Enter “22” d. Domain: Enter “https://radxdatahub.nih.gov/” Note: FileZilla may have a different UI and processes than other sFTP clients. Figure 9: sFTP Window in FileZilla 7. Connect to the RAD Data Hub Server. The status will indicate if the connection was successful. If the connection failed, contact the RADx Data Hub Support Team. 8. Find and select the zip folder containing multiple studies, created in Step 7, and drag it into the RADx Data Hub window within FileZilla to securely transfer and upload to the RADx Data Hub. Figure 10: File Transfer Pane File Upload Standard Operating Procedures (SOP) 9. Depending on the upload size, study files may not be visible in the RADx Data Hub for a few moments. You will receive an email if the upload for each study was successful and can return to the Data Hub to complete the submission. If you have not received an email check the RADx Data Hub to see if the files were submitted. If not, you will need to upload the files again. Verify that you followed the steps for uploading files correctly and contact the RADx Data Hub Support Team if problems continue. 10. Return to the RADx Data Hub, go to the navigation menu, select “Data Submitter,” and select the Data Submission option. 11. Locate a study in the dashboard and click the link in the ID column. Each study from the sFTP package will have its own submission ID. Proceed through the submission workflow for every study uploaded. Step 2: Categorize Files Once the files are uploaded, they need to be categorized. The system will make an initial attempt of categorization by analyzing the file names in accordance with the Data File Naming Convention Guidance to automatically categorize and bundle them (Note: A bundle consists of a non-harmonized or harmonized data file, a metadata file, and data dictionary file). The results of the initial categorization attempt will be presented on the File Categorization page in three tables: Bundled Files, Study Documents, and Unassigned Files (see Figure 11). If you upload via sFTP and files are able to be categorized into full bundles using the naming conventions, you will skip this step and validation will begin immediately. Note: The below example shows harmonized files not categorized. The process will be slightly different if, for example, only metadata and data dictionary files were not categorized. File Upload Standard Operating Procedures (SOP) Figure 11: File Categorization Page To categorize and bundle the files in the Unassigned Files table: 1. If the Unassigned Files table contains any data files, select the appropriate category for such files (see Figure 12). The system records the category and makes it available under “Add Data Files” in the Uploaded Files table. Figure 12: Unassigned Files Table File Upload Standard Operating Procedures (SOP) 2. Scroll up to the Bundled Files table. Click “Add Data Files,” and select the appropriate files. The system will display them in the Uploaded Files table. Figure 13: Bundled Files Table 3. Scroll down to the Unassigned Files table. Select a category for any uncategorized, unassigned data dictionary and metadata files. The system will record the category and make it available in the Add Unassigned Files dropdown. Figure 14: File Type Dropdown in the Unassigned Files Table 4. Scroll up to the Bundled Files table. Click “Add Unassigned File” and select from the list of categorized data dictionary and metadata files. The system will add the file to the appropriate bundle.