RADx Data Hub User Tutorial

USER TUTORIAL RADx Data Hub Tutorial Introduction and Overview Overview The NIH Rapid Acceleration of Diagnostics Data Hub (RADx Data Hub) is a centralized data repository that provides access to analytic tools and de-identified COVID-19 data from the RADx Initiative. The RADx Data Hub better supports scientific efforts to understand COVID-19 and factors associated with disparities in morbidity and mortality in underserved and vulnerable populations, by allowing researchers to discover, access, and perform analyses of COVID-19 datasets in a cloud-enabled platform. Introduction This tutorial provides in-depth, step-by-step instructions on how to use the features and functionality of the RADx Data Hub most effectively. If you still have questions after reading the tutorial, consult some of the other support documentation (e.g. the Frequently Asked Questions page or the Glossary) or reach out directly by using the “Contact Us” link in the main navigation bar or the footer. Target Audience The primary audience for this tutorial is external researchers. Internal NIH staff should consult the appropriate Standard Operating Procedures (SOPs) on the Resource Center page. The Features of the RADx Data Hub The RADx Data Hub contains several different features to help you get the most out of the system and better meet your research needs. These include: ● Common Page Navigation Tools: The RADx Data Hub features navigation tools (e.g. the navigation bar, the footer) that will lead you to different pages and features within the Hub. ● Home Page: The Home page is a one-stop shop for many of the key resources on the RADx Data Hub. From this page, you can quickly reach educational documents (e.g. the Frequently Asked Questions [FAQ], the User Tutorial), get information on news, funding opportunities, events, and study updates, search for studies, and view statistics on the information in the Hub. ● Study Explorer: The Study Explorer contains a number of discovery features (e.g. search, sorting, filtering) to help you quickly and easily search across studies to find datasets of interest and learn more about variables contained therein. ● Study Overview Pages: Each study in the RADx Data Hub has its own Study Overview page, which contains study metadata, variables, data files, and downloadable documents. ● Variables Overview Pages: Each variable in the RADx Data Hub has its own Variable Overview page, which contains detailed variable information and a list of studies containing the variable. By viewing these pages, you can understand the data’s context and structure before you request access—helping you make informed decisions with confidence. ● Variables Catalog: This tabular resource lists all variables in each data file for each study. By viewing this resource, you can gain a deeper understanding of the key variables in a study to help you determine whether it aligns with your research goals before requesting access to the study. ● Support Resources: The system gives you multiple ways to submit a support request, so you can ask questions, report bugs, and request in-depth assistance from the Support team on complex questions. You can use the “Need Support?” link in the navigation bar or in the footer. ● User Registration: To access certain features, such as the Approved Data tab, you will need to first register with the RADx Data Hub. After you have registered, you can login using the “Login” button in the top-right of every page to access role-based features. ● Approved Data: After you have been approved for data, you can access the “My Approved Data” tab. From here, you can apply for a workbench instance, download data, or transfer it to your workbench instance. ● Public Data: The Public Data page has synthetic data files, which you can practice using our “Analytics Workbench” feature. Study Explorer General The publicly available Study Explorer lets you search RADx study metadata to find studies for your research. In the Study Explorer, you can: ● View available RADx Data Hub studies ● Perform free-text searches ● Navigate results ● Refine results through sorting and filtering ● Perform cross-entity search across studies and variables View Available RADx Data Hub Studies To view available RADx Data Hub studies and variables, click “Study Explorer” in the upper navigation bar. You will be taken to the Study Explorer. It has two tabs: Studies and Variables. By default, the Studies tab is displayed first, where you can see all findable RADx Data Hub studies, presented, by default, in Table View. Figure 1: Study Explorer Link in the Navigation Bar and default Study Explorer View with Studies and Variables Tabs In the top right of each tab you will find several controls (Figure 2) including: ● Studies and Variables tab that allows you to switch between study and variable search ● List/Table View toggle that allows you to switch between List View (which presents results in a vertically arranged list) and Table View (which presents results in a tabular format) ● Download Results button downloads the search results as a csv file ● Manage Columns button allows you to choose columns to hide or show. Figure 2: View and Download Results Controls Performing Free-Text Searches & Viewing Search Results You can perform free-text searches by entering custom queries in the search bar. To perform a free-text search in the Study Explorer: 1. Click "Study Explorer" in the navigation bar 2. Locate the Search bar (Figure 3) 3. Enter your free-text query 4. Press "Enter" or click the magnifying glass icon to view results, sorted by relevance based on your query Tip: You can also search from the Home Page Figure 3: Study Explorer Search Bar Navigate Through Search Results After performing a search, use the page navigator in the top right of the Study Explorer to move through search result pages (Figure 4). To navigate through pages of the results, you can: ● Option 1: Click the Forward or Backward Arrows in the Page Navigator to move one page forward or one page backward in the results. ● Option 2: Click Individual Page Numbers (typically in the format: 1, 2, … X) to go to an individual page. Figure 4: Page Navigator In addition to changing the search results view, you can also change the number of results per page. To do this, click the “Show” dropdown at the top or bottom right (Figure 5). Then, select the number of results to show per page, and the page will automatically update. Figure 5: "Number of Results per Page" and Sorting Controls The same controls are available on the Variables tab, but the sorting options are going to be different (Figure 6). Figure 6: "Number of Results per Page" and Sorting Controls on the Variables tab Refining Results Through Sorting and Filtering Sorting and filtering search results can further refine a search. To sort in the Study Explorer: 1. Locate the sorting options in the top right. (Figure 5) 2. Pick either "Ascending" or "Descending" in the sort order dropdown 3. Use the sort by dropdown to select a field for sorting, and results will dynamically update. Filtering is more complex than sorting but can help further refine a search. The filter pane is made up of two primary components: filter categories and filter values. Filter categories (e.g. “Has Data Files” or “RADx Data Program”) are high-level buckets that include multiple filter values. Filter values are the specific criteria by which you can filter search results. For example, the filter values in the “Has Data Files” category are “Yes” or “No.” Three techniques to narrow or refine a search using filters: Table 1: Different Search Techniques After deciding a filtering technique, select values by expanding the accordion for the desired filter category (Figure 7). Next, click the checkbox next to the filter value, and results will dynamically update. To remove a single filter value, click the checkbox a second time or click the “X” button on the filter badge above the filter pane. To remove all filters, press “Reset Search” above the filter pane. Note: The numbers to the right of the filter values represent the number of results a selected value will return. Figure 7: Filter Box Performing Cross-Entity Searches Two tabs of the Study Explorer allow users to perform cross-entity searches. That is, you can search for studies or variables, and the results will display linked studies/variables. For example, if you want to search for studies focused on Essential workers, you select “Essential Workers” value in the “Study Population Focus” filter, and your search returns the 12 studies focused on this particular population. In the Studies tab, you can see the variables for a particular study by clicking the “View list of variables” icon next to that study (Figure 8). Figure 8: View list of Variables icon next to the study name System will display list of variables contained in this study’s files (Figure 9). Figure 9: List of Variables modal Using the Variables Tab to search will give variables related to your search term. For example, if you wanted to see what variables are related to age, you would enter age, and it would generate a list of all variables related to age across all studies. Similarly to the Studies Tab, in the Variables Tab, you will see the same icon (Figure 8). But clicking on it will expand the list of studies, where files contain that particular variable (Figure 10). Figure 10: List of Studies modal Both modals have a button “Explore studies/variables in search”. Clicking this button will bring you to a respective tab of the Study Explorer populated with search results for this study/variable. Study Overview Each study has an overview page, which contains key documents, metadata, and variable and file information. To reach the Study Overview page, locate a study in the Study Explorer or Variables Catalog and click on the “Study Name”. On the Study Overview page, you can: ● View study information ● View variable information ● Learn how to request study access ● Download publicly available documents ● Learn about data files and download resources View Study Information Study Overview contains comprehensive study information, it begins with the study name, and below that, a Study Information section lists several attributes to help you gain a high-level understanding of the study. Figure 1: Overview Page View Variable Information The Variable Information section provides the total number of unique variables across all data files for the study. It also provides a list of all variable names and labels with a link to the specific Variable Overview page if available so users can easily access detailed variable information. Learn How to Request Study Access Request study data access, including harmonized and non-harmonized data file access in dbGaP. Click the How to Request Access button in the top right of the Data Files section to view an expandable data access checklist. Navigate to the Request Data Access section of the tutorial to learn more. Figure 1: “Need Support?” Link in Navigation Bar 2. Complete the required fields, indicated with an asterisk Figure 2: Support Request Form 3. Choose the appropriate option under "Request Type" to route the request Figure 3: Request Type Dropdown ○ General Feedback - Provide Data Hub site feedback ○ Technical - Report bugs or other technical problems ○ Feature Request - Provide new feature suggestions ○ Engagement - Request a RADx Data Hub training, presentation, or demonstration ○ Workbench Support - Request help on the Analytics Workbench 4. Under “Request Title,” briefly describe the request 5. Provide specific details about the request under “Request Details” 6. Click “Submit” to complete the support request. You will receive an automated email confirming the Support Team received the request. This will include the ticket number and any further instructions. Soon after submission, a member of the Support Team will contact you with further questions or possible resolutions. User Registration Create an Account To create a RADx Data Hub account, you need a Researcher Auth Service (RAS) Identity Provider (IdP) account, specifically an NIH Login or eRA Commons account. Once you have an account with either IdP, you can register with the RADx Data Hub following the steps below: 1. To register, click “Login” in the top right corner on any RADx Data Hub page. The system will display a modal with a Login/Sign Up using RAS button (Figure 1). Figure 1: Login Modal 2. Press “Login/Sign-Up Using RAS,” and you will be directed to the RAS Sign-In page (Figure 2). Figure 2: RAS Login Page 3. Enter your eRA or NIH Login credentials, and the system will redirect you to the RADx Data Hub User Registration page. (Figure 3). Figure 3: User Registration Form 4. Fill in the required fields. The system automatically displays First Name, Last Name, Middle Initial (M.I.), and Email, based on your RAS information. If these are incorrect, please contact RAS. 5. Click the Institution dropdown and add an institution. If you cannot find your institution, press “Click here to add an institution,” fill out the required fields (Figure 4), and press “Add Institution.” After that, the institution should appear in the Institution dropdown. Figure 4: Add Institution Form 6. Carefully read the Terms and Conditions (Figure 5), and click the provided box to accept conditions. Figure 5: Terms & Conditions 7. Press “Submit” to finish registering. The system will automatically log you in, redirect you to the Home page, and send an email confirming the registration. Note: RADx Data Hub leverages dbGaP to manage study access. When registering for the RADx Data Hub, useregister using the same RAS credentials used for dbGaP (or a linked account). Failure to do this will make it impossible for the system to show your authorized studies available in “My Approved Data.” Login to the RADx Data Hub 1. Click “Login” in the top right corner on any RADx Data Hub page. The system will display an additional modal with a Login/Sign Up option (Figure 1). Figure 1: Login/Sign-Up Modal 1. Press “Login/Sign-Up Using RAS,” and you will be directed to the RAS sign-in page (Figure 2). Figure 2: RAS Sign-In Page 2. Select the NIH Login or eRA option and enter login credentials. The system will redirect you to the home page after correctly entering your credentials. Note: If you have a login.gov account linked to an NIH Login or eRA account used for dbGaP, you may use that to login. Request Data Access Requirements The RADx Data Hub requires eRA Commons authentication and dbGaP authorization to access controlled data. Data requestors must have an eRA Commons (or NIH login) account with PI status to submit a request. Non-PIs must have a PI submit a request in dbGaP on their behalf. Once the PI is granted data access, the PI can grant team members access by logging into dbGaP and adding them as a downloader. Important notes ● Your eRA* or NIH Login used for dbGaP must match your RADx Data Hub login. ○ Users should use an eRA account, if possible ● Non-PIs must have an eRA account to be added as a data downloader Resources: ● eRA Commonds Frequently Asked Questions (FAQs) ● eRA Help and Tutorials ● NIH / eRA Helpdesk / Ticketing System Requesting Access to Studies To gain study data access, including harmonized and non-harmonized data files, PIs must request dbGaP access. 1. Log into the RADx Data Hub using your dbGaP eRA or NIH Login. 2. Locate a study through the Study Explorer, and click on the Study Name to view the Study Overview page. Note: To request access to more than one study, record the dbGaP Study Accession of each study. Later, you will search for and add the dbGaP Study Accession IDs of interest in dbGaP. 3. Click on the dbGaP Study Accession link in the Study Info box on the Study Overview page. This will bring you to the dbGaP Study Overview page. 4. In dbGaP, submit a data access request for the study(ies). Use dbGaP’s Important Links and Information section for guidance. Note: To request access to more than one study, add datasets by navigating to the “Choose Datasets” tab of the Project Request, and type in the dbGaP Study Accession IDs for each study individually. 5. Once you receive a study access confirmation email from dbGaP, return to the RADx Data Hub. Login using the same eRA or NIH Login as for dbGaP, and navigate to My Approved Data. Resources: ● dbGaP: Apply for Controlled Access Data Video ● dbGaP: Frequently Asked Questions (FAQs) ● dbGaP Helpdesk ● Tips for Preparing A Successful Data Access Request Adding Downloaders Downloaders must meet the following requirements prior to the PI adding them as a downloader: ● Have an eRA Commons account ● Have logged into dbGaP at least once After gaining data access, the PI (data request submitter) must log back into dbGaP and: 1. Navigate to the Downloaders tab under Authorized Access 2. Use the First and Last name boxes to search for the downloader 3. Select a team member's name, and add them to the project using Set Downloader After the PI adds a team member as a downloader, the team member will receive a welcome email from dbGaP. Then, the team member can login into the RADx Data Hub using the same dbGaP eRA account as in dbGaP, and navigate to My Approved Data. Resources: Jupyter Lab General Figure 1: Labeled Jupyter Notebook Interface The main elements of JupyterLab editor are: 1. Notebook: A document containing analysis code, outputs, and any additional markdown or text. 2. Cell: A single section of a notebook where to enter code, markdown, or text. 3. Toolbar: Perform the most common notebook actions, including: ○ Save ○ Insert cell below ○ Cut selected cell ○ Copy selected cell ○ Paste from clipboard ○ Run selected cell ○ Interrupt the kernel ○ Restart the kernel ○ Restart the kernel and run all cells ○ Change cell type (i.e. Code, Markdown, Raw) ○ Launch terminal 4. Environment: Displays the current notebook kernel type 5. File Browser: Displays lists of folders, notebooks, and other files ○ The Personal Studio environment is a private. personal Amazon EFS directory 6. Left sidebar: Contains tabs to access the following functionalities: ○ File Browser: Displays lists of folders, notebooks, and other files ○ Running Terminals and Kernels: View current kernels and terminals running in JupyterLab Optionally shut down all or select resources (i.e., notebooks, terminals, kernels, apps, and instances) ○ Git: Connects to a Git repository for Git tool and operation access ○ Table of Contents: Automatically generated for each notebook, Markdown file, or Python file open to navigate the document’s structure with clickable entries ○ Extension Manager: Enables and manages third-party JupyterLab extensions ○ Jupyter AI: A JupyterLab tool to explore generative AI models and integrate them into notebooks Create and Launch a JupyterLab Space The default workspace environment is a ml.t3.medium (2 vCPU, 4 GiB memory) instance type. To create a new JupyterLab space: When the Workbench is launched, select “JupyterLab” from the “Overview” section, or select “JupyterLab” from the “Applications” in the left panel (Figure 2). Figure 2: Workbench Applications Highlighting JupyterLab Select "+ Create JupyterLab space" in the upper right corner of the JupyterLab page ● In the “Create JupyterLab space” dialog, specify a name for the space in the “Name” field. To finish, click “Create space.” ● Note: Because the platform is shared, workspaces must have a unique name. If the workspace name already exists, the following error will appear at the bottom of the page (Figure 3) Figure 3: Error Message for Workspace Name Exists To launch a JupyterLab space: 1. From the Workbench Home page, select “JupyterLab” from the Overview section, or select “JupyterLab” from “Applications” in the left panel (Figure 2). 2. Select “Run” in the Action column of the JupyterLab space to start the workspace (Figure 4). This may take up to a minute to start. Figure 4: Start Running JupyterLab Space 3. Once the status changes to “Running”, select the “Open” icon in the Action column to launch JupyterLab in a new tab (Figure 5). Figure 5: Open JupyterLab Space To create a new notebook: 1. From the landing page, select “File,” “New,” and “Notebook” (Figure 6). ○ In the “Select Kernel” dialog, select a kernel on the dropdown menu. To finish, click “Select”, which launches the notebook. Figure 6: Launch Notebook from File Menu 2. From the Launcher page, click a preferred kernel in the Notebook section (Figure 7). Figure 7: Launch Notebook Using Launcher Upload and Download Files To upload files from a local machine into a JupyterLab space: 1. In the left sidebar, choose the “File Browser” icon 2. In the File Browser, choose the "Upload Files” icon 3. Select the files to upload and choose “Open” 4. Once the file appears in the home folder, double-click the file to open it in a new tab To download a file locally: 1. In the left sidebar, choose the “File Browser” icon 2. Right click the file and select “Download” To Download an entire file locally: 1. From the menu, choose “File,” “New,” and “Terminal”, which will launch a Terminal in a new JupyterLab tab 2. Type the following command replacing folder_name and /path/to/folder: zip -r -X folder_name.zip /path/to/folder 3. Once the folder is zipped and it appears in the File Browser, right click the .zip file and select “Download” Clone a Git Repository Git repositories can be cloned into the JupyterLab home folder using the following steps: 1. Select the Git icon in the left sidebar. 2. Choose “Clone a Repository.” 3. In the Clone Git Repository window, enter the Git URL (for example, https://github.com/aws/amazon-sagemaker-examples.git) 4. Under “Project directory to clone into,” enter the path to the local directory where the cloned directory should exist, otherwise Studio will clone the repository into the home directory. 5. Choose “Clone,” which will automatically open a new terminal window and clone the repository. This may take up to a minute depending on the repository size. 6. If the repository requires credentials, a prompt will appear to enter a username and personal GitHub account access token. 7. When complete, the File Browser will open, displaying the cloned repository. 8. Choose the Git icon to view the Git user interface, which tracks the repository. 9. To track a different repository, open the repository in the file browser and click the Git icon. Create a Persistent Conda Environment Environments can be customized by installing and removing extensions and packages as needed. Any installed extensions and packages installed on the environment will persist. To create persistent conda environments in the JupyterLab application, use the following steps: 1. Open a JupyterLab space. 2. From the landing page, select “File,” “New,” and “Terminal”. 3. Within the terminal, create a new conda environment, replacing myenv with the desired environment name: conda create -n myenv 4. Activate the environment conda activate myenv 5. Install any necessary packages for the environment, for example: conda install numpy pandas 6. Install the ipykernel to create a kernel option. This step can be skipped if it has already been installed: conda install ipykernel 7. Add the new conda environment to the Jupyter kernel, changing the --display-name option as preferred: python -m ipykernel install –user --name myenv --display-name "MyEnvironment" 8. Verify installation of the kernel: jupyter kernelspec list 9. When a notebook is launched, the new kernel should appear. If the kernel is not listed, close the tab and reopen the JupyterLab space Access Public Data To access curated public and synthetic datasets on the RADx Data Hub’s Data Access page, follow the Public Data Tutorial Datasets from the AWS Registry of Open Data, an AWS-hosted repository of more than 400 publicly available datasets, can be copied into a JupyterLab environment using the following steps: 1. Identify a dataset of interest and find the associated Amazon Resource Name (ARN). ○ For example: NIH NCBI Sequence Read Archive (SRA) ○ ARN: arn:aws:s3:::sra-pub-src-1 ○ The bucket name is sra-pub-src-1 2. From the JupyterLab landing page, select “File,” “New,” then “Terminal.” 3. Enter the following command: aws s3 sync s3://sra-pub-src-1 . 4. Replace sra-pub-src-1 with a selected dataset bucket name Change Environment Notebooks launch with the minimum instance type available by default. The minimum instance type is appropriate for most tasks, however, a larger instance can be requested by submitting a Support Request. Follow the instructions in the User Support Requests Tutorial and select “Workbench Support” when choosing a Request Type. Please provide as much detail as possible in the request for the support team to determine the best suitable environment. For more detailed information about available instance types and their performance capabilities, see Available Studio Instance Types. File Sync If an added Workbench file does not appear in the File Browser of JupyterLab, the workspace should be resynced. Close the JupyterLab tab, and refresh the My Approved Data page. Then, follow the steps to relaunch the JupyterLab page. If the files still do not appear, the workspace may need to be manually synced with the following steps: 1. From the File menu, click “File,” “New,” and “Terminal.” 2. Enter the following into the Terminal: ./s3sync.sh If the files in a workspace are still missing, please submit a Support Request. Data Wrangler General By default, Data Wrangler uses the m5.4xlarge (16 vCPU, 64 GiB memory) instance type. To request a difference compute instance, please see the Change environment section. If a Data Wrangler instance has been provisioned, a Data Wrangler flow can be created using the following steps: 1. From the Workbench Home page, select “Canvas” from “Applications” in the left panel. 2. Click “Run Canvas” to start the instance. This may take up to 8 minutes. 3. Once the instance status has changed to “Running”, click the “Open Canvas” button to launch Canvas in a new tab. 4. Select the “Data Wrangler” application in the sidebar of Canvas applications and click “Create a data flow” which will open a dialog to rename the data flow for the analysis. 5. Click import data (Figure 1) and select the data file type (i.e., Tabular or Image). Select one of the data source options from the dropdown, or upload files directly. See Import to learn more about AWS data import options. 6. Data Wrangler can now be used to add transforms, analyze, and visualize your data. To learn more, see Transform Data and Analyze and Visualize. 7. To export a data flow, click Export from the data flow page. To learn more about exporting data transformations to other platforms, see Export. Figure 1: Import Data to Data Wrangler Flow To stop running Canvas, click “Stop Canvas” on the Canvas homepage (Figure 2). Click the checkbox and the “Stop Canvas” button that appears to confirm shutdown. Be sure to save all work and data flows prior to shutdown. Figure 2: Stop Canvas Application SAS Viya General Figure 1: SAS Interface From the My Approved Data or Public Data page, a “Launch SAS” button will appear if a license has been granted. To launch the SAS platform, click “Launch SAS.” To start a new program: 1. From the main menu, click “New”, and “SAS Program”. A new blank program will open in the work area To save a SAS program: 1. Click the “Save” icon on the work area toolbar 2. Select the location where files will be saved 3. Enter the name of the program 4. Click "Save" To open a SAS program: 1. Click the "Open" icon on the main menu bar 2. Find the location of the program in the left panel 3. Select the SAS program to open in the right panel 4. Click “Open” and the program will appear in the work area To run just a portion of a SAS program, highlight the portion to run. To run the entire SAS program, no code needs to be highlighted: 1. Click the "Run" icon on the work area toolbar 2. Open the “Log” tab to confirm the program ran correctly