##### Setup ##### ************* Setup (Cloud) ************* CloudReg is designed to be used in the cloud but components of the CloudReg pipeline can also be run locally. For instructions on full local use see here. For cloud setup please see below. We chose to work with Amazon Web Services (AWS) and the below setup instructions are for that. Requirements ------------ - AWS account - IAM Role and User with credentials to access EC2 and S3 - S3 Bucket to store raw data - S3 Bucket to store processed data (can be the same bucket as above) - EC2 instance with Docker - EC2 instance with MATLAB - Local computer (used to send commands to cloud services) - (Optional) CloudFront CDN with HTTP/2 enabled for fast visualization - (Optional) Web Application Firewall for IP-address restriction on data access. Create AWS account ------------------ Follow `instructions `_ to create AWS account or use existing AWS account. All of the following AWS setup instructions should be performed within the same AWS account. Create IAM Role --------------- 1. Log into `AWS console `_ 2. Navigate to `IAM section `_ of console 3. Click on *Roles* in the left sidebar 4. Click *Create Role* 5. Click *AWS Service* under *Type of Trusted Entity* 6. Click *EC2* as the AWS Service and click *Next* 7. Next to *Filter Policies*, search for *S3FullAccess* and *EC2FullAccess* and click the checkbox next to both to add them as policies to this role. 8. Click *Next* 9. Click *Next* on the *Add Tags* screen. Adding tags is optional. 10. On the *Review Role* screen, choose a role name, like *cloudreg_role*, and customize the description as you see fit. 11. Finally, click *Create Role* Create IAM User --------------- 1. Log into `AWS console `_ 2. Navigate to `IAM section `_ of console 3. Click on *Users* in the left sidebar 4. Click *Add User* 5. Choose a User name like *cloudreg_user*, check *Programmatic Access*, and click *Next* 6. Click on *Attach existing policies directly* and search for and add *S3FullAccess* and *EC2FullAccess*, and click Next 7. Click *Next* on the *Add Tags* screen. Adding tags is optional. Then click *Next* 8. On the *Review* screen, verify the information is correct and click *Create User* 9. On the next screen, download the autogenerated password and key and keep them private and secure. We will need these credentials later when running the pipeline. Create S3 Bucket ---------------- 1. Log into `AWS console `_ 2. Navigate to `S3 section `_ of console 3. Click *Create Bucket* 4. Choose a bucket name and be sure to choose the bucket region carefully. You will want to pick the region that is geographically closest to you for optimal visualization speeds. Record the region you have chosen. 5. Uncheck *Block All Public Access*. We will restrict access to the data using CloudFront and a Firewall. 6. The remaining settings can be left as is. Click *Create Bucket* Set up CORS on S3 Bucket containing processed data/results ---------------------------------------------------------- 1. Log into `AWS console `_ 2. Navigate to `S3 section `_ of console 3. Click on the S3 Bucket you would like to add CORS to. 4. Click on the *Permissions* tab 5. Scroll to the bottom and click *Edit* under *Cross-origin resource sharing (CORS)* 6. Paste the following text:: [ { "AllowedHeaders": [ "Authorization" ], "AllowedMethods": [ "GET" ], "AllowedOrigins": [ "*" ], "ExposeHeaders": [], "MaxAgeSeconds": 3000 } ] 7. click *Save Changes* Set up Docker EC2 instance ---------------------------- 1. Log into `AWS console `_ 2. Navigate to `EC2 section `_ of console 3. In the left sidebar, click *Instances*. Make sure you change the region (top right, middle drop-down menu) to match that of your raw data and processed data S3 buckets. 4. Click *Launch Instances* 5. In the search bar, enter the following: *ami-098555c9b343eb09c*. This is an Amazing Machine Image (AMI) called *Deep Learning AMI (Ubuntu 18.04) Version 38.0*. Click Select when this AMI shows up. 6. The default instance type should be *t2.micro*, if not choose change it to that type. Leave the remaining choices as their defaults and click *Review and Launch*. 7. Verify the EC2 instance information is correct and click *Launch*. 8. When the key pair pop-up appears, select *Choose an existing key pair* if you have `already created one `_, or select *Create a new key pair* if you do not already have one. Follow the instructions on-screen to download and save the key pair. 9. Follow `AWS tutorial `_ to connect to this EC2 instance through the command line. 10. Once you have connected to the instance via SSH, create the `cloud-volume credentials file `_ on the instance using the CLI text editor of your choice. 11. Install docker-compose by running :code:`sudo curl -L "https://github.com/docker/compose/releases/download/1.28.0/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose; sudo chmod +x /usr/local/bin/docker-compose` 12. Run :code:`sudo shutdown now` to turn off the EC2 instance 13. Record the "Instance ID" of this CloudReg instance (this can be found in the EC2 console). We will need this when running the pipeline. Set up MATLAB EC2 instance -------------------------- 1. Follow instructions `here `_ on setting up MATLAB on an EC2 instance. Be sure to create this instance in the same region as your S3 buckets. Be sure to use the same SSH key you created for the CloudReg EC2 instance. 2. After creating this instance, navigate to the EC2 console and record the "Instance ID" of this MATLAB instance. We will need this when running the pipeline. Set up AWS CloudFront --------------------- 1. Log into `AWS console `_ 2. Navigate to `CloudFront section `_ of console 3. Click "Create Distribution" and then click "Get Started". 4. Click in the "Origin Domain Name" box and select the S3 bucket you previously created to store preprocessed data for visualization. Once you select your S3 bucket from the drop-down menu, the Origin ID should populate automatically. 5. Leave all other default parameters under "Origin Settings". 6. See the video below on how to set up the remaining parameters. 7. After following the video, click "Create Distribution". NOTE: Be sure to save the CloudFront URL that is created for that distribution. It can be found at the CloudFront console homepage after clicking on the distribution you created. It should appear next to "Domain Name". .. raw:: html Set up AWS Web Application Firewall ----------------------------------- 1. Before setting up the Web Application Firewall, please find the IP address(es) you would like to give access to. Oftentimes this information can be discovered by emailing IT at your institution or going to `whatismyip `_ for just your IP address. 2. Log into `AWS console `_ 3. Navigate to `WAF section `_ of console. This link will redirec you to WAF classic in order to implement our firewall. 4. In the drop-down menu next to "Filter", select "Global (CloudFront)". 5. Click "Create Web ACL". 6. Choose a name that is unique for your web ACL and leave the CloudWatch metric name and Region Name as is. 7. Click on the drop-down next "AWS resource to associate" and choose the CloudFront distribution you created previously. 8. Click "Next" 9. To the right of "IP Match Conditions", click "Create Condition". 10. Choose a unique name and leave the region as "Global". 11. Next IP address range, input the IP range that you obtained in step 1. You can verify this range with a `CIDR calculator `_ 12. Click "Create" at the bottom right and then click "Next". 13. Click "Create Rule" to the right of "Add rules to web ACL". 14. Choose a name and leave the other 2 parameters as default. 15. Under "Add conditions", choose "does" and "originate from an IP address in" 16. Under the third drop-down, choose the rule you created in step 14. 17. Under "If a request matches all of the conditions in a rule, take the corresponding action", choose allow. 18. Under "If a request doesn't match any rules, take the default action" choose "block all requests that don't match rules" 19. Click "Review and Create" and then on the next page choose, "Confirm and create". Local machine setup ------------------- On a local machine of your choice follow the instructions below. The following instructions should be followed from within a terminal window (command line). The below steps only need to be done the FIRST TIME you set up the pipeline. 1. Install `Docker `_ 2. Make sure Docker is open and running. 3. Open a new Terminal window. 4. Pull the CloudReg docker image: :code:`docker pull neurodata/cloudreg:local` ************* Setup (Local) ************* CloudReg is designed to be used in the cloud but components of the CloudReg pipeline can also be run locally. Instructions for local setup are below. Requirements ------------ - Local Machine - MATLAB license Local Machine Setup ------------------- On a local machine of your choice follow the instructions below. The following instructions should be followed from within a terminal window (command line). The below steps only need to be done the FIRST TIME you set up the pipeline. 1. Install `Docker `_. 2. Install `MATLAB `_. 3. Make the :code:`matlab` command available from the terminal. 4. Open a new Terminal window. 5. Make sure Docker is open and running. 6. Pull the CloudReg Docker image: :code:`docker pull neurodata/cloudreg:local`.