Objectives:
Important Caveat: For many of the above steps, your instructor will have completed these tasks for you. If you are learning on your own, or trying this after the workshop, see the exercises and instructions below.
The most important thing about The Cloud is choice - instead of purchasing a physical computer, you can obtain on-demand computing at almost any scale. This power comes with advantages and disadvantages:
Advantages of Cloud Computing * Access large amounts of computing power on demand * Full administrative rights - install anything * Use pre-configured images (software already installed)
Disadvantages of Cloud Computing * Cloud computing costs money (you must keep track of your costs) * If you need help, you may not have a local system administrator * Images may be poorly documented (you may not be clear on what is installed, or how to use it)
There are several cloud providers to choose from. Some scientific clouds may either be free or allocate resources competitively. Commercial clouds are can be very powerful, but choice can be overwhelming. We will cover as much as we you need to get through the Data Carpentry lessons, but you will ultimately need to learn things not covered here so see the documentation below:
Tip: Keep in mind, if you are attending a workshop this will have already been done for you!
We will provide instructions for working on the Amazon and the Atmosphere cloud. Follow the instructions for your platform of choice.
Prerequisites
* You can use some of Amazon Web Services for free, or see if you qualify for an AWS Grant (See: https://aws.amazon.com/grants/ ) if you are using AWS for education. The free level of service will not be sufficient for working with the amount of data we are using for our lessons.
For 'Choose and Instance Type' select t2.medium; then click 'Review and Launch'
Tip: You can select larger instances, but these will cost more to run. Also for the 'Review Instance Launch' you will get some warning about instance security. We will not go into detail here about security groups. However, you should know that when you launch an instance, you will want to take steps to make sure no one maliciously accesses your instance. For example, never give your security credentials (next step) to another user.
For the 'Review Instance Launch' step, click 'Launch'
You will be asked to Select an existing key pair or create a new key pair. Select 'Proceed without a key pair' and select the acknowledgement you are given. Then click 'Launch Instances'
You instance will now be launched. You should follow the links to 'Create billing alerts'.
Instructions for MAC
$ ssh dcuser@your.amazon.dns
You should now be connected to your personal instance. You can confirm this with the following commands; whoami
,pwd
,ls
, which should yield the following results:
Last login: Thu Jul 30 13:21:08 2015 from 8.sub-70-197-200.myvzw.com
$ whoami
dcuser
$ pwd
/home/dcuser
$ ls
dc_sample_data Downloads openrefine-2.6-beta.1 Trimmomatic-0.32
Desktop FastQC R
$
Instructions for PC
You should now be connected to your personal instance. You can confirm this with the following commands; whoami
,pwd
,ls
, which should yield the following results:
Last login: Thu Jul 30 13:21:08 2015 from 8.sub-70-197-200.myvzw.com
$ whoami
dcuser
$ pwd
/home/dcuser
$ ls
dc_sample_data Downloads openrefine-2.6-beta.1 Trimmomatic-0.32
Desktop FastQC R
$
Please remember, for as long as this instance is running, you will be charged for your usage. You can see an estimate of the current charge from your AWS EC2 dashboard by clicking your name (Account name) on the upper right of the dashboard and selecting 'Billing & Cost Management'. DO NOT FORGET TO TERMINATE YOUR INSTANCE WHEN YOU ARE DONE
When you are finished with your instance, you must terminate. Follow the following steps.
Under 'Instance State' select terminate.
Warning: This will delete any data on this instance, so you must move any data you wish to save off the instance.
Tip: You can use iCommands to move data between your computer, a cloud instance, and the iPlant Data Store. iCommands is installed on the Data Carpentry Amazon AMI. You can download and see documentation for iCommands here - there is also some documentation on setting up iCommands in the Atmosphere section below
Select 'Yes, Terminate' to terminate the instance.
Prerequisites
Click Launch and then select the following options in the launch wizard:
Tip: You can select a larger instance, but you must have sufficient resources (no exceeded quotas). If Atmosphere is at high capacity, you may not be able to launch an instance if the instance size exceeds what is currently free on the cloud provider
Your instance should be ready in 10-15 minutes. When your instance status is listed as 'active' you will be able to connect.
Instructions for Mac\Linux
Open the terminal application and use 'ssh' to connect. Your command will be:
$ ssh iplantusername@your.atmosphere.ipaddress
Your computer will be unable to verify the authenticity of the host... type yes to continue connecting
When prompted for a password, enter your iPlant username.
You should now be connected to your personal instance. You can confirm this with the following commands; whoami
,pwd
, which should yield the following results:
Welcome to Ubuntu 12.04.2 LTS (GNU/Linux 3.2.0-37-virtual x86_64)
New release '14.04.2 LTS' available.
Run 'do-release-upgrade' to upgrade to it.
_ _ _
/ \ | |_ _ __ ___ ___ ___ _ __ | |__ ___ _ __ ___
/ _ \| __| '_ ` _ \ / _ \/ __| '_ \| '_ \ / _ \ '__/ _ \
/ ___ \ |_| | | | | | (_) \__ \ |_) | | | | __/ | | __/
/_/ \_\__|_| |_| |_|\___/|___/ .__/|_| |_|\___|_| \___|
|_|
iPlant Collaborative
The user manual is located here: http://goo.gl/2pT72
For assistance, contact support@iplantcollaborative.org.
Last login: Fri Aug 14 10:16:50 2015 from dhcp140-78.cshl.edu
iplantusername@vm65-164:~$ whoami
iplantusername
iplantusername@vm65-164:~$ pwd
/home/iplantusername
Note: In the above example 'iplantusername' will be your actual iPlant username.
Instructions for PC
You should now be connected to your personal instance. You can confirm this with the following commands; whoami
,pwd
, which should yield the following results:
Welcome to Ubuntu 12.04.2 LTS (GNU/Linux 3.2.0-37-virtual x86_64)
New release '14.04.2 LTS' available.
Run 'do-release-upgrade' to upgrade to it.
_ _ _
/ \ | |_ _ __ ___ ___ ___ _ __ | |__ ___ _ __ ___
/ _ \| __| '_ ` _ \ / _ \/ __| '_ \| '_ \ / _ \ '__/ _ \
/ ___ \ |_| | | | | | (_) \__ \ |_) | | | | __/ | | __/
/_/ \_\__|_| |_| |_|\___/|___/ .__/|_| |_|\___|_| \___|
|_|
iPlant Collaborative
The user manual is located here: http://goo.gl/2pT72
For assistance, contact support@iplantcollaborative.org.
Last login: Fri Aug 14 10:16:50 2015 from dhcp140-78.cshl.edu
iplantusername@vm65-164:~$ whoami
iplantusername
iplantusername@vm65-164:~$ pwd
/home/iplantusername
Note: In the above example 'iplantusername' will be your actual CyVerse username.
The sample dataset is NOT included on the Atmosphere instance. The very first time you load the instance, you will need to copy data from the public CyVerse Data Store into your instance using the following instructions.
Prerequisites * You must be connected to your Atmosphere instance
iCommands will allow you to quickly transfer data into your Atmosphere instance:
Initialize iCommands using the following command
$ iinit
You will then be asked to setup your account and will need to enter the following information
Prompt | Entry |
---|---|
irodsHost | data.iplantcollaborative.org |
port | 1247 |
zone | iplant |
irodsUserName | your iplant username |
Current iRODS password | your iplant password |
Verify that you have connected to your iPlant Data Store; view the contents of your home directory using the following the ils
command:
$ ils
Enter the following command:
$ iget -rPVT /iplant/home/shared/iplant_training/data_carpentry_ngs/dc_sampledata_lite .
using ls
you should be able to verify you have downloaded the dcsampledatalite directories and files.
Tip: You can use iCommands to move data between your computer, a cloud instance, and the CyVerse Data Store. iCommands is installed on the Data Carpentry Amazon AMI. You can download and see documentation for iCommands here
When you are finished with your instance, you must terminate. Follow the following steps.
Click on the instance name to get to the 'Actions' menu. You may then select the 'Delete' button to terminate the instance.
Warning: This will delete any data on this instance, so you must move any data you wish to save off the instance (for example by using iCommands).
After reading the warning and confirming you are ready, click 'Terminate' to terminate the instance.