Organizing Genomics Projects
Getting your project started
Project organization is one of the most important parts of a sequencing project, but is often overlooked in the excitement to get a first look at new data. While it's best to get yourself organized before you begin analysis, it's never too late to start.
You should approach your sequencing project in a very similar way to how you do a biological experiment, and ideally, begins with experimental design. We're going to assume that you've already designed a beautiful sequencing experiment to address your biological question, collected appropriate samples, and that you have enough statistical power. For all of those steps, collecting specimens, extracting DNA, prepping your samples, you've likely kept a lab notebook that details how and why you did each step, but documentation doesn't stop at the sequencer!
Every computational analysis you do is going to spawn many files, and inevitability, you'll want to run some of those analysis again. Genomics projects can quickly accumulates hundreds of files across tens of folders. Do you remember what PCR conditions you used to create your sequencing library? Probably not. Similarly, you probably won't remember whether your best alignment results were in Analysis1, AnalysisRedone, or AnalysisRedone2; or which quality cutoff you used.
Luckily, recording your computational experiments is even easier than recording lab data. Copy/Paste will become your best friend, sensible file names will make your analysis traversable by you and your collaborators, and writing the methods section for your next paper will be a breeze. Let's look at the best practices for documenting your genomics project.
Your future self will thank you.
Exercise
In this exercise we will setup a filesystem for the project we will be using over the next few days. We will also introduce you to some helpful shell commands/programs/tools:
mkdirhistorytail|nano>>
A. Create a file system for a project
Inspired by the guide below, we will start by create a directory that we can use for the rest of the workshop:
First, make sure that you are in your home directory:
$ pwd
/home/dcuser
# Hopefully you got the above output '/home/dcuser'
Tip: Remember, when we give a command, rather than copying and pasting, just type it out. Also the '$' indicates we are at the command prompt, do not include that in your command.
Tip If you were not in your home directory, the easiest way to get there is to enter the command cd which always returns you to home.
Next, try making the following directories using the mkdir command
- dc_workshop
- dc_workshop/docs
- dc_workshop/data
- dc_workshop/results
Verify that you have created the directories;
$ ls -R dc_workshop
if you have created these directories, you should get the following output from that command:
dc_workshop/:
data docs results
dc_workshop/data:
dc_workshop/docs:
dc_workshop/results:
B. Document your activity on the project
The history command is a convenient way to document the all the commands you have used while analyzing and manipulating your project. Let's document the work we have done to create these folders.
- View the commands that you have used so far during this session using
history:bash $ history
The history likely contains many more commands that you have used just for these projects. Let's view the last several commands so that focus on just what we need for the project. View the last n lines of your history (where n = approximately the last few lines you think relevant - for our example we will use the last 7:
$ history | tail -n7As you may remember from the shell lesson, the pipe
|sends the output of history to the next program, in this case, tail. We have used the -n option to give the last 7 lines.Using your knowledge of the shell use the append redirect
>>to create a file called dcworkshoplogXXXXXX_XX.txt (Use the four-digit year, two-digit month, and two digit day, e.g. dcworkshoplog201507_30.txt)You may have noticed that your history may contain the
historycommand itself. To remove this redundancy from our log, lets use thenanotext editor to fix the file:bash $ nano dc_workshop_logFrom the nano screen, you should be able to use your cursor to navigate, type, and delete any redundant lines.Add a dateline and comment to the line where you have created the directory e.g.
# 2015_07_30 # Created sample directories for the Data Carpentry workshopNext, remove any lines of the history that are not relevant. Just navigate to those lines and use your delete key.
Close nano by hitting 'Control' and the 'X' key at the same time; notice in nano this is abbreviated '^X'; nano will ask if you want to save; hit 'Y' for yes. When prompted for the 'File Name to Write' we can hit 'Enter' to keep the same name and save.
Now that you have created the file, move the file to 'dc_workshop/docs' using the
mvcommand.
C. Intro to Markdown
As you start to build your notebook, there is an easy way to format these notes - Markdown.
Markdown is a simplified version of HTML, the code that is behind most webpages on the internet.
One reason to markdown is that it is a simple way to create documents that anyone can access. You may be very use to using word processing programs like Microsoft Word. While these are very nice, they are not free and not everyone has access. Also, when we are doing work on the computer usually using Linux we would not be able to read these Microsoft documents.
To write something in Markdown, all you need to do is write text like you always do, but you will format it using some special symbols. Using an editor
It helps to have an editor so that you can see what your markdown will look like when it is rendered (final version that people see). Suggestions for free software include:
Markdown basics
See a really good guide at: https://help.github.com/articles/markdown-basics/ Paragraphs
Regular paragraphs are no different than typing on any text editor/word processor. Just type.
Adding Headings Use the '#' key to make a line a heading (subtitle). For really big fonts use just one '#' the more '#'s' used, the smaller the subtitle.
Example 1
Example 2
Example 3
Example 4
Bolding and emphasis
Surrounding a word or phrase with one set of asterisks () makes it italic. Using two sets **makes it bold.
Lists
Lists can be * Unordered * Start with a single asterisk * Have one item per line.
To make an ordered list: 1. Start the list with one number, followed by a period 2. On the next line, just use the next number.
Links and Images
A link is properly written in markdown using a pair of square brackets '[]' followed by round parentheses '()'. The text of the link will go in the square brackets, and the actual URL will go in the parentheses:
If you want an image, just follow the link text in the brackets with an exclamation mark ''
Questions:
1. What is the default number of lines that tail displays?
2. What is the difference between '>' and '>>'