Count how many MiSeq reads derived from each surface of the flowcell

I recently had call to perform one of those tasks that I think others might, yet not be entirely sure how to go about it.

Specifically, in troubleshooting a MiSeq run's poor yield, I wanted to see whether there were significantly more reads derived from one of the flow cell surfaces (top or bottom) relative to the other. The reason I did this was my FWHM (full cluster width at half maximum, a measure of the focus during imaging) was noticeably higher for that surface.

I mean, I have no idea if ~3 is that much worse than ~2.8-2.9, but there's no harm in checking right?

This is very easily achieved as all of the information required to work it out is contained within the FASTQ reads themselves, in tile section of the identifier line of each each.

Therefore with a quick bit of basic bash we can find out exactly how many reads derived from each surface.

# Get all index reads (as the shortest) in one file
zcat *I1*z > I1.fq

# Extract the identifier lines with sed
 # and grep for those with a '1' at the right position
 # This indicated they derived from the top surface
sed '2~4d;3~4d;4~4d' I1.fq | grep ^.............................1 -c

# Do the same for '2', i.e. the bottom surface
sed '2~4d;3~4d;4~4d' I1.fq | grep ^.............................2 -c

And there you have it. Simple, quick and effective.

(As it turned out I have almost equal numbers derived from both surfaces, so it wasn't to blame in my case, but this might be useful for other situations!)

Extracting sample names from Illumina FASTQ filenames

Another post in the vein of "minor analysis task and my solution": how to extract the sample name or identifier (i.e. what you called a given file in the samplesheet) from a list of Illumina FASTQ file names.

So you might have something that looks like this (which I've cobbled together from a couple of runs from different machines that I pulled off Basespace):
 
322_S18_R2_001.fastq.gz 
1c8_P6_act_S11_R2_001.fastq.gz
5c4_P8_res_S1_R1_001.fastq.gz

In this exercise, I wanted the first part of the IDs, which are what we labelled the actual samples with. In an ideal world this should be very easy: we could just use sed or cut to delete everything after the first instance of a character used in the text that Illumina appends to the sample name. However different people (or the same people at different times) use different naming conventions, which can often include underscores, the character that separates the string fields that get added.

Therefore what we need to do is not delete from say the first pattern, but from the nth pattern from the end, which turned out to be a slightly trickier problem (at least to one such as myself that doesn't find sed to be the most intuitive program in the world). Here's what I came up with:

sed 's/\(.*\)_\(.*_\)\(.*\)_/\1|\2/;s/|.*//' FilenameList.txt

What this does is find the third underscore to last (just before S[XX]_R[X].fastq.gz string) and replace it with a pipe character ('|'), which should never appear in a filename, before finding this pipe character and deleting everything after it. This produces the following output:
 
322
1c8_P6_act
5c4_P8_res

This feels a bit hacky to me, as really I feel like I should be able to do this in one substitution, but the first part of the command actually selects the latter half (so if anyone knows how to tidy this up I'd love to hear it!).

It's also worth pointing out that the output of different machines or demultiplexing settings will differ in the number of underscores, and so the command will need to change slightly. For instance, the above example was run on NextSeq data demultiplexed using bcl2fastq --no-lane-splitting. For an example of a file containing lane information ('L001' etc), here's the equivalent on some MiSeq filenames I pulled up.

# This 
324bAB_S8_L001_R1_001.fastq.gz 
076-PBMCs-alpha_S5_L001_R1_001.fastq.gz 

# After this
sed 's/\(.*\)_\(.*_\)\(.*\)_\(.*\)_/\1|\2/;s/|.*//' FilenameList.txt   

# Becomes this 
324bAB
076-PBMCs-alpha

The reason I was actually doing this was because I had a number of different NextSeq runs, some of which contained samples sequenced from the same original libraries (to increase the depth), so I needed to get the names of all samples sequenced in each run. I had each run in separate directories, and then ran this in the folder above:

for i in */
do echo $i
ls $i | grep R1 | grep fastq.gz | sed 's/\(.*\)_\(.*_\)\(.*\)_/\1|\
2/;s/|.*//' > ${i%?}.fl
done

This goes through all the directories, finds the R1 FASTQ files (as there are other files in there, and this gives me only one entry per sample) and strips their sample names out as above, and writes these to a textfile with the suffix '.fl'.

The key to finding TCR sequences in RNA-seq data

I had previously written a short blog post touching on how I'd tried to mine some PBMC RNA-seq data (from the ENCODE project) for rearranged T-cell receptor genes, to try and open up this huge resource for TCR repertoire analysis. However, I hadn't gotten very far, on account of finding very few TCR sequences per file.

That sets the background for an extremely pleasant surprise this morning, when I found that Scott Brown, Lisa Raeburn and Robert Holt from Vancouver (the latter of whom being notable for producing one of the very earliest high-throughput sequencing TCR repertoire papers) had published a very nice paper doing just that!

This is a lovely example of different groups seeing the same problem and coming up with different takes. I saw an extremely low rate of return when TCR-mining in RNA-seq data from heterogeneous cell types, and gave up on it as a search for needles in a haystack. The Holt group saw the same problem, and simply searched more haystacks!

This paper tidily exemplifies the re-purposing of biological datasets to allow us to ask new biological questions (something that I consider a practical and moral necessity, given the complexity of such data and the time and costs involved in their generation).

Moreover, they do some really nice tricks, like estimating TCR transcript proportions in other data sets based on constant region usage, investigate TCR diversity relative to CD3 expression, testing on simulated RNA-seq data sets as a control, looked for public or known-specificity receptors and inferred possible alpha-beta pairs by checking all each sample's possible combinations for their presence in at least one other sample (somewhat akin to Harlan Robins' pairSEQ approach).

All in all, a very nice paper indeed, and I hope we see more of this kind of data re-purposing in the field at large. Such approaches could certainly be adapted for immunoglobulin genes. I also wonder if, given whole-genome sequencing data from mixed blood cell populations, we might even be able to do a similar analysis on rearranged variable antigen receptors from gDNA.

Heterogeneity in the polymerase chain reaction

I've touched briefly on some of the insights I made writing my thesis in a previous blog post. The other thing I've been doing a lot of over the last year or so is writing and contributing to papers. I've been thinking that it might be nice to write a few little blog posts on these, to give a little background information on the papers themselves, and maybe (in the theme of this blog) share a little insight into the processes that went into making them.

The paper I'll cover in this piece was published in Scientific Reports in October. I won't go into great detail on this one, not least because I'm only a (actually, the) middle author on it: this was primarily the excellent work of my friends and colleagues Katharine Best and Theres Oakes, who performed the bulk of the analysis and wet-lab work respectively (although I also did a little of both). Also, our supervisor Benny Chain summarised the findings of the article itself on his own blog, which covers the principles very succinctly.

Instead, I thought I'd write this blog to share that piece of information that I always wonder about when I read a paper: what made them look at this, what put them on this path? This is where I think I made my major contribution to this paper, as (based on my recollections) it began with observations made during my PhD.

My PhD primarily dealt with the development and application of deep-sequencing protocols for measuring T-cell receptor (TCR) repertoires (which, when I started, there were not many published protocols for). As a part of optimising our library preparation strategies, I thought that we might use random nucleotide sequences in our PCR products – which were originally added to increase diversity, overcoming a limitation in the Illumina sequencing technology – to act as unique molecular barcodes. Basically, adding random sequences to our target DNA before amplification uniquely labels each molecule. Then, in the final data we can infer that any matching sequences that share the same barcode are probably just PCR duplicates, if we have enough random barcodes*, meaning that sequence was less prevalent in the original sample than one might think based on raw read counts. Not only does this provide better quantitative data, but by looking to see whether different sequences share a barcode we can find likely erroneous sequences produced during PCR or sequencing, improving the qualitative aspects of the data as well. Therefore we thought (and still do!) that we were on to a good thing.

(Please note that we are not saying that we invented this, just that we have done it: it has of course been done before, both in RNA-seq (e.g. Fu et al, 2011 and Shiroguchi et al, 2012) at large and in variable antigen receptor sequencing (Weinstein et al, 2009), but it certainly wasn't widespread at the time; indeed there's really only one other lab I know of even now that's doing it (Shugay et al, 2014).)

However, in writing the scripts to 'collapse' the data (i.e. remove artificial sequence inflation due to PCR amplification, and throw out erroneous sequences) I noticed that the degree to which different TCR sequences were amplified followed an interesting distribution:

Here I've plotted the raw, uncollapsed frequency of a given TCR sequence (i.e. the number of reads containing that TCR, here slightly inaccurately labelled 'clonal frequency') against that value divided by the number of random barcodes it associated with, giving a 'duplication rate' (not great axis labels I agree, but this is pulling the plots straight out of a lab meeting I gave three years ago). The two plots show the same data, with a shortened X axis on the right to show the bulk of the spread better.

We can see that above a given frequency – in this case about 500, although it varies – we observe a 'duplication rate' around 70. This means that above a certain size, sequences are generally amplified at a rate proportional to their prevalence (give or take the odd outlier), or that for every input molecule of cDNA it gets amplified and observed seventy times. This is the scenario we'd generally imagine for PCR. However, below that variable threshold there is a very different, very noisy picture, where the amount to which a sequence is found to be amplified and observed is not related to the collapsed prevalence. This was the bait on the hook that lead our lab down this path.

Now, everyone knows PCR doesn't behave like it does in the diagrams, like it should. That's what everyone always says (usually as they stick another gel picture containing mysteriously sized bands into their labbooks). However, people have rarely looked at what's actually going on. There's a bit of special PCR magic that goes on, and a million different target and reaction variables that might affect things: you just optimise your reaction until your output looks like what you'd expect. It's only with the relatively recent advances in DNA sequencing technology that we can actually look at exactly what molecules are being made in the reaction that we can start to get actual data showing how just un-like the schematics the reaction can in fact behave.

This is exactly what Katharine's paper chases up, applying the same unique molecular barcoding strategy to TCR sequences from both polyclonal and monoclonal** T-cells. I won't go into the details, because hey, you can just read the paper (which says it much better), but the important thing is that this variability is not due to the standard things you might expect like CG content, DNA motifs or amplicon length, because it happens even for identical sequences. It doesn't matter how well you control your reactions, the noise in the system breeds variability. This makes unique molecular barcoding hugely important, at least if you want accurate relative quantitation of DNA species across the entire dynamic range of your data.

* Theoretically about 16.7 million in our case, or 4¹², as we use twelve random nucleotides in our barcodes.

** Although it's worth saying that while the line used, KT-2, is monoclonal, that doesn't mean the TCR repertoire is exactly as clean as you'd expect. T-cell receptor expression in T-cell lines is another thing that isn't simple as the textbook diagrams pretend.