# Variant of CPM for datasets with library sizes of fewer than 1 mil molecules

```

Now we will compare various single-cell DE methods. We will focus on methods

that performed well in Soneson and Robinson's [2019; CITE] detailed comparison

of differential expression methods for single-cell data. Note that we will only

be running methods which are available as R-packages and run relatively quickly.

that performed well in Soneson and Robinson's [@Soneson2018-hy] detailed

comparison of differential expression methods for single-cell data. Note that we

will only be running methods which are available as R-packages and run

relatively quickly.

### Kolmogorov-Smirnov test

...

...

@@ -103,10 +109,14 @@ length(sigDE)

sum(GroundTruth$DE %in% sigDE)

# Number of KS-DE genes that are true DE genes

sum(GroundTruth$notDE %in% sigDE)

# Number of KS-DE genes that are truly not-DE

# Number of KS-DE genes that are truly not-DE

```

As you can see many more of our ground truth negative genes were identified as DE by the KS-test (false positives) than ground truth positive genes (true positives), however this may be due to the larger number of notDE genes thus we typically normalize these counts as the True positive rate (TPR), TP/(TP + FN), and False positive rate (FPR), FP/(FP+TP).

As you can see many more of our ground truth negative genes were identified as

DE by the KS-test (false positives) than ground truth positive genes (true

positives), however this may be due to the larger number of notDE genes thus we

typically normalize these counts as the True positive rate (TPR), TP/(TP + FN),

Now we can see the TPR is much higher than the FPR indicating the KS test is identifying DE genes.

So far we've only evaluated the performance at a single significance threshold. Often it is informative to vary the threshold and evaluate performance across a range of values. This is then plotted as a receiver-operating-characteristic curve (ROC) and a general accuracy statistic can be calculated as the area under this curve (AUC). We will use the ROCR package to facilitate this plotting.

Now we can see the TPR is OK, but the FDR is very high--- much higher than we

would normally deem acceptable.

So far we've only evaluated the performance at a single significance threshold.

Often it is informative to vary the threshold and evaluate performance across a

range of values. This is then plotted as a receiver-operating-characteristic

curve (ROC) and a general accuracy statistic can be calculated as the area under

this curve (AUC). We will use the ROCR package to facilitate this plotting.

```{r ks-roc-plot, fig.cap="ROC curve for KS-test."}

# Only consider genes for which we know the ground truth

...

...

@@ -133,24 +149,33 @@ ROCR::plot(perf)

aucObj <- ROCR::performance(pred, "auc")

aucObj@y.values[[1]] # AUC

```

Finally to facilitate the comparisons of other DE methods let's put this code into a function so we don't need to repeat it:

Finally to facilitate the comparisons of other DE methods let's put this code

into a function so we don't need to repeat it:

```{r}

DE_Quality_AUC <- function(pVals) {

DE_Quality_AUC <- function(pVals, plot=TRUE) {

pVals <- pVals[names(pVals) %in% GroundTruth$DE |

names(pVals) %in% GroundTruth$notDE]

truth <- rep(1, times = length(pVals));

truth[names(pVals) %in% GroundTruth$DE] = 0;

pred <- ROCR::prediction(pVals, truth)

perf <- ROCR::performance(pred, "tpr", "fpr")

ROCR::plot(perf)

if (plot)

ROCR::plot(perf)

aucObj <- ROCR::performance(pred, "auc")

return(aucObj@y.values[[1]])

}

```

### Wilcox/Mann-Whitney-U Test

The Wilcox-rank-sum test is another non-parametric test, but tests specifically if values in one group are greater/less than the values in the other group. Thus it is often considered a test for difference in median expression between two groups; whereas the KS-test is sensitive to any change in distribution of expression values.

### Wilcoxon/Mann-Whitney-U Test

The Wilcoxon-rank-sum test is another non-parametric test, but tests

specifically if values in one group are greater/less than the values in the

other group. Thus it is often considered a test for difference in median

expression between two groups; whereas the KS-test is sensitive to any change in

distribution of expression values.

```{r wilcox-plot, fig.cap="ROC curve for Wilcox test.", message=FALSE, warning=FALSE}

pVals <- apply(

...

...

@@ -173,17 +198,34 @@ We've already used edgeR for differential expression in Chapter

gene expression and uses a generalized linear model (GLM) framework, the enables

us to include other factors such as batch to the model.

```{r edger-plot, fig.cap="ROC curve for edgeR.", message=FALSE}

`edgeR`, like many DE methods work best with pre-filtering of very lowly

experessed genes. Here, we will keep genes with non-zero expression in at least

30 cells.

The first steps for this analysis then involve

```{r edger-plot, fig.cap="Biological coefficient of variation plot for edgeR.", message=FALSE}

There are many different imputation methods available we will consider three

fast, published methods:

...

...

@@ -50,7 +54,9 @@ imputed all inflated zeros.

### scImpute

To test `scImpute`, we use the default parameters and we apply it to the Deng dataset that we have worked with before. scImpute takes a .csv or .txt file as an input:

To test `scImpute`, we use the default parameters and we apply it to the Deng

dataset that we have worked with before. scImpute takes a .csv or .txt file as

@@ -107,11 +115,14 @@ __Exercise:__ Based on the PCA and the clustering results, do you think that imp

### DrImpute

We can do the same for DrImpute. DrImpute runs on a log-normalized expression matrix directly in R, we generate this matrix using scater, then run DrImpute. Unlike scImpute, DrImpute considers the consensus imputation across a range of ks using two differ correlation distances:

We can do the same for DrImpute. DrImpute runs on a log-normalized expression

matrix directly in R, we generate this matrix using scater, then run DrImpute.

Unlike scImpute, DrImpute considers the consensus imputation across a range of

ks using two differ correlation distances:

```{r}

deng <- normalize(deng)

res <- DrImpute(deng@assays[["logcounts"]], ks=8:12)

__Exercise:__ Check the sc3 clustering of the DrImpute matrix, do you think that imputation using `DrImpute` is a good idea for the Deng dataset?

__Exercise:__ Check the sc3 clustering of the DrImpute matrix, do you think that

imputation using `DrImpute` is a good idea for the Deng dataset?

__Exercise:__ What is the difference between `scImpute` and `DrImpute` based on the PCA and clustering analysis? Which one do you think is best to use?

__Exercise:__ What is the difference between `scImpute` and `DrImpute` based on

the PCA and clustering analysis? Which one do you think is best to use?

Next, you can evaluate the markers of Thymus T cell in Thymus stromal cell

```{r, eval=FALSE}

evaluateMarkers(

scfind_index,

gene.list = as.character(t_cell_markers$genes),

cell.types = "Thymus.stromal cell",

sort.field = "f1"

)

```

```{r, eval=FALSE}

# By default, the marker evaluation takes all cell types in the dataset as background cell type, but you can use the argument `background.cell.types` to fine tune the evaluation