Zhihui
před 4 roky
revize
3f1c24338b
7 změnil soubory, kde provedl 306 přidání a 0 odebrání
binární
.DS_Store
Zobrazit soubor
+ 120
- 0
README.md
Zobrazit soubor
| @@ -0,0 +1,120 @@ | |||
| # RNA Sequencing Quality Control Pipeline | |||
| > Author： Li Zhihui | |||
| > | |||
| > E-mail：18210700119@fudan.edu.cn | |||
| > | |||
| > Git: | |||
| > | |||
| > Last Updates: 2020/07/13 | |||
| ## 安装指南 | |||
| ``` | |||
| # 激活choppy环境 | |||
| source activate choppy | |||
| # 安装app | |||
| choppy install lizhihui/test_dataportol1 | |||
| ``` | |||
| ## App概述——中华家系1号标准物质介绍 | |||
| 建立高通量全基因组测序的生物计量和质量控制关键技术体系，是保障测序数据跨技术平台、跨实验室可比较、相关研究结果可重复、数据可共享的重要关键共性技术。建立国家基因组标准物质和基准数据集，突破基因组学的生物计量技术，是将测序技术转化成临床应用的重要环节与必经之路，目前国际上尚属空白。中国计量科学研究院与复旦大学、复旦大学泰州健康科学研究院共同研制了人源中华家系1号基因组标准物质（**Quartet，一套4个样本，编号分别为LCL5，LCL6，LCL7，LCL8，其中LCL5和LCL6为同卵双胞胎女儿，LCL7为父亲，LCL8为母亲**），以及相应的全基因组测序序列基准数据集（“量值”），为衡量基因序列检测准确与否提供一把“标尺”，成为保障基因测序数据可靠性的国家基准。人源中华家系1号基因组标准物质来源于泰州队列同卵双生双胞胎家庭，从遗传结构上体现了我国南北交界的人群结构特征，同时家系的设计也为“量值”的确定提供了遗传学依据。 | |||
| 该Quality_control APP用于转录组测序（RNA Sequencing，RNA-Seq）数据的质量评估，包括原始数据质控、比对数据质控和基因表达数据质控。 | |||
| ## 流程与参数 | |||
|  | |||
| ## App输入文件 | |||
| inputSamplesFile | |||
| ``` | |||
| #read1 #read2 #sample_id #adapter_sequence #adapter_sequence_r2 | |||
| ``` | |||
| read1 是阿里云上fastq read1的地址 | |||
| read2 是阿里云上fastq read2的地址 | |||
| sample_id 是指样本的命名 | |||
| adapter_sequence 是R1端需要去除的接头 | |||
| adapter_sequence_r2 是R2端需要去除的接头 | |||
| 所有上传的文件应有规范的命名 | |||
| ## App输出文件 | |||
| 1.上游质控参数 | |||
| | 列名 | 说明 | 范围 | | |||
| | -------------------------- | ---- | ---- | | |||
| | SampleID | | | | |||
| | #Date | | | | |||
| | #LibraryPrep | | | | |||
| | Replicate | | | | |||
| | Sample | | | | |||
| | #SequenceMachine | | | | |||
| | #SequenceSite | | | | |||
| | #SequenceTech | | | | |||
| raw reads | |||
| | Total_Reads_After_Trimming | | | | |||
| | GC_content |* | | | |||
| | Human.percentage | | | | |||
| | #ERCC.percentage | | | | |||
| | EColi.percentage | | | | |||
| | Adapter.percentage | | | | |||
| | #Vector.percentage | | | | |||
| | rRNA.percentage | | | | |||
| | Virus.percentage | | | | |||
| | Yeast.percentage | | | | |||
| | Mitoch.percentage | | | | |||
| | Phix.percentage | | | | |||
| | No.hits.percentage | | | | |||
| | GC_content_bamqc | | | | |||
| | Mapping_Ratio | * | | | |||
| | Insert_size_median | * | | | |||
| | Insert_size_peak | * | | | |||
| error rate | |||
| average length | |||
| 3’5‘ gene cover | |||
| duplication | |||
| strand bias | |||
| 2.下游质控参数 | |||
| | Quality metrics | Category | Description | Reference value | | |||
| | ----------------------------------------- | ----------- | ------------------------------------------------------------ | --------------- | | |||
| | Number of detected genes | One group | This metric is used to estimate the detection abundance of one sample. | (**, 58,395] | | |||
| | Detection Jaccard index (JI) | One group | Detection JI is the ratio of number of the genes detected in both replicates than the number of the genes detected in either of the replicates. This metric is used to estimate the repeatability of one sample detected gene from different replicates. | [0.8, 1] | | |||
| | Coefficient of variation (CV) | One group | CV is calculated based on the normalized expression levels in all 3 replicates of one sample for each genes. This metric is used to estimate the repeatability of one sample expression level from different replicates. | [0, 0.2] | | |||
| | Correlation of technical replicates (CTR) | One group | CTR is calculated based on the correlation of one sample expression level from different replicates. | [0.95, 1] | | |||
| | Signal-to-noise Ratio (SNR) | More groups | Signal is defined as the average distance between libraries from the different samples on PCA plots and noise are those form the same samples. SNR is used to assess the ability to distinguish technical replicates from different biological samples. | [5, inf) | | |||
| | Sensitivity of detection | One group | Sensitivity is the proportion of "true" detected genes from reference dataset which can be correctly detected by the test set. | [0.96, 1] | | |||
| | /Reference dependent | | | | | |||
| | Specificity of detection | One group | Specificity is the proportion of "true" non-detected genes from reference dataset which can be correctly not detected by the test set. | [0.94, 1] | | |||
| | /Reference dependent | | | | | |||
| | Consistency ratio of relative expression | Two groups | Proportion of genes that falls into reference range (mean ± 2 fold SD) in relative ratio (log2FC). | [0.82, 1] | | |||
| | /Reference dependent | | | | | |||
| | Correlation of relative log2FC | Two groups | Pearson correlation between mean value of reference relative ratio and test site. | [0.96,1] | | |||
| | /Reference dependent | | | | | |||
| | Sensitivity of DEGs | Two groups | Sensitivity is the proportion of "true" DEGs from reference dataset which can be correctly identified as DEG by the test set. | [0.80, 1] | | |||
| | /Reference dependent | | | | | |||
| | Specificity of DEGs | Two groups | Specificity is the proportion of "true" not DEGs from reference dataset which can be can be correctly identified as non-DEG by the test set. | [0.95, 1] | | |||
| | /Reference dependent | | | | | |||
| ## 结果展示与解读 | |||
+ 46
- 0
defaults
Zobrazit soubor
| @@ -0,0 +1,46 @@ | |||
| { | |||
| "idx": "oss://pgx-reference-data/reference/hisat2/grch38_snp_tran/", | |||
| "gtf": "oss://pgx-reference-data/reference/annotation/Homo_sapiens.GRCh38.93.gtf", | |||
| "idx_prefix": "genome_snp_tran", | |||
| "hisat2.docker": "registry.cn-shanghai.aliyuncs.com/pgx-docker-registry/hisat2:v2.1.0-2", | |||
| "hisat2.cluster": "OnDemand bcs.a2.3xlarge img-ubuntu-vpc", | |||
| "samtools.docker": "registry.cn-shanghai.aliyuncs.com/pgx-docker-registry/samtools:v1.3.1", | |||
| "samtools.cluster": "OnDemand bcs.a2.large img-ubuntu-vpc", | |||
| "stringtie.docker": "registry.cn-shanghai.aliyuncs.com/pgx-docker-registry/stringtie:v1.3.4", | |||
| "stringtie.cluster": "OnDemand bcs.a2.large img-ubuntu-vpc", | |||
| "fastp.docker": "registry.cn-shanghai.aliyuncs.com/pgx-docker-registry/fastp:0.19.6", | |||
| "fastp.cluster": "OnDemand bcs.a2.large img-ubuntu-vpc", | |||
| "fasta": "GRCh38.d1.vd1.fa", | |||
| "fastqc.disk_size": "150", | |||
| "gtf": "oss://pgx-reference-data/reference/annotation/Homo_sapiens.GRCh38.93.gtf", | |||
| "fastqscreen.cluster_config": "OnDemand bcs.b2.3xlarge img-ubuntu-vpc", | |||
| "fastqc.cluster_config": "OnDemand bcs.b2.3xlarge img-ubuntu-vpc", | |||
| "qualimapBAMqc.disk_size": "500", | |||
| "fastqc.docker": "registry.cn-shanghai.aliyuncs.com/pgx-docker-registry/fastqc:v0.11.5", | |||
| "fastqscreen.docker": "registry.cn-shanghai.aliyuncs.com/pgx-docker-registry/fastqscreen:0.12.0", | |||
| "screen_ref_dir": "oss://pgx-reference-data/fastq_screen_reference/", | |||
| "fastq_screen_conf": "oss://pgx-reference-data/fastq_screen_reference/fastq_screen.conf", | |||
| "multiqc.cluster_config": "OnDemand bcs.b2.3xlarge img-ubuntu-vpc", | |||
| "multiqc.docker": "registry-vpc.cn-shanghai.aliyuncs.com/pgx-docker-registry/multiqc:v1.8", | |||
| "qualimapBAMqc.docker": "registry.cn-shanghai.aliyuncs.com/pgx-docker-registry/qualimap:2.0.0", | |||
| "qualimapBAMqc.cluster_config": "OnDemand bcs.a2.7xlarge img-ubuntu-vpc", | |||
| "fastqscreen.disk_size": "100", | |||
| "multiqc.disk_size": "100", | |||
| "ref_dir": "oss://chinese-quartet/quartet-storage-data/reference_data/", | |||
| "trim_front1": "0", | |||
| "trim_tail1": "0", | |||
| "max_len1": "0", | |||
| "trim_front2": "0", | |||
| "trim_tail2": "0", | |||
| "max_len2": "0", | |||
| "adapter_sequence": "AGATCGGAAGAGCACACGTCTGAACTCCAGTCA", | |||
| "adapter_sequence_r2": "AGATCGGAAGAGCGTCGTGTAGGGAAAGAGTGT", | |||
| "disable_adapter_trimming": "0", | |||
| "length_required": "50", | |||
| "UMI": "0", | |||
| "umi_len": "0", | |||
| "umi_loc": "umi_loc", | |||
| "length_required": "20", | |||
| "qualified_quality_phred": "20", | |||
| "disable_quality_filtering": "0" | |||
| } | |||
+ 47
- 0
inputs
Zobrazit soubor
| @@ -0,0 +1,47 @@ | |||
| { | |||
| "{{ project_name }}.inputSamplesFile": "{{ inputSamplesFile }}", | |||
| "{{ project_name }}.idx": "{{ idx }}", | |||
| "{{ project_name }}.gtf": "{{ gtf }}", | |||
| "{{ project_name }}.idx_prefix": "{{ idx_prefix }}", | |||
| "{{ project_name }}.hisat2.docker": "{{ hisat2.docker }}", | |||
| "{{ project_name }}.hisat2.cluster": "{{ hisat2.cluster }}", | |||
| "{{ project_name }}.samtools.docker": "{{ samtools.docker }}", | |||
| "{{ project_name }}.samtools.cluster": "{{ samtools.cluster }}", | |||
| "{{ project_name }}.stringtie.docker": "{{ stringtie.docker }}", | |||
| "{{ project_name }}.stringtie.cluster": "{{ stringtie.cluster }}", | |||
| "{{ project_name }}.fastp.docker": "{{ fastp.docker }}", | |||
| "{{ project_name }}.fastp.cluster": "{{ fastp.cluster }}", | |||
| "{{ project_name }}.fasta": "{{ fasta }}", | |||
| "{{ project_name }}.fastqc.disk_size": "{{ fastqc.disk_size }}", | |||
| "{{ project_name }}.gtf": "{{ gtf }}", | |||
| "{{ project_name }}.fastqscreen.cluster_config": "{{ fastqscreen.cluster_config }}", | |||
| "{{ project_name }}.fastqc.cluster_config": "{{ fastqc.cluster_config }}", | |||
| "{{ project_name }}.qualimapBAMqc.disk_size": "{{ qualimapBAMqc.disk_size }}", | |||
| "{{ project_name }}.fastqc.docker": "{{ fastqc.docker }}", | |||
| "{{ project_name }}.fastqscreen.docker": "{{ fastqscreen.docker }}", | |||
| "{{ project_name }}.screen_ref_dir": "{{ screen_ref_dir }}", | |||
| "{{ project_name }}.fastq_screen_conf": "{{ fastq_screen_conf }}", | |||
| "{{ project_name }}.multiqc.cluster_config": "{{ multiqc.cluster_config }}", | |||
| "{{ project_name }}.multiqc.docker": "{{ multiqc.docker }}", | |||
| "{{ project_name }}.qualimapBAMqc.docker": "{{ qualimapBAMqc.docker }}", | |||
| "{{ project_name }}.qualimapBAMqc.cluster_config": "{{ qualimapBAMqc.cluster_config }}", | |||
| "{{ project_name }}.fastqscreen.disk_size": "{{ fastqscreen.disk_size }}", | |||
| "{{ project_name }}.multiqc.disk_size": "{{ multiqc.disk_size }}", | |||
| "{{ project_name }}.ref_dir": "{{ ref_dir }}", | |||
| "{{ project_name }}.trim_front1": "{{ trim_front1 }}", | |||
| "{{ project_name }}.trim_tail1": "{{ trim_tail1 }}", | |||
| "{{ project_name }}.max_len1": "{{ max_len1 }}", | |||
| "{{ project_name }}.trim_front2": "{{ trim_front2 }}", | |||
| "{{ project_name }}.trim_tail2": "{{ trim_tail2 }}", | |||
| "{{ project_name }}.max_len2": "{{ max_len2 }}", | |||
| "{{ project_name }}.adapter_sequence": "{{ adapter_sequence }}", | |||
| "{{ project_name }}.adapter_sequence_r2": "{{ adapter_sequence_r2 }}", | |||
| "{{ project_name }}.disable_adapter_trimming": "{{ disable_adapter_trimming }}", | |||
| "{{ project_name }}.length_required": "{{ length_required }}", | |||
| "{{ project_name }}.UMI": "{{ UMI }}", | |||
| "{{ project_name }}.umi_loc": "{{ umi_loc }}", | |||
| "{{ project_name }}.umi_len": "{{ umi_len }}", | |||
| "{{ project_name }}.length_required": "{{ length_required }}", | |||
| "{{ project_name }}.qualified_quality_phred": "{{ qualified_quality_phred }}", | |||
| "{{ disable_quality_filtering }}.umi_loc": "{{ disable_quality_filtering }}" | |||
| } | |||
binární
tasks/.DS_Store
Zobrazit soubor
+ 67
- 0
tasks/fastp.wdl
Zobrazit soubor
| @@ -0,0 +1,67 @@ | |||
| task fastp { | |||
| String sample_id | |||
| File read1 | |||
| File read2 | |||
| String adapter_sequence | |||
| String adapter_sequence_r2 | |||
| String docker | |||
| String cluster | |||
| String umi_loc | |||
| Int trim_front1 | |||
| Int trim_tail1 | |||
| Int max_len1 | |||
| Int trim_front2 | |||
| Int trim_tail2 | |||
| Int max_len2 | |||
| Int disable_adapter_trimming | |||
| Int length_required | |||
| Int umi_len | |||
| Int UMI | |||
| Int qualified_quality_phred | |||
| Int length_required | |||
| Int disable_quality_filtering | |||
| command <<< | |||
| ##1.Disable_quality_filtering | |||
| if [ ${disable_quality_filtering} -gt 0 ] | |||
| then | |||
| mv ${read1} {sample_id}_R1.fastq.tmp1.gz | |||
| mv ${read2} {sample_id}_R2.fastq.tmp1.gz | |||
| else | |||
| fastp --thread 4 --trim_front1 ${trim_front1} --trim_tail1 ${trim_tail1} --max_len1 ${max_len1} --trim_front2 ${trim_front2} --trim_tail2 ${trim_tail2} --max_len2 ${max_len2} -i ${read1} -I ${read2} -o ${sample_id}_R1.fastq.tmp1.gz -O ${sample_id}_R2.fastq.tmp1.gz -j ${sample_id}.json -h ${sample_id}.html | |||
| ##2.UMI | |||
| if [ ${UMI} -gt 0 ] | |||
| then | |||
| mv ${sample_id}_R1.fastq.tmp1.gz ${sample_id}_R1.fastq.tmp2.gz | |||
| mv ${sample_id}_R2.fastq.tmp1.gz ${sample_id}_R2.fastq.tmp2.gz | |||
| else | |||
| fastp --thread 4 -U --umi_loc=${umi_loc} --umi_len=${umi_len} --trim_front1 ${trim_front1} --trim_tail1 ${trim_tail1} --max_len1 ${max_len1} --trim_front2 ${trim_front2} --trim_tail2 ${trim_tail2} --max_len2 ${max_len2} -i ${sample_id}_R1.fastq.tmp1.gz -I ${sample_id}_R2.fastq.tmp1.gz -o ${sample_id}_R1.fastq.tmp2.gz -O ${sample_id}_R2.fastq.tmp2.gz -j ${sample_id}.json -h ${sample_id}.html | |||
| ##3.Trim | |||
| if [ ${disable_adapter_trimming} -gt 0 ] | |||
| then | |||
| fastp --thread 4 -l ${length_required} -q ${qualified_quality_phred} -u ${length_required} --adapter_sequence ${adapter_sequence} --adapter_sequence_r2 ${adapter_sequence_r2} --detect_adapter_for_pe --trim_front1 ${trim_front1} --trim_tail1 ${trim_tail1} --max_len1 ${max_len1} --trim_front2 ${trim_front2} --trim_tail2 ${trim_tail2} --max_len2 ${max_len2} -i ${sample_id}_R1.fastq.tmp2.gz -I ${sample_id}_R2.fastq.tmp2.gz -o ${sample_id}_R1.fastq.gz -O ${sample_id}_R2.fastq.gz -j ${sample_id}.json -h ${sample_id}.html | |||
| else | |||
| mv ${sample_id}_R1.fastq.tmp2.gz ${sample_id}_R1.fastq.gz | |||
| mv ${sample_id}_R2.fastq.tmp2.gz ${sample_id}_R2.fastq.gz | |||
| fi | |||
| >>> | |||
| runtime { | |||
| docker: docker | |||
| cluster: cluster | |||
| systemDisk: "cloud_ssd 40" | |||
| dataDisk: "cloud_ssd 200 /cromwell_root/" | |||
| } | |||
| output { | |||
| File json = "${sample_id}.json" | |||
| File report = "${sample_id}.html" | |||
| File Trim_R1 = "${sample_id}_R1.fastq.gz" | |||
| File Trim_R2 = "${sample_id}_R2.fastq.gz" | |||
| } | |||
| } | |||
+ 26
- 0
workflow.wdl
Zobrazit soubor
| @@ -0,0 +1,26 @@ | |||
| import "./tasks/fastp.wdl" as fastp | |||
| workflow {{ project_name }} { | |||
| File inputSamplesFile | |||
| Array[Array[File]] inputSamples = read_tsv(inputSamplesFile) | |||
| String fasta | |||
| String idx_prefix | |||
| File screen_ref_dir | |||
| File fastq_screen_conf | |||
| File idx | |||
| File gtf | |||
| File ref_dir | |||
| scatter (quartet in inputSamples){ | |||
| call fastp.fastp as fastp { | |||
| input: | |||
| sample_id=quartet[2], | |||
| read1=quartet[0], | |||
| read2=quartet[1] | |||
| } | |||
| } | |||
| } | |||
Načítá se…