Reproducible OOM Kill when using the bioinformatics package "GetBaseCountsMultiSample" on WLS2 with Ubuntu 22.04 and native Ubuntu

Question

I just started out running some bioinformatic scripts for variant calling from a friend of mine and I am still very new to this. Unfortunately, I get hard stuck when I get to the part of the script where I have to use GetBaseCountsMultiSample (https://github.com/zengzheng123/GetBaseCountsMultiSample) to calculate the base counts in the tumor and normal BAM file for all the sites in my VCF file. Every time I use the tool it terminates after loading the second BAM file and the output in the terminal is "killed".

I used WSL2 with Ubuntu 22.04 on Windows 10 with 80GB RAM and my command is:

cd ~/Output_Strelka
~/packages/GetBaseCountsMultiSample-1.2.3/GetBaseCountsMultiSample \
--fasta ~/Output_Strelka/MoCaSeq_ref/GRCh38.p12.reduced.fa --maq 1 \
--bam Tumor:M15/results/bam/M15RNA.Tumor.bam \
--bam Normal:M15/results/bam/M15.Normal.bam \
--vcf M15/results/rescued/M15.StrelkaRNA.vcf \
--output M15/results/rescued/M15.StrelkaRNA.M15.StrelkaRNA.vcf \
--thread 1

This is the output in the terminal:

[INFO] Loading reference sequence: /home/chiv/Output_Strelka/MoCaSeq_ref/GRCh38.p12.reduced.fa
[INFO] Finished loading reference sequence
[INFO] Loading variants file: M15/results/rescued/M15.StrelkaRNA.vcf
[INFO] 347217 variants has been loaded from file: M15/results/rescued/M15.StrelkaRNA.vcf
[INFO] Sorting variants
[INFO] Indexing variants
[INFO] Processing bam file: M15/results/bam/M15.Normal.bam
[INFO] Processing bam file: M15/results/bam/M15RNA.Tumor.bam
Killed

Here are some of the error messages that I retrieved from the log file:

/var/log/kern.log:Jul 30 20:33:43 DESKTOP-9R7FHMK kernel: [34890.795897] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=GetBaseCountsMu,pid=100281,uid=1000
/var/log/kern.log:Jul 30 20:33:43 DESKTOP-9R7FHMK kernel: [34890.795920] Out of memory: Killed process 100281 (GetBaseCountsMu) total-vm:49172696kB, anon-rss:39340272kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:95440kB oom_score_adj:0
/var/log/kern.log:Jul 30 20:36:38 DESKTOP-9R7FHMK kernel: [35066.182716] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=GetBaseCountsMu,pid=100477,uid=1000
/var/log/kern.log:Jul 30 20:36:38 DESKTOP-9R7FHMK kernel: [35066.182726] Out of memory: Killed process 100477 (GetBaseCountsMu) total-vm:49466188kB, anon-rss:39645536kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:95896kB oom_score_adj:0
/var/log/kern.log:Jul 30 21:15:20 DESKTOP-9R7FHMK kernel: [37388.345857] GetBaseCountsMu invoked oom-killer: gfp_mask=0x1100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
/var/log/kern.log:Jul 30 21:15:20 DESKTOP-9R7FHMK kernel: [37388.345877]  oom_kill_process.cold+0xb/0x10
/var/log/kern.log:Jul 30 21:17:17 DESKTOP-9R7FHMK kernel: [37505.417357] GetBaseCountsMu invoked oom-killer: gfp_mask=0x1100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
/var/log/kern.log:Jul 30 21:17:17 DESKTOP-9R7FHMK kernel: [37505.417377]  oom_kill_process.cold+0xb/0x10

After loading the second BAM file my ram goes from 20gb to 73gb usage in a matter of 30s, remains there for 3 minutes and then the task gets killed. CPU usage is about 25% the entire time.

I asked my friend that I got the script from but his suggestions did not solve the problem. He tried all my input files on his machine to rule out that this is the cause and on his system it works. Here is what I tried to do to fix the problem:

1. I tried to run the task in a docker container from my friend but same problem
2. I tried with Ubuntu 18.04 and even WSL1 as well as different g++ compilers to install the package
3. I used the --max_block_size and --max_block_dist options of GetBaseCountsMultiSample
4. I tried to disable OOM Kill and use sudo sysctl vm.overcommit_memory=2, but with the second option it stops with the error "terminate called after throwing an instance of 'std::bad_alloc' what(): std::bad_alloc Aborted"
5. I installed a native Ubuntu 22.04 LTS in dual boot with the windows boot manager to rule out that the problem comes from WSL but same error
6. In a hail merry I removed my 16GB RAM Kit to have only my 64 GB Kit with two sticks but same issue
7. I ran the same code on another computer at work with WSL2 Ubuntu 22.04 and 64GB Ram but same error
8. I also used the newest version of GetBaseCountsMultiSample (and not just the version my friend uses) but to no avail

What somewhat worked: I reduced the file size of my BAM files with:

samtools view -h M15.Normal.bam | head -n 1000 | samtools view -bS - > M15.Normal.small.bam
samtools view -h M15RNA.Tumor.bam | head -n 1000 | samtools view -bS - > M15RNA.Tumor.small.bam

--> this actually worked without issue with GetBaseCountsMultiSample

I then fractioned my BAM files into individual chromosomes. Chromosome 1 worked (size 230MB) but Chromosome 2 (size 408 MB) already gave the OOM Kill error. Even fractioning Chr2 further in files below 200MB did not help.

As I said earlier, my input files work on my friends linux system with GetBaseCountsMultiSample. As far as I understand, dual booting linux should solve the problem if it is windows-related or could that still be an issue. I am extremely grateful for any advice.

Thanks everyone for their time and apologies for the long post. I wanted to give as much information about the error and my attempts.

Answer 1

Thank you Patrick H. for your helpful comments. I managed to fix it now with the advice from my friend:

I created a swap file with 300+ GB and monitored the ram consumption during usage. Apparently the program uses a ridiculous amount of ram independent of the size of the input files. When it ramps up it uses all my 80GB of installed ram and 200GB of my swap file (this is the same for small bam files and full-sized bam files). But with this it manages to complete the task.

Thanks again and I hope that helps others who run into similar issues.