Downloading multiple fasta files from ncbi

Question

I'm trying to download all fasta files associated with one organism from ncbi.

I tried wget -r -l3 -A "*.fna.gz" ftp://ftp.ncbi.nlm.nih.gov/genomes/refseq/bacteria/Microcystis_aeruginosa/ to get all files ending in .fna.gz from the third level down, but then it just rejects everything with the following output:

Removed “ftp.ncbi.nlm.nih.gov/genomes/refseq/bacteria/Microcystis_aeruginosa/latest_assembly_versions/.listing”. Rejecting “GCF_000010625.1_ASM1062v1”. Rejecting “GCF_000307995.1_ASM30799v2”. Rejecting “GCF_000312165.1_ASM31216v1”. Rejecting “GCF_000312185.1_ASM31218v1”. Rejecting “GCF_000312205.1_ASM31220v1”. Rejecting “GCF_000312225.1_ASM31222v1”. Rejecting “GCF_000312245.1_ASM31224v1”. Rejecting “GCF_000312265.1_ASM31226v1”. Rejecting “GCF_000312285.1_ASM31228v1”. Rejecting “GCF_000312725.1_ASM31272v1”. Rejecting “GCF_000330925.1_MicAerT1.0”. Rejecting “GCF_000332585.1_MicAerD1.0”. Rejecting “GCF_000412595.1_spc777-v1”. Rejecting “GCF_000599945.1_Mic70051.0”. Rejecting “GCF_000787675.1_ASM78767v1”. Rejecting “GCF_000981785.1_ASM98178v1”.

Any ideas on why it's rejecting these directories? Thanks for your help.

Answer 1

Not exactly sure why it's rejecting your request, but when I was still doing this type of thing, I found that if I don't download queries in smaller batches, the NCBI server timed me out and blocked my IP for a while before I could download again. This doesn't seem to be the same problem that your seeing, but maybe this script might get the same thing done. Let me know if this helps.

#!/usr/bin/env python

from Bio import Entrez

search_term = raw_input("Organism name: ")

Entrez.email = "[email protected]"   # required by NCBI
search_handle = Entrez.esearch(db="nucleotide", term=search_term, usehistory="y")
search_results = Entrez.read(search_handle)
search_handle.close()

gi_list = search_results["IdList"]
count = int(search_results["Count"])
webenv = search_results["WebEnv"]
query_key = search_results["QueryKey"]

batch_size = 5    # download sequences in batches so NCBI doesn't time you out

with open("ALL_SEQ.fasta", "w") as out_handle:
    for start in range(0, count, batch_size):
        end = min(count, start+batch_size)
        print "Going to download record %i to %i" % (start+1, end)
        fetch_handle = Entrez.efetch(db="nucleotide", rettype="fasta", retmode="text",retstart=start, retmax=batch_size, webenv=webenv, query_key=query_key)
        data = fetch_handle.read()
        fetch_handle.close()
        out_handle.write(data)

print ("\nDownload completed")

Answer 2

I found a perl script that gets me close to accomplishing this task from here . Unfortunately, this script is just returning the ID's of the genomes, and not the actual sequences.

For example, the head of my output is:

gi|425458296|ref|NZ_CAIN00000000.1|NZ_CAIN01000000 Microcystis aeruginosa PCC 9808, whole genome shotgun sequencing project

gi|425448636|ref|NZ_CAIK00000000.1|NZ_CAIK01000000 Microcystis aeruginosa PCC 7941, whole genome shotgun sequencing project

Any perl users know what's going on?

use strict;
use LWP::Simple;
my ($name, $outname, $url, $xml, $out, $count, $query_key, $webenv, $ids);
my @genomeId;
my $base = 'http://eutils.ncbi.nlm.nih.gov/entrez/eutils/';
my $limit = 'wgs[prop]+AND+srcdb+refseq[prop])';
my @species = ('Microcystis aeruginosa');

foreach my $s (@species) {
  undef @genomeId;
  $query_key = $webenv = '';
  $s =~ s/ /+/g;

  # ESearch
  $url = $base . "esearch.fcgi?db=genome&term=$s";
  $xml = get($url);
  $count = $1 if ($xml =~ /<Count>(\d+)<\/Count>/);

  if ($count > 30) {
    $url = $base . "esearch.fcgi?db=genome&term=$s&retmax=$count";
    $xml = get($url);
  }

  while ($xml =~ /<Id>(\d+?)<\/Id>/gs) {
    push(@genomeId, $1);
  }

  $ids = join(',', @genomeId);

  # ELink
  $url = $base . "elink.fcgidbfrom=genome&db=nuccore&cmd=neighbor_history&id=$ids&term=$limit";
  $xml = get($url);
  $query_key = $1 if ($xml =~ /<QueryKey>(\d+)<\/QueryKey>/);
  $webenv = $1 if ($xml =~ /<WebEnv>(\S+)<\/WebEnv>/);

  # EFetch
  $url = $base . "efetch.fcgidb=nuccore&query_key=$query_key&WebEnv=$webenv&rettype=fasta&retmode=text";
  $out = get($url);

  open (OUT, ">$s.fna");
  close OUT; 
}