LevelDB key,value from csv

Question

I've huge csv file database of ~5M rows having below fields

start_ip,end_ip,country,city,lat,long 

I am storing these in LevelDB using start_ip as the key and rest as the value.

How can I retrieve records for keys where

( ip_key > start_ip and ip_key < end_ip )

Any alternative solution.

Answer 1

I assume that your keys are the hash values of the IP and the hashes are 64-bit `unsigned' integers, but if that's not the case then just modify the code below to account for the proper keys.

void MyClass::ReadRecordRange(const uint64 startRange, const uint64 endRange)
{
    // Get the start slice and the end slice
    leveldb::Slice startSlice(static_cast<const char*>(static_cast<const void*>(&startRange)), sizeof(startRange));
    leveldb::Slice endSlice(static_cast<const char*>(static_cast<const void*>(&endRange)), sizeof(endRange));

    // Get a database iterator
    shared_ptr<leveldb::Iterator> dbIter(_database->NewIterator(leveldb::ReadOptions()));

    // Possible optimization suggested by Google engineers 
    // for critical loops. Reduces memory thrash.
    for(dbIter->Seek(startSlice); dbIter->Valid() && _options.comparator->Compare(dbIter->key(), endSlice)<=0); dbIter->Next())
    {
        // get the key
        dbIter->key().data();

        // get the value
        dbIter->value().data();

        // TODO do whatever you need to do with the key/value you read
    }
}

Note that _options are the same leveldb::Options with which you opened the database instance. You want to use the comparator specified in the options so that the order in which you read the records is the same as the order in the database.

If you're not using boost or tr1, then you can either use something else similar to the shared_ptr or just delete the leveldb::Iterator by yourself. If you don't delete the iterator, then you'll leak memory and get asserts in debug mode.