Large Volume Data Searches

Question

I have been asked to look into developing a solution for logging for a large organisation. I was thinking having some sort of service that has a msg queue of logs and inserts them into some sort of database but there is lie's the rub.

Oracle and SQLServer don't seem to be able to handle very large amounts of data and performing quick searches on them. They seem to be more focused on having large number of transactions doing small things than small number of transactions doing big searches. Would you be able to give me a nudge in the correct direction on some software / databases we could implement for this specific purpose?

Answer 1

The answer is not Relational DB's.

Look into http://hadoop.apache.org/

Answer 2

Oracle and SQL Server are capable of performing extremely fast searches on huge amounts of data - you just need to ensure your database design and the methods you use to access that data are designed to be as performant as possible.