Would it be a good idea to read big data (query that returns billions of results) using an indexer(elastic search/solr) overtop of cassandra? Or would it be more preformant to ask cassandra directly? I am only wondering about reading data, not about updating and deleting.
Should indexers only be used for searches that return smaller sets of data?
I guess in a nutshell my question is when is it better to query an indexer over a big data database - more specifically cassandra when the query narrows down the potential reaults? Does this mean if the query returns a wide range of results that it aould be better to query cassandra directly?
Do you mean, reading the data, indexing it, then reading it again from the index? Then definitely reading once would be better. i.e. asking Cassandra directly. Unless, you want to use ElasticSearch linguistic capabilities. If your query doesn't account for natural language, then go with reading directly from Cassandra.
Yes, search engines are optimized for this types of queries. Search engines solve 2 main issues: 1. Returning relevant results various types of filtering and natural languages capabilities. e.g. searching for "USA" and finding "United States of America" 2. Scoring the results in such a way that the most relevant (by some ranking function such as TD-IDF or BM25
When a search query executed only the id's of the document are returned and are assembled from the store part of the index, which is the most expensive search engine operation (besides optimizing perhaps :P ).
In a nutshell, if you can narrow the results from Cassandra in the same way as ElasitcSearch query, then you don't need ElasticSearch.