Coldfusion 8 - Problems with indexing large data using verity

Question

I am currently running coldfusion 8 with verity running on a K2 server. I am using a query to index several different columns with my table using cfindex. One of the columns is a large varchar type.

It seems that when the data is being indexed only the first 30KB is being stored, resulting in no results being brought back if I search for anything after that. I tried moving several different phrases and words further up in the data, within the 30KB and the results then appear.

I then carried out more verity tests using the browse command in the command prompt to see whats actually in the collection.

i.e. Coldfusion8\verity\collections\\parts browse 0000001.ddd

I found out that the body being indexed (CF_BODY) never exceeds the size of 32000.

Can anyone tell me if there is a fixed index size per document for verity?

Many thanks,

Richard

Answer 1

Punch line

Version 6 has operator limits:

• up to 32 764 children in one "topic" for ANY operator
• up to 64 children for NEAR

Exceeding these values doesn't necessarily give error message. When you search, you're certain you don't exceed them?

Source

Verity documentation, Appendix B: Query limits says there are two limitations: search time and operator's. Quote below is whole section telling about the latter, straight from the book.

Verity Query Language and Topic Guide, Version 6.0:

Note the following limits on the use of operators:

There can be a maximum of 32,764 children for the ANY operator. If a topic exceeds this limit, the search engine does not always return an error message.

The NEAR operator can evaluate only 64 children. If a topic exceeds this limit, the search engine does not return an error message.

For example, assume you have created a large topic that uses the ACCRUE operator with 8365 children. This topic exceeds the 1024 limit for any ACCRUE-class topic and the 16000/3 limit for the total number of nodes. In this case, you cannot substitute ANY for ACCRUE, because that would cause the topic to exceed the 8,000 limit for the maximum number of children for the ANY operator. Instead, you can build a deeper tree structure by grouping topics and creating some named subnodes.