find an entry in hazelcast list quickly

Question

I have a hazelcast Ilist,The student class contains 5 properties like(id,name,address,number,school).Now there are 10k records in the list,how can I find the student whose name is tony and number is 001 quickly except for for loop.I hnow if it is a Imap I can use predicate to filter ,but it is a list ,I didn't find a predicate for Ilist.Any help ,thanks a lot.

Answer 1

Unfortunately there is no way to do this with some kind of a predicate or other magic. You have to do a loop. To speed it up, however, you should run this search on the member that contains the list. Partitioning is defined by the name of the list though. You can basically write yourself a small "query engine" to utilize the Hazelcast predicates on top of a list.

I created a basic example, you can most probably optimize it though.

A simple student class:

public class Student implements Serializable {
    private long id;
    private String name;
    private String address;
    private String number;
    private String school;

    public long getId() { return id; }

    public void setId(long id) { this.id = id; }

    public String getName() { return name; }

    public void setName(String name) { this.name = name; }

    public String getAddress() { return address; }

    public void setAddress(String address) { this.address = address; }

    public String getNumber() { return number; }

    public void setNumber(String number) { this.number = number; }

    public String getSchool() { return school; }

    public void setSchool(String school) { this.school = school; }

    @Override
    public String toString() {
        return "Student{" + "id=" + id
            + ", name='" + name + '\''
            + ", address='" + address + '\''
            + ", number='" + number + '\''
            + ", school='" + school + '\'' + '}';
    }
}

The search executor:

public class StudentSearch {

    private final IExecutorService executorService;

    public StudentSearch(HazelcastInstance hazelcastInstance) {
        this.executorService = 
            hazelcastInstance.getExecutorService("student_search");
    }

    public Student findFirstByNameAndNumber(String listName,
                                            String name,
                                            String number)
            throws Exception {
        Predicate namePredicate = Predicates.equal("name", name);
        Predicate numberPredicate = Predicates.equal("number", number);
        Predicate predicate = Predicates.and(namePredicate, numberPredicate);

        StudentSearchTask task = new StudentSearchTask(listName, predicate);
        Future<Student> future = executorService.submitToKeyOwner(task, listName);
        return future.get();
    }

    private static class StudentSearchTask
            implements Callable<Student>,
                       DataSerializable,
                       HazelcastInstanceAware {

        private HazelcastInstance hazelcastInstance;

        private String listName;
        private Predicate predicate;

        public StudentSearchTask() {
        }

        public StudentSearchTask(String listName, Predicate predicate) {
            this.listName = listName;
            this.predicate = predicate;
        }

        @Override
        public void setHazelcastInstance(HazelcastInstance hazelcastInstance) {
            this.hazelcastInstance = hazelcastInstance;
        }

        @Override
        public Student call() throws Exception {
            IList<Student> list = hazelcastInstance.getList(listName);
            Optional<Map.Entry<String, Student>> first =
                list.stream()
                    .map(this::makeMapEntry)
                    .filter(predicate::apply)
                    .findFirst();

            return first.orElse(makeMapEntry(null)).getValue();
        }

        @Override
        public void writeData(ObjectDataOutput out) throws IOException {
            out.writeUTF(listName);
            out.writeObject(predicate);
        }

        @Override
        public void readData(ObjectDataInput in) throws IOException {
            listName = in.readUTF();
            predicate = in.readObject();
        }

        private Map.Entry<String, Student> makeMapEntry(Student student) {
            return new QueryEntry(listName, student);
        }
    }

    // Used to query the list entries
    private static class QueryEntry
            implements Map.Entry<String, Student>,
                       Extractable {

        private final String key;
        private final Student value;

        private QueryEntry(String key, Student value) {
            this.key = key;
            this.value = value;
        }

        @Override
        public Object getAttributeValue(String attributeName)
                throws QueryException {
            if ("number".equals(attributeName)) {
                return value.getNumber();
            } else if ("name".equals(attributeName)) {
                return value.getName();
            }
            return null;
        }

        @Override
        public AttributeType getAttributeType(String attributeName) 
                throws QueryException {
            return AttributeType.STRING;
        }

        @Override
        public String getKey() {
            return key;
        }

        @Override
        public Student getValue() {
            return value;
        }

        @Override
        public Student setValue(Student value) {
            throw new UnsupportedOperationException();
        }
    }
}

And finally how to run this code:

List<Student> students = hz.getList(listName);
addStudents(students);
StudentSearch search = new StudentSearch(hz);
Student result = search
    .findFirstByNameAndNumber(listName, "Tony", "001");
System.out.println(result);

I hope this helps a bit :)

Answer 2

I can not get what id in the object is for as you need to query id+name (therefore id is not intended to be unique?). Why you store them in a Set if you know you will need to query them (please give more info).

As you pointed out there are not predicates in Set. IMHO this is because entries not associated with a Key can not be indexed. Without possibility to ad an index (Or at leas range-scan on key) the concept of predicates crumble down, as any query will still iterate over the entire set. As far as i can see, you do not have many options:

If you are using set must to have unique entries, don't!

In this case, move it to a map, an use a key as anyone, for example your object id. If there may be id, duplicates, you can make more complex key like id+name or even hash the entire object. Once you have to put a new Object make the key and check if it is already present if so fallback with your custom logic. Map will give you all the Indexes and predicates you can wish for.

From the other hand if for some reasons that are not under your control you Must use set... then you can do it in many ways but I will suggest as follow:

1. Listen on any modification to the Set (Or if is static or consistency is not a concern scan periodically the set

2. Build your custom index

How to build the index:

It really depend upon the performances you want, the RAM impact you can accept and how different queries may be. (Let assume that you only queries are always the same eg "name equals to").

MultiMap<String, String> index
// index.put(name, key)

You structure your index by adding, removing entries on each Set modification, using in your MultiMap the object.name as key and actual Key in the Set as value in the multimap. Once you search for a give name you simply do as follow (pseudo-pseudo code)

MultiMap<String, String> index;
Map<String, your_object_class> your_set;

function getByName(String name)
{
  List<String> name_key_set index.get( name );
  List<your_object_class> out;
  for(String key : name_key_set)
    out.add(index.get(key));
  return out;
}

IMO there is nothing you can call query on a Set ( referring to query as a clever way to retrieve data and not a brute force iteration ) as any such system will require key=>value entries.

With further information we can help you better :)