How to access element(having a unique identifier) in a vector using a map in constant time?

Question

I am having a class Person, which has a vector of another class Badge(with a unique Id). To access all the elements in the vector, I can iterate over the elements, which takes O(n) time, n being the size of the vector. I wanted to do it in constant time, so I created a map, with Badge ids as keys and pointer to Badge as values. Here, the pointer points to an element of the vector. More specifically:

class Badge
{
public:
    Badge(string, int);
    const int& getBadgeId() const;
    const string& getBadgeName() const;
    int getRoyaltyPoints() const;
    void setRoyaltyPoints(int royaltyPoints);

private:
    int badgeId;
    string badgeName;
    int royaltyPoints;
    int getUniqueBadgeId();
};

class Person
{
public:
    Person(string);
    const unordered_map<int, Badge*>& getBadgeMap() const;
    const vector<Badge>& getBadges() const;
    const string& getName() const;
    const string& getPersonId() const;
    void addBadge(Badge&);
    void updateRoyalty(Badge&, int);

private:
    string personId;
    string name;
    vector<Badge> badges;
    unordered_map<int, Badge*> badgeMap;
    string getUniquePersonId();
};

The functions involved:

/* Adds a badge to the badges collection and updates the map */
void Person::addBadge(Badge &badge)
{
    this->badges.push_back(badge);
    this->badgeMap[badge.getBadgeId()] = &this->badges.back();
    //this->badgeMap[badge.getBadgeId()] = &this->badges[this->badges.size() - 1];
}

/* Updates the royalty points in a badge */
void Person::updateRoyalty(Badge &badge, int newValue)
{
    Badge* bd = this->badgeMap[badge.getBadgeId()];
    bd->setRoyaltyPoints(newValue);
}

The complete code can be found here: https://www.ideone.com/hjUT6e

On running the function, most of the times, a runtime error is encountered. Sometimes, the code works as expected, and other times, the updateRoyalty() function has no effect on the royalty points, which means it's as good as the function wasn't called at all.

Is there any way to address this weird behavior? I can think of two ideas:

1) Use std::find() function to search for the element in the vector and overload the = operator. But it takes O(n) time as well

2) In place of Badge *, use Badge as the value in the unordered_map. There are two problems here. First, Badge has no constructor called Badge(), it has a parameterized constructor. So the compiler complains. Second, I need to overload [] operator, write my own hash function. I am fairly new to C++, so trying to keep things simple here(if possible, language independent).

Kindly let me know how can I achieve this

Thank you

EDIT: The question has been marked duplicate and closed. The duplicate question What happen to pointers when vectors need more memory and realocate memory? is a totally different question. It's related to resizing the vector memory, whereas this is about storing pointer(as a value in map) to a vector element. I request to reopen this question.