How to create Hotjar like application?

Question

Is there any tutorial or document explaining to create the hotjar like application ? Can anyone provide any insights in this? i want to create the application which analyses user behaviour.. like

• upto how much percentage of the page the user has scrolled

• in which part of the DOM has the user clicked

• and create report using heatmap.js for just one of my static site/page

Ive made the reports using the static datas and heatmaps Now i just want to track the user activities like scroll points, mouse hover/click points which can be differnet as for different screen sizes or devices... Is there any apis or js framework to help?

Answer 1

You won't find a single tutorial on this as you won't find a "Build a web search engine yourself" tutorials. Website user tracking is a very complex topic. Developing such a solution will require huge effort and investments. It would require expensive infrastructure as well (servers to collect data from the users and process them).

Additionaly there are some risks and problems to it as well. User privacy is a hot topic now because of questionable morality of user tracking. Web users awareness grows all the time and a lot of users choose to opt-out of web-tracking. The industry follows and expands users' possibilities to do that (both technical and legal).

If you still want to proceed - learn as much as you can about web user tracking. Search for "user tracking methods", "user tracking techniques", "web analitics" buzz-phrases.

Then when it comes to implementation:

1. In the browser (client-side)

   • Implement individual users identification in order to classify their actions across website/multiple websites (fingerprinting). In fact this might require some work on the server too.
   • Record as much of user's interactions as possible - clicking, scrolling, dragging, keypresses, navigation between pages, typing in the inputs (watch out for sensitive data - passwords, addresses) etc. This data will be the base for user behavior analysis on the server later on. Also, combined with the captured state of the DOM (initial and later mutations), this allows us to create "recordings" of user browsing sessions. I've put the word "recordings" in quotes because, contrary to what many think, these videos are usually not created by recording user's screen. This would be far too compilcated and bandwith-heavy. Instead they are composed from the collected pieces of data I mentioned earlier. Read more about this (IMHO the most interesting) topic in this answer to "How does HotJar generate their recordings?".
   • Implement sending the above information SECURELY to the server for analysis.
   • Again: make your solution SECURE. The data you will be collecting is highly sensitive and shouldn't fall into the wrong hands for your and site users' sake. You'll probably have to fund a Bug Bounty program (I warned you it'd be expensive!) like HotJar did.
   • Figure out a way to inject your tracking application into the website's code (in all popular browsers - don't forget the legacy ones too!). In most cases it requires the site owner to put a small piece of JS on every page of their website. Read Erik Näslund's (HotJar architect) answer to "Why do websites like Hotjar and Google Analytics use complex tracking code instead of just a tag?" to find out more about how this script looks like and why.

2. On the server (the relatively easy part)

   • Implement data processing and produce reports - heatmaps, session recordings, clickstreams.

I did a very simple POC covering some of the above-mentioned client-side stuff some time ago. It's a simple script that watches DOM changes and user events (some of them) and logs them to the console when injected to a web page. In a full-blown solution this would be sent to the server for processing (instead of being written to the console).

These recorded events (DOM changes along with timestamps and user input/events) could be used to reliably reproduce what the users see in the browser window and how they interact with it.