Generating heatmap layer for milions of points

Question

I'm using heatmap layer from Google Maps to display a heatmap, however, I now have too many points and it stopped working, because browser cannot handle it anymore. I've found that they provide Fusion Tables, but they're also limited: to 100k rows, which is way too low. I need to render heatmap of milions or maybe even more points. I'd be perfect if my server can have some PHP script to render a heatmap, for example, once a day. And then a client from js will just download a preloaded heatmap (on the map like google maps, but may be different map too). Is this possible with some existing technology (can be commercial)?

Answer 1

All you need is to pre-clusterize your points into smaller number of points and pass these groups to Google Maps as if it were your original points. So, if you had 3 nearby points with values 3, 5 and 10, you create one point with value 18 at weighted average of their coordinates. Do it for all your dataset and you will reduce it 3 times. Change 3 to any appropriate value to further reduce your dataset.

The easy way to cluster your data is to use geohashing[1]. Here is nice geohashing library for PHP[2]. You may calculate a set of geohashes of different precision for each of your points only once when adding, and then use desired precision to reduce your dataset using simple GROUP BY. Example (meta):

use Lvht\GeoHash;

class DataPoint extends ActiveRecord {
    public geo_hash_precision4, geo_hash_precision5, geo_hash_precision6;
    public function save() {
        $this->geo_hash_precision4 = GeoHash::encode($this->lat,$this->lon, 0.0001);
        $this->geo_hash_precision5 = GeoHash::encode($this->lat,$this->lon, 0.00001);
        $this->geo_hash_precision6 = GeoHash::encode($this->lat,$this->lon, 0.000001);
        parent::save();
    }
}

class DataSet extends ActiveQuery {
    /**
     * @param int $p Desired precision
     */
    public function getValues($p = 6) {
        $data = $this->rawQuery("SELECT geo_hash_precision{$p}, SUM(value) FROM {$this->table} GROUP BY geo_hash_precision{$p}");
        // Add bounding box WHERE to reduce set for map size
        foreach ($data as $row) {
            list($minLon, $maxLon, $minLat, $maxLat) = GeoHash::decode($row["geo_hash_precision{$p}"]);
            $row['lat'] = ($maxLat - $minLat) / 2;
            $row['lon'] = ($maxLon - $minLon) / 2;
            unset($row["geo_hash_precision{$p}"]);
        }
    }
}

1. https://en.wikipedia.org/wiki/Geohash
2. https://github.com/lvht/geohash

Answer 2

I use heatmap.js for visualization. It's really fast and can handle alot of points. Not sure though if 1.5 mil points will work.

What I can imagine is to use one of the Javascript solutions with node.js to prerender. A quick google search about 'heatmap js nodejs prerender' found me this https://github.com/substack/node-heatmap and this https://mango-is.com/blog/engineering/pre-render-d3-js-charts-at-server-side/