KCF-tracker reports tracking failure when object is rotated

Question

I'm working on a project where I have to detect and track small cars using colors as feature representation of the object (see below image). For detection, I used histogram back_projection See: https://docs.opencv.org/master/dc/df6/tutorial_py_histogram_backprojection.html

The code for detection is shown below:

def back_projectcam1(histm,hsv_histt,hsvt): #hist: model histogram, hsv_histt: target image 
 histogram, hsvt: hsv image of the target image
 track_track=None
 Mask = cv2.calcBackProject([hsvt],[0,1],histm,[0,180,0,256],1)
 Open=cv2.morphologyEx(Mask,cv2.MORPH_OPEN,kernel1)
 Filter=cv2.filter2D(Open,-1,kernel3,Open)
 
 Close=cv2.morphologyEx(Filter,cv2.MORPH_CLOSE,kernel1)
 _,thresh = cv2.threshold(Close,10,255,cv2.THRESH_BINARY)

 contours =cv2.findContours(Mask, cv2.RETR_EXTERNAL,cv2.CHAIN_APPROX_SIMPLE)
 if contours[2] is not None:
  cntss = imutils.grab_contours(contours)
  cnt=max(cntss, key=cv2.contourArea) 
  x,y,w,h = cv2.boundingRect(cnt)
  track_track=x,y,w,h
 return track_track,thresh

For tracking I used KCF-tracker (Kernalized Correlation Filter).

My algorithm works as follows:

-A detection occurs only at certain frames (every 50 frames) and the resulted bounding box is assigned to the KCF-tracker as the new initial bounding box.

The idea behind this is to make the process faster as when only using detection.

When I run the code every seems works well regarding speed and accuracy:

Except when the car rotates(means drive on the curved railroad), the tracker fails as shown:

I suspect that KCF.tracker fails when the object being traced rotates because it's simply a correlation, not a convolution filter (kernel is not rotated!). However, this is just an assumption and I might be wrong.

So to solve this, I simply must make the time between every two detection smaller (eg. every 20 frames instead of 50. However, if I do so, the speed of the process drops!

So my question is:

How can I efficiently tell my algorithm to increase the number of detections only when the car drives on the curve?

I've tried to define that statically (but saying that when the car reaches the curve pixels, do more detections), however this was not efficient.