hadto

it's pretty stupid,
search the neighborhood of the last know location
find the pixel with the closest match to the target color
search a smaller area with the found pixel as a center
find the rectangular boundaries of the color area according to a threshold
make the final location the center of the rectangle
it worked a lot better when I was testing it with a paper set