In classical Q-learning, the objective is to maximize the sum of discounted rewards through iteratively using the Bellman equation as an update, in an attempt to estimate the action value function of the optimal policy. In this paper, we extend the well-established loss by introducing the hindsight factor, an additional loss which integrates the historic temporal difference in action-value as part of the reward. The effect of this modification is examined in a deterministic continuous state space function estimation problem, resulting in an evident reduction in overestimation and improved stability. The underlying effect of the hindsight factor is modeled as an adaptive learning rate which is adjusted based on the actionvalue. The proposed method outperfoms variations of Q-learning, with an overall higher average reward and lower action values, which supports the deterministic evaluation, and proves that the hindsight factor contributes to lower overestimation errors.