Constitution doesn't say someones president until a mid-term election. Nor does it say a mid-term election is by its very nature a "referendum on the president". You know why? Because if the founding fathers wanted a referendum on the president half way through his term, they would have proposed that. This is just a way for the legislature to subvert power away from the Executive, a clear violation of checks and balances.