Not following a mazhab

People in my area follow different imams and they claim that their act is correct and perfect than the other. They also say to me that you follow one imam only, reply to them that I will follow what is there is in Quran and sunnah (hadith-sahih + sanan+ other books-with references) irrespective of following one imam and making sects (as said in Quran" Vo ati ullaha Vo ati ur rasula") and they don't fluctuate w.r.t time as it is followed by prophet in times. Am I correct in this case of following Quran and hadith(also includes imams +sahaba+mufti+fuqaha) or following a single imam?