Yes, we do.
Why? Because to be the best, you have to play the best. You can win all of your games, but if it is against third rate teams, does it really mean anything?
I believe we call it Strength of Schedule. It makes a major difference on which teams are selected for the CFP.
You want to be a part of the strongest conference (because that is the teams you will play the most) so your strength of schedule makes your team more attractive to the CFP committee.
This is also where the round robin tournament would help decide the strongest conference and the strongest strength of schedule.