Not sure I’d put USC in top five. Private school. Pacific time zone…. Competition from pro-football for attention (again). And primarily there are vastly better things to do in SoCal on autumn afternoons. I recall staying over on a business trip in Manhattan Beach. We went to a bar for lunch and a UCLA game was on the TV and I thought oh yeah, it’s Saturday. I’m not sure if I’d even watch the Huskers (although I did do so in Santa Barbara, once).
I lived in Philadelphia for 12 years. I was much more likely to see a Big 12 game on TV than the Pac10/12.
When it comes to college sports, the West Coast may as well be its own country. It's just too far removed from the bulk of the U.S. population, which functions in either the Central or East time zones.
I also lived in Southern California for 3-4 years. It's just not a rabid sports region (as you say, too many other forms of entertainment). There's lots of people and lots of money -- thus sports exist. But the majority of fan bases only care if they are winning. Otherwise, there's better options to spend time. USC wouldn't even be considered if not for the 4-5 years under Pete Carroll which brought them national attention that they hadn't seen since the 70s/early-80s.