It's not an either/or scenario. Players AND refs decide games together. Neither happen in a vaccuum. It's not a black or white situation.
The players are accountable for making the appropriate plays. The refs are responsible for making the appropriate calls.
If they were not that significant to games, they wouldn't get graded and assigned to the playoffs based on these ratings.