You should have to be good at your job. You should be given a reasonable amount of time to get there and fair warning if you’re falling behind, but you have to be good at your job. Incompetence hurts people other than your boss — Everyone has a customer, after all.
Maybe you struggle simply because you’re in the wrong place. That doesn’t make you a C- or D-player, it just makes you an A-player of baseball who’s found themselves on a football pitch.
But if you just suck? Nobody owes you a sustained living to… exist.
Where you are correct, though, is that a number of the people making these judgements are average themselves and don’t know it because the toxic behaviours that make people terrible managers and leaders are considered desirable due to Jobs- and Bezos-worship. Leadership itself is judged on short-term results, and a “leader” is given a nice bonus and a car if he’s made the CEO look good in the annual earnings call because he basically separated his subordinates from their families.