But with that in mind, Simon Cowell and his big wig producers are reportedly considering revealing the phone votes every week in order to show viewers how their favourites are doing.
The system is already used on X Factor US, where the ranking of contestants is revealed every week, opposed to just showing who has and hasn't made it through like they've always done over here.
As well as encouraging people to vote, it is thought that it would help viewers to see when their favourite acts are heading for the danger zone.
"If the viewers knew popular acts like Ella and James Arthur were pulling in low numbers of votes, it would encourage them to pick up the phone," a show insider apparently told The Sun.
"Revealing the voting numbers each week has really brought a new dynamic to the US show. Simon is looking to introduce it in the UK next year as part of a major shake-up."
Although it doesn't sound like it's going to happen this series, it might avoid the type of situation like last week, when we were fairly close to breaking things off with our telly.
What do you think about all this?