Boo, you wh*re.
The music mogul - who decided he'd rather get his judge on for the US version of the show - was said to be considering sitting on both judging panels this year, British AND American.
It was even suggested that Simon would use his private plane to jet from one to the other, and that the mentor system - where each judge looks after a group of finalists - would be scrapped so that he would have enough time to take part in two programmes.
However, according to The Sun, Simon denied the reports when asked if he was planning a return at the Britain's Got Talent auditions in Cardiff yesterday.
A source said: "Simon shook his head. People weren't happy."
Speculation is already building around the judging line up for this year's X Factor UK - after the last series saw it's lowest viewing figures since 2006.
Apaz, Nicole Scherzinger - who won in 2012 with act James Arthur - has already been signed up for a second stint on the show, while Tulisa was forced to deny reports that she'd quit earlier this month.
Also being thrown into the rumour mill is ex-judge Sharon Osbourne, who is said to be keen to make her X Factor return.
*small voice* but... we just want Simon back. *smaller voice* please?
What do you think about all this - disappointed that Uncle Si won't be sitting on the panel this year?
Images: PA/ITV Pictures