With both leadership positions spilled, the foreign affairs minister Julie Bishop's position as deputy Liberal leader was also opened, but she was the only one to nominate and was therefore re-elected.
Mr Dutton attended a leadership meeting with Mr Turnbull earlier on Tuesday.
In a move that repeats the pattern of previous leadership spills such as Paul Keating's two ballots against Bob Hawke during 1991, Mr Dutton has resigned from cabinet and go to the backbench to consider his next steps.
Dutton has consistently shied away from admitting his desire to take on Turnbull, with Liberal MP Christopher Pyne telling Sunrise host Kochie just hours before the party room voted, that Dutton remained loyal to the Liberal leader.
Mr Turnbull had brought on the vote himself in a bid to resolve speculation that his leadership was under threat.
Mr Dutton previously said he would continue to support Mr Turnbull as long as he was a member of the cabinet.
Prime Minister Jacinda Ardern said the resignation would not have any impact on the issues the government continues to raise across the ditch.
Mr Dutton has not indicated if he will mount another challenge.
SBS News understands Labor leader Bill Shorten told colleagues voters were asking "what the hell is going on?"
Malcolm Turnbull called the spill after what was clearly massive pressure from the Dutton camp.
Malcolm Turnbull is being assured he has the numbers to resist any immediate push from Peter Dutton after the Government dumped its carbon emissions reduction target to placate internal dissidents.
Abbott was then unseated in a party coup by Turnbull in 2015.
"They want us to know that we should be concentrating on the things that are important to them". Abbott told the ABC outside Parliament House on Monday night.
Damian Drum, a lawmaker in The Nationals' party, a junior coalition partner, called on Abbott to resign from Parliament instead of destabilizing the government.
Climate change was to be the undoing of Turnbull the first time around. "And it was a conversion of convenience this morning".