The coalition only happened though because an established party with huge name recognition benefitted from a very specific set of political circumstances.
The reason the Lib Dems did incredibly well in that election was partially because the traditional left-centre-right alignment of modern British political parties had gone out of whack and partially because the Tories weren't particularly convincing. Lots of young, lefty voters were turned off by Labour's shift to the right, Iraq, tuition fees etc. saw the Lib Dems as their best option and switched, which partially mitigated the impact of a smaller than expected number of floating voters switching to the Tories. Consequently, whilst Labour's vote collapsed to 29%, the Tories still only managed about 36% and they weren't able to gobble up the Lib Dem-Tory marginals/three-horse race seats they needed to build a majority.
Third parties holding the balance of power in our political system is fantastically rare and it took a very odd set of circumstances for it to happen in 2010. Given that they utterly trashed their reputation in coalition, it seems very unlikely that the Lib Dems will return to anything approaching the level of popularity they did in 2005-2010 anytime soon, barring some sort of electoral pact with Labour. In terms of a new party making enough of a splash in parliamentary democracy to become a major player/power-broker on a regular basis - the last time a party did that was Labour in 1918, and that only happened because new voting laws more than doubled the electorate by giving non-property owning men the vote. The prospects of it happening now are incredibly remote.