This handy Wikipedia list tells me the oldest buildings and structures in Toronto. After our discussion yesterday about the Mirvish - Geehry condo project, I clicked over to learn about the oldest building in Toronto.
It's The Scadding Cabin, found on the western grounds of the CNE. It was built by the Queen's York Rangers in 1794 on behalf of John Scadding, who served as clerk (essentially, an executive assistant) to the first lieutenant governor of Upper Canada, John Graves Simcoe.
Rumour has it they're going to tear down The Scadding Cabin for a new condo tower... but don't quote me on that.