The original house at Castle Park, formerly known as Castle Perrin, was probably built in the 1820s, evidenced by its late-Georgian windows. By 1830 it was owned by Alderman Arthur Perrin, who resided at Bullock Castle and rented Castle Perrin to members of the British administration based in Dublin.
In the 1850s the Right Hon John Richards was in residence and later the Hon James O’Brien, second justice in the court of the Queen’s bench lived at the house. Perrin was responsible for enlarging the house, and added the castellated façade that faces Dublin Bay.
On March 29th, 1904, the Irish Times newspaper announced that a castellated Dalkey residence, set in thirty-four acres, had been sold for the sum of £4,070. The purchaser was an unknown Englishman, Mr Wilfred P Toone.