The estate was owned by several families during the hundred years or so before the house was destroyed by fire. In 1841 Jane Granger of Wigton died and left the property and land to her cousin Henry Granger of London; a merchant with interests in Liverpool and Australia. By 1845 Henry owned a large amount of land and many properties in High Ireby. He continued to build up the estate and by the 1860's when he was needed to find the sum of £20,000 he was forced to sell the estate. It was eventually bought by John Boustead in 1870. This was the same John Boustead who had recently bought Armathwaite Hall from the Vane family.
In 1906, Ireby Grange was then bought by James Gurney and is still owned by the family. In 1931 Hugh Walpole visited Ireby Grange and found it in a dilapidated state. The Grange featured in as the principle house in his third book 'The Fortress' in his Lakeland saga.
The Grange was later destroyed by fire in January 1957. The Cumberland News of Friday, January 18th 1957 reported:-
'A 100 year-old fifty two roomed three story mansion - the Grange at High Ireby, near Wigton - was gutted by fire on Sunday. The mansion was better known to most people as " the Fortress" in Hugh Walpole's Herries Series. The owner Mr Norman William Gurney, a retired land agent, near London, used the mansion for only 6 - 8 weeks each year in the summer. Only four rooms were furnished. He was last in residence in August and it has been unoccupied apart from his annual visits. The mansion came into his possession upon the death of his father, Mr James Gurney, a well known Wigton councillor, about eighteen years ago.' 'Smoke was spotted coming from the roof at about 7.30 am by Mr Joe Hayton, who farms with his family nearby. He and another neighbour went to the house but left quickly to raise the alarm when they saw there was nothing they could do. The Wigton brigade was soon on the scene and the sub-officer J. Hutton immediately sent to Workington for assistance. Flames could be seen round the roof eaves. By noon the whole of the main roof and the roof of the square tower to the front topped by a weather vane had fallen in and the whole of the house was described as a shambles. The fire is believed to have started in the top story.'
The Grange stood in ruins for a while and much of the stone was used to fill the barytes mine shafts at Ruthwaite to make them safe.