The castle was on the south side of the town near the edge of the high, steep precipice overlooking the River Torridge, now called Castle Hill. Its commanding position, with strong natural defences to the south and far-reaching views of the surrounding countryside, can still be appreciated.
In the Middle Ages it was an important site occupied for about four centuries. As the heavily fortified property of the lords of Torrington it was the most imposing secular residence in the locality, as well as being the administrative centre of numerous rural estates.
Castle Street leads to the old castle site where there is now a bowling green, car park and the castle mound which is all that remains of the old fortification.
The original castle was built in the 12th century and the site is first mentioned in 1139 when it was attacked in the civil war waged in the reign of King Stephen (1135-54). Henry de Tracy, a supporter of King Stephen, took the castle from its lord, William de Toriton, but William’s family later regained control and kept the castle until 1227. It seems the first castle was built without royal licence and in 1228 the Sheriff of Devon ordered it to be pulled down and the ditches around it to be filled up level with the ground. A second castle was built, with permission, by Richard de Merton in 1338 on the same site.
The castle of 1139 had a tower which was possibly situated on the earthwork to the west of the bowling green. There is later reference to a bailey and the site may have been of the well-known motte (mound) and bailey (courtyard) pattern. (The name of the Barley Grove car park could well be a corruption of ‘bailey’). Despite the extensive destruction of the site in 1228, this mainly affected its defences for the chambers, hall, kitchen, grange and cowshed were mentioned in 1343 and the chapel still survived in the 16th century having been used as a school house since 1485. This was eventually demolished in 1780. Another school house was built on the same site which, in later years, became the Blue Coat School, then the Eric Palmer Community Centre. This closed in 2010 and the building is now a health and fitness centre.
The east end of the castle site, said to be the location of the castle keep, is now occupied by the bowling green. The castellated walls together with the arrow slits seen here were, however, constructed by Lord Rolle in around 1846 and he built Town Mills (by New Bridge on the A386) in the same flamboyant style. When the old bowling pavilion was pulled down in 1987 to make way for a new clubhouse, an archaeological study was made and the masonry foundations of part of a domestic building were discovered, the tail of a rampart of clay and stone was located and considerable quantities of medieval pottery were recovered.