The fort was built in the 17th century (around the 1630s) by Raja Hasan Khan of the Amacha dynasty. The name Fong-Khar means Palace on the Rock, because the original building rests on a platform around a large cone-shaped boulder. It is actually a group of three structures built at different times, with around 40 rooms including kitchens, servants' quarters and a family mosque.
The fort was built by craftsmen brought in to work in the Balti and Kashmiri styles, using local stone, mud and wood. By the early 20th century it was no longer used as a royal residence and slowly fell into ruin, until the Aga Khan Trust took on the restoration, carried out by Aga Khan Cultural Service Pakistan from about 1999 to 2004/05. The restored fort opened to visitors in the mid-2000s and is now a heritage museum and boutique hotel managed by Serena Hotels. Its garden and pavilion are popular places to rest.