It was the medieval tower from the 1200s that the castle commander Erik Rosenkrantz reconstructed in the 1560s, partly with stone from the demolished monastery at Lyse. The king, Fredrik II, had ordered Erik Rosenkrantz to pull down the old ramshackle tower and build a new one. Rich, gothic building details show that Mr. Erik must have viewed the medieval tower as a good model on which to continue building. The research made by Gerard Fischer following the war destruction in 1944 have confirmed that this must be “the little castle by the sea” which Magnus Håkonsson (Lagabøte) had constructed.
What Rosenkrantz did was to combine the little castle with castle officer Jørgen Hanssøn’s circular wall and ante-castle from the beginning of 1500, make it higher, and construct a new an contemporary façade facing south – facing Bryggen (the harbour) and the town. While the little castle and the hall had as their models English constructions, now they were Scottish. It has been documented that Rosenkrantz used Scottish bricklayers.