Although parts of Rye such as St Mary's Church (from 1220) and Ypres tower (a castle built in 1258 to guard the port) date from the thirteenth century, a great deal of it was destroyed by the French in the fourteenth and fifteenth centuries. As a result, much had to be rebuilt and there are a large number of sixteenth-century timber-framed buildings, with the addition of some attractive Georgian architecture to be found.

The eastern most port in Sussex, Rye was designated one of the Cinque Ports along with Winchelsea in 1346, although as the sea receded its importance as a port slowly diminished; Rye now stands a mile or two (1.6 - 3.2 km) from the sea.

Strand Quay, once the commercial heart of the town, sits next to the River Tillingham, a tributary of the River Rother that is still navigable from the sea. On warm summer days it is possible to see French and Dutch flags displayed on visiting boats. The old warehouses are still there but now contain shops, restaurants, a public house and museums instead of the cargoes, which would once have been unloaded here.

History is all around in Rye. In 1340, four gates were built when the port was fortified to protect
it from the French, although today only one remains. Called Landgate, it can be found in the north-east corner of the town. The original Mermaid Inn in Mermaid Street was destroyed in 1370 and the building that stands here now is from the fifteenth century. The attractive Georgian fašade of the George Hotel hides a sixteenth-century building, and opposite is the Old Grammar School which was given to the town by Sir Thomas Peacocke in the seventeenth century.

Rye has also had its share of famous visitors and residents. Queen Elizabeth I visited in 1573 and George I was put up in Lamb House by Mayor James Lamb in 1726. The same building also saw American novelist Henry James and later author E. F. Benson as residents; Benson was the creator of Mapp and Lucia. John Fletcher the dramatist was born in Fletcher's House in 1579 and Samuel Jeaks, the author of the 'Charter of the Cinque Ports' and was born in 1623, also came from Rye.

Rye is one of the most picturesque towns in Sussex. It has held on to its past, yet has moved forward with the times. New businesses have taken over the old buildings in a sympathetic way, leaving Rye as one of the finest examples of an ancient port to be found.