Park in the main car park on the Port Gaverne Road and take the coastal path down to the harbour.
Follow the road round the harbour and up the steep Roscarrock Hill on the other side. This turns into a footpath and carries on up to the top of the cliffs above Lobber Point where there are fine views back over the village and towards Tintagel.
From here the path drops down to the remote rock inlet of Pine Haven and a beautiful unspoiled stretch of coastline.
Whist there are alternative inland routes back to Port Isaac if, like me, the temptation of staying next to the cliff edge is always more than you can resist, then all you can do is retrace your steps.
Either that or keep going until you run out of coastal path. These days I think that will take you to Chewton Bunny on the Dorset/Hampshire border (the other side of Highcliffe Castle), which is perhaps a bit more than a stroll.