Landmark localisation in medical imaging has achieved great success using deep encoder-decoder style networks to regress heatmap images centered around the target landmarks. However, these networks are large and computationally expensive. Moreover, their clinical use often requires human interaction, opening the door for manual correction of low confidence predictions. We propose PHD-Net: a lightweight, multi-task Patch-based network combining Heatmap and Displacement regression. We design a simple Candidate Smoothing strategy to fuse its two-task outputs, generating the final prediction with quantified confidence. We evaluate PHD-Net on hundreds of Short Axis and Four Chamber cardiac MRIs, showing promising results.