Cardiac magnetic resonance imaging (CMRI) provides non-invasive characterization of the heart and surrounding tissues. It is an important tool for the prognosis of pulmonary arterial hypertension (PAH), a disease with heterogeneous presentation that makes survival likelihood prediction a challenging task. In this paper, we propose a Geodesically Smooothed Tensor feature learning method (GST) that utilizes not only the heart but also its surrounding tissues to characterize disease severity for improving prognosis. Specifically, GST includes structures surrounding the heart by geodesic rings which were incrementally smoothed with Gaussian filters. This provides additive insight while modulating for patient positional differences for a subsequent tensor-based feature learning pipeline. We performed evaluation on Four Chamber and Short Axis CMRI from 150 individuals with confirmed PAH and 1-year mortality census (27 deceased, 123 alive). The proposed GST method improved AUC and Cox difference at 4-years post-imaging (Cox4YD) over the standardized measurement of right ventricular end systolic volume index (RVESVi: AUC: 0.58; Cox4YD: 0.18) on the Four Chamber protocol (AUC: 0.77; Cox4YD: 0.35). Only AUC was improved over RVESVi in the Short Axis scans (AUC: 0.77; Cox4YD: 0.16).