Head and neck cancer patients can experience significant side effects from therapy. Accurate risk stratification allows for proper determination of therapeutic dose and minimization of therapy induced damage to healthy tissue. Radiomics models have proven their power for detection of useful tumors characteristics that can be used for patient prognosis. We studied the ability of deep learning models for segmentation of gross tumor volumes (GTV) and prediction of a risk score for progression free survival based on positron emission tomography/computed tomography (PET/CT) images. A 3D Unet-like architecture was trained for segmentation and achieved a Dice similarity score of 0.705 on the test set. A transfer learning approach based on video clip data, allowing for full utilization of 3 dimensional information in medical imaging data was used for prediction of a tumor progression free survival score. Our approach was able to predict progression risk with a concordance index of 0.668 on the test data. For clinical application further studies involving a larger patient cohort are needed.