Pollen allergies have become one of the most wide-spread afflictions that impact quality of life. This has made the need for automatic pollen detection, classification and monitoring a very important topic. This paper introduces a new public annotated image data-set of pollen with almost 45 thousand samples obtained from an automatic instrument. In this work we apply some of the best performing convolutional neural networks architectures on the task of pollen classification as well as some fully convolutional networks optimized for image segmentation on complex microscope images. We obtain an F1 scores of 0.95 on the new data-set when the best trained model is used as a fully convolutional classifier and a class mean Intersection over Union (IoU) of 0.88 when used as an object detector.