- Instance segmentation, the classification and localization of objects in an image, is a problem in cellular biophysics due to the physiological relevance of cell morphology. Particularly, cancer cells migrate in tissue space by changing their body shape similarly to how humans extend their limbs to swim through water. Monitoring such shape changes in a dense cell population requires high throughput methods to process and analyze microscopy timelapse video frames. This thesis explores the use of a deep learning algorithm in the instance segmentation of MDA-MB-231 cancer cells at varying densities. We employ Mask R-CNN and characterize the model's performance with varying hyperparameters, including learning rate, gradient clip norm, learning momentum, non-max suppression, and anchor ratios. We apply performance metrics, namely the precision-recall curve and mean average precision, to validate the model's detections. We then explore how well Mask R-CNN generalizes from a low cell density training set to high density images. Our results demonstrate Mask R-CNN as a reliable instance segmentation model to segment dense cell populations, and serves as a starting point to develop an automated pipeline for cell experiments.
- Key Words: Deep Learning, Mask R-CNN, Cancer Cell, Cellular Biophysics