Graduate Thesis Or Dissertation
 

A Deep Action Segmentation and Its Explanation with a Dictionary of Meaningful Attention Maps

Public Deposited

Downloadable Content

Download PDF
https://ir.library.oregonstate.edu/concern/graduate_thesis_or_dissertations/x346d975c

Descriptions

Attribute NameValues
Creator
Abstract
  • This thesis addresses the problem of temporal action segmentation in videos, where the goal is to label every video frame with the appropriate action class present. We focus on the domain of NFL football videos, where action classes represent common football play types. For action segmentation, we use a temporal convolutional network (TCN) that accounts for temporal context for labeling every frame, given a sequence frame deep features as input. Toward better understanding the TCN's decision-making process, we also compute TCN’s visual attention maps at the pixel level with the excitation back-propagation algorithm. As the attention maps highlight the most discriminative frame parts, we hypothesize that frequent space-time patterns of attention maps correspond to meaningful concepts in the football domain (e.g., characteristic spatial formation of the players, part of a yard line). Thus, computing the attention maps and discovering a dictionary of their frequent patterns could be used to provide meaningful explanations about TCN’s predictions. For dictionary learning, we first use a deep spatio-temporal auto-encoder to project the input attention maps onto a latent (encoded) feature space, where the attention maps can then be clustered. The resulting clusters can be assigned a semantic meaning by visual inspection for subsequent explanations about TCN’s performance. We present a quantitative evaluation of TCN on challenging NFL football videos, and visualizations of the discovered dictionary of discriminative frame parts.
License
Resource Type
Date Issued
Degree Level
Degree Name
Degree Field
Degree Grantor
Commencement Year
Advisor
Committee Member
Academic Affiliation
Rights Statement
Publisher
Language
Embargo reason
  • Intellectual Property (patent, etc.)
Embargo date range
  • 2018-01-17 to 2019-02-17

Relationships

Parents:

This work has no parents.

In Collection:

Items

Thumbnail Title Date Uploaded Visibility Actions
file details TrigkakisDimitrios2018.pdf 2018-01-15 Public Download