------------------- # GENERAL INFORMATION ------------------- 1. Title of Dataset Explaining Reinforcement Learning To Mere Mortals: An Empirical Study - Supplementary Materials 2. Creator Information Name: Anderson, Andrew A. Institution: Oregon State University College, School or Department: College of Engineering Address: Email: anderan2@oregonstate.edu ORCID: 0000-0003-4964-6059 Role:Lead author 3. Contact Information Name: Anderson, Andrew A. Institution: Oregon State University College, School or Department: College of Engineering Address: Email: anderan2@oregonstate.edu ORCID: 0000-0003-4964-6059 Role: Lead author ------------------- CONTEXTUAL INFORMATION ------------------- 1. Abstract for the dataset This dataset is the supplementary materials for the research paper "Explaining Reinforcement Learning to Mere Mortals: An Empirical Study". In an age where AI has become more pervasive, making decisions for us, it's reasonable to expect it to explain itself. What should these explanations look like? This study takes a look at the effectiveness of two visual explanation types: saliency maps (the eyeballs of the AI) and decomposed reward bars (its crystal ball for a predicted score in the future). 2. Context of the research project that this dataset was collected for. Our Research Questions were as follows: 1. Which treatment is better (and how) at enabling people to describe how the system works? 2. Which treatment is better (and how) at enabling people to predict what the system will do? 3. Date of data collection: 2018-09-28 --> 2018-10-25 4. Geographic location of data collection: 44.5638, -123.2794 5. Funding sources that supported the collection of the data: DARPA #N66001-17-2-4030 NSF #1314384 -------------------------- SHARING/ACCESS INFORMATION -------------------------- 1. Licenses/restrictions placed on the data: This work is licensed under a Creative Commons Attribution 4.0 International License 2. Links to publications related to the dataset: This dataset is supplementary material for the following paper:Explaining Reinforcement Learning to Mere Mortals: An Empirical Study Andrew Anderson, Jonathan Dodge, Amrita Sadarangani, Zoe Juozapaitis Evan Newman, Jed Irvine, Souti Chattopadhyay, Alan Fern, Margaret Burnett Accepted at: International Joint Conference on Artificial Intelligence (IJCAI) 2019 (Macao, China). Appropriate Citation (APA): Anderson, A., Dodge, J., Sadarangani, A., Juozapaitis, Z., Newman, E., Irvine, J., ... & Burnett, M. (2019). Explaining Reinforcement Learning to Mere Mortals: An Empirical Study. In Conference proceedings IJCAI. Appropriate Citation (Bibtex): @inproceedings{papazoglou1991next, title={Explaining Reinforcement Learning to Mere Mortals: An Empirical Study}, author={Anderson, Andrew and Dodge, Jonathan and Sadarangani, Amrita and Juozapaitis, Zoe and Newman, Evan and Irvine, Jed and Chattopadhyay, Souti and Fern, Alan and Burnett, Margaret}, booktitle={Conference proceedings IJCAI}, pages={}, year={2019}, } 3. Recommended citation for the data: Anderson, A. A. (2019). Explaining Reinforcement Learning To Mere Mortals: An Empirical Study - Supplementary Materials (Version 1) [Data set]. Oregon State University. https://doi.org/10.7267/tt44ps61c 4. Dataset Digital Object Identifier (DOI) 10.7267/tt44ps61c 5. Limitations to reuse Under what conditions to use dataset: Anyone can use our code for the purposes of replicating our study or training their own artificial intelligence (agent) in our small Real-Time Strategy (RTS) domain. -------------------------- VERSIONING AND PROVENANCE -------------------------- 1. Last modification date 2018-05-17 2. Links/relationships to other versions of this dataset: There are no other versions of this dataset. The source code for this study can be found in https://github.com/SCAII/Sky-install​ 3. Was data derived from another source? No 4. Additional related data collected that was not included in the current data package: Participant information that we are still analyzing. -------------------------- METHODOLOGICAL INFORMATION -------------------------- 1. Readme to run 4 quadrants game study: To set up the study, open: Reproduce Scaii study.pdf Note: ReplayFiles contains the exact same replay files we used in our study. They do not require you to train another agent or modify any source code. Place them where they need to go, given to you in Reproduce Scaii study.pdf Note: QuestionFiles contains the questions that we asked for the study. Instructions on their setup for the system is provided in Reproduce Scaii study.pdf For all treatments: Go to SupplementalMaterials/Scripts/Door Print off DoorPersonChecklist.docx Provide to your door person, follow all protocols Go to ../Helper Print off HelperChecklist.docx Provide to your in-session helpers for setting up the lab. Tells them what they can/cannot answer. Shows them how to tear down the lab. Note: Camtasia is a screencapture software, but you can use your own. Go to ../Tutorializer If running the Control: print T0 Script Final + Emergencies.pdf If running the Saliency treatment: print T1 Script Final + Emergencies.pdf If running the Rewards treatment: print T2 Script Final + Emergencies.pdf If running the Everything treatment: print T3 Script Final + Emergencies.pdf IF YOU WANT TO TRAIN THE AGENT: go to Reproduce SCAII study, section 5. IF YOU WANT THE SALIENCY CODE THAT WE USED: Go to SupplementalMaterials/SaliencyCode open saliencyCode.docx follow the link to the GitHub repository 2. Methods for processing the data: When running participants, the system generates logged data at the same level as where replays are stored. Their name will be of the form: answers_<>_<> 3. Instrument- or software-specific information needed to interpret the data: The source code for this study can be found in https://github.com/SCAII/Sky-install 5. Environmental/experimental conditions: The system is designed to run on Windows machines only. We ran ours on machines running Windows X, each with at least 8GB of RAM and having ~500GB of main memory to store the replay files. We used Camtasia for our screen capture software, though any screen-capture software will suffice. 6. Describe any quality-assurance procedures performed on the data: We validated our data files by parsing the data and convering it to a .csv. We then performed visual scans to make sure that each decision point had answers to all questions. 7. People involved with sample collection, processing, analysis and/or submission: Andrew Anderson & Jonathan Dodge were responsible for collection, processing, analysis, and submission of the data to IJCAI. Evan Newman, Jed Irvine, & Andrew Anderson were responsible for coding the UI and enabling the features for the participants. Amrita Sadarangani was responsible for the saliency computations used in the study. Zoe Juozapaitis was responsible for training the RL agent that we used. Kin-Ho Lam was responsible for the logging software and the question files. Souti Chattopadhyay was responsible for being the door person of the study, and she devoted many many hours to sandboxing our study and ensuring we had high-quality data to analyse. Margaret Burnett and Alan Fern were responsible for keeping their graduate students sane and motivated. --------------------- DATA & FILE OVERVIEW --------------------- File List: 1. Zipped file scaii.zip has the following contents: scaii folder Contains the source code for the system .scaii/backends contains the map data that the system uses. .scaii/bin contains scaii_core.dll, a core part of the system. .scaii/city_attack_static contains attack_enemy.lua. .scaii/git contains the SCAII folder, the main folder that holds all the source code. .scaii/glue contains the python environment that glues the system together. .scaii/replays contains the treatment question files, replay files used in OSU's study, and also where the answer files appear. NOTE: The name of this folder starts with a period. When scaii.zip is unzipped it will generate a folder named .scaii that will appear as a hidden folder in Unix systems like Linux and Mac. 2. Zipped file SupplementalMaterials.zip has the following contents a. Reproduce SCAII study.pdf Contains all the information you need to replicate our study b. QuestionFiles folder Contains all question files for our study, where T0 -> Control, T1 -> Saliency, T2 -> Rewards, T3 -> Everything General format can be found in Reproduce SCAII study.pdf c. ReplayFiles folder Contains all the replay files Oregon State University used for their study. d. SaliencyCode folder Contains file saliencyCode.pdf with the link to our GitHub repo for the saliency code we used. e. Scripts folder Scripts/Door/ Contains all information that the door person needs to help in the study in file DoorPersonChecklist_Script.pdf. The door person is responsible for checking people in to the session and following up with potential late/missing participants. They should always try to reschedule late/missing participants. Scripts/Helper/ File HelperChecklist_Script.pdf contains all information that the in-session helpers need to set up the sessions, what to do during the sessions, and how to save the files after and tear-down the session. Scripts/Tutorializer Folder Contains thee 4 scripts to read VERBATIM during each session, T0 -> Control, T1 -> Saliency, T2 -> Rewards, T3 -> Everything. Emergencies.pdf contains the protocols for emergencies (bathroom breaks, interruptions, etc). d. Study Docs/Post Task Questionnaires folder Contains the 4 pdfs for the post-task questionnaires, to be distributed after participants have completed the tasks on their screens. ----------------------------------------- CODE-SPECIFIC INFORMATION: ----------------------------------------- 1. Installation Instructions in Reproduce SCAII study.pdf 2. Requirements All information can be found at: https://github.com/SCAII/Sky-install​ 3. Usage Found in the file: Reproduce Scaii study.pdf 4. Support The authors will not support others that want to use these scripts. 5. Contributing The code is in a public repository, and other researchers are welcome to pull our code and manipulate it to their needs. However, we request that other researchers do not attempt to merge their code with our original without the explicit, written consent of Mr. Jed Irvine, whom can be reached at: irvine@eecs.oregonstate.edu