This document describes the procedure used for a pilot of NIST’s Assessing Risks and Impacts of AI (ARIA) evaluation: ARIA 0.1. Five organizations participated, submitting a total of 7 AI applications to be evaluated. In this document, we first describe the design of the three evaluation scenarios (TV Spoilers, Meal Planner, Pathfinder) and the three testing levels (model testing, red teaming, field testing). We then discuss the methods used for assessment via dialogue annotation and tester questionnaires. Finally, we describe our approach to measuring validity of AI applications using measurement trees.