Evaluating AI and AI Evaluations URI

In this course, we will aim to answer one key question, through a multidisciplinary examination:

We will answer this question by examining how to measure what an AI can do robustly and reliably. We will study both evaluations of AI and evaluations of AI evaluations and along the way, you will build a benchmark.

We will begin with a review of model evaluation for simpler ML models, emphasizing the more abstract goals that underlie the standard evaluation procedures. Then we will use these underlying goals to build an understanding of the tools available for evaluating more complex systems. Then we will study various AI benchmarks to see what specific AI abilities we can make rigorous claims about. We will also compare how these rigorous claims get translated into nonscientific literature.

Complete this form to request a permission number

Course structure¶

Spring 2026: TTh 5:00-6:15pm

There will be 1-3 papers or freely available textbook chapters to read most weeks. Most class sessions will be focused on a discussion of the paper(s) assigned for that class session.

While we study up-to-date papers of benchmarks and new model evaluation techniques, students will prepare short, lightning-talk style presentations of papers. For these class sesssions, several students each student will read and present different papers and we will discuss them as a group.

Evaluation¶

Students will have to:

• present lightning-talk (5min) style presentations of papers

• engage in course discussions

• complete reading reflections/article critques

• meet project milestones

• complete a project developing a (part of) an AI benchmark

The grade will be based on two components:

• 40% course community contributions

• 60% benchmarking project

within each of these categories some things will be graded on completion only (full points for doing it in good faith & on time) and some will be graded on quality (a more traditional score will be provided). For quality items a rubric will be provided 1 week or more in advance.

Remember, this is a 4 credit cass with only 3 contact hours per week. This means that you should be spending a minimum of 9 hours per week on things for the course outide of class time, but most weeks probably closer to 12. When project milestones are due, expecations for community contribution will be a bit lower, and between milestones especially when I am giving feedback, you’ll have more community contributions expected.

community contribution evaluation¶

Community Contributions will be based on

• preparing for class, every time, so whatever different days I request thigns, there will be different ways this looks. I will ask for different evidence on different days and basically record 1 thing per day for you. (completion- based on check)

• glossary and shared notes contributions (completion-number)

• contribution to shared infrastructure (completion -number and scope)

• lightning talk presentations (quality/binary)

• contributions to class discussion & prismia responses (completion -number and quality)

All of these are grouped together becasue for each of you your contributions will look different, because we are all contributing to the community, each specific contribution will be different.

For final grading, you will receive one of:

• meet/exceeds expecation (full credit);

• insconsistent/slighlty below (half credit);

• far below expecation (no credit).

There will be a chance to contest your standing here at the end of the semester if you do not reach meet/exceeds expecatations and want something between

I will let you know at least every 2 weeks if I think your contributions are below, meeting, or exceeding expectations. You can cancel out a below expectation with an exceeding expectation in order to get your average up to on track. You cannot have more than 2 below expecations in a row and still get meet/exceeds in your final grade (so you cannot just exceed at the beginning and then quit)

project evaluation¶

This will be evaluated by, percentatges are of the project grade not of the final course grade

• overall repo (quality 30%)

• your specific contributions to your project based on evidence and team mate review (quality; 20%)

• project milestones (completion 20%)

• report (quality 20%)

• presentation (quality 10%)

Project¶

The project will be completed by meeting several milestones, these milestones are not likely to change and changes will be announced with a lot of warning

LLM use¶

All work must reflect the students understanding. LLM assistants may be used to improve writing quality for assignments where writing quality will be assessed. However, when quality will be assessed, concision and proper style will also be required. Any submitted writing that contains classic “bot” phrasing or that is overly verbose and off topic will not be assessed and earn zero credit.

At the instructor’s discretion, any submitted work may be re-assessed by oral exam to ensure that the student actually understands.

Tools¶

This class meets in person, synchronously.

• we will use prismia chat to share materials. Join link on brightspace

• Most course work will be done in GitHub repos, you may choose to work in Github.com, a codespace, or locally for most work. Join link for github is on Brightspace.

• A reading list will developed collaboratively via zotero

• paper writing will be on overleaf via github, Sarah will create overleaf projects as needed so they get paid features via URI

• this site and other public materials from benchmarks will use myst markdown

Overall Schedule¶