Beavercreek, Ohio – Scientists at Parallax Advanced Research are collaborating with PacMar Technologies on the Testing Artificial Intelligence Learning in Open-world Novelty Scenarios (TALONS) project, which has developed a novelty generation and evaluation verification and validation system for AI. The purpose of TALONS is to future proof AI by preparing it for potentially unknown circumstances and dynamics, known as “novelty,” relevant to an agent’s operation.
“Being able to create novelty in new dimensions is critical to the development of AI systems that are robust to the real world. In the domain of self-driving cars, TALONS addresses the novelty concern directly in a measurable, meaningful fashion,” said Dr. Darrell Lochtefeld, VP & Division Manager, Intelligent Systems Division, Parallax Advanced Research.
Developing AI for open worlds is one of the hardest problems in the field. What makes open worlds so difficult is novelty–which includes “unknown unknowns”–things we don’t even know that we don’t know. Most AI systems today are trained with past data, and, therefore, they often fail to recognize, characterize, and respond to novelty competently. Current verification and validation (V&V) methods lack a capability to represent, generate, and test AI against unknown unknowns. This limits V&V systems to generate only those novel conditions that are conceived of by human subject matter experts.
A basic premise of TALONS research is that encountering novelty/unpredictable situations is inevitable for all AI systems deployed in the real world. Although AI systems are becoming ubiquitous in various products in both military and commercial sectors, existing AI systems are unable to contend successfully with unexpected changes and situations.
Parallax scientists are providing the theoretical foundations to build an automated, scalable novel world environment generator. As part of this, they are also building a novel scenario generator capable of generating new scenarios for a newly generated world. These novel worlds can be realized in high-fidelity simulated environments, such as Car Learning to Act (CARLA) – an autonomous vehicle simulator, to measure AI performance to novelty. The theoretical underpinnings of the novelty generator are domain independent – meaning, TALONS can generate novel worlds in any domain, even human-made domains like monopoly.
An unpredictable situation in autonomous driving is when a car attempts to drive on ice for the first time when it’s not designed to operate in such an environment.
A key feature of the TALONS novelty generator is the use of transformations to introduce novelty both at the world environment level and the scenario generation level. Because the search spaces are massive, Parallax scientists have devised methods for automatic sampling from the space of novel environments, the transformations taking place, and agent responses in random scenarios.
With CARLA, the TALONS system can provide novelty in high-fidelity simulations of realistic vehicle interactions set in diverse and challenging dimensions within targeted training periods. This combination ensures adequate opportunities for agent novelty response evaluation to result in a greater understanding of an AI’s ability to deal with novelties. A key innovation is that novelty is generated randomly from an enormous space, removing flawed human biases about what unknowns are possible.
“Novelty–unknown unknowns–is a tricky problem to solve, because if we can write a computer program to generate novelty, how can it be novel?” said Dr. Dustin Dannenhauer, Parallax AI scientist and program manager of TALONS. “Our approach to solve this problem is to represent a strict formal environment definition for a starting environment, perturb it in a structured way over and over, thereby creating a massive (infinite, if needed) number of possible environments, many more than a human could enumerate in their lifetime.”
Parallax takes the approach of representing as many environments as possible, formally, and then relying on sampling techniques to find interesting novel environments that are worth using to test the AI.
“In our approach, human bias, if needed, is given during the sampling of the environment space, rather than encoding many assumptions into the entire environment generation process from the beginning,” said Dr. Dannenhauer. “This enables the TALONS approach to consider greater novel environment diversity than existing approaches.”
Parallax is developing software for automatic novelty generation based on concepts and prototypes previously developed. The team will evaluate both in-house baseline agents and third-party AI agents with respect to novelty robustness and the evaluations have used CARLA; this well-known high-fidelity simulator built using the Unreal Video Game Engine offers, among other features, a high-fidelity physics engine.
Upon successful completion, the TALONS novelty generator will be capable of generating diverse scenarios that introduce different forms of novelty with controlled characteristics, and these scenarios will be operationally effective at testing third-party AI agents’ robustness to the introduced novelty.
"We need new benchmarks to test AI’s ability to adapt to unforeseen situations, and the TALONS project is an important step in this direction,” said Dr. Dannenhauer.
Parallax is a 501(c)(3) nonprofit that tackles global challenges by accelerating innovation and developing technology and solutions through strategic partnerships with government, industry and academia across Ohio and the Nation. Together with academia, Parallax accelerates innovation that leads to new breakthroughs. Together with government, Parallax tackles critical global challenges and delivers new solutions. Together with industry, Parallax develops groundbreaking ideas and speeds them to market.