2024 AutoTest Technical Conference

As a result of the exponential increase in complexity of rule-based systems processing a large amount of data, the move to AI and data driven processes is inevitable. This process requires a shift of testing and assurance paradigms. Implementing AI models in automotive systems requires a novel approach of testing and documenting the process of decision making. Depending on the level of safety, different levels of interpretability and test coverage, as well as documentation are required. Fueled by the proposed Acts on AI, the need for explanations of the system is imminent. Keysight Technologies, a market leader in test and measurement, proposes a novel holistic framework combining tests covering the complete lifecycle of an AI System, from conception to the last day of deployment, standardizing test executions and providing insights into the decision-making process of the Model under Test (MuT). While being aligned with requirements stated by governmental institutions, the proposed solution enables AI Engineers as well as Domain Experts to join forces and improve their models based on insights and recommendations offered by the framework. Separating the lifecycle into five steps (Problem understanding and Data Analysis, Feature Engineering, Model Training, Deep Model Evaluation, Model Inference) allows integration of the framework into different software engineering processes, like V-Model or Waterfall, necessary for safety critical systems, and platforms.