Creating Transformative and Trustworthy AI Systems Requires a Community Effort

As the SEI leads the community effort toward human-centered, robust, secure, and scalable AI, we are learning what is needed to move toward transformative and trustworthy AI systems. In this post, we describe how professionalizing the practice of AI engineering and developing the AI engineering discipline can increase the dependability and availability of AI systems. We also share what’s needed in the AI engineering community and how to get involved.

Voices calling for an AI engineering discipline are growing. Government entities such as the Defense Innovation Unit (DIU) are launching initiatives like the Responsible AI Guidelines to embed trust and social responsibility into DoD AI innovation activities. On a related front, research entities such as the IEEE Computer Society (CS) are launching special issue journals like AI Engineering to share practical experiences and research results for developing AI-intensive systems. Similarly, private sector entities including IBM and Coursera are partnering to release educational programming to train workforce members to build transformative and trustworthy AI systems. In addition to these institutional efforts, researchers such as Hannah Kerner, James Llinas, and Andrew Moore are championing the need for an applied discipline of AI engineering.

In partnership with the Office of the Director of National Intelligence (ODNI), we at the Carnegie Mellon University (CMU) Software Engineering Institute (SEI) are leading a national initiative to advance the discipline of AI engineering to increase utility and dependability of AI systems. We have hosted workshops and a symposium, published white papers and software artifacts, and shared resources on how to produce human-centered, robust and secure, and scalable AI systems. In the months ahead, we will continue to grow the AI engineering community by hosting discussions and fostering collaborations. With this work more than a year underway, we would like to share some insights we’ve gained and invite ideas and feedback in this blog post.

AI Systems Need to Shift from Brittle to Dependable

Organizations of all sizes and across all sectors are investing in AI technologies at an unprecedented rate to transform business and mission outcomes and to unlock competitive advantages. These AI investments are increasingly being implemented in high-stakes and high-availability scenarios, requiring sophisticated reliability engineering for operational assurance and responsible usage. Unfortunately, the return on AI investments is remarkably risky – Gartner estimates that nearly 85 percent of AI projects will fail in 2022. AI incident trackers, such as the AI Incident Database (AIID), are cataloging associated harms from failed AI endeavors (such as the self-driving Uber crash) and capturing examples of the real, sometimes irreversible, damage caused by brittle AI systems.

Incidents in the AIID, along with the examples of AI deployed in high-stakes and high-availability scenarios, call for shifting the mindset of AI system development from an ad hoc craft to a dependable engineering practice that is optimized to maximize value and minimize risk associated with the engineering construction. Traditional engineering disciplines have turned to practice professionalization as an enabler for optimizing this balance at a societal level.

Professionalizing the Practice is One Way Forward

Practice professionalization serves to standardize expectations for the performance of services and provides increased protections and channels for resolving issues. Consider the trust we place in our doctors, our lawyers, and even the engineers who design and construct our homes. We rely on their expertise to ensure that the products and services we receive are dependable and useful. It has become increasingly clear that society seeks to rely upon AI systems embedded in everyday infrastructure, including in high-stakes and high-availability applications, such as recommender systems in judicial sentencing, object detection systems in satellite surveillance, and optimization systems in financial forecasts.

As organizations integrate AI technology into these complex systems, rigorous engineering is required to balance system design tradeoffs and to avoid unintended consequences. Professional engineering practices (such as civil engineering) cultivate and uphold these rigorous standards (such as structural safety requirements) to facilitate quality engineering. Professional practice resources, such as certifications, accreditation systems, codes of practice, and professional development, offer vehicles to mature the collective state of the practice. For AI engineering, professional practice resources will provide practitioners tools to integrate AI technology into complex and dynamic systems (such as test and evaluation criteria for continuous ML monitoring).