Model Cards

The value of a shared understanding of AI models

Whether it’s knowing the nutritional content in our food, the conditions of our roads, or a medication’s interaction warnings, we rely on information to make responsible decisions. But what about AI? Despite its potential to transform so much of the way we work and live, users are often interacting with AI and machine learning models without a clear understanding of how they function. For example, under what conditions does the model perform best and most consistently? Does it have blind spots? If so, where? Without model transparency, such questions can be surprisingly difficult to answer.

Model cards: A step towards AI transparency

We’re excited to continue to build model and data transparency. Model cards are an idea we originally explored in a Google research paper in 2018, and one that has been used across the industry to organize essential facts of machine learning models in a structured way.

Take a dog breed classifier, for instance. Its model card might specify that the model is based on a convolutional neural network that outputs bounding boxes along with the breed’s label. It could then build on this basic foundation with insight into the factors that help ensure optimal performance. What kind of photos tend to yield the most accurate results? Can it handle partially obscured dogs? What about dogs that are extremely close, extremely far away, or seen from unusual angles? Even simple guidelines like these can make a difference, helping us leverage a model’s capabilities and steer clear of its limitations.

Other models might benefit from very different kinds of information. A model card for a language translator, for example, may provide guidance around jargon, slang and dialects, or measure its tolerance for differences in spelling. There are many forms that transparent documentation can take, and we encourage a flexible approach that allows for variation in model type and evaluation specifics.

Model cards may even have the potential to help investigate issues like unfair bias. For instance, does a model perform consistently across a diverse range of people, or does it vary in unintended ways as characteristics like skin color or region change? Model cards can bring clarity to these kinds of disparities, encouraging developers to consider their impact on a diverse range of people from the start of their app’s development process—and keep them in mind throughout.

Model cards take many different forms depending on the use case. Given the rapid evolution of AI technologies and as new, industry-wide benchmarks for performance and safety emerge, model card structures and content must be flexible. They’re not one-size-fits-all, and may also appear within other transparency artifacts, such as technical reports that offer in-depth technical details alongside the summaries featured in model cards.

Here, we offer some examples of recent model cards for different types of models:

An evolving approach to model cards

In our 2018 research paper, we proposed some basic model card fields that would help provide model end users with the information they need to evaluate how and when to use a model. Many of the fields we first proposed remain vital categories of metadata that are found in model cards across the industry today; however, as AI models changed and evolved, we recognized that we needed new types of metadata and information included in model cards, as well as new processes on how to generate model cards.

The model card fields suggested in our 2018 research paper "Model Cards for Model reporting".

In 2020, we introduced the Model Card Toolkit (MCT) to help streamline the creation of model cards. We made the MCT widely available to help promote the development of responsible AI, and support developers in more easily compiling the information that goes into model cards. Much like the model card fields we proposed in 2018, the MCT remains a valuable tool; however, the evolution of our approach to model cards didn’t end with the MCT.

Previous iterations of our model cards, such as one for an API model and another to predict 3D facial surface geometry, and more detailed model reports, such as one for a face detection model and another for an object detection model, adequately conveyed important information about those respective models.

However, as generative AI models have advanced, we have adapted our most recent model cards, such as the card for our highest quality text-to-image model, to reflect the rapidly evolving landscape of AI development and deployment. While these model cards still contain some of the same categories of metadata we originally proposed in 2018, they also prioritize clarity, practical usability, and include a comprehensive assessment of a model’s intended usage, limitations, risks and mitigations, and ethical and safety considerations.

As models continue to evolve, we will work to recognize the key commonalities between models, which can be leveraged as inputs for model cards. Although there is no perfect model card format that will work for every model, by identifying these commonalities, while also remaining flexible in our approach, we can use model cards to support shared understanding and increased transparency around how models work.

A technical artifact designed for developers

As AI models have grown more complex, so has the type of information and data that is included in model cards. Today’s model cards are technical artifacts designed for developers and other more technical audiences. Developers can use model cards to help design applications that emphasize a model’s strengths while avoiding or informing end users of its weaknesses. Despite this technical focus, model cards can still provide useful insights for companies, regulators, and end users.

AI transparency is a common good

Transparency in AI should be a common good. That’s why model cards aren’t intended to be a Google product, but a shared, evolving framework shaped by a range of voices. This includes users, developers, civil society stakeholders, and companies across the industry.

Help us advance the conversation around AI

We believe model cards will play a foundational role in the future of AI, including generative AI. The experiences of developers, policy makers, analysts, advocates, and users will play an essential role in shaping this resource.