AI Has a Carbon Footprint - and It's Massive! %

Researchers at the University of Massachusetts, Amherst, found that energy required to train a common AI model was equivalent to nearly five times the lifetime emissions of the average American car, including its manufacture!

The paper specifically focused on determining carbon emissions for the subfield of AI called NLP (Natural-Language Processing). The researchers looked at the field’s four most-contributing models: the Transformer, ELMo, BERT, and GPT-2. Power consumption for each model was measured as it trained on a single GPU for a day, which was then used to calculate the total energy consumed during the entire training, and the equivalent of pounds of carbon dioxide based on the average US energy mix.

The results showed that as the model size increased, so did its computational and environmental costs; but as the models used an additional “neural architecture search” for increased accuracy, the carbon dioxide emissions skyrocketed to 626,155 pounds in the case of Transformer.

“Training a single model is the minimum amount of work you can do,” says Emma Strubell, the lead author of the paper. AI researchers would usually develop a new model from scratch, or adapt an existing model to a new data set, either of which can require many more rounds of training and tuning.

To put things into perspective, Strubell and her colleagues found that building and testing the final-paper-worthy model required training of 4,789 models over six months. In terms of CO2, these values equated to more than 78,000 pounds of emissions, and it is alarming that such results represent typical work in the field.

The intensity of resources required for paper-worthy results has made it difficult for people in academia to continue contributing to research. Emma and her coauthors hope that their paper’s findings help in the investment for more efficient hardware and algorithms, so that researches in academia and the industry both get equal access to resources.