City Research Online

Improving Fairness using Vision-Language Driven Image Augmentation

D'Incà, M., Tzelepis, C. ORCID: 0000-0002-2036-9089, Patras, I. & Sebe, N. (2024). Improving Fairness using Vision-Language Driven Image Augmentation. In: Proceeding of the 2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). 2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 4-8 Jan 2024, Waikoloa, Hawaii. doi: 10.1109/WACV57701.2024.00463


Fairness is crucial when training a deep-learning discriminative model, especially in the facial domain. Models tend to correlate specific characteristics (such as age and skin color) with unrelated attributes (downstream tasks), resulting in biases which do not correspond to reality. It is common knowledge that these correlations are present in the data and are then transferred to the models during training (e.g., [35]). This paper proposes a method to mitigate these correlations to improve fairness. To do so, we learn interpretable and meaningful paths lying in the se- mantic space of a pre-trained diffusion model (DiffAE) [27] – such paths being supervised by contrastive text dipoles. That is, we learn to edit protected characteristics (age and skin color). These paths are then applied to augment images to improve the fairness of a given dataset. We test the proposed method on CelebA-HQ and UTKFace on several downstream tasks with age and skin color as protected characteristics. As a proxy for fairness, we compute the difference in accuracy with respect to the protected characteristics. Quantitative results show how the augmented images help the model improve the overall accuracy, the aforementioned metric, and the disparity of equal opportunity. Code is available at:

Publication Type: Conference or Workshop Item (Paper)
Additional Information: © 2024 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.
Publisher Keywords: Training, Measurement,Correlation, Image color analysis, Semantics, Natural languages, Skin
Subjects: Q Science > QA Mathematics > QA75 Electronic computers. Computer science
Departments: School of Science & Technology > Computer Science
[thumbnail of wacv24_tzelepis.pdf]
Text - Accepted Version
Download (9MB) | Preview
Supplementary Materials:


Add to AnyAdd to TwitterAdd to FacebookAdd to LinkedinAdd to PinterestAdd to Email


Downloads per month over past year

View more statistics

Actions (login required)

Admin Login Admin Login