Towards fair decentralized benchmarking of healthcare AI algorithms with the Federated Tumor Segmentation (FeTS) challenge
Zenk, M. ORCID: 0000-0002-8933-5995, Baid, U.
ORCID: 0000-0001-5246-2088, Pati, S.
ORCID: 0000-0003-2243-8487 , Linardos, A., Edwards, B.
ORCID: 0000-0002-0957-9149, Sheller, M.
ORCID: 0000-0002-6571-0850, Foley, P.
ORCID: 0000-0001-9401-3088, Aristizabal, A., Zimmerer, D., Gruzdev, A., Martin, J.
ORCID: 0000-0003-2476-7639, Shinohara, R. T., Reinke, A.
ORCID: 0000-0003-4363-1876, Isensee, F., Parampottupadam, S.
ORCID: 0009-0009-9401-887X, Parekh, K., Floca, R.
ORCID: 0000-0003-3218-3377, Kassem, H.
ORCID: 0000-0001-5830-8890, Baheti, B., Thakur, S., Chung, V.
ORCID: 0000-0002-5622-7998, Kushibar, K.
ORCID: 0000-0001-7507-5208, Lekadir, K., Jiang, M.
ORCID: 0000-0003-4228-8420, Yin, Y., Yang, H., Liu, Q., Chen, C.
ORCID: 0000-0002-6040-6833, Dou, Q., Heng, P-A.
ORCID: 0000-0003-3055-5034, Zhang, X., Zhang, S.
ORCID: 0000-0002-8719-448X, Khan, M. I., Azeem, M. A., Jafaritadi, M.
ORCID: 0000-0002-4085-4057, Alhoniemi, E., Kontio, E.
ORCID: 0000-0002-0606-0106, Khan, S. A., Mächler, L., Ezhov, I.
ORCID: 0000-0002-0862-6513, Kofler, F.
ORCID: 0000-0003-0642-7884, Shit, S.
ORCID: 0000-0003-4435-7207, Paetzold, J. C., Loehr, T., Wiestler, B., Peiris, H., Pawar, K., Zhong, S.
ORCID: 0000-0001-6542-649X, Chen, Z.
ORCID: 0000-0002-0173-6090, Hayat, M., Egan, G., Harandi, M., Isik Polat, E.
ORCID: 0000-0002-0728-5390, Polat, G.
ORCID: 0000-0002-1499-3491, Kocyigit, A.
ORCID: 0000-0001-5003-4127, Temizel, A.
ORCID: 0000-0001-6082-2573, Tuladhar, A.
ORCID: 0000-0002-3942-2732, Tyagi, L., Souza, R., Forkert, N. D.
ORCID: 0000-0003-2556-3224, Mouches, P., Wilms, M.
ORCID: 0000-0001-8845-360X, Shambhat, V., Maurya, A., Danannavar, S. S., Kalla, R., Anand, V. K., Krishnamurthi, G., Nalawade, S., Ganesh, C., Wagner, B.
ORCID: 0000-0003-2835-986X, Reddy, D., Das, Y., Yu, F. F., Fei, B., Madhuranthakam, A. J., Maldjian, J., Singh, G., Ren, J., Zhang, W., An, N.
ORCID: 0009-0003-4255-7675, Hu, Q., Zhang, Y., Zhou, Y., Siomos, V., Tarroni, G.
ORCID: 0000-0002-0341-6138, Passerrat-Palmbach, J.
ORCID: 0000-0003-3178-9502, Rawat, A., Zizzo, G., Kadhe, S. R., Epperlein, J. P., Braghin, S.
ORCID: 0000-0001-5519-1674, Wang, Y., Kanagavelu, R., Wei, Q., Yang, Y., Liu, Y.
ORCID: 0000-0002-1590-2029, Kotowski, K., Adamski, S., Machura, B.
ORCID: 0000-0001-7953-7684, Malara, W., Zarudzki, L., Nalepa, J.
ORCID: 0000-0002-4026-1569, Shi, Y.
ORCID: 0000-0002-8721-8509, Gao, H., Avestimehr, S., Yan, Y., Akbar, A. S.
ORCID: 0000-0002-6011-7011, Kondrateva, E., Yang, H., Li, Z., Wu, H-Y., Roth, J., Saueressig, C., Milesi, A., Nguyen, Q. D., Gruenhagen, N. J., Huang, T-M., Ma, J.
ORCID: 0000-0002-9739-0855, Singh, H. S. H., Pan, N-Y., Zhang, D., Zeineldin, R. A., Futrega, M., Yuan, Y., Conte, G. M.
ORCID: 0000-0001-5926-7517, Feng, X., Pham, Q. D., Xia, Y.
ORCID: 0000-0001-9273-2847, Jiang, Z.
ORCID: 0000-0001-5713-1675, Luu, H. M., Dobko, M., Carré, A., Tuchinov, B., Mohy-ud-Din, H., Alam, S., Singh, A., Shah, N.
ORCID: 0000-0002-4429-8414, Wang, W., Sako, C.
ORCID: 0000-0003-3243-3954, Bilello, M., Ghodasara, S.
ORCID: 0000-0002-5332-3132, Mohan, S.
ORCID: 0000-0002-4025-115X, Davatzikos, C.
ORCID: 0000-0002-1025-8561, Calabrese, E.
ORCID: 0000-0002-1464-0354, Rudie, J., Villanueva-Meyer, J., Cha, S., Hess, C.
ORCID: 0000-0002-5132-5302, Mongan, J., Ingalhalikar, M., Jadhav, M.
ORCID: 0000-0002-0561-7144, Pandey, U.
ORCID: 0000-0003-3289-0901, Saini, J., Huang, R. Y., Chang, K., To, M-S., Bhardwaj, S., Chong, C., Agzarian, M.
ORCID: 0000-0002-2562-0340, Kozubek, M.
ORCID: 0000-0001-7902-589X, Lux, F.
ORCID: 0000-0002-3707-8157, Michálek, J.
ORCID: 0000-0002-5591-1894, Matula, P.
ORCID: 0000-0003-4125-1597, Ker^kovský, M.
ORCID: 0000-0003-0587-9897, Kopr^ivová, T.
ORCID: 0000-0003-3820-2647, Dostál, M.
ORCID: 0000-0003-1740-9227, Vybíhal, V.
ORCID: 0000-0002-5154-5591, Pinho, M. C., Holcomb, J., Metz, M.
ORCID: 0000-0002-1459-5741, Jain, R.
ORCID: 0000-0002-4879-0457, Lee, M. D.
ORCID: 0000-0001-8816-1076, Lui, Y. W.
ORCID: 0000-0002-9984-9164, Tiwari, P., Verma, R., Bareja, R., Yadav, I., Chen, J.
ORCID: 0000-0003-4907-0061, Kumar, N., Gusev, Y.
ORCID: 0000-0001-7371-4715, Bhuvaneshwar, K.
ORCID: 0000-0003-4015-7056, Sayah, A.
ORCID: 0000-0002-9683-3802, Bencheqroun, C., Belouali, A.
ORCID: 0000-0002-2780-2500, Madhavan, S., Colen, R. R., Kotrotsou, A., Vollmuth, P., Brugnara, G.
ORCID: 0000-0003-2461-1407, Preetha, C. J.
ORCID: 0000-0002-9875-1982, Sahm, F.
ORCID: 0000-0001-5441-1962, Bendszus, M., Wick, W.
ORCID: 0000-0002-6171-634X, Mahajan, A.
ORCID: 0000-0001-6606-6537, Balaña, C.
ORCID: 0000-0003-0771-0390, Capellades, J., Puig, J., Choi, Y. S.
ORCID: 0000-0002-6034-9912, Lee, S-K.
ORCID: 0000-0001-5646-4072, Chang, J. H.
ORCID: 0000-0003-1509-9800, Ahn, S. S.
ORCID: 0000-0002-0503-5558, Shaykh, H. F.
ORCID: 0000-0002-2690-7685, Herrera-Trujillo, A.
ORCID: 0000-0002-9775-9405, Trujillo, M., Escobar, W., Abello, A., Bernal, J.
ORCID: 0000-0003-3167-5134, Gómez, J., LaMontagne, P.
ORCID: 0000-0002-6752-8518, Marcus, D. S., Milchenko, M.
ORCID: 0000-0002-4022-1081, Nazeri, A.
ORCID: 0000-0001-6983-0641, Landman, B.
ORCID: 0000-0001-5733-2127, Ramadass, K.
ORCID: 0000-0002-4610-3860, Xu, K.
ORCID: 0000-0002-2664-255X, Chotai, S.
ORCID: 0000-0001-5981-3231, Chambless, L. B., Mistry, A., Thompson, R. C., Srinivasan, A., Bapuraj, J. R.
ORCID: 0000-0003-3933-203X, Rao, A., Wang, N.
ORCID: 0000-0001-9796-4532, Yoshiaki, O., Moritani, T., Turk, S., Lee, J., Prabhudesai, S., Garrett, J.
ORCID: 0000-0002-8152-736X, Larson, M., Jeraj, R., Li, H., Weiss, T.
ORCID: 0000-0002-5533-9429, Weller, M.
ORCID: 0000-0002-1748-174X, Bink, A.
ORCID: 0000-0002-2163-3400, Pouymayou, B.
ORCID: 0000-0002-8060-7827, Sharma, S., Tseng, T-C., Adabi, S., Xavier Falcão, A.
ORCID: 0000-0002-2914-5380, Martins, S. B.
ORCID: 0000-0002-2894-3911, Teixeira, B. C. A.
ORCID: 0000-0003-4769-6562, Sprenger, F.
ORCID: 0000-0002-1631-3517, Menotti, D.
ORCID: 0000-0003-2430-2030, Lucio, D. R., Niclou, S. P.
ORCID: 0000-0002-3417-9534, Keunen, O.
ORCID: 0000-0003-2203-7026, Hau, A-C.
ORCID: 0000-0002-4412-2355, Pelaez, E.
ORCID: 0000-0001-9355-5440, Franco-Maldonado, H.
ORCID: 0000-0002-0178-8157, Loayza, F.
ORCID: 0000-0002-6283-3679, Quevedo, S.
ORCID: 0000-0001-5585-0270, McKinley, R.
ORCID: 0000-0001-8250-6117, Slotboom, J.
ORCID: 0000-0001-5121-9852, Radojewski, P., Meier, R., Wiest, R.
ORCID: 0000-0001-7030-2045, Trenkler, J.
ORCID: 0000-0001-6060-4234, Pichler, J., Necker, G., Haunschmidt, A.
ORCID: 0000-0001-9864-5189, Meckel, S.
ORCID: 0000-0001-6468-4526, Guevara, P.
ORCID: 0000-0001-9988-400X, Torche, E., Mendoza, C.
ORCID: 0000-0002-3718-0629, Vera, F.
ORCID: 0000-0002-2196-7267, Ríos, E.
ORCID: 0000-0002-0194-9504, López, E.
ORCID: 0000-0003-3526-5271, Velastin, S. A., Choi, J., Baek, S., Kim, Y., Ismael, H., Allen, B.
ORCID: 0000-0002-4893-7105, Buatti, J. M.
ORCID: 0000-0001-8499-3721, Zampakis, P.
ORCID: 0000-0002-5104-9619, Panagiotopoulos, V.
ORCID: 0000-0001-9298-2985, Tsiganos, P.
ORCID: 0000-0002-3279-4947, Alexiou, S., Haliassos, I.
ORCID: 0000-0002-3278-0588, Zacharaki, E. I.
ORCID: 0000-0001-8228-0437, Moustakas, K.
ORCID: 0000-0001-7617-227X, Kalogeropoulou, C.
ORCID: 0000-0002-9725-1791, Kardamakis, D. M., Luo, B., Poisson, L. M.
ORCID: 0000-0002-3409-6536, Wen, N., Vallières, M.
ORCID: 0000-0001-7639-8172, Loutfi, M. A. L.
ORCID: 0009-0005-3555-0963, Fortin, D., Lepage, M.
ORCID: 0000-0002-5363-9487, Morón, F.
ORCID: 0000-0003-3745-2177, Mandel, J.
ORCID: 0000-0001-5916-5668, Shukla, G., Liem, S., Alexandre, G. S., Lombardo, J., Palmer, J. D.
ORCID: 0000-0003-3311-0129, Flanders, A. E., Dicker, A. P.
ORCID: 0000-0003-0733-3337, Ogbole, G., Oyekunle, D.
ORCID: 0000-0003-3957-0206, Odafe-Oyibotha, O., Osobu, B., Shu’aibu Hikima, M., Soneye, M., Dako, F., Dorcas, A., Murcia, D., Fu, E., Haas, R., Thompson, J. A.
ORCID: 0000-0003-2991-5194, Ormond, D. R.
ORCID: 0000-0001-7027-2915, Currie, S.
ORCID: 0000-0002-0154-6588, Fatania, K.
ORCID: 0000-0003-2421-1083, Frood, R.
ORCID: 0000-0003-2681-9922, Simpson, A. L.
ORCID: 0000-0002-4387-8417, Peoples, J. J.
ORCID: 0000-0003-0191-7446, Hu, R., Cutler, D.
ORCID: 0000-0001-7523-714X, Moraes, F. Y., Tran, A.
ORCID: 0000-0003-4596-3321, Hamghalam, M., Boss, M. A.
ORCID: 0000-0002-9492-767X, Gimpel, J.
ORCID: 0000-0003-4245-9429, Kattil Veettil, D.
ORCID: 0000-0002-0317-5477, Schmidt, K., Cimino, L., Price, C., Bialecki, B.
ORCID: 0000-0003-1835-7449, Marella, S., Apgar, C., Jakab, A.
ORCID: 0000-0001-6291-9889, Weber, M-A.
ORCID: 0000-0003-3918-8066, Colak, E., Kleesiek, J.
ORCID: 0000-0001-8686-0682, Freymann, J. B., Kirby, J. S., Maier-Hein, L., Albrecht, J., Mattson, P.
ORCID: 0000-0002-5984-238X, Karargyris, A.
ORCID: 0000-0002-1930-3410, Shah, P., Menze, B.
ORCID: 0000-0003-4136-5690, Maier-Hein, K. & Bakas, S.
ORCID: 0000-0001-8734-6482 (2025).
Towards fair decentralized benchmarking of healthcare AI algorithms with the Federated Tumor Segmentation (FeTS) challenge.
Nature Communications, 16(1),
article number 6274.
doi: 10.1038/s41467-025-60466-1
Abstract
Computational competitions are the standard for benchmarking medical image analysis algorithms, but they typically use small curated test datasets acquired at a few centers, leaving a gap to the reality of diverse multicentric patient data. To this end, the Federated Tumor Segmentation (FeTS) Challenge represents the paradigm for real-world algorithmic performance evaluation. The FeTS challenge is a competition to benchmark (i) federated learning aggregation algorithms and (ii) state-of-the-art segmentation algorithms, across multiple international sites. Weight aggregation and client selection techniques were compared using a multicentric brain tumor dataset in realistic federated learning simulations, yielding benefits for adaptive weight aggregation, and efficiency gains through client sampling. Quantitative performance evaluation of state-of-the-art segmentation algorithms on data distributed internationally across 32 institutions yielded good generalization on average, albeit the worst-case performance revealed data-specific modes of failure. Similar multi-site setups can help validate the real-world utility of healthcare AI algorithms in the future.
Publication Type: | Article |
---|---|
Additional Information: | This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. |
Subjects: | Q Science > QA Mathematics > QA75 Electronic computers. Computer science R Medicine > RA Public aspects of medicine > RA0421 Public health. Hygiene. Preventive Medicine R Medicine > RC Internal medicine > RC0254 Neoplasms. Tumors. Oncology (including Cancer) |
Departments: | School of Science & Technology School of Science & Technology > Computer Science |
SWORD Depositor: |
Available under License Creative Commons: Attribution International Public License 4.0.
Download (8MB) | Preview
Export
Downloads
Downloads per month over past year