Design without data? A study of methodological transparency in contemporary design science
DOI:
https://doi.org/10.47909/978-9916-9331-4-5.116Keywords:
design research, research methods, bibliometrics, qualitative research, data-driven design, methodological transparencyAbstract
This study investigated the methodological landscape of contemporary design science by analyzing 7,511 articles published across 10 leading journals in the field. The objective of this study was twofold: first, to ascertain the prevalence of qualitative, quantitative, and other forms of inquiry, and second, to reflect on the implications of methodological choices within design scholarship. The utilization of OpenAlex for the collection of metadata and ChatGPT-4o for the classification of abstracts based on method-related keywords enabled the study to categorize articles as quantitative, qualitative, mixed methods, or inconclusive. The findings indicated that a mere 5.8% of the articles employed quantitative methods, while 14.28% utilized qualitative methods. Notably, 77.78% of the articles exhibited an absence of clear methodological signals, indicating a deficiency in methodological transparency. The application of topic modeling to inconclusive works revealed a preponderance of research that was conceptual, practice-based, or speculative in nature. These findings lent further credence to ongoing discourse regarding the dearth of methodological transparency and the underutilization of empirical strategies in design. The study’s conclusion asserted that enhancing methodological articulation and establishing shared standards fortified the credibility and interdisciplinary recognition of design as a scientific field.
Downloads
References
Babbie, E. (2016). The practice of social research. Cengage Learning.
Barrett, E., & Bolt, B. (Eds.). (2010). Practice as research: Approaches to creative arts enquiry. I.B. Tauris.
Biggs, M. A. R., & Büchler, D. (2007). Rigor and practice-based research. Design Issues, 23(3), 62–69. https://doi.org/10.1162/desi.2007.23.3.62
Borgdorff, H. (2012). The conflict of the faculties: Perspectives on artistic research and academia. Leiden University Press.
Cole, N. L., Ulpts, S., Bochynska, A., Kormann, E., Good, M., Leitner, B., & Ross-Hellauer, T. (2024). Reproducibility and replicability of qualitative research: An integrative review of concepts, barriers and enablers. Center for Open Science. https://doi.org/10.31222/osf.io/n5zkw
Creswell, J. W., & Clark, V. L. P. (2018). Designing and conducting mixed methods research. SAGE Publications, Inc.
Creswell, J. W., & Creswell, J. D. (2018). Research design: Qualitative, quantitative, and mixed methods approaches. SAGE Publications, Inc.
Cross, N. (2001). Designerly ways of knowing: Design discipline versus design science. Design Issues, 17(3), 49–55. https://doi.org/10.1162/074793601750357196
Denzin, N. K., & Lincoln, Y. S. (Eds.). (2011). The SAGE handbook of qualitative research. SAGE Publications, Inc.
Dunne, A., & Raby, F. (2013). Speculative everything: Design, fiction, and social dreaming. The MIT Press.
Ebel, P., Gülle, K. J., Lingenfelder, C., & Vogelsang, A. (2023). Exploring millions of user interactions with ICEBOAT: Big data analytics for automotive user interfaces. In AutomotiveUI '23: Proceedings of the 15th International Conference on Automotive User Interfaces and Interactive Vehicular Applications (pp. 81–92). https://doi.org/10.1145/3580585.3607158
Escudero-Mancebo, D., Fernández-Villalobos, N., Martín-Llorente, Ó., & Martínez-Monés, A. (2023). Research methods in engineering design: A synthesis of recent studies using a systematic literature review. Research in Engineering Design, 34(2), 221–256. https://doi.org/10.1007/s00163-022-00406-y
Frayling, C. (1993). Research in art and design. Royal College of Art Research Papers, 1(1). https://researchonline.rca.ac.uk/384/
Greene, J. C., Caracelli, V. J., & Graham, W. F. (1989). Toward a conceptual framework for mixed-method evaluation designs. Educational Evaluation and Policy Analysis, 11(3), 255–274. https://doi.org/10.3102/01623737011003255
Harris, J. K., Combs, T. B., Johnson, K. J., Carothers, B. J., Luke, D. A., & Wang, X. (2019). Three changes public health scientists can make to help build a culture of reproducible research. Public Health Reports, 134(2), 109–111. https://doi.org/10.1177/0033354918821076
Hilbert, M., & López, P. (2011). The world’s technological capacity to store, communicate, and compute information. Science, 332(6025), 60–65. https://doi.org/10.1126/science.1200970
Johannessen, L. K., Keitsch, M. M., & Pettersen, I. N. (2019). Speculative and critical design—Features, methods, and practices. Proceedings of the Design Society: International Conference on Engineering Design, 1(1), 1623–1632. https://doi.org/10.1017/dsi.2019.168
Mejeh, M., Hagenauer, G., & Gläser-Zikuda, M. (2023). Mixed methods research on learning and instruction—Meeting the challenges of multiple perspectives and levels within a complex field. Forum Qualitative Sozialforschung/Forum: Qualitative Social Research, 24(1), Article 1. https://doi.org/10.17169/fqs-24.1.3989
Oppermann, M., & Munzner, T. (2020). Data-first visualization design studies. In 2020 IEEE Workshop on Evaluation and Beyond—Methodological Approaches to Visualization (BELIV) (pp. 74–80). https://doi.org/10.1109/beliv51497.2020.00016
Pilcher, N., & Cortazzi, M. (2024). “Qualitative” and “quantitative” methods and approaches across subject fields: Implications for research values, assumptions, and practices. Quality & Quantity, 58(3), 2357–2387. https://doi.org/10.1007/s11135-023-01734-4
Pinto, A. L., Teixeira, J. M., & Velasco, J. L. (2025). Understanding design metrics: A theoretical model for application and evaluation. AWARI, 6, 1–10. https://doi.org/10.47909/awari.833
Quiñones-Gómez, J. C., Mor, E., & Chacón, J. (2025). Data-driven design in the design process: A systematic literature review on challenges and opportunities. International Journal of Human–Computer Interaction, 41(4), 2227–2252. https://doi.org/10.1080/10447318.2024.2318060
Teixeira, J. M., & Velasco, J. L. (2024). Design para Negócios Digitais: Qualitativo x Quantitativo [Google Presentation slides]. Undergraduate Program of Design—University of Santa Catarina. https://docs.google.com/presentation/d/1ESCASgoeOIqYqiW0gJmxeqceMv-e1JuHcsxc_uAB2WM/edit?usp=sharing
Tenny, S., Brannan, J. M., & Brannan, G. D. (2025). Qualitative study. In StatPearls. StatPearls Publishing. http://www.ncbi.nlm.nih.gov/books/NBK470395/
Thelwall, M., & Nevill, T. (2021). Is research with qualitative data more prevalent and impactful now? Interviews, case studies, focus groups and ethnographies. Library & Information Science Research, 43(2), Article 101094. https://doi.org/10.1016/j.lisr.2021.101094
Timperley, C. S., Herckis, L., Goues, C. L., & Hilton, M. (2021). Understanding and improving artifact sharing in software engineering research. Empirical Software Engineering, 26(4). https://doi.org/10.1007/s10664-021-09973-5
Toh, W. X., Hashemi Farzaneh, H., Kaiser, M. K., & Lindemann, U. (2014). Analysis of empirical studies in design research. In DS 77: Proceedings of the DESIGN 2014 13th International Design Conference (pp. 59–70).
Tong, A., Sainsbury, P., & Craig, J. (2007). Consolidated criteria for reporting qualitative research (COREQ): A 32-item checklist for interviews and focus groups. International Journal for Quality in Health Care, 19(6), 349–357. https://doi.org/10.1093/intqhc/mzm042
Van Turnhout, K., Bennis, A., Craenmehr, S., Holwerda, R., Jacobs, M., Niels, R., Zaad, L., Hoppenbrouwers, S., Lenior, D., & Bakker, R. (2014). Design patterns for mixed-method research in HCI. In NordiCHI '14. Proceedings of the 8th Nordic Conference on Human-Computer Interaction: Fun, Fast, Foundational. https://doi.org/10.13140/2.1.4701.2643
Velasco, J. L., Teixeira, J. M., & Pinto, A. L. (2025). Validating personas for better communication: A structured model for low-resource contexts. Iberoamerican Journal of Science Measurement and Communication, 5(3), Article 3. https://doi.org/10.47909/ijsmc.236
Walny, J., Frisson, C., West, M., Kosminsky, D., Knudsen, S., Carpendale, S., & Willett, W. (2020). Data changes everything: Challenges and opportunities in data visualization design handoff. IEEE Transactions on Visualization and Computer Graphics, 26(1), 12–22. https://doi.org/10.1109/tvcg.2019.2934538
Weichbroth, P. (2019). A mixed-methods measurement and evaluation methodology for mobile application usability studies. Annals of Computer Science and Information Systems, 20, 101–106. https://doi.org/10.15439/2019F299
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2025 Jefferson Lewis Velasco, Adilson Luiz Pinto, Júlio Monteiro Teixeira
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
This is an open access article distributed under the terms of the Creative Commons Attribution-NonCommercial 4.0 International License (CC BY-NC 4.0) which permits copying and redistributing the material in any medium or format, adapting, transforming and building upon the material as long as the license terms are followed.