The most critical factor in the validity and reliability of any machine learning (ML) model is the quality of the data used for its training. The principle of "Garbage In, Garbage Out" (GIGO) is fundamental here. If the training data is sourced from an unreliable provider, it may be inaccurate, biased, or incomplete. This foundational flaw will be learned and amplified by the model, rendering its predictions untrustworthy, regardless of how well it is designed or tested. An IS auditor's primary concern is to ensure the integrity and reliability of the system's foundation, which for an ML model, is its training data.

A. A decrease in accuracy on a different population indicates a generalization problem, which is a concern, but it is a performance symptom, not a foundational flaw like unreliable source data.

C. The use of open-source programming languages (e.g., Python, R) is a standard, accepted, and prevalent practice in ML development and does not represent an inherent risk.

D. Testing a model with a holdout dataset drawn from the same population as the training data is a correct and standard procedure for validating model performance.

1. ISACA

"Auditing Artificial Intelligence" White Paper

2019: Page 13

under the "Data" section

states

"The quality of the data used to train the AI system is critical to its performance... The GIGO (garbage in

garbage out) principle applies. If the data used to train the system is biased

incomplete or of poor quality

the system will not perform as expected." This directly identifies data quality and source as a critical audit concern.

2. Munoko

I.

Brown-Liburd

H. and Vasarhelyi

M. "Auditing machine learning models: A framework for auditors and data scientists." Managerial Auditing Journal

Vol. 35 No. 5

2020

pp. 649-667: In Section 3.1

"Data Understanding

" the framework emphasizes that a key audit step is to "assess the reliability of the data source" and evaluate the data collection process to ensure data quality. This highlights the primacy of the data source in an ML audit. (DOI: https://doi.org/10.1108/MAJ-08-2019-2382)

3. ISACA

"A Framework for Auditing AI

" ISACA Journal

Volume 6

2019: The article outlines an audit approach where a primary step is to "understand and assess the data used by the AI." It specifies that the "auditor should verify the source

lineage and quality of the data

" reinforcing that the data's origin is a top-priority audit point.