What is the key characteristic of a cost function in machine learning?

The key characteristic of a cost function in machine learning is that it measures overall prediction error during training. This function quantifies how well the model's predictions align with the actual outcomes, providing a numerical value that represents the model's performance. The cost function is crucial during the training phase as it guides the optimization process; the goal is to minimize this function to enhance the accuracy of the model. By evaluating the predictions against the true values, the cost function helps inform adjustments to the model, allowing it to learn from errors and improve over time.

Understanding this concept is vital because it underpins the entire training process in machine learning models. During training, the model iteratively adjusts its parameters based on the feedback provided by the cost function, refining its predictions towards greater accuracy. This contrasts with other aspects of the learning process that may not focus primarily on error measurement or involve other stages of the model's lifecycle.