What reinforcement learning algorithm learns the value of actions based solely on states, without needing a model of the environment?

The reinforcement learning algorithm that learns the value of actions based solely on states, without needing a model of the environment, is indeed Q-learning. This algorithm falls under the category of model-free methods, meaning it does not require a prior understanding of the dynamics of the environment it operates in. Instead, Q-learning focuses on learning a value function that can be used to make decisions on which actions to take in different states.

Q-learning works by estimating the optimal action-value function through exploration and exploitation. It updates the Q-values based on the rewards received after taking actions and the maximum estimated future rewards, allowing it to learn from experiences directly. This helps the agent to improve its policy by gradually refining its understanding of the relationship between actions and rewards in various states.

In contrast, while Deep Q-Networks utilize neural networks to approximate the Q-values, they still follow the principles of Q-learning. SARSA, on the other hand, is an on-policy algorithm, meaning it also requires a model of the environment to evaluate actions based on the policy currently being followed. The Monte Carlo Method calculates value estimates based on averaged returns over episodes but does not directly learn state-action values without explicitly considering the actions taken. This makes Q-learning distinctive in its approach to being a