A decision tree is a machine learning model that makes predictions by asking a series of simple yes-or-no questions about the input data, structured like a flowchart, until it arrives at a final answer.
How a Tree Is Built
Starting from a root node containing all the training data, the algorithm searches for the question that best splits the data into groups that are as pure as possible, meaning each resulting group leans strongly toward one outcome. This splitting process repeats at each new branch, creating deeper levels of questions, until the branches reach a stopping point, such as a maximum depth or a minimum group size.
Strengths and Weaknesses
- Strength: highly interpretable, since you can trace the exact path of questions that led to a prediction.
- Strength: requires little data preprocessing compared to many other algorithms.
- Weakness: a single deep tree can overfit easily, memorising quirks of the training data.
- Weakness: small changes in the data can produce a very different tree structure.
Because a single tree can be unstable, practitioners often combine many trees together into an ensemble, such as a random forest or a gradient boosted tree model, which trades away some interpretability in exchange for noticeably better and more reliable accuracy.