Understanding of a complex scene goes way beyond top-down perception. When people operate in a natural scene, they can detect and recognize objects and relations using context, they can predict how objects and people will move next, and even reason why they behave as they do. We develop algorithms that allow smart agents to learn how to reason about their environment. Below are several recent studied that we published about this topic, including zero-shot learning from reasoning-by-elimination, and recognizing new combinations of known components.