Most sleep researchers recommend cutting off caffeine at least six hours before your intended sleep time. For people who are caffeine-sensitive, even morning coffee can affect sleep quality. If you struggle with sleep and drink caffeine after noon, this is worth experimenting with first.
Caffeine blocks adenosine receptors, the chemical that creates sleep pressure. It does not eliminate adenosine. The pressure builds up behind the block and can cause a rebound feeling of fatigue when caffeine wears off.