The Portuguese giants, who won the old European Cup twice in the 1960s, were knocked out by goals from Dimitrios Giannoulis and Andrija Zivkovic.
Benfica finished second in the Portuguese top division but had a tough draw at PAOK, who finished second in the Greek Super League.
Having seen the domestic league and cup double go to Porto last term.
Benfica missed a string of chances against PAOK in a one-sided first half as they enjoyed 72% of possession.
They were punished after the break as Dimitrios Giannoulis poked in the opener from close range in the 63rd minute after a one-two with Chuba Akpom and substitute Zivkovic made it 2-0 in the 75th with a sweet shot inside the near post.
Rafa Silva pulled one back with a stoppage time header but it came too late to avoid an embarrassing defeat which will pile pressure on manager Jorge Jesus, who returned to Benfica during the close season for a second spell in charge.
Having been in charge from 2009 to 2015, Jesus went on to win the elusive double of the Brazilian league title and Copa Libertadores with Flamengo in 2019 but now faces a mammoth task to get Benfica back to winning ways.
They will not have a chance to overcome the defeat because the third qualifying round is being contested with single-leg ties.
Benfica will begin their 2020/21 Primeira Liga campaign with an away trip to F.C. Famalicão on Friday.
