While this practice seems unthinkable today, across the Mediterranean, ancient Greek and Roman structures were salvaged for building materials in subsequent centuries. Given the prevalence of the Catholic Church in the Middle Ages, pagan buildings were at best a curiosity. The Parthenon in Rome is said to have only survived because it was converted into a church.
The Pentelic marble used to construct ancient Athens proved to be too alluring for Byzantine builders. You can see blatant example of stone theft in the piece-meal construction of the 13th century Panagia Gorgoepikoos Church in Athens. While the materials were stolen, the care with which pieces were selected and incorporated suggests some appreciation for classical art.