The Codebook
Both encoder and decoder must have the same codebook.
The codebook must be useful for many images and be stored someplace.
The codebook must be designed properly to be effective.
Design requires a representative training set.
These are major drawbacks to VQ.