TRichView.com

Is it possible to have an image included by reference to a physical file in the RVF without the image content being included in the RVF?

It is possible, but only for all images in the document.
Exclude rvfoSavePicturesBody from RVFOptions, and images will not be saved inside files. When loading file, OnRVFPictureNeeded event occurs.

Does this also work if the content of the TRichViewEdit is RTF?

No, for RTF it's a bit more difficult.
RTF files can contain references to external pictures via INCLUDEPICTURE field.
You need to implement saving pictures as this field type. It can be done using OnSaveItemToFile event.
The example can be found in the demo projects, in Demos\DelphiUnicode\Assorted\Graphics\SharedImages\ demo.