Factor Objects do not yet support empty matrices.