The ARR Summoner quests seem to contradict that. A person may have an affinity for a particular crystal, but it seems like the power and knowledge can be siphoned out regardless.
V'aleera's Wiki - https://wiki.ffxiv-roleplayers.com/pages...eera_Lhuil
V'aleera's Tumblr - valeeralhuil.tumblr.com
V'aleera's Tumblr - valeeralhuil.tumblr.com