Any excess ingestion of halogen ions is potentially neurologically toxic.
The fluoride ions in fluoroapatite are bound in the crystal lattice of the apatite, more so than the original hydroxide ions they replace, (which is why it is a good thing)
so fluoroapatite has no neurotoxic potential whatsoever. (unless lets say you were to grind up the teeth of a few hundred people and eat it... yuck...)
The hexametaphosphate ring that forms the base of the apatite lattice can be more of a threat in another form, accumulated phosphates in joints has been indicated in some forms of arthritis by nucleating apatite formation.
I would not be concerned about fluoride concentrations in drinking water and toothpaste and fluoride rinses. The health benefits of preserving your teeth far outweighs any miniscule neurotoxicity of ingestion of trace fluoride concentrations. (trust me, I have bad teeth and they can make you very sick, deathly ill as a matter of fact...)
Personally, I'm a little more concerned by the brominated vegetable oils present in soft drinks used to keep flavouring oils in suspension. (any pop that is cloudy) It is entirely possible that halogenated oils and fatty acids could have significant neurological effects if accumulated in any quantity. Think of the lipid bilayer of cell membranes and the nerve cell axons and synapse faces in particular, the inclusion of those big "fluffy" halogen atoms could significantly hamper the normal functions of the ion pumps in said membranes. Actually considering this, I can envision certain toxicity scenarios involving excessive halogen ingestion with the presence of fats and heavy metals, however the ionic form of the halogens is relatively unreactive and quite unlikely to halogenate the organics in the digestion system. (otherwise the strong hydrochloric acid concentration in our stomachs would be a problem).
Have a look at group seven of the periodic table and you'll see the halogens that are relatively abundandant and sometimes critical in biological systems.
Take iodine as an example, it is necessary for proper thyriod function (thyroxin contains two iodine atoms if I recall correctly).
Chloride of course is critical to our function especially our neurological function and overall osmotic balance across innumerable membranes.
Any of the halogens in their diatomic gas phase are significantly toxic of course, as on contact with water they form their corresponding acids. (e.g. mustard gas)
Astatine, as I recall has a number of radioacticve isotopes so I would certainly steer clear of that one...
But I don't think it occurs in nature. However the stable istotopes and isotopic abundances of the other halogens are invaluable in chemical research and hence medical diagnostics.