Mod is short for "MODification".
It's a player-developed-and-player sponsored project that is dedicated to change the game. Either to expand the game, or to make the game go in a new direction.
They are generally safe. (since if worse comes to worst, you can always resintall the game) and the notable ones are often as popular (or MORE popular) than the original. (i.e. CounterStrike, the defacto king of FPS started as a mod to Half-Life)
In this case, Dominion War is developed by a player called "the pelican". you can get more info from his site.
It's listed in the race name because it was PLANNED to be included in the final version, but the time/budget ran out before they can be properly included. (same with Cardassians)