Here are my final beta tests. I haven't figured it all out yet. I'm still confident there is room for improvement.
As you can see, they look awful from F1 view. I'll explain why below.
Here's what I know:
The nebulas use the skybgnd (pink) and skybgnd-mono (white) textures in the Textures folder to create the nebulas. The original images are 256x256 and get blown up a lot when mapped onto the game screen. This is part of the reason they look terrible.
The nebula "model" is an internal model, like the star model, and can't be changed or manipulated without the source code and/or other tools. From what I can tell, the model is a cylinder with the ship at the center. Zoom and Tilt do not affect your perspective of the nebula, i.e. the cylinder pitches up and down with you and changing your optical distance from the ship leaves the rest of the screen unchanged. Panning right and left will change your view around the inside of the cylinder. As a result only the middle part of the image you place on the model will ever be seen. More testing needs to be done on this as I haven't found the "perfect " fit when it comes to image size yet.
The texture map is a wrap around the cylinder, like making a tube of a drawing you made and placing it around your ship like scenery. Only this scenery is partially transparent, and there are two cylinders that overlap each other, one wrapped around the other so you have an inner and an outer cylinder. The outer cylinder is of course larger than the inner one. This is why you will occasionally see multiple images like you are drunk and also why F1 view looks so terrible. One image is overlapping the other, but is slightly larger. Because of this a generic cloud type of shot is best, since any recognizable landmarks will be seen many times.
Taldren solved this problem by making the original graphic a wrap around, i.e. each side transitions into the adjacent side of another copy so that it looks like a continuous particle field, and by overlapping the cylinders, made the effect random enough so you don't notice the pattern while fighting. If you look just at the nebulas you can see it pretty clearly. This means that any images you use must wrap at the sides or you will have a giant seam running from the top to the bottom of your screen every time you make a full rotation. The program maps several iterations of the image around the cylinder, meaning that smaller images will repeat several times before a complete rotation back to start. I'm not exactly sure how many it tries to force in there, but it does seem less the wider the image you try to place.
I have no idea how big the canvas is. Wider images get compressed some, but I haven't found a goldilocks zone yet. From what I have seen anything below about 1280 in height seems to start to look bad, but I have had images pushing 5000 in width and it still took it. Like I stated before, the top and bottom seem to get cut off no matter what the size, at least with images over 1200 in height. I didn't really experiment with smaller since they look terrible at those sizes. The images I used for the shots are 4096x1660, but I'll bet they could be smaller, at least in width, and still look the same. After a certain point it doesn't seem to make any appreciable difference how big it is.
Quality of the image in game is tricky. I went through about 30-35 images and some looked great as static shots, but rendered in the game like someone barfed on my monitor (I have an LED monitor that sees a ton of colors and it is very unforgiving). Others, like the ones I posted, seemed to hold up better. All I can say is try everything because you never know. It would be so cool to be able to disable the space backgrounds when in a nebula or have the background pointed to an empty model when a nebula is detected. They tend to interfere, but some actually make the shot better, so there is give and take on that point.
The best course would be for someone with the skill to make a new generic nebula graphic that solves all of these problems and still looks good. That won't be me though. I can only hack stuff together.