Normalization failure

@heno Heno thanks for reporting this. Could you upload the small data set to https://upload.astropixelprocessor.com/ using username upload4 and password the same so Mabula can have a look? Please create a folder called heno_master_flat_bug and put the files in that directory. Many thanks in advance.

Wouter

I have uploaded all the files in use during these sessions except only 5 light frames from (of 400) each camera/telescope and only 5 of the (30) flat frames that created the flat frame master for the ASI1600MM camera.
I accidentally uploaded 4 light files to the /AstroPixelProcessorUpload area. These can be deleted as I have reloaded them to the heno_master_flat_bug folder.
Please let me know if you need anything else. Hope you are able to figure this one out.

Helge.

Thanks Helge. We will have a look at the files as soon as possible. And don’t worry about the files outside of your directory. We will clean them out. 

Wouter

Not sure if Mabula have had time to look at this, but I think I now know what is going on.
Have been testing a little myself today and find that this do not happen if I use only the b/w files from the ASI1600MM alone and leave the colored set of files out. When I do that, APP correctly generates a b/w master flat. But if I include the colored files and have the Force CFA checked APP somehow gets its wires crossed and generate a "colored" flat as describe in the first post. It also generates a colored MD I just noticed. But the MDF is b/w. Funny thing is, when I open them in any other program they seem to be greyscale.

So to cut it short.
The problem seem to be in the creation of the b/w MF and MD when the force CFA is checked for the colored set of images.
I just uploaded the finished image generated by APP. Open it in APP and zoom in to 200% and you see all the colored, diagonal stripes.
Hope this helps.

Helge

Helge,

Force CFA indeed attempts to demosaic ALL images because it is intended to be used with color images that do not have the CFA pattern in the FITS header. So it makes sense that APP tries to demosaic the mono images. Do your color images have the CFA pattern in the FITS header? If yes then please disable force CFA and load the mono and color images as you did before. Then normalization should be done without an error.

Wouter

@wvreeven
The colour camera is an ASI294MC and it requires Force CFA to be checked to produce color images. This I have tried many times. The essence of what you're saying is that if your color camera need Force CFA to be checked, you cannot include color and mono images in the same integration.
That is a serious drawback, I think. I would suggest that Force CFA check mark is moved to where a session is spesified so that APP can be told which lightframes/session nead CFA to be forced and which do not. 
However, that is only part of the problem. I have run a great number of tests using only the monocrome data from ASI1600MM. I have had some very strange integration result. Conclusion is that this is a numbers game, so to speak. The three enclosd images are 10, 100 and 420 frames integrated with exactly the same settings, zoomed to 400%. You can clearly see that something goes sideways when the number of frames increase. If I am doing something wrong, I certainly have no clue what it is. All settings are default settings.

I you look at the integration file I uploaded, you see the same vertical stripes. If I integrate mono and color together, these stripes are colored.
Right now I cannot integrate my monocrome files, with or without the colored ones. I'm stuck.

Helge 

@heno

With more and more lights, the background noise drops more and more. This is plainly visible by the fact that more and more faint stars become visible. With a lower background level, artifacts in the lights or calibration files become visible as well.

How do you take the lights? Do you apply dithering? If not, is there perhaps a drift in the guiding that may cause the stripy noise pattern? Note that applying correct dithering is essential for fighting low noise artifacts like this.

As for the Foce CFA issue: I'll communicate this to Mabula. For now please consider integrating the mono and color lights separately and then integrating the two resulting images again.

@wvreeven
I don't think I will immediately accept that explanation. Yes, more stars become visible, but these artifacts seem to develope from nothing. There is nothing visible in either light frames or calibration files.
With respect to dithering, I can not say 100% sure, but normally my lights are dithered. Saying that, I have taken dozens of images without dithering and I have never seen anything like this. I don't think dithering is or lack thereof is the reason.
I'm running a 420 frame test now with just the light frames, no calibration, to exclude that the calibration frames are causing this. We'll see tomorrow.

Helge

So I have established one thing for sure, this is NOT an APP issue. I get exactly (far as I can tell) the same results in DSS. 
So what is going on? 
I am convinced it is neither dithering (or lack of) nor drift that is causing this. This because the color frames and the mono frames were shot at the same time using a dual scope setup. If it was any of the above causing this, it would have been visible in both image sets. It is not.
Flexure between the scopes is also unlikely since the guide scope is mounted on the mono imaging train. Then the color images should have the artefacts.
I also have established that the artifacts are independant of the use of calibration frames, being darks, flats, darkflats or any combination or none.
The mono frames were shot with an Altair 80mm APO, a Baader Contrast booster filter and ASI1600MM cool using NINA software.
I have also used the ASI1600 with a SW Esprit 120ED and Baader LRGBHa filters using SGP software. These integrations show no signs of artifacts, but I only have about 30 of each color. It should still be noticeable if it was there. This sort of acquit the camera I think, leaving me with the scope, the Booster filter and the imaging software.  
Any suggestion (cause or how to solve) in this matter will be highly appreciated.

Helge

So I think I figured out what is going on (not 100% sure) and it has to do with dithering, but not in the way anybody, at least not me, would think.
As I mentioned above, these image runs are captured with a dual setup using NINA software, the 1.11 development version.
I had set it up to use synchronized dithering (when dithering the main scope the other pause imaging) every 5th frame (as I remember). This does not work. In fact sync dithering does not work at all in the development version according to the guys at Discord. So I have been dithering the main/color imaging train while the monocrome imaging train was imaging. Every 5th frame is then corrupt. I'm guessing that this is the cause of the stripes.  But I'll be happy to hear other suggestions. 
You could say that NINA was the cause, but only because I did not know how to use it.

Helge

So it probably indeed is a failure of dithering in that case, like Wouter mentioned that is indeed usually the reason for getting rid of these artefacts.

@heno If I understand it correctly, your Altair 80 mm APO has a focal length of 480 mm and the Esprit 120ED 840 mm, which is almost double the focal length of the Altair APO. If you dither with SGP then the dithering in number of pixels with the Altair APO will be almost twice as small as  with the Esprit. Proper dithering involves dithering over enough pixels so to me it looks like you are not dithering enough with the Altair APO.

@wvreeven
No, I'm not using the Esprit and the Altair together. I'm using a RASA8 (400mm) together with the Altair (480mm). The problem was not that I have not dithered, or not dithered enough pixels. The problem was that I was  dithering the RASA train while the Altair train was imaging. They were not synced as I thought they were. So the dither on the Altair train happened in the middle of a frame. That of course will cause trouble. 

Helge

@heno You mentioned the Esprit in one of your previous comments and this is the first time you mention the RASA. Hence my confusion. In any case, you seem to be getting down to the cause of the artifacts in the mono images.

@wvreeven
Sorry for the confusion, not intended. 
At least I hope I found the reason for the artifacts. I will of course test this at first opportunity, whenever this might be. The rain is pouring. 😥