Normalization failu...
 
Share:
Notifications
Clear all

Normalization failure  

  RSS

 Heno
(@heno)
Red Giant Customer
Joined: 3 years ago
Posts: 64
December 3, 2020 20:04  

I think I may have come across a bug in APP. I'm having a monster stack of 820 lights from two different telescopes and two different cameras, one color, one monocrome. For all the following runs, force CFA and flip x/y descriotors are checked and same optical instrument is unchecked. 
So from the beginning, Run1:
Loaded all lights and individual calibration frames (darks, flats, darkflats and existing BPM) into two sessions. Started the process and when I went to bed it was working on integrations, no error messages of any kind. Unfortunately my hard disk run out of space during the night, not APP's fault.
So this morning I cleaned it up and restarted the process (run2) only this time I used the already created master frames. Nothing else changed. Everything works fine up to normalization of the monocrome frames, session2. (This takes many hours). Then I have this error:

FlatFieldCalibrationError

If I understand this message correctly, APP suggests that the light frames from the ASI1600MM are colored. They certainly are not.
But there was no other alternative then to cancel normalization. I shut down APP to start from scratch in the next attempt, just in case. Run 3:
This time I loaded only 5 light frames of each kind to speed up the process, all the created masters except the monocrome flat frames which I loaded as individual flats. Process went straight through to integration, no error messages. Normalization worked as it should.
Just to be 100% sure I ran it again, including the new master flat, not the individual flats, and the normalization failed.
I guess this one is for @mabula-admin.


ReplyQuote
(@wvreeven)
Black Hole Admin
Joined: 2 years ago
Posts: 484
December 3, 2020 20:56  

@heno Heno thanks for reporting this. Could you upload the small data set to https://upload.astropixelprocessor.com/ using username upload4 and password the same so Mabula can have a look? Please create a folder called heno_master_flat_bug and put the files in that directory. Many thanks in advance.

Wouter


ReplyQuote
 Heno
(@heno)
Red Giant Customer
Joined: 3 years ago
Posts: 64
December 5, 2020 11:00  

I have uploaded all the files in use during these sessions except only 5 light frames from (of 400) each camera/telescope and only 5 of the (30) flat frames that created the flat frame master for the ASI1600MM camera.
I accidentally uploaded 4 light files to the /AstroPixelProcessorUpload area. These can be deleted as I have reloaded them to the heno_master_flat_bug folder.
Please let me know if you need anything else. Hope you are able to figure this one out.

Helge.


ReplyQuote
(@wvreeven)
Black Hole Admin
Joined: 2 years ago
Posts: 484
December 5, 2020 12:01  

Thanks Helge. We will have a look at the files as soon as possible. And don’t worry about the files outside of your directory. We will clean them out. 

Wouter


ReplyQuote
 Heno
(@heno)
Red Giant Customer
Joined: 3 years ago
Posts: 64
December 7, 2020 12:56  

Not sure if Mabula have had time to look at this, but I think I now know what is going on.
Have been testing a little myself today and find that this do not happen if I use only the b/w files from the ASI1600MM alone and leave the colored set of files out. When I do that, APP correctly generates a b/w master flat. But if I include the colored files and have the Force CFA checked APP somehow gets its wires crossed and generate a "colored" flat as describe in the first post. It also generates a colored MD I just noticed. But the MDF is b/w. Funny thing is, when I open them in any other program they seem to be greyscale.

So to cut it short.
The problem seem to be in the creation of the b/w MF and MD when the force CFA is checked for the colored set of images.
I just uploaded the finished image generated by APP. Open it in APP and zoom in to 200% and you see all the colored, diagonal stripes.
Hope this helps.

Helge


ReplyQuote
(@wvreeven)
Black Hole Admin
Joined: 2 years ago
Posts: 484
December 7, 2020 13:37  

Helge,

Force CFA indeed attempts to demosaic ALL images because it is intended to be used with color images that do not have the CFA pattern in the FITS header. So it makes sense that APP tries to demosaic the mono images. Do your color images have the CFA pattern in the FITS header? If yes then please disable force CFA and load the mono and color images as you did before. Then normalization should be done without an error.

Wouter


ReplyQuote
 Heno
(@heno)
Red Giant Customer
Joined: 3 years ago
Posts: 64
December 13, 2020 15:38  

@wvreeven
The colour camera is an ASI294MC and it requires Force CFA to be checked to produce color images. This I have tried many times. The essence of what you're saying is that if your color camera need Force CFA to be checked, you cannot include color and mono images in the same integration.
That is a serious drawback, I think. I would suggest that Force CFA check mark is moved to where a session is spesified so that APP can be told which lightframes/session nead CFA to be forced and which do not. 
However, that is only part of the problem. I have run a great number of tests using only the monocrome data from ASI1600MM. I have had some very strange integration result. Conclusion is that this is a numbers game, so to speak. The three enclosd images are 10, 100 and 420 frames integrated with exactly the same settings, zoomed to 400%. You can clearly see that something goes sideways when the number of frames increase. If I am doing something wrong, I certainly have no clue what it is. All settings are default settings.

10 lightframes
100 lightframes
420 lightframes

I you look at the integration file I uploaded, you see the same vertical stripes. If I integrate mono and color together, these stripes are colored.
Right now I cannot integrate my monocrome files, with or without the colored ones. I'm stuck.

Helge 


ReplyQuote
(@wvreeven)
Black Hole Admin
Joined: 2 years ago
Posts: 484
December 13, 2020 16:15  

@heno

With more and more lights, the background noise drops more and more. This is plainly visible by the fact that more and more faint stars become visible. With a lower background level, artifacts in the lights or calibration files become visible as well.

How do you take the lights? Do you apply dithering? If not, is there perhaps a drift in the guiding that may cause the stripy noise pattern? Note that applying correct dithering is essential for fighting low noise artifacts like this.

As for the Foce CFA issue: I'll communicate this to Mabula. For now please consider integrating the mono and color lights separately and then integrating the two resulting images again.


ReplyQuote
 Heno
(@heno)
Red Giant Customer
Joined: 3 years ago
Posts: 64
December 13, 2020 19:40  

@wvreeven
I don't think I will immediately accept that explanation. Yes, more stars become visible, but these artifacts seem to develope from nothing. There is nothing visible in either light frames or calibration files.
With respect to dithering, I can not say 100% sure, but normally my lights are dithered. Saying that, I have taken dozens of images without dithering and I have never seen anything like this. I don't think dithering is or lack thereof is the reason.
I'm running a 420 frame test now with just the light frames, no calibration, to exclude that the calibration frames are causing this. We'll see tomorrow.

Helge


ReplyQuote
 Heno
(@heno)
Red Giant Customer
Joined: 3 years ago
Posts: 64
December 14, 2020 15:59  

So I have established one thing for sure, this is NOT an APP issue. I get exactly (far as I can tell) the same results in DSS. 
So what is going on? 
I am convinced it is neither dithering (or lack of) nor drift that is causing this. This because the color frames and the mono frames were shot at the same time using a dual scope setup. If it was any of the above causing this, it would have been visible in both image sets. It is not.
Flexure between the scopes is also unlikely since the guide scope is mounted on the mono imaging train. Then the color images should have the artefacts.
I also have established that the artifacts are independant of the use of calibration frames, being darks, flats, darkflats or any combination or none.
The mono frames were shot with an Altair 80mm APO, a Baader Contrast booster filter and ASI1600MM cool using NINA software.
I have also used the ASI1600 with a SW Esprit 120ED and Baader LRGBHa filters using SGP software. These integrations show no signs of artifacts, but I only have about 30 of each color. It should still be noticeable if it was there. This sort of acquit the camera I think, leaving me with the scope, the Booster filter and the imaging software.  
Any suggestion (cause or how to solve) in this matter will be highly appreciated.

Helge

This post was modified 1 month ago by Heno

ReplyQuote
 Heno
(@heno)
Red Giant Customer
Joined: 3 years ago
Posts: 64
December 14, 2020 22:02  

So I think I figured out what is going on (not 100% sure) and it has to do with dithering, but not in the way anybody, at least not me, would think.
As I mentioned above, these image runs are captured with a dual setup using NINA software, the 1.11 development version.
I had set it up to use synchronized dithering (when dithering the main scope the other pause imaging) every 5th frame (as I remember). This does not work. In fact sync dithering does not work at all in the development version according to the guys at Discord. So I have been dithering the main/color imaging train while the monocrome imaging train was imaging. Every 5th frame is then corrupt. I'm guessing that this is the cause of the stripes.  But I'll be happy to hear other suggestions. 
You could say that NINA was the cause, but only because I did not know how to use it.

Helge


ReplyQuote
(@vincent-mod)
Universe Admin
Joined: 4 years ago
Posts: 2909
December 15, 2020 10:31  

So it probably indeed is a failure of dithering in that case, like Wouter mentioned that is indeed usually the reason for getting rid of these artefacts.


ReplyQuote
(@wvreeven)
Black Hole Admin
Joined: 2 years ago
Posts: 484
December 15, 2020 10:54  

@heno If I understand it correctly, your Altair 80 mm APO has a focal length of 480 mm and the Esprit 120ED 840 mm, which is almost double the focal length of the Altair APO. If you dither with SGP then the dithering in number of pixels with the Altair APO will be almost twice as small as  with the Esprit. Proper dithering involves dithering over enough pixels so to me it looks like you are not dithering enough with the Altair APO.


ReplyQuote
 Heno
(@heno)
Red Giant Customer
Joined: 3 years ago
Posts: 64
December 15, 2020 13:58  

@wvreeven
No, I'm not using the Esprit and the Altair together. I'm using a RASA8 (400mm) together with the Altair (480mm). The problem was not that I have not dithered, or not dithered enough pixels. The problem was that I was  dithering the RASA train while the Altair train was imaging. They were not synced as I thought they were. So the dither on the Altair train happened in the middle of a frame. That of course will cause trouble. 

Helge


ReplyQuote
(@wvreeven)
Black Hole Admin
Joined: 2 years ago
Posts: 484
December 15, 2020 14:18  

@heno You mentioned the Esprit in one of your previous comments and this is the first time you mention the RASA. Hence my confusion. In any case, you seem to be getting down to the cause of the artifacts in the mono images.


ReplyQuote
 Heno
(@heno)
Red Giant Customer
Joined: 3 years ago
Posts: 64
December 15, 2020 19:49  

@wvreeven
Sorry for the confusion, not intended. 
At least I hope I found the reason for the artifacts. I will of course test this at first opportunity, whenever this might be. The rain is pouring. 😥 

 


ReplyQuote
Share: