Mosaic normalizatio...
 
Share:
Notifications
Clear all

Please note our new Downloads page here

2023-01-19: APP 2.0.0-beta13 has been released !

!!! Big performance increase due to optimizations in integration !!!

and upgraded development platform to GraalVM 22.3 based on openJDK19

 

We are very close now to  releasing APP 2.0.0 stable with a complete printable manual...

 

Astro Pixel Processor Windows 64-bit

Astro Pixel Processor macOS Intel 64-bit

Astro Pixel Processor macOS Apple M Silicon 64-bit

Astro Pixel Processor Linux DEB 64-bit

Astro Pixel Processor Linux RPM 64-bit

Mosaic normalization issue (NAN or normalization failed)

Page 1 / 2

(@dav78)
Main Sequence Star Customer
Joined: 4 years ago
Posts: 85
Topic starter  

Hi,

 

I'm working on a mosaic (35 tiles for now) and at normalization step, for some tiles normalization failed, and for others it's NAN - 0 (I don't know what it means).

All the tiles are processed with same masters, and are taken with same settings.

I've made a screen shot of the integration result and tiles informations.

 

APP1

 

My settings are basically the same I usually use for processing mosaic (2500-3000 stars, dynamic distortion correction, advanced mode, LNC MBB...).

I tried with less tiles (9), but some tiles failed to normalize too. If I try again, the failed tiles may change. And if I change the reference image, and choose one that was previously failed at normalization step, all files are normalize (with those 9 tiles at least).

APP2

 

For more than two years using APP, I never saw this issue.

What have changed since is my camera (ASI533), and my computer (iMac 27 with Big Sur OS, before I was with MacBook Air and Mojave OS).

APP 1.082 and 1.083beta, same results (worth with 1.083beta2 as sometime star analysis fails with most of the images).

 

Any idea of what's happening here and how to solve it ?

 

Thanx !

David

 

 


ReplyQuote
(@vincent-mod)
Universe Admin
Joined: 6 years ago
Posts: 5218
 

Mmm, I think it probably is a data issue. What happens if you don't use any calibration frames on a panel (just test a panel that fails)?


ReplyQuote
(@dav78)
Main Sequence Star Customer
Joined: 4 years ago
Posts: 85
Topic starter  

Thank you @vincent-mod for your answer.

 

I've made a few other tests.

I've taken the same 9 panels.

- With individual lights not calibrated, it works (auto ref image is n°6)

- With individual lights calibrated, it works (auto ref image is n°6)

- With integrated lights (5x3min) calibrated, it doesn't work (auto ref image is n°4)

- With integrated lights (5x3min) calibrated, it works (manual ref image is n°6, that previously failed to normalize)

 

So, it seems that it's not a calibration issue.

The normalization of integrated panels seems to fail depending on the reference image chosen.

My integration settings here are : median, winsorized 6-3 with local normalization rejection checked, bayerdrizzle.

 

Here are the screenshots of the trials with integrated panels.

So maybe it's an integration issue, but I don't understand why and where it the fault...

One more thing, when I checked neutralize background in the histogram adjustment, the picture becomes white. Maybe there is a relation here, or not.

 

Capture d’écran 2021 10 13 à 11.17
Capture d’écran 2021 10 13 à 11.02

ReplyQuote
(@vincent-mod)
Universe Admin
Joined: 6 years ago
Posts: 5218
 

Mmm, the all white result might signal an issue in the background maybe. Is your offset set correctly for instance? Oh and one other possibility, are you using your calibration file twice? Like on the single frames and then again with the integrated files? If so, APP will substract twice and that is also an issue.


ReplyQuote
(@dav78)
Main Sequence Star Customer
Joined: 4 years ago
Posts: 85
Topic starter  

I don't know what the offset setting is as I use an ASIAIR PRO (it's a factory setting for the offset if I'm right).

And no, I haven't use the calibration twice. 

Maybe something with the masterbias, but it doesn't explain why some panels are correctly normalized one time, and the other time not...

 

Are they anywhere temp files that APP is using on a Mac OS ?

I mean, I was running two versions of APP on my Mac, maybe there is a conflict somewhere.

 


ReplyQuote
(@vincent-mod)
Universe Admin
Joined: 6 years ago
Posts: 5218
 

Running 2 APP versions and using the same working directory I never tested, but I can imagine that might not be the best thing to do indeed.

If things are not improving, I can have a look at the data if you want.


ReplyQuote
(@dav78)
Main Sequence Star Customer
Joined: 4 years ago
Posts: 85
Topic starter  

I made some other tests.

I've tried by creating an other masterbias with different files, same issue.

The question is why does normalization works with calibrated single frames but not with 5 stacked frames.

When I unchecked neutralize background at the 5 stage, the normalization worked for every stacked files, but the back ground of the mosaic doesn't seem to be uniform (it's darker on some frames).

@Vincent-mod, what data do you need to make further investigations ?

David

 


ReplyQuote
(@vincent-mod)
Universe Admin
Joined: 6 years ago
Posts: 5218
 

Those are good questions indeed. So, let me ask for the data to investigate. 🙂

Go to https://upload.astropixelprocessor.com and for the username and password, use: upload

Create a directory named “dav78-mosaicBackgroundIssue” and upload in there. Just upload like 10 lights per panel and your master calibration files. Thanks!


ReplyQuote
(@dav78)
Main Sequence Star Customer
Joined: 4 years ago
Posts: 85
Topic starter  

I'm sending the files right now : master calibration files, and 9 panels (5 frames each, I haven't more), the 9 mentioned above.

Thanx 😉

 

David


ReplyQuote
(@vincent-mod)
Universe Admin
Joined: 6 years ago
Posts: 5218
 

Excellent, I'll download them today and will have a look later.


ReplyQuote
(@vincent-mod)
Universe Admin
Joined: 6 years ago
Posts: 5218
 

So 1 thing I notice about the data is that the stars look a bit weird, square, meaning (I think) you're undersampling quite a bit. Having issues with beta 2 on analyzing them (which is a known problem, though I only now run into that). So, let me download 1.082 first as well. 🙂 I'm doing the first tests without bias, simply because the exposure time is 0.01s and I know that some sensors don't like that (they don't behave linear in that time). I'd do a bias with an exposure of something like 0.5s.

And another thing, the master-calibration files were made in 16-bit, which was the norm till 1.082 indeed, but we noticed that in some cases this caused an issue. Nowadays we create them in 32-bit (in 1.082 you have to switch that on, it's standard in a version above). Maybe you can do that as well, remake the masters in 32-bit, skip the bias and test.

 


ReplyQuote
(@vincent-mod)
Universe Admin
Joined: 6 years ago
Posts: 5218
 

Ok, so using the above setup (1.082, no use of the bias, 1600 stars to analyze, advanced normalization mode, mosaic registration, LNC3 1 it, 10% MBB) I get the following result;

MosaicResult

Switching on background normalization on the right-hand panel doesn't influence it (tiny bit but not in a degrading way at all).

Stretching the mosaic to the max, adding saturation to the max and making some Light Pollution Corrections;

MosaicLPC

(Above is the LPC box placement, I lowered the stretch again in this screenshot)

And the end-result;

MosaicEndResult

So my guess is that the bias was causing an issue. My advice is to either only use dark-flats, darks, flats and a BPM (Bad Pixel Map) or when a bias is used, take a higher exposure length per sub of say 0.5s.


ReplyQuote
(@dav78)
Main Sequence Star Customer
Joined: 4 years ago
Posts: 85
Topic starter  

Hi @vincent-mod, thank you for your feedback.

I've always used masters in 16-bit with this camera and never saw any issue (single panels or mosaics) and the masterbias was made with the shortest exposure time (has it was recommended, but things may have changed with newest cameras).

My other masterbias is 0,1 sec, and the issue is the same.

I will try with masterbias at 0,5 sec and 32-bit masters. For dark flats, how does it improve the calibration ? The masterbias is there for reading noise, right ? As I never make my flats with the same exposure time, it will be a huge work to make a master dark flat library ^^

For the picture made above, have used single frames or stacked frames ?

For the normalization of the mosaic, the background normalization box was checked ?

 

And yes, I'm under sampling, but not that much, my camera is ASI533 with 3,76 microns pixels, and the lens is a Samyang 135mm f2. Maybe not enough frame for an efficient bayerdrizzle...

 

 

 


ReplyQuote
(@vincent-mod)
Universe Admin
Joined: 6 years ago
Posts: 5218
 

For the above I used the single frames, more because it's just a few per panel. The only difference was that I excluded the bias. All settings were standard, I didn't change anything apart from setting it up for a mosaic.

For drizzling to work, you indeed need more data, the more the better really. But it would work very nicely I think.


ReplyQuote
(@dav78)
Main Sequence Star Customer
Joined: 4 years ago
Posts: 85
Topic starter  

@vincent-mod Ok, thanx.

Single frames mosaic works perfectly with full calibration (flats, darks and bias).

So it works too without masterbias considering your picture.

Maybe it would be usefull to try to reproduce the issue with stacked frames and full calibration to analyse what APP is saying (if you have the time of course).

What does NAN means ?

 

It tried today with stacked frames calibrated with masterflat and masterdark (without masterbias). It worked (but I don't know how it can affect the resulting picture).

 

The question remain the same : why does normalization fails with stacked calibrated frames and works with single calibrated frames (with same masters)? And why, in my case, the success of the normalization can depend on the reference image chosen?

Considering our results, the masterbias seams to be part of the issue.

 

In the mean time I will try : 32-bit masters, a new masterbias with 0,5 sec exposures, and masterdarkflat.

 

David 

 

 

 

 

 


ReplyQuote
(@vincent-mod)
Universe Admin
Joined: 6 years ago
Posts: 5218
 

Sure, I can test it with the panels as well, please allow for a bit of time as there is more data for me to analyze. 🙂 I think there isn't a huge issue here, maybe first try out the new bias or even don't use a bias but only darks and a bad pixel map, that may work better for your sensor anyway.

 NaN means "Not a Number", so it couldn't produce a result for some reason.


ReplyQuote
(@dav78)
Main Sequence Star Customer
Joined: 4 years ago
Posts: 85
Topic starter  

I'm back with some more tests.

@vincent-mod, as you advised it, I have created a masterdarkflat instead of a masterbias, and 32-bit masters (all done with 1-083beta2)

Unfortunately the issue is still there : for some stacked calibrated frames the normalization failed, but sometimes it succeeds, depending on the reference image chosen. And even when the normalization succeeds, the result isn't great. 

So the masterbias wasn't the cause, but maybe the masterflat. I don't know.

 

One more thing about star detection with the beta2. Star detection fails most of the time, but in my case, AAP algorithm was chosen (by default). But, as my pictures where taken with duo band filter, I've chosen the HaOiii color algorithm, and star detection worked! That's strange because the algorithm shouldn't influence the star detection, right ?

 

Capture d’écran 2021 11 01 à 22.11
Capture d’écran 2021 11 01 à 21.30

 

 


ReplyQuote
(@vincent-mod)
Universe Admin
Joined: 6 years ago
Posts: 5218
 

I don't think it should matter, but this can be something that has to do with the issue we have in beta2, so I'm not sure on that one. For the other problem I'm not sure how to deal with that, so I'll ask Mabula to also have a closer look. Does the automatic reference image work? So when you don't change it? Also, I never actually asked why you want to manually pick a different frame, it may be that APP already selects the optimum for that and then it does work?


Dav78 liked
ReplyQuote
(@mabula-admin)
Quasar Admin
Joined: 6 years ago
Posts: 3426
 

Hi Dav @dav78,

I am looking at your data. I think the issue is in your bias and darks. They don't seem to be fully compatible with your lights... Did you create them with the same software and the same software version as your lights?

Can you upload some of the single bias, darks and flats that you used to create the masters?

Reason for me to suspect this is that allthough you have exposed for 180seconds with I think broadband illumination of the sensor, many pixels are clipped to the black point after bias/dark subtraction. This should not occur on data like this. The adaptive pedestal kicks in to prevent the black clipping, but still the sky background even with the pedestal is very low...

Then depending on the reference frame for the mosaic, some of the panels will be problematic because the data normalization causes too many pixels to become 0... ending in failed normalization on NaN infinite SNR results, because noise can't be calculated properly.

Mabula

 


ReplyQuote
(@mabula-admin)
Quasar Admin
Joined: 6 years ago
Posts: 3426
 

Hi @dav78,

With all your calibration data, so bias, darks and flats i get no problems to create the mosaic in APP 1.082 with automatic reference by APP. Reference was panel 10 for the mosaic.

With Panel 18, like in your screenshots where it goes wrong, I also don't get any problems with 5) regular normalization mode

Mosaic ref panel18 all good normal normalization

With 5) advanced normalization mode I get the problem however:

AdvancedNormalization triggers issue

So this means the following, advanced normalization only uses the background and dispersion in the regions where the mosaic panels overlap with each other, instead of calculating this for the whole individual mosaic panel.

Clearly, with these background values in de overlap areas, when the frames are normalized in the normalization engine, the noise and SNR calculations fail on some of them, because now too many pixels get clipped the black point... which is indicative of a data calibration issue still.

I guess we really need to see the individual bias, dark and flats to get any explanation for the issue. Can you upload them?

Mabula

 


Dav78 liked
ReplyQuote
Page 1 / 2
Share: