[Solved] Strange calibrate star color behavior

Hi @geordan,

Thank you very much for your question.

If I look at the color plots, the star population only seems to have stars in a perfect bent line instead of nicely scattered around the model. That looks very weird. You need to check the actual colors and how you did the RGB Combine composite, the source of the problem will be there I would assume.

Were you able to solve the issue, or is it still a mystery?

Mabula

Hi,

I used the RGB Combine tool in APP.  This happens no matter which RGB/LRGB formula I use.  It's happening consistently for me now.

For reference, my workflow up to this point is very simple:

- integrate

- remove light pollution

- combine RGB

The actual colors don't look particularly unusual but I'm not sure what to look for.

Hi,

Any additional advice for this?  I don't know what information about my images or workflow that will be helpful to figure out what's going on here.

I have been having problems with the RGB combine tool for many months, even though it used to work fine for me too. If I do a simple RGB combine for stars I get an even more bizarre outcome than you when I run the star colour calibration tool (see image below). If I perform the RGB star combination in Pixinsight then run the star colour tool in APP then I see normal behaviour so the issue is definitely with RGB combine not the star colour tool.

I can't use the RGB tool for creating narrowband palette combinations any more either as documented in a previous post which was not resolved ( https://www.astropixelprocessor.com/community/main-forum/excessive-auto-stretch-in-rgb-combine/#post-33349 ). I now have to do all my narrowband colour combinations in Set Astro Suite.

Jon

Hi @geordan and @jdwood,

Can you both share a dataset with me, so I can dig into this? When i combine filter data in the RGB Combine Tool all seems to be fine, so it must be related somehow to how you create your stacks/channels before you supply them to the RGB Combine Tool. If I can see some of your stacks/channels that create this issue, I will be quickly able to solve this going forward.

Can you upload data here:

https://upload.astropixelprocessor.com/

username: uploadData

password: uploadTestData

Please make a folder with your name and issue and upload the data into those folders. I will have a look as soon as I see the data on our server.

I have also opened an issue on ToDo list.

Thank you very much for sharing this and please accept my apologies for the late reply from my side.

Mabula

thanks @mabula-admin for looking into this

I'm away from my imaging PC at present but will try to get some files to you in a couple of days.

cheers, Jon

I've uploaded subs and calibration images, plus the integrations I've made which resulted in the strange behavior.  (I confirmed that it still behaves this way in beta41.)  Let me know if there's more data you need from me.

Thanks for looking into this!

Posted by: @jdwood

↑

thanks @mabula-admin for looking into this

I'm away from my imaging PC at present but will try to get some files to you in a couple of days.

cheers, Jon

Thanks Jon @jdwood ! Look forward to your upload.

Mabula

Posted by: @geordan

↑

I've uploaded subs and calibration images, plus the integrations I've made which resulted in the strange behavior.  (I confirmed that it still behaves this way in beta41.)  Let me know if there's more data you need from me.

Thanks for looking into this!

Thanks @geordan,

We have received the upload in good order, I will investigate in the coming days and will get back to you about this.

Mabula

Hi @mabula-admin,

I took some new subs last night just to convince myself that I was not just doing something silly - I have uploaded three Master stacks, one each for R, G and B. There is no nebula here - just a star-field.

If I take these three subs and register/normalise them, and then use the RGB combine tool to combine them, things look superficially OK after background neutralisation. If I then try to use the calibrate star colour tool though I end up with something which is not easily interpreted - see below.

An obvious answer might be that my filter wheel is somehow not working as expected but I have seen this on other scopes and strangely, if I take those same 3 Masters and combine them in Pixinsight (using ChannelCombination), then return the resulting file to APP and run the star colour tool on that, everything now looks perfectly normal and as it should be.  Very odd...

Hope some of this makes sense 😀 

Jon

Hi Jon @jdwood,

Thank you very much for the upload, we have received it in good order.

Indeed, that just looks very weird, something fishy is going on and I will make sure it is solved. I will get back to you once I know more.

Mabula

Hi @geordan,

I am testing your data. I see the problem clearly in Calibrate Star Colors, it looks like R,G,B are not really R,G,B somehow, that is why the Calibrate Star Color tool is able to get all the stars on one line... I will analyse all frames that you uploaded to see if I can make sense of it somehow. 

I have one question for now: You have uploaded individual lights and the calibration masters which is great, that really helps me. But there are only MasterFlats and MasterDarkFlats no MasterBias nor MasterDark, is this on purpose? Without a masterbias or masterdark you are not getting optimal calibration I think? Or would you use a MasterDarkFlat as MasterBias ?

Mabula

Hi Geordan,

I think I understand what is happening, you must have made a mistake somehow when loading your filter data to process the R,G,B and L stacks. I just made the L,R,G,B stacks myself, when stretched I can see that the nebula shows different in the L,R,G,B integrations so they do indicate different filters being used in capturing the data.

Then I make a LRGB composite in RGB Combine and load it into the Star Color Calibration Tool after remove light pollution, and it works as expected actually. I will compare your L,R,G,B stacks to mine to hopefully understand where the mistake was made. This is what I get with Calibrate Star Colors on a simple LRGB composite, in the RGB Combine I used both normalization with Multiply-Scale and no normalization, both results work fine in Calibrate Star Colors:

Hi @jdwood & @geordan,

Update, okay, I am now able to reproduce the issue also. The Calibrate Star Colors tool becomes unstable actually, I see that now.  Somehow, the way the stacks were created by Geordan trigger this. When I create the stacks, the issue is not created. I will digg deeper now to solve this 😉

Mabula

Hi @jdwood & @geordan,

I have found the issue ;-), it is related to the image being background calibrated or not. If you explicitely perform background calibration before sending the image to the Star Color Calibration Tool, it will work properly.

The issue is caused by the FITS headers that are inhereted from the individual channels that were background calibrated. The RGB Combine Tool stored the tag in the fits header of the compositie and it shouldn't. So when you start Star Color Calibration, you are not warned properly that you need to do Background calibration first. I will fix this behaviour now.

Thank you very much for reporting this.

Mabula

Hi @jdwood & @geordan,

I have located the problem in the RGB Combine Tool (and all other lokations in APP that could cause this problem) and I have fixed it. I can properly process the L,R,G,B stacks of Geordan now to a LRGB composite and then do proper star color calibration without getting weird results. I will now also look at Jon's data to see if that works properly as well now.

• FIXED CALIBRATE STAR COLORS & RGB COMBINE TOOLS

  As reported here https://www.astropixelprocessor.com/community/main-forum/strange-calibrate-star-color-behavior/#post-33826   , the Calibrate Star Color tool could exhibit very weird behaviour creating fully red results instead of a nicely calibrated result. The bug was caused by the RGB Combine tool not remove old FITS tags from previously background calibrated files. This issue has been fixed. The problem caused Calibrate Star Colors to think that the image was properly background calibrated, but it was not and needed to be properly background calibrated first before running Calibrate Star Colors.

Hi John , I have also fixed the issue with your RGB data in the RGB Combine Tool:

This is what the RGB Combine TOOL makes of it in 2.0.0-beta42:

And then in Calibrate Star Colors:

This is what we expect. The issue was similiar but slighly different than @geordan 's issue. It both was caused by applying Light Pollution correction before going to the RGB Combine Tool. The bugs were that the sstored calibration values in the fits header were no longer correct after renormalizing the frames and sending them into the RGB Combine Tool. This is all fixed now. Will have a good look at your narrowband data now to make sure the stretching is good as well.

Mabula

Thanks for investigating this!  I'm glad the root cause was found.  And that lines up with my workflow: I do light pollution removal, then go into RGB Combine.  In the meantime (before the next release), it sounds like I can do an explicit background calibration after light pollution removal in order to set the proper FITS headers?

Posted by: @geordan

↑

Thanks for investigating this!  I'm glad the root cause was found.  And that lines up with my workflow: I do light pollution removal, then go into RGB Combine.  In the meantime (before the next release), it sounds like I can do an explicit background calibration after light pollution removal in order to set the proper FITS headers?

You are most welcome @geordan,

Yes, do an explicit background calibration and load that result into the Star Color Calibration tool, it should work correctly then 😉 still.

excellent thanks @mabula-admin! Yes - I normally do a LP correction on each master stack before combination. Maybe that is not the best way but I just got into the habit as, at least for narrowband data, they are often collected on different nights with different moon phase etc.

Hi Jon, @jdwood

Of course, depending on the data, different workflows can work out better 😉

I will soon release beta42 with all the fixes !

Mabula

Hi @geordan & @jdwood,

I have prepared a test version for 2.0.0-beta42 with modifications that possibly fix a crash on windows 11 as reported in another topic. I have created a test version for Windows.

This test version includes all the fixes for your problems, so please give it a go and let me know if all is okay now 😉

You can download it here:

https://appreleaseserver.s3.eu-central-1.amazonaws.com/2.0.0-beta42test1/AstroPixelProcessor-2.0.0-beta42test1-Windows-64-bit.zip

This version includes additional improvements as stated in the release notes for beta42:

https://www.astropixelprocessor.com/community/release-information/astro-pixel-processor-2-0-0-beta42-release-notes-work-in-progress/#post-342260

Can you test beta42test1 and let me know if this has solved the issue or not?

Thanks,

Mabula

thanks @mabula-admin 

Happy to test a new version but, unfortunately, I am a Mac user (at least for processing) 🙂 

Jon

I did a reprocess from the beginning with beta42 (integrate then light pollution removal then RGB Combine), then when I went to do calibrate star color it popped up a dialog telling me that the background was not yet calibrated, and gave me the options to either do light pollution removal or calibrate background.  Choosing either light pollution removal or calibrate background resulted in correct calibrate star color behavior.

It might be surprising for the user to see the dialog if they went through this workflow (since I had done light pollution removal on the individual channels), but this does force correct behavior.

Should I be doing light pollution removal per integrated channel?  I assumed so, given that there might be different gradients per channel, but now I'm not sure.

Hi @jdwood,

Ah okay, no worries, @geordan confirms all is okay now and I have tested it many times. I will go ahead and release beta42 for all platforms now. It should be available later today on the website.

Mabula

@geordan 

I did a reprocess from the beginning with beta42 (integrate then light pollution removal then RGB Combine), then when I went to do calibrate star color it popped up a dialog telling me that the background was not yet calibrated, and gave me the options to either do light pollution removal or calibrate background. Choosing either light pollution removal or calibrate background resulted in correct calibrate star color behavior.

That is actually the correct behaviour yes. It is fine to do light pollution removal or  background calibration on the individual stacks, but if you load them into a tool that will change data like RGB Combine with it's formula it is critical that background calibration is done after such a tool to get correct results in the Star Color Calibration tool, the stored backgrounds from the individual channels can be way off after RGB Combine so they need to be recalculated. So yes, this confirms all is okay now and APP's behaviour is as it should.

Actually, in my experience and what I always advice users: It is better to do Light Pollution Removal on the RGB composite then doing it on the individual channels. I get bettter results consistenly, maybe because I know how the tool works inside out. Technically, if you do Light Pollution Removal on monochrome data that you will feed into a composite, it is very easy to do it incorrectly and possible get rid of some good data.

If you do Light Pollution Removal only on the RGB composite, you are much more able to see if the correction per channel is correctly because you can see the color changes and by oversaturating in the tool with the preview filter, you can really see well what a good correction will be preserving all signal and getting rid of the unwanted gradients. 

I do have a couple of datasets that do work better in the other workflow, so I guess it depends a little on experience and also the data itself 😉

Mabula

Posted by: @jdwood

↑

thanks @mabula-admin 

Happy to test a new version but, unfortunately, I am a Mac user (at least for processing) 🙂 

Jon

Hi Jon @jdwood,

Please check the new release 2.0.0-beta42, which has all the fixes 😉

You can find it in the downloads section: https://www.astropixelprocessor.com/downloads/

Mabula