Multi-Session Calib...
 
Share:

20th March 2020 - APP 1.078 has been released !

2019 November: Complete LRGB Tutorial of NGC292, The Small Magellanic Cloud by Christian Sasse, Astronomer in Charge of iTelescope.net

2019 September: Astro Pixel Processor and iTelescope.net celebrate a new Partnership!

[Solved] Multi-Session Calibration Doesn't Work - 1.075  

Page 2 / 2
  RSS

(@mabula-admin)
Quasar Admin
Joined: 3 years ago
Posts: 2364
January 3, 2020 18:17  

Dear Jon @midnight_lightning,

Thank you very much, I am now working on and testing with your data 😉

I had to finish extensive work on the Star Analysis module first for the next release.

I will let you know what I find and correct if needed 😉

Mabula

Main developer of Astro Pixel Processor and owner of Aries Productions


ReplyQuote
(@mabula-admin)
Quasar Admin
Joined: 3 years ago
Posts: 2364
January 3, 2020 20:37  

Dear Jon @midnight_lightning,

I have processed all the data that you have uploaded to our server.

Indeed, I have noticed a small glitch when the masters are actually applied to the lights. After creation of all the masters in 2) Calibrate, not all lights were assigned all masters as they are supposed to. I will fix that. It was corrected by clicking on the re-assing masters to lights button in 2) Calibrate however.

Besides that, I did not see any bug or big problem:

  • all calibration masters were created and correctly
  • the resulting light frame integrations were all correct and looked very nice
  • no missing frames anywhere, the frame count in 1) LOAD was correct with the numbers of frames that were actually there in your dataset. The frame number counts in the masters and light frame itnegrations was correct as well, indicating that no data was missing.

CorrectComposite

But I do noticed some issues that might explain why you think things are not going correctly ? and they have to do with how things are reported in the bottom frame list panel:

  • the frame numbers don't add up nicely... the numbers are actually scrambled... I need and will fix that now
  • the filters and sessions are not nicely sorted, which I will fix as well

incorrect numbering

My compliments on a very nice dataset 😉 it looks great !

I will do some additional testing and I will make sure that master frame assigned works flawlessly in the next release APP 1.076.

Kind regards,

Mabula

Main developer of Astro Pixel Processor and owner of Aries Productions


ReplyQuote
(@mabula-admin)
Quasar Admin
Joined: 3 years ago
Posts: 2364
January 4, 2020 14:38  

Dear Jon @midnight_lightning,

The error with masterframe assignment has been fixed, it was a concurrency issue. If mutliple master flats were created in 2) Calibrate, like in your case, 7 different masterflats, not all MasterFlats were directly assigned when finishing the calibration task. It will work robustly now in APP 1.076 😉

Mabula

 

Main developer of Astro Pixel Processor and owner of Aries Productions


ReplyQuote
(@midnight_lightning)
Red Giant Customer
Joined: 2 years ago
Posts: 66
January 4, 2020 19:08  

Hi Mabula, thanks for looking at this.

I have run through pre-processing again this afternoon in light of your comments and except for a question and a couple of observations it seemed to work ok providing I load by file type - All Lights, All Flats, All Darks, All Dark Flats. 

  •  Question. After Calibration the MDF's had not been assigned to the lights in the file list - is this correct? I tried to Reassign Masters as you mentioned above but it didn't change anything, all the other masters looked to be assigned correctly so perhaps you don't show MDF assignment? 
  • The filename of the MD’s in the APP folder does not show which sessions they apply to, they just show the latest session number. e.g. my Ha Darks apply to session 3 and 4 but the filename only mentions session 4. It's odd because the “Frame” name in the APP file list does show the session 3 and 4. Could this be fixed to avoid confusion?

     

However, I suspect the main issue I had was due to the method of loading files.

Your mentioned earlier “To be clear, how frames are mapped should not depend on the loading order of course (unless you load the same frames twice), and also no frames should be dropped after you have loaded all of your frames. If you would load the same frames twice and assign them differently the second time, then yes, the first batch is dropped of course because you give them a new assignment.”

When I load data by Session rather than file type I do load the same data multiple times - e.g. I allocate Darks file 1 to Blue Session 1, and then later I allocate the same Darks file 1 to Session 2 Green.

This makes sense to me because the Darks File 1 is applicable in both cases but if I understand your statement, because I have loaded the same file twice then the first assignment is dropped. This is what I saw back in march with files that I had loaded disappearing.

I have run so many different permutations that I am struggling to remember what does what but I suspect the problem I had was loading the same files twice was causing files to be dropped. Logically, in my mind anyway, it should work regardless of the approach to loading because essentially a matrix of file associations is being built.

If I load by file type, even though I only load the Darks file once it gets assigned to Blue Session 1 and Green Session 2.        

However if I load by Session, I explicitly load the darks file twice, once to Blue Session 1 and once to Green session 2 but the program logic appears to drop the original assignment on the basis the file has been reassigned. 

So it appears to me that the method of loading files is critical to correct operation.

Is this making any sense?

Thanks

Jon

 

 

 

 


ReplyQuote
(@mabula-admin)
Quasar Admin
Joined: 3 years ago
Posts: 2364
January 6, 2020 14:29  
Posted by: @midnight_lightning

Hi Mabula, thanks for looking at this.

I have run through pre-processing again this afternoon in light of your comments and except for a question and a couple of observations it seemed to work ok providing I load by file type - All Lights, All Flats, All Darks, All Dark Flats. 

  •  Question. After Calibration the MDF's had not been assigned to the lights in the file list - is this correct? I tried to Reassign Masters as you mentioned above but it didn't change anything, all the other masters looked to be assigned correctly so perhaps you don't show MDF assignment? 

Hi Jon @midnight_lightning,

I have done multiple tests now in APP 1.076 (the next release) and all works fine each time that i run it 😉 (I am running it one more time while writing this). MDFs are never assigned to lights because they are used for the calibration of the flat frames. When the masterflat is created, APP checks if the flats can be calibrated by either bias or darkflats or both and will automatically assing them to the flats while creating the masterflats. So that is correct, you will never see assignment of the MDFs to lights, because they serve another purpose.

  • The filename of the MD’s in the APP folder does not show which sessions they apply to, they just show the latest session number. e.g. my Ha Darks apply to session 3 and 4 but the filename only mentions session 4. It's odd because the “Frame” name in the APP file list does show the session 3 and 4. Could this be fixed to avoid confusion?

Indeed, you are totally right, I have just checked, the bug was there in MB,MD,MDF,MF creation, it is fixed now ;-). The Session String was not created properly for the naming of the file.

Calibration Masters FileName Shows Correct Channels and Sessions

I did not notice it earlier, because I was loading the darks for all channels and all sessions, which does work on your dataset, but you should be able to have darks specifically assigned to channels and sessions, because of temperature differences between darks/masterdarks and the file name should show it correctly. It will now ;-).

However, I suspect the main issue I had was due to the method of loading files.

Your mentioned earlier “To be clear, how frames are mapped should not depend on the loading order of course (unless you load the same frames twice), and also no frames should be dropped after you have loaded all of your frames. If you would load the same frames twice and assign them differently the second time, then yes, the first batch is dropped of course because you give them a new assignment.”

When I load data by Session rather than file type I do load the same data multiple times - e.g. I allocate Darks file 1 to Blue Session 1, and then later I allocate the same Darks file 1 to Session 2 Green.

This makes sense to me because the Darks File 1 is applicable in both cases but if I understand your statement, because I have loaded the same file twice then the first assignment is dropped. This is what I saw back in march with files that I had loaded disappearing.

I have run so many different permutations that I am struggling to remember what does what but I suspect the problem I had was loading the same files twice was causing files to be dropped. Logically, in my mind anyway, it should work regardless of the approach to loading because essentially a matrix of file associations is being built.

If I load by file type, even though I only load the Darks file once it gets assigned to Blue Session 1 and Green Session 2.        

However if I load by Session, I explicitly load the darks file twice, once to Blue Session 1 and once to Green session 2 but the program logic appears to drop the original assignment on the basis the file has been reassigned. 

So it appears to me that the method of loading files is critical to correct operation.

Is this making any sense?

Thanks

Jon

 

Okay, yes, that fully explains the issue that you experienced...I now understand what happened 😉

You will need to load your frames once and completely assing them while doing it. If you load them twice and give another assignment on the second load, the first assignment is lost.

I deliberately implemented it like this. If you load the frames wrong, simply load them again with the correct assignment and then the first assignment is overruled. I guess this has to do with how one manages his/her files.

Make clear for yourself which frames belong to which session(s) and channel(s) and load them just once with all correct assignments, then all should be okay 😉

(I am nearly ready to release a first Beta of the upcoming release with a lot of changes and improvements.)

Kind regards,

Mabula

 

Main developer of Astro Pixel Processor and owner of Aries Productions


ReplyQuote
(@midnight_lightning)
Red Giant Customer
Joined: 2 years ago
Posts: 66
January 6, 2020 18:30  

Thanks Mabula.

I don't feel its very intuitive or user friendly to have a load sequence remove an assignment that the user has explicitly made, I've wasted well over a week on this since March trying to work out what was happening. At least I now know how to avoid the issue.

I wonder how many other people load data by session as I did and just haven't noticed that their Darks etc were disappearing and only got used for the last session? If it has to work this way perhaps the program could issue a warning message/explanation when someone uses the same file in another session and the earlier one is removed?

Looking forward to the new release!

Jon


ReplyQuote
(@mabula-admin)
Quasar Admin
Joined: 3 years ago
Posts: 2364
January 6, 2020 23:08  

Hi Jon @midnight_lightning,

Yes, fair enough, I must admit, it never occured to me that one would  load the calibration frames multiple times and each time would assign a different filter or session. Why not load and do all needed assignments all at once? That is faster  for sure and also less error prone I think/feel.

The way that you uploaded your data to me, was perfectly organized to do it just like that 😉 If you load, it's logical that you can select only 1 fitler and 1 session per light. So it all comes down to the calibration frames. If you know that the 1200s darks  are for H-alpha session 3, and 4, you can directly load them as such, right? You can select multiple filters and session when doing the assignment.

To implement it such that your method will work as well, needs actually some additional work. I will put it on my ToDo list.

Apologies for failing to understand your load workflow and how this affected things. I do get it now 😉

The next release will be a very nice release, your dataset helped me a lot in the past few days to get additional improvements for multi-session processing:

https://www.astropixelprocessor.com/community/release-information/astro-pixel-processor-1-076-preparing-next-release/

"

  • IMPROVED, FRAME LIST PANEL, the frame numbers per frame type (lights, flats etc) are now always ascending no matter how you sort the frames. The first image showed incorrect frame numbering from top to bottom:

incorrect numbering
frame list panel frame number ascending

  • FIXED, 2) CALIBRATE in MULTI-SESSION/FILTER modes, when multiple masterframes were created of a single type, like multiple MasterFlats for different sessions/filters, then due to a concurrency issue, not all masters were directly assigned to the correct light frames when finishing the Create Master Calibration frames task. This is now fixed.
  • IMPROVED/FIXED, FRAME LIST PANEL, the column with the file name of the frame that was loaded has been adjusted. If the file name contains the selected work directory, only the file path without the work directory is shown. Furthermore, a bug has been fixed, if you would change the column order by dragging with your mouse, the panel could crash.. this will no longer happen.
  • IMPROVED, FRAME LIST PANEL, MULTI-CHANNEL/SESSION, to improve the readability of the frame list panel when processing in multi-channel and/or multi-session modes, all frames per filter are always grouped. Within each filtergroup, the different sessions are grouped. Furthermore, the filter groups will be shown with their own color to further improve readability. Finally, each frame type (lights, bias, dark, BPM etc) will have their own color in the frame list panel.

Frame List Panel filter groups with session groups 2
Frame List Panel filter groups with session groups
Frame List Panel filter groups with session groups 3

  • FIXED, CALIBRATION MASTERS MULTI-SESSION, in multi-session mode, masters created for specific sessions did not show the correct sessions in the file naming. This is correct name:

Calibration Masters FileName Shows Correct Channels and Sessions

  • IMPROVED, FRAME QUALITY SELECTION IN MULTI-CHANNEL/FILTER MODE, when selecting a % of frames in 6) INTEGRATE, (lights to stack % ) in the multi-channel mode, the frame % is applied per fitlergroup now. This is a big improvement, because previously, too many frames would be removed of the filter with the least quality instead of removing the worst frames per filter. The selection is not done per session, because you simply want the best frames per filter normally. In the following screenshot, you see a frame % applied of 75% to a selection of Red, Green, Blue and H-alpha frames over 5 different sessions, notice that 75% is set for integration per fitler now. You can see this from the INTEGRATE mark in the quality score column:

Frame List Panel filter Quality Selection per filter

"

Kind regards,

Mabula

Main developer of Astro Pixel Processor and owner of Aries Productions


ReplyQuote
(@midnight_lightning)
Red Giant Customer
Joined: 2 years ago
Posts: 66
January 7, 2020 08:55  

Hi Mabula, 

Thanks for responding. I agree that your method of loading is more efficient and will use it going forwards, I was just trying to highlight that assuming everyone loads in this way may be unsafe.

The new release looks great, lots of nice improvements so looking forwards to getting it, I do like APP pre-processing, it's way ahead of anything else available in terms of usability and good results.

Just an observation on the Frame Quality Selection. The 1.076 change is a definite improvement but dropping a percentage of frames is still a fairly arbitrary approach. With few clear nights in the UK and 20 minute exposures for NB I am loath to drop any frames and manually check each one which is very time consuming. It would be great if APP could do the checking and only drop or identify "poor" frames. I'm not sure how/whether this could be accomplished but perhaps rather than dropping a percentage it might be possible to use a deviation method to identify "outlier" frames that are say x% below average - with the user selecting x. Perhaps its too complicated, outside obvious poor frames I often struggle to know what to keep and what to discard and rely on the Integration "Quality" weight. If it could be done it would be a great time saver. 

Incidentally I did a test a while ago to compare processing all lights vs the best lights - using Quality Integration. I couldn't see any difference in the resulting masters. I assume its the Quality setting that made this work so well, it would be interesting to repeat the test using Equal, perhaps when I get some spare time.

Keep up the good work, thanks again 🙂

Jon

 


ReplyQuote
Page 2 / 2
Share: