[Solved] Multi-Session Calibration Doesn't Work - 1.075

Forwarded directly to Mabula..

@vincent-mod

Thanks Vincent. 

Is there a work around, I haven’t been able to process any new data since February?

Not sure as I never noticed this issue, Mabula is back from being abroad ands said he would attend to the forum shortly, let's wait and see what he has to say, that'll be faster in this case. 🙂 We'll get it sorted.

@vincent-mod

thanks Vincent

Any update?

I've waited since March for an update on this (also see previous ticket) and am going to have to learn how to do it in Pixinsight if it cannot be resolved.

I only bought APP for the pre-processing and have to say that I am disappointed with the level of support I am getting with this issue.

I understand your frustration.

Would it be possible for you to put together a minimal set of files that shows this behaviour, i.e. possibly one Light and one Flat for each filter and each session? That could help to reproduce the problem and for us to possibly help you find a work around so at least you can move forward with processing your images.

Clear skies, Wouter

Again, sorry for the delay; I've reached out to Mabula a few times but he hasn't attended the forum yet. I guess he's very busy at the moment, I messaged him again. @mabula-admin

Hi Wouter, I started writing this hours ago and have made some progress so please see end first - unless you work with Mabula there probably isn't much you can help with but thanks for asking. 

I set up a Drop Box folder with one of each frame type along the lines you suggested, however I thought I would try a test loading of this data before posting and oddly it all loaded without any issues. I ran the complete process through to integration - all fine.

I'm wondering if this is because there were relatively few files in this test?

The data is here https://www.dropbox.com/sh/kqe94ilyogzl3j7/AAB_Yst6dR6EWK0Rm3yDtZPLa?dl=0   (Sorry had to delete this data to be able to supply full data)

I then tried to load the full data set again and got a different result to the last time I ran the full data. I didn't get any Flat frames go missing this time although I did lose some Bias and Dark frames from the files list. I'm wondering whether the missing files were actually dropped and not used, or just omitted from the list to save space.

e.g. Although I loaded (the same) BIAS for each Session and Filter, I'm left with one entry for BIAS which shows allocation to "All Channels Session 5" - I did use the same BIAS files for all sessions and Channels so although this shows "Session 5" perhaps it is actually used for all sessions which would be correct? I'm left guessing which is not great.

Similarly for Darks, I loaded Darks individually for all sessions and filters but ended up with Darks for Session 4 and Session 5 only (all channels) in the files list. Did it use these for the other sessions as well - who knows?

I kept a spreadsheet showing what was in the flies list after loading each Session's data.

I tried to run the calibration step after Load anyway but it gave this error.

. I have checked all subs for correct Filter/Type in Fits header in case there is a data issue but all look fine. I OK'd the flats error and let it run through and whilst it produced Masters for Flats and Dark Flats it only produced MD's for Session 4 and Session 5

I don't understand why Flats disappeared from the earlier run and not this one but there are various permutations of loading data - e.g. Using "Filter Header" or specifically naming the filter, loading sequence - perhaps I did something different.

Just tried another full load and this time everything loaded as expected - I suspect the difference was down to the Loading Procedure. I usually load data one session at a time, so I load Session 1 Lights, Flats, DF's, Darks, Bias. Then go to Session 2 and do the same and so on. (Yes I know there is some duplication where Dark and Bias data is reused but it fits my workflow and library structure to do it that way).

However, this time I loaded the Lights first, for all sessions, then the Flats for all sessions etc. When it came to loading BIAS I only had to select All Channels and All Sessions once. With Darks it was All Channels and S1, S2, S5 for the 300s Darks and All Channels with S3, S4 for the 1200s Darks. 

So, the data appears to have loaded ok and I have run through to Integration where the Masters look ok except for the Master Darks. Again, I only have two MD's for the five sessions, One for the 1200s subs labelled as Session 4 and one for the 300s subs labelled as Session 5. I don't know if these were also used for processing Sessions 1, 2 and 3 but if so it would help if they were labelled as having done so. (Note - when I ran the test session, at the top of this post, I got all 5 MD's, one for each session as expected)

Given the results of my testing today, I've lost count of how many days I've spent on this, it appears that that you get inconsistent results depending on how you Load data - I don't know what the trigger is.

There is also still the question of Darks, and sometimes Flast, disappearing from the files list and the final masters. I just noticed that in the full test whilst I only got 2 of 5 MD files produced, the files list indicates they were used correctly across all 5 sessions. 

e.g. One MD file was labelled "Session 5" but I can see from the file list it was associated with Sessions 1, 2 and 5 which is correct.

Hope this helps resolution.

Thanks

Jon

Hi Jon,

Good to see that you are making progress. And thanks for explaining in so much detail how you load all your frames. To answer your question: I don’t work with Mabula but I do use APP a lot, also with multiple sessions and multiple filters, so I thought I’d chime in and at least offer a helping hand.

Learning to use a new user interface can be tricky and despite Mabula’s efforts to make APP as easy to use as possible there clearly is room for improvement. That is possibly why you end up with different file lists if the subs are loaded in a different order. My guess is that the UI has been designed with a specific use in mind (being: dark and bias subs can be reused over different sessions as long as the gain+offset (or ISO), temperature and exposure time are the same plus loading first all light subs, then flat subs, then dark subs and finally bias subs) and it doesn’t handle loading them in a different order well. I don’t know if this means that the master dark and master bias have been used for all sessions or only 4 and 5 (this is for Vincent or Mabula to answer) but I expect that they have been used for all sessions.

So, yes, it takes quite a bit of effort to get to learn APP and I agree that over the past months the responsiveness of Mabula (and at times of Vincent as well) has been poor at best but I can only imagine how hard it is to develop an application like APP alone whilst depending almost completely solely on one person to provide feedback to forum users.

I hope that now either Vincent or Mabula can take some time to answer your remaining questions and that you find joy again in using APP.

Clear skies, Wouter

Hi Wouter, many thanks for stepping in to help, much appreciated.

I really like APP and have been suggesting it to people since I bought it but having paid for a full license I am disappointed with the level of  support since I logged this in March.  

The basic pre-processing Load logic facilitates a very nice method of building a matrix to map Lights, Sessions and Calibration files. It's fairly intuitive to use so I don't think this is about learning to use an interface. If the user explicitly says use these calibration frames for this session and these lights it shouldn't matter what order the files are loaded.

Only Mabula will know what the issue is but it feels like an engineering issue to me, hope it gets looked at soon.

Thanks again.

 Jon

Hi everyone, thanks Wouter. Yes I agree, support is lacking a bit and indeed it's difficult to have just 1 person answer all posts. I have my own knowledge which I try to use the best way I can in answering questions, but I'm also not all-knowing. Besides that, this is not my main job and we're in the process of relocation with my family, so that's why sometimes I'm offline for a week and you suddenly see a barrage of answers after that. 😉 I'm talking about this with Mabula shortly.

Hi Jon @midnight_lightning, Wouter @wvreeven & Vincent @vincent-mod,

First of all, Jon, apologies for keeping you in the dark for so long. I will put this issue very high on my list to get sorted out for the next release.

To be able to sort this out properly, it would be most helpfull if you can share your data, so I can run it myself to see what is happening.

To be clear, how frames are mapped should not depend on the loading order of course (unless you load the same frames twice), and also no frames should be dropped after you have loaded all of your frames. If you would load the same frames twice and assign them differently the second time, then yes, the first batch is dropped of course because you give them a new assignment.

It could be that the frame assignment for filters and sessions does not work consistently like you indicate, if so, then this needs to be sorted a.s.a.p. 😉

Would this data set be enough for me to test if fully you think?

I set up a Drop Box folder with one of each frame type along the lines you suggested, however I thought I would try a test loading of this data before posting and oddly it all loaded without any issues. I ran the complete process through to integration - all fine.

I'm wondering if this is because there were relatively few files in this test?

The data is here https://www.dropbox.com/sh/kqe94ilyogzl3j7/AAB_Yst6dR6EWK0Rm3yDtZPLa?dl=0  /a>  

I then tried to load the full data set again and got a different result to the last time I ran the full data. I didn't get any Flat frames go missing this time although I did lose some Bias and Dark frames from the files list. I'm wondering whether the missing files were actually dropped and not used, or just omitted from the list to save space.

Wouter, thank you very much for your assistance. That is highly appreciated 😉

Kind regards,

Mabula

@mabula-admin

Hi Mabula, thanks for replying.

The test data on the earlier link was just one file each from Lights and calibration frames, and ran through without issue when I tried it, so not much use for verification.

I have put the full data in folders to share but unfortunately it is nearly 6GB after zipping and I don't have enough cloud storage so will have to transfer it in chunks. 

Can you try and download these and if successful I will clear down my One Drive and send the Dark Flats on another link.

https://1drv.ms/u/s!AhXsgtBn5aZUgYQyoiukHP7LwZ-MFw?e=ldiMSy  - Flats 

https://www.dropbox.com/sh/ovu74tlwzhpn75s/AAC10TaPyslYeoKb6ehbXd1da?dl=0 Lights, Bias, Darks

Just thinking through the testing I did yesterday, and your comments, it may be that the processing is working ok and that its the file list and file naming that is confusing. Note, I will have loaded the same data twice in some cases, particularly when I load each session's data in turn.

I have loaded the data in many different ways, e.g. by loading each session at a time, including loading some files multiple times (e.g. Bias that applied to all Channels, all sessions) and also by Frame Type, all the lights, then all the Darks etc.

So, for example, when I loaded data one session at a time I expected to see it build up in the file list as I loaded it but some files started to disappear - which ties in with your explanation about duplicates being removed. But in this case I would expect the remaining file to be labelled to show which sessions/channels it applies to - the APP Loading Sequence.txt in an earlier post shows how the files get dropped and how naming changes and fails to state what sessions/channels it applies to.

This is also reflected in the files produced in the final integration - I got two Master Darks, for S4 and S5, but nothing for S1, S2, S3. With hindsight it looks like the S4 file also covers S3, and the S5 file covers S1 and S2 - just needs to say this in the name to be consistent with other file naming.

I did notice that the file list, after Integration, indicates that the correct files have been applied to each session/channel which again suggests processing is ok and its the naming conventions that have thrown me.    

That said, when I loaded data by session yesterday it threw an error in calibration (see earlier post). When I reran it loading data by file type it ran through ok. Same files just different loading approach.

Note. The data supplied covers several multi-session permutations however doesn't include any sessions where multiple darks may be necessary, e.g. if different exposure times were used for RGB and Narrow Band on the same night.

Hope this helps.

Kind regards

Jon

@mabula-admin

Hi Mabula, the files that I loaded for you a couple of days ago are taking up 100% of of my Dropbox and Onedrive allocations. Be good if you could download them so I can then add the dark flats before clearing them down so I can use them for other things.

Thanks

Jon

I'll drop him a message as well just to be sure, he's very busy preparing a meeting. You can't use our server by any chance?

@vincent-mod

I don't know about using your server, what would I need to do?

You can upload it to this address, log-in with username and password: appuser

Create a directory with your name and "Multi-session-Calib" added to the name. Put the files into there. Thanks!

@vincent-mod

Thanks Vincent, took a few attempts due to timeouts but all uploaded now 🙂

Jon

Thank you! Either me or Mabula will have a good look at it.

Dear Jon @midnight_lightning,

Thank you very much, I am now working on and testing with your data 😉

I had to finish extensive work on the Star Analysis module first for the next release.

I will let you know what I find and correct if needed 😉

Mabula

Dear Jon @midnight_lightning,

I have processed all the data that you have uploaded to our server.

Indeed, I have noticed a small glitch when the masters are actually applied to the lights. After creation of all the masters in 2) Calibrate, not all lights were assigned all masters as they are supposed to. I will fix that. It was corrected by clicking on the re-assing masters to lights button in 2) Calibrate however.

Besides that, I did not see any bug or big problem:

• all calibration masters were created and correctly
• the resulting light frame integrations were all correct and looked very nice
• no missing frames anywhere, the frame count in 1) LOAD was correct with the numbers of frames that were actually there in your dataset. The frame number counts in the masters and light frame itnegrations was correct as well, indicating that no data was missing.

But I do noticed some issues that might explain why you think things are not going correctly ? and they have to do with how things are reported in the bottom frame list panel:

• the frame numbers don't add up nicely... the numbers are actually scrambled... I need and will fix that now
• the filters and sessions are not nicely sorted, which I will fix as well

My compliments on a very nice dataset 😉 it looks great !

I will do some additional testing and I will make sure that master frame assigned works flawlessly in the next release APP 1.076.

Kind regards,

Mabula

Dear Jon @midnight_lightning,

The error with masterframe assignment has been fixed, it was a concurrency issue. If mutliple master flats were created in 2) Calibrate, like in your case, 7 different masterflats, not all MasterFlats were directly assigned when finishing the calibration task. It will work robustly now in APP 1.076 😉

Mabula

Hi Mabula, thanks for looking at this.

I have run through pre-processing again this afternoon in light of your comments and except for a question and a couple of observations it seemed to work ok providing I load by file type - All Lights, All Flats, All Darks, All Dark Flats. 

•  Question. After Calibration the MDF's had not been assigned to the lights in the file list - is this correct? I tried to Reassign Masters as you mentioned above but it didn't change anything, all the other masters looked to be assigned correctly so perhaps you don't show MDF assignment? 
• The filename of the MD’s in the APP folder does not show which sessions they apply to, they just show the latest session number. e.g. my Ha Darks apply to session 3 and 4 but the filename only mentions session 4. It's odd because the “Frame” name in the APP file list does show the session 3 and 4. Could this be fixed to avoid confusion?

   

However, I suspect the main issue I had was due to the method of loading files.

Your mentioned earlier “To be clear, how frames are mapped should not depend on the loading order of course (unless you load the same frames twice), and also no frames should be dropped after you have loaded all of your frames. If you would load the same frames twice and assign them differently the second time, then yes, the first batch is dropped of course because you give them a new assignment.”

When I load data by Session rather than file type I do load the same data multiple times - e.g. I allocate Darks file 1 to Blue Session 1, and then later I allocate the same Darks file 1 to Session 2 Green.

This makes sense to me because the Darks File 1 is applicable in both cases but if I understand your statement, because I have loaded the same file twice then the first assignment is dropped. This is what I saw back in march with files that I had loaded disappearing.

I have run so many different permutations that I am struggling to remember what does what but I suspect the problem I had was loading the same files twice was causing files to be dropped. Logically, in my mind anyway, it should work regardless of the approach to loading because essentially a matrix of file associations is being built.

If I load by file type, even though I only load the Darks file once it gets assigned to Blue Session 1 and Green Session 2.        

However if I load by Session, I explicitly load the darks file twice, once to Blue Session 1 and once to Green session 2 but the program logic appears to drop the original assignment on the basis the file has been reassigned. 

So it appears to me that the method of loading files is critical to correct operation.

Is this making any sense?

Thanks

Jon

Posted by: @midnight_lightning

Hi Mabula, thanks for looking at this.

I have run through pre-processing again this afternoon in light of your comments and except for a question and a couple of observations it seemed to work ok providing I load by file type - All Lights, All Flats, All Darks, All Dark Flats. 

•  Question. After Calibration the MDF's had not been assigned to the lights in the file list - is this correct? I tried to Reassign Masters as you mentioned above but it didn't change anything, all the other masters looked to be assigned correctly so perhaps you don't show MDF assignment? 

Hi Jon @midnight_lightning,

I have done multiple tests now in APP 1.076 (the next release) and all works fine each time that i run it 😉 (I am running it one more time while writing this). MDFs are never assigned to lights because they are used for the calibration of the flat frames. When the masterflat is created, APP checks if the flats can be calibrated by either bias or darkflats or both and will automatically assing them to the flats while creating the masterflats. So that is correct, you will never see assignment of the MDFs to lights, because they serve another purpose.

• The filename of the MD’s in the APP folder does not show which sessions they apply to, they just show the latest session number. e.g. my Ha Darks apply to session 3 and 4 but the filename only mentions session 4. It's odd because the “Frame” name in the APP file list does show the session 3 and 4. Could this be fixed to avoid confusion?

Indeed, you are totally right, I have just checked, the bug was there in MB,MD,MDF,MF creation, it is fixed now ;-). The Session String was not created properly for the naming of the file.

I did not notice it earlier, because I was loading the darks for all channels and all sessions, which does work on your dataset, but you should be able to have darks specifically assigned to channels and sessions, because of temperature differences between darks/masterdarks and the file name should show it correctly. It will now ;-).

However, I suspect the main issue I had was due to the method of loading files.

Your mentioned earlier “To be clear, how frames are mapped should not depend on the loading order of course (unless you load the same frames twice), and also no frames should be dropped after you have loaded all of your frames. If you would load the same frames twice and assign them differently the second time, then yes, the first batch is dropped of course because you give them a new assignment.”

When I load data by Session rather than file type I do load the same data multiple times - e.g. I allocate Darks file 1 to Blue Session 1, and then later I allocate the same Darks file 1 to Session 2 Green.

This makes sense to me because the Darks File 1 is applicable in both cases but if I understand your statement, because I have loaded the same file twice then the first assignment is dropped. This is what I saw back in march with files that I had loaded disappearing.

I have run so many different permutations that I am struggling to remember what does what but I suspect the problem I had was loading the same files twice was causing files to be dropped. Logically, in my mind anyway, it should work regardless of the approach to loading because essentially a matrix of file associations is being built.

If I load by file type, even though I only load the Darks file once it gets assigned to Blue Session 1 and Green Session 2.        

However if I load by Session, I explicitly load the darks file twice, once to Blue Session 1 and once to Green session 2 but the program logic appears to drop the original assignment on the basis the file has been reassigned. 

So it appears to me that the method of loading files is critical to correct operation.

Is this making any sense?

Thanks

Jon

 

Okay, yes, that fully explains the issue that you experienced...I now understand what happened 😉

You will need to load your frames once and completely assing them while doing it. If you load them twice and give another assignment on the second load, the first assignment is lost.

I deliberately implemented it like this. If you load the frames wrong, simply load them again with the correct assignment and then the first assignment is overruled. I guess this has to do with how one manages his/her files.

Make clear for yourself which frames belong to which session(s) and channel(s) and load them just once with all correct assignments, then all should be okay 😉

(I am nearly ready to release a first Beta of the upcoming release with a lot of changes and improvements.)

Kind regards,

Mabula

Thanks Mabula.

I don't feel its very intuitive or user friendly to have a load sequence remove an assignment that the user has explicitly made, I've wasted well over a week on this since March trying to work out what was happening. At least I now know how to avoid the issue.

I wonder how many other people load data by session as I did and just haven't noticed that their Darks etc were disappearing and only got used for the last session? If it has to work this way perhaps the program could issue a warning message/explanation when someone uses the same file in another session and the earlier one is removed?

Looking forward to the new release!

Jon

Hi Jon @midnight_lightning,

Yes, fair enough, I must admit, it never occured to me that one would  load the calibration frames multiple times and each time would assign a different filter or session. Why not load and do all needed assignments all at once? That is faster  for sure and also less error prone I think/feel.

The way that you uploaded your data to me, was perfectly organized to do it just like that 😉 If you load, it's logical that you can select only 1 fitler and 1 session per light. So it all comes down to the calibration frames. If you know that the 1200s darks  are for H-alpha session 3, and 4, you can directly load them as such, right? You can select multiple filters and session when doing the assignment.

To implement it such that your method will work as well, needs actually some additional work. I will put it on my ToDo list.

Apologies for failing to understand your load workflow and how this affected things. I do get it now 😉

The next release will be a very nice release, your dataset helped me a lot in the past few days to get additional improvements for multi-session processing:

https://www.astropixelprocessor.com/community/release-information/astro-pixel-processor-1-076-preparing-next-release/

"

• IMPROVED, FRAME LIST PANEL, the frame numbers per frame type (lights, flats etc) are now always ascending no matter how you sort the frames. The first image showed incorrect frame numbering from top to bottom:

• FIXED, 2) CALIBRATE in MULTI-SESSION/FILTER modes, when multiple masterframes were created of a single type, like multiple MasterFlats for different sessions/filters, then due to a concurrency issue, not all masters were directly assigned to the correct light frames when finishing the Create Master Calibration frames task. This is now fixed.
• IMPROVED/FIXED, FRAME LIST PANEL, the column with the file name of the frame that was loaded has been adjusted. If the file name contains the selected work directory, only the file path without the work directory is shown. Furthermore, a bug has been fixed, if you would change the column order by dragging with your mouse, the panel could crash.. this will no longer happen.
• IMPROVED, FRAME LIST PANEL, MULTI-CHANNEL/SESSION, to improve the readability of the frame list panel when processing in multi-channel and/or multi-session modes, all frames per filter are always grouped. Within each filtergroup, the different sessions are grouped. Furthermore, the filter groups will be shown with their own color to further improve readability. Finally, each frame type (lights, bias, dark, BPM etc) will have their own color in the frame list panel.

• FIXED, CALIBRATION MASTERS MULTI-SESSION, in multi-session mode, masters created for specific sessions did not show the correct sessions in the file naming. This is correct name:

• IMPROVED, FRAME QUALITY SELECTION IN MULTI-CHANNEL/FILTER MODE, when selecting a % of frames in 6) INTEGRATE, (lights to stack % ) in the multi-channel mode, the frame % is applied per fitlergroup now. This is a big improvement, because previously, too many frames would be removed of the filter with the least quality instead of removing the worst frames per filter. The selection is not done per session, because you simply want the best frames per filter normally. In the following screenshot, you see a frame % applied of 75% to a selection of Red, Green, Blue and H-alpha frames over 5 different sessions, notice that 75% is set for integration per fitler now. You can see this from the INTEGRATE mark in the quality score column:

"

Kind regards,

Mabula

Hi Mabula, 

Thanks for responding. I agree that your method of loading is more efficient and will use it going forwards, I was just trying to highlight that assuming everyone loads in this way may be unsafe.

The new release looks great, lots of nice improvements so looking forwards to getting it, I do like APP pre-processing, it's way ahead of anything else available in terms of usability and good results.

Just an observation on the Frame Quality Selection. The 1.076 change is a definite improvement but dropping a percentage of frames is still a fairly arbitrary approach. With few clear nights in the UK and 20 minute exposures for NB I am loath to drop any frames and manually check each one which is very time consuming. It would be great if APP could do the checking and only drop or identify "poor" frames. I'm not sure how/whether this could be accomplished but perhaps rather than dropping a percentage it might be possible to use a deviation method to identify "outlier" frames that are say x% below average - with the user selecting x. Perhaps its too complicated, outside obvious poor frames I often struggle to know what to keep and what to discard and rely on the Integration "Quality" weight. If it could be done it would be a great time saver. 

Incidentally I did a test a while ago to compare processing all lights vs the best lights - using Quality Integration. I couldn't see any difference in the resulting masters. I assume its the Quality setting that made this work so well, it would be interesting to repeat the test using Equal, perhaps when I get some spare time.

Keep up the good work, thanks again 🙂

Jon