Multi-Session Calib...
 
Share:

20th March 2020 - APP 1.078 has been released !

2019 November: Complete LRGB Tutorial of NGC292, The Small Magellanic Cloud by Christian Sasse, Astronomer in Charge of iTelescope.net

2019 September: Astro Pixel Processor and iTelescope.net celebrate a new Partnership!

[Solved] Multi-Session Calibration Doesn't Work - 1.075  

Page 1 / 2
  RSS

(@midnight_lightning)
Red Giant Customer
Joined: 2 years ago
Posts: 66
November 23, 2019 17:10  

I have reported this before but the multi-session pre-processing doesn't work. I previously mentioned Darks but the same thing happens with Flats. Its long and complicated so I will just give one example - something I have just experienced using 1.075. Similar issues occur with Darks, and though I haven't checked perhaps also with Dark Flats.

My Data

Session 1 - Blue and Red (300s exposures)

Session 2 - Green and Red (300s exposures)

Session 3 - Ha (1200s exposure)

Session 4 - Ha (1200s exposure)

Session 5 - Blue (300s exposure)

New Flats / Dark Flats were taken for each session (because I set up each night and dust etc moves about - I don't re-use Flats).

Darks of 1200s and 300s were taken after the first session and used for all subsequent sessions.

I loaded the Lights and Flats for the above using Multi-session and Multichannel. The lights all loaded fine but some of the Flats have been dropped from the list as follows:

Session 1 - ERROR - Blue Flats Loaded Ok. The Green Flats loaded initially but then subsequently went missing.

Session 2 - OK - The Green and Red flats loaded correctly. 

Session 3 - ERROR - No Flats loaded for Session 3. (I suspect they were there after loading Session 3 and then subsequently removed when I loaded Session4)

Session 4 - ERROR - Flats have been loaded but they are the ones for Session 3!

Session 5 - Ok - The correct flats are loaded for Blue data.

It looks like there is some logic that says if there is more than one Flats file for the same colour filter then just use one of them.

That's no good, particularly if sessions are months or years apart and dust bunnies etc are going to be completely different. It's also an issue for Darks as cameras change over time and sometimes different cameras are used in different session. 

Can this be looked at urgently please? 

Someone gave me a work-around for this issue but I cant get my head around it. Could anyone provide a workflow for processing multi-session multi-channel that ensures the correct calibration files are used for each session?

Thanks

Jon

 

UPDATE - I have attached a copy of the APP file list showing what I see after loading the data files above for the 5 sessions.

 

 

 

 

This topic was modified 4 months ago by Vincent Groenewold - Moderator
This topic was modified 4 months ago 2 times by midnight_lightning

ReplyQuote
(@vincent-mod)
Quasar Admin
Joined: 3 years ago
Posts: 1435
November 25, 2019 12:30  

Forwarded directly to Mabula..

This post was modified 4 months ago by Vincent Groenewold - Moderator

ReplyQuote
(@midnight_lightning)
Red Giant Customer
Joined: 2 years ago
Posts: 66
November 25, 2019 14:38  

@vincent-mod

Thanks Vincent. 

Is there a work around, I haven’t been able to process any new data since February?


ReplyQuote
(@vincent-mod)
Quasar Admin
Joined: 3 years ago
Posts: 1435
November 27, 2019 12:46  

Not sure as I never noticed this issue, Mabula is back from being abroad ands said he would attend to the forum shortly, let's wait and see what he has to say, that'll be faster in this case. 🙂 We'll get it sorted.


ReplyQuote
(@midnight_lightning)
Red Giant Customer
Joined: 2 years ago
Posts: 66
November 27, 2019 13:17  

@vincent-mod

thanks Vincent


ReplyQuote
(@midnight_lightning)
Red Giant Customer
Joined: 2 years ago
Posts: 66
December 3, 2019 10:37  

Any update?

I've waited since March for an update on this (also see previous ticket) and am going to have to learn how to do it in Pixinsight if it cannot be resolved.

I only bought APP for the pre-processing and have to say that I am disappointed with the level of support I am getting with this issue.

This post was modified 4 months ago 2 times by midnight_lightning

ReplyQuote
(@wvreeven)
Neutron Star Customer
Joined: 2 years ago
Posts: 162
December 3, 2019 11:30  

I understand your frustration.

Would it be possible for you to put together a minimal set of files that shows this behaviour, i.e. possibly one Light and one Flat for each filter and each session? That could help to reproduce the problem and for us to possibly help you find a work around so at least you can move forward with processing your images.

 

Clear skies, Wouter


ReplyQuote
(@vincent-mod)
Quasar Admin
Joined: 3 years ago
Posts: 1435
December 3, 2019 13:31  

Again, sorry for the delay; I've reached out to Mabula a few times but he hasn't attended the forum yet. I guess he's very busy at the moment, I messaged him again. @mabula-admin


ReplyQuote
(@midnight_lightning)
Red Giant Customer
Joined: 2 years ago
Posts: 66
December 3, 2019 18:44  

Hi Wouter, I started writing this hours ago and have made some progress so please see end first - unless you work with Mabula there probably isn't much you can help with but thanks for asking. 

I set up a Drop Box folder with one of each frame type along the lines you suggested, however I thought I would try a test loading of this data before posting and oddly it all loaded without any issues. I ran the complete process through to integration - all fine.

I'm wondering if this is because there were relatively few files in this test?

The data is here https://www.dropbox.com/sh/kqe94ilyogzl3j7/AAB_Yst6dR6EWK0Rm3yDtZPLa?dl=0   (Sorry had to delete this data to be able to supply full data)

I then tried to load the full data set again and got a different result to the last time I ran the full data. I didn't get any Flat frames go missing this time although I did lose some Bias and Dark frames from the files list. I'm wondering whether the missing files were actually dropped and not used, or just omitted from the list to save space.

e.g. Although I loaded (the same) BIAS for each Session and Filter, I'm left with one entry for BIAS which shows allocation to "All Channels Session 5" - I did use the same BIAS files for all sessions and Channels so although this shows "Session 5" perhaps it is actually used for all sessions which would be correct? I'm left guessing which is not great.

Similarly for Darks, I loaded Darks individually for all sessions and filters but ended up with Darks for Session 4 and Session 5 only (all channels) in the files list. Did it use these for the other sessions as well - who knows?

I kept a spreadsheet showing what was in the flies list after loading each Session's data.

 

I tried to run the calibration step after Load anyway but it gave this error.

Calibrate Error

. I have checked all subs for correct Filter/Type in Fits header in case there is a data issue but all look fine. I OK'd the flats error and let it run through and whilst it produced Masters for Flats and Dark Flats it only produced MD's for Session 4 and Session 5

I don't understand why Flats disappeared from the earlier run and not this one but there are various permutations of loading data - e.g. Using "Filter Header" or specifically naming the filter, loading sequence - perhaps I did something different.

Just tried another full load and this time everything loaded as expected - I suspect the difference was down to the Loading Procedure. I usually load data one session at a time, so I load Session 1 Lights, Flats, DF's, Darks, Bias. Then go to Session 2 and do the same and so on. (Yes I know there is some duplication where Dark and Bias data is reused but it fits my workflow and library structure to do it that way).

However, this time I loaded the Lights first, for all sessions, then the Flats for all sessions etc. When it came to loading BIAS I only had to select All Channels and All Sessions once. With Darks it was All Channels and S1, S2, S5 for the 300s Darks and All Channels with S3, S4 for the 1200s Darks. 

So, the data appears to have loaded ok and I have run through to Integration where the Masters look ok except for the Master Darks. Again, I only have two MD's for the five sessions, One for the 1200s subs labelled as Session 4 and one for the 300s subs labelled as Session 5. I don't know if these were also used for processing Sessions 1, 2 and 3 but if so it would help if they were labelled as having done so. (Note - when I ran the test session, at the top of this post, I got all 5 MD's, one for each session as expected)

Given the results of my testing today, I've lost count of how many days I've spent on this, it appears that that you get inconsistent results depending on how you Load data - I don't know what the trigger is.

There is also still the question of Darks, and sometimes Flast, disappearing from the files list and the final masters. I just noticed that in the full test whilst I only got 2 of 5 MD files produced, the files list indicates they were used correctly across all 5 sessions. 

e.g. One MD file was labelled "Session 5" but I can see from the file list it was associated with Sessions 1, 2 and 5 which is correct.

Hope this helps resolution.

Thanks

Jon

This post was modified 4 months ago 4 times by midnight_lightning

ReplyQuote
(@wvreeven)
Neutron Star Customer
Joined: 2 years ago
Posts: 162
December 3, 2019 19:22  

Hi Jon,

Good to see that you are making progress. And thanks for explaining in so much detail how you load all your frames. To answer your question: I don’t work with Mabula but I do use APP a lot, also with multiple sessions and multiple filters, so I thought I’d chime in and at least offer a helping hand.

Learning to use a new user interface can be tricky and despite Mabula’s efforts to make APP as easy to use as possible there clearly is room for improvement. That is possibly why you end up with different file lists if the subs are loaded in a different order. My guess is that the UI has been designed with a specific use in mind (being: dark and bias subs can be reused over different sessions as long as the gain+offset (or ISO), temperature and exposure time are the same plus loading first all light subs, then flat subs, then dark subs and finally bias subs) and it doesn’t handle loading them in a different order well. I don’t know if this means that the master dark and master bias have been used for all sessions or only 4 and 5 (this is for Vincent or Mabula to answer) but I expect that they have been used for all sessions.

So, yes, it takes quite a bit of effort to get to learn APP and I agree that over the past months the responsiveness of Mabula (and at times of Vincent as well) has been poor at best but I can only imagine how hard it is to develop an application like APP alone whilst depending almost completely solely on one person to provide feedback to forum users.

I hope that now either Vincent or Mabula can take some time to answer your remaining questions and that you find joy again in using APP.

 

Clear skies, Wouter


ReplyQuote
(@midnight_lightning)
Red Giant Customer
Joined: 2 years ago
Posts: 66
December 4, 2019 10:32  

Hi Wouter, many thanks for stepping in to help, much appreciated.

I really like APP and have been suggesting it to people since I bought it but having paid for a full license I am disappointed with the level of  support since I logged this in March.  

The basic pre-processing Load logic facilitates a very nice method of building a matrix to map Lights, Sessions and Calibration files. It's fairly intuitive to use so I don't think this is about learning to use an interface. If the user explicitly says use these calibration frames for this session and these lights it shouldn't matter what order the files are loaded.

Only Mabula will know what the issue is but it feels like an engineering issue to me, hope it gets looked at soon.

Thanks again.

 Jon

This post was modified 4 months ago by midnight_lightning

ReplyQuote
(@vincent-mod)
Quasar Admin
Joined: 3 years ago
Posts: 1435
December 4, 2019 11:31  

Hi everyone, thanks Wouter. Yes I agree, support is lacking a bit and indeed it's difficult to have just 1 person answer all posts. I have my own knowledge which I try to use the best way I can in answering questions, but I'm also not all-knowing. Besides that, this is not my main job and we're in the process of relocation with my family, so that's why sometimes I'm offline for a week and you suddenly see a barrage of answers after that. 😉 I'm talking about this with Mabula shortly.


ReplyQuote
(@mabula-admin)
Quasar Admin
Joined: 3 years ago
Posts: 2366
December 4, 2019 12:56  

Hi Jon @midnight_lightning, Wouter @wvreeven & Vincent @vincent-mod,

First of all, Jon, apologies for keeping you in the dark for so long. I will put this issue very high on my list to get sorted out for the next release.

To be able to sort this out properly, it would be most helpfull if you can share your data, so I can run it myself to see what is happening.

To be clear, how frames are mapped should not depend on the loading order of course (unless you load the same frames twice), and also no frames should be dropped after you have loaded all of your frames. If you would load the same frames twice and assign them differently the second time, then yes, the first batch is dropped of course because you give them a new assignment.

It could be that the frame assignment for filters and sessions does not work consistently like you indicate, if so, then this needs to be sorted a.s.a.p. 😉

Would this data set be enough for me to test if fully you think?

I set up a Drop Box folder with one of each frame type along the lines you suggested, however I thought I would try a test loading of this data before posting and oddly it all loaded without any issues. I ran the complete process through to integration - all fine.

I'm wondering if this is because there were relatively few files in this test?

The data is here https://www.dropbox.com/sh/kqe94ilyogzl3j7/AAB_Yst6dR6EWK0Rm3yDtZPLa?dl=0  

I then tried to load the full data set again and got a different result to the last time I ran the full data. I didn't get any Flat frames go missing this time although I did lose some Bias and Dark frames from the files list. I'm wondering whether the missing files were actually dropped and not used, or just omitted from the list to save space.

Wouter, thank you very much for your assistance. That is highly appreciated 😉

 

Kind regards,

Mabula

Main developer of Astro Pixel Processor and owner of Aries Productions


ReplyQuote
(@midnight_lightning)
Red Giant Customer
Joined: 2 years ago
Posts: 66
December 4, 2019 20:46  

@mabula-admin

Hi Mabula, thanks for replying.

The test data on the earlier link was just one file each from Lights and calibration frames, and ran through without issue when I tried it, so not much use for verification.

I have put the full data in folders to share but unfortunately it is nearly 6GB after zipping and I don't have enough cloud storage so will have to transfer it in chunks. 

Can you try and download these and if successful I will clear down my One Drive and send the Dark Flats on another link.

https://1drv.ms/u/s!AhXsgtBn5aZUgYQyoiukHP7LwZ-MFw?e=ldiMSy  - Flats 

https://www.dropbox.com/sh/ovu74tlwzhpn75s/AAC10TaPyslYeoKb6ehbXd1da?dl=0 Lights, Bias, Darks

Just thinking through the testing I did yesterday, and your comments, it may be that the processing is working ok and that its the file list and file naming that is confusing. Note, I will have loaded the same data twice in some cases, particularly when I load each session's data in turn.

I have loaded the data in many different ways, e.g. by loading each session at a time, including loading some files multiple times (e.g. Bias that applied to all Channels, all sessions) and also by Frame Type, all the lights, then all the Darks etc.

So, for example, when I loaded data one session at a time I expected to see it build up in the file list as I loaded it but some files started to disappear - which ties in with your explanation about duplicates being removed. But in this case I would expect the remaining file to be labelled to show which sessions/channels it applies to - the APP Loading Sequence.txt in an earlier post shows how the files get dropped and how naming changes and fails to state what sessions/channels it applies to.

This is also reflected in the files produced in the final integration - I got two Master Darks, for S4 and S5, but nothing for S1, S2, S3. With hindsight it looks like the S4 file also covers S3, and the S5 file covers S1 and S2 - just needs to say this in the name to be consistent with other file naming.

I did notice that the file list, after Integration, indicates that the correct files have been applied to each session/channel which again suggests processing is ok and its the naming conventions that have thrown me.    

That said, when I loaded data by session yesterday it threw an error in calibration (see earlier post). When I reran it loading data by file type it ran through ok. Same files just different loading approach.

Note. The data supplied covers several multi-session permutations however doesn't include any sessions where multiple darks may be necessary, e.g. if different exposure times were used for RGB and Narrow Band on the same night.

Hope this helps.

Kind regards

Jon

 

 


ReplyQuote
(@midnight_lightning)
Red Giant Customer
Joined: 2 years ago
Posts: 66
December 6, 2019 09:39  

@mabula-admin

Hi Mabula, the files that I loaded for you a couple of days ago are taking up 100% of of my Dropbox and Onedrive allocations. Be good if you could download them so I can then add the dark flats before clearing them down so I can use them for other things.

Thanks

Jon

 


ReplyQuote
(@vincent-mod)
Quasar Admin
Joined: 3 years ago
Posts: 1435
December 6, 2019 10:17  

I'll drop him a message as well just to be sure, he's very busy preparing a meeting. You can't use our server by any chance?

This post was modified 4 months ago by Vincent Groenewold - Moderator

ReplyQuote
(@midnight_lightning)
Red Giant Customer
Joined: 2 years ago
Posts: 66
December 7, 2019 11:18  

@vincent-mod

I don't know about using your server, what would I need to do?

 


ReplyQuote
(@vincent-mod)
Quasar Admin
Joined: 3 years ago
Posts: 1435
December 7, 2019 16:35  

You can upload it to this address, log-in with username and password: appuser

Create a directory with your name and "Multi-session-Calib" added to the name. Put the files into there. Thanks!


ReplyQuote
(@midnight_lightning)
Red Giant Customer
Joined: 2 years ago
Posts: 66
December 8, 2019 23:36  

@vincent-mod

Thanks Vincent, took a few attempts due to timeouts but all uploaded now 🙂

Jon


ReplyQuote
(@vincent-mod)
Quasar Admin
Joined: 3 years ago
Posts: 1435
December 9, 2019 15:01  

Thank you! Either me or Mabula will have a good look at it.


ReplyQuote
Page 1 / 2
Share: