Flat Field Calibration can not be performed correctly (Dark Flats don't match)

Mmm, if they indeed are identical in all settings as the flats, that should normally work. Are the dimensions identical too? Just as an extra check. Are the flats and the dark-flats taken with the same program?

@vincent-mod

Yes, all the same dimension and all taken with SGPro which I have used for years. I bought a QHY600 CMOS a year ago and during summer have had issues getting the temperature down for flats (very short exp and lots of heat from reading the sensor). Just ran a test using one channel and one session and it failed the same way. Checked the flats and dark flats headers and both had two each subs at -9.9C when all the rest were at -10C. The camera was set for -10C. Could this be teh issue, would 0.1C trigger an error? 

I will rerun the test tomorrow but using only the -10c subs frames and see if that works.

No, the temp doesn't matter. Just before I ask you to upload some data, could you show a screenshot of the fits  headers for both? I'll ask for data tomorrow.

@vincent-mod 

I just reran the integration but dropped the Dark Flats and Flats that were -9.9C and it ran through. I'm pretty sure I used all the same settings. 

UPDATE. I didn't make it clear here that I only ran through the data for Session 1, single Channel - this is important - see below.

I have 25 each of Flats and Darks Flats, do you want all 50 headers? Do you know of an easy way to extract the headers and any preferred format - e.g. do you want 50 jpeg screen shots or is there a better way?

Thanks

That is very interesting, I wouldn't have expected that. Could you just show a screenshot of the -0.9C en one of the ones that do work?

@vincent-mod

I cant say definitively there isn't something else at play here. The run that failed with all subs had multiple sessions and channels. The two successful runs I did with the -9.9C frames removed were single channel, single session, so there is a difference there which could also account for the results. I guess the next test would be to go through all subs in the full data set, remove any -9.9C frames and then rerun all sessions and channels. Probably needs to be done but it may be a while before I get time to do it.      

Can you let me know when you have finished with the headers so I can delete them, or just delete them yourself - dont want my "address" staying on line 🙂

Thanks, checked and removed. 😉 Nothing weird to see there so I'm at a loss a bit. Could you indeed do a small test with exactly the same settings for the files instead of multiple sessions etc? Just to make all parameters the same.

@vincent-mod

I think I already did that Vincent unless I am mid-understanding? The only settings that I’m not 100% sure were whether I chose Winsor or sigma rejection although am am pretty sure it was sigma for all tests. 

If it is something else you need could elaborate pls?

Thanks

jon

No, if you tested things like that that's fine. I'm going to ask for the data, just to see the details better. If you don't mind. 😉

Go to https://upload.astropixelprocessor.com and use upload as username and upload as password.

Create a directory named “midnight-lightning-calIssue” and upload in there. Thank you!

@vincent-mod 

Ok. Its an unfinished 4 panel mosaic so I have data for 3 panels so far and 6 sessions including 2 channels with multiple calibration files for each session. Each sub is 120mb.

Just wanted to check you need everything before I start. Just wondering if 1 panel, 1 channel, one session would do to start?

Well, I think just a few flats, some darkflats that work and some that caused an issue would be fine. Say, 10 for each.

@vincent-mod

Data added.

Flats - Frames 2,4,7,8 are all -9.9C (should have been -10C)

Dark Flats - Frames 9 and 17 are -9.9C

Darks - Frames 4,6,10 are -9.9C

Hope this helps.

UPDATE. I have just been through all calibration frames and changed the fits header to -10C where it originally showed -9.9C. I then ran all of the data through Calibration again and it failed with the same message as in the original post above. I need to do some more checking but it is starting to look as if the temperature difference was not the issue. The other thing I noticed is that I have the "Separate Darks by  Exposure" option ticked - I am not clear whether this just refers to "Darks" or also "Dark Flats" for which I have "Dark" in the fits header. Actually, could this be an issue, I used to give Dark Flats a header Type of "Flat" (I think erroneously) but it was causing issues in other software so I started using "Dark" in the header. I loaded them into APP using the correct Loader type so perhaps APP uses this rather than the header anyway - I don't know? 

@vincent-mod

Just spent the past 4 hours running tests and am finally getting somewhere - I don't think its anything to do with Temperature, it seems to be due to multi-session processing. Could it be mixing up the Flats and dark Flats from OIII and Ha?

This is my loading table for the first two sessions and was used in the following tests. Note "P1S1" stands for "Panel 1 (Mosaic) Session 1"

These are the tests I ran - Note when loading Flats and darks I always use "Apply Filter Header Tag"

So the outcome is as follows:

• Running Session 1 - Ha (Single Channel Single Session) works - regardless of whether the -9.9C calibrations frames are used. 
• Running Session 2 - OIII (Single Channel Single Session) works ( I also ran this with and without the -9.9C calibrations frames). 
• Running the same Session 1 and Session 2 Data (Same settings) as Muliti-Channel/Multi-Session fails with the Flats Calibration Error.

I think this is as far as I can go without understanding the code. One avenue perhaps worth exploring is there is a big difference in exposure times for the Ha (62s) and OIII(5s) Flats/Dark_Flats.

Clutching at straws here but..... is it possible that the wrong dark flats  are being used for calibrating the Flats in multi-channel/session processing (e.g. OIII Flats are being calibrated with Ha Dark_Flats), but that there is an exposure tolerance in place that usually means it doesn't trigger an error? (Or possibly using the right Dark_flats but mixing up/using the wrong exposure?)

@midnight_lightning

Not 100% sure, but if the fits header doesn't have a proper type (like flat or dark) that is recognized by APP, you have to properly load it with the correct buttons. Otherwise indeed it could be that APP uses it in a wrong way. Again, not super sure as I never had that particular situation.

Excellent analysis there, when running multiple sessions you also assign the darflats, flats etc. to the proper session, I guess you did this correctly?

@vincent-mod I definitely used the correct Lights/Darks/Flats..... loaders correctly for the appropriate files 🙂

Posted by: @vincent-mod

Excellent analysis there, when running multiple sessions you also assign the darflats, flats etc. to the proper session, I guess you did this correctly?

I'm pretty sure I did but can never be 100% sure, I did it several times and very carefully, it is easy to make a mistake. 

I wrote the loader chart to help eliminate errors and the individual single channel/single tests would have failed (hopefully) if I had got an error in the loader chart mappings (e.g. mixing OIII and Ha calibration frames)

I also checked all the files looked right after loading and couldn't see anything obvious.

I'm hoping there will be enough there for Mabula, or yourself, to narrow down possible causes but let me know if you need anything else - preferably before I forget what I have done so far, which at my age isn't long  🤣 

haha Well no worries. It is a bit difficult to find out if the darkflats etc indeed are just fine. Maybe we may need some more data to try and replicate your issue with Ha and OIII. Could you add some data for that to the folder? And add the master calibration files as well. I, or Wouter/Mabula, can then try your workflow.

@vincent-mod

I found the problem - I was on the right lines with my findings above but for a totally unexpected reason. 😫 🤣 😎 

I decided to blink all the data and check fits headers again before uploading. It was in the fits header, I didn't see it last time because I was checking all the data was correct, this time I noticed some missing data and that's the issue. 

I use APP for pre-processing and PI for post-processing and for years have assigned Dark Flats as "Type = Flat" - all good. 

In the past few months I have been using PI for pre-processing because it allows me to save projects and with complex multi-session/channel processing I find loading data to APP has a high risk of making a mistake and it provides little support in checking for mistakes. Also, as there is no way to save the loaded data for another day its a chore if you have to do it more than once (Sorry but saving projects has been requested a long time 😏 ). 

PI requires Dark Flats to have "Type=Dark" so I changed my SGPro process to name them as "Dark" rather than "Flat". 

My current project is a Mosaic and APP is simply the best software for Mosaic pre-processing that I am aware of, so I went back to APP for this project. 

When I loaded my Dark-Flats as usual I used the "Assign Filter Header" option - that was the issue. Because I had told SGPro to use "Type=Dark" it hadn't bothered to add the filter data to the header. Which in turn meant APP couldn't use the header. When I assigned the filter explicitly it worked. 

So it looks like it was trying to use the wrong Dark_Flats to calibrate the Flats. 

It's a nasty little gotcha, took me most of two days to work it out but essentially it's user error, although the software could provide more support. I think there is a an argument that when a filter is in use SGPro should record it in the header regardless of the sub being a Dark though I can understand why they don't.  

If I am right about the cause of the error I have a couple of suggestions for APP.

It would be good if the code could throw an error if the "Assign Filter Header" is used and there is no Filter Type in the header.

When the processing failed I was only using two sessions for the test with one channel in each session. The Flats and Dark Flats were unique to each session (i.e. not used across both sessions) and they were definitely assigned to the correct session. So, I am struggling to understand why this in itself didn't override the lack of filter type in the header i.e. There was only one set of Flats and one set of Dark Flats in each session - if only data assigned to that session was used I cant see how the error (as in original post) was generated. I could be wrong but it appears that, for example, the processing tried to use Ha Dark Flats from Session 1 to calibrate OIII Flats from session 2? It's maybe a moot point but it just makes me wonder if the files are being assigned correctly even when the correct filter type data is in the header. 

WBPP (the pre-processing script) in PI produces a diagram showing how calibration frames are used. It shows which Dark-Flats are used to calibrate which Flats, which Flats are used to Calibrate which lights and which Darks are used to calibrate which Lights. It gives a lot of confidence that data has been loaded correctly and assigned correctly by the software, especially when processing multi-session, multi-channel datasets. Perhaps APP could have something similar. It wouldn't need to be fancy, even a .csv/Excel export file showing how calibration files are being used/mapped would be very useful in checking that files have been loaded and assigned correctly (Something as simple as my Load Mapping chart above would be help).

Anyway, thanks for your help Vincent, I'm glad I found this before taking up any more of your time.

Please consider the new feature requests above as well as the existing one for saving projects - I think not being able to at least save the Load Table is a significant weakness and just doing this should be a minor development - I assume you already have a matrix of load data, maybe that could be saved and used to reload data providing the source files haven't moved location? 

Thanks Again.            

Jon 

PS. I am still uploading the data requested in case you want to take a closer look. 

Well Jon, I wouldn't have been able to solve this one I think. 🙂 These are the most trickiest as it totally depends on exactly what you did in combination with SGP in this case. Thanks a lot for your amazing analysis and I'll certainly forward your suggestions to Mabula.