RAW Files for Proce...
 
Share:
Notifications
Clear all

2022-05-29: APP 2.0.0-beta2 has been released !

Release notes

Download links per platform:

windows 2.0.0-beta2

macOS x86_64 2.0.0-beta2

macOS arm64 M1 2.0.0-beta2

Linux DEB 2.0.0-beta2

Linux RPM 2.0.0-beta2

[Sticky] RAW Files for Processing Practice

Page 2 / 3

(@1cm69)
Red Giant Customer
Joined: 5 years ago
Posts: 133
Topic starter  
Posted by: Wei-Hao Wang

Ah, unfortunately I ended my role in the APP beta test team because I don’t have much time to test it. So I no longer have a working APP installed in my computer. Someone else here might help.

OK, I'll wait for Mabula  @mabula-haverkamp-administrator


ReplyQuote
(@mabula-admin)
Quasar Admin
Joined: 5 years ago
Posts: 3180
 
Posted by: 1CM69

OK, I have just had another go at processing this data & I do not understand what is going wrong.

Every set of data I have tried to process in APP has shown major issues with the Flats.

I am following Mabula's @mabula-haverkamp-administrator steps here:  https://www.astropixelprocessor.com/a-dslr-data-calibration-workflow/ to the letter and nothing is working.

For this latest attempt I am using data from the folder - NIKON_D800 > 2016 > 20161106

and as specified in the LOG for this data I am using NEF files:

6111 - 6122 for BIAS ISO800 1/8000s

6281 - 6312 for FLAT ISO ISO800 1/20s

6323 - 6334 for Dark ISO800 180s

6262 - 6271 for LIGHT ISO800 180s

This is how I process:

  1. Load F & D to create BPM (this also creates MF & MD which I delete) then clear the load
  2. load F, B & BPM to create MF - then clear the load
  3. Load D to create MD - then clear load
  4. Load L, BPM, MF & MD and hit Calibrate

At this stage I took a screenshot of the Linear data & also the Calibrated data:

lin

 

cal

 

You can easily see the vignette in the Linear data but this should disappear in the Calibrated data but it doesn't, it just changes colour.

 

So what is going on??

@1cm69 & @wei-hao_wang,

Hi Kirk & Wei-Hao,

I do know what's wrong here with the flat-field calibration.

If you load flats, they need to be normalised before they are integrated into the masterflat.

The normalization engine has changed a bit in 1.059 and now the normalization of the flats unfortunately has a bug. 

I have seen this myself on several datasets now and it has been reported by several users. It causes the masterflat to undercorrect, showing a light frame that looks "over corrected". Since flat-fielding is about dividing the masterflat, a visual overcorrection, actually is a undercorrection in the vignetting profile that is in the masterflat.

I am working with the highest priority now to release APP 1.060. It will have a completely upgraded calibration engine and this problem should be completely gone. The new engine, will have less strict rules and will include dark scaling as well.

Kind regards,

Mabula


ReplyQuote
(@1cm69)
Red Giant Customer
Joined: 5 years ago
Posts: 133
Topic starter  
Posted by: Mabula Haverkamp - Admin
Posted by: 1CM69

OK, I have just had another go at processing this data & I do not understand what is going wrong.

Every set of data I have tried to process in APP has shown major issues with the Flats.

I am following Mabula's @mabula-haverkamp-administrator steps here:  https://www.astropixelprocessor.com/a-dslr-data-calibration-workflow/ to the letter and nothing is working.

For this latest attempt I am using data from the folder - NIKON_D800 > 2016 > 20161106

and as specified in the LOG for this data I am using NEF files:

6111 - 6122 for BIAS ISO800 1/8000s

6281 - 6312 for FLAT ISO ISO800 1/20s

6323 - 6334 for Dark ISO800 180s

6262 - 6271 for LIGHT ISO800 180s

This is how I process:

  1. Load F & D to create BPM (this also creates MF & MD which I delete) then clear the load
  2. load F, B & BPM to create MF - then clear the load
  3. Load D to create MD - then clear load
  4. Load L, BPM, MF & MD and hit Calibrate

At this stage I took a screenshot of the Linear data & also the Calibrated data:

lin

 

cal

 

You can easily see the vignette in the Linear data but this should disappear in the Calibrated data but it doesn't, it just changes colour.

 

So what is going on??

@1cm69 & @wei-hao_wang,

Hi Kirk & Wei-Hao,

I do know what's wrong here with the flat-field calibration.

If you load flats, they need to be normalised before they are integrated into the masterflat.

The normalization engine has changed a bit in 1.059 and now the normalization of the flats unfortunately has a bug. 

I have seen this myself on several datasets now and it has been reported by several users. It causes the masterflat to undercorrect, showing a light frame that looks "over corrected". Since flat-fielding is about dividing the masterflat, a visual overcorrection, actually is a undercorrection in the vignetting profile that is in the masterflat.

I am working with the highest priority now to release APP 1.060. It will have a completely upgraded calibration engine and this problem should be completely gone. The new engine, will have less strict rules and will include dark scaling as well.

Kind regards,

Mabula

Phew!! Thanks Mabula for getting back to me - thought I had totally messed something up.

Looking forward to next release.

Regards..,

Kirk


ReplyQuote
(@mabula-admin)
Quasar Admin
Joined: 5 years ago
Posts: 3180
 
Posted by: 1CM69
Posted by: Mabula Haverkamp - Admin
Posted by: 1CM69

OK, I have just had another go at processing this data & I do not understand what is going wrong.

Every set of data I have tried to process in APP has shown major issues with the Flats.

I am following Mabula's @mabula-haverkamp-administrator steps here:  https://www.astropixelprocessor.com/a-dslr-data-calibration-workflow/ to the letter and nothing is working.

For this latest attempt I am using data from the folder - NIKON_D800 > 2016 > 20161106

and as specified in the LOG for this data I am using NEF files:

6111 - 6122 for BIAS ISO800 1/8000s

6281 - 6312 for FLAT ISO ISO800 1/20s

6323 - 6334 for Dark ISO800 180s

6262 - 6271 for LIGHT ISO800 180s

This is how I process:

  1. Load F & D to create BPM (this also creates MF & MD which I delete) then clear the load
  2. load F, B & BPM to create MF - then clear the load
  3. Load D to create MD - then clear load
  4. Load L, BPM, MF & MD and hit Calibrate

At this stage I took a screenshot of the Linear data & also the Calibrated data:

lin

 

cal

 

You can easily see the vignette in the Linear data but this should disappear in the Calibrated data but it doesn't, it just changes colour.

 

So what is going on??

@1cm69 & @wei-hao_wang,

Hi Kirk & Wei-Hao,

I do know what's wrong here with the flat-field calibration.

If you load flats, they need to be normalised before they are integrated into the masterflat.

The normalization engine has changed a bit in 1.059 and now the normalization of the flats unfortunately has a bug. 

I have seen this myself on several datasets now and it has been reported by several users. It causes the masterflat to undercorrect, showing a light frame that looks "over corrected". Since flat-fielding is about dividing the masterflat, a visual overcorrection, actually is a undercorrection in the vignetting profile that is in the masterflat.

I am working with the highest priority now to release APP 1.060. It will have a completely upgraded calibration engine and this problem should be completely gone. The new engine, will have less strict rules and will include dark scaling as well.

Kind regards,

Mabula

Phew!! Thanks Mabula for getting back to me - thought I had totally messed something up.

Looking forward to next release.

Regards..,

Kirk

Hi Kirk @1cm69,

Yes, I am to blame here, I had to double check this before releasing 1.059. As soon as work is completed on the new calibration engine, I'll release it 😉

I will also put this problem in a seperate topic so everybody is aware that there is a bad bug here in 1.059.

Mabula


ReplyQuote
(@1cm69)
Red Giant Customer
Joined: 5 years ago
Posts: 133
Topic starter  
Posted by: Mabula Haverkamp - Admin
Posted by: 1CM69
Posted by: Mabula Haverkamp - Admin
Posted by: 1CM69

OK, I have just had another go at processing this data & I do not understand what is going wrong.

Every set of data I have tried to process in APP has shown major issues with the Flats.

I am following Mabula's @mabula-haverkamp-administrator steps here:  https://www.astropixelprocessor.com/a-dslr-data-calibration-workflow/ to the letter and nothing is working.

For this latest attempt I am using data from the folder - NIKON_D800 > 2016 > 20161106

and as specified in the LOG for this data I am using NEF files:

6111 - 6122 for BIAS ISO800 1/8000s

6281 - 6312 for FLAT ISO ISO800 1/20s

6323 - 6334 for Dark ISO800 180s

6262 - 6271 for LIGHT ISO800 180s

This is how I process:

  1. Load F & D to create BPM (this also creates MF & MD which I delete) then clear the load
  2. load F, B & BPM to create MF - then clear the load
  3. Load D to create MD - then clear load
  4. Load L, BPM, MF & MD and hit Calibrate

At this stage I took a screenshot of the Linear data & also the Calibrated data:

lin

 

cal

 

You can easily see the vignette in the Linear data but this should disappear in the Calibrated data but it doesn't, it just changes colour.

 

So what is going on??

@1cm69 & @wei-hao_wang,

Hi Kirk & Wei-Hao,

I do know what's wrong here with the flat-field calibration.

If you load flats, they need to be normalised before they are integrated into the masterflat.

The normalization engine has changed a bit in 1.059 and now the normalization of the flats unfortunately has a bug. 

I have seen this myself on several datasets now and it has been reported by several users. It causes the masterflat to undercorrect, showing a light frame that looks "over corrected". Since flat-fielding is about dividing the masterflat, a visual overcorrection, actually is a undercorrection in the vignetting profile that is in the masterflat.

I am working with the highest priority now to release APP 1.060. It will have a completely upgraded calibration engine and this problem should be completely gone. The new engine, will have less strict rules and will include dark scaling as well.

Kind regards,

Mabula

Phew!! Thanks Mabula for getting back to me - thought I had totally messed something up.

Looking forward to next release.

Regards..,

Kirk

Hi Kirk @1cm69,

Yes, I am to blame here, I had to double check this before releasing 1.059. As soon as work is completed on the new calibration engine, I'll release it 😉

I will also put this problem in a seperate topic so everybody is aware that there is a bad bug here in 1.059.

Mabula

Mabula, wouldn't it be an idea to have a BUG section on the forum where you can list known bugs?


ReplyQuote
(@1cm69)
Red Giant Customer
Joined: 5 years ago
Posts: 133
Topic starter  

Purely as a matter of interest I just ran the M42 data through DSS.

Loading all the files, as stated above, at once and letting DSS do it's thing.

Then I tweaked the Autosave file in PS.

Not too far away from the PI version

M42 Data via DSS

 

I did take a note of DSS processing steps

  • creates a MB from the Bias files
  • subtracts the MB from all Darks
  • creates MD
  • subtracts MB from Flats
  • computes Flat calibration params
  • applies Flat calibration params
  • creates MF
  • registers Lights
  • subtracts MD
  • applies MF
  • stacks final image

 


ReplyQuote
(@mabula-admin)
Quasar Admin
Joined: 5 years ago
Posts: 3180
 

Hi Kirk,

Thank you for sharing, I am making good progress now on APP's new calibration engine.

I will download Wei Hao's (@wei-hao_wang) data as well for testing and will show you the result in the coming days.

Cheers,

Mabula


1CM69 liked
ReplyQuote
(@1cm69)
Red Giant Customer
Joined: 5 years ago
Posts: 133
Topic starter  

No problem at all Mabula  😉 


ReplyQuote
(@mabula-admin)
Quasar Admin
Joined: 5 years ago
Posts: 3180
 

Hi Kirk & Wei-Hao @1cm69 & @wei-hao_wang,

I have started testing the new data calibration engine (will be part of APP 1.060) of APP with Wei-Hao's data.

I started with downloading the Canon5D2 2011_07_31 folder. I found M17 data with matching flats and darks.

I downloaded bias frames for both ISO 200 & ISO 1600 for the lights and the flats.

I have calibrated using all bias, darks with dark scaling, and flats and I have no problem with the flat-field calibration.

This is great data by Wei-Hao 😉 thanks for sharing !

Wei Hao Wang M17 MtHo Huan processing Mabula LQ

I think the google drive download errored as well, because I can't find the data of M16 that Kirk showed in the downloaded zip...

Next test will be the Nikon D800 data 😉

Kind regards,

Mabula


ReplyQuote
(@1cm69)
Red Giant Customer
Joined: 5 years ago
Posts: 133
Topic starter  

That’s a nice result Mabula. 

I found that downloading from the Google drive was hit & miss at times. 

I ended up opening the data folders & selected 10 files at a time to download, this seemed to work. 


ReplyQuote
(@mabula-admin)
Quasar Admin
Joined: 5 years ago
Posts: 3180
 

Hi Kirk &Wei-Hao, @1cm69 @wei-hao_wang

I have processed the Nikon D800 M42-A7 data using the files as mentioned in Wei-Hao's log file.

Indeed, APP had trouble with flat-field correction on this dataset. So I tested and have adjusted several things in the APP flat-field modules, the problem was due to 2 factors:

  • a technical problem in calculating the location and scale of each flat using multiple cores.
  • the reference lokation and scale used for normalizing the flats between each other is adjusted. The flat is chosen that has median location of all flats.

 

To compare the result APP gives now, I also ran the data through PI, like Wei-Hao did as well:

These are the 2 stacked results, only stretched with auto DPP in APP with saturation enabled. Very strong stretch with much saturation to clearly see everything.

Left is APP (AAD debayer, corrected bad pixels with BPM), right is PI (VNG debayer, still shows hot pixels..):

APP integration
PI integration

So APP does perform the flat-calibration now correctly 😉 I will run some more tests on more data of Wei-Hao and others before releasing APP 1.060.

The Nikon D800 sensor has a shutter technique problem showing of in the masterflat's rejection map, and to top it off, Wei-Hao's Nikon D800 apparently has different sensor dimensions as the original D800 model. Nikon or the sensor manufacturer has changed the sensor dimensions, but the camera is sold under the same name. (perhaps the sensor manufacturer has changed the sensor model?)

I have caught this exception in APP for D800 (and D800E) camera's for APP 1.060, so APP will correctly remove the border. PI (and thus dcraw) is still showing the much thicker top raw raw border on the top side as you can see in the PI integration. The new sensor dimensions are 64 larger in height, and 64 more pixels need to be removed from the top border which dcraw isn't doing yet.

Both issues are shown in the comparison, the dark stripe shown in the masterflat's rejection map reveals uneven illumination on parts of the sensor. Without data rejection on the masterflat this issue is shown identically on the integration, so it's no issue caused by data rejection.:

compare

Kirk, thank you again for pointing out the flat-fielding issue. Flat-fielding worked correctly on a lot of other datasets, but it clearly wasn't robust enough for other datasets as this dataset showed.

Mabula


ReplyQuote
(@1cm69)
Red Giant Customer
Joined: 5 years ago
Posts: 133
Topic starter  

Great news Mabula, sounds like it’s coming along a treat. 

Kirk


ReplyQuote
(@wei-hao_wang)
Brown Dwarf Customer
Joined: 5 years ago
Posts: 11
 

Hi Mabula,

The dimension difference should be caused by the firmware hack.  Otherwise the camera is very standard, just like all other D800s.

Good to see APP's calibration working well now.

Cheers,
Wei-Hao


ReplyQuote
(@mabula-admin)
Quasar Admin
Joined: 5 years ago
Posts: 3180
 

Hi @Wei-Hao_wang and Kirk @1cm69 ,

Yes, indeed, at Nikon Hacker:

https://nikonhacker.com/viewtopic.php?f=2&t=2319&p=16478&hilit=D800+dimension#p16478

Update:
Some user report conflict with existing tethering software. In most cases, it's due to the software not supporting dynamic RAW data format, as capture itself usually succeed before program crash when downloading the overscan NEF data. It's possible that such as image dimension, or compression mode was hard coded into the software. DigiCamControl (latest 1.2) will support dark current enabled DSLRs.

I have added the following in the release Notes of APP 1.060:

FIXED, NIKON D800, D800E NEF overscan support,  the RAW image dimensions of Nikon DSLRs hacked by Nikon Hacker firmware's can be different. This is due to loading NEF overscan data with the hacked firmware. This is handled correctly for these models now, thanks to data from Wei-Hao Wang @wei-hao_wang.

(It is easy to support other hacked Nikon models correctly, I just need to see a single raw of such an hacked model.)

Okay, back to calibrating Wei-Hao's data... I have gone a step further now. The flat analysis is now more sophisticated and can even reject the bad flats that are present using robust statistics to compare the flats for location and dispersion of the flat histograms 😉 .

About 50% of the flats (15 out of 32) are rejected in this case automatically, so the D800 sensor shutter is a problem to take into account. Left is with the bad flats removed, right is just processed with all provided flats:

Bad Flat detection enabled
All flats used

3 examples of flats: 1 good and 2 bad:

Good Flat
Bad Flat 2
Bad Flat

Clearly by looking at the flats, the problem is apparent. In the integration it caused a dark band without removing the bad flats.

I'll now proceed with other data of Wei-Hao and other's to further test the robustness of all of this..

Kind regards,

Mabula

 


ReplyQuote
(@1cm69)
Red Giant Customer
Joined: 5 years ago
Posts: 133
Topic starter  

This is brilliant work Mabula, can’t wait to try the new version.  😀 


ReplyQuote
(@wei-hao_wang)
Brown Dwarf Customer
Joined: 5 years ago
Posts: 11
 

Hi Mabula,

It's interesting to see those flat images.  I think the horizontal bands are a combined effect of the camera shutter and the flickering of the illumination source (a flat LED panel).  What I usually did is to average enough of them (more than 20 or even 30), so the effect can be averaged out.  To achieve this, I specifically set to have no frame/pixel rejection for averaging the flats.

That being said, I do not recall seeing flat images this bad in a regular basis.  On the camera's LCD screen, most of them look OK, if not all.  At some point I need to check whether this is a particularly bad set of flats, or indeed most of my flats taken with the LED panel are like this.

Cheers,
Wei-Hao


1CM69 liked
ReplyQuote
(@mabula-admin)
Quasar Admin
Joined: 5 years ago
Posts: 3180
 

Hi @wei-hao_wang,

Yes, it's the first time I see it like this.. 50% of the flats of the set have these problems. Indeed, must be combination of both shutter and the illumination source I think. Longer exposures should make it much less problematic (which tends to be the case for a lot of flat-field problems.)

I agree, it might be better to not reject pixels in the pixelstacks and average the frames. But in addition, I managed to add a robust detection method for these deviating frames and it works very well. The deviating frames are not integrated in the masterflat.

The way I accomplish this is by assuming that any regular and correct flat will have a certain/fixed ratio (within a certain error window) between

  • a central value of a specific corner area of the flat
  • to the central value of the center area of the flat

I calculate this ratio for two opposite corners. Then the deviating frames are easily detected by comparing the 2 ratios per flat to the ratio's of other flats.

In a flat set without deviating flats , only 2.5-5% of the flats are not used. In this flat set, 50% is rejected, I did a visual verification and it was 100% correct.

Need to do some more testing though 😉

Mabula


1CM69 liked
ReplyQuote
(@wei-hao_wang)
Brown Dwarf Customer
Joined: 5 years ago
Posts: 11
 
Posted by: Wei-Hao Wang

Hi Mabula,

I understand your method.  It can even be generalized to divide the image into a few large grids, which is equivalently shrinking the image to very low resolution.  Then the "low-res" images can be quickly compared with each other to find outliers.

However, no matter in this grid method, or your center vs corner method, there is a danger.  In relatively wide-angle imaging (with focal length less than 300 mm or so), sometimes we do twilight sky flat.  However, there could be gradient in the twilight, no matter where we point our lens to.  The standard way to solve this problem is to point the camera at many different parts of the sky as well as to rotate the camera angle. Then the many flats can be averaged and hopefully the sky gradient will be averaged out.  In this situation, it is expected that every flat looks somewhat different from each other, and every flat is still needed.  The rejection method you mentioned may remove some of the flats in this situation.  

A potential solution is to allow the users to turn off this option. However, adding too many options to the UI will confuse the users and make the program less user friendly.

Cheers,
Wei-Hao

Thanks @wei-hao_wang

Yes, I fully agree, this separate rejection method will be off by default 😉 for the mentioned reason. I will think about leaving it as an option in the GUI or not. Probably this issue is best dealt by the photographer himself understanding the issue with the sensor and flatpanel combined so I might not include it in APP's next release.

Thank you for your thoughts on this matter 😉

Mabula

 


ReplyQuote
(@mabula-admin)
Quasar Admin
Joined: 5 years ago
Posts: 3180
 

Hi @Wei-Hao_wang and Kirk @1cm69,

This is my processing of Wei-Hao's Nikon D800 NGC253 data of 2016-11-04 with all of the supplied flats processed with APP 1.060-beta up until star color calibration. The flat-field correction is working well (no under correction in the corners) except for part of the sensor chamber showing at the bottom. Left is the whole Field Of View of the data, right is a crop 😉 : One of the dust spots isn't calibrated out, since this apparently isn't in all subs..

Wei Hao Wang NGC253 Processing Mabula
Wei Hao Wang NGC253 Processing Mabula crop

This is only 10 x 488seconds, so I should probably integrate with median integration since I do see a satellite stripe... this one is with average integration and no outlier rejection.

Cheers,

Mabula


1CM69 liked
ReplyQuote
(@mabula-admin)
Quasar Admin
Joined: 5 years ago
Posts: 3180
 

Version 2 of Wei-Hao's NGC253 with median integration to get rid of the satellite stripe and a darker background now:

Wei Hao Wang NGC253 Processing Mabula crop ver2

Cheers,

Mabula


1CM69 liked
ReplyQuote
Page 2 / 3
Share: