The problem with the data upload limit for attachments has been fixed. I have restored it to 30MegaBytes. A recent forum software upgrade was responsible for the changed limit. Please accept my apologies for missing this when upgrading the forum sofware, Mabula.
[Sticky] DSLR: how to check the linear histogram of your data - Or how long can I expose my flats?
DSLR camera's show you a distorted view of what your sensor measured. And knowing what our sensor measured is pretty important in our hobby.
Several data processing steps occur in-camera starting with the Raw data to create the image that you see on the display of your camera. Luckily, with APP we can verify all steps and actually see them 😉 So I will give a clear example with all steps using a CR2 flat frame of a Canon EOS 750D captured by Kirk @1cm69 . The frame can be downloaded here so you can follow the steps:
This topic is strongly related to the the question: how can I check in APP where the peak of the linear histogram is of my flats, to know how long I can actually expose my flats?
Step 1) start APP and load the CR2 frame, doubleclick on the frame to load it into the image viewer:
Bear in mind that APP automatically applies a DDP stretch, so you are seeing the flat already stretched with some DDP parameters.
Step 2) first we are going to load the frame linearly without any stretching:
- Preview filter on the right side: disable DDP
- Preview filter on the right side: and set B(lack) slider to 0 !
So we have disabled DDP stretching and we have set the black point to zero.
Some remarks to interpret what you see and what this means in APP :
- this is the linear unstretched data as captured by the sensor !
- The CR2 internal blackpoint (which is an offset just like CCD camera's) is not subtracted in APP in the linear image loader. All Canon DSLRs have an internal black point, see section 5.2 at: http://lclevy.free.fr/cr2/#wb Different models have different values, some have a value around 1024 ADU, some have a value around 2048 ADU. These are 14bits values.
- the camera is 14bits, and APP shows the 14bits in 16bits in the image viewer. So the actual ADU count in 14bits needs to be divided by 4 from what the image viewer shows.
- in creation of a masterflat from this, APP will use the 14bits data and calibrate in 14bits. After calibration, the data is converted to 16bits.
Check the histogram of the linear data:
- the green channel's peak is just past the middle of the histogram.
- blue and red are a bit weaker, but both are not far behind.
This flat frame was exposed with ISO100 and 2,5 seconds exposure time. If 5 seconds was used, the histogram would have hit the right side in the green channel and clipping would have occured which we need to prevent. This flat frame is well illuminated 😉
Step 3) now let APP show the conversion of the linear data to non-linear sRGB using the camera's model specific 3x3 matrix (camera -> XYZ -> sRGB) and the sRGB log 2.2 gamma curve:
- above the image viewer, select image instead of linear in the image viewer dropdown-box. This loads the image in the non-linear sRGB or Adobe 1998 mode for DSLR data.
Attention ! : this non-linear image loader is only used in the image viewer. And the non-linear image loader is NEVER used in (Canon) DSLR image data calibration and further processing ! So the non-linear image loaders are only used when you select image in the image viewer mode dropdownbox AND load an image into the image viewer.
If image is selected in the image viewer dropdownbox and you start with calibration and/or further processing, the data will only be processed by Linear Image loaders.
So the setting above the image viewer only has influence on how the data is loaded into the image viewer when you double-click on a frame in the frame list.
Step 4) like 3 but with Camera White Balance on, we should now see the non-linear sRGB data with white balance applied. This will be shown by the camera's display:
- in 0) RAW/FITS, select Camera White Balance
- reload the image by either double clicking the frame in the frame list panel, or by reselecting image in the image viewer mode dropdown-box
Check the histogram of this image and compare it to the histogram of the previous image without CameraWhite Balance, you can see that Camera White Balance applied gives the impression that the Red channel is almost/already clipping the histogram on the right side when it actually isn't 😉
The RGGB Camera White Balance parameters are hidden in the CR2's Makernote : ColorData tag field:
- select details above the image viewer, this will show you the extensive metadata of a Canon CR2 frame. Other brands like Nikon also have a MakerNote, and Exif. Those are separate metadata blocks containing different sort of information
This final image is what your camera will show, now let's compare it to step 2) which shows the linear data without any stretching (and no gamma conversion and matrix and White Balance):
Conclusion: the non-linear image of your camera is a lot different than what your sensor has recorded.
- linear data has still plenty of room before clipping of the data would occur on the right side of the histogram in the green channel. Exposure time could almost be doubled.
- The non-linear seems to be clipping in the red channel on the right hand side.
- Colors are different for 2 reasons, Camera White Balance and the 3x3 matrix to go from sensor/camera colors to XYZ neutral color space to sRGB colorspace.
- Non-linear data is stretched with a specific sRGB gamma curve which is almost equal to log 2.2.
how to check the linear histogram of your data - Or how long can I expose my flats?
- load the frame
- disable DDP
- set B(lack) slider to 0
- load the image in the linear image viewer mode
- and look at the histogram 😉 !
Don't focus on getting as close as you can to a certain median value of the flat frame. Looking at the histogram tells you everything you need to know !
AIM for the peaks to be around half way in the full linear range.
The flat in this example looks very well exposed, ALL 3 peaks of the RGB channels are close to half-way of the histogram. No clipping is happening on either side.
Be aware that for older camera models especially, exposing your flats past half way can give rise to problems. Older sensors tend to be not linear anymore when you expose to 2/3s of the histogram, give or take.
Main developer of Astro Pixel Processor and owner of Aries Productions
Everyone: let me know if all of this is clear 😉
Having just read through on my phone as I’m away from my PC, it seems all to make sense Mabula.
I’ll do a proper run through following your examples when next at my PC.
I dare say that I’ll be pestering you again before too long.
Thank you so much for doing this, much appreciated.
Hi Mabula @mabula-haverkamp-administrator I have now had time to go through this properly in front of my PC & it is fantastic, thank you.
Very easy to follow.
What I noticed is the image that you chose to use just happened to be one where I was experimenting with EV settings after reading somewhere that this may help with capturing Flats.
This image has a +2EV setting so because Aperture cannot be changed & ISO was set at 100 the equivalent shutter speed is 0.625s at EV0 as opposed to the 2.5s at +2EV.
I used the Exposure Calculator at: https://www.scantips.com/lights/exposurecalc.html
In theory this holds water but in practice after taking a number of captures at EV0 starting at 0.8s up to 2.5s I have found that there are variations & for me & my particular setup 1.0s frames at ISO100 with EV0 seem to give the best result.
Plus, altering the EV compensation can only be done physically on camera & there is the risk of either forgetting to do it & wasting valuable data or even leaving the EV at some other point except 0 and then carrying on capturing other data or calibration files, thus ruining them.
So after looking at this I decided that I would not use any EV compensation at all to negate this risk.
The other thing that I wanted to change was the use of T-shirts. For the frame that I captured used in this thread, I had to use 2 T-shirts to achieve an acceptable exposure length. To change from using the 2 T-shirts I added a 5mm thick opal acrylic sheet: https://www.amazon.co.uk/gp/product/B00WWPX1BQ to my light panel instead.
If you don't mind Mabula, I am going to PM you some further information & questions as to not muddy this thread.
Also, following Mabula's post about Internal Blackpoint, I also found this very in-depth page: https://sno.phy.queensu.ca/~phil/exiftool/TagNames/Canon.html
that explains all the data held inside Canon tags & using this you are able to determine the internal blackpoint and for my DSLR: https://sno.phy.queensu.ca/~phil/exiftool/TagNames/Canon.html#ColorData7
it is located at index 276 of 'MarkerNote - ColorData', visible in APP