15th Feb 2024: Astro Pixel Processor 2.0.0-beta29 released - macOS native File Chooser, macOS CMD-Q fixed, read-only Fits on network fixed and other bug fixes
7th December 2023: added payment option Alipay to purchase Astro Pixel Processor from China, Hong Kong, Macau, Taiwan, Korea, Japan and other countries where Alipay is used.
Clearing previous left over calibration/integration info?
I want to change the flats used on a previous integration session and I also want to add more light frames. Moved the unwanted MF(master flat) and MD(master flat dark) out of the APP working directory as they were to be replaced, left only BPM,MB,MD (for lights), I then copied all my original lights onto my latest acquisition folder which included all my additional lights,new flats, and flat darks and then loaded the 3 masters from the app working directory. (btw, should I even be loading a master bias being I have matched dark set and APP does no dark scaling anyways?) Issue is that after calibrating I noticed some/many of the lights don't show a D for darks having been performed down in the file list window. Rerunning calibration didn't seem to offer anything different except overwriting the masters it had already made. If I clear all the files out or reload app and reload lights and just it's matching Master Dark it quickly relists showing the D without having to actually redo calibration, but still only on some files. To heighten the mystery it's not just on the new lights frame, happens to both original and the new light frames. Not spotting any rhyme or reason for this. I went ahead and hit calibrate with just the lights and the master dark and but still no luck. Am including pic of the list with lights and darks loaded and then one showing the list and log afterwards..
This does look strange, can you send me the masterdark and some of the light frames? Ones that get the D and ones that don't?
Did you change the "exposure tolerance" in 2) calibrate for the master dark ?
Normal behaviour should be that the master dark is matched (so get the D mark in the frame column) when ISO and exposure time (within the exposure tolerance %) match.
Nothing of exposure tolerance was changed from whatever is default. A correction worth mention though is that the lights from both sessions are not exact match with my dark set. Darks were 300sec and lights are all 270sec. Not sure if the first run of them had the D not showing up. Is the D a sign that it tried and then successfully did calibrate or is it just a sign that it tried?.. or is it also a sign that it tried but could be a fail? (see idk if it's even trying) They are matched iso but 30sec in time off. So it sounds your saying exposure tolerance may well fix this right up.. What would you think is a number to change it to? The temps run from 5-8c on the lights and 5-7 (mostly 7c) on the darks if ya think that might matter. I'm to understand though that far as any future plans to scale darks go they can scale by time if bias subtracted but they don't scale well by temp due to non linear aspects, is that correct
Th default value of the exposure tolerance is 10%. 10% of 300seconds is 30, so in that case exposures between 270 -330 will match with the 300 second masterwork. Some lights are just in this range, sow will be just out of the range. If you increase the exposure tolerance to 15% for instance, then everything between 255 - 345 will match with the 300 second masterwork, that should fix it for you 😉
Yes dark scaling for time will be easily to do, but for temperature is very hard indeed, due to non-linear behaviour of signals like amp-glow.
Update: I changed the exposure tolerance from 10(default?) to 15 and all lights acquired a D. Comparing linear to L-calibrate I was able to see that yes there is calibration going on. There isn't much noise this cam down at 5-8c due to the tec peltier cooling but there is some hot white tracking noise/snow I would like to see gone. If I don't dither between shots they can run rampant into wavy streaks if settings aren't just right in the process. Apparently cooling and mismatched darks can only do so much to rid it all. The darks look to have taken out the few red and blue pixles but the whites only about 50% of them. Some get fainter by half or so but others stay just as bright. I would think it'll all pretty much go once integrated but if I should change an iteration or kappa setting while making the master or while running the master on the light that be great. Just used defaults to do the master, maybe I should turn on rejection or such??
Here's a sub showing linear on left and dark calibrated on right:
Ahh we cross posted.. but hey it looks like I read your mind in choosing the 15%.. 😉 umm why does the system show me these posts are around midnight your time when it's 3pm my time..hmm
There was some other questions above if ya get a chance..such as- (btw, should I even be loading a master bias being I have matched dark set and APP does no dark scaling anyways?)
First try to create a Bad Pixel Map if you haven't done so already. that should help a lot with the non-linear hot pixels you are referring to I would think
Attempted putting an edit to last post but that ability was lost..
Apparently what I thought were plenty big changes in adu for my flats don't end me with much change if viewing a single light sub calibrated via l-calibrate viewer so clearing and updating in APP must all be working ok in the background as it should. Unticking and ticking various flat versions in the file list seems to be working...However I for one would enjoy explanation why it is that both my last integrations on both the Iris Nebula and the Omega Nebula don't at all show the level of overcorrection that a single shows in l-calibrate. I assume I still need to drop down in adu some but perhaps I got to do a full integration to know it.
Following is a screen grab of 3 different level flats put against a single light viewed in l-calibrate. The first on left is my latest level that I have not yet integrated fully with all the lights, second is one I used on a first Iris Nebula intergration (final stack did not end up nearly as overcorrected as the viewer would lead us to believe), then third is a longer 2sec flat made for previous Omega Nebula shot. it's way brighter still in the viewer against at least this Iris data but I included it for consideration as it did rather fine against the Omega data.Why does l-calibrate not correlate with what the final is going to be? Seems to currently be way off base..
Regarding flats and the ADU value that you're playing with. As long as you expose your flats for several seconds and make sure that the peak of the flat's histogram is in the linear domain of your sensor (usually anywhere between 0,10-0,75 of the dynamic range of the sensor for the newer DSLRs) the exact ADU value shouldn't have an influence on how flats turn out. If this is still the case, then the method that you use in making flats is probably not working or not very effective in sampling the vignetting function of your optics I am afraid.
In flat calibration, APP will maintain the correct ADU values for your light frames and will not destroy the ratios between color channels, unlike other programs out there.
The l-calibrate function shows your frames calibrated, if you load masters then they will be used directly in the l-calibrated mode if they match with the light frames. In 2) Calibrate you indeed make new masters and it prepares APP for automatic workflow from 2) to 6).
The images that you show, really look a bit like you are subtracting the bias pedestal /offset twice from your lights? If you are using a masterdark and a masterbias for your calibration, then try to only use the masterdark.
The l-calibrate mode should confirm you that calibration is working, I always check this and will only proceed with processing if it does 😉
I am not aware of cases where l-calibrate does not match the calirbated frames that are loaded into the integration. If it doesn't match like you indicate, then between loading l-calibrate and the final integration you most likely have altered things regarding the loaded master frames?
What was changed recent for the master dark was the going from 10 to 15% exposure tolerance due to the mismatched darks being 300sec and the Iris lights being 270sec but that ought not matter for this flats issue. See the 3 flat pic above did not have any bias or darks loaded at all. I stepped down from having any of the other files to try and figure out the issues. I'm still assuming that once the integration is ran without that Norton crash I described in other thread that I will end up with something very similar to my first low integration result as far as a tad bit of overcorrection in corners. It had the BPM, MB, MD (all 3 made in APP for other objects), then also the flats and flat darks. If I reading you right your saying I should end up similar to what l-calibrate is showing, Hence we're really back to square one in whether or not something is going on with APP not clearing something currently. There was no bias or darks that example above of the overcorrection even in the list and program had been restarted. While I totally get what your saying about APP not destroying the channel ratios and possibility of the flats acquisition not grabbing all the optical train aspects I gotta wonder what it does with lower and higher adu count flats.. It will of course change the result more or less right?. I've had plenty of successful flat correcting on other programs this setup minus the filter/fr combo.. Well now I'm back at that position down to just the fr in there (due to that illumination circle reflection issue) and am using pretty decent practices for flats creation. No pollution filter, daylight bulb, exposures between 1.3-2sec, diffuser on light instead of scope, dew shield in place, only taken at night with no intruding lights. Following pics may better describe what I mean:
This is linear view in APP of fully integrated stack with the middle flat of previous 3pic. I trust you agree it is not what that l-calibrate shows so plenty happened to smooth it out during integration or there's a current clearing of info issue. Dust bunnies are taken care of but it's green in corners and a tad brighter.Hey we can almost live with that via tools and cropping, The amp glow made it through on the right, mismatched darks didn't do any justice aparently.
If equalized/stretched in Photoshop to really show off the errors I get this:
It for sure highlights the darks/amp glow issue and but far as flats go I think it starts showing midway out bottom left the overcorrected vignette? Top left darker and whole bottom brighter on around to the glow I suppose is actual sky gradient/pollution. Bunnies are gone so that's good. What ya make of these, overcorrected a tad?
Here was the outcome after processing:
It took some serious green tweaking. What with the issues of flats, darks, and being red zone photgraphy wide open no filter I could not get much definition out of the dust. Dust in dark sky locations block the stars and can actually be pulled out and defined with longer integrations.
The completed integration didn't work out. To our Dropbox I've uploaded the masters used on it along with 6 of the lights(3 from one night, 3 from another), 5 dark flats, 5 flats should you wish to try it from scratch.
While I totally get what your saying about APP not destroying the channel ratios and possibility of the flats acquisition not grabbing all the optical train aspects I gotta wonder what it does with lower and higher adu count flats.. It will of course change the result more or less right?. If equalized/stretched in Photoshop to really show off the errors I get this:
It for sure highlights the darks/amp glow issue and but far as flats go I think it starts showing midway out bottom left the overcorrected vignette? Top left darker and whole bottom brighter on around to the glow I suppose is actual sky gradient/pollution. Bunnies are gone so that's good. What ya make of these, overcorrected a tad?Here was the outcome after processing:
It took some serious green tweaking. What with the issues of flats, darks, and being red zone photgraphy wide open no filter I could not get much definition out of the dust. Dust in dark sky locations block the stars and can actually be pulled out and defined with longer integrations.
Hi Kevin, if flat calibration doesn't do normalization correctly then the channel ratio's are destroyed and different ADU values for your flats will definitely lead to different results, which is actually pretty bad and unwanted.
This is NOT the case in flat calibration in APP, as long as the flats are created properly and the flats intensities are in the linear domain of the sensor, then flats with different ADU values and flats with different color ratios will still have the same outcome due to proper normalization of the data in the flat-field correction:
I know that other application don't do this. A raw linear light frame with a blue background will still have a blue background after flat field calibration in APP independently from the color of the flats. In other applications the light frame can have a totally different color... it can even become red...
If you are still failing to get good flat correction (you are refering to overcorrection of the vignetting), then I am afraid you are still having problems with getting the bias offset subtracted once and only once for both light and flat frames. It will be the most likely cause of the error. Just send me a couple of light frames and the frames that you are using in calibration.
I know these rules are rather strict and stricter than what other programs do and use, I will soon make adjustments to make APP smarter in this regard 😉 Your data could possibly help very nicely in testing this.
The completed integration didn't work out. To our Dropbox I've uploaded the masters used on it along with 6 of the lights(3 from one night, 3 from another), 5 dark flats, 5 flats should you wish to try it from scratch.
Hi Kevin,
Indeed, I just checked the files that you uploaded to dropbox. The problem is the masterdark for your lights that you are using. It has the bias subtracted. So it will only work if
you use the masterbias iso400 together with the masterdark.
Or make a new masterdark and don't subtract the bias from it 😉
The flats and flatdarks are fine. So the flat has the bias offset/pedestal subtracted like it should.
In your case, using the supplied masterdark, the bias pedestal isn't subtracted from the lights, because the masterdark was already bias subtracted.
screenshot showing the darks, the black point of the preview slider is set at 0, so you can see that this dark is already bias subtracted. The FITS header info confirms this as well:
FITS HDUs: 1 HDU1 - SIMPLE = T / Java FITS: Fri Jun 02 21:40:09 PDT 2017 HDU1 - BITPIX = 16 / bits per data value HDU1 - NAXIS = 2 / number of axes HDU1 - NAXIS1 = 5280 / size of the n'th axis HDU1 - NAXIS2 = 3528 / size of the n'th axis HDU1 - EXTEND = T / Extensions are permitted HDU1 - BSCALE = 4.000183116645303 / scaling to 16bit HDU1 - BZERO = 32768.0 / offset data range to that of unsigned short HDU1 - DATE = '2017-06-03T04:42:29' / creation date of master dark HDU1 - SOFTWARE= 'Astro Pixel Processor by Aries Productions' / software HDU1 - VERSION = '1.041 ' / Astro Pixel Processor version HDU1 - CALFRAME= 'master dark' / master dark frame HDU1 - INSTRUME= 'CANON EOS REBEL T4I' / instrument name HDU1 - CFAIMAGE= 'RGGB ' / Color Filter Array pattern HDU1 - GAIN = 400.0 / gain or ISO depending on instrument HDU1 - EXPTIME = 300.20001 / exposure time (s) HDU1 - MEAN-R = ' 4.44E+00' / mean of channel-R HDU1 - MEAN-G1 = ' 4.46E+00' / mean of channel-G1 HDU1 - MEAN-G2 = ' 4.01E+00' / mean of channel-G2 HDU1 - MEAN-B = ' 4.05E+00' / mean of channel-B HDU1 - MED-R = ' 4.00E+00' / median of channel-R HDU1 - MED-G1 = ' 4.00E+00' / median of channel-G1 HDU1 - MED-G2 = ' 3.00E+00' / median of channel-G2 HDU1 - MED-B = ' 3.00E+00' / median of channel-B HDU1 - SIGMA-R = ' 1.42E+01' / standard deviation of channel-R HDU1 - SIGMA-G1= ' 1.53E+01' / standard deviation of channel-G1 HDU1 - SIGMA-G2= ' 1.33E+01' / standard deviation of channel-G2 HDU1 - SIGMA-B = ' 1.20E+01' / standard deviation of channel-B HDU1 - NOISE-R = ' 1.47E+00' / MRS gaussian noise estimate of channel-R HDU1 - NOISE-G1= ' 1.47E+00' / MRS gaussian noise estimate of channel-G1 HDU1 - NOISE-G2= ' 1.32E+00' / MRS gaussian noise estimate of channel-G2 HDU1 - NOISE-B = ' 1.31E+00' / MRS gaussian noise estimate of channel-B HDU1 - END
Oh Wow!.. That's rather counter intuitive. If I'm making masters they should be kept discrete for their future use as individuals in any set without worry if something of another calibration has been subtracted and saved to it.. Exception would be if I specifically told it to save a master with bias or dark subtracted. I aasumed the putting of the masters against each other and the lights is done strictly per session during the integration. How do other apps do this?? When I had first made that master dark I had not yet known the app doesn't scale darks. Far as I ever understood it that is the only reason for the bias files and/or for short length flats. There was no real need for it on that creation as I always have had dark flats for my flats. By these analogy of making masters with other calibrations built into them I suppose I also have flat masters floating around with the dark flat bias and dark signal removed also? They were all made original with dark flats but during these experimentation were put to the lights just by theirselves, actually wouldn't that be just fine as we want that signal out and it wouldn't be getting done twice? So whats their issue in being so overcorrected in l-calibrate? There ever a case when we wouldn't want the calibration of the flats saved to them? I don't really see a case where that would come to play but with bias and darks yes. I assume if I were to put existing master flat and a existing master dark flat into a new lights integration that the app would just ignore the dark flat master or would it go ahead and waste computer resource and time in going ahead and reperforming the same effort of overwriting already calibrated stuff? Think I seen it as a question pop up in program so all is fine I guess on that..
If I'm making masters they should be kept discrete for their future use as individuals in any set without worry if something of another calibration has been subtracted and saved to it
Well, to be honest, I can't really agree with you on this point, you need to categorize your masters if you want to use them later for other objects, you need to store the details of how you created it. In the future I'll add more header info so you can verify what actually has happened to the master.
The reason that the masters can be created differently is because different sensors ask for different calibration workflows and APP enables you to do just that. Understanding the different calibration paths that are possible will help a lot to get the most optimal calibration of your data.
I have been thinking a lot about making the calibration in APP more user friendly, and I do consider this very important so I'll start work on this after the weekend. It's clear that most errors occur in failing to understand how to subtract the bias offset of both lights and flat frames once and only once. I intend to create a warning system in APP to help the user in this regard to get the most optimal calibration with help from APP's suggestions. So APP should detect if the bias offset subtraction is done correctly and only once, for both lights and flats. If the supplied calibration frames can't do this, due to mismatch of exposure and/or bias/gain values, than APP will warn the user about it. I think that would help a lot.
Thanks for getting back to this.. Yea I understand various systems might call for different routes of calibration to a successful outcome. Older cam versus newer cam/lower noise, non cooled versus cooled sensor, short exposure versus longer (amp glow), etc. can equal one imager deciding to only use bias and no darks at all or vica versa..No darks for dslr is currently common in the dslr relm for example. Many believe they end up adding more noise..
For sure the master headers need to be upfront about it IF another master been subtracted from file, like maybe MD(-mb) and MF(-mdmb), as there always gonna be different cases of failed calibrations or decisions by tinkers to change workflow. Gotta have ability for pure masters discrete of each other. Can be automatic or by button, auto is good but doesn't matter which I guess, as long as the process doesn't involve user being limited to only having the one type of calibrations files in the list to do it. I doubt anybody interested in partial loading of calibration files to get discrete masters. Sounds simple my end to say it, probably not so easy your end to implement. ..The warning system sounds great 😉
So does this mean that the master bias and master flat that I produce can not be saved as such and then used in a different data set? Does this mean that I need to do new masters each time I run through the complete re-processing stage?
Please tell me in simple yes or no!!! I can't understand what on earth has been said already!!
Hi Sara, We will of course defer to Mabula when he gets back to this but I think in my case it meant that if I make a dark master while also having bias frames that session that I had better include the master bias on any future use of that master dark. Your case you only mention bias and flats but so I wonder why no mention of darks...
Realizing I have asked Mabula a ton of questions and considering this thread ended up taking a couple big twisty turns I gonna leave it where it is with his goal of making it user friendly with warnings and changes to header/file naming.
Hi Sara, We will of course defer to Mabula when he gets back to this but I think in my case it meant that if I make a dark master while also having bias frames that session that I had better include the master bias on any future use of that master dark. Your case you only mention bias and flats but so I wonder why no mention of darks...
Realizing I have asked Mabula a ton of questions and considering this thread ended up taking a couple big twisty turns I gonna leave it where it is with his goal of making it user friendly with warnings and changes to header/file naming.
Thanks for getting back to this.. Yea I understand various systems might call for different routes of calibration to a successful outcome. Older cam versus newer cam/lower noise, non cooled versus cooled sensor, short exposure versus longer (amp glow), etc. can equal one imager deciding to only use bias and no darks at all or vica versa..No darks for dslr is currently common in the dslr relm for example. Many believe they end up adding more noise..
For sure the master headers need to be upfront about it IF another master been subtracted from file, like maybe MD(-mb) and MF(-mdmb), as there always gonna be different cases of failed calibrations or decisions by tinkers to change workflow. Gotta have ability for pure masters discrete of each other. Can be automatic or by button, auto is good but doesn't matter which I guess, as long as the process doesn't involve user being limited to only having the one type of calibrations files in the list to do it. I doubt anybody interested in partial loading of calibration files to get discrete masters. Sounds simple my end to say it, probably not so easy your end to implement. ..The warning system sounds great 😉
So does this mean that the master bias and master flat that I produce can not be saved as such and then used in a different data set? Does this mean that I need to do new masters each time I run through the complete re-processing stage?
Please tell me in simple yes or no!!! I can't understand what on earth has been said already!!
Hi Sara,
No it doesn't ;-), you need to know however if a master bias is subtracted from the darks that created a masterdark for instance.
Usually masterbias frames can be used for more than 1 year to good effect.
And masterflats can be used for other projects/objects as well as long as nothing changes with the optical configuration.
BPMs only need to be created once in a very long time is my experience.
Master darks also can be used over and over again. But you do want to know if they are bias subtracted or not and to which temperature they belong. So you need to save them for further use with those details.
And as I indicated to Kevin, I'll soon start work on improving and smartening up the calibration engine 😉
Hi Sara, We will of course defer to Mabula when he gets back to this but I think in my case it meant that if I make a dark master while also having bias frames that session that I had better include the master bias on any future use of that master dark. Your case you only mention bias and flats but so I wonder why no mention of darks...
Indeed, that's what I meant. Sara can get good calibration with her sensor with only bias, flats and BPM calibration. It means that her sensor doesn't seem to have significant Amp glow or fixed patterns in the dark currents, which would warrant dark current calibration.
Realizing I have asked Mabula a ton of questions and considering this thread ended up taking a couple big twisty turns I gonna leave it where it is with his goal of making it user friendly with warnings and changes to header/file naming.
I realize that the calibration enigne, as it is, causes some confusion here and there. I will try to make it smarter, so the user will be "guided" in the calibration process to achieve the best calibration as possible.
Hi Sara, We will of course defer to Mabula when he gets back to this but I think in my case it meant that if I make a dark master while also having bias frames that session that I had better include the master bias on any future use of that master dark. Your case you only mention bias and flats but so I wonder why no mention of darks...
Realizing I have asked Mabula a ton of questions and considering this thread ended up taking a couple big twisty turns I gonna leave it where it is with his goal of making it user friendly with warnings and changes to header/file naming.
