15th Feb 2024: Astro Pixel Processor 2.0.0-beta29 released - macOS native File Chooser, macOS CMD-Q fixed, read-only Fits on network fixed and other bug fixes
7th December 2023: added payment option Alipay to purchase Astro Pixel Processor from China, Hong Kong, Macau, Taiwan, Korea, Japan and other countries where Alipay is used.
I made a decent stack yesterday, composed of two sessions, totaling 80 frames. Went through each frames manually to remove any bad ones, selected full integration, winsor 3 5iter, lnc 1d 1iter, mbb 5%. Woke up and the process failed due to some frames not passing star analysis.
I removed all frames but two, one of each session, added both MF, MD, MB and BPM. It seems that APP was attempting to star analyse my flat frames and obviously it finds no stars!
Here is a log snippet:
09:46:59 - GENERAL IMAGE LOADER: loading frame C:\Users\Shlomi\Pictures\digiCamControl\MB-ISO_gain_800.0-exp_2.5E-4s-40subs-NIKON_D3300-6016x4016-NR-avg.fits
09:47:06 - GENERAL IMAGE LOADER: frame C:\Users\Shlomi\Pictures\digiCamControl\MB-ISO_gain_800.0-exp_2.5E-4s-40subs-NIKON_D3300-6016x4016-NR-avg.fits was loaded successfully
09:47:06 - 2) CALIBRATE: Adaptive Data Pedestal re-initialized at: 0
09:47:06 - GENERAL IMAGE LOADER: loading frame C:\Users\Shlomi\Pictures\digiCamControl\Checking guiding\Bubble Nebula\APP\MD-ISO_gain_800.0-exp_121.800003s-26subs-NIKON_D3300-6016x4016-NR-avg-all_sessions.fits
09:47:13 - 2) CALIBRATE: Adaptive Data Pedestal re-initialized at: 0
09:47:13 - GENERAL IMAGE LOADER: loading frame C:\Users\Shlomi\Pictures\digiCamControl\Checking guiding\Bubble Nebula\APP\MF-lights1-lights2-ISO_gain_800.0-exp_0.02s-27subs-NIKON_D3300-6016x4016-NR-noBl-avg.fits
09:47:13 - GENERAL IMAGE LOADER: frame C:\Users\Shlomi\Pictures\digiCamControl\Checking guiding\Bubble Nebula\APP\MD-ISO_gain_800.0-exp_121.800003s-26subs-NIKON_D3300-6016x4016-NR-avg-all_sessions.fits was loaded successfully
09:47:22 - GENERAL IMAGE LOADER: frame C:\Users\Shlomi\Pictures\digiCamControl\Checking guiding\Bubble Nebula\APP\MF-lights1-lights2-ISO_gain_800.0-exp_0.02s-27subs-NIKON_D3300-6016x4016-NR-noBl-avg.fits was loaded successfully
09:47:22 - GENERAL IMAGE LOADER: loading frame C:\Users\Shlomi\Pictures\digiCamControl\BPM-NIKON_D3300-6016x4016.fits
09:47:23 - 2) CALIBRATE: performing calibration in 32bits normalized floats...
09:47:23 - 2) CALIBRATE: converting data to 32bits normalized floats...
09:47:23 - GENERAL IMAGE LOADER: frame C:\Users\Shlomi\Pictures\digiCamControl\BPM-NIKON_D3300-6016x4016.fits was loaded successfully
09:47:25 - 2) CALIBRATE: converting MasterDark to 32bits normalized floats...
09:47:26 - 2) CALIBRATE: Adaptive Data Pedestal raised to: 3.212E-04
09:47:27 - 2) CALIBRATE: Adaptive Data Pedestal raised to: 6.423E-04
09:47:27 - 2) CALIBRATE: Adaptive Data Pedestal raised to: 9.635E-04
09:47:30 - GENERAL IMAGE LOADER: frame C:\Users\Shlomi\Pictures\digiCamControl\Checking guiding\Bubble Nebula\Lights1\DSC_0022.nef was loaded successfully
09:47:30 - 3) ANALYSE STARS: DSC_0022.nef : starting analysis of stars in frame
09:47:30 - 3) ANALYSE STARS: DSC_0022.nef: starting...
09:47:30 - 3) ANALYSE STARS: DSC_0022.nef: getting luminosity channel...
09:47:30 - 3) ANALYSE STARS: DSC_0022.nef: creating star map...
09:47:32 - 2) CALIBRATE: Adaptive Data Pedestal: enabled
09:47:32 - 2) CALIBRATE: Adaptive Data Pedestal set at: 9.635E-04
09:47:32 - 2) CALIBRATE: Adaptive Data Pedestal re-initialized at: 0
09:47:32 - GENERAL IMAGE LOADER: loading frame C:\Users\Shlomi\Pictures\digiCamControl\Checking guiding\Bubble Nebula\APP\MF-lights3-ISO_gain_800.0-exp_0.02s-22subs-NIKON_D3300-6016x4016-NR-noBl-avg.fits
09:47:40 - 3) ANALYSE STARS: DSC_0022.nef: identifying star candidates...
09:47:40 - 3) ANALYSE STARS: DSC_0022.nef: star map created successfully
09:47:40 - 3) ANALYSE STARS: DSC_0022.nef: identifying star candidates: initial FWHM estimate: 6.98 pixels
09:47:40 - 3) ANALYSE STARS: DSC_0022.nef: identifying star candidates: target noise level is at kappa 185.0
09:47:40 - 3) ANALYSE STARS: DSC_0022.nef: identifying star candidates: target luminosity of stars is 9.727E-01
09:47:41 - 3) ANALYSE STARS: DSC_0022.nef: identifying star candidates: initially identified 1004 of possible star candidates
09:47:41 - 3) ANALYSE STARS: DSC_0022.nef: probing star positions and luminosities...
09:47:41 - 3) ANALYSE STARS: DSC_0022.nef: identified star candidates successfully
09:47:41 - 3) ANALYSE STARS: DSC_0022.nef: probed star positions and luminosities successfully
09:47:41 - 3) ANALYSE STARS: DSC_0022.nef: probing local background, noise and FWHM for all stars...
09:47:41 - 3) ANALYSE STARS: DSC_0022.nef: estimated Full Width at Half Maximum (FWHM) of star profiles : 6.68 +- 1.00 pixels...
09:47:41 - 3) ANALYSE STARS: DSC_0022.nef: analysing all stars candidates with star size areas equal to or larger than 33 pixels...
09:47:41 - 3) ANALYSE STARS: DSC_0022.nef: probed local background, noise and FWHM for all stars successfully
09:47:41 - 3) ANALYSE STARS: DSC_0022.nef: performed & collecting 2D general gaussian star profile regression and IW centroiding on all stars successfully
09:47:41 - 3) ANALYSE STARS: DSC_0022.nef: removed 70 star candidates stars from the initial star candidates
09:47:41 - 3) ANALYSE STARS: DSC_0022.nef: 934 star candidates left
09:47:41 - 3) ANALYSE STARS: DSC_0022.nef: removed stars that are too close to each other successffully: 434 stars removed
09:47:41 - 3) ANALYSE STARS: DSC_0022.nef: got all star details succesfully
09:47:41 - 3) ANALYSE STARS: DSC_0022.nef: probed local background, noise and FWHM for all stars successfully
09:47:41 - 3) ANALYSE STARS: DSC_0022.nef: correcting star map and star list for possible bad star detections...
09:47:41 - 3) ANALYSE STARS: DSC_0022.nef: number of removed duplicate stars 0
09:47:41 - 3) ANALYSE STARS: DSC_0022.nef: final number of identified and fully analysed stars 500
09:47:41 - 3) ANALYSE STARS: DSC_0022.nef: receiving final details...
09:47:41 - 3) ANALYSE STARS: DSC_0022.nef: closed
09:47:41 - 3) ANALYSE STARS: DSC_0022.nef : received star analysis results of frame
09:47:41 - 3) ANALYSE STARS: # stars 500
09:47:41 - 3) ANALYSE STARS: quality score 479.37
09:47:41 - 3) ANALYSE STARS: Full Width at Half Maximum of the average analyzed star
09:47:41 - 3) ANALYSE STARS: FWHM minimum: 5.13
09:47:41 - 3) ANALYSE STARS: FWHM maximum: 5.47
09:47:41 - 3) ANALYSE STARS: received 1 of 4 frames to analyse
09:47:41 - 3) ANALYSE STARS:
09:47:41 - GENERAL IMAGE LOADER: loading frame C:\Users\Shlomi\Pictures\digiCamControl\Checking guiding\Bubble Nebula\APP\MF-lights1-lights2-ISO_gain_800.0-exp_0.02s-27subs-NIKON_D3300-6016x4016-NR-noBl-avg.fits
09:47:43 - 2) CALIBRATE: performing calibration in 32bits normalized floats...
09:47:43 - 2) CALIBRATE: converting data to 32bits normalized floats...
09:47:43 - GENERAL IMAGE LOADER: frame C:\Users\Shlomi\Pictures\digiCamControl\Checking guiding\Bubble Nebula\APP\MF-lights3-ISO_gain_800.0-exp_0.02s-22subs-NIKON_D3300-6016x4016-NR-noBl-avg.fits was loaded successfully
09:47:45 - 2) CALIBRATE: Adaptive Data Pedestal raised to: 3.212E-04
09:47:46 - 2) CALIBRATE: Adaptive Data Pedestal raised to: 6.423E-04
09:47:46 - 2) CALIBRATE: Adaptive Data Pedestal raised to: 9.635E-04
09:47:49 - GENERAL IMAGE LOADER: frame C:\Users\Shlomi\Pictures\digiCamControl\Checking guiding\Bubble Nebula\Lights3 - battery change - m.flip\DSC_0149.nef was loaded successfully
09:47:49 - 3) ANALYSE STARS: DSC_0149.nef : starting analysis of stars in frame
09:47:49 - 3) ANALYSE STARS: DSC_0149.nef: starting...
09:47:49 - 3) ANALYSE STARS: DSC_0149.nef: getting luminosity channel...
09:47:49 - 3) ANALYSE STARS: DSC_0149.nef: creating star map...
09:47:59 - 3) ANALYSE STARS: DSC_0149.nef: star map created successfully
09:47:59 - 3) ANALYSE STARS: DSC_0149.nef: identifying star candidates...
09:47:59 - 3) ANALYSE STARS: DSC_0149.nef: identifying star candidates: initial FWHM estimate: 7.41 pixels
09:47:59 - 3) ANALYSE STARS: DSC_0149.nef: identifying star candidates: target noise level is at kappa 117.0
09:47:59 - 3) ANALYSE STARS: DSC_0149.nef: identifying star candidates: target luminosity of stars is 6.110E-01
09:48:00 - 3) ANALYSE STARS: DSC_0149.nef: identifying star candidates: initially identified 1002 of possible star candidates
09:48:00 - 3) ANALYSE STARS: DSC_0149.nef: probing star positions and luminosities...
09:48:00 - 3) ANALYSE STARS: DSC_0149.nef: identified star candidates successfully
09:48:00 - 3) ANALYSE STARS: DSC_0149.nef: probed star positions and luminosities successfully
09:48:00 - 3) ANALYSE STARS: DSC_0149.nef: probing local background, noise and FWHM for all stars...
09:48:00 - 3) ANALYSE STARS: DSC_0149.nef: estimated Full Width at Half Maximum (FWHM) of star profiles : 7.02 +- 1.05 pixels...
09:48:00 - 3) ANALYSE STARS: DSC_0149.nef: analysing all stars candidates with star size areas equal to or larger than 37 pixels...
09:48:00 - 3) ANALYSE STARS: DSC_0149.nef: probed local background, noise and FWHM for all stars successfully
09:48:01 - 3) ANALYSE STARS: DSC_0149.nef: performed & collecting 2D general gaussian star profile regression and IW centroiding on all stars successfully
09:48:01 - 3) ANALYSE STARS: DSC_0149.nef: removed 160 star candidates stars from the initial star candidates
09:48:01 - 3) ANALYSE STARS: DSC_0149.nef: 842 star candidates left
09:48:01 - 3) ANALYSE STARS: DSC_0149.nef: got all star details succesfully
09:48:01 - 3) ANALYSE STARS: DSC_0149.nef: removed stars that are too close to each other successffully: 338 stars removed
09:48:01 - 3) ANALYSE STARS: DSC_0149.nef: probed local background, noise and FWHM for all stars successfully
09:48:01 - 3) ANALYSE STARS: DSC_0149.nef: correcting star map and star list for possible bad star detections...
09:48:01 - 3) ANALYSE STARS: DSC_0149.nef: number of removed duplicate stars 1
09:48:01 - 3) ANALYSE STARS: DSC_0149.nef: final number of identified and fully analysed stars 503
09:48:01 - 3) ANALYSE STARS: DSC_0149.nef: receiving final details...
09:48:01 - 3) ANALYSE STARS: DSC_0149.nef: closed
09:48:01 - 3) ANALYSE STARS: DSC_0149.nef : received star analysis results of frame
09:48:01 - 3) ANALYSE STARS: # stars 503
09:48:01 - 3) ANALYSE STARS: quality score 399.94
09:48:01 - 3) ANALYSE STARS: Full Width at Half Maximum of the average analyzed star
09:48:01 - 3) ANALYSE STARS: FWHM minimum: 5.75
09:48:01 - 3) ANALYSE STARS: FWHM maximum: 5.81
09:48:01 - 3) ANALYSE STARS: received 2 of 4 frames to analyse
09:48:01 - 3) ANALYSE STARS:
09:48:01 - GENERAL IMAGE LOADER: loading frame C:\Users\Shlomi\Pictures\digiCamControl\Checking guiding\Bubble Nebula\APP\MF-lights3-ISO_gain_800.0-exp_0.02s-22subs-NIKON_D3300-6016x4016-NR-noBl-avg.fits
09:48:02 - GENERAL IMAGE LOADER: frame C:\Users\Shlomi\Pictures\digiCamControl\Checking guiding\Bubble Nebula\APP\MF-lights1-lights2-ISO_gain_800.0-exp_0.02s-27subs-NIKON_D3300-6016x4016-NR-noBl-avg.fits was loaded successfully
09:48:02 - 3) ANALYSE STARS: MF-lights1-lights2-ISO_gain_800.0-exp_0.02s-27subs-NIKON_D3300-6016x4016-NR-noBl-avg.fits : starting analysis of stars in frame
09:48:02 - 3) ANALYSE STARS: MF-lights1-lights2-ISO_gain_800.0-exp_0.02s-27subs-NIKON_D3300-6016x4016-NR-noBl-avg.fits: starting...
09:48:02 - 3) ANALYSE STARS: MF-lights1-lights2-ISO_gain_800.0-exp_0.02s-27subs-NIKON_D3300-6016x4016-NR-noBl-avg.fits: getting luminosity channel...
09:48:02 - 3) ANALYSE STARS: MF-lights1-lights2-ISO_gain_800.0-exp_0.02s-27subs-NIKON_D3300-6016x4016-NR-noBl-avg.fits: creating star map...
09:48:11 - 3) ANALYSE STARS: MF-lights1-lights2-ISO_gain_800.0-exp_0.02s-27subs-NIKON_D3300-6016x4016-NR-noBl-avg.fits: star map created successfully
09:48:11 - 3) ANALYSE STARS: MF-lights1-lights2-ISO_gain_800.0-exp_0.02s-27subs-NIKON_D3300-6016x4016-NR-noBl-avg.fits: identifying star candidates...
09:48:11 - 3) ANALYSE STARS: MF-lights1-lights2-ISO_gain_800.0-exp_0.02s-27subs-NIKON_D3300-6016x4016-NR-noBl-avg.fits: no final details to receive
09:48:11 - 3) ANALYSE STARS: MF-lights1-lights2-ISO_gain_800.0-exp_0.02s-27subs-NIKON_D3300-6016x4016-NR-noBl-avg.fits: closed
09:48:12 - 3) ANALYSE STARS: MF-lights1-lights2-ISO_gain_800.0-exp_0.02s-27subs-NIKON_D3300-6016x4016-NR-noBl-avg.fits : received star analysis results of frame
09:48:12 - 3) ANALYSE STARS: analysis failed
09:48:12 - 3) ANALYSE STARS: received 3 of 4 frames to analyse
09:48:12 - 3) ANALYSE STARS:
09:48:12 - 3) ANALYSE STARS: MF-lights1-lights2-ISO_gain_800.0-exp_0.02s-27subs-NIKON_D3300-6016x4016-NR-noBl-avg.fits: identified star candidates successfully
09:48:14 - GENERAL IMAGE LOADER: frame C:\Users\Shlomi\Pictures\digiCamControl\Checking guiding\Bubble Nebula\APP\MF-lights3-ISO_gain_800.0-exp_0.02s-22subs-NIKON_D3300-6016x4016-NR-noBl-avg.fits was loaded successfully
09:48:14 - 3) ANALYSE STARS: MF-lights3-ISO_gain_800.0-exp_0.02s-22subs-NIKON_D3300-6016x4016-NR-noBl-avg.fits : starting analysis of stars in frame
09:48:14 - 3) ANALYSE STARS: MF-lights3-ISO_gain_800.0-exp_0.02s-22subs-NIKON_D3300-6016x4016-NR-noBl-avg.fits: starting...
09:48:14 - 3) ANALYSE STARS: MF-lights3-ISO_gain_800.0-exp_0.02s-22subs-NIKON_D3300-6016x4016-NR-noBl-avg.fits: getting luminosity channel...
09:48:14 - 3) ANALYSE STARS: MF-lights3-ISO_gain_800.0-exp_0.02s-22subs-NIKON_D3300-6016x4016-NR-noBl-avg.fits: creating star map...
09:48:23 - 3) ANALYSE STARS: MF-lights3-ISO_gain_800.0-exp_0.02s-22subs-NIKON_D3300-6016x4016-NR-noBl-avg.fits: identifying star candidates...
09:48:23 - 3) ANALYSE STARS: MF-lights3-ISO_gain_800.0-exp_0.02s-22subs-NIKON_D3300-6016x4016-NR-noBl-avg.fits: star map created successfully
09:48:23 - 3) ANALYSE STARS: MF-lights3-ISO_gain_800.0-exp_0.02s-22subs-NIKON_D3300-6016x4016-NR-noBl-avg.fits: no final details to receive
09:48:23 - 3) ANALYSE STARS: MF-lights3-ISO_gain_800.0-exp_0.02s-22subs-NIKON_D3300-6016x4016-NR-noBl-avg.fits: closed
09:48:23 - 3) ANALYSE STARS: MF-lights3-ISO_gain_800.0-exp_0.02s-22subs-NIKON_D3300-6016x4016-NR-noBl-avg.fits : received star analysis results of frame
09:48:23 - 3) ANALYSE STARS: analysis failed
09:48:23 - 3) ANALYSE STARS: received 4 of 4 frames to analyse
09:48:23 - 3) ANALYSE STARS:
09:48:23 - 3) ANALYSE STARS: finished
09:48:23 - 3) ANALYSE STARS: MF-lights3-ISO_gain_800.0-exp_0.02s-22subs-NIKON_D3300-6016x4016-NR-noBl-avg.fits: identified star candidates successfully
09:54:28 - 3) ANALYSE STARS: received all results
I triple checked those master-flats are NOT frames, did that twice, the behavior is consistent. Trying to go back to beta4 to see if this works there, but any clues why would this happen?
Thanks!
Shlomi
One more point, could we please not fail the integration? If star analysis fails on any frame, I much rather that frame be excluded implicitly (with some warning at the end of the process), so at least I get some results.
Thanks 🙂
An update:
Just before reinstalling the old version, I tried to sanitize the test.. I restarted APP, added only two frames from each session and all calibration frames. That seemed to have worked fine. I then added the rest of the images, and it worked fine again.. Now I am really confused, the previous problem happened 3 times to me (on the same APP instance though), but now it seems to be fine. Any thoughts?
Maybe you accidentally loaded flats into the lights module? If the frames don't contain any information about them being flats, lights or darks, APP will assume they are how you loaded them in.
I double and triple checked.. I had 80 lights listed on the side screen in the 1 load part, but when star analysis starts it claimed to have started 82 tasks - in the first run. In the second run I manually deselected ALL frames and then manually selected only two, it still started 4 tasks tried to process the flats as lights..
Since I already restarted the app I probably lost the log, unless it is being saved somewhere automatically. I'd love to help make this product better!
If not, I'll let you know when and if this happens again.
Thanks for responding!
Hi @shlomi & @vincent-mod ,
I suspect the follwing happened, it should explain APP's behaviour especially after a restart... : in 1) LOAD there is a selectbox for automatic recognition of Masters, if that is turned off, a Masterflat that is loaded with the lights button will be loaded as a light and thus APP will process it as such. The selectbox is enabled by default so you can load your masters with any button and they will be recognized as such through their metadata.
Now in your case, letting APP continue processing after this problem does not make sense I think, because your masterflat was not properly loaded, it was loaded as a light. And so your lights will not be calibrated by the Masterflat and thus your result will never be like it can be, right?
I do agree, that I can add an option to force APP just to continue in case of a problem with a frame, instead of it breaking down, right?
Kind regards,
Mabula
Hi Mabula,
Thanks for looking into that! I actually did manage to find the logs of the original attempt only.. Its actually better, since I had NO master flats at that point, they were generated during the calibration stage! So I could have never added them as frames.. Please check out the log, if this was a user error on my behalf, it would be great to know and perhaps be able to recognized such mistakes automatically for other users. If not, at least it will help point us to the source of the error.
In the logs, the first failed is at timestamp 08:39:07. Checkout the file name - it was automatically generated by previous steps.
Hope this helps!
It does look like it is analyzing the masterflats for stars indeed, we will have a closer look at the log and will maybe ask you for some of these subs. Thanks!
Hi,
So last night I attempted again with a 2 panel mosaic. The same problem appeared, this time I took some more snapshots and logs for you to figure this out. Two problems:
1. Flats and master flats are failing star analysis even though the flats were definitely not selected as lights, and the master flats were generated during the processing
2. Some times, trying to load master bias and BPM does not ask for which session to assign them to. I have to remove them and reload multiple times to get the selection dialog.
I had this two problems happening three times after killing and restarting the app. It worked at the end by changing the workflow - I loaded a single session and star analysis, which showed the right amount of frames, then canceled the operation and loaded the second session. This seemed to work fine..
Please let me know if you could use any more information, I would love to be able to help you solve this. It wasted long hours for me..
Thanks,
Shlomi
Let me clarify the log files uploaded above:
flat-star-failed2.txt - The first attempt, shows multiple failures during the star analysis phase..
flat-star-failed3-127-no-125.txt - The second attempt: shows that only a total of 125 frames were added, but 127 were being analyzed:
07:44:06 - FRAME DETAILS UPDATER: adding 49 new frames...
...
07:44:20 - FRAME DETAILS UPDATER: adding 76 new frames...
...
07:47:18 - 3) ANALYSE STARS: 127 image loaders created...
Here you could also see that I tried to add the BPM and bias multiple times, however I did not see in the logs anything about the session dialog being opened, or even that I removed the file from the list.
Thanks,
Shlomi
Thank you, I'll ask Mabula to specifically look at this issue.
