20 January 2021: Soon to be released APP 1.083-beta2 : improved comet registration, updated tools, new Star Reducer Tool and more...
16 November 2020 - Wouter van Reeven has officially joined the Astro Pixel Processor Team as a moderator on our forum, welcome Wouter 🙂 !
Null Pointer exception on multi-sessions...I might have found a bug/solution
I've been getting this strange Null Pointer exception bug anytime I try to do a multi-session integration. It didn't matter what settings I used, it would do it. Being a programmer myself, I decided to try and troubleshoot my problem. I tried it on both Mac and Windows, same issue. I tried using different images, same issue. I tried different settings, no change. At this point I was really scratching my head. I decided to try each session individually, every single one of them worked. Now, I was even more confused.
I started going through every little detail, trying to think about what I would likely mess up if I were to make the program. When I started thinking like that, a lightbulb went off in my head...it's likely being caused by mixing master flats with individual flats. On the 4 sessions I was trying to integrate, 3 of them already had master flats, but 1 of them didn't. I converted the individual flats to a master and ran everything again. This time it worked perfectly. While I don't know what the exact bug is, it seems to have something to do with mixing sessions that use master flats with sessions that uses individual flats.
TLDR; If you're getting a null pointer exception and you're trying to do a multi-session integration, make sure each session uses a master flat.
Really liking the way you think and analyze. 🙂 Could you give us a step by step workflow to see if we can reproduce that nullptr? That would be really helpful and I'll give Mabula a heads-up on it to see if that can be improved.
@vincent-mod haha thanks for the kind words. I know how hard it can be to find random bugs like this, especially ones that seem to be very specific use case. I can surely post my steps to recreate this error. I'm not sure how much of this is required to get the error, but it's what I did to get it.
1. Open APP and select a directory
2. Go to step 0, select pattern: RGGB and check "force Bayer"
3. Go back to step 1, uncheck multi-channel, enter anything for the name, load in your light frames. For the light frames, they were all fit files. After I loaded them in, I selected session 1. I then loaded in more light files into session 2 and so on. I tried my light files in separate folders as well as combined, didn't seem to matter.
4. Load in flat files (not MasterFlat) for one of the sessions. I loaded in 2 fit files as a test.
5. Load in MasterFlat for the other sessions. My master flats were tiff files (not sure if that matters)
6. Load in MasterDarks and MasterDarkFlats. I used 1 of each for each session, except the session from above. So it should look something like this
session 1: lights, 2 or more flats, 1 master dark, 1 master dark_flat
session 2: lights, 1 master flat, 1 master dark, 1 master dark_flat
session 3: lights, 1 master flat, 1 master dark, 1 master dark_flat
session 4: lights, 1 master flat, 1 master dark, 1 master dark_flat
7. Go to step 6 and integrate
It should throw the error in a few minutes. The way I was able to fix it was by converting the 2 flats into a master flat tiff file and using that instead. My hunch is that it has nothing to do with the filetype itself, but the mixture of combining multiple sessions and some of the sessions using Flats while others use MasterFlats. If I have some time later I'll play around with it some more and post my findings.
If you need me to explain anything or test something, I'd be more than willing to.
Lovely thanks very much. I'll try to reproduce this, if that doesn't work I might ask you for a few of your data frames. 🙂
Thanks for the excellent help!
@vincent-mod So I've done more testing and It might be slight different than I stated above. It seems that it throws the same error if I have more than 1 session and I choose a fit file for my flat. I tried using a stacked fit file for my master flat and it threw the error. I converted the image to tiff and it worked just fine. I've noticed this same result on 2 projects so far.
@vincent-mod I honestly have no idea what the problem is anymore. I was able to integrate 3 sessions without a problem. I added another session, which had tiffs for their master files, and it threw the same error. Does the order in which the files are loaded in matter? Do I have to load in all the lights at the same time, followed by all the flats and so on? That seems to be the only thing that would be different.
Edit: Nope, that doesn't seem to have fixed the problem. Man this is throwing me for a loop. It almost seems random at this point. It's like playing integration roulette haha. I have noticed that it only ever happens on sessions. I've never seen it happen on single sessions.
Edit 2: Now it seems completely random. I had it error out again. I wanted to watch it and see exactly where it was erroring out, so I made no changes, just ran the integration again. This time it worked just fine.
@vincent-mod I honestly have no idea what the problem is anymore. I was able to integrate 3 sessions without a problem. I added another session, which had tiffs for their master files, and it threw the same error. Does the order in which the files are loaded in matter?
That's unforunate. I have the "feeling" that it may be the tiff file itself in some cases. Seems APP has a bit strict requirement for loading those (in the sense that Mabula is following the guidelines for that format, but not all software is programmed nicely to do the same). Are you able to load raw files instead of tiff to test that? Mabula is going to look into the tiff module at least.
@vincent-mod So I ran the same project that has been giving me the same error consistently. I accidentally loaded the masters in under the regular section (instead of loading it into MasterFlat, I loaded them into Flat and so on). I ran the same project, and it ran just fine.
Could my issue have been caused by APP not seeing my masters as being masters or something like that? Is there any real difference between loading a master in the regular section vs the master section?
It might just be a fluke, and would have integrated either way, it has fooled be before haha.
Might be it treats it not like a master if you select them under the regular darks option. Not 100% sure though as I never did that on purpose. 😉