[Sticky] Data calibration principles/rules - must read !

Yes, though Bias frames or dark flats are needed for flats. If you use dark flats, bias is indeed not required.

That is good information to know and begs a further question. DarkFlats must be taken at each imaging session, where as Bias can be done anytime. A database of Bias Frames can be taken at the various Gains/Offsets/Temps used for lights and stored for use.

Is there a compelling preference between DarkFlats and Bias. I am using an ASI 1600 MC Cool

Timothy,

Flats need to be taken evenry session. DarkFlats, like Bias and Darks, only need to be taken once every year or so. For my ASI1600MM and now my ASI6200MM I have created a library of Darks and DarkFlats that I use for processing with APP. I don’t use Bias anymore.

HTH, Wouter

@tmyers,@wvreeven,

For my camera (QHY163m) which shares the same sensor as ZWOASI1600MM, I use conventional a master bias to calibrate flats. I've never encountered with overcorrecting flat again since I set exposure time properly, and checked the correct version of camera's driver. If you have updated camera's driver, you may need to take new biases, darks, and flats.

Cheers,

Kijja

Concerning the flats i found these videos from Guik:

and especially this:

where he explains that focus doesnˋt matter!

what do you think about this?

would be very nice if he is right because for me that means NOT to take flats after every session; except i have changed the filter!?

CS Didi

Focus does matter. 🙂 Why? Because dust on your sensor will change shape when you’re using a different focus and will not be calibrated very well if at all using flats. It all depends on how good you’re looking at your data and I also think, since flats are indeed the most challenging to get right, some people try to reason they’re not that vital to not having to go through the trouble. 😉

@vincent-mod

i saw those videos from Cuiv too and I think that he was explaining that small variations in focus will not matter...

i.e. throughout an imaging session focus changes whether your setup does it automatically depending on temperature changes or if you have it to check focus every hour. 

This means that all your lights from that session will have varying focuser positions and you cannot stop after each incremental focus change and take Flats, I personally take mine at the end of a session and this may be a different focus position to when the session started. 

These though would be very small changes in focus. 

Regards..,

Very small differences are fine indeed, but the way he is stating this in the video is not that clear. Many will take this as “focus doesn’t matter”, it does. Also regarding the frequency of taking flats. If you have a fixed setup, it might be fine to skip a few sessions. When you’re always setting your gear up, I would certainly take them every session. You can ruin an entire night of imaging (been there) when suddenly a little dust particle made its way on the sensor, while setting up the gear and using old flats.

@vincent-mod / @1cm69

Thanks to both of you; very clear statement!

So i will continue my current workflow and take the flats after the lights in each session

CS Didi

@vincent-mod / @1cm69

Thanks to both of you; very clear statements!

So i will continue my current workflow and take the flats after the lights in each session

CS Didi

Hi,

To decide if focus change is likely to make significant (ie visible) impact with your particular imaging set-up I am thinking it surely cannot be a too difficult when next you are making flats to take a few exposures racked in and out from your normal focus range,  and then to use these to make master flats from each set and use difference maths to compare the master flats.  

I think I shall try this but I'm very doubtful I will see much more then very faint changes in illumination levels across the grey scale images - certainly no more than might be visible when clouds or other light pollution changes drift across the field of view when taking lights.   There again, maybe some  set-ups are much more sensitive to focus changes.

Would be interested to hear if anyone else tries this.

Mike

Illumination won't be affected that much I think, it's the dust that will. But a nice comparison is nice to see indeed, just to determine how big the variation can be. I would indeed take flats with a range of focus positions away from "perfect" focus and then calibrate the same subframe. Zooming in on the dust spot will then show if it calibrated nicely away or not.

Hi Vincent,

To do this experiment really well one might need to deliberately arrange eg some pieces of dry paper or cotton thread on the primary lens, filter wheel etc.   On my flats, apart from gradual vignetting change I see only very faint smudges from smears and dust motes. 

To see any really noticeable effect from focus changes I think you would need something in the light path that would cast a fairly hard-edged shadow on the sensor and this has  to change its relative shape/position on the sensor as the focus changes

Mike

Hello guys,

My camera Stellina is producing FITS files with both BAYERPAT= 'RGGB ' and BAYERPAT= 'BGGR '.

Is it ok to mix both BAYERPAT  for lights and darks in the APP flow process (Fits header are properly filled) using 'pattern = supported' settings ?

Thx !

Mmm, no idea, do you happen to know why it produces two separate bayer patterns? That doesn’t make a lot of sense to me.

Posted by: @vincent-mod

Mmm, no idea, do you happen to know why it produces two separate bayer patterns? That doesn’t make a lot of sense to me.

Not received any official answer yet from Stellina team but it seems to be related to field rotator when you pass zenith.

A Bayer pattern remains the same on a sensor so that would be very strange. I don’t know about the Stellina, but a response from them would be nice to get down to this.

Hi all, I just bought a Stellina, have a lot of experience, but not so much time, so an ideal companion. Indeed, Stellina (Vaonis) does flip the Bayer patterns from BGGR <=> RGGB depending on his position towards the Zenit! There is no warning for that, Stellina just does it. A nice, useful document, written for DSS though, but very useful, also for APP! 

So, if you want to process your photos in a correct way, one must make darks for each type (e.g. one set for BGGR and one set for RGGB). Once I got a "greenish" result ... because I calibrated my RGGB photos with an BGGR master dark :-(.

I use now AvisFV 2.0 to very quickly run through my fits => split RGGB series and BGGR series. remove cloud and plane photos on sight (takes only a minute or 2 for 2000 photos to give you an idea)

I just tried now to process them, which worked, BUT APP merged the stacks correctly, i.e. treat the right colour channels once I will integrating them. The only difficulty is that e.g. quality is analysed per session, so it is impossible to compare the quality between sessions. It could well be that it might be useful to skip a certain session.

Therefor I am now saving the calibration of the sessions and import ALL as light with no darks and then try to integrate. Once I analysed the stars, I was able to see the quality differences between the sessions as a whole.

Is there another way to do this right away?  because this latest procedure is very time consuming....
it would be handy to chose between analyse per session and analyse as a whole.

J

Dear Mabula, Vincent, @mabula-admin @vincent-mod

with respect to my post I executed all possibilities and tried 5 times to develop the best possible picture to my knowledge.

Is there a way to see the quality of your images with respect to all pictures or is it only within the session? If I process the calibrated files as lights without darks, (because they were processed which was necessary since I have BGGR and RGGB fits) ...

What is the advantage of having multi sessions despite of adding multiple darks/bias/flats to a certain session? The only possibility is If we use different installation or even different scope setting ...

To give you and Mabula an idea of the look and feel of Vaonis Stellina fits, I just uploaded 1 Night of NGC7635

https://www.dropbox.com/sh/h2gy65u2bp1c4xw/AAA-J5y1ALQ5JTq71uu_CEpZa?dl=0

You have my conscent to use my date as you see fit.

this was my result:

(Note SQM 17.65 - Bortle 8!), lunar 70%, temp average 5°, humidity average 92%

Vaonis Stellina - 400mm 80mm ED Apo refractor (F5) + CLS filter integrated

with fixed Sony CMOS 6MP + mechanical derotator

Posted by: @vincent-mod

A Bayer pattern remains the same on a sensor so that would be very strange. I don’t know about the Stellina, but a response from them would be nice to get down to this.

If you do a 180° rotation of your sensor, the pattern ABC switches to CBA.

This is what Stellina does with the field derorator.

My question is, do you think I can mix light and dark with both pattern mentionned in fits header and APP makes a sort, or should I use 2 sessions one per pattern ?

Thx.

@flamingo

I wrote a Workflow on this for Stellina as a variant on capitaine Nautilus for DSS

I must work on the content and layout, but basis is good:

  No, you may not mix this up and yes what you write is logical and correct. Fact is, why does not Stellina alter the firmware so the ABC => CBA is "stored" as ABC, this would save a huge amount of time.

You should split it up in 2 sessions and apply the representative darks to the corresponding lights

you may not mix them up. If you do, you will get a very coloured, undesired result.

@janwalschap

Thx !

Posted by: @janwalschap

What is the advantage of having multi sessions despite of adding multiple darks/bias/flats to a certain session? The only possibility is If we use different installation or even different scope setting ...

That is the advantage, you can then indeed assign specific calibration files to a certain session if they are different. This enables people to combine the processing of that data.

To assess the quality of your data, what you can do is to load in your data and calibration frames, go to tab 5, normalize. And normalize all data. Then in the list below, you can right-click and select to make a graph. Here you can make graphs out of things like star shape, noise, signal to noise, quality etc. You can also sort the list based on these parameters to get a quick look at the worst frames.

@vincent-mod - Hi Vincent, I know 😉 I wrote that in my manual. But I will try to express myself better, IF we use the same telescope, same setup but different days, presuming fits are of same format, we can easily load in all in 1 session, there is no advantage too split up, on the contrary, correct?

But My main issue stays: Suppose I have RGGB and BGGR and I load them in 2 sessions and I make the graph or the sort, you will see that quality comparison seems to be done "per session" and not as a "total". i.e. you don't have a clue if the session of one day is better or worse than the others. I presume this is a feature that is not foreseen yet.

Therefore, I load BGGR and RGGB sessions, calibrate them. Start again and load all calibrated images of the 2 sessions as lights, since you do not need your (darks, flats, bias ....) anymore, they have been subtracted already (If I'm right). If I do then the normalisation or star analysis (here I do most of the time my first preselection), you can compare alle images to each other. Is my idea/workflow correct?

=> I could see a serious quality drop on all RGGB images, therefor I'm not 100% sure of my statements.

One worry I have is that even the calibrated images stay in their original format i.e. BGGR stays BGGR and RGGB stays RGGB, so even if they are calibrated, may I mix them up then? Accordingly, to my test, it works, but what does happen exactly on the "image format level", i.e. how does APP treat them? 

If so, Will App foresee in the future a possibility to compare quality and all other parameters to each other 1) per session (as it is now) or 2) in total (as a whole)?

Posted by: @janwalschap

@vincent-mod - Hi Vincent, I know 😉 I wrote that in my manual. But I will try to express myself better, IF we use the same telescope, same setup but different days, presuming fits are of same format, we can easily load in all in 1 session, there is no advantage too split up, on the contrary, correct?

Yes, that would be fine for a singe session indeed. Unless you have flats that are specific to a certain night, then I would split them up. it's about correcting the data to be pretty much the same and then combining it all.

But My main issue stays: Suppose I have RGGB and BGGR and I load them in 2 sessions and I make the graph or the sort, you will see that quality comparison seems to be done "per session" and not as a "total". i.e. you don't have a clue if the session of one day is better or worse than the others. I presume this is a feature that is not foreseen yet.

Yes, the graphs need a upgrade for things like this. Right now you just get all the data that is loaded in indeed.

Therefore, I load BGGR and RGGB sessions, calibrate them. Start again and load all calibrated images of the 2 sessions as lights, since you do not need your (darks, flats, bias ....) anymore, they have been subtracted already (If I'm right). If I do then the normalisation or star analysis (here I do most of the time my first preselection), you can compare alle images to each other. Is my idea/workflow correct?

Yes that will work, they are not calibrated again (if you don't load the calibration masters of course). To get that flow better, in the next release you can skip certain steps you already did so APP should continue where you left off.

One worry I have is that even the calibrated images stay in their original format i.e. BGGR stays BGGR and RGGB stays RGGB, so even if they are calibrated, may I mix them up then? Accordingly, to my test, it works, but what does happen exactly on the "image format level", i.e. how does APP treat them? 

The mosaic pattern is just for APP to get a proper image, so if the patterns are correct APP will treat them like that and the result will be a color image that is the same as the one with the other pattern.

If so, Will App foresee in the future a possibility to compare quality and all other parameters to each other 1) per session (as it is now) or 2) in total (as a whole)?

Yes, in a future update this will be possible and is already on Mabula's to-do list. 🙂

Hope that answers some of your questions?

Vincent

@vincent-mod,

Thx Vincent, indeed, it was an elaborated question 😉 but important for Stellina Users. It's good for you that you know this kind of images are created. (as I mentioned above)

you may use my Stellina data of the bubble experiment (mentioned above):

https://www.dropbox.com/sh/h2gy65u2bp1c4xw/AAA-J5y1ALQ5JTq71uu_CEpZa?dl=0

I got an extensive 1-day training from Mabula himself last year @ our observatory in Belgium, it was brilliant.

I feedback this info towards Stellina users as well, since many of them want to move further than you can do with DSS.

Keep up the good work!

I'll have a look later. Would also be nice for Stellina to not flip the mosaic pattern, that's basically the issue here and doesn't make sense. 🙂 But anyway, like you figured out, it can indeed be done with some careful workflows. Thanks for making that guide!

@vincent-mod

I just got feedback that there is no software layer between the opto-mechanical part of Stellina  and the storage, this means Stellina cannot adapt this issue and will always output fits in BGGR and once it flips over the zenith to  RGGB

@mabula-admin

Is the calibration information now in the Metadata of the masters? If so, what APP version was this introduced? I'm not seeing it in the masters I'm creating, and I don't see any indication in the console that MasterBias or MasterDarkFlats are being used when creating my MasterFlats.

Same with making the MasterDarkFlats. How would I know if they contain bias offset or if a MasterBias was subtracted in the making? Just very unclear what was done for each of my masters and how to confirm I'm combining everything correctly.

Thanks for any help you can provide regarding this topic.

-Chris

Yes, this may need attention and is already mentioned to Mabula. A lot of the information would not make sense in the end integration as many parameters are there for the separate subs. But it may need more information on the process.