[Sticky] Data calibration principles/rules - must read !

@vincent-mod

Euh? I don't understand the question: the info in my dropbox contains:

PURE RAW (untouched) data from Vaonis Stellina  (device software Version V.21 dd 30/10/2020)

this is important to future versions to compare to each other

(Reason: So Mabula and his team kan see for them selves what output this device really produces, this info is saved directly to an USB stick attached in the copartment of the Stellina without any interference of whatever)

=> This means the metadata is exactly as it comes out of the Stellina Telecope
21/11/2020 17:17 <DIR> _Session 1 BGGR
21/11/2020 17:17 <DIR> Darks Session 1_BGGR

PURE RAW (untouched) data from Vaonis Stellina  device (after passing the zenith)

21/11/2020 17:17 <DIR> Darks Session 2 RGGB

21/11/2020 17:19 <DIR> _Session 2 RGGB

11/11/2020 10:25 6,393,600 BPM-stellina-bb468e-3072x2080.fits

Results I got after integretion by applying the method mentioned in the directory header:

22/11/2020 09:02 <DIR> integrated session 1 and 2 integration as one session many good images
22/11/2020 09:02 <DIR> integrated session 1 and 2 integration as one session only best images
21/11/2020 17:17 <DIR> integrated Session 1 integration only results

in the root, I placed:

Session 1 => BGGR + BGGR darks

Session 2 => RGGB + RGGB darks

Saved then the calibrated files

Do you have sufficient information with this?

Was this your question?

My main question was:

for the moment we have to treat this as 2 separate sessions which is logical (RGGB and BGGR session)

BUT for the moment each session is evaluated to each other within the session

eg within session one, my images may range from quality 420 => 117  (eg)

the second session may have a quality ranging from 397 => 50 (eg)

They are shown individually with respect to a reference, bu every session has his reference.

But what is the quality comparte to the images of session 1 and 2

to do that, I callibrate all my images, save those calibrated images (in the root of this dropbox link)

then reload all immages as 1 session (becaus all are callibrated)

If I go then to the star analysis or normalisation, I can compare all  to 1 reference

hence it's much more relevant to eliminate the worst or non acceptible images

I Hope I did it right or used the right procedure? That's the only ting I'm not sure of

To let you gain some time, I uploaded those calibrated files in the root 

People at APP told me a button will be there in the future which would allow us to compare the quality per session or as one whole session, which would aid a lot in skipping the worst frames

@vincent-mod

Correction, due to help from colleague astronomers, I made an error in my thinking process!

I can indeed do all what I described; the BMP will be processed as one despite the session split up.

Seen the sensor data is dumped inverted in the 2nd session, data of a sensor and inverted sensor are combined in the BPM! The only solution is to edit the BGGR and convert it to RGGB or vice versa so one has on set of Bayer identical frames. Someone has written an image data flip (ie including RGGB <=> BGGR) program in python (& astropy). Then you will notice that a part of the image is upside down, but APP is capable to handle this "meridian flip" looking image.  The flip is here not at the meridian but near to the zenith, depending on the frame orientation at start.

I reprocessed the frames with APP, and the result is very good!

As I don't have a Stellina and I almost never see them, it's still quite puzzling to me how to tackle the data. But at least it's great to hear you found a solution! Is that script on github by any chance?

@vincent-mod, HI Vincent, not yet, but I'll attach some research to this post. including the astropy of our Collegue astronomer Paul De Backer connected to workgroups @ Urania, Hove, province Antwerp, Belgium. It's his contribution.

note:

I gave it the extention txt, please rename to .py to let the python script work. More info is in the pdf doc.

You don't need a Stellina ;-), I posted the data for you on  my dropbox, it's still available.

Oh yes I know, thanks for the data. 🙂 I mainly meant that, since there aren't others using it a lot here, I haven't much experience with it. And this is quite the unique issue to have.

Thanks for the script, I'll also have a look at it, who knows we might make a "Stellina" button. haha Not a big chance though as this should be fixed by Stellina I think.

@vincent-mod I understand now what you ment ;-).  And indeed proportionally, the number of people using APP and Stellina is small, but Since I wrote a small accessible tutorial, a lot of more experienced Vaonis Stellina users are testing APP right now! According to the data to get teh stacking even better, do you have recommendations in APP settings?

In addendum my last one, M1 

processing of  15 december

Messier 1, Crab nevel

(5.9-8.1°C and 91.2-94.4% Humidity)

634 usable light FITS (of 10 sec gain 200, always by default) van de 1104 (tgv 2/8 alto cumulus)

117 Dark & Dark flats + 111 Flats

Extremely good SQM at my location (18.65-19.14) Bortle7, due to heavy rain just before observations.

Moon disk illumination 2.1%

APP post processing all RGGB, Topaz Denoize, light cosmetic touch in lightroom 

@vincent-mod @Mabula-Admin, just an impression what my input did:

Make your flats with a low ISO or gain+offset to be able to increase the exposure length of the flats. They will be of better quality then.

The same can be said for the opposite, i.e. make the gain higher to lower the read noise. They will be of a better (but for different reasons) quality then. In fact, this is the advise given by SharpCap. I would say the right answer is both. Raise the gain and lower the light to get low noise and longer exposures. It's a balance, right?

Running an integration with the new 1.083b4 I got the calibration/clipping warning.  This is a surprise since all gain, offset, time, and temperature are matched properly and I see no signs of leaks.  Messing around a bit I noticed that when I exclude bias files I don't get the message.  Could this be because I didn't include the bias when I made the master darks?  I thought I did all in one go but it is possible I am misremembering.  I guess if that were the case and I included the bias it would wind up being subtracted twice, once explicitly and once because it remains in the dark?

Unrelated question: can I omit darkflat master(or the master bias for flats if I used bias rather than darkflats) from future calibrations once I have made a master flat - it seems like this should be OK since it is removed in the master flat and the master flat is used for scaling not subtraction?

You can use darkflats for the flats and darks, bpm for the rest indeed. I'm not sure what happened in your workflow for that warning to pop up, but calibrating something twice might make sense. Also, when you have a sensor that doesn't behave well when shooting with the shortest exposure times, you can also get issues. And, the warning also pops up if your exposure length per subs is just too short, which happens easily with narrowband.

@mabula-admin  Hello

According to the principles presented at the beginning of this thread, there are three ways to calibrate FLAT and LIGHT.
I think I can choose one of three.

I am confused between 2. and 3.
Both instruct me to calibrate using master dark, but 2. use one master dark, 3. use two master darks.

Why is there one master dark and two master darks?

Please tell me the case that I need two master darks.

☕ ☕ 

Hi @mo-mo,

The rules as described on the topic's first post are old. For the flats you need to read darks as flatdarks.

But 2 masterdarks (for lights) are needed of you have shot lights of different exposure times. Liker 300 seconds and 600 seconds. Then you would also create darks of 300 and 600 seconds which will result in 2 masterdarks.

Mabula

@mabula-admin 

Thank you for your answer.

Are there any other changes?
I think this principle is important, so I would like to know other changes.

Thank you

I am somewhat confused about callibration.

What I do is apply flats and flatdarks only.

Is this ok as I have had some warnings pop up when callibrating. I tried including darks but the results were awful. I am using an ASI533 cooled to minus 10C.

Thanks

Flatdarks for flats is perfectly fine yes, flatdarks are basically almost like bias frames. If you use a dark that causes issues in the data, then you may have a problem with those darks.

Hi everybody,

I have already worked through this post several times to understand calibration principles. Thanks already!

While trying to calibrate my flats with a masteries of the same iso, I don't seem to find any output in the console window, that the Masteries has actually been "applied". Are there special steps to be taken for APP to calibrate the flats with the masterbias or does loading all relevant files (Flats, Masterbias, BPM) suffice?

Additionally, when trying to calibrate my lights, according to the rules laid out in the first post, with the masterflat and an additional Masterbias (in my case iso settings remain unchanged for any frames), the calibration preview shows a highly colour/luminance-distorted preview. This only changes when removing the Masterbias. Am I missing a step?

Thanks in advance