The Pygac API

Reader Base Classes

Common functionality shared by multiple readers.

Reader

Generic reader for GAC and LAC data.

Can’t be used as is, has to be subclassed to add specific read functions.

exception pygac.reader.NoTLEData

Raised if no TLE data available within time range.

class pygac.reader.Reader(interpolate_coords=True, adjust_clock_drift=True, tle_dir=None, tle_name=None, tle_thresh=7, creation_site=None, custom_calibration=None, calibration_file=None)

Reader for GAC and LAC format, POD and KLM platforms.

abstract property QFlag

KLM/POD specific quality indicators.

property calibration

Get the property ‘calibration’.

classmethod can_read(filename, fileobj=None)

Read the GAC/LAC data.

Args:

filename (str): Path to GAC/LAC file fileobj: An open file object to read from. (optional)

Retruns:

result (bool): True if the reader can read the input

correct_scan_line_numbers()

Remove scanlines with corrupted scanline numbers.

This includes:

• Scanline numbers outside the valide range

• Scanline numbers deviating more than a certain threshold from the ideal case (1,2,3,…N)

Example files having corrupt scanline numbers:

• NSS.GHRR.NJ.D96144.S2000.E2148.B0720102.GC

• NSS.GHRR.NJ.D96064.S0043.E0236.B0606162.WI

• NSS.GHRR.NJ.D99286.S1818.E2001.B2466869.WI

Returns:

Intermediate and final results (for plotting purpose)

correct_times_median(year, jday, msec)

Replace invalid timestamps with statistical estimates (using median).

Assumes that the majority of timestamps is ok.

Args:

year: Year jday: Day of the year msec: Milliseconds since 00:00

Returns:

Corrected year Corrected day of the year Corrected milliseconds

correct_times_thresh(max_diff_from_t0_head=360000, min_frac_near_t0_head=0.01, max_diff_from_ideal_t=10000)

Correct corrupted timestamps using a threshold approach.

The threshold approach is based on the scanline number and the header timestamp. It also works if the majority of scanlines has corrupted timestamps.

The header timestamp is used as a guideline to estimate the offset between timestamps computed from the scanline number and the actual scanline timestamps in the data. If header timestamp and scanline timestamps do not match, no correction is applied.

Once the offset has been estimated, one can calculate the ideal timestamps based on the scanline number. Timestamps deviating more than a certain threshold from the ideal timestamps are replaced by the ideal timestamps.

Example files having corrupt timestamps:

• NSS.GHRR.NA.D81193.S2329.E0116.B1061214.WI

• NSS.GHRR.NL.D01035.S2342.E0135.B0192627.WI

Args:

max_diff_from_t0_head (int): Threshold for offset estimation: A

scanline timestamp matches the header timestamp t0_head if it is within the interval

[t0_head - max_diff_from_t0_head, t0_head + max_diff_from_t0_head]

around the header timestamp.

min_frac_near_t0_head (float): Specifies the minimum fraction of

scanline timestamps matching the header timestamp required for applying the correction.

max_diff_from_ideal_t (float): Threshold for timestamp correction:

If a scanline timestamp deviates more than max_diff_from_ideal_t from the ideal timestamp, it is regarded as corrupt and will be replaced with the ideal timestamp.

Returns:

Intermediate and final results (for plotting purpose)

data_set_pattern = re.compile('\\w{3}\\.\\w{4}\\.\\w{2}.D\\d{5}\\.S\\d{4}\\.E\\d{4}\\.B\\d{7}\\.\\w{2}')

property filename

Get the property ‘filename’.

classmethod fromfile(filename, fileobj=None)

Create Reader from file (alternative constructor)

Args:

filename (str): Path to GAC/LAC file

Kwargs:

fileobj (file object): Open file object to read from

Note:

The fileobj is useful when dealing with tar archives, where the filename is given by the tarinfo object, but the extracted file object’s property ‘name’ is set to the filename of the archive.

get_angles()

Get azimuth and zenith angles.

Azimuth angle definition is the same as in pyorbital, but with different units (degrees not radians for sun azimuth angles) and different ranges.

Returns:

sat_azi: satellite azimuth angle degree clockwise from north in range ]-180, 180]

sat_zentih: satellite zenith angles in degrees in range [0,90]

sun_azi: sun azimuth angle degree clockwise from north in range ]-180, 180]

sun_zentih: sun zenith angles in degrees in range [0,90]

rel_azi: absolute azimuth angle difference in degrees between sun and sensor in range [0, 180]

get_attitude_coeffs()

Return the roll, pitch, yaw values

get_calibrated_channels()

Calibrate and return the channels.

get_counts()

Get the counts.

Returns:

np.array: The counts, with channel 3a and 3b split if necessary.

abstract get_header_timestamp()

Read start timestamp from the header.

Returns:

datetime.datetime: Start timestamp

get_lonlat()

Compute lat/lon coordinates.

TODO: Switch to faster interpolator?

get_midnight_scanline()

Find the scanline where the UTC date increases by one day.

Returns:

int: The midnight scanline if it exists and is unique.

None, else.

get_miss_lines()

Find missing scanlines.

I.e. scanlines which were dropped for some reason or were never recorded.

Returns:

Indices of missing scanlines

get_qual_flags()

Read quality flags.

get_sat_angles()

Get satellite angles.

Returns:

Azimuth, elevation (degrees)

get_sun_earth_distance_correction()

Get the julian day and the sun-earth distance correction.

abstract get_telemetry()

KLM/POD specific readout of telemetry.

get_times()

Read scanline timestamps and try to correct invalid values.

Note:

Also sets self.utcs and self.times!

Returns:

UTC timestamps

get_tle_file()

Find TLE file for the current satellite.

get_tle_lines()

Find closest two line elements (TLEs) for the current orbit.

Raises:

IndexError, if the closest TLE is more than pygac.GACReader.tle_thresh() days apart

abstract get_tsm_pixels(channels)

Determine pixels affected by the scan motor issue.

Channel selection is POD/KLM specific.

is_tsm_affected()

Determine whether this orbit is affected by the scan motor problem.

Returns:

bool: True if the orbit is affected, False otherwise.

lineno2msec(scan_line_number)

Compute ideal scanline timestamp based on the scanline number.

Assumes a constant scanning frequency.

Args:

scan_line_number: Specifies the scanline number (1-based)

Returns:

Corresponding timestamps in milliseconds since 1970-01-01 00:00, i.e. the first scanline has timestamp 0.

property mask

Mask for corrupt scanlines.

mask_tsm_pixels(channels)

Mask pixels affected by the scan motor issue.

abstract postproc(channels)

Apply KLM/POD specific postprocessing.

abstract read(filename, fileobj=None)

Read the GAC/LAC data.

Args:

filename (str): Path to GAC/LAC file fileobj: An open file object to read from. (optional)

abstract classmethod read_header(filename, fileobj=None)

Read the file header.

Args:

filename (str): Path to GAC/LAC file fileobj: An open file object to read from. (optional)

Returns:

archive_header (struct): archive header header (struct): file header

Note:

This is a classmethod to avoid throwaway instances while checking if the reader corresponds to the input file.

read_tle_file(tle_filename)

Read TLE file.

save(start_line, end_line, output_file_prefix='PyGAC', output_dir='./', avhrr_dir=None, qual_dir=None, sunsatangles_dir=None)

Convert the Reader instance content into hdf5 files

property times

The UTCs as datetime.datetime

static tle2datetime64(times)

Convert TLE timestamps to numpy.datetime64.

Args:

times (float): TLE timestamps as %y%j.1234, e.g. 18001.25

static to_datetime(datetime64)

Convert numpy.datetime64 to datetime.datetime.

Args:

datetime64 (numpy.datetime64): Numpy timestamp to be converted.

Returns:

datetime.datetime: Converted timestamp

static to_datetime64(year, jday, msec)

Convert timestamps to numpy.datetime64.

Args:

year: Year jday: Day of the year (1-based) msec: Milliseconds since 00:00

Returns:

numpy.datetime64: Converted timestamps

abstract property tsm_affected_intervals

Specify time intervals being affected by the scan motor problem.

Returns:

dict: Affected time intervals. A dictionary containing a list of

(start, end) tuples for each affected platform. Both start and end must be datetime.datetime objects.

update_meta_data()

Add some metd data to the meta_data dicitonary.

exception pygac.reader.ReaderError

Raised in Reader.read if the given file does not correspond to it

pygac.reader.inherit_doc(cls)

Make a class method inherit its docstring from the parent class.

Copied from http://stackoverflow.com/a/8101598/5703449 .

GAC format reader

Generic reader for GAC data.

Can’t be used as is, has to be subclassed to add specific read functions.

class pygac.gac_reader.GACReader(*args, **kwargs)

Reader for GAC data.

scan_freq = 0.002

LAC format reader

The LAC reader.

class pygac.lac_reader.LACReader(*args, **kwargs)

Reader for LAC data.

scan_freq = 0.006

POD series reader

POD file reading.

Reads L1b GAC/LAC data from POD series of satellites (NOAA-14 and earlier). Format specification can be found in chapters 2 & 3 of the POD user guide.

class pygac.pod_reader.PODReader(interpolate_coords=True, adjust_clock_drift=True, tle_dir=None, tle_name=None, tle_thresh=7, creation_site=None, custom_calibration=None, calibration_file=None)

The POD reader.

QFlag

alias of POD_QualityIndicator

correct_scan_line_numbers()

Correct the scan line numbers.

static decode_timestamps(encoded)

Decode timestamps.

Returns:

year day of year milliseconds since 00:00

get_header_timestamp()

Get the timestamp from the header.

Returns:

A datetime object containing the timestamp from the header.

Raises:

A ValueError if the timestamp is corrupt.

get_telemetry()

Get the telemetry.

Returns:

prt_counts: np.array ict_counts: np.array space_counts: np.array

get_tsm_pixels(channels)

Determine pixels affected by the scan motor issue.

Uses channels 1, 2, 4 and 5. Neither 3a, nor 3b.

postproc(channels)

No POD specific postprocessing to be done.

read(filename, fileobj=None)

Read the data.

Args:

filename (str): Path to GAC/LAC file fileobj: An open file object to read from. (optional)

Returns:

header: numpy record array

The header metadata

scans: numpy record array

The scanlines

classmethod read_header(filename, fileobj=None)

Read the file header.

Args:

filename (str): Path to GAC/LAC file fileobj: An open file object to read from. (optional)

Returns:

archive_header (struct): archive header header (struct): file header

spacecraft_names = {1: 'noaa11', 2: 'noaa6', 3: 'noaa14', 4: 'noaa7', 5: 'noaa12', 6: 'noaa8', 7: 'noaa9', 8: 'noaa10', 25: 'tirosn'}

spacecrafts_orbital = {1: 'noaa 11', 2: 'noaa 6', 3: 'noaa 14', 4: 'noaa 7', 5: 'noaa 12', 6: 'noaa 8', 7: 'noaa 9', 8: 'noaa 10', 25: 'tiros n'}

class pygac.pod_reader.POD_QualityIndicator(value)

Quality Indicators.

Source:

POD guide Table 3.1.2.1-2. Format of quality indicators.

ASCEND_DESCEND = 33554432

BIT_SLIPPAGE = 524288

BIT_SYNC_STATUS = 8388608

CALIBRATION = 134217728

CH_3_CONTAMINATION = 262144

CH_4_CONTAMINATION = 131072

CH_5_CONTAMINATION = 65536

DATA_GAP = 536870912

DATA_JITTER = 268435456

FATAL_FLAG = 2147483648

FLYWHEELING = 1048576

FRAME_SYNC_LOCK = 2097152

NO_EARTH_LOCATION = 67108864

PSEUDO_NOISE = 16777216

SYNC_ERROR = 4194304

TIME_ERROR = 1073741824

TIP_PARITY_1 = 32768

TIP_PARITY_2 = 16384

TIP_PARITY_3 = 8192

TIP_PARITY_4 = 4096

TIP_PARITY_5 = 2048

pygac.pod_reader.main_pod(reader_cls, filename, start_line, end_line)

Generate a l1c file.

KLM series reader

Read KLM data.

Reads L1b GAC/LAC data from KLM series of satellites (NOAA-15 and later). Format specification can be found in section 8 of the KLM user guide.

class pygac.klm_reader.KLMReader(interpolate_coords=True, adjust_clock_drift=True, tle_dir=None, tle_name=None, tle_thresh=7, creation_site=None, custom_calibration=None, calibration_file=None)

Reader for KLM data.

QFlag

alias of KLM_QualityIndicator

get_ch3_switch()

Channel 3 identification.

0: Channel 3b (Brightness temperature 1: Channel 3a (Reflectance) 2: Transition (No data)

get_header_timestamp()

Get the timestamp from the header.

Returns:

A datetime object containing the timestamp from the header.

Raises:

A ValueError if the timestamp is corrupt.

get_telemetry()

Get the telemetry.

Returns:

prt_counts: np.array ict_counts: np.array space_counts: np.array

get_tsm_pixels(channels)

Determine pixels affected by the scan motor issue.

Uses channels 1, 2, 4 and 5. Neither 3a, nor 3b.

postproc(channels)

Apply KLM specific postprocessing.

Masking out 3a/3b/transition.

read(filename, fileobj=None)

Read the data.

Args:

filename: Path to GAC/LAC file fileobj: An open file object to read from. (optional)

Returns:

header: numpy record array

The header metadata

scans: numpy record array

The scanlines

classmethod read_header(filename, fileobj=None)

Read the file header.

Args:

filename (str): Path to GAC/LAC file fileobj: An open file object to read from. (optional)

Returns:

archive_header (struct): archive header header (struct): file header

spacecraft_names = {2: 'noaa16', 4: 'noaa15', 6: 'noaa17', 7: 'noaa18', 8: 'noaa19', 11: 'metopb', 12: 'metopa', 13: 'metopc'}

spacecrafts_orbital = {2: 'noaa 16', 4: 'noaa 15', 6: 'noaa 17', 7: 'noaa 18', 8: 'noaa 19', 11: 'metop 01', 12: 'metop 02', 13: 'metop 03'}

class pygac.klm_reader.KLM_QualityIndicator(value)

Quality Indicators.

Source: KLM guide

• Table 8.3.1.3.3.1-1. Format of packed LAC/HRPT Data Sets (Version 2, pre-April 28, 2005).

• Table 8.3.1.3.3.2-1. Format of LAC/HRPT Data Record for NOAA-N (Version 5, post-November 14, 2006, all spacecraft).

• Table 8.3.1.4.3.1-1. Format of packed GAC Data Record for NOAA KLM (Version 2, pre-April 28, 2005).

• Table 8.3.1.4.3.2-1. Format of GAC Data Record for NOAA-N (Version 4, post-January 25, 2006, all spacecraft).

Notes:

• Table 8.3.1.3.3.1-1. and Table 8.3.1.4.3.1-1. define bit: 21 as “frame sync word not valid”

• Table 8.3.1.3.3.2-1. and Table 8.3.1.4.3.2-1. define bit: 21 as “flywheeling detected during this frame”

BIT_SLIPPAGE = 1048576

BIT_SYNC_STATUS = 16777216

CALIBRATION = 268435456

CH_3B_RS = 128

CH_3B_RS_ANOMALY = 64

CH_3_CONTAMINATION = 192

CH_4_CONTAMINATION = 48

CH_4_RS = 32

CH_4_RS_ANOMALY = 16

CH_5_CONTAMINATION = 12

CH_5_RS = 8

CH_5_RS_ANOMALY = 4

CLOCK_UPDATE = 67108864

DATA_GAP = 536870912

DATA_JITTER = 2

FATAL_FLAG = 2147483648

FLYWHEELING = 2097152

FRAME_SYNC_LOCK = 4194304

INSTRUMENT_CHANGE = 33554432

NO_EARTH_LOCATION = 134217728

PSEUDO_NOISE = 1

SYNC_ERROR = 8388608

SYNC_INVALID = 2097152

TIME_ERROR = 1073741824

TIP_PARITY = 256

pygac.klm_reader.main_klm(reader_cls, filename, start_line, end_line)

Generate a l1c file.

Actual Reader Implementations

Actual reader implementations building upon the base classes.

GAC POD reader

Reader for GAC POD data.

class pygac.gac_pod.GACPODReader(*args, **kwargs)

The GAC POD reader class.

The scan_points attributes provides the position of the longitude and latitude points to compute relative to the full swath width.

The offset attribute tells where in the file the scanline data starts.

pygac.gac_pod.main(filename, start_line, end_line)

Generate a l1c file.

GAC KLM reader

Reader for GAC KLM data.

class pygac.gac_klm.GACKLMReader(*args, **kwargs)

The GAC KLM reader class.

The offset attribute tells where in the file the scanline data starts.

pygac.gac_klm.main(filename, start_line, end_line)

Generate a l1c file.

LAC POD reader

Reader for LAC POD data.

class pygac.lac_pod.LACPODReader(*args, **kwargs)

The LAC POD reader.

The scan_points attributes provides the position of the longitude and latitude points to compute relative to the full swath width.

The offset attribute tells where in the file the scanline data starts.

pygac.lac_pod.main(filename, start_line, end_line)

Generate a l1c file.

LAC KLM reader

Reader for LAC KLM data.

class pygac.lac_klm.LACKLMReader(*args, **kwargs)

The LAC KLM reader.

The offset attribute tells where in the file the scanline data starts.

pygac.lac_klm.main(filename, start_line, end_line)

Generate a l1c file.

Calibration

Calibration coefficients and generic calibration functions

class pygac.calibration.Calibrator(spacecraft, custom_coeffs=None, coeffs_file=None)

Factory class to create namedtuples holding the calibration coefficients.

Attributes:

fields: coefficient names Calibrator: namedtuple constructor default_coeffs: dictonary containing default values for all spacecrafts

class Calibrator(dark_count, gain_switch, s0, s1, s2, b, centroid_wavenumber, space_radiance, to_eff_blackbody_intercept, to_eff_blackbody_slope, date_of_launch, d, spacecraft, version)

b

Alias for field number 5

centroid_wavenumber

Alias for field number 6

d

Alias for field number 11

dark_count

Alias for field number 0

date_of_launch

Alias for field number 10

gain_switch

Alias for field number 1

s0

Alias for field number 2

s1

Alias for field number 3

s2

Alias for field number 4

space_radiance

Alias for field number 7

spacecraft

Alias for field number 12

to_eff_blackbody_intercept

Alias for field number 8

to_eff_blackbody_slope

Alias for field number 9

version

Alias for field number 13

static date2float(date, decimals=5)

Convert date to year float.

Argument

date (datetime.datetime) - date decimals (int or None) - rounding precision if None, do not round (default=5)

Return

date_float (float) - date as year

Note

rounding to the 5th decimal reproduces the original float values from patmos-x

Example:

date2float(‘2000-07-02’) == 2000.5 because the 2nd of July was the middle day of the leap year 2000

default_coeffs = None

default_file = None

default_version = None

fields = ['dark_count', 'gain_switch', 's0', 's1', 's2', 'b', 'centroid_wavenumber', 'space_radiance', 'to_eff_blackbody_intercept', 'to_eff_blackbody_slope', 'date_of_launch', 'd', 'spacecraft', 'version']

classmethod read_coeffs(coeffs_file)

Read calibration coefficients for all satellites from file.

Args:

coeffs_file (str): path to coefficients file

Returns:

coeffs (dict): dictionary containing coefficients for all satellites version (str): version of the coefficients (None if unknown)

version_hashs = {'689386c822de18a07194ac7fd71652ea': {'name': 'PATMOS-x, v2017r1, with provisional coefficients for MetOp-C', 'status': CoeffStatus.PROVISIONAL}, '963af9b66268475ed500ad7b37da33c5': {'name': 'PATMOS-x, v2017r1', 'status': CoeffStatus.NOMINAL}}

class pygac.calibration.CoeffStatus(value)

Indicates the status of calibration coefficients.

EXPERIMENTAL = 'experimental'

NOMINAL = 'nominal'

PROVISIONAL = 'provisional'

pygac.calibration.calibrate_solar(counts, chan, year, jday, cal, corr=1)

Do the solar calibration and return scaled radiance.

Arguments:

counts (array) - raw counts for the given channels (options 1, 2[, 3A if available & active]) chan (array) - pygac internal channel index array year (int) - year jday (int) - day of year cal (namedtuple) - spacecraft specific calibration coefficients, see Calibrator

Optional:

corr (float) - depricated - reflectance correction multiplier (default = 1)

Returns:

r_cal (array) - scaled radiance

Note:

This function and documentation follows the time-dependent solar calibration as described in: Heidinger, A.K., W.C. Straka III, C.C. Molling, J.T. Sullivan, and X. Wu, (2010). “Deriving an inter-sensor consistent calibration for the AVHRR solar reflectance data record.”, International Journal of Remote Sensing, 31:6493 - 6517.

pygac.calibration.calibrate_thermal(counts, prt, ict, space, line_numbers, channel, cal)

Do the thermal calibration and return brightness temperatures (K).

Arguments:

counts (array) - counts for the given channel (options: 3B (if active), 4, 5) prt (array) - counts of the Platinum Resistance Thermometers (PRT) ict (array) - counts of the In-orbit Calibration Targets (ICT) space (array) - counts of cold space line_numbers (array) - line number index channel (array) - pygac internal channel index array cal (namedtuple) - spacecraft specific calibration coefficients, see Calibrator

Note:

This function and documentation follows steps 1 to 4 from the KLM User’s Guide (Robel, J. (2009). NOAA KLM user’s guide with NOAA-N,-P supplement. NOAA KLM Users Guide - August 2014 Revision) section 7.1.2.4 “Steps to Calibrate the AVHRR Thermal Channels”, and the smoothing approach by Trishchenko (2002). The correction method for the non-linear response of the Mercury-Cadmium-Telluride detectors used for channels 4 and 5 is based on Walton et al. (1998)