The Pygac API

Reader Base Classes

Common functionality shared by multiple readers.

Reader

Generic reader for GAC and LAC data.

Can’t be used as is, has to be subclassed to add specific read functions.

exception pygac.reader.DecodingError

Raised when decoding of some value fails.

exception pygac.reader.NoTLEData

Raised if no TLE data available within time range.

class pygac.reader.Reader(interpolate_coords=True, adjust_clock_drift=True, tle_dir=None, tle_name=None, tle_thresh=7, creation_site=None, custom_calibration=None, calibration_file=None, header_date='auto', calibration_method='noaa', calibration_parameters=None, correct_scanlines=True, reference_image=None, dem=None, compute_lonlats_from_tles: bool = False)

Reader for GAC and LAC format, POD and KLM platforms.

abstract property QFlag

KLM/POD specific quality indicators.

property calibrated_dataset

Return the calibrated dataset.

classmethod can_read(filename, fileobj=None)

Read the GAC/LAC data.

Parameters:

• filename (str) – Path to GAC/LAC file

• fileobj – An open file object to read from. (optional)

Retruns:

result (bool): True if the reader can read the input

correct_scan_line_numbers()

Remove scanlines with corrupted scanline numbers.

This includes:

• Scanline numbers outside the valid range

• Scanline numbers deviating more than a certain threshold from the ideal case (1,2,3,…N)

Example files having corrupt scanline numbers:

• NSS.GHRR.NJ.D96144.S2000.E2148.B0720102.GC

• NSS.GHRR.NJ.D96064.S0043.E0236.B0606162.WI

• NSS.GHRR.NJ.D99286.S1818.E2001.B2466869.WI

Returns:

Intermediate and final results (for plotting purpose)

correct_times_median(year, jday, msec)

Replace invalid timestamps with statistical estimates (using median).

Assumes that the majority of timestamps is ok.

Parameters:

• year – Year

• jday – Day of the year

• msec – Milliseconds since 00:00

Returns:

Corrected year Corrected day of the year Corrected milliseconds

correct_times_thresh(max_diff_from_t0_head=360000, min_frac_near_t0_head=0.01, max_diff_from_ideal_t=10000)

Correct corrupted timestamps using a threshold approach.

The threshold approach is based on the scanline number and the header timestamp. It also works if the majority of scanlines has corrupted timestamps.

The header timestamp is used as a guideline to estimate the offset between timestamps computed from the scanline number and the actual scanline timestamps in the data. If header timestamp and scanline timestamps do not match, no correction is applied.

Once the offset has been estimated, one can calculate the ideal timestamps based on the scanline number. Timestamps deviating more than a certain threshold from the ideal timestamps are replaced by the ideal timestamps.

Example files having corrupt timestamps:

• NSS.GHRR.NA.D81193.S2329.E0116.B1061214.WI

• NSS.GHRR.NL.D01035.S2342.E0135.B0192627.WI

Parameters:

• max_diff_from_t0_head (int) –

  Threshold for offset estimation: A scanline timestamp matches the header timestamp t0_head if it is within the interval

  [t0_head - max_diff_from_t0_head, t0_head + max_diff_from_t0_head]

  around the header timestamp.

• min_frac_near_t0_head (float) – Specifies the minimum fraction of scanline timestamps matching the header timestamp required for applying the correction.

• max_diff_from_ideal_t (float) – Threshold for timestamp correction: If a scanline timestamp deviates more than max_diff_from_ideal_t from the ideal timestamp, it is regarded as corrupt and will be replaced with the ideal timestamp.

Returns:

Intermediate and final results (for plotting purpose)

create_counts_dataset()

data_set_pattern = re.compile('\\w{3}\\.\\w{4}\\.\\w{2}.D\\d{5}\\.S\\d{4}\\.E\\d{4}\\.B\\d{7}\\.\\w{2}')

property filename

Get the property ‘filename’.

classmethod fromfile(filename, fileobj=None)

Create Reader from file, alternative constructor.

Parameters:

filename (str) – Path to GAC/LAC file

Kwargs:

fileobj (file object): Open file object to read from

Note

The fileobj is useful when dealing with tar archives, where the filename is given by the tarinfo object, but the extracted file object’s property ‘name’ is set to the filename of the archive.

get_angles()

Get azimuth and zenith angles.

Azimuth angle definition is the same as in pyorbital, but with different units (degrees not radians for sun azimuth angles) and different ranges.

Returns:

satellite azimuth angle degree clockwise from north in range ]-180, 180]

sat_zenith: satellite zenith angles in degrees in range [0,90]

sun_azi: sun azimuth angle degree clockwise from north in range ]-180, 180]

sun_zenith: sun zenith angles in degrees in range [0,90]

rel_azi: absolute azimuth angle difference in degrees between sun and sensor in range [0, 180]

Return type:

sat_azi

get_attitude_coeffs()

Return the roll, pitch, yaw values.

get_calibrated_channels()

Calibrate and return the channels.

get_calibrated_dataset()

Create and calibrate the dataset for the pass.

get_counts()

Get the counts.

Returns:

The counts, with channel 3a and 3b split if necessary.

Return type:

np.array

abstractmethod get_header_timestamp()

Read start timestamp from the header.

Returns:

Start timestamp

Return type:

datetime.datetime

get_lonlat()

Compute lat/lon coordinates.

TODO: Switch to faster interpolator?

get_midnight_scanline()

Find the scanline where the UTC date increases by one day.

Returns:

The midnight scanline if it exists and is unique.

None, else.

Return type:

int

get_miss_lines()

Find missing scanlines.

I.e. scanlines which were dropped for some reason or were never recorded.

Returns:

Indices of missing scanlines

get_qual_flags()

Read quality flags.

get_sat_angles()

Get satellite angles.

Returns:

Azimuth, elevation (degrees)

get_sun_earth_distance_correction()

Get the julian day and the sun-earth distance correction.

abstractmethod get_telemetry()

KLM/POD specific readout of telemetry.

get_times()

Read scanline timestamps and try to correct invalid values.

Returns:

UTC timestamps

get_tle_file()

Find TLE file for the current satellite.

get_tle_lines()

Find closest two line elements (TLEs) for the current orbit.

:raises IndexError, if the closest TLE is more than pygac.GACReader.tle_thresh() days apart:

abstractmethod get_tsm_pixels(channels)

Determine pixels affected by the scan motor issue.

Channel selection is POD/KLM specific.

is_tsm_affected()

Determine whether this orbit is affected by the scan motor problem.

Returns:

True if the orbit is affected, False otherwise.

Return type:

bool

lineno2msec(scan_line_number)

Compute ideal scanline timestamp based on the scanline number.

Assumes a constant scanning frequency.

Parameters:

scan_line_number – Specifies the scanline number (1-based)

Returns:

00, i.e. the first scanline has timestamp 0.

Return type:

Corresponding timestamps in milliseconds since 1970-01-01 00

lonlat_interpolator(lons, lats, cols_subset=None, cols_full=None)

Interpolate from lat-lon tie-points to pixel locations

Parameters:

• lons – Longitude tie-points

• lats – Latitude tie-points

Returns:

pixel_longitudes, pixel_latitudes

property mask

Mask for corrupt scanlines.

mask_tsm_pixels(ds)

Mask pixels affected by the scan motor issue.

abstractmethod postproc(ds)

Apply KLM/POD specific postprocessing.

abstractmethod read(filename, fileobj=None)

Read the GAC/LAC data.

Parameters:

• filename (str) – Path to GAC/LAC file

• fileobj – An open file object to read from. (optional)

read_as_dataset(file_to_read)

abstractmethod classmethod read_header(filename, fileobj=None)

Read the file header.

Parameters:

• filename (str) – Path to GAC/LAC file

• fileobj – An open file object to read from. (optional)

Returns:

archive header header (struct): file header

Return type:

archive_header (struct)

Note

This is a classmethod to avoid throwaway instances while checking if the reader corresponds to the input file.

read_tle_file(tle_filename)

Read TLE file.

save(start_line, end_line, output_file_prefix='PyGAC', output_dir='./', avhrr_dir=None, qual_dir=None, sunsatangles_dir=None)

Convert the Reader instance content into hdf5 files.

property times

Get the UTCs as datetime.datetime.

static tle2datetime64(times)

Convert TLE timestamps to numpy.datetime64.

Parameters:

times (float) – TLE timestamps as %y%j.1234, e.g. 18001.25

static to_datetime(datetime64)

Convert numpy.datetime64 to datetime.datetime.

Parameters:

datetime64 (numpy.datetime64) – Numpy timestamp to be converted.

Returns:

Converted timestamp

Return type:

datetime.datetime

static to_datetime64(year, jday, msec)

Convert timestamps to numpy.datetime64.

Parameters:

• year – Year

• jday – Day of the year (1-based)

• msec – Milliseconds since 00:00

Returns:

Converted timestamps

Return type:

numpy.datetime64

abstract property tsm_affected_intervals

Specify time intervals being affected by the scan motor problem.

Returns:

Affected time intervals. A dictionary containing a list of

(start, end) tuples for each affected platform. Both start and end must be datetime.datetime objects.

Return type:

dict

update_meta_data()

Add some meta data to the meta_data dicitonary.

exception pygac.reader.ReaderError

Raised in Reader.read if the given file does not correspond to it.

exception pygac.reader.TimestampMismatch

Raise when matching timestamps doesn’t work.

pygac.reader.inherit_doc(cls)

Make a class method inherit its docstring from the parent class.

Copied from http://stackoverflow.com/a/8101598/5703449 .

GAC format reader

Generic reader for GAC data.

Can’t be used as is, has to be subclassed to add specific read functions.

class pygac.gac_reader.GACReader(*args, **kwargs)

Reader for GAC data.

lonlat_sample_points = array([  4,  12,  20,  28,  36,  44,  52,  60,  68,  76,  84,  92, 100,        108, 116, 124, 132, 140, 148, 156, 164, 172, 180, 188, 196, 204,        212, 220, 228, 236, 244, 252, 260, 268, 276, 284, 292, 300, 308,        316, 324, 332, 340, 348, 356, 364, 372, 380, 388, 396, 404])

max_scanlines = 15000

scan_freq = 0.002

LAC format reader

The LAC reader.

class pygac.lac_reader.LACReader(*args, **kwargs)

Reader for LAC data.

lonlat_sample_points = array([  24,   64,  104,  144,  184,  224,  264,  304,  344,  384,  424,         464,  504,  544,  584,  624,  664,  704,  744,  784,  824,  864,         904,  944,  984, 1024, 1064, 1104, 1144, 1184, 1224, 1264, 1304,        1344, 1384, 1424, 1464, 1504, 1544, 1584, 1624, 1664, 1704, 1744,        1784, 1824, 1864, 1904, 1944, 1984, 2024])

max_scanlines = 65535

scan_freq = 0.006

POD series reader

POD file reading.

Reads L1b GAC/LAC data from POD series of satellites (NOAA-14 and earlier). Format specification can be found in chapters 2 & 3 of the POD user guide.

class pygac.pod_reader.PODReader(interpolate_coords=True, adjust_clock_drift=True, tle_dir=None, tle_name=None, tle_thresh=7, creation_site=None, custom_calibration=None, calibration_file=None, header_date='auto', calibration_method='noaa', calibration_parameters=None, correct_scanlines=True, reference_image=None, dem=None, compute_lonlats_from_tles: bool = False)

The POD reader.

QFlag

alias of POD_QualityIndicator

classmethod choose_header_based_on_timestamp(header_date, fd_)

Choose the header dtype based on the timestamp.

compute_clock_offsets()

correct_scan_line_numbers()

Correct the scan line numbers.

static decode_timestamps(encoded)

Decode timestamps.

Returns:

year day of year milliseconds since 00:00

get_header_timestamp()

Get the timestamp from the header.

Returns:

A datetime object containing the timestamp from the header.

Raises:

A ValueError if the timestamp is corrupt. –

classmethod get_start_date(fd_)

Get the start time from the filestream.

get_telemetry()

Get the telemetry.

Returns:

np.array ict_counts: np.array space_counts: np.array

Return type:

prt_counts

get_tsm_pixels(channels)

Determine pixels affected by the scan motor issue.

Uses channels 1, 2, 4 and 5. Neither 3a, nor 3b.

postproc(channels)

No POD specific postprocessing to be done.

read(filename, fileobj=None)

Read the data.

Parameters:

• filename (str) – Path to GAC/LAC file

• fileobj – An open file object to read from. (optional)

Returns:

numpy record array

The header metadata

scans: numpy record array

The scanlines

Return type:

header

classmethod read_header(filename, fileobj=None, header_date='auto')

Read the file header.

Parameters:

• filename (str) – Path to GAC/LAC file

• fileobj – An open file object to read from. (optional)

• header_date – date to use to choose the header. Defaults to “auto” to use the data to pick the header corresponding to the date of the file.

Returns:

archive header header (struct): file header

Return type:

archive_header (struct)

spacecraft_names = {1: 'noaa11', 2: 'noaa6', 3: 'noaa14', 4: 'noaa7', 5: 'noaa12', 6: 'noaa8', 7: 'noaa9', 8: 'noaa10', 25: 'tirosn'}

spacecrafts_orbital = {1: 'noaa 11', 2: 'noaa 6', 3: 'noaa 14', 4: 'noaa 7', 5: 'noaa 12', 6: 'noaa 8', 7: 'noaa 9', 8: 'noaa 10', 25: 'tiros n'}

class pygac.pod_reader.POD_QualityIndicator(*values)

Quality Indicators.

Source:

POD guide Table 3.1.2.1-2. Format of quality indicators.

ASCEND_DESCEND = 33554432

BIT_SLIPPAGE = 524288

BIT_SYNC_STATUS = 8388608

CALIBRATION = 134217728

CH_3_CONTAMINATION = 262144

CH_4_CONTAMINATION = 131072

CH_5_CONTAMINATION = 65536

DATA_GAP = 536870912

DATA_JITTER = 268435456

FATAL_FLAG = 2147483648

FLYWHEELING = 1048576

FRAME_SYNC_LOCK = 2097152

NO_EARTH_LOCATION = 67108864

PSEUDO_NOISE = 16777216

SYNC_ERROR = 4194304

TIME_ERROR = 1073741824

TIP_PARITY_1 = 32768

TIP_PARITY_2 = 16384

TIP_PARITY_3 = 8192

TIP_PARITY_4 = 4096

TIP_PARITY_5 = 2048

pygac.pod_reader.main_pod(reader_cls, filename, start_line, end_line)

Generate a l1c file.

KLM series reader

Read KLM data.

Reads L1b GAC/LAC data from KLM series of satellites (NOAA-15 and later). Format specification can be found in section 8 of the KLM user guide.

class pygac.klm_reader.KLMReader(interpolate_coords=True, adjust_clock_drift=True, tle_dir=None, tle_name=None, tle_thresh=7, creation_site=None, custom_calibration=None, calibration_file=None, header_date='auto', calibration_method='noaa', calibration_parameters=None, correct_scanlines=True, reference_image=None, dem=None, compute_lonlats_from_tles: bool = False)

Reader for KLM data.

QFlag

alias of KLM_QualityIndicator

get_ch3_switch()

Channel 3 identification.

0: Channel 3b (Brightness temperature) 1: Channel 3a (Reflectance) 2: Transition (No data)

get_header_timestamp()

Get the timestamp from the header.

Returns:

A datetime object containing the timestamp from the header.

Raises:

A ValueError if the timestamp is corrupt. –

get_telemetry()

Get the telemetry.

Returns:

np.array ict_counts: np.array space_counts: np.array

Return type:

prt_counts

get_tsm_pixels(channels)

Determine pixels affected by the scan motor issue.

Uses channels 1, 2, 4 and 5. Neither 3a, nor 3b.

postproc(ds)

Apply KLM specific postprocessing.

Masking out 3a/3b/transition.

read(filename, fileobj=None)

Read the data.

Parameters:

• filename – Path to GAC/LAC file

• fileobj – An open file object to read from. (optional)

Returns:

numpy record array

The header metadata

scans: numpy record array

The scanlines

Return type:

header

classmethod read_header(filename, fileobj=None)

Read the file header.

Parameters:

• filename (str) – Path to GAC/LAC file

• fileobj – An open file object to read from. (optional)

Returns:

archive header header (struct): file header

Return type:

archive_header (struct)

spacecraft_names = {2: 'noaa16', 4: 'noaa15', 6: 'noaa17', 7: 'noaa18', 8: 'noaa19', 11: 'metopb', 12: 'metopa', 13: 'metopc'}

spacecrafts_orbital = {2: 'noaa 16', 4: 'noaa 15', 6: 'noaa 17', 7: 'noaa 18', 8: 'noaa 19', 11: 'metop 01', 12: 'metop 02', 13: 'metop 03'}

class pygac.klm_reader.KLM_QualityIndicator(*values)

Quality Indicators.

Source: KLM guide

• Table 8.3.1.3.3.1-1. Format of packed LAC/HRPT Data Sets (Version 2, pre-April 28, 2005).

• Table 8.3.1.3.3.2-1. Format of LAC/HRPT Data Record for NOAA-N (Version 5, post-November 14, 2006, all spacecraft).

• Table 8.3.1.4.3.1-1. Format of packed GAC Data Record for NOAA KLM (Version 2, pre-April 28, 2005).

• Table 8.3.1.4.3.2-1. Format of GAC Data Record for NOAA-N (Version 4, post-January 25, 2006, all spacecraft).

Notes

• Table 8.3.1.3.3.1-1. and Table 8.3.1.4.3.1-1. define bit: 21 as “frame sync word not valid”

• Table 8.3.1.3.3.2-1. and Table 8.3.1.4.3.2-1. define bit: 21 as “flywheeling detected during this frame”

BIT_SLIPPAGE = 1048576

BIT_SYNC_STATUS = 16777216

CALIBRATION = 268435456

CH_3B_RS = 128

CH_3B_RS_ANOMALY = 64

CH_3_CONTAMINATION = 192

CH_4_CONTAMINATION = 48

CH_4_RS = 32

CH_4_RS_ANOMALY = 16

CH_5_CONTAMINATION = 12

CH_5_RS = 8

CH_5_RS_ANOMALY = 4

CLOCK_UPDATE = 67108864

DATA_GAP = 536870912

DATA_JITTER = 2

FATAL_FLAG = 2147483648

FLYWHEELING = 2097152

FRAME_SYNC_LOCK = 4194304

INSTRUMENT_CHANGE = 33554432

NO_EARTH_LOCATION = 134217728

PSEUDO_NOISE = 1

SYNC_ERROR = 8388608

SYNC_INVALID = 2097152

TIME_ERROR = 1073741824

TIP_PARITY = 256

pygac.klm_reader.main_klm(reader_cls, filename, start_line, end_line)

Generate a l1c file.

Actual Reader Implementations

Actual reader implementations building upon the base classes.

GAC POD reader

Reader for GAC POD data.

class pygac.gac_pod.GACPODReader(*args, **kwargs)

The GAC POD reader class.

The scan_points attributes provides the position of the longitude and latitude points to compute relative to the full swath width.

The offset attribute tells where in the file the scanline data starts.

pygac.gac_pod.main(filename, start_line, end_line)

Generate a l1c file.

GAC KLM reader

Reader for GAC KLM data.

class pygac.gac_klm.GACKLMReader(*args, **kwargs)

The GAC KLM reader class.

The offset attribute tells where in the file the scanline data starts.

lonlat_sample_points = array([  4.5,  12.5,  20.5,  28.5,  36.5,  44.5,  52.5,  60.5,  68.5,         76.5,  84.5,  92.5, 100.5, 108.5, 116.5, 124.5, 132.5, 140.5,        148.5, 156.5, 164.5, 172.5, 180.5, 188.5, 196.5, 204.5, 212.5,        220.5, 228.5, 236.5, 244.5, 252.5, 260.5, 268.5, 276.5, 284.5,        292.5, 300.5, 308.5, 316.5, 324.5, 332.5, 340.5, 348.5, 356.5,        364.5, 372.5, 380.5, 388.5, 396.5, 404.5])

pygac.gac_klm.main(filename, start_line, end_line)

Generate a l1c file.

LAC POD reader

Reader for LAC POD data.

class pygac.lac_pod.LACPODReader(*args, **kwargs)

The LAC POD reader.

The scan_points attributes provides the position of the longitude and latitude points to compute relative to the full swath width.

The offset attribute tells where in the file the scanline data starts.

pygac.lac_pod.main(filename, start_line, end_line)

Generate a l1c file.

LAC KLM reader

Reader for LAC KLM data.

class pygac.lac_klm.LACKLMReader(*args, **kwargs)

The LAC KLM reader.

The offset attribute tells where in the file the scanline data starts.

pygac.lac_klm.main(filename, start_line, end_line)

Generate a l1c file.

Calibration