The Pygac API¶
Reader Base Classes¶
Common functionality shared by multiple readers.
Reader¶
Generic reader for GAC and LAC data.
Can’t be used as is, has to be subclassed to add specific read functions.
- exception pygac.reader.NoTLEData¶
Raised if no TLE data available within time range.
- class pygac.reader.Reader(interpolate_coords=True, adjust_clock_drift=True, tle_dir=None, tle_name=None, tle_thresh=7, creation_site=None, custom_calibration=None, calibration_file=None)¶
Reader for GAC and LAC format, POD and KLM platforms.
- abstract property QFlag¶
KLM/POD specific quality indicators.
- property calibration¶
Get the property ‘calibration’.
- classmethod can_read(filename, fileobj=None)¶
Read the GAC/LAC data.
- Args:
filename (str): Path to GAC/LAC file fileobj: An open file object to read from. (optional)
- Retruns:
result (bool): True if the reader can read the input
- correct_scan_line_numbers()¶
Remove scanlines with corrupted scanline numbers.
- This includes:
Scanline numbers outside the valid range
Scanline numbers deviating more than a certain threshold from the ideal case (1,2,3,…N)
- Example files having corrupt scanline numbers:
NSS.GHRR.NJ.D96144.S2000.E2148.B0720102.GC
NSS.GHRR.NJ.D96064.S0043.E0236.B0606162.WI
NSS.GHRR.NJ.D99286.S1818.E2001.B2466869.WI
- Returns:
Intermediate and final results (for plotting purpose)
- correct_times_median(year, jday, msec)¶
Replace invalid timestamps with statistical estimates (using median).
Assumes that the majority of timestamps is ok.
- Args:
year: Year jday: Day of the year msec: Milliseconds since 00:00
- Returns:
Corrected year Corrected day of the year Corrected milliseconds
- correct_times_thresh(max_diff_from_t0_head=360000, min_frac_near_t0_head=0.01, max_diff_from_ideal_t=10000)¶
Correct corrupted timestamps using a threshold approach.
The threshold approach is based on the scanline number and the header timestamp. It also works if the majority of scanlines has corrupted timestamps.
The header timestamp is used as a guideline to estimate the offset between timestamps computed from the scanline number and the actual scanline timestamps in the data. If header timestamp and scanline timestamps do not match, no correction is applied.
Once the offset has been estimated, one can calculate the ideal timestamps based on the scanline number. Timestamps deviating more than a certain threshold from the ideal timestamps are replaced by the ideal timestamps.
- Example files having corrupt timestamps:
NSS.GHRR.NA.D81193.S2329.E0116.B1061214.WI
NSS.GHRR.NL.D01035.S2342.E0135.B0192627.WI
- Args:
- max_diff_from_t0_head (int): Threshold for offset estimation: A
scanline timestamp matches the header timestamp t0_head if it is within the interval
[t0_head - max_diff_from_t0_head, t0_head + max_diff_from_t0_head]
around the header timestamp.
- min_frac_near_t0_head (float): Specifies the minimum fraction of
scanline timestamps matching the header timestamp required for applying the correction.
- max_diff_from_ideal_t (float): Threshold for timestamp correction:
If a scanline timestamp deviates more than max_diff_from_ideal_t from the ideal timestamp, it is regarded as corrupt and will be replaced with the ideal timestamp.
- Returns:
Intermediate and final results (for plotting purpose)
- data_set_pattern = re.compile('\\w{3}\\.\\w{4}\\.\\w{2}.D\\d{5}\\.S\\d{4}\\.E\\d{4}\\.B\\d{7}\\.\\w{2}')¶
- property filename¶
Get the property ‘filename’.
- classmethod fromfile(filename, fileobj=None)¶
Create Reader from file, alternative constructor.
- Args:
filename (str): Path to GAC/LAC file
- Kwargs:
fileobj (file object): Open file object to read from
- Note:
The fileobj is useful when dealing with tar archives, where the filename is given by the tarinfo object, but the extracted file object’s property ‘name’ is set to the filename of the archive.
- get_angles()¶
Get azimuth and zenith angles.
Azimuth angle definition is the same as in pyorbital, but with different units (degrees not radians for sun azimuth angles) and different ranges.
- Returns:
sat_azi: satellite azimuth angle degree clockwise from north in range ]-180, 180]
sat_zenith: satellite zenith angles in degrees in range [0,90]
sun_azi: sun azimuth angle degree clockwise from north in range ]-180, 180]
sun_zenith: sun zenith angles in degrees in range [0,90]
rel_azi: absolute azimuth angle difference in degrees between sun and sensor in range [0, 180]
- get_attitude_coeffs()¶
Return the roll, pitch, yaw values.
- get_calibrated_channels()¶
Calibrate and return the channels.
- get_counts()¶
Get the counts.
- Returns:
np.array: The counts, with channel 3a and 3b split if necessary.
- abstract get_header_timestamp()¶
Read start timestamp from the header.
- Returns:
datetime.datetime: Start timestamp
- get_lonlat()¶
Compute lat/lon coordinates.
TODO: Switch to faster interpolator?
- get_midnight_scanline()¶
Find the scanline where the UTC date increases by one day.
- Returns:
- int: The midnight scanline if it exists and is unique.
None, else.
- get_miss_lines()¶
Find missing scanlines.
I.e. scanlines which were dropped for some reason or were never recorded.
- Returns:
Indices of missing scanlines
- get_qual_flags()¶
Read quality flags.
- get_sat_angles()¶
Get satellite angles.
- Returns:
Azimuth, elevation (degrees)
- get_sun_earth_distance_correction()¶
Get the julian day and the sun-earth distance correction.
- abstract get_telemetry()¶
KLM/POD specific readout of telemetry.
- get_times()¶
Read scanline timestamps and try to correct invalid values.
- Note:
Also sets self.utcs and self.times!
- Returns:
UTC timestamps
- get_tle_file()¶
Find TLE file for the current satellite.
- get_tle_lines()¶
Find closest two line elements (TLEs) for the current orbit.
- Raises:
IndexError, if the closest TLE is more than
pygac.GACReader.tle_thresh()
days apart
- abstract get_tsm_pixels(channels)¶
Determine pixels affected by the scan motor issue.
Channel selection is POD/KLM specific.
- is_tsm_affected()¶
Determine whether this orbit is affected by the scan motor problem.
- Returns:
bool: True if the orbit is affected, False otherwise.
- lineno2msec(scan_line_number)¶
Compute ideal scanline timestamp based on the scanline number.
Assumes a constant scanning frequency.
- Args:
scan_line_number: Specifies the scanline number (1-based)
- Returns:
Corresponding timestamps in milliseconds since 1970-01-01 00:00, i.e. the first scanline has timestamp 0.
- property mask¶
Mask for corrupt scanlines.
- mask_tsm_pixels(channels)¶
Mask pixels affected by the scan motor issue.
- abstract postproc(channels)¶
Apply KLM/POD specific postprocessing.
- abstract read(filename, fileobj=None)¶
Read the GAC/LAC data.
- Args:
filename (str): Path to GAC/LAC file fileobj: An open file object to read from. (optional)
- abstract classmethod read_header(filename, fileobj=None)¶
Read the file header.
- Args:
filename (str): Path to GAC/LAC file fileobj: An open file object to read from. (optional)
- Returns:
archive_header (struct): archive header header (struct): file header
- Note:
This is a classmethod to avoid throwaway instances while checking if the reader corresponds to the input file.
- read_tle_file(tle_filename)¶
Read TLE file.
- save(start_line, end_line, output_file_prefix='PyGAC', output_dir='./', avhrr_dir=None, qual_dir=None, sunsatangles_dir=None)¶
Convert the Reader instance content into hdf5 files.
- property times¶
Get the UTCs as datetime.datetime.
- static tle2datetime64(times)¶
Convert TLE timestamps to numpy.datetime64.
- Args:
times (float): TLE timestamps as %y%j.1234, e.g. 18001.25
- static to_datetime(datetime64)¶
Convert numpy.datetime64 to datetime.datetime.
- Args:
datetime64 (numpy.datetime64): Numpy timestamp to be converted.
- Returns:
datetime.datetime: Converted timestamp
- static to_datetime64(year, jday, msec)¶
Convert timestamps to numpy.datetime64.
- Args:
year: Year jday: Day of the year (1-based) msec: Milliseconds since 00:00
- Returns:
numpy.datetime64: Converted timestamps
- abstract property tsm_affected_intervals¶
Specify time intervals being affected by the scan motor problem.
- Returns:
- dict: Affected time intervals. A dictionary containing a list of
(start, end) tuples for each affected platform. Both start and end must be datetime.datetime objects.
- update_meta_data()¶
Add some metd data to the meta_data dicitonary.
- exception pygac.reader.ReaderError¶
Raised in Reader.read if the given file does not correspond to it.
- pygac.reader.inherit_doc(cls)¶
Make a class method inherit its docstring from the parent class.
Copied from http://stackoverflow.com/a/8101598/5703449 .
GAC format reader¶
Generic reader for GAC data.
Can’t be used as is, has to be subclassed to add specific read functions.
LAC format reader¶
The LAC reader.
POD series reader¶
POD file reading.
Reads L1b GAC/LAC data from POD series of satellites (NOAA-14 and earlier). Format specification can be found in chapters 2 & 3 of the POD user guide.
- class pygac.pod_reader.PODReader(interpolate_coords=True, adjust_clock_drift=True, tle_dir=None, tle_name=None, tle_thresh=7, creation_site=None, custom_calibration=None, calibration_file=None)¶
The POD reader.
- QFlag¶
alias of
POD_QualityIndicator
- correct_scan_line_numbers()¶
Correct the scan line numbers.
- static decode_timestamps(encoded)¶
Decode timestamps.
- Returns:
year day of year milliseconds since 00:00
- get_header_timestamp()¶
Get the timestamp from the header.
- Returns:
A datetime object containing the timestamp from the header.
- Raises:
A ValueError if the timestamp is corrupt.
- get_telemetry()¶
Get the telemetry.
- Returns:
prt_counts: np.array ict_counts: np.array space_counts: np.array
- get_tsm_pixels(channels)¶
Determine pixels affected by the scan motor issue.
Uses channels 1, 2, 4 and 5. Neither 3a, nor 3b.
- postproc(channels)¶
No POD specific postprocessing to be done.
- read(filename, fileobj=None)¶
Read the data.
- Args:
filename (str): Path to GAC/LAC file fileobj: An open file object to read from. (optional)
- Returns:
- header: numpy record array
The header metadata
- scans: numpy record array
The scanlines
- classmethod read_header(filename, fileobj=None)¶
Read the file header.
- Args:
filename (str): Path to GAC/LAC file fileobj: An open file object to read from. (optional)
- Returns:
archive_header (struct): archive header header (struct): file header
- spacecraft_names = {1: 'noaa11', 2: 'noaa6', 3: 'noaa14', 4: 'noaa7', 5: 'noaa12', 6: 'noaa8', 7: 'noaa9', 8: 'noaa10', 25: 'tirosn'}¶
- spacecrafts_orbital = {1: 'noaa 11', 2: 'noaa 6', 3: 'noaa 14', 4: 'noaa 7', 5: 'noaa 12', 6: 'noaa 8', 7: 'noaa 9', 8: 'noaa 10', 25: 'tiros n'}¶
- class pygac.pod_reader.POD_QualityIndicator(value)¶
Quality Indicators.
- Source:
POD guide Table 3.1.2.1-2. Format of quality indicators.
- ASCEND_DESCEND = 33554432¶
- BIT_SLIPPAGE = 524288¶
- BIT_SYNC_STATUS = 8388608¶
- CALIBRATION = 134217728¶
- CH_3_CONTAMINATION = 262144¶
- CH_4_CONTAMINATION = 131072¶
- CH_5_CONTAMINATION = 65536¶
- DATA_GAP = 536870912¶
- DATA_JITTER = 268435456¶
- FATAL_FLAG = 2147483648¶
- FLYWHEELING = 1048576¶
- FRAME_SYNC_LOCK = 2097152¶
- NO_EARTH_LOCATION = 67108864¶
- PSEUDO_NOISE = 16777216¶
- SYNC_ERROR = 4194304¶
- TIME_ERROR = 1073741824¶
- TIP_PARITY_1 = 32768¶
- TIP_PARITY_2 = 16384¶
- TIP_PARITY_3 = 8192¶
- TIP_PARITY_4 = 4096¶
- TIP_PARITY_5 = 2048¶
- pygac.pod_reader.main_pod(reader_cls, filename, start_line, end_line)¶
Generate a l1c file.
KLM series reader¶
Read KLM data.
Reads L1b GAC/LAC data from KLM series of satellites (NOAA-15 and later). Format specification can be found in section 8 of the KLM user guide.
- class pygac.klm_reader.KLMReader(interpolate_coords=True, adjust_clock_drift=True, tle_dir=None, tle_name=None, tle_thresh=7, creation_site=None, custom_calibration=None, calibration_file=None)¶
Reader for KLM data.
- QFlag¶
alias of
KLM_QualityIndicator
- get_ch3_switch()¶
Channel 3 identification.
0: Channel 3b (Brightness temperature 1: Channel 3a (Reflectance) 2: Transition (No data)
- get_header_timestamp()¶
Get the timestamp from the header.
- Returns:
A datetime object containing the timestamp from the header.
- Raises:
A ValueError if the timestamp is corrupt.
- get_telemetry()¶
Get the telemetry.
- Returns:
prt_counts: np.array ict_counts: np.array space_counts: np.array
- get_tsm_pixels(channels)¶
Determine pixels affected by the scan motor issue.
Uses channels 1, 2, 4 and 5. Neither 3a, nor 3b.
- postproc(channels)¶
Apply KLM specific postprocessing.
Masking out 3a/3b/transition.
- read(filename, fileobj=None)¶
Read the data.
- Args:
filename: Path to GAC/LAC file fileobj: An open file object to read from. (optional)
- Returns:
- header: numpy record array
The header metadata
- scans: numpy record array
The scanlines
- classmethod read_header(filename, fileobj=None)¶
Read the file header.
- Args:
filename (str): Path to GAC/LAC file fileobj: An open file object to read from. (optional)
- Returns:
archive_header (struct): archive header header (struct): file header
- spacecraft_names = {2: 'noaa16', 4: 'noaa15', 6: 'noaa17', 7: 'noaa18', 8: 'noaa19', 11: 'metopb', 12: 'metopa', 13: 'metopc'}¶
- spacecrafts_orbital = {2: 'noaa 16', 4: 'noaa 15', 6: 'noaa 17', 7: 'noaa 18', 8: 'noaa 19', 11: 'metop 01', 12: 'metop 02', 13: 'metop 03'}¶
- class pygac.klm_reader.KLM_QualityIndicator(value)¶
Quality Indicators.
Source: KLM guide
Table 8.3.1.3.3.1-1. Format of packed LAC/HRPT Data Sets (Version 2, pre-April 28, 2005).
Table 8.3.1.3.3.2-1. Format of LAC/HRPT Data Record for NOAA-N (Version 5, post-November 14, 2006, all spacecraft).
Table 8.3.1.4.3.1-1. Format of packed GAC Data Record for NOAA KLM (Version 2, pre-April 28, 2005).
Table 8.3.1.4.3.2-1. Format of GAC Data Record for NOAA-N (Version 4, post-January 25, 2006, all spacecraft).
Notes:
Table 8.3.1.3.3.1-1. and Table 8.3.1.4.3.1-1. define bit: 21 as “frame sync word not valid”
Table 8.3.1.3.3.2-1. and Table 8.3.1.4.3.2-1. define bit: 21 as “flywheeling detected during this frame”
- BIT_SLIPPAGE = 1048576¶
- BIT_SYNC_STATUS = 16777216¶
- CALIBRATION = 268435456¶
- CH_3B_RS = 128¶
- CH_3B_RS_ANOMALY = 64¶
- CH_3_CONTAMINATION = 192¶
- CH_4_CONTAMINATION = 48¶
- CH_4_RS = 32¶
- CH_4_RS_ANOMALY = 16¶
- CH_5_CONTAMINATION = 12¶
- CH_5_RS = 8¶
- CH_5_RS_ANOMALY = 4¶
- CLOCK_UPDATE = 67108864¶
- DATA_GAP = 536870912¶
- DATA_JITTER = 2¶
- FATAL_FLAG = 2147483648¶
- FLYWHEELING = 2097152¶
- FRAME_SYNC_LOCK = 4194304¶
- INSTRUMENT_CHANGE = 33554432¶
- NO_EARTH_LOCATION = 134217728¶
- PSEUDO_NOISE = 1¶
- SYNC_ERROR = 8388608¶
- SYNC_INVALID = 2097152¶
- TIME_ERROR = 1073741824¶
- TIP_PARITY = 256¶
- pygac.klm_reader.main_klm(reader_cls, filename, start_line, end_line)¶
Generate a l1c file.
Actual Reader Implementations¶
Actual reader implementations building upon the base classes.
GAC POD reader¶
Reader for GAC POD data.
- class pygac.gac_pod.GACPODReader(*args, **kwargs)¶
The GAC POD reader class.
The scan_points attributes provides the position of the longitude and latitude points to compute relative to the full swath width.
The offset attribute tells where in the file the scanline data starts.
- pygac.gac_pod.main(filename, start_line, end_line)¶
Generate a l1c file.
GAC KLM reader¶
Reader for GAC KLM data.
- class pygac.gac_klm.GACKLMReader(*args, **kwargs)¶
The GAC KLM reader class.
The offset attribute tells where in the file the scanline data starts.
- pygac.gac_klm.main(filename, start_line, end_line)¶
Generate a l1c file.
LAC POD reader¶
Reader for LAC POD data.
- class pygac.lac_pod.LACPODReader(*args, **kwargs)¶
The LAC POD reader.
The scan_points attributes provides the position of the longitude and latitude points to compute relative to the full swath width.
The offset attribute tells where in the file the scanline data starts.
- pygac.lac_pod.main(filename, start_line, end_line)¶
Generate a l1c file.
LAC KLM reader¶
Reader for LAC KLM data.
- class pygac.lac_klm.LACKLMReader(*args, **kwargs)¶
The LAC KLM reader.
The offset attribute tells where in the file the scanline data starts.
- pygac.lac_klm.main(filename, start_line, end_line)¶
Generate a l1c file.
Calibration¶
Calibration coefficients and generic calibration functions
- class pygac.calibration.Calibrator(spacecraft, custom_coeffs=None, coeffs_file=None)¶
Factory class to create namedtuples holding the calibration coefficients.
- Attributes:
fields: coefficient names Calibrator: namedtuple constructor default_coeffs: dictonary containing default values for all spacecrafts
- class Calibrator(dark_count, gain_switch, s0, s1, s2, b, centroid_wavenumber, space_radiance, to_eff_blackbody_intercept, to_eff_blackbody_slope, date_of_launch, d, spacecraft, version)¶
- b¶
Alias for field number 5
- centroid_wavenumber¶
Alias for field number 6
- d¶
Alias for field number 11
- dark_count¶
Alias for field number 0
- date_of_launch¶
Alias for field number 10
- gain_switch¶
Alias for field number 1
- s0¶
Alias for field number 2
- s1¶
Alias for field number 3
- s2¶
Alias for field number 4
- space_radiance¶
Alias for field number 7
- spacecraft¶
Alias for field number 12
- to_eff_blackbody_intercept¶
Alias for field number 8
- to_eff_blackbody_slope¶
Alias for field number 9
- version¶
Alias for field number 13
- static date2float(date, decimals=5)¶
Convert date to year float.
- Argument
date (datetime.datetime) - date decimals (int or None) - rounding precision if None, do not round (default=5)
- Return
date_float (float) - date as year
- Note
rounding to the 5th decimal reproduces the original float values from patmos-x
- Example:
date2float(‘2000-07-02’) == 2000.5 because the 2nd of July was the middle day of the leap year 2000
- default_coeffs = None¶
- default_file = None¶
- default_version = None¶
- fields = ['dark_count', 'gain_switch', 's0', 's1', 's2', 'b', 'centroid_wavenumber', 'space_radiance', 'to_eff_blackbody_intercept', 'to_eff_blackbody_slope', 'date_of_launch', 'd', 'spacecraft', 'version']¶
- classmethod read_coeffs(coeffs_file)¶
Read calibration coefficients for all satellites from file.
- Args:
coeffs_file (str): path to coefficients file
- Returns:
coeffs (dict): dictionary containing coefficients for all satellites version (str): version of the coefficients (None if unknown)
- version_hashs = {'689386c822de18a07194ac7fd71652ea': {'name': 'PATMOS-x, v2017r1, with provisional coefficients for MetOp-C', 'status': CoeffStatus.PROVISIONAL}, '963af9b66268475ed500ad7b37da33c5': {'name': 'PATMOS-x, v2017r1', 'status': CoeffStatus.NOMINAL}}¶
- class pygac.calibration.CoeffStatus(value)¶
Indicates the status of calibration coefficients.
- EXPERIMENTAL = 'experimental'¶
- NOMINAL = 'nominal'¶
- PROVISIONAL = 'provisional'¶
- pygac.calibration.calibrate_solar(counts, chan, year, jday, cal, corr=1)¶
Do the solar calibration and return scaled radiance.
- Arguments:
counts (array) - raw counts for the given channels (options 1, 2[, 3A if available & active]) chan (array) - pygac internal channel index array year (int) - year jday (int) - day of year cal (namedtuple) - spacecraft specific calibration coefficients, see Calibrator
- Optional:
corr (float) - depricated - reflectance correction multiplier (default = 1)
- Returns:
r_cal (array) - scaled radiance
- Note:
This function and documentation follows the time-dependent solar calibration as described in: Heidinger, A.K., W.C. Straka III, C.C. Molling, J.T. Sullivan, and X. Wu, (2010). “Deriving an inter-sensor consistent calibration for the AVHRR solar reflectance data record.”, International Journal of Remote Sensing, 31:6493 - 6517.
- pygac.calibration.calibrate_thermal(counts, prt, ict, space, line_numbers, channel, cal)¶
Do the thermal calibration and return brightness temperatures (K).
- Arguments:
counts (array) - counts for the given channel (options: 3B (if active), 4, 5) prt (array) - counts of the Platinum Resistance Thermometers (PRT) ict (array) - counts of the In-orbit Calibration Targets (ICT) space (array) - counts of cold space line_numbers (array) - line number index channel (array) - pygac internal channel index array cal (namedtuple) - spacecraft specific calibration coefficients, see Calibrator
- Note:
This function and documentation follows steps 1 to 4 from the KLM User’s Guide (Robel, J. (2009). NOAA KLM user’s guide with NOAA-N,-P supplement. NOAA KLM Users Guide - August 2014 Revision) section 7.1.2.4 “Steps to Calibrate the AVHRR Thermal Channels”, and the smoothing approach by Trishchenko (2002). The correction method for the non-linear response of the Mercury-Cadmium-Telluride detectors used for channels 4 and 5 is based on Walton et al. (1998)