Records and the File Format Table

NIPS 360 FFS data is stored as a data set which is a collection of logical records.

This module provides functionality for reading logical records from a binary stream/file and parsing individual fields from different types of logical records.

Logical Records

class nips.logical_record.LogicalRecord(offset: int, length: int, os_control: bytes, delete_code: bytes, type: chr, raw: bytes)

Logical Record

Data stored in NIPS 360 FFS data files are stored in variable length logical records. This class is a Python representation of logical records.

See also Appendix A.1 of the NIPS 360 FFS User’s Manual Volume 1 - Introduction to File Concepts.

offset

Position of record in input file/stream. This is useful for debugging and finding the location of records in the original file.

Type:

int

length

Length of record.

Type:

int

os_control

Two bytes reserved for OS usage (OS Control bytes).

Type:

bytes

delete_code

Deletion Code Field (one byte).

Type:

bytes

type

Type of the record in ASCII (decoded from EBCDIC).

Type:

chr

raw

Binary representation of the entire record.

Type:

bytes

nips.logical_record.read(io, fixed_physical_record_length=None, ascii_record_length=False)

Read logical records from a binary stream or file.

This function reads all logical records from all physical records in the provided input.

Logical records are blocked into physical records (also known as blocks) that contain multiple logical records. The length of physical records is encoded in the first four bytes of the physical record.

In some cases the first four bytes indicating the length of the physical record is not present, instead blocks have a fixed size. This can be set with fixed_physical_record_length.

In some cases the logical record lenght is encoded as ASCII digits. The argument ascii_record_length can be set to read logical record lenght as ASCII digits.

The function will check if the first four characters are valid ASCII digits. If this is the case, ascii_record_length will be automatically set to True and fixed_physical_record_length to 1004.

Parameters:

  • io (io.RawIOBase) – File from where to read Logical Records. Must be seekable.

  • fixed_physical_record_length (None | int) – If specified

  • ascii_record_length (bool) –

Yields:

LogicalRecord – Logical records contained in the input stream.

File Format Table

class nips.fft.FileFormatTable

Bases: object

NIPS File Format Table

The File Format Table holds the description of the format of the data set.

classification_record

Classification Record.

Type:

ClassificationRecord)

data_file_control_record

Data File Control Record.

Type:

DataFileControlRecord

element_format_records

Holds the Element Format Records for the Fixed Set and any Periodic Sets.

Type:

list(list(ElementFormatRecords)))

class nips.fft.ClassificationRecord(logical_record)

Bases: LogicalRecord

NIPS 360 FFS Classification Record

Carries user-defined classification label

class nips.fft.UserDataFileRecordStructure(length: int, binary_words: int, binary_block_position: int)
class nips.fft.DataFileControlRecord(logical_record)

Bases: LogicalRecord

Data File Control Record

Holds necessary information on the organization and format of the element format records.

record_control_group_position
Type:

int

record_control_group_length
Type:

int

set_id_position
Type:

int

set_id_length
Type:

int

subset_control_group_position
Type:

int

subset_control_group_length
Type:

int

number_of_periodic_sets
Type:

int

element_format_record_significant_data_position
Type:

int

fixed_set_structure
Type:

UserDataFileRecordStructure

periodic_set_structure
Type:

list[UserDataFileRecordStructure]

Element Format Record

class nips.fft.ElementFormatRecord(logical_record, data_file_control_record)

Bases: LogicalRecord

Element Format Record

element_name

Element name.

Type:

str

element_set_identification

Element set identification that is set to 0 for the Fixed Set and the Periodic Set identifier for Periodic Sets.

Type:

int

element_type_identification

Element type identification byte. This contains various flags that are decoded to attributes such as group.

Type:

int

group

True if the element format record defines a group. Decoded from the element type identification byte.

Type:

bool

control

True if field or group is used for record or subset control. Decoded from the element type identification byte.

Type:

bool

system

True if element format record defines a system-generated field/group. Decoded from the element type identification byte.

Type:

bool

restricted

Field/group may not be used by the analyst. Decoded from the element type identifcation byte.

Type:

bool

fixed_length_field

True if element format record defines a fixed length field. Decoded from the element type identification byte.

Type:

bool

variable_length_field

True if element format record defines a variable length field. Decoded from the element type identiication byte.

Type:

bool

variable_set_field

True if element format record defines a variable set field. Decoded from the element type identification byte.

Type:

bool

element_location

Location of Element in Data File Record.

Type:

int

element_length

Length of Element.

Type:

int

element_mode_specification

Data Value Mode specification.

Type:

str

input_subroutine_conversion_name
Type:

str

output_subroutine_conversion_name
Type:

str

element_label_location
Type:

int

element_label_length
Type:

int

element_label
Type:

str

edit_mask_location
Type:

int

edit_mask_length
Type:

int

size_on_output
Type:

int

field_names_location
Type:

int

number_of_fields_making_up_the_group
Type:

int

field_structure
Type:

list[FieldStructure]

User Data Records

class nips.data_file_record.Field(raw, element_format_record)

A field of a Data File Record holding an Element value.

raw

The physical representation of the field.

Type:

bytes

data_value_mode

The Data Value Mode of the field.

Type:

DataValueMode

element_format_record

The Element Format Record that defines this Element and field.

Type:

ElementFormatRecord

class nips.data_file_record.DataFileRecord(logical_record, data_file_control_record, element_format_records)

Bases: LogicalRecord, Mapping

Data File Record

Represents a NIPS 360 FFS Data File Record - a record that contains user data.

key

Record key.

Type:

str

record_control_group

Record Control Group.

Type:

str

subset_control_group

Subset Control Group.

Type:

str

set_id

The Set ID. For Fixed Set 0 and for Periodic Sets the Periodic Sets the Periodic Set ID.

Type:

int

fields

A mapping from element names to fields.

Type:

dict