Conversion of NIPS Data Files

Some notes on the process of converting NIPS Data Files to other formats.

Blocks

NIPS data files are organized into blocks containing multiple logical records. This is done to optimize tape usage and read times (see Section 8.8 Blocked Records of Programming the ibm 360). The choice of block size might provide some insight into core memory available to the processing system.

Applications such as NIPS 360 FFS File Structuring are capable of reading blocked or unblocked file records (see Section 2.5.1.1 File Organization of Users Manual Volume 3: File Maintenance fm).

NIPSTRAN does not preserve blocking information when converting to ASCII records.

The NIPS as RDF conversion indirectly preserves this information as it tracks the offset and lenght of a logical record. This can be used to figure out where blocks.

ASCII encoded record length

Some NIPS data files (e.g. from Pacification Attitude Analysis System (PAAS) Data File, March 1970 - December 1972) have the record length encoded in the first 4 bytes as ASCII. There are also no block headers present instead the block lenght is fixed to 1004 bytes.

PyNIPS can handle this (see 80be4f0f75. This was implemented by figuring out what NIPSTRAN does. We have not been able to find documentation about this encoding yet.

Duplicate FFT

The NIPS file HES.HES71.72.NIPS has multiple File Format Tables. See HES.HES71.72.NIPS for more information.

Invalid Set IDs

It seems as if the encoded set if of some records in HES.HES71.72.NIPS are invalid. See HES.HES71.72.NIPS for more information.

Binary representation of records

Technically the translation to RDF (see NIPS as RDF) is lossless as it contains the raw binary representation of the records. All other fields are decoded from this raw binary representation:

@prefix nips: <http://computingthesocial.net/ns/nips/> .
@prefix ns1: <urn:eris:> .
@prefix qb: <http://purl.org/linked-data/cube#> .
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .

<urn:eris:BIAV2ZOXYNSVD275UXPTKSDVVJYHDNTOS5WDL46GDJP6BKFOZB3EGTP4HVJYXSVBZPTUE6MNHSBYBYRGVNG3D6UAS6UL42AK3WVTBSP4QM> a nips:DataFileRecord ;
    nips:delete_code "AA=="^^xsd:base64Binary ;
    nips:length 184 ;
    nips:offset 13910 ;
    nips:type_code "R" ;
    qb:dataSet <urn:eris:BIALQWXK3UQZVE3ZN3KH33JZUVFSFVBZ5CJSWNLUEUHGZFPQIUD65IGSZAKOKF6LYNFZGZWUMQZDGGNRU3XTGCR2H6LEGL25D5JFQJPOIQ>,
        <urn:eris:BIAMKEQTHMVHNDZMG3M4VKBB4VHIFVLPDWLSZ5Y4I5CASOXNOTGJ4AGMNNGVDMO6O5AUQF3ZU3XIY5NJCPH2K6PX3UXVNFTGBU65JARPQA> ;
    rdf:value "ALgAAADZ8fDx8PHw8fDz9vfw9QAAAAAABgAAAAAAAAAAAAFlAAAAAAAAAAAAAAAAAAAAAPHV48jkwdVgxMHkQEBAQEBAQMjByWDB1UBAQEBAQEBA8PDz6MT09vf1+PbV5cNAQEDlw0BA5fJAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEDlw9XoQEBA2UBAQEDV6Ojo8EBAQEBAQEBAQEBAQEBAQEBAQEDw9eXlQEBAQEBAAAAAALQAAA==