Description of the Raw Data Format at BL4/ID2
This is a very fast written description, of what is stored in the header together with the image data. For a summary of all keywords that are currently in use with SAXS data look into the file ~boesecke/programs/work/include/SaxsDefinition.h. This C-header file contains also all transformations that are in use. The actual raw data file of the gasfilled detector, that is produced by spec is defined at the end (see also the C-program: ~boesecke/programs/work/include/GetDldData.C

Center_1 = 23 ;
Center_2 = 24 ;
DDummy = 0.1 ;
Dim_1 = 1024 ;
Dim_2 = 1024 ;
Dummy = -1 ;
Image = 1 ;
Offset_1 = 0 ;
Offset_2 = 0 ;
Psize_1 = 0.025 ;
Psize_2 = 0.026 ;
SampleDistance = 0.995386 ;
SaxsDataVersion = 1.0 ;
Title = 0 ;
WaveLength = 7.69043e-11 ;

The actual structure was implemented on the base of the ESRF data format,  which has been constructed Peter Daly. The ESRF data format does not have a very consistent definition and has been criticised for it. The programs used at BL4/SAXS are using the ESRF data format only as a way to store the data. The definition given below are independent of the format and can be implemented in other data bases later. Even the programs are not based on  the ESRF data format and can be updated to an other format, if necessary.  The data is stored in an ASCII header followed by a binary section. A basic header contains the following keywords (apart from the ESRF data format keywords which define the backbone) are:

/n{/n
...
HeaderID = EH:000001:000000:000000 ;
ByteOrder = HighByteFirst ;
DataType = FloatValue ;
Size = <length of binary data in bytes>;
...
Center_1 = 23 ;/n
Center_2 = 24 ;/n
DDummy = 0.1 ;/n
Dim_1 = 1024 ;/n
Dim_2 = 1024 ;/n
Dummy = -1 ;/n
Image = 1 ;/n
Offset_1 = 0 ;/n
Offset_2 = 0 ;/n
Psize_1 = 0.025 ;/n
Psize_2 = 0.026 ;/n
SampleDistance = 0.995386 ;/n
SaxsDataVersion = 1.0 ;/n
Title = 0 ;/n
WaveLength = 7.69043e-11 ;/n
      }/n
Exactly <Size> bytes of <binary data>

Coordinate System

The origin (0,0) is at the lower left corner. The first pixel touches with its lower left corner the origin. The uper edges of the first pixel are 1.0 in both  directions. All following pixels are defined accordingly. This definition makes the handling of binning procedures easier. 
Remark: This is like it should be, but for the moment there is still a shift by 0.5, 0.5, i.e. the center of the first pixel is (0.0,0.0)).
	Therefore: SaxsDataVersion = 1.0 ;

Description of keywords

ESRF Data Format Keywords
HeaderID, ByteOrder, DataType, Size 
These are internal keywords of the ESRF data format and will not be discussed.

The header must be padded with blanks that its size is a multiple of 512 bytes.

An ESRF data format file can contain an unlimited number of header/binary data 
blocks.

SAXS Format Keywords
The following keywords are followed by integer values:

- Image:
sequence number of the data in a single data file. The data is processed according to the number following after Image (default 1, but can be any integer number). The image number must be unique in the data file.

- SaxsDataVersion = 1.0:
Version of the SAXS data format (must be 1.0)

- Dim_1, Dim_2:
Dimension of the actual data array (column, row) 

The following keywords are followed by float values:
- Dummy:
Dummy value
- DDummy:
All value between Dummy-DDummy and Dummy+DDummy are not valid.

- Offset_1, Offset_2:
Actual start position (in pixel) the data (in pixels)
- Center_1, Center_2:
Position of the primary beam on the detector (in pixels)
- Psize_1, Psize_2:
Pixel size in meter
- SampleDistance: 
Distance between sample and detector center 
  (Center_1, Center_2).

- WaveLength:
wavelength in meter
- Title:
A single line string the describes the data.

Remark: The definition of the center as the point where the primary beam (virtually) hits the detector is only valid if the detector is perpendicular to the primary beam.

Raw Data Format Keywords

The previously defined keywords are used through all the treatment of the data file, raw data keywords are only available in the raw data file.

The raw data files contain a lot of other keywords which are defined in the following way. HM stands for Historgramming Memory, HS is an abbreviation for Histogramming Scaler.

DetectorName = X-ray Image Intensifier with FRELON CCD ;
-should be a unique name of the used detector

DetectorPosition = 0.995139 ;
+ Position of the detector on the bench in meter (inside the detector tube) 
  The sample distance can be calculated from the DetectorPosition with an
  offset. E.g. if the detector is at home (closest distance): 
  DetectorPosition = 0. The  SampleDistance of the X-ray Image Intensifier
  from the Sample is DetectorPosition + 1.xxx meter. 

HMFile = /data/opid02/inhouse/ccd220897/flatfield2208970002 ;
- full path of the raw data file (info only)

HMFrame = 1 ;
+histogramming memory frame one (number is  converted to image number) 

HMStartTime = Fri Aug 22 19:18:51 1997 ;
-starting time of the first frame (info only) 

HS32C01 = 1.09726e+07 ;
HS32C02 = 1.17135e+07 ;
HS32C03 = 0 ;
HS32C04 = 369815 ;
HS32C05 = 1.35419e+07 ;
...
HS32C15 = 7.67908e+06 ;
...
HS32C32 = 0 ;
+content of 32 channel histogramming scaler 1 to 32 (HS32C01 to HS32C32)

HS32Depth = 24 ;
+depth of the value (usally 24 bit)

HS32F01 = 7.56e+07 ;
HS32F02 = 7.27e+07 ;
HS32F03 = 3.73e+07 ;
HS32F04 = 1.1913e+09 ;
HS32F05 = 2.11339e+07 ;
HS32F06 = 6.9124e+08 ;
...
HS32F15 = 1e-06 ;
...
HS32F31 = 1 ;
HS32F32 = 1 ;
+multiplication factor for channel 1 to 32 

HS32Len = 32 ;
+number of the used channels

HS32N01 = PIN1 ;
HS32N02 = PIN2 ;
HS32N03 = PIN3 ;
HS32N04 = PIN41 ;
HS32N05 = PIN42 ;
HS32N06 = PIN5 ;
HS32N07 = PIN6 ;
HS32N08 = PIN7 ;
...
HS32N15 = time ;
...
HS32N31 = 0 ;
HS32N32 = 0 ;
+names of channel 1 to 32

HS32Z01 = 44.76 ;
HS32Z02 = 26.92 ;
...
HS32Z15 = 0 ;
...
HS32Z31 = 0 ;
HS32Z32 = 0 ;
+ zero values of the channels

HSAnode = 6 ;
+channel number in which contains the anode counts (channel 1 to 32)

HSAnodeS = 231 ;
+alternative channel that contains divided the anode counts (0 if not used)

HSI0 = 4 ;
+channel number of the I0 monitor

HSI0S = 3 ;
+alternative channel of the I0 monitor that contains divided counts (0 = unused)

HSI1 = 6 ;
+channel number of the counter for the transmitted intensity I1 

HSI1S = 8 ;
+alternative counter for I1

HSTime = 15 ;
+channel which contains the counting time 

HSTimeS = 23 ;
+alternative channel of the counting time

MachineInfo =   Ie=141.81mA,gap46=21.80mm,taper46= 0.01mm,gap26=200.00mm,taper26
= 0.00mm ;
-single line string with machine information (info only)

OpticsInfo = optics ;
-single line string with optics information (info only)

StationInfo = id2 ;
-single line string with station information (info only)

SubTitle = 3to4 ;
-single line string with sample or experiment information (info only)

DetectorInfo = 
- single line string with detector information (not used for the moment)

Example:
The time is measured in channel 15 (HSTime = 15). The value HS32Z15 is 
not used for the time:

Channel 15 has the name: time
The calibrated value of channel 15 is:
(HS32C15)*HS32F15 = 7.67908e+06*1e-06 =  7.67908 seconds

Channel 1 hast the name: PIN1
The calibrated value of channel 1 is (the value HS32Z15 is not used for the 
time):
(HS32C01-HS32Z01*time)*HS32F01 = (1.09726e+07 counts-44.76 counts/sec*7.67908 sec)*7.56e+07 photons/count =   8.29503e+14 photons 

Remark: Normalization of the data to absolute scattering intensities Together with the absolute detector response, these values are sufficient to normalize the data automatically to absolute scattering intensities. 

Spec raw data file of the gasfilled detector data
 The first line of a block contains its number of lines,  the last line contains -1 as a separator

The header filename is followed by hm (for histogramming memory file)  The scaler data are defined in the Scalar header block. The binary data are defined in the Image header block. The current version number is 2.2. Older version are checked by a string comparison. The run number is just a number between 1 and 999, which is used to distinguish subsequently taken data. One line per item is used
                General Header: 4, run number, 2.2 (version number) 
                               	and separator(-1)

                Scalar header:  Length of header (including separator),
                                scalar memory ID (2 for scalar, 3 for
                                                  histogramming memory),
                                first channel,
                                last channel,
                                first scalar,
                                last scalar,
                                two zero for padding,
                                length of scalar data in bytes,
                                separator (-1),
                Scalar data:    Length of scalar data block (including sep.),
                                scalar data ID (5 for scalar data)
                                (last scalar - first scalar) *
                                  (last channel - first channel) lines of data
                                separator (-1),

                Image header:   length of header (including separator),
                                3 (presumably signifying hist. memory ???),
                                data length in bits (16 or 32),
                                x size,
                                y size,
                                number of images,
                                two zero for padding,
                                size of each image in bytes,
                                x offset
                                y offset
                                separator (-1)

                Timing Header:  length of header (including separator),
                                "TIMING"
                                DLD_HMSTARTTIME
                                for (i=first_frameno;i<=last_frameno;i++) {
                                  DLD_HMDELTATIME
                                  }
                                separator (-1)

                Scaler Calibration Header:
                                length of header (including separator),
                                "SCALERCALIB"
                                DLD_SCALER_DEPTH
                                DLD_SCALER_I0
                                DLD_SCALER_I1
                                DLD_SCALER_ANODE
                                DLD_SCALER_TIME
                                first_scalerno
                                last_scalerno
                                for (i=first_scalerno-1;i<last_scalerno;i++) {
                                  DLD_SCALER_NAME
                                  DLD_SCALER_ZERO
                                  DLD_SCALER_CALIB
                                  }
                                DLD_SCALER_I0S
                                DLD_SCALER_I1S
                                DLD_SCALER_ANODES
                                DLD_SCALER_TIMES
                                separator (-1)

                Experiment Header:
                                length of header (including separator),
                                "EXPERIMENT"
                                DLD_CENTER_1
                                DLD_CENTER_2
                                DLD_PIXSIZE_1
                                DLD_PIXSIZE_2
                                DLD_WAVELENGTH
                                DLD_SAMPLEDISTANCE
                                DLD_TITLE
                                DLD_SUBTITLE
                                DLD_DETECTORPOSITION
                                separator (-1)
 
                Info Header:
                                length of header (including separator),
                                "INFO"
                                DLD_DETECTORTYPE
                                DLD_MACHINEINFO
                                DLD_OPTICSINFO
                                DLD_STATIONINFO
                                DLD_PROPOSALINFO
                                separator (-1)

EXAMPLE:
It is actually an CCD file but there is not a big difference.
(Only data size).


4
634
2.2
-1
10
2
1
32
1
1
0
0
128
-1
35
5
1219
865
0
2287
23015
19152
3182
3590
0
0
0
0
1021533
499
20000357
9766
0
0
0
0
0
3337
20479643
20479643
0
0
0
0
0
0
0
0
-1
12
3
16
1024
1024
1
0
0
2097152
0
0
-1
5
TIMING
Wed May  7 19:30:45 1997
0.006
-1
110
SCALERCALIB
25.585
5
7
13
15
1
32
PIN1
20.99
7.5617e+07
PIN2
1.49
72700000
PIN3
0
37300000
PIN41
106.15
4.29349e+08
PIN42
933.65
1.03513e+07
PIN5
1014.9
-65985562914.776428
PIN6
238.35
1.95772e+06
PIN7
150.2
126627372650.48988
I0
0
9670000
I02
0
1
I1
0
1
I2
0
1
anode
0
1
anode2
0
1
time
1e-06
time2
0
0.000205
thc1
0
0.002048
thc2
206.8
2009.7
vfc23
0
1
vfc24
0
1
thc5
0
1
vfc26
0
1
vfc27
0
1
thcsa
0
0.0005
 
0
1
 
0
1
 
0
1
 
0
1
 
0
1
 
0
1
 
0
1
 
0
1
10
7
14
16
-1
12
EXPERIMENT
256
256
0.000366
0.000361
9.88717e-11
0.999968
working proteins
olivier
0.160975
-1
8
INFO
X-ray Image Intensifier with FRELON CCD
  Ie= 0.00mA,gap46=199.98mm,taper46= 0.02mm,gap26=20.40mm,taper26= 0.00mm
optics
id2
stopped flow
-1


