High Data-Rate Macromolecular Crystallography

CBFlib: Crystallographic Binary File library
[more/less ...] CBFlib-0.9.5.17.tar.gz, CBFlib_bleeding_edge on githib.com

Compression Filters for HDF5 (introduced with HDF5 1.8.11) [more/less ...] HDF group instructions lz4 source bslz4 source

Eiger Filter installation instructions

Mac OSX lz4 and bslz4 installation instructions, courtesy Sascha Grimm, Dectris, Ltd. [more/less ...]

Homebrew
========
brew install hdf5 # install hdf5 library, might not be necessary
brew install lz4 # install lz4 library
git clone https://github.com/nexusformat/HDF5-External-Filter-Plugins/tree/master/LZ4 
# download hdf lz4 filter source
cd LZ4 # change directory to lz4 source
./configure # prepare the compilation
make # compile
This should produce the file ./plugins/lib/libh5lz4.dylib. Copy this file into your hdf5 plugin folder e.g. cp libh5lz4.dylib 
/usr/local/hdf5/lib/plugin/.
Now, the decompression in hdf5 should work. Please keep in mind that the directories and files might differ for your system installation and 
must be modified accordingly. It might be necessary to supply the h5 and lz4 installation path to the configure command ( e.g. ../configure 
--with-hdf5=/temp/hdf5 --with-lz4lib=/temp/lz4).


Fink
===
I did that by emulating the source instructions below in fink.


For source junkies
==============

download the latest versions of gcc, mpc, mpfr, gmp and isl
http://gcc.gnu.org/mirrors.html
http://www.multiprecision.org/index.php?prog=mpc&page=download
http://www.mpfr.org/mpfr-current/#download
http://gmplib.org/
ftp://gcc.gnu.org/pub/gcc/infrastructure/

compile and install gmp, mofr, mpc and isl successively

to achieve that
cd gmp* (or whatever package)
mkdir build
cd build
../configure [with prefix, see list below]
make -j 4
sudo make install

configure flags are
gmp:
--prefix=/usr/local/gcc-5.3.0 --with-gmp=/usr/local/gcc-5.3.0
mpfr:
--prefix=/usr/local/gcc-5.3.0 --with-gmp=/usr/local/gcc-5.3.0
mpc:
--prefix=/usr/local/gcc-5.3.0 --with-gmp=/usr/local/gcc-5.3.0 --with-mpfr=/usr/local/gcc-5.3.0
isl:
--prefix=/usr/local/gcc-5.3.0 --with-gmp-prefix=/usr/local/gcc-5.3.0

then install gcc:
cd gcc*
mkdir build && cd build
../configure --prefix=/usr/local/gcc-5.3.0 --enable-checking=release --with-gmp=/usr/local/gcc-5.3.0 --with-mpfr=/usr/local/gcc-5.3.0 
--with-mpc=/usr/local/gcc-5.3.0 --enable-languages=c,c++,fortran --with-isl=/usr/local/gcc-5.3.0 --program-suffix=-5.3.0
(takes 1 h)
make -j 4
sudo make install

get bitshuffle:
clone https://github.com/kiyo-masui/bitshuffle.git
cd bitshuffle

modify INCLUDE_DIRS and LIBRARY_DIRS in setup.py to point to gcc-5.3.0:
vi setup.py
modify:
if sys.platform == 'darwin':
    pass
    # putting here both macports and homebrew paths will generate
    # "ld: warning: dir not found" at the linking phase
    #INCLUDE_DIRS += ['/opt/local/include'] # macports
    #LIBRARY_DIRS += ['/opt/local/lib']     # macports
    #INCLUDE_DIRS += ['/usr/local/include'] # homebrew
    #LIBRARY_DIRS += ['/usr/local/lib']     # homebrew
elif sys.platform.startswith('freebsd'):
    pass
    #INCLUDE_DIRS += ['/usr/local/include'] # homebrew
    #LIBRARY_DIRS += ['/usr/local/lib']     # homebrew
INCLUDE_DIRS += ['/usr/local/gcc-5.3.0/include'] # gcc 5.3
LIBRARY_DIRS += ['/usr/local/gcc-5.3.0/lib']     # gcc 5.3

pip install .

get_eiger_stats.csh -- a little script to report Eiger image counts, compression filters and compression ratios. [more/less ...]

#!/bin/csh
set start=$1
if ( "${start}xx" == "--helpxx" || "${start}xx" == "-hxx" ) then
  echo "echo get_eiger_stats.csh  H.J. Bernstein, 30 Sep 2016"
  echo ""
  echo "usage:"
  echo "  get_eiger_stats.csh [start] "
  echo "    searches the directory tree from start, default '.'."
  echo "    for file names ending in '_master.h5; and uses"
  echo "    eiger2cbf and h5dump to extract the count of images"
  echo "    the data block sizes, the compression filter and "
  echo "    the compression ratios.  The list is sorted by image"
  echo "    count."
  echo ""
  echo "  Assumes eiger2cbf and h5dump are in PATH "
  echo "  Assumes HDF5_PLUGIN_PATH and LD_LIBRARY_PATH are set "
  echo "  appropriately "
  exit
else if ( "${start}xx" == "xx" ) then
  set start = "."
endif
set flist=`find $1 -name \*_master.h5 | paste -s`
#echo $flist
foreach file ($flist)
  if ( -r $file ) then
    set images=`eiger2cbf $file |& paste -s| sed '$s/^.*details.//' |tail`
    echo -n $images 'images in:' $file ':'   >> /tmp/eigerstats$$
    h5dump -H -p $file| grep -e COMPRESSION -e "filter;" | cut -d ';' -f 1 \
    | sed '1,$s/^ *//'| sed '1,$s/COMMENT HDF5//' | sed '1,$s/ COMPRESSION//'\
    |paste -sd ' \n' |paste -s >> /tmp/eigerstats$$
  endif
end
cat /tmp/eigerstats$$ | sort -n | sed '1,$s/SIZE/\n        SIZE/g'

Data Set Deposition

Integrated Resource for Reproducibility in Macromolecular Crystallography, courtesy of Wladek Minor, University of Virginia
[more/less ...]
If you have a dataset that would be useful in understanding HDRMX issues, please deposit it with the Integrated Resource for Reproducibility in Macromolecular Crystallography (IRRMC, http://proteindiffraction.org) and send a note about it to the hdrmx-bb list.
proteindiffraction.org

Data Processing Software Packages for HDRMX Data

DIALS: Diffraction Integration for Advanced Light Sources
The DIALS project is a collaborative endeavour to develop new diffraction integration software to meet the data analysis requirements presented by these recent advances.
[more/less ...]
X-ray crystallography for structural biology has benefited greatly from a number of advances in recent years including high performance pixel array detectors, new beamlines capable of delivering micron and sub-micron focus and new light sources such as X-FELs. The DIALS project is a collaborative endeavour to develop new diffraction integration software to meet the data analysis requirements presented by these recent advances. There are three end goals: to develop an extensible framework for the development of algorithms to analyse X-ray diffraction data; the implementation of algorithms within this framework and finally a set of user facing tools using these algorithms to allow integration of data from diffraction experiments on synchrotron and free electron sources.
- DIALS West: dials.lbl.gov
- Dials East: dials.diamond.ac.uk
DIALS apache webserver for spotfinding using the eiger
[more/less ...]
Follow these instructions to install an apache webserver configured to run the distl and dials spotfinders. For more information on the distl spotfinder server, see http://cci.lbl.gov/labelit/html/client_server.html. The version described here allows the beamline operator to use either distl or the dials spotfinder to assess the quality of single images, typically during a raster scan of a crystal cluster. Use of the apache server allows automatic load balancing and scaling of many parallel requests, typical for a high throughput environment.
distl/dials apache server
XDS: XDS is a program package written for the "reduction" of 2-dimensional data images ("frames") obtained from crystals irradiated with monochromatic X-rays
[more/less ...]
The package mainly consists of the programs XDS (which processes a single dataset), XSCALE (for scaling of several datasets) , and XDSCONV (for conversion of a reflection output file to other formats). Additional auxiliary programs are cellparm, 2cbf and merge2cbf. XDS-viewer, for visualizing frames, can be downloaded separately.
The author of the program package is Wolfgang Kabsch (Wolfgang dot Kabsch at mpimf-heidelberg dot mpg dot de). http://www.mpimf-heidelberg.mpg.de/~kabsch/xds/html_doc/downloading.html (if the download from that site fails, there's a copy at ftp://turn5.biologie.uni-konstanz.de/xds ).
Documentation is at http://www.mpimf-heidelberg.mpg.de/~kabsch/xds ...
XDSwiki

Eiger Data Tags

In this space we will be providing information on the data and metadata tags used in handling Eiger images. There is a gradually evolving set of documents available on the Dectris website www.dectris.com. Dectris has kindly given permission for us to consolidate and summarize some of that information here for conventience. This is a work in progress. [more/less ...]

This is a table of the Dectris Software Releases of " EIGER Detector HDF5 NeXus Format" through release 1.3.0. Partial information from the NeXus NXmx application definition has been added, but also see the changes in Eiger Data Tag Proposed "Gold Standard", below, for proposed changes. You may sort on a column by clicking on the heading. Please report comments and corrections to the hdrmx-bb list.

Version	NeXus Path	Name	Type	Units	Description	Full CBF	mini CBF
< 1.2.0	/(entry):NXentry/(instrument):NXinstrument/(beam):NXbeam/wavelength	wavelength	NX_FLOAT	A	wavelength of the beam in the case of monochromatic beam	_diffrn_radiation_wavelength.wavelength 0.7085	Wavelength 0.7085 A
< 1.2.0	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/geometry/orientation	orientation	NX_FLOAT		To be described by Dectris
< 1.2.0	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/geometry/translation	translation	NX_FLOAT		To be described by Dectris
≥ 1.2.0	/(entry):NXentry/(instrument):NXinstrument/(beam):NXbeam/incident_wavelength	incident_wavelength	NX_FLOAT	A	Wavelength of the beam in the case of monochromatic beam	_diffrn_radiation_wavelength.wavelength	Wavelength
	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/beam_center_x	beam_center_x	NX_FLOAT		Beam center in x in pixels. This is necessary for efficiency correction. Default value: beam_center_x = 0.5* x_pixels_in_detector	_diffrn_detector_element.reference_center_fast	Beam_x
	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/beam_center_y	beam_center_y	NX_FLOAT		Beam center in y in pixels. This is necessary for efficiency correction. Default value: beam_center_y = 0.5* y_pixels_in_detector	_diffrn_detector_element.reference_center_slow	Beam_y
	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/bit_depth_readout	bit_depth_readout	NX_INT		Bit depth of the readout file	_array_structure.encoding_type
	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/count_time	count_time	NX_FLOAT	s	time per image, time during which detector counts X-rays	_diffrn_scan.integration_time 0.0012887	Exposure_time 0.0012887 s
	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/countrate_correction_applied	countrate_correction_applied	NX_INT		Application of rate correction
	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/description	description	NX_CHAR		name/manufacturer/model/etc. information	_diffrn_detector.type "Dectris Eiger 16M"	Detector "Dectris Eiger 16M"
	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/detectorSpecific/auto_summation	auto_summation	NX_INT		Auto-summation was used, i.e. images are composed of summed frames
	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/detectorSpecific/calibration_type	calibration_type	NX_CHAR		Dectris specific
	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/detectorSpecific/countrate_correction_bunch_mode	countrate_correction_bunch_mode	NX_CHAR		Bunch mode of the count rate correction. Default value: continuous
	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/detectorSpecific/data_collection_date	data_collection_date	NX_CHAR		Date and time of data collection in ISO 8601	_diffrn_scan_frame.date "2015-11-10T17:17:02.057611"	2015-11-10T17:17:02.057611
	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/detectorSpecific/detectorModule_000/countrate_correction_count_cutoff	countrate_correction_count_cutoff	NX_UINT		Maximum number of counts possible due to count rate limitation after count rate correction	_array_intensities.overload 1.4861e+06	Count_cutoff 1.4861e+06
	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/detectorSpecific/detectorModule_000/countrate_correction_cutoff	countrate_correction_cutoff	NX_FLOAT		To be described by Dectris
	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/detectorSpecific/detectorModule_000/countrate_correction_lookup_table	countrate_correction_lookup_table	NX_UINT		Lookup table used for count rate correction
	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/detectorSpecific/detectorModule_000/countrate_correction_table	countrate_correction_table	NX_FLOAT		To be described by Dectris
	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/detectorSpecific/detectorModule_000/data_origin	data_origin	NX_UINT		Position of module in image coordinates
	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/detectorSpecific/detectorModule_000/data_rotation	data_rotation	NX_UINT		To be described by Dectris
	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/detectorSpecific/detectorModule_000/data_size	data_size	NX_UINT		Size of module in number of pixels
	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/detectorSpecific/detectorModule_000/flatfield	flatfield	NX_FLOAT		Dectris specific	_diffrn_detector.details	Flat_field: FF_p2m0109_E15000_T7500_vrf_m0p30.tif
	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/detectorSpecific/detectorModule_000/hv	hv	NX_FLOAT		Dectris specific
	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/detectorSpecific/detectorModule_000/pixel_mask	pixel_mask	NX_UINT		Dectris specific
	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/detectorSpecific/detectorModule_000/rxbias_0	rxbias_0	NX_UINT		Dectris specific
	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/detectorSpecific/detectorModule_000/rxbias_1	rxbias_1	NX_UINT		Dectris specific
	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/detectorSpecific/detectorModule_000/rxbias_2	rxbias_2	NX_UINT		Dectris specific
	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/detectorSpecific/detectorModule_000/rxbias_3	rxbias_3	NX_UINT		Dectris specific
	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/detectorSpecific/detectorModule_000/starven	starven	NX_UINT		Dectris specific
	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/detectorSpecific/detectorModule_000/starvep	starvep	NX_UINT		Dectris specific
≥ 1.2.0	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/detectorSpecific/detectorModule_000/stream	stream	NX_UINT		Dectris specific
	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/detectorSpecific/detectorModule_000/tgstv	tgstv	NX_UINT		Dectris specific
	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/detectorSpecific/detectorModule_000/trimbit	trimbit	NX_UINT		Dectris specific	_diffrn_detector.details	Trim_file: p2m0109_E15000_T7500_vrf_m0p30.bin
	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/detectorSpecific/detectorModule_000/vcal	vcal	NX_UINT		Dectris specific
	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/detectorSpecific/detectorModule_000/vccn	vccn	NX_UINT		Dectris specific
	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/detectorSpecific/detectorModule_000/vccp	vccp	NX_UINT		Dectris specific
	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/detectorSpecific/detectorModule_000/vcmp_0	vcmp_0	NX_UINT		Dectris specific
	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/detectorSpecific/detectorModule_000/vcmp_1	vcmp_1	NX_UINT		Dectris specific
	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/detectorSpecific/detectorModule_000/vcmp_2	vcmp_2	NX_UINT		Dectris specific
	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/detectorSpecific/detectorModule_000/vcmp_3	vcmp_3	NX_UINT		Dectris specific
	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/detectorSpecific/detectorModule_000/vdda	vdda	NX_UINT		Dectris specific
	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/detectorSpecific/detectorModule_000/vddc	vddc	NX_UINT		Dectris specific
	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/detectorSpecific/detectorModule_000/vddd	vddd	NX_UINT		Dectris specific
	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/detectorSpecific/detectorModule_000/vdddio	vdddio	NX_UINT		Dectris specific
	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/detectorSpecific/detectorModule_000/vish	vish	NX_UINT		Dectris specific
	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/detectorSpecific/detectorModule_000/vrfp	vrfp	NX_UINT		Dectris specific
	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/detectorSpecific/detectorModule_000/vrfs	vrfs	NX_UINT		Dectris specific
	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/detectorSpecific/detectorModule_000/vtrm	vtrm	NX_UINT		Dectris specific
	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/detectorSpecific/detector_readout_period	detector_readout_period	NX_FLOAT	s	To be described by Dectris
	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/detectorSpecific/element	element	NX_CHAR		To be described by Dectris
	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/detectorSpecific/flatfield	flatfield	NX_FLOAT		Flatfield correction data
	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/detectorSpecific/frame_count_time	frame_count_time	NX_FLOAT	s	To be described by Dectris See /(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/count_time	_diffrn_scan_frame.integration_time 0.0012887	Exposure_time 0.0012887 s
	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/detectorSpecific/frame_period	frame_period	NX_FLOAT	s	To be described by Dectris See /(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/frame_time	_diffrn_scan_frame.integration_period 0.0012987	Exposure_period 0.0012987 s
	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/detectorSpecific/frames_sum	Nframes_sum	NX_UINT		To be described by Dectris
	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/detectorSpecific/nimages	Nimages	NX_UINT		Number of images
	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/detectorSpecific/nsequences	Nsequences	NX_UINT		To be described by Dectris
	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/detectorSpecific/ntrigger	Ntrigger	NX_UINT		To be described by Dectris
	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/detectorSpecific/number_of_excluded_pixels	Number_of_excluded_pixels	NX_UINT		Number of excluded (unreliable) pixels	_diffrn_detector.details 293	N_excluded_pixels = 293
	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/detectorSpecific/photon_energy	photon_energy	NX_FLOAT	eV	Photon energy, default value: photon_energy = 2* threshold_energy
	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/detectorSpecific/pixel_mask	pixel_mask	NX_UINT		Contains a bit field for each pixel to signal dead, blind or high or otherwise unwanted or undesirable pixels	_diffrn_detector.details	Excluded_pixels: badpix_mask.tif
	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/detectorSpecific/software_version	software_version	NX_CHAR		Software version used for data acquisition
	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/detectorSpecific/summation_nimages	summation_nimages	NX_UINT		To be described by Dectris
	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/detectorSpecific/trigger_mode	trigger_mode	NX_CHAR		Trigger mode, e.g. externally_triggered, software control
	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/detectorSpecific/x_pixels_in_detector	x_pixels_in_detector	NX_UINT		Number of pixels in detector in x direction	_array_structure_list.dimension
	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/detectorSpecific/y_pixels_in_detector	y_pixels_in_detector	NX_UINT		Number of pixels in detector in y direction	_array_structure_list.dimension
	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/detector_distance	detector_distance	NX_FLOAT	m	Distance to detector	_diffrn_measurement.sample_detector_distance 300	Detector_distance 0.300 m
	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/detector_number	detector_number	NX_CHAR		Detector serial number
	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/detector_readout_time	detector_readout_time	NX_FLOAT	s	Readout time of the detector for non-continuous readout (PILATUS) or dead time of continuous readout (EIGER). This defines the minimum time after which the detector can count again and acquire the next frame.	_diffrn_radiation.dtime .000003	Tau = .000003 s
	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/efficiency_correction_applied	efficiency_correction_applied	NX_INT		Application of sensor efficiency correction
	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/flatfield_correction_applied	flatfield_correction_applied	NX_INT		Application of flatfield correction
	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/frame_time	frame_time	NX_FLOAT	s	Time interval between start of image acquisitions. This defines the speed of data collection and is inversely proportional to frame rate, the frequency of image acquisition. This parameter is recorded as "exposure_period" for PILATUS.	_diffrn_scan_frame.integration_period 0.0012987	Exposure_period 0.0012987 s
≥ 1.2.0	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/geometry/orientation/value	value	NX_FLOAT		To be described by Dectris
≥ 1.2.0	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/geometry/translation/distances	distances	NX_FLOAT		To be described by Dectris
	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/pixel_mask_applied	pixel_mask_applied	NX_INT		Application of pixel mask to data
	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/sensor_material	sensor_material	NX_CHAR		Sensor material, e.g. Si
	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/sensor_thickness	sensor_thickness	NX_FLOAT	m	Thickness of the sensor	_diffrn_detector.layer_thickness 0.320	Silicon sensor, thickness 0.000320 m
	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/threshold_energy	threshold_energy	NX_FLOAT	eV	Threshold energy of the detector	_diffrn_detector.threshold 0	Threshold_setting 0 eV/font>
	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/virtual_pixel_correction_applied	virtual_pixel_correction_applied	NX_INT		Application of virtual pixel correction
	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/x_pixel_size	x_pixel_size	NX_FLOAT	m	Pixel size in x	loop_ _array_element_size.index _array_element_size.size 1 .000075 2 .000075	Pixel_size 75e-6 m x 75e-6 m
	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/y_pixel_size	y_pixel_size	NX_FLOAT	m	Pixel size in y	loop_ _array_element_size.index _array_element_size.size 1 .000075 2 .000075	Pixel_size 75e-6 m x 75e-6 m
NXmx 1.4	/(entry):NXentry/title	title	NX_CHAR		optional title
NXmx 1.4	/(entry):NXentry/start_time	start_time	NX_DATE_TIME		optional collection start time See /(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/detectorSpecific/data_collection_date	_diffrn_scan.date_start
NXmx 1.4	/(entry):NXentry/end_time	end_time	NX_DATE_TIME		optional collection end time See /(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/detectorSpecific/data_collection_date	_diffrn_scan.date_end
NXmx 1.4	/(entry):NXentry/definition	definition	NX_CHAR		"NXmx"
NXmx 1.4	/(entry):NXentry/(instrument):NXinstrument/(attenuator):NXattenuator/attenuator_transmission	attenuator_transmission	NX_NUMBER		unitless attenutator transmission
NXmx 1.4	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/depends_on	depends_on	NX_CHAR		the axis at the end of nested axis chain on which the detector is mounted	loop_ _diffrn_detector_axis.detector_id _diffrn_detector_axis.axis_id (see imgCIF dictionary)
NXmx 1.4	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/depends_on	depends_on	NX_CHAR		the axis at the end of nested axis chain on which the detector is mounted	loop_ _diffrn_detector_axis.detector_id _diffrn_detector_axis.axis_id (see imgCIF dictionary)
NXmx 1.4	/(entry):NXentry/(data):NXdata/data_nnnnnn	data_nnnnnn	NX_NUMBER		the image data	loop_ _array_data.array_id _array_data.binary_id _array_data.data (see imgCIF dictionary)
NXmx 1.4	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/data_nnnnnn	data_nnnnnn	NX_NUMBER		a link to the image data in (data):NXdata	loop_ _array_data.array_id _array_data.binary_id _array_data.data (see imgCIF dictionary)
NXmx 1.4	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/description	name/manufacturer/model/etc. information	NX_CHAR			_diffrn_detector.type "Dectris Eiger 16M"	Detector "Dectris Eiger 16M"
NXmx 1.4	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/distance	Distance from the sample to the beam center. This value is a guidance only, the proper geometry can be found following the depends_on axis chain.	NX_FLOAT			_diffrn_measurement.sample_detector_distance 300	Detector_distance 0.300 m
NXmx 1.4	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/dead_time	dead_time	NX_FLOAT	s	See /(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/detector_readout_time	_diffrn_detector.dtime .000003	Tau = .000003
NXmx 1.4	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/saturation_value	saturation_value	NX_NUMBER		The value at which the detector goes into saturation. Data above this value is known to be invalid.	_array_intensities.overload 1.4861e+06	Count_cutoff 1.4861e+06
NXmx 1.4	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/gain_setting	gain_setting	NX_CHAR		The gain setting of the detector. This influences background.	_array_intensities.details "Gain_setting: low gain"	Gain_setting: low gain
NXmx 1.4	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/threshold_energy	threshold_energy	NX_FLOAT	eV	Single photon counter detectors can be adjusted for a certain energy range in which they work optimally. This is the energy setting for this.	_diffrn_detector.threshold 0	Threshold_setting 0 eV/font>
NXmx 1.4	/(entry):NXentry/(sample):NXsample/(beam):NXbeam/incident_wavelength	incident_wavelength	NX_NUMBER	A	wavelength of the beam in the case of monochromatic beam	_diffrn_radiation_wavelength.wavelength 0.7085	Wavelength 0.7085 A
NXmx 1.4	/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/beam_center_x	beam_center_x	NX_FLOAT	mm	This is the x position where the direct beam would hit the detector. This is a length, not a pixel position, and can be outside of the actual detector.	_diffrn_detector_element.reference_center_fast 165	Beam_center_x = 165 mm
NXmx 1.4	/(entry):/(entry):NXentry/(instrument):NXinstrument/(detector):NXdetector/beam_center_y	beam_center_y	NX_FLOAT	mm	This is the y position where the direct beam would hit the detector. This is a length, not a pixel position, and can be outside of the actual detector.	_diffrn_detector_element.reference_center_slow 153	Beam_center_y = 153 mm
NXmx 1.4	/(entry):NXentry/(sample):NXsample/(beam):NXbeam/flux	flux	NX_NUMBER	ph/s	flux incident on beam plane area	_diffrn_radiation.flux 0	Flux 0 ph/s
NXmx 1.4	/(entry):NXentry/(sample):NXsample/(beam):NXbeam/incident_polarisation_stokes[np, 4]	"Watts/meter^2"	NX_NUMBER		Stokes polarization components in the order I, Q U, V	_diffrn_radiation.polarizn_Stokes_I SVECI _diffrn_radiation.polarizn_Stokes_Q SVECQ _diffrn_radiation.polarizn_Stokes_U SVECU _diffrn_radiation.polarizn_Stokes_V SVECV
NXmx 1.4	/(entry):NXentry/(sample):NXsample/(transformations):NXtransformations				Suggested location for sample goniometer or other axes (transformations)
Pilatus						_diffrn_detector.energy_range_low 0 _diffrn_detector.energy_range_high 0	Energy_range (0, 0) eV
Pilatus						_diffrn_measurement.ample_detector_voffset -0.02	Detector_Voffset -0.00002 m
Pilatus						_diffrn_radiation.filter_transmission 1.0	Filter_transmission 1.0000
Pilatus						loop_ _diffrn_scan_frame_axis.axis_id _diffrn_scan_frame_axis.angle GONIOMETER_SCAN 0.090	Start_angle 0.0900 deg.
Pilatus						loop_ _diffrn_scan_frame_axis.axis_id _diffrn_scan_frame_axis.axis_description GONIOMETER_SCAN "X, CW"	Oscillation_axis X, CW
Pilatus						loop_ _diffrn_scan_frame_axis.axis_id _diffrn_scan_frame_axis.angle_increment GONIOMETER_SCAN 0.01	Angle_increment 0.0100 deg.
Pilatus						loop_ _diffrn_scan_frame_axis.axis_id _diffrn_scan_frame_axis.angle DETECTOR_TWOTHETA 0	Detector_2theta 0.0000 deg.
Pilatus						loop_ _diffrn_scan_frame_axis.axis_id _diffrn_scan_frame_axis.angle GONIOMETER_ALPHA 0	Alpha 0.0000 deg.
Pilatus						loop_ _diffrn_scan_frame_axis.axis_id _diffrn_scan_frame_axis.angle GONIOMETER_OMEGA 0	Omega 0.0000 deg.
Pilatus						loop_ _diffrn_scan_frame_axis.axis_id _diffrn_scan_frame_axis.angle GONIOMETER_KAPPA 0	Kappa 0.0000 deg.
Pilatus						loop_ _diffrn_scan_frame_axis.axis_id _diffrn_scan_frame_axis.angle GONIOMETER_CHI 0	Chi 0.0000 deg.
Pilatus						loop_ _diffrn_scan_frame_axis.axis_id _diffrn_scan_frame_axis.angle_increment GONIOMETER_CHI 0.01	Phi_increment 0.0100 deg.
Pilatus						loop_ _diffrn_scan_frame_axis.axis_id _diffrn_scan_frame_axis.angle GONIOMETER_pHI 0	Phi 0.0000 deg.
Pilatus						loop_ _diffrn_scan_frame_axis.axis_id _diffrn_scan_frame_axis.angle_increment GONIOMETER_PHI 0.01	Phi_increment 0.0100 deg.
Pilatus						_diffrn_scan_frame.oscillations 1	N_oscillations 1

Eiger Data Tag Proposed "Gold Standard"

A new Eiger Data Tag Proposed "Gold Standard" has been proposed and discussed at several recent HDRMX meetings over the past year, and is expected to be put forward for formal adooption at the HDRMX meeting at Diamond Light Source, 6-7 November 2019. [more/less ...]

The proposed revised NXmx application definition is available in http://github.com/HDRMX/definitions. The necessary proposed matching changes to Mark Koennecke's cnxvalidate program are available in http://github.com/HDRMX/cnxvalidate. A sample gold standard Eiger X2 dataset is available at https://zenodo.org/record/3484187

Eiger FileWriter Interface

A generic frame-reading API for a runtime-loadable library, courtesy of K. Diederichs², V. Boccone¹, A. Förster¹, M. Mathes¹, V. Pilipp¹, ¹Dectris Ltd. (Baden, Switzerland) ²Universität Konstanz (Germany)
[more ...], https://github.com/dectris/neggia
eiger2cbf: EIGER HDF5 to miniCBF converter by Takanori Nakane
[more/less ...]
eiger2cbf is a simple program that converts diffraction images from EIGER in the HDF5 format to the miniCBF format. This program is intended to be used with MOSFLM. You do not need this program to process images with DIALS; DIALS can process HDF5 images directly (DIALS 1.2.0, which comes with CCP4 7.0.13).
https://github.com/biochem-fan/eiger2cbf,
fcb_eiger_reader: Code to read HDF5 Eiger images in Fortran
[more/less ...]
Much of FCB_EIGER_READER is derived from eiger2cbf Copyright 2016 Takanori Nakane. To install this kit, you first need to have installed a complete HDF5 kit with fortran support, and to have installed the bitshuffle and lz4 filters into a static library
fcb_eiger_reader github repository, fcb_eiger_reader20160719.tar.gz
h5nxmxmv: HDF5 NeXus nxMX File Renaming Utility, courtesy of Ezequiel Panepucci, Simon Gregor Ebner SLS Paul Scherrer Institut
[more/less ...]
This script is meant to make it easier to rename an HDF5 file conforming to the NeXus/nxMX application definition.
This script will:
1. create a new hard-link from source -> destination datafile
2. update all of the External Links contained in the master file to point to the renamed destination datafile
3. remove old source datafile hardlink from the filesystem
4. rename master file to its new filename
This script will NOT:
- handle overwriting files - it will refuse to proceed if it detects the possibility of overwriting files
  TIP check your folder thoroughly!!
- rename across different filesystems
  TIP rename locally and copy it afterwards
CAVEAT: Your data is precious right? Test this script in your OWN environment before using it.
https://github.com/paulscherrerinstitute/h5nxmxmv,

Eiger Streaming Interface

Spot-finding from the streaming interface, courtesy of Keitaro Yamashita, SPring-8
[more/less ...]
Eiger X 9M has been installed on BL32XU (our beamline) and working. We are using streaming mode for spot-finding and monitor mode for checking latest image.
For spot-finding, we are using modified version of Cheetah, which is available on https://github.com/keitaroyam/cheetah (eiger-zmq branch). Unfortunately it is not stand-alone and needs a result receiver program via Zmq. The program is not available online but I will make it open-source as soon as possible.
modified version of Cheetah (eiger-zmq branch)
[more/less ...]
We modify master.h5 to apply byteshuffle+gzip compression and add metadata about our non-conventional goniometer rotation direction (so-called reverse-phi), which is read by DIALS and generate_XDS.INP.
By compression, the master file size is reduced to ~51 MB (from original size of ~200 MB) through h5repack -f SHUF -f GZIP=1. The metadata is modified as advised by Andreas. /entry/sample/goniometer/omega is copied to /entry/sample/transformations/omega and attributes of units, transformation_type, vector, offset, and depends_on are added. The 'vector' attribute is (-1, 0, 0) for "reverse-phi" setting. You can see the code for this:
modifying master

Streamed Data From Eiger 16M, courtesy Ezequiel Panepucci, SLS Paul Scherrer Institut
[more/less ...]

Streamed Data from an EigerX 16M
================================

Header Messages - one per series
--------------------------------
* 000000_000.raw
* 000000_001.raw
```javascript
    {
        "auto_summation":1,
        "beam_center_x":1908.5810546875,
        "beam_center_y":2188.41552734375,
        "bit_depth_readout":12,
        "calibration_type":"standard",
        "count_time":0.016656000167131424,
        "countrate_correction_applied":1,
        "countrate_correction_bunch_mode":"continuous",
        "data_collection_date":"2016-06-02T10:55:45.424234",
        "description":"Dectris Eiger 16M",
        "detector_distance":0.15001900494098663,
        "detector_number":"E-32-0100",
        "detector_orientation":[-1,0,0,0,-1,0],
        "detector_readout_period":0.0075177969411015511,
        "detector_readout_time":9.9999997473787516e-06,
        "detector_translation":[0.14314357936382294,0.16413117945194244,-0.15001900494098663],
        "efficiency_correction_applied":0,
        "element":"",
        "flatfield_correction_applied":1,
        "frame_count_time":0.0012720000231638551,
        "frame_period":0.0012819999828934669,
        "frame_time":0.016666000708937645,
        "nframes_sum":13,
        "nimages":25,
        "ntrigger":23,
        "number_of_excluded_pixels":1207641,
        "photon_energy":12398.4228515625,
        "pixel_mask_applied":1,
        "sensor_material":"Si",
        "sensor_thickness":0.00031999999191612005,
        "software_version":"1.5.2",
        "summation_nimages":1,
        "threshold_energy":6199.21142578125,
        "trigger_mode":"exts",
        "virtual_pixel_correction_applied":1,
        "wavelength":0.99999970197677612,
        "x_pixel_size":7.5000003562308848e-05,
        "x_pixels_in_detector":4150,
        "y_pixel_size":7.5000003562308848e-05,
        "y_pixels_in_detector":4371
    }
```

Image Messages - one per frame
------------------------------

* *_000.raw : htype: dimage-1.0
* *_001.raw : htype: dimage_d-1.0
* *_002.raw : bs-lz4 image blob
* *_000.raw : htype: dconfig-1.0
* *_000.raw : image_appendix
```javascript
    {
        "high_resolution_limit": null,
        "distance": 0.150019,
        "two_theta": 0.0,
        "uid": 10003,
        "gid": 10003,
        "spotfinder": true,
        "low_resolution_limit": 50.0,
        "minimum_signal_height": 2.5,
        "beam_position_y": 2188.4156164266183,
        "beam_position_x": 1908.5810367547858,
        "minimum_spot_area": 5,
        "pixel_size": 0.075,
        "tracking_id": "88e62fe9-6a1d-4b66-aea0-758fba397c48",
        "beamline": "x06sa",
        "wavelength": 0.9999981596301908,
        "user": "e10003",
        "file_template": "/sls/X06SA/data/e10003/Data10/rasters/20160602/88e62fe9-6a1d-4b66-aea0-758fba397c48/raster_%06d.cbf",
        "rebin_frame": 0
    }
```

* 10hz_1.5e12
    * exposure time: 0.1 s
    * photon flux: 1.5e12
* 1hz_1.5e12
    * exposure time: 1.0 s
    * photon flux: 1.5e12
* 40hz_1.5e12
    * exposure time: 0.025 s
    * photon flux: 1.5e12
* 60hz_dump_1.5e12
    * exposure time: 0.016666 s
    * photon flux: 1.5e12
* 60hz_dump_3e10phs
    * exposure time: 0.016666 s
    * photon flux: 3e10
* lcp
    * exposure time: 0.025 s
    * photon flux: 1.5e12
* lcp100hz
    * exposure time: 0.010 s
    * photon flux: 1.5e12

data

Intensity Measurement:

Tom Caradoc-Davies, Daniel Eriksson, and David Aragao have prepared a very helpful document on the protocol they use for Eiger intensity measurement and Daniel has released the code they use for data analysis. [more/less ...]
Document: "Using Eiger for high-frequency beam intensity measurement"(PDF)
Code: https://github.com/AustralianSynchrotron/Sum-Eiger-Frames.
We will allocate a section of the HDRMX web page here for links to the raw data that beamlines that wish to share such data have used for their intensity tests. If you are interested in sharing such data please upload it to zenodo (https://zenodo.org) or other suitable web site that can provide a stable doi or url, and then send us that link for posting on the HDRMX web site and for Daniel to list in a file in their github repository.

Meta Data samples

James Holton's Gallery James Holton's little "gallery" of sample data sets from various beamlines. [more/less ...]
James wrote: My goal here was to somehow connect different "file formats" to the correct way to process them. Currently it is all by "example" with the processing in tarballs. But I think it would be great if I could also have a "proper CBF header" or "neXus spec" for each of these, also downloadable. My problem is I'm not really sure how to generate these files.
What would you think of specifying such things with an HDF5 master file? That is, a master file that points not to other HDF5 files, but to miniCBF, or SMV or mccd, or whatever files. This I believe will go a log way to acceptance of the new representations. Beamline scientists in general are loathe to change their header formats because this tends to generate hundreds of emails from angry users with titles like "wrong beam center". However, if we quietly introduced an HDF5 master file that contained the proper neXus information for the relevant beamline, then rather quickly front-end programs like generate_XDS.INP would start telling people that they will work much better when provided a master file than they will with just the raw image heders. If we can then convert the output of XDS, mosflm, HKL, etc. into a master file, then we will have closed the loop.
Can you help me convert the processing results form my "gallery" into neXus specifications?
Thanks, and let me know if you need anything I'm not providing.
http://bl831.als.lbl.gov/example_data_sets/

NeXus Format

NeXus is the preferred data format specification for Eiger images. NeXus is developed as an international standard by scientists and programmers representing major scientific facilities in Europe, Asia, Australia, and North America in order to facilitate greater cooperation in the analysis and visualization of neutron, x-ray, and muon data.
[more/less ...]
NeXus is an effort by an international group of scientists motivated to define a common data exchange format for neutron, X-ray, and muon experiments. NeXus is built on top of the scientific data format HDF5 and adds domain-specific rules for organizing data within HDF5 files in addition to a dictionary of well-defined domain-specific field names. The NeXus data format has two purposes:
- raw data: NeXus defines a format that can serve as a container for all relevant data associated with a scientific instrument or beamline. This is a very important use case.
- processed data: NeXus defines standards in the form of application definitions for the exchange of data between applications. NeXus provides structures for raw experimental data as well as for processed data.
NXmx -- the NXmx application definition defines the terms used in Eiger images. [more/less ...]
The NXmx application definition is intended to provide a NeXus template to be populated with data from MX experiments. If you follow this structure, you will have tools available to validate the completeness of the metadata to help to achieve efficient processing.
NXmx specification
NXmx -- proposed Gold Standard revisions November 2019. [more/less ...]
Here are proposed revisions to the NXmx application definition for the proposed Gold Standard. These are drafts, subject to discussion by NIAC and COMCIFS and possible approval.
NXmx Gold Standard draft,
NXmx -- proposed revisions October 2016. [more/less ...]
Here are proposed revisions to the NXmx application definition and the NXtransformations base class definition to accommodate new fields for the Eiger and to clarify NeXus/HDF5 support for multi-module detectors. These are drafts, subject to discussion at the 2016 NIAC meeting and possible approval.
NXmx 1.5 draft after NIAC revisions, prior NXmx 1.5 draft, NXtransformations 1.1 draft
prior NXtransformations 1.1 draft
NXmx 1.5 NXDL xml source, NXtransformations 1.1 NXDL xml source
NXmx 1.5 - 1.4 differences, NXtransformations 1.1 - 1.0 differences

Useful Test Cases

A set of 720 Eiger 9M images that are difficult to compress, courtesy of Martin Savko, SOLEIL
[more/less ...]
Regarding our discussion about the compression factors we observe with bslz4: in that respect the worst compression ratio that we have ever observed -- only 1.73 -- was achieved with the following dataset
The data were collected in standard mode, with the flux of ~1.2e12 ph/s on a very large sample (snapshot in the attachment) with beam size of 5 x 10 um (vertical x horizontal). The overall data-rate associated with this dataset was not that large: only ~450MB/s as the frame rate was only 20 Hz. The shared directory contains both compressed archive of the original hdf5 dataset and the compressed cbfs.
data