Running SHADOW with Excon batch system:			        apr 1995

Purpose of Excon:
The Excon software was designed to run a series of programs much like you
would run an experiment in a lab.  It gives the user the ability to set
the sequence of events, extract data at any point, and set variables to
change with a loop statement.  The calculations are then run in batch mode
to save time and CPU, with the specified outputs saved in table form.  

Purpose of SHADOW+Excon:
To run SHADOW calculations iteratively in batch mode with the ability to
vary one or more namelist parameters each run.  Also runs power
calculations previously handles with MAKE_PWR.  Excon acts as the SHADOW
user by changing a variable, re-running SHADOW, and extracting the desired
file or data.  It can do this in a loop, for a set of values or both for
as many variables as desired.  Currently, you can run PREPLOT to extract
data or save the binary files for later processing.  For example, you can
run the standard power calculations using PWR_DENS, or you can set one of
the mirrors to rotate, set the source energy or divergence to change,
mirror radius, etc.

Instructions (quick and dirty):
-Set SHADOW path with .shadowrc file.
-Run all preprocessors, i.e., PREREFL, BRAGG, MAKE_ID, etc to set up data
files for SHADOW.
-Using MENU (must at least save system from MENU to create systemfile.dat) 
set up source and optical system.
-If running PWR calculations, generate SOURCE, SYSTEM and run PWR_DENS
once to set up the parameter file and the Excon state file.
-Run system-to-excon to generate the Excon experiment file, shadow.exp,
and the G-files that correspond to the SHADOW namelist files.  
THIS MUST BE RUN EVERYTIME YOU CHANGE A VALUE WITH MENU AND SAVE THE
SYSTEM OR THE CHANGES WILL NOT BE INCORPORATED IN THE NEXT EXCON RUN.

% system-to-excon -o shadow.exp -s start.00 [systemfile.dat]

where shadow.exp is the experiment file, -s takes the source namelist
file, and the systemfile.dat from SHADOW MENU is read.  If you have not
renamed systemfile.dat, there is no need to specify it on the command
line.
-Edit the experiment file to:
	1. add programs to the sequence such as seed, pwr_dens, and
	preplot.
	2. change single parameters, if desired, without re-entering MENU
	mode.
	3. define experiment using ``vary'' statements on namelist
	variables.
	4. add entries to OUTPUTS lines to save values, files, and output
	from various programs.
-Run Excon.
% excon -e shadow.exp -o table1
The defaults for these parameters are shadow.exp for the experiment file
and table for the output file name root.

Instructions (VERBOSE mode):
-Set SHADOW path with .shadowrc file.
-Run all preprocessors, i.e., PREREFL (mirror reflectivity), BRAGG
(crystal calculations), MAKE_ID (insertion devices), JNTPSCALC (surface
roughness), etc to set up data files for SHADOW.
-Using MENU (must at least save system from MENU to create systemfile.dat) 
set up source and optical system.
-If running PWR calculations, generate SOURCE, SYSTEM and run PWR_DENS
once to set up the parameter file and the Excon state file (created
automatically).
-Run system-to-excon to generate the Excon experiment file, shadow.exp,
and the G-files that correspond to the SHADOW namelist files.  

% system-to-excon -o shadow.exp -s start.00 [systemfile.dat]

where shadow.exp is the experiment file, -s takes the source namelist
file, and the systemfile.dat from SHADOW MENU is read.  If you have not
renamed systemfile.dat, there is no need to specify it on the command
line.

The experiment file controls the sequence of events within each iteration
of SHADOW as well as controlling the entire experiment by changing
specified variables and extracting output.  The user also has the option
to override single namelist variables in the experiment file.

The G-files are named start.00.G, for example, and contain all the
variables in the namelist.  They can be edited tochange parameters, if
desired, to avoid re-running system-to-excon and re-customizing the
experiment file.

*** experiment file created by system-to-excon ***

# PROGRAM SEQUENCE 
# 
# if you have only one SOURCE, no need to tack on a number at the end, but
# must do so for the Optical elements, since EXCON has to know how to
# address each one; eg., the following is a valid experiment sequence:
# source -> trace1 -> trace2 -> trace3 -> trace4
#
seed1 -> source -> trace1 -> trace2 -> trace3 -> preplot1


# INPUTS
#
# the following G files were created from the SHADOW MENU definitions. 
# DO NOT MESS WITH THE FOLLOWING SECTION WITH THE G FILES. 
#
source.$GFILE		= start.00.G
trace1.$GFILE		= .//./start.01.G
trace2.$GFILE		= .//./start.02.G
trace3.$GFILE		= .//./start.03.G
pwr_dens.statefile	= ./pwr_dens.state.g
# must add above line if doing PWR_DENS calculations.  This is the
# only reason you would change a line in this section!

#
# you could override ALL the defaults here for all the tools. Any
# parameter not specified here will default to the entries in the G
# files above.
#
# source.paramName	 = value
# trace1.paramName	 = value
# trace2.paramName	 = value
# etc etc etc
#
trace1.fmirr		= 1	#override start.01 to make a spherical
				#mirror


# EXPERIMENT (INCLUSIVE)
# vary source.paramName from X to Y by Z
# vary trace1.paramName from 86.0 to 89.5 by 0.2
vary trace1.rmirr from 1000 to 1005 by 0.5	# Change the radius of
						# the spherical mirror
						# and look at the output 
						# of the beamline
						# w/PREPLOT.

# OUTPUTS (table and graph are reserve words)
# table1.columns = source.paramName, trace1.paramName, trace2.paramName
table1.columns = trace1.rmirr, preplot1.fileout

*** end of experiment file ***

-Edit the experiment file to:
1. add programs to the sequence such as seed, pwr_dens, or preplot.  
Valid programs in Excon sequence:
source:  runs SHADOW SOURCE generation on source namelist specified in
         system-to-excon.
trace#:  runs SHADOW TRACE on optical element #.
seed:    runs SEED utility to change the starting random number seed in
         the source namelist file.
preplot#:runs SHADOW utility PREPLOT at specified point in sequence.  By
         default, it will take as input the star.xx file of the previous 
	 element or the begin.dat file if it is after the source.  Default
	 values for running the program are: extracting two columns, X
	 and Z (1 and 3), excluding the losses, writing an ascii file, and
	 output in a file preplot.  The following lines need to be added
	 to the variables section specifying the correct values if you 
	 wish to use values other than the defaults:
	 preplot.icol		= 2	# number of columns to extract
					# 1,2, or 3 are valid answers
	 preplot.kol1		= 1	# first column
	 preplot.kol2		= 3	# second column
	 preplot.kol3		= -1	# third column
	 preplot.iener		= 1	# eV=1,Ang=0,cm-1=2
					# only if column 11 chosen
	 preplot.kloss		= 0	# exclude losses
	 preplot.iterm		= 3	# save to ascii file
	 preplot.filein		= NOT SPECIFIED
	 preplot.fileout	= preplot.$UNIQUE
pwr_dens:runs SHADOW utility PWR_DENS on file specified in initial run of
	 PWR_DENS (outside of Excon).  Requires running seed so that
	 initial rays are not identical for each run.  Also requires
	 adding:
	 pwr_dens.statefile	= ./pwr_dens.state.g
	 to the experiment file INPUTS section.  The above line is the
	 only thing you should change in this section.  The output file of
	 PWR_DENS is saved automatically, so there is no reason to do so
	 in the reports section.
cleanup: deletes screen files created by Excon, but not deleted each run.
         Saves on disk space for multiple runs.  Takes no input.

2. change single parameters, if desired, without re-entering MENU mode.
For example, in the preceeding experiment file, we have overridden the
mirror shape in the start.01 file and made it spherical.
3. define experiment using ``vary'' statements on namelist variables.
Shown, we vary the mirror radius from 1000 to 1005 in steps of 0.5.

Valid experiment statements:
vary tool.param from value_min to value_max by value_step
	Runs SHADOW with tool.param (i.e. trace2.theta_i (incidence
	angle)) starting at value_min and incrementing by value_step
	until value_max is reached.
vary tool.param set value1, value2, value3, ..., valuen
	Runs SHADOW with tool.param set to be each of the values in the
	set once.
vary together tool.param1 and tool.param2
	Groups variables to be varied together.  Both param1 and param2
	will be changed at the same time according to their respective
	vary statements.  If this statement is not used, every possible
	combination of param1 and param2 will be run.  You can also create
	groups of parameters that vary at the same time by using multiple
	vary together statements.  For example, if we add the statement:
vary together tool.param2 and tool.param3
	parameters 1, 2, and 3 will all change at the same time.

4. add entries to OUTPUTS lines to save values, files, and output from 
various programs.  Intermediate files will not be saved unless you 
include the filename in the output table.  Saved files will be given 
unique names according to the original SHADOW name, the OE number and 
the iteration.
In the example experiment file, we save a table that contains the mirror
radius for that run (trace1.rmirr), and the output file of PREPLOT
corresponding to that radius (preplot1.fileout).

-Run Excon.
% excon -e shadow.exp -o table
or if using defaults, simply
% excon
The defaults for these parameters are shadow.exp for the experiment file
and table for the output file name root.

EXAMPLE:

1. The first example will take an existing beamline configuration and
rotate the third mirror to simulate scanning.  We will change the random
number seed to avoid streaking, and use preplot to extract the X and Z
columns of the final image.  We start with the following files:
>
start.00	start.02	systemfile.dat
start.01	start.03
>
> system-to-excon -o shadow.exp -s start.00 
Saving existing experiment file shadow.exp to shadow.exp.bak ... Done.
Creating GFILES ...
    start.00 --> start.00.G
    .//./start.01 --> .//./start.01.G
    .//./start.02 --> .//./start.02.G
    .//./start.03 --> .//./start.03.G
 
The EXCON experiment template is in the file ``shadow.exp''.
 
> ls
shadow.exp	start.01	start.02.G	systemfile.dat
start.00	start.01.G	start.03
start.00.G	start.02	start.03.G
> cat shadow.exp
# PROGRAM SEQUENCE 
# 
# if you have only one SOURCE, no need to tack on a number at the end, but
# must do so for the Optical elements, since EXCON has to know how to
# address each one; eg., the following is a valid experiment sequence:
# source -> trace1 -> trace2 -> trace3 -> trace4
#
source -> trace1 -> trace2 -> trace3
 
 
# INPUTS
#
# the following G files were created from the SHADOW MENU definitions. 
# DO NOT MESS WITH THE FOLLOWING SECTION WITH THE G FILES. 
#
source.$GFILE           = start.00.G
trace1.$GFILE           = .//./start.01.G
trace2.$GFILE           = .//./start.02.G
trace3.$GFILE           = .//./start.03.G
 
#
# you could override ALL the defaults here for all the tools. Any
# parameter not specified here will default to the entries in the G
# files above.
#
# source.paramName       = value
# trace1.paramName       = value
# trace2.paramName       = value
# etc etc etc
#
 
 
 
# EXPERIMENT (INCLUSIVE)
# vary source.paramName from X to Y by Z
# vary trace1.paramName from 86.0 to 89.5 by 0.2
 
 
 
# OUTPUTS (table and graph are reserve words)
# table1.columns = source.paramName, trace1.paramName, trace2.paramName

>
The above file shows the default experiment file created for a given
system.  We now need to edit it and add the customization for our 
experiment. (shown below)

# PROGRAM SEQUENCE 
# 
# if you have only one SOURCE, no need to tack on a number at the end, but
# must do so for the Optical elements, since EXCON has to know how to
# address each one; eg., the following is a valid experiment sequence:
# source -> trace1 -> trace2 -> trace3 -> trace4
#
#seed -> source -> trace1 -> trace2 -> trace3
seed -> source -> trace1 -> trace2 -> trace3 -> preplot

# INPUTS
#
# the following G files were created from the SHADOW MENU definitions. 
# DO NOT MESS WITH THE FOLLOWING SECTION WITH THE G FILES. 
#
source.$GFILE		= start.00.G
trace1.$GFILE		= start.01.G
trace2.$GFILE		= start.02.G
trace3.$GFILE		= start.03.G

#
# you could override ALL the defaults here for all the tools. Any
# parameter not specified here will default to the entries in the G
# files above.
#
# source.paramName	 = value
# trace1.paramName	 = value
# trace2.paramName	 = value
# etc etc etc
#
trace3.f_move		= 1	# sets flag for mirror movement

# EXPERIMENT (INCLUSIVE)
# vary source.paramName from X to Y by Z
# vary trace1.paramName from 86.0 to 89.5 by 0.2
vary trace3.x_rot from -0.20 to 0.20 by 0.10

# OUTPUTS (table and graph are reserve words)
# table1.columns = source.paramName, trace1.paramName, trace2.paramName
table1.columns = trace3.x_rot, preplot.fileout, preplot.$GFILE

Excon can now be run.  The following files will result:
> excon
> ls
effic.01           preplot+3.g        rmir.03            start.02
effic.02           preplot+4.g        seed.state         start.02.G
effic.03           preplot+5.g        shadow.exp         start.03
optax.01           preplot.preplot+1  shadow.exp.bak     start.03.G
optax.02           preplot.preplot+2  start.00           systemfile.dat
optax.03           preplot.preplot+3  start.00.G         table1
preplot+1.g        preplot.preplot+4  start.01
preplot+2.g        preplot.preplot+5  start.01.G

Since we told excon to save the PREPLOT output and G-files, they appear.
Most other intermediate files are deleted to save space.  We could have
put trace1.starfile in the table to save all the image files after the
first element.  table1 contains the following:
> cat table1
-0.200000 preplot.preplot+1 preplot+1.g 
-0.100000 preplot.preplot+2 preplot+2.g 
0.000000 preplot.preplot+3 preplot+3.g 
0.100000 preplot.preplot+4 preplot+4.g 
0.200000 preplot.preplot+5 preplot+5.g 

The first values are those of the variable x_rot, then the name of the
PREPLOT output file containing the image for that rotation angle, and 
the name of the PREPLOT G-file.  The preplot.preplot+n files are two
columns ascii and can be plotted with Primvs or any other local graphics
package.

2. The second example will show a calculation of incoming power at a final
image.  Previously, the program MAKE_PWR would have been used.  We start 
with the basic files, and add gold.dat and be.dat, the files with the 
optical constants for the gold coating on the mirrors and the beryllium 
window.  These were generated with PREREFL.
> ls
be.dat          start.00        start.02        systemfile.dat
gold.dat        start.01        start.03
>
We now have to run SHADOW completely once and run PWR_DENS. We should end
up with:
> ls
PWR_DENS_PAR      end.01            optax.01          star.02
be.dat            end.02            optax.02          star.03
begin.dat         end.03            optax.03          start.00
effic.01          gold.dat          pwr_dens.state.g  start.01
effic.02          mirr.01           pwr_out           start.02
effic.03          mirr.02           screen.0301       start.03
end.00            mirr.03           star.01           systemfile.dat
>
We have the usual output of PWR_DENS, plus a file pwr_dens.state.g.  
The file contains the parameters for the power calculations
like the g-files corresponding to the source and optical elements.
At this point, we can create our experiment file with system-to-excon.
We need to add to the sequence, the INPUTS and the experiment section.  It
should look like the following when done:  Notice the line in the
INPUTS section.  Power calculations are the only reason you would make a
change to the section of the experiment file.  Excon needs to know about
the power g-file containing the parameters.
>

# PROGRAM SEQUENCE 
# 
# if you have only one SOURCE, no need to tack on a number at the end, but
# must do so for the Optical elements, since EXCON has to know how to
# address each one; eg., the following is a valid experiment sequence:
# source -> trace1 -> trace2 -> trace3 -> trace4
#
seed -> source -> trace1 -> trace2 -> trace3 -> pwr_dens


# INPUTS
#
# the following G files were created from the SHADOW MENU definitions. 
# DO NOT MESS WITH THE FOLLOWING SECTION WITH THE G FILES. 
#
source.$GFILE		= start.00.G
trace1.$GFILE		= .//./start.01.G
trace2.$GFILE		= .//./start.02.G
trace3.$GFILE		= .//./start.03.G
pwr_dens.statefile	= ./pwr_dens.state.g
#
# you could override ALL the defaults here for all the tools. Any
# parameter not specified here will default to the entries in the G
# files above.
#
# source.paramName	 = value
# trace1.paramName	 = value
# trace2.paramName	 = value
# etc etc etc
#



# EXPERIMENT (INCLUSIVE)
# vary source.paramName from X to Y by Z
# vary trace1.paramName from 86.0 to 89.5 by 0.2
vary pwr_dens.loop from 1 to 10 by 1


# OUTPUTS (table and graph are reserve words)
# table1.columns = source.paramName, trace1.paramName, trace2.paramName


You can save any of the output -- the PWR_DENS output files are saved
automatically.  Excon can now be run.

> excon

At the end of the run, you will have the output of PWR_DENS just like with
MAKE_PWR.

Namelist Variables:
To effectively use the "vary" statements, the namelist variables must be
defined.  The file ./shadow/doc/Namelist.doc lists all the
variables and the brief description of what they are and their allowed
values.


Questions on how to run, bug reports, etc, contact Chris Welnak
						cwelnak@xraylith.wisc.edu
						ph: 608 265 6074
						fax: 608 265 3811

