Saving/Retrieving Warp Data in External Files

Here we outline several possible ways to save data from a Warp simulation into an external binary or ascii text file and how to read data (from other Warp simulations or other sources) stored in a binary or ascii file into a Warp simulation. In the sections Binary File and ASCII Text File below we overview how to save data associated with a variable "var" defined in a Warp simulation into an external binary or ascii text file and how to restore it in another Warp simulation. The external files could also, of course, be opened with other software tools. The variable var can be any Warp variable (including array element) with attributes or a user defined variable in the python interface. As a special case we cover separately below, Reading an ASCII Text Data File, to illustrate how potentially large data sets stored in ascii text files can be efficiently read into Warp. This situation often occurs when reading in externally calculated field data for lattice elements such as gridded magnetic field data exported from a design code for magnetic optics or gridded electromagnetic field data from a rf cavity design code.

The case of saving complete data from a simulation for possible restart/continuation of simulations is covered in Saving/Restarting Warp Simulations.

Binary File

Warp can save variables in efficient binary form using a convenient wrapper around the standard python pickle package. Other binary formats for saves are, if loaded, available within Warp. However, the pickle package has become standard within python and is presently recommended for Warp applications. To save the variable var in a binary pickle file "save.pkl" as "var":

fout = PWpickle.PW("save.pkl")

fout.var = var

fout.close()

To read the variable var back into a Warp simulation:

fin = PRpickle.PR("save.pkl")

var = fin.var

fin.close()

The pickle package can be used directly from python as well. PWpickle and PRpickle are convenient since the variable name associated with the data is saved, giving a nice attribute-like, pythonic way of writing and reading the data.

ASCII Text File

Saving variables in ascii text files is not as efficient in terms of storage as binary, but it offers advantages: the saved data is human readable, the files are highly portable, and the format is robust for long-term storage since a special reader is not required. Text files can be created using the python interface in numerous ways. Several simple ways potentially convenient in Warp simulations are outlined below.

Standard python commands to open and write strings to ascii text files can be employed to save variables. For example:

fout = open("save.txt","w")

fout.write("var = %s \n" % var)

fout.close()

In this example var is written into a string using python string formatting syntax with the line return "\n" (using write, the line return must be explicitly given). The python write() method writes a string to the file - standard formatting and string processing techniques can be applied to generate the string. In the example above, "%s" denotes a string, effectively generating open format. Detailed information can be found in the python documentation for the string module. Common format options include:

%s String

%d Decimal (int)

%i Integer

%e Floating Point (exponential)

%f Floating Point (decimal)

%g Floating Point (decimal, no padded zeros)

For specific number formats with specified digits before and after the decimal point, follow the conventions such as:

%4.3f Floating Point, 4 digits, 3 past decimal

To write two Warp variables (var1,var2) in a string in exponential format with specific digits, the write() method in the previous example can be replaced with

fout.write("var1 = %.8e; var2 = %.8e \n" %(var1,var2))

to save the two numbers in exponential format with 8 digits past the decimal point.

Alternatively, one can redirect standard output to the external ascii text file "save.txt" and save variables in python syntax to read back in:

fout = open("save.txt","w")

sys.stdout = fout

print "var = ",var

sys.stdout = sys.__stdout__

fout.close()

Note that standard output is reset to the default value after the variable write: otherwise all subsequent standard output will be written to save.txt.

In both of these ascii text file save examples, the variable var has been saved in python syntax within the ascii text file "save.txt" so it can then be read into a Warp simulation using:

execfile("save.txt")

Alternatively, variables can be saved in successive lines in the text file and then read back in the known order. This, of course, requires careful format control. Useful python commands in such procedures include:

fin = open("save.txt","r")

str = fin.read() # read entire file into string for processing/extraction

fin.close()

to read the entire file into the string str for processing/extraction, or substitute

str = fin.readline()

to read the next line in the file (when called successively) into string str. If readline() reads in the value of var as a string in floating point (or exponential) format, it can be converted to floating point form and stored in var using

var = float(fin.readline())

Yet another way to save data from runs in ascii format in unix/linux based operating systems is to pipe standard output from the terminal window where a simulation is made into a file. For example, a simulation defined in "script.py" can be run non-interactively with output redirected to an ascii text file "run_log.txt" by executing the script as:

% python script.py > run_log.txt

Any formatted writes of variables will then be saved to the text log file along with all status output writes of the simulation.

This can also provide a useful way to make a log of a simulation including saving limited data of interest generated in formatted print statements using techniques described in the examples above.

Reading an ASCII Text Data File

Here we cover how field data from a potentially large formatted ascii text file can be read into Warp with an illustrative example as a special case. In principle, techniques described above in ASII Text File could be applied. But the procedures described here are more efficient and straightforward to apply for large data sets. The prototypical situation where this occurs is gridded field data which might be produced and exported by a design code for magnetic optics such as Poisson or TOSCA and imported into Warp to use in a "bgrd" gridded field lattice element (see subsection Lattice Element Scripts within Lattice). Similar situations also occur for gridded field data describing an rf cavity. Data sets in such situations may be large for 3d field components stored on a fine resolution grid. Although large data sets are more efficiently stored in binary format, ascii is often used for exporting/archiving data from magnet design codes since ascii is very stable and there are a plethora of modern and older magnetic design codes which are employed. Also, being able to open/examine the data with an ordinary text editor is often useful. It is common to write a special purpose Warp "simulation" script to open and pre-process the large ascii text file containing the field data exported, and then save the results in arrays properly structured for use in Warp in binary pickle format files (see Binary File above) which is then used to import the data in actual simulations.

Suppose we have an ascii format text file "field.txt" containing 3d magnetic field data for Bx, By, Bz on a uniform x, y, z mesh. Numbers are separated by spaces and each line within field.txt represents a data point with the format:

x y z Bx By Bz

We suppose that each data entry has fortran-like format and that the ordering of the data in the file is unknown other than it is a uniform mesh with (presumably) all points defined. A common variant of field.txt might have data points in each line separated by commas "," rather than spaces " ". Often there will be several lines at the top of the field.txt containing comments to help describe the data. Unfortunately, lines of data in field.txt may appear to be formatted differently depending on the platform that the file was produced because Windows, Unix/OSX, and older Mac OSs employ different conventions for indicating the end of a line in a text file. Text editors one might employ to look at the data are often smart enough to automatically fix such problems. If it creates issues, scripts can be readily found to "fix" the file by changing the line feeds to the form appropriate for the OS used. Note also,the example as posed here contains much redundant data (further reducing the efficiency of the storage) since the mesh is assumed uniform and the x, y, z coordinates of each mesh point are given. A more efficient alternative is to give mesh bounds and integers giving the mesh locations (integers replacing floating point numbers for the coordinates to reduce storage) or to specify the grid ordering of the field data itself (no coordinate storage). Examples provided here can be modified to cover such situations. However, in these case, care must be taken to properly understand grid indexing contentions employed (0 based, 1 based, or other) and mesh bounds for proper reading of the data. Field data for idealized lattice elements will also often have symmetries which can be exploited to reduce storage requirements. For simplicity, we do not cover such cases here but the methods presented can be modified to be consistent with reduced symmetry data sets.

First, to read in the data we use a fairly general script getdatafromtextfile() written to simplify reading in mesh data from text files. For our example, data is read in using:

[x,y,z,Bx,By,Bz] = getdatafromtextfile("field.txt",nskip=8,dims=[6,None])

Here, we assume that field.txt contains 8 comment lines in the header which are skipped, and dims=[6,None] means that the data will be read in line-by-line until the end of the file is reached (via the None setting) with 6 entries per line. The 1d arrays x, y, z, Bx, By, Bz will contain the mesh coordinates and corresponding field data with each common array element corresponding to a unique mesh location, i.e., x[10], y[10], z[10] corresponds to the 11'th mesh point stored in the data vectors with Bx[10], By[10], Bz[10] the corresponding 3d field components at the mesh point. If field.txt contains comma "," separated data entries, the option separator = "," can be employed in getdatafromtextfile(). Other options are detailed in doc(getdatafromtextfile).

To obtain mesh increments and sizes from the x, y, z mesh data:

dx = average(diff(unique(x)))

dy = average(diff(unique(y)))

dz = average(diff(unique(z)))

nx = nint( ( x.max()-x.min() )/dx )

ny = nint( ( y.max()-y.min() )/dy )

nz = nint( ( z.max()-z.min() )/dz )

Here, various Numpy functions are employed to efficiently difference [diff()] and then average [average()] unique [unique()] x-mesh data entries to find the mesh increment dx. Then the max minus min x-coordinate range of the mesh data are divided by the mesh increment dx and rounded to the nearest integer function [nint()] to find the number of mesh points nx. Note that (nx+1)*(ny+1)*(nz+1) corresponds to the length of the data vectors [i.e,(nx+1)*(ny+1)*(nz+1) = len(x)]. This coding avoids possible roundoff errors that might result from using particular neighboring pairs of data points to calculate the grid increments. Although we presume a uniform mesh with no missing entries, the coding can also deal with redundant data entries. Alternatively to the calculation above, the mesh information (both size and bounds) might be separately read into Warp rather than calculated directly from the data entries.

Next, we efficiently load the vectors of field data entries Bx, By, Bz onto uniform mesh arrays bx, by, bz with python convention (0 based) grid indexing while assuming no underlying order of the data in the ascii text file, as follows:

bx = fzeros((nx+1,ny+1,nz+1) )

by = fzeros((nx+1,ny+1,nz+1) )

by = fzeros((nx+1,ny+1,nz+1) )

ix = nint( ( x-x.min() )/dx )

iy = nint( ( y-y.min() )/dy )

iz = nint( ( z-z.min() )/dz )

ii = ix + (nx+1)*iy + (nx+1)*(ny+1)*iz

bx.ravel(order='F').put(ii,Bx)

by.ravel(order='F').put(ii,By)

bz.ravel(order='F').put(ii,Bz)

Here, a fortran memory order data array bx are created with fzeros() for efficient fetching within Warp. 0-based grid indices ix are calculated from the x coordinate vector and mesh increment dx. Then data in the field vectors Bx, By, Bz are placed into the 3d arrays bx, by, bz using ravel() and put() with the Fortran ordered 1d-index ii with put() and ravel(). Note that although the arrays are created with Fortran memory order, the procedure assumes no ordering of the data itself. The resulting bx, by, dz 3d field arrays can then be loaded into the lattice using the addnewbgrd() field element script as described in the subsection Lattice Element Scripts within Lattice.

A less efficient, but more elementary and still general approach to the data loading outlined above can be accomplished by replacing the code from ii = ... down with:

for ii in len(x):

i = ix[ii]

j = iy[ii]

k = iz[ii]

bx[i,j,k] = Bx[ii]

by[i,j,k] = By[ii]

bz[i,j,k] = Bz[ii]

For cases where the field data stored reflects symmetries, the reduced data can be read in consistent with the symmetries and appropriate Warp options set to employ the reduced data sets if such symmetries are an option of the optical element in Warp. Or the data can be extended consistent with the symmetries when being read in. We do not cover these cases since there are a large number of possibilities with details depending on the specific situation.