Software Coordinating Committee Conference Call
		      April 19, 2007 1:00 PM EDT
			  Recorder: C. DeTar


Present: 

   DeTar, Levkova, Fowler, Scholz, Pochinski, Brower, Clark, 
   Zhang, Osborn, Edwards, Jung, Holmgren, Simone, Watson, 
   Alan Porterfield, Mawhinney

Absent (??):

   Khoriaty, Efstathiadis, Gottlieb, Basak, Joo, Renner, 

=====================================================================
** Action items

1. Still finishing up the Propagator File Format
     (The usual suspects: Carleton, Osborn,Chulwo, Bob M, et al)

     We discussed the updated document appended below.
     Here are the points we agreed on:
       (1) We need a variant of file format 2 that 
           replaces the source Dirac field with a source
           complex field.
       (2) The LHPC file format should describe the source
           as a propagator field.
       (3) The idea of writing records with data restricted to
           a hypercube seems useful.

** DeTar: I will make these changes to the file format specification
   and circulate it.
   

2. Back to setting a strategy for Multi-threading/Multi-core plans:
    (Robert, Jie, Andrew, Rob et al)

   Brower: Wouldn't you want to use threads first for Level 3
     software?  Would you want to rewrite QMP so it is aware
     of global memory?
   Pochinski: QMP always treats local memory as global.
   Edwards: That would be for the multiprocess model 
   Holmgren: MPI would do it automatically in that case.
   Edwards: So we don't need to bring in Level 3 in this
     discussion.
   Simone: QMP could be modified to avoid a copy.
   Holmgren: Shared memory assignments for the fields would 
     do it.
   Edwards: Currently QMP allocates memory and then opens
     a channel for communication.
   Pochinski: Actually it is only half a channel.  Currently we
     allocate only sends. To avoid copies you have to allocate both
     send memory and receive memory and add another "I'm done" call
     for the receiver to tell the sender it is safe to alter the
     values.
   DeTar: Better than calls would be locking memory.
   Fowler: On the Intels if you could bring all the data
     into cache, the copies would be handled efficiently.
   Brower: It would be good as an exercise to see how to code
     Dslash in threads and see what impact it would have on QMP.
   Fowler: Our group is talking about run time systems for multicore 
     chips.  We'd be interested in using a good example,
     for our proposal.
   Brower: How about QMP?

Committee conference concluded at 3:00 PM EDT.  
Next call Apr 26 at 1:00 PM EDT

----------------------------------------------------------------------


		USQCD Propagator File Format Proposal
			    April 17, 2007
		 DeTar, Edwards, Osborn, Jung, Simone


Currently we have three file formats on the table.

General considerations

We use the DeGrand-Rossi gamma matrix convention with the unimproved
sign for gamma_2.

The binary data is in lexicographic site order (same as with the gauge
field).

We use three fields in these files (and a choice of precision for
each) with the following names.  The order of words for a given
lattice site is

  USQCD_F_Complex  USQCD_D_Complex

    re/im

  USQCD_F3_DiracFermion USQCD_D3_DiracFermion

    sink spin  (slower)
    sink color (faster)
    re/im

  USQCD_F3_DiracPropagator  USQCD_D3_DiracPropagator

    sink spin    (slowest)
    source spin  (faster)
    sink color   (faster still)
    source color (fastest)
    re/im

----------------------------------------------------------------------
A twelve-field propagator file with one scalar source:
(13 logical records)
----------------------------------------------------------------------

1. USQCD_DiracFermion_ScalarSource_12Sink

  Here the source is given by one complex scalar field and the same
  scalar field is used for each source color and spin.

  Logical record 1: The complex scalar source
     (Do we allow a single-precision source and double precision solutions?)

  Logical records 2 - 13: DiracFermion sink fields
      Solutions for each choice of source color and spin
      Required order: color varies faster than spin.
        (spin, color) = (0,0) (0,1) (0,2) (1,0) , ...


----------------------
File metadata
----------------------
<?xml version=\"1.0\" encoding=\"UTF-8\"?>
   <usqcdPropFile>
      <version>1.0</version>             # prop file version
      <type>USQCD_DiracFermion_ScalarSource_12Sink</type>  # Propagator file format type
      <info>                             
                     # Collaboration discretion
      </info>
   </usqcdPropFile>

----------------------
Source record metadata
----------------------

(QIO_RecordInfo datatype = USQCD_F_Complex or USQCD_D_Complex)
(QIO_RecordInfo precision = F or D)

<usqcdSourceInfo>
  <version>1.0</version>
  <info> collaboration use </info>
</usqcdSourceInfo>

-------------------------------
Propagator/sink record metadata
-------------------------------

(QIO_RecordInfo datatype = USQCD_F3_DiracFermion or USQCD_D3_DiracFermion)
(QIO_RecordInfo precision = F or D)

<?xml version=\"1.0\" encoding=\"UTF-8\"?>
   <usqcdPropInfo>
      <version>1.0</version>             # prop file version
      <spin> </spin>
      <color> </color>
      <info> collaboration use </info>
   </usqcdPropInfo>

----------------------------------------------------------------------
A multiple-field propagator file with the source and sink fields paired.
----------------------------------------------------------------------

2. USQCD_DiracFermion_Source_Sink_Pairs

  Here we specify the full source field and its solution field.
  There can be any number of such pairs.

  Logical record 1: DiracFermion field (source)
      Source #1
  Logical record 2: DiracFermion field (sink)
      Solution #1
  Logical record 3: DiracFermion field (source)
      Source #2
  Logical record 4: DiracFermion field (sink)
      Solution #2
  etc.

----------------------
File metadata
----------------------
<?xml version=\"1.0\" encoding=\"UTF-8\"?>
   <usqcdPropFile>
      <version>1.0</version>
      <type>USQCD_DiracFermion_Source_Sink_Pairs</type>
      <info>                             
                     # Collaboration discretion
      </info>
   </usqcdPropFile>

----------------------
Source record metadata  
----------------------

(QIO_RecordInfo datatype = USQCD_F3_DiracFermion or USQCD_D3_DiracFermion)
(QIO_RecordInfo precision = F or D)

<?xml version=\"1.0\" encoding=\"UTF-8\"?>
  <usqcdSourceInfo>
    <version>1.0</version>
    <spin> optional - collaboration use </spin>
    <color> optional - collaboration use </color>
    <info> collaboration use </info>
  </usqcdSourceInfo>

-------------------------------
Propagator/sink record metadata
-------------------------------

(QIO_RecordInfo datatype = USQCD_F3_DiracFermion or USQCD_D3_DiracFermion)
(QIO_RecordInfo precision = F or D)

<?xml version=\"1.0\" encoding=\"UTF-8\"?>
   <usqcdPropInfo>
      <version>1.0</version>             # prop file version
      <info> collaboration use </info>
   </usqcdPropInfo>

----------------------------------------------------------------------
The LHPC propagator format (modified slightly to make more standard)
(1 logical record)
----------------------------------------------------------------------

3. LHPC_DiracPropagator

  Logical record 1:

    Here a single record contains the full propagator.

    (Should we require a complex scalar field for the source as
     we did in type 1?)


-------------
File metadata
-------------

<?xml version=\"1.0\" encoding=\"UTF-8\"?>
   <usqcdPropFile>
      <version>1.0</version>             # prop file version
      <type>LHPC_DiracPropagator</type>      # Propagator file format type
      <info> collaboration use </info>
   </usqcdPropFile>


-------------------------------
Propagator/sink record metadata
-------------------------------

(QIO_RecordInfo datatype = USQCD_F3_DiracPropagator 
                        or USQCD_D3_DiracPropagator)
(QIO_RecordInfo precision = F or D)

<?xml version=\"1.0\" encoding=\"UTF-8\"?>
   <usqcdPropInfo>
      <version>1.0</version>             # prop file version
      <info> collaboration use </info>
   </usqcdPropInfo>