New Features as of version 0.93 The main feature of this release is allowing all components to support Python Pickling Version 2 (the binary protocol that tends to be a lot faster for serialization). Most versions of Python in the last few years support Python Pickling Protocol 2 as a built-in, and since it is at least typically 2x faster, we recommend using this protocol. NOTE: In order to have MidasTalker/Server/Yeller/Talker code use the new Python Pickling Protocol 2, you need to revisit code that use those classes and __FORCE__ them to use the new Serialization (and probably ArrayDisposition). Otherwise, ALL CLASSES default to Python Pickling Protocol 0 for backwards compatibility. The old interfaces still work, but they have been augmented to enable new options for the serialization protocol. Python Area: All MidasTalker and MidasListeners now support using Pickling Protocol 2. All examples have the serialization as an option so you can see how to set the different serialization. Also, all MidasListeners and MidasYellers support Pickling Protocol 2, as do the examples. For example: >>> from midastalker import * # Make sure PYTHONPATH is set properly >>> a = MidasTalker("somehost", 6666, SERIALIZE_P2) # This midastalker will talk to a server that uses Pickling Pro. 2 All the Midas* classes have been slightly rewritten so that their help pages from Python are cleaner. I.e.: >>> import midastalker >>> help(midastalker) A new feature is that the serialization allows Python arrays (import array) to be shuttled around. When using any of the Midas* classes, you can specify your "array disposition": for sending homogeneous data (i.e., Vectors from M2k or Array from OpenContainers). Thus, you can send Vectors as (1) plain Python Lists (inefficient but supported on all platforms), (2) Numeric Arrays (efficient but requires you build your Python Numeric package support so you can import Numeric) (2) Python Arrays (efficient and usually built-in but doesn't work well with cPickle on versions 2.3.x and below). If a Midas* asks for a feature it can't support (for example, asking for SERIALIZE_P2 on Python 2.2.x), it will throw an exception indicating what options you need to choose instead (for example, choose SERIALIZE_P2_OLD instead for Python 2.2.x). Most of the examples (ending with _ex*.py) support several options for controlling how data is sent around. By default, all data is serialized using Python Pickling Protocol 0, which is the most backwards compatible protocol but also the slowest. You can manually change how data flows in all the examples by setting one of three options on the command line: --ser=n - Set the serialization type of how the example sends data over the socket: this can have a huge performance difference. The options are: 0 : Use Python Pickling Protocol 0 - slower, but works w/all versions of PTOOLS, Python 1 : Use no serialization (just send the string data) - useful if you want to send raw data 2 : Use Python Pickling Protocol 2 - fast, but doesn't work with all versions -2 : As Python Pickling Protocol 2, but as Python 2.2.x - Python 2.2.x dumps Protocol 2 subtlely differently so this allows you to talk to older versions The default is Python Pickling Protocol 0 for backwards compatibility but it is strongly suggested moving to Protocol 2 for speed. --sock=n - Set whether socket communications use dual socket (--sock=2) or single socket (--sock=1) mode. To get around a bug in VMS TCP/IP, dual sockets needed to be used for biderectional communication. Modern sockets don't have this problem. Dual socket is the usual default, as that's the default of Midas 2K and X-Midas, but we suggest using Single Socket mode when you can. Most examples default to --sock=2 --arrdisp=n - Set the array disposition if you use serialization (Pickling Protocol 0 or 2). This is how the examples sends 'homogeneous data' around the system (i.e., M2k Vectors, PTOOLS Arrays, Python arrays, Numeric arrays). The options are: 0 : Shuttle all homogeneous data as Numeric Arrays (like the Numeric package from Python) 1 : Shuttle all homogeneous data as Python Lists (this loses their 'array-like' quality but is the most backwards compatible because all versions of PTOOLS and Python support Python Lists) 2 : Shuttle all homogeneous data as Python Arrays (like the Array package from Python) The default is 1 for backwards compatibility, but if you are using XMPY, Numeric comes built and it is strongly recommended to use 0 (for speed). If you are using Python but do not have the Numeric package AND you have Python 2.4.x or above, it is recommended to use 2 C++ Area: All C++ Midas* classes support Protocol 2 and can talk directly to any of the Python classes. There are also options for specifying the (a) the serialization you want and (b) the 'array disposition'. Note that the C++ classes should echo the structure of the Python classes. Added valprotocol2.h.cc, p2common.h, cpickle.h files to support the new protocol 2. Added the p2_test.cc,output and the P2TestFiles for repeatable tests to make sure the Pickling Protocol 2 works for Vals. All the C++ examples have added --ser, --sock and --arrdisp options to the command line parameters, just like the Python examples. See above for a full description. Added a README to the C++ area which describes these options. A minor interface change from the C++/opalutils.h: Use ReadTabFromOpalFile and ReadValFromOpalFile (with a capital 'F' in file) instead of ReadTabFromOpalfile and ReadValFromOpalfile. Minor bug fix in the same file: extraneous output to cerr has been deleted. M2k Area: Updated the OpalPythonDaemon and OpalSocketMsg so that they can use now use PythonP2, PythonP2WithNumeric, PythonP2WithArray, PythonP2Old or PythonP2OldWithNumeric as their serialization. The Adaaptive serialization also recognizes the new Python Protocol 2 (as well as the older Python 2.2.x protocol 2 which we number -2). The code for OpalPythonDaemon and OpalPythonSocketMsg is untouched, but the m2opalsockmsgnethdr .h and .cc had to be changed (and 4 new files p2common.h, cPickle.h, m2opalprotocol2.h,.cc). The unit.cfg was updated to reflect these changes and additions. We also added a Midas2k test (opalp2_test) and associated output. X-Midas Area: All the Python Pickling Protocol 2 from the C++ area has been folded into the Midas* classes in the ptools option tree. Updated primtives valpipe, fanintab and pipe2tab tab2opal with changes from Chuck Pendergraph and John Goodell Most host primitives have been updated to allow you to select a Python Pickling Protocol. The explain pages have been updated as well. All primitives which use the Midas* classes support the X-Midas switches /SER=n, /SOCK=n and /ARRDISP=n just like the C++ and Python examples (see the Python area above). Extensive testing has been done to make sure Python and C++ and Midas 2k and X-Midas all talk together. Certain large scale apps have served as a testbed to allow us a large degree of confidence in this release. TODO: We are hoping a future version of MidasServer and MidasTalker can use SERIALIZE_DISCOVER and be able to detect the serialization being used and just do the right thing. TODO: Add better error messages for when Tabs and Arrs don't parse correctly TODO: Update Tabs to make copies and compares slightly faster TODO: Fix OCArray for removeAll to be faster