I’m testing against “known good” components – the next release branch.

How to run tests of current development line of a library (a Boost library, e.g. Boost.Geometry) against Boost release branch? Assuming all the library development happens in Boost trunk, let’s say in ${DEV}/boost/_svn/trunk where ${DEV} is your base workshop location, do this:

• Checkout Boost release branch

  mkdir -p ${DEV}/boost/_svn/branches/release
  cd ${DEV}/boost/_svn/branches/release
  svn co https://svn.boost.org/svn/boost/branches/release/ .
  

• Stay in ${DEV}/boost/_svn/branches/release
• Build and install Boost.Build v2 from the release branch:

  cd tools/build/v2
  # On Windows, run bootstrap.bat
  ./bootstrap.sh
  # On Windows, I use C:\usr as prefix for my Boost installation
  ./b2 --prefix=/usr/local install
  # Note, this command will also install b2 utility in ${prefix}/bin
  

• Switch Boost.Geometry directories to trunk:

  svn switch https://svn.boost.org/svn/boost/trunk/boost/geometry boost/geometry
  svn switch https://svn.boost.org/svn/boost/trunk/libs/geometry libs/geometry
  

• Run tests and generate full log in file:

  export BGLOG=boost-geometry-test.log
  date > $BGLOG & b2 >> $BGLOG 2<&1 & date >> $BGLOG
  
  # On Windows, I run this with logging this way:
  set BGLOG=%CD%\boost-geometry-test.log
  dateu.exe > %BGLOG% & b2 >> %BGLOG% 2<&1 & dateu.exe >> %BGLOG%
  

  Note, the dateu.exe is renamed Unix utility date.exe installed from GnuWin32 packages. I just like it.

Inspect the log file for test failures, for example by quick check of number of passed tests:

$ grep "\*passed\*" boost-geometry-test.log | wc -l
111

That’s it.

Try this out – it will help a lot.

The new list is available at geometry at lists dot boost dot org. It is also mirrored at Nabble as Boost.Geometry.

The old ggl at lists dot osgeo dot org will be wiped out soon.

If anyone experiences any problems with the new geometry at lists.boost.org, please contact me directly.

Big thanks to the OSGeo Foundation for hosting the ggl mailing list and thanks to Boost admins for help with the new server.

Lately, I have found a tool written in Ruby by Eloy Durán. It is git-svn-mirror – a command-line tool that automates the task of creating a Git mirror for a SVN repository, and keeping it up-to-date. I installed Eloy’s tool from Ruby gems and played with it for a while. I really liked it.

I skimmed the Ruby code of git-svn-mirror and found out it makes use of bare repositories in Git. A Git bare repository stores just the contents of the .git directory, without any files checked out around it. Long story short, this script does almost exactly what I need and if there is something it does not do, then I can add it.

I have never written a single line of code in Ruby and I don’t feel like I need to learn it now. So, I decided to port git-svn-mirror to Python. I have just pushed pygit-svn-mirror 0.1 based on git-svn-mirror 0.1 to the repository at GitHub. I have tried to follow command line interface and overall code structure of the original version in Ruby. I have also preserved the original license and Eloy’s copyright.

There is README.md file included with detailed guide on how to use the pygit-svn-mirror. Basically, there are two commands: init and update. For each command, --help option will display required and supported arguments.

For example, creating mirror of Subversion repository of PROJ.4 project at GitHub involves the following commands:

mkdir /path/to/proj4/mirror
cd /path/to/proj4/mirror
git-svn-mirror.py init \
  --from=https://svn.osgeo.org/metacrs/proj/ \
  --to=git@github.com:<USRNAME>/proj.4.git

and to update the mirror from its workbench directory:

cd /path/to/proj4/mirror
git-svn-mirror.py update

or from any folder but with workbench location pointed explicitly:

git-svn-mirror.py update -w /path/to/proj4/mirror

Feedback, bug reports and patches highly appreciated.

Finally, big thanks to Eloy Durán for the original git-svn-mirror written in Ruby.

I have very little experience with Boost.Build v2 internals, so the patch may need further improvements. It works in my environment on Windows 7 (64-bit) with Visual Studio 2010 Express Edition installed alongside the Visual Studio 11 Developer Preview.

If you want to use Visual Studio 11 as Boost.Build toolset, just put the following entry in your user-config.jam file:

using msvc : 11.0 ;

To verify if the toolset is recognised correctly, use b2 debugging options:

b2 --debug-configuration --debug-building --debug-generator buffer.cpp
notice: found boost-build.jam at D:/dev/boost/_svn/trunk/boost-build.jam
notice: loading Boost.Build from D:/dev/boost/_svn/trunk/tools/build/v2
notice: Searching C:\Windows C:\Users\mloskot C:\Users\mloskot D:\dev\boost\_svn\trunk\tools/build/v2 D:/dev/boost/_svn/trunk/tools/build/v2/kernel D:/dev/
boost/_svn/trunk/tools/build/v2/util D:/dev/boost/_svn/trunk/tools/build/v2/build D:/dev/boost/_svn/trunk/tools/build/v2/tools D:/dev/boost/_svn/trunk/tool
s/build/v2/contrib D:/dev/boost/_svn/trunk/tools/build/v2/. for site-config configuration file site-config.jam .
notice: Loading site-config configuration file site-config.jam from D:/dev/boost/_svn/trunk/tools/build/v2/site-config.jam .
notice: Searching C:\Users\mloskot C:\Users\mloskot D:\dev\boost\_svn\trunk\tools/build/v2 D:/dev/boost/_svn/trunk/tools/build/v2/kernel D:/dev/boost/_svn/
trunk/tools/build/v2/util D:/dev/boost/_svn/trunk/tools/build/v2/build D:/dev/boost/_svn/trunk/tools/build/v2/tools D:/dev/boost/_svn/trunk/tools/build/v2/
contrib D:/dev/boost/_svn/trunk/tools/build/v2/. for user-config configuration file user-config.jam .
notice: Loading user-config configuration file user-config.jam from C:/Users/mloskot/user-config.jam .
notice: [msvc-cfg] msvc-11.0 detected, command: 'C:\Program Files (x86)\Microsoft Visual Studio 11.0\VC\bin\cl.exe'
notice: [msvc-cfg] msvc-10.0 detected, command: 'C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\bin\cl.exe'
notice: [msvc-cfg] msvc-10.0express detected, command: 'C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\bin\cl.exe'
notice: will use 'C:\Program Files (x86)\Microsoft Visual Studio 11.0\VC\bin\cl.exe' for msvc, condition <toolset>msvc-11.0
notice: [msvc-cfg] condition: '<toolset>msvc-11.0/<architecture>/<address-model>', setup: 'call "C:\Program Files (x86)\Microsoft Visual Studio 11.0\VC\vcv
arsall.bat" x86 >nul
...

and look for the following message reported:

notice: will use 'C:\Program Files (x86)\Microsoft Visual Studio 11.0\VC\bin\cl.exe' for msvc, condition <toolset>msvc-11.0

# catcher code
import sys
class StdoutCatcher:
   def __init__(self):
      self.data = ''
   def write(self, stuff):
      self.data = self.data + stuff
catcher = StdoutCatcher()
sys.stdout = catcher

This Python code can be executed by embedded Python interpreter using PyRun_SimpleString, then the output can be accessed by fetching __main__ module attributes:

PyObject* m = PyImport_AddModule("__main__");
char const* code = "... catcher code here...";
PyRun_SimpleString(code);
PyRun_SimpleString("print(3.14)");
PyObject* catcher = PyObject_GetAttrString(m, "catcher");
PyObject* output = PyObject_GetAttrString(catcher, "data");
// get textual data contained in output

Such mix of Python and C code is neither convenient to use nor states a flexible solution. I simply don’t like this prosthesis, especially if I need to frequently switch between number output sinks.

So, I have come up with better solution which allows me to directly bind any callable C++ entity. The syntax I mean looks and feels like this:

int main()
{
    PyImport_AppendInittab("emb", emb::PyInit_emb);
    Py_Initialize();
    PyImport_ImportModule("emb");

    PyRun_SimpleString("print(\'hello to console\')");

    // here comes the ***magic***
    std::string buffer;
    {
        // switch sys.stdout to custom handler
        emb::stdout_write_type write =
            [&buffer] (std::string s) { buffer += s; };

        emb::set_stdout(write);
        PyRun_SimpleString("print(\'hello to buffer\')");
        PyRun_SimpleString("print(3.14)");
        PyRun_SimpleString("print(\'still talking to buffer\')");
        emb::reset_stdout();
    }

    PyRun_SimpleString("print(\'hello to console again\')");
    Py_Finalize();

    // output what was written to buffer object
    std::clog << buffer << std::endl;
}

This allows me to handle sys.stdout.write with C++ free function, class member function, named function objects or even anonymous functions as in the example above where I use C++11 lambda.

Complete implementation of the emb module in C/C++ using plain Python C API is available from my Python workshop at GitHub:

git clone git://github.com/mloskot/workshop.git

The complete code is enclosed in python/emb/emb.cpp file. Note, this is a minimal example to present the essential concept. In production-ready code, it certainly needs more attention around reference counting of PyObject, getting rid of global state, and so one.

Recently, I’ve started using Visual Studio 11 Developer Preview and discovered the new style debugger visualizers this new edition introduces. I decided to port Barend’s work to the new XML-based format in .natvis files.

All my visualizers, not only for Boost.Geometry but also for other libraries, are available in visualstudio11 Git repository on my GitHub. The .natvis files are not overly complicated and the installation is dead easy, see README file attached.

Comments, suggestions and fixes are welcome.

Meanwhile, I have become annoyed by manually fixing the solution files. Here is quick .bat script I call from my cmake-{project}.bat proxy scripts I maintain for every project configured with CMake. So the fix is applied automatically every time I run CMake:

@echo off
REM Fix broken Visual Studio 11 .sln generated by CMake 2.8.6 (or earlier)
set SED="C:\Program Files (x86)\GnuWin32\bin\sed.exe"
set SLN=%1
copy %SLN% %SLN%.copy
%SED% "s/Studio 2011/Studio 11/g" %SLN%.copy > %SLN%
del /Q %SLN%.copy

In previous versions of the Visual Studio, all templates of debugger visualizers have to be saved in single file: %VSINSTALLDIR%\Common7\Packages\Debugger\autoexp.dat. In case one wants to extend or improve the templates definition, maintaining everything in one system-wide file is neither practical nor safe.

The visualizers in Visual Studio 11 seem to solve the maintenance hassle by allowing more modular approach. First nice feature is that custom templates can be defined without touching any of the installed Visual Studio files and can be put in %USERPROFILE%\Documents\Visual Studio 11\Visualizers. Second nice feature I’ve just discovered is that definitions of visualizers can be split into several separate .natvis files. It is especially useful if one wants to maintain visualizers grouped according to libraries (e.g. Loki, Boost, POCO).

Basically, %USERPROFILE%\Documents\Visual Studio 11\Visualizers works like a package of custom visualizers now:

Having the visualizers diagnostics enabled, whole process of parsing and loading .natvis files will be traced in Visual Studio 11 output window:

No more copying and pasting debugger visualizers for Boost and others into big muddy autoexp.dat.

BTW, who knows where does NATVIS come from, hmm? :-)

Unfortunately, it seems the new debugger visualizers have not been documented anywhere. At the time of writing this blog, Google reports one link about defaultvis.natvis, it is the brief post on Visual C++ forum.

Luckily, there is a short manual included as comment in header of the %ProgramFiles%\Microsoft Visual Studio 11.0\Common7\Packages\Debugger\Visualizers\defaultvis.natvis file. It says:

1) give better diagnostic reporting.
a) Set the name-value entry under the Debugger registry key: EnableNatvisDiagnostics which is a REG_DWORD to a value of 1 this will output each expression string that is evaluated under the native visualizer into the output window and is good for debugging type definition typos in .natvis files.

The first thing is to figure out where to set the EnableNatvisDiagnostics flag. The Visual Studio 11 installer does not put it into the registry. Also, there are several Debugger keys. However, a few experiments proved the flag should be set under HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\VisualStudio\11.0\Debugger:

Once it is set, every time Visual Studio 11 debugger is launched and use of visualized types is determined, the .natvis files are parsed and evaluated templates validated. The process is reported in the Output window:

If anything goes wrong, a type definition template is incorrect and its evaluation fails, it is also reported in the Output window:

It will be a very useful feature especially when defining visualizers of complex types. Previous versions of Visual Studio used old format of templates defined in autoexp.dat file which stated its own DSL based on regular expressions, so the syntax was complex and fragile, and as Andy Pennel has confessed, not documented. (Here kudos to Avery Lee who did great job reverse-engineering the autoexp.dat syntax.)

At least, this time folks get the XML Schema for the Visual Studio debugger visualizers.