[cnsidero]

I have a cellSet that I wan't to perform some simple statistics on (as post not while the calculation is running), e.g. compute the mean and standard deviation of temperature on a set of cells. I'd rather do this with an OF'ish tool as my experiences haven't been too positive with Paraview in this regard. Also this something I'd like to automated so OF lends itself to this well.

It seems swak4Foam - with funkyDoCalc or possible the new funkyPythonPostProc - should do the trick but not quite sure how to do this.

I suppose this is question is for you Bernhard - just a gentle nudging in the right direction would be helpful.

Thanks, Chris

[gschaider]

Originally Posted by cnsidero

I have a cellSet that I wan't to perform some simple statistics on (as post not while the calculation is running), e.g. compute the mean and standard deviation of temperature on a set of cells. I'd rather do this with an OF'ish tool as my experiences haven't been too positive with Paraview in this regard. Also this something I'd like to automated so OF lends itself to this well.

It seems swak4Foam - with funkyDoCalc or possible the new funkyPythonPostProc - should do the trick but not quite sure how to do this.

I suppose this is question is for you Bernhard - just a gentle nudging in the right direction would be helpful.

Thanks, Chris

Disclaimer: a lot of the features here are not in the latest release. But I only discuss features that are in the publicly available development version

The absolutely easiest thing would be a new utility

fieldReport T -doSets -time 0: -nrOfQuantiles 4 -distributionDirectory TDistributions -csvName Tdata.csv

That prints a number of statistics (min, max, average, sums) to the terminal, writes distributions to a directory and generates a CSV-file with all the numbers. I think it is quite obvious from the options what is what.
What it doesn't do is calculate the standard-deviation. Instead with the nrOfQuantiles 4 it calculates the values for which 25%, 50% (==Median) and 75% of the values are smaller (further discussion of the feature see below). That utility is fine if you only need single numbers and want to post-process them yourself. And it might give you more numbers than you wanted (BTW: most numbers come in two variations. Unweighted or weighted - in this case with the cell size)

With funkyDoCalc you only get the numbers you asked for and you are more flexible. In the simplest case you write something like this in a dictionary (I'm doing this from memory so some keywords might be wrong - I only write this stuff. This doesn't mean that I know how to use it)

THeater {
    valueType cellSet;
    setName theHeater;
    expression "T";
    accumulations (
       min
       max
       average
       weightedAverage
       weightedQuantile0.25
       weightedQuantile0.75
    );
}

This misses the standard deviation but has the quantiles (which I think are a more stable measure for the width of the distribution anyway. For skewed distributions it might happen that average+stdDev>max). A standard-deviation weighted by cell-size (don't shoot me if this is wrong) might be

THeaterDev {
    valueType cellSet;
    setName theHeater;
    expression "sqrt(TSqAvg-TAvg*TAvg)";
    variables (
        "totalVolume=sum(vol());"
        "TAvg=sum(T*vol())/totalVolume;"
        "TSqAvg=sum(T*T*vol())/totalVolume;"
    );
    accumulations (
       min   // doesn't matter. The expression should be uniform
    );
}

The idea with funkyPythonPostProc is to calculate all necessary "raw" data with the swak-facilites and send them to Python as Numpy-arrays. Use case (and there is an example for this in the Sources) is for instance to do a linear regression of p dependent on x (p(x)=k x+d). The example goes on and collects and prints k and d as a function of time. Nice side-effect of that utility is that it can drop you to a python-shell where you can interactively play with numbers (using IPython - this is for the Conneseurs)

[cnsidero]

Bernard,

Good thing all I asked for was a "nudging" ... wow! Thanks for the super detailed reply. fieldReports will get me what I want right now and I'll migrate to using funkyDoCalc for future calculations.

funkyPythonPostProc looks beautiful. Had I known about it before I could have used it on a project I just finished up where I was doing some non-linear LSQ regressions using lmfit (which uses scipy.optimize.leastsq under the hood). I think it might have saved me the pain of using Paraview to strip away the regions of interest.

-Chris

[gschaider]

Originally Posted by cnsidero

Bernard,

Good thing all I asked for was a "nudging" ... wow! Thanks for the super detailed reply.

Well. I like to brag about my stuff No. Seriously. I think that graphical postprocessing is overrated compared to numbers. So I'm quite fond of these tools.

Originally Posted by cnsidero

fieldReports will get me what I want right now and I'll migrate to using funkyDoCalc for future calculations.

funkyPythonPostProc looks beautiful. Had I known about it before I could have used it on a project I just finished up where I was doing some non-linear LSQ regressions using lmfit (which uses scipy.optimize.leastsq under the hood). I think it might have saved me the pain of using Paraview to strip away the regions of interest.

Well. If it eases your pain: I only finished implementing FP^3 (don't know if that abbreviation will stick) a couple of weeks ago. Probably too late for your project anyway.

But feel free to experiment. I'd appreciate any feedback before roling it into a release

[cnsidero]

Originally Posted by gschaider

Nice side-effect of that utility is that it can drop you to a python-shell where you can interactively play with numbers (using IPython - this is for the Conneseurs)

You peaked my interest here. It's the manipulatedPitzDaily example me thinks. I set useNumpy and useIPython to true in controlDict but when I run it, these warnings appear at the beginning of the run:

--> FOAM Warning :
^[[?1034h--> FOAM Warning :
    From function pythonInterpreterWrapper::importLib(const word &name)
    in file pythonInterpreterWrapper.C at line 830
    Could not import IPython in "::adaptRelaxation"
--> FOAM Warning :
    From function pythonInterpreterWrapper::pythonInterpreterWrapper
    in file pythonInterpreterWrapper.C at line 76
    Importing of IPython failed. Falling back to regular shell for "::adaptRelaxation"
--> FOAM Warning :
    From function pythonInterpreterWrapper::importLib(const word &name)
    in file pythonInterpreterWrapper.C at line 830
    Could not import rlcompleter in "::adaptRelaxation"
--> FOAM Warning :
    From function pythonInterpreterWrapper::pythonInterpreterWrapper
    in file pythonInterpreterWrapper.C at line 334
    No dictionary 'importLibs' found in "::adaptRelaxation"

Is there some additional OF or shell variables I need set to get it to work?

-Chris

[gschaider]

Originally Posted by cnsidero

You peaked my interest here. It's the manipulatedPitzDaily example me thinks. I set useNumpy and useIPython to true in controlDict but when I run it, these warnings appear at the beginning of the run:

--> FOAM Warning :
^[[?1034h--> FOAM Warning :
    From function pythonInterpreterWrapper::importLib(const word &name)
    in file pythonInterpreterWrapper.C at line 830
    Could not import IPython in "::adaptRelaxation"
--> FOAM Warning :
    From function pythonInterpreterWrapper::pythonInterpreterWrapper
    in file pythonInterpreterWrapper.C at line 76
    Importing of IPython failed. Falling back to regular shell for "::adaptRelaxation"
--> FOAM Warning :
    From function pythonInterpreterWrapper::importLib(const word &name)
    in file pythonInterpreterWrapper.C at line 830
    Could not import rlcompleter in "::adaptRelaxation"
--> FOAM Warning :
    From function pythonInterpreterWrapper::pythonInterpreterWrapper
    in file pythonInterpreterWrapper.C at line 334
    No dictionary 'importLibs' found in "::adaptRelaxation"

Is there some additional OF or shell variables I need set to get it to work?

-Chris

No. In fact if unset useIPython will be assumed to be true (I added this as an opt-out). Usually I'd have said "Ah. No Ipython installed" but the fact that rlcompleter fails too points to a fundamental problem (or a reaaaally old Python-version. Python 2.4 already knows that lib).

But if you set

        interactiveAfterExecute true;

you get dropped to a very primitive Python-shell, right? On that shell try these things

import sys
print sys.version
print sys.path
import rlcompleter
import IPython
print IPython.__version__

and compare these to the behaviour you get on a "regular" Python-shell.

Also make sure that the libraries you set with the SWAK_PYTHON_LINK-variable in swakConfiguaration point to the "right" libs (I don't know a sure-fire recipe to find that out). My suspicion is that the linked python is not the same Python that the libs are installed for (if you can't import sys then we're in deep trouble. But I'm confident. After rlcompleter the code imports readline and THAT seems to work - or you didn't bother me with that warning)

[cnsidero]

Originally Posted by gschaider

        interactiveAfterExecute true;

you get dropped to a very primitive Python-shell, right?

Correct.

Originally Posted by gschaider

On that shell try these things

import sys
print sys.version
print sys.path
import rlcompleter
import IPython
print IPython.__version__

and compare these to the behaviour you get on a "regular" Python-shell.

From the python dropped into by swak:

>>> import sys
>>> print sys.version
2.6.6 (r266:84292, May 27 2013, 05:35:12)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-3)]
>>> print sys.path
['/usr/lib/python2.6/site-packages/pip-1.3.1-py2.6.egg', '/usr/lib/python2.6/site-packages/lmfit-0.7.2-py2.6.egg', '/usr/lib64/python2.6/site-packages/openmpi', '/usr/lib64/python26.zip', '/usr/lib64/python2.6', '/usr/lib64/python2.6/plat-linux2', '/usr/lib64/python2.6/lib-tk', '/usr/lib64/python2.6/lib-old', '/usr/lib64/python2.6/lib-dynload', '/usr/lib64/python2.6/site-packages', '/usr/lib64/python2.6/site-packages/gst-0.10', '/usr/lib64/python2.6/site-packages/gtk-2.0', '/usr/lib64/python2.6/site-packages/webkit-1.0', '/usr/lib64/python2.6/site-packages/wx-2.8-gtk2-unicode', '/usr/lib/python2.6/site-packages', '/usr/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg-info', '/usr/lib/python2.6/site-packages/IPython/Extensions']
>>> import rlcompleter
>>> import IPython
Traceback (most recent call last):
  File "test", line 1, in <module>

  File "/usr/lib/python2.6/site-packages/IPython/__init__.py", line 58, in <module>
    __import__(name,glob,loc,[])
  File "/usr/lib/python2.6/site-packages/IPython/Shell.py", line 37, in <module>
    from IPython import ultraTB, ipapi
  File "/usr/lib/python2.6/site-packages/IPython/ultraTB.py", line 93, in <module>
    from IPython import Debugger, PyColorize, ipapi
  File "/usr/lib/python2.6/site-packages/IPython/Debugger.py", line 34, in <module>
    from IPython import PyColorize, ColorANSI, ipapi
ImportError: cannot import name PyColorize
>>> print IPython.__version__
Traceback (most recent call last):
  File "test", line 1, in <module>

NameError: name 'IPython' is not defined

and regular python:

>>> import sys
>>> print sys.version
2.6.6 (r266:84292, May 27 2013, 05:35:12)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-3)]
>>> print sys.path
['', '/usr/lib/python2.6/site-packages/pip-1.3.1-py2.6.egg', '/usr/lib/python2.6/site-packages/lmfit-0.7.2-py2.6.egg', '/usr/lib64/python2.6/site-packages/openmpi', '/usr/lib64/python26.zip', '/usr/lib64/python2.6', '/usr/lib64/python2.6/plat-linux2', '/usr/lib64/python2.6/lib-tk', '/usr/lib64/python2.6/lib-old', '/usr/lib64/python2.6/lib-dynload', '/usr/lib64/python2.6/site-packages', '/usr/lib64/python2.6/site-packages/gst-0.10', '/usr/lib64/python2.6/site-packages/gtk-2.0', '/usr/lib64/python2.6/site-packages/webkit-1.0', '/usr/lib64/python2.6/site-packages/wx-2.8-gtk2-unicode', '/usr/lib/python2.6/site-packages', '/usr/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg-info']
>>> import rlcompleter
>>> import IPython
>>> print IPython.__version__
0.10
>>>

Difference seems to be when swak drops into the python it adds '/usr/lib/python2.6/site-packages/IPython/Extensions' to the path.

Originally Posted by gschaider

Also make sure that the libraries you set with the SWAK_PYTHON_LINK-variable in swakConfiguaration point to the "right" libs (I don't know a sure-fire recipe to find that out). My suspicion is that the linked python is not the same Python that the libs are installed for (if you can't import sys then we're in deep trouble. But I'm confident. After rlcompleter the code imports readline and THAT seems to work - or you didn't bother me with that warning)

This seems to be correct.

[gschaider]

Good news (or probably bad): you're not alone

Originally Posted by cnsidero

Correct.



From the python dropped into by swak:

>>> import sys
>>> print sys.version
2.6.6 (r266:84292, May 27 2013, 05:35:12)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-3)]
>>> print sys.path
['/usr/lib/python2.6/site-packages/pip-1.3.1-py2.6.egg', '/usr/lib/python2.6/site-packages/lmfit-0.7.2-py2.6.egg', '/usr/lib64/python2.6/site-packages/openmpi', '/usr/lib64/python26.zip', '/usr/lib64/python2.6', '/usr/lib64/python2.6/plat-linux2', '/usr/lib64/python2.6/lib-tk', '/usr/lib64/python2.6/lib-old', '/usr/lib64/python2.6/lib-dynload', '/usr/lib64/python2.6/site-packages', '/usr/lib64/python2.6/site-packages/gst-0.10', '/usr/lib64/python2.6/site-packages/gtk-2.0', '/usr/lib64/python2.6/site-packages/webkit-1.0', '/usr/lib64/python2.6/site-packages/wx-2.8-gtk2-unicode', '/usr/lib/python2.6/site-packages', '/usr/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg-info', '/usr/lib/python2.6/site-packages/IPython/Extensions']
>>> import rlcompleter
>>> import IPython
Traceback (most recent call last):
  File "test", line 1, in <module>

  File "/usr/lib/python2.6/site-packages/IPython/__init__.py", line 58, in <module>
    __import__(name,glob,loc,[])
  File "/usr/lib/python2.6/site-packages/IPython/Shell.py", line 37, in <module>
    from IPython import ultraTB, ipapi
  File "/usr/lib/python2.6/site-packages/IPython/ultraTB.py", line 93, in <module>
    from IPython import Debugger, PyColorize, ipapi
  File "/usr/lib/python2.6/site-packages/IPython/Debugger.py", line 34, in <module>
    from IPython import PyColorize, ColorANSI, ipapi
ImportError: cannot import name PyColorize
>>> print IPython.__version__
Traceback (most recent call last):
  File "test", line 1, in <module>

NameError: name 'IPython' is not defined

You're on RHEL 6, right? (versions look like my CentOS 6. Just the build date is later because ... it is CentOS)

Originally Posted by cnsidero

and regular python:

>>> import sys
>>> print sys.version
2.6.6 (r266:84292, May 27 2013, 05:35:12)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-3)]
>>> print sys.path
['', '/usr/lib/python2.6/site-packages/pip-1.3.1-py2.6.egg', '/usr/lib/python2.6/site-packages/lmfit-0.7.2-py2.6.egg', '/usr/lib64/python2.6/site-packages/openmpi', '/usr/lib64/python26.zip', '/usr/lib64/python2.6', '/usr/lib64/python2.6/plat-linux2', '/usr/lib64/python2.6/lib-tk', '/usr/lib64/python2.6/lib-old', '/usr/lib64/python2.6/lib-dynload', '/usr/lib64/python2.6/site-packages', '/usr/lib64/python2.6/site-packages/gst-0.10', '/usr/lib64/python2.6/site-packages/gtk-2.0', '/usr/lib64/python2.6/site-packages/webkit-1.0', '/usr/lib64/python2.6/site-packages/wx-2.8-gtk2-unicode', '/usr/lib/python2.6/site-packages', '/usr/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg-info']
>>> import rlcompleter
>>> import IPython
>>> print IPython.__version__
0.10
>>>

OK. This is quite a conservative IPython

Originally Posted by cnsidero

Difference seems to be when swak drops into the python it adds '/usr/lib/python2.6/site-packages/IPython/Extensions' to the path.

That is probably from the aborted IPython import (before your attempt)

Originally Posted by cnsidero

This seems to be correct.

Yep. The rlcompleter lead me to this. Sorry for doubting you.

The thing is: exactly the same thing happens to me on my CentOS-machine.

The story is the following: usually I develop new stuff on my MacBook (where Python and all are shiny and new). Additionally I test it on a virtual Ubüntü-machine. And when all is stable I test it on the Workstation in the office (with the conservative but stable CentOSG). I DID test a previous version on the CentOS machines. I somehow fixed that problem but it seems the intermediate updates have broken it again.

The problem is that the old IPython on RHEL-machines behaves a bit differently from the current one (I'm not sure. But the thing is that IPython is rather terminal-oriented and the old version doesn'T seem to like it if initialized with no terminal in sight. Fun fact: if you set 'useIPython false;' then rlcompleter seems to import without problems. If you then do 'import IPython' on the embedded Python-shell you get a different error message)

I'll look into this. But don't expect a solution this week. Sorry

[cnsidero]

Originally Posted by gschaider

Good news (or probably bad): you're not alone

You're on RHEL 6, right? (versions look like my CentOS 6. Just the build date is later because ... it is CentOS)

Yup.

Originally Posted by gschaider

OK. This is quite a conservative IPython

As you know, RHEL tends to quite conservative. In general, Python and the majority of it's modules are new enough but iPython and matplotlib tend to lack some of the new things and can cause issues. I run Fedora 19 in a virtual machine which has newer everything for cases that I need it.

Originally Posted by gschaider

The thing is: exactly the same thing happens to me on my CentOS-machine.

The story is the following: usually I develop new stuff on my MacBook (where Python and all are shiny and new). Additionally I test it on a virtual Ubüntü-machine. And when all is stable I test it on the Workstation in the office (with the conservative but stable CentOSG). I DID test a previous version on the CentOS machines. I somehow fixed that problem but it seems the intermediate updates have broken it again.

The problem is that the old IPython on RHEL-machines behaves a bit differently from the current one (I'm not sure. But the thing is that IPython is rather terminal-oriented and the old version doesn'T seem to like it if initialized with no terminal in sight. Fun fact: if you set 'useIPython false;' then rlcompleter seems to import without problems. If you then do 'import IPython' on the embedded Python-shell you get a different error message)

I'll look into this. But don't expect a solution this week. Sorry

No expectations here. I'm just exploring it as it looks really interesting. I'll keep poking around it with and help where I can (testing, docs, etc). Let me know if there's anything in particular I can help with.

fieldReports and funkyDoCalc is more than meeting my needs for now. Looking into parameterizing their usage in a python script with PyFoam. Quick question in that regard, using the funky* expressions (or any other method), is there a way to query the number of cells in the internalField, boundaryField, or cell set?

Thanks again, Chris

[gschaider]

Originally Posted by cnsidero

As you know, RHEL tends to quite conservative. In general, Python and the majority of it's modules are new enough but iPython and matplotlib tend to lack some of the new things and can cause issues. I run Fedora 19 in a virtual machine which has newer everything for cases that I need it.

I've been experimenting with installing a separate python27, putting it into a virtualenv and install ipython, matplotlib, numpy and scipy in there (they install quite easily with pip there). The plan is to set SWAK_PYTHON_LINK to that. Only have to find a way to initialize it correctly to the virtualenv and then we're in business

Originally Posted by cnsidero

No expectations here. I'm just exploring it as it looks really interesting. I'll keep poking around it with and help where I can (testing, docs, etc). Let me know if there's anything in particular I can help with.

Well. Pointing out that it fails on a production-Linux (as opposed to a "everything works right after installation but not 3 months later"-Ubuntu) was helpful enough

Originally Posted by cnsidero

fieldReports and funkyDoCalc is more than meeting my needs for now. Looking into parameterizing their usage in a python script with PyFoam. Quick question in that regard, using the funky* expressions (or any other method), is there a way to query the number of cells in the internalField, boundaryField, or cell set?

There is no special function. The workaround I'd use is "sum(1)" (works everywhere. Mind: in parallel runs this is the number of cells on ALL processor - not just one - but that is what one usually wants I'd think)

Just discovered during writing: I recently implemented it in the accumulators (size) but it is not available in expressions. fieldReport prints it automatically

[gschaider]

Originally Posted by cnsidero

Yup.



As you know, RHEL tends to quite conservative. In general, Python and the majority of it's modules are new enough but iPython and matplotlib tend to lack some of the new things and can cause issues. I run Fedora 19 in a virtual machine which has newer everything for cases that I need it.



No expectations here. I'm just exploring it as it looks really interesting. I'll keep poking around it with and help where I can (testing, docs, etc). Let me know if there's anything in particular I can help with.

Just to let you (and everybody who might stumble upon this thread) know: the development-version is now fixed: old IPython will work too (the problem was basically that is expected a sys.argv which is not there in an embedded Python. On the other hand new IPython versions us the non-existence of that to determine exactly whether the this is an embedded run)

Anyway: what doesn't work in old IPython is tab-completion (in new one it does). This seems to be a common problem: Although tab-completion is configured in the embeded standard-shell it doesn't work there either (although it does the same things that make it work in a non-embedded python-shell). So: tab-completion only in new IPython

[cnsidero]

I updated and rebuilt. Running the tutorial drops me into an iPython shell successfully.

However when I list the variables with "whos" it reports "Interactive namespace is empty". A quick search on stackoverflow suggests iPython needs to be invoked with -i parameter.

[gschaider]

Originally Posted by cnsidero

I updated and rebuilt. Running the tutorial drops me into an iPython shell successfully.

However when I list the variables with "whos" it reports "Interactive namespace is empty". A quick search on stackoverflow suggests iPython needs to be invoked with -i parameter.

No. -i is if you run s script like this "ipython -i test.py" that when the script ends you're dropped to the ipython-shell instead of the terminal-shell

I'm quite surprised that %whos lists the variables with a new IPython (As I understand it %whos only shows the variables that were interactively defined on the shell and all these variables were injected not from the shell). And with the old IPython this is not the case? (currently got no possibility to test it with an old one)