Educating the world

Our blog has over 10,000 readers a month

Compiling omniORB 4.1.5 and omniORBpy 3.5 with Python 2.7

January 24th, 2011

Python 2.7 (final) was released on 3rd July 2010, which is almost half a year ago. One of the awful things about Python is that major releases are not compatible with each other. That is the extension modules compiled against Python 2.6 do not play nicely with the extension modules compiled against Python 2.7. The upshot of this, is that if you want to upgrade your version of Python you have to hunt around looking for extension modules compiled against your new version. In spite of Python 2.7 being 7 months old (1 year + 1 month if you count back to the release of Python 2.7 alpha 1), there isn’t as much extension module support as you’d expect; you still have to hunt and most of it is on unofficial websites i.e. not in the download pages of its project.

One such project is the CORBA implementation omniORB. It’s download pages only contain the Python 2.6 version. While I can use ./configure to build projects under unix with ease the building of omniORBpy under windows was much more troublesome. I can see why no one has attempted it thus far.

So for all you lucky people out there here is how you do it.

You need to have Python installed but you probably have that already so I’m not going to talk about that here.

We need to compile the omniORB library first then we can compile the omniORBpy against that.

Firstly we must create a directory to download our source software into:

mkdir C:\python-modules

Then download and unpack the source code. omniORB is housed at SourceForge so go to the download page at:

http://sourceforge.net/projects/omniorb/files/

and download the latest omniORB (omniORB-4.1.5.tar.bz2) and omniORBpy (omniORBpy-3.5.tar.bz2)
We also need the openssl implementation which can be downloaded from:

http://code.google.com/p/openssl-for-windows/downloads/list

The latest one is openssl-0.9.8k_WIN32 (at the time of writing).

Unpack them all into C:\python-modules so that we have three folders C:\python-modules\omniORB-4.1.5, C:\python-modules\omniORBpy-3.5 and C:\python-modules\openssl-0.9.8k_WIN32.

omniORBpy-3.5 needs to be in a special place to move it to C:\python-modules\omniORB-4.1.5\src\lib\omniORBpy (removing the version number from the end of the directory name).

The build process needs Cygwin which I already had installed. I tried to use it in the build but it didn’t work, the “make” just hung. So I just stuck to the instructions and used omniORB’s cut-down version. You can download this from http://omniorb.sourceforge.net/att_packages/gnu-win32-lite.zip and unpack it into c:\gnuwin32.

Now that we have all the software downloaded, we can start configuring the build.

Edit C:\python-modules\omniORB-4.1.5\config\config.mk and uncomment the platform you are using in my case it is x86_win32_vs_9
Next load the platform specific file from C:\python-modules\omniORB-4.1.5\mk\platforms\x86_win32_vs_9.mk. As we will be building in a Cygwin environment, we need to specify paths in Cygwin format. Change the variable PYTHON to point to the correct location of the Python executable. In my case I have Python 2.7 installed (but anything above Python 1.5 is fine)

PYTHON = /cygdrive/c/Python27/python

OpenSSL transport support is disabled by default because you need the Open SSL libraries which we have now so uncomment the OPEN_SSL_ROOT location. The path has to be a relative path. I tried all combinations of windows-style full paths and cygwin-style full paths and none of them worked. “make clean” doesn’t reset the build like it should do :( So if there is a problem during the build you have to unpack the sources from the zips again to create a fresh source tree. Set the ssl root to:

OPEN_SSL_ROOT = ../../../../../../openssl-0.9.8k_WIN32

which is a relative reference to C:\python-modules\openssl-0.9.8k_WIN32 from C:\python-modules\omniORB-4.1.5\src\lib\omniORB\orbcore\ssl !

Now that we are configured we can actually start the build.

Open a command window and add c:\gnuwin32 to the start of your path:

set PATH=c:\gnuwin32\bin;%PATH%

Run the visual studio batch file to set up the build environment:

“C:\Program Files\Microsoft Visual Studio 9.0\VC\vcvarsall.bat”

Not quite sure what this does but you need it. I think it has something to do with translating cygwin paths to windows paths

C:\gnuwin32\bin\checkmounts C:\gnuwin32

Change to the omniORB source directory and run the build:

cd C:\python-modules\omniORB-4.1.5\src
make export

The build of omniORB will take about 30 minutes.

Next up is to compile the omniORBpy. Change to the omniORBpy source directory and run the build from there:

cd C:\python-modules\omniORB-4.1.5\src\lib\omniORBpy
make export

The build of omniORBpy will take about 3 minutes.

Once the build has finished you need to create a module extension package so that you can give it to other people so they don’t have to go through all the hassle of downloading and compiling it all up.

The pre-built omniORBpy using Python 2.6 resident on SourceForge has a problem which means you need to adjust your PYTHONPATH and your actual PATH in order to use it. I use Clearcase which has everything on different drives so having to hard code drive letters made it almost impossible to use in a shared environment. Fear not I have done the 2 hours of reading really boring documentation so you don’t have to and constructed a omniORBpy 3.5 (omniORB 4.1.5) with Python 2.7 module extension that doesn’t require you to update your PATH or PYTHONPATH by making use of the site pth features and the nugget of information I found buried on a forum that said that a pyd (Python dynamic library) would pick up DLLs from the same directory it lived in automatically.

Unfortunately the build system is not set up enough so that you can make an installable deliverable to give out like you get with some projects (for example lxml). This means you have to do it by hand and it’s a real pain in the arse!

I’ve written a program to make the omniORBpy extension placing everything in the right place. It’s a bit rough but it works. You can download it from here:

http://www.bigsoft.co.uk/blog_images/omniORBpy-make-dist.py.txt

Well, thanks for reading all the way to the end (if you did!) and your prize is omniORBpy 3.5 with omniORB 4.1.5 binding using Python 2.7.1. Enjoy.

Installation instructions

  1. Download http://www.bigsoft.co.uk/blog_images/omniORBpy-3.5.win32-py2.7.zip
  2. Unpack omniORBpy-3.5.win32-py2.7.zip into C:\Python27\Lib\site-packages

Simple Python example programs with explanations

January 19th, 2011

Been doing a lot of Python programming recently and have found that I keep writing the same kinds of utilities. I thought I’d define them all here in functions so that I could cut and paste them as and when I need them. If you’re lucky I’ll write a little explanation (or tutorial) of how they work for the novice Pythoners out there.

Display sorted environment

Code

import os
def printenv():
  s = sorted(os.environ.items())
  for e in s:
    print "%s=%s" % e

os.environ is a map (collection of key/value pairs) so items turns it into a list of tuples so that it can be iterated over in a for loop. Each tuple contains the key and value of the map entry. The body of the for loop could also be written:

Code

map_key = s[0]
map_value = s[1]
print "%s=%s" % (map_key, map_value)

or

Code

map_key, map_value = s
print "%s=%s" % (map_key, map_value)

but you would only be adding extra stages.

Editting the N900 custom dictionary

January 11th, 2011

Nokia’s N900 has quite a nifty way of helping you type. If you enter a word during an SMS, an email or on a web page it tries to predict what the word will be an gives you the option to auto-complete.

The phone has 3 dictionaries to help do this. The dictionaries have a precedence. As you type, the word is checked against the personal dictionary which is a list of words that you have only used a couple of times, these are generally recent words. If the sequence is not in there, then it is checked against the used dictionary which contains personal words that have been reused or selected from a previous auto-complete. If it isn’t in there it is checked against the uneditable system dictionary. As you type more letters you are more likely to get to the one you are looking for or the end of the word. If the end of the word was reached and the word is not a member of any of the dictionaries then it is added to the personal dictionary. Neat!

One of the problems with this is that the personal dictionary usually fills up with typos and it becomes really easy to turn those typos into used dictionary entries. Another problem is that the Nokia N900 didn’t have any way to edit the auto-complete dictionary other than to delete the dictionary files, switch off predictive text and switch it back on again (because it cached the dictionaries in memory). The files are located here:

/home/user/.osso/dictionaries/.personal.dictionary
/home/user/.osso/dictionaries/.used.dictionary

As you can imagine this was a pain. I started writing an App to edit these files for you and then I got wind of one that was hidden in the develop-extras repository. It’s been in there for a while and hasn’t changed so I think it’s pretty stable.

Here’s a few instructions on installing it from the developer’s extra software repository:

  1. Load in the App Manager.
  2. From Uninstall/Download/Update screen open the drop down menu.
  3. Select Application catalogues.
  4. Select New.
  5. Enter Catalogue name: Maemo extras-devel
  6. Enter Web address: http://reposityory.maemo.org/extras-develop
  7. Enter Distribution: fremantle
  8. Enter Components: free non-free
  9. Make sure the Disabled box is unchecked
  10. Then click Save.
  11. The phone will then check for updates and download the latest lists from all of the repositories.
  12. Click the Download picture in the centre of the Application Manager.
  13. Click the Utilities section button.
  14. Type “au” to narrow the search and Auto-Complete Editor will appear in the list.
  15. Click the App, say that you agree with the Terms and Conditions and click Continue.
  16. The App will download and install in a few seconds.
  17. The App will appear as AutoComp Edit in the App list.

Splitting an MP4 into 2 halves

December 15th, 2010

I recorded a friend’s stag do on a new video camera that produced a 4.5 GB video file. I wanted to stream this down to the telly and play it on the Xbox but the wifi bandwidth wasn’t quite beefy enough, so it was stuttery.

I then entered a world of hell trying to put it onto a removable media but none of that worked.

  1. First I tried to copy the 4.5GB on to a FAT32 memory stick, but the file system format wouldn’t take a file bigger than 4GB.
  2. Then I formatted the memory stick to NTFS which would accept the extra large file, but the XBox wouldn’t read this - weird thanks Microsoft!
  3. Next I thought of DVDs. ULead DVD burning software can only burn Joliet for the data file system and this wouldn’t accept the large file size either.

Then I had a brainwave - yes another one! - why don’t I just cut the large MP4 file in half! So simple. Went for a quick search on the internet. I really had not realised how much crud there was out there to do this. Everyman and his dog wants $34.99 for some (really) crappy application that splits an AVI or MPG file into pieces. I tried a couple and most either didn’t work or had a demo that only split the first 2 minutes, not to mention all the really dodgie warez sites that kept popping up.

Then I had another brainwave - why don’t I use FFMPEG it’s free. The first problem was to find a build of it that runs under Windows and the second problem is out of the gazillion different command line switches and options which ones will I need. So here follows the splitting MP4s guide for anyone who doesn’t want to pay $34.99 for someone’s school project.

If you run Linux then you should be quite capable of downloading the source from SourceForge, and compiling it up yourself, so you can skip to the FFMPEG command line section below. If you are running Windows or MacOs then you probably need a really simple guide so hopefully this should do!

You will need a copy of 7-Zip to uncompress the deliverable before you can run it.

  1. Go to: http://ffmpeg.arrozcru.org/autobuilds/
  2. Download the Latest static builds for MacOs or Windows.
  3. Use 7-Zip to unpack the file into a local directory.
  4. That’s it, there’s no installer!

Ok, now that the Linux boys/girl has re-joined us again we’ll talk about how you split an MP4 file into 2 halves.

From the command line enter the command:

ffprobe stagdo-1080p.mp4

This should produce the duration of the video file (I’ve highlighted it, because I’m nice).

FFprobe version SVN-r25924, Copyright © 2007-2010 the FFmpeg developers
built on Dec 10 2010 04:13:08 with gcc 4.4.2
configuration: –enable-gpl –enable-version3 –enable-libgsm –enable-pthreads –enable-libvorbis –enable-libtheora –enable-libspeex –enable-libmp3lame –enable-libopenjpeg –enable-libschroedin
ger –enable-libopencore_amrwb –enable-libopencore_amrnb –enable-libvpx –disable-decoder=libvpx –arch=x86 –enable-runtime-cpudetect –enable-libxvid –enable-libx264 –extra-libs=’-lx264 -lpthrea
d’ –enable-librtmp –extra-libs=’-lrtmp -lpolarssl -lws2_32 -lwinmm’ –target-os=mingw32 –enable-avisynth –cross-prefix=i686-mingw32- –cc=’ccache i686-mingw32-gcc’ –enable-memalign-hack
libavutil 50.34. 0 / 50.34. 0
libavcore 0.16. 0 / 0.16. 0
libavcodec 52.99. 1 / 52.99. 1
libavformat 52.88. 0 / 52.88. 0
libavdevice 52. 2. 2 / 52. 2. 2
libavfilter 1.68. 1 / 1.68. 1
libswscale 0.12. 0 / 0.12. 0
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from ’stagdo-1080p.mp4′:
Metadata:
major_brand : isom
minor_version : 512
compatible_brands: isomiso2avc1mp41
creation_time : 1970-01-01 00:00:00
encoder : Lavf52.56.0
Duration: 02:28:07.92, start: 0.000000, bitrate: 4567 kb/s
Stream #0.0(eng): Video: h264, yuv420p, 1920x800 [PAR 1:1 DAR 12:5], 4400 kb/s, 23.98 fps, 23.98 tbr, 2997 tbn, 47.95 tbc
Metadata:
creation_time : 1970-01-01 00:00:00
Stream #0.1(eng): Audio: aac, 48000 Hz, stereo, s16, 159 kb/s
Metadata:
creation_time : 1970-01-01 00:00:00

From the example we can see the video is 2 hours 28 minutes 7 seconds and 92/100th of a second. We want 2 halves so divide by 2, if you want thirds you’ll have to divide by 3, if you want quarters…. just kidding!

A bit of arithmetic and we can work out the start and end times of each section you would like to copy out of the main MP4 file.

FFMPEG only ran on one of my 8 cores so if you open 2 command line boxes you can run them at the same time on different cores, so it took the same time to decode both halves as it would have done to decode one half.

ffmpeg -i stagdo-1080p.mp4 -ss 00:00:00 -t 01:04:06 -vcodec copy stagdo-1.mp4
ffmpeg -i stagdo-1080p.mp4 -ss 01:04:06 -t 01:24:06 -vcodec copy stagdo-2.mp4

      -i - input file name. I used MP4 but FFMPEG supports many different video and audio file formats.
      -ss - start time in hours, minutes and seconds.
      -t - time or duration in hours, minutes and seconds.
      -vcodec - video codec to use. We use copy which just copies the data exactly as it’s written.
      Finally the name of the output file.

For completeness here is the output of what the FFMPEG looks like:

FFmpeg version SVN-r25924, Copyright © 2000-2010 the FFmpeg developers
built on Dec 10 2010 04:13:08 with gcc 4.4.2
configuration: –enable-gpl –enable-version3 –enable-libgsm –enable-pthreads –enable-libvorbis –enable-libtheora –enable-libspeex –enable-libmp3lame –enable-libopenjpeg –enable-libschroedin
ger –enable-libopencore_amrwb –enable-libopencore_amrnb –enable-libvpx –disable-decoder=libvpx –arch=x86 –enable-runtime-cpudetect –enable-libxvid –enable-libx264 –extra-libs=’-lx264 -lpthrea
d’ –enable-librtmp –extra-libs=’-lrtmp -lpolarssl -lws2_32 -lwinmm’ –target-os=mingw32 –enable-avisynth –cross-prefix=i686-mingw32- –cc=’ccache i686-mingw32-gcc’ –enable-memalign-hack
libavutil 50.34. 0 / 50.34. 0
libavcore 0.16. 0 / 0.16. 0
libavcodec 52.99. 1 / 52.99. 1
libavformat 52.88. 0 / 52.88. 0
libavdevice 52. 2. 2 / 52. 2. 2
libavfilter 1.68. 1 / 1.68. 1
libswscale 0.12. 0 / 0.12. 0
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from ’stagdo-1080p.mp4′:
Metadata:
major_brand : isom
minor_version : 512
compatible_brands: isomiso2avc1mp41
creation_time : 1970-01-01 00:00:00
encoder : Lavf52.56.0
Duration: 02:28:07.92, start: 0.000000, bitrate: 4567 kb/s
Stream #0.0(eng): Video: h264, yuv420p, 1920x800 [PAR 1:1 DAR 12:5], 4400 kb/s, 23.98 fps, 23.98 tbr, 2997 tbn, 47.95 tbc
Metadata:
creation_time : 1970-01-01 00:00:00
Stream #0.1(eng): Audio: aac, 48000 Hz, stereo, s16, 159 kb/s
Metadata:
creation_time : 1970-01-01 00:00:00
Output #0, mp4, to ’stagdo-1.mp4′:
Metadata:
major_brand : isom
minor_version : 512
compatible_brands: isomiso2avc1mp41
creation_time : 1970-01-01 00:00:00
encoder : Lavf52.88.0
Stream #0.0(eng): Video: libx264, yuv420p, 1920x800 [PAR 1:1 DAR 12:5], q=2-31, 4400 kb/s, 2997 tbn, 23.98 tbc
Metadata:
creation_time : 1970-01-01 00:00:00
Stream #0.1(eng): Audio: aac, 48000 Hz, stereo, s16, 64 kb/s
Metadata:
creation_time : 1970-01-01 00:00:00
Stream mapping:
Stream #0.0 -> #0.0
Stream #0.1 -> #0.1
Press [q] to stop encoding
frame=92212 fps=251 q=-1.0 Lsize= 2044520kB time=3845.97 bitrate=4354.9kbits/s
video:2011880kB audio:30079kB global headers:0kB muxing overhead 0.125418%

Each 1 hour 4 minute (and 6 second) section took about 6 minutes to copy. Once the files were split they both fitted on my FAT32 memory stick.

The first half played very happily on the XBox, but unfortunately the second half was about 0.25 seconds out of audio sync, so if anyone knows what I can do to adjust the audio, I’d be grateful if they would leave a comment.

Laptops and PC desktops

December 9th, 2010

Being in I.T. means you suffer from the same problems as doctors - bare with me on this! You meet a doctor at a party and the first thing you ask is are you a GP (General Practitioner) as opposed to some weird area of medicine or biology, or even English literature (Phd) that you are going to know nothing about. If they say they are a G.P. then the second question is usually along the lines of “I’ve got this spot/growth/whatever” or “I’ve had this pain in my whatever” and the party for them turns into another day at the clinic.

Well being in I.T. can be a bit like that only it tends to be “I’ve got this problem with my soundcard” (if you were around in the 80s) or “I think I have a virus, can you fix it?” or “What is the best computer to buy?”

Well I thought I’d answer the final one here so I can refer the friends of friends away from me and towards my blog so I don’t have to talk to them any more.

First you must ask yourself if you would like a laptop, a desktop or dare I say it…. a tablet.

When you say “tablet” everyone thinks of the iPad but it depends on whether you are a form over function type of person who has bought into all the hype. If you are then you are only reading this article to hear someone say how wonderful they are, and you won’t find that here, so you might as well stop reading now. If you are a realist there are plenty of other laptops which are tablets as well, like the Dell Inspiron Duo. Tablets have come a long way in recent times so anything that supports the Android operating system will keep you happy. I quite like the ASUS Eee Pad which is a cross between a tablet and a netbook.

So now we are on the question: laptop or desktop?

These days there aren’t too many reasons why you would buy a desktop rather that a laptop. Now-a-days you can plug a bigger screen or keyboard into the laptop to make a desktop set-up. Laptops are generally more expensive, easier to steal and slightly slower.

For someone who is only interested in surfing/email/office, all computers these days are good enough. You won’t notice the difference between the top of the range and the bottom of the range. If you already have a computer the chances are that it is a couple of years old and has lost it sparkle. Literally, anything new will be about 10 or 20 times faster and better than what you currently have. It will be capable of running the operating system it comes with without any problems.

The main advantage of a laptop and a desktop is mobility; being able to carry it from the sitting room to the kitchen adds up to a more convenient experience.

Pretty much all laptops these days are much-of-a-muchness. When buying a laptop you just need to ask a couple of questions: does it look nice? is it nice to type on? is it not too heavy? The best way to answer these questions is to go somewhere like PC World or Comet and have a play. Pick something you like!