Educating the world

Our blog has over 10,000 readers a month

Conditionally replacing line contents

July 17th, 2014

I answer a lot of Unix shell scripting questions for friends (I know, sad isn't it) but as I haven't written a blog article for a while I thought I document my thoughts in case the rest of world needs something similar.

Question

have a file of IP addresses, bit like this

192.168.20.0/24
192.167.20.254
10.0.2.254
216.200.119.0/16

basically if the line doesn't have a / in it, I want to add a /32 at
the end, can you do this in vi at all ?

I've been titting around with grep -v to do an inverse search, but
can't get sed to adjust the file in place etc. hence I think VI might
be better

Answer
There are many solutions to this kind of problem. Let's assume that the network addresses are contained in a file called file.txt.

If the order was not important then a really simple solution would be this:

$ grep -v "/" file.txt | sed 's>$>/32>' > out.txt
$ grep "/" file.txt >> out.txt

  • Filter file.txt and only give back the lines without a "/" in them. For each of those replace the end with "/32" and put the contents in the file out.txt.
  • Line 2 filters the file.txt, giving back the lines with a "/" in them and then append those lines to the file out.txt.

You will loses the order as doctored lines will appear at the top and undoctored lines will appear at the bottom. If the values were sorted you could just sort out.txt at the end and restore the order (eg sort out.txt > out.sorted.txt).

The trouble with editing the file "in place" is that there is a condition rule for each line. I'm not sure you could do it using vi or ed because you are not dealing with a range of lines in a buffer which is generally how those applications work.

I typically solve this kind of problem with a bit of awk. Awk was one of the precursors to Perl but I prefer it. Perl just became a monolithic mess of difficult to remember syntax for no real benefit over what awk already did.

$ awk '{ if ( $0 ~ /\// ) print $0; else print $0 "/32" }' < file.txt > out.txt

  • Take the file.txt as input, if the line contains a "/" print the line untouched else print the line with "/32" on the end. Do this for each line and put the output in out.txt.

The condition (in the if) is a regular expression match so that everything within the // is used in the match, in our case we are looking for a "/" so we must delimited it with a backslash \/.

There a little awk tutorial here that you might find useful:
http://www.grymoire.com/Unix/Awk.html

I'm going to New York, where should I go?

June 9th, 2014

I've probably had a months worth of holidays and business trips to New York over the last couple of years and have got to know the place pretty well. So I've compiled this list of cul'cha, tours and cheap things to do that I just roll out whenever anyone asks me what to do in New York City.

1) New York City Duck Tour

http://www.lovetoeatandtravel.com/site/us/ny/ny_tours/duck.htm
https://www.google.co.uk/search?q=duck+amphibious+vehicle+tour&tbm=isch
All the information I found about it said it has been discontinued but that was a couple of years ago. There might be something similar that has replaced it. Basically they drive you around New York pointing out the sights in an amphibious DUKW (pronounced "duck") vehicle then it drives into the Hudson and you take a boat ride along Manhattan Island then back the the start. It cost about $20 each but loads of fun.

2) The Staten Island Ferry

http://www.siferry.com/
Costs about $5 to get to Staten Island then it's free to come back. It takes you right past the Statue of Liberty and is about 20 times cheaper than a Statue of Liberty cruise. You can come back anytime so you can have a look around Staten Island. There's a couple of nice cafes once you get out of the ferry port.

3) Guggenheim Art museum

http://www.guggenheim.org/new-york
Really nice spiral staircase and there's usually something good on.

4) The Metropolitan Museum of Art

http://www.metmuseum.org/
Set on the edge of Central Park (the nice end). Lovely gardens out the back.

5) Fancy a run?

http://www.centralpark.com/guide/sports/running.html
Most of the joggers run around "Jacqueline Kennedy Onassis Reservoir" which is the largest body of water in Central Park. There is a path that goes all the way around with plenty of water fountains along the route. It's about 1.5 miles around so you shouldn't have a problem. There's plenty to see along the way.

6) Central park open air theatre

http://www.centralparknyc.org/things-to-see-and-do/attractions/delacorte-theater.html
The only problem is that tickets are like gold dust. They sell them on the day and people start queuing really early so check before you go.

7) Grand Central Terminal (Station)

http://www.grandcentralterminal.com/
You've probably seen it in films but it got renovated and cleaned up a few years ago to a very high standard. Plenty of posh bars, coffee shops and nice places to eat.

8) Greenwich Village

http://en.wikipedia.org/wiki/Greenwich_Village
Bohemian part of New York. Lots of art shops, music shops, eateries.

9) Film and TV tours

http://onlocationtours.com/locations/ghostbusters/
http://onlocationtours.com/tour/sex-and-the-city/

10) Basildon Room - Waldorf Astoria

If you have been to Basildon Park (just outside Reading, UK) there is a huge mirror in one of the rooms and the tour guides say that the sister mirror is in the Waldorf Astoria Hotel. Well it is, in the Basildon Room and if you ask nicely at the desk they will let you in to see it (for free). It's a nice excuse to get you into the hotel which is a spectacle in itself.

Well that's some of the things I've enjoyed doing on my trips to NYC. Have fun and let me know how it goes.

Saving user name and password

March 28th, 2014

I have a log in form that requires a user name and password. The form won't remember my passwords because the submission form has autocomplete switched off. Here is an example of a similar form:

<form method="POST" name="loginform" autocomplete="off" action="/cgi-bin/security.cgi">
<input name="un" type="text" />
<input name="pw" type="password" autocomplete="off" />
</form>

This means that the person who created the login form has specifically prevented my browser from saving the form user names and passwords. This might be ok in the wild but if you have lots of test systems then it can be a bit of a pain. As my old pal says, "if it's got a chip in it, you can re-program it". This is where I learnt about Bookmarklets.

A bookmarklet is a bookmark stored in a web browser that contains JavaScript commands to extend the browser's functionality.

These are very handy for getting around the restrictions build into the D.O.M.. So from the example above we want to switch on the autocomplete for the form and the password field so that the web browser will allow me to save the form data ready for next time.

So here is what you do in Firefox but it should work in any browser.

  1. Right click on the bookmark bar and select New Bookmark.
  2. Give it a Name, doesn't matter what.
  3. In the Location enter the following Javascript:

    <a href="javascript:(function(){var fm=document.getElementsByTagName('loginform'); var pw=document.getElementsByName('pw'); for(i=0;i<fm.length;i++){fm[i].setAttribute('autocomplete','on');} for(i=0;i<pw.length;i++){pw[i].setAttribute('autocomplete','on');}})()">Autocomplete on</a>

  4. Click Add.
  5. Make sure your browser is set up to remember passwords then go to your logging page.
  6. Click the bookmark in your bookmark bar. This will alter the DOM and switch the auto complete attributes to on.
  7. Fill in the User Name and Password on the form and click Login.
  8. Your browser will ask you if you want to remember the credentials, so say that you do.

When you return to the page the browser will ignore the autocomplete in the form because it already has the values, so double click inside the User Name field and select the username. The password will be filled in automatically. Then all you need to do is click the Login buttons and you're in!

Who's who in Islam

March 12th, 2014

What's the difference between Sunnis and Shi'a in Islam?

BBC News report 20th December 2013:

The division in Islam goes back to a dispute over who should succeed the prophet Muhammad after his death in 632. Those who wanted his position to be inherited by his closes associate became Sunnis. Those who wanted him to be followed by his descendants became Shi'a.

Just is in the split in the Christian church between Catholics and Protestants, it's been as much about power as [it has been about] religion.

Windows XAMPP, Apache, PHP, libssl2 pain

February 25th, 2014

Just spent most of the day trying to get XAMPP, Apache, PHP, OpenSSL and libssl2.dll working on Windows. Yep, DLL hell as usual. In the end I found a pure PHP implementation of SSH which worked out of the box on Windows without a problem.

http://phpseclib.sourceforge.net/

Enjoy!