Start
   Blogaria
   Bored
   bsgen
   c-conf
   Cookies
   cycliclog
   Dialwhatever
   dnspb
   fch
   fe
   fswatch
   HammerServer
   jpeginfo
   kalk
   Lectures
   Microproxy
   msc
   Nasapics
   Off The Grid
   PGPkey
   Posters
   SafeEdit
   Simple listserv
   syscheck
   tcp-fwd
   Wallpapers
   xml tools
Karel as an adult



Blogaria

Just a collection of random rants that I can't really place anywhere else.

2016-07-20 Fswatch 1.09
2016-05-04 Another fswatch update
2016-03-24 fswatch updated
2016-02-22 OTG Price Claimed
2016-02-01 File System Watcher fswatch
2016-01-05 Commandline flag parsing in Go
2015-11-19 Phoning the Dutch Vehicle Registration
2015-11-09 FE 1.05 released
2015-09-28 Bad Horse
2015-09-24 A generic attribute holder class in Python
2015-08-05 Autorotating jpegs according to EXIF tags


2015-04-15 0MQ in Python

In the never-started-series-on-cool-modules-in-Python, here's 0MQ.

What's 0MQ?


0MQ (zero-MQ) is a really nice and stable networked message transport system. It has bindings for most popular langugages, and apparently is very widely used. Check out http://zeromq.org/ for a good overview. Of course there are bindings for Python too, so here'a s try-out.

Publish and subscribe


PUB/SUB is a very basic paradigm, where messages are published, and an arbitrary number of subscribers receives certain messages, based on the topic of the message.

Publishers

A 0MQ publisher is responsible for binding a publishing channel on a given TCP port. Next, objects are published, which subscribers can read. Each published message has a topic, which is simply a number, and a payload (buffer, string, JSON object, serialized Python object, whatever). Every message that the publisher makes available, must have both the topic and the payload. As far as the publisher is concerned, messages are simply "put on the line" and whoever wants them, receives them.

Here's a simple publisher that selects a random payload word (from the "lorem ipsum" dictionary) and a random topic, which is a number between 0 and 9. The message, consisting of the topic and the payload, is published on a TCP port that is stated on the commandline. Note also the somewhat liberal usage of the words socket and bind in the zmq module: these names suggest POSIX C-like behavior, while in fact, the zmq variants do much more. According to 0MQ's documentation, the bulk of the heavy lifting is done by zmq's socket module. In the case of the publisher, the socket is of -obviously- of the publishing type; hence, it's created using the type specification zmq.PUB.

The below publisher runs ad infinitum and pauses for 1 second between all messages in order to make the demo clearer.

#!/usr/bin/env python3

import random
import sys
import time
import zmq

txt = '''lorem ipsum dolor sit amet consectetur adipiscing elit sed
do eiusmod tempor incididunt ut labore et dolore magna aliqua ut enim
ad minim veniam quis nostrud exercitation ullamco laboris nisi ut
aliquip ex ea commodo consequat duis aute irure dolor in reprehenderit
in voluptate velit esse cillum dolore eu fugiat nulla pariatur
excepteur sint occaecat cupidatat non proident sunt in culpa qui
officia deserunt mollit anim id est laborum'''
words = []
for line in txt.split(sep='\n'):
    words += line.split()

if len(sys.argv) != 2:
    print('''Usage: publisher PORT
    PORT is the TCP port to send messages
    Generated topics are 0-9, payloads are random words''', file=sys.stderr)
    sys.exit(1)

ctx = zmq.Context()
sock = ctx.socket(zmq.PUB)
sock.bind('tcp://*:%s' % sys.argv[1])

while True:
    topic = random.randint(0, 9)
    payload = words[random.randint(0, len(words) - 1)]
    print('topic %d: payload %s' % (topic, payload))
    sock.send_string('%d %s' % (topic, payload))
    time.sleep(1)

Subscribers

Coding a subscriber (client) is even easier than the publisher. Again, a zmq socket is created and connected to the publisher's TCP address. There's a method socket.setsockopt_string(...) that allows the client to subscribe to only a given topic. In the below code fragment this is the second command line argument; the first being the TCP port of the publisher.

Note also that thea subscriber must create their socket using type zmq.SUB, which of course matches the pub/sub pair of a zmq.PUB-type publisher socket.

#!/usr/bin/env python3

import sys
import zmq

if len(sys.argv) != 3:
    print('''Usage: subscriber PORT TOPIC
    PORT   is the TCP port to listen to
    TOPIC  is the publisher-sent topic, 0-9''', file=sys.stderr)
    sys.exit(1)

ctx = zmq.Context()
sock = ctx.socket(zmq.SUB)
sock.connect('tcp://localhost:%s' % sys.argv[1])
sock.setsockopt_string(zmq.SUBSCRIBE, sys.argv[2])
print('Subscriber created on port %s, topic %s' % (sys.argv[1], sys.argv[2]))

while True:
    print(sock.recv_string())

Test

With the above code, it's possible to start one publisher and several subscribers, each interested in a topic numbered 0-9. For example, you could:
  • Start the publisher in one xterm window;
  • Start a subscriber in another xterm window with the second command line argument 1 to only receive messages with topic 1;
  • Start another subscriber in another xterm window with the second command line argument 2 to only receive messages with topic 2;
  • And so on.
You can try this out yourself, and/or you can have a look at the following screen recording. The upper left terminal is the publisher; the other ones are subsribed to topics 1, 2 and 3. As you see the topics 1, 2 or 3 being at random published, you see them being displayed in the subscribers.

Ventilator, workers and a sink


A second neat and hugely useful paradigm is that of a ventilator, workers and a sink. This builds up to a fully networked distributed task processing scheme:
  • There is one ventilator, which prepares tasks that should be executed and makes them available;
  • There are workers, which pick up the tasks and process them. Once they have a result, they make it available to the next step;
  • There is one sink where all workers' results are collected.
The really neat thing about this, is that there must be at least one worker, but there of course may be more, which increases parallelism and speeds up the processing of the total work. But there's more: workers can be added while the ventilator and the sink are running. New workers that spring into life simply grab the next task and start chugging along.

This is shown in the below diagram:

                >---------  Worker >--------  
	       /    	  	   	     \ 	    
              >-----------  Worker >-----------	    
	     /	 	    		       \    
  Ventilator >------------  Worker >----------> Sink
	     \	 	  		       /
	      >-----------  Worker >-----------	
	       \	  	 	     / 
		>---------  Worker >--------- 
					     

Socket types

Again, 0mq's socket types determine what goes where and how. Here two types are relevant: zmq.PUSH and zmq.PULL. The push-type socket makes information available to others; the pull-type socket is of course the reverse and collects information. Using this very simple to understand scheme, the socket types for all components are:
  • The ventilator requires a zmq.PUSH socket on a given TCP port where tasks are made available;
  • Each worker have two types of sockets: one zmq.PULL socket to fetch the ventilator's tasks, and one zmq.PUSH socket to make its results available to the sink. This second socket must also run on some TCP port, but different from the ventilator's socket;
  • Finally, the sink has a zmq.PULL socket to collect the workers' results.

An example

Onward to code. The following set of a ventilator, workers and a sink has the following (very simple) task:
  • The ventilator pushes to the workers a series of integers in the 10-million range. These values are chosen in this range to make sure that the workers' task requires some processing time.
  • The workers are responsible for determining whether the value that they pull from the ventilator is a prime or not. They do this in a very simplistic manner (it has to take time, right, else the demo wouldn't work well!). They simply divide the received value with all numbers between 2 and value-1. If the reminder of the division is zero, then the value isn't a prime; otherwise, it is. (Yes, I know that there are tons of optimizations already there, but that's not the point now.)

    When the value ends up being a prime, then the value is pushed to the sink.

  • Finally, the sink pulls prime numbers from the workers and displays them.

The Ventilator

Below is the code of the ventilator. It is started with one command line argument; the TCP port where values are pushed, and hence, where workers can pull them. There's a nice press enter to continue hurdle to ensure that you start the workers and the sink.
#!/usr/bin/env python3

import sys
import zmq

if len(sys.argv) != 2:
    print('Usage: ventilator PORT', file=sys.stderr)
    sys.exit(1)

ctx = zmq.Context()

print('Starting ventilator on port', sys.argv[1])    
sender = ctx.socket(zmq.PUSH)
sender.bind("tcp://*:%s" % sys.argv[1])

print('Start the sink, then some workers, once they run hit enter')
input()

for nr in range(10000000, 20000000):
    sender.send_string('%i' % nr)

print('All test numbers sent')

Workers

Workers are started with two command line arguments: the ventilator's port (from which values are pulled) and the sink's port (to which primes are pushed). Next, workers pull values from the ventilator and test them for primeness. In order to display its actions, a worker will print its process ID and primes on stdout.
#!/usr/bin/env python3

import os
import sys
import zmq

if len(sys.argv) != 3:
    print('Usage: worker VENTILATOR-PORT SINK_PORT', file=sys.stderr)
    sys.exit(1)

ctx = zmq.Context()

print('Connecting receiver to ventilator on port', sys.argv[1])
receiver = ctx.socket(zmq.PULL)
receiver.connect('tcp://localhost:%s' % sys.argv[1])

print('Connecting sender to sink on port', sys.argv[2])
sender = ctx.socket(zmq.PUSH)
sender.connect('tcp://localhost:%s' % sys.argv[2])

print('Worker ready, ventilator can start pushing')

while True:
    nr = int(receiver.recv())
    isprime = 1
    for tester in range(2, nr - 1):
        if nr % tester == 0:
            isprime = 0
            break
    if isprime:
        print('Worker at PID %d: %d is a prime' % (os.getpid(), nr))
        sender.send_string('%d' % nr)

The Sink

The sink is a very simple end point component. All it does, is connect to the workers' socket, pull results, and display them. There is one command line argument: the workers' TCP port.
#!/usr/bin/env python3

import sys
import zmq

if len(sys.argv) != 2:
    print('Usage: sink PORT', file=sys.stderr)
    sys.exit(1)

ctx = zmq.Context()

print('Starting sink on port', sys.argv[1])
receiver = ctx.socket(zmq.PULL)
receiver.bind("tcp://*:%s" % sys.argv[1])

while True:
    print(receiver.recv_string(), 'is a prime')   

Demo

In order to try this out, you should at least start two workers, otherwise there will be no parallelism. For fun, you can add a third worker while the ventilator is chugging along. In the below screen capture there are four windows: the ventilator, the sink, and two workers. Enjoy!

More stuff


There is of course a ton of more information. The above two examples barely scratch the surface. If this has whetted your appetite, visit zeromq.org and enjoy!

2015-04-13 FE update
2015-02-23 Perl Course notes moved
2015-01-13 For my dad
2014-12-11 Peugeot 307 te koop
2014-11-20 Combining tornado.web StaticFileHandler with dynamic handlers
2014-11-05 Pushing scripts to servers using ssh
2014-11-03 Python3 argparse Mini HOWTO
2014-10-30 Handling secure httponly cookies with Python3 and tornado.web
2014-10-29 Overloading libc Calls
2014-10-27 Python Bootcamp Notes
2014-10-06 Custom Squid Proxy Authentication
2014-08-30 CyclicLog
2014-05-02 Playing with Python
2014-04-08 Kalk 1.38
2014-03-18 Closing C++ lecture
2014-02-26 Spring Cleanup
2014-02-26 Gravity error
2014-02-03 More on the knight tour
2014-01-28 The circular knight tour
2014-01-24 A Perl IBAN Checker
2013-10-16 Go2SEPA is live
2013-07-24 AJAXy Fileuploads with JQuery
2013-05-23 The Ticket
2013-04-24 Who is leeching my Facebook
2013-04-15 Time for a compact Coolpix
2013-04-15 Selling my Peavey amp
2013-03-28 encfs Helper Revisited
2013-03-15 c conf 1.17
2013-02-25 Flitsers op de Nederlandse wegen
2013-01-14 OTG Revisited
2013-01-11 Kalk revisited
2012-12-12 God is coming
2012-11-29 Strangely disturbing
2012-11-27 How to sneeze on a motorbike
2012-11-23 Een aanslag uit de toekomst
2012-11-22 More voluntary layoffs
2012-10-17 Browser fingerprint
2012-10-15 Site layout revamped
2012-10-14 Crossroads performing well
2012-10-08 The saddle seat
2012-09-30 Sudo Exploit
2012-09-26 Technomona is live
2012-09-19 A Perl to HTML Prettyprinter
2012-09-17 Playing with Moose
2012-08-22 How do you export photos from Adobe Photoshop Elements
2012-08-21 Security hole at the Empire State Building
2012-07-15 Matrix Sunglasses
2012-07-12 Two Morgans
2012-07-12 Review of the BMW R1200RT Motorbike
2012-06-13 Passwords: maak het hackers moeilijk
2012-06-13 Mijndomein foutje
2012-06-07 YOLNT
2012-06-06 Human Origins
2012-05-23 Me on Facebook
2012-05-16 Voluntary Layoffs
2012-05-07 RBS Bike Tour
2012-04-27 Room with a view
2012-04-23 Browser cookies and Javascript revisited
2012-02-17 Schrodingers Cat
2012-02-14 KPN heeft problemen
2012-01-12 Memebase forever
2012-01-11 Strange squares
2011-12-22 TVV Ondernemingsportaalnl.com zuigt ezel
2011-12-08 Dilbert vs Skype
2011-11-29 The uncanny resilience of bulshytt
2011-11-23 Another silly Trojan attempt
2011-10-29 ACTA is coming our way
2011-10-28 Burgernet in the Netherlands
2011-10-27 Facepalm art
2011-10-26 Do not drag this image
2011-10-22 Off The Grid Challenge
2011-10-12 PI like a boss
2011-10-07 Once upon a time
2011-07-13 Dutch eticket system for trains
2011-07-12 Is Hell exothermic or endothermic
2011-04-27 Optical Illusions
2011-04-19 Odd lyrics
2011-04-16 Band Revival at MON
2011-03-13 Protests in the Middle East and you
2011-03-10 Mac OSX Hotkey for locking your system
2011-02-12 dnspb 0.06 is out
2011-02-08 Would I buy this fridge
2011-02-06 InstaYouth
2011-02-05 The Thinker is back
2011-01-17 Math challenge
2011-01-11 Zero tolerance and zero intelligence
2011-01-05 My interest income in 1991
2011-01-01 Your horoscope by Eddie
2010-12-22 New York City Tours might be half price for you
2010-12-20 Weather Forecast
2010-12-14 World Economy Collapse explained in 3 minutes
2010-12-13 The Salvation Army and its choice of toys
2010-12-08 Elizabeth thinks highly of me
2010-12-06 Should I trust my government with my data
2010-12-05 Announcing dnspb
2010-12-03 Realistic piechart
2010-11-26 Crossroads 2.71 is out
2010-11-24 8 bit Starwars
2010-11-17 Six to eight black men
2010-11-16 Canada wants backdoors and data and everything
2010-11-11 Autumn storm over the Netherlands
2010-10-08 USA wants backdoors to everything
2010-10-05 Sudoku solver in Perl
2010-10-02 Finally wrote up a Syscheck page
2010-09-28 Neon sign fail
2010-09-27 The Renault Eco Team
2010-09-23 Crossroads 2.68 is out
2010-09-20 How to suppress Flash cookies
2010-09-15 Meanwhile on Facebook
2010-09-09 The Yes Men Fix The World
2010-09-07 ed is not dead
2010-08-26 Installing Perl modules in a non root environment
2010-08-22 Magic self leviation
2010-08-20 Google Chrome does not support offline Gmail
2010-08-19 The number 48
2010-08-12 Welsh trout mini HOWTO
2010-08-04 Fooling a NetCache proxy into fetching forbidden files
2010-07-30 The world will end on May 21, 2011
2010-07-28 Hiding or showing a textbox with image animation using JQuery
2010-07-27 Manipulating browser cookies using Javascript
2010-07-25 Survival of the fittest book
2010-07-23 Pastafarians in Spain
2010-07-22 You have two sheep
2010-07-09 Highway bank fire
2010-07-08 Setting up a remote git repository
2010-07-06 Bye bye trusted old Macbook
2010-06-28 John Cleese on Football
2010-06-23 ABN Amro and the Pathetic Customer Service Dept.
2010-06-22 Wally does not like criticism
2010-06-14 Soccermatch Netherlands vs Denmark
2010-06-13 Lazy Cat
2010-06-08 Reading public Buzz using the Google API
2010-06-07 A Personal Letter from Steve Martin
2010-06-05 Sushi Saturday
2010-06-04 Suppressing the Enter key with Javascript
2010-05-31 Temporal spacial anomaly on the Dutch highway
2010-05-23 Greenhost will not log your traffic
2010-05-10 Jarlsberg Webapp Exploits
2010-05-04 A Thought Experiment
2010-05-03 SafeEdit information updated
2010-05-01 Microproxy now supports ftp
2010-04-30 What could get Data angry
2010-04-29 Lego Mindstorm solving the Rubik Cube
2010-04-28 Crossroads 2.65 is out
2010-04-17 Goggomobil in its natural habitat
2010-04-14 Bacon Time
2010-04-11 104 More friends to connect with
2010-04-10 Bacteria infested radio reporter
2010-04-07 The Kubat STAR
2010-03-30 Homework Essay
2010-03-29 C++ mutexes again
2010-03-20 Weird Eyechart
2010-03-15 Microproxy 1.01
2010-03-05 Microproxy
2010-03-03 Sven Kramer and the wrong lane
2010-02-26 Endearing Babe Magnet
2010-02-17 Speed of light measured using chocolate and a microwave
2010-02-17 Never again expires after 65 years
2010-02-16 encfs on the Mac
2010-02-15 Hyves.nl and sexual predators
2010-02-10 Funny textbook
2010-02-09 DNS failing after sleep wake cycle
2010-02-06 Blast from the past
2010-01-28 Simple and straight Perl HTTP::Proxy
2010-01-15 Avatar the Movie
2010-01-08 Slightly NSFW Linux Ad
2010-01-07 WTF
2010-01-05 Stop Software Patents in the EU
2009-12-05 HammerServer 1.02
2009-11-28 Perls Automagical Autoloading
2009-10-07 Office Poster
2009-10-06 The nr 1 Nerdjoke
2009-10-04 WoW Startscript for my Mac
2009-09-27 HammerServer section is online
2009-09-26 The BING HQ
2009-09-26 Digging a WOW Tunnel
2009-06-29 Wee Todd
2009-06-23 The On Off Switch Revisited
2009-06-22 Meatspace
2009-05-30 My old houses
2009-05-11 LOLcats are funny
2009-05-11 Civic Duty WIN
2009-05-10 Vote for the baby, Sky Radio promo FAIL
2009-05-05 My secure data center
2009-02-15 My Valentine is sending me a dot exe
2009-02-05 MacPorts trash: .mp_123456 savefiles cleaning
2009-02-01 Truecrypt 6 on Linux and the ext3 filesystem
2009-01-28 www versus nl.youtube.com
2009-01-27 Songsmith and The Police
2009-01-25 My own Ministery of Silly Walks
2009-01-09 CoolIris Mini HOWTO
2008-11-04 UDP and DNS balancing
2008-11-02 Life in graphs
2008-11-01 Skeined yet?
2008-10-30 New Crossroads on the horizon
2008-10-28 Thread safe or not
2008-10-15 WOW patch 3 on a case sensitive MacOSX filesystem
2008-10-15 Surprising C++ optimizations
2008-10-14 Weird system message
2008-10-08 Data mining against terrorism does not work
2008-09-16 Crossroads at the top of Freshmeat.net
2008-09-09 Stupid spammers at Computable
2008-09-06 Spam prevention with Postfix and Postgrey
2008-09-03 The Gnomish Flying Machine
2008-08-27 Bank customer data on eBay
2008-08-26 Mutexes in C++ Threads
2008-08-22 4M dataloss in the UK last year
2008-08-21 Dropping spam with Postfix and Spamassassin
2008-08-18 Bayes and the War on Photography
2008-08-13 Good marital advice
2008-08-12 Squid proxy for personal usage
2008-08-11 Posix threads in C++
2008-08-09 Crossroads mailing list
2008-08-08 Crossroads 2.00 is out
2008-08-01 Fail Pics
2008-07-14 The Fish Dance
2008-07-01 Big Bother and Massive Data Storage
2008-06-30 MMV One of omitted Unix tools
2008-06-08 Even anonymous breadcrumbs can give you away
2008-05-29 Crossroads in Argentina
2008-05-20 The Party at the Company Outing
2008-05-19 Crossroads 1.80 is out
2008-05-18 Where does technical innovation really come from
2008-05-16 Corporate bs generator
2008-05-15 Even the Vatican has to adapt
2008-05-12 Big Brother is watching your dog
2008-05-09 666 all over the place
2008-04-17 Security and privacy are incompatible
2008-04-16 The Hallmark E Card
2008-04-15 Crosroads Solaris port is out
2008-04-04 Identity theft can cost you dearly
2008-04-03 Crossroads can already do that
2008-03-31 A dagerous safari
2008-03-28 Why some Java J2EE projects are inefficient
2008-03-26 The Hummingbird
2008-03-25 The Easter delusion
2008-03-18 McAfee detects mass hack of 200.000 webpages
2008-03-17 More predictive statistics
2008-03-10 Backwards conclusions even on Slashdot
2008-02-18 A fractal photograph
2008-02-15 Kaprekar revisited
2008-02-14 Kaprekar numbers
2008-02-12 A tale of the criminal ineptitude
2008-02-10 Irritating Selfregistered users in PHPBB
2008-02-08 B2B Spam in the Netherlands
2008-02-06 Surprising iSight Capture
2008-02-05 Breadcrumbs at WickedLasers.com
2008-01-29 iSight Capture Utility
2008-01-28 The Male Brain
2008-01-26 Searching for the next Uri Geller
2008-01-24 Opt in for b2b spam
2008-01-14 Bokito Revisited
2008-01-13 Top Crossroads User
2008-01-12 World of Warcraft Dancing
2008-01-12 Justice dispensed better late than never
2008-01-11 Jeremy Clarkson and Identity Theft
2008-01-10 Terrorism in the Netherlands
2007-12-07 The mind and bodysnatchers are among us
2007-12-05 Bruce Schneier and Hildo
2007-12-04 Bye bye, good Christian soul
2007-12-03 Confusing mail message
2007-11-30 Medion MD 85276 reviewed
2007-11-29 Recent cases of data exposure
2007-11-20 Bayes bites
2007-11-19 Japan starts fingerprinting foreigners
2007-11-14 Privacy, Yahoo and the Strange World
2007-11-14 Privacy, Fall through algorithms, and Securing data
2007-11-07 European airlines to retain data
2007-11-03 BloggEd
2007-10-30 Wilders and Marktplaats.nl
2007-10-28 The goldplated Mac
2007-10-26 More morons
2007-10-26 Dilbert nails it again
2007-10-23 Rough yet funny
2007-10-05 Another silly Trojan mail
2007-10-01 So ugly it is beautiful
2007-09-28 Here is a nickel kid
2007-09-23 Spy Shredder
2007-08-29 Web svn view 1.08
2007-08-24 Caught in THE Process
2007-08-21 Stupid Trojan attack
2007-08-21 Back in 1994
2007-08-20 A girly iPod
2007-08-17 Crossroads for RDP connections
2007-08-15 Firewall art
2007-08-14 jpeginfo
2007-08-13 Good People
2007-08-07 The Real Crossroads
2007-07-30 BBC Documentaries in the Netherlands
2007-07-12 No problems with Crossroads so far
2007-07-11 Politically correct ad nauseam
2007-07-02 Waka Waka Poem
2007-07-02 Voyage of the rubber ducks
2007-06-28 The On Off Switch
2007-06-27 No free lunch
2007-06-25 Crossroads web interface
2007-06-25 Blinkenlights
2007-06-21 There is no silver bullet
2007-06-18 Motto of the week
2007-06-18 Do not feed the troll
2007-06-17 Which programming language are you
2007-06-13 Crossroads support request
2007-06-12 Bokito glasses
2007-06-07 Apache mod_proxy balancer description
2007-06-05 A ticketnumber is not support
2007-06-05 403 Hammertime
2007-06-04 Playground Fun
2007-05-24 Ascii man
2007-05-07 Cannot find the damn server
2007-05-02 The BFG200
2007-04-27 Crossroads Top User
2007-03-30 Crossroads Usage
2007-03-25 The guy with the dark motorhelmet
2007-03-22 The Process and The Result
2007-03-21 Quotes attributed to Jos
2007-03-20 A really nice comment about Crossroads
2007-03-18 Kubat in the air