[ python ] gevent pool

Installation

sudo apt-get install python-dev
pip install gevent

NOTE : need python-dev, otherwise we will get the following error.

error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

ref : https://stackoverflow.com/questions/26053982/error-setup-script-exited-with-error-command-x86-64-linux-gnu-gcc-failed-wit

Example

import gevent

def job(url, q)

## do whatever

 

# limit ourselves to max 10 simultaneous outstanding requests

N = 10

pool = gevent.pool.Pool(N)

q = gevent.queue.Queue(100)

map(lambda i: pool.spawn(job, '%s.com', q), xrange(0, N))

pool.join()

 

gevent.queue – Synchronized queues

multi-producer, multi-consumer queues that work across greenlets.

>>> queue = gevent.queue.Queue()
>>> queue.put(1)
>>> queue.put(2)
>>> queue.put(StopIteration)
>>> for item in queue:
... print(item)
1
2

or 

while row:

row = queue.get()

if not row:

break

sleep

gevent.sleep(seconds)

ref : http://www.gevent.org/,

https://github.com/gevent/gevent/blob/eef54f46d50495b2e6bc2a9b3321b3b4185eb497/examples/dns_mass_resolve.py,

http://www.gevent.org/gevent.queue.html

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s