Make the standard library cooperative.
- A little monkey likes to do just the same as you and you: When you stand up very tall, Monkey stands up very tall. When you go to throw a ball, Monkey goes to throw a ball. When you try to touch your toes, Monkey tries to touch his toes. When you wrinkle up your nose, Monkey wrinkles up his nose. When you jump up in the air, Monkey jumps up in.
- JMonkeyEngine is a modern developer friendly game engine written primarily in Java. Its minimalistic and code first approach makes it perfect for developers who want the support of a game engine while retaining full control over their code with the ability to extend and adapt the engine to their workflow.
“Media Monkey is one killer application for the serious music collector. For those who have large collections of music will find the organization features are some of the best out there.” — How-To Geek: MediaMonkey Review “When it comes to value for money and sheer depth of features, it really is hard to beat MediaMonkey.
The primary purpose of this module is to carefully patch, in place,portions of the standard library with gevent-friendly functions thatbehave in the same way as the original (at least as closely as possible).
The primary interface to this is the
patch_all()
function, whichperforms all the available patches. It accepts arguments to limit thepatching to certain modules, but most programs should use thedefault values as they receive the most wide-spread testing, and some monkeypatches have dependencies on others.Patching should be done as early as possible in the lifecycle of theprogram. For example, the main module (the one that tests against
__main__
or is otherwise the first imported) should begin withthis code, ideally before any other imports:A corollary of the above is that patching should be done on the mainthread and should be done while the program is single-threaded.
Tip
Some frameworks, such as gunicorn, handle monkey-patching for you.Check their documentation to be sure.
Warning
Patching too late can lead to unreliable behaviour (for example, somemodules may still use blocking sockets) or even errors.
Querying¶
Sometimes it is helpful to know if objects have been monkey-patched, and inadvanced cases even to have access to the original standard library functions. Thismodule provides functions for that purpose.
Plugins and Events¶
Beginning in gevent 1.3, events are emitted during the monkey patching process.These events are delivered first to
gevent.events
subscribers, and thento setuptools entry points.The following events are defined. They are listed in (roughly) the orderthat a call to
patch_all()
will emit them.Each event class documents the corresponding setuptools entry point name. Theentry points will be called with a single argument, the same instance ofthe class that was sent to the subscribers.
You can subscribe to the events to monitor the monkey-patching process andto manipulate it, for example by raising
gevent.events.DoNotPatch
.You can also subscribe to the events to provide additional patching beyond whatgevent distributes, either for additional standard library modules, orfor third-party packages. The suggested time to do this patching is inthe subscriber for
gevent.events.GeventDidPatchBuiltinModulesEvent
.For example, to automatically patch psycopg2 using psycogreenwhen the call to patch_all()
is made, you could write code like this:In your
setup.py
you would register it like this:For more complex patching, gevent provides a helper methodthat you can call to replace attributes of modules with attributes of yourown modules. This function also takes care of emitting the appropriate events.
Use as a module¶
Sometimes it is useful to run existing python scripts or modules thatwere not built to be gevent aware under gevent. To do so, this modulecan be run as the main module, passing the script and its arguments.For details, see the
main()
function.Changed in version 1.3b1: Added support for plugins and began emitting will/did patch events.
get_original
(mod_name, item_name)[source]¶Retrieve the original object from a module.
Silent Library Monkey
If the object has not been patched, then that object will still beretrieved.
- mod_name (str) – The name of the standard library module,e.g.,
'socket'
. Can also be a sequence of standard librarymodules giving alternate names to try, e.g.,('thread','_thread')
;the first importable module will supply all item_name items. - item_name – A string or sequence of strings naming theattribute(s) on the module
mod_name
to return.
The original value if a string was given for
item_name
or a sequence of original values if asequence was passed.is_module_patched
(mod_name)[source]¶Check if a module has been replaced with a cooperative version.
mod_name (str) – The name of the standard library module,e.g.,
'socket'
.is_object_patched
(mod_name, item_name)[source]¶Check if an object in a module has been replaced with acooperative version.
- mod_name (str) – The name of the standard library module,e.g.,
'socket'
. - item_name (str) – The name of the attribute in the module,e.g.,
'create_connection'
.
main
()[source]¶gevent.monkey - monkey patch the standard modules to use gevent.
USAGE:
python-mgevent.monkey[MONKEYOPTIONS][--module](script|module)[SCRIPTOPTIONS]
If no MONKEY OPTIONS are present, monkey patches all the modules as if by calling
patch_all()
.You can exclude a module with –no-<module>, e.g. –no-thread. You canspecify a module to patch with –<module>, e.g. –socket. In the lattercase only the modules specified on the command line will be patched.The default behavior is to execute the script passed as argument. If you wishto run a module instead, pass the
--module
argument before the module name.Changed in version 1.3b1: The script argument can now be any argument that can be passed to
runpy.run_path
,just like the interpreter itself does, for example a package directory containing __main__.py
.Previously it had to be the path toa .py source file.Changed in version 1.5: The
--module
option has been added.MONKEY OPTIONS:
--verbose--[no-]socket,--[no-]dns,--[no-]time,--[no-]select,--[no-]thread,--[no-]os,--[no-]ssl,--[no-]subprocess,--[no-]sys,--[no-]builtins,--[no-]signal,--[no-]queue,--[no-]contextvars
patch_all
(socket=True, dns=True, time=True, select=True, thread=True, os=True, ssl=True, subprocess=True, sys=False, aggressive=True, Event=True, builtins=True, signal=True, queue=True, contextvars=True, **kwargs)[source]¶Library Monkeys
Do all of the default monkey patching (calls every other applicablefunction in this module).
A true value if patching all modules wasn’t cancelled, a falsevalue if it was.
Changed in version 1.1: Issue a
warning
if this function is called multiple timeswith different arguments. The second and subsequent calls will only add morepatches, they can never remove existing patches by setting an argument to False
.Changed in version 1.1: Issue a
warning
if this function is called with os=False
and signal=True
. This will cause SIGCHLD handlers to not be called. This maybe an error in the future.Changed in version 1.3b1: Defined the return values.
Changed in version 1.3b1: Add
**kwargs
for the benefit of event subscribers. CAUTION: gevent may addand interpret additional arguments in the future, so it is suggested to use prefixesfor kwarg values to be interpreted by plugins, for example, patch_all(mylib_futures=True)
.Changed in version 1.3.5: Add queue, defaulting to True, for Python 3.7.
Changed in version 1.5: Remove the
httplib
argument. Previously, setting it raised a ValueError
.Changed in version 1.5a3: Add the
contextvars
argument.Changed in version 1.5: Better handling of patching more than once.
patch_builtins
()[source]¶Make the builtin
__import__()
function greenlet safe under Python 2.Note
This does nothing under Python 3 as it is not necessary. Python 3 featuresimproved import locks that are per-module, not global.
patch_dns
()[source]¶Replace DNS functions in
socket
withcooperative versions.This is only useful if
patch_socket()
has been called and isdone automatically by that method if requested.patch_module
(target_module, source_module, items=None)[source]¶Replace attributes in target_module with the attributes of thesame name in source_module.
The source_module can provide some attributes to customize the process:
__implements__
is a list of attribute names to copy; if not present,the items keyword argument is mandatory.__implements__
must only havenames from the standard library module in it._gevent_will_monkey_patch(target_module,items,warn,**kwargs)
_gevent_did_monkey_patch(target_module,items,warn,**kwargs)
These two functions in the source_module are called if they exist,before and after copying attributes, respectively. The “will” functionmay modify items. The value of warn is a function that should be calledwith a single string argument to issue a warning to the user. If the “will”function raisesgevent.events.DoNotPatch
, no patching will be done. These functionsare called before any event subscribers or plugins.
items (list) – A list of attribute names to replace. Ifnot given, this will be taken from the source_module
__implements__
attribute.A true value if patching was done, a false value if patching was canceled.
patch_os
()[source]¶Monkey Library Wow
Replace
os.fork()
with gevent.fork()
, and, on POSIX,os.waitpid()
with gevent.os.waitpid()
(if theenvironment variable GEVENT_NOWAITPID
is not defined). Doesnothing if fork is not available.Caution Akvis smartmask 10 portable.
This method must be used with
patch_signal()
to have proper SIGCHLD
handling and thus correct results from waitpid
.patch_all()
calls both by default.Caution
For
SIGCHLD
handling to work correctly, the event loop must run.The easiest way to help ensure this is to use patch_all()
.patch_queue
()[source]¶On Python 3.7 and above, replace
queue.SimpleQueue
(implementedin C) with its Python counterpart.patch_select
(aggressive=True)[source]¶Replace
select.select()
with gevent.select.select()
and select.poll()
with gevent.select.poll
(where available).If
aggressive
is true (the default), also remove otherblocking functions from select
.select.devpoll()
(Python 3.5+)
patch_signal
()[source]¶Make the
signal.signal()
function work with a monkey-patchedos
.Caution
This method must be used with
patch_os()
to have proper SIGCHLD
handling. patch_all()
calls both by default.Library Monkey Pro
Caution
For proper
SIGCHLD
handling, you must yield to the event loop.Using patch_all()
is the easiest way to ensure this.patch_socket
(dns=True, aggressive=True)[source]¶Replace the standard socket object with gevent’s cooperativesockets.
dns (bool) – When true (the default), also patch addressresolution functions in
socket
. See Name Resolution (DNS) for details.patch_ssl
() → None[source]¶Replace
ssl.SSLSocket
object and socket wrapping functions inssl
with cooperative versions.This is only useful if
patch_socket()
has been called.patch_subprocess
()[source]¶Replace
subprocess.call()
, subprocess.check_call()
,subprocess.check_output()
and subprocess.Popen
withcooperativeversions
.Note
On Windows under Python 3, the API support may not completely matchthe standard library.
patch_sys
(stdin=True, stdout=True, stderr=True)[source]¶Patch sys.std[in,out,err] to use a cooperative IO via athreadpool.
This is relatively dangerous and can have unintended consequencessuch as hanging the process or misinterpreting control keyswhen
input()
and raw_input()
are used. patch_all()
does not call this function by default.Library Monkey Pro
This method does nothing on Python 3. The Python 3 interpreterwants to flush the TextIOWrapper objects that make upstderr/stdout at shutdown time, but using a threadpool at thattime leads to a hang.
patch_thread
(threading=True, _threading_local=True, Event=True, logging=True, existing_locks=True) → None[source]¶Replace the standard
thread
module to make it greenlet-based.- threading (bool) – When True (the default),also patch
threading
. - _threading_local (bool) – When True (the default),also patch
_threading_local.local
. - logging (bool) – When True (the default), also patch lockstaken if the logging module has been configured.
- existing_locks (bool) – When True (the default), and theprocess is still single threaded, make sure that any
threading.RLock
(and, under Python 3,importlib._bootstrap._ModuleLock
)instances that are currently locked can be properly unlocked. Important: This is abest-effort attempt and, on certain implementations, may not detect alllocks. It is important to monkey-patch extremely early in the startup process.Setting this to False is not recommended, especially on Python 2.
Caution
Monkey-patching
thread
and usingmultiprocessing.Queue
orconcurrent.futures.ProcessPoolExecutor
(which uses aQueue
) will hang the process.Changed in version 1.1b1: Add logging and existing_locks params.
patch_time
()[source]¶Replace
time.sleep()
with gevent.sleep()
.