Python如何实现子进程并行处理？
- 内容介绍
- 文章标签
- 相关推荐
本文共计595个文字，预计阅读时间需要3分钟。
希望能够并行运行多个进程，并且能够随时使用stdout。应该如何操作？我是否需要为每个subprocess.Popen()调用运行线程，还是可以在一个线程中完成？假设您有一个随机打印行为的脚本：
pythonimport subprocessimport threadingimport timeimport random
def random_print(): while True: print(random.choice([Hello, World, Python, Subprocess])) time.sleep(random.uniform(0.1, 1))
if __name__==__main__: # 创建一个线程来运行随机打印脚本 t=threading.Thread(target=random_print) t.start()
# 创建并启动多个子进程 for _ in range(5): subprocess.Popen([python, random_script.py])
我希望能够并行运行多个进程,并且能够随时使用stdout.我该怎么办？我是否需要为每个subprocess.Popen()调用运行线程,什么？ 你可以在一个线程中完成.假设您有一个随机打印行的脚本：
#!/usr/bin/env python #file: child.py import os import random import sys import time for i in range(10): print("%2d %s %s" % (int(sys.argv[1]), os.getpid(), i)) sys.stdout.flush() time.sleep(random.random())
并且您希望在输出可用后立即收集输出,您可以在POSIX系统上使用select作为@zigg suggested：
#!/usr/bin/env python from __future__ import print_function from select import select from subprocess import Popen, PIPE # start several subprocesses processes = [Popen(['./child.py', str(i)], stdout=PIPE, bufsize=1, close_fds=True, universal_newlines=True) for i in range(5)] # read output timeout = 0.1 # seconds while processes: # remove finished processes from the list (O(N**2)) for p in processes[:]: if p.poll() is not None: # process ended print(p.stdout.read(), end='') # read the rest p.stdout.close() processes.remove(p) # wait until there is something to read rlist = select([p.stdout for p in processes], [],[], timeout)[0] # read a line from each process that has output ready for f in rlist: print(f.readline(), end='') #NOTE: it can block
更便携的解决方案(适用于Windows,Linux,OSX)可以为每个进程使用读取器线程,请参阅Non-blocking read on a subprocess.PIPE in python.
这是基于os.pipe()的解决方案,适用于Unix和Windows：
#!/usr/bin/env python from __future__ import print_function import io import os import sys from subprocess import Popen ON_POSIX = 'posix' in sys.builtin_module_names # create a pipe to get data input_fd, output_fd = os.pipe() # start several subprocesses processes = [Popen([sys.executable, 'child.py', str(i)], stdout=output_fd, close_fds=ON_POSIX) # close input_fd in children for i in range(5)] os.close(output_fd) # close unused end of the pipe # read output line by line as soon as it is available with io.open(input_fd, 'r', buffering=1) as file: for line in file: print(line, end='') # for p in processes: p.wait()
本文共计595个文字，预计阅读时间需要3分钟。
希望能够并行运行多个进程，并且能够随时使用stdout。应该如何操作？我是否需要为每个subprocess.Popen()调用运行线程，还是可以在一个线程中完成？假设您有一个随机打印行为的脚本：
pythonimport subprocessimport threadingimport timeimport random
def random_print(): while True: print(random.choice([Hello, World, Python, Subprocess])) time.sleep(random.uniform(0.1, 1))
if __name__==__main__: # 创建一个线程来运行随机打印脚本 t=threading.Thread(target=random_print) t.start()
# 创建并启动多个子进程 for _ in range(5): subprocess.Popen([python, random_script.py])
我希望能够并行运行多个进程,并且能够随时使用stdout.我该怎么办？我是否需要为每个subprocess.Popen()调用运行线程,什么？ 你可以在一个线程中完成.假设您有一个随机打印行的脚本：
#!/usr/bin/env python #file: child.py import os import random import sys import time for i in range(10): print("%2d %s %s" % (int(sys.argv[1]), os.getpid(), i)) sys.stdout.flush() time.sleep(random.random())
并且您希望在输出可用后立即收集输出,您可以在POSIX系统上使用select作为@zigg suggested：
#!/usr/bin/env python from __future__ import print_function from select import select from subprocess import Popen, PIPE # start several subprocesses processes = [Popen(['./child.py', str(i)], stdout=PIPE, bufsize=1, close_fds=True, universal_newlines=True) for i in range(5)] # read output timeout = 0.1 # seconds while processes: # remove finished processes from the list (O(N**2)) for p in processes[:]: if p.poll() is not None: # process ended print(p.stdout.read(), end='') # read the rest p.stdout.close() processes.remove(p) # wait until there is something to read rlist = select([p.stdout for p in processes], [],[], timeout)[0] # read a line from each process that has output ready for f in rlist: print(f.readline(), end='') #NOTE: it can block
更便携的解决方案(适用于Windows,Linux,OSX)可以为每个进程使用读取器线程,请参阅Non-blocking read on a subprocess.PIPE in python.
这是基于os.pipe()的解决方案,适用于Unix和Windows：
#!/usr/bin/env python from __future__ import print_function import io import os import sys from subprocess import Popen ON_POSIX = 'posix' in sys.builtin_module_names # create a pipe to get data input_fd, output_fd = os.pipe() # start several subprocesses processes = [Popen([sys.executable, 'child.py', str(i)], stdout=output_fd, close_fds=ON_POSIX) # close input_fd in children for i in range(5)] os.close(output_fd) # close unused end of the pipe # read output line by line as soon as it is available with io.open(input_fd, 'r', buffering=1) as file: for line in file: print(line, end='') # for p in processes: p.wait()