JAVA高级面试进阶训练营视频教程

admin · 发表于 2021-7-21 09:54:15

python 如果有导入numpy模块的import语句，会导致默认将多进程程序的每个进程都绑定到同一个CPU core上，

失去了多进程在多核CPU上的性能优越性，这和CPU affinity(CPU亲和性)有关，解决办法：

导入affinity包，执行：

affinity.set_process_affinity_mask(0,2**multiprocessing.cpu_count()-1)

以下是英文文档原文，供参考：

Python refuses to use multiple cores – solution

I was trying to get parallel Python to work and I noticed that if I run two Python scripts simultaneously – say, in two different terminals – they use the same core. Hence, I get no speedup from multiprocessing/parallel Python. After some searching around, I found out that in some circumstances importing numpy causes Python to stick all computations in one core. This is an issue with CPU affinity, and apparently it only happens for some mixtures of Numpy and BLAS libraries – other packages may cause the CPU affinity issue as well.

There’s a package called affinity (Linux only AFAIK) that lets you set and get CPU affinity. Download it, run python setup.py install, and run this in Python or ipython:

 
         In [ 
         1 
         ]:  
         import 
         affinity 
        
         In [ 
         2 
         ]: affinity.get_process_affinity_mask( 
         0 
         ) 
        
         Out[ 
         2 
         ]:  
         63

This is good: 63 is a bitmask corresponding to 111111 – meaning all 6 cores are available to Python. Now running this, I get:

 
         In [ 
         4 
         ]:  
         import 
         numpy as np 
        
         In [ 
         5 
         ]: affinity.get_process_affinity_mask( 
         0 
         ) 
        
         Out[ 
         5 
         ]:  
         1

So now only one core is available to Python. The solution is simply to set the CPU affinity appropriately after import numpy, for instance:

 
         import 
         numpy as np 
        
         import 
         affinity 
        
         import 
         multiprocessing 
        
         affinity.set_process_affinity_mask( 
         0 
         , 
         2 
         * 
         * 
         multiprocessing.cpu_count() 
         - 
         1 
         )

		自动登录	找回密码
密码			立即注册

JAVA高级面试进阶训练营视频教程	Java架构师系统进阶VIP课程	分布式高可用全栈开发微服务教程	Go语言视频零基础入门到精通	Java架构师3期(课件+源码)
Java开发全终端实战租房项目视频教程	SpringBoot2.X入门到高级使用教程	大数据培训第六期全套视频教程	深度学习（CNN RNN GAN）算法原理	Java亿级流量电商系统视频教程
互联网架构师视频教程	年薪50万Spark2.0从入门到精通	年薪50万！人工智能学习路线教程	年薪50万大数据入门到精通学习路线	年薪50万机器学习入门到精通教程
仿小米商城类app和小程序视频教程	深度学习数据分析基础到实战	最新黑马javaEE2.1就业课程	从 0到JVM实战高手教程	MySQL入门到精通教程

JAVA高级面试进阶训练营视频教程

Java架构师系统进阶VIP课程

python 导入numpy 导致多进程绑定同一个CPU问题解决方法

Python refuses to use multiple cores – solution