0

Python性能分析

 2 years ago
source link: https://maxyoung.fun//blog/Python%E6%80%A7%E8%83%BD%E5%88%86%E6%9E%90.html
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.

Python性能分析

性能优化的重点在于定位瓶颈在哪儿
用time库手动打标, 可以看耗时, 但是比较麻烦
下面介绍两个工具:

cProfile

https://docs.python.org/3/library/profile.html

Python >= 3.7
import cProfile, pstats, io
from pstats import SortKey
pr = cProfile.Profile()
pr.enable()
# ... do something ...
pr.disable()
s = io.StringIO()
sortby = SortKey.CUMULATIVE
ps = pstats.Stats(pr, stream=s).sort_stats(sortby)
ps.print_stats()
print(s.getvalue())
Python < 3.7
import cProfile, pstats, io
pr = cProfile.Profile()
pr.enable()
# ... do something ...
pr.disable()
s = io.StringIO()
ps = pstats.Stats(pr, stream=s).sort_stats("cumulative")
ps.print_stats(10)
print(s.getvalue())

line_progfiler

pip install line_profiler

from scripts.revenue_manager import get_manager 
from common.collect import collect, Collect
from line_profiler import LineProfiler
manager = get_manager("one_time", 9, "2021-06")
# 实例化LineProfiler
lp = LineProfiler()
# 我们有时候需要看主函数调用的子函数的运行情况
# 我们可以添加需要监控的函数, 函数可以是单独的函数, 也可以是类的函数
lp.add_function(collect)
lp.add_function(Collect.__init__)
# 将需要运行的函数作为参数, 添加到LineProfiler里
lp_wrapper = lp(manager.pull_data)
# 运行, 这里可以加上manager.pull_data的参数
lp_wrapper("7")
# 将信息打印出来
lp.print_stats()

信息大概如下:

Total time: 22.8278 s
File: common/collect.py
Function: collect at line 27
Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
    27                                           def collect(revenue_mode)
    67         1          3.0      3.0      0.0      collector = Collect(source
    68         1   19628932.0 19628932.0     86.0        direction, add_conditions, print_log=print_log)
    79         1    3198725.0 3198725.0     14.0     data, is_success = collect_cdkey_expire(collector)
    87         1          2.0      2.0      0.0      return {'data': data, 'is_success': is_success}

Total time: 19.3315 s
File: common/collect.py
Function: __init__ at line 210

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   210                                               def __init__(self, source)
   213         1        262.0    262.0      0.0          self.connection = MongoConnection.get_client()
   239         1   18754966.0 18754966.0     97.0        self.fin_cat_id_l3.split(","))).all()
   240         1      98748.0  98748.0      0.5          self.sku_codes = [sku.sku_code for sku in sku_list]

Total time: 27.2546 s
Function: pull_data at line 696

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   696                                               def pull_data(self):
   704         1     212100.0 212100.0      0.8          OneTimeOrigin.objects.filter(
   705         1          6.0      6.0      0.0              source="cdkey_expire",
   706         1     173688.0 173688.0      0.6              grant_time__lt=self.date_range_to_str).delete()
   718         1   24631955.0 24631955.0     90.4          one_time_pull_data_from_tenon()
   ...

我们可以看到三部分数据, 前面两个是我们通过add_function添加的监控函数, 最后一个是运行的主函数
主要看Time(运行时间), %Time(百分比)


report

Recommend

About Joyk


Aggregate valuable and interesting links.
Joyk means Joy of geeK