53
Python多线程爬图&Scrapy框架爬图-KaliArch
source link: http://blog.51cto.com/kaliarch/2162411
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
一、背景对于日常Python爬虫由于效率问题，本次测试使用多线程和Scrapy框架来实现抓取斗图来表情。由于IO操作不使用CPU，对于IO密集（磁盘IO/网络IO/人家交互IO）型适合用多线程，对于计算密集型：建议用多进程。进程：优点：充分利用多核CPU（能够同时进行多个操作）缺点：系统资源消耗大，重新开辟内存空间线程：优点：共享内存，IO操作可以创造出并发操作缺点：抢占资源，请求上下文切换消耗时
Recommend
About Joyk
Aggregate valuable and interesting links.
Joyk means Joy of geeK