爬蟲背景:
由于原來的數據庫中有1.5億左右的用戶id，但是其中有1.2億的用戶資料是不完整的(沒有粉絲數量，點贊數量等，算是無用數據)，現在老板要求將這些沒有資料的用戶更新信息，咋辦？
剛開始的想法是使用主從模式+scrapy爬取，但是寫著寫著覺得麻煩(寫python的都很懶，scrapy還是比較臃腫的)，然后突然想到，python中的多線程，處理爬蟲這種存在大量io的操作時，多線程是非常有用的，而且省服務器資源(其他的爬蟲也在服務器，能省一點是一點，畢竟是自己用的，太卡的話上班都不爽)，開始干！

不怕笑話，就2個函數。。。。
我覺得代碼不重要，重要的是處理的方式，以及多多利用多線程的這種方法，各種原因不想搭scrapy的時候，只需要幾行代碼，就能將爬蟲速度提升n倍，豈不美哉？
一.
說一下我的爬蟲思路(架構？)，不存在花里胡哨的操作與各種專業名詞，只講究實用
1.我的整個爬蟲所用到的有django1.8.2 + requests + redis + pgsql 由于這里django的作用很小(代碼中有講)，就不講了
2.django操作pg存數據，redis作為隊列
3.由于使用了redis作為隊列，所以不存在多線程中的鎖(Lock)的隱患，隊列本身就是安全的，鎖來鎖去什么的不用管
4.我的這個模式，自認為還是比較好用的，簡單快捷，剩下的會在下面代碼中注釋說明，只copy了部分代碼塊

二.
說一下我這個里面的多線程的注意事項吧：
1.將爬蟲用while True包裹起來，目的是為了保持線程的存活，至于單個線程的退出條件，可以用for，while之類的條件來限制
2.函數中不要用return，不要用return，不要用return，重要的事情說三遍

            
              @staticmethod
    def get_user_info_from_app_new(**kwargs):
        while True:
            user_id = redis_client.rpop(REDIS_DOUYIN_NULL_INFO_USER)  # 從redis中取一個user_id
            retry_list = []
            for retry in range(150):  # 額，由于設備號不足的原因，導致一個user_id的請求最多重復請求150次，只要其中任意一次成功，就ok
                url_as, url_cp, iid, device_id, openudid = DouyinCrawler.get_params_for_app(**kwargs)
                device_info = redis_client.srandmember(REDIS_DOUYIN_SET_DEVICE_TYPE_BRAND)  # str 每一次請求都是隨機不同的設備型號
                device = device_info.split(';')
                d_type = device[0]
                d_brand = device[1]
                # print 'device_type', d_type
                # print 'device_brand', d_brand
                url = 'https://aweme-eagle.snssdk.com/aweme/v1/user/?user_id={}&retry_type=no_retry&iid={}&device_id={}&ac=wifi&channel=xiaomi&aid=1128&app_name=aweme&version_code=179&version_name=1.6.8&device_platform=android&ssmix=a&device_type={}&device_brand={}&language=zh&os_api=22&os_version=5.1.1&openudid={}&manifest_version_code=179&resolution=720*1280&dpi=240&update_version_code=1792&_rticket={}&ts={}&as={}&cp={}'.format(
                    user_id, iid, device_id, d_type, d_brand, openudid, int(time.time() * 1000), int(time.time()), url_as,
                    url_cp)
                req = requests.get(url, headers=app_headers)
                try:
                    jdata = json.loads(req.content)
                except ValueError as e:
                    logging.info('retry for no json data decode, {}, {}'.format(user_id, req.content))
                    continue
                if jdata['status_code'] != 0 or not jdata.get('user'):
                    logging.error('get user info from app error, {}, {}, times:{}'.format(user_id, jdata, retry))
                    retry_list.append(retry)
                    # return None
                else:
                    try:
                        douyin_id = user_id
                        user = jdata.get('user')
                        city = user.get('city')
                        if user.get('cover_url') is None and user.get("nickname", u'已重置') == u"已重置":
                            logging.info('user has been limited, {}, {}'.format(user_id, jdata))
                        locations = user.get('location')
                        activity_digg_count = user['activity']['digg_count']
                        use_music_count = user['activity']['use_music_count']
                        apple_account = user['apple_account']
                        head_url = user['avatar_larger']['url_list'][0]
                        gender = user.get('gender')
                        bind_phone = user['bind_phone']
                        birthday = user['birthday']
                        country = user['country']
                        province = user['province']
                        constellation_id = user['constellation']
                        verify_info = user['custom_verify']
                        dongtai_count = user['dongtai_count']
                        like_num = user['favoriting_count']
                        follower_quantity = user['follower_count']
                        followed_quantity = user['following_count']
                        ins_id = user['ins_id']
                        mplatform_followers_count = user['mplatform_followers_count']
                        name = user['nickname']
                        room_id = user['room_id']
                        musician_digg_count = user.get('original_musician', {}).get('digg_count')
                        musician_music_count = user.get('original_musician', {}).get('music_count')
                        musician_music_used_count = user.get('original_musician', {}).get('music_used_count')
                        share_qrcode_uri = user['share_qrcode_uri']
                        short_id = user['short_id']
                        signature = user['signature']
                        total_digg_count = user['total_favorited']
                        uid = user['uid']
                        unique_id = user['unique_id']
                        weibo_name = user.get('weibo_name')
                        weibo_url = user.get('weibo_url')
                        feed_num = user.get('aweme_count')

                        with_fusion_shop_entry = user.get('with_fusion_shop_entry')
                        with_item_commerce_entry = user.get('with_commerce_entry')
                        verification_type = user.get('verification_type')
                        custom_verify = user.get('custom_verify')
                        enterprise_verify_reason = user.get('enterprise_verify_reason')

                        payloads = {
                            'user': user,
                            'city': city,
                            'locations': locations,
                            'activity_digg_count': activity_digg_count,
                            'use_music_count': use_music_count,
                            'apple_account': apple_account,
                            'head_url': head_url,
                            'gender': gender,
                            'bind_phone': bind_phone,
                            'birthday': birthday,
                            'country': country,
                            'province': province,
                            'constellation_id': constellation_id,
                            'verify_info': verify_info,
                            'dongtai_count': dongtai_count,
                            'like_num': like_num,
                            'follower_quantity': follower_quantity,
                            'followed_quantity': followed_quantity,
                            'ins_id': ins_id,
                            'mplatform_followers_count': mplatform_followers_count,
                            'name': name,
                            'room_id': room_id,
                            'musician_digg_count': musician_digg_count,
                            'musician_music_count': musician_music_count,
                            'musician_music_used_count': musician_music_used_count,
                            'share_qrcode_uri': share_qrcode_uri,
                            'short_id': short_id,
                            'signature': signature,
                            'total_digg_count': total_digg_count,
                            'uid': uid,
                            'unique_id': unique_id,
                            'weibo_name': weibo_name,
                            'weibo_url': weibo_url,
                            'feed_num': feed_num,
                            'with_fusion_shop_entry': with_fusion_shop_entry,
                            'with_item_commerce_entry': with_item_commerce_entry,
                            'verification_type': verification_type,
                            'custom_verify': custom_verify,
                            'enterprise_verify_reason': enterprise_verify_reason,
                        }
                        try:
                            # django中有很好用的方法，創建或更新資料，都不用管去重的問題
                            dd_user, is_create = DouyinUser.objects.update_or_create(douyin_id=user_id, defaults={
                                'name': payloads['name'],
                                'verify_info': payloads['verify_info'],
                                'signature': payloads['signature'],
                                'follower_quantity': payloads['follower_quantity'],
                                'followed_quantity': payloads['followed_quantity'],
                                'total_digg_count': payloads['total_digg_count'],
                                'feed_num': payloads['feed_num'],
                                'like_num': payloads['like_num']
                            })
                            print dd_user, 'UPDATE USER INFO SUCCESS'
                            # return payloads  # 注釋了 最好不要有return這個東西
                        except:
                            continue
                    except:
                        # 加入到抓取失敗的隊列
                        # redis_client.lpush(REDIS_DOUYIN_USER_GET_FAILED_LIST, user_id)
                        continue

            
          

主函數

            
                  def get_new(self):
        for i in range(10):  # 創建線程池，大小10，想要多少就多少，電腦性能決定
            tt = Thread(target=DouyinCrawler.get_user_info_from_app_new)  #  個人不建議在這里傳參
            tt.start()  # 不加鎖，不回收，直接start

            
          

是的，多加2行就搞定了，速度提升10倍！
最后：
注意！！！！！！！
windows上，把主函數放在main函數里面調用