Deploy ML models at Scale

ML Model Deployment ( Source )

Let’s assume that you have built a ML model and that you are happy with its performance. Then the next step is to deploy the model into production. In this blog series I will cover how you can deploy your model for large scale consumption with in a scalable Infrastructure using AWS using docker container service.

In this blog I will start with the first step of building an API framework for the ML model and running it in you local machine. For the purpose of this blog, let’s consider the Sentiment classification model builthere. In order to deploy this model we will follow the below steps:

1. Convert the model into  .hdf5 file or  .pkl file
2. Implement a Flask API
3. Run the API

Convert the model into “ .hdf5” file or ‘.pkl’ file

In case the model is a built on sklearn, it would be best to save it as a ‘.pkl’ file. Alternatively if it is a deep learning model then it is recommended to save the model as a ‘HDF’ file. The main difference between ‘.pkl’ and ‘.hdf’ is, pickle requires a large amount of memory to save a data structure to disk, where as HDF is designed to efficiently store large data sets.

Save model in Pickle(.pkl):

from sklearn.externals import joblib  <em># Save to file</em>

# Fit the model (example of a model in sklearn)
model = LogisticRegression()
model.fit(X_train, Y_train)

#working directory
joblib_file = "best_model.pkl"   
joblib.dump(model, joblib_file)

Save model in HDF(.hdf5):

Once you have trained your deep learning model in keras or tensorflow you can save the model architecture and its weights using a .hdf5 file system.

# save the best model and early stopping

saveBestModel = keras.callbacks.ModelCheckpoint('/best_model.hdf5', monitor='val_acc', verbose=0, save_best_only=True, save_weights_only=False, mode='auto', period=1)

Implement a Flask API

Step 1:Load the saved model (as per previous section) by one of the following methods depending on the type of file i.e. ‘hdf5’ or ‘pkl’.

HDF5:

from keras.models import load_model
import h5py
prd_model = load_model('best_model.hdf5')

PKL:

from sklearn.externals import joblib
loaded_model = joblib.load('best_model.pkl')

Step 2:Import flask and create a flask application object as shown below:

from flask import Flask, url_for, request

app = Flask(__name__)

Step 3:The next step is to build a test API function which returns the “ API working” string. This can be used to ensure the health of the API when it is deployed in production. Here we use ‘@app.route’, which is a python decorator ( a decorator is a function that takes another function and extends the behaviour of the latter function without explicitly modifying it)

@app.route(‘/apitest')
def apitest():
    return ‘API working’

Step 4:The next step is to build a “POST” request api for processing requests to our sentiment model. We are using the path name “/sentiment”. The function reads the json input and converts it into pandas dataframe. It extracts the relevant fields from the json and calls the “get_sentiment_DL” function for processing. “get_sentiment_DL” function contains the trained model which has been loaded via ‘hdf5’ file. It finally will return back the results of the model in the form of json result.

# main API code
@app.route(‘/sentiment’, methods=[‘POST’])
def sentiment():
   if request.method == ‘POST’:
   text_data = pd.DataFrame(request.json)
   text_out = get_sentiment_DL(prd_model, text_data, word_idx)

   text_out = text_out[[‘ref’,’Sentiment_Score’]]

#Convert df to dict and then to Json
   text_out_dict = text_out.to_dict(orient=’records’)
   text_out_json = json.dumps(text_out_dict, ensure_ascii=False)

return text_out_json

Step 5: The detailed model processing steps will be performed by “get_sentiment_DL” function. In the case of our deep learning sentiment model we are passing:

1. best_model: the loaded model

2. text_data: Input text for sentiment classification

3. word_idx: Word index from the GloVe file (details of the modelhere).

def get_sentiment_DL(best_model, text_data, word_idx):

'''Modle Processing'''

return sentiment_score

Step 6:Add the below section to run the app. Here Host is set as “0.0.0.0” as we are hosting in our local server. However you can configure it to your network settings. Debug can be set to True at the time of building the API functionality. The Port is set to 5005, however this can be configured as per your requirement.

if __name__ == “__main__”:
  app.run(host=”0.0.0.0", debug=False, port=5005)

Run the API

To run the API, open a command line window and go to the directory where the code is stored. Run the python script by running the below command (“sentiment.py” is the name of the file with the above API implementation).

python sentiment.py

On running the above command you will be able to see the below result in your command line window:

Once the API is running you can test the API by going to your browser and typing “0.0.0.0:5005/apitest”. You will get the below result in you browser.

Browser — API Test

You can now pass any data to the API using python as shown below. The address in our case is “ http://0.0.0.0:5005/sentiment ”. The results of the model will be returned and stored in “response” field.

import requests
import json
import urllib.parse

data_json = '''{"ref":[1], "text":["I am well"]}'''

head= {“Content-type”: “application/json”}

response = requests.post(‘<a href="https://www.seelabs.app/face_rec_train%27" data-href="https://www.seelabs.app/face_rec_train'" rel="nofollow noopener noopener" target="_blank">http://0.0.0.0:5005/sentiment'</a>, json = data_json, headers = head)

result = response.content.decode(‘utf8’)

Conclusion

In conclusion, we have covered steps to deploy the model into an api service in your local computer.

The next step is to deploy this in a cloud server as a micro service. In following blog’s I will cover the use of container service such as docker and deploy it in AWS.