Build, train and track machine learning models using Superwise and Layer

Deploying machine learning models is a small part of the entire machine learning life cycle. Once a model is in production, you have to monitor it to ensure it’s working as expected.

Build, train and track machine learning models using Superwise and Layer
Getting started with Superwise and Layer on AWS Sagemaker

Deploying machine learning models is a small part of the entire machine learning life cycle. Once a model is in production, you have to monitor it to ensure it’s working as expected. The behavior of a model can change in unexpected ways, especially when data in the real world changes. For instance, when the data the model was trained on is significantly different from the data used for making predictions. This leads to model drift. Drift can be caused by, among other things, changes in the behavior of people, changes in data distribution, and errors in data collection.

This article will illustrate how you can use Layer and Amazon SageMaker to deploy a machine learning model and track it using Superwise. Amazon SageMaker enables you to build, train and deploy machine learning models.

Layer helps you build, train and track all your machine learning project metadata, including ML models and datasets‍ with semantic versioning, extensive artifact logging, and dynamic reporting with local↔cloud training.

Superwise provides tools for ensuring continuous model observability to keep your ML healthy in production. The platform enables you to create alerts using ML monitoring templates. It also offers root cause investigation by correlating and grouping anomalies to discover why models are misbehaving. Furthermore, you can use Superwise to optimize models with production observability insights to identify retaining opportunities and strategies, underperforming segments, and biases in ML decision-making processes.

Create an Amazon SageMaker notebook instance

This example is meant to run on an Amazon SageMaker instance. Start by creating one.

When creating, ensure that you select notebook-al2-v1 for the platform identifier because it has a newer Python version required to install Superwise.

Fetch a model from Layer

With the instance ready, install Layer, SageMaker, and Superwise.

pip install sagemaker superwise layer

For this illustration, let’s use a regression model that is already trained on the California housing dataset. Check out this example to learn how to train a model from scratch with Layer. Let’s import the model using Layer:

Deploying the model on Sagemaker requires a model.joblib file. We, therefore, have to persist the model.

Deploying a model with SageMaker requires a script containing the following functions:

  • model function named model_fn
  • input function named input_fn
  • prediction function named predict_fn

The function is attached to this repo.

Deploy the model to Sagemaker

Deploying the model to Amazon SageMaker requires a tar file containing the model.joblib . Build the tar file and upload it to your s3 bucket. Next, build a SKLearnModel using the uploaded artifact and deploy it to an endpoint using the deploy command.

After deployment, test the endpoint to confirm that it’s working as expected.

Tracking the Layer model with Superwise

You need a Superwise account to start tracking the model. After creating one, obtain your client ID and secret from your profile. Then initialize Superwise.

Create the Model entity

Next, create a Superwise model entity.

Create a Schema object

Add the prediction value, a timestamp, and the label to the training features and use the DataFrame index as the record ID. In practice, you should use an ID representing the problem you are tackling. This baseline data is used for creating a schema object. Superwise uses this to interpret the data––for instance, in understanding which column represents predictions and the labels.

Create a Version object

The next step is to create a Version object for the model we are tracking.

Monitoring ongoing predictions

Now that we have a Version of the model setup with a Baseline, we can start sending ongoing model predictions to Superwise to monitor the model’s performance in a production setting.

We will treat the test split of the data as our “ongoing predictions” for this demo.

Next, log the production data to Superwise and check the status.

Don’t forget to delete the endpoint used for the demonstration.

Final thoughts

In this article, we have demonstrated how to integrate a Sagemaker-based development workflow with Superwise and Layer. We have also seen how to set up Superwise to start tracking your models by registering and providing a baseline for the model’s behavior. Finally, we have looked at sending new predictions from your model to Superwise, simulating a post-deployment scenario. We are confident that with this new partnership, your production models will never drift again.

Resources

Don’t forget to join our Slack community, and follow us on Twitter and Linkedin for resources, events, and much more.

Subscribe to Layer

Don’t miss out on the latest issues. Sign up now to get access to the library of members-only issues.
jamie@example.com
Subscribe