Explainable Generic ML Pipeline with MLflow

Intro

One common challenge in MLOps is the hassle of migrating between various algorithms or frameworks. To tackle the challenge, this is my second article on the topic of generic model building using mlflow.pyfunc.

In my previous article, I offered a beginner-friendly step-by-step demo on creating a minimalist algorithm-agnostic model wrapper.

Algorithm-Agnostic Model Building with MLflow

To further our journey, by the end of this article, we will build a much more sophisticated ML pipeline with the below functionalities:

1. This pipeline supports both classification (binary) and regression tasks. It works with scikit-learn models and other algorithms that follow the scikit-learn interface (i.e., fit, predict/predict_proba).
2. Incorporating a fully functional Pre-Processor that can be fitted on train data and then used to transform new data for model consumption. This pre-processor can handle both numeric and categorical features and handle missing values with various imputation strategies.
3. Adding an explainer to shed light on the model's reasoning, which is invaluable for model selection, monitoring and implementation. This task can be tricky due to the varying implementations of SHAP values across different ML algorithms. But, all good, we will address the challenge in this article.

   Tags: Databricks Hands On Tutorials Machine Learning Mlflow Mlops

   Add Fav

Comment