How to Turn Your AI Idea Into a Scalable Product: A Technical Guide
How to Turn Your AI Idea Into a Scalable Product: A Technical Guide
Have you ever had a great idea for an AI-powered app or data science product?
I know I have. I've got a note on my iPhone called Ideas with 50+ ideas.
But how do you turn your idea into a scalable product with real-life users?
Sure, you might know how to develop an ML model or fine-tune an LLM. But a model is no use to anyone if it's stuck inside a Jupyter notebook or running on localhost.
This guide will show you how to go from an idea to a production product.
If you're a wantrepreneur or startup employee, this will give you the knowledge needed to get off localhost, launch your product, and start acquiring users.
As you'll see, there is no single way to build an AI-powered product – there are many possible options. My aim is not to advocate for a particular strategy or deep-dive into the code. Rather, my aim is to give you a broad overview of the options from a technical architecture perspective. That way, when you read future blogs that showcase a particular deployment strategy, you'll have the knowledge and confidence to critically evaluate that blog and decide whether it's the best option for your application.
Models: Use your own, or call someone else's?
The core of any data/AI product is the model.
When I say "model," you might immediately think of a Large Language Model (LLM) like GPT-3.5 or a multimodal one like GPT-4o. But data science encompasses a lot more than GenAI, and there are many different types of models.
For example, an AI "model" could be:
- A Large Language Model (LLM) like GPT-3.5, which generates text
- A Machine Learning (ML) classifier which generates numerical predictions based on patterns observed in the training data
- An ML time series model which generates stepwise forecasts based on historical trends
… or something else. You might also have a non-AI model which nonetheless does some clever data science:
- An optimisation model (e.g., a linear Programming model), which generates the optimal grouping/order/solution for a set of inputs
- A rule-based/deterministic model, which follows complex pre-defined rules to make classifications or decisions
My point is: there are lots of possibilities!
But regardless of the model type, there are three high-level strategies for building and deploying:
- Build your own model locally and deploy it
- Call someone else's pre-existing model (e.g., GPT-3.5) via an API
- A hybrid of the first two
Let's talk through these options.
Option 1: Develop your own model locally
If you want a model that's tailored to your specific data or task, you may want to develop your own model locally on your computer (or in a hosted environment/notebook on a platform like Google Colab or Vertex AI).
For example, you might:
- train a LightGBM regressor for a specialised task like mobile phone price prediction,
- fine-tune a pre-trained RoBERTa classifier,
- fine-tune an open-source LLM like Llama-3, or
- build a Fantasy Football optimisation algorithm
What all these share in common is that, after training/fine-tuning your custom model, you'll end up with a file containing your model (e.g., model.txt) or parameters.
For your custom model to be useful, you now have to regularly run that model and generate fresh predictions/outputs.
There are a few ways to do this.
Deploy the model as a standalone "microservice"
One strategy is to deploy your model as its own microservice on the web using a Python framework like FastAPI, Flask, or Django Rest Framework.
For example, you could wrap your model in a FastAPI application and deploy it on a service like AWS, Render or Heroku. Others would then be able to interact with your model by making API calls to the app's endpoints.
If a user wanted to interact with your deployed app directly (e.g., sending cURL requests from their terminal/notebook to your app), the setup would look like this:
More likely, however, you'll want a friendly UI which enables users to interact with your model/app indirectly (via another application/website with a nice visual interface):
If you'd like a more detailed example of this strategy in action, I highly recommend the following tutorial, which shows how to deploy a basic scikit-learn model using FastAPI, Heroku and Docker.
Serve a machine learning model using Sklearn, FastAPI and Docker
Users can get the model to generate predictions by sending GET requests to the /predict route, whereafter the model would return the prediction.
Integrate the model directly into your monolith application
For small models, deploying a standalone app to host your model can sometimes be overkill. An alternative strategy is to integrate the model directly into your main application (a "monolith-first" architecture).
(Just be careful that your model's not too huge: it could stall/crash your entire application.)
This is a nice example of integrating an ML model directly into a Django application:
Creating a Machine Learning Based Web Application Using Django
Run the model locally/offline, and save the outputs to a database
Sometimes it's overkill to "productionise" a model on the web, and it's perfectly adequate to run your model locally at regular intervals and just send its outputs to a database instead.
This might sound like a low-tech strategy, but don't mistake low-tech for low-value.
This is what solopreneur Pieter Levels did in the early days of his startup PhotoAI. His users uploaded photos to his database via TypeForm, he downloaded those photos and manually ran them through his local model, and then he sent the outputs back to his users.
It's kind of brilliant, right?
There's minimal technical faff (great for validating ideas), and users still get to interact with your model. It's just that the interaction is indirect via the database which stores your model's outputs, rather than directly through the model's API endpoints.
Run the model online as a serverless function (e.g., via AWS Lambda or GitHub Actions)
This option is a sort of halfway house between the previous strategies.
Essentially, it involves creating a score.py file (or equivalent) that generates predictions using your model.txt file, deploying both to a platform like GitHub or AWS Lambda, and setting up a cron job to have the score.py script run at regular intervals. The model is "online" in the sense that you don't run it locally (it's being run at regular intervals on a remote server), but you're not deploying the model as an application/web service, and it's not available via API. My previous article provides a detailed walkthrough of how to do this:
Is this the best option? Honestly, it will depend on your application. Personally, I'm a big fan of the GitHub Actions strategy as a quick and low-maintenance way to deploy your model, but different applications will have different requirements. Just remember:
Don't mistake low-tech for low-value.
Option 2: Call some else's model
Not all AI products need a custom model.
Often, it's more than adequate to send requests to a generic pre-trained model (like GPT-4o or Claude 3.5 Sonnet) with some clever prompts:
The chief advantage of using a public model is that you don't need to host or monitor the model yourself. You simply make API calls to the model from within your main application, and monitor your app's usage via the provider's dashboard (e.g., OpenAI's Usage Dashboard).
This setup is ideal is you want to focus on the application itself, rather than on building super custom models or worrying about their deployment. This is a great example of how to implement this approach using GPT-3.5, Django, and HTMX.
Option 3: Hybrid options
Remote fine-tuning
Fine-tuning generally happens locally (or in a hosted notebook), giving you direct access to the fine-tuned model file.
With certain platforms/models, however, you won't get access to the raw model file after fine-tuning. One example is OpenAI.
This approach is much less common than options 1 and 2, but it's still an option. If you do want to explore fine-tuning a model on a platform like OpenAI's, you'd just call those fine-tuned models via an API in the same way as you would with a generic hosted LLM like GPT-3.5 or 4.
Deploying chains as standalone apps
If you want to build complicated workflows/chains using pre-trained models like GPTs, it might make sense to deploy the chain as a standalone app (instead of calling the model directly from within your application).
This example illustrates this nicely. The developer calls OpenAI models from within a complex LangChain workflow, and then deploys the chain as a standalone app via FastAPI.
Which option is right for me?
Honestly, it depends. And I don't just say that as a cop-out. I say it because, well, it depends!
If you want to get something up-and-running and deploy your application as quickly as possible, calling someone else's model via an API (or deploying your own as an API with FastAPI) is an excellent shout. It's the quickest way to get your application out there for the world to see, and it's a popular approach with data scientists who don't want to faff around with frontend frameworks and complex deployment pipelines. This is the approach I personally take to starting many new projects – get something up-and-running using a minimalist framework like FastAPI or Django, then worry about the frontend and application logic once you've got the bare bones of the app working.
One final piece of advice: don't over-engineer it. The true end-goal of a machine learning project is not to "deploy a model"; it's to make something that's useful, and for that, it has to be used. In other words, you have to get users! This is the approach I took with my SQL tutorial website theSQLgym – I started by getting the basic project up-and-running, and then I iteratively improved the app while it was live, based on real users' feedback.
Next steps
This guide has focused on how to deploy/productionise models, but there's a lot more to an app than just the model. If you're interested in learning more about full-stack development of AI apps (e.g., how to develop websites), you might be interested in my previous article:
That's it!
Thanks for reading. I hope you found this helpful, and free to connect with me on Twitter or LinkedIn!
