Itinerary

• Part I: Introducing the AI Strategy Framework
• Part II: Crafting a Compelling AI Product Vision and Narrative
• Part III: Data Collection - The Essence of AI
• Part IV: Ensuring Reliable and Accessible Storage
• Part V: Data Exploration and Transformation - Obtaining Clean Data
• Part VI: Insights & Analysis - See the Unseen
• Part VII: Machine Learning - The Continuous Improvement Cycle (👈You’re here)
• Part VIII: AI & Deep Learning - Reaching the Pinnacle

An Algorithmic Home Run

In 2002, algorithms changed baseball forever. This was the year that Oakland A’s general manager, Billy Beane, began to employ data-based decision making to construct his team.

Beane needed a way to overcome budget constraints and compete with teams like the Yankees who had budgets several times larger. This story is written about in Michael Lewis’ 2003 book and re-enacted in the popular 2011 film "Moneyball."

Historically, scouts would travel all across the country looking for particular skills. According to the MLB’s own scouting guide, these include things like speed, quickness, arm strength, hitting ability, and mental toughness.

Relying on the human ‘gut instinct’ to evaluate skill and future success was the status quo. This approach allowed many old men to watch baseball for a living.

The Shift to Data Science

A transition is occurring, and it seems like Beane was well ahead of the business community. He adopted data science approaches to save money while producing amazing results.

The first thing that Beane did was to flip the evaluation metrics on their head. He defined, in clear terms, what you actually want a high-quality player to produce: runs.

Calculating Success

Beane's goal was to win the World Series, and to win the World Series, you need to win games. To win games, you need to score more runs than your opponent.

Instead of thinking about the factors that affected scoring runs directly, the baseball community focused on weak correlates that were difficult to measure. So Beane ran some calculations with the help of his team of data analysts.

They found that to make the playoffs with a high probability, you needed at least 99 wins. To get 99 wins, you need a run differential (runs scored minus runs against) of 169.

Putting numbers to these seemingly obvious facts allowed the Oakland A’s staff to begin to uncover decision-making protocols that could replace the historical ‘eye-test.’

What is an Algorithm?

This brings us to the concept of the algorithm. It’s an equation that, given different inputs, produces a reliable output.

It’s taking in numbers and statistics - it’s taking in data. In this case, it’s meant to formalize the scouting process of evaluating players and predicting their future success.

Creating the OBP Algorithm

Following the chain of causation further, we need to score runs to make the playoffs. So we need players on the team who score runs.

In baseball, to score a run, you need to first get on base. You get on base by either getting a hit or getting a ‘walk.’ So Beane’s algorithm becomes:

On Base Percentage = (Hits + Walks) x Total Bases / Total At Bats

Implementing the Algorithm

By putting in actual data on players around the league, Beane could find players who had the highest ‘On-base Percentage’ or OBP. The algorithm could stack rank the players based on Beane's criteria.

This systematic and data-driven approach helped him put together a team that would get on base enough to get enough runs to win 99 games and make the playoffs. He created a way to find value in an area that had historically relied on the inconsistent evaluations of humans.

As we move through today's discussion, it can be easy to get lost in the vocabulary of Machine Learning and data science. There are many different algorithms that are more complex than Beane’s OBP calculation.

But complexity does not equal value. Beane took a simple algorithm and turned it into a playoff team with the smallest budget in the league.

He didn’t need complexity; he needed clear thinking and a clear target to help him formulate his strategy. It’s important not to pursue complexity for the sake of complexity.

Maintain focus on your vision, and the rest will fall into place.

What is Machine Learning Anyways?

Using our example of the OBP algorithm created by Billy Beane, we get only a piece of the picture. Beane took all of his own knowledge of baseball, added in some math, and manually created an equation.

Instead of Beane creating the equation, machine learning dynamically learns the equation from the data itself. Machine learning algorithms like decision trees, support vector machines, or logistic regression provide a framework for learning from data.

Machine Learning Framework

Given the input data set, these algorithms modify their structure to best fit the underlying data. They provide an equation that takes in new, previously unseen information, and gives an accurate prediction.

A quick break here to clear up some differences in what's called shallow machine learning vs. deep learning vs. AI. Shallow machine learning methods do not use layers of computation and the famous backpropagation algorithm to uncover their equations.

Shallow vs. Deep Learning

Shallow machine learning uses more straightforward methods with fewer steps and less computation to get to a reliable answer. These types of algorithms are a good first step because they are (slightly) less complicated, use less compute, and have easier implementations using third-party packages or platforms.

Now back to our baseball analogy. So what would an OBP machine learning algorithm look like?

Preparing the Dataset

We’ve talked a lot about cleaning up the input dataset before we even get to this step. We’d have to clean up our baseball stats data and make a clear hypothesis.

Imagine we are the general manager attending the draft to find new high-quality players that will score lots of runs and help the team make the playoffs. We gather all of the stats from the player's high school or college career.

Then we collect or calculate the standard statistics mentioned above, like hits, walks, and batting averages. From here, we may want to do some feature engineering to create new columns or data points from the raw data.

Now we define a target, the thing we’re trying to predict. This is both an art and a science.

Defining the Target

What is the piece of information we want to know? In our situation, we want to know if the player is going to get on base enough to help the team score enough runs, get enough wins, and make the playoffs.

So we look at past years' data and see the stats of players who were previously drafted. We then use their OBP as the target.

This means our machine learning algorithm will come up with an equation that looks something like this:

Prediction of rookie season OBP = Player Stats + Engineered Features

Training: Teaching Your Model to Sit and Lie Down

So we’ve cleaned up our dataset, engineered our features, and defined our ‘label’ or what we want the model to predict. Now, after thousands of words in this blog series, we are finally training a machine learning model.

I’m sorry it has come to such an anti-climatic point in our story. This is by far the easiest part of the entire process.

Training the Model

The application and training of several different types of machine learning algorithms can be done with only a few lines of code. Packages like Sci-Kit Learn allow data scientists to quickly uncover which approach is getting them the best results and does not require much effort.

The work comes in the evaluation and experimentation with your training set. Taking an approach that passes the data through several models and then combining their outputs is called an ensemble and can often provide higher accuracy than deploying a single model in isolation.

Keep in mind, though, that in the early stages when building your first model, simplicity is key. Training is the easy part, actually deploying the model and ensuring it works as expected is the hard part.

Let’s talk a little bit about what is happening in training. You’re passing a dataset that includes what you believe to be the predictive factors that will help your model output the correct answer.

Understanding the Process

In our case, we think that providing statistics about a baseball player will help us to predict their future OBP percentage during their rookie season in the big leagues. So we are providing different statistics and relying on the algorithm to manipulate these numbers into an output number - On Base Percentage.

Different models do this in different ways, but they all have something called a “loss function.” Loss functions tell your model how much it sucks.

Minimizing the Loss

Minimizing the loss function across all of your training examples is what allows the model to converge on the equation that is going to best predict the correct output. By passing an example of player statistics through the algorithm with a certain set of parameters, we get our output prediction of OBP.

We then evaluate this against the ‘loss function’ and can modify the parameters such that they will get us closer to the right answer. Repeating this hundreds, thousands, or millions of times allows the model to see lots of examples and tune itself closer to the equation that will make the difference between the actual OBP and its prediction as small as possible.

Evaluating the Model

Once the training cycle is complete, it’s time to evaluate the model. We do this by showing it examples it’s never seen before - the validation set.

The validation set allows us to have a set of data that has been held back from the algorithm during training to see how it may work in the ‘real world.’ We pass the input statistics of our baseball players into our trained algorithm and it gives us an answer.

We repeat this for all examples in our training set and see how many it got right vs. wrong. This becomes our accuracy score.

When we say our model is 95% accurate, this is what it means. The model has evaluated a bunch of examples and got 95% of its predictions correct.

When the data scientist feels confident that the model is accurate enough to provide the business value set out in the AI Strategy Narrative, it’s time to move to deployment. Getting the model into the world and having real humans interact with it.

Deployment: Bring Your Model into the World

This is the true pinnacle of the machine learning layer of our AI Strategy Framework. We often assume the completion of training as being the summit because at that point we have a model that is predictive, and we’ve accomplished a lot.

In reality, creating a way to get data to that model, capturing its output, and surfacing the output in a useful way to achieve our vision is really the hardest part. It's difficult because the model requires data to be delivered to it in a certain format and with the cleaning transformations applied.

Challenges in Deployment

It's also challenging because the machine learning tech stack often uses different programming languages and a different tech stack than the rest of the software solution. This results in brand new infrastructure and development operations work.

It most often means hiring different engineers with different skills. They need the ability to work with machine learning languages and technologies as well as strong capabilities in integrating these seamlessly with more traditional software.

MLOps: Machine Learning Operations

Things like containerization, serialization, serverless architectures, and other complex topics begin to surface as part of the deployment plan. This is called MLOps or Machine Learning Operations, the process of continuous training and model deployment.

This is really where you can reap the fruits of your labor in the prior steps of the AI Strategy Framework. If you’ve maintained your discipline and kept the complexity low, you give your MLOps engineers a fighting chance of standing up your machine learning infrastructure.

Having everything work together is only part of the success metrics you’re looking for. You need to have created value for your end user and a way to measure the business impact of all the steps you’ve been through.

By ensuring you can quickly get the model out and measure its impact, you buy yourself a second chance to continue to experiment and improve the model and the customer experience. For those who introduce too much complexity too early, you may never even get to measure the impact.

Bringing the Example to Life

Let’s bring all of this to life using our baseball example. We have our model that predicts rookie season on base percentage.

We’ve kept it so simple that all we need to do is gather all of the stats of draft-eligible players, run them through the model, and get its output prediction. On draft day, we’ll have a list that shows the stack ranking of predicted OBP for each player.

Perhaps the second version would be built to predict players' OBP for the rest of the season after updating their stats for each game. This would require additional model training on different data sets.

The deployment of the second model would be much more complex. We would need to pass data to the correct model after each game, updating the OBP predictions.

Here we can see that depending on the use case and how many different moving parts we have, the deployment problem can grow exponentially. Monitoring and updating the model regularly becomes crucial.

Monitoring: Hiring a Babysitter

The only constant is change. Machine learning algorithms are based on data, and their training occurred on only a small slice of time.

As the world changes, the structure of the underlying data may change, resulting in the machine learning model needing to adjust its equation to continue to perform. This is where monitoring becomes important.

Importance of Ongoing Evaluation

As the model works within your product or application, you want to be able to see what predictions it is producing, and compare these to reality. By getting an ongoing evaluation of your model accuracy across time, you can monitor for what's called ‘model drift.’

This means that the model no longer represents reality and needs to be re-trained on more recent examples. Let's imagine our model has done an incredible job of predicting OBP for our draft picks in their rookie season.

But something dramatic happens, like the league introducing a pitch clock. Now instead of the pitcher being able to take their time, they are on the clock, and this changes the game dynamics.

By monitoring your model across time, you can see its drift, and you can make a plan to re-train exclusively using player stats with the pitch clock. This also leads us to understand further complexity in MLOps.

Continuous Monitoring and Retraining

Not only is there a responsibility for deploying the model, but there is also a responsibility to monitor it, collect recent data samples, and allow for automatic re-training of the model. All of these items take significant time and effort and customized development work.

Take it slow, and appreciate the gift of having a simple data pipeline, with simple transformations, and a simple ML model as your first start.

One More Thing: Supervised vs. Unsupervised Learning

As we've navigated the realms of data, algorithms, and machine learning, we've primarily focused on models and strategies that rely on clear targets or outcomes—what's known as supervised learning. Here, each piece of data comes with a label, like a guidebook for the algorithm to learn from, aiming to predict these labels for new, unseen data.

It's like teaching a child to identify animals by showing them pictures with names underneath; soon enough, they start recognizing lions and elephants in books they've never opened. But what about the pictures without names?

Enter the world of unsupervised learning, a type of machine learning that deals with unlabeled data, finding patterns and structures on its own. Imagine walking into an artist's studio filled with canvases of different shades of blue.

No one needs to tell you they're variations of the same color; you group them together naturally. Unsupervised learning algorithms do something similar, categorizing data into clusters without pre-existing labels, uncovering hidden patterns we might not have noticed.

Key Methods in Unsupervised Learning

Here are two key ways unsupervised learning diverges from supervised:

• ‍Clustering: Like organizing books on a shelf by genre without knowing the titles, clustering groups similar data points together. It’s used in market segmentation, social network analysis, and organizing vast libraries of content.‍
• Dimensionality Reduction: When data is a sprawling city, dimensionality reduction is the subway system making it navigable. It simplifies data, retaining only the most informative features, making complex datasets more understandable and models more efficient.

The distinction between supervised and unsupervised learning reminds us that machine learning is not just about prediction; it's also about discovery. While supervised learning models thrive on labeled data, making precise predictions and classifications, unsupervised learning ventures into the unknown, identifying hidden structures and patterns that can lead to new insights and strategies.

Keep it simple

1. Focus on Problem-Solving, Not Algorithms: Begin with a clear understanding of the problem you aim to solve with machine learning. Don't get caught up in the complexity or variety of algorithms available. The best algorithm is the one that effectively addresses your specific challenge, even if it's one of the simplest.
2. Quality Data Over Complex Models: Machine learning models are only as good as the data they learn from. Prioritize acquiring and maintaining high-quality, relevant data. A simple model trained on well-curated, representative data can outperform a complex model trained on poor-quality data.
3. Iterate and Learn: Machine learning is an iterative process. Start with a basic model to establish a benchmark. Use this initial model to learn about the behavior of your data and the performance of your algorithm. Gradually refine your model based on feedback and results, not by immediately adding complexity.

Wrapping up

This article was written to ensure your AI Strategy is not a swing and a miss. Machine learning is complex. Even the best companies have struggled to see results from large budgets and smart people spending years trying to implement ML solutions. You can do better, though. By understanding the components of ML that are simple, and the parts that can explode in complexity, you’re already off to a great start. Use the AI Strategy Framework. Keep things simple with each step up the pyramid, and by the time you reach the ML layer, you’ll be ready to see some value and make progress. If you trust the process, and stay committed, a World Series Championship is in your future. 

Need Help?

If you're seeking to unlock the full potential of AI within your organization but need help, we’re here for you. Our AI strategies are a no-nonsense way to derive value from AI technology. Reach out. Together we can turn your AI vision into reality.

‍