Agenda Machine Learning Week 2024

Phoenix, AZ June 4-7, 2024

In this keynote, you’ll learn how open-source models can help you build high-quality AI applications, generative or not, while giving you more flexibility, control, and ROI than closed-model APIs. We’ll highlight the latest and greatest models, and show you how to get started with them in minutes. Along the way, you’ll also learn about the technical ecosystem that Hugging Face is fostering, from models and datasets, to cloud integrations and hardware acceleration.

The greatest tool is the hardest to use. Machine learning is the world’s most important general-purpose technology – but it’s notoriously difficult to launch. Outside Big Tech and a handful of other leading companies, machine learning initiatives routinely fail to deploy, never realizing value. What’s missing? A specialized business practice suitable for wide adoption.

In this keynote, Machine Learning Week Founder Eric Siegel presents a six-step practice called bizML for ushering machine learning projects from conception to deployment. This disciplined approach, which is the topic of Siegel’s new book, The AI Playbook, serves both sides: It empowers business professionals and it establishes a sorely needed strategic framework for data professionals.

For those of us who’ve spent any time arguing with customer service chatbots or screaming “REPRESENTATIVE!” to an interactive voice response (IVR) line, the hype around generative AI improving customer experience might seem hard to believe. Sure, ChatGPT can do some impressive things, but do we really think AI can make the painful process of dealing with our insurance companies, wireless carriers or utilities any easier? The answer is yes, but not by building a better bot.

This presentation will explore how the real breakthrough created by GenAI is making the entire operation smarter and more personalized, so there are fewer reasons to deal with customer service in the first place.

They say “You get what you measure and track”. A data culture is no exception. Many organizational studies show a lack of a data culture are holding business organizations back from fully adopting AI and Advanced Analytics (ML). An organization develops a data culture, which encompasses the common language, norms, and values for utilizing data. In this talk, we will discuss the organizational mechanism on how to develop, measure, and drive an organization’s data cultural capabilities using Advanced Analytics.

Automated Machine Learning – so-called AutoML —  has received considerable attention in recent years and is poised to take enterprise analytics to the next level. Most often, however, automation has been limited to the model-building algorithms themselves, such as hyper-parameter tuning and model ensembles. It appears that Insufficient progress has been made with the most time-consuming parts of the machine learning process: data preparation, model interpretation and model deployment. This talk will describe why attention in these steps has been slow in coming and practical recommendations for automating them.

Data science is becoming increasingly relevant in the field of manufacturing, primarily in the areas of Materials Requirement Planning (MRP), Master Production Schedule (MPS) and Distribution Requirements Planning (DRP). This abstract focuses on the application and influence of AI and data science in these areas. Data science leverages statistical models, machine learning, forecasting and optimization to analyze and interpret complex data sets. In the manufacturing context, these techniques can be used to optimize operations, predict demand, and improve supply chain management. Specifically, MRP can utilize data science to optimize Raw material inventory levels, reduce lead times, and anticipate material requirements from downstream operations. MPS can leverage data science to enhance production scheduling and planning, minimize downtime, and improve capacity planning. DRP can benefit from data science by improving the accuracy of distribution plans, reducing stockouts, and minimizing holding costs. The integration of data science into MRP, MPS and DRP can lead to more efficient and effective manufacturing processes, thus enabling companies to gain a competitive edge in today’s data-driven business environment. However, the successful implementation of data science in manufacturing requires strong data infrastructure, skilled personnel, and a culture that encourages data-driven decision making.

A modernization journey is underway at New York Life where we’re bending the company’s core around data, tech, and AI as partners and enablers to deliver exceptional experiences for our clients and agents. The arrival of Generative AI has been a catalyst moment as we work to leverage this nascent technology to further drive organizational transformation. As we continue to work toward systems-level disruption, today’s singles and doubles will build to homeruns. This requires a reboot of technology, process, and people.

From a people perspective, continuous upskilling is essential, particularly as it relates to embracing the power of AI and other new technologies. This talk will feature insights and lessons learned from leading a variety of learning initiatives for New York Life’s employees over the past seven years. This includes the rollout of workshops, learning pathways, expos, hackathons, gamification and other formats and methods to create engaging and effective learning experiences that:

• meet employees where they are in terms of expertise and interest
• dispel concerns, particularly concerning GenAI
• generate excitement among employees
• motivate and reward learners and incentivize others to join

Those seeking new ideas and inspiration for learning initiatives are sure to come away with applicable tips and strategies.

As Machine Learning continues to forge its way into diverse industries and applications, optimizing computational resources, particularly memory, has become a critical aspect of effective model deployment. This session, “Memory Optimizations for Machine Learning,” aims to offer an exhaustive look into the specific memory requirements in Machine Learning tasks and the cutting-edge strategies to minimize memory consumption efficiently.We’ll begin by demystifying the memory footprint of typical Machine Learning data structures and algorithms, elucidating the nuances of memory allocation and deallocation during model training phases. The talk will then focus on memory-saving techniques such as data quantization, model pruning, and efficient mini-batch selection. These techniques offer the advantage of conserving memory resources without significant degradation in model performance.Additional insights into how memory usage can be optimized across various hardware setups, from CPUs and GPUs to custom ML accelerators, will also be presented.

In an era where personalization has become the norm, businesses must evolve to meet the increasing expectations of consumers. The session delves into the realm of hyper-personalization, exploring cutting-edge strategies that go beyond surface level customization to deeply engage and resonate with individual customers. We will discuss business context and high-level model architecture of a hyper-personalization framework in the presentation.

In this workshop, Sergey Yurgenson, FeatureByte’s Head of Data Science and Kaggle Grandmaster, will lead attendees through an exploration of the fusion of human ingenuity and AI prowess in feature engineering. The session offers a hands-on demonstration of FeatureByte’s platform, showcasing a feature engineering use case and the transformative potential of GenAI Copilots. Attendees will see how AI augmentation accelerates feature ideation, comparing traditional human-only methods with GenAI-assisted approaches.

Through interactive engagement, participants will gain practical insights into optimizing feature selection and extraction with AI. Attendees will also receive complimentary trial access to FeatureByte’s platform for direct experimentation, enhancing their understanding of the discussed concepts. Join us to discover how GenAI-assisted feature engineering is reshaping data science, providing actionable techniques to amplify your data-driven insights.

Agile processes designed for software engineering often create friction for ML teams.  We’ll talk through some ideas on handle that friction, including how to set deliverables for AI/ML teams, how to handle the uncertainty/experimental cycle of AI/ML when interfacing with product and engineering teams, and how important expectations are to running an “agile” ML team.

There’s a fundamental problem with the typical model development process: It evaluates models in terms of technical metrics like AUC/precision/recall without also including business metrics like profit/ROI/savings – the stuff that actually matters to the company.

This is a serious problem – if you aren’t measuring business value, you’re not pursuing business value. Further, those technical metrics fail to provide your client/stakeholder meaningful visibility – she doesn’t care about AUC. How is she supposed to authorize deployment?

That’s why Machine Learning Week founder Eric Siegel recently co-founded Gooder AI. It addresses this fundamental issue by way of its SaaS product, the first full-scale platform for machine learning validation – to maximize the value of models by testing and visualizing their business performance.

Spoiler alert: Unlike technical measures, a model’s business performance (profit, savings, etc.) depends on how you use it. So assessing the business value requires a specialized visualization solution, one that allows you to interactively try out what-if deployment scenarios. This includes setting various business inputs, which are subject to change, and moving the decision threshold to estimate the potential deployed value.

Come watch Eric demo Gooder AI and show how it drives ML deployment to maximize business impact.

With the explosion of Data and AI capabilities, “Citizen Data Scientists” have popped up all across our organizations. The first reaction may be to try to squelch their activities in an effort to control the risk of inaccurate data storytelling and misinformation. But a more effective strategy is to embrace these data practitioners and enable them with tools and best practices to further accelerate the data and AI journey. Hear how Paychex has launched the Data and AI Community of Practice, leveraging proven techniques to engage the community and drive organizational value.

ML platforms help enable intelligent data-driven applications and maintain them with limited engineering effort. Upon sufficiently broad adoption, such platforms reach economies of scale that bring greater component reuse while improving efficiency of system development and maintenance. For an end-to-end ML platform with broad adoption, scaling relies on pervasive ML automation and system integration to reach the quality we term self-serve; a quality we define with ten requirements and six optional capabilities. With this in mind, we identify long-term goals for platform development, discuss related tradeoffs and future work. Our reasoning is illustrated on two commercially-deployed end-to-end ML platforms that host hundreds of real-time use cases at Meta — one general-purpose and one specialized.

To stay competitive as technology evolves and user interactions become more direct, it is essential to find data-driven ways to continuously improve customers’ experiences. In this talk, we’ll showcase an example of how we helped a company in the hospitality industry employ machine learning models to generate real-time predictions for how long their customers would have to wait for an order. We’ll highlight the components of the solution, which utilizes gradient boosting quantile regression, as well as key data inputs, feature engineering and selection techniques, and model monitoring considerations.

Many industries are facing economic headwinds. In such an environment, technology investments become challenging. How do you determine how much to invest in data, traditional AI, Generative AI, and other priorities? In this session, Bipin Chadha will discuss these questions and explore how CSAA Insurance Group is approaching the tradeoffs

He will present his data and AI strategy, highlighting how CSAA is:

• Investing in data infrastructure and data governance
• Enabling data professionals to generate insights that drive value for the business
• Investing in data science and AI capabilities and modeling governance to become more data driven
• Investing in experimentation with the emerging Generative AI capabilities

There are several techniques available in R to evaluate the performance of a classification model. This presentation will demonstrate a sample classification model followed by some practical insights related to selected model performance evaluation techniques available in R. Specific focus on reports that can be generated to summarize model performance results will be addressed. The presentation will also recommend further areas of application and research.

A case study will be presented which describes a process for identifying relevant topics to a customer in a specific domain based on a corpus of user comments. We will review in detail how the process is created, how the customer interacts with the process to curate their own labeled data, and how that data is used for downstream analysis and modeling tasks. We will describe how we monitor the results for accuracy and how we manage changes over time. We will show how this process is domain agnostic and easily repeatable.

One of the biggest challenges facing AI today is the fact that, despite volumes of guidance detailing how AI should be safe, ethical, and responsible, many organizations don’t have established standards and practices to enforce responsible AI.  Responsible AI requires firm model development governance standards, including algorithms allowed, processes followed, testing completed, and auditability to ensure accountability to meeting responsible AI standards. In this keynote address, FICO CAO Scott Zoldi will discuss three pillars of Responsible AI: explainable AI, ethical AI, and auditable AI in the context of life-altering decisions derived from AI models. Establishing a corporate model development standard and enforcing adherence through auditable AI not only allows meeting regulatory rules and guidance, but further establishes customer trust and safe usage of AI.

Meet new people for dinner at a restaurant in walking distance and network in a relaxed setting with good food and favorite beverages (everyone pays for their own dinner). We will publish the restaurants closer to the date.

As a UPS Manager of 43 years, Jack Levis witnessed tremendous change.  UPS became an Airline, a Global Logistics Player, and integral to the E-Commerce explosion. Of all the changes Jack witnessed, Data and Technology was arguably the largest and most impactful. As a Senior Director responsible for the Digitization and Advanced Analytics of Operations, Jack oversaw much of that technology transformation. The impact was tremendous with many hundreds of millions of dollars in cost savings while simultaneously providing additional services to customers.

In this talk, Jack will discuss lessons learned in successfully managing the breakthrough innovation.  While the tools, algorithms, and mathematics are important, he will also discuss the need for psychology in terms of culture, leadership, and change management. Using examples from his journey, Jack will describe Ten Steps to Innovation through Math and Psychology. 

Join us for an insightful journey into the heart of PayPal’s dynamic Machine Learning & Data Analytics community. In this talk, we’ll unravel the story behind the creation and growth of our community, exploring the strategies and initiatives that transformed it into a thriving hub of innovation.

Discover how we fostered collaboration and knowledge-sharing among diverse talents, creating an environment that empowers professionals to harness the power of machine learning and data analytics. From the inception of our community to the implementation of impactful projects, we’ll delve into the challenges faced and lessons learned, offering valuable insights for those looking to cultivate similar communities in their organizations.

Whether you’re a seasoned data scientist, a curious newcomer, or a leader eager to promote a culture of continuous learning, this talk promises to provide practical takeaways and inspiration. Join us and be part of the conversation that shapes the future of data-driven excellence at PayPal.

Ever wanted some really practical tips and tricks on how to make Machine Learning work for you? Perhaps a little less code and a bit more nuts and bolts? Maybe even some insider insights from a career spanning almost four decades? In this session, you will get lightning-fast, perfectly-practical prediction tips that you can apply right away. You don’t need a technical background for the session, and although practitioners are welcome. You will leave with insights you’re unlikely to hear anywhere else in this entertaining session.

Our business partners are besieged by statements of how data science is the secret to “auto-magically” understanding customers and predicting their decisions. However, the Venn diagram defining data science to our business partners continues to explode with overlapping expectations and responsibilities, all while the practice evolves from an individualized to industrialized scale.
With a constantly changing landscape of ever advancing capabilities and tools, how’s a company to successfully deliver lasting and quantifiable value from data science engagements? By learning from our mistakes!
In this talk, Corwin Smith will share several real examples where, with the best of intentions (and approaches), data science projects failed, what we should have done differently and how we future proofed it from happening again.

In the field of Artificial Intelligence and Machine Learning (AI/ML), a significant challenge lies not only in developing accurate models but also in effectively utilizing them to drive business decisions. This presentation aims to discuss the application of causal ML techniques in influencing important business decisions. By leveraging causal ML, we can uncover the causal relationships between variables, enabling us to understand the impact of various factors on business outcomes. This understanding empowers organizations to make informed decisions and optimize their strategies for success.

Furthermore, this presentation will highlight the importance of engaging business partners from the outset of the AI/ML implementation process. By actively involving them, we gain valuable insights into their existing manual decision-making processes. This understanding allows us to develop an iterative plan that aligns with their needs and preferences, ensuring their comfort and buy-in throughout the implementation journey. Additionally, a rigorous test and learn approach is designed to measure the success of the implemented AI/ML solution, providing tangible evidence of its effectiveness in driving business decisions. This collaborative and iterative approach not only enhances the adoption of AI/ML but also fosters a culture of continuous improvement and data-driven decision-making within organizations.

Events in recent years have led to a fresh wave of discussions about racial justice and equality in the United States. This has led to an increased focus in the insurance industry and regulatory community on bias and equity. However, a lack of consistent data collection is often a significant barrier to the study of disproportionate impacts and equity across race/ethnicity cohorts in various contexts.

In this presentation, we describe a range of techniques for developing probabilistic estimates or predictions of individual race and/or ethnicity. We will show how to apply some of these methods to a simulated dataset to illustrate how to use them in practice. In addition, we will share results from a case study that assesses the predictive performance of these probabilistic estimates using an actual dataset from the insurance industry that has self-reported race/ethnicity recorded.

Neural Networks are flexible, powerful, and often very competitive in accuracy.  They are criticized primarily for being uninterpretable black boxes, but their chief weakness is that backpropagation makes them unrepeatable.  That is, their final coefficient values will differ, from one run to the next, even if the NN structure, meta-parameters, and data are held constant!  And unlike multi-colinear regressions, the varied NN coefficient sets aren’t just alternative ways — in an over-parameterized model — of producing similar predictions.  Instead, the predictions can vary a disquieting amount and often “converge” to a significantly worse training fit than is possible.  

What happens if one instead employs a global optimization algorithm to train a NN?  Untapped descriptive power should be unleashed, encouraging use of simpler structures to avoid overfit.  And, with randomness removed, the results will be repeatable.  We’ll demonstrate initial results for (the relatively small) NNs practical to optimize.

Machine learning projects can’t deploy without business-side buy-in… nor can they achieve deployment-ready maturity without business-side input and collaboration. Often, data scientists deliver a viable model, but the stakeholders aren’t ready for the pass and fumble the ball. How must ML professionals prep them and prep the model accordingly? And, perhaps more importantly, how do project leaders engage stakeholders so that they readily supply their business-side expertise to help guide the project from conception through launch? In this session, our expert panelists will weigh in.

Machine learning projects require careful balance, but relying solely on one team from start to finish can be detrimental. This talk delves into the advantages of segmenting the machine learning development process, using a real-world use case of AI-centered development for human resources (with Opptly). Discover how breaking down the process and the teams responsible leads to a more favorable experience and improved outcomes for everyone involved. Get ready to split the AI atom and unlock the potential of machine learning!

Data discovery and understanding is the second step in the machine learning development cycle, right after problem identification. Meanwhile, data lakes are growing in size and scope, making the task of identifying the local neighborhood of tables relevant to a given problem harder. Additionally, for those who do consulting, this is a constant problem with every new customer engagement. In order to address this bottleneck we present a new, fully automated solution to data discovery and understanding in data lakes and data warehouses.

The resilience of countries has never been more important in light of the Russian invasion of Ukraine. This presentation describes the general concept of Country Resilience and the baseline indicators that are measuring it. It will further describe how Machine Learning and AI can be used to simulate, track, alert, and assist in recovery from an unexpected interruptions.

In last year’s Rexer Analytics Data Science Survey, data scientists reporting three interesting things:

1. their models are rarely getting deployed, 
2. return on investment was the single most important metric for measuring projects and 
3. ultimately decision makers were unwilling to approve the operational changes needed to get to deployment.

In this session, Kate will share six lessons she learned as a tech founder that changed her mindset and helped her build successful data science “MVP”s (minimal viable products) that pass the “pitch” stage with her stakeholders.

Explore the intersection of logic and AI in “Decoding Paradoxes.” Learn how fallacies like the Streetlight Effect and Survivorship Bias influence AI decision-making and algorithmic biases. Discover how these paradoxes affect AI development and deployment. Join us to deepen your understanding of AI’s future. Topics include:

• Cognitive fallacies that often lead to biased models and flawed decision-making
• Strategies for training models to recognize and mitigate the influence of fallacies
• The challenges in interpreting machine learning models affected by fallacies and methods to enhance model transparency
• The ethical dimensions of cognitive fallacies in ML
• Real-world case studies and examples where cognitive fallacies have played a significant role.

Thought leaders in machine learning Dean and Karl, field questions from the audience about strategies for machine learning projects, best practices, and tips, drawing from their decades of experience as consultants and company executives.