Machine Learning System Design Interview

Product Information

Valerii: We can run an A/B test to see the online performance, right? How will we see that? How long do we need to run A/B tests, etc? So all these things have to be considered. Okay. Now, let's say I told you about the basic features. What about feature engineering? Like I said, linear regression doesn't take nonlinearity into account. Can I do that with basic feature engineering? Probably, if you have enough data, just having a polynomial of the second degree, which helps you find an overlap between features – how they interact with each other – is enough. Because if you have trillions of data points, you can do that, because sparsity is not an issue here. And so on, and so on, and so on and so on. ( 16:43) Alexey: The typical components of a machine learning system – this is the first part of the question – are things like data pipelines, data preparation, things to calculate features? ( 49:35) The book consists of two parts. The first part provides an overview of the machine learning interview process, what types of machine learning roles are available, what skills each role requires, what kinds of questions are often asked, and how to prepare for them. This part also explains the interviewers’ mindset and what kind of signals they look for.

Valerii: Well, I was able, to some extent. I managed this. Because look, I mean, come on. Batch norms – there is some normalization. So? Okay. ( 33:19) Preparing for ML system design interviews However, based on my experience and research, it’s very common for consumer big tech companies to ask questions about recommender systems in their system design. This makes sense: ML aims to solve a multitude of complex problems. It has made rapid progress in areas like speech understanding, search ranking, and credit card fraud detection. Companies are leveraging these technologies across industries from healthcare and agriculture to manufacturing and retail.Hammer out timing SLAs (eg. we’ll incorporate user actions into recommendations within X seconds/minutes/hours) Alexey: Okay. If we take an ecommerce company – a small one – then we can think about what kind of questions they may ask candidates. It could be about designing a search system, designing a recommender system –the typical things that they do. However, when it comes to Facebook, Facebook does so many different things, so you can never know exactly what kind of domain you might get. They might ask you to design a newsfeed, for example. Or they might ask you to design a point of interest recommender system, or a fraud detection system for WhatsApp, right? It could be anything. ( 36:21) You also want to bring up technical scaling requirements (don’t make assumptions, it’s key to clarify this out loud):

Get Book Machine Learning, Multi Agent And Cyber Physical Systems - Proceedings Of The 15th International Flins Conference (Flins 2022) by Qinglin Sun,Jie Lu,Xianyi Zeng,Etienne E Kerre,Tianrui Li Pdf Make sure you bring up how you would launch the system and actually evaluate whether it’s achieving its business objectives. This is almost always via A/B testing, which has lots of its own nuances. Talk about which metrics you’d measure and statistical tests you’d perform for an A/B test. You can go into some depth talking about ramping patterns and issues that arise with A/B testing. Model Lifecycle Management Want to learn Data Science from scratch with the support of a mentor and a learning community? Join this Study Circle for free: https://community.aigents.co/spaces/9010170/ Obviously there’s many more items here. Notice that the concepts are still vague, and would require clarification to actually use in a model. Eg. don’t just leave a feature as ‘history of items liked’, that’s not a numeric value you can train a model with. Feature RepresentationAlexey: I have an example from my personal experience of being interviewed at one of these companies on system design. I had the question to design a system for finding places of interest. So let's say I go to London – I go to whatever central square you have in London, and the system would need to give me all the points of interest, all the closest interesting places. ( 22:05) The standard development cycle of machine learning includes data collection, problem formulation, model creation, implementation of models, and enhancement of models. It is in the company’s best interest throughout the interview to gather as much information as possible about the competence of applicants in these fields. There are plenty of resources on how to train machine learning models and how to deploy models with different tools. However, there are no common guidelines for approaching machine learning system design from end to end. This was one major reason for designing this course. April 29th: I launched mlengineer.io blog so you can get latest machine learning interview experience. Suppose you just heard from the recruiter that you passed your phone screen. Congratulations! Your reward is a full day of interviews, including systems design. How can you prepare ahead of time? We’ve trained our model with the hyperparameters that led to the best evaluation metrics in our holdout data. Should we just launch this to the entire user base? Unless you’re fresh out of school, you should know the answer is NO!

Within each data source, you can iterate on the types features available. It’s good to call out some example specific features, but it would take too long to be exhaustive about these. Eg. for a Facebook user you have features like:

Alexey: Yeah. But where do we actually design systems? Or this is what you mean by that? Do we need to say “This system is doing this and then there is another system?” Or it’s about designing the… ( 25:30) Congrats! You have learned about implementing introductory ML system concepts and how to approach interview questions based on system design concepts. There’s still a lot to learn about ML system design. Valerii: Well, you can find me on LinkedIn. Just type in my name, you use a y instead of ii. With the new rules, it should be ii at the end. ( 1:00:12) You should also discuss how you’d organize the data for evaluation, eg. k-fold, holdout etc. You should discuss the metrics you’d use to compare models. For classifiers you should discuss what’s more important, precision or recall, especially considering their effect on users. How does it affect the user experience to have a ‘false positive’ versus missing the ‘right’ answer? Modelling Techniques Alexey: I think this is what happened to me, but this is something that I prepared for later. So, you said that important interviews for detecting, or assessing your level are: behavioral interview, system design interview, and machine learning system design interview. Can you tell us – what is the difference between system design and machine learning system design? ( 13:36)