#2 Project update - General information regarding tools and packages in Python

In this journal, I will go over some general information and the tool that I will use to do this project. This is what I've searched and learned so far.

In order to complete this project, we will go over 7 steps. That is also considered to be the typical steps to build a machine learning model in general.

                               1. Import the Data

Kaggle.com is a great source for dataset, where we can find a lot of precious dataset. In this project, I will use the Movie List dataset. In this step, we will use Numpy package in Python for creating dataframe to hold the dataset. 

Numpy stands for Numerical Python, this is a general-purpose array-processing package in Python. We will utilize this package to deal with array. 

I will attach here the link of Numpy in case someone wants to get to know more about it:

  • https://www.w3schools.com/python/numpy

                               2. Clean the Data

Clean the data is a very important step, because our model is unable to work with missing data. Therefore, we need to prepare the data carefully by the way filling out the missing data by the mean value or dropping them.

In this step, we will use Pandas package in Python for cleaning and analyzing the data. "Pandas is a Python package that provides fast, flexible, and expressive data structures designed to make working with "relational" or "labeled" data both easy and intuitive".

There are 5 methods that I plan to use in this project: filter, sort_value, agg, query, and assign.

Here is some links related to Pandas package in Python that I've learned from:

  • https://www.w3schools.com/python/pandas/
  • https://pypi.org/project/pandas/
  • https://www.sharpsightlabs.com/blog/python-pandas/

                               3. Split the Data into Training/Test Sets

In machine learning, we will use 80% data for training the model and 20% for testing. 

                               4. Create a Model

                               5. Train the Model

Besides some other packages, we will learn Scikit-learn (sklearn in short) for creating and training the model. There are a ton of interesting and useful modules and methods in sklearn package that we need to explore. I think this package contributes 70% to the success of this project. This is must-learn package for students who want to proficient in Python especially and machine learning generally.

I am still learning this package, so I will update more about it in the next journal. Here is some sources that I used for learning sklearn package. 

  • https://www.activestate.com/resources/quick-reads/what-is-scikit-learn-in-python/
  • https://www.machinelearningplus.com/nlp/cosine-similarity/
  • https://www.youtube.com/watch?v=ueKXSupHz6Q&t=154s

                               6. Make Predictions (Not learned and searched yet)

                               7. Evaluate and Improve (Not learned and searched yet)

Step 4 and 5 will be 2 steps that will take me more time to learn and complete, they're considered to be the core section of this project. I will continue do more research and learn about them, then I will make a better update in the next journal. 

Comments

  1. EricaOctober 18, 2021 at 6:41 AM

    Bao, great work. Have you implemented these strategies yet or still working out your specific analysis pathway? I can’t wait to see your project in greater detail.

    ReplyDelete
    Replies
    1. Bao Huynh's BlogOctober 20, 2021 at 2:29 PM

      I've never implemented these strategies before this project. But I am working on it, you can expect to see more detail information in the next journal.

      Thanks for your feedback, I greatly appreciate it!

      Bao.

      Delete

Post a Comment

Popular posts from this blog

#8 Sklearn - Python package - Model evaluate metrics for regression

Image
    One of the most important steps in building prediction model is evaluating how accurate our model is by the way using metrics to calculate the amount of deviation between predicted value and actual value. In this journal, we will dive into 4 metrics, they are Mean Absolute Error, Mean Square Error, Root Mean Spare Error, and R-squared score. All the functions we use that come from sklearn Python package (from sklearn import metrics).        Mean Absolute Error   (MAE) is the mean of the absolute value of the errors. In other words, we use the differences of each actual value with predicted value then divide by its number of observations. It measures the average of the residuals in the dataset. That means the weight of all the errors are the same. For example, the difference of first pair yi1 and y^i1 is 5, the difference of second pair yi2 and y^i2 is 1. The average of them will be 3. Then the error will be off by 3. So, "MAE  is bes...
Read more

#3 Project Progress: Import and Clean the Data in Python

Image
I am so excited to share with you about my first project in Python related to Data Science. In order to complete this project, first of all, we need to import the data. Jupyter notebook is a interactive computing notebook environment, that is came with Anaconda - a Python distribution for scientific computing. We will run our code on this notebook. We will use read_csv() method in pandas package to read the csv files. My file here is Data.csv, that I download from Kaggle ( TMDB 5000 Movie Dataset | Kaggle ). df.head() uses for printing some head rows of data. df.shape() uses for printing the shape of data, you can see in this data set, we have 8403 rows and 20 different features.  After importing the data, we need to clean them to make sure we won't mess them up when we manipulate to build a model. We have total 20 features here but we won't use them all. So, we need to decide what features are important and what are not. Because I want to build a content-based movie recommenda...
Read more

#10 Fighting The Semantic Gap On CBIR Systems

Image
      CBIR which stands for Content Based Image Retrieval is a technique that involves querying an image. Which you then retrieve matches to the queried image based on its similarity results. It typically involves comparing the low-level features of an image, including color, texture, and shape. However, what if you want to compare high level features? High level features mean, the semantic meaning behind an image that humans are basically able to discern. For example, say you want, a tall happy person, or a happy small dog. A person would be able to pick out images pretty easily from a set that match these criteria, a system not so much. What exactly can be done to bridge this Semantic Gap?       The academic article “Fighting the Semantic Gap on CBIR Systems through New Relevance Feedback Techniques” mentions using relevance feedback techniques to combat this.” The paper describes the process as “Therefore, by gathering the users´ indications, algori...
Read more