Data Science: Natural Language Processing (NLP) in Python

Explore regression in Python by building linear regression and a random forest regressor on the airfoil dataset, using a train-test split, and making predictions.

Explore future topics in machine learning and deep learning, including supervised classification and regression, unsupervised clustering, reinforcement learning, hyperparameters, training versus test sets, and real-world applications.

Describe the Markov property as the key assumption behind Markov models, illustrating sequence probability factorization with the chain rule and Bayes rule, and note its usefulness despite Box's quote.

Explore why maximum likelihood estimates can fail with unseen word pairs and apply smoothing techniques such as +1 and epsilon, and use log probabilities to assess sequence likelihoods.

Build a text classifier using Markov models on line samples from Edgar Allan Poe and Robert Frost. Convert text to integers with tokenization and apply add one smoothing.

Explore how Markov models power NLP by predicting the next value from past values, from first-order to higher-order, with Python implementations and applications in finance, reinforcement learning, and speech recognition.

Explore the substitution cipher, where a one-to-one mapping encrypts and decrypts messages, illustrated with a concrete example of encoding and decoding.

Build a character-level language model with n-grams and a Markov assumption to score real versus unreal sentences. Use add-one smoothing, compute sequence probabilities, and decode by maximizing log-likelihood.

Encode a substitution cipher as a DNA string and mutate by swapping. Evaluate fitness with the log likelihood of the decrypted message and select the best.

Demonstrate creating a substitution cipher by building a true mapping from two letter lists, randomizing one side, and pairing letters with zip, to test a decryption algorithm in a simulation.

Explore a genetic algorithm that evolves a dna pool of 20 shuffled strings to decrypt a cipher, scoring by log likelihood of decoded text and selecting top five across generations.

The lecture demonstrates a cipher decryption workflow in Python, using log likelihood to compare decoded and true messages, analyze wrong letters, and plot objective per iteration to illustrate convergence.

Explore cipher decryption using substitution ciphers, build a character-level language model with the Markov assumption and add one smoothing, and apply genetic algorithms to optimize results.

Learn to focus on text classification concepts rather than the underlying algorithms, as spam detection exercises show that any suitable classifier works, with Naive Bayes performing well for text features.

Detect SMS spam by transforming text with tf-idf and count vectorizers, and compare Naive Bayes and boosting classifiers on the spam collection data.

Reclassify the sms spam dataset and build a text classification pipeline using count vectorization and train-test split, achieving about 98% accuracy; visualize spam and ham word clouds and analyze misclassifications.

Share feedback through a simple suggestion box form to tailor the course to your background, course, and difficulty. Highlight missing explanations or topics to shape future sessions and materials.

Turn tokens into a binary bag-of-words feature vector of size v using a word-to-index dictionary, then normalize by document length to obtain proportions for logistic regression.

Explore how to improve sentiment analysis in NLP with Python by examining data shuffling, baselines, benchmarks, and the role of interpretability.

Explore why more NLP material isn't only about NLTK and learn to build your own feature vectors using machine learning models like naive Bayes and SVM with scikit-learn.

Explore the math behind latent semantic analysis with singular value decomposition, revealing latent factors and reducing redundancy through dimensionality reduction.

Apply latent semantic analysis in Python to preprocess book titles, build a term document matrix, and visualize clusters with a two-dimensional scatter plot.

Create an article spinner in Python by building trigram probabilities from Amazon electronics reviews parsed with Beautiful Soup, then sample randomly to generate rewrites and assess limitations.