Benchmarking, Improving AI Model - BLEU, TER, GLUE and more

Debunk the myth of a universal ai model; emphasize task-specific models, proper benchmarking, and fine-tuning with relevant data to avoid garbage in, garbage out.

Explore how natural language processing enables computers to understand and generate human language across text, voice, and video, covering natural language understanding and generation with foundation models.

Explore the four core machine learning types—supervised, unsupervised, reinforced learning, and deep learning—through neural networks, unstructured data, and the role of large language models.

Learn how supervised learning uses labeled data to train a model, test with unlabeled data, and refine through retraining to improve dog or not dog predictions.

Reinforcement learning enables an artificial intelligence to learn from feedback by observing environment state, taking actions, and earning rewards or penalties through trial and error, as in a self-driving car.

Explain what token is in large language models as a unit of text that can be characters, words, or spaces, and show how tokenization affects prompt length and token counts.

Install python on Windows via the Microsoft Store to automatically resolve dependencies, then verify the installation in the terminal and confirm python is in PATH for benchmarking tasks.

Install Python dependencies with pip by downloading get-pip.py and running it to install pip. Then use pip install to add packages like open AI for the hands on demo.

Explore the Hugging Face Transformers library, install PyTorch and Transformers, and use pipelines to benchmark and compare pre-trained models for sentiment analysis.

Recall measures the proportion of true positives among actual positives, highlighting false negatives; illustrated by cancer screening and fraud detection, with a case showing 80% recall on 100 positives.

Learn to calculate perplexity for a text using PyTorch and Transformers with a GPT-2 model and its tokenizer from Hugging Face, including setup and benchmarking.

Learn to run a Glue benchmark by preparing an open model with a tokenizer, loading libraries, tokenizing data, fine-tuning on the training set, and submitting predictions to the Glue leaderboard.

Benchmark ChatGPT against the SST-2 sentiment task using the OpenAI API, then evaluate accuracy, precision, recall, and F1 on the first 50 items to compare models.

Explore retrieval augmented generation, known as rag, and how external databases, vector embeddings, and memory extension reduce drift and hallucinations, with retriever, documentation, and generation steps.

The Rag framework evaluation emphasizes fluency, relevance, coherence, and concision, using an LLM as a judge within a deep eval framework to benchmark Rag pipelines.

Analyze how training time drives cost and performance by examining data size, neural network and training hyperparameters, including epochs, learning rate, and loss functions.