Benchmarking, Improving AI Model - BLEU, TER, GLUE and more

Explore non-functional testing for large language models, including ethics, adversarial testing such as prompt injection and data poisoning, and evaluating human-like behavior, consistency, and robustness in conversation.

Learn how to tune foundation models by modifying data and algorithms, running pre-trained tests, retraining on new data, validating metrics, and deploying with ongoing monitoring.

Debunk the myth of a universal ai model; emphasize task-specific models, proper benchmarking, and fine-tuning with relevant data to avoid garbage in, garbage out.

Explore what intelligence means and how artificial intelligence uses machine learning, natural language processing, computer vision, and speech to perform tasks that require understanding.

Explore how natural language processing enables computers to understand and generate human language across text, voice, and video, covering natural language understanding and generation with foundation models.

Explore the four core machine learning types—supervised, unsupervised, reinforced learning, and deep learning—through neural networks, unstructured data, and the role of large language models.

Learn how supervised learning uses labeled data to train a model, test with unlabeled data, and refine through retraining to improve dog or not dog predictions.

Reinforcement learning enables an artificial intelligence to learn from feedback by observing environment state, taking actions, and earning rewards or penalties through trial and error, as in a self-driving car.

Explain what token is in large language models as a unit of text that can be characters, words, or spaces, and show how tokenization affects prompt length and token counts.

Install python on Windows via the Microsoft Store to automatically resolve dependencies, then verify the installation in the terminal and confirm python is in PATH for benchmarking tasks.

Install Python dependencies with pip by downloading get-pip.py and running it to install pip. Then use pip install to add packages like open AI for the hands on demo.

Install miniconda to create isolated python environments, activate and deactivate them, manage dependencies with conda and yaml files, and validate setup across Windows or Mac.

install node.js and npm from the installer at nodejs.org, then verify with node -v and npm -v and adjust PATH so node and npm bin are accessible.

Discover how to access ChatGPT via API by securing a subscription with prepaid credit, adding a credit/debit card in billing, and creating an API key.

Explore the Hugging Face community page to discover open-source models (over 1,000,000), datasets, and spaces, learn to filter and run models, and preview Transformers with evaluate.

Explore the Hugging Face Transformers library, install PyTorch and Transformers, and use pipelines to benchmark and compare pre-trained models for sentiment analysis.

Explore how to load and use any hugging face model with transformers and pipelines, run local benchmarks (BLEU, TER, GLUE), and test diffusion and text-to-speech workflows.

Learn to use the Hugging Face evaluate package to benchmark machine learning models with metrics like accuracy, precision, recall, F1, and perplexity, and to run BLEU, TER, and GLUE benchmarks.

Accuracy measures the proportion of correct predictions, calculated as correct divided by total predictions. Imbalanced data and small tests can distort results, underscoring the need for relevant, high-quality training data.

Learn how to measure precision of LLMs by identifying true positives and false positives. Explore how false positives and false negatives impact model quality, illustrated with a spam detection example.

Recall measures the proportion of true positives among actual positives, highlighting false negatives; illustrated by cancer screening and fraud detection, with a case showing 80% recall on 100 positives.

Learn to calculate perplexity for a text using PyTorch and Transformers with a GPT-2 model and its tokenizer from Hugging Face, including setup and benchmarking.

Explore k-fold cross validation, splitting data into five folds with four training and one testing set, and compare basic and stratified folding to prevent bias in imbalanced data.

Learn to run a Glue benchmark by preparing an open model with a tokenizer, loading libraries, tokenizing data, fine-tuning on the training set, and submitting predictions to the Glue leaderboard.

Run and visualize a GLUE benchmark on a BERT model from Hugging Face using evaluate_glue.py, showing accuracy around 50% and the impact of training on a downstream task.

Benchmark ChatGPT against the SST-2 sentiment task using the OpenAI API, then evaluate accuracy, precision, recall, and F1 on the first 50 items to compare models.

Explore retrieval augmented generation, known as rag, and how external databases, vector embeddings, and memory extension reduce drift and hallucinations, with retriever, documentation, and generation steps.

Explore four popular rag techniques: standard rag, corrective rag, speculative rag, and graph rag. Understand how document scoring and fact checking with multi-document search improve accuracy.

Explore ragas retrieval validation framework to evaluate large language models using context precision, context recall, and relevance metrics, comparing prompt responses to ground-truth references with sample documents.

The Rag framework evaluation emphasizes fluency, relevance, coherence, and concision, using an LLM as a judge within a deep eval framework to benchmark Rag pipelines.

Analyze how training time drives cost and performance by examining data size, neural network and training hyperparameters, including epochs, learning rate, and loss functions.

Discover how memory and token limits affect chat models, explaining context, input and output tokens, and manual memory management when testing apis.