API Performance Testing With K6, Github Copilot, and ChatGPT

In this lesson, we learned about large language models (LLMs), a type of artificial intelligence that understands and generates human-like language. We also discussed how LLMs work and the importance of providing a clear and specific prompt when using an LLM as an assistant. Additionally, we emphasized the need for healthy skepticism when relying on LLM-generated output, as errors and unintended meanings may occur. Overall, understanding LLMs and prompt engineering can be a valuable tool in various domains, such as healthcare, finance, education, and entertainment.

The lesson emphasizes the dynamic nature of large language models (LLMs) like ChatGPT and Github Copilot, highlighting their rapid evolution and non-deterministic behavior. Students are cautioned that responses may vary, requiring validation and adjustment. While the course utilizes paid versions for accuracy, free versions are available. Additionally, it discusses differences in responses between ChatGPT and Copilot, underscoring the need for critical evaluation and human expertise.

This lesson informs the users that the prompts and source code for the course are available for download from the "Resources & References" section. However, simpler or less important prompts are not available for download. If there is a minor difference between downloadable items and video, it should not be a problem.

In this lesson, we learned about the principles of prompt engineering, which involves designing effective prompts as input for the AI assistant. The first principle is to write clear and specific instructions, which can be achieved by using delimiters to separate distinct parts of the input. Clear and specific instructions help guide the model towards the desired output, producing detailed and relevant responses.

In this lesson, we learned how AI-powered assistants like ChatGPT and Copilot can help us write code by taking English prompts and returning the source code syntax. These assistants can also help new learners understand syntax and explain code to a certain extent. While these AI assistants may not be 100% accurate, they can be a great starting point for those who are new to programming and need help with syntax and code comprehension.

Software testing is a crucial process that ensures the quality, reliability, and performance of software. It involves evaluating software behavior against predefined requirements. Test cases and test scripts help ensure consistency and repeatability in testing procedures. Various types of testing tools, such as automation, performance, and bug tracking tools, help overcome challenges in software testing.

This lesson introduces two AI assistants in software engineering and testing: ChatGPT and GitHub Copilot. ChatGPT is designed for natural language tasks, while GitHub Copilot focuses on code-related tasks. Best practices for integrating them into software engineering workflows are provided, including complementary usage, continuous feedback, and ethical considerations. It's important to be mindful of the limitations of these tools and review their outputs to ensure they meet project standards and requirements.

In this lesson, we learned about using ChatGPT as a web application to generate text. We also discussed the importance of having a GitHub account for using GitHub Copilot and how to set it up with Visual Studio Code. We explored different ways to interact with GitHub Copilot, such as using the chat feature, typing source code comments, and using regular source code. Additionally, we learned about the built-in functions available on GitHub Copilot, such as "/explain" and "/fix."

This lesson outlines the source code structure that will be used throughout the course. Each lesson will have a folder named after the lesson title, containing relevant files and scripts. Additionally, each folder will have a subfolder named "ai," which will contain files related to the AI assistant. The lecture number may vary, but the order of the folders will follow the order of the lectures.

In software development, functional and non-functional requirements are crucial. Functional requirements describe what functions the software must have, while non-functional requirements describe how the system should perform. Performance testing is a type of testing that measures the qualitative aspects of a user's experience, such as latency and reliability. There are several types of performance testing, including capacity testing, soak testing, load testing, spike testing, stress testing, smoke testing, and scalability testing. Ensuring the system meets both functional and non-functional requirements is essential for delivering a good user experience.

K6 is an open-source performance testing tool that helps developers and testers measure and analyze software system performance. It simulates virtual users (VUs) and their activities to test non-functional aspects such as load, stress, and scalability. The tool offers various terminologies like throughput, latency, iterations, ramp-up, ramp-down, user flows, and load profiles to understand the testing process better. K6 is highly relevant for load, stress, and soak testing, which helps to evaluate software system performance under expected traffic.

In this lesson, we learn how to install k6, a load testing tool, by accessing the k6 website and following the installation guide for various operating systems.

In this lesson, we create a basic K6 script, "hello k6," using Visual Studio Code. We learn to handle file naming conventions in JavaScript, execute HTTP calls, and log responses. Running tests via command line or VS Code extension is demonstrated, ensuring a smooth start to your K6 journey. Explore various methods to create and run K6 scripts effortlessly, setting the stage for comprehensive testing.

In this lesson, we leverage AI assistants like Github Copilot to create a K6 script, "Hello k6." Copilot provides code samples and explanations, facilitating script creation.

K6, a command-line tool, offers various functionalities accessed via terminal commands. Utilize "k6" to list available functions, run with "k6 space [command]." Explore options with "--help" for detailed command usage. Tailor test scenarios with options like virtual users ("-u"), iterations ("-i"), or duration ("--duration"). Harness "--stages" for complex load profiles, ramp-up, and ramp-down scenarios. Learn through practical examples for efficient K6 script execution and performance testing.

Use Github Copilot to generates K6 command line in an instance.

In K6 scripts, the "sleep" syntax introduces pauses to simulate realistic user behavior, preventing overload and respecting rate limits. Use "sleep" to mimic human actions, like sequential browsing or form filling. Implement dynamic sleep times for authenticity, utilizing "randomIntBetween" to generate varied pauses. Utilize Github Copilot's autocomplete feature for efficient code composition.

Work with AI Assistant to use K6 sleep function

In K6 scripts, defining test options within the script itself offers simplicity, repeatability, and version control benefits. For instance, specify virtual users and iterations for controlled testing. Utilize stages for nuanced load profiles. Alternatively, configure options via separate JSON files for clean, reusable scripts. Command-line flags override script and configuration file options. Explore K6's extensive test options via official documentation for comprehensive testing capabilities.

Use GitHub Copilot as an AI assistant to generate K6 test options.

Docker simplifies software installation by managing dependencies and ensuring compatibility across different machines and operating systems. Instead of dealing with potential errors and lengthy installations, Docker uses images containing all necessary components, making deployment easier. In this lesson, we learn Docker's basic concepts, including installing Docker, downloading Docker images, and running containers. This approach streamlines the setup process, particularly beneficial for deploying applications hassle-free.

The lesson introduces using Postman to consume REST APIs. Postman is a versatile API development environment that offers features like execution, testing, documentation, and automation. The lesson covers basic functionalities such as executing requests, saving them as collections, and importing API documentation. Additionally, we learn to import API documentation into Postman for easier access to APIs.

K6 is a performance testing tool aiding in HTTP request-response testing. It simulates user behavior, supporting various HTTP methods, query parameters, headers, and request bodies. Additionally, K6 parses JSON responses, enabling precise result analysis. With its simplicity and versatility, K6 proves invaluable for API testing, allowing developers to assess application performance under different loads and conditions with ease and accuracy.

The lesson covers interacting with AI assistants to create JavaScript scripts for HTTP requests in the k6 environment. It demonstrates generating scripts for GET, POST, PUT, PATCH, and DELETE requests using curl commands or Postman samples, and refining them with Copilot's assistance on Grafana K6. Despite occasional AI assistant inaccuracies, it illustrates how to handle requests, manipulate data, and manage K6 scripts

The lesson explains K6 testing with checks, comparing expected outcomes to actual results. Despite failures, tests continue, monitoring unsuccessful checks. Demonstrated examples cover HTTP response status codes, response times, and JSON content checks. Emphasizing interpretation of test outputs, it highlights refining checks for comprehensive API testing.

Interact with GitHub Copilot to create K6 scripts. Start by generating scripts from curl commands. Then, add checks for status codes, response durations, and message fields. Utilize Copilot to refine checks with complex validations for JSON fields. Run scripts with 20 vus for 100 iterations to verify functionality.

The K6 result output comprises various metrics crucial for performance testing, aiding in bottleneck identification and resource optimization. Metrics types include Counter, Gauge, Rate, and Trend, each offering insights into different aspects of system behavior. For HTTP testing, specific metrics like request duration are vital. Monitoring trends over time allows for thorough analysis, ensuring application reliability and scalability.

The lesson discusses reading K6 test output, focusing on metrics such as average, minimum, median, and maximum response times and percentiles like p(90) and p(95) to understand system performance under varying conditions. Additionally, it explains the significance of percentiles compared to averages, emphasizing their importance in identifying outliers and potential issues. The lesson outlines interpreting k6 test results, including scenarios, checks performed, data received/sent, HTTP metrics, and test runtime summary, while clarifying that K6 doesn't provide server-side metrics.

K6 offers diverse output options such as CSV, JSON files, and integration with visualization software. It includes a built-in web dashboard for real-time monitoring. The process involves creating configuration files for customized stages and enabling the dashboard through environment variables. The dashboard provides real-time visualization of test metrics and allows exporting static HTML reports.

HTTP metrics default to basic statistics, but "summary-trend-stats" allows customization like hiding the median and showing 99th percentiles. It's set via command line, script, or JSON, with different naming conventions

The lesson outlines using thresholds in k6 testing for performance evaluation. It explains that thresholds define pass or fail criteria for metrics like error rates and response times, which users can set in k6 scripts or separate configuration files. The script demonstrates applying thresholds to API tests, showing how to handle passing and failing conditions, including options to abort tests based on threshold results. It emphasizes the importance of using checks and thresholds together for efficient performance test automation, illustrated through practical examples and configurations.

The lesson highlights the complexities of test outcomes and how K6 tags can aid in filtering results. Demonstrating through HTTP requests, it emphasizes categorizing responses, setting thresholds, and utilizing system tags like "expected_response" for performance analysis. It explains configuring thresholds for various scenarios, breaking down metrics by URL, and applying tags to checks for detailed analysis, providing comprehensive insights into performance testing strategies.

Use GitHub Copilot to generate tags for various K6 use cases

In the lesson, we tackle the challenge of differentiating metrics when dealing with similar URLs. For instance, when accessing an endpoint with optional parameters, we send requests with various delays. We explore using custom tags to categorize these requests. By adding tags to requests and checks, we effectively analyze performance based on specific behaviors, addressing potential errors and ensuring accurate metric breakdowns.

Use GitHub Copilot to generate custom tags in K6

In this lesson, we learn about our approach for the upcoming lessons. We will work with the AI assistant, such as Copilot while creating the script. There will be some demonstration videos on how to use ChatGPT to generate K6 scripts in the later section of the course. However, working with Copilot, which is integrated directly into the visual studio code, is more convenient.

To debug K6 scripts, utilize K6 checks to identify issues during execution. Alternatively, employ console.log statements to examine variables. To view detailed request and response data, use the --http-debug flag, with options for basic or full output. Ensure logs show relevant information, such as usernames, passwords, and response statuses. Utilize flags like "http-debug" with the "full" value to display request and response bodies for thorough debugging.

In the lesson, we explored various methods to prepare test data. Initially, we utilized JavaScript Maps to generate usernames and passwords randomly. Then, we experimented with JavaScript arrays for test data preparation, also considering external files in CSV or JSON format. To optimize resource usage, we adopted K6's "SharedArray" from the "k6/data" module for efficient data sharing across Virtual Users (VUs). Finally, we verified data generation from arrays, CSV, and JSON files using K6's http debug mode.

Data correlation involves extracting values from HTTP responses to use in subsequent requests. For example, obtaining an access token from a login endpoint to access secured APIs or retrieving product UUIDs to interact with specific products. We can simulate this process using tools like Postman or K6 scripts. This process ensures proper sequencing and validation of API interactions, enhancing testing effectiveness.

We can use K6 to test HTTPS applications with self-signed certificates. This ensures successful execution and passing checks, which is particularly useful for testing internal or development server applications with self-signed certificates.

HTTP status codes include 3xx for redirection. By accessing a sample endpoint, we observe redirection in browsers and Postman. K6 scripting allows testing redirects, which is important for various scenarios. A K6 script verifies redirects and final responses, which is crucial for testing scenarios. Failure to follow redirects leads to test failures, highlighting the importance of proper redirection handling in testing.

Web applications can optimize loading time by making API calls sequentially or in parallel. For sequential calls, inspecting network activity demonstrates each API finishing before the next starts, resulting in slower page rendering. Conversely, parallel calls demonstrate simultaneous API execution, significantly reducing page rendering time. K6's HTTP batch allows parallel requests with various syntax options for sending multiple requests. Sequential and parallel request handling in K6 scripts affect testing efficiency and page loading speed, showcased through code examples and execution analysis.

Using K6's HTTP batch increases the built-in HTTP requests metric by N per iteration, but counting the batch calls requires custom metrics. With Counter from k6 metrics, batch calls are tracked. Other metrics like Gauge, Rate, and Trend offer diverse monitoring capabilities. Examples showcase custom metric creation, adjustment, and utilization, enhancing performance analysis beyond built-in metrics.

K6 introduces "groups" for organizing test cases within a script file, enhancing readability and management. Nested groups enable hierarchical structuring. Basic syntax involves importing "group" from k6 module, creating groups, and writing scripts inside. Group_duration metric measures total time taken by group's HTTP requests. Checks can be added within groups, aiding in performance evaluation. Thresholds set performance criteria for each group, supporting detailed analysis. Custom JSON configuration files further streamline metrics management.

In K6 scripting, code within the default function constitutes Virtual User (VU) code, executed repeatedly throughout the test duration. While this ensures consistent behavior, repeating all code in default can overload resources and cause issues like repeated logins. Utilizing K6's lifecycle functions - init, setup, default, and teardown - strategically prepares, executes, and cleans up test environments, optimizing performance and maintaining data integrity. Examples include accessing health check endpoints in setup and cleanup operations in teardown.

In the lesson, we utilized the BASE_URL variable for application paths, allowing easy migration by altering one line in the K6 script. K6 environment variables further enhance flexibility, permitting dynamic URL changes and eliminating hardcoding. With environment variables, we seamlessly switch between environments, ensuring script adaptability. Operating system or command-line settings dictate variable priority, impacting script behavior.

In this lesson, we explore using ChatGPT to generate K6 scripts, comparing it with GitHub Copilot. The lesson highlights how ChatGPT, albeit sometimes less effective than Copilot, can still be utilized for K6. Differences in outputs may arise due to variations in ChatGPT versions.