Show And Tell: Sikuli - Pattern-Matching and Automation

Explore Sikuli pattern matching and automation by building image-based scripts. Take screenshots, locate UI elements, click, type hello world, and run the script.

Explore what happens under hood of the sikuli workflow: a secure bundle directory containing a python script, image files, and a dml file, with python and java APIs via JNI.

Learn to build a naive Sikuli script that automates opening the calculator via the start menu, typing calc, and performing multiplication, using image pattern matching, veto waits, and robust screenshots.

Use the pattern class to bind an image with similarity and offset attributes for precise, reusable matches; replace multiple screenshots with a single keypad image to reduce redundancy and errors.

Explore region and pattern objects in Sikuli, defining rectangular screen areas with x and y coordinates and height, and test intersections of regions and patterns using exist and highlight.

Learn how Sikuli's pattern-matching uses the match object and find operations to locate UI elements, interact with them via click, right-click, and drag-and-drop, and handle multiple matches.

Master the application class to automate programs by opening, focusing, closing apps, and identifying the front or focused windows; learn to hook, focus, and highlight windows like Chrome.

Learn Sikuli pattern matching and automation by using security settings to highlight actions, adjust mouse and typing delays, and log script execution.

Learn to query system and environment variables with Sikuli, detect the operating system (Windows, Mac, Linux), and display versions, while testing keyboard caps lock and mouse coordinates.

Enable interactive automation by using Sikuli prompts, input boxes, and drop-downs to gather user data, handle selections, and display feedback through message and error dialogs.

Study Python conditionals using if, elif, and else with a colon and indented blocks. Understand comparison operators, truth evaluation, and end statements for unhandled cases.

Automate dynamic calculations with a Sikuli security script that drives a calculator, using user input to select add, subtract, multiply, or divide via pattern matching and keypad offsets.

Demonstrate using for loops in Sikuli to automate a calculator task with a Python-like range, clicking numbers and plus signs, and speeding up execution by adjusting delays.

Showcases Sikuli hotkeys to run a long-lived loop controlled by a global variable. Stop the loop via a control key combination, while the loop depends on a screen capture condition.

Invoke Sikuli from Java using Eclipse by loading the Sikuli API jar, creating a screen object, and matching images to click and type hello world.

Explore how Sikuli and Selenium combine to automate a browser using Java, leveraging visual pattern matching to locate the search bar and enter text on a targeted site.

Learn to script Microsoft Word with Sikuli by locating the executable, escaping backslashes, and opening a new blank document to type text via GUI automation.

Show how to perform drag-and-drop in Sikuli by defining a region, capturing a source image, and moving it to a destination with a two-line script, then verify the file appears.

Utilize Sikuli pattern-matching and automation to manage system power settings by locating unchecked checkboxes in the settings panel, taking screen grabs of the region, and toggling battery and brightness options.

Use Sikuli's visual pattern matching to bulk delete Gmail emails by selecting all on each page, clicking delete, and looping until the inbox is empty.

Explore Sikuli’s pattern-matching and automation by locating icons in a crowded folder, building a pattern list from screenshots, searching the screen for matches, and highlighting each instance.

Automate emptying the recycle bin with Sikuli pattern-matching that tolerates inaccuracy, capturing screenshots, right-clicking the recycle bin, and confirming the dynamic delete dialog.

Learn to build a Sikuli-based Facebook automation script that monitors a specific user’s status updates, triggers a pop-up alert when a new post appears, and runs in the background.

Use Sikuli pattern-matching and automation to monitor Skype availability, replacing infinite loops with a vanish function and a 3600-second window to trigger a pop-up when the user is online.

Use Sikuli to perform image searches inside large PDFs by capturing a target image on the fly, scrolling until a match appears, then switch apps and notify once found.

The lecture introduces optical character recognition in Sikuli and demonstrates text-based search by converting images to text within a selected region of a PDF document.

Demonstrates OCR-based search to copy and extract data from an Excel spreadsheet region, including numbers 1 to 11, converting the region's contents for use in automation.

Automates mass data downloads with Sikuli pattern-matching by defining a region, locating green downward icons, iterating matches, and clicking with a rightward offset while waiting for screen changes.

Automate a simple memory game with Sikuli pattern-matching by locating the game region, finding cards, looping to click matching symbols, and finishing when all pairs are matched.