
Introduce qualia, a poset data quality platform, to increase data quality, reduce cycle time, and automate data monitoring, testing, and scheduled or triggered alerts for data stewards.
Transition from reactive mindsets to proactive data quality validation that catches anomalies before end users see them. Establish transparency, standards, and automation to ensure 100 percent data completeness across sources.
Explore testing scenarios such as source-of-truth tests, previous-period and reference data comparisons, snapshots, and validation of patterns or dates with static or dynamic thresholds, plus hard and soft business rules.
Explore Khalif connectors for relational databases, Snowflake and Redshift, Flash files, and text data; connect Cassandra, MongoDB, Hive, and APIs via REST, then test dashboards with Qlik Sense and Tableau.
Learn how automation validations can be scheduled or triggered to control external jobs, enable root cause analysis, and execute events based on results via a command-line API.
Explore the alerting component of quali up, notify data stewards, track data quality over time, and publish results by outcome across Slack, Splunk, emails, and tickets.
Learn how to log in to quilliup using a tenant text input for security, enter your username, and sign out by selecting your user ID across screens.
Navigate the interface by collapsing the left sidebar, using the home button and environment identifier, and view test executions, past, warned, and failed, via the gauge and about screen.
Explore how tests are the basic building blocks of quality gates, examine the gates module, and learn how tests validate source-to-target checks, KPIs, and scenarios, with background and scheduled validations.
Parse test results from the test canvas, examine the checksum and counts of distinct values in source and target, and diagnose a failed test against a threshold.
Share test results via email by configuring recipients, using a default template with runtime variables, and optionally attach an Excel file for failed runs.
Examine a sample email showing results, with a subject naming the test and status, a table of variables and tags, and attachments including the query and a timestamped Excel file.
Demonstrates how to export test results to an Excel file from the test canvas, locate the export to Excel button, and download the file after confirming the save location.
Understand how datasets are defined and compared, view and edit source data, and fetch data from relational queries, Excel, flat files, MongoDB, Cassandra, or API calls, then compare with dashboards.
Create a dataset from a relational source by selecting a data source, choosing an entity, inspecting table definitions, and using quick sample data to verify retrieval without sql.
Add filters to datasets built with a query builder by defining a condition on the entity name, selecting operators, and applying a value or variable.
Create test metadata by using the test page and blue button to design a custom test, configure scenario variables, severity, and tags, then save and edit if needed.
Explore data validation tests, learn how to apply rules from a dataset, including null checks, formatting checks, and greater-than-or-equal comparisons, with UI filtering and export options.
Quilliup's alpha to omega lecture shows creating data validation quick rules without coding by selecting checkboxes for not null and unique keys, including composite key uniqueness.
Create and test data validation user-defined rules, including a number format check on product cost and a rule ensuring price is greater than or equal to cost.
Explore the data profiling function in data validation tests, which scans a dataset to reveal fields, data types, nulls, uniques, and min max values to guide rule creation.
Create a data validation test with an ad hoc dataset, define quick and user-defined rules via the expression editor, and run the data profile to verify price >= cost.
Learn how tags, color-coded boxes with text, are user-definable and can be added to any entity; create new tags and tag groups to organize by test type or context.
Learn how variables store information in Quilly up, with examples like an email list, showing how changing a single variable propagates to all tests and is evaluated at execution time.
Create a fixed variable on the variables page by naming it, selecting the text data type (string), and entering comma-separated email addresses to use as the default value.
Create dynamic variables from a database table by converting a fixed email list into a single-value dynamic variable, using a query and dataset filters for runtime tests.
Explore dynamic variables in a simple project by using the query builder to create, filter, and concatenate fields, then sample data and evaluate single-value versus list-value results.
Finalize dynamic variables in a relational data source and show how each type—string, concatenate fields, and concatenate values—renders their values.
Create complex data sets by pasting a custom SQL query into the dataset builder, validate the query, select fields, and save your relational or RDBMS based dataset.
Differentiate library datasets from ad hoc datasets by using predefined library data that cannot be edited in tests, but can be copied or added to the shared library.
Explore UI test types, including compare tests across data sets, data validation tests, KPI checks, and existence checks with a free query rule.
Create and customize an existence test to verify a record in a dataset by filtering a library data set, adjusting the threshold, and logging results to Splunk.
Set up and validate thresholds to flag zero-record entries using global and per-table rules, comparing KPI range data to absolute thresholds to differentiate passes, fails, and warnings.
Learn to define thresholds by the number of rows returned, using a highest threshold where zero passes, one warns, and more than one fails.
Create a two way threshold rule in the global box with lower bound zero and upper bounds 100000 for warnings, 400000 for passes, up to two million, then run test.
Evaluate two or more data sets using compare tests, configure aggregations with group by, auto-mapped fields, and join types to validate differences and flag data quality issues.
Adjust test toolbox settings to control performance, memory via chunk size, thread counts, missing keys and results, plus timeout and logging, for efficient analysis and history retention.
log test results to a database table by selecting a predefined datasource and schema, enabling logging to create summary and details tables for each test id.
Explore how the two log tables—summary and detail—capture test executions, compare source and target record counts, and reveal test-specific field mappings in the logs database.
Learn to create a KPI range test by selecting a KPI rule, configuring group by entity with row counts, and applying thresholds to define success or failure.
Learn to share test results via Splunk by configuring settings, group names, and enabling sending for tests. Splunk is enabled by default and shares results only when you elect it.
Understand how versioning saves and restores previous test versions, highlights changes, and lets you keep or discard updates to avoid mistakes.
Create execution flow dependencies by connecting tests on a canvas with draggable arrows to define serial or parallel execution. Double-click arrows to change dependency types or delete to remove.
Discover how execution flows organize tests, notes, and scripts on the canvas, establishing serial and parallel runs and parent-child dependencies to group objects.
Explore creating loops in execution flows by enabling circularity and selecting a start node. Use a dynamic variable to set iteration counts for flexible, repeated execution.
Enable and configure scheduling for execution flows with the scheduler button, set start and expiration dates, options like weekdays at 5 am, or a custom cron expression.
Create and manage notes and annotations within an execution flow by dropping new notes onto the canvas, dragging them, and deleting with the trash button.
Explore what scripts are and how to create them, whether on the command line or against a database, and how they extend Queally by triggering actions, external events, and validations.
Create a cmd script in Windows to run command line tasks, such as updating a file’s date time with a touch file. Verify code in shell or PowerShell before saving.
Create a Linux acmd script by typing commands directly into the shell, unlike Windows, verify them there first, then copy and paste into the script box and save.
Create and run a script to execute a create schema against a database, using a command line script type, selecting a data source, and using dynamic variables to generate schemas.
Create and manage SQL query scripts for a relational data source, including adding, editing, and deleting scripts. Learn how to run scripts with or without output on the bascule server.
Explore monitor sidebar, open the gauge, filter by a time period (last 30 days by default), view the score, and note successes, warnings, and failures, with links to future videos.
Explore the full monitor to view KPI boxes, filter by status, and use auto-refresh with favorites for access; open test results and search by name, id, flow, script, or tags.
Learn how the url encodes the project id and entity id when navigating projects and execution flows, helping deployments and precise entity access.
Generate authentication tokens in the Quantic module. Use the token to trigger external events via Kirkwall in Aquileia, and share it with the service account.
Explore data connectors and data sources in the gates module, including relational database management systems and flat files, with an upcoming focus on creating each type.
Create an RDBMS data source by adding it, choosing a driver, and importing a JDBC string. Test the connection, enable write logging, and assign to projects.
Add file paths as data sources by linking to a folder at the root of the C drive. Set a subfolder, define a filter and extension, test, assign, and save.
learn how to manage projects in settings by creating, editing, and deleting them, assign data sources and roles, and configure permissions for each project.
Learn to manage roles in quality gates, create a role for signed users, and save changes with a description, while a future video covers adding roles to projects and permissions.
Create tag groups and tags by adding a tag group with a name and color, then add tags, test colors on text, and save for use.
Understand how to manage users in the administration panel, including login users and service accounts, view their type as active directory or application users, and add, edit, or reset passwords.
Create and manage user groups in the administration module to assign permissions to modules. Manage users through groups to simplify access when dozens of users need home page testing.
Learn to create an applicative user not tied to Active Directory in the administration module, enter name, username, and optional email, and view the new user on the home page.
Explore system properties in the administration module, adjusting test results locations, environment names, and page sizes to optimize performance and reliability across dev, test, and prod environments.
Create network locations as data sources for flat files and Excel validation, configure a root path in system settings, and test the connection before adding files.
quilliup is a data quality platform, running as a web application, which comprises multiple modules. Quality Gates, the main module, will be the focus of this course. Quality Gates increases and maintains data quality, decreases cycle time, and automates manual processes. Quality Gates has advanced alerting features and can integrate with any ecosystem, including those using other monitoring platforms like Splunk. The Quality Gates validation methodology consists of three steps: test, automate, alert. This course will guide you through every aspect of Quality Gates.
Future sections will illuminate the other modules in quilliup.