Palantir Foundry Ontology: Data Flow with AIP & Workshop

An orientation, starting with what an ontology is, so you see what Foundry is trying to achieve. We move on to Foundry's Ontology and the crucial concept of Object Types. We differentiate Object Types from the similar-sounding Object Sets and instances, and then end by navigating through the key tabs of the Foundry Ontology, dwelling on permissions and user edits.

Explains what an ontology is using the example of a zoo. Understand that it predates Palantir by millennia, and that it is a top-down construct, which can change over time and may have different levels of political acceptability. Briefly see how this is implemented in Foundry.

Introduces the Foundry-specific implementation of ontology. Create an Object Type in the ontology using the valid designator dataset from the course on Pipelines. See that it is important to assess our ontology for coherence: we should not blindly add ever more object types.

Foundry has a lot of references to "objects" : Object Types, Object Sets, Object Instances and plain old objects. Here we see the difference between these concepts in Foundry, and we also compare properties and objects to the corresponding columns and rows within Datasets. See that Object Types are not equivalent to datasets: they are mapped to, and sit on top of, datasets.

A guided tour of the ontology UI, showing the scope of this course to be Overview, Properties, Security, Datasources, Capabilities and Object Views tabs (although we do touch upon global interfaces later in the course). Explains how the permission system works; how end user edits are handled; what the capabilities and object views tabs are for.

In this section, we run a number of experiments to see how the Ontology data flows through to an AIP agent widget that we set up inside a Workshop app. We find the Ontology data does not flow through as we might expect, and we end the section by seeing the best way to get the AIP agent to work.

Walks through creating an AIP agent in Workshop, using an Object Set based on our Ontology Object Type, and two derived variables to hold the input and output data. We get our widget working with an initial prompt. Since this is a course on the Ontology and not on Workshop or AIP, we use the Legacy model of the AIP agent widget (not AIP Agent Studio), which is easier to teach since all the settings live in Workshop.

Tests whether Ontology property descriptions influence AIP responses and reasoning. We place misleading information into property descriptions to challenge the response given by the AIP agent, to see whether this will affect the response. We also strip out part of the prompt within Workshop, which told the agent it was an expert, for some surprising results.

We examine whether changing the Object Type description - the single high level description for the whole data concept - flows through to the AIP agent widget. We find it does, and we see how powerful it can be. We provide misinformation in the Object Type description and see whether this can overpower the general aviation knowledge that the AIP agent can access.

Value Types in the Ontology let us constrain string patterns - for example, that an email address should always contain an @ sign. We add a value type on runway designators which clearly lay out the "rules" governing validity of these string patterns. We see whether the AIP agent widget can access these value types to enrich its response.

An interface lets different Object Types in the Ontology share key data shapes and properties. We set up a global interface for designator, and share it between out dataset and the Runway dataset, and see whether the AIP agent can pick up on this linkage.

If Ontology metadata flowed into the AIP agent widget, this would create a maintenance problem: the validity of the designator is determined by a function, which could be wrong, or might later be changed or improved. Separate Ontology metadata must be manually kept in synch - poor design. Far better for us to give the AIP agent direct access to the function or process used in Pipeline Builder to generate the outcome. This way, even if the function is wrong, the AIP agent will correctly tell users what IS happening and not what SHOULD happen. This video shows how to do this.

In this section, we focus on the relationship between Workshop and the Ontology, and we learn that Workshop is highly Ontology-aware – in fact, Ontology-dependent. We examine the best Workshop widgets for showcasing Ontology data and links and we learn how to traverse those links within Workshop so we can filter one Object Type by another with minimal effort. Finally we learn how and where to edit a central curated widget of an Object Type – called an Object View.

We see that Workshop can only work with data exposed through Object Sets, not directly with datasets. Those Object Sets, in turn, always originate from Ontology Object Types. Even when an Object Set is derived from an AIP response, it is still shaped and constrained by the Ontology. There is no way to bypass the Ontology in Workshop: it consumes object types, their properties, and the links between them. Using the example carrier data, we then show how quickly a fully functional filter panel can be generated from an Object Set in just a few seconds.

We implement multiple widgets in Workshop that are highly adept at showcasing Object Types and links between them. We cover the Links, Property Lists, Object Table and Edit history widgets. We end by dwelling on the importance of Object Views, which is the subject of its own video later.

In Workshop we can filter one Object Type’s data by a different Object Type using traversals. For example, we can see all the flights that occurred for a particular carrier, using the example data, because these Object Types are linked. In this video we see how to set up Object Sets with traversals, to enable filtering one dataset by another without even importing the second Object Type into Workshop.

Object Views in Workshop seem like magic: you drag in this widget and see a fully formed comprehensive, meaningful view of an Object Type. We see how this happens and how to edit an Object Type within the Ontology. This means data owners can create curated, intelligent views of the data they know, for report creators to depend upon with zero effort.

We summarise the key benefits of the Ontology, and see why it is vastly preferable to a regular database in which all tables are logically equivalent. We see where property descriptions flow – given they don’t really go into Workshop or AIP. We see how the Ontology acts as a buffer and stops applications breaking every time a pipeline fails. We examine the benefits of a single central view of all our data and learn that shared properties and interfaces can unlock the ability to slice and dice our data in new ways.

Property descriptions are metadata set within the Ontology. We’ve seen earlier that they are not consumed within AIP Agent widget, and in fact, they are rarely consumed within Workshop at all. So where do they end up and what’s the point of them? In this video, we examine Object Explorer to understand the benefit of single, central metadata for our organisation.

The Ontology enables evolution without disruption, letting data and processes change without breaking downstream usage – or at least making failures clear and loud, rather than silent. We set an enum constraint on our Output data and then introduce a new value outside of that constraint via Pipeline Builder. We see what breaks, and where – and discuss why this is a good thing.

The Ontology is a way of promoting only the golden datasets to a higher state of visibility for applications to use. This is a huge win for organisations, which previously depended on key staff to remember or document which data mattered. This central location allows data governance to really mean something, with processes that are repeatable and shareable.

A brief look back at what we have learned, and a focus on possible next steps.