Data Integration with the MarkLogic NoSQL Database

Store and index XML and JSON documents in MarkLogic by reconstituting relational data into whole documents, then load with optional schemas and immediate indexing for fast queries.

Learn how to load documents into a MarkLogic NoSQL database using multiple options, including MarkLogic Content Pump, the Java API, the REST API, and XQuery/JavaScript methods, with practical steps.

Split a multiple-record SML document into individual contact documents using MarkLogic Content Pump, dynamically generate URIs from contact elements, and assign them to the raw collection.

Learn how to transform between JSON and XML in MarkLogic, using transform and transfer functions, customize whitespace handling, and mix JSON and XML documents within the same database.

Learn to load records from a delimited text file into MarkLogic, turning each row into a document with configurable id, uri, and document type (XML or JSON), organized into collections.

Discover transforming data during ingestion with the transform option and MarkLogic content pump, or post-load with the data movement SDK, using JavaScript or XQuery; normalize dates to international format.

Learn to transform dates to international format during data loading in MarkLogic using a transformation module loaded into the customer 360 modules database.

Store data as documents in MarkLogic with immediate indexing; use MarkLogic Content Pump to load, transform, split XML into documents, and convert between JSON and XML, including loading CSV rows.

Load factsheet documents into the customer 360 database using MarkLogic content pump, importing 32 documents into the raw collection and verifying them via the console.

Align data from multiple sources by harmonizing differing document structures, using templates and queries to present a consistent view and improve query performance.

Learn data harmonization techniques to harmonize marketing contacts from multiple data sources in a MarkLogic NoSQL database by querying differently structured documents to retrieve names and contact details across systems.

Learn how the envelope pattern harmonizes data from different sources by wrapping documents in a source element and an envelope, creating a canonical structure for cross-source queries on marketing contacts.

Update json documents by adding data to an envelope using node insert before, rest api patch, or java api patch, with a canonical vocabulary for consistent querying.

Learn to insert data into envelope documents using node insert before/after, create canonical sections with company, first name, and last name, and assign documents to the canonical collection for querying.

Explore querying envelope documents in MarkLogic with path expressions to target canonical values, such as a company with campus by X, and fetch first and last names across data sources.

Learn batch processing of millions of documents with core, applying XQuery or JavaScript transformations to selected documents and replacing originals in canonical and transformed collections.

Use Corb to batch transform marketing documents in the raw collection, envelope them, and set up a canonical section for the customer 360 database, validating seven documents meet criteria.

Use the data movement sdk with the MarkLogic Java api to batch load and transform documents asynchronously across a scalable cluster.

Learn to load and transform documents in MarkLogic using the Data Movement SDK, envelop content into a canonical and source section, and deploy a transformation module via curl.

Learn how to set up a Java project in Eclipse, configure Maven, and include the MarkLogic Java API from Maven for batch processing with the Data Movement SDK.

Create a batch load and transform pipeline with the data movement sdk, building utils and load-transform classes, configuring properties, and running the Java app to load and transform documents.

Learn to enhance batch processing with the data movement SDK by updating a transformation file to map organization to company via an if-then-else, then rerun and validate updated documents.

Master data management resolves entities across systems by reconciling name variations and using alternative data like address or phone, with a spelled-out double Metafont function for phonetic matching.

Explore data harmonization in MarkLogic, using the envelope pattern to add data without altering the original, and batch processing with core and the data movement SDK, plus resolving entities.

Complete a hands-on lab to add canonical data to marketing contact documents' envelopes section using Korb transformations. Configure properties, run, and verify updates.

Use triples to harmonize data by linking entities and aliases, enabling graph queries across relationships like Napa biking tours and Lord Voldemort as Tom Riddle, with triples embedded in documents.

Learn to store and index triples in MarkLogic, choose between independent and embedded triples, and load them via RDF files or through REST, Java, Node, and template-driven extraction.

Learn how to store and index embedded triples in MarkLogic using triple store documents, loading JSON documents with triples, and how these triples populate the triple index for queryable data.

Explore querying triples in MarkLogic with sparkle/SPARQL via the Optik API, using subject, predicate, and object patterns to retrieve real-time board member and investor data.

Learn to query contact documents in MarkLogic to return company names, first names, last names, and emails, and use same-as rules or inferencing to retrieve related records efficiently.

Query triples with XQuery to map company names to friendly labels. Return first names, last names, and emails, and integrate triples with your MarkLogic application code.

Harmonize data using triples, leverage the triple index, and build sparkle queries. Integrate sparkle into your application code for end-to-end semantic data handling.

Explore template driven extraction in MarkLogic 9, dynamically generating rows or triples from documents to enable SQL queries against generated views and support business intelligence.

Create and validate templates that extract rows from each order item using xquery or json, define columns like product, price, and quantity with proper types and validate with tdd:validate.

Test templates against documents with the TE:no data extracts function, test multiple documents and templates, and use nullable to handle optional notes, yielding rows with product price and quantity.

Define an optional notes column in the template and gracefully handle missing notes, then add a calculated row total column by multiplying quantity by price.

Apply an optional, nullable column named company in the template, deriving its value from billing.company via a data path. Test this against sample orders to verify results.

Define variables in templates and nest templates to reference variables with a dollar sign, use a top-down context from slash order, and improve performance by traversing the document tree.

Generate a MarkLogic view by inserting templates with the TDD template inserts function, loading into the schema database, then query the view with sequel or Optik API queries.

Insert templates into the customer 360 database using tdd template insert to generate a purchases view in the customer 360 schemas database, then query with sequel or optik api.

Disable the template by setting enabled to false to clear the Rose index, then delete the template document from the schemas database.

Leverage tde templates to harmonise data from two ordering systems, enabling a single view, cross-system aggregation, and consistent column names and scalar types for unified queries.

Create and validate a second TDE template for bookings, test with data extract, insert into the purchases view, and harmonize data from two ordering systems without altering source documents.

Explore template driven extraction to generate rows and triples, test and define templates, build views for sequel and Optik API queries, and harmonise data across systems.

Create a factsheet template by defining the factsheet root and exposing data for a business intelligence tool via a sequel datasource or Optik API, enabling joins with order documents.

MarkLogic enables sequel queries by exposing views and supporting sql 92, joins, and aggregates; connect via query console, application code, or the odbc driver for bi tools.

Use the query console to run SQL queries against the purchases view, experimenting with select, where, and order by, and group by product to sum quantities for best sellers.

Explore joining data from multiple views using inner joins, specifying from, on, and order by to create unified rows; practice linking tech support calls with factsheets and purchasers data.

Set up an odbc app server to enable sql queries from MarkLogic to the customer 360 database by configuring port 3366 and naming the server, then verify it appears.

Install the MarkLogic ODBC driver for Linux or Windows, choose the 32/64-bit version, install via RPM on Linux, set up a data source name, and test the connection with isql.

Install the marklogic odbc driver on linux, become root, modify the template to set port 3366, apply changes, and test the connection with isql to run a sample query.

Use mlsql to test sequel queries against the MarkLogic ODBC server, login as admin, run queries with semicolons, and exit with \q.

Run SQL queries from XQuery or JavaScript in a MarkLogic app and filter results by document criteria. Use CTX functions to craft criteria, like Gere's or chain, or property reason.

Explore creating sequel queries in marklogic, including joins and aggregates, enabling external access for business intelligence tools, and running queries from javascript applications with document criteria.

Explore how Optik API enables fluent querying of rows, documents, and triples in MarkLogic using JavaScript, XQuery, Java, or REST, with the pro cycling customer 360 dashboard as an example.

Learn to write Optik API queries by building and executing query plans, using from view, where, order by, limit, group by, aggregate, and join operations to derive totals and insights.

Import the Optik API workspace, query the Purchasers view to select product, price, and quantity, apply filters, and group by product to sum quantities.

Learn to build optic API joins between factsheets and marketing contacts views, select and order columns, apply where filters, and view results in javascript, guided by the optic API documentation.

Use document criteria with the Optik API to filter query results by words or phrases in documents, and enable stemming in MarkLogic 9 to improve matches.

Create a document join with the optic API to fetch original source documents from joined rows, enabling a master-detail view via column definitions and the node function.

Perform a document join to fetch full documents by joining the doc id column, using the optic API to return both row data and the joined document content.

Explore how the Optik API creates triples and uses inner joins with the purchases view to compute sales totals for companies where Michael Ross is a board member.

Explore including triples with unions, distinct, and optional left outer joins; perform inner joins with a purchases view and group by company to calculate total sales.

Modify a triples query with the Optik API by declaring a company variable and adding a spouse join to return vip people, including board members, founders, investors, and all customers.

This lecture guides building a JavaScript app with the Optic API to unify customer data into a 360 dashboard, configure the MarkLogic database, and install npm dependencies.

Apply index settings and launch the customer 360 app with the Optic API, explore data via dashboards, searches, and triple visualizations with Laad Live.

Explore the optical API in XQuery and its fluent interface, using the arrow operator, where and select clauses, and inner joins to surface board member company sales.

Learn to send Optik API queries through the MarkLogic REST API by serializing them as JSON, posting to the REST endpoint, and receiving results in JSON, XML, or CSV.

Explore the optic API for fluent queries in a marklogic multi-model database, enabling document criteria, master-detail functionality, and triples with joins, to power a customer 360 dashboard via JSON serialization.

Model data with entity services in mark logic 9, create a model descriptor, load it into the database, and generate conversion functions to align documents with the model.

Create a model descriptor file defining product, order, and customer entity types, with an info section (title and version) and a definitions section; deploy to MarkLogic to manifest the model.

Define an entity type in the model descriptor by naming entities, adding properties with data types, marking fields optional or required, and using arrays for recommendations and optional primary keys.

Learn how to load a model descriptor into the MarkLogic NoSQL database, assign it to a collection, and persist it as triples, with hands-on Javascript and SPARQL walkthrough.

Generate model-driven components from the model descriptor, including a conversion module, templates, a schema file, index settings, and search api options for entity instances.

Customize the conversion module to map source document fields to model properties, load the converter into the 16 modules database, and convert raw product documents into compliant instances with attachments.

Use the conversion module to transform raw product documents from the customer 360 database into canonical product instances, then insert them into the stash product collection via 8055 rest API.

Run search queries against instances described by your model using the core search API to retrieve results. Generate extraction templates from your model descriptor and explore other artifacts.

Define an entity type with a model descriptor, load it into a MarkLogic NoSQL collection, generate a model and artifacts, customize a conversion module, and query compliant documents.

Explore the data hub framework, an open source tool with a graphical interface to define entities, create input flows, harmonize data, and browse data.

Wraps up data integration with MarkLogic by reviewing harmonisation of documents and triples, bringing in data, templates, and querying with SQL and object API, plus data modeling and setup topics.