SAP HANA SYSTEM REPLICATION SETUP with Hands-on Lab Access

General Prerequisites for Configuring SAP HANA System Replication

Before you configure SAP HANA system replication, several prerequisites must be fulfilled.

System Requirements

• The primary and secondary systems must both be installed and correctly configured; verify that both systems are independently up and running.

• SAP HANA systems can only be replicated as the whole system, which means that the system database and all tenant databases are part of the system replication. A takeover can only be performed as a whole system. A takeover on the level of a single tenant database isn’t possible.

• The configuration of active hosts in the primary and secondary system must be the same, which means that the number of active hosts, the names of the host roles, failover groups, and worker groups must be identical in both systems. So, if there’s a standby host on the primary system it need not be available on the secondary system and vice-versa.

• System replication between two systems on the same host isn’t supported. Check that the host names in the primary system are different to the host names used in the secondary system. You can see the SAP HANA host name for each host in the environment variable SAP_RETRIEVAL_PATH (/usr/sap/<SID>/HDB<InstNo>/<hostname>) and with the python script landscapeHostConfiguration.py. For more information, see Host Name Resolution for System Replication and Checking the Status with landscapeHostConfiguration.py in the SAP HANA Administration Guide.

  If the host names of the primary and the secondary system are the same (for example, because two systems are used that have identical host names), change the host names used on the secondary system. For more information, see Rename an SAP HANA System Host in the SAP HANA Administration Guide.

• To secure the system replication communication channel between the primary and the secondary system, configure the ini parameters [system_replication_communication] / listeninterface and allowed_sender as described in Host Name Resolution for System Replication.

• For SAP HANA system replication, a port offset value of 10000 is configured to reserve ports for system replication communication.Note

  Note that values for port ranges do not need to be maintained manually. This can be done automatically by the SAP Host Agent which includes port reservation functionality and optimizes the relevant Linux kernel parameters. Refer to Linux Kernel Parameters in the Lifecycle Management section of the SAP HANA Administration Guide and the following SAP Notes:

  • 401162 - Linux: Avoiding TCP/IP port conflicts and start problems, this describes setting up the SAP Host Agent.

  • 2382421 - Optimizing the Network Configuration on HANA- and OS-Level

• If a new tenant database is created in a running SAP HANA system replication, it must be backed up to participate in the replication. Afterward, the initial data shipping is started automatically for this tenant database. If a takeover is done while the initial data shipping is running and not finished, this new tenant database won’t be operational after takeover and will have to be recovered with backup and recovery. See the SAP HANA Database Backup and Recovery section of the SAP HANA Administration Guide.

• The secondary system must have the same SAP system ID (<SID>) and instance number as the primary system.

  Note

  The primary system replicates all relevant license information to the secondary system. An additional license isn’t required. For more information, see SAP Note 2211663 The license changes in an SAP HANA database after the deregistration of the secondary site.

• During an upgrade of the system replication landscape, the software version of the current secondary system must be equal or newer to the version of the current primary system.

  Note

  During a failback, the roles of your systems in the system replication landscape switch. Make sure in this case that your primary system doesn’t have a newer software version than the secondary system.

  Note

  For Active/Active (read enabled) setups, the SAP HANA versions must be the same on the primary and the secondary system. Use this setup mainly during the upgrade process of the system replication landscape.

Endianness

In a system replication landscape the systems on all sites must run on platforms with the same byte order. System replication is supported between Intel little-endian and IBM Power little-endian systems.

In SAP HANA the following byte order is supported in the corresponding SAP HANA and Linux versions:

Supported Byte Order

SAP HANA Version

Linux OS Version

Byte Order

SPS11 & SPS12

Linux Intel SLES 11

little-endian

Linux Power SLES 11

big-endian

SAP HANA 2.0 SPS00

Linux Intel SLES 12

little-endian

Linux Power SLES 12

System replication is supported between Intel little-endian (SAP HANA SPS12 or SAP HANA 2.0 SPS 00) and IBM Power little-endian (SAP HANA 2.0 SPS 00).

Configuration

• All configuration steps have to be executed on the master name server node only.

• The .ini file configuration must be similar for both systems. Any changes made manually, or by SQL commands on one system should be manually duplicated on the other system.

  Automatic configuration parameter checks alert you to configuration differences between the two systems.

  Note

  To keep the ini file configuration similar on both systems, the INI parameter checker is per default configured to check for differences. Additionally, it can be configured to replicate parameter changes from the primary system to the secondary system.

• Ensure that log_mode is set to normal in the persistence section of the global.ini file. In log mode normal, the log segments are automatically backed up. This ensures that the database can be backed up to the most recent point in time.

Authorization

• You must be logged on to both systems as the operating system user (user <sid>adm) or you have provided its credentials when prompted.

• You need the operating system user to set up a system replication landscape, to perform a takeover and a failback, as well as to disable system replication with the SAP HANA cockpit. For more information, see Operating System User <sid>adm and Connect to a Database With SSO or SAP HANA Credentials.

• Before you configure SAP HANA system replication, you must copy the system PKI SSFS .key and the .dat file from the primary system to the secondary system:

  /usr/sap/<SID>/SYS/global/security/rsecssfs/data/SSFS_<SID>.DAT

  /usr/sap/<SID>/SYS/global/security/rsecssfs/key/SSFS_<SID>.KEY

  For more information, see SAP Note 2369981 Required configuration steps for authentication with HANA System Replication.

  If you installed XS advanced, you must also copy the XSA SSFS .key and the .dat file from the primary system to the secondary system in the following directories:

  /usr/sap/<SID>/SYS/global/xsa/security/ssfs/data/SSFS_<SID>.DAT

  /usr/sap/<SID>/SYS/global/xsa/security/ssfs/key/SSFS_<SID>.KEY

  For more information, see SAP Note 2300936 Host Auto-Failover & System Replication Setup with SAP HANA extended application services, advanced model.

  The copied files become active during system restart. Therefore, it’s recommended to copy them when the secondary system is offline (for example, before registration).

• In preparation for maintenance tasks (for example, near zero downtime upgrades), configure a user in the local userstore under the SRTAKEOVER key. For more information, see Configure a User Under the SRTAKEOVER Key in the SAP HANA Administration Guide.

• For the local secure store the primary and secondary require completely independent installations of LSS. The file shares that LSS needs must be different for the two databases.

Dynamic Tiering

• If you plan to add SAP HANA dynamic tiering to your landscape in the future, seeSAP Note 2447994 before you enable HANA system replication. SAP HANA dynamic tiering requires certain communication ports, operation modes, and replication modes.

• SAP HANA dynamic tiering isn’t supported with multitarget system replication. For more information about SAP HANA system replication with SAP HANA dynamic tiering, see SAP Note 2447994.

Replication Modes for SAP HANA System Replication

While registering the secondary system, you need to decide which replication mode to use.

SAP HANA offers different modes for the replication of the redo log:

Replication modesLog Replication ModeDescriptionSynchronous in-memory (SYNCMEM)

The primary system commits the transaction after it gets a reply that the log was received by the secondary system and stored in memory. The delay for the transaction in the primary system is smaller, because it only includes the time for transmitting the data. The disk I/O speed on the secondary system doesn't influence the primary's performance.

When the connection to the secondary system is lost, the primary system continues the transaction processing and writes the changes only to the local disk.

Data loss can occur in the following situations:

• When the primary and the secondary system fail at the same time while the secondary system is connected. The data is not written to disk – neither on the primary nor on the secondary system.

• When a takeover is executed while the secondary system is unavailable. The data that arrived on the secondary is outdated compared to the data on the primary.

This option provides better performance because it is not necessary to wait for disk I/O on the secondary system, but it is more vulnerable to data loss.

Synchronous on disk (SYNC)

The primary system waits with committing the transaction until it gets a reply that the log is persisted in the secondary system. This option guarantees immediate consistency between both systems, at a cost of delaying the transaction by the time for data transmission and persisting in the secondary system.

When the connection to the secondary system is lost, the primary system continues the transaction processing and writes the changes only to the local disk. No data loss occurs in this scenario as long as the secondary system is connected. Data loss can occur, when a takeover is executed while the secondary system is disconnected.

Additionally, this replication mode can run with a full sync option. This means that log write is successful when the log buffer has been written to the log file of the primary and the secondary system. When the secondary system is disconnected (for example, because of network failure), the primary system suspends the transaction processing until the connection to the secondary system is reestablished. No data loss occurs in this scenario. You can set the full sync option for system replication with the parameter [system_replication]/enable_full_sync.Note

If SAP HANA system replication runs in the SYNC replication mode with the full sync option enabled, and if the connection to the secondary site is interrupted, no write operations on the primary site are possible. The operation of creating a tenant database, for example, will wait until the connection to the secondary is reestablished or the SQL statement times out.

For more information about how to enable the full sync option, see Full Sync Option for System Replication.

Asynchronous (ASYNC)The primary system commits the transaction after sending the log without waiting for a response. Here we have no delay because the data transmission is asynchronous to the transaction in the primary system.

This option provides better performance because it is not necessary to wait for log I/O on the secondary system. Database consistency across all services on the secondary system is guaranteed. However, it is more vulnerable to data loss. Data changes may be lost during takeover.

Note

If you plan to add SAP HANA dynamic tiering to your landscape in the future, please check supported replication modes in SAP Note 2447994 before you enable SAP HANA system replication.

The replication mode can be changed without going through a full data shipping from the primary system to the secondary system afterwards. For more information, see Changing the Replication Mode.

Operation Modes for SAP HANA System Replication

While registering the secondary system, you need to decide in which operation mode to run SAP HANA system replication.

System replication can be run in three operation modes: delta_datashipping, logreplay or logreplay_readaccess. Depending on the configured operation mode, the database sends different types of data packages to the secondary system. For more information, see Data Transferred to the Secondary System.

Operation Mode

Description

delta_datashipping

This mode establishes a system replication where occasionally (per default every 10 minutes) a delta data shipping takes place in addition to the continuous log shipping.

The secondary system persists the received log entries but it does not replay them until it has to take over. To shorten the log replay time, data snapshots are transmitted from time to time from the primary to the secondary system. The data snapshots are transferred asynchronously as differential backups (data backup deltas) triggered by the secondary system, which asks for a data backup delta with changes since the last one. During takeover the redo log needs to be replayed up to the last arrived delta data shipment.

logreplay

In this operation mode, a redo log shipping is done after system replication was initially configured with one full data shipping.

The redo log is continuously replayed on the secondary system immediately after arrival making this step superfluous during takeover. Since the log is continuously replayed, the secondary system can take over immediately, if the primary system fails. With continuous log replay, the log entries are sent from the redo log buffers in memory. When the secondary system is temporarily disconnected, the primary system must not claim and overwrite the log segments that have not been replicated yet. This is achieved by retaining these log segments up to a configurable maximum retention size. When the maximum retention size is reached, the log segments are reclaimed and overwritten with new log to prevent a standstill of the primary system. After such a situation, a full data snapshot needs to be transferred again, when the secondary system is connected again.

Because this operation mode does not require delta data shippings, the amount of data that needs to be transferred to the secondary system is reduced.

logreplay_readaccess

This mode is required for replication to an Active/Active (read enabled) secondary system.

Using this operation mode while configuring your system replication, read access becomes possible on the secondary system by establishing a direct connection to the secondary system or by providing a SELECT statement from the primary system with a HINT. For more information , see also Client Support for Active/Active (Read Enabled) and SAP HANA SQL Reference Guide.

This operation mode is similar to the logreplay operation mode regarding the continuous log shipping, the redo log replay on the secondary system, as well as the required initial full data shipping and the takeover. As with the logreplay operation mode, the redo log is replicated to the secondary system and continuously replayed to keep the secondary system synchronized.

Limitations

Before you begin preparing a replication strategy for an SAP HANA system, consider the following important aspects regarding the operation modes logreplay and logreplay_readaccess:

• Registering a secondary with operation mode logreplay against a primary running on an SAP HANA revision less than or equal to SPS10 will not work, because the primary does not yet support this feature. Furthermore, for operation mode logreplay_readaccess the primary must be running on a revision SAP HANA 2 SPS00 or higher.

• Only the operation mode delta_datashipping will work when registering the original primary (failback) after upgrade of the secondary during a near zero downtime upgrade from an SAP HANA revision less than or equal to SPS 10 to SPS 11, because the former primary’s version does not yet support logreplay.

• If the connection to the secondary is not available, the primary system will keep writing the redo log segments in the online log area to be prepared for the delta log shipping after the connection is reestablished. These log segments are marked as RetainedFree until the secondary is in sync again. In this case there is a risk that the log volume may run full. To prevent this:

  • If a secondary is not used anymore, it must be unregistered (sr_unregister).

  • If a takeover to the secondary was done, the former primary should be disabled (sr_disable).

  For more information, see How to Avoid Log Full Situations in LogReplay: Managing the Size of the Log File.

• In a multitier or multitarget system replication it is not possible to combine the logreplay and delta_datashipping operation modes.

• For multitarget system replication only the logreplay and logreplay_readaccess modes are supported.

• The logreplay operation modes do not support history tables.

Note

If you plan to add SAP HANA dynamic tiering to your landscape in the future, please check supported operation modes in SAP Note 2447994 before you enable SAP HANA system replication.

Note

When selecting an operation mode, keep in mind that the selected operation mode impacts the network throughput requirements of the communication channel used in SAP HANA system replication. For more information about this, see Network Recommendations.

To configure system replication with two hosts, you may have to change the host names.

In this example a single host system is used. In multi-host systems all hosts have to be renamed.

Note

To rename hosts in a production system replication landscape, system replication must be first deactivated. This means you have to first unregister and disable the secondary system before renaming any hosts. Once you have renamed the hosts then you can enable recovery mode again and register the secondary system with the primary system to re-activate system replication.

Procedure

1. Enable system replication on the primary system, with the hostname ej11.

   cd /usr/sap/<sid>/HDB<instancenr>/exe

   
   

   ./hdbnsutil -sr_enable --name=dcsite1

2. Stop the secondary system. The primary system can stay online.As <sid>adm

   /usr/sap/hostctrl/exe/sapcontrol -nr <instance_number> –function StopSystem HDB

3. Register the secondary system with the following command:

   cd /usr/sap/<sid>/HDB<instancenr>/exe

   
   

   ./hdbnsutil -sr_register

   --name=dcsite2

   --remoteHost=ej11

   --remoteInstance=50

   --replicationMode=sync

   --operationMode=logreplay

   Also see SAP Note 611361 Hostnames of SAP servers

4. Start the secondary system. This initiates the initial data transfer.As <sid>adm

   /usr/sap/hostctrl/exe/sapcontrol -nr <instance_number> –function StartSystem HDB

10.10.1.120    hanadb.support.com hanadb

10.10.1.26      hanadba.training.com hanadba

hdbnsutil -sr_enable --name='SITEA'

hdbnsutil -sr_enable --name='SITEB'

---------------------------------------------------------------------------------------------------------------------------------

Replication setup from SITEA to SITEB -Not working need to add ip of hanadba in hanadb server

1]hdbnsutil -sr_enable --name='SITEA'   --run this on primary(hanadba)

hdbnsutil -sr_enable --name='hanadba'

2]HDB stop                --run this on sec(hanadb)

3]Register SITEB

hdbnsutil -sr_register --name='SITEB' --remoteHost=hanadba --remoteInstance=01 --replicationMode=syncmem --operationMode=logreplay

4]HDB start              run this on sec(hanadb)

----------------------------------------------------------------------------------------------------------------------------------

Replication setup from SITEB to SITEA

Primary -hanadb 2.00.059.05.1662044871

Secondary -hanadba 2.00.066.00.1671096120

1]hdbnsutil -sr_enable --name=hanadba   --run this on primary(hanadb)

2]HDB stop                --run this on sec(hanadba)

3]Register SITEB

hdbnsutil -sr_register --name=hanadb --remoteHost=hanadba --remoteInstance=02 --replicationMode=syncmem --operationMode=logreplay

4]HDB start              run this on sec(hanadba)

hdbnsutil -sr_state

python systemReplicationStatus.py

/usr/sap/HP6/SYS/global/security/rsecssfs/data

/usr/sap/HP6/SYS/global/security/rsecssfs/key

Copy SSFS Keys :

scp SSFS_HP6.KEY hp6adm@hanadb:/usr/sap/HP6/SYS/global/security/rsecssfs/key

scp SSFS_HP6.DAT hp6adm@hanadb:/usr/sap/HP6/SYS/global/security/rsecssfs/data

vim /hana/shared/HP2/global/hdb/custom/config/global.ini

hdbnsutil -sr_register --remoteHost=hanadba --remoteInstance=01 --replicationMode=sync --name=hanadb --operationMode=logreplay

UnRegsiter

hdbnsutil -sr_unregister --name='SITEB'

HDB stop

HDB start

HDB stop

hdbnsutil -sr_register --name='SITEA' --remoteHost=hanadba --remoteInstance=00 --replicationMode=syncmem --operationMode=logreplay

HDB start

hdbnsutil -sr_takeover --suspendPrimary

hdbnsutil -sr_unregister --name='SITEA'

hdbnsutil -sr_disable

VIEWS:

M_SYSTEM_REPLICATION_TAKEOVER_HISTORY

M_SERVICE_REPLICTION

M_SYSTEM_REPLICATION

Configure a SAP HANA Multitarget System Replication

You can configure a multitarget system replication in which a primary system replicates data changes to more than one secondary system.

Context

In this example, primary system A in data center 1 replicates data changes to secondary system B in the same data center. Primary system A also replicates data changes to secondary system C in data center 2. Secondary system C is a source system for a further secondary system D located in the same data center with system C.

Procedure

1. On primary system A in data center 1:

   1. Create backups.

   2. Enable system replication.

2. On the local secondary system B in data center 1:

   1. Stop the system.

   2. Register it to A.

   3. Start the system.

3. On the remote secondary system C in data center 2:

   1. Stop the system.

   2. Register it to A.

   3. Start the system.

4. On the remote secondary system D in data center 2:

   1. Stop the system.

   2. Register it to C.

   3. Start the system.

Several solutions are available when the systems involved in a multitarget system replication configuration fail.

We are using the setup described in Multitarget System Replication to exemplify the procedure. In this setup, primary system A replicates data changes to secondary system B located in the same data center. Primary system A also replicates data changes to the secondary system C located in data center 2. Secondary system C is a source system for a further secondary system D located in the same data center with system C.

Failure on Primary System A

When primary system A fails, proceed as follows:

1. Take over on secondary system B in data center 1.

2. Register secondary system C in data center 2 to the new primary system B in data center 1. Then, register secondary system D in data center 2 to secondary system C.

3. After the failure on the previous primary system A is solved, register it to the new primary system B in data center 1.

Alternatively, you can set the global.ini/[system_replication]/register_secondaries_on_takeover parameter to true and take over on secondary system B in data center 1. As a result, secondary system C in data center 2 will register automatically to the new primary system B in data center, while secondary system D in data center 2 will register automatically to secondary system C. After the failure on the previous primary system A is solved, register it to the new primary system B in data center 1. See also the additional options described in Automatic Registration After Takeover.

Failure of Data Center 1

When all the systems in data center 1 fail, proceed as follows:

1. Take over on secondary system C in data center 2.

2. After the failure on the previous primary system is solved, register system A to the new primary system C in data center 2.

3. Register secondary system B as tier 3 to system A in data center 1.

Perform a Takeover with hdbnsutil

You can perform a takeover on your secondary system with the hdbnsutil command line tool.

Prerequisites

• The secondary system must be fully initialized.

  You can check this in M_SERVICE_REPLICATION or in the SAP HANA studio Administration Console Landscape System Replication. The secondary system is ready for takeover if all services display REPLICATION_STATUS ACTIVE.

• The takeover command can be executed both when the secondary system is offline or online.

Note

If you are performing a takeover as part of a planned downtime, you should first make sure that the primary system has been fully stopped before performing a takeover to the secondary system.

Procedure

As <sid>adm enter the following command on the secondary system to enable the secondary system to take over and become the primary system:

cd /usr/sap/<sid>/HDB<instancenr>/exe

./hdbnsutil -sr_takeover [--comment="Your Comment"]

Use the --comment to add a reason for the takeover (for example, to distinguish between a planned or unplanned takeover). This comment is displayed in the M_SYSTEM_REPLICATION_TAKEOVER_HISTORY monitoring view in the COMMENTS column.

If the system is offline, the takeover is actually carried out when the system is started again.

Perform a Failback with hdbnsutil

You can perform a failback using the hdbnsutil command line tool.

Prerequisites

• You need the operating system user to perform a failback. For more information, see Operating System User <sid>adm.

• The former primary system is not running.

• The current primary system is running.

Context

This is the same procedure as is used for setting up a normal secondary described in steps 2 and 3 of Configure SAP HANA System Replication with hdbnsutil. To fail back, attach your original primary system as new secondary system to the current primary system. These are identified in the following parameters of the command: --remoteHost=<new primary hostname>, --name=<new secondarySiteName>. Refer also to the hdbnsutil example Example: Configure SAP HANA System Replication where the primary and secondary are identified as dcsite1 and dcsite2 respectively.

Procedure

When the former primary is available again and offline, it can be registered as the new secondary.

hdbnsutil -sr_register --remoteHost=<new primary hostname>

--remoteInstance=<instance number>

--replicationMode=<sync/syncmem/async>

--operationMode=<delta_datashipping|logreplay|logreplay_readaccess>

--name=<new secondarySiteName>