Home » System Center Data Protection Manager

Category Archives: System Center Data Protection Manager

Self-service data recovery with Data Protection Manager, Service Manager and Orchestrator

January 3, 2013 / Leave a comment

In my sandbox I test a lot of solutions, management packs, integration packs and ideas. They don’t always work out the way expected 🙂 The result is that I often need to restore a database from backup. I use Data Protection Manager to protect my databases, Service Manager to order the restore and Orchestrator as the “doer”. In this blog post I want to share a example how to make restore of database a bit easier.

In the Service Manager self-service portal I have a request offering named Restore Database.

  • Restore to Original Location. If this checkbox is enabled the database will be restored to original instance. In most cases I restore to a network folder. Network Folder is used in this integration pack in the same way that it is used in the Data Protection Manager user interface. Choosing Network Folder recovers to a local path on a server that have the DPM agent installed. I have configured my runbook (also included in this blog post) to always recover to C:\RESTORE on the target machine
  • Target server. If I select to restore to a network folder, default, I input a server name. For example if I want to restore the Orchestrator database to my Orchestrator database server I input SCO12SP1-SQL01 in the Target Server text box. The database backup will then be restored to C:\RESTORE in the SCO12SP1-SQL01 server.
  • Recovery Point to Restore. In this query based list I can select which DPM recovery point to restore. I have a runbook (also included in this blog post) that create CI of each recovery point.

20130102_DPM_SelfService05

Service Manager invokes the “1.2 Restore” runbook in Orchestrator. The runbook is divided into two tracks depending if restoring to a network folder or to original location. Both Data Source ID and Recovery Source ID, used to recover the SQL database, is stored on the Backup CI in Service Manager so we dont need to get them from DPM within the runbook. In general the runbook restore the database and updates the service request.

20130102_DPM_SelfService02Runbook “1.1 Create Backup CIs” is the second runbook in this example. It is used to create backup CI objects in Service Manager. The backup class is a custom class that I have created with the Service Manager authoring tool. The runbook runs every hour and creates new/updates/deletes CIs of the backup class.

  • Every hour.  Invokes the runbook every hour
  • Get Existing Backup. Gets all objects of the backup class in Service Manager. If there are any objects the “Set Verified to FALSE” activity change the verified property of all the backup CIs to FALSE.
  • Junction. Used to merge possible multiple threads to one
  • Get Data Source for System Center DBs. In my DPM server I have a protection group named “System Center Databases”, this activity gets all data sources for that protection group
  • Get All Recovery Point. This activity gets all recovery points for the data sources returned by the “Get Data Sources for System Center DBs” activity
  • Check if Backup Exist. This activity checks in Service Manager if there is a backup CI, with Active status, for the current BackupID. BackupID is a property of the backup class that I use to give all recovery points a unique ID, the backup ID contains of <Protection Group Name>.<Production Server Name>.<Recovery Time Point in Time>.<Data source Name>
  • If a backup CI object already exist the runbook change the verified property of the backup CI object to TRUE
  • If no backup CI object exist a new backup CI is created and a relationship to the server is created
  • Junction. Used to merge possible multiple threads to one
  • Get Non Verified Backups. This activity gets all backup CIs that has not been verified (verified property equals FALSE) and deletes them with the “Delete Backup” activity

 

20130102_DPM_SelfService01

 

Backup CIs listed in the Service Manager Console

20130102_DPM_SelfService03

Backup CI

20130102_DPM_SelfService04Relationship between windows server and backup CI

20130102_DPM_SelfService06

 

When the runbook is done the service request is updated with some information, which can be read from the Service Manager self-service portal. As you can see the database has been restored to C:\RESTORE on the SCO12SP1SQL-01 server. A very Quick and easy way to roll back a database.

20130102_DPM_SelfService07

 

You can download my example files here, 20130103_DPM. Note that this is provided “AS-IS” with no warranties at all. This is not a production ready management pack or solution for your production environment, just a idea and an example.

DPM 2010 – SQL Server End-user recovery

October 30, 2009 / 2 Comments on DPM 2010 – SQL Server End-user recovery

Data Protection Manager (DPM) 2010 (currently in beta) allows backup administrators to authorize SQL Server database owners to recover their databases without intervention from the backup administrator.

To do this, the DPM administrator must create and manage DPM roles. A DPM role allows backup administrators to control what an end-user can recover and which instances of SQL Server they can recover to. A DPM role for SQL Server End-User Recovery (SQL Server EUR) includes the following:

  • Users: A security group that represents a set of end users.
  • Objects: SQL Server databases that can be recovered and instances of SQL Server that have been identified for an alternate instance recovery.
  • Recovery permissions for alternate instance recovery, which a DPM administrator uses for managing SQL Server EUR functionalities used by end-users.

 

Source: DPM 2010 beta documentation

The first thing you need to do is to install the SQL Server end-user recovery client on the protected machine, in this example hq-opsmgr28. The EUR client application enables end-users to perform recoveries of databases.

DPM_EndUser01

After that you need to create a role, that is done, in the current beta, with DPM Command shell. The following example creates a role named “Operations Manager SQL”.

$Role = New-DPMRole -name “OpsMgrSQL” -description “Operations Manager SQL” -DPMServerName HQ-DPM55
Add-DPMSecurityGroup -DPMRole $Role -SecurityGroups “Hq\OpsMgrSQL”
$DatabaseForEndUserRecovery = $null
$ListOfPGs = Get-ProtectionGroup -DPMServerName HQ-DPM55
$ListOfPGs | ForEach-Object {if ($_.FriendlyName -eq “SQL Protection Group”) {$PG = $_; break }}
$DatasourceInPG = Get-Datasource $PG
$DatasourceInPG | ForEach-Object {if ($_.LogicalPath -eq “HQ-OPSMGR28\OperationsManager”) {$DatabasesForEndUserRecovery +=,$_}}
Add-DPMRecoveryItem -DPMRole $Role -type SQLDatabase -datasource $DatabasesForEndUserRecovery
Add-DPMRecoveryItem -DPMRole $Role -type SQLInstance -sqlinstances “HQ-OPSMGR28”
$RecoveryTargetInstance = New-DPMRecoveryTarget -Type SQLINSTANCE -RecoveryTarget “HQ-OPSMGR28\Recovery” -RecoveredFilesPath C:\TEMP
Add-DPMRecoveryTarget $Role $RecoveryTargetInstance
Set-DPMRole -DPMRole $Role

This script will create a new role named OpsMgrSQL. It will add the security group HQ\OpsMgrSQL to this role. Get the list of protection groups and all data sources in SQL Protection Group, and then add the OperationsManager database to the list of databases we want to allow the role to recover . Then add the list to the role as recovery item. Add the required instance whose database we want to recover, and add the target instance and recovery target. Finally we save the role.

Members of the HQ\OpsMgrSQL group can now connect to the DPM server HQ-DPM55 and restore the OperationsManager database (from the hq-opsmgr28 server). The OperationsManager database is protected by the SQL Protection Group and can be restored to the HQ-OPSMGR28\Recovery SQL instance only.

From the end-user recovery wizard, in the protected server, the end-user can now recover the database

DPM_EndUser06

 

DPM_EndUser07

 

DPM_EndUser08

 

DPM_EndUser09

DPM_EndUser10

DPM_EndUser11

DPM_EndUser12

DPM_EndUser03

DPM_EndUser05

 

The OperationsManager database is restored and back online again. Performed by the SQL administrators without any delay or wait for backup administrators.

Look for new databases (…with a pinch of DPM)

August 25, 2009 / Leave a comment

With SQL Server Audit, SQL Server 2008 introduces an important new feature that provides a true auditing solution for enterprise customers. While SQL Trace can be used to satisfy many auditing needs, SQL Server Audit offers a number of attractive advantages that may help DBAs more easily achieve their goals such as meeting regulatory compliance requirements. These include the ability to provide centralized storage of audit logs and integration with System Center, as well as noticeably better performance. Perhaps most significantly, SQL Server Audit permits fine-grained auditing whereby an audit can be targeted to specific actions by a principal against a particular object. This paper provides a comprehensive description of the new feature along with usage guidance and then provides some practical examples. Source MSDN

If you want to get an alert when a new database is created in SQL 2008 you will first need to configure auditing on SQL side, and then a rule in Operations Manager to generate an alert. Configure a new audit with audit destination equals Application log or security log. If you select security log you might need to configure some extra security permissions. Create a new server audit policy, configure audit action type equals DATABASE_CHANGE_GROUP.

sqlAudit01

Next step is to create the rule that will pickup the SQL event and generate an alert. Create a new event based rule, target it to for example SQL Servers, to monitor all your SQL machines. Configure the rule to look for event ID 33205 including CREATE and DATABASE in the event description.

sqlAudit02

In the expression of the rule, we use “.” to tell Operations Manager “any character before, around or after the two keywords, CREATE and DATABASE.

sqlAudit03

 

Another step that you might want to do is to verify that you backup the new SQL database. I modified a power shell script and the result is that it will connect to your DPM server and a SQL box, it will then ask you if you want to add any of the unprotected databases on that server to a protection group in DPM. 

param([string] $ProductionServer, [string] $PGName)
if(!$ProductionServer)
{
$ProductionServer = read-host "Enter the production server name (a SQL server protected by DPM)"
}
if(!$PGName)
{
$PGName = read-host "Enter the name of your existing SQL protection group name"
}

$dpmservername = read-host “Enter the name of your DPM server”

connect-dpmserver $dpmservername
$dpmservername

$PGList = @(Get-ProtectionGroup $dpmservername)

foreach ($PG in $PGList)
{
if($PG.FriendlyName -eq $PGName)
{
write-host “Found protection group $PGName”
$MPG = Get-ModifiableProtectionGroup $PG
$PGFound=$true
}
}

if(!$PGfound)
{
write-host “Protection Group $PGName does not exist”
exit 1
}

$PSList=@(Get-ProductionServer $dpmservername)
$DsList = @()

foreach ($PS in $PSList)
{
if($PS.NetBiosName -eq $ProductionServer)
{
write-host “Running Inquiry on” $PS.NetbiosName
$DSlist += Get-Datasource -ProductionServer $PS -Inquire
$PSFound=$true
}
}

if(!$PSfound)
{
“Production Server $PS does not exist”
exit 1
}

$protectedDsList = @()
foreach ($ds in $dslist)
{
if($ds.ToString(“T”, $null) -match “SQL” -and !$ds.Protected)
{
$toadd = read-host “Do you want to protect the” $ds.Name “database? (y/n)?”
If ($toadd -eq “y”)
{
$protectedDsList += $ds
Add-ChildDatasource -ProtectionGroup $MPG -ChildDatasource $ds
$x=Get-DatasourceDiskAllocation -Datasource $ds
Set-DatasourceDiskAllocation -Datasource $x -ProtectionGroup $MPG
}
}
}

Set-ReplicaCreationMethod -ProtectionGroup $MPG -Now

if($protectedDsList.Length)
{
write-host “Adding new SQL DBs to” $MPG.FriendlyName
Set-protectiongroup $MPG
}

disconnect-dpmserver $dpmservername
“Exiting from script”

sqlAudit04

(tested in a sandbox, so I am aware that the ops mgr databases are not protected and all the test databases) If you want to integrate the script into Ops Mgr you should read this post from David Allen.

Using Pre-Backup and Post-Backup Scripts in DPM

October 19, 2008 / 1 Comment on Using Pre-Backup and Post-Backup Scripts in DPM

You can configure DPM to run scripts before (pre) and after (post) each DPM backup job. This can used to prepare the data source for backup and also to start a service again after backup. 

On each agent machine, in the DPM folder under \DPM\Scripting there is a file named ScriptingConfig.xml. This is the file that you use to configure pre-backup scripts and post-backup scripts. These scripts are run with the local system account. The script needs to be configured on each agent, or you could use some find of deployment system to deploy the configuration file to multiple machines.

In my test I simply stopped the print spooler (spooler) service and started it again after backup. I created two CMD files in C:\Scripts. The files include “net start spooler” and “net stop spooler”. My test machine ScriptConfig.xml looks like this (click to view the full image):

The BackupCommandLine lines is for command-line parameters to be passed to the script. The DataSourceName attribute is the data source volume or name, for example a storage group name in Exchange.

I have not found a good way to see if the pre-backup and post-backup script ran successfully. But if you have Operations Manager in your environment, you can configure your scripts to echo to event viewer. Then you can use the missing event monitor to make sure the scripts are run successfully every X minute, for example if you do a backup every hour, then you want an event every hour too.

Remember to include your scripts in your disaster recovery plan.

 

Microsoft Hyper-V Server and System Center

October 7, 2008 / 1 Comment on Microsoft Hyper-V Server and System Center

I installed a Microsoft Hyper-V Server a couple of days ago. Hyper-V Server is a version of Windows Server 2008 core that includes the Hyper-V role, all other roles are disabled. There is a new simple interface to configure the server. You can also use the Hyper-V MMC and System Center Virtual Machine Manager to administer your Hyper-V server. More info about Hyper-V Server here.

I installed an agent from Operations Manager 2007 without any problem. However I found out that Ops Mgr found the machine as 2008 full installation (installation type = full), not as a core machine (installation type = core). Not the worst thing, but it is good to know. You can see it if you create a new state view target Windows Server 2008 Computers.

I also tried to install a Data Protection Manager 2007 agent on my Hyper-V server. First DPM could not identify my Hyper-V server as a known Windows version. In Active Directory I could see that my Hyper-V server operating system is set to “Hyper-V Server”, not “Windows Server 2008 Enterprise” as my other core machines. When I changed it to “Windows Server 2008” there was no problem adding it to DPM and installing the agent. I guess that the DPM query does not recognize “Hyper-V Server” as a supported operating system, but that works too after this “workaround”.

Protecting Exchange 2007 with DPM 2007

September 21, 2008 / 1 Comment on Protecting Exchange 2007 with DPM 2007

This week I have been doing some tests with Microsoft Data Protection Manager 2007 (DPM). Data Protection Manager 2007 is a member of the Microsoft System Center family of management products, the product family that I work primary with. Focused on the primary Microsoft server workloads, DPM 2007 was specifically built to protect and recover SQL Server, Microsoft Exchange Server, SharePoint Portal Server, Microsoft Virtual Server, as well as Windows file services. I have been testing protection of Microsoft Exchange 2007. You can find more info about DPM here.

ESEUTIL offload and log file database integrity

I configure my DPM server to do express full backup (all blocks that has changed since last express full backup, update the replica on the DPM server) every night, and synchronization every 15 minute. That will give me 96 recovery points per day, one recovery point for each full backup or synchronization.

To ensure consistent backups, DPM can run ESEUTIL against the Exchange data replica on the server. In other words DPM offloads the overhead from the Exchange server and still ensures a consistent, fully supported backup. After the synchronization is complete DPM sends a message to VSS on the Exchange server to truncate the logfiles. You can review this in the local event views on the Exchange server. You can also use the ESEUTIL determine which logs have been committed to the exchange database, this is good for troubleshooting scenarios. For example if you have the following logfiles on you Exchange server

  • E02000000CD
  • E02000000CE
  • E02000000CF
  • E02000000D0

and you run the ESEUTIL /mk command you can see that for example E02000000D0 is the first uncommitted log for the appropriate database, that means that all the C* log files will be truncated after the next synchronization. There will always be some log files in the folder , even if you have a idle Exchange server, a idle stand-alone mailbox server can generate up to 96 logs during a day. More info about that here

According to this KB you need to copy a number of files to your DPM server before it can run ESEUTIL. That is correct, but I never found an Exosal.dll file on my Exchange server and the database check seems to be working without it.

 

“Latest” recovery point 

Another feature I tested is the “latest” recovery point. It means that DPM first will first restore the database to the latest syncronization, in my case 15-minute recovery point, then reapply all log files that are still on the Exchange server. I tried this by first sending an e-mail between two users, in a time slot between two synchronizations. I then shut down the Exchange server and removed the data volume. I added a new blank disk and restarted the Exchange server. From my DPM server I started the recovery wizard and selected “latest” as recovery point, for my storage group. When the recovery was complete and I started Outlook at my workstation I could see that the e-mail that I had sent between two synchronizations was in my inbox. That means that DPM had first restored the latest synchronization and then reapplied Exchange log files.

 

 

Recovering a mailbox

You can easy restore a mailbox with the recovery wizard. Double click the storage group, select a mailbox and click restore. Before you do this, note that DPM will not setup a recovery storage group for you. You will need to do that on your own and also configure the “database can be overwritten by restore” flag. When the database is restored, the work in DPM is finished. You now need to use Exchange tools to mount the database in the recovery storage group and recover the mailbox data. MS Exchange Troubleshooting Assistant in Exchange 2007 can help you with that, for example to merge two mailboxes. If you want to restore the mailbox to another mailbox (not the same GUID), for example if you deleted the mailbox and create a new mailbox for the user, you can use the “restore-Mailbox” power shell command. The DPM restore is always a mailbox database (edb) and log files. You need to use Exchange tools to get the data into a PST.

 

Last year I did a test on DPM System Recovery Tool. The DPM System Recovery Tool (SRT) is a tool within Microsoft System Center Data Protection Manager 2007. SRT helps you recover systems that no longer start at all. Instead of having to first restore the operating system, then install all patches, updates, applications and then restore the application backup, SRT can roll back a complete system. you can read about that here.

I found a couple of useful links when protecting Exchange with DPM.
• Using Eseutil to Determine Which Logs Have Been Committed, link
• Hexa to Binary and Decimal converter/convertor, link

DPM MP Reports

January 2, 2008

I have installed the System Center Data Protection Manager 2007 (DPM) management pack for Operations Manager 2007 and notice that there are no reports included in the management pack, but there are a number of reports in the DPM 2007 Administrator Console. If you want to view these reports from the Ops Mgr console you can create a new Web Page View (a true cowboy solution).

  1. Right-click Monitoring\Monitoring in the console and choose to create a new Web Page View
  2. Input a suitable name, for example DPM Reports
  3. Input a suitable description
  4. Input target website. Input http://dpmserver/Reports. Replace “dpmsever” with the name of your DPM server for example http://emea-dpm-01/reports.

In this example I am running DPM db on a “real” SQL server, not the built-in SQL software that comes with DPM. You will have to give your operators permissions to view DPM reports. You can do that from SQL Report Manager

  1. Browse to http://dpmserver/Reports
  2. Click “Show Details”
  3. Click “Edit” in front of “DPMReports”, to get the DPMReports folder properties
  4. Click “Security” on the left side
  5. Click “New Tole Assignment”
  6. In the New Role Assignment window, input the security group or user account, for example “EMEA\OpsMgrAdmins” (EMEA is my domain)
  7. Assign the new group or user a suitable role for example “Browser”
  8. Click OK and your are done

 

 

 

Bare Metal Recovery

December 16, 2007 / 1 Comment on Bare Metal Recovery

The DPM System Recovery Tool (SRT) is a tool within Microsoft System Center Data Protection Manager 2007. SRT helps you recover a system that no longer start at all. Instead of having to first restore the operating system, then install alla patches, updates, applications and then restore the application backup, SRT can roll back a complete system.

SRT use a storage method which only stores one copy of any file with a unique content address computed by a 128-bit MD5 cryptographic algorithm. This means that if you have a number of similar machines the first machine will take the same amount of space as the space that you backup. But the other machines will only need 2% to 5% of space.

Before you can restore anything with SRT you will have to create a recovery point schedule. This schedule will tell DPM how often to backup (create recovery points) the server and also the which recovery set to use. The recovery set define what to include in the recovery point. SRT includes two default recovery sets:     

  • System Volumes (system files, system settings and boot files)
  • Disk Layout (disk information, including master boot code, partition table, partition and volume information)

You can create custom recovery sets, example to protect extra volumes, from DPM System Recovery Tool.

After you have create a schedule and you have at least one recovery point sucessfully created you can restore your machine. You can administrate both schedules, recovery sets and recovery points from DPM System Recovery Tool.

To do a bare metal restore you need to create a client boot CD. The client boot CD will be used to boot the target machine and contact the DPM server. This CD is not a unique for each machine, you can create it in DPM Recovery Center.

To restore your machine you can do like this

  1. Make sure the orginal machine is down and offline.
  2. Boot the target machine with the client boot CD (SRTBootClient.iso)
  3. DPM System Recovery Tool will start and you will be asked for TCP/IP Configuration
  4. After you have input TCP/IP configuration the system will wait for a signal from the DPM server
  5. Start Recovery Center at your DPM server and select your server. You can see all information about the recovery points in recovery center. Click Roll Back System to start the recovery. The DPM System Recovery Tool Rollback Wizard will start.
  6. In the Rollback Wizard you can select recovery point, recovery sets to restore and disks to include. In the end of the wizard you start the rollback.
  7. After a short while your target machine will be rebooted and the roll back is complete. You will get a report in HTML format and also information about the result in the recovery center.

Some screenshots from the process

 

Client Boot
 

 

System Recovery Tool Rollback Wizard

Event ID 12302: Volume Shadow Copy Service error

November 26, 2007 / 1 Comment on Event ID 12302: Volume Shadow Copy Service error

I ran into a issue today with Data Protection Manager 2007. When I tried to create a new protection group and select members I got

The Operation Failed bacuse of a protection agent failure
Retry the operation
ID: 998
Details Unknow error (0x80042318) (0x80042318)

on the member I found this error

Volume Shadow Copy Service error: An internal inconsistency was detected in trying to contact shadow copy service writers.  Please check to see that the Event Service and Volume Shadow Copy Service are operating properly.

The workaround I found was to (on the member server)

  1. Backup the follwing regkey: HKLM\Software\Microsoft\EventSystem\{26c409cc-ae86-11d1-b616-00805fc79216}\Subscriptions
  2. Erase the regkey
  3. Reboot.
  4. Verify that things are working by running “vssadmin.exe list writers”

The deleted regkey will be regenerated after reboot.

Remeber to install article ID 940349 on the server you are trying to protect, else you will get an error about pre requirements when trying to select members to a proction group. At this page you can see all Software Prerequisites for different kind of workload.

Links During October

October 31, 2007 / 1 Comment on Links During October

  • New MP, Microsoft Live Communications Server 2005 Management Pack for Operations Manager 2007, link
  • New MP, Office Communications Server 2007 Management Pack for MOM 2005, link
  • New MP, Microsoft Compute Cluster Server 2003 Management Pack for Operations Manager 2007, link
  • New MP, Microsoft Active Directory Federation Services 2003 Management Pack for Operations Manager 2007, link
  • New MP, Microsoft BizTalk Server 2006 R2 Management Pack for Microsoft Operations Manager 2005, link
  • New MP, Microsoft Windows Server 2000/2003 Operating System Management Pack, link
  • New MP, Microsoft Windows Server 2000/2003 DHCP Management Pack for Operations Manager 2007, link
  • New MP, Microsoft Forefront Client Security Health Management Pack for Microsoft Operations Manager 2005 SP1, link
  • New MP, Microsoft Windows Server 2000/2003 DHCP Management Pack for Operations Manager 2007, link
  • New MP, Microsoft Exchange Server 2007 Management Pack for Operations Manager 2007, link
  • New MP, Server Virtualization Management Pack for Microsoft System Center Operations Manager 2007, link

 

  • A new trial (120-days) version of Data Protection Manager, download here
  • New Tool, OpsMgr 2007 Database and Data Warehouse Size Calculator, download here

 

  • New KB, The installation of the System Center Operations Manager 2007 Reporting Server feature on a domain controller is now supported, link
  • New KB, How to use diagnostic tracing in System Center Operations Manager 2007 and in System Center Essentials 2007, link
  • New KB, Error message when you try to generate a report in System Center Operations Manager 2007: “Cannot initialize report”, link
  • New KB, A computer agent unexpectedly generates heartbeat alerts after you put it into Maintenance mode in System Center Operations Manager 2007, link
  • New KB, How to collect and monitor UNIX Syslogs in System Center Operations Manager 2007 or in System Center Essentials 2007, link
  • New KB, Discovery information is missing after you delete and then reimport a management pack in Microsoft System Center Operations Manager 2007 , link
  • New KB, Error message when you try to download a file by using the Background Intelligent Transfer Service: “Content file download failed”, link
  • New KB, A management group that contains an Exchange Server computer may intermittently stop receiving new alerts in Microsoft System Center Operations Manager 2007, link
  • New KB, You receive many error alerts after you import the Exchange Server Management Pack for Microsoft Operations Manager 2005, link
  • New KB, The Exchange 2003 Management Pack Configuration Wizard may crash when the “Message Tracking” option is turned on, link
  • New KB, Best practices to use when you configure overrides in System Center Operations Manager 2007, link
  • New KB, SCOM Reporting installation fails when the DB name contains special characters, link
  • New KB, SCOM Installation fails on 64 bit Windows Server, link
  • New KB, Management servers that are running the release version of System Center Operations Manager 2007 do not process the agent event data from an agent of System Center Operations Manager 2007 Service Pack 1, link

 

Post navigation

← Older posts