using BCP to transfer a large amount of data in SQL Server

I would like to transfer hundred millions of records between table in different database server. I tried to use DTS export/import and it’s not fast enough for my needs but i forget that there is a BCP command in SQL which is SQL Server format file.

The speed in my local machine is (22795.13 rows per sec)

Sample of the command: (You need to run it in command prompt), you can find bcp.exe in folder (C:\program files\Microsoft SQL Server\100\Tools\Binn) – The 100 is for SQL Server 2008, for SQL server 2005 it’s 90

Sample

1. You need to generate the BCP File from the source table

bcp MyDatabase.dbo.MyTable out C:\BCP_MyTable -n -S localhost -T -e[BCP_MyTable_ERROR]

*Using -T is for trusted connection

2. You need to import the BCP file to the destination table

bcp MyDatabase.dbo.MyTable in C:\BCP_MyTable -n -S localhost -T -e[BCP_MyTable_ERROR]

for further command line reference click here

*UPDATE:

you can configure your server to to reduce the amount of transaction log during bulk copy/insert transaction by executing

EXEC SP_DBOPTION MyDatabase, ‘SELECT INTO/BULKCOPY’, TRUE

bcp MyDatabase.dbo.MyTable in C:\BCP_MyTable /b 20000 -n -S localhost -T -e[BCP_MyTable_ERROR]

What I did was to add extra parameter of /b and the number 20000 (you should play around with the number to see the best one fit your situation) after it means the number of the records per transaction. Please be careful if you don’t put /b parameter, sql server by default will commit all the records at once. but if you put /b parameter then it will commit the transaction per x amount of records specified. If you transfer large data, you shouldn’t commit all at once because it will take sometime. In my case I transferred 18 millions of records

I’d recommend you to save the BCP file and copy the file over if it’s in different server because you need to be careful with the bandwidth of the network and the risk if the network connection is interrupted

Restore SQL Server database through network path

I had a problem where I need to restore database using shared folder on the network. I tried to copy it locally to my local harddrive but then the connection is not stable enough. Hence I decided to create a script

Single Media

USE MASTER
DECLARE @fromdatabase varchar(50)

DECLARE @todatabase varchar(50)
DECLARE @sql as nvarchar(4000)
DECLARE @BackupPath as nvarchar(250)



/***************************************/
/* Change this string to match the database
	name you want to create a backup of  */
Set @fromdatabase = 'DailySupport-1'
Set @todatabase = 'MyDatabaseName'
/***************************************/



/***************************************/
/*  Which db server are you using?     */
/***************************************/

Set @BackupPath='\\sharednetwork\SQL\DatabasesBackup\'

/***************************************/
/***************************************/


/** Kill off any existing connections **/
SET @sql = ' use master'
SET @sql = @sql + ' EXEC sp_KILLSPIDS ' + @todatabase + ';'
Print @sql
EXEC sp_executesql @sql


/** Perform the restore **/
SET @sql = 'RESTORE DATABASE ' + @todatabase
SET @sql = @sql + ' FROM DISK = N''' + @BackupPath + '' + @fromdatabase + '.bak'''
SET @sql = @sql + ' WITH FILE = 1,NOUNLOAD, REPLACE, STATS=10 ;'
Print @sql
EXEC sp_executesql @sql

Multiple Media(Sometimes the backup source is stripped multiple files hence if you try to run it with above script you will get this error “The media set has 2 media families but only 1 are provided. All members must be provided.”)

USE MASTER
DECLARE @fromdatabase varchar(50)
DECLARE @fromdatabase2 varchar(50)

DECLARE @todatabase varchar(50)
DECLARE @sql as nvarchar(4000)
DECLARE @BackupPath as nvarchar(250)



/***************************************/
/* Change this string to match the database
	name you want to create a backup of  */
Set @fromdatabase = 'DailySupport-1'
SET @fromdatabase2 = 'DailySupport-2'
Set @todatabase = 'MyDatabaseName'
/***************************************/



/***************************************/
/*  Which db server are you using?     */
/***************************************/

Set @BackupPath='\\sharednetwork\SQL\DatabasesBackup\'

/***************************************/
/***************************************/


/** Kill off any existing connections **/
SET @sql = ' use master'
SET @sql = @sql + ' EXEC sp_KILLSPIDS ' + @todatabase + ';'
Print @sql
EXEC sp_executesql @sql


/** Perform the restore **/
SET @sql = 'RESTORE DATABASE ' + @todatabase
SET @sql = @sql + ' FROM DISK = N''' + @BackupPath + '' + @fromdatabase + '.bak'''
SET @sql = @sql + ', DISK = N''' + @BackupPath + '' + @fromdatabase2 + '.bak'''
SET @sql = @sql + ' WITH FILE = 1,NOUNLOAD, REPLACE, STATS=10 ;'
Print @sql
EXEC sp_executesql @sql

How to set variable from Dynamic SQL

Here is the scenario, You have a stored procedure that builds dynamic insert SQL which means the “INSERT” statement is being composed on the fly based on the table name but at the same time you need to get SCOPE_IDENTITY or the last identity number inserted for further processing (e.g History table)

	DECLARE @nvcSQL			NVARCHAR(MAX)
	DECLARE @pvcTableName	VARCHAR(255)
	DECLARE @pvcColumn		VARCHAR(255)
	DECLARE @pvcValue		NVARCHAR(MAX)

	--This is used to actually get the primary key for identity record
	DECLARE @dvcPrimaryKey VARCHAR(255)
	SET @dvcPrimaryKey = ''

	DECLARE @dintPrimaryKey INT
	SELECT @dvcPrimaryKey = ISNULL(NAME,'') FROM SYS.COLUMNS WHERE OBJECT_NAME(OBJECT_ID) = @pvcTableName AND Is_Identity = 1


	-- Only execute when there is a @pvcValue.
	IF @pvcValue  ''
	BEGIN
		SELECT @nvcSQL = 'INSERT INTO ' + @pvcTableName + '(' + @pvcColumn+ ') VALUES (' + @pvcValue + ')'
		SELECT @nvcSQL = @nvcSQL + ';' + 'SELECT @dintPrimaryKey = SCOPE_IDENTITY()'
		--PRINT @nvcSQL

		EXEC sp_executesql @query = @nvcSQL, @params = N'@dintPrimaryKey INT OUTPUT', @dintPrimaryKey = @dintPrimaryKey OUTPUT
		--EXEC (@nvcSQL)
	END

	IF @dvcPrimaryKey  ''
	BEGIN
		SELECT (@dvcPrimaryKey + ' = ' + CAST(@dintPrimaryKey AS VARCHAR(10)))  AS PrimaryKey
	END
	ELSE
	BEGIN
		SELECT '' AS PrimaryKey
	END

Question: why do we need to use sp_executesql instead of EXEC?because EXEC just execute a SQL command without returning/expecting any result from it

TRY….CATCH Rollback Transaction In SQL Server 2005

This feature has been sometime in SQL Server 2005 in SQL Server 2000 you normally use @@TRANCOUNT to detect any exception but now in SQL Server 2005 you can use try catch block.

In this snippet, there are 2 INSERT statement and 1 UPDATE statement. I’m expecting when there is any failure (e.g the UPDATE statement fails) then all the preceding INSERT/UPDATE/DELETE within the TRANSACTION block will be canceled

e.g

BEGIN TRY
BEGIN TRANSACTION transABC

INSERT INTO TABLEA (col1,col2,col3) VALUES ('a','b','c')
INSERT INTO TABLEB (col1,col2,col3) VALUES ('a','b','c')

UPDATE TABLEA SET col2='abcde' WHERE col1 = 'a'

COMMIT TRANSACTION transABC
END TRY

BEGIN CATCH
IF @@TRANCOUNT > 0
ROLLBACK TRANSACTION transABC --RollBack in case of Error

-- Raise an error with the details of the exception
DECLARE @ErrMsg nvarchar(4000), @ErrSeverity int
SELECT @ErrMsg = ERROR_MESSAGE(),
@ErrSeverity = ERROR_SEVERITY()

RAISERROR(@ErrMsg, @ErrSeverity, 1)

END CATCH

SQL Server Function using CLR

UPDATED: I’ve added one function to write from BLOB in SQL Server table to the disk straight away

I thought this article might be useful for anyone that wants to implement .NET code to SQL server level. In this case I really need CLR because I want to do compression of images and I believe it’s not possible to do that using pure SQL server stored procedure and I’m trying to avoid creating a .NET application just for compression of images through row by row processing.

Here we start:

This is your C#/.NET code, you need to declare it as SQL function or SQL stored procedure

using System.Data.SqlTypes;
using Microsoft.SqlServer.Server;
using Zip = ICSharpCode.SharpZipLib.Zip.Compression;

namespace CLRCompressionFunctions
{
    public partial class CompressionCore
    {
        [SqlFunction()]
        public static SqlBytes fn_Compress(SqlBytes uncompressedBytes)
        {
            if (uncompressedBytes.IsNull)
                return uncompressedBytes;

            MemoryStream memory = new MemoryStream();

            ICSharpCode.SharpZipLib.Zip.Compression.Streams.DeflaterOutputStream stream =
                new ICSharpCode.SharpZipLib.Zip.Compression.Streams.DeflaterOutputStream(memory, new Zip.Deflater(Zip.Deflater.BEST_COMPRESSION), 131072);

            stream.Write(uncompressedBytes.Buffer, 0, Convert.ToInt32(uncompressedBytes.Length));
            stream.Flush();
            stream.Close();
            stream = null;

            return new SqlBytes(memory.ToArray());
        }

        [SqlFunction()]
        public static SqlBytes fn_Decompress(SqlBytes compressedBytes)
        {
            if (compressedBytes.IsNull)
                return compressedBytes;

            ICSharpCode.SharpZipLib.Zip.Compression.Streams.InflaterInputStream stream =
                new ICSharpCode.SharpZipLib.Zip.Compression.Streams.InflaterInputStream(new MemoryStream(compressedBytes.Buffer));
            MemoryStream memory = new MemoryStream();
            byte[] writeData = new byte[4096];
            int size;

            while (true)
            {
                size = stream.Read(writeData, 0, writeData.Length);
                if (size > 0)
                {
                    memory.Write(writeData, 0, size);
                }
                else break;
            }
            stream.Flush();
            stream.Close();
            stream = null;

            return new SqlBytes(memory.ToArray());
        }

[SqlFunction()]
        public static SqlString fn_WriteFile(SqlString path, SqlBytes bytesFile, SqlBoolean isCompressed)
        {
            string returnString = string.Empty;

            try
            {
                //check if the file exists or not
                FileStream myFStream = new FileStream(path.ToString(), FileMode.OpenOrCreate, FileAccess.ReadWrite);

                SqlBytes fileBytes = bytesFile;

                if (isCompressed)
                {
                    fileBytes = fn_Decompress(bytesFile);
                }

                int Length = 256;
                Byte[] buffer = new Byte[Length];

                Stream readStream = fileBytes.Stream;

                int bytesRead = readStream.Read(buffer, 0, Length);

                // write the required bytes
                while (bytesRead > 0)
                {
                    myFStream.Write(buffer, 0, bytesRead);
                    bytesRead = readStream.Read(buffer, 0, Length);
                }

                readStream.Close();
                myFStream.Close();
                returnString = "File is written successfully";
            }
            catch (Exception ex)
            {
                returnString = ex.ToString();
            }

            return new SqlString(returnString);
        }
    }
}

Installation time to your SQL Server, You need to register your assembly(.dll) as well as referenced Assembly to SQL Server

ALTER DATABASE TestAssembly SET TRUSTWORTHY ON
GO
EXEC sp_configure 'clr enabled', 1;
RECONFIGURE WITH OVERRIDE;
GO
CREATE ASSEMBLY [ICSharpCode.SharpZipLib.dll]
                  FROM 'D:\Applications\FileCompressorApp\CLRCompressionFunctions\Deployment\ICSharpCode.SharpZipLib.dll'
                  WITH PERMISSION_SET = UNSAFE
GO
CREATE ASSEMBLY [CLRCompressionFunctions]
                  FROM 'D:\Applications\FileCompressorApp\CLRCompressionFunctions\Deployment\CLRCompressionFunctions.dll'
                  WITH PERMISSION_SET = EXTERNAL_ACCESS
GO

PERMISSION_SET = SAFE only if you don’t want the assembly accessing external resources such as writing to disk, but in this case the function is used to write into the disk
e.g How about if you want to use/register System.Drawing to your assembly? Yes you can do it by using

CREATE ASSEMBLY [System.Drawing.dll]
FROM 'C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\System.Drawing.dll'
WITH PERMISSION_SET = UNSAFE

You need to enable the CLR on SQL server in order to use your function

Now you need to create your function based on your assembly

CREATE FUNCTION [fn_Compress]           (
                 @uncompressedBytes varbinary(MAX))
            RETURNS varbinary(MAX)
            AS    EXTERNAL NAME CLRCompressionFunctions.[CLRCompressionFunctions.CompressionCore].fn_Compress
GO
CREATE FUNCTION [fn_Decompress]           (
                 @uncompressedBytes varbinary(MAX))
            RETURNS varbinary(MAX)
            AS    EXTERNAL NAME CLRCompressionFunctions.[CLRCompressionFunctions.CompressionCore].fn_Decompress
GO
CREATE FUNCTION [fn_WriteFile]           (
                 @path nvarchar(4000),
				 @bytesFile varbinary(MAX),
				 @bitCompressed bit)
			RETURNS nvarchar(4000)
            AS    EXTERNAL NAME CLRCompressionFunctions.[CLRCompressionFunctions.CompressionCore].fn_WriteFile
GO

Usage, It’s the same as you call a function in SQL Server

SELECT dbo.[fn_Compress](testimage) FROM tblImages
SELECT dbo.[fn_Decompress](imgFileContent) FROM TABLE_NAME
GO
SELECT dbo.[fn_WriteFile]('c:\test.pdf',imgFileContent, 0) FROM TABLE_NAME
GO 

Self Inner Join with Group By in SQL Server

This is the case:
I have a table and basically this table has many redundancy. The solution is to create a link table and this table will contains record association. So what I need to do is to get all the ID’s Populated to the link table and link it to a single unique record on the original table. The logic that I used is to get the value from the column based on the lowest ID.

INSERT INTO tblExpReceiptFileAssociation(intExpenseDtlId, intFileID)
	SELECT te.intExpenseDtlID, ta.LinkFileID
	FROM tblExpReceiptFile te
	INNER JOIN
		(
			SELECT vcFileName, bintFileSize, chCreateStaffCode,
				   CONVERT(VARCHAR,sdCreateDate,101) as DateCreated,
				   MIN(intFileID) as LinkFileID
			FROM
				tblExpReceiptFile
			GROUP BY
				vcFileName, bintFileSize,
				chCreateStaffCode, CONVERT(VARCHAR,sdCreateDate,101)
		) ta
	ON
		te.vcFileName = ta.vcFileName
	AND
		te.bintFileSize = ta.bintFileSize
	AND
		te.chCreateStaffCode = ta.chCreateStaffCode
	AND
		CONVERT(VARCHAR,te.sdCreateDate,101) = ta.DateCreated

Linq.Binary(SQL Image Data Type) Type to File Stream

I got a column with Data Type of Image in SQL Server. What I would like to do is to omit this data to a FileStream or to a file. I tried to read the data using LINQ and I found the data type detected for that particular column is System.Data.Linq.Binary. I was expecting it to be Byte Data type. So I need to convert the Binary to byte then to File Stream. But I found a simpler way by using “ToArray” properties and cast it back to byte solves my problem.

    foreach (tblExpReceiptFile file in ExpReceiptFactory.tblExpReceiptFileSelect())
            {
                string fileName = file.vcFileName.Replace(" ","_");
                FileStream fileStream = File.Create(DirectoryPath + @"\" + fileName + ".pdf");
                fileStream.Write((byte[])file.imgFileContent.ToArray(), 0, ((byte[])file.imgFileContent.ToArray()).Length);
                fileStream.Close();
            }

SQL Methods in LINQ

I’m trying to implement DateDiff in LINQ. I would like to get all records with the same date and ignoring the timestamp.

Here is my code snippet which doesn’t work

 public int tblExpReceiptFileUniqueID(string chCreateStaffCode, string vcFileName,
                                                long intFileSize, DateTime? sdCreateDate)
        {
            var expReceiptFile = from ef in DataContext.tblExpReceiptFiles orderby ef.intFileID
                                 where ef.chCreateStaffCode == chCreateStaffCode
                                 && ef.vcFileName == vcFileName
                                 && ef.bintFileSize == intFileSize
                                 && ((DateTime)ef.sdCreateDate).Date == sdCreateDate.Value.Date
                                 select ef;

            tblExpReceiptFile expReceiptFileRec = expReceiptFile.First();

            return expReceiptFileRec.intFileID;
        }

I’m trying to cast the column to “DateTime” and use the “Date” property but I got this error
“The Member ‘System.DateTime.Date’ has no supported translation to SQL”

I thought of “DATEDIFF” function in SQL server and I’m trying to get my head around in implementing this using LINQ and I found that we can use a library called “SQLClient”

using System.Data.Linq.SqlClient;

Code Snippet which is working

    public int tblExpReceiptFileUniqueID(string chCreateStaffCode, string vcFileName,
                                                long intFileSize, DateTime? sdCreateDate)
        {
            var expReceiptFile = from ef in DataContext.tblExpReceiptFiles orderby ef.intFileID
                                 where ef.chCreateStaffCode == chCreateStaffCode
                                 && ef.vcFileName == vcFileName
                                 && ef.bintFileSize == intFileSize
                                 && SqlMethods.DateDiffDay(ef.sdCreateDate, sdCreateDate.Value.Date) == 0
                                 select ef;

            tblExpReceiptFile expReceiptFileRec = expReceiptFile.First();

            return expReceiptFileRec.intFileID;
        }

Check Existence of temp table in memory

How to check whether the temp table is exists on the memory or not?The reason why you need this is because your stored procedure can throw the error when you try to drop a temp table which is not exists anymore on the memory. The best practice to drop a temp table is normally to check the existence of the table on the memory then we drop the table if it is exists

--Drop the table after usage
IF object_id('tempdb..#tmpStandardCostRate') IS NOT NULL
BEGIN
   DROP TABLE #tmpStandardCostRate
END

Find Index in all the tables SQL Server

This is a query to find all the indexes in your table including when it was last updated

Find all indexes on all tables

SELECT
	OBJECT_NAME(OBJECT_ID) AS 'Table Name',
	[name] as 'Statistic',
	STATS_DATE(object_id, index_id) AS 'Last Updated Statistics Date'
FROM
	sys.indexes
ORDER BY
	 STATS_DATE(object_id, index_id)
DESC

Find all indexes on a particular table

SELECT
	OBJECT_NAME(OBJECT_ID) AS 'Table Name',
	[name] as 'Statistic',
	STATS_DATE(object_id, index_id) AS 'Last Updated Statistics Date'
FROM
	sys.indexes
WHERE
        OBJECT_NAME(OBJECT_ID)  = 'YourTableName'

This SQL query returning the information about number of rows in the table as well as number of update/insert/delete after the last index has been rebuilt

SELECT
	'TABLE ' = substring(sysobjects.name,1,30) , ' INDEX ' = substring(sysindexes.name,1,30)
	,sysIndexes.rowcnt, sysindexes.rowmodctr
	,[last updated]=STATS_DATE(sysobjects.id, sysindexes.indid)
        ,user_seeks,user_scans,user_lookups,user_updates
FROM	sysobjects
INNER JOIN sysindexes ON sysobjects.id = sysindexes.id  AND sysindexes.indid > 0
INNER JOIN sys.dm_db_index_usage_stats iusage ON iusage.object_id = sysobjects.id  AND iusage.index_id = sysindexes.indid
WHERE
	sysobjects.xtype = 'U'
AND
	iusage.database_id = (SELECT dbid FROM master.dbo.sysdatabases WHERE [name] = db_name())
ORDER BY sysobjects.name