Design and implementation of SCSI target emulator

Download Design and implementation of SCSI target emulator

Post on 05-Jul-2016




1 download

Embed Size (px)


<ul><li><p>TSINGHUA SCIENCE AND TECHNOLOGYISSN 1007-0214 07/21 pp38-43Volume 11, Number 1, February 2006</p><p>Design and Implementation of SCSI Target Emulator*</p><p>PAN Jiaming (), SHU Jiwu ()**, ZHANG Suqin (), ZHENG Weimin ()</p><p>Department of Computer Science and Technology, Tsinghua University, Beijing 100084, China </p><p>Abstract: A SCSI target emulator is used in a storage area network (SAN) environment to simulate the </p><p>behavior of a SCSI target for processing and responding to I/O requests issued by initiators. The SCSI target </p><p>emulator works with general storage devices with multiple transport protocols. The target emulator utilizes a </p><p>protocol conversion module that translates the SCSI protocols to a variety of storage devices and</p><p>implements the multi-RAID-level configuration and storage visualization functions. Moreover, the target</p><p>emulator implements RAM caching, multi-queuing, and request merging to effectively improve the I/O </p><p>response speed of the general storage devices. The throughput and average response times of the target</p><p>emulator for block sizes of 4 KB to 128 KB are 150% faster for reads and 67% faster for writes than the</p><p>existing emulator. With a block size of 16 KB, the I/O latency of the target emulator is only about 20% that of </p><p>the existing emulator.</p><p>Key words: storage area network (SAN); iSCSI; SCSI target emulator </p><p>Introduction</p><p>With the explosion in the amount of information, theperformance and scalability of storage systems have become critical issues along with the system cost-effectiveness. Storage area network (SAN)[1] is anew storage system model with excellent performanceand scalability that has been widely used in recentyears. FC-SAN is based on the fibre channel transportprotocol, which allows SCSI commands and datatransmits on fibre channels. IP-SAN is usually basedon the iSCSI protocol[2,3] that describes a means of transporting the SCSI packets over TCP/IP, providingfor an interoperable solution which takes advantage ofthe existing Internet infrastructure, Internet manage-ment facilities, and address distance limitations.</p><p>The SCSI protocol[4] is used to transmit data betweeninitiators and targets. A traditional SCSI target is a SCSI device such as a SCSI disk or tape driver, buttraditional SCSI targets cannot be used in a SANenvironment.</p><p>Palekar et al. [5] proposed a SCSI target emulator forSAN environment. The target emulator receives SCSI commands and data sent by initiators from the storagenetwork, then processes and responds to them. The target emulator then passes the SCSI commands and data to the SCSI subsystem. However, thedisadvantage of Palekars emulator is that it can onlymake use of devices with a SCSI interface.</p><p>The IntraStor system[6] is a three-tier storage system.The lowest system tier is the IP disk controllers (IDCs) with the disks attached. The middle tier is the storagecontroller and manager (SCM). The upper tier is theapplication host. The application host provides accessto the SCM using the iSCSI protocol. The SCM givesaccess and manages the IDCs using a block orientedtransport protocol named xBlock. The disadvantage ofthe IntraStor system is that multiple protocol</p><p>Received: 2004-04-15; revised: 2005-01-04 </p><p>Supported by the National High-Tech Research and Development</p><p>(863) Program of China (No. 2001AA111110)</p><p>To whom correspondence should be addressed.</p><p>E-mail: shujw@tsinghua.; Tel: 86-10-62795215</p></li><li><p>PAN Jiaming () et alDesign and Implementation of SCSI Target Emulator 39</p><p>conversions may degrade the performance. Moreover,the IDCs must implement an independent protocolconversion for each storage device.</p><p>This paper introduces a target emulator with the goalof building a scalable, flexible, manageable, andcost-effective data storage system. The emulator wasdesigned for multiple transport protocols, especiallythe iSCSI protocol, and with different storage devices.The target emulator implements a protocol conversionmodule which is able to translate SCSI commandscarried by different SCSI transport protocols to avariety of storage devices. Many measures, such asRAM caches, multiple queues, and request merging,are used to optimize the reading and writingperformance.</p><p>The emulator improves on existing target emulatorsin several areas. First, the conversion module is able toconvert the SCSI request to a variety of devices, so anydisks can be used in the target. Secondly, the emulatoris able to accept virtual disks as storage devices, suchas MD RAID volumes[7] and LVM logical volumes[8].Therefore, the target system storage based on thisemulator could be configured as a system with a multi-RAID-level disk array and storage visualization.Finally, RAM caching, multiple queuing, and requestmerging are used to greatly improve the reading andwriting speeds. Therefore, the emulator performance is greatly improved.</p><p>1 SAN System</p><p>The emulator is implemented on a in-house SANsystem TH-MSNS[9,10], in which the target simulatormodule is a key component[11]. The target simulatormodule shown in Fig. 1 runs on the target side,processing and responding to SCSI requests.</p><p>Fig. 1 SAN I/O path with STML</p><p>The SCSI target simulator module was a low-levelfront-end protocol-specific target driver to transmit</p><p>SCSI commands and data. The driver hands off theSCSI commands and data to the SCSI target mid-leveldriver. The mid-level driver processes and executes theSCSI commands and hands the responses back to the low-level driver. The low-level driver is protocolspecific while the mid-level driver is accessible tomultiple low-level drivers implementing differentprotocols.</p><p>2 SCSI Target Emulator Design </p><p>The target emulator design is based on the targetsimulator module of TH-MSNS. The mid-level driveris unchanged; therefore, the SCSI requests for SCSI devices will be passed to the SCSI subsystem as beforewith a routine added to forward SCSI requests forgeneral storage devices. The mid-level driver in thetarget emulator is compatible with the various existinglow-level drivers implementing various transportprotocols.</p><p>The target emulator also is a SCSI protocolconversion module to process the SCSI requests forgeneral storage devices. The conversion module isresponsible for receiving SCSI requests for generalstorage devices from the mid-level driver andtranslating these requests to the devices. The actualreading from and writing to the storage devices areperformed by the I/O handler module. The accessingspeed to the general storage devices is enhanced by aRAM cache driver named dCache in the I/O handlermodule. In addition, measures such as multiplequeuing and request merging are used to optimize the reading and writing for the general storage devices.The target emulator architecture is shown in Fig. 2.</p><p>Fig. 2 SCSI target emulator architecture</p></li><li><p>Tsinghua Science and Technology, February 2006, 11(1): 38-4340</p><p>The target emulator includes the following three keymodules:</p><p>1) SCSI target mid-level module This modulereceives the SCSI commands and data from the low-level drivers and returns the responses (data and/or status) back to the driver. If the SCSI request targetis a SCSI device, the mid-level module passes therequest to the scsi_mod driver directly, whichprocesses the command and sends it onto thecorresponding SCSI target device. If the SCSI command target is a general storage device, themid-level module forwards the request to the protocolconversion module.</p><p>2) SCSI protocol conversion module Thismodule is the bridge between the mid-level moduleand the general storage devices. It translates therequests received from the mid-level module to avariety of general storage devices and sets up blockrequests for general storage devices. Finally, thismodule passes the block requests to the I/O handlermodule.</p><p>3) I/O handler module This module isresponsible for actually performing the I/O operations.The module also manages the storage devices, such as ATA disks and virtual disks. Each disk has anindependent request queue and an I/O operation thread,which results in multiple queues. The benefit ofmultiple queues is that the independent storage devicescan be read or written simultaneously, which improvesthe concurrent access performance. In addition, the I/O handler module will merge the requests in the requestqueues if possible to reduce the storage disk accessfrequency.</p><p>The dCache driver is a RAM cache driverimplemented in the I/O handler module which utilizesa RAM buffer to cache data. The RAM buffer accesstime is much faster than the magnetic disk access time.Therefore, the dCache considerally reduces the I/O response time which dramatically improves theperformance.</p><p>In addition, the I/O handler module provides a configuration interface which interacts withconfiguration tools so users can manage the storagedevices (e.g., register or un-register storage devices).</p><p>3 Implementation </p><p>The mid-level module in the target emulator is based</p><p>on the mid-level module of the TH-MSNS targetsimulation module, so the implementation of themid-level module will not be described here. Thissection only describes the implementation of theprotocol conversion module and the dCache driver, themultiple queuing, and the request merging.</p><p>3.1 SCSI protocol conversion module</p><p>When the mid-level module receives a SCSI commandfor one of the general storage devices from thelow-level driver, the mid-level module sets up a SCSI request which is transmitted to the conversion module.</p><p>Each storage device has a unique device ID in thetarget emulator. Each (target, logical unit number(LUN)) pair is mapped to a device ID in theconversion module. When the conversion modulereceives a SCSI request, the module first maps the (target, LUN) pair to the device ID. Then, theconversion module evaluates the SCSI request. If the request is a query-type request, it fills the SCSI request with the desired device information and returns therequest for a read or write request, the conversionmodule parses the LBA and the transfer length fromthe SCSI command descriptor block[12,13]. Then, theconversion module combines the LBA, the transferlength, and the data into a structure called a blockdevice request (BDR) which is sent to the I/O handlermodule. When the I/O handler module successfullycompletes reading or writing, the conversion modulefills the request and returns it to the mid-level module.</p><p>3.2 dCache driver</p><p>The dCache is a cache driver for general devices in theI/O handler module. The RAM buffer used by dCacheis allocated and initialized when the target emulator isset up. The buffer is organized as pages with page sizeof 1 KB.</p><p>A table named pageTable in the dCache driverdescribes the pages. Each entry in the pageTablerepresents a page and includes the following majorfields.</p><p>Status: indicate the page status as free, dirty or valid,meaning that the page is not in use, the page containsdata that has not been written to the disks, or the pagecontains data that has been written to the disks. A free page can be allocated freely, while a dirty page cannot</p></li><li><p>PAN Jiaming () et alDesign and Implementation of SCSI Target Emulator 41</p><p>be replaced, but a valid page can be replaced. If thereare no free pages, the dCache will replace a valid page.The replacement policy follows the LRU algorithm.</p><p>Counter: counter used by the LRU algorithm.(DeviceID, LBA): corresponding device ID and</p><p>page LBA.Address: page start address.The pages in the dCache are addressed by the</p><p>(device ID, LBA) pair. A hashtable named LBATablewith the key (device ID, LBA) pair is used to speed upthe search with the result being the page correspondingto the (device ID, LBA) pair.</p><p>All free pages are organized as a linked list calledfree_list. In the same way, dirty pages are linked with dirty_list and valid pages are linked with valid_list. Allcontinuous pages are linked together.</p><p>Upon receiving a write request, the dCache searchesthe LBATable with the (device ID, LBA) pair. If anentry is found, the corresponding page is overwrittenby the incoming data. Otherwise, the dCache gets a page from the free_list or the valid_list and copies theincoming data into the page. When the copy is complete, dCache updates the relative data structuresand informs the protocol conversion module.</p><p>For a read request, the dCache searches theLBATable. If the data is found in the RAM buffer, thedata is copied from the RAM buffer to the requesting buffer. Otherwise, the dCache forwards the BDR to theI/O handler module for reading and gets a page fromthe free_list or the valid_list. As the data is written intothe requesting buffer, it is also written into the page.</p><p>The operation of moving data from a higher-levelstorage device to a lower-level storage device isdefined as a destage operation. A destage thread wasimplemented to perform the destage tasks. The destagethread is initialized when the emulator sets up andcontinues to watch at the dCache. The destage threadremains asleep most of the time. The destage thread isactivated when the number of dirty pages exceeds thethreshold value or a storage device is not busy. Whenthe number of dirty pages exceeds the threshold value,the destage thread migrates the data from the dirty pages to the disks and marks the page state as valid.When the destage thread detects no I/O operations to astorage device, it searches the dirty pages for thisdevice and destages a page.</p><p>3.3 Multiple queuing and request merging </p><p>The concurrent access performance is improved bygiving each storage device its own I/O request queueand I/O operation thread. The I/O request queue is a FIFO. When the dCache needs to read or write disks,the dCache forwards the BDR to the correspondingqueue according to its device ID. The I/O operationthread sleeps until the I/O handler module detects thatthe queue is not empty.</p><p>The I/O handler module uses request merging topreprocess the requests. The I/O handler moduleanalyzes the data according to their LBAs and mergesrequests without data correlation. If the LBAs ofseveral write requests are the same, the I/O handlermodule discards the previous requests and only writesthe last request to the disks. If the LBAs of several write requests in the queue are continuous, theserequests are merged into one request so that the datawill be written together to the disks.</p><p>4 Performance Analysis</p><p>The SCSI target emulator was implemented on theTH-MSNS. The read/write throughput and the averageI/O response time of the emulator were then comparedwith Palekars target emulator.</p><p>The testing used only an initiator and a targetconnected by a gigabit ethernet. The initiator and targetconfigurations are described in Table 1.</p><p>Table 1 Initiator and target configurations</p><p>Initiator Target</p><p>CPU Intel(R) Xeon(TM) CPU 2.40 GHzu 2</p><p>Intel(R) Xeon(TM)</p><p>CPU 2.40 GHz </p><p>Memory 1 GB 1 GB OS Red Hat Linux 7.3 Red Hat Linux 7.3 kernel 2.4.18-3smp kernel 2.4.22-up</p><p>The SCSI target system first used a software RAID-0 volume based on the SCSI target emulator.The tests evaluated the throughput as well as theaverage response time for block sizes of 4 KB, 16 KB,256 KB,...</p></li></ul>