Apparatus, system, and method for redundant write caching

48 Pages·2014·4.61 MB·English

USOO8706968B2
(12) Unlted States Patent (10) Patent N0.: US 8,706,968 B2
Flynn (45) Date of Patent: Apr. 22, 2014
(54) APPARATUS, SYSTEM, AND METHOD FOR 5,261,068 A 11/1993 Gaskins et a1.
REDUNDANT WRITE CACHING 5,291,496 A 3/1994 Andaleon et a1.
5,307,497 A 4/1994 Felgenbaum et a1.
. 5,313,475 A 5/1994 Cromer et a1.
(75) Inventor: DaVId Flynn, Sandy, UT (Us) 5,325,509 A 6/1994 Lautzenheiser
5,392,427 A 2/1995 Barrett et al.
(73) Assignee: Fusion-lo, Inc., Salt Lake City, UT (US) 5,404,485 A 4/1995 Ban
5,438,671 A 8/1995 Miles
( * ) Notice: Subject to any disclaimer, the term of this (Continued)
patent is extended or adjusted under 35
U-S-C- 154(b) by 928 days- FOREIGN PATENT DOCUMENTS
(21) App1.No.: 12/847,952 EP 1100001 5/2001
EP 1418502 5/2004
(22) Filed: Jul. 30, 2010 (Continued)
(65) Prior Publication Data OTHER PUBLICATIONS
US 2011/0022801A1 Jan, 27, 2011 “Hamming Codes for NAND Flash Memory Devices”, Micron
TN-29-08: Technical Note, 2005, pp. 7, Boise, Idaho, US.
Related US. Application Data (Continued)
(63) Continuation-in-part of application No. 11/952,123, _ _ _ _ _
?led on Dec. 6, 2007, now Pat. No. 8,019,938. 1;;2 ;”?! Emmi” 7 Plelrvfe'MIdIf/l Batting H n Kn
orney, gen , 0r zrm * enne . 0 on; 1 on
(60) Provisional application No. 61/240,573, ?led on Sep. McConkie
8, 2009.
51 I t Cl (57) ABSTRACT
( ) GnobF 2/06 (2006 01) An apparatus, system, and method are disclosed for redun
(52) U 8 Cl ' dant write caching. The apparatus, system, and method are
' ' ' _ _ _ _ provided with a plurality of modules including a write request
USPC 711/120’ 711/122’ 711/162’ 771111/13131121'01231’ module, a ?rst cache write module, a second cache write
_ _ _ ' module, and a trim module. The write request module detects
(58) Fleld 0f Clasm?catlon searCh a write request to store data on a storage device. The ?rst
USPC """" " 711/103’ 170’ E12'002’ 122’ 120’ 162’ cache write module writes data of the write request to a ?rst
_71_1/E12'024’ E12'021’ 714/131 1_'101’ E111 cache. The second cache write module writes the data to a
See apphcanon ?le for Complete SearCh hlswry' second cache. The trim module trims the data from one of the
(56) References Cited ?rst cache and the second cache in response to an indicator
that the storage device stores the data. The data remains
U.S. PATENT DOCUMENTS available in the other of the ?rst cache and the second cache to
service read requests.
4,571,674 A 2/1986 Hartung
4,980,861 A 12/1990 Hefdt et al~ 20 Clalms, 12 Drawrng Sheets
sak
Flrsl Cache Second Cache
192
Phys Log Clean/ Vslldl Phys Log 000111 Vslldl
Add. Add. Dirly Invalid g1; Add. Add [My Invalid 32";
5122 m m m E: m m m —
m0 2103 [M. y Villd 01000111 m0 4221 Clean Vali. d o01n1“0
m1 2902 Clean Valld 0m10u0 m1 0200 Clean lnvall. d c100m1
a 22 6 0290 Clean Vell. d m10m01 m2 5013 le. ly Vall. d 0m10“1
soa “ 4321 clean Invali. d 0m1m10 Ms 2103 Dll'ly Vall. d m10m11
m4 5013 M. y Valld 01100111 m4 1000 Clean Vall. d 11100100
Storage Device
m
Address Data Audra: Data Address Data
E m m m m M
1069 1100
£123 1010
501a 1101
me 0101
I 3902 0100
2153 szss 1001
5.1.212 54;: 0010
US 8,706,968 B2
Page 2
(56) References Cited 6,710,901 3/2004 Pastor
6,715,046 3/2004 Shoham et al.
US. PATENT DOCUMENTS 6,728,851 4/2004 Estakhri et al.
6,735,546 5/2004 Scheuerlein
5,469,555 11/1995 Ghosh et al. 6,745,292 6/2004 Stevens
5,499,354 3/1996 Aschoffet al. 6,751,155 6/2004 Gorobets
5,504,882 4/1996 Chai 6,754,774 6/2004 Gruner et al.
5,551,003 8/1996 Mattson et al. 6,754,800 6/2004 Wong et al.
5,559,988 9/1996 Durante et al. 6,757,800 6/2004 Estakhri et al.
5,586,291 12/1996 Lasker et al. 6,760,806 7/2004 Jeon
5,594,883 1/1997 Pricer 6,775,185 8/2004 Fujisawa et al.
5,596,736 1/1997 Kems 6,779,088 8/2004 Benveniste et al.
5,603,001 2/1997 Sukegawa et al. 6,785,776 8/2004 Arimilli et al.
5,680,579 10/1997 Young et al. 6,785,785 8/2004 Piccirillo et al.
5,701,434 12/1997 Nakagawa 6,801,979 10/2004 Estakhri et al.
5,734,861 3/1998 Cohn et al. 6,804,755 10/2004 Selkirk et al.
5,745,671 4/1998 Hodges 6,816,917 11/2004 Dicorpo et al.
5,745,792 4/1998 Jost 6,839,819 1/2005 Martin
5,754,563 5/1998 White 6,845,053 1/2005 Chevallier
5,787,486 7/1998 Chin et al. 6,845,428 1/2005 Kedem
5,797,022 8/1998 Shimotono et al. 6,871,257 3/2005 Conley et al.
5,809,527 9/1998 Cooper et al. 6,877,076 4/2005 Cho et al.
5,809,543 9/1998 Byers et al. 6,880,049 4/2005 Gruner et al.
5,822,759 10/1998 Treynor 6,883,068 4/2005 Tsirgotis et al.
5,835,935 11/1998 Estakhri et al. 6,883,069 4/2005 Yoshida
5,845,313 12/1998 Estakhri et al. 6,883,079 4/2005 Priborsky
5,845,329 12/1998 Onishi et al. 6,910,170 6/2005 Choi et al.
5,860,083 1/1999 Sukegawa 6,912,537 6/2005 Selkirk et al.
5,893,138 4/1999 Judd et al. 6,912,618 6/2005 Estakhri et al.
5,907,856 5/1999 Estakhri et al. 6,922,754 7/2005 Liu et al.
5,924,113 7/1999 Estakhri et al. 6,928,505 8/2005 Klingman
5,930,815 7/1999 Estakhri et al. 6,938,133 8/2005 Johnson et al.
5,960,462 9/1999 Solomon et al. 6,977,599 12/2005 Widmer
5,961,660 10/1999 Capps, Jr. et al. 6,978,342 12/2005 Estakhri et al.
6,000,019 12/1999 Dykstal et al. 6,981,070 12/2005 Luk et al.
6,061,511 5/2000 MarantZ et al. 6,996,676 2/2006 Megiddo
6,073,232 6/2000 Kroeker et al. 7,010,652 3/2006 Piccirillo et a1.
6,101,601 8/2000 Mathews et al. 7,010,663 3/2006 George et al.
6,105,076 8/2000 Beardsley et al. 7,013,376 3/2006 Hooper, III
6,128,695 10/2000 Estakhri et al. 7,013,379 3/2006 Testardi
6,141,249 10/2000 Estakhri et al. 7,043,599 5/2006 Ware et al.
6,145,051 11/2000 Estakhri et al. 7,047,366 5/2006 Ezra
6,170,039 1/2001 Kishida 7,050,337 5/2006 Iwase et al.
6,170,047 1/2001 Dye 7,057,936 6/2006 Yaegashi et al.
6,172,906 1/2001 Estakhri et al. 7,058,769 6/2006 Danilak
6,173,381 1/2001 Dye 7,069,380 6/2006 Ogawa et al.
6,185,654 2/2001 Van Doren 7,076,723 7/2006 Saliba
6,209,088 3/2001 Reneris 7,089,391 8/2006 Geiger et al.
6,223,308 4/2001 Estakhri et al. 7,096,321 8/2006 Modha
6,230,234 5/2001 Estakhriet al. 7,111,140 9/2006 Estakhri et al.
6,240,040 5/2001 Akaogi et al. 7,130,956 10/2006 Rao
6,278,633 8/2001 Wong et al. 7,130,957 10/2006 Rao
6,279,069 8/2001 Robinson et al. 7,143,228 11/2006 Lida et al.
6,289,413 9/2001 Rogers et al. 7,149,947 12/2006 MacLellan et al.
6,330,688 12/2001 Brown 7,162,571 1/2007 Kilian et al.
6,334,173 12/2001 Won et al. 7,165,144 * 1/2007 Choubal et al. ............. .. 711/118
6,356,986 3/2002 Solomon et al. 7,167,944 1/2007 Estakhri
6,370,631 4/2002 Dye 7,167,953 1/2007 Megiddo
6,385,710 5/2002 Goldman et al. 7,177,197 2/2007 Cernea
6,393,513 5/2002 Estakhri et al. 7,178,081 2/2007 Lee et al.
6,404,647 6/2002 Minne 7,181,572 2/2007 Walmsley
6,412,080 6/2002 Fleming et al. 7,194,577 3/2007 Johnson et al.
6,418,478 7/2002 Ignatius et al. 7,197,657 3/2007 Tobias
6,418,509 7/2002 Yanai et al. 7,203,815 4/2007 Haswell
6,424,872 7/2002 Glanzer et al. 7,219,197 5/2007 Hatakeyama
6,477,617 11/2002 Golding 7,219,238 5/2007 Saito et al.
6,515,928 2/2003 Sato et al. 7,227,777 6/2007 Roohparvar
6,516,380 2/2003 Kenchammana-Hoskote 7,234,082 6/2007 Lai et al.
6,519,185 2/2003 Harari et al. 7,237,141 6/2007 Fredin
6,523,102 2/2003 Dye et al. 7,243,203 7/2007 Scheuerlein
6,552,955 4/2003 Miki 7,246,179 7/2007 Camara
6,567,889 5/2003 DeKoning 7,248,691 7/2007 Pandit et al.
6,587,915 7/2003 Kim 7,257,129 8/2007 Lee et al.
6,601,211 7/2003 Norman 7,263,591 8/2007 Estakhri et al.
6,608,793 8/2003 Park et al. 7,280,536 10/2007 Testardi
6,625,685 9/2003 Cho et al. 7,293,196 * 11/2007 Hicken et al. ............... .. 711/120
6,633,956 10/2003 Mitani 7,305,520 12/2007 Voigt
6,671,757 12/2003 Multer et al. 7,337,201 2/2008 Yellin et al.
6,675,349 1/2004 Chen 7,340,558 3/2008 Lee et al.
US 8,706,968 B2
Page 3
(56) References Cited 2006/0064556 A1 3/2006 Aasheim et a1.
2006/0075057 A1 4/2006 Gildea etal.
U_g_ PATENT DQCUMENTS 2006/0090048 A1 4/2006 Okumoto etal.
2006/0106968 A1 5/2006 WooiTeoh
7,340,566 B2 3/2008 Voth 2006/0106990 A1 5/2006 Benhase et a1.
7,340,581 B2 3/2008 Gorobetsetal. 2006/0143396 A1 6/2006 Cabot
7,356,651 B2 4/2008 Liu etal. 2006/0152981 A1 7/2006 Ryu
7 360 015 B2 4/2008 Mathews et a1, 2006/0184736 A1 8/2006 Benhase et a1.
733602037 B2 4/2008 Higaki et 31‘ 2006/0224849 A1 10/2006 RezaulIslam etal.
7366 808 B2 4/2008 Kano et al‘ 2006/0230295 A1 10/2006 Schumacher et a1.
7:370:16; B2 5/2008 Yang et 31‘ 2006/0248387 A1 11/2006 Nicholson
7,424,593 B2 9/2008 Estakhriet 31‘ 2006/0265624 A1 11/2006 Moshayedi
7,437,510 B2 10/2008 Rosenbluth etal. 2007/0016699 A1 1/2007 Mlnarnl
7441081 B2 10/2008 Humlicek 2007/0050571 A1 23/2007 Nakamura et a1.
734412090 B2 10/2008 Estakhrietal, 2007/0118676 A1 5/2007 Kano etal.
7,447,847 B2 11/2008 Louie etal. 2007/0124474 A1 5/2007 Margulis
7450 420 B2 11/2008 Sinclair 2007/0124540 A1 5/2007 van Riel
7’460’432 B2 @2008 Warmr 2007/0150689 A1 6/2007 Pandit
7’463’521 B2 @2008 Li 2007/0162830 A1 7/2007 Steketal.
7’480’766 B2 1/2009 Gorobets 2007/0198770 A1 8/2007 Horiietal.
7’487’235 B2 2/2009 Andrews et 31‘ 2007/0204128 A1 8/2007 Lee etal.
734872320 B2 2/2009 Bansai 2007/0204197 A1 8/2007 Yokokawa
7,500,000 B2 3/2009 Groves et al. 2007/0230253 A1 10/2007 Klm
7523 249 B1 4/2009 Estakhri et 31‘ 2007/0233455 A1 10/2007 Zlmmer et a1.
7’529’905 B2 5/2009 Sinclair 2007/0233937 A1 10/2007 Coulson etal.
7’536’491 B2 5/2009 Kano et 31‘ 2007/0233938 A1 10/2007 ChO etal.
7’549’013 B2 6/2009 Estakhrietal, 2007/0234021 A1 10/2007 Ruberg etal.
7’552’271 B2 6/2009 Sinclair 2007/0245094 A1 10/2007 Lee etal.
735802287 B2 8/2009 Aritome 2007/0245217 A1 10/2007 Valle
7,631,138 B2 12/2009 Gonzalez etal. 2007/0250660 A1 10/2007 6111
7640 390 B2 12/2009 Iwamura et 31‘ 2007/0271468 A1 11/2007 McKenney et a1.
736442239 B2 1/2010 Ergan 2007/0271572 A1 11/2007 Gupta etal.
7,660,911 B2 2/2010 McDaniel 2007/0274150 A1 11/2007 Gorobets
7 689 803 B2 3/2010 Kan et 31‘ 2007/0276897 A1 11/2007 Tameshige et a1.
7’725’628 B1 5/2010 Phan 2007/0300008 A1 12/2007 Rogers
737253661 B2 5/2010 Liu et 31‘ 2008/0005748 A1 1/2008 Mathewetal.
7856 528 B1 12/2010 Frost et a1, 2008/0010395 A1 1/2008 Mylly et al.
738732782 B2 1/2011 Terry 2008/0016301 A1 1/2008 Chen
7,908,501 B2 3/2011 Kim et a1, 2008/0043769 A1 2/2008 Hirai
8,234,450 B2 7/2012 Glover et al. ............... .. 711/122 Zoos/0052483 A1 2/2008 Raqgmlan etal~
2002/0053009 A1 5/2002 Selkirk etal. Zoos/0059752 A1 3/2008 Senzawa
2002/0069317 A1 6/2002 Chowetal' 2008/0059820 A1 3/2008 Vaden et al.
2002/0069318 A1 6/2002 Chowetal‘ 2008/0080243 A1 4/2008 Edahiro et a1.
2002/0194451 A1 12/2002 Mukaidaet al‘ 2008/0091876 A1 4/2008 Fujibayashi et a1.
2003/0061296 A1 3/2003 Craddock etal. Zoos/0098159 A1 4/2008 3098 Girl
2003/0070034 A1 4/2003 Friedmann et al‘ 2008/0104344 A1 5/2008 Shlmozono et a1.
2003/0093741 A1 5/2003 Argon etal. Zoos/0117686 A1 5/2008 Yarnflda
2003/0131182 A1 7/2003 Kumaret 31‘ 2008/0120303 A1 5/2008 Selklrket al.
2003/0163630 A1 8/2003 Aasheim etal. Zoos/0120469 A1 5/2008 Komegay
2003/0163633 A1 8/2003 Aasheim Zoos/0123211 A1 5/2008 Chng etal
2003/0163663 A1 8/2003 Aasheim et a1, 2008/0126700 A1 5/2008 El-Bataletal.
2003/0198084 A1 10/2003 Fujisawa etal. Zoos/0126852 A1 5/2008 Brandyberry etal
2004/0059870 A1 3/2004 Ash 2008/0133963 A1 6/2008 Katano et al.
2004/0064647 A1 4/2004 Deriin etal. Zoos/0137658 A1 6/2008 Wang
2004/0093463 A1 5/2004 Shang 2008/0140819 A1 6/2008 Balleyet a1.
2004/0186946 A1 9/2004 Lee 2008/0183965 A1 7/2008 Shigaetal.
2004/0268359 A1 12/2004 Hanes 2008/0195801 A1 8/2008 Cheon etal.
2005/0002263 A1 1/2005 Iwase etal. Zoos/0201535 A1 8/2008 Ham
2005/0015539 A1 1/2005 Horii 2008/0205286 A1 8/2008 L1 etal.
2005/0018527 A1 1/2005 Gorobets Zoos/0209090 A1 8/2008 Kan? et 31;
2005/0027951 A1 2/2005 Piccirillo etal. Zoos/0229046 A1 9/2008 Ramborskl
2005/0055497 A1 3/2005 Estakhriet 31‘ 2008/0235443 A1 9/2008 Chowetal.
2005/0076107 A1 4/2005 Goudet 31‘ 2008/0266973 A1 10/2008 Sekaretal.
Zoos/0132259 A1 6/2005 Emmotet 31‘ 2008/0276040 A1 11/2008 Moritoki
2005/0144361 A1 6/2005 Gonzales etal. Zoos/0294847 A1 11/2008 Mawyamaetal
2005/0149618 A1 7/2005 Cheng Zoos/0313364 A1 12/2008 Flynn _
Zoos/0149819 A1 7/2005 Hwang 2009/0043952 A1 2/2009 Estakhrl etal.
2005/0177672 A1 8/2005 Rao 2009/0070541 A1 3/2009 Yochai
2005/0177687 A1 8/2005 R30 2009/0083478 A1 3/2009 Kunimatsu etal.
2005/0193166 A1 9/2005 Johnson etal. 2009/0083485 A1 3/2009 Cheng
2005/0229090 A1 10/2005 Shen etal. 2009/0089485 A1 4/2009 Yeh
2005/0235132 A1 10/2005 Karretal. 2009/0089518 A1 4/2009 Hobbet
2005/0240713 A1 10/2005 Wu 2009/0091996 A1 4/2009 Chen etal.
2005/0246510 A1 11/2005 Retnamma etal. 2009/0125650 A1 5/2009 Sebire
2005/0257213 A1 11/2005 Chu etal. 2009/0144496 A1 6/2009 Kawaguchi
2005/0276092 A1 12/2005 Hansen etal. 2009/0150605 A1 6/2009 Flynn etal.
2006/0004955 A1 1/2006 Ware etal. 2009/0150621 A1 6/2009 Lee
2006/0026339 A1 2/2006 Rostampour 2009/0157956 A1 6/2009 Kano
2006/0059326 A1 3/2006 Aasheim 2009/0204750 A1 8/2009 Estakhrietal.
US 8,706,968 B2
Page 4
(56) References Cited Megiddo, Nimrod, “ARC: A Self-Tuning, Low Overhead Replace
ment Cache”, USENIX Association, Mar. 31-Apr. 2, 2003, pp. 17,
U.S. PATENT DOCUMENTS Proceedings of FAST ’03 2nd USENIX Conference on File and
Storage Technologies.
2009/0228637 A1 9/2009 Moon et al. Whyte, Barry, “IBM SAN Volume Controller 4.2.1 Cache Partition
2009/0235017 A1 9/2009 Estakhri et a1. ing”, Redpaper, IBM Corporation, Apr. 30, 2003, pp. 12, REDP
2009/0276654 A1 1 1/ 2009 Butterworth et al. 4426-00, Poughkeepsie, New York, US.
2009/0300277 A1 12/2009 J eddeloh
Stokes, Jon, “Intel’s New Flash Tech to Bring Back Turbo Memory,
2009/0313453 A1 12/2009 Stefanus et al.
for Real”, http://arstechnica.com/hardware/news/2009/09/intels
2009/0327602 A1 12/2009 Moore et al.
2009/0327804 A1 12/ 2009 Moshayedi new-?ash-tech-to-b . . . .
2010/0017556 A1 1/2010 Chin et al. Rosenblum, Mendel, “The Design and Implementation of a Log
2010/0023674 A1 1/2010 Aviles structured File System”, 1992.
2010/0023676 A1 1/2010 Moon et al. Casey, Michael, “SAN Cache: SSD in the SAN”, Computer Tech
2010/0023682 A1 1/2010 Lee et a1. nology Review’s Storage Inc., Quarter 1, 2000, pp. 4, SolidData,
2010/0030946 A1 2/2010 Kano et al. Santa Clara, California, US.
2010/0077194 A1 3/ 2010 Zhao et al. “HyniX 48-GB Flash MCP”, Slashdot, pp. 17, http://hardware.
2010/0102999 A1 4/2010 Lee et a1. slashdot.org/ article.pl?sid:07/ 09/ 06/ 1462 1 8&from:rss.
2010/0106917 A1 4/ 2010 Ruberg et al. “Sybase: Maximizing Performance through Solid State File-Cach
ing”, SolidData Best Practices Guide, May 2000, pp. 4, Santa Clara,
FOREIGN PATENT DOCUMENTS California, US.
“Bulletproof Memory for RAID Servers, Part 1, 2, 3”, Agigatech,
EP 1814039 8/2007 2009, pp. 13, http://agigatech:com/blog/bulletproof-memory-for
GB 0123416 11/2002
raid-servers.
JP 04-242848 8/1992
Wu, Michael, “eNVy: A Non-Volatile, Main Memory Storage Sys
JP 08-153014 6/1996
tem”, 1994, pp. 12, ASPLOS VI-10/94 San Jose, California, US.
JP 2000259525 9/2000
IBM, “Method to Improve Reliability of SSD Arrays”, Prior Art
JP 2009122850 6/2009
KR 2010022811 3/2010 Database, Nov. 5, 2009, www.ip.com.
WO 9419746 9/1994 “Method for Fault Tolerance in Nonvolatile Storage”, Prior Art Data
WO 9518407 7/1995 base, Feb. 3, 2005, www.ip.com.
WO 9612225 4/1996 “ioDriveiFrequently Asked Questions”, Clustered Storage Solu
WO 0131512 5/2001 tions: Products, 2008, http://www.clusteredstorage.com/clusteredi
WO 0201365 1/2002 storageisolutionshtml.
WO 2008070173 6/2008 U.S. Appl. No. 11/952,123 Of?ce Action, May 5,2010.
WO 2008073421 6/2008 Windows PC Accelerators, XP-002476842, Microsoft Corporation,
WO 2010036757 4/2010
Nov. 30, 2006, pp. 1-16.
OTHER PUBLICATIONS Application No. PCT/US2007/025049, International Search Report,
May 14, 2008.
The Advantages of Object-Based StorageiSecure, Scalable, Application No. PCT/US2007/025049, International Preliminary
Dynamic Storage Devices, Seagate Research Technology Paper, Apr. Report on Patentability, Mar. 11, 2009.
“White Paper: S2A9550 Overview”, DataDirect Networks, www.
2005, pp. 7, Scotts Valley, California, US.
datadirectnet.com, 2007, 17 pages.
Wacha, Rosie, “Improving RAID-Based Storage Systems with Flash
“Pivot3 RAIGE Cluster: Technology Overview”, White Paper, www.
Memory”, First Annual ISSDIVUSRL Research Symposium, Oct. pivot3,com, Jun. 2007, pp. 1-17.
20-21, 2009, pp. 21. Plank, James S. “A Tutorial on Reed-Solomon coding for Fault Tol
Wang, Feng, “OBFS: A File System for Object-based Storage erance in RAID-like Systems”, Dept of Computer Science, Univ. of
Devices”, 21st IEEE/12th NASA Goddard Conference on Mass Stor Tennessee, Sep. 1997, pp. 995-1012.
age Systems and Technologies, Apr. 2004, pp. 18, College Park, Morgenstern, David, “Is There a Flash Memory RAID in Your
Maryland, US. Future?”, Nov. 2006.
US. Appl. No. 11/952,123, Notice ofAllowance, May 5, 2011. “File System Primer”, http://wiki.novell.com/indeX.php?FileiSys
“DataDirect Storaage Systems Selected for NCSA Cluster”, temiPrimer, Jul. 2006.
HPCWire, Aug. 15, 2003, pp. 2, vol. 12, No. 32, http://www.hpcwire. Singer, Dan, “Implementing MLC NAND Flash for Cost-Effective,
com/hpcwire/hpcwireWWW/03/0815/105731.html. High-Capacity Memory”, M-systems, White Paper, Sep. 2003.
“Introducing Box Hill’s Fibre Box”, Box Hill Systems Corporation, “Introduction to Samsungs Linux Flash File System”iRFS,
Samsung Electronics, Nov. 2006.
Jan. 16, 1997, p. 5.
Mesnier, Ganger, Riedel, “Object-Based Storage”, IEEE Comm
“jZip A Free WinZip Alternative”, First JZip by Greg Kowal, 2012.
Magazine, Aug. 2003.
Application No. PCT/US2012/022244, International Search Report
Hensbergen, “Dynamic Policy Disk Caching for Storage Network
and Written Opinion, Jul. 30, 2012.
ing”, Nov. 2006.
Yerrick, Damian, “Block Device”, downloaded Mar. 1, 2010, pp. 3,
Application No. 2007800509734, Of?ce Action, Jan. 26, 2011.
http;//www.pineight.com/ds/block/.
Application No. 078676616, Of?ce Action, Oct. 6, 2011.
Weber, Ralph 0., “Information TechnologyiSCSI Object-Based
“Design and Use Considerations for NAND Flash Memory”, Micron
Storage Device Commands (OSD)”, Seagate Technology, Jul. 30,
TN-29-17:NAND Flash Memory Introduction, 2006, pp. 8, Boise,
2004, p. 171, Project T10/1355-D, Revision 10, Reference No. ISO/ Idaho, US.
IEC 14776-391 : 200x ANSI INCITS.***200X. “Am29DL322D/323D/324D”, Spansion Data Sheet, Oct. 7,2004, 58
Makulowich, John, “Strategies, Players and Emerging Markets”, pages, Pub. No. 21534, Revision D, Amendment 7, Fujitsu.
Washington Technology, Jun. 26, 1997, pp. 6, http:// Ismail, Ari, “Performance Boosting and Workload Isolation in Stor
washingtontechnolo gy.com/ Articles/ 1 997/ 06/ 26/ Strategies-Play age Area Networks with SANCache”, Proceedings of the 23rd IEEE/
14th NASA Goddard Conference on Mass Storage Systems and
ers-and-Emerging-Ma . . . .
U.S. Appl. No. 12/885,285, Notice ofAllowance, Oct. 17, 2012. Technologies, May 2006, pp. 263-273, Hewlett Packard Laborato
Application No. PCT/US2011/046005, International Search Report ries.
and Written Opinion, Apr. 6, 2012. Bandulet, Christian, “Object-Based Storage Devices”, Sun Devel
Application No. PCT/US2010/048321, International Search Report oper Network Site, Jul. 2007, http://developers.sun.com/solaris/ar
and Written Opinion, Apr. 28, 2011. ticles/osd.html.
US 8,706,968 B2
Page 5
(56) References Cited Johnson, Michael K., “An Introduction to Block Device Drivers”,
Jan. 1, 1995, pp. 6.
OTHER PUBLICATIONS Leventhal, Adam, “Flash Storage Memory”, Communications of the
ACM, Jul. 2008, pp. 47-51, vol. 61, No. 7.
“BiTMICRO Introduces E-Disk PMC Flash Disk Module”, Military Volos, Haris, “Mnemosyne: Lightweight Persistent Memory”,
& Aerospace Electronics East, 2004, http://www.bitmicro.com/
ASPLOS’ll, Mar. 5-11, 2011, Newport Beach, California, US.
pressinewsireleasesi200405 18iprt.php.
“NAND Flash 101: An Introduction to NAND Flash and How to
“EMC Virtual Infrastructure for Microsoft ApplicationsiData Cen
ter Solution Enabled by EMC SymmetriXV-Max andVMware ESX3. Design it in to Your Next Product”, Micron, Nov. 2006, pp. 27, Boise,
5”, White Paper, Apr. 2009, pp. 30. Idaho, US.
Feresten, Paul, “NETAPP Thin Provisioning: Better for Business”, Coburn, Joel, “NV-Heaps: Making Persistent Objects Fast and Safe
NETAPP White Paper, Mar. 2007, pp. 11, WP-7017-0307. with Next-Generation, Non-Volatile Memories”, ASPLOS’ 1 1, Mar.
“How NTFS Works”, Mar. 28, 2003, pp. 34. 5-11, 2011, pp. 13, Newport Beach, California, US.
“Information TechnologyiSCSI Obj ect-Based Storage Device Brandon, Daniel, “Sparse Matrices in CS Education”, Consortium
Commands (OSD)”, American National Standards Institute, Jul. 30, for Computing Sciences in Colleges, 2009, pp. 6.
2004, pp. 187, Project T10/1355-D, Revision 10, New York, New Application No. PCT/US2011/047659, International Search Report
York, US. and Written Opinion, Apr. 9, 2012.
“Intel Turbo Memory with User Pinning”, Intel NAND Storage Solu
tions, 2008, pp. 4. * cited by examiner
US. Patent
Apr. 22, 2014 Sheet 1 0f 12 US 8,706,968 B2
100
Host Device
11—4
Direct Cache Module
m
A
First Cache Second Cache
Solid-State Solid-State
Storage Storage
Controller Controller
m m
Write Data Solid-State Write Data Solid-State
Pipeline Storage Pipeline Storage
@ m M m
Read Data Read Data
Pipeline Pipeline
@ @
Storage Device
m
FIG. 1

US. Patent
Apr. 22, 2014 Sheet 3 0f 12 US 8,706,968 BZ
300 \
/\|Buffer Control E Data Bus A
t
J‘ ’\ _ I\
_‘" H 7
Output Buffer l Cntl Q
< 4 V
T — Cntl Q _ Input Buffer
Read Program Cntl Q ? %
Module 0 ? ‘ Wri-t e Program
w 0 Cntl Q
<- - > Module
1 2 ? 310
Alignment ,3} T
M dule 0 Q Q) .
0 ? 2; ’ Cntl Q _ Compressmn
@ 3 ? Module ?
8 + I D! V
N Decompressron “ Cntl Q g Cn? Q Encryption
LBCDoolumcoaacsilkn
g Module @ E g 342 Module 314
Q 4 w — + —
g Decryption 0 Cntl Q 2 Garb. Collector
_| Module ? ? % Bypass 316
0 V
Garb. Collector 0 _____________ __ Q Gnu Q Packetizer
Bypass m 342 ' 302
+ _ _
I V
Depacketizer ‘ Cntl Q Gnu Q _ Media Encrypt
2+4 ? ? Module m
I V
MGdla Decrypt 0 on" Q ‘ ECC Generator
Module Q E T Q
ECC Correction Bank lnterleave Write+Buffer
< ; Modu|e Controller 320
— a E —
4 t 1 l
Read Sync. ' ' Write Sync
_ _ Buffer __ _ Sync. Buffer _ _ Buffer _ __ _ _
a w E
‘ $ ‘ W 't D t
Read Data r' e a a
Pipeline Sg’é??iriis Pipe'ine
108 106
SCDotlmooarciakng e — E —
—t +
SSS Controller MUX
m .7 m
4 V
Storage "0 Bus A Storage Control Bus &
FIG. 3
US. Patent Apr. 22, 2014 Sheet 4 0f 12 US 8,706,968 B2
400 \
Control & Status R
Write Erase
Agent Agent
? %
.5
cu
E
o
D
X
0
2
O
(I)
:5
In
To
0
o
Bank lnterleave
S c. Buffer
E
R ' M | 4 Status NAND Bus cu
emappmg Odue? Capture Controller E
o
D
ress & Command l/ g Q
X
0
Storage 2
Bus
Controller
2
Mux3 O m
U)
Storage "0 Bus m Storage Control Bus m
FIG. 4
US. Patent
Apr. 22, 2014 Sheet 5 0f 12 US 8,706,968 B2
Host Device
11—4
User Application
@
Storage Client
Direct Cache Module
m
Block l/O Emulation Direct Interface
@ @
First Cache Second Cache
E m
Logical-to-Physical Logical-to-Physical
Translation m Translation m
Storage Device
m
FIG. 5