Apparatus, system, and method to increase data integrity in a redundant storage system

54 Pages·2014·4.25 MB·English

US008832528B2
(12) United States Patent (10) Patent N0.: US 8,832,528 B2
Thatcher et al. (45) Date of Patent: *Sep. 9, 2014
(54) APPARATUS, SYSTEM, AND METHOD TO (58) Field of Classi?cation Search
INCREASE DATA INTEGRITY IN A USPC ....... .. 714/766, 758, 770, 799, 805, 710, 723,
REDUNDANT STORAGE SYSTEM 714/761
See application ?le for complete search history.
(75) Inventors: Jonathan Thatcher, Liberty Lake, WA
(US); David Flynn, Sandy, UT (US); (56) References Cited
Joshua Aune, South Jordan, UT (US);
U.S. PATENT DOCUMENTS
Jeremy Fillingim, Salt Lake City, UT
(US); Bill Inskeep, West Valley City, UT
4,633,471 A * 12/1986 Pereraet a1. ................ .. 714/758
(US); John Strasser, Syracuse, UT 4,740,968 A 4/1988 Aichelmann, Jr.
(US); Kevin Vigor, Salt Lake City, UT
(Continued)
(Us)
FOREIGN PATENT DOCUMENTS
(73) Assignee: Fusion-i0, Inc., Salt Lake City, UT (US)
EP 569236 11/1993
(*) Notice: Subject to any disclaimer, the term of this 61-226853 8/1986
patent is extended or adjusted under 35
(Continued)
U.S.C. 154(b) by 215 days.
OTHER PUBLICATIONS
This patent is subject to a terminal dis
claimer. Of?ce Action in Chinese Application No. 2009801280134 issued
Apr. 1, 2013, 21 pages.
(21) Appl. N0.: 13/321,484
(Continued)
(22) PCT Filed: May 18, 2010
Primary Examiner * Fritz Alphonse
(74) Attorney, Agent, or Firm * Meyertons, Hood, Kivlin,
(86) PCT No.: PCT/US2010/035320
Kowert & Goetzel, PC.
§ 371 (0X1),
(2), (4) Date: Feb. 3, 2012 (57) ABSTRACT
In various embodiments, an apparatus, system, and method
(87) PCT Pub. No.: WO2010/135370 may increase data integrity in a redundant storage system. In
PCT Pub. Date: Nov. 25, 2010 one embodiment, a request is received for data stored at a
storage system having a plurality of storage elements, Where
(65) Prior Publication Data one or more of the plurality of storage elements include parity
information. A determination is made that one of the plurality
US 2012/0131413 A1 May 24,2012 of storage elements is unavailable, the unavailable storage
element being a functional storage element and including at
Related US. Application Data
least a portion of the data. Responsive to the determination,
(63) Continuation-in-part of application No. 12/468,041, the data is reconstructed based on at least a portion of the
?led on May 18, 2009, noW Pat. No. 8,281,227. parity information and data from one or more of the plurality
of storage elements other than the unavailable storage ele
(51) Int. Cl. ment; a response is provided to the request such that the
G11C 29/00 (2006.01) response includes the reconstructed data.
(52) US. Cl.
USPC .......................... .. 714/766; 714/758; 714/805 20 Claims, 22 Drawing Sheets
cuem
1.1.1
COMDMSI
.142
Sella'swne Storage Devlue
102
thdrétate Siner
Con?uller
1m
Wme Date
F) ' sounsme Storage
125
Reduction Demqu Rewnhgurahon
Apparatus Apparatus Apparalus
15 LE m
US 8,832,528 B2
Page 2
(56) References Cited 6,934,878 B2 8/2005 Massa et 31.
6,938,133 B2 8/2005 Johnson @131.
US PATENT DOCUMENTS 6,944,791 B2 9/2005 Humlicek et a1.
6,959,413 B2 10/2005 Humliceket al.
5,208 813 A 5/1993 Stallmo 6,973,604 B2 12/2005 Davis et 31.
52333618 A 8/1993 Glideret 31‘ 6,976,187 B2 12/2005 Arnottetal.
5,261,068 A 11/1993 Gaskins et al. 6,985,992 B1 1/2006 (311998; etal~
5,265,098 A 11/1993 Manson et al‘ 7,010,652 B2 3/2006 P1001r1110 et al.
5390187 A 2/1995 Stallmo 7,024,586 B2 4/2006 Kleiman et a1.
5’404’485 A 4/1995 Ban 7,032,142 B2 4/2006 Fujiokaetal.
5’438’671 A 8/1995 Miles 7,043,599 B1 5/2006 Ware et a1.
534792653 A 12/1995 Jones 7,050,337 B2 5/2006 Iwase et a1.
5,488,701 A 1/1996 Bradyet a1. 7,076,606 B2 7/2006 Orsley
5,490,149 A 2/1996 Nylander-Hill .............. .. 714/5.1 7,089,391 B2 8/2006 GelgeretaL
5,548,711 A 8/1996 Brantetal‘ 7,100,004 B2 8/2006 Johnsonetal.
5,553,231 A 9/1996 Papenberg et a1. 7,133,965 B2 11/2006 (3111911
5581567 A 12/1996 Chenetal‘ 7,146,461 B1 12/2006 Klselevetal.
5,581,690 A 12/1996 Ellis et a1. 7,181,572 B2 2/2007 Walmsley
5,596,708 A 1/1997 Weber 7,191,379 B2 3/2007 Adelmann etal.
5,617,530 A 4/1997 Stallmo et a1. 7,194,577 B2 3/2007 Johnson etal
5,642,494 A 6/1997 Wang et 31‘ 7,219,238 B2 5/2007 S211t0 et a1.
5,701,434 A 12/1997 Nakagawa 7,222,257 B1 5/2007 131%
5,729,497 A 3/1998 Pawlowski 7,224,604 B2 5/2007 Lasser
5,740,188 A 4/1998 Olarig 7,227,797 B2 6/2007 Thayeret a1.
5,754,567 A 5/1998 Norman 7,243,203 B2 7/2007 Scheuerlein
5,799,140 A 8/1998 Niijima et :11. 7,243,293 B2 7/2007 Chen
5,819,109 A 10/1998 Davis 7,308,532 B1 12/2007 Woodetal.
5,835,935 A 11/1998 Estakhri 613.1. 7,315,976 B2 1/2008 Holt
5,841,795 A 11/1998 Olarigetal‘ 7,334,156 B2 2/2008 Landetal.
5,845,329 A 12/1998 Onishi etal‘ 7,340,581 B2 3/2008 Gorobets et 31.
5,911,779 A 6/1999 Stallmo et a1. 7,350,101 B1 3/2008 Nguyen etal~
5,922,080 A 7/1999 Olarig 7,386,758 B2 6/2008 Kltamura
5,960,462 A 9/1999 Solomon et a1. 7,409,492 B2 8/2008 Tan'ilka etal~
5,974,502 A 10/1999 DeKoning et a1. 7,426,655 B2 9/2008 ShalketaL
5,978,953 A 11/1999 012mg 7,428,691 B2 9/2008 Ouchl
6,000,006 A 12/1999 Bruce etal‘ 7,451,266 B2 11/2008 Shinagawaet 31.
6,000,019 A 12/1999 Dykstal et :11. 7,480,766 B2 1/2009 Gorqbets
6,003,144 A 12/1999 Olarig et 31‘ 7,526,686 B2 4/2009 K_01v1c_k, Jr. et a1.
6,018,817 A 1/2000 Chen et a1. .................. .. 714/762 7529905 B2 5/2009 $111919“
6,024,486 A 2/2000 Olarig et 31‘ 7,574,554 B2 8/2009 Tanakaet a1.
6,044,483 A 3/2000 Chen etal, ,,,,,,,,,,,,,,,,,, ,, 714/762 7,631,138 B2 12/2009 Gonzalez etal~
6,070,249 A 5/2000 Lee 7,721,140 B2 5/2010 Dell et a1.
6,079,029 A 6/2000 Iwatani etal‘ 7,721,182 B2 5/2010 Joseph etal.
6158017 A 12/2000 Han et al‘ 7,797,503 B2 9/2010 Bellofatto et a1.
6’170’037 B1 1/2001 Blumenau 7,856,528 B1 12/2010 Frost et a1.
6,170,039 B1 1/2001 Kishida 7,865,809 B1 1/2011 Lee etal.
631703047 B1 1/2001 Dye 7,941,696 B2 5/2011 Frost et a1.
6,173,381 B1 1/2001 Dye 7,966,550 B2 6/2011 Mokhlesietal.
6,185,654 B1 2/2001 Van Doren 7,975,209 B2 7/2011 Chin et a1.
6,219,807 B1 4/2001 Ebiharaetal. 8,019,938 B2 9/2011 Flynn etal~
6,223,301 B1 4/2001 Santeleret a1. 8,019,940 B2 9/2011 Flynn etal
6233 717 B1 5/2001 Choi 2002/0002690 A1 1/2002 Lalet a1.
6’279’138 B1 @2001 Jadav et 31‘ 2002/0069317 A1 6/2002 Chow et a1.
6’282’671 B1 @2001 Islametal, 2002/0069318 A1 6/2002 Chowetal.
6’330’688 B1 12/2001 Brown 2002/0194530 A1 12/2002 Santeleretal.
6’356’986 B1 3/2002 Solomon et 31‘ 2003/0023924 A1 1/2003 Davis 613.1.
633703631 B1 4/2002 Dye 2003/0120863 A1 6/2003 Lee etal.
6385 710 B1 5/2002 Goldman et 31‘ 2003/0135808 A1 7/2003 Kakuta et 31.
634123080 B1 @2002 Fleming etal‘ 2003/0145270 A1 7/2003 Holt
6,418,478 B1 7/2002 Ignatius et al‘ 2003/0165076 A1 9/2003 Gorobets et 31.
6430 702 B1 8/2002 Santeler et a1, 2003/0182592 A1 9/2003 Massa et 31.
6:523:10; B1 200% Dye et 31‘ 2003/0198084 A1 10/2003 Fujisawa et al.
6557123 B1 4/2003 Wiencko Jr‘et a1, 2004/0015771 A1 1/2004 Lasser et a1.
6’587’915 B1 700% Kim ’ 2004/0037120 A1 2/2004 Uysal et a1.
6’601’211 B1 7/2003 Norman 2004/0163027 A1 8/2004 MacLaren et a1.
636253685 B1 9/2003 Cho et 31‘ 2004/0163028 A1 8/2004 Olarig
6,671,757 B1 12/2003 Multeretal. 2004/0168101 A1 8/2004 Kubo
6,754,774 B2 6/2004 Gruner et a1, 2005/0002263 A1 1/2005 Iwase et a1.
6,775,185 B2 8/2004 Fujisawa et a1, 2005/0015539 A1 1/2005 Horiiet a1.
6,779,088 B1 8/2004 Benveniste et al‘ 2005/0027951 A1 2/2005 Piccirillo et 31.
6,785,785 B2 8/2004 Piccirillo etal, 2005/0193166 A1 9/2005 Johnson etal.
6,850,443 B2 2/2005 Lofgren et a1. 2005/0246510 A1 11/2005 Retnamma 6131.
6,851,082 B1 2/2005 Corbett 2006/0004955 A1 1/2006 Ware et a1.
6,871,257 B2 3/2005 Conley et a1. 2006/0041718 A1 2/2006 Ulrich @131.
6,871,295 B2 3/2005 Ulrich e131. 2006/0129760 A1 6/2006 Subbarao
6,877,076 B1 4/2005 Cho et a1. 2006/0129899 A1 6/2006 Cochran et a1.
6,880,049 B2 4/2005 Gruner et a1. 2006/0143503 A1 6/2006 Shaik et a1.
6,883,079 B1 4/2005 Priborsky 2006/0190683 A1 8/2006 Takase
6,883,131 B2 4/2005 Acton 2006/0265636 A1 11/2006 Hummler
US 8,832,528 B2
Page 3
(56) References Cited Chen, Scott, “What Types of ECC Should Be Used on Flash
Memory?”, Internet Article, Nov. 27, 2007, pp. 6, XP002535760,
U.S. PATENT DOCUMENTS WWW.spansion.com/applicationinotes/TypesiofiECCiUsedi
oniFlashiANiO liepdf.
2006/0282747 A1 12/2006 Hummler Application No. PCT/U S2009/ 044414, International Preliminary
2007/0016699 A1 1/ 2007 Minami
Report on Patentability, Nov. 25, 2010.
2007/0103992 A1 5/2007 Sakui et al.
Application No. 2011-509798, Of?ce Action, Jun. 14, 2012.
2007/0180328 A1 8/2007 Cornwell et al.
2007/0198770 A1 8/ 2007 Horii et al. Application No. 2011-509798, Of?ce Action, Oct. 18, 2012.
2007/0233949 A1 10/2007 Hart Application No. 097477814, Of?ce Action, Jul. 5, 2012.
2007/0255889 A1 11/2007 Yogev et al. Application No. 097477814, Of?ce Action, Jan. 24, 2011.
2007/0276988 A1 11/2007 Luo et al. US. Appl. No. 12/467,914, Of?ce Action, Apr. 22, 2011.
2007/0283079 A1 12/2007 Iwamura et al. US. Appl. No. 12/467,914, Of?ce Action, Oct. 14, 2011.
2007/0297252 A1 12/ 2007 Singh US. Appl. No. 12/467,914, Notice ofAllowance, Feb. 7, 2012.
2008/0016392 A1 1/2008 Earl et al. US. Appl. No. 12/468,040, Of?ce Action, Dec. 15,2011.
2008/0022156 A1 1/ 2008 Sutardja
U.S. Appl. No. 12/468,040, Notice ofAllowance, Jul. 5, 2012.
2008/0034253 A1 2/ 2008 Hoya et al.
US. Appl. No. 13/645,353, Of?ce Action, Nov. 21, 2012.
2008/0034270 A1 2/ 2008 Onishi et al.
US. Appl. No. 12/468,041, Of?ce Action, Mar. 15,2012.
2008/0065827 A1 3/ 2008 Byrne et al.
2008/0091876 A1 4/2008 Fujibayashi et al. Application No. PCT/U S2010/ 035318, International Preliminary
2008/0168225 A1 7/ 2008 O’Connor Report on Patentability, Dec. 1, 2011.
2008/0201616 A1 8/ 200 8 Ashmore US. Appl. No. 12/468,041, Notice ofAllowance, Aug. 13, 2012.
2008/0250270 A1 10/2008 Bennett Application No. PCT/U S2010/ 035320, International Preliminary
2008/0256292 A1 10/2008 Flynn etal. Report on Patentability, Dec. 1, 2011.
2008/0266962 A1 10/2008 Jeon et al. Application No. PCT/US2010/035320, International Search Report
2009/0037652 A1 2/2009 Yu et al. and Written Opinion, Dec. 29, 2010.
2009/0125788 A1 5/2009 Wheeler et al. Application No. PCT/US2010/035318, International Search Report
2009/0164836 A1 6/2009 Carmichael
and Written Opinion, Dec. 24, 2010.
2009/0187785 A1 7/2009 Gonzalez et al.
“IBM Chipkill Memory”, IBM, 1999, pp. 6, Research Triangle Park,
2009/0282301 A1 11/2009 Flynn etal.
North Carolina, US.
“ASPMC-660”, ASINE, downloaded Nov. 18, 2009, pp. 3, http://
FOREIGN PATENT DOCUMENTS WWW.asinegroup.com/products/aspmc660.htrnl.
Bhargava, Bharat, “A Model for Adaptable Systems for Transaction
JP 06-067912 11/1994
Processing”, Decemb34 1989, pp. 17, IEEE Trransactions on Knowl
JP 2000259359 9/2000
edge and Data Engineering, vol. 1, No. 4.
JP 2001251146 9/2001
JP 2003233638 8/2003 “Method for Preventing Catastrophic Failures in RAID Volumes
JP 2006018373 1/2006 Using Dynamic Recon?guration”, Jun. 15, 2005, pp. 6,
JP 2012514274 6/2012 PriorArtDatabase IPCOM000125746D, WWW.ip.com.
WO 2006029243 3/2006 “NAND Flash 101: An Introduction to NAND Flash and How to
WO 2006050455 5/2006 Design It in to Your Next Product”, Micron Technical Note, 2006, pp.
WO 2009140700 11/2009 25, TN-29-19: NAND Flash 101 Introduction.
Alvarez, Guillermo A., “Tolerating Multiple Failures in RAID Archi
OTHER PUBLICATIONS
tectures With Optimal Storage and Uniform Declustering”, down
Application No. PCT/US2009/044414, International Search Report loaded Mar. 16, 2009, pp. 11, Department of Computer Science and
and Written Opinion, Jul. 21, 2009. Engineering, University of California, San Diego, La Jolla, Califor
“Error Correction Code in Single Level Cell NAND Flash Memories, nia, US.
Wacha, Rosie, “Improving RAID-Based Storage Systems With Flash
AN1823”, Internet Article, Nov. 17, 2004, pp. 16, XP002535761,
WWW.datasheetarchive.com. Memory”, First Annual ISSDIVUSRL Research Symposium, Oct.
“Bad Block Management in NAND Flash Memories, AN1819”, 20-21, 2009, pp. 21.
Internet Article, May 19, 2004, pp. 8, XP002535765, WWW.
datasheetarchive.com. * cited by examiner
US. Patent Sep. 9, 2014 Sheet 1 0f 22 US 8,832,528 B2
100
Client
M
Client Client
Computer
M M
Network
M
Computer
M
Solid-State Storage Device
M
Solid-State Storage
Controller
M
Write Data
Pipeline Solid-State Stora e
M CID 1% 9
Read Data
Pipeline
M
Reduction Detection Reconfiguration
Apparatus Apparatus Apparatus
M M @
FIG. 1
US. Patent
Sep 9 2014 Sheet 2 0f 22 US 8,832,528 B2
' a
200
_ g m A Storage Control Bus 212a
(D “'6 T) +
4, g E glo E BankO Bank 1___ Bank N
Application
m *g e 32 214a 214b 214m
Specific Processors
0 V0 1 _> SSS 0.0 SSS 0.1 _> SSS ON
E 0 i 216a 218a 220a
6 g 8 ¢ ¢ -- o
m“ a, +SSSi.O +SSSi.1 *SSSLN
5’ 216b 218b 220b
Controllers ¢ E5| 0 =0 - c = c
L 6 ‘ = I I I
Peer M “@2805 §+ SSSM.O ,SSSMA +SSSMN
5
m 5 N g g 5 % 21 :m 22;)m
¢ 0 w ..
Storage Controllers
z c Solid-State Storage 110a
o 51:) 4»
3
8t m I Storage Control Bus m
2 E 7‘; * Bank 0 Bank 1 Bank N
3| co U, 2 g 5 g 214a 214b 214n
SB@yustse m
N gogggggl ssso.o ssso.1 SSSON
RCoDntMroAl
N2l4e4 E M > B 3 S “g ‘ 216a 218a 220a
“5 O a c c e
m w
4, MiSC- Logic ‘I, + see 1.0 + sss 1.1 sss 1.N
? 5 216b 218b 220b
CPUs Master 9 't ' ¢ : c
*— +> Controller ° M sss M.1 sss M.N
% a g ’ Siamo * 218m ’ 220m
‘5 ‘L, DMA Cntrller 5 ¢ ¢ ¢
Con4tro—ll»ers
0 E
8 Solid-State Storage 110n
EMxetm.. 250
'3 Memory
3 1+ Controller 2 .
0 (Q\l a L Dynamic
@N Random
9 I ¢ Memory Array
9 Static Mem. @
t 6%? (DRAM)
Array T6
EMxetm.. 252
2 4+ Mgmt Mgmt Bus Solid-State
% _ Controller : : E Storage Device
w m (IZC) m
FIG. 2
US. Patent Sep. 9, 2014 Sheet 3 0f 22 US 8,832,528 B2
300 \
/\|Buffer Control @ Data Bus &
k A
J\ l\ _ l\
Output Buffer Cntl Q
330 ‘ 342 ‘ "
T — ) Cntl Q Input Buffer
? @
Read Program Cntl Q ‘
Module 4* 342 4 W .
_ rlte Program
@ O Cntl Q
<l' > - > Module
* 2 E 310
Alignment i’l —
c Cntl Q ‘5 7
'6 MOdu'e “ ? ‘ E, , Cntl Q Compression
5 ? 8 ? Module ?
0 8 * . 01 +
.6 u Decompresslon ‘ Cntl Q ‘ g , Gnu Q _ Encryption
g :5 Module ? _ :2 ? Module m
m E + w +
5 g Decryption ‘ Cntl Q ‘ % Garb. Collector
(—5 -I MOdu'e g E E Bypass 316 *
0 4
Q 0 +
_' Garb' COHGCtor O Cntl Q Packetizer
Bypass ? ’ 3 302
4 _ _
. i
DepaCket'zer a Gnu Q ( , Cntl Q ’ Media Encrypt.
% _ ? Module ?
. i
MGCIIG Decrypt. ‘ Cl'i? Q ( ECC Generator
Module E E 304
4 v T
ECC Correction Bank lnterleave Write Buffer
< > Module Controller 320
_ E E —
V * t t i
Read Sync. ' ' Write Sync
— — — — — — — Buffer — Sync' BUHer — Buffer - —
a a w
E ‘ $ ‘ W 't D t
“1 Read Data H e a a
E Pipeline Sgra?e|:3us Pipeline
O 108 on ro er 106
? — 3E —
8g Cmontr oller I v a
g V V
5 Storage "0 Bus m Storage Control Bus @
FIG. 3

US. Patent Sep. 9, 2014 Sheet 5 0f 22 US 8,832,528 B2
450
Storage Storage Storage Storage Storage
Element 1 Element 2 Element 3 Element 4 Element n
402a M 4020 402d M
ILEB PEB 1 PEB 2 PEB 3 PEB 4
:? 404a 4040 4040 404d - .
l.__-_______ . _ _ _ _ _ _ _ _.__________
|| LP Page 0 Page 0 Page 0 Page 0
|| 1% _ 406a _ _ __ 4_06_b 4000 r 4090_ _ _ _ _
"CEhigk D O D 0 ' P 0 II D 0 ciigk
:IL456 " 4(Ea __ _ 408b | m n 456
H D 1 P 1 D 1 D 1
|| 408a 408 4080 408d
ll : I I E
II P m D m D m D m
II 4089 408b 4080 408d
l|_________- __________-_-_________
: Page 1 Page 1 Page 1 Page 1
I 410a 410b 4100 410d
| D 0 D 0 D 0 D 0
| 408a 408b 4080 408d
' D 1 D 1 D 1 P 1
: 408a 408b 4080 4089
| I I I I
| D m D m P m D m
| 408a 408b 4089 408d
l
l
l
I Page m Page m Page m Page m
I 412a M 4120 M
l D o P 0 D 0 D 0
| m 4089 4080 408m
l
I P 1 D 1 D 1 D 1
I m M 4086 m
l E E E :
| D m D m D m D m
| 408a m m _0§_m4
l
I- _ _ _ _ _ _ _ . _ _ _ _ _ . _ _ _ _ _ . _ _ _ _ _ ._ _ _
US. Patent Sep. 9, 2014 Sheet 6 0f 22 US 8,832,528 B2
500
Reduction Apparatus
E
Receive Module Data Read Module
E E
Regeneration Module ECC Module
E m
FIG. 5
US. Patent
Sep. 9, 2014 Sheet 7 0f 22 US 8,832,528 B2
600
Reduction Apparatus
Data Read Module
Receive Module g
Q .
Read Rotation
Module
m
Regeneration Module ECC Module
% @
ECC Check Module Substitution Module
E w
Device Error
Periodic Read Module
Determination Module
m
@
Parity Rotation Module ECC Correction Module
m %
FIG. 6