Apparatus, system, and method to increase data integrity in a redundant storage system

55 Pages·2015·4.14 MB·English

USOO8281227B2
(12) Unlted States Patent (10) Patent No.: US 8,281,227 B2
Thatcher et al. (45) Date of Patent: Oct. 2, 2012
(54) APPARATUS, SYSTEM, AND METHOD TO 5,390,187 A 2/ 1995 Stallmo
INCREASE DATA INTEGRITY INA 2 EELS
REDUNDANT STORAGE SYSTEM 534793653 A 12/1995 Jones
5,488,701 A 1/1996 Brad et al.
(75) Inventors: Jonathan Thatcher, Lehi, UT (US); 5,553,231 A 9/1996 Papeiiberg et a1,
David Flynn, Sandy, UT (US); Joshua 5,581,567 A 12/1996 Chen et al.
Anne, South Jordan, UT (Us); Jeremy 5,581,690 A * 12/1996 Ellis et a1. .................. .. 714/622
Fillingim, Salt Lake City, UT (US); Bill (Continued)
Inskeep, West Valley City, UT (US);
John Strasser, Syracuse, UT (Us); FOREIGN PATENT DOCUMENTS
Kevin Vigor, Salt Lake City, UT (US) EP 0569236 A2 11/1993
Continued
(73) Assignee: Fusion-10, Inc., Salt Lake City, UT (US) ( )
OTHER PUBLICATIONS
( * ) Notice: Subject to any disclaimer, the term of this
patent is extended or adjusted under 35 PCT/US2009/044414 International Preliminary Report on Patent
U.S.C. 154(b) by 743 days. ability, NOV 25, 2010
(21) Appl. No.: 12/468,041 (Commued)
(22) Filed_ May 18 2009 Primary Examiner * Shelly A Chase
' ’ (74) Attorney, Agent, or Firm * KunZler LaW Group, PC
(65) Prior Publication Data
(57) ABSTRACT
Us 2010/0293440 A1 NOV' 18’ 2010 An apparatus, system, and method are disclosed to increase
(51) Int Cl data integrity in a redundant storage system. The receive
G11C29/00 (2006 01) module receives a read request to read data from a logical
(52) U 5 Cl ' 714/770 page spanning an array of N+P number of storage elements.
(58) Field of """""" 63i7 6 4 The array of storage elements includes N number of the
H69 770 773’ storage elements each storing a portion of an ECC chunk and
S 1. . ?l f 1 h h. T ’ P number of the storage elements storing parity data. The data
ee app Icanon e or comp ete seam may read module reads data from at least a portion of a physical
. page on each of X number of storage elements of the N+P
(56) References Clted number of storage elements Where X equals N. The regenera
U S PATENT DOCUMENTS tion module regenerates missing data. The ECC module
' i _ determines if the read data and any regenerated missing data
2 ‘SAtlgllllginann’ Jr' includes an error. The read data combined With any regener
532333618 A 8/1993 Glider et 31‘ ated missing data includes the ECC chunk.
5,261,068 A 11/1993 Gaskins et al.
5,265,098 A 11/1993 Mattson et al. 21 Claims, 22 Drawing Sheets
BOO
Receive Request lo Read
Data
Determine X Number of
Storage Elements to Read
Read Data from X Number
of Storage Elements
31 0
Parity Data? i l Regenerate Missing Data
Data Errors?
No
Yes
814 Correctable Using Correct Error Using ECC
ECC.7 Y
No
818 Read StoraRgee aEdl ements not
Determine Storage Element
820 Causing Errors
Substitute Data into ECC
Chunk
US 8,281,227 B2
Page 2
US. PATENT DOCUMENTS 7,243,203 B2 7/2007 Scheuerlein
5,596,708 A 1/1997 Weber 7’243’293 B2 7/2007 Chen
7,308,532 B1 12/2007 Wood et a1.
5,701,434 A 12/1997 Nakagawa
. 7,315,976 B2 1/2008 Holt
5,729,497 A 3/1998 Pawlowskl
. 7,334,156 B2 2/2008 Land et a1.
5,740,188 A 4/1998 Olang
7,340,581 B2 3/2008 Gorobets et 31.
5,754,567 A 5/1998 Norman
7,350,101 B1 3/2008 Nguyen et a1.
5,799,140 A 8/1998 N1111maetal. -
. 7,386,758 B2 6/2008 Knamnra
578197109 A 10/1998 DAVIS 7409 492 B2 8/2008 Tanaka et a1
5,835,935 A 11/1998 Estakhrietal. ’ ’ - '
5 841795 A 11/1998 01 . 7,426,655 B2 9/2008 Sha1k et a1.
7 7 Aug. 7,451,266 B2 11/2008 Shinagawaetal.
578457329 A 12/1998 Onlsh‘ 6‘ ’11' 7526 686 B2 4/2009 Kolvick etal
5,922,080 A 7/1999 Olarig 7,’ 529,’ 905 B2 5/2009 S1- ncla1- r '
5,960,462 A 9/1999 Solomon et a1. 7 574 554 B2 8/2009 Tanaka et a1
5,974,502 A 10/1999 DeKoning etal. .......... .. 711/114 7’721’140 B2 5,2010 Denetal '
5,978,953 A 11/1999 Olarig 7,’ 721,’ 182 B2 5/2010 Joseph eta' l.
670007006 A 12/1999 Bruce et 31' 7856 528 B1 12/2010 Frost et a1
6,000,019 A 12/1999 Dykstal et a1. 7 7 '
- 7,941,696 B2 5/2011 Frost et a1.
6,003,144 A 12/1999 Olang 7966 550 B2 60011 M khl . 1
6,024,486 A 2/2000 Olarig 7 7 °_ “let”
6,070,249 A 5/2000 Lee 7,975,209 B2 7/2011 Ch1netal.
6,079,029 A 6/2000 Iwatani et a1. 8,019,938 B2 9/2011 Flynn etal~
6,158,017 A 12/2000 Han etal. 8,019,940 B2 9/2011 Flynn etal~
6,170,037 B1 1/2001 Blumenau 2002/0002690 A1 1/2002 Laiet a1.
6,170,039 B1 1/2001 Kishida 2002/0069317 A1 6/2002 Chow et a1.
6,170,047 B1 1/2001 Dye 2002/0069318 A1 6/2002 Chow et a1.
6,173,381 B1 1/2001 Dye 2002/0194530 A1 12/2002 Santeleretal.
6,185,654 B1 2/2001 Van Doren 2003/0023924 A1 1/2003 Davis etal.
6,219,807 B1 4/2001 EblharaetaL 2003/0120863 A1 6/2003 Lee etal.
6,223,301 B1 4/2001 SanyeleretaL 2003/0135808 A1 7/2003 Kakutaetal.
5% $88} $3131 t l 2003/0145270 A1 7/2003 Holt
7 7 a V6 a~ 2003/0165076 A1 9/2003 Gorobetsetal.
275232;; 5% $388} Elmetal 2003/0182592 A1 9/2003 Massaetal.
7 7 TOW“ 2003/0198084 A1 10/2003 Fujisawa et a1.
6,356,986 B1 3/2002 Solomon et a1.
2004/0015771 A1 1/2004 Lasseretal.
6,370,631 B1 4/2002 Dye
6385 710 B1 5,2002 Goldman at 31 2004/0037120 A1 2/2004 Lysaletal.
6,418,478 B1 7/2002 Ignatius et a1. 2004/0163028 A1 57/2004 Olarlg
6,430,702 B1 8/2002 Santeleret a1. iggg/gégggé :1 535882 $1130 1
6,523,102 B1 2/2003 Dye etal. wase eta,
6,557,123 B1 4/2003 Wiencko, Jr. et a1. 2005/0015539 A1 1/2005 Horii et a1.
6,587,915 B1 7/2003 Kim 2005/0027951 A1 2/2005 Piccirillo etal.
6,601,211 B1 7/2003 Norman 2005/0193166 A1 9/2005 Johnson etal.
6,625,685 B1 9/2003 Cho et a1. 2005/0246510 A1 11/2005 Retnamma etal.
6,671,757 B1 12/2003 Multeretal. 2006/0004955 A1 1/2006 Ware
6,754,774 B2 6/2004 GrgneretaL 2006/0041718 A1 2/2006 Ulrich etal.
6,775,185 B2 57/2004 1310181th al~ 2006/0129760 A1 6/2006 Subbarao
677797088 B1 8/2004 BFHYénlste eta1~ 2006/0129899 A1 6/2006 Coohran et a1.
677857785 B2 8/2004 Plccmllo etal 2006/0143503 A1 6/2006 Shaik et a1.
6,850,443 B2 2/2005 Lofgren et a1.
2006/0190683 A1 8/2006 Takase
6,851,082 B1 2/2005 Corbett
6871295 B2 3,2005 Ulrich at 31 2006/0265636 A1 11/2006 Hummler
678777076 B1 4,2005 Cho etal ' 2006/0282747 A1 12/2006 Hummler
6,’ 880,’ 049 B2 4/2005 G 1' 1. 2007/0016699 A1 1/2007 Mina_m i
6 883 079 B1 400% Prrilllngya 2007/0103992 A1 5/2007 Saku1 et a1.
638833131 B2 4/2005 Acton 2007/0180328 A1 8/2007 Cornwelletal.
6,934,878 B2 8/2005 Massa etal. 5883/8522347‘3 :1 13/588; goriiet?
6,938,133 B2 8/2005 Johnson et a1. art
6,944,791 B2 9/2005 Humlicek et a1. ......... .. 714/622 2007/0255889 A1 11/2007 Yogev et a1.
6,959,413 B2 10/2005 Humlicek et a1. 2007/0276988 A1 11/2007 Luo et a1.
6,976,187 B2 12/2005 Arnottetal. 2007/0283079 A1 12/2007 Iwamura et a1.
6,985,992 B1 1/2006 Chang etal. 2007/0297252 A1 12/2007 Singh
7,010,652 B2 3/2006 Pi99iri119 etal 2008/0016392 A1 1/2008 Earl et a1.
7,024,586 B2 4/2006 K191man etal~ 2008/0022156 A1 1/2008 Sutardja
7,032,142 B2 4/2006 1311119191 etal~ 2008/0034253 A1 2/2008 Hoya et a1.
770437599 B1 5/2006 Ware “*4 2008/0034270 A1 2/2008 Onishi et a1.
770507337 B2 500% Ml“ “AL 2008/0065827 A1 3/2008 Byrne et a1.
7,076,606 B2 7/2006 Ors. ley 2008/0091876 A1 4/2008 Fuj.l. bayash1. et a1.
7,089,391 B2 8/2006 Gelgeretal. ,
7100004 B2 800% J 2008/0168225 A1 7/2008 0 Connor
, , ohnson et a1.
7,133,965 B2 11,2006 Chien 2008/0201616 A1 8/2008 Ashmore
7,146,461 B1 12,2006 Kiselev etal‘ 2008/0250270 A1 10/2008 Bennett
7,181,572 B2 2/2007 Walmsley 2008/0256292 A1 10/2008 Flynn 6t {11.
7,191,379 B2 3/2007 Adelmann et al‘ 2008/0266962 A1 10/2008 Jeon et 31.
7,194,577 B2 3/2007 Johnson et 31, 2009/0037652 A1 2/2009 Yu et a1.
7,219,238 B2 5/2007 Saito et a1, 2009/0125788 A1 5/2009 Wheeler et a1.
7,222,257 B1 5/2007 Dibb 2009/0164836 A1 6/2009 Carmichael
7,224,604 B2 5/2007 Lasser 2009/0187785 A1 7/2009 Gonzalez et a1.
7,227,797 B2 6/2007 Thayer et a1. 2009/0282301 A1 11/2009 Flynn et a1.
US 8,281,227 B2
Page 3
FOREIGN PATENT DOCUMENTS Chen, What Types of ECC Should Be Used on Flash Memory, Span
sion Application Note, Nov. 27, 2007, downloaded from www.span
JP 2000259359 A 9/2000 sion.com/applicationinotes/TypesiofiECCiUsedioniFlashi
WO 2006029243 A1 3/2006
WO 2006050455 5/2006 ANi0 liepdf on Jul. 2, 2009.
IBM Chipkill Memory, Enhancing IBM Net?nity Server Reliability,
OTHER PUBLICATIONS Feb. 1999, pp. 1-6.
Alvarez, Tolerating Multiple Failures in RAID Architectures with
PCT/U S20 1 0/ 03 5318 International Search Report and Written Opin
Optimal Storage and Uniform Declustering, Department of Com
ion, Dec. 24, 2010. puter Science and Engineering, University of California, San Diego,
PCT/U S20 1 0/ 03 5320 International Search Report and Written Opin 1997, pp. 62-72.
ion, Dec. 29, 2010. Bhargava, A Model for Adaptable Systems for Transaction Process
ASPMCi660, Asine Group, http://www.asinegroup.com/products/ ing, IEEE Transactions on Knowledge and Data Engineering, Dec.
aspmc660.html, copyright 2002, downloaded on Nov. 18, 2009. 1989, pp. 433-449, vol. 1, No. 4, Downloaded on Mar. 16, 2009.
BiTMICRO Introduces E-Disk PMC Flash Disk Module at Military Method for Preventing Catastrophic Failures in RAID Volumes
& Aerospace Electronics East, May 18, 2004, BiTMICRO, http:// Using Dynamic Recon?guration, IP.com Prior Art Database Techni
www.bitmicro .com/pressinewsireleasesi200405 18iprt.php.
cal Disclosure, Jun. 15, 2005, IPCOM000125746D.
NAND Flash 101: An Introduction to NAND Flash and How to Examination Report, Application No. 097477814, May 7, 2012.
Design It in to Your Next Product, Micron, pp. 1-28, Micron Tech Of?ce Action from USPTO, U.S. Appl. No. 12/468,040, Dec. 15,
nology Nov. 2006. 201 1.
PCT/U S2009/ 044414, International Search Report and Written PCT/US2010/035318, IPRP, Dec. 1, 2011.
Opinion, Jul. 21, 2009. PCT/US2010/035320, IPRP, Dec. 1,2011.
Bad Block Management in NAND Flash Memories, AN1819 Appli PCT/US2010/035320, International Search Report and Written
cation Note, May 19, 2004, downloaded from w. Opinion, Dec. 29, 2010.
datasheetarchive.com on Jul. 2, 2009. PCT/US2010/035318, International Search Report and Written
Error Correction Code in Single Level Cell NAND Flash Memories, Opinion, Dec. 24, 2010.
AN1823 Application Note, Nov. 17, 2004, downloaded from w.
datasheetarchive.com on Jul. 2, 2007. * cited by examiner
US. Patent
Oct. 2, 2012 Sheet 1 0f 22 US 8,281,227 B2
100
Client
M
Client Client
Computer
m m
Network
M
Computer
Solid-State Storage Device
m
Solid-State Storage
Controller
M
Write Data
Pipeline
Solid-State Storage
E
E
Read Data
Pipeline
@
Reduction Detection Reconfiguration
Apparatus Apparatus Apparatus
M @ E
FIG. 1
US. Patent
Oct. 2, 2012 Sheet 2 0f 22 US 8,281,227 B2
N O C)
/
_ Q m A Storage Control Bus 212a
w 9 w 5 FE 4‘ -
4, U, 5, all e C Bank 0 Bank 1___ Bank N
ASpppelicciaftiiocn ‘0 ‘g \— g‘c“ % m m
Processors
0 v0 ‘ r + see 0.0 _> sss 0.1 * sss ON
258 0 A i 216a 218a 220a
F, 53' A 8| 1 1 1
4-) ii: N : ‘—
5‘“ ‘3', +sss1.0 sssm ,sssm
5 5 216b 218b 220b
Controllers ¢ 5 0 :¢ : Q : ¢
L N = I I I
Peer E ngl g a, SSSMO SSSM1 SSSMN
5 4-) 3 bc O(\l ‘ rug Om1 -> - -> . .> .
m 0 ‘1 § § 21gm 212m 22;)m
¢ 13 en
Storage Controllers
2 C Solid-State Storage 110a
4-) (T) “I 45+
:1: N I
3 (\l I
8| “0 T Storage Control Bus 212n
2 E a; * Bank 0 Bank 1 Bank N
00 co 2 c x “'6 214a 214b 214m
SBmyustse m o1 _ t a 6 <- % U _ _ _
RCoDntMroAl N c“ w ,5 O c 52 4 SSS 0.0 SSS 0.1 SSS O.N
% M =% 8 Srco‘ ’ 216a " 218a ' 220a
m
0 Misc- Logic ‘2 sss 1.0 sss 1.1 sss 1.N
E 5 216b 218b 220b
Master g 5‘ E G 5‘
CPUS
m
L 0 Controller ° sss M.O sss M.’| sss M.N
% @ é” ’ m ’ m ’ m
E DMA Cntrller 5 ¢ ¢ ¢
0 0 a
Con4tr—ollV)er s 8 Solid-State Storage m
EMxetm.. E
'5 Memory
8 N 4-> Cont2r2o8l ler 4- _|_y Dynaml. c
‘1’ o — Random
CDN
9 ¢ Memory Array
2 Static Mem. @
g Array Q (DRAM)
Array E, (SRAM)
EMxetm.. 252
g 4-» Mgmt Mgmt Bus Solid-State
'5 _ Controller 4 > E Storage Device
w m (120) m
FIG. 2
US. Patent 0e. 2, 2012 Sheet 3 or 22 US 8,281,227 B2
300 \
| Buffer Control @ Data Bus u
A L M
J\ I\ _ :\
Output Butter ‘ Cntl Q ‘
\V
% ? ) Cntl Q Input Buffer
Read Program Cntl Q ? @
Module 4 ? a Wri-t e P'r ogram
a O Cntl Q
<r > -> Module
1 53 a 310
Alignment L’L —
C Cntl Q (T, t
'& MOdU'e ‘ 342 ‘ f5 Cntl Q I Compression
g E — 9 ? Module ?
0 g * . 05 *
.5 N Decompressmn ‘ Cntl Q ‘ g Gnu Q Encryption
8 g Module E _ g ? ' Module m
w _ I i
5 g Decryptlon ‘ Cntl Q ‘ 2 Garb. Collector 4“
Tu _, Module E E b Bypass 316
8 1 S +
_l Garb. Collector ‘ _____________ __ O Cnii Q Packetizer
Bypass m ’ 342 302
I _ _
I v
Depacketlzer ‘ Cntl Q ‘ , Cntl o I, Media Encrypt.
% ? ? Module m
I +
Media Decrypt. ‘ Cntl Q ‘ ECC Generator
Module & ? T 304
+ v T
ECC Correction Bank Interleave Write Buffer
< > Module Controller 320
_ E E —
V , , , 1
Read Sync. ' ' Write Sync
_ _ _ _ _ _ _ Buffer _ Sync. Butter _ _ Buffer _ _ _
c a , E , ?
._ M I I
"1 Read Data ' ' Write Data
5 Pipeline Sg??ggtfs Pipeline
Q 108 106
{I — ? —
O
5 SSS Controller l ' MUX
g, m >\ @
(D
:6 V V
<7) Storage l/O Bus m Storage Control Bus ?
FIG. 3

US. Patent
Oct. 2, 2012 Sheet 5 0f 22 US 8,281,227 B2
450
Storage Storage Storage Storage Storage
Element 1 Element 2 Element 3 Element 4 Element n
402a 402b 4020 402d 402n
_ ___--- _ ___: ———- -
D0 DO I P0 I DO Chunk D0 I|l
408a 408b I 4080 |I 408d 456 408m II |
“ _D_1_ - _ " _P_1_ D1 _ D_1_ " an “ _D_1_ I:
m 4089 4080 4 8d 408m : I
I I I I I | l
P m D m D m D m D m | l
4080 408b 4080 408d 408m | l
- l
_____ ____________ _-____ ___ ____J
Page 1 Page 1 Page 1 Page 1 Page 1 :
410a 410b 4100 410d 410m I
D 0 D O D 0 D O P 0 |
408a 408b 4080 408d 4080 |
D 1 D 1 D 1 P 1 D 1 :
408a 408b 4080 40 Q 408m I
5 5 5 5 5 I
D m D m P m D m D m |
408a 408b 4089 408d 408m |
l
l
l
Page m Page m Page m Page m Page m l
412a 412b 4120 412d 412m l
D O P O D O D O D 0 :
408a ?g M m m I
P 1 D 1 D 1 D 1 D 1 l
40 Q 408b 4080 408m 408m l
: : : : : l
Dim Dim Dim Dim P-m :
408a 408b 4080 408m 4080 I
l
US. Patent 0“. 2, 2012 Sheet 6 or 22 US 8,281,227 B2
500
Reduction Apparatus
m
Receive Module Data Read Module
? ?
Regeneration Module ECC Module
@ @
FIG. 5
US. Patent
Oct. 2, 2012 Sheet 7 of 22 US 8,281,227 B2
600
Reduction Apparatus
Data Read Module
?
Receive Module
?
Read Rotation
Module
?
Regeneration Module ECC Module
@ @
ECC Check Module Substitution Module
@ @
Device Error
Periodic Read Module
Determination Module
@
@
Parity Rotation Module ECC Correction Module
m m
FIG. 6