Architecture for Enterprise Business Intelligence

443 Pages·2012·17.23 MB·English

Architecture for
Enterprise Business
Intelligence
an overview of the microstrategy
platform architecture for big data,
cloud bi, and mobile applications
Architecture for
Enterprise Business
Intelligence
an overview of the microstrategy
platform architecture for big data,
cloud bi, and mobile applications
Table of Contents

INTRODUCTION
1 Foreword 3
2 Platform Design 7
2.1 MicroStrategy’s Philosophy – Delivering an Architecture for the Long Term 7
2.2 Architectural Tenets for Enterprise Business Intelligence 8
2.3 Complete BI Functionality Delivered on an Organically-developed Architecture 10
2.4 Enterprise BI on an Integrated Backplane and a Unified Web Interface 11
2.5 Summary 14
3 Product Architecture 17
3.1 MicroStrategy has a Single Unified Architecture 17
3.2 MicroStrategy Intelligence Server 19
3.3 MicroStrategy Web 19
3.4 MicroStrategy Mobile 19
3.5 MicroStrategy Report Services 20
3.6 MicroStrategy OLAP Services 20
3.7 MicroStrategy Transaction Services 20
3.8 MicroStrategy Distribution Services 21
3.9 MicroStrategy Office 21
3.10 MicroStrategy Desktop 21
3.11 MicroStrategy Architect 22
3.12 MicroStrategy MultiSource Option 22
3.13 MicroStrategy Clustering Option 23
3.14 MicroStrategy Object Manager 23
3.15 MicroStrategy Integrity Manager 23
3.16 MicroStrategy Enterprise Manager 24
3.17 MicroStrategy Command Manager 24
3.18 MicroStrategy SDK 24
3.19 Summary 25
PLATFORM COMPONENTS
4 Metadata and the Logical Model 29
4.1 A Complete Metadata is Key for Efficient BI Processing 29
4.2 Configuration Objects 31
4.3 Data Abstraction Objects 32
4.4 Business Abstraction Objects 35
4.5 Report Components 37
4.6 Report and Document Definitions 38
4.7 Visual Analyses 41
4.8 Delivery Objects 41
4.9 Comprehensive Documentation of Metadata 43
4.10 Multi-Lingual BI Applications 44
4.11 Object Change Journaling 47
4.12 Benefits of the Metadata Design 48
4.13 Summary 52
5 MicroStrategy Intelligence Server Architecture 55
5.1 Powering the MicroStrategy Business Intelligence Platform 56
5.2 Intelligence Server Engines and Query Flow 57
5.3 Automatic Resource Allocation 59
5.4 64-bit Business Intelligence 60
5.5 Cluster-Capable Intelligence Server 60
5.6 Optimized for Heterogeneous Data Sources 61
5.7 Sophisticated Analysis 66
5.8 In-Memory Intelligent Cubes 71
5.9 Multi-Level Shared Report Caching 74
5.10 Scheduling Reports and Administrative Tasks 77
5.11 Exporting Data 78
5.12 Formatting Data 79
5.13 Simplified Central Administration 81
5.14 Summary 85
6 MicroStrategy Web Architecture 87
6.1 Core Design Principles 87
6.2 Optimize Performance with a Service-Oriented Architecture 87
6.3 Open, Layered Design Optimizes Performance, Integration, and Customization 89
6.4 MicroStrategy Web is built for Integration and Customization 90
6.5 A Single, Platform-Independent Code Base Provides Flexibility and Consistency 92
6.6 All Data in the BI Application is Secure 93
6.7 Employ Leading Edge Technologies to Maximize Ease-of-Use and Interactivity 95
6.8 Seamless Support for Enterprise Portals 97
6.9 Data Integrated with Geo-Spatial Information Systems 99
6.10 Summary 100
7 MicroStrategy Mobile Architecture 103
7.1 Core Design Principles 103
7.2 Deliver Mobile Apps in a Multi-Tiered Architecture 103
7.3 MicroStrategy Mobile App Architecture 105
7.4 MicroStrategy Mobile Server Architecture 107
7.5 All Data in the MOBILE BI APPLICATION is Secure 108
7.6 Leading Edge Technologies Maximize Ease-of-Use and Interactivity 109
7.7 Summary 110
8 Report Scheduling and Delivery 113
8.1 Designing for High-Volume Automated Report Delivery 113
8.2 Self-Service vs. Centrally-Managed Subscriptions 115
8.3 Exception Reporting 117
8.4 High Performance Report Deliveries 118
8.5 Summary 120
PLATFORM FUNCTIONALITY
9 Security 125
9.1 Securing BI Applications 125
9.2 User Authentication 126
9.3 User Authorization 128
9.4 Multi-Tier Web Architecture and Transmission Security 134
9.5 Mobile Security 137
9.6 Summary 141
55 10 Reliability and Fault Tolerance 143
56 10.1 Stateful and Stateless Architectures 143
57 10.2 Shared, unified metadata 145
59 10.3 Communication between Cluster Nodes 145
60 10.4 MicroStrategy Clustering Capabilities 146
60 10.5 Fault Tolerance with Built-in Failover 149
61 10.6 Adaptation to Changing Conditions 149
66 10.7 Summary 150
71
11 Scalability and Performance 153
74
11.1 High-Performance Initiative 153
77
11.2 Multi-Level Shared Caching 154
78
11.3 In-Memory Cubes 157
79
11.4 Query Optimization 159
81
11.5 Distributable Execution 167
85
11.6 64-bit Business Intelligence 169
87 11.7 Multi-Threaded Processing 170
11.8 Efficient Communications 172
87
11.9 Performance Tuning 174
87
11.10 Proven Scalability and Performance 174
89
11.11 Summary 175
90
92 12 Closed-Loop Business Intelligence 177
93
12.1 Closed-Loop Applications Reduce the Time to Act 177
95
12.2 Closed-Loop BI Using MicroStrategy Transaction Services 178
97
12.3 Enabling transactional functionality in MicroStrategy 180
99
12.4 Closed-Loop BI Using Dynamic Datamarts 183
100
12.5 Summary 185
103 13 Data Mining and Predictive Analysis 187
103 13.1 Integrating Data Mining Into Business Intelligence 188
103 13.2 Data Mining Services in MicroStrategy 190
105 13.3 Data Mining Algorithms 191
107 13.4 Benefits of Integrating Business Intelligence and Data Mining 198
108 13.5 Summary 199
109
14 Heterogeneous Data Access 201
110
14.1 Data is Stored in Many Different Places 201
113 14.2 Relational Data Sources 202
14.3 Multi-Dimensional Sources 205
113
14.4 Data Access to Other Data Sources 206
115
14.5 Joining Data from Heterogeneous Data Sources 208
117
14.6 Summary 211
118
120 15 Exporting Data and Reports 213
15.1 MicroStrategy Supports Flexible Exporting for All Uses 213
15.2 MicroStrategy Provides Efficient Exporting of Data 214
15.3 1.3 MicroStrategy Office Integrates With Excel, Word, and PowerPoint 217
15.4 Summary 220
125
125
USER INTERACTIVITY
126
128 16 End User Experience 225
134 16.1 Consistent Reports, Functionality, and Security through Many User Interfaces 225
137 16.2 The Business Intelligence Application Spectrum 230
141 16.3 MicroStrategy User Experience Design Tenets 231
16.4 Dashboards and Scorecards 234
16.5 Visual Insight Analysis 236
16.6 Transactions 237
16.7 Enterprise reporting 238
16.8 Ad-hoc Reporting and OLAP Analysis 239
16.9 Advanced and Predictive Analysis 245
16.10 Alerts and proactive notifications 246
16.11 User Functionality by User Role 247
16.12 Customizing MicroStrategy Functionality for the end user 250
16.13 Summary 250
17 Developer Experience 253
17.1 Multiple Interfaces for BI Development 253
17.2 MicroStrategy Architect 253
17.3 Map Database Structures to Qualitative and Quantitative Business Terms 257
17.4 MicroStrategy Desktop 260
17.5 Business Abstraction Objects 261
17.6 Report Objects 265
17.7 Report creation using the dynamic SQL engine 265
17.8 Creating Reports from Intelligent Cubes 267
17.9 Build and Execute Reports with the Dynamic MDX Engine 268
17.10 Operational Reports using the Freeform SQL engine 269
17.11 1.11 Transaction Reports 270
17.12 Building Graphs in MicroStrategy Desktop 270
17.13 Data Mining and Predictive Reports in MicroStrategy Desktop 271
17.14 Report Services Document Creation 272
17.15 Advanced Visualizations 274
17.16 Creating Dashboards and Reports in Web 276
17.17 Creating Pixel-Perfect Dashboards and Scorecards over the Web 281
17.18 Transaction Documents 283
17.19 Customization / SDK Developer 285
17.20 Summary 286
18 Administrator Experience 289
18.1 In-depth Administration across the Whole Platform 289
18.2 Real-Time Monitoring and Control 290
18.3 Error Logging and Diagnostic Files 292
18.4 MicroStrategy Enterprise Manager 293
18.5 Centralized Administration 294
18.6 Change Journaling 297
18.7 MicroStrategy Command Manager 300
18.8 MicroStrategy Object Manager 303
18.9 Integrity Manager 307
18.10 MicroStrategy Web Administration 310
18.11 MicroStrategy Mobile Administration 313
18.12 Health Center 315
18.13 System Management Software Integration 316
18.14 Summary 317
IMPLEMENTING BI APPLICATIONS
19 Developing Business Intelligence Applications 321
19.1 Ad-Hoc Data Discovery 321
19.2 Application Project Lifecycle 323
234 19.3 Evolve from Departmental to Enterprise BI Applications 326
236 19.4 Estimating the Capacity Requirements of the BI Application 330
237 19.5 Summary 334
238
20 Portable Analytic Modules 337
239
20.1 Rapid Application Development Framework 337
245
20.2 Portable Analytic Applications 338
246
20.3 MicroStrategy BI Developer Kit Analytic Modules 339
247
20.4 Summary 347
250
250 21 Extensibility and the SDK 349
253 21.1 Access All BI Functionality in a Service-Oriented Architecture with Open APIs 349
21.2 MicroStrategy Web API 351
253
21.3 Task Framework 356
253
21.4 MicroStrategy Mobile API 357
257
21.5 MicroStrategy Visualization API 358
260
21.6 MicroStrategy Office API 360
261
21.7 MicroStrategy Intelligence Server API 361
265
21.8 Comprehensive Documentation That Describes the API in Detail 361
265
21.9 Portal Integration 363
267
21.10 Web Services Development 363
268
21.11 Sample Applications 365
269
21.12 Customization and Extension Examples 367
270
21.13 Summary 373
270
271
272 APPENDICES
274
276 Appendix A: Schema Support 377
281 Appendix B: Very Large Database (VLDB) Properties 381
283 Appendix C: Analytical Functions 389
285 Appendix D: Supported Graph Types 392
286 Appendix E: Performance Counters and Key Performance Indicators 394
Appendix F: Supported Data Sources 402
289
Appendix G: Operating Platforms 405
289 Appendix H: Security Privileges 410
290 Appendix I: API Classes, Methods, Properties, and Interfaces 412
292
293
294
297
300
303
307
310
313
315
316
317
321
321
323
INTRODUCTION
IN
T
R
O
D
U
C
T
IO
N
1
FOREWARD
Partnering With Our Customers to Achieve Enterprise Business Intelligence
The MicroStrategy® architecture that you see today is the culmination of roughly 5,000 man-years of
engineering effort and well over half a billion dollars in engineering investment. More significant is the fact
that the MicroStrategy architecture embodies the cumulative engineering advances gained through over 20
years of experience in supporting the most demanding BI applications in the world.
Our customers drive our technology development. Ever since MicroStrategy’s first customer implementation,
we have been challenged to solve the most technically difficult problems in BI. After countless
implementations for over thousands of customers, we fully understand the value of a close partnership
of co-development and BI innovation that is continually fueled by our customers’ uniquely aggressive and
ever-expanding requirements. The effort we have invested in supporting our customers’ initiatives and
applications has propelled our software faster and farther than any competing BI technology on the market.
Very early we realized that our customers’ BI requirements would continue to grow as their data volumes
and user populations increased, and as their business requirements became broader and sophisticated. In
fact, it became clear to us that our customers were driving the technology toward the goal of “Enterprise-
wide Business Intelligence.” We took this goal to heart when we designed our new product architecture
– beginning with MicroStrategy 7™ in 2000, and evolving to MicroStrategy 8™ by 2005 and significantly
expanding our platform capabilities with MicroStrategy 9™ in 2009.
Nearly every BI vendor now claims to have an “integrated BI architecture” suitable for Enterprise BI. What
is often overlooked is the fact that MicroStrategy’s end goal was not integration – it was the starting point.
We view integration not as a feature but as a requirement of a scalable, maintainable, and high performance
architecture. Any architecture with duplicative or redundant metadata, data structures, processes, queues,
caches, and thread management facilities has inherent performance inefficiencies, is difficult to manage, and
will be more costly to maintain. Given that performance and scalability have always been the dominant limiting
factors to successfully deploy enterprise systems, an architecture designed from the start for Enterprise BI was
needed – not one forcibly mashed together from a set of existing products and legacy code.
As such, we determined that we needed to discard every line of code in our previous architecture (version
6 and prior) and start over from scratch. While this business decision was considered risky at the time, the
strategy was necessary if we wanted to meet the long-term needs of our customers. It allowed us to start by
designing an architecture built specifically for performance and maintainability. Additionally, it allowed us to
build new features as part of an integrated architecture, rather than worry about integrating legacy products
after the fact. Interestingly, none of the competitive BI technologies followed our approach. Instead, they
decided to make incremental gains towards integration, and chose architectures built primarily to achieve
legacy integration rather than for performance and flexibility. In retrospect, our strategy has turned out to
have extraordinarily beneficial consequences for both MicroStrategy and our customers.
We have continued to innovate and advance the BI platform to keep up with our customers’ and the
market’s requirements. Full-featured business user and developer Web interfaces, dashboards, data
3
IN
T
R discovery, visualizations, in-memory cubes, mobile BI, Office integration, data input and write-back, and
O
D
advanced analytics and data mining have been added. The result of this 20 year journey has resulted in a
U
C
T mature, Industrial-Strength Business Intelligence™ platform that is available as on-premises installations or on
IO
MicroStrategy’s Cloud infrastructure.
N
Our architecture can be compared to an iceberg where 90% of the mass lies under the surface, hidden from casual
view. Often, our customers do not begin to appreciate the full depth and thoughtfulness of our product’s capabilities
until they have used the technology for several years, and grown their BI implementation into a world-class application
beyond their initial expectations and vision.
We offer this book as a means of formally presenting our architecture − coded by our engineers and co-developed by
our customers for over 20 years. I hope the content will help provide a level of detail that is lacking in most BI literature
today, and provide some clarity as to what you can achieve with the MicroStrategy Platform. We believe that we have
a unique offering and are confident that as you dig into the detail, you will conclude the same.
Jeff Bedell
Chief Technology Officer, MicroStrategy

4