Next generation BI - Part two

Technical issues


Comment In the previous article in this series, I outlined some of the major features that you want to see in a next generation BI solution.

Most of this is not complex: new visualisation capabilities, integration with search and so on are not, at least in principle, difficult, though companies providing sophisticated visualisation capabilities may need to provide default functionality for users that do not have high powered graphics cards.

However, where there is a major issue is in being able to support any ad hoc query against any data at any time (and in context).

There are two reasons for this. The first is that conventional OLAP and ROLAP implementations do not understand the relationships that exist between the data. Or rather, they do, but only implicitly.

What I mean is that in a relational database, for example, relationships are defined through the use of primary and foreign keys and in the database schema. In other words, relationship data is built into the structure. The same is true of OLAP cubes, in which dimensions and hierarchies are fundamental to the definition of those cubes. So, relationships are hard-wired by the developer: if relationships have not been defined (and many are not) then you cannot enquire against them, and this is precisely the limiting factor in today's generation of BI products.

To resolve this issue, relationship information needs to be abstracted from the data to which it relates and it needs to be stored separately (though it may be, perhaps partly, replicated in the database structure) so that relationships can be determined dynamically. In practice, this means you must be able to discover relationships as the data is loaded into the database or data warehouse or, where the data has already been loaded, then after the event.

There is another step that is required. Determining that there is a relationship between two pieces of data that you can exploit is one thing - actually getting that data may be another. The reason for this is that conventional BI tools work against aggregated data. That is, sales are summarised by product by store, say, and it is this information that is stored in your OLAP cubes.

However, if you want to report on something the developer was not expecting, the relevant aggregates may not be available. Therefore, any next generation solution must have one of two features: either it must store all possible aggregates, which in turn implies that all data must be stored within a single cube as opposed to the multiple cubes that are typically used today; or it must be able to calculate new aggregates on the fly and, in order to ensure decent performance, this means in memory.

In practice, since you may wish to compare data that comes from different data sources (which implies a requirement for enterprise information integration) you will always need to have the ability to calculate some aggregates on the fly and, therefore, in memory.

On a slightly different topic: we are increasingly seeing the need for real-time operational BI. Currently, this is often deployed using a separate platform from the traditional query environment. This is because traditional environments have static data and dynamic queries whereas real-time BI has static queries and dynamic data (events). Needless to say, the underlying platform should be invisible to the user who should be able to use the same set of query capabilities regardless of the underlying platform. Indeed, you should be able to combine information from the two environments.

Finally, there is the question of migration from, or reuse of, existing environments. As we shall see in the last article in this series, some of the technology needed to support this is precisely the same software needed to extract relationship information from OLAP cubes and relational databases.

Copyright © 2006, IT-Analysis.com


Other stories you might like

  • Your snoozing iOS 15 iPhone may actually be sleeping with one antenna open
    No, you're not really gonna be hacked. But you may be surprised

    Some research into the potentially exploitable low-power state of iPhones has sparked headlines this week.

    While pretty much no one is going to utilize the study's findings to attack Apple users in any meaningful way, and only the most high-profile targets may find themselves troubled by all this, it at least provides some insight into what exactly your iOS handheld is up to when it's seemingly off or asleep. Or none of this is news to you. We'll see.

    According to the research, an Apple iPhone that goes asleep into low-power mode or is turned off isn't necessarily protected against surveillance. That's because some parts of it are still operating at low power.

    Continue reading
  • China will produce one in five of the chips it uses in 2026, says analyst
    Well short of planned 70 percent domestic capacity

    China’s integrated circuit (IC) production has failed to keep pace with its appetite for silicon, with market research firm IC Insights predcicting the nation will produce only one in five ICs it uses in 2026.

    That figure is a increase from 2021's one in six, and reflects eight percent compound annual growth rate from 2021 to 2026. But it means China will miss its own targets for locally-made-and-consumed silicon.

    “Although China has been the largest consuming country for ICs since 2005, it does not necessarily mean that large increases in IC production within China would immediately follow, or ever follow” said the firm in a bulletin on Wednesday.

    Continue reading
  • Tencent happily parting ways with loss-making cloud customers
    Cutting costs across sprawling business as COVID makes life hard in China

    Chinese tech giant Tencent has recorded its first ever quarter-to-quarter revenue fall, warned that COVID-19 lockdowns will hurt messing with its business, and cautioned against assumptions that Beijing is ready to enthusiastically support tech companies.

    On its Q1 2022 earnings call yesterday, the company offered more explanation of its shifting cloud strategy.

    Chief strategy officer James Mitchell told investors the company is pleased to have shown loss-making cloud customers the door, and “proactively scaled back … deeply discounted infrastructure-only contracts for basic services such as cloud compute and content delivery network.” Projects that had high costs and/or relied on sub-contractors have also been scaled back.

    Continue reading
  • Will this be one of the world's first RISC-V laptops?
    A sneak peek at a notebook that could be revealed this year

    Pic As Apple and Qualcomm push for more Arm adoption in the notebook space, we have come across a photo of what could become one of the world's first laptops to use the open-source RISC-V instruction set architecture.

    In an interview with The Register, Calista Redmond, CEO of RISC-V International, signaled we will see a RISC-V laptop revealed sometime this year as the ISA's governing body works to garner more financial and development support from large companies.

    It turns out Philipp Tomsich, chair of RISC-V International's software committee, dangled a photo of what could likely be the laptop in question earlier this month in front of RISC-V Week attendees in Paris.

    Continue reading
  • Did ID.me hoodwink Americans with IRS facial-recognition tech, senators ask
    Biz tells us: Won't someone please think of the ... fraud we've stopped

    Democrat senators want the FTC to investigate "evidence of deceptive statements" made by ID.me regarding the facial-recognition technology it controversially built for Uncle Sam.

    ID.me made headlines this year when the IRS said US taxpayers would have to enroll in the startup's facial-recognition system to access their tax records in the future. After a public backlash, the IRS reconsidered its plans, and said taxpayers could choose non-biometric methods to verify their identity with the agency online.

    Just before the IRS controversy, ID.me said it uses one-to-one face comparisons. "Our one-to-one face match is comparable to taking a selfie to unlock a smartphone. ID.me does not use one-to-many facial recognition, which is more complex and problematic. Further, privacy is core to our mission and we do not sell the personal information of our users," it said in January.

    Continue reading
  • Meet Wizard Spider, the multimillion-dollar gang behind Conti, Ryuk malware
    Russia-linked crime-as-a-service crew is rich, professional – and investing in R&D

    Analysis Wizard Spider, the Russia-linked crew behind high-profile malware Conti, Ryuk and Trickbot, has grown over the past five years into a multimillion-dollar organization that has built a corporate-like operating model, a year-long study has found.

    In a technical report this week, the folks at Prodaft, which has been tracking the cybercrime gang since 2021, outlined its own findings on Wizard Spider, supplemented by info that leaked about the Conti operation in February after the crooks publicly sided with Russia during the illegal invasion of Ukraine.

    What Prodaft found was a gang sitting on assets worth hundreds of millions of dollars funneled from multiple sophisticated malware variants. Wizard Spider, we're told, runs as a business with a complex network of subgroups and teams that target specific types of software, and has associations with other well-known miscreants, including those behind REvil and Qbot (also known as Qakbot or Pinkslipbot).

    Continue reading
  • Supreme Court urged to halt 'unconstitutional' Texas content-no-moderation law
    Everyone's entitled to a viewpoint but what's your viewpoint on what exactly is and isn't a viewpoint?

    A coalition of advocacy groups on Tuesday asked the US Supreme Court to block Texas' social media law HB 20 after the US Fifth Circuit Court of Appeals last week lifted a preliminary injunction that had kept it from taking effect.

    The Lone Star State law, which forbids large social media platforms from moderating content that's "lawful-but-awful," as advocacy group the Center for Democracy and Technology puts it, was approved last September by Governor Greg Abbott (R). It was immediately challenged in court and the judge hearing the case imposed a preliminary injunction, preventing the legislation from being enforced, on the basis that the trade groups opposing it – NetChoice and CCIA – were likely to prevail.

    But that injunction was lifted on appeal. That case continues to be litigated, but thanks to the Fifth Circuit, HB 20 can be enforced even as its constitutionality remains in dispute, hence the coalition's application [PDF] this month to the Supreme Court.

    Continue reading
  • How these crooks backdoor online shops and siphon victims' credit card info
    FBI and co blow lid off latest PHP tampering scam

    The FBI and its friends have warned businesses of crooks scraping people's credit-card details from tampered payment pages on compromised websites.

    It's an age-old problem: someone breaks into your online store and alters the code so that as your customers enter their info, copies of their data is siphoned to fraudsters to exploit. The Feds this week have detailed one such effort that reared its head lately.

    As early as September 2020, we're told, miscreants compromised at least one American company's vulnerable website from three IP addresses: 80[.]249.207.19, 80[.]82.64.211 and 80[.]249.206.197. The intruders modified the web script TempOrders.php in an attempt to inject malicious code into the checkout.php page.

    Continue reading

Biting the hand that feeds IT © 1998–2022