• David Wang, Director of Product Marketing at Databricks


• 11.11.2016 09:00 am
data

We are proud to announce that Omega Point, a leading data analytics provider, has selected Databricks for its ability to apply advanced analytics to synthesize insights from disparate large-scale datasets.

Inferring insights from a multitude of large-scale data to inform business-critical decisions is an increasingly common practice across industries. Omega Point is yet another example of this trend in financial services, where a sliver of timely information could have millions of dollars of impact.

Omega Point offers finance and strategy professionals a software platform called Portfolio Intelligence that enables them to understand, visualise, and optimise their portfolios. Its offering taps into satellite imagery, web traffic, earnings transcript sentiment, and other large-scale data sets known as “economic exhaust” or “alternative datasets” to uncover portfolios’ exposure to 50+ relevant market factors using cutting edge data science.

Extracting insights from “alternative datasets” requires cutting-edge data processing capabilities to infer valuable information from a plethora of seemingly low-value data. Omega Point selected Apache Spark because of its scalability and flexibility to tackle advanced analytics. However, their initial attempt to deploy Spark by integrating a self-managed PaaS with open source data science tools ran into roadblocks because of reliability and performance problems. As a result, the release of the Omega Point Portfolio Intelligence Platform was severely delayed.

Omega Point chose Databricks to power the backbone of its production data pipeline. On a daily basis, Databricks pulls over 120 sources of data directly from Omega Point’s Amazon S3 account and through a sequence of over 80 production Spark jobs, and produces relevant indicators used to assess current economic and financial market trends in clients’ portfolios. Databricks also enabled Omega Point’s data science team to build a systematic learning environment, where Spark’s MLlib and popular open source libraries such as numpy, pandas, scipy, sklearn, matplotlib could be seamlessly integrated to develop sophisticated models rapidly.

With Databricks, Omega Point gained a high-performance and reliable Spark platform that addressed their reliability and performance bottlenecks. Databricks sped up the release cycle, improved data throughput, and reduced the debugging time for the data engineering and science teams. Omega Point’s deployment and maintenance costs dropped by 75% while its production uptime increased by more than 80%. Ultimately, Databricks enabled Omega Point to accelerate the release of core feature of its flagship product by six months.