Which Cloud Data Platform Should You Pick?

For contemporary businesses, selecting an appropriate data platform is paramount. Such platforms are essential not only for storing and securing corporate data but also for acting as analytical powerhouses, generating crucial insights for key strategic decisions.

The market offers numerous solutions, continually advancing with artificial intelligence. Nevertheless, five key contenders—Databricks, Snowflake, Amazon Redshift, Google BigQuery, and Microsoft Fabric—are identified as the top choices for enterprises today.

Databricks Overview

Emerging in 2013 from the minds behind Apache Spark, the open-source analytics platform, Databricks quickly became a major force in the data sector. The company is credited with coining and developing the ‘data lakehouse’ concept, an innovative approach merging data lake and data warehouse functionalities to enhance how organizations manage their data assets.

A data lakehouse unifies what were traditionally distinct architectures: data lakes, designed for vast quantities of raw data, and data warehouses, holding categorized structured data. This consolidated framework empowers organizations to conduct queries across all data sources simultaneously and to manage associated workloads effectively.

Now recognized as a distinct category, the lakehouse architecture is extensively adopted and integrated into numerous IT environments.

Databricks champions itself as a ‘data+AI’ leader, claiming to offer the industry’s sole platform with a cohesive governance framework spanning both data and AI, alongside a singular query engine for ML, BI, SQL, and ETL operations.

The Databricks Data Intelligence Platform heavily emphasizes machine learning and AI tasks, with deep roots in the Apache Spark ecosystem. Its adaptable and open environment accommodates nearly all data types and workloads.

To facilitate the agentic AI paradigm, Databricks introduced Agent Bricks, powered by Mosaic, enabling users to deploy tailored AI agents and systems that leverage their specific data and requirements. Organizations can utilize retrieval-augmented generation (RAG) to construct agents on proprietary data, employing Databricks’ vector database for memory functions. 

Core Platform: The flagship product from Databricks is its Data Intelligence Platform. This platform is inherently cloud-native, engineered from inception for cloud environments, and designed to comprehend the underlying meaning (semantics) of business data, which underpins its ‘intelligence’ capabilities.

Built upon a lakehouse architecture, the platform utilizes open-format software interfaces like Delta Lake and Apache Iceberg, ensuring standardized interactions and seamless interoperability. It also integrates Databricks’ Unity Catalog, which provides unified management for access control, quality assurance, data exploration, auditing, data lineage, and security.

Powering the platform is DatabricksIQ, the company’s Data Intelligence Engine. This engine employs generative AI for semantic understanding, leveraging breakthroughs from MosaicML, a company acquired by Databricks in 2023.

Deployment Method: Databricks operates as a cloud-native platform, featuring strategic alliances with leading cloud service providers such as Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform (GCP).

Pricing: Databricks employs a pay-as-you-go pricing structure, eliminating upfront expenses. Users are billed exclusively for consumed services, measured with “per second granularity.” Unit-based pricing varies for different services, including data engineering, data warehousing, interactive workloads, AI, and operational databases (from $0.07 to $0.40). Discounts are available through committed use contracts for customers agreeing to specific usage thresholds.

Challenges/Trade-offs: Databricks can be more intricate to operate and less “plug-and-play” compared to serverless alternatives. As it’s fundamentally an Apache Spark-based platform, it necessitates more management and tuning effort than environments designed for simpler operation. Its pricing models may also exhibit greater complexity.

Further Databricks Insights

• Its integrated stack facilitates data pipelines, feature engineering, business intelligence (BI), ML training, and other intricate operations on a single storage layer.
• The platform’s compatibility with open formats and engines, such as Delta and Iceberg, prevents vendor lock-in regarding storage.
• Unity Catalog delivers a centralized governance layer, where data descriptions and tags enable the platform to comprehend an organization’s specific data semantics.
• Agent Bricks and MLflow collectively provide a robust toolkit for AI and machine learning development.

Snowflake Analysis

Established in 2013, Snowflake is recognized as a trailblazer in cloud data warehousing. It functions as a unified hub for both structured and semi-structured data, readily accessible to organizations for analysis and business intelligence (BI) purposes.

Snowflake is regarded as a primary rival to Databricks. Indeed, in a direct counterpoint to the data lakehouse innovator, Snowflake asserts that its platform has consistently functioned as a hybrid combining data warehouse and data lake capabilities.

Core Platform: Snowflake defines itself as an ‘AI Data Cloud,’ engineered to oversee all data-centric operations within an enterprise. Similar to Databricks, its platform is cloud-native, integrating storage, scalable compute resources, and various cloud services into a single environment.

Snowflake facilitates AI model creation, notably via its Cortex AI agent-builder platform, along with advanced analytics and other intensive data workloads. Its Snowgrid cross-cloud layer ensures global connectivity across diverse regions and cloud providers, promoting consistent performance. Concurrently, the Snowflake Horizon governance layer oversees access, security, privacy, compliance, and interoperability.

Snowpipe and Openflow functionalities enable real-time data ingestion, integration, and streaming. Snowpark Connect facilitates migration and compatibility with Apache Spark codebases. Furthermore, Cortex AI provides a secure environment for users to execute large language models (LLMs) and develop generative AI and agentic applications.

Deployment Method: Aligned with Databricks, Snowflake maintains alliances with major industry players, delivering its solution as Software-as-a-Service (SaaS) across AWS, Azure, GCP, and other cloud environments. Notably, a key strategic alliance with Microsoft specifically permits customers to acquire and operate Azure Databricks directly, facilitating integration with additional Azure services.

Pricing: Snowflake utilizes a consumption-based pricing model. Compute resources are billed using credits, starting at $2, with costs varying by subscription tier (standard, enterprise, business critical, or virtual private Snowflake) and cloud region. Data storage within Snowflake incurs a separate monthly charge, determined by average usage.

Snowflake Strengths: Snowflake presents itself as a ready-to-use, fully managed SQL platform. It is designed for data-intensive applications, offering robust governance and requiring minimal manual configuration or tuning.

Moreover, Snowflake persistently advances its offerings for the agentic AI landscape. Snowflake Intelligence, for example, empowers users to query their data and receive responses using natural language. Cortex AI delivers secure access to prominent Large Language Models (LLMs), enabling teams to invoke models, execute text-to-SQL commands, and implement RAG within Snowflake while safeguarding their data privacy.

Snowflake Challenges/Trade-offs

• Its proprietary storage and compute engine offer less openness and control compared to a lakehouse setup.
• The credit-based pricing and additional serverless features can make cost prediction and management challenging.
• Some users have noted less robust support for unstructured data and real-time data streaming.

Further Considerations for Snowflake

• Its elastic compute architecture delivers robust performance for a wide array of users, data volumes, and workloads within a unified, scalable engine.
• Infrastructure management is minimal, as Snowflake abstracts away numerous complexities including optimization, planning, and authentication.
• Storage offers seamless interoperability, granting users unified and unsiloed access to their data.
• Snowgrid functionalities operate across multiple regions and clouds (including AWS, Azure, GCP, and others), enabling effortless data sharing, workload portability, and uniform global policy enforcement.

Source: Foundry

Amazon Redshift Deep Dive

Amazon Web Services (AWS) Redshift offers a fully managed, petabyte-scale cloud data warehouse. It is engineered to supersede intricate and costly on-premises legacy data infrastructure.

Core Platform: Amazon Redshift functions as a data warehouse optimized for querying and extensive analytics on vast datasets. Its foundational architecture relies on two key principles: columnar storage and massively parallel processing (MPP). Data is structured across distinct nodes (columns), and MPP enables rapid, concurrent processing of these datasets.

Redshift leverages standard SQL for interacting with relational database data and seamlessly integrates with extract, transform, load (ETL) solutions, such as AWS Glue, for data management and preparation. Its Amazon Redshift Spectrum capability allows users to query data directly from Amazon Simple Storage (Amazon S3) files, bypassing the need to first load data into tables.

Furthermore, Amazon Redshift ML enables developers to construct and train Amazon SageMaker machine learning (ML) models directly from their Redshift data using straightforward SQL commands.

Redshift benefits from profound integration within the AWS ecosystem, ensuring effortless interoperability with a wide array of other AWS services.

Deployment Method: Amazon Redshift is entirely managed by AWS and is available through two deployment models: provisioned, which involves a fixed rate for dedicated resources regardless of usage, and serverless, a pay-per-use model.

Pricing: Redshift provides two pricing tiers corresponding to its deployment options: provisioned and serverless. Provisioned instances start at $0.543 per hour, while serverless usage commences at $1.50 per hour. Both models are capable of scaling to petabytes of data and accommodating thousands of simultaneous users.

Redshift’s Strengths: The primary advantage of AWS Redshift lies in its deep integration within the extensive AWS ecosystem. It readily connects with services such as S3, Glue, SageMaker, Kinesis data streaming, and other AWS offerings. This inherent compatibility makes it an ideal solution for organizations already heavily invested in AWS, enabling secure data access, combination, and sharing with reduced data movement or duplication.

Additionally, AWS has launched Amazon Q, a generative AI assistant offering specialized functionalities for software developers, BI analysts, and other professionals utilizing AWS. Amazon Q allows users to inquire about their data to facilitate decision-making, accelerate tasks, and ultimately boost productivity.

Redshift Challenges/Trade-offs

• Ecosystem Lock-in: Although Redshift integrates seamlessly into the AWS ecosystem, it may not be suitable for organizations pursuing multi-cloud or cloud-agnostic strategies.
• Management Overhead: Despite being an AWS-managed service, users often report that it requires more manual oversight than some alternatives. Tasks like data compaction (e.g., vacuum operations), routine ETL process checks, and ongoing monitoring for irregular queries can be necessary, potentially affecting service performance.

Further Considerations for Redshift

• Developers appreciate Redshift’s ease of use, attributed to its SQL-based foundation.
• The platform achieves high performance and scalability through its columnar architecture, separation of compute and storage, and Massively Parallel Processing (MPP).
• AWS provides adaptable deployment choices: provisioned clusters for consistent workloads and serverless options for fluctuating demand.
• Its Zero-ETL functionalities streamline data ingestion, eliminating the need for intricate pipelines and enabling near real-time analytics.

Google BigQuery Overview

Originally conceived as a fully managed cloud data warehouse, Google BigQuery has evolved into an autonomous data and AI platform, offered by Google, that automates the complete data lifecycle.

Core Platform: Google BigQuery is a serverless, distributed, columnar data warehouse tailored for extensive, petabyte-scale workloads and SQL-driven analytics. It operates on Google’s Dremel execution engine, which dynamically allocates resources for queries, enabling rapid analysis of terabytes of data with optimized resource consumption.

BigQuery separates its compute layer (Dremel) from its storage layer, storing data columnarly within Google’s distributed file system, Colossus. Data ingestion from operational systems, logs, SaaS applications, and various other sources is commonly managed through extract, transform, load (ETL) tools.

Leveraging standard SQL commands, BigQuery empowers developers to readily train, evaluate, and deploy ML models for functions such as linear regression and time-series forecasting (for prediction), and k-means clustering (for analytics). When integrated with Vertex AI, the platform is capable of performing advanced predictive analytics and executing AI workflows directly on warehouse data.

Moreover, BigQuery supports the integration of agentic AI, including pre-fabricated agents for data engineering, data science, analytics, and conversational analytics. Developers also have the option to craft custom agents using APIs and agent development kit (ADK) integrations.

Deployment Method: Google BigQuery is a fully managed service, defaulting to a serverless architecture. This design means users are not required to provision or administer individual servers or clusters.

Pricing: BigQuery offers three distinct pricing structures. A free tier grants users up to 1 tebibyte (TiB) of query capacity monthly. On-demand pricing (per-TiB) bills customers according to the volume of data processed by each query. Capacity pricing (per slot-hour) assesses charges based on the allocated compute capacity, quantified in slots (virtual CPUs) over duration.

Strengths: BigQuery’s deep integration with the GCP ecosystem makes it a straightforward option for organizations already significant users of Google’s product suite. The platform is inherently scalable, delivers high speed, and is genuinely serverless, freeing customers from the burdens of infrastructure management or provisioning.

GCP consistently drives AI innovation: BigQuery ML (BQML) empowers analysts to construct, train, and deploy ML models using simple SQL commands directly within the platform interface, while Vertex AI can be employed for more sophisticated MLOps and agentic AI workflows.

BigQuery Challenges/Trade-offs

• The expense for intensive workloads can be hard to forecast, necessitating rigorous partitioning and clustering strategies.
• Users have noted challenges related to testing and schema inconsistencies during extract, transform, load (ETL) operations.

Additional BigQuery Considerations

• Its architecture, which separates storage (Colossus) from compute (Dremel engine), allows BigQuery to analyze petabytes of data within seconds.
• Google autonomously manages resource allocation, maintenance, and scaling, freeing teams from operational concerns.
• The platform offers adaptable payment schemes, catering to both consistent and intermittent workload patterns.
• Support for standard SQL enables analysts to query data using their existing skill sets, eliminating the need for further training.

Microsoft Fabric Review

Microsoft Fabric is a Software-as-a-Service (SaaS) data analytics platform that consolidates data warehousing, real-time analytics, and business intelligence (BI). It is underpinned by OneLake, Microsoft’s “logical” data lake, which employs virtualization to deliver a unified data view across disparate systems.

Core Platform: Fabric is offered as a SaaS solution, with all workloads executing on OneLake, Microsoft’s data lake constructed upon Azure Data Lake Storage (ADLS). Fabric’s integrated catalog ensures centralized management of data lineage, discovery, and governance for all analytical assets, including tables, lakehouses, warehouses, reports, and ML tools.

Various workloads operate directly on OneLake, enabling seamless chaining without requiring data movement between services. These encompass a data factory (offering pipelines, dataflows, connectors, and ETL/ELT for data ingestion and processing); a lakehouse, equipped with Spark notebooks and pipelines for data engineering in a Delta format; and a data warehouse, featuring SQL endpoints, T-SQL compatibility, clustering and identity columns, alongside migration utilities.

Additionally, real-time intelligence, powered by Microsoft’s Eventstream and Activator tools, enables the ingestion of telemetry and other Fabric events without requiring code. This facilitates data monitoring and automated actions for teams. Microsoft Power BI is natively integrated with OneLake, and its DirectLake feature allows direct querying of lakehouse data, bypassing the need for import or redundant storage.

Fabric also supports integration with Azure Machine Learning and Foundry, allowing users to develop, deploy models, and conduct inferencing directly on Fabric datasets. Moreover, the platform incorporates native Microsoft Copilot agents. These agents assist users in tasks such as drafting SQL queries, constructing notebooks and pipelines, summarizing information, generating insights, and populating code and documentation.

Microsoft advocates for a “medallion” lakehouse architecture within Fabric, a format designed to progressively enhance data structure and quality. The company describes this as a “three-stage” process for cleaning and organizing data, which ultimately results in data that is “more reliable and easier to utilize.”

These three stages comprise: Bronze (where raw, ingested data is stored verbatim); Silver (where data is cleaned, errors corrected, formats standardized, and duplicates eliminated); and Gold (where data is meticulously curated and prepared for integration into reports and dashboards).

Deployment Method: Fabric is delivered as a fully managed Software-as-a-Service (SaaS) solution by Microsoft, hosted entirely within its Azure cloud computing infrastructure.

Pricing: Fabric employs a capacity-based licensing model (FSKUs), offering two billing alternatives: a flexible pay-as-you-go option, billed by the second, allowing for dynamic scaling or pausing; and reserved capacity plans, which are prepaid for 1 to 3 years and can provide 40% to 50% cost savings for consistent workloads. Storage in OneLake is usually billed independently.

Microsoft Fabric Strengths

• Its deliberate design as an all-encompassing SaaS solution means a singular platform manages ingestion, lakehouse, warehouse, and real-time ML and BI operations.
• The integrated Copilot feature aids in expediting routine tasks (like documentation or SQL coding), a capability users often highlight as superior to competitor AI tools that lack such deep integration.
• Microsoft endorses and details the medallion architecture, utilizing lake views to automate data progression through bronze, silver, and gold stages.

Microsoft Fabric Challenges/Trade-offs

• As a relatively new platform (generally available in 2023), users have expressed that some features seem underdeveloped, and both documentation and best practices are still maturing.
• It carries the potential for vendor lock-in within the Microsoft ecosystem, which might deter organizations seeking more open, multi-cloud solutions such as Databricks or Snowflake.
• Given its capacity/consumption-based pricing structure, diligent FinOps practices are advisable to prevent unexpected costs.

Further Considerations for Microsoft Fabric

• The Direct Lake mode enables Power BI to perform analyses on vast datasets directly from OneLake’s memory, bypassing the “import/refresh” cycles typical of other platforms.
• This Zero-ETL capability empowers Fabric to virtualize data from sources like Snowflake, Databricks, or Amazon S3. This means you can view and query your Snowflake tables directly within Fabric without any data transfer.
• Copilot Integration: Built-in AI assistants empower users to write Spark code, develop data factory pipelines, and even create comprehensive Power BI reports using natural language prompts.

Conclusion

Selecting the optimal cloud data platform represents a strategic choice far more comprehensive than mere storage and retrieval. While prominent vendors now integrate data storage, governance frameworks, and sophisticated AI functionalities, they diverge in aspects such as operational intricacy, ecosystem compatibility, and cost structures.

Ultimately, the ideal platform hinges on an organization’s specific cloud strategy, operational sophistication, diversity of workloads, AI objectives, and preference regarding ecosystem commitment versus architectural adaptability.