From raw data to intelligent actions: inside our next-gen AI analytics data lake stack

At Canonical, we believe organisations should be able to unlock the full value of their data without giving up control.

That principle is at the heart of our next-generation enterprise data lake stack. It’s designed for organizations that need to build advanced analytics and AI capabilities while maintaining control over their data, infrastructure and technology choices, with security and governance built into the platform from the outset.

We have been developing and deploying this stack with customers for some time. Their experience has helped us turn that vision into a mature platform for complex, business-critical data workloads. In particular, we see organizations using it to address three increasingly important challenges:

• Building self-governed analytics architectures that reconcile transactional systems, data silos, and the central data lake, enabling HTAP (hybrid transactional-analytical processing) for near-real-time decision-making without surrendering control of sensitive data.
• Scaling self-governed AI and LLM deployments, including behind-the-firewall agentic workloads where data, models, and inference must remain within infrastructure controlled by the organization.
• Delivering secure, AI-enabled conversational analytics, where users can interact naturally with enterprise data while strong identity, access control and governance safeguards determine exactly what data they can access.

Our broader vision is that self-governance should not require organizations to sacrifice innovation. They should be able to adopt modern analytics and AI while retaining control over where their data resides, who can access it, how it is processed, and which technologies they depend upon.

While the stack is comprehensive, there are features that specifically focus on addressing the needs of four key stakeholder groups: the platform operations team, the data engineering team, data scientists and analysts, and the compliance team. We will primarily focus on those capabilities today.

Core architecture and design philosophy

As you can see from the diagram above, there are a lot of features packed in. In this post, we’ll break them down in more detail.

The solution we’ve built is a considered, clean, integrated system founded on open-source software, including the extensive management and automation logic.

The entire AI and analytics data lake stack ships on Ubuntu Server and is built on proven, scalable foundational capabilities including: 

• A CIS (Center for Internet Security) level 1 and 2-capable Kubernetes distribution for cluster management;
• Ceph, a software defined storage system for scale-out storage on commodity hardware with full support for erasure coding and storage tiering;
• A standards-compliant federated identity management system;
• A comprehensive observability stack for logging, monitoring, profiling and alerting;
• Automated wire encryption and certificate management;
• Robust secrets management; 
• And tools and automation for system backup.

We took a number of bold bets in designing the system. For example, the past philosophy for data lake architecture used to be “bring the compute to the data” – or in other words, co-location of the storage and the data processing on converged systems. We chose instead to decouple storage and compute, for a number of reasons: 

• First, data center networking is substantially more capable now than it was 20 years ago when Apache Hadoop was initially developed, so the latency concerns that drove the converged architecture of Apache Hadoop no longer hold true. 
• Second, decoupling the compute tier from the storage tier allows independent (and optionally, dynamic) scaling of CPU and GPU compute resources, making strategies like capacity bursting possible and cost effective. 
• Third, decoupling long-term storage from compute can significantly reduce our users’ operating costs in public cloud contexts regardless of capacity bursting. 
• And finally, the S3 API has become the de-facto cloud storage standard, making S3-compatible, disaggregated storage the logical and obvious choice.

Naturally, the system is oriented towards compliance with open standards. We’ve both defined, and adopted, a number of open standards that we believe provide a familiar yet superior experience for our users. At the low level for example, our machinery for hardened container images – Rocks OCI container image build language, Chiseled packaging system and Pebble system daemon offer revolutionary levels of security and convenience. For lakehouse data catalog and access credentials vending, we’re shipping Apache Polaris, which implements the widely adopted Iceberg Catalog REST API standard, ensuring a governed yet open ecosystem where data remains accessible.

Data ingestion and integration

The stack includes powerful capabilities for data integration. For CDC (changed data capture), we deliver a full distribution of Apache Kafka, including the Kafka Connect integration engine, which has connectors for popular database systems including PostgreSQL, MySQL, MongoDB, and OpenSearch. This capability enables near-realtime ingestion from upstream core transactional systems to the data lake.

For SaaS systems, the stack includes Airbyte with prebuilt connectors for services like Snowflake, GitHub, SalesForce and Slack, while for easy, UI-driven bespoke data integrations we offer Apache NiFi. The system is founded on Apache Spark, so data engineers have the full power and flexibility of a distributed, parallel compute runtime for building advanced batch or streaming ETL (extract, transform, load) data processing jobs.

Lakehouse foundations

With the heart of the system being Apache Spark, the data lake stack delivers a unified engine for managing unstructured, semi-structured, and fully structured data. Spark’s capability set is enabled through both a Python-style distributed dataframe SDK and via an industry-standard, ISO-compliant SQL (structured query language) interpreter. Spark offers capabilities for data processing at extreme scale – covering graph, vector and tabular data; the system can also process text and binary.

In terms of lakehouse storage engines for efficient data and file management at scale, the system delivers support for the Apache Iceberg storage engine as a first-class citizen. The Delta Lake engine is also supported, giving users freedom and choice.

In terms of hardware, Apache Spark offers benefits to users with access to NVIDIA GPU accelerators, who benefit from native support for the NVIDIA Spark-RAPIDS extension, which is fully supported and enables significantly improved performance for many data processing operations. Fractional allocation through NVIDIA GPU virtualization technology is possible.

To simplify Spark job management, users’ analytical queries are submitted and coordinated through Apache Kyuubi, a highly available multi-user SQL gateway that offers flexible resource management options and coordinates job processing on the backend Spark data processing clusters.

And to simplify workload orchestration and workflow management, we ship Apache Airflow as a key part of the distribution. Airflow facilitates scheduling and managing complex DAGs (directed acyclic graphs) as an enterprise job scheduler whilst sophisticated workflows with more complex logic can alternatively be developed with Temporal, which is also available as part of the stack.

Federated data access and governance

A key design goal for the data lake stack was to enable compliance teams with comprehensive tooling for federated data governance. We wanted to enable users to combine data from online, transactional database systems such as PostgreSQL with lakehouse data in a consistent and controlled way. That’s why unified access management and powerful policy definition are enabled by Trino and Apache Ranger respectively; with Ranger you can access audit logging to OpenSearch, a high performance, open-source search engine ideal for SIEM (security incident event management), graph and vector search applications.

So how does this work in practice? Trino offers horizontally scalable data virtualization for unified, SQL-based data access. BI tools access Trino through standard ODBC/JDBC connectors, whilst the Apache Ranger centralized policy engine enforces granular row and column based filter policies, sensitive data redaction and data masking.

Beyond regulating access and policy enforcement, comprehensive governance also requires complete visibility into where data originates and how it moves across the system. To meet this need, the stack integrates DataHub for lineage tracking and as a user browsable data catalogue. Lineage tracking enables governance teams to ensure full data traceability from origin to destination for compliance.

Finally, today’s access demands are no longer limited to human users. AI agents also need to be granted access in a governed way, so we ship integrated MCP (model context protocol) servers for agentic access to DataHub, Apache Superset, and Trino.

Business intelligence, advanced analytics, and AI

As alluded to above, we offer an integrated solution for Apache Superset – a scalable, high performance business intelligence system that enables users to build traditional visualizations, charts, and dashboards. However we don’t stop there. A key design goal for the system is to enable data scientists with powerful tools for machine learning as well as AI model pre-training, fine-tuning, and reinforcement learning.

That’s why the platform ships with a full AI development suite based on Kubeflow. Kubeflow is an industry leading solution for developing and operationalizing advanced data science workloads, from data-analysis using interactive Notebooks, to more structured processes to run model-training and fine-tuning using Kubeflow Pipelines, Trainer, and Katib. We integrate Kubeflow with Feast for offline and online feature store management, and we integrate MLFlow for LLM and ML model observability, experiment tracking and as a private model registry.

Self-governed genAI LLM serving, with duplex performance and extreme scale

Organizations are increasingly finding that they don’t want to send every prompt to cloud services and frontier labs. When integrating agent-driven, conversational analytics over highly sensitive corporate data, the risks associated with a downstream data breach can be unacceptable.

Thus an effective, high performance solution for serving open-weight models like Kimi, GLM, DeepSeek, Llama, and Qwen is also included in the data lake stack, based on the vLLM model serving engine, adapted for low latency, duplex performance by disaggregating the model’s prefill and decode phases using LLM-d and KServe. Horizontal scalability and autoscaling features are also there.

Airgapped platform operations, cost control and scalability

The entire data lake stack is designed for self-governed deployment as a first principle, and is airgapped capable. This means the complete system can run disconnected from the public internet, either in an isolated public cloud tenancy or on-premise running on dedicated physical systems.

For server lifecycle management, the stack works with Canonical MAAS (Metal as a Service), a dynamic systems provisioning engine, which is certified for a wide range of hardware from industry-leading vendors like HPE, Dell, SuperMicro and NVIDIA.

Because the system is founded on Kubernetes and disaggregates the persistent storage tier, users can benefit from cost control techniques such as bursting and autoscaling through various Kubernetes autoscaler implementations. The AI and analytics data lake stack offers broad support for Microsoft Azure, Amazon AWS, Google Cloud and VMware Kubernetes platforms in addition to the included Canonical Kubernetes distribution.

For the long road ahead

The stack is immediately available and includes additional major capabilities not mentioned here. As a committed champion of open-source software delivering more than 22 years of technology innovation, Canonical offers the full system with up to 15 years of support and maintenance through Ubuntu Pro, Canonical’s comprehensive subscription for open source security, so that customers can be assured of the longevity of their investment with peace of mind.

If you’d like to learn more we’d love to hear from you. Contact us now to arrange a call.