Beyond the LLM: Understanding the AI Stack for AR+IQ

Photo by Steve Johnson on Unsplash

Most teams building AI-powered products fall into the same trap. They plug a large language model (LLM) like GPT or Claude into a user interface, ship an app, and call it done. But for serious applied AI, especially for domains like AR+IQ, this is only the beginning.

At iSPARX™ and through the AR+IQ platform, we operate across the entire AI stack. Here’s a simple framework that guides our architectural thinking, with real-world examples from our work in immersive media, cultural storytelling, and augmented reality (AR) experiences.  .

1. LLM (Large Language Model) — The Base Layer

Think: GPT-4o, Claude 3, Mistral.

This is the general-purpose language generation layer. It’s where most AI builders start. LLMs excel at:

  • Summarising documents

  • Drafting text content

  • Powering FAQs

  • Supporting conversational interfaces

In AR+IQ:

We use LLMs to:

  • Automatically generate metadata for artworks in AR+artist™

  • Produce draft cultural narratives for AR+story™ modules

  • Create first-pass text content for AR+concierge™ pop-up installations 

Limitations:

  • No real-time awareness of current events or updates

  • No direct access to internal or enterprise data unless integrated further

  • Struggles with executing tasks or workflows

  • Lacks consistency in long-running interactions

Example failure mode:

If a user asks about the status of their AR+artist™ exhibit deployment, a pure LLM can’t query deployment data directly.


2. RAG (Retrieval-Augmented Generation) — The Knowledge Layer

Think: Vector databases + LLM.

RAG bridges internal data and AI generation. The model retrieves relevant context before answering.

In AR+IQ:

We embed RAG into:

  • The AR+concierge™ knowledge base for geospatial event data

  • Retrieval of cultural consultation documents (e.g. kaupapa Māori protocols, Indigenous IP frameworks) 

  • Querying system logs, location data, and user interaction telemetry 

RAG is good for:

  • Providing precise answers based on internal documentation

  • Building context-aware document copilots for internal staff

  • Serving structured content to artists, cultural partners, and event producers

Limitations:

  • Still passive — it cannot act or perform transactions

  • Complex multi-step workflows are still out of reach

  • Limited real-time interaction with live systems

Example:

When planning an AR+story™ guided experience at Ruatoki, RAG allows instant access to co-created cultural content libraries .


3. AI Agents — The Action Layer

Think: Orchestration models that can call APIs, monitor systems, and execute tasks.

In AR+IQ:

AI Agents are deployed to:

  • Automate AR+workshop scheduling across venues 

  • Manage participant onboarding and consent tracking (critical for ethics compliance) 

  • Update CMS content dynamically for changing installations 

Agents are good for:

  • Scheduling logistics across AR+concierge™ deployments

  • Generating real-time analytics reports for event partners 

  • Integrating ticketing, loyalty, and ecommerce platforms directly into AR+IQ pop-ups

Limitations:

  • Single-agent models struggle with complex multi-agent workflows

  • Long-term planning and adaptation require higher-order coordination

Example:

An agent automatically coordinates with venue management systems to update event calendars for GigBox.Ai installations .


4. Agentic AI — The System Orchestration Layer

Think: Teams of agents, each with specialised roles, operating with memory, coordination, and autonomy.

In AR+IQ:

We are piloting Agentic AI for:

  • Coordinated deployment of multi-location exhibitions with live telemetry feedback 

  • Dynamic adjustment of AR scene composition based on real-time crowd flow data 

  • Multi-agent content creation pipelines that balance artist input, cultural protocols, and technical deployment 

Agentic AI is good for:

  • Orchestrating complex AR+IQ deployments across multiple stakeholders

  • Managing long-running adaptive AR environments

  • Enabling self-healing AR experiences that adjust based on user behaviour

Limitations:

  • Complex to design and manage

  • Often unnecessary for simple use cases

  • Requires extensive domain knowledge to train and supervise

Example:

During the Pātaka Museum installation, agents collaborated to manage live updates, visitor engagement metrics, and narrative adjustments driven by participant feedback.


The Quick Guide: Choosing the Right AI Layer for AR+IQ

Problem >> Solution

You need customer Q&A over internal documents? >> Use RAG

You need automated workshop scheduling? >> Use Agents

You need end-to-end orchestration across multiple locations and teams? >> Use Agentic AI


The Reality: Most Teams Only Ship the First Layer

Many startups stop at LLM + UI. In iSPARX™, that approach would barely scratch the surface of what AR+IQ delivers. For true applied AI in immersive, cultural, geospatial experiences, the full stack is not optional — it’s foundational  .

We don’t build chatbots.

We build fully autonomous systems that understand context, execute real-world actions, and deliver culturally-anchored experiences.


In short:

LLM is the conversation.

RAG is the knowledge.

Agents are the hands.

Agentic AI is the brain.


At iSPARX™, we aren’t just “applying AI” — we are redefining how audiences interact with space, story, and culture. AR+IQ is built for 22nd-century creative industries.



Architecting AI for AR+IQ: Beyond the LLM Layer

When most AI startups say they’re “AI-powered,” they mean:

  • They called OpenAI’s API.

  • Wrapped it in a frontend.

  • Shipped.

For CTOs building scalable, production-grade platforms like AR+IQ, that approach is not even V0. It’s a prototype at best. Applied AI requires layered architectures — integrating multiple AI modalities, agents, orchestration logic, and deeply embedded domain knowledge.

At iSPARX™, AR+IQ is not a chatbot. It’s a multi-modal, agentic AI system designed to operate in highly contextualised, spatial, and culturally-sensitive environments across multiple sectors: arts, tourism, education, and commerce   .

Below is the simplified stack we’ve operationalised:


1. LLM Foundation Layer — Generative Core

Function:

Autoregressive language models trained on massive corpora.

Examples in stack: GPT-4o, Claude 3, Mistral 7B

Primary Roles in AR+IQ:

  • Drafting AR+story™ narrative scaffolds 

  • Summarising post-installation feedback reports

  • Generating dynamic textual overlays for AR+artist™ and AR+guide™

  • Assisting in metadata auto-generation for the CMS 

Architecture Considerations:

  • Hosted LLM endpoints (OpenAI/Azure OpenAI) via secured API layer

  • Wrapped with safety filters to enforce cultural protocol constraints 

  • Fine-tuned internal embeddings for domain-specific semantic precision

Weaknesses:

  • No system state awareness

  • Cannot call APIs or control spatial environments directly

  • Prone to hallucination without grounding


2. RAG Layer — Retrieval Augmented Knowledge Integration

Function:

Real-time grounding using proprietary indexed datasets and vector embeddings.

Examples in stack:

  • Azure Cognitive Search

  • Pinecone VectorDB

  • Custom Faiss deployments for embedded cultural knowledge graphs 

Use Cases in AR+IQ:

  • Indigenous knowledge retrieval for AR+concierge™ installations 

  • Regulatory compliance retrieval for cross-border deployments

  • Internal technical documentation retrieval for field techs

Architecture Considerations:

  • Embedding pipelines running on Azure Functions with daily refresh cycles

  • Vector databases with hybrid search (semantic + keyword + filtering)

  • Secure multi-tenant isolation to separate clients’ knowledge bases

Weaknesses:

  • Still passive — requires orchestration layer to act on retrieved data

  • Latency spikes under heavy complex queries


3. AI Agent Layer — Orchestrated Action Units

Function:

Encapsulated API-calling agents with single-domain responsibility.

Examples in stack:

  • Custom-built agents via LangChain + Azure OpenAI Orchestration

  • Workflow orchestrators using Temporal.io or Durable Functions

  • Integration to live operational systems (Venue CMS, CRM, logistics platforms)

Use Cases in AR+IQ:

  • Deployment automation for AR+IQ pop-up installations 

  • Real-time ethics compliance checks during participant onboarding 

  • Live workshop scheduling and resource allocation 

  • Dynamic telemetry monitoring for live spatial interactions 

Architecture Considerations:

  • Each agent operates with scoped permissions for auditability

  • Secure API token vaulting using Azure KeyVault

  • Real-time queue management for parallelised agent execution (Azure Service Bus)

Weaknesses:

  • Agents do not coordinate — orchestration logic must handle multi-agent state

  • Performance drops under complex emergent workflows

4. Agentic AI Layer — Distributed Autonomous Coordination

Function:

Multi-agent systems with persistent shared state, role-based task allocation, and dynamic reasoning.

Examples in stack:

  • Multi-agent coordination via Autogen/OpenDevin-like systems (internal prototype)

  • Memory layers via Redis Streams & event sourcing logs

  • Custom reasoning models for cultural protocols compliance (ethical constraints enforcer)

Use Cases in AR+IQ:

  • Coordinating real-time updates across multi-location AR+IQ deployments 

  • Managing cross-border asset approvals, cultural authority consultations, and ethics board reporting 

  • Negotiating optimal visitor flows based on real-time AR+concierge™ telemetry 

Architecture Considerations:

  • Temporal state machines with compensation strategies

  • Fine-grained task decomposition into micro-agents with context-limited reasoning

  • Persistent memory layers for longitudinal deployment state awareness

Weaknesses:

  • High operational complexity

  • Requires human-in-the-loop supervisory agents for governance

  • Limited generalisability — domain-specific engineering required


The Architectural Meta-Pattern

The Architectural Meta-Pattern

This is a layered system design approach for applied AI systems operating in real-world domains (AR, immersive media, multi-user environments, cultural storytelling, etc). Each layer builds capability beyond the previous one. You can think of it as stacked capabilities, each solving a class of problem:


Why We Can’t Just “Use GPT-4o”

The majority of real-world AR+IQ deployments involve:

  • Real-time geospatial coordination 

  • Multi-stakeholder cultural governance 

  • Physical environment state management (venue mapping, visitor flows, weather data) 

  • Long-running, stateful, inter-agent negotiation

These are non-trivial systems problems. They require distributed systems expertise, domain-specific ontologies, real-time sensor integration, and deeply ethical architecture considerations  .


CTO Takeaway

If your AI product simply calls GPT, you’re building a demo.

If your AI product runs RAG + Agents + Agentic AI across real-world, live spatial deployments like AR+IQ — you’re building operational AI.

The AR+IQ stack is built for:

  • Enterprise scale

  • Regulatory safety

  • Cultural safety

  • Multi-modal experiences across live physical spaces.


At iSPARX™, we are engineering 22nd century creative infrastructure, not chatbots.

JoFF Rae

Producer & Creative / New Media Artist with international cognisance in experiential media, arts & entertainment / developer of creative projects // of Ati Awa / Ko Taranaki te māunga / from Aotearoa / live in New Zealand / reside in the Wellington region / produce via Auckland / work from home, office & studio / presently active in Auckland, Wellington, Calgary, New York, LA, Melbourne & elsewhere / working on working remotely from Costa Del Sol / creative by any means necessary! / Guilty of ART!//

http://www.isparx.group
Next
Next

iSPARX™ AR+ Environmental Visualisation Platform