This snapshot establishes the camera-to-result recognition flow and related tests while checking in the project skill/docs assets required for the configured local tooling.
930 lines
26 KiB
Markdown
930 lines
26 KiB
Markdown
---
|
|
name: axiom-realitykit
|
|
description: Use when building 3D content, AR experiences, or spatial computing with RealityKit. Covers ECS architecture, SwiftUI integration, RealityView, AR anchors, materials, physics, interaction, multiplayer, performance.
|
|
license: MIT
|
|
metadata:
|
|
version: "1.0.0"
|
|
---
|
|
|
|
# RealityKit Development Guide
|
|
|
|
**Purpose**: Build 3D content, AR experiences, and spatial computing apps using RealityKit's Entity-Component-System architecture
|
|
**iOS Version**: iOS 13+ (base), iOS 18+ (RealityView on iOS), visionOS 1.0+
|
|
**Xcode**: Xcode 15+
|
|
|
|
## When to Use This Skill
|
|
|
|
Use this skill when:
|
|
- Building any 3D experience (AR, games, visualization, spatial computing)
|
|
- Creating SwiftUI apps with 3D content (RealityView, Model3D)
|
|
- Implementing AR with anchors (world, image, face, body tracking)
|
|
- Working with Entity-Component-System (ECS) architecture
|
|
- Setting up physics, collisions, or spatial interactions
|
|
- Building multiplayer or shared AR experiences
|
|
- Migrating from SceneKit to RealityKit
|
|
- Targeting visionOS
|
|
|
|
Do NOT use this skill for:
|
|
- SceneKit maintenance (use `axiom-scenekit`)
|
|
- 2D games (use `axiom-spritekit`)
|
|
- Metal shader programming (use `axiom-metal-migration-ref`)
|
|
- Pure GPU compute (use Metal directly)
|
|
|
|
---
|
|
|
|
## 1. Mental Model: ECS vs Scene Graph
|
|
|
|
### Scene Graph (SceneKit)
|
|
|
|
In SceneKit, nodes own their properties. A node IS a renderable, collidable, animated thing.
|
|
|
|
### Entity-Component-System (RealityKit)
|
|
|
|
In RealityKit, entities are **empty containers**. Components add data. Systems process that data.
|
|
|
|
```
|
|
Entity (identity + hierarchy)
|
|
├── TransformComponent (position, rotation, scale)
|
|
├── ModelComponent (mesh + materials)
|
|
├── CollisionComponent (collision shapes)
|
|
├── PhysicsBodyComponent (mass, mode)
|
|
└── [YourCustomComponent] (game-specific data)
|
|
|
|
System (processes entities with specific components each frame)
|
|
```
|
|
|
|
**Why ECS matters**:
|
|
- **Composition over inheritance**: Combine any components on any entity
|
|
- **Data-oriented**: Systems process arrays of components efficiently
|
|
- **Decoupled logic**: Systems don't know about each other
|
|
- **Testable**: Components are pure data, Systems are pure logic
|
|
|
|
### The ECS Mental Shift
|
|
|
|
| Scene Graph Thinking | ECS Thinking |
|
|
|---------------------|--------------|
|
|
| "The player node moves" | "The movement system processes entities with MovementComponent" |
|
|
| "Add a method to the node subclass" | "Add a component, create a system" |
|
|
| "Override `update(_:)` in the node" | "Register a System that queries for components" |
|
|
| "The node knows its health" | "HealthComponent holds data, DamageSystem processes it" |
|
|
|
|
---
|
|
|
|
## 2. Entity Hierarchy
|
|
|
|
### Creating Entities
|
|
|
|
```swift
|
|
// Empty entity
|
|
let entity = Entity()
|
|
entity.name = "player"
|
|
|
|
// Entity with components
|
|
let entity = Entity()
|
|
entity.components[ModelComponent.self] = ModelComponent(
|
|
mesh: .generateBox(size: 0.1),
|
|
materials: [SimpleMaterial(color: .blue, isMetallic: false)]
|
|
)
|
|
|
|
// ModelEntity convenience (has ModelComponent built in)
|
|
let box = ModelEntity(
|
|
mesh: .generateBox(size: 0.1),
|
|
materials: [SimpleMaterial(color: .red, isMetallic: true)]
|
|
)
|
|
```
|
|
|
|
### Hierarchy Management
|
|
|
|
```swift
|
|
// Parent-child
|
|
parent.addChild(child)
|
|
child.removeFromParent()
|
|
|
|
// Find entities
|
|
let found = root.findEntity(named: "player")
|
|
|
|
// Enumerate
|
|
for child in entity.children {
|
|
// Process children
|
|
}
|
|
|
|
// Clone
|
|
let clone = entity.clone(recursive: true)
|
|
```
|
|
|
|
### Transform
|
|
|
|
```swift
|
|
// Local transform (relative to parent)
|
|
entity.position = SIMD3<Float>(0, 1, 0)
|
|
entity.orientation = simd_quatf(angle: .pi / 4, axis: SIMD3(0, 1, 0))
|
|
entity.scale = SIMD3<Float>(repeating: 2.0)
|
|
|
|
// World-space queries
|
|
let worldPos = entity.position(relativeTo: nil)
|
|
let worldTransform = entity.transform(relativeTo: nil)
|
|
|
|
// Set world-space transform
|
|
entity.setPosition(SIMD3(1, 0, 0), relativeTo: nil)
|
|
|
|
// Look at a point
|
|
entity.look(at: targetPosition, from: entity.position, relativeTo: nil)
|
|
```
|
|
|
|
---
|
|
|
|
## 3. Components
|
|
|
|
### Built-in Components
|
|
|
|
| Component | Purpose |
|
|
|-----------|---------|
|
|
| `Transform` | Position, rotation, scale |
|
|
| `ModelComponent` | Mesh geometry + materials |
|
|
| `CollisionComponent` | Collision shapes for physics and interaction |
|
|
| `PhysicsBodyComponent` | Mass, physics mode (dynamic/static/kinematic) |
|
|
| `PhysicsMotionComponent` | Linear and angular velocity |
|
|
| `AnchoringComponent` | AR anchor attachment |
|
|
| `SynchronizationComponent` | Multiplayer sync |
|
|
| `PerspectiveCameraComponent` | Camera settings |
|
|
| `DirectionalLightComponent` | Directional light |
|
|
| `PointLightComponent` | Point light |
|
|
| `SpotLightComponent` | Spot light |
|
|
| `CharacterControllerComponent` | Character physics controller |
|
|
| `AudioMixGroupsComponent` | Audio mixing |
|
|
| `SpatialAudioComponent` | 3D positional audio |
|
|
| `AmbientAudioComponent` | Non-positional audio |
|
|
| `ChannelAudioComponent` | Multi-channel audio |
|
|
| `OpacityComponent` | Entity transparency |
|
|
| `GroundingShadowComponent` | Contact shadow |
|
|
| `InputTargetComponent` | Gesture input (visionOS) |
|
|
| `HoverEffectComponent` | Hover highlight (visionOS) |
|
|
| `AccessibilityComponent` | VoiceOver support |
|
|
|
|
### Custom Components
|
|
|
|
```swift
|
|
struct HealthComponent: Component {
|
|
var current: Int
|
|
var maximum: Int
|
|
|
|
var percentage: Float {
|
|
Float(current) / Float(maximum)
|
|
}
|
|
}
|
|
|
|
// Register before use (typically in app init)
|
|
HealthComponent.registerComponent()
|
|
|
|
// Attach to entity
|
|
entity.components[HealthComponent.self] = HealthComponent(current: 100, maximum: 100)
|
|
|
|
// Read
|
|
if let health = entity.components[HealthComponent.self] {
|
|
print(health.current)
|
|
}
|
|
|
|
// Modify
|
|
entity.components[HealthComponent.self]?.current -= 10
|
|
```
|
|
|
|
### Component Lifecycle
|
|
|
|
Components are value types (structs). When you read a component, modify it, and write it back, you're replacing the entire component:
|
|
|
|
```swift
|
|
// Read-modify-write pattern
|
|
var health = entity.components[HealthComponent.self]!
|
|
health.current -= damage
|
|
entity.components[HealthComponent.self] = health
|
|
```
|
|
|
|
**Anti-pattern**: Holding a reference to a component and expecting mutations to propagate. Components are copied on read.
|
|
|
|
---
|
|
|
|
## 4. Systems
|
|
|
|
### System Protocol
|
|
|
|
```swift
|
|
struct DamageSystem: System {
|
|
// Define which components this system needs
|
|
static let query = EntityQuery(where: .has(HealthComponent.self))
|
|
|
|
init(scene: RealityKit.Scene) {
|
|
// One-time setup
|
|
}
|
|
|
|
func update(context: SceneUpdateContext) {
|
|
for entity in context.entities(matching: Self.query,
|
|
updatingSystemWhen: .rendering) {
|
|
var health = entity.components[HealthComponent.self]!
|
|
if health.current <= 0 {
|
|
entity.removeFromParent()
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
// Register system
|
|
DamageSystem.registerSystem()
|
|
```
|
|
|
|
### System Best Practices
|
|
|
|
- **One responsibility per system**: MovementSystem, DamageSystem, RenderingSystem — not GameLogicSystem
|
|
- **Query filtering**: Use precise queries to avoid processing irrelevant entities
|
|
- **Order matters**: Systems run in registration order. Register dependencies first.
|
|
- **Avoid storing entity references**: Query each frame instead. Entity references can become stale.
|
|
|
|
### Event Handling
|
|
|
|
```swift
|
|
// Subscribe to collision events
|
|
scene.subscribe(to: CollisionEvents.Began.self) { event in
|
|
let entityA = event.entityA
|
|
let entityB = event.entityB
|
|
// Handle collision
|
|
}
|
|
|
|
// Subscribe to scene update
|
|
scene.subscribe(to: SceneEvents.Update.self) { event in
|
|
let deltaTime = event.deltaTime
|
|
// Per-frame logic
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## 5. SwiftUI Integration
|
|
|
|
### RealityView (iOS 18+, visionOS 1.0+)
|
|
|
|
```swift
|
|
struct ContentView: View {
|
|
var body: some View {
|
|
RealityView { content in
|
|
// make closure — called once
|
|
let box = ModelEntity(
|
|
mesh: .generateBox(size: 0.1),
|
|
materials: [SimpleMaterial(color: .blue, isMetallic: false)]
|
|
)
|
|
content.add(box)
|
|
|
|
} update: { content in
|
|
// update closure — called when SwiftUI state changes
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
### RealityView with Camera (iOS)
|
|
|
|
On iOS, `RealityView` provides a camera content parameter for configuring the AR or virtual camera:
|
|
|
|
```swift
|
|
RealityView { content, attachments in
|
|
// Load 3D content
|
|
if let model = try? await ModelEntity(named: "scene") {
|
|
content.add(model)
|
|
}
|
|
}
|
|
```
|
|
|
|
### Loading Content Asynchronously
|
|
|
|
```swift
|
|
RealityView { content in
|
|
// Load from bundle
|
|
if let entity = try? await Entity(named: "MyScene", in: .main) {
|
|
content.add(entity)
|
|
}
|
|
|
|
// Load from URL
|
|
if let entity = try? await Entity(contentsOf: modelURL) {
|
|
content.add(entity)
|
|
}
|
|
}
|
|
```
|
|
|
|
### Model3D (Simple Display)
|
|
|
|
```swift
|
|
// Simple 3D model display (no interaction)
|
|
Model3D(named: "toy_robot") { model in
|
|
model
|
|
.resizable()
|
|
.scaledToFit()
|
|
} placeholder: {
|
|
ProgressView()
|
|
}
|
|
```
|
|
|
|
### SwiftUI Attachments (visionOS)
|
|
|
|
```swift
|
|
RealityView { content, attachments in
|
|
let entity = ModelEntity(mesh: .generateSphere(radius: 0.1))
|
|
content.add(entity)
|
|
|
|
if let label = attachments.entity(for: "priceTag") {
|
|
label.position = SIMD3(0, 0.15, 0)
|
|
entity.addChild(label)
|
|
}
|
|
} attachments: {
|
|
Attachment(id: "priceTag") {
|
|
Text("$9.99")
|
|
.padding()
|
|
.glassBackgroundEffect()
|
|
}
|
|
}
|
|
```
|
|
|
|
### State Binding Pattern
|
|
|
|
```swift
|
|
struct GameView: View {
|
|
@State private var score = 0
|
|
|
|
var body: some View {
|
|
VStack {
|
|
Text("Score: \(score)")
|
|
|
|
RealityView { content in
|
|
let scene = try! await Entity(named: "GameScene")
|
|
content.add(scene)
|
|
} update: { content in
|
|
// React to state changes
|
|
// Note: update is called when SwiftUI state changes,
|
|
// not every frame. Use Systems for per-frame logic.
|
|
}
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## 6. AR on iOS
|
|
|
|
### AnchorEntity
|
|
|
|
```swift
|
|
// Horizontal plane
|
|
let anchor = AnchorEntity(.plane(.horizontal, classification: .table,
|
|
minimumBounds: SIMD2(0.2, 0.2)))
|
|
|
|
// Vertical plane
|
|
let anchor = AnchorEntity(.plane(.vertical, classification: .wall,
|
|
minimumBounds: SIMD2(0.5, 0.5)))
|
|
|
|
// World position
|
|
let anchor = AnchorEntity(world: SIMD3<Float>(0, 0, -1))
|
|
|
|
// Image anchor
|
|
let anchor = AnchorEntity(.image(group: "AR Resources", name: "poster"))
|
|
|
|
// Face anchor (front camera)
|
|
let anchor = AnchorEntity(.face)
|
|
|
|
// Body anchor
|
|
let anchor = AnchorEntity(.body)
|
|
```
|
|
|
|
### SpatialTrackingSession (iOS 18+)
|
|
|
|
```swift
|
|
let session = SpatialTrackingSession()
|
|
let configuration = SpatialTrackingSession.Configuration(tracking: [.plane, .object])
|
|
let result = await session.run(configuration)
|
|
|
|
if let notSupported = result {
|
|
// Handle unsupported tracking on this device
|
|
for denied in notSupported.deniedTrackingModes {
|
|
print("Not supported: \(denied)")
|
|
}
|
|
}
|
|
```
|
|
|
|
### AR Best Practices
|
|
|
|
- Anchor entities to detected surfaces rather than world positions for stability
|
|
- Use plane classification (`.table`, `.floor`, `.wall`) to place content appropriately
|
|
- Start with horizontal plane detection — it's the most reliable
|
|
- Test on real devices; simulator AR is limited
|
|
- Provide visual feedback during surface detection (coaching overlay)
|
|
|
|
---
|
|
|
|
## 7. Interaction
|
|
|
|
### ManipulationComponent (iOS, visionOS)
|
|
|
|
```swift
|
|
// Enable drag, rotate, scale gestures
|
|
entity.components[ManipulationComponent.self] = ManipulationComponent(
|
|
allowedModes: .all // .translate, .rotate, .scale
|
|
)
|
|
|
|
// Also requires CollisionComponent for hit testing
|
|
entity.generateCollisionShapes(recursive: true)
|
|
```
|
|
|
|
### InputTargetComponent (visionOS)
|
|
|
|
```swift
|
|
// Required for visionOS gesture input
|
|
entity.components[InputTargetComponent.self] = InputTargetComponent()
|
|
entity.components[CollisionComponent.self] = CollisionComponent(
|
|
shapes: [.generateBox(size: SIMD3(0.1, 0.1, 0.1))]
|
|
)
|
|
```
|
|
|
|
### Gesture Integration with SwiftUI
|
|
|
|
```swift
|
|
RealityView { content in
|
|
let entity = ModelEntity(mesh: .generateBox(size: 0.1))
|
|
entity.generateCollisionShapes(recursive: true)
|
|
entity.components.set(InputTargetComponent())
|
|
content.add(entity)
|
|
}
|
|
.gesture(
|
|
TapGesture()
|
|
.targetedToAnyEntity()
|
|
.onEnded { value in
|
|
let tappedEntity = value.entity
|
|
// Handle tap
|
|
}
|
|
)
|
|
.gesture(
|
|
DragGesture()
|
|
.targetedToAnyEntity()
|
|
.onChanged { value in
|
|
value.entity.position = value.convert(value.location3D,
|
|
from: .local, to: .scene)
|
|
}
|
|
)
|
|
```
|
|
|
|
### Hit Testing
|
|
|
|
```swift
|
|
// Ray-cast from screen point
|
|
if let result = arView.raycast(from: screenPoint,
|
|
allowing: .estimatedPlane,
|
|
alignment: .horizontal).first {
|
|
let worldPosition = result.worldTransform.columns.3
|
|
// Place entity at worldPosition
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## 8. Materials and Rendering
|
|
|
|
### Material Types
|
|
|
|
| Material | Purpose | Customization |
|
|
|----------|---------|---------------|
|
|
| `SimpleMaterial` | Solid color or texture | Color, metallic, roughness |
|
|
| `PhysicallyBasedMaterial` | Full PBR | All PBR maps (base color, normal, metallic, roughness, AO, emissive) |
|
|
| `UnlitMaterial` | No lighting response | Color or texture, always fully lit |
|
|
| `OcclusionMaterial` | Invisible but occludes | AR content hiding behind real objects |
|
|
| `VideoMaterial` | Video playback on surface | AVPlayer-driven |
|
|
| `ShaderGraphMaterial` | Custom shader graph | Reality Composer Pro |
|
|
| `CustomMaterial` | Metal shader functions | Full Metal control |
|
|
|
|
### PhysicallyBasedMaterial
|
|
|
|
```swift
|
|
var material = PhysicallyBasedMaterial()
|
|
material.baseColor = .init(tint: .white,
|
|
texture: .init(try! .load(named: "albedo")))
|
|
material.metallic = .init(floatLiteral: 0.0)
|
|
material.roughness = .init(floatLiteral: 0.5)
|
|
material.normal = .init(texture: .init(try! .load(named: "normal")))
|
|
material.ambientOcclusion = .init(texture: .init(try! .load(named: "ao")))
|
|
material.emissiveColor = .init(color: .blue)
|
|
material.emissiveIntensity = 2.0
|
|
|
|
let entity = ModelEntity(
|
|
mesh: .generateSphere(radius: 0.1),
|
|
materials: [material]
|
|
)
|
|
```
|
|
|
|
### OcclusionMaterial (AR)
|
|
|
|
```swift
|
|
// Invisible plane that hides 3D content behind it
|
|
let occluder = ModelEntity(
|
|
mesh: .generatePlane(width: 1, depth: 1),
|
|
materials: [OcclusionMaterial()]
|
|
)
|
|
occluder.position = SIMD3(0, 0, 0)
|
|
anchor.addChild(occluder)
|
|
```
|
|
|
|
### Environment Lighting
|
|
|
|
```swift
|
|
// Image-based lighting
|
|
if let resource = try? await EnvironmentResource(named: "studio_lighting") {
|
|
// Apply via RealityView content
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## 9. Physics and Collision
|
|
|
|
### Collision Shapes
|
|
|
|
```swift
|
|
// Generate from mesh (accurate but expensive)
|
|
entity.generateCollisionShapes(recursive: true)
|
|
|
|
// Manual shapes (prefer for performance)
|
|
entity.components[CollisionComponent.self] = CollisionComponent(
|
|
shapes: [
|
|
.generateBox(size: SIMD3(0.1, 0.2, 0.1)), // Box
|
|
.generateSphere(radius: 0.1), // Sphere
|
|
.generateCapsule(height: 0.3, radius: 0.05) // Capsule
|
|
]
|
|
)
|
|
```
|
|
|
|
### Physics Body
|
|
|
|
```swift
|
|
// Dynamic — physics simulation controls movement
|
|
entity.components[PhysicsBodyComponent.self] = PhysicsBodyComponent(
|
|
massProperties: .init(mass: 1.0),
|
|
material: .generate(staticFriction: 0.5,
|
|
dynamicFriction: 0.3,
|
|
restitution: 0.4),
|
|
mode: .dynamic
|
|
)
|
|
|
|
// Static — immovable collision surface
|
|
ground.components[PhysicsBodyComponent.self] = PhysicsBodyComponent(
|
|
mode: .static
|
|
)
|
|
|
|
// Kinematic — code-controlled, participates in collisions
|
|
platform.components[PhysicsBodyComponent.self] = PhysicsBodyComponent(
|
|
mode: .kinematic
|
|
)
|
|
```
|
|
|
|
### Collision Groups and Filters
|
|
|
|
```swift
|
|
// Define groups
|
|
let playerGroup = CollisionGroup(rawValue: 1 << 0)
|
|
let enemyGroup = CollisionGroup(rawValue: 1 << 1)
|
|
let bulletGroup = CollisionGroup(rawValue: 1 << 2)
|
|
|
|
// Filter: player collides with enemies and bullets
|
|
entity.components[CollisionComponent.self] = CollisionComponent(
|
|
shapes: [.generateSphere(radius: 0.1)],
|
|
filter: CollisionFilter(
|
|
group: playerGroup,
|
|
mask: enemyGroup | bulletGroup
|
|
)
|
|
)
|
|
```
|
|
|
|
### Collision Events
|
|
|
|
```swift
|
|
// Subscribe in RealityView make closure or System
|
|
scene.subscribe(to: CollisionEvents.Began.self, on: playerEntity) { event in
|
|
let otherEntity = event.entityA == playerEntity ? event.entityB : event.entityA
|
|
handleCollision(with: otherEntity)
|
|
}
|
|
```
|
|
|
|
### Applying Forces
|
|
|
|
```swift
|
|
if var motion = entity.components[PhysicsMotionComponent.self] {
|
|
motion.linearVelocity = SIMD3(0, 5, 0) // Impulse up
|
|
entity.components[PhysicsMotionComponent.self] = motion
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## 10. Animation
|
|
|
|
### Transform Animation
|
|
|
|
```swift
|
|
// Animate to position over duration
|
|
entity.move(
|
|
to: Transform(
|
|
scale: SIMD3(repeating: 1.5),
|
|
rotation: simd_quatf(angle: .pi, axis: SIMD3(0, 1, 0)),
|
|
translation: SIMD3(0, 2, 0)
|
|
),
|
|
relativeTo: entity.parent,
|
|
duration: 2.0,
|
|
timingFunction: .easeInOut
|
|
)
|
|
```
|
|
|
|
### Playing USD Animations
|
|
|
|
```swift
|
|
if let entity = try? await Entity(named: "character") {
|
|
// Play all available animations
|
|
for animation in entity.availableAnimations {
|
|
entity.playAnimation(animation.repeat())
|
|
}
|
|
}
|
|
```
|
|
|
|
### Animation Playback Control
|
|
|
|
```swift
|
|
let controller = entity.playAnimation(animation)
|
|
controller.pause()
|
|
controller.resume()
|
|
controller.speed = 2.0 // 2x playback speed
|
|
controller.blendFactor = 0.5 // Blend with current state
|
|
```
|
|
|
|
---
|
|
|
|
## 11. Audio
|
|
|
|
### Spatial Audio
|
|
|
|
```swift
|
|
// Load audio resource
|
|
let resource = try! AudioFileResource.load(named: "engine.wav",
|
|
configuration: .init(shouldLoop: true))
|
|
|
|
// Create entity with spatial audio
|
|
let audioEntity = Entity()
|
|
audioEntity.components[SpatialAudioComponent.self] = SpatialAudioComponent()
|
|
let controller = audioEntity.playAudio(resource)
|
|
|
|
// Position the audio source in 3D space
|
|
audioEntity.position = SIMD3(2, 0, -1)
|
|
```
|
|
|
|
### Ambient Audio
|
|
|
|
```swift
|
|
entity.components[AmbientAudioComponent.self] = AmbientAudioComponent()
|
|
entity.playAudio(backgroundMusic)
|
|
```
|
|
|
|
---
|
|
|
|
## 12. Performance
|
|
|
|
### Entity Count
|
|
|
|
- **Under 100 entities**: No concerns
|
|
- **100-1000 entities**: Monitor with RealityKit debugger
|
|
- **1000+ entities**: Use instancing and LOD strategies
|
|
|
|
### Instancing
|
|
|
|
```swift
|
|
// Share mesh and material across many entities
|
|
let sharedMesh = MeshResource.generateSphere(radius: 0.01)
|
|
let sharedMaterial = SimpleMaterial(color: .white, isMetallic: false)
|
|
|
|
for i in 0..<1000 {
|
|
let entity = ModelEntity(mesh: sharedMesh, materials: [sharedMaterial])
|
|
entity.position = randomPosition()
|
|
parent.addChild(entity)
|
|
}
|
|
```
|
|
|
|
RealityKit automatically batches entities with identical mesh and material resources.
|
|
|
|
### Component Churn
|
|
|
|
**Anti-pattern**: Creating and replacing components every frame.
|
|
|
|
```swift
|
|
// BAD — component allocation every frame
|
|
func update(context: SceneUpdateContext) {
|
|
for entity in context.entities(matching: query, updatingSystemWhen: .rendering) {
|
|
entity.components[ModelComponent.self] = ModelComponent(
|
|
mesh: .generateBox(size: 0.1),
|
|
materials: [newMaterial] // New allocation every frame
|
|
)
|
|
}
|
|
}
|
|
|
|
// GOOD — modify existing component
|
|
func update(context: SceneUpdateContext) {
|
|
for entity in context.entities(matching: query, updatingSystemWhen: .rendering) {
|
|
// Only update when actually needed
|
|
if needsUpdate {
|
|
var model = entity.components[ModelComponent.self]!
|
|
model.materials = [cachedMaterial]
|
|
entity.components[ModelComponent.self] = model
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
### Collision Shape Optimization
|
|
|
|
- Use simple shapes (box, sphere, capsule) instead of mesh-based collision
|
|
- `generateCollisionShapes(recursive: true)` is convenient but expensive
|
|
- For static geometry, generate shapes once during setup
|
|
|
|
### Profiling
|
|
|
|
Use Xcode's RealityKit debugger:
|
|
- **Entity Inspector**: View entity hierarchy and components
|
|
- **Statistics Overlay**: Entity count, draw calls, triangle count
|
|
- **Physics Visualization**: Show collision shapes
|
|
|
|
---
|
|
|
|
## 13. Multiplayer
|
|
|
|
### Synchronization Basics
|
|
|
|
```swift
|
|
// Components sync automatically if they conform to Codable
|
|
struct ScoreComponent: Component, Codable {
|
|
var points: Int
|
|
}
|
|
|
|
// SynchronizationComponent controls what syncs
|
|
entity.components[SynchronizationComponent.self] = SynchronizationComponent()
|
|
```
|
|
|
|
### MultipeerConnectivityService
|
|
|
|
```swift
|
|
let service = try MultipeerConnectivityService(session: mcSession)
|
|
// Entities with SynchronizationComponent auto-sync across peers
|
|
```
|
|
|
|
### Ownership
|
|
|
|
- Only the **owner** of an entity can modify it
|
|
- Request ownership before modifying shared entities
|
|
- Non-Codable component data does not sync
|
|
|
|
---
|
|
|
|
## 14. Anti-Patterns
|
|
|
|
### Anti-Pattern 1: UIKit-Style Thinking in ECS
|
|
|
|
**Time cost**: Hours of frustration from fighting the architecture
|
|
|
|
```swift
|
|
// BAD — subclassing Entity for behavior
|
|
class PlayerEntity: Entity {
|
|
func takeDamage(_ amount: Int) { /* logic in entity */ }
|
|
}
|
|
|
|
// GOOD — component holds data, system has logic
|
|
struct HealthComponent: Component { var hp: Int }
|
|
struct DamageSystem: System {
|
|
static let query = EntityQuery(where: .has(HealthComponent.self))
|
|
func update(context: SceneUpdateContext) {
|
|
// Process damage here
|
|
}
|
|
}
|
|
```
|
|
|
|
### Anti-Pattern 2: Monolithic Entities
|
|
|
|
**Time cost**: Untestable, inflexible architecture
|
|
|
|
Don't put all game logic in one entity type. Split into components that can be mixed and matched.
|
|
|
|
### Anti-Pattern 3: Frame-Based Updates Without Systems
|
|
|
|
**Time cost**: Missed frame updates, inconsistent behavior
|
|
|
|
```swift
|
|
// BAD — timer-based updates
|
|
Timer.scheduledTimer(withTimeInterval: 1/60, repeats: true) { _ in
|
|
entity.position.x += 0.01
|
|
}
|
|
|
|
// GOOD — System update
|
|
struct MovementSystem: System {
|
|
static let query = EntityQuery(where: .has(VelocityComponent.self))
|
|
func update(context: SceneUpdateContext) {
|
|
for entity in context.entities(matching: Self.query,
|
|
updatingSystemWhen: .rendering) {
|
|
let velocity = entity.components[VelocityComponent.self]!
|
|
entity.position += velocity.value * Float(context.deltaTime)
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
### Anti-Pattern 4: Not Generating Collision Shapes for Interactive Entities
|
|
|
|
**Time cost**: 15-30 min debugging "why taps don't work"
|
|
|
|
Gestures require `CollisionComponent`. If an entity has `InputTargetComponent` (visionOS) or `ManipulationComponent` but no `CollisionComponent`, gestures will never fire.
|
|
|
|
### Anti-Pattern 5: Storing Entity References in Systems
|
|
|
|
**Time cost**: Crashes from stale references
|
|
|
|
```swift
|
|
// BAD — entity might be removed between frames
|
|
struct BadSystem: System {
|
|
var playerEntity: Entity? // Stale reference risk
|
|
|
|
func update(context: SceneUpdateContext) {
|
|
playerEntity?.position.x += 0.1 // May crash
|
|
}
|
|
}
|
|
|
|
// GOOD — query each frame
|
|
struct GoodSystem: System {
|
|
static let query = EntityQuery(where: .has(PlayerComponent.self))
|
|
|
|
func update(context: SceneUpdateContext) {
|
|
for entity in context.entities(matching: Self.query,
|
|
updatingSystemWhen: .rendering) {
|
|
entity.position.x += Float(context.deltaTime)
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## 15. Code Review Checklist
|
|
|
|
- [ ] Custom components registered via `registerComponent()` before use
|
|
- [ ] Systems registered via `registerSystem()` before scene loads
|
|
- [ ] Components are value types (structs), not classes
|
|
- [ ] Read-modify-write pattern used for component updates
|
|
- [ ] Interactive entities have `CollisionComponent`
|
|
- [ ] visionOS interactive entities have `InputTargetComponent`
|
|
- [ ] Collision shapes are simple (box/sphere/capsule) where possible
|
|
- [ ] No entity references stored across frames in Systems
|
|
- [ ] Mesh and material resources shared across identical entities
|
|
- [ ] Component updates only occur when values actually change
|
|
- [ ] USD/USDZ format used for 3D assets (not .scn)
|
|
- [ ] Async loading used for all model/scene loading
|
|
- [ ] `[weak self]` in closure-based subscriptions if retaining view/controller
|
|
|
|
---
|
|
|
|
## 16. Pressure Scenarios
|
|
|
|
### Scenario 1: "ECS Is Overkill for Our Simple App"
|
|
|
|
**Pressure**: Team wants to avoid learning ECS, just needs one 3D model displayed
|
|
|
|
**Wrong approach**: Skip ECS, jam all logic into RealityView closures.
|
|
|
|
**Correct approach**: Even simple apps benefit from ECS. A single `ModelEntity` in a `RealityView` is already using ECS — you're just not adding custom components yet. Start simple, add components as complexity grows.
|
|
|
|
**Push-back template**: "We're already using ECS — Entity and ModelComponent. The pattern scales. Adding a custom component when we need behavior is one struct definition, not an architecture change."
|
|
|
|
### Scenario 2: "Just Use SceneKit, We Know It"
|
|
|
|
**Pressure**: Team has SceneKit experience, RealityKit is unfamiliar
|
|
|
|
**Wrong approach**: Build new features in SceneKit.
|
|
|
|
**Correct approach**: SceneKit is soft-deprecated. New features won't be added. Invest in RealityKit now — the ECS concepts transfer to other game engines (Unity, Unreal, Bevy) if needed.
|
|
|
|
**Push-back template**: "SceneKit is in maintenance mode — no new features, only security patches. Every line of SceneKit we write is migration debt. RealityKit's concepts (Entity, Component, System) are industry-standard ECS."
|
|
|
|
### Scenario 3: "Make It Work Without Collision Shapes"
|
|
|
|
**Pressure**: Deadline, collision shape setup seems complex
|
|
|
|
**Wrong approach**: Skip collision shapes, use position-based proximity detection.
|
|
|
|
**Correct approach**: `entity.generateCollisionShapes(recursive: true)` takes one line. Without it, gestures won't work and physics won't collide. The "shortcut" creates more debugging time than it saves.
|
|
|
|
**Push-back template**: "Collision shapes are required for gestures and physics. It's one line: `entity.generateCollisionShapes(recursive: true)`. Skipping it means gestures silently fail — a harder bug to diagnose."
|
|
|
|
---
|
|
|
|
## Resources
|
|
|
|
**WWDC**: 2019-603, 2019-605, 2021-10074, 2022-10074, 2023-10080, 2023-10081, 2024-10103, 2024-10153
|
|
|
|
**Docs**: /realitykit, /realitykit/entity, /realitykit/realityview, /realitykit/modelentity, /realitykit/anchorentity, /realitykit/component
|
|
|
|
**Skills**: axiom-realitykit-ref, axiom-realitykit-diag, axiom-scenekit, axiom-scenekit-ref
|