Add scan flow MVP and local Axiom skill workspace
This snapshot establishes the camera-to-result recognition flow and related tests while checking in the project skill/docs assets required for the configured local tooling.
This commit is contained in:
Matthias
929
.claude/skills/axiom-realitykit/SKILL.md
Normal file
929
.claude/skills/axiom-realitykit/SKILL.md
Normal file
@@ -0,0 +1,929 @@
|
||||
---
|
||||
name: axiom-realitykit
|
||||
description: Use when building 3D content, AR experiences, or spatial computing with RealityKit. Covers ECS architecture, SwiftUI integration, RealityView, AR anchors, materials, physics, interaction, multiplayer, performance.
|
||||
license: MIT
|
||||
metadata:
|
||||
version: "1.0.0"
|
||||
---
|
||||
|
||||
# RealityKit Development Guide
|
||||
|
||||
**Purpose**: Build 3D content, AR experiences, and spatial computing apps using RealityKit's Entity-Component-System architecture
|
||||
**iOS Version**: iOS 13+ (base), iOS 18+ (RealityView on iOS), visionOS 1.0+
|
||||
**Xcode**: Xcode 15+
|
||||
|
||||
## When to Use This Skill
|
||||
|
||||
Use this skill when:
|
||||
- Building any 3D experience (AR, games, visualization, spatial computing)
|
||||
- Creating SwiftUI apps with 3D content (RealityView, Model3D)
|
||||
- Implementing AR with anchors (world, image, face, body tracking)
|
||||
- Working with Entity-Component-System (ECS) architecture
|
||||
- Setting up physics, collisions, or spatial interactions
|
||||
- Building multiplayer or shared AR experiences
|
||||
- Migrating from SceneKit to RealityKit
|
||||
- Targeting visionOS
|
||||
|
||||
Do NOT use this skill for:
|
||||
- SceneKit maintenance (use `axiom-scenekit`)
|
||||
- 2D games (use `axiom-spritekit`)
|
||||
- Metal shader programming (use `axiom-metal-migration-ref`)
|
||||
- Pure GPU compute (use Metal directly)
|
||||
|
||||
---
|
||||
|
||||
## 1. Mental Model: ECS vs Scene Graph
|
||||
|
||||
### Scene Graph (SceneKit)
|
||||
|
||||
In SceneKit, nodes own their properties. A node IS a renderable, collidable, animated thing.
|
||||
|
||||
### Entity-Component-System (RealityKit)
|
||||
|
||||
In RealityKit, entities are **empty containers**. Components add data. Systems process that data.
|
||||
|
||||
```
|
||||
Entity (identity + hierarchy)
|
||||
├── TransformComponent (position, rotation, scale)
|
||||
├── ModelComponent (mesh + materials)
|
||||
├── CollisionComponent (collision shapes)
|
||||
├── PhysicsBodyComponent (mass, mode)
|
||||
└── [YourCustomComponent] (game-specific data)
|
||||
|
||||
System (processes entities with specific components each frame)
|
||||
```
|
||||
|
||||
**Why ECS matters**:
|
||||
- **Composition over inheritance**: Combine any components on any entity
|
||||
- **Data-oriented**: Systems process arrays of components efficiently
|
||||
- **Decoupled logic**: Systems don't know about each other
|
||||
- **Testable**: Components are pure data, Systems are pure logic
|
||||
|
||||
### The ECS Mental Shift
|
||||
|
||||
| Scene Graph Thinking | ECS Thinking |
|
||||
|---------------------|--------------|
|
||||
| "The player node moves" | "The movement system processes entities with MovementComponent" |
|
||||
| "Add a method to the node subclass" | "Add a component, create a system" |
|
||||
| "Override `update(_:)` in the node" | "Register a System that queries for components" |
|
||||
| "The node knows its health" | "HealthComponent holds data, DamageSystem processes it" |
|
||||
|
||||
---
|
||||
|
||||
## 2. Entity Hierarchy
|
||||
|
||||
### Creating Entities
|
||||
|
||||
```swift
|
||||
// Empty entity
|
||||
let entity = Entity()
|
||||
entity.name = "player"
|
||||
|
||||
// Entity with components
|
||||
let entity = Entity()
|
||||
entity.components[ModelComponent.self] = ModelComponent(
|
||||
mesh: .generateBox(size: 0.1),
|
||||
materials: [SimpleMaterial(color: .blue, isMetallic: false)]
|
||||
)
|
||||
|
||||
// ModelEntity convenience (has ModelComponent built in)
|
||||
let box = ModelEntity(
|
||||
mesh: .generateBox(size: 0.1),
|
||||
materials: [SimpleMaterial(color: .red, isMetallic: true)]
|
||||
)
|
||||
```
|
||||
|
||||
### Hierarchy Management
|
||||
|
||||
```swift
|
||||
// Parent-child
|
||||
parent.addChild(child)
|
||||
child.removeFromParent()
|
||||
|
||||
// Find entities
|
||||
let found = root.findEntity(named: "player")
|
||||
|
||||
// Enumerate
|
||||
for child in entity.children {
|
||||
// Process children
|
||||
}
|
||||
|
||||
// Clone
|
||||
let clone = entity.clone(recursive: true)
|
||||
```
|
||||
|
||||
### Transform
|
||||
|
||||
```swift
|
||||
// Local transform (relative to parent)
|
||||
entity.position = SIMD3<Float>(0, 1, 0)
|
||||
entity.orientation = simd_quatf(angle: .pi / 4, axis: SIMD3(0, 1, 0))
|
||||
entity.scale = SIMD3<Float>(repeating: 2.0)
|
||||
|
||||
// World-space queries
|
||||
let worldPos = entity.position(relativeTo: nil)
|
||||
let worldTransform = entity.transform(relativeTo: nil)
|
||||
|
||||
// Set world-space transform
|
||||
entity.setPosition(SIMD3(1, 0, 0), relativeTo: nil)
|
||||
|
||||
// Look at a point
|
||||
entity.look(at: targetPosition, from: entity.position, relativeTo: nil)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. Components
|
||||
|
||||
### Built-in Components
|
||||
|
||||
| Component | Purpose |
|
||||
|-----------|---------|
|
||||
| `Transform` | Position, rotation, scale |
|
||||
| `ModelComponent` | Mesh geometry + materials |
|
||||
| `CollisionComponent` | Collision shapes for physics and interaction |
|
||||
| `PhysicsBodyComponent` | Mass, physics mode (dynamic/static/kinematic) |
|
||||
| `PhysicsMotionComponent` | Linear and angular velocity |
|
||||
| `AnchoringComponent` | AR anchor attachment |
|
||||
| `SynchronizationComponent` | Multiplayer sync |
|
||||
| `PerspectiveCameraComponent` | Camera settings |
|
||||
| `DirectionalLightComponent` | Directional light |
|
||||
| `PointLightComponent` | Point light |
|
||||
| `SpotLightComponent` | Spot light |
|
||||
| `CharacterControllerComponent` | Character physics controller |
|
||||
| `AudioMixGroupsComponent` | Audio mixing |
|
||||
| `SpatialAudioComponent` | 3D positional audio |
|
||||
| `AmbientAudioComponent` | Non-positional audio |
|
||||
| `ChannelAudioComponent` | Multi-channel audio |
|
||||
| `OpacityComponent` | Entity transparency |
|
||||
| `GroundingShadowComponent` | Contact shadow |
|
||||
| `InputTargetComponent` | Gesture input (visionOS) |
|
||||
| `HoverEffectComponent` | Hover highlight (visionOS) |
|
||||
| `AccessibilityComponent` | VoiceOver support |
|
||||
|
||||
### Custom Components
|
||||
|
||||
```swift
|
||||
struct HealthComponent: Component {
|
||||
var current: Int
|
||||
var maximum: Int
|
||||
|
||||
var percentage: Float {
|
||||
Float(current) / Float(maximum)
|
||||
}
|
||||
}
|
||||
|
||||
// Register before use (typically in app init)
|
||||
HealthComponent.registerComponent()
|
||||
|
||||
// Attach to entity
|
||||
entity.components[HealthComponent.self] = HealthComponent(current: 100, maximum: 100)
|
||||
|
||||
// Read
|
||||
if let health = entity.components[HealthComponent.self] {
|
||||
print(health.current)
|
||||
}
|
||||
|
||||
// Modify
|
||||
entity.components[HealthComponent.self]?.current -= 10
|
||||
```
|
||||
|
||||
### Component Lifecycle
|
||||
|
||||
Components are value types (structs). When you read a component, modify it, and write it back, you're replacing the entire component:
|
||||
|
||||
```swift
|
||||
// Read-modify-write pattern
|
||||
var health = entity.components[HealthComponent.self]!
|
||||
health.current -= damage
|
||||
entity.components[HealthComponent.self] = health
|
||||
```
|
||||
|
||||
**Anti-pattern**: Holding a reference to a component and expecting mutations to propagate. Components are copied on read.
|
||||
|
||||
---
|
||||
|
||||
## 4. Systems
|
||||
|
||||
### System Protocol
|
||||
|
||||
```swift
|
||||
struct DamageSystem: System {
|
||||
// Define which components this system needs
|
||||
static let query = EntityQuery(where: .has(HealthComponent.self))
|
||||
|
||||
init(scene: RealityKit.Scene) {
|
||||
// One-time setup
|
||||
}
|
||||
|
||||
func update(context: SceneUpdateContext) {
|
||||
for entity in context.entities(matching: Self.query,
|
||||
updatingSystemWhen: .rendering) {
|
||||
var health = entity.components[HealthComponent.self]!
|
||||
if health.current <= 0 {
|
||||
entity.removeFromParent()
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Register system
|
||||
DamageSystem.registerSystem()
|
||||
```
|
||||
|
||||
### System Best Practices
|
||||
|
||||
- **One responsibility per system**: MovementSystem, DamageSystem, RenderingSystem — not GameLogicSystem
|
||||
- **Query filtering**: Use precise queries to avoid processing irrelevant entities
|
||||
- **Order matters**: Systems run in registration order. Register dependencies first.
|
||||
- **Avoid storing entity references**: Query each frame instead. Entity references can become stale.
|
||||
|
||||
### Event Handling
|
||||
|
||||
```swift
|
||||
// Subscribe to collision events
|
||||
scene.subscribe(to: CollisionEvents.Began.self) { event in
|
||||
let entityA = event.entityA
|
||||
let entityB = event.entityB
|
||||
// Handle collision
|
||||
}
|
||||
|
||||
// Subscribe to scene update
|
||||
scene.subscribe(to: SceneEvents.Update.self) { event in
|
||||
let deltaTime = event.deltaTime
|
||||
// Per-frame logic
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. SwiftUI Integration
|
||||
|
||||
### RealityView (iOS 18+, visionOS 1.0+)
|
||||
|
||||
```swift
|
||||
struct ContentView: View {
|
||||
var body: some View {
|
||||
RealityView { content in
|
||||
// make closure — called once
|
||||
let box = ModelEntity(
|
||||
mesh: .generateBox(size: 0.1),
|
||||
materials: [SimpleMaterial(color: .blue, isMetallic: false)]
|
||||
)
|
||||
content.add(box)
|
||||
|
||||
} update: { content in
|
||||
// update closure — called when SwiftUI state changes
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### RealityView with Camera (iOS)
|
||||
|
||||
On iOS, `RealityView` provides a camera content parameter for configuring the AR or virtual camera:
|
||||
|
||||
```swift
|
||||
RealityView { content, attachments in
|
||||
// Load 3D content
|
||||
if let model = try? await ModelEntity(named: "scene") {
|
||||
content.add(model)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Loading Content Asynchronously
|
||||
|
||||
```swift
|
||||
RealityView { content in
|
||||
// Load from bundle
|
||||
if let entity = try? await Entity(named: "MyScene", in: .main) {
|
||||
content.add(entity)
|
||||
}
|
||||
|
||||
// Load from URL
|
||||
if let entity = try? await Entity(contentsOf: modelURL) {
|
||||
content.add(entity)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Model3D (Simple Display)
|
||||
|
||||
```swift
|
||||
// Simple 3D model display (no interaction)
|
||||
Model3D(named: "toy_robot") { model in
|
||||
model
|
||||
.resizable()
|
||||
.scaledToFit()
|
||||
} placeholder: {
|
||||
ProgressView()
|
||||
}
|
||||
```
|
||||
|
||||
### SwiftUI Attachments (visionOS)
|
||||
|
||||
```swift
|
||||
RealityView { content, attachments in
|
||||
let entity = ModelEntity(mesh: .generateSphere(radius: 0.1))
|
||||
content.add(entity)
|
||||
|
||||
if let label = attachments.entity(for: "priceTag") {
|
||||
label.position = SIMD3(0, 0.15, 0)
|
||||
entity.addChild(label)
|
||||
}
|
||||
} attachments: {
|
||||
Attachment(id: "priceTag") {
|
||||
Text("$9.99")
|
||||
.padding()
|
||||
.glassBackgroundEffect()
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### State Binding Pattern
|
||||
|
||||
```swift
|
||||
struct GameView: View {
|
||||
@State private var score = 0
|
||||
|
||||
var body: some View {
|
||||
VStack {
|
||||
Text("Score: \(score)")
|
||||
|
||||
RealityView { content in
|
||||
let scene = try! await Entity(named: "GameScene")
|
||||
content.add(scene)
|
||||
} update: { content in
|
||||
// React to state changes
|
||||
// Note: update is called when SwiftUI state changes,
|
||||
// not every frame. Use Systems for per-frame logic.
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. AR on iOS
|
||||
|
||||
### AnchorEntity
|
||||
|
||||
```swift
|
||||
// Horizontal plane
|
||||
let anchor = AnchorEntity(.plane(.horizontal, classification: .table,
|
||||
minimumBounds: SIMD2(0.2, 0.2)))
|
||||
|
||||
// Vertical plane
|
||||
let anchor = AnchorEntity(.plane(.vertical, classification: .wall,
|
||||
minimumBounds: SIMD2(0.5, 0.5)))
|
||||
|
||||
// World position
|
||||
let anchor = AnchorEntity(world: SIMD3<Float>(0, 0, -1))
|
||||
|
||||
// Image anchor
|
||||
let anchor = AnchorEntity(.image(group: "AR Resources", name: "poster"))
|
||||
|
||||
// Face anchor (front camera)
|
||||
let anchor = AnchorEntity(.face)
|
||||
|
||||
// Body anchor
|
||||
let anchor = AnchorEntity(.body)
|
||||
```
|
||||
|
||||
### SpatialTrackingSession (iOS 18+)
|
||||
|
||||
```swift
|
||||
let session = SpatialTrackingSession()
|
||||
let configuration = SpatialTrackingSession.Configuration(tracking: [.plane, .object])
|
||||
let result = await session.run(configuration)
|
||||
|
||||
if let notSupported = result {
|
||||
// Handle unsupported tracking on this device
|
||||
for denied in notSupported.deniedTrackingModes {
|
||||
print("Not supported: \(denied)")
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### AR Best Practices
|
||||
|
||||
- Anchor entities to detected surfaces rather than world positions for stability
|
||||
- Use plane classification (`.table`, `.floor`, `.wall`) to place content appropriately
|
||||
- Start with horizontal plane detection — it's the most reliable
|
||||
- Test on real devices; simulator AR is limited
|
||||
- Provide visual feedback during surface detection (coaching overlay)
|
||||
|
||||
---
|
||||
|
||||
## 7. Interaction
|
||||
|
||||
### ManipulationComponent (iOS, visionOS)
|
||||
|
||||
```swift
|
||||
// Enable drag, rotate, scale gestures
|
||||
entity.components[ManipulationComponent.self] = ManipulationComponent(
|
||||
allowedModes: .all // .translate, .rotate, .scale
|
||||
)
|
||||
|
||||
// Also requires CollisionComponent for hit testing
|
||||
entity.generateCollisionShapes(recursive: true)
|
||||
```
|
||||
|
||||
### InputTargetComponent (visionOS)
|
||||
|
||||
```swift
|
||||
// Required for visionOS gesture input
|
||||
entity.components[InputTargetComponent.self] = InputTargetComponent()
|
||||
entity.components[CollisionComponent.self] = CollisionComponent(
|
||||
shapes: [.generateBox(size: SIMD3(0.1, 0.1, 0.1))]
|
||||
)
|
||||
```
|
||||
|
||||
### Gesture Integration with SwiftUI
|
||||
|
||||
```swift
|
||||
RealityView { content in
|
||||
let entity = ModelEntity(mesh: .generateBox(size: 0.1))
|
||||
entity.generateCollisionShapes(recursive: true)
|
||||
entity.components.set(InputTargetComponent())
|
||||
content.add(entity)
|
||||
}
|
||||
.gesture(
|
||||
TapGesture()
|
||||
.targetedToAnyEntity()
|
||||
.onEnded { value in
|
||||
let tappedEntity = value.entity
|
||||
// Handle tap
|
||||
}
|
||||
)
|
||||
.gesture(
|
||||
DragGesture()
|
||||
.targetedToAnyEntity()
|
||||
.onChanged { value in
|
||||
value.entity.position = value.convert(value.location3D,
|
||||
from: .local, to: .scene)
|
||||
}
|
||||
)
|
||||
```
|
||||
|
||||
### Hit Testing
|
||||
|
||||
```swift
|
||||
// Ray-cast from screen point
|
||||
if let result = arView.raycast(from: screenPoint,
|
||||
allowing: .estimatedPlane,
|
||||
alignment: .horizontal).first {
|
||||
let worldPosition = result.worldTransform.columns.3
|
||||
// Place entity at worldPosition
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 8. Materials and Rendering
|
||||
|
||||
### Material Types
|
||||
|
||||
| Material | Purpose | Customization |
|
||||
|----------|---------|---------------|
|
||||
| `SimpleMaterial` | Solid color or texture | Color, metallic, roughness |
|
||||
| `PhysicallyBasedMaterial` | Full PBR | All PBR maps (base color, normal, metallic, roughness, AO, emissive) |
|
||||
| `UnlitMaterial` | No lighting response | Color or texture, always fully lit |
|
||||
| `OcclusionMaterial` | Invisible but occludes | AR content hiding behind real objects |
|
||||
| `VideoMaterial` | Video playback on surface | AVPlayer-driven |
|
||||
| `ShaderGraphMaterial` | Custom shader graph | Reality Composer Pro |
|
||||
| `CustomMaterial` | Metal shader functions | Full Metal control |
|
||||
|
||||
### PhysicallyBasedMaterial
|
||||
|
||||
```swift
|
||||
var material = PhysicallyBasedMaterial()
|
||||
material.baseColor = .init(tint: .white,
|
||||
texture: .init(try! .load(named: "albedo")))
|
||||
material.metallic = .init(floatLiteral: 0.0)
|
||||
material.roughness = .init(floatLiteral: 0.5)
|
||||
material.normal = .init(texture: .init(try! .load(named: "normal")))
|
||||
material.ambientOcclusion = .init(texture: .init(try! .load(named: "ao")))
|
||||
material.emissiveColor = .init(color: .blue)
|
||||
material.emissiveIntensity = 2.0
|
||||
|
||||
let entity = ModelEntity(
|
||||
mesh: .generateSphere(radius: 0.1),
|
||||
materials: [material]
|
||||
)
|
||||
```
|
||||
|
||||
### OcclusionMaterial (AR)
|
||||
|
||||
```swift
|
||||
// Invisible plane that hides 3D content behind it
|
||||
let occluder = ModelEntity(
|
||||
mesh: .generatePlane(width: 1, depth: 1),
|
||||
materials: [OcclusionMaterial()]
|
||||
)
|
||||
occluder.position = SIMD3(0, 0, 0)
|
||||
anchor.addChild(occluder)
|
||||
```
|
||||
|
||||
### Environment Lighting
|
||||
|
||||
```swift
|
||||
// Image-based lighting
|
||||
if let resource = try? await EnvironmentResource(named: "studio_lighting") {
|
||||
// Apply via RealityView content
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 9. Physics and Collision
|
||||
|
||||
### Collision Shapes
|
||||
|
||||
```swift
|
||||
// Generate from mesh (accurate but expensive)
|
||||
entity.generateCollisionShapes(recursive: true)
|
||||
|
||||
// Manual shapes (prefer for performance)
|
||||
entity.components[CollisionComponent.self] = CollisionComponent(
|
||||
shapes: [
|
||||
.generateBox(size: SIMD3(0.1, 0.2, 0.1)), // Box
|
||||
.generateSphere(radius: 0.1), // Sphere
|
||||
.generateCapsule(height: 0.3, radius: 0.05) // Capsule
|
||||
]
|
||||
)
|
||||
```
|
||||
|
||||
### Physics Body
|
||||
|
||||
```swift
|
||||
// Dynamic — physics simulation controls movement
|
||||
entity.components[PhysicsBodyComponent.self] = PhysicsBodyComponent(
|
||||
massProperties: .init(mass: 1.0),
|
||||
material: .generate(staticFriction: 0.5,
|
||||
dynamicFriction: 0.3,
|
||||
restitution: 0.4),
|
||||
mode: .dynamic
|
||||
)
|
||||
|
||||
// Static — immovable collision surface
|
||||
ground.components[PhysicsBodyComponent.self] = PhysicsBodyComponent(
|
||||
mode: .static
|
||||
)
|
||||
|
||||
// Kinematic — code-controlled, participates in collisions
|
||||
platform.components[PhysicsBodyComponent.self] = PhysicsBodyComponent(
|
||||
mode: .kinematic
|
||||
)
|
||||
```
|
||||
|
||||
### Collision Groups and Filters
|
||||
|
||||
```swift
|
||||
// Define groups
|
||||
let playerGroup = CollisionGroup(rawValue: 1 << 0)
|
||||
let enemyGroup = CollisionGroup(rawValue: 1 << 1)
|
||||
let bulletGroup = CollisionGroup(rawValue: 1 << 2)
|
||||
|
||||
// Filter: player collides with enemies and bullets
|
||||
entity.components[CollisionComponent.self] = CollisionComponent(
|
||||
shapes: [.generateSphere(radius: 0.1)],
|
||||
filter: CollisionFilter(
|
||||
group: playerGroup,
|
||||
mask: enemyGroup | bulletGroup
|
||||
)
|
||||
)
|
||||
```
|
||||
|
||||
### Collision Events
|
||||
|
||||
```swift
|
||||
// Subscribe in RealityView make closure or System
|
||||
scene.subscribe(to: CollisionEvents.Began.self, on: playerEntity) { event in
|
||||
let otherEntity = event.entityA == playerEntity ? event.entityB : event.entityA
|
||||
handleCollision(with: otherEntity)
|
||||
}
|
||||
```
|
||||
|
||||
### Applying Forces
|
||||
|
||||
```swift
|
||||
if var motion = entity.components[PhysicsMotionComponent.self] {
|
||||
motion.linearVelocity = SIMD3(0, 5, 0) // Impulse up
|
||||
entity.components[PhysicsMotionComponent.self] = motion
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 10. Animation
|
||||
|
||||
### Transform Animation
|
||||
|
||||
```swift
|
||||
// Animate to position over duration
|
||||
entity.move(
|
||||
to: Transform(
|
||||
scale: SIMD3(repeating: 1.5),
|
||||
rotation: simd_quatf(angle: .pi, axis: SIMD3(0, 1, 0)),
|
||||
translation: SIMD3(0, 2, 0)
|
||||
),
|
||||
relativeTo: entity.parent,
|
||||
duration: 2.0,
|
||||
timingFunction: .easeInOut
|
||||
)
|
||||
```
|
||||
|
||||
### Playing USD Animations
|
||||
|
||||
```swift
|
||||
if let entity = try? await Entity(named: "character") {
|
||||
// Play all available animations
|
||||
for animation in entity.availableAnimations {
|
||||
entity.playAnimation(animation.repeat())
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Animation Playback Control
|
||||
|
||||
```swift
|
||||
let controller = entity.playAnimation(animation)
|
||||
controller.pause()
|
||||
controller.resume()
|
||||
controller.speed = 2.0 // 2x playback speed
|
||||
controller.blendFactor = 0.5 // Blend with current state
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 11. Audio
|
||||
|
||||
### Spatial Audio
|
||||
|
||||
```swift
|
||||
// Load audio resource
|
||||
let resource = try! AudioFileResource.load(named: "engine.wav",
|
||||
configuration: .init(shouldLoop: true))
|
||||
|
||||
// Create entity with spatial audio
|
||||
let audioEntity = Entity()
|
||||
audioEntity.components[SpatialAudioComponent.self] = SpatialAudioComponent()
|
||||
let controller = audioEntity.playAudio(resource)
|
||||
|
||||
// Position the audio source in 3D space
|
||||
audioEntity.position = SIMD3(2, 0, -1)
|
||||
```
|
||||
|
||||
### Ambient Audio
|
||||
|
||||
```swift
|
||||
entity.components[AmbientAudioComponent.self] = AmbientAudioComponent()
|
||||
entity.playAudio(backgroundMusic)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 12. Performance
|
||||
|
||||
### Entity Count
|
||||
|
||||
- **Under 100 entities**: No concerns
|
||||
- **100-1000 entities**: Monitor with RealityKit debugger
|
||||
- **1000+ entities**: Use instancing and LOD strategies
|
||||
|
||||
### Instancing
|
||||
|
||||
```swift
|
||||
// Share mesh and material across many entities
|
||||
let sharedMesh = MeshResource.generateSphere(radius: 0.01)
|
||||
let sharedMaterial = SimpleMaterial(color: .white, isMetallic: false)
|
||||
|
||||
for i in 0..<1000 {
|
||||
let entity = ModelEntity(mesh: sharedMesh, materials: [sharedMaterial])
|
||||
entity.position = randomPosition()
|
||||
parent.addChild(entity)
|
||||
}
|
||||
```
|
||||
|
||||
RealityKit automatically batches entities with identical mesh and material resources.
|
||||
|
||||
### Component Churn
|
||||
|
||||
**Anti-pattern**: Creating and replacing components every frame.
|
||||
|
||||
```swift
|
||||
// BAD — component allocation every frame
|
||||
func update(context: SceneUpdateContext) {
|
||||
for entity in context.entities(matching: query, updatingSystemWhen: .rendering) {
|
||||
entity.components[ModelComponent.self] = ModelComponent(
|
||||
mesh: .generateBox(size: 0.1),
|
||||
materials: [newMaterial] // New allocation every frame
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
// GOOD — modify existing component
|
||||
func update(context: SceneUpdateContext) {
|
||||
for entity in context.entities(matching: query, updatingSystemWhen: .rendering) {
|
||||
// Only update when actually needed
|
||||
if needsUpdate {
|
||||
var model = entity.components[ModelComponent.self]!
|
||||
model.materials = [cachedMaterial]
|
||||
entity.components[ModelComponent.self] = model
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Collision Shape Optimization
|
||||
|
||||
- Use simple shapes (box, sphere, capsule) instead of mesh-based collision
|
||||
- `generateCollisionShapes(recursive: true)` is convenient but expensive
|
||||
- For static geometry, generate shapes once during setup
|
||||
|
||||
### Profiling
|
||||
|
||||
Use Xcode's RealityKit debugger:
|
||||
- **Entity Inspector**: View entity hierarchy and components
|
||||
- **Statistics Overlay**: Entity count, draw calls, triangle count
|
||||
- **Physics Visualization**: Show collision shapes
|
||||
|
||||
---
|
||||
|
||||
## 13. Multiplayer
|
||||
|
||||
### Synchronization Basics
|
||||
|
||||
```swift
|
||||
// Components sync automatically if they conform to Codable
|
||||
struct ScoreComponent: Component, Codable {
|
||||
var points: Int
|
||||
}
|
||||
|
||||
// SynchronizationComponent controls what syncs
|
||||
entity.components[SynchronizationComponent.self] = SynchronizationComponent()
|
||||
```
|
||||
|
||||
### MultipeerConnectivityService
|
||||
|
||||
```swift
|
||||
let service = try MultipeerConnectivityService(session: mcSession)
|
||||
// Entities with SynchronizationComponent auto-sync across peers
|
||||
```
|
||||
|
||||
### Ownership
|
||||
|
||||
- Only the **owner** of an entity can modify it
|
||||
- Request ownership before modifying shared entities
|
||||
- Non-Codable component data does not sync
|
||||
|
||||
---
|
||||
|
||||
## 14. Anti-Patterns
|
||||
|
||||
### Anti-Pattern 1: UIKit-Style Thinking in ECS
|
||||
|
||||
**Time cost**: Hours of frustration from fighting the architecture
|
||||
|
||||
```swift
|
||||
// BAD — subclassing Entity for behavior
|
||||
class PlayerEntity: Entity {
|
||||
func takeDamage(_ amount: Int) { /* logic in entity */ }
|
||||
}
|
||||
|
||||
// GOOD — component holds data, system has logic
|
||||
struct HealthComponent: Component { var hp: Int }
|
||||
struct DamageSystem: System {
|
||||
static let query = EntityQuery(where: .has(HealthComponent.self))
|
||||
func update(context: SceneUpdateContext) {
|
||||
// Process damage here
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Anti-Pattern 2: Monolithic Entities
|
||||
|
||||
**Time cost**: Untestable, inflexible architecture
|
||||
|
||||
Don't put all game logic in one entity type. Split into components that can be mixed and matched.
|
||||
|
||||
### Anti-Pattern 3: Frame-Based Updates Without Systems
|
||||
|
||||
**Time cost**: Missed frame updates, inconsistent behavior
|
||||
|
||||
```swift
|
||||
// BAD — timer-based updates
|
||||
Timer.scheduledTimer(withTimeInterval: 1/60, repeats: true) { _ in
|
||||
entity.position.x += 0.01
|
||||
}
|
||||
|
||||
// GOOD — System update
|
||||
struct MovementSystem: System {
|
||||
static let query = EntityQuery(where: .has(VelocityComponent.self))
|
||||
func update(context: SceneUpdateContext) {
|
||||
for entity in context.entities(matching: Self.query,
|
||||
updatingSystemWhen: .rendering) {
|
||||
let velocity = entity.components[VelocityComponent.self]!
|
||||
entity.position += velocity.value * Float(context.deltaTime)
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Anti-Pattern 4: Not Generating Collision Shapes for Interactive Entities
|
||||
|
||||
**Time cost**: 15-30 min debugging "why taps don't work"
|
||||
|
||||
Gestures require `CollisionComponent`. If an entity has `InputTargetComponent` (visionOS) or `ManipulationComponent` but no `CollisionComponent`, gestures will never fire.
|
||||
|
||||
### Anti-Pattern 5: Storing Entity References in Systems
|
||||
|
||||
**Time cost**: Crashes from stale references
|
||||
|
||||
```swift
|
||||
// BAD — entity might be removed between frames
|
||||
struct BadSystem: System {
|
||||
var playerEntity: Entity? // Stale reference risk
|
||||
|
||||
func update(context: SceneUpdateContext) {
|
||||
playerEntity?.position.x += 0.1 // May crash
|
||||
}
|
||||
}
|
||||
|
||||
// GOOD — query each frame
|
||||
struct GoodSystem: System {
|
||||
static let query = EntityQuery(where: .has(PlayerComponent.self))
|
||||
|
||||
func update(context: SceneUpdateContext) {
|
||||
for entity in context.entities(matching: Self.query,
|
||||
updatingSystemWhen: .rendering) {
|
||||
entity.position.x += Float(context.deltaTime)
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 15. Code Review Checklist
|
||||
|
||||
- [ ] Custom components registered via `registerComponent()` before use
|
||||
- [ ] Systems registered via `registerSystem()` before scene loads
|
||||
- [ ] Components are value types (structs), not classes
|
||||
- [ ] Read-modify-write pattern used for component updates
|
||||
- [ ] Interactive entities have `CollisionComponent`
|
||||
- [ ] visionOS interactive entities have `InputTargetComponent`
|
||||
- [ ] Collision shapes are simple (box/sphere/capsule) where possible
|
||||
- [ ] No entity references stored across frames in Systems
|
||||
- [ ] Mesh and material resources shared across identical entities
|
||||
- [ ] Component updates only occur when values actually change
|
||||
- [ ] USD/USDZ format used for 3D assets (not .scn)
|
||||
- [ ] Async loading used for all model/scene loading
|
||||
- [ ] `[weak self]` in closure-based subscriptions if retaining view/controller
|
||||
|
||||
---
|
||||
|
||||
## 16. Pressure Scenarios
|
||||
|
||||
### Scenario 1: "ECS Is Overkill for Our Simple App"
|
||||
|
||||
**Pressure**: Team wants to avoid learning ECS, just needs one 3D model displayed
|
||||
|
||||
**Wrong approach**: Skip ECS, jam all logic into RealityView closures.
|
||||
|
||||
**Correct approach**: Even simple apps benefit from ECS. A single `ModelEntity` in a `RealityView` is already using ECS — you're just not adding custom components yet. Start simple, add components as complexity grows.
|
||||
|
||||
**Push-back template**: "We're already using ECS — Entity and ModelComponent. The pattern scales. Adding a custom component when we need behavior is one struct definition, not an architecture change."
|
||||
|
||||
### Scenario 2: "Just Use SceneKit, We Know It"
|
||||
|
||||
**Pressure**: Team has SceneKit experience, RealityKit is unfamiliar
|
||||
|
||||
**Wrong approach**: Build new features in SceneKit.
|
||||
|
||||
**Correct approach**: SceneKit is soft-deprecated. New features won't be added. Invest in RealityKit now — the ECS concepts transfer to other game engines (Unity, Unreal, Bevy) if needed.
|
||||
|
||||
**Push-back template**: "SceneKit is in maintenance mode — no new features, only security patches. Every line of SceneKit we write is migration debt. RealityKit's concepts (Entity, Component, System) are industry-standard ECS."
|
||||
|
||||
### Scenario 3: "Make It Work Without Collision Shapes"
|
||||
|
||||
**Pressure**: Deadline, collision shape setup seems complex
|
||||
|
||||
**Wrong approach**: Skip collision shapes, use position-based proximity detection.
|
||||
|
||||
**Correct approach**: `entity.generateCollisionShapes(recursive: true)` takes one line. Without it, gestures won't work and physics won't collide. The "shortcut" creates more debugging time than it saves.
|
||||
|
||||
**Push-back template**: "Collision shapes are required for gestures and physics. It's one line: `entity.generateCollisionShapes(recursive: true)`. Skipping it means gestures silently fail — a harder bug to diagnose."
|
||||
|
||||
---
|
||||
|
||||
## Resources
|
||||
|
||||
**WWDC**: 2019-603, 2019-605, 2021-10074, 2022-10074, 2023-10080, 2023-10081, 2024-10103, 2024-10153
|
||||
|
||||
**Docs**: /realitykit, /realitykit/entity, /realitykit/realityview, /realitykit/modelentity, /realitykit/anchorentity, /realitykit/component
|
||||
|
||||
**Skills**: axiom-realitykit-ref, axiom-realitykit-diag, axiom-scenekit, axiom-scenekit-ref
|
||||
Reference in New Issue
Block a user