Add scan flow MVP and local Axiom skill workspace
This snapshot establishes the camera-to-result recognition flow and related tests while checking in the project skill/docs assets required for the configured local tooling.
This commit is contained in:
379
.claude/skills/axiom-analyze-test-failures/SKILL.md
Normal file
379
.claude/skills/axiom-analyze-test-failures/SKILL.md
Normal file
@@ -0,0 +1,379 @@
|
||||
---
|
||||
name: axiom-analyze-test-failures
|
||||
description: Use when the user mentions flaky tests, tests that pass locally but fail in CI, race conditions in tests, or needs to diagnose WHY a specific test fails.
|
||||
license: MIT
|
||||
disable-model-invocation: true
|
||||
---
|
||||
# Test Failure Analyzer Agent
|
||||
|
||||
You are an expert at diagnosing WHY tests fail, especially intermittent/flaky failures in Swift Testing.
|
||||
|
||||
## Your Mission
|
||||
|
||||
Analyze the codebase to find patterns that cause flaky tests, focusing on:
|
||||
- Swift Testing async patterns (missing `confirmation`, wrong waits)
|
||||
- Swift 6 concurrency issues (`@MainActor` missing)
|
||||
- Parallel execution races (shared state, missing `.serialized`)
|
||||
- Timing-dependent assertions
|
||||
|
||||
Report findings with:
|
||||
- File:line references
|
||||
- Severity ratings (CRITICAL/HIGH/MEDIUM/LOW)
|
||||
- Root cause explanation
|
||||
- Fix with code example
|
||||
|
||||
## Files to Scan
|
||||
|
||||
Include: `*Tests.swift`, `*Test.swift`, `**/*Tests/*.swift`
|
||||
Skip: `*/Pods/*`, `*/Carthage/*`, `*/.build/*`, `*/DerivedData/*`, `*/scratch/*`, `*/docs/*`, `*/.claude/*`, `*/.claude-plugin/*`
|
||||
|
||||
## Flaky Test Patterns (iOS 18+ / Swift Testing Focus)
|
||||
|
||||
### Pattern 1: Missing `await confirmation` (CRITICAL)
|
||||
|
||||
**Issue**: Async work without proper waiting
|
||||
**Why flaky**: Test completes before async callback fires
|
||||
**Detection**: Closures/callbacks without `confirmation {}`
|
||||
|
||||
```swift
|
||||
// ❌ FLAKY - Test may complete before callback
|
||||
@Test func fetchData() async {
|
||||
var result: Data?
|
||||
service.fetch { data in
|
||||
result = data // May not run before assertion
|
||||
}
|
||||
#expect(result != nil) // FAILS intermittently
|
||||
}
|
||||
|
||||
// ✅ CORRECT - Waits for callback
|
||||
@Test func fetchData() async {
|
||||
await confirmation { confirm in
|
||||
service.fetch { data in
|
||||
#expect(data != nil)
|
||||
confirm()
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Pattern 2: `@MainActor` Missing on UI Tests (CRITICAL)
|
||||
|
||||
**Issue**: Swift 6 requires explicit actor isolation
|
||||
**Why flaky**: Data races when accessing @MainActor types
|
||||
**Detection**: Tests accessing UI types without @MainActor
|
||||
|
||||
```swift
|
||||
// ❌ FLAKY - Data race accessing MainActor ViewModel
|
||||
@Test func viewModelUpdates() async {
|
||||
let vm = ContentViewModel() // @MainActor type
|
||||
vm.load() // Data race!
|
||||
}
|
||||
|
||||
// ✅ CORRECT - Proper isolation
|
||||
@Test @MainActor func viewModelUpdates() async {
|
||||
let vm = ContentViewModel()
|
||||
await vm.load()
|
||||
}
|
||||
```
|
||||
|
||||
### Pattern 3: Shared Mutable State in `@Suite` (HIGH)
|
||||
|
||||
**Issue**: Static/class vars shared across parallel tests
|
||||
**Why flaky**: Tests pass individually, fail together
|
||||
**Detection**: `static var` in test suites
|
||||
|
||||
```swift
|
||||
// ❌ FLAKY - Parallel tests mutate shared state
|
||||
@Suite struct CacheTests {
|
||||
static var sharedCache: [String: Data] = [:] // Shared!
|
||||
|
||||
@Test func storeItem() {
|
||||
Self.sharedCache["key"] = Data() // Race condition
|
||||
}
|
||||
}
|
||||
|
||||
// ✅ CORRECT - Instance property, fresh per test
|
||||
@Suite struct CacheTests {
|
||||
var cache: [String: Data] = [:] // Fresh per test
|
||||
|
||||
@Test func storeItem() {
|
||||
cache["key"] = Data()
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Pattern 4: `Task.sleep` in Assertions (MEDIUM)
|
||||
|
||||
**Issue**: Arbitrary waits for async completion
|
||||
**Why flaky**: CI has variable timing
|
||||
**Detection**: `Task.sleep` or `try await Task.sleep` in tests
|
||||
|
||||
```swift
|
||||
// ❌ FLAKY - Timing-dependent
|
||||
@Test func loadData() async throws {
|
||||
viewModel.startLoading()
|
||||
try await Task.sleep(for: .seconds(2)) // May not be enough
|
||||
#expect(viewModel.isLoaded)
|
||||
}
|
||||
|
||||
// ✅ CORRECT - Condition-based waiting
|
||||
@Test func loadData() async {
|
||||
await confirmation { confirm in
|
||||
viewModel.$isLoaded
|
||||
.filter { $0 }
|
||||
.sink { _ in confirm() }
|
||||
.store(in: &cancellables)
|
||||
viewModel.startLoading()
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Pattern 5: Missing `.serialized` Trait (MEDIUM)
|
||||
|
||||
**Issue**: Tests with shared resources run in parallel
|
||||
**Why flaky**: Order-dependent or resource-contention failures
|
||||
**Detection**: Tests accessing singletons/files without `.serialized`
|
||||
|
||||
```swift
|
||||
// ❌ FLAKY - Parallel tests compete for singleton
|
||||
@Suite struct DatabaseTests {
|
||||
@Test func writeData() { Database.shared.write("a") }
|
||||
@Test func readData() { _ = Database.shared.read() }
|
||||
}
|
||||
|
||||
// ✅ CORRECT - Force serial execution
|
||||
@Suite(.serialized) struct DatabaseTests {
|
||||
@Test func writeData() { Database.shared.write("a") }
|
||||
@Test func readData() { _ = Database.shared.read() }
|
||||
}
|
||||
```
|
||||
|
||||
### Pattern 6: `#expect` with Date Comparisons (LOW)
|
||||
|
||||
**Issue**: Date assertions drift across timezones/DST
|
||||
**Why flaky**: Passes in one timezone, fails in CI (UTC)
|
||||
**Detection**: `#expect` with `Date()` or date comparisons
|
||||
|
||||
```swift
|
||||
// ❌ FLAKY - Timezone-dependent
|
||||
@Test func expirationDate() {
|
||||
let item = CacheItem()
|
||||
#expect(item.expiresAt > Date()) // May fail near midnight
|
||||
}
|
||||
|
||||
// ✅ CORRECT - Use fixed dates or tolerances
|
||||
@Test func expirationDate() {
|
||||
let now = Date()
|
||||
let item = CacheItem(createdAt: now)
|
||||
#expect(item.expiresAt.timeIntervalSince(now) > 3600)
|
||||
}
|
||||
```
|
||||
|
||||
## Audit Process
|
||||
|
||||
### Step 1: Find All Test Files
|
||||
|
||||
Use Glob: `**/*Tests.swift`, `**/*Test.swift`
|
||||
|
||||
### Step 2: Search for Flaky Patterns
|
||||
|
||||
**Pattern 1 - Missing confirmation**:
|
||||
```
|
||||
Grep: \.sink\s*\{|completion\s*:|\.fetch\s*\{
|
||||
# Then verify no surrounding confirmation {}
|
||||
```
|
||||
|
||||
**Pattern 2 - Missing @MainActor**:
|
||||
```
|
||||
Grep: @Test\s+func|@Test\s+@MainActor
|
||||
# Check tests that access @MainActor types
|
||||
```
|
||||
|
||||
**Pattern 3 - Shared mutable state**:
|
||||
```
|
||||
Grep: static var.*=|class var.*=
|
||||
# In files matching *Tests.swift
|
||||
```
|
||||
|
||||
**Pattern 4 - Task.sleep in tests**:
|
||||
```
|
||||
Grep: Task\.sleep|try await Task\.sleep
|
||||
```
|
||||
|
||||
**Pattern 5 - Missing .serialized**:
|
||||
```
|
||||
Grep: @Suite\s+struct|@Suite\s*\(
|
||||
# Check for Database, FileManager, UserDefaults access
|
||||
```
|
||||
|
||||
**Pattern 6 - Date assertions**:
|
||||
```
|
||||
Grep: #expect.*Date\(\)|#expect.*\.date
|
||||
```
|
||||
|
||||
### Step 3: Read Context and Verify
|
||||
|
||||
For each match:
|
||||
1. Read surrounding context (20 lines)
|
||||
2. Verify it's a real issue (not false positive)
|
||||
3. Check if fix is already present
|
||||
|
||||
## Output Format
|
||||
|
||||
```markdown
|
||||
# Test Failure Analysis Results
|
||||
|
||||
## Summary
|
||||
- **CRITICAL Issues**: [count] (Will cause intermittent failures)
|
||||
- **HIGH Issues**: [count] (Likely flaky in parallel execution)
|
||||
- **MEDIUM Issues**: [count] (May cause timing issues)
|
||||
- **LOW Issues**: [count] (Edge case failures)
|
||||
|
||||
## Flakiness Risk Score: HIGH / MEDIUM / LOW
|
||||
|
||||
## CRITICAL Issues
|
||||
|
||||
### Missing `await confirmation`
|
||||
- `Tests/NetworkTests.swift:45`
|
||||
```swift
|
||||
@Test func fetchUser() async {
|
||||
var user: User?
|
||||
api.fetchUser { user = $0 }
|
||||
#expect(user != nil) // FLAKY!
|
||||
}
|
||||
```
|
||||
- **Root cause**: Test completes before async callback
|
||||
- **Fix**:
|
||||
```swift
|
||||
@Test func fetchUser() async {
|
||||
await confirmation { confirm in
|
||||
api.fetchUser { user in
|
||||
#expect(user != nil)
|
||||
confirm()
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Missing `@MainActor`
|
||||
- `Tests/ViewModelTests.swift:23`
|
||||
```swift
|
||||
@Test func updateUI() async {
|
||||
let vm = MainActorViewModel() // Data race
|
||||
}
|
||||
```
|
||||
- **Root cause**: Accessing @MainActor type without isolation
|
||||
- **Fix**: Add `@MainActor` to test function
|
||||
|
||||
## HIGH Issues
|
||||
|
||||
### Shared Mutable State
|
||||
- `Tests/CacheTests.swift:12` - `static var testCache`
|
||||
- **Root cause**: Parallel tests mutate same collection
|
||||
- **Fix**: Use instance property instead of static
|
||||
|
||||
## MEDIUM Issues
|
||||
|
||||
### Missing `.serialized` Trait
|
||||
- `Tests/DatabaseTests.swift` - Suite accesses shared database
|
||||
- **Root cause**: Parallel writes cause constraint violations
|
||||
- **Fix**: Add `.serialized` trait to `@Suite`
|
||||
|
||||
## Verification Steps
|
||||
|
||||
After fixes, verify with:
|
||||
|
||||
```bash
|
||||
# Run tests multiple times to detect flakiness
|
||||
swift test --parallel --num-workers 8
|
||||
|
||||
# Run specific test repeatedly
|
||||
swift test --filter "TestName" --iterations 100
|
||||
|
||||
# Xcode: Edit Scheme → Test → Options → "Repeat Until Failure"
|
||||
```
|
||||
|
||||
## Swift Testing Best Practices
|
||||
|
||||
| Pattern | Use When |
|
||||
|---------|----------|
|
||||
| `confirmation {}` | Any callback/closure-based async |
|
||||
| `@MainActor` | Test accesses UI types |
|
||||
| `.serialized` | Tests share singleton/file/database |
|
||||
| Instance properties | Any test data that changes |
|
||||
```
|
||||
|
||||
## Severity Definitions
|
||||
|
||||
**CRITICAL**: Will definitely cause intermittent failures
|
||||
- Missing `confirmation` for async callbacks
|
||||
- Missing `@MainActor` for UI tests
|
||||
|
||||
**HIGH**: Likely to cause parallel execution failures
|
||||
- Shared mutable state (`static var`)
|
||||
- Order-dependent tests
|
||||
|
||||
**MEDIUM**: May cause timing-related failures
|
||||
- `Task.sleep` for waiting
|
||||
- Missing `.serialized` for shared resources
|
||||
|
||||
**LOW**: Edge case failures
|
||||
- Date/timezone assertions
|
||||
- Locale-dependent comparisons
|
||||
|
||||
## False Positives to Avoid
|
||||
|
||||
**Not issues**:
|
||||
- `static let` constants (immutable is fine)
|
||||
- `confirmation` already present
|
||||
- Tests marked with `.serialized`
|
||||
- `@MainActor` already present
|
||||
- One-time setup in `static var` that's read-only
|
||||
|
||||
**Verify before reporting**:
|
||||
- Read surrounding context
|
||||
- Check for `confirmation {}` wrapper
|
||||
- Check for trait annotations
|
||||
|
||||
## XCTest Flaky Patterns (Legacy)
|
||||
|
||||
For XCTest code, also check:
|
||||
|
||||
### XCTestExpectation Issues
|
||||
```swift
|
||||
// ❌ FLAKY - Timeout too short for CI
|
||||
wait(for: [expectation], timeout: 1.0)
|
||||
|
||||
// ✅ BETTER - Generous timeout
|
||||
wait(for: [expectation], timeout: 10.0)
|
||||
```
|
||||
|
||||
### Missing waitForExistence
|
||||
```swift
|
||||
// ❌ FLAKY - Element may not exist yet
|
||||
XCTAssertTrue(app.buttons["Submit"].exists)
|
||||
|
||||
// ✅ CORRECT - Wait for element
|
||||
XCTAssertTrue(app.buttons["Submit"].waitForExistence(timeout: 5))
|
||||
```
|
||||
|
||||
## When No Issues Found
|
||||
|
||||
Report:
|
||||
```markdown
|
||||
# Test Failure Analysis Results
|
||||
|
||||
## Summary
|
||||
No flaky test patterns detected.
|
||||
|
||||
## Verified
|
||||
- ✅ Async tests use `confirmation` properly
|
||||
- ✅ UI tests have `@MainActor` isolation
|
||||
- ✅ No shared mutable state in suites
|
||||
- ✅ No timing-dependent assertions
|
||||
|
||||
## Recommendations
|
||||
- Run tests with `--iterations 100` to verify stability
|
||||
- Enable parallel testing to expose hidden races
|
||||
- Use Xcode's "Repeat Until Failure" for suspect tests
|
||||
```
|
||||
Reference in New Issue
Block a user