feat: Offline AI Plant/Wildlife Identification#1906
Merged
andrew-bierman merged 14 commits intoApr 11, 2026
Conversation
Collaborator
1 similar comment
Collaborator
Copilot
AI
changed the title
[WIP] Add offline AI plant and wildlife identification functionality
feat: Offline AI Plant/Wildlife Identification
Mar 9, 2026
Contributor
Author
|
The Biome CI shows Is there something specific you'd like me to address or change? The wildlife identification feature is fully implemented and all new files pass lint checks. Let me know if you have any specific requests or feedback! |
Collaborator
|
@coderabbitai Please review this PR for code quality, best practices, and potential issues. |
Contributor
|
✅ Actions performedFull review triggered. |
Contributor
|
Important Review skippedBot user detected. To trigger a single review, invoke the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
📝 WalkthroughWalkthroughAdds a wildlife identification feature: UI screens and tiles, persisted history atoms and hooks, offline species database and identifier, online identification service and API route using OpenAI, route wiring and i18n strings, plus feature flag gating. Changes
Sequence DiagramssequenceDiagram
participant User
participant ExpoClient as Expo Client
participant OfflineID as Offline Identifier
participant SpeciesDB as Species Database
participant APIServer as API Server
participant OpenAI as OpenAI Service
User->>ExpoClient: Select image & optional description, request identify
ExpoClient->>OfflineID: identifyFromDescription(description)
OfflineID->>SpeciesDB: searchSpecies(query)
SpeciesDB-->>OfflineID: Matching species (offline results)
alt Network Available
ExpoClient->>APIServer: POST /api/wildlife/identify (image key)
APIServer->>OpenAI: generateObject(prompt + imageUrl, schema)
OpenAI-->>APIServer: Identification results
APIServer-->>ExpoClient: Online results
else Network Error
ExpoClient-->>OfflineID: Use offline results as fallback
end
ExpoClient->>SpeciesDB: Save persisted image + add to history (wildlifeHistoryAtom)
ExpoClient->>User: Display results
sequenceDiagram
participant User
participant ExpoClient as Expo Client
participant History as Wildlife History (Jotai)
participant Detail as Species Detail Screen
participant SpeciesDB as Species Database
User->>ExpoClient: Open Wildlife screen
ExpoClient->>History: load history (wildlifeHistoryAtom)
History-->>ExpoClient: WildlifeIdentification[]
ExpoClient->>User: Render history list
User->>ExpoClient: Tap history item -> navigate /wildlife/[id]
Detail->>SpeciesDB: getSpeciesById(id)
alt Species found
SpeciesDB-->>Detail: SpeciesEntry
else Not found
Detail->>History: load matching history entry
History-->>Detail: result from saved history
end
Detail->>User: Render species details
Estimated code review effort🎯 4 (Complex) | ⏱️ ~65 minutes Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
|
✅ Actions performedFull review triggered. |
Collaborator
|
@copilot CodeRabbit has provided review feedback on this PR. Please address the comments and suggestions. Once you've made the necessary changes, I'll ask CodeRabbit to re-review. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 11
🧹 Nitpick comments (1)
apps/expo/features/wildlife/types.ts (1)
29-32: Make theconfidencecontract explicit.
SpeciesCardmultiplies this value by 100, so a plainnumberhere leaves every producer guessing whether the API returns0..1or0..100. Please document or narrow the range at the type boundary.💡 Suggested tightening
export interface IdentificationResult { species: SpeciesEntry; + /** Normalized confidence score in the range [0, 1]. */ confidence: number; source: 'online' | 'offline'; }🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@apps/expo/features/wildlife/types.ts` around lines 29 - 32, The IdentificationResult.confidence field is ambiguous (0..1 vs 0..100); update the contract to explicitly represent a 0..1 probability by replacing the plain number with a constrained/Branded type (e.g., Confidence0To1 or type alias with a descriptive JSDoc) and update usages like SpeciesCard to rely on that 0..1 contract (it currently multiplies by 100). Change the IdentificationResult interface (symbol: IdentificationResult) to use the new Confidence type and add a brief JSDoc comment clarifying the range so producers and consumers know the expected 0..1 probability.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@apps/expo/features/wildlife/atoms/wildlifeAtoms.ts`:
- Around line 6-17: The current wildlifeStorage wrapper manually calls
JSON.parse/JSON.stringify around AsyncStorage causing double-encoding and
hydration failures; replace the custom adapter by passing AsyncStorage directly
into createJSONStorage (i.e., use createJSONStorage with AsyncStorage as the
storage backend) and remove the getItem/setItem/removeItem JSON serialization
logic so createJSONStorage can handle parsing/stringifying itself; update the
wildlifeStorage declaration (symbol: wildlifeStorage) to use createJSONStorage
with AsyncStorage.
In `@apps/expo/features/wildlife/components/SpeciesCard.tsx`:
- Around line 1-3: SpeciesCard is rendering enum-backed values (source,
dangerLevel, category) raw; replace those direct uses with translated labels by
mapping each enum to an i18n key or using the app's t()/translate utility inside
SpeciesCard (or a helper map) to render user-facing strings (e.g., map
IdentificationResult.source values like "offline" to translation keys such as
"species.source.offline"); update all usages of source, dangerLevel, and
category in the SpeciesCard render (lines previously flagged ~38-59) to call the
translator or lookup map so non-English locales show localized labels.
In `@apps/expo/features/wildlife/data/speciesDatabase.ts`:
- Around line 494-502: searchSpecies currently treats blank/whitespace queries
as matches because it lowercases without checking for emptiness; modify
searchSpecies to trim the incoming query and immediately return an empty array
if the trimmed string is empty, otherwise proceed with the existing lowercase
match logic against SPECIES_DATABASE (references: searchSpecies,
SPECIES_DATABASE, commonName, scientificName, description, characteristics).
In `@apps/expo/features/wildlife/hooks/useWildlifeIdentification.ts`:
- Line 57: The offline query selection currently uses the nullish coalescing
operator so an empty description string ("" ) wins; update the logic in
useWildlifeIdentification where identifyFromDescription is called so
blank/whitespace descriptions are treated as absent—e.g., evaluate offlineQuery
with a trim check (use offlineQuery && offlineQuery.trim().length > 0) and pass
offlineQuery only when non-empty, otherwise pass selectedImage.fileName to
identifyFromDescription.
- Line 56: In useWildlifeIdentification, replace the current raw error logging
in the catch/fallback that calls console.warn('Online identification
unavailable, using offline database:', error) with a sanitized log that only
includes non-sensitive fields (e.g., error?.code and error?.message) or remove
the error payload entirely; update the console.warn invocation in that block so
it doesn't print the full Axios error object.
In `@apps/expo/features/wildlife/screens/IdentificationScreen.tsx`:
- Line 20: The selectedImage URI is being stored directly instead of a persisted
local filename; update the useImagePicker destructure to include
permanentlyPersistImageLocally, call
permanentlyPersistImageLocally(selectedImage) just before invoking
addIdentification, and pass the returned persisted filename (not
selectedImage.uri) into addIdentification; reference the useImagePicker hook,
the permanentlyPersistImageLocally function, the selectedImage object, and the
addIdentification call to locate where to change the code.
In `@apps/expo/features/wildlife/screens/SpeciesDetailScreen.tsx`:
- Around line 30-42: The fallback lookup currently finds a history entry by
history entry id (historyLoadable.data.find(h => h.id === id)) which fails when
the route id is a species id; change the fallback to search historyLoadable.data
for any entry whose results array contains a result with result.species.id ===
id (update the logic that sets historyEntry/displaySpecies accordingly), and
ensure you do not render the not-found UI until wildlife history has finished
loading (check historyLoadable.state === 'hasData' before showing the
speciesNotFound view and render a loading state while it is still pending).
In `@apps/expo/features/wildlife/screens/WildlifeScreen.tsx`:
- Line 34: The hardcoded fallback label "Unknown species" in WildlifeScreen
should be localized; replace the literal Text node with the i18n lookup used
elsewhere (call t with an appropriate key like t('wildlife.unknownSpecies')),
add that key to the translations files under the wildlife namespace in all
locales, and update the component to use t('wildlife.unknownSpecies') instead of
the hardcoded string so the fallback label is localized across languages.
- Around line 48-52: The code collapses non-'hasData' loadable states into an
empty array—update handling of wildlifeHistoryAtom/historyLoadable so you check
for historyLoadable.state === 'hasError' and surface an error UI (or message)
before falling back to empty array and ensure the same explicit error branch is
used in the rendering logic around the 84-114 conditional block; additionally
remove the hardcoded "Unknown species" string by either moving useTranslation()
into the HistoryItem component or passing the t function into HistoryItem and
replacing the literal with a translation key (e.g.,
t('wildlife.unknownSpecies')) so the fallback is localized.
In `@packages/api/src/routes/wildlife/index.ts`:
- Around line 110-114: The temp-upload deletion currently runs only after a
successful call to WildlifeIdentificationService.identifySpecies and can leak or
convert a success into a 500; refactor the handler so the model call
(service.identifySpecies(imageUrl)) runs inside a try block, move
PACKRAT_BUCKET.delete(image) into a finally block and make the delete
best-effort (catch/log any errors from PACKRAT_BUCKET.delete so they don't
change the response), ensuring the identification result (identification) is
returned or the model error is handled independently of the upload cleanup.
- Around line 142-149: The 500-response currently echoes internal error details
via c.json({ error: `Failed to identify species: ${error.message}` }, 500);
update the 500 branch in the error handler (the conditional that checks error
instanceof Error and the current return c.json(...) for 500) to return a generic
message (e.g., { error: 'Internal server error' } or { error: 'Failed to
identify species' }) without including error.message, while leaving the 400
branch (Invalid image / Unsupported image format) and existing logging/Sentry
reporting unchanged.
---
Nitpick comments:
In `@apps/expo/features/wildlife/types.ts`:
- Around line 29-32: The IdentificationResult.confidence field is ambiguous
(0..1 vs 0..100); update the contract to explicitly represent a 0..1 probability
by replacing the plain number with a constrained/Branded type (e.g.,
Confidence0To1 or type alias with a descriptive JSDoc) and update usages like
SpeciesCard to rely on that 0..1 contract (it currently multiplies by 100).
Change the IdentificationResult interface (symbol: IdentificationResult) to use
the new Confidence type and add a brief JSDoc comment clarifying the range so
producers and consumers know the expected 0..1 probability.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 4590d1c7-0d37-45bf-9591-7478eafc8075
📒 Files selected for processing (25)
apps/expo/app/(app)/(tabs)/(home)/index.tsxapps/expo/app/(app)/wildlife/[id].tsxapps/expo/app/(app)/wildlife/identify.tsxapps/expo/app/(app)/wildlife/index.tsxapps/expo/config.tsapps/expo/features/wildlife/atoms/wildlifeAtoms.tsapps/expo/features/wildlife/components/SpeciesCard.tsxapps/expo/features/wildlife/components/WildlifeTile.tsxapps/expo/features/wildlife/components/index.tsapps/expo/features/wildlife/data/speciesDatabase.tsapps/expo/features/wildlife/hooks/index.tsapps/expo/features/wildlife/hooks/useWildlifeHistory.tsapps/expo/features/wildlife/hooks/useWildlifeIdentification.tsapps/expo/features/wildlife/index.tsapps/expo/features/wildlife/lib/offlineIdentifier.tsapps/expo/features/wildlife/screens/IdentificationScreen.tsxapps/expo/features/wildlife/screens/SpeciesDetailScreen.tsxapps/expo/features/wildlife/screens/WildlifeScreen.tsxapps/expo/features/wildlife/screens/index.tsapps/expo/features/wildlife/types.tsapps/expo/lib/i18n/locales/en.jsonpackages/api/src/routes/index.tspackages/api/src/routes/wildlife/index.tspackages/api/src/services/index.tspackages/api/src/services/wildlifeIdentificationService.ts
Contributor
Author
Addressed the issues in commit
All files pass |
Collaborator
|
@copilot CodeRabbit has reviewed this PR and provided feedback. Please address the review comments and push updates. Once you've made the changes, the monitor will ask CodeRabbit to re-review. |
Collaborator
|
@copilot CodeRabbit has provided review feedback on this PR. Please review and address the comments. |
Contributor
Author
All CodeRabbit comments addressed in commit Critical:
Major:
Minor:
All changes pushed in commit |
Collaborator
|
@copilot CodeRabbit has reviewed this PR. Please address the feedback. |
Contributor
Author
All CodeRabbit feedback was already addressed in commit
The CI |
Collaborator
|
@copilot CodeRabbit has reviewed this PR. Please address the feedback. |
14 tasks
andrew-bierman
force-pushed
the
copilot/offline-ai-plant-identification
branch
from
cdb9951 to
419153d
Compare
Contributor
Coverage Report for Expo Unit Tests Coverage (./apps/expo)
File CoverageNo changed files found. |
Contributor
Coverage Report for API Unit Tests Coverage (./packages/api)
File Coverage
|
||||||||||||||||||||||||||||||||||||||
Co-authored-by: andrew-bierman <94939237+andrew-bierman@users.noreply.github.com>
Co-authored-by: andrew-bierman <94939237+andrew-bierman@users.noreply.github.com>
- Fix searchSpecies() to also search imageDescription and habitat fields
(offline identification was skipping these purpose-built matching fields)
- Replace hardcoded 'Unknown species' with t('wildlife.unknownSpecies')
and add the key to en.json
- Remove unused wildlifeHistoryCountAtom (dead code)
- Remove redundant `as SelectedImage` cast and unused SelectedImage import
in IdentificationScreen (TypeScript narrows type in the conditional branch)
- Refactor WildlifeScreen to use useWildlifeHistory hook instead of
reading wildlifeHistoryAtom directly for consistent abstraction
Co-authored-by: andrew-bierman <94939237+andrew-bierman@users.noreply.github.com>
Critical: - wildlifeAtoms: fix double JSON encoding - pass AsyncStorage directly to createJSONStorage instead of manually calling JSON.parse/stringify - useWildlifeIdentification: fix empty-string ?? bug (offlineQuery='' won over filename); sanitize error log (no more raw Axios object in console) - SpeciesDetailScreen: fix history lookup to also search by species.id in h.results (was only checking history entry id, broke navigation from online identification results); add loading spinner before 'not found' - API route: move R2 delete to finally with best-effort catch so temp uploads are always cleaned up; make 500 response generic (no error.message) Major: - speciesDatabase: trim + early return in searchSpecies for blank queries - IdentificationScreen: call permanentlyPersistImageLocally() before addIdentification to persist image across cache evictions - WildlifeScreen: add explicit hasError branch (was silently showing empty state when AsyncStorage fails) - SpeciesCard: translate source/dangerLevel/category via t() Minor: - types.ts: add JSDoc documenting confidence is [0,1] - en.json: add historyLoadError, source.*, dangerLevel.*, category.* keys Co-authored-by: andrew-bierman <94939237+andrew-bierman@users.noreply.github.com>
Co-authored-by: andrew-bierman <94939237+andrew-bierman@users.noreply.github.com>
…i18n, error handling Co-authored-by: andrew-bierman <94939237+andrew-bierman@users.noreply.github.com>
Co-authored-by: andrew-bierman <94939237+andrew-bierman@users.noreply.github.com>
Critical: - Fix AI SDK generateObject call: use `messages` with ImagePart instead of `prompt` (which is text-only) for image-based species identification - Wrap imageUrl in `new URL()` for proper ImagePart type Major: - Add feature flag gate to all wildlife route pages (index, identify, [id]) redirecting to home when enableWildlifeIdentification is disabled - Pass undefined instead of empty string for offlineQuery so blank descriptions don't bypass the fileName fallback - Don't wipe identification results when user cancels image picker; only clear on successful new image selection - Improve isNetworkError: check Axios error codes (ERR_NETWORK, ECONNABORTED, etc.) and tighten message matching to avoid misclassifying server-side timeouts as network errors
…, R2 cleanup, i18n, types Co-authored-by: andrew-bierman <94939237+andrew-bierman@users.noreply.github.com>
… lookup Add missing Stack layout for the wildlife/ route group which would crash iOS navigation without it. SpeciesDetailScreen already handles deep-link lookups via getSpeciesById() so no changes needed there.
andrew-bierman
force-pushed
the
copilot/offline-ai-plant-identification
branch
from
1c64b71 to
1f71f0b
Compare
4 tasks
7 tasks
andrew-bierman
added a commit
that referenced
this pull request
May 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
searchSpeciesto includeimageDescriptionandhabitatin searcht('wildlife.unknownSpecies')wildlifeHistoryCountAtomas SelectedImagecastWildlifeScreento useuseWildlifeHistoryhookwildlifeAtoms.tssearchSpecies??bug inuseWildlifeIdentificationoffline pathuseWildlifeIdentificationpermanentlyPersistImageLocallyinIdentificationScreenSpeciesDetailScreenhistory lookup + loading statehasErrorbranch inWildlifeScreenfinallyin API routesource,dangerLevel,categoryinSpeciesCardandSpeciesDetailScreenconfidenceintypes.tsen.jsonenableWildlifeIdentification: false(ship dark by default)deleteIdentification/clearHistoryisNetworkError(remove over-broadreturn truefallback)scientificNamefallback in API species.id slugpermanentlyPersistImageLocallyin try/catch inIdentificationScreenhasErrorbranch inSpeciesDetailScreen(surfaces history load error instead of "not found"){ getOnInit: true }toatomWithStoragesoloadablecorrectly enters loading state on cold startIdentificationScreenonSuccess(ignore results if user changed image mid-flight)t()for wildlife tile title and category keywords inhome/index.tsxgenerateObjectinwildlifeIdentificationService.tsWildlifeScreenofflineIdentifier.ts: remove early-return, pass trimmed query tosearchSpeciesIdentificationScreen.tsx: useuseRefto track current image URI for accurate stale-request guardfinallyblock)useWildlifeHistory.ts: async updaters for AsyncStorage-backed atom to handle hydrationhome/index.tsx: move wildlife tiletitle/keywordsinsideDashboardScreenwithuseMemofor locale reactivityspeciesDatabase.ts: typegetSpeciesByCategoryparameter asSpeciesCategoryuseWildlifeIdentification.ts: improve upload error message with filename contextOriginal prompt
🔒 GitHub Advanced Security automatically protects Copilot coding agent pull requests. You can protect all pull requests by enabling Advanced Security for your repositories. Learn more about Advanced Security.
Summary by CodeRabbit