Sync & background work

Path shorthands: A: = app/src/main/java/org/platformcommons/urvi/, D: = data/src/main/java/org/mindtree/app/agricultor/data/.

The idea in one paragraph

Field workers often have no network. So the app never sends a change straight to the server. It saves the change in the local database and writes a "to do" note. Later, when there is a network, background jobs turn each note into an HTTP request, send it, and copy the server's new ids back into the local database. Some notes must wait for others (a crop plan cannot be sent before its farmer exists on the server), so the app tracks those dependencies too.

End-to-end flow

1. User saves something (new farmer, land, crop plan, photo, endorsement...). data/…/data/repo/DataRepository.kt writes the entity to Room and calls SyncLedgerUtils.addInSyncLedger[WithDependencies].
2. Level 1, the ledger. A row goes into Room table sync_ledger (entity_name, entity_id = app UUID, sync_method, sync_op_id, sync_status = SYNC_NOT_INITIALIZED). If it must wait for another row, a sync_dependency row (parent_id -> child_id) is added.
3. SyncWork runs (WorkManager, every 15 minutes, unique periodic_sync; also kicked from Home). It picks ledger rows that have no pending parents and no sync_op_id yet.
4. An entity worker builds the request. For example PortfolioLineItemWorker looks up server ids that now exist in Room (the asset id from the asset code), and builds a CoreAPI Operation: URL, method, body, file params, response-handler class name, and the ledger row as metaJson.
5. Level 2, the offline-core queue. CoreAPI.dooWithRetry(op) stamps the session id and a retry policy and stores the operation in SQLite SYNC_QUEUE_DB, table OPERATION (plus INTEGRATOR for library-level dependencies). The ledger row becomes IN_SYNC_QUEUE with sync_op_id set.
6. The Android SyncAdapter drains the queue. It runs in a separate :sync process (enabled in app/…/urvi/view/home/HomeActivity.kt:706-731). SyncQueueProcessor.processSyncQueue() loads pending rows oldest first and sends each one.
7. A response handler applies the result (data/…/data/sync/rh/*, base class BaseResponseHandler.kt).
   • Success (2xx): write server ids into Room, mark the ledger row SYNC_DONE, delete its sync_dependency rows. Children are now free and go out on the next SyncWork run.
   • Failure: log it, and usually answer LATER (try again after a delay). A few cases answer NEVER (drop).
8. Clean-up. Garbage collection deletes finished queue rows after 30 seconds. SYNC_DONE ledger rows are deleted by SyncLedgerDao.

User save
  -> Room entity + sync_ledger (+ sync_dependency)
  -> SyncWork (15 min) -> <Entity>Worker -> CoreAPI Operation
  -> SYNC_QUEUE_DB.OPERATION
  -> SyncAdapter (:sync process) -> HTTP
  -> <X>RH.onSuccess: server ids -> Room, ledger SYNC_DONE, children unblocked
     <X>RH.onError : ErrorLogEntity, LATER (retry) or NEVER (drop)

Ordering and dependencies

Ordering is enforced before an operation exists: a child is not even built until its parent is done, so it can be built with the parent's real server id. There are no temporary ids on the server. The client keeps its own appId (UUID); for farmers the aliasId is that UUID.

Some writes skip the ledger and go straight into the level-2 queue: annual action plans (2025-26 POST/PUT, 2024-25 PATCH), ATM harvest, and assessment (AIA) uploads.

Retries

• Retry limit comes from ref-data KEY_SYNC_RETRY_THRESHOLD_LIMIT (default 1176 attempts). Delay comes from KEY_RETRY_DELAY_SECONDS (default 180 s), constant, with no backoff.
• After the limit the operation is expired and its dependents are expired too (the library passes the two arguments in the wrong order, SyncQueueProcessor.java:276).
• A 401 stops the whole run. The library retries once on 401 or 503 inside a single send.
• An exception inside a response handler becomes NEVER: the change is silently dropped.
• A DELETE answered with "already deactivated" is retried anyway, because the success branch is commented out, so it loops.

Conflicts

There is no conflict detection. PUT and PATCH send whole objects, so the last write wins. A 409 is only treated as "server already saw this request". The one special case is the compact-block lock: if the server says a line item "is a part of compactBlock. Hence, Editing is not allowed.", the app refetches the item and overwrites the local copy.

Errors and the error log

• Every failure is saved in Room table ErrorLogEntity (URL, body, headers, raw response) by BaseResponseHandler.logErrorInDb.
• Every 10th attempt, the operation is copied to Firestore SYNC_LOG.
• A 403 after more than 2 attempts on a farmer that already has a server id becomes NEVER, with a Firestore log.
• Some handlers fix the stored request in place (updateOperation), for example default totalQuantity=0 on an "Integrity" error, or fix owningEntityId from the alias.
• Asset-mismatch errors schedule FetchFarmerAssetDataWorker to refetch the asset and mark a sync error in the UI.
• The Pending Sync screen (app/…/urvi/view/pendingsync/*) lists queued operations with labels and per-item retry. A failure broadcast (SYNC_FAIL) is also sent, with no receiver.
• Home repairs ledger rows that say IN_SYNC_QUEUE but whose operation has vanished (HomeViewModel.repairSyncLedger).

Image and file uploads

• Uploads are multipart operations with a file param: crop photo, crop / land / ATM KML geofence, ATM photo, farmer photo, endorsement abstract.
• Files live in the app cache and files folders (commit 14abc4ad).
• Each upload needs the parent's server id, so it waits for the parent create.
• If the image path is an http URL (a server copy), the worker downloads it and uploads it again.
• Firebase fallback: after N failed crop-photo uploads (ref-data FIREBASE_ATTACHMENT_FALLBACK_UPLOAD_FLAG), the photo goes to Firebase Storage (userId/farmerId/pliId/pliId.ext), its metadata to Firestore PLI_IMAGE, and the ledger row is marked done. The backend never sees that photo.

Live response handlers

All in data/…/data/sync/rh/. "Depends on" is the ledger dependency (see above).

Live background workers

Workers that exist but never run are listed on the stale page.

Push messages (FCM)

• The receiver is app/…/urvi/fcm/FirebaseMessageReceiver.kt. The device token is only logged, never sent to a server, so nobody can push to a specific user.
• The app subscribes to topics urvi_endorsement_status, urvi_endorsement_comment and NOTIFICATION.
• Endorsement-status and comment messages are shown only if that farmer is in the phone's local DB and the sender is someone else. They are also saved to the Notifications table. Anything else shows its title and body.
• The sender was the app itself: after an endorsement or comment syncs, the handler calls the legacy FCM API with a server key. That API was shut down in 2024 and the key is never loaded, so endorsement push is dead. Polling (15 min endorsements, hourly comments) is the only thing that works.
• Remote Config supplies show_force_sync, show_analytics_tab and current_season_value.

How sync is triggered

Problems

1. Two queues for one job. The Room ledger and the library's SQLite queue both track state, and drift between them needs a repair step on Home.
2. Two processes can drain the queue at once (the :sync adapter and in-app force sync), with no lock.
3. No conflict detection: last write wins, silently.
4. Retries are effectively infinite (1176 × 180 s, about 2.5 days of constant retries) with no backoff, then a silent expiry.
5. Silent drops: an exception in a handler becomes NEVER.
6. Client-side id resolution: about 40 handlers back-fill server ids; one failure leaves partial server state.
7. The queue stores class names and Java-serialized files, so renaming or obfuscating a handler breaks queued data, and a new app cannot read the old queue.
8. Uploads wait for parents, re-upload server images, and can bypass the backend via Firebase.
9. PII leaves the device in queue snapshots (every 6 h) and Firestore logs.
10. Push is dead, and the original design put a server key on the device.
11. Worker name collision drops either analytics or endorsement refresh.

What the new outbox should do

• One Room table outbox: id (UUID, also the Idempotency-Key), entity type, op, payload JSON, dependsOn, status, attempts, nextAttemptAt, lastError, createdAt. No reflection, no class names in the DB.
• One WorkManager CoroutineWorker: unique work, network required, exponential backoff, expedited for "Sync now". Drop the SyncAdapter, StubContentProvider and the :sync process.
• Send batches to POST /v2/sync/batch, keyed by client appId. The App Service resolves parent references and returns {appId, serverId, status, error, serverVersion} per item. The client stops back-filling ids.
• Upload files first to POST /v2/files and reference the fileId, so uploads never wait for a parent's server id.
• Detect conflicts: send a version or updatedAt precondition; the server returns a typed conflict error (the compact-block lock becomes one of these).
• Bounded retries, never silent: after the max attempts, mark the row failed, show it on the Pending Sync screen, and allow manual retry.
• Expose state as a Flow (pending count, failures) instead of broadcasts.
• Server-side push by user and VO, using device tokens registered with PUT /v2/devices/{id}/push-token.
• Migration: ship a final old-app release that forces a full drain of SYNC_QUEUE_DB and shows the pending count before cutover, because the new app cannot read the old queue.