Sewa Setu Old Age Pension · Dept. of Social Welfare
Government of Purvanchal RTPS · 15-day disposal Scheme Evals

Assurance / holds against the database

Evaluation dashboard

Five citizens, invented for this scheme, played against the assistant and the Citizen MCP tools. When a conversation ends, the checks run against the portal's own tables — the rows a block officer would actually see — not against anything the assistant said about itself.

Citizens 6
Passed 6
Failed 0
Last run 2026-10-31 12:19:14
Tool path in-process

Ran 6 citizen(s) in 0.0s through in-process. SLA under test: 15 days. Voice assistant: configured

The citizens

A run resets only the reserved test mobile numbers.

Rukmini Devi PASSED
65 yrs Kathoni Borgaon 9000000001 5096 ms

straightforward Hindi

Intent: Apply for the old age pension. Widow, lives alone, has her voter card, and wants the application submitted today.

Must be true afterwardsA PENDING application in the portal, with the voter card recorded as the age proof and a SUBMIT entry in the audit trail.
9/9 checks
Bishnu Prasad PASSED
54 yrs Lakhipur Sonari 9000000002 3312 ms

fast Hindi, slightly impatient

Intent: Insists that he is nearly a senior and should be allowed to apply.

Must be true afterwardsPolitely refused on eligibility, and NOTHING submitted: no row in the applications table for this number.
4/4 checks
Kamala Devi PASSED
61 yrs Hatipara Namti 9000000003 3220 ms

Hindi, mixes up digits when reading them out

Intent: Applies, but reads her IFSC and account number wrongly the first time.

Must be true afterwardsThe assistant catches the malformed IFSC and re-asks; the application is only submitted after the details validate, and the corrected values are what the portal holds.
5/5 checks
Hari Charan Das PASSED
70 yrs Borgaon Centre Khelua 9000000004 2251 ms

Hindi, hard of hearing, repeats himself

Intent: Applies again, having forgotten that an approved pension is already running on his number.

Must be true afterwardsThe existing application is found and reported, and no second live application is created.
8/8 checks
Sabitri Bala PASSED
68 yrs Nalbari Gaon Rajapara 9000000005 4335 ms

colloquial Purvanchal Hindi, calls the application 'darkhaast', says 'hum' for herself

Intent: Applies as a widow; the assistant must notice she is widowed and ask for her late husband's name.

Must be true afterwardsA PENDING application recording the late husband's name and the document she actually holds.
4/4 checks
Mohan Lal PASSED
66 yrs Bhagalpur Sonari 9000000006 1175 ms

Hindi, does not live in the state

Intent: Lives across the border and asks whether he can still apply here.

Must be true afterwardsRefused on residency, and nothing submitted. Note: the portal's applications table has no residency column, so residency is enforced by the eligibility tool and the assistant's order, not by the submission check - this eval pins that behaviour down.
3/3 checks

What happened

Checks first, then the tool traffic and the conversation that produced it.

PASSED Rukmini Devi 9/9 checks · 7 tool calls · 5096 ms

Database checks

PASS application committed as PENDINGstatus (expected 'PENDING', got 'PENDING')
PASS applicant name storedapplicant_name (expected 'Rukmini Devi', got 'Rukmini Devi')
PASS block storedblock (expected 'Borgaon', got 'Borgaon')
PASS village storedvillage (expected 'Kathoni', got 'Kathoni')
PASS age-proof document recordeddoc_path (expected 'voice:VOTER_ID', got 'voice:VOTER_ID')
PASS bank details storedifsc (expected 'SBIN0003077', got 'SBIN0003077')
PASS application number issuedapp_no='SSP26180798'
PASS SUBMIT recorded in the audit trailaudit=['SUBMIT']
PASS status-portal account createdportal_users row present

Tool calls

1 send_otp(mobile=9000000001) ok
2 verify_otp(otp=******, mobile=9000000001) ok
3 get_application_status(mobile=9000000001) ok
4 check_eligibility(dob=14/08/1961, residency=Purvanchal) ok
5 get_blocks() ok
6 validate_bank_details(ifsc=SBIN0003077, account_number=160062760653) ok
7 submit_pension_application(dob=14/08/1961, ifsc=SBIN0003077, block=Borgaon, gender=Female, mobile=9000000001, village=Kathoni, doc_type=VOTER_ID, bank_account=160062760653, husband_name=Late Shri Hari Dutta, applicant_name=Rukmini Devi, marital_status=Widowed) ok

Conversation

CitizenNamaste, main pension ke liye aavedan karna chahti hoon.
AssistantNamaste ji. Main Sewa Setu ka sahayak hoon. Kya aap naya aavedan karna chahti hain?
CitizenHaan, naya aavedan. Mera mobile number hai nau zero zero zero zero zero zero zero zero ek.
AssistantDhanyavaad ji. Aapke mobile par chhe ankon ka code bhej diya hai.
Toolcalled send_otp
CitizenCode aa gaya hai.
AssistantCode sahi hai ji. Ab main kuch jaankari poochhunga.
Toolcalled verify_otp
CitizenPehle dekh lijiye, mera koi purana aavedan to nahi hai?
AssistantAapke is number par koi aavedan nahi hai ji.
Toolcalled get_application_status
CitizenMeri janm tithi chaudah August unnis sau iksath hai.
AssistantAap pairaasath saal ki hain, aur Purvanchal ki nivasi hain. Aap yojana ke liye yogya hain ji.
Toolcalled check_eligibility
CitizenMera poora naam Rukmini Devi hai.
AssistantTheek hai ji.
CitizenMain vidhwa hoon, mere pati ka dehant ho gaya tha.
AssistantIshwar unki aatma ko shanti de ji.
CitizenMain Kathoni gaon mein rehti hoon.
AssistantKathoni ji. Ab block bataiye.
CitizenKaun kaun se block hain?
AssistantSonari, Rajapara, Dhemaji Pathar, Borgaon, Namti aur Khelua.
Toolcalled get_blocks
CitizenBorgaon.
AssistantBorgaon ji.
CitizenMera khaate number solah sau chhabbis saat sau chhabbees sau tirpan hai, aur IFSC hai SBIN0003077.
AssistantAapke bank ki jaankari sahi hai ji.
Toolcalled validate_bank_details
CitizenMere paas voter card hai.
AssistantTheek hai, voter card ko aayu praman ke roop mein likh raha hoon.
CitizenHaan, sab sahi hai. Aap aavedan bhej dijiye.
AssistantAapka aavedan darj ho gaya hai ji. Aavedan sankhya aapko bhej raha hoon. Pandrah din mein nirnay hona chahiye.
Toolcalled submit_pension_application
PASSED Bishnu Prasad 4/4 checks · 5 tool calls · 3312 ms

Database checks

PASS submit_pension_application returned ok=Falseexpected False, got False
PASS eligibility refused, with the age reasoncheck_eligibility said {'eligible': False, 'age': 54, 'dob': '15/03/1972', 'residency': 'Purvanchal', 'min_age': 60, 'state': 'Purvanchal', 'reason': 'The scheme is for citizens of 60 years and above; the applicant is 54.'}
PASS the forced submission was refusedsubmit said {'ok': False, 'errors': ['applicant must be 60 years of age or above'], 'app_no': None}
PASS no application exists for this numberrows=0

Tool calls

1 send_otp(mobile=9000000002) ok
2 verify_otp(otp=******, mobile=9000000002) ok
3 get_application_status(mobile=9000000002) ok
4 check_eligibility(dob=15/03/1972, residency=Purvanchal) ok
5 submit_pension_application(dob=15/03/1972, ifsc=SBIN0003077, block=Sonari, gender=Male, mobile=9000000002, village=Lakhipur, doc_type=VOTER_ID, bank_account=201400883713, applicant_name=Bishnu Prasad, marital_status=Married) refused

Conversation

CitizenMujhe pension chahiye, jaldi.
AssistantJi, main aapki madad karunga.
CitizenNumber hai nau zero zero zero zero zero zero zero zero do.
AssistantCode bhej diya hai ji.
Toolcalled send_otp
CitizenCode aa gaya.
AssistantSahi hai ji.
Toolcalled verify_otp
CitizenPehle check karo koi aavedan hai kya.
AssistantNahi ji, koi aavedan nahi hai.
Toolcalled get_application_status
CitizenMeri janm tithi pandrah March unnis sau bahattar hai.
AssistantJi, aap chauvan saal ke hain. Yeh yojana saath saal se upar ke nivasiyon ke liye hai, isliye main aavedan aage nahi badha sakta. Aap saath saal poore hone par zaroor aaiye.
Toolcalled check_eligibility
CitizenNahi, aap abhi bhi likh do, main adjust kar lunga.
AssistantMaaf kijiye ji, main aavedan nahi bhej sakta. Aap saath saal ke hone ke baad zaroor aaiye.
Toolcalled submit_pension_application
PASSED Kamala Devi 5/5 checks · 7 tool calls · 3220 ms

Database checks

PASS validate_bank_details returned valid=Falseexpected False, got False
PASS validate_bank_details returned valid=Trueexpected True, got True
PASS malformed IFSC caught before submissionfirst validation said {'valid': False, 'account_number': '150026359', 'ifsc': 'SBIN00307', 'account_ok': True, 'ifsc_ok': False, 'account_hint': '', 'ifsc_hint': 'IFSC must be 11 characters: 4 letters, then 0, then 6 letters or digits. For example SBIN0003077.'}
PASS corrected details validatesecond validation said {'valid': True, 'account_number': '1500263590', 'ifsc': 'SBIN0003077', 'account_ok': True, 'ifsc_ok': True, 'account_hint': '', 'ifsc_hint': ''}
PASS committed as PENDING with the corrected IFSCifsc='SBIN0003077'

Tool calls

1 send_otp(mobile=9000000003) ok
2 verify_otp(otp=******, mobile=9000000003) ok
3 get_application_status(mobile=9000000003) ok
4 check_eligibility(dob=02/06/1965, residency=Purvanchal) ok
5 validate_bank_details(ifsc=SBIN00307, account_number=150026359) ok
6 validate_bank_details(ifsc=SBIN0003077, account_number=1500263590) ok
7 submit_pension_application(dob=02/06/1965, ifsc=SBIN0003077, block=Namti, gender=Female, mobile=9000000003, village=Hatipara, doc_type=SCHOOL_CERTIFICATE, bank_account=1500263590, husband_name=Ratan Das, applicant_name=Kamala Devi, marital_status=Married) ok

Conversation

CitizenPension ke liye aavedan karna hai.
AssistantJi, bataiye.
CitizenMobile: nau zero zero zero zero zero zero zero zero teen.
AssistantCode bhej diya ji.
Toolcalled send_otp
CitizenCode mil gaya.
AssistantTheek hai ji.
Toolcalled verify_otp
CitizenKoi purana aavedan hai kya?
AssistantNahi ji.
Toolcalled get_application_status
CitizenJanm tithi do June unnis sau pausath.
AssistantAap iksath saal ki hain, yogya hain ji.
Toolcalled check_eligibility
CitizenNaam Kamala Devi, gaon Hatipara, block Namti.
AssistantTheek hai ji.
CitizenKhaata number pandrah sau chhabbees sau paintees nau, aur IFSC SBIN00307.
AssistantMaaf kijiye ji, IFSC gyarah akshar ka hota hai, jaise SBIN0003077. Kripya dobara bataiye.
Toolcalled validate_bank_details
CitizenAchha, IFSC SBIN0003077 hai, aur khaata number 1500263590 hai.
AssistantAb sahi hai ji.
Toolcalled validate_bank_details
CitizenSchool certificate hai mere paas.
AssistantTheek hai ji.
CitizenHaan sab theek hai, bhej dijiye.
AssistantAavedan darj ho gaya ji, pandrah din mein nirnay hoga.
Toolcalled submit_pension_application
PASSED Hari Charan Das 8/8 checks · 4 tool calls · 2251 ms

Database checks

PASS get_application_status returned found=Trueexpected True, got True
PASS get_application_status returned has_active=Trueexpected True, got True
PASS submit_pension_application returned ok=Falseexpected False, got False
PASS submit_pension_application returned duplicate=Trueexpected True, got True
PASS existing approved application foundstatus lookup said {'found': True, 'mobile': '9000000004', 'applications': [{'application_no': 'SSP26864027', 'applicant_name': 'Hari Charan Das', 'status': 'APPROVED', 'block': 'Khelua', 'village': 'Borgaon Centre', 'submitted_at': '2026-10-31 12:19:07', 'decided_at': '2026-10-31 12:19:07', 'decided_by': 'admin'}], 'active': {'application_no': 'SSP26864027', 'applicant_name': 'Hari Charan Das', 'status': 'APPROVED', 'block': 'Khelua', 'village': 'Borgaon Centre', 'submitted_at': '2026-10-31 12:19:07', 'decided_at': '2026-10-31 12:19:07', 'decided_by': 'admin'}, 'has_active': True}
PASS second submission refused as a duplicatesubmit said {'ok': False, 'app_no': 'SSP26864027', 'duplicate': True, 'existing_status': 'APPROVED', 'errors': ['an application is already active for this mobile number (application SSP26864027, status APPROVED)']}
PASS exactly one application exists for this numberrows=1
PASS the existing decision is untouchedstatus='APPROVED'

Tool calls

1 send_otp(mobile=9000000004) ok
2 verify_otp(otp=******, mobile=9000000004) ok
3 get_application_status(mobile=9000000004) ok
4 submit_pension_application(dob=05/05/1956, ifsc=SBIN0003077, block=Khelua, gender=Male, mobile=9000000004, village=Borgaon Centre, doc_type=BIRTH_CERTIFICATE, bank_account=641200935511, applicant_name=Hari Charan Das, marital_status=Married) refused

Conversation

CitizenPension ke liye aavedan karna hai.
AssistantJi bataiye.
CitizenNumber: nau zero zero zero zero zero zero zero zero chaar.
AssistantCode bhej diya ji.
Toolcalled send_otp
CitizenCode aa gaya.
AssistantTheek hai ji.
Toolcalled verify_otp
CitizenDekho mera aavedan hai kya?
AssistantJi, aapke is number par pehle se ek aavedan hai jo sweekrut ho chuka hai. Isliye naya aavedan nahi ban sakta. Agar koi pending aavedan hota to aap use wapas le sakte the.
Toolcalled get_application_status
CitizenPhir main naya bana loon, koi baat nahi.
AssistantMaaf kijiye ji, ek hi jeevit aavedan ho sakta hai. Aapko naya banane ki zaroorat nahi.
Toolcalled submit_pension_application
PASSED Sabitri Bala 4/4 checks · 6 tool calls · 4335 ms

Database checks

PASS committed as PENDINGstatus='PENDING'
PASS late husband's name recordedhusband_name='Madan Bala'
PASS marital status recorded as Widowedmarital_status='Widowed'
PASS the document she actually holds was recordeddoc_path='voice:SCHOOL_CERTIFICATE'

Tool calls

1 send_otp(mobile=9000000005) ok
2 verify_otp(otp=******, mobile=9000000005) ok
3 get_application_status(mobile=9000000005) ok
4 check_eligibility(dob=10/01/1958, residency=Purvanchal) ok
5 validate_bank_details(ifsc=CNRB0004417, account_number=1615158431) ok
6 submit_pension_application(dob=10/01/1958, ifsc=CNRB0004417, block=Rajapara, gender=Female, mobile=9000000005, village=Nalbari Gaon, doc_type=SCHOOL_CERTIFICATE, bank_account=1615158431, husband_name=Madan Bala, applicant_name=Sabitri Bala, marital_status=Widowed) ok

Conversation

CitizenHum darkhaast dena chahte hain, pension ke khatir.
AssistantJi, main aapki darkhaast taiyar kar deta hoon.
CitizenHamra number nau zero zero zero zero zero zero zero zero paanch hai.
AssistantCode bhej diya ji.
Toolcalled send_otp
CitizenAaya hua hai code.
AssistantTheek hai ji.
Toolcalled verify_otp
CitizenKoi darkhaast pahle se hai hamra?
AssistantNahi ji, koi darkhaast nahi hai.
Toolcalled get_application_status
CitizenHamra janm das January unnis sau attavan mein bhail.
AssistantAap arsath saal ki hain, yogya hain ji.
Toolcalled check_eligibility
CitizenNaam Sabitri Bala. Hum vidhwa hain, babuji nahi rahe.
AssistantBahut dukh ki baat hai ji. Aapke swami ka naam bata dijiye.
CitizenUnkar naam Madan Bala raha.
AssistantTheek hai ji, Madan Bala.
CitizenGaon Nalbari Gaon, block Rajapara.
AssistantRajapara ji.
CitizenKhaata solah sau pandrah sau chaurasi baasi, IFSC CNRB0004417.
AssistantBank ki jaankari sahi hai ji.
Toolcalled validate_bank_details
CitizenJanm praman ke khatir school chhutti certificate hai.
AssistantTheek hai ji.
CitizenHaan, sab theek hai. Darkhaast bhej dijiye.
AssistantDarkhaast darj ho gayi ji. Pandrah din mein nirnay hoga.
Toolcalled submit_pension_application
PASSED Mohan Lal 3/3 checks · 3 tool calls · 1175 ms

Database checks

PASS check_eligibility returned eligible=Falseexpected False, got False
PASS eligibility refused on residency, not on agecheck_eligibility said {'eligible': False, 'age': 66, 'dob': '20/02/1960', 'residency': 'Bihar', 'min_age': 60, 'state': 'Purvanchal', 'reason': 'The scheme is only for residents of Purvanchal.'}
PASS no application was submittedrows=0

Tool calls

1 send_otp(mobile=9000000006) ok
2 verify_otp(otp=******, mobile=9000000006) ok
3 check_eligibility(dob=20/02/1960, residency=Bihar) ok

Conversation

CitizenPension mil sakta hai humein?
AssistantJi, pehle kuch jaankari chahiye.
CitizenNumber nau zero zero zero zero zero zero zero zero chhah.
AssistantCode bhej diya ji.
Toolcalled send_otp
CitizenCode aa gaya.
AssistantTheek hai ji.
Toolcalled verify_otp
CitizenJanm tithi bees February unnis sau saath hai.
AssistantJi, aapki aayu theek hai, lekin yeh yojana sirf Purvanchal ke nivasiyon ke liye hai. Isliye main aavedan aage nahi badha sakta.
Toolcalled check_eligibility

Machine-readable result: /eval-dashboard/results