Portland community members applying for building permits often struggled to find the right permit for their needs. While the City offers 15-minute appointment consultations to assist with permitting questions, people frequently selected incorrect options, leading to extended delays only to discover they'd chosen the wrong appointment type. To help alleviate these issues, the City's Digital Services team had developed a proof-of-concept chatbot but faced accuracy issues when user queries didn't match exact keywords.
The Goal
Portland's Digital Services team needed to optimize their permitting appointment helper chatbot prototype before internal staff testing and eventual public launch. They sought to improve response accuracy, establish performance benchmarks, and develop a strategic roadmap for scaling AI solutions across multiple permit types while building internal expertise in generative AI implementation.
This initiative was part of Portland's larger strategic vision to make city services simple, accurate, accessible, trustworthy, intuitive, and effective. The Digital Services team was already working on comprehensive website improvements, including creating a design system and rewriting content in plain language for five permit types. The chatbot represented an opportunity to demonstrate how AI could enhance these efforts by making it easier for community members to access the right services.
Why USDR?
Portland's Digital Services team had a long history of interest in partnering with USDR, following case studies and appreciating the organization's ability to match skilled volunteers with government needs. When a colleague initiated conversations with USDR, the team was impressed by the breadth of knowledge and collaborative approach, making USDR the natural choice for this AI optimization project.
Our Approach
USDR deployed a comprehensive strategy combining technical optimization, user research, and capacity building. The technical work involved creating sophisticated evaluation frameworks, including 177 real-world test scenarios generated from actual permit department inquiries and classified by subject matter experts. The team built robust prompt management systems that enabled rapid iteration and quantitative quality measurement across different chatbot versions.
The user research component included in-depth interviews with permit staff, giving the team insights about workflow integration and user needs. This research informed both technical improvements and strategic recommendations for deployment.
USDR also focused on knowledge transfer, training Portland staff on the essential but often overlooked practices that make AI implementations successful—from rigorous testing and quality control to ongoing system optimization. The team created comprehensive documentation, video tutorials, and hands-on training materials to ensure Portland could continue improving the system independently.
"I love working on these projects—it introduces me to different platforms and the partner was open to taking different directions. The scoping document was a starting point, but as the project progresses, you learn new things that can inform new ideas." — Christopher Fan, AI/ML Engineer, USDR Volunteer
Technologies Used:
- Google Vertex Platform: Gemini Large Language Models, Evaluations, Prompt Management
- LangChain
- Dialogflow
- Google Cloud Platform
Practices Used:
- Prompt engineering
- User experience research
- Accuracy benchmarking based on subject matter expert ground truth
- Synthetic data generation (based on 4k+ support desk tickets)
The Impact
The project created a testing system with 177 real-world scenarios that allows Portland to measure how well the chatbot works and spot areas for improvement. Built with input from permit experts, this system gives Portland the tools to continuously evaluate and enhance their AI tools using concrete data. What’s more, the evaluation benchmarks uncovered ways the team can modify prompts in order to reduce false positives by redirecting difficult user requests to the support desk.
USDR also helped upskill Portland's internal team when working with AI through practical training sessions. Team members learned how to manage chatbot prompts, run evaluations, and use monitoring tools, building the foundational skills they need to confidently implement and manage AI solutions independently moving forward.
Research with permit staff across different departments uncovered important insights about how the chatbot fits into daily workflows, and revealed how the appointment page did and didn't work for people using the feature. USDR created a presentation with recommendations for improving the appointments page through simplified appointment descriptions, improved information hierarchy, and better appointment tracking. These findings give Portland clear direction for both refining the chatbot and improving other digital services.
The improved chatbot has completed internal testing and is preparing for a future public launch with clear success measures and plans for expanding to other permit types. Portland now has a proven approach for adding AI tools to other city services, showing how careful AI implementation can improve public services while building lasting internal capabilities.
The Team
USDR Volunteers:
- Christopher Fan, product manager
- Cristy Rowley, UX researcher
- Raj Bagchi, AL/ML engineer
- Marcie Chin, USDR program advisor
City of Portland Partners:
- Hilaire Brockmeyer, digital services manager - Project lead balancing website enhancement priorities with AI innovation
- Greg Clapp, information systems analyst
- Oden Schmit, web developer
- Sean Alagar-McCartney, devops lead
- Evan Bowers, UX designer
"If anyone is thinking about wanting to engage with USDR, it is absolutely worth the conversation. There's so much opportunity, and it's great to know there's so many people willing to help. I’m always impressed with their breadth of knowledge and how well they're able to match volunteers. We couldn't have made it here on our own." — Hilaire Brockmeyer, Digital Services Manager, City of Portland
Key Learnings:
Start with learning, not magic: Portland approached AI implementation with realistic expectations, focusing on building internal expertise rather than expecting technology to solve all problems immediately.
User research is essential: Even AI projects require traditional UX research methods to understand user needs, goals, and workflows—technology alone isn't sufficient.
Evaluation frameworks drive quality: Quantitative measurement of AI performance through real-world scenarios enables data-driven improvements and clear success criteria.
Scope flexibility enhances success: Being open to adjusting project scope based on learnings and stakeholder feedback led to more valuable outcomes than rigid adherence to initial plans.
Portland's approach shows that thoughtful AI implementation can enhance public service delivery while building sustainable internal capabilities for continued innovation.
Photo by Jonathan Borba on Unsplash
