Feedback loop

In order to continuously improve the relevancy of results, FastSearch tracks all user queries and search result ratings. Search result ratings are strong signals of user preference and can be directly trained on to improve both cross-encoder (ranking) and bi-encoder (retrieval) models. User queries improve the domain coverage of the query corpus and can be used generate more diverse query+result candidates for data labeling.

Feedback UI

FastSearch was designed as a Single Page App (SPA) to increase the likelihood of users providing feedback by eliminating all navigation across multiple pages, making it easy for users to scan results and activate videos in place, and placing the feedback component right next to the video player.

Clicking on an item In the result list opens an embedded YouTube player queued to its exact timestamp 1. Directly underneath the player are feedback components 2 and links to related course materials 3. This reduces the chance that a user finds a relevant search result, views it in another page, and never comes back to rate it.

Storing Feedback

All user feedback is written to a MySQL database. User queries are stored asynchronously, via background tasks scheduled on the FastAPI framework event loop, in order to reduce the latency created by database connection and transaction.

Coming in FastSearch 2.0

The resiliency of feedback storage will be improved by switching to an asynchronous queue such as AWS SQS or Kafka.

Also, as usage of FastSearch grows, clickstream data – including results-clicked-per-query, result watch time, result-query abandonment – will be tracked. These events will be used as weak signals to fill in the gaps when users don’t provide explicit feedback. For example, a search result with a long watch-time and high cross-encoder score likely indicates a relevant search result, especially if the user closed the page after. This can be modeled using clickstream events as features to improve the ranking model used to generate pseudo labels during training.