Skip to main content
LeoNex News

Automated review summaries with a feedback loop: LLMs in production at Grab

Illustration: Automated review summaries with a feedback loop: LLMs in production at Grab
In this article

The Automated Merchant Review Summary System turns handwritten store reviews into concise summaries. Consumers can view an overall summary or choose a specific topic, while merchants (stores, selling partners) learn what customers appreciate and which areas need improvement. The pipeline combines representative sampling, summarization by a large language model (LLM), automated quality checks and user feedback.

The problem with long review lists

A long, unstructured review list causes two related problems. Consumers spend time looking for comments that match their priorities, while merchants must aggregate comments one by one to spot recurring themes. A store may have hundreds or thousands of reviews; reading only the first few easily gives an incomplete picture, while the average star rating offers very little context. Two restaurants with the same number of stars can still differ in food quality, delivery experience, portion size or perceived value.

On the merchant side, reading and categorizing every review is time-consuming and gets harder as volume grows. Recurring complaints or positive points can get buried among individual comments.

Three existing approaches all have limits:

  • Manual analysis: reading each review and drawing conclusions yourself. The conclusions depend on which reviews the reader chooses to read and how they interpret them, and this approach is not feasible as volume grows.

  • Aggregate star ratings: they give a quick sense of overall opinion but do not explain why, so merchants see overall performance without seeing the themes that need attention.

  • Keyword search and filters: useful when the reader already knows what to look for, but they do not surface unexpected themes, and literal matching can separate phrases that mean the same thing. For example, the three phrases "too cold", "not warm enough" and "lukewarm" all describe a food temperature problem.

Summary generation pipeline architecture

The summary generation pipeline prepares the source review data, produces the summary, then checks the output before publishing. The system creates one overall summary and topic-specific summaries such as taste, packaging and portion size. The overall system has three flows: generating and validating summaries; maintaining summaries; and displaying summaries and collecting feedback.

The first step is data preparation and representative sampling. The pipeline computes the distribution of the merchant's ratings, and a sentiment model assigns each handwritten review a score from one to five. Stratified sampling then selects a smaller set that follows this distribution, reducing the risk of one score group dominating the summary. However, matching the rating distribution does not guarantee that the sample is representative of every period, language, customer group or aspect of the merchant's experience.

Next, the system sends the sampled reviews to an LLM along with instructions to produce two types of output:

  • A concise summary of the sampled reviews.

  • Topic-specific summaries for groups such as taste, packaging and portion size.

The system can also update the set of topics when new topics appear in consumer feedback.

Quality control with an LLM

In the quality control (QC) step, a separate LLM receives the generated summary together with the sampled reviews, then checks whether the source reviews support each claim. A summary that fails the validation criteria is held back and not published. Passing this automated evaluation step reduces, but does not eliminate, the risk of unsupported claims appearing.

Maintaining summaries over time

The maintenance flow updates stored summaries as new reviews and consumer feedback arrive. The system monitors both sources and regenerates a summary when either reaches a configured threshold. There are two triggers:

  1. New-review trigger: a monitoring service counts the reviews submitted since the latest summary. When the count reaches the configured threshold, the service starts the generation and validation flow.

  2. User-feedback trigger: the interface records whether consumers found the summary helpful. When negative feedback reaches the configured threshold, the system generates and validates a new summary.

On the display side, the interface retrieves the validated summary for the merchant's review page and records whether consumers found it helpful.

Impact and lessons learned

The main lesson is that summary quality depends on more than the text-generation step. The source sample must reflect the intended audience, every claim must be grounded in source reviews, and the output must change in response to the feedback beneath it. Automated LLM validation is only one layer of defense, not the sole proof of accuracy, and the feedback-driven regeneration mechanism needs safeguards against abuse to remain trustworthy.

This pipeline's scope is consumer-facing summaries. A misleading or incomplete summary can affect how consumers perceive a merchant, but the complaint and dispute mechanism for merchants sits outside the pipeline and is not part of the flow described.

Key takeaways

More in AI Engineering

Illustration: Context engineering and guardrails to "tame" coding agents
AI-ENG AI Engineering

Context engineering and guardrails to "tame" coding agents

Coding agents such as Cursor, Windsurf, Cline or Claude Code often deliver impressive results early on, but as a project grows with many files and branching business logic, they can break working features, pull in conflicting libraries or violate the architecture. The cause lies in an unstructured way of using them…

4 min read

Illustration: Multi-agent: when to split agents and when not to
AI-ENG AI Engineering

Multi-agent: when to split agents and when not to

In AI agent systems, breaking things into multiple agents sounds modern, but in practice it often makes testing and debugging harder and drives up costs. The guiding principle: start with one agent, split only for a specific reason, and always measure to confirm that decision. The principles below…

2 min read

Email newsletter

Receive a selection of new LeoNex News articles by email.

We only use this address to send the newsletter.

Receive a selection of new LeoNex News articles by email. You will receive a confirmation email; the subscription only takes effect after you confirm. Every email includes an unsubscribe link. Read the privacy policy