Automated review summaries with a feedback loop: LLMs in production at Grab

In this article
The Automated Merchant Review Summary System turns handwritten store reviews into concise summaries. Consumers can view an overall summary or choose a specific topic, while merchants (stores, selling partners) learn what customers appreciate and which areas need improvement. The pipeline combines representative sampling, summarization by a large language model (LLM), automated quality checks and user feedback.
The problem with long review lists
A long, unstructured review list causes two related problems. Consumers spend time looking for comments that match their priorities, while merchants must aggregate comments one by one to spot recurring themes. A store may have hundreds or thousands of reviews; reading only the first few easily gives an incomplete picture, while the average star rating offers very little context. Two restaurants with the same number of stars can still differ in food quality, delivery experience, portion size or perceived value.
On the merchant side, reading and categorizing every review is time-consuming and gets harder as volume grows. Recurring complaints or positive points can get buried among individual comments.
Three existing approaches all have limits:
Manual analysis: reading each review and drawing conclusions yourself. The conclusions depend on which reviews the reader chooses to read and how they interpret them, and this approach is not feasible as volume grows.
Aggregate star ratings: they give a quick sense of overall opinion but do not explain why, so merchants see overall performance without seeing the themes that need attention.
Keyword search and filters: useful when the reader already knows what to look for, but they do not surface unexpected themes, and literal matching can separate phrases that mean the same thing. For example, the three phrases "too cold", "not warm enough" and "lukewarm" all describe a food temperature problem.
Summary generation pipeline architecture
The summary generation pipeline prepares the source review data, produces the summary, then checks the output before publishing. The system creates one overall summary and topic-specific summaries such as taste, packaging and portion size. The overall system has three flows: generating and validating summaries; maintaining summaries; and displaying summaries and collecting feedback.
The first step is data preparation and representative sampling. The pipeline computes the distribution of the merchant's ratings, and a sentiment model assigns each handwritten review a score from one to five. Stratified sampling then selects a smaller set that follows this distribution, reducing the risk of one score group dominating the summary. However, matching the rating distribution does not guarantee that the sample is representative of every period, language, customer group or aspect of the merchant's experience.
Next, the system sends the sampled reviews to an LLM along with instructions to produce two types of output:
A concise summary of the sampled reviews.
Topic-specific summaries for groups such as taste, packaging and portion size.
The system can also update the set of topics when new topics appear in consumer feedback.
Quality control with an LLM
In the quality control (QC) step, a separate LLM receives the generated summary together with the sampled reviews, then checks whether the source reviews support each claim. A summary that fails the validation criteria is held back and not published. Passing this automated evaluation step reduces, but does not eliminate, the risk of unsupported claims appearing.
Maintaining summaries over time
The maintenance flow updates stored summaries as new reviews and consumer feedback arrive. The system monitors both sources and regenerates a summary when either reaches a configured threshold. There are two triggers:
New-review trigger: a monitoring service counts the reviews submitted since the latest summary. When the count reaches the configured threshold, the service starts the generation and validation flow.
User-feedback trigger: the interface records whether consumers found the summary helpful. When negative feedback reaches the configured threshold, the system generates and validates a new summary.
On the display side, the interface retrieves the validated summary for the merchant's review page and records whether consumers found it helpful.
Impact and lessons learned
The main lesson is that summary quality depends on more than the text-generation step. The source sample must reflect the intended audience, every claim must be grounded in source reviews, and the output must change in response to the feedback beneath it. Automated LLM validation is only one layer of defense, not the sole proof of accuracy, and the feedback-driven regeneration mechanism needs safeguards against abuse to remain trustworthy.
This pipeline's scope is consumer-facing summaries. A misleading or incomplete summary can affect how consumers perceive a merchant, but the complaint and dispute mechanism for merchants sits outside the pipeline and is not part of the flow described.


