Booking.com
Booking.com B.V.
Government orders to act against illegal content
Article 15(1)(a)
Notices received from users and flaggers
Article 16
Own-initiative moderation
Article 15(1)(c) and (d)
802,337Actions under terms & conditions
—Actions against illegal content
7.28%Share taken solely by automated means (ToS)
Restriction types applied (terms & conditions)
Account-level actions
Article 15(1)(d)
| Account suspensions | 2,865 |
|---|---|
| Account terminations | 294 |
| Total account actions | 3,159 |
Automated detection accuracy
Booking.com reports the accuracy of its automated detection. These are its own figures, measured against its own method and denominators, and are not comparable with other providers'. The detection tool or method is shown as filed.
| Tool or method | Scope | Accuracy | Precision | Recall |
|---|---|---|---|---|
| — | Own-initiative | — | 39.5% | 63.8% |
| — | Total number | 99.6% | 39.5% | 63.8% |
| The accuracy rate is calculated for our core models only and excludes our ancillary models. | Own-initiative | 99.6% | — | — |
Show per-language figures (23)
| Tool or method | Language | Accuracy | Precision | Recall |
|---|---|---|---|---|
| — | bg | 99.9% | — | — |
| — | cs | 99.9% | 35.5% | 35.8% |
| — | da | 99.9% | 56.0% | 67.3% |
| — | de | 99.9% | 16.1% | 46.3% |
| — | el | 99.7% | 35.8% | 41.1% |
| — | en | 99.8% | 49.1% | 88.7% |
| — | es | 99.7% | 31.5% | 35.0% |
| — | et | 99.9% | 21.9% | 66.2% |
| — | fi | 100.0% | 11.4% | 62.9% |
| — | fr | 99.9% | 53.2% | 66.4% |
| — | hr | 99.9% | 39.2% | 78.6% |
| — | hu | 99.9% | 20.0% | 50.6% |
| — | it | 99.9% | 28.0% | 88.2% |
| — | lt | 99.9% | 40.9% | 78.0% |
| — | lv | 99.9% | 16.5% | 50.5% |
| — | nl | 99.9% | 37.5% | 40.1% |
| — | pl | 99.9% | 42.7% | 69.4% |
| — | pt | 99.8% | 33.8% | 33.1% |
| — | ro | 99.9% | 49.4% | 35.9% |
| — | ru | 99.8% | 37.9% | 66.8% |
| — | sk | 99.9% | 59.4% | 48.1% |
| — | sv | 99.9% | 25.0% | 58.8% |
| Refers to the total number of measures not taken by automated means. | bg | — | 28.5% | 64.5% |
Full per-tool and per-language detection figures are inExplore (automated_means_accuracy).
In Booking.com's words
Booking.com's long-form answers to the standard qualitative questions every platform must answer (Article 42 of the DSA). How it moderates content, how it measures accuracy, how its teams are resourced. Its own words. Expand each to read. (Short notes pinned to individual figures are under "Footnotes from Booking.com" below.)
High-level description of the content moderation governance structure
We employ content moderators (both in-house and vendor-based), managed by team leads, to review text and photo content from our customers, partners and travelers. Most uploaded content undergoes pre-moderation, using a combination of machine learning and human moderation before the content is made public. A portion of the content available on our platform undergoes post-moderation, the content is moderated after it’s been published. Decisions are made based on content policies and guidelines, with quality checks conducted by our Policies Associates, managed by the Policy Lead, to ensure fair and consistent enforcement. Rejected content is removed, providing the content owner a clear reason which policy violation led to the removal and an option for the user appeal. All content rejections are reported to the European Commission Database. We have also established escalation paths for stakeholders such as Customer Service and Partner Services, handled by our in-house content moderators and associates.
Meaningful and comprehensible information regarding content moderation engaged in at the providers' own initiative
Our core ML algorithms swiftly review various types of content, such as guest reviews, partner responses, photos and more. Most content is approved within seconds and goes live on Booking.com. Anything not approved by our core ML algorithms is sent to our moderators for further review when potential guideline violations are detected. Approved content is accessible on our platform and apps, while policy-violating content won't be published. Our decision can be appealed or the content can be resubmitted.
Methodology used to compute the number of human resources dedicated to content moderation
We employ bilingual content moderators based on content volume, prioritizing languages with higher moderation needs. While we assess language requirements based on volume, we ensure all our content moderators (whether in-house or vendor-based) are proficient in English and capable of moderating other languages using translation tools.
Qualifications of the human resources dedicated to content moderation
1 Senior Manager, 1 Policy Lead, 2 Senior Leads, 1 Vendor Management and Operations Lead, 1 Enablement Lead Risk, 6 Associates, 15 Content Moderators, 33 Vendor Content Moderators. All moderators, internal and externally employed go through a 3 week moderation onboarding process and need to reach a specific quality score to perform moderation.
Qualitative description of indicators of accuracy and possible rate of error of automated means
Accuracy:
The accuracy of the moderation model reflects the overall proportion of correct predictions — both correctly identified policy violations and correctly identified safe content — across the total set of evaluated inputs. Accuracy is calculated based on a manually labeled test dataset consisting of diverse samples of user-generated content, balanced across multiple moderation policies.
Accuracy = (True Positives + True Negatives) / Total Samples. This is computed using held-out test data not seen during model training, with data labeled by human annotators for quality.
Precision:
Precision indicates how many of the items flagged as violating were actually correct, helping reduce false positives. It measures the reliability of automated content removals. Precision is measured on content automatically flagged for removal and compared against gold-standard annotations from human moderators.
Precision = True Positives / (True Positives + False Positives). A high precision score reflects that the model is making confident, correct removals, minimizing over-enforcement.
Recall
Recall captures the model’s ability to find all relevant violations — in other words, how much harmful content it successfully detects. Measured on datasets that include a known number of true violations, including edge cases and borderline content examples.
Recall = True Positives / (True Positives + False Negatives). This tells us how effective the model is at capturing all harmful content that should be removed.
Qualitative description of the automated means
The core ML models Booking.com applies are content classifier models which are tuned to identify context. The ancillary ML models use numerous different data points and fraud indicators to detect inauthentic listings and inauthentic reviews.
Safeguards applied to the use of automated means
We perform constant random sampling of the automatically approved items and send them to moderators to ensure that the quality of ML automated content approvals is within the acceptable range.
Specification of the precise purposes to apply automated means
The core ML models are designed to detect both illegal content and content that violates Booking’s content policies. Illegal and violating content identified by the core ML models is sent to our content moderators for human review. Content moderators will make the final decision. The ancillary ML models are designed and used to detect inauthentic listings and inauthentic reviews at scale.
Summary of the content moderation engaged in at the providers’ own initiative
Booking.com’s long-held values as well as our guidelines and terms and conditions for all users of our platform - travellers and supply partners - are designed to foster safe and welcoming travel experiences for all. To maintain that environment and ensure the safety of our travellers and supply partners, we take action upon content that violates the law, our Content Guidelines or Terms and Conditions. We have a fast turnaround on our content moderation, using both moderation by automated Machine Learning (ML) models and manual review.
Support given to human resources dedicated to content moderation
Internal employees can reach out to the employee assistance program that provides counselling services, practical information and digital content to support employees’ mental, physical, social and nancial well-being. All moderators have followed a course about dealing with distressing content, designed to increase awareness, learn about prevention of potential issues and to understand what support is available. Our external partner contracted for moderation has a consistent approach towards employee safety utilising SGS audits and participating with renown partners.
Training given to human resources dedicated to content moderation
When new policies are launched or a new content moderator is onboarded, training decks and videos are provided to introduce the new content policies. Content moderators spend on average approximately 6 hours monthly receiving training, reviewing content guidelines and policy clarications, reviewing their errors and asking questions. Frequently asked questions are compiled, and grey areas are claried on a regular basis.
Raw data
Every figure on this page comes from Booking.com's filing as loaded into RTFP's public database. You can query the underlying data directly via thepublic API. The original filing is linked from thesources page.
Footnotes from Booking.com
Short notes Booking.com pinned to specific figures in its filing. Definitions, clarifications and corrections written against individual numbers, shown verbatim. (For its longer descriptions of how it moderates, see "In Booking.com's words" above.)
Show 28 notes
Active monthly recipients
- Number of average monthly active recipients during the reporting periodBooking reports these as estimates: a range based on a margin applied to the best available data, rounded to the nearest 100,000, subject to statistical uncertainty and possible later revision, and counting users to whom information was displayed even without a transaction. Figures appear to be in millions: the ">45" total matches the 45,000,000 VLOP threshold, so e.g. "16.4 - 20" reads as 16,400,000 to 20,000,000.
Article 16 notices
- Actions on the basis of termsIncludes the number of properties delisted.
- Actions on the basis of termsNo reports included violent content, hence we report zero removals.
- Actions on the basis of termsOur content labelling allows us to count the amount of pieces of content removed under harassment only, hence the number of actions reported.
- Actions on the basis of termsOur terms and conditions incorporate policies on regulated content. Additionally, our content removal process is applied globally, meaning that the removal of content affects users both within and outside the EEA. As a result, we have made the decision to report all content removals on a global scale, in accordance with the service's terms and conditions.
- Actions on the basis of termsThe amount of content removed under our confidential data and privacy policies.
- Actions on the basis of termsThe amout of pieces of content removed under hate and discrimination. Harassment is reported under Cyber_Violence.
- Items in noticesAt present, our reporting forms are designed to allow users to submit a notice concerning only one single item of alleged illegal content or listing at a time. The number of specific items of information included in the total number of notices, therefore, corresponds directly to the number of notices received. If, during the investigation prompted by a user's report, additional content violating our policies is identified, it is addressed and reported separately in accordance with our terms and conditions
- Median time to actionWe calculate the median resolution time based on reports that have been closed. The median is the value at the center of the distribution when all resolution times for closed reports are listed in ascending order. The median time reflects reports that were received and subsequently closed.
- Notices receivedBooking.com has 3 reporting forms for user notices: illegal content, intellectual property and illegal listings
- Notices receivedOur current reporting form allows users to report harassment under a larger category "Hate, discrimination and harasssment", hence we cannot count how many tickets are opened specifically for Harassment
- Notices receivedOur current reporting form allows users to report harassment under a larger category "Hate, discrimination and harasssment", hence we include the count of harassment related reports here
- Notices receivedOur current reporting form allows users to report harassment under a larger category "Violent, offensive and restricted content", hence we include the count of violence related related reports in STATEMENT_CATEGORY_NOT_SPECIFIED_NOTICE which includes restricted content and offensive content. No reports related to violence were recieved.
- Notices receivedThe amount of reports recieved in our illegal listings form.
- Notices receivedThe amount of reports recieved under our illegal content category Personal and other confidential data
- Notices receivedThe amount of reports recieved under our illegal content category Spam, misleading content and deceptive practices.
- Notices receivedThis category includes reports recieved under the previous "Category: scope of platform", such as Sexual Content, Overbookings, Guest Cancellations, Conflict of interest, Promotional content, Photo & Editorial guidelines, Partner Name Change, Biased content, Offensive/ shocking content and Nudity.
- Notices received (Trusted Flaggers)Trusted Flaggers must register to receive the illegal content reporting form. During the verification process, agents ensure that the email address, name, or organization used to register matches any of the Trusted Flaggers appointed by the EC. If the registration step is invalid, the ticket is closed and the user is informed that they cannot submit illegal content reports as a Trusted Flagger. Although the form is sometimes misused by users and no actual Trusted Flagger notices have been submitted, during the audit period, we received one registration that we considered valid. However, no report has been received.
Complaints, appeals & disputes
- Complaint regarding a decision not to take action on a notice submitted in accordance with Article 16Median time is calculated in hours.
- Complaint regarding a decision to remove or disable access to or restrict visibility of informationMedian time is calculated in hours.
- Number of complaints submitted to the internal-complaints mechanismMedian time is calculated in hours.
Government orders
- Article 10: median time to inform of receiptUpon receipt of an order via the ServiceNow portal, an automatic confirmation is sent immediately to the authority. As such, the median time for acknowledging receipt is considered instantaneous.
- Article 9 orders receivedThere are cases when one request targets multiple illegal properties.
- Article 9: median time to inform of receiptUpon receipt of an order via the ServiceNow portal, an automatic confirmation is sent immediately to the authority. As such, the median time for acknowledging receipt is considered instantaneous.
Human resources
- Number of external moderators contracted by the providerVendor Content Moderator
- Number of internal moderators employed by the providerContent Moderators in Booking.com
- Number of total moderators with sufficient linguistic expertiseEmployed and Contracted Content Moderators
- Number of total moderators with sufficient linguistic expertiseNumber of Employed and Contracted Content Moderators with expetise per language