New York

London

Shanghai

オファー

もっと詳しく知る

Modern Slavery Act

Our Technologies

Investors

Tax strategy

Stay abreast with our updates and sign up to our newsletter.

Facebook

Twitter

LinkedIn

オファーのお問い合わせ

Home

ホームページ

Technology

テクノロジー

Success stories

サクセスストーリー

Resources

資力

For agencies

代理店の場合

About Us

私たちに関しては

Team

チーム

ニュース

Career

キャリア

ブログ

Our technologies

**What is RTB House Crawler?**
- RTB House Crawler is a software powered by Deep Learning that reads web pages and analyzes them to place the most relevant ads.
Why does RTB House visit your site?
To support contextual advertising, we need to analyze the public web content by crawling web pages. This lets us categorize pages and adjust advertisements.

**What kind of data does RTB House collect?**
- We collect meta tags, text sections (some or all, depending on usability) from HTML documents, and language-related headers from responses returned by the server.

**When will the site be indexed?**
- We try to access a given page as soon as it’s published - the time may vary depending on how fast we find out that it exists. Later on, the data might be refreshed, but it’s not a rule - it mostly depends on the site's popularity.

**How can I authorize/exclude the crawler?**
- We follow the rules defined in robots.txt for default user agents, meaning if a given page is not disallowed, it will be indexed.

If you want to block robots on all sites, add the following:
User-agent: *
Disallow: /

If you want to block a particular path with exceptions:  
User-agent: *
Allow: /path/allowed
Disallow: /path

You can also block everything with exceptions:
User-agent: *
Allow: /path/exception2
Allow: /path/exception1
Disallow: /

More information about creating a robots.txt file https://developers.google.com/search/docs/crawling-indexing/robots/create-robots-txt

**How long does it take to update the robots.txt published by a domain?**
- The crawler checks the robots.txt file once a week for each domain, meaning if the robots.txt is updated, we will sync its status within a week or less.

**More information**
- If you need to know more about the crawler, please email us at crawler@rtbhouse.com.

RTB House Crawler

data created by MaxMind, available at https://www.maxmind.com

GeoLite2

Our products include the following technologies

**Technologies that we use in** our products 

Create truly personalized marketing campaigns powered by Deep Learning technology that cut through the online noise. 100% ready for the cookieless world.

Technology Powered by Deep Learning

 | RTB House

TECHNOLOGIES

Our technologies