Real-time bidding (RTB) data is time-series data consisting of discrete auction events ordered by timestamp. The required schema for all RTB data uploaded to the Metamarkets Real-time Data Ingestion (RDI) platform are based on OpenRTB API Specification Version 2.5.

The schema specified in this document is intended for use with RTB exchange and Supply Side Platform (SSP) data. Use this schema to format your RTB data for RDI.

Schema Overview

Data for RDI consists of lines of auction and impression records (or other events, such as clicks), one record per line. These records should be structured according to the schema shown in the following sections. This schema is based on the Open RTB API Specification Version 2.5 standard.

The RTB bid responses are included with the RTB bid request to form an auction summary record. Impression and/or click records may be sent in separate real-time feeds; Metamarkets RDI performs a join in real-time of these feeds to the downstream information. The following table is an overview of the auction summary record and select objects contained within it.

Impression, Click, or Conversion records are sent independently of the auction summary record and are described by the below:

• Impression win record – Indicates that the impression was delivered/served, and typically includes financial information about the Publisher Revenue, Advertiser Charge Price, and any exchange fees. RDI joins this information back to the auction summary record during processing.
• Click record – Indicates that a click occurred, and may simply consist of a timestamp and the ID of the original request to which the click pertains. RDI joins this information back to the original request.
• Conversion record - Indicates that the desired intent of the advertisement went to completion and typically includes the original auction summary record with the conversion object appended.

For a detailed view of these records, see the JSON example provided in sections below.

Tips for Better Performing Dashboards

All of the data you deliver to Metamarkets will be processed and eventually surfaced back to you in the form of a dashboard. As you think about your requirements and begin planning what data you'll deliver, it's advisable to keep the following tips in mind to reduce errors and help assure that that dashboard performs well. The data you send has a direct impact on dashboard performance.

Start With As Few Metrics As Possible

Metrics are numbers. More metrics increase the amount of data you must send, require storage, and potentially involve computations that can slow down queries. Metrics can be added at a later time if needed.

Avoid the Uniques and Big Dimensions

When possible, avoid the following, as they can significantly increase data volume and be expensive to query:

• Tracking uniques
• Creating dimensions with a large number of values (more than 100,000)
• Having multi-valued dimensions (such as categories or keywords)

Avoid Null Values

In general, null values should not be included. Null fields, arrays, or nested objects should not be included in any of the objects.

The timestamp field is required. Any records with missing or incorrectly formatted timestamps will be thrown away by Metamarkets RDI. Because Metamarkets will be joining the records shown below using the unique identifier for the request as the primary key, any records missing the "request_id" field will also not be surfaced on the Metamarkets dashboard.

Dimensions in the Auction Summary Record

The following tables define the dimensions found in the auction summary record. Each table corresponds to one of the following:

• The set of dimensions immediately under the auction summary record (top-level dimensions)
• Dimensions immediately under an object encapsulated by the auction summary record
• Dimensions immediately under a child object of an object encapsulated by the auction summary record

Dimensions Immediately Under the Auction Summary Record

Impression Object

The following dimensions are immediately under the imp object:

The following dimensions are encapsulated by the banner object:

The following dimensions are encapsulated by the video object:

The following dimensions are encapsulated by the audio object:

App and Site Objects

An auction summary record contains either an app or a site object. These two objects share fields, except as indicated in the Dimension column.

Device Object

The following dimensions are immediately under the device object:

The following dimensions are encapsulated by the geo object:

The following dimension is encapsulated by an ext object:

User Object

The following dimensions are encapsulated by the user object:

The following dimensions are encapsulated by the data object:

The following dimensions are encapsulated by the segment object:

Bid Response Object

The tables below define the dimensions and objects immediately under the bid response object. For elaboration on any field see Open RTB 2.5 Section 4.

Dimension Immediately Under the Bid Response Object

The following dimensions are encapsulated by the seatbid object:

The following dimensions are encapsulated by the bid object:

Metrics in the Auction Summary Record

Metrics are fields with numeric values that can be aggregated or used in computations.

Raw Metrics

Raw metrics are those directly included (or implied) in the raw data delivered to RDI.

Derived and Computed Metrics

Certain metrics can be derived or computed by Metamarkets, either at processing or query time.

Sample Auction Summary Record (with Nested Bids)

The example below shows how an auction summary event (a single record) should be structured. If the auction is successful, the identity of the winning bidder should be denoted with an "auction_winner" flag in the bid response object. This structure also allows for communicating additional data about bids, such as duration of the response and information about bid errors (including timeouts, bad JSON, blocked creatives, etc.). While the example below represents a simple auction summary record for mobile, video, and private marketplace, data should be added to the request in accordance with the OpenRTB API Specification (v2.5).

Note: Example JSON records are "pretty printed" for readability. However, records must be delivered to RDI in a flat format, one record per line, each record separated from the next by a newline character.

    {
      "id": "AFEWSEBD5EB5FI32DASFCD452BB78DVE",
      "timestamp": "2014-03-05T04:58:23.000Z",
      "at": 2,
      "bcat": ["IAB26","IAB25"],
      "imp": [
        {
          "tagid": "231",
          "displaymanager": "MyRenderer",
          "displaymanagerver": "v2",
          "secure": 0,
          "bidfloor": 0.1,
          "banner": {
            "h": 320,
            "w": 50,
            "pos": 3,
            "expandable": [2,3],
            "api": [3,4]
          }
        }
      ],
      "app": {
        "id": "12312312",
        "name": "Unicornssay",
        "domain": "unicornssay.com",
        "bundle": "bundlename",
        "cat": [
          "IAB1"
        ],
        "publisher": {
          "id": "DSA1394D42D3",
          "name": "Unicornssay"
        }
      },
      "site": {
        "id": "1345135123",
        "name": "Unicornssay",
        "domain": "unicornssay.com",
        "cat": [
          "IAB1",
          "IAB2"
        ],
        "publisher": {
          "name": "Publisher A",
          "id": "pub12345"
        }
      },
      "device": {
        "geo": {
          "city": "US  SFO",
          "country": "USA",
          "region": "CA",
          "zip": "94107",
          "type": 1,
          "metro": "807"
        },
        "connectiontype": 2,
        "devicetype": 1,
        "osv": "4.2.1",
        "os": "iOS",
        "model": "iPhone 3GS",
        "make": "Apple",
        "carrier": "Verizon",
        "ip": "192.168.1.8",
        "didsha1": "AA003EABFB29E6F759F3BDAB34E50BB11",
        "dpidmd5": "A4003FABFB52E6F8A4003FABFB52E6F81",
        "ext": {
          "uid": "AA000DFE74168477C70D291f57"
        }
      },
      "user": {
        "id": "456789876567897654678987656789",
        "yob": 1987,
        "gender": "M",
        "data": [
          {
            "id": "123",
            "name": "bluesky",
            "segment": [
              {
                "id": "abc1",
                "name": "gender",
                "value": "male"
              }
            ]
          }
        ]
      },
      "bid_responses": [
        {
          "timestamp": "2014-03-05T04:58:23.200Z",
          "status": 1,
          "total_duration": 43,
          "bidder_id": "1921",
          "bidder_name": "Real Ads",
          "cur": "USD",
          "seatbid": [
            {
              "seat": "512",
              "bid": [
                {
                  "status": 1,
                  "clear_price": 1.1,
                  "attr": [
                    1,
                    2,
                    3,
                    4,
                    5,
                    6,
                    7,
                    12
                  ],
                  "crid": "1234",
                  "cid": "229",
                  "id": "1",
                  "impid": "102",
                  "price": 5.43,
                  "adid": "314",
                  "adomain": [
                    "realtime4real.mmx.org"
                  ],
                  "iurl": "http://adserver.com/pathtosampleimage"
                }
              ]
            }
          ]
        },
        {
          "timestamp": "2014-03-05T04:58:23.200Z",
          "status": 1,
          "total_duration": 99,
          "bidder_id": "1325",
          "bidder_name": "JumpUp",
          "cur": "USD",
          "seatbid": [
            {
              "seat": "12",
              "bid": [
                {
                  "status": 3,
                  "attr": [
                    1,
                    12
                  ],
                  "crid": "12333",
                  "cid": "2232",
                  "id": "1",
                  "impid": "102",
                  "price": 1.1,
                  "adid": "33",
                  "adomain": [
                    "carscarscars.com"
                  ],
                  "iurl": "http://adserver.com/pathtosampleimage"
                }
              ]
            }
          ]
        },
        {
          "timestamp": "2014-03-05T04:58:23.800Z",
          "status": 3,
          "total_duration": 303,
          "bidder_id": "2144",
          "bidder_name": "Turning.com",
          "cur": "USD"
        }
      ]
    }

As shown in the example above, the auction summary record provides a way to denote the statuses/outcomes of bid activity, which is then used to provide rich analytics on bidding health in your Metamarkets dashboard. At the response level, these codes help identify issues that prevent bids from being included in the auction (timeouts, HTTP errors, bad JSON, etc.). At the bid level, the codes enable the tracking of granular bid statuses/outcomes such as types of blocks, above/below floor, etc.

BidResponse-level Status ("bid_responses[].status")

1 = Valid
2 = Timeout
3 = Invalid JSON
4 = HTTP Error

Note: In the event of a timeout, it may also be useful to analyze the response durations. You should populate the "total_duration" field in the response record to add this visibility. For responses with "status" other than "Valid", the seatBid object may be omitted.

Bid-level Status ("seatbid[].bid[].status")

1 = Won
2 = Lost on Price (i.e., a normal loss)
3 = Below Floor
4 = Markup Delivery Failure (i.e., forfeited the win by not returning markup in the bid or the win notice)
5 = Unscreenable (i.e., did not send the minimal attributes to make it screenable by sellers, such as "iurl", "adomain", etc.)
6 = Blocked by Publisher
7 = Unverified Creative (i.e., publisher requires human review of creatives and this creative has not yet been reviewed)
8 = Blocked Advertiser (i.e., violated the block list of advertiser domains)
9 = Blocked Content Category (i.e., violated the block list of content categories)
10 = Blocked Creative Attribute (i.e., violated the block list of creative attributes)

Mapping Explicit Values to IDs

While categorical dimension fields (e.g., publisher or campaign name) can be provided directly in the raw data, you could instead choose to pass an ID value, which RDI can then map to the appropriate string value using Handling Lookup Tables you provide. Because the OpenRTB API Spec v2.2 is supported, OpenRTB-compliant values may also be provided (e.g., integer values "1" through "7" for Ad Position).

Sample Impression Records

The example below shows the format for an impression record. While exchanges often need to deliver impression and revenue information separately from the associated auction and bid data, you can include this information in the auction summary record if preferred. Provided that the Charge Price and Publisher Revenue are included, RDI can compute the exchange/SSP revenue amount. If impressions are sent separately from the auction summary record, an additional RDI endpoint will be provided.

Note The value of request_id is the auction ID. The timestamp applies to the impression.

    {
        "timestamp": "2014-03-05T04:58:23.200Z",
        "request_id": "AFEWSEBD5EB5FI32DASFCD452BB78DVE",
        "charge_price": 1.15,
        "publisher_revenue": 1.05
    }

Sample Click Records

Clicks can also be delivered in a subsequent record. This record consists of a timestamp and the ID of the original request to which the click pertains. In the event that clicks can be delivered, an additional RDI endpoint will be provided.

Note The value of request_id is the auction ID. The timestamp applies to the time of the click.

    {
        "timestamp": "2014-03-05T04:58:23.200Z",
        "request_id": "AFEWSEBD5EB5FI32DASFCD452BB78DVE"
    }

Sample Conversion Records

Depending on your business, you may track different events after the auction such as app downloads or purchases. These conversions may happen hours (or even days) after the auction or impression, precluding a join with the auction record because it missed the real-time window period: 20 minutes, and even the batch fix-up window period: 4 hours. To best manage this, we recommend appending all the dimensions & values from the original auction when posting conversion events. These conversion events should be posted to a separate endpoint provided by Metamarkets and can include additional dimensions/metrics within the conversions object. At minimum, the timestamp field should reflect when the conversion occurred and must be included in the top-level timestamp. Below is an example of an abbreviated conversion JSON:

    {
      "id": "AFEWSEBD5EB5FI32DASFCD452BB78DVE",
      "timestamp": "2014-03-07T03:15:00.000Z",
      "auction_timestamp": "2014-03-07T03:15:00.000Z",
      "at": 2,
      "bcat": ["IAB26","IAB25"],
      "app": { ... },
      "imp": [ ... ],
      "device": { ... },
      "user": { ... },
      "bid_responses": [ ... ],
      "conversions":  {
            "conversion_charge": 10,
            "conversion_type": "Subscription"
              }
    }

With identical dimensions, the original Auction Record & the Conversion Record are unioned together and surfaced immediately on the dashboard.

Providing Data for Uniques Calculations

Metamarkets uses the hyperloglog algorithm to approximate the number of unique users based off of unique IDs provided in the auction summary record. While device or user IDs may be present in the request, you may need to vary which unique identifier should be used for a particular auction. As such, you may also provide an additional dimension for use in the uniques calculations. Common IDs used include:

• An internal exchange ID
• A device ID available in the request
• A third-party ID such as from a DMP
• Any other unique ID designated by you for uniques calculations

Data Delivery Requirements and Processing Summary

Upon receiving the auction summary, impression and click records, Metamarkets RDI performs a join, ultimately storing all data in denormalized form.

In order to perform the join with real-time processing, a collection window of up to 20 minutes is used. Data delivered in time for real-time processing is surfaced on the Metamarkets dashboard immediately at the end of the 20-minute collection window.

Impressions and clicks that come in more than 20 minutes after the request are still captured, but are not surfaced on the dashboard until a batch fix-up job runs. Batch fix-up jobs run approximately every 4 hours, allowing for impressions, clicks or late data to be received and surfaced on the dashboard.

When the batch fix-up job runs for a particular hour, all data received for that hour and the previous hour is also reprocessed, creating a join window of up to 2 hours to include clicks and impressions that occur near the time of the original request.