Real-time bidding (RTB) data is time-series data consisting of discrete auction events ordered by timestamp. The required schema for all RTB data uploaded to the Metamarkets Realtime Data Ingestion (RDI) platform are based on OpenRTB API Specification Version 2.5.

The schema specified in this document is intended for use with Demand Side Platform (DSP) data. Use this schema to format your RTB data for RDI.

Schema Overview

Data for RDI consists of lines of auction and impression records (or other events, such as clicks), one record per line. These records must be structured according to the schema shown in this section. This schema is based on the Open RTB API Specification Version 2.5 standard.

The RTB bid responses are included with the RTB bid request to form an auction summary record. Impression and/or click records may be sent in separate realtime feeds; Metamarkets RDI performs a join in real time of these feeds to the downstream information. The following list is an overview of these records, as well as selected objects embedded in those records:

Impression, Click, or Conversion records are sent independently of the auction summary record and are described by the below:

• Impression win record – Indicates that the impression was delivered/served, and typically includes financial information about the Publisher Revenue, Advertiser Charge Price, and any exchange fees. RDI joins this information back to the auction summary record during processing.
• Click record – Indicates that a click occurred, and may simply consist of a timestamp and the ID of the original request to which the click pertains. RDI joins this information back to the original request.
• Conversion record - Indicates that the desired intent of the advertisement went to completion and typically includes the original auction summary record with the conversion object appended.

For a detailed view of these records, see the JSON example provided in sections below.

Tips for Better Performing Dashboards

All of the data you deliver to Metamarkets will be processed and eventually surfaced back to you in the form of a dashboard. As you think about your requirements and begin planning what data you'll deliver, it's advisable to keep the following tips in mind to reduce errors and help assure that that dashboard performs well. The data you send has a direct impact on dashboard performance.

Start With As Few Metrics As Possible

Metrics are numbers. More metrics increase the amount of data you must send, require storage, and potentially involve computations that can slow down queries. Metrics can be added at a later time if needed.

Avoid the Uniques and Big Dimensions

When possible, avoid the following, as they can significantly increase data volume and be expensive to query:

• Tracking uniques
• Creating dimensions with a large number of values (more than 100,000)
• Having multi-valued dimensions (such as categories or keywords)

Avoid Null Values

In general, null values should not be included. Null fields, arrays, or nested objects should not be included in any of the objects.

The timestamp field is required. Any records with missing or incorrectly formatted timestamps will be thrown away by Metamarkets RDI. Because Metamarkets will be joining the records shown below using the unique identifier for the request as the primary key, any records missing the "request_id" field will also not be surfaced on the Metamarkets dashboard.

Dimensions in the Auction Summary Record

The following tables define the dimensions found in the auction summary record. Each table corresponds to one of the following:

• The set of dimensions immediately under the auction summary record
• Dimensions immediately under an object encapsulated by the auction summary record
• Dimensions immediately under a child object of an object encapsulated by the auction summary record

Dimensions Immediately Under the Auction Summary Record

Advertiser Object

The following dimensions are immediately under the advertiser object:

Impression Object

The following dimensions are immediately under the imp object:

The following dimensions are encapsulated by the banner object:

App and Site Objects

An auction summary record contains either an app or a site object. These two objects share fields, except as indicated in the Dimension column.

Device Object

The following dimensions are immediately under the device object:

The following dimensions are encapsulated by the geo object:

The following dimension is encapsulated by an ext object:

User Object

The following dimensions are encapsulated by the user object:

Data Object

The following dimensions are immediately under the data object:

The following dimensions are encapsulated by the segment object:

Bid Response Object

The tables below define the dimensions and objects immediately under the bid response object. For elaboration on any field see Open RTB 2.5 Section 4.

Dimension Immediately Under the Bid Response Object

The following dimensions are encapsulated by the seatbid object:

The following dimensions are encapsulated by the bid object:

Metrics in the Auction Summary Record

Metrics are fields with numeric values that can be aggregated or used in computations.

Raw Metrics

Raw metrics are those directly included (or implied) in the raw data delivered to RDI.

Derived and Computed Metrics

Certain metrics can be derived or computed by Metamarkets, either at processing or query time.

If there are other computed metrics that you would like included in your Metamarkets dashboard, contact your account manager.

Sample Auction Summary Record (with Nested Bids)

The example below shows how an auction summary event (a single record) should be structured. If the auction is successful, the identity of the winning bidder should be denoted with an "auction_winner" flag in the bid response object. This structure also allows for communicating additional data about bids, such as duration of the response and information about bid errors (including timeouts, bad JSON, blocked creatives, etc.). While the example below represents a simple auction summary record for mobile, video, and private marketplace, data should be added to the request in accordance with the OpenRTB API Specification (v2.5).

Note Example JSON records are "pretty printed" for readability. However, records must be delivered to RDI in a flat format, one record per line, each record separated from the next by a newline character.

    {
      "id": "AFEWSEBD5EB5FI32DASFCD452BB78DVE",
      "timestamp": "2014-03-05T04:58:23.000Z",
      "at": 2,
      "bcat": ["IAB26","IAB25"],
      "imp": [
        {
          "tagid": "231",
          "displaymanager": "MyRenderer",
          "displaymanagerver": "v2",
          "bidfloor": 0.1,
          "banner": {
            "h": 320,
            "w": 50,
            "pos": 3
          }
        }
      ],
      "app": {
        "id": "12312312",
        "name": "Unicornssay",
        "domain": "unicornssay.com",
        "bundle": "bundlename",
        "cat": [
          "IAB1"
        ],
        "publisher": {
          "id": "DSA1394D42D3",
          "name": "Unicornssay"
        }
      },
      "site": {
        "id": "1345135123",
        "name": "Unicornssay",
        "domain": "unicornssay.com",
        "cat": [
          "IAB1",
          "IAB2"
        ],
        "publisher": {
          "name": "Publisher A",
          "id": "pub12345"
        }
      },
      "device": {
        "geo": {
          "city": "US  SFO",
          "country": "USA",
          "region": "CA",
          "zip": "94107",
          "type": 1,
          "metro": "807"
        },
        "connectiontype": 2,
        "devicetype": 1,
        "osv": "4.2.1",
        "os": "iOS",
        "model": "iPhone 3GS",
        "make": "Apple",
        "carrier": "Verizon",
        "ip": "192.168.1.8",
        "ext": {
          "did": "12-3131-13913931-AD32"
        }
      },
      "user": {
        "id": "456789876567897654678987656789",
        "yob": 1987,
        "gender": "M",
        "data": [
          {
            "id": "123",
            "name": "bluesky",
            "segment": [
              {
                "id": "abc1",
                "name": "gender",
                "value": "male"
              }
            ]
          }
        ]
      },
      "bid_responses": [
        {
          "timestamp": "2014-03-05T04:58:23.200Z",
          "total_duration": 43,
          "bidder_id": "1921",
          "bidder_name": "Real Ads",
          "cur": "USD",
          "seatbid": [
            {
              "seat": "512",
              "bid": [
                {
                  "advertiser": {
                    "cat": ["IAB2","IAB9"],
                    "aid": "12345"
                  },
                  "attr": [
                    1,
                    2,
                    3,
                    4,
                    5,
                    6,
                    7,
                    12
                  ],
                  "crid": "1234",
                  "cid": "229",
                  "id": "1",
                  "impid": "102",
                  "price": 5.43,
                  "adid": "314",
                  "adomain": [
                    "realtime4real.mmx.org"
                  ],
                  "iurl": "http://adserver.com/pathtosampleimage"
                }
              ]
            }
          ]
        }
      ]
    }

Mapping String Values to IDs

While categorical dimension fields (e.g., publisher or campaign name) can be provided directly in the raw data, you could instead choose to pass an ID value, which RDI can then map to the appropriate string value using Handling Lookup Tables you provide. Because the OpenRTB API Spec v2.5 is supported, OpenRTB-compliant values may also be provided (e.g., integer values "1" through "7" for Ad Position).

Win Notification Record

While request & bid information should be delivered simultaneously via the auction summary record, a win notification may be delivered as a separate record. This record should consist of the timestamp of the event, the original request_id (to be used in the join), and the charge price for the inventory. A separate RDI endpoint will be provided for win notification events.

Note The value of request_id is the auction ID. The timestamp applies to the time of the win.

    {
        "timestamp": "2014-03-05T04:58:23.200Z",
        "request_id": "AFEWSEBD5EB5FI32DASFCD452BB78DVE",
        "charge_price": 1.15
    }

The Impression Record

The example below shows the format for an impression record. This record is used to indicate that the impression was delivered after being won. It should consist of the timestamp of the impression and a request_id (to be used for joining back to the original request). The click record may also contain financial/pricing information if the DSP charges customers based on clicks. An additional RDI endpoint will be provided for impression events.

    {
        "timestamp": "2014-03-05T04:58:23.200Z",
        "request_id": "AFEWSEBD5EB5FI32DASFCD452BB78DVE",
        "advertiser_charge_price": 1.2
    }

Click Records

Clicks can also be delivered in a subsequent record. This record consists of a timestamp and the ID of the original request to which the click pertains. The click record may also contain financial/pricing information if the DSP charges customers based on clicks. In the event that clicks can be delivered, an additional RDI endpoint will be provided.

Note The value of request_id is the auction ID. The timestamp applies to the time of the click.

    {
        "timestamp": "2014-03-05T04:58:23.200Z",
        "request_id": "AFEWSEBD5EB5FI32DASFCD452BB78DVE",
        "advertiser_charge_price": 1.2
    }

Conversion Records

Depending on your business, you may track different events after the auction such as app downloads or purchases. These conversions may happen hours (or even days) after the auction or impression, precluding a join with the auction record because it missed the real-time window period: 20 minutes, and even the batch fix-up window period: 4 hours. To best manage this, we recommend appending all the dimensions & values from the original auction when posting conversion events. These conversion events should be posted to a separate endpoint provided by Metamarkets and can include additional dimensions/metrics within the conversions object. At minimum, the timestamp when the conversion occured should be included in the conversion object. Below is an example of an abbreviated conversion JSON:

With identical dimensions, the original Auction Record & the Conversion Record are unioned together and surfaced immediately on the dashboard.

{
      "id": "AFEWSEBD5EB5FI32DASFCD452BB78DVE",
      "timestamp": "2014-03-05T04:58:23.000Z",
      "at": 2,
      "bcat": ["IAB26","IAB25"],
      "app": { ... },
      "imp": [ ... ],
      "device": { ... },
      "user": { ... },
      "bid_responses": [ ... ],
      "conversions":  {
            "conversion_timestamp": "2014-03-07T03:15:00.000Z",
            "conversion_charge": 10,
            "conversion_type": "Subscription"
              }
    }

Providing Data for Uniques Calculations

Metamarkets uses the hyperloglog algorithm to approximate the number of unique users based off of unique IDs provided in the auction summary record. While device or user IDs may be present in the request, you may need to vary which unique identifier should be used for a particular auction. As such, you may also provide an additional dimension for use in the uniques calculations. Common IDs used include:

• An internal exchange ID
• A device ID available in the request
• A third-party ID such as from a DMP
• Any other unique ID designated by you for uniques calculations

Data Delivery Requirements & Processing Summary

Upon delivery of auction summary, impression and click records, Metamarkets RDI will perform a join, ultimately storing all data in denormalized form.

In order to perform the join with real-time processing, a collection window of up to twenty minutes will be used. Data delivered in time for real-time processing will be surfaced on the Metamarkets dashboard immediately at the end of the 20 minute collection window. Impressions and clicks that come in more than 20 minutes after the request will still be captured, but will not be surfaced on the dashboard until the batch fix-up job runs.

A batch fix-up job will be set to run on a delay of at least 4-hours, allowing for impressions, clicks or late data to be received and surfaced on the dashboard.

When the batch fix-up job runs for a particular hour, all data received for that hour and the previous hour will also be reprocessed, creating a join window of up to 2 hours to allow for clicks and impressions that occur in the hour following the original request.