Integration Guide for Exchanges

Real-time bidding (RTB) data is time-series data consisting of discrete auction events ordered by timestamp. The required schema for all RTB data uploaded to the Metamarkets Real-time Data Ingestion (RDI) platform are based on OpenRTB API Specification Version 2.5.

The schema specified in this document is intended for use with RTB exchange and Supply Side Platform (SSP) data. Use this schema to format your RTB data for RDI.

Schema Overview

Data for RDI consists of lines of auction and impression records (or other events, such as clicks), one record per line. These records should be structured according to the schema shown in the following sections. This schema is based on the Open RTB API Specification Version 2.5 standard.

The RTB bid responses are included with the RTB bid request to form an auction summary record. Impression and/or click records may be sent in separate real-time feeds; Metamarkets RDI performs a join in real-time of these feeds to the downstream information. The following table is an overview of the auction summary record and select objects contained within it.

ObjectDescription
BidRequestTop-level object that includes the remaining listed objects.
ImpAn array of objects each containing a description of a creative that is being bid on. Only the first object is used. This object includes banner, video, and audio child objects.
AppFor in-app impressions this object contains metadata on the application and publisher. Used in absence of site object.
SiteFor page impressions this object contains metadata on the site and publisher. Used in absence of app object.
DeviceMetadata on the user's device and location. This object includes geo and ext child objects.
UserMetadata on the user. This objects includes the data child object which encapsulated segment.
BidResponseContains bid response data including seatbid and bid objects.
ExtThis object can be a child of any other object. It encapsulated non-standard fields that have been pre-approved by Metamarkets.

Impression, Click, or Conversion records are sent independently of the auction summary record and are described by the below:

  • Impression win record – Indicates that the impression was delivered/served, and typically includes financial information about the Publisher Revenue, Advertiser Charge Price, and any exchange fees. RDI joins this information back to the auction summary record during processing.
  • Click record – Indicates that a click occurred, and may simply consist of a timestamp and the ID of the original request to which the click pertains. RDI joins this information back to the original request.
  • Conversion record - Indicates that the desired intent of the advertisement went to completion and typically includes the original auction summary record with the conversion object appended.

For a detailed view of these records, see the JSON example provided in sections below.

Tips for Better Performing Dashboards

All of the data you deliver to Metamarkets will be processed and eventually surfaced back to you in the form of a dashboard. As you think about your requirements and begin planning what data you'll deliver, it's advisable to keep the following tips in mind to reduce errors and help assure that that dashboard performs well. The data you send has a direct impact on dashboard performance.

Start With As Few Metrics As Possible

Metrics are numbers. More metrics increase the amount of data you must send, require storage, and potentially involve computations that can slow down queries. Metrics can be added at a later time if needed.

Avoid the Uniques and Big Dimensions

When possible, avoid the following, as they can significantly increase data volume and be expensive to query:

  • Tracking uniques
  • Creating dimensions with a large number of values (more than 100,000)
  • Having multi-valued dimensions (such as categories or keywords)

Avoid Null Values

In general, null values should not be included. Null fields, arrays, or nested objects should not be included in any of the objects.

The timestamp field is required. Any records with missing or incorrectly formatted timestamps will be thrown away by Metamarkets RDI. Because Metamarkets will be joining the records shown below using the unique identifier for the request as the primary key, any records missing the "request_id" field will also not be surfaced on the Metamarkets dashboard.

Dimensions in the Auction Summary Record

The following tables define the dimensions found in the auction summary record. Each table corresponds to one of the following:

  • The set of dimensions immediately under the auction summary record (top-level dimensions)
  • Dimensions immediately under an object encapsulated by the auction summary record
  • Dimensions immediately under a child object of an object encapsulated by the auction summary record

Dimensions Immediately Under the Auction Summary Record

DimensionField (Data Type)Definition
Timestamptimestamp (string)REQUIRED. The timestamp of the event in ISO format (e.g. YYYY-MM-DDTHH:mm:SSZ"). The timezone should be UTC.
Request IDid (string)REQUIRED. The unique auction identifier. Appears as "id" at the top of the auction record, and as "request_id" in the Impression Win and Click records.
Auction Typeat (integer)The type of auction: 1 = First price, 2 = Second price.
Blocked Attributesbattr (array of integers)Blocked creative attributes. Use List 5.3 of the OpenRTB API Specification for values.
Blocked IAB Categoriesbcat (string)Blocked advertiser categories, based on IAB content categories. For example: IAB1 = "Arts & Entertainment". Use List 5.1 of the OpenRTB API Specification for values.
Blocked Creative Typesbtype (array of integers)Blocked creative types. Use List 5.2 of the OpenRTB API Specification for values.

Impression Object

The following dimensions are immediately under the imp object:

DimensionField (Data Type)Definition
Bid Floor Pricebidfloor (double)Price-floor of auction in CPM.
Is Securesecure (boolean)Indicates whether the impression requires secure assets ("1") and must use HTTPS, or not ("0"). If this field is omitted, then no secure assets are required and HTTP is used.
Placement Nametagid (string)Ad placement ID (ad tag).
SDK Namedisplaymanager (string)Name of ad mediation partner, SDK technology, or native player responsible for rendering ad (typically video or mobile).
SDK Versiondisplaymanagerver (string)Version of ad mediation, SDK technology, or native player responsible for rendering ad.

The following dimensions are encapsulated by the banner object:

DimensionField (Data Type)Definition
Ad Positionpos (integer)The actual position of the ad. For example: 1 = "Above the fold". Use List 5.4 of the OpenRTB API Specification for values.
Ad Sizeh, w (integer)Height and width of an impression in pixels. These exist as separate fields in the data you deliver, but are concatenated before being surfaced.
Expandableexpandable (array of integers)Directions an ad can expand. For example, 1 = "Left", 2 = "Right". Use List 5.5 of the OpenRTB API Specification for values.
Supported API frameworksapi (array of integers)Supported API frameworks for the impression.

The following dimensions are encapsulated by the video object:

DimensionField (Data Type)Definition
Min/Max Durationminduration, maxduration (integer)Minimum/maximum video ad duration in seconds.
Video Bid Response Protocolsprotocols (array of integers)Array of supported bid response protocols. Use List 5.8 of the OpenRTB API Specification for values.
Video Start Delaysstartdelay (integer)Indicates the start delay in seconds for pre-roll, mid-roll, or post-roll ad placements. Refer to List 5.12 in the OpenRTB API Specification for values.
Playback Methodsplaybackmethod (array of integers)Allowed playback methods. If none specified, assume all are allowed. Refer to List 5.10 in the OpenRTB API Specification for values.
Content Delivery Methodsdelivery (array of integers)Supported delivery methods (e.g., streaming, progressive). If none specified, assume all are supported. Refer to List 5.15 in the OpenRTB API Specification.

The following dimensions are encapsulated by the audio object:

DimensionField (Data Type)Definition
Min/Max Durationminduration/maxduration (integer)Minimum/maximum ad duration in seconds.
Audio Bid Response Protocolsprotocols (integer array)Array of supported audio protocols. Refer to List 5.8 in Open RTB Specification for values.
Audio Start Delaysstartdelay (integer)Start delay in seconds for pre-roll, mid-roll, or post-roll ad replacements. Refer to List 5.12 in Open RTB Specification for values.
Audio Content Delivery Methoddelivery (integer array)Supported delivery methods. If none listed, assume all are allowed. Refer to List 5.14 in Open RTB Specification for values.

App and Site Objects

An auction summary record contains either an app or a site object. These two objects share fields, except as indicated in the Dimension column.

DimensionField (Data Type)Definition
App Bundlebundle (string)Application bundle, an ID intended to remain unique across multiple exchanges.
App Store URLstoreurl (string)The app-store URL for the installed application, to comply with IAB Quality Assurance Guidelines.
App Versionversion (string)Application version.
App/Site Domaindomain (string)Either the application domain (for example, "unicornssayapp.com") for in-app ads, or the site domain (for example, "unicornssay.com") for page ads.
Application/Site Categorycat (array of strings)The IAB content category, such as "IAB1". Supply category codes only, not descriptions. For example: IAB1 = "Arts & Entertainment" (IAB1 is the code). Use List 5.1 of the OpenRTB API Specification for values.
Application/Site IDid (string)The ID of the adspace/app or site, which can be used to look up the app or site name.
Application/Site Namename (string)The name of the adspace/app or site, which can be also mapped from a lookup table.
Publisher IDid (string)The publisher's ID on the exchange, which can be used in a lookup table to map the publisher's name.
Publisher Namename (string)The publisher's name, which can also be mapped from a lookup table.

Device Object

The following dimensions are immediately under the device object:

DimensionField (Data Type)Definition
Connection Typeconnectiontype (integer)The device's connection type. For example, 1 = "Ethernet". Use List 5.22 of the OpenRTB API Specification for values.
Data Provider Namename (string)Audience data provider.
Device Carriercarrier (string)The name of the service carrier, which can be also mapped from a lookup table.
Device Makemake (string)The name of the device manufacturer. For example: "Apple".
Device Modelmodel (string)The device model. For example: "iPhone 5s".
Device Typetype (integer)Device type. For example: 1 = "Mobile/Tablet". Use List 5.21 of the OpenRTB API Specification for values.
Do Not Trackdnt (boolean)Do Not Track flag setting. 1 = "true" (flag is on), 0 = "false" (flag is off).
OSos (string)Device operating system. For example: "iOS".
User ID Typemacmd5, dpidmd5, dpidsha1, didmd5, didsha1, macmd5, macsha1, ifa (string)The types of user IDs supplied. One or more of these fields can be supplied, and aggregated sum of the occurrence of each type is surfaced in the Metamarkets dashboard. Their values, which are not surfaced, are the actual IDs.

The following dimensions are encapsulated by the geo object:

DimensionField (Data Type)Definition
Countrycountry (string)Use ISO 3166 country codes.
Location Typetype (integer)Specifies the origin of geographic info. For example: 1 = "GPS/Location Services". Use List 5.20 of the OpenRTB API Specification for values.
DMAdma (string)Device Nielson Designated Market Areas code.
Metrometro (string)Metropolitan code.
Regionregion (string)Region of visitor. Use ISO 3166-2.

The following dimension is encapsulated by an ext object:

DimensionField (Data Type)Definition
Unique IDuid (string)Unique ID is used for calculating total unique users. See this section for details on providing a user identifier for use in calculating the unique metric.

User Object

The following dimensions are encapsulated by the user object:

DimensionField (Data Type)Definition
Gendergender (string)Values can be M, F, O (other), or NULL (unknown).
User IDid (string)Unique consumer ID of this user on the exchange.
Year of Birthyob (integer)Year of birth as a four-digit integer.

The following dimensions are encapsulated by the data object:

DimensionField (Data Type)Definition
Data Provider IDid (string)Unique exchange-specific identifier for the data provider.
Data Provider Namename (string)Audience data provider.

The following dimensions are encapsulated by the segment object:

DimensionField (Data Type)Definition
Segment IDid (string)Unique identifier for a data provider's segment applicable to the user.
Segment Namename (string)Audience segment name.
Segment Valuevalue (string)The value for the corresponding audience segment name.

Bid Response Object

The tables below define the dimensions and objects immediately under the bid response object. For elaboration on any field see Open RTB 2.5 Section 4.

Dimension Immediately Under the Bid Response Object

DimensionField (Data Type)Definition
Bid Request IDid (string)Bid request ID to which the Bid Response corresponds
Bidder IDbidder_id (string)Individual bidder's ID.
Currencycur (string)Bid currency. Default is USD.
No Bid Reasonnbr (integer)Reason for not bidding. Refer to [OpenRTB List 5.24
Response Statusstatus (integer)Indicates validity of responses. See below section for response status codes. If the response status is not "1" (valid response), do not include the seatbid array for that response.
Seat Bidseatbid (object array)Array containing seatbid fields.

The following dimensions are encapsulated by the seatbid object:

DimensionField (Data Type)Definition
Bidbid (object array)Array containing bid fields.
Seatseat (string)The ID of the entity using the bidders.

The following dimensions are encapsulated by the bid object:

DimensionField (Data Type)Definition
Advertiseradomain (array of strings)The advertiser's top-level domain or domains.
Bid IDid (string)ID of the individual bid (at bid level).
Bid Priceprice (float)Bid prices in CPM.
Bid Statusstatus (integer)Indicates winning and failed bids. See below section for bid status codes.
Campaign IDcid (string)The campaign ID, which can be used in a lookup table to map the campaign name.
Clear Priceclear_price (double)The auction clear price (second-highest price for second-price auctions). Omit in cases of non-winning bids.
Creative Attributesattr (array of integers)Attributes describing an ad in terms of capabilities and restrictions. Use Table 5.3 of the OpenRTB API Specification for values.
Creative IDcrid (string)Creative ID if an ad is specified.

Metrics in the Auction Summary Record

Metrics are fields with numeric values that can be aggregated or used in computations.

Raw Metrics

Raw metrics are those directly included (or implied) in the raw data delivered to RDI.

MetricField (Data Type)Definition
Bid Requestsrequest_id (string)A count of request IDs.
Bid Responsesbid_responses (object array)A count of the bid responses.
ClicksClick RecordA count of click events based on the presence of a click record.
ConversionsConversion RecordIncluding this object, which contains timestamp and ID fields, indicates a conversion has occurred.
Video Durationvideo objectDuration of video ad in seconds.
Video Completion Ratesubsequent event record (e.g. Click Record)Percent completion rate.
Floor Pricebidfloor (double)The auction floor price, present in the auction summary record.
ImpressionsImpression RecordA count of impressions. Only the first impression in the imp object is counted.
Revenueadvertiser_charge_price (double)Amount charged to the DSP customers. This field may be passed in the impression, click or conversion record as shown below.
Spendcharge_price (double)Amount charged to the buyer/DSP.
Winning BidsWin Notice RecordA count of winning bids.

Derived and Computed Metrics

Certain metrics can be derived or computed by Metamarkets, either at processing or query time.

MetricFormulaDefinition
Avg. Bid Floorsum(bid_floor) / requestsThe average will be calculated for requests where a bid floor is present.
Avg. Winning Bid Pricesum(winning_bid_price) / winning bidsWill be determined using the bid price from the auction summary records when a win notification record is also received.
CTR(Clicks / Impressions) * 100The click-through rate.
eCPA(spend / conversions) * 1000A count of request IDs.
eCPC(Revenue / Clicks) * 1000The price per thousand clicks.
eCPM(spend / impressions) * 1000The price per thousand impressions.
Fill Rate(Impressions / Winning Bids) * 100The percentage of won requests for which an impression is served/delivered.
ProfitAdvertiser Charge Price - SpendThe amount charged to the DSP's customer less the amount spent on the impression.
Profit Margin(Profit / Revenue) * 100
Win Rate(Winning Bids / Bids) * 100The win rate for the bidder.

Sample Auction Summary Record (with Nested Bids)

The example below shows how an auction summary event (a single record) should be structured. If the auction is successful, the identity of the winning bidder should be denoted with an "auction_winner" flag in the bid response object. This structure also allows for communicating additional data about bids, such as duration of the response and information about bid errors (including timeouts, bad JSON, blocked creatives, etc.). While the example below represents a simple auction summary record for mobile, video, and private marketplace, data should be added to the request in accordance with the OpenRTB API Specification (v2.5).

Note: Example JSON records are "pretty printed" for readability. However, records must be delivered to RDI in a flat format, one record per line, each record separated from the next by a newline character.

    {
      "id": "AFEWSEBD5EB5FI32DASFCD452BB78DVE",
      "timestamp": "2014-03-05T04:58:23.000Z",
      "at": 2,
      "bcat": ["IAB26","IAB25"],
      "imp": [
        {
          "tagid": "231",
          "displaymanager": "MyRenderer",
          "displaymanagerver": "v2",
          "secure": 0,
          "bidfloor": 0.1,
          "banner": {
            "h": 320,
            "w": 50,
            "pos": 3,
            "expandable": [2,3],
            "api": [3,4]
          }
        }
      ],
      "app": {
        "id": "12312312",
        "name": "Unicornssay",
        "domain": "unicornssay.com",
        "bundle": "bundlename",
        "cat": [
          "IAB1"
        ],
        "publisher": {
          "id": "DSA1394D42D3",
          "name": "Unicornssay"
        }
      },
      "site": {
        "id": "1345135123",
        "name": "Unicornssay",
        "domain": "unicornssay.com",
        "cat": [
          "IAB1",
          "IAB2"
        ],
        "publisher": {
          "name": "Publisher A",
          "id": "pub12345"
        }
      },
      "device": {
        "geo": {
          "city": "US  SFO",
          "country": "USA",
          "region": "CA",
          "zip": "94107",
          "type": 1,
          "metro": "807"
        },
        "connectiontype": 2,
        "devicetype": 1,
        "osv": "4.2.1",
        "os": "iOS",
        "model": "iPhone 3GS",
        "make": "Apple",
        "carrier": "Verizon",
        "ip": "192.168.1.8",
        "didsha1": "AA003EABFB29E6F759F3BDAB34E50BB11",
        "dpidmd5": "A4003FABFB52E6F8A4003FABFB52E6F81",
        "ext": {
          "uid": "AA000DFE74168477C70D291f57"
        }
      },
      "user": {
        "id": "456789876567897654678987656789",
        "yob": 1987,
        "gender": "M",
        "data": [
          {
            "id": "123",
            "name": "bluesky",
            "segment": [
              {
                "id": "abc1",
                "name": "gender",
                "value": "male"
              }
            ]
          }
        ]
      },
      "bid_responses": [
        {
          "timestamp": "2014-03-05T04:58:23.200Z",
          "status": 1,
          "total_duration": 43,
          "bidder_id": "1921",
          "bidder_name": "Real Ads",
          "cur": "USD",
          "seatbid": [
            {
              "seat": "512",
              "bid": [
                {
                  "status": 1,
                  "clear_price": 1.1,
                  "attr": [
                    1,
                    2,
                    3,
                    4,
                    5,
                    6,
                    7,
                    12
                  ],
                  "crid": "1234",
                  "cid": "229",
                  "id": "1",
                  "impid": "102",
                  "price": 5.43,
                  "adid": "314",
                  "adomain": [
                    "realtime4real.mmx.org"
                  ],
                  "iurl": "http://adserver.com/pathtosampleimage"
                }
              ]
            }
          ]
        },
        {
          "timestamp": "2014-03-05T04:58:23.200Z",
          "status": 1,
          "total_duration": 99,
          "bidder_id": "1325",
          "bidder_name": "JumpUp",
          "cur": "USD",
          "seatbid": [
            {
              "seat": "12",
              "bid": [
                {
                  "status": 3,
                  "attr": [
                    1,
                    12
                  ],
                  "crid": "12333",
                  "cid": "2232",
                  "id": "1",
                  "impid": "102",
                  "price": 1.1,
                  "adid": "33",
                  "adomain": [
                    "carscarscars.com"
                  ],
                  "iurl": "http://adserver.com/pathtosampleimage"
                }
              ]
            }
          ]
        },
        {
          "timestamp": "2014-03-05T04:58:23.800Z",
          "status": 3,
          "total_duration": 303,
          "bidder_id": "2144",
          "bidder_name": "Turning.com",
          "cur": "USD"
        }
      ]
    }

As shown in the example above, the auction summary record provides a way to denote the statuses/outcomes of bid activity, which is then used to provide rich analytics on bidding health in your Metamarkets dashboard. At the response level, these codes help identify issues that prevent bids from being included in the auction (timeouts, HTTP errors, bad JSON, etc.). At the bid level, the codes enable the tracking of granular bid statuses/outcomes such as types of blocks, above/below floor, etc.

BidResponse-level Status ("bid_responses[].status")

1 = Valid
2 = Timeout
3 = Invalid JSON
4 = HTTP Error

Note: In the event of a timeout, it may also be useful to analyze the response durations. You should populate the "total_duration" field in the response record to add this visibility. For responses with "status" other than "Valid", the seatBid object may be omitted.

Bid-level Status ("seatbid[].bid[].status")

1 = Won
2 = Lost on Price (i.e., a normal loss)
3 = Below Floor
4 = Markup Delivery Failure (i.e., forfeited the win by not returning markup in the bid or the win notice)
5 = Unscreenable (i.e., did not send the minimal attributes to make it screenable by sellers, such as "iurl", "adomain", etc.)
6 = Blocked by Publisher
7 = Unverified Creative (i.e., publisher requires human review of creatives and this creative has not yet been reviewed)
8 = Blocked Advertiser (i.e., violated the block list of advertiser domains)
9 = Blocked Content Category (i.e., violated the block list of content categories)
10 = Blocked Creative Attribute (i.e., violated the block list of creative attributes)

Mapping Explicit Values to IDs

While categorical dimension fields (e.g., publisher or campaign name) can be provided directly in the raw data, you could instead choose to pass an ID value, which RDI can then map to the appropriate string value using Handling Lookup Tables you provide. Because the OpenRTB API Spec v2.2 is supported, OpenRTB-compliant values may also be provided (e.g., integer values "1" through "7" for Ad Position).

Sample Impression Records

The example below shows the format for an impression record. While exchanges often need to deliver impression and revenue information separately from the associated auction and bid data, you can include this information in the auction summary record if preferred. Provided that the Charge Price and Publisher Revenue are included, RDI can compute the exchange/SSP revenue amount. If impressions are sent separately from the auction summary record, an additional RDI endpoint will be provided.

Note The value of request_id is the auction ID. The timestamp applies to the impression.

    {
        "timestamp": "2014-03-05T04:58:23.200Z",
        "request_id": "AFEWSEBD5EB5FI32DASFCD452BB78DVE",
        "charge_price": 1.15,
        "publisher_revenue": 1.05
    }

Sample Click Records

Clicks can also be delivered in a subsequent record. This record consists of a timestamp and the ID of the original request to which the click pertains. In the event that clicks can be delivered, an additional RDI endpoint will be provided.

Note The value of request_id is the auction ID. The timestamp applies to the time of the click.

    {
        "timestamp": "2014-03-05T04:58:23.200Z",
        "request_id": "AFEWSEBD5EB5FI32DASFCD452BB78DVE"
    }

Sample Conversion Records

Depending on your business, you may track different events after the auction such as app downloads or purchases. These conversions may happen hours (or even days) after the auction or impression, precluding a join with the auction record because it missed the real-time window period: 20 minutes, and even the batch fix-up window period: 4 hours. To best manage this, we recommend appending all the dimensions & values from the original auction when posting conversion events. These conversion events should be posted to a separate endpoint provided by Metamarkets and can include additional dimensions/metrics within the conversions object. At minimum, the timestamp field should reflect when the conversion occurred and must be included in the top-level timestamp. Below is an example of an abbreviated conversion JSON:

    {
      "id": "AFEWSEBD5EB5FI32DASFCD452BB78DVE",
      "timestamp": "2014-03-07T03:15:00.000Z",
      "auction_timestamp": "2014-03-07T03:15:00.000Z",
      "at": 2,
      "bcat": ["IAB26","IAB25"],
      "app": { ... },
      "imp": [ ... ],
      "device": { ... },
      "user": { ... },
      "bid_responses": [ ... ],
      "conversions":  {
            "conversion_charge": 10,
            "conversion_type": "Subscription"
              }
    }

With identical dimensions, the original Auction Record & the Conversion Record are unioned together and surfaced immediately on the dashboard.

Providing Data for Uniques Calculations

Metamarkets uses the hyperloglog algorithm to approximate the number of unique users based off of unique IDs provided in the auction summary record. While device or user IDs may be present in the request, you may need to vary which unique identifier should be used for a particular auction. As such, you may also provide an additional dimension for use in the uniques calculations. Common IDs used include:

  • An internal exchange ID
  • A device ID available in the request
  • A third-party ID such as from a DMP
  • Any other unique ID designated by you for uniques calculations

Data Delivery Requirements and Processing Summary

Upon receiving the auction summary, impression and click records, Metamarkets RDI performs a join, ultimately storing all data in denormalized form.

In order to perform the join with real-time processing, a collection window of up to 20 minutes is used. Data delivered in time for real-time processing is surfaced on the Metamarkets dashboard immediately at the end of the 20-minute collection window.

Impressions and clicks that come in more than 20 minutes after the request are still captured, but are not surfaced on the dashboard until a batch fix-up job runs. Batch fix-up jobs run approximately every 4 hours, allowing for impressions, clicks or late data to be received and surfaced on the dashboard.

When the batch fix-up job runs for a particular hour, all data received for that hour and the previous hour is also reprocessed, creating a join window of up to 2 hours to include clicks and impressions that occur near the time of the original request.