Connector Catalog

Overview

Within Beckon, there are several pre-defined connectors users can implement in their deployments. This article provides relevant information that relates to those connectors. 

Youtube - Targeted Queries API

YouTube - Bulk Reports API

Facebook

Facebook-Ads 

Twitter

Twitter-Ads

Google

Doubleclick

Google Adwords

Google Webmaster

Sprinklr

Saleforce

Maestro

Apigee

LinkedIn

BigQuery

Pardot

Email Connector

SFTP Connector

Generic Connector

Information

YouTube - Targeted Queries Application Programming Interface (API)

Choose a channel that is accessible for the oauth-ed user as a profile. The connector needs access to the YouTube account in read-only mode.

The list of available metrics and dimensions is specified in YouTube Analytics and Reporting APIs. For time-series data, you can only choose metrics as defined by YouTube. If you need to get other supported dimensions, then ‘day’ may not apply, e.g., for the ‘Top Videos’ report, we can specify video as a dimension, but in that case, you need to go to properties of the connector and replace ‘day’ with ‘video’ for date-dimension.

The following properties can be specified as applicable:

Property

Example

Comment

channel_id

UCVcoho9uaKUkDOYQxH5XNGA

This is used to override the channel determined by the profile selected. The override feature is mainly for backward compatibility and not recommended.

sort

-views

Some combinations of metric/dimensions require sorting, e.g., ‘Playback location detail’ requires you to use sorting with -views or -estimatedMinutesWatched.

max_results

200

Some reports require specification of this property, e.g., ‘Top Videos’ can have a maximum of 200 or ‘Playback location detail’ can have a max of 25 results. The API call will fail if this property is not specified in those cases.

filters

video==dMH0bHeiRNg;country==IT

It is possible to add filtering optionally, or when the report that you are using requires to determine the filter.

For more information, please refer to YouTube Analytics and Reporting APIs.

YouTube - Bulk Reports API

Choose the type of report when you create the connector as a profile.

YouTube Bulk Reports connector is slightly different from other connectors. This connector sets up a job on YouTube's servers when the connector is initially created using QuickStart Wizard. The reports are generated on YouTube's servers after approximately 48 hours. The initial reports will have historical data, and then, the data will start coming for each day. The trick is to begin the cron-schedule for the connector after two days (since nothing is available till then) or allow the connector to fetch-fail for first two days.

The report creation starts only after the job is created for the first time (or if the client has used the API before).

The connector always pulls the latest of the reports that YouTube creates. So, creating a cron-schedule for this connector is essential to avoid missing any data. The cron-timeframe should typically be set to ‘LATEST” which leads to getting the latest created report in each cron job run on that connector.

If you use the interactive ‘fetch-now’ button, the connector will retrieve all the reports that have any data in that range. This is done based on how YouTube's servers created the reports, especially the historical ones, which are created when you set up the connector job for the first time. The process is more consistent and starts to generate one daily report eventually. Historical data for up to 180 days before creating the job is available. That historical data is made available in a month after the job is created.

For more information, please refer to YouTube Reporting API - Data Model.

Facebook

The connector needs to have an account which has permissions for reading insights and managing pages. QuickStart asks the user to choose the page as a profile for the connector.

Facebook's connector is typically used for page level insights. It can also be used for post-level insights if we know the post-id or we are getting all posts in a given page_id. The profile selection shows all available pages for the oauth-ed user.

The metric names indicate the dimensions that they are compatible with. So, we should add exactly those dimensions with that metric in the right order, otherwise, they will not be associated with the correct mapping (e.g., page_impressions_by_age_gender_unique must have age and gender added as dimensions (in that order) with that metric.

We can handle compatibility rules in reference data. So, make sure to add these compatibility rules when you find a proper effective use case that is not automatically selected in Quickstart. Additionally, if you choose more than one metrics in Quickstart Wizard, make sure they support the same dimensions in the same order.

It is essential to check if the metrics are specified as ‘lifetime’ or ‘day’ in Facebook's API reference. If ‘lifetime’ is supported, then date-dimension should be changed to ‘lifetime’ from ‘day’ in the properties tab. Also, if you choose multiple metrics in Quickstart Wizard, they all need to support either ‘lifetime’ or ‘day’ as date-dimension.

The following properties can be specified as applicable:

Property

Example

Comment

post_id

991280604246393

This is used to specify post_id for post-level insights

page_id

 

Optionally, you can override page_id as a property instead of using it from the selected profile. This property can also have a comma-separated list of multiple pages (that are accessible by the same oauth).

page_name

 

This is typically used when page_id property above is specified. If page_id is a comma-separated list, then page_name is also a comma-separated list in the same order.

Date dimension

lifetime or day

Based on what the metric supports in Facebook's API.

facebook_page_posts

true

This is required to change the processor of the connector to PagePosts to get analytics for posts in the given page. The default is false.

facebook_post_type

video

When facebook_page_posts is true, facebook_post_type filters posts based on the specified type.

facebook_excluded_post_ids

991280604246393,991280604246394

Comma-separated list of post ids that are included. This works with the facebook_post_type property.

facebook_included_post_ids

991280604246393,991280604246394

It's a comma-separated list of excluded post_ids. All other posts on the page are included for an analytics query. This works with the facebook_post_type property.

facebook_max_post_count

20

When facebook_page_posts is true, facebook_max_post_count limits posts based on the specified number.

throttleTime

5000

This works with facebook_post_type property. The throttleTime is used to sleep between the query for each post. The time is specified in milliseconds.

facebook.use_usertoken

true

The default is false so that pageToken is used instead of userToken.

For more information, refer to this article

Facebook-Ads

The connector uses the ad-account that is selected as part of a profile.

Level

The connector chooses a level based on selected dimensions as follows:

Priority

Dimension includes

Level

1

ad_id or ad_name

ad

2

adset_id or adset_name

adset

3

campaign_id or campaing_name

campaign

The level will be selected based on priority if multiple dimensions are included from the above list. The insights will be aggregated based on the level. By default, we get insights for an ad-account (unless entity_id is specified) at the given level.

Breakdowns

Dimensions that are used as ‘breakdowns’ can be specified with a comma-separated property. Add that as a dimension and specify that in the property. Only certain combinations of breakdowns are allowed as described in the Facebook marketing API documentation.

Following properties can be specified as applicable:

Property

Example

Comment

entity_id

Ad_account_id or campaign_id or adset_id or ad_id

This is used to specify the object for which insights are fetched.
The default is to use account_id from the profile selection.

breakdowns

age,gender

  • age
  • country
  • gender
  • frequency_value
  • hourly_stats_aggregated_by_advertiser_time_zone
  • hourly_stats_aggregated_by_audience_time_zone
  • impression_device
  • place_page_id
  • placement
  • device_platform
  • product_id
  • region

action_breakdowns

 

  • action_canvas_component_name
  • action_carousel_card_id
  • action_carousel_card_name
  • action_destination
  • action_device
  • action_link_click_destination
  • action_reaction
  • action_target_id
  • action_type
  • action_video_sound
  • action_video_type

max_rows_per_call

 

The default is 100, but it can be changed to a smaller or larger values
to control number of calls. Facebook allows a max of 500 (at this time)
for this parameter.

filter

[{'field':'ad.delivery_info','operator':'IN',
'value':['active','archived','completed','inactive',
'limited','not_delivering','not_published',
'pending_review','permanently_deleted',
'recently_completed','recently_rejected','rejected', 
'scheduled']}]

Filtering string as documented in Facebook Ads API.
The example shows a typical filter required for getting
historical data since API will not return data for all
delivery types. This was a suggestion from Facebook's team.

For more information, please refer to Facebook's API Statistics.  

Note: we do not support action_breakdowns as of now.

Twitter

Twitter's connector supports getting data associated with tweets or users, which are selected as a profile.

  • The user data includes : "id", "screen_name", "created_date", "date", "favorites_count", "followers_count", "friends_count", "statuses_count", "lists_count"
  • The tweets data includes: "id", "created_date", "date", "url", "favorites_count", "country", "retweets_count", "media_url", "text"

The QuickStart Wizard allows you to map all or a subset of these metrics and dimensions. It pulls all of them regardless. Also, note that startDate and endDate have no effect on this pull.

Twitter-Ads

The profile-selection shows each campaign in an ad-account.

Segmentation type

Segmentation type is derived from the first dimension selected in the connector. The possible values are:

  • PLATFORMS,
  • LOCATIONS,
  • GENDER,
  • INTERESTS,
  • KEYWORDS,
  • CITIES,
  • REGIONS,
  • POSTAL_CODES,
  • DEVICES,
  • PLATFORM_VERSIONS

Ad-level

Ad-level decides the granularity of the data that is pulled by the connector.

  • ACCOUNT: Account level metrics.
  • CAMPAIGN: Campaign metrics.
  • ACCOUNT_WITH_CAMPAIGNS: Campaign data with associated account data.
  • CAMPAIGN_WITH_LINEITEMS: Lineitem data with associated campaign and account data.
  • ACCOUNT_WITH_PROMOTED_TWEETS: Promoted tweets data related to an account.
  • CAMPAIGN_LIST: List all campaigns for the selected account.
  • LINEITEM_LIST: List all lineitems (ad-groups) for a selected account.

The API limits to getting data for a maximum of twenty (20) campaigns or lineitems when used in ACCOUNT_WITH_CAMPAIGN or CAMPAIGN_WITH_LINEITEM level.

The connector retrieves all the available metrics (some of them will have blank values based on selected objective).

Property

Example

Comment

twitterAds.level

ACCOUNT

Ad-level decides the granularity of data download.

twitterAds.ad_objective

TWEET_ENGAGEMENTS

Ad objective is deduced from the selected campaigns from the profile. The lineitems in that campaign is searched for an objective. This property (when specified) overrides the objective. Ad-objective decides the metric-groups that are applicable for data pull.

twitterAds.use_servable_campaigns

true

The default is ‘false’. When set to true, we will use only valid ‘servable’ campaigns for ACCOUNT_WITH_CAMPAIGN level.

twitterAds.max_entity_count

50

The default is the twenty most recent entities

Google Analytics

Google Analytics' connector is supported using QuickStart.

A QuickStart profile selection shows a tree three levels: Account, Web-property, ProfileGoogle Analytics connector gets metadata regarding metrics and dimensions using an API call, and the reference data is merged into that information. A maximum of ten metrics and seven dimensions (one used for the date out of these seven) are supported by this API in one connector request.

Sampling

The connector handles sampling of data using a technique of making multiple calls with smaller duration. The requests are first made with a given start/end date. If it leads to sampling, we reduce the range to monthly, followed by weekly and daily until we get data without sampling. This is done transparently and automatically by the connector for each pull.

Templatized columns

The custom dimensions/metrics and other templatized columns provide metadata information on minimum and maximum index. This index is different (200 max instead of 20 max) when GA-360 is used. The connector looks at that metadata to show available values in quickstart.

Following properties can be set to customize this connector: 

Property

Example

Comment

ga.sort_string

ga:sessions

This is used typically for top-n kind of reports from GA. The sort string decides the ordering based on comma separated list of dimensions or metrics. The default order will be ascending. To specify descending order, use a “-” (minus sign) before the dimenson/metric in the sort_string.

ga:max_rows

10

This is typically used in combination with ga:sort_string to get a top-n report.

 

Doubleclick

Profile selection shows user-profiles with account name, (optional) sub-account name and user-name. The connector creates a report, runs it asynchronously and then downloads it when it is ready. If you need to get data for metrics/dimensions that are supported using a different report-type, it can be specified with the following property. 

Property

Example

Comment

doubleclick.report_type

REACH

The default report type is STANDARD. But one can specify FLOOFLIGHT or REACH optionally in this property.

doubleclick.enable_all_dimension_combinations

true

The default is false. This property is applicable to only REACH report_type. If set to true, we allow for a report with all dimension combinations as supported by doubleclick API. The report will take a much longer time to create and download.

 

Google Adwords

Note: Disable all ad blocker apps/extensions in order to see the connector in the QuickStat in Beckon.

Google Adwords supports a large number of reports that are shown for selection as a profile in QuickStatThe connector also gets metadata using an API call based on the report-type selected in the profile.

Google Adwords supports a hierarchy of accounts starting with the root account, which can have multiple manager-accounts or advertiser accounts. Each manager account can have many advertiser accounts accessible to it. The API calls are supported for advertiser accounts. They will fail for manager account.

Property

Example

Comment

adwords.customer_id

 

When the oauth-ed account has multiple advertiser accounts accessible, we can specify which customer_id to use to choose an account.

adwords.include_zero_impressions

true

The default is false. If it is set to true, we will also retrieve rows with 0 impression values for all dimension combinations specified in the request, That is typically used for getting the structure or all values of a dimension, but hardly useful in Beckon.

 For more information, please refer to Adwords API.

Google Webmaster (Google Search Console)

The profile selection shows all verified sites for your account. The connector retrieves clicks, impressions, position as metrics (even if you un-select them), amongst others. If you select multiple dimensions for segmentation, the numbers may not match with a different dimension combination. The Webmaster Analytics User Interface (UI) allows users to choose only one dimension. 

The following dimensions are suppprted: queries, pages, countries, devices and dates. Search-type is always set to ‘web’ while making the analytics query.

Property

Example

Comment

webmasterSearchConsole.searchType

image or video or web

The default is ‘web’ which should work for most cases

webmasterSearchConsole.aggregationType

auto or byPage or byProperty

The default is ‘auto’.

Sprinklr

Beckon needs to create a separate ‘app’ and register with Sprinklr. This typically gets approved in a couple of days after the client shows proof of purchase for a given API type. Once we get it approved, we need to update the account_config table with client-id and client-secret for that account.

QuickStart supports sprinkler connectors for Reports-api, Listening-api and Paid-ads-api.

Profile selection shows reposts supported by all three types of APIs. The authenticated user (and their account) should have access to the chosen API from Sprinklr.

Custom Dimensions are supported by the connector. It is possible to get a list of all custom dimensions by using the bootstrap API for OUTBOUND_CUSTOM_FIELDS using a generic connector and then get the label and fieldname from the response. We then need to add this to reference data by postfixing account-name with a delimiter of “::” between them. The custom dimensions are shown to all clients and hence the last part of that is used to specify the client name that has defined it in sprinklr. We need to also specify ‘ACCOUNT_CUSTOM_PROPETY’ in the reference data if the custom property is for ‘Account’ asset-type (in Sprinklr). The format of custom dimensions in reference data is as follows:

Brand::56b4dbd0db24fc4e08000000::customProperties::coke 

or

KO Response Type::56fe7d57a6fd015d96000006::ACCOUNT_CUSTOM_PROPETY::coke

Property

Example

Comment

filter_dimension

Brand::56b4dbd0db24fc4e08000000::customProperties::coke

Name of dimension to filter on. This name should match reference data for that dimension

filter_op

IN

Filter operator

filter_value

abc,def

A single value or a comma-separated list of values that should be used in a filter

 

Salesforce

Salesforce's connector is based on specifying Salesforce Object Query Language (SOQL) to retrieve data from Salesforce. The data can then be sent to the transformer. SOQL support includes relationship queries for child-to-parent relationships are used.

We support custom objects, and custom attributes through SOQL.

Property

Example

Comment

bigquery.query

SELECT count(1) c, state FROM [bigquery-public-data:usa_names.usa_1910_current] group by state order by c desc limit 10

SQL query that needs to be executed on BigQuery

salesforce.query_script

 

A script that returns a string that provides a SOQL query to be run. The script gets access to the connectorConfig object as well as startDate and endDate to make query string.

 

Maestro

Please refer to Maestro API for detail on this.

Loopback documentation for request/response formats can be found in Querying Data

The metrics and dimensions should be compatible with the selected endpoint.

Property

Example

Comment

maestro_endpoint

 

This property decides which API endpoint gets called. Please refer to the reference doc for various available endpoints.

profile_id

123444443,1111123

Comma separated list of all profiles

 

Apigee

The Apigee related connectors (apigee-GA, apigee-facebook, apigee-twitter, and apigee-youtube) were used by the brand Coke. They are moving away from that to Maestro. Some of those connectors are still being used in production though. 

Note: credentials may be required to access the below link. 

Property

Example

Comment

X-KO-ACCOUNT

CL3

Account as set up by coke in apigee

apigee_uri

https://api.coke.com/googleAnalytics/v1/api

Possible values: https://api.coke.com/googleAnalytics/v1/api

https://api.coke.com/facebook/v1/api/v2.2

https://api.coke.com/youtubeAnalytics/v1/api

profile_id

11113344

Profile_id for Google Analytics.

apigee_twitter_path

 

Used to append apigee_uri in case of apigee-twitter.

period

day

Used by apigee-facebook to decide a day or lifetime period.

 

LinkedIn

LinkedIn is supported in the form of an oauth adapter. The user authenticating for this must be an admin for the API to work. LinkedIn has no way to refresh a token without a user session. The token will expire after sixty days, and the user will need to reauthenticate after that. Also, the last token remains valid until another token is retrieved even if it is done before the expiry time, which is why it is safer to restrict having one adapter per account per oauth-user.

A generic connector needs to be built for making API calls to get data from LinkedIn. The generic connector should be using api.auth_type=oauth and then be associated with a LinkedIn adapter.

BigQuery

BigQuery's connector supports running a query on a given account (that is oauth-ed) for a given project. 

Property

Example

Comment

bigquery.query

SELECT count(1) c, state FROM [bigquery-public-data:usa_names.usa_1910_current] group by state order by c desc limit 10

SQL query that needs to be executed on BigQuery

bigquery.query_script

 

A script that returns a string that provides a query to be run. The script gets access to the connectorConfig object as well as startDate and endDate to make query string.

bigquery.use_legacy_sql

true

This defaults to false. But, based on whether legacy SQL is specified for a query, this flag should be turned true/false.

 

Pardot

Not yet officially supported by Beckon (backend exists for a future connector).

Email Connector

All attachments that are sent by a valid beckon user (registered in Beckon's app) are checked to see if any of email-connectors match their criteria. The email endpoint is automatically generated for each account and can be viewed in the ‘Company Settings’ page for that account. The criteria is specified in email connectors using regular expressions as defined below:

Property

Example

Comment

Email Subject Regex

mysubect.*

It can be blank if you do not want to use that in selecting the connector

Email From Regex

.*joe@client.com.*

It can be blank if you do not want to use that in selecting the connector

Email To Regex

.*a@b.com.*

It can be blank if you do not want to use that in selecting the connector

Email CC Regex

.*a@b.com.*

It can be blank if you do not want to use that in selecting the connector

Email Filename Regex

file.*csv

It can be blank if you do not want to use that in selecting the connector

email.fetchBody

true

The default is false. When set to true, the email body will be used as a document (instead of attachments) to be sent to the transformer and imported into Beckon.

 

SSH File Transfer Protocol (SFTP) Connector

This connector is used to access external (non-Beckon) SFTP servers to pull data. We support accessing external SFTP servers by using public/private key-pair based authentication. The external SFTP should be configured by sending them our public key. The private key is used in creating our connector along with server name, port and user-name information.

 

Property

Example

Comment

SFTP Directory Name

read_write

Name of the directory on the third-party SFTP server from where files are pulled.

SFTP Directory Script

 

Javascript expression that will result in a directory name on the third-party SFTP server.

SFTP Filename Regex

(d|s).*

Regex to choose the file(s) to be pulled.

SFTP Filename Script

 

Javascript expression that will result in a file name regex on third-party SFTP server.

 

Generic Connector

A generic connector provides a way to access third party data using REST based apis. It supports basic authentication and oauth as well as custom login mechanisms that can be scripted.

These connectors can be scheduling on a cron like other beckon connectors, and the user has control over upload-action.

There are two main phases for downloading data from a third-party API:

  1. Login-phase: The login phase establishes authentication with the API and retrieves an access-token to be used during API calls.
  2. Request-phase: The request phase makes the actual API request to get data.

The generic connector provides extension points in both phases. The extensibility is supported by allowing a user-provided script to be executed before or after the phase.

The script that executes before the login phase can create a request that will be sent to the third-party as an auth request. The script that executes after the login phase can use the data retrieved in login-request and parse out access-token or any other information.

The user can also provide a script that creates the api-request (GET or POST) with all of the query, header, post parameters. The script that runs after the api phase uses the data and formats/extracts response into a CVS format. The response is stored into a document by the connector and then sent to an appropriate transformer.

Pagination in API requests is supported in response-handler by returning a map with a nextRequest key to specify the next request and throttleTime key to specify sleep time between multiple requests. The response-handler can return a string with data on the last request when there are no more pages available.

The scripts get called with following variables in the engine bindings:

Binding variable

Comment

logger

Can be used to log messages at various levels

connectorConfig

The whole connector object (with all nested objects). One can also access

workarea

This Map can be used to store state between calls to the scrip

startDate

The start date of the current download request

endDate

The end date of the current download request

Configuration parameters

Property

Example

Comment

api.method

GET

GET/POST

api.auth_type

basic

basic/oauth/custom/none

api.url

http://abc.com/api

REST api endpoint

api.login.url

http://abc.com/api

REST api login endpoint. Not used commonly.

api.header.<header_name>

api.header.Content-Type

Specified the value for any HTTP header sent with the request

api.postparam.<header_name>

api.postparam.account

Specified the value for HTTP post parameters  sent with the request

api.login.header.<header_name>

api.login.header.Content-Type

Specified the value for any HTTP header sent with the login request. Not commonly used.

api.login.postparam.<header_name>

api.login.postparam.account

Specified the value for HTTP post parameters  sent with the login request. Not commonly used.

api.script.language

javascript

javascript/ruby/groovy

api.login.script.request

 

Login request

api.login.script.response

 

Login response handler

api.script.request

 

API request

api.script.response

 

API response handler

api.rate_limit_error_code

403

HTTP error status code that represents api rate limit error

api.max_number_of_pages

50

The default when this property is not specified is 1000. If the value is negative, then there is no limit on the number of pages retrieved.

use_binary_response

true

When the script expects the content to be byte array instead of a String. The default is ‘false’.

file_extension

zip

Decides extension for the downloaded file. The default is ‘csv’.

file_name

myDocument

Name of document that will be used instead of the generated name



Comments

0 comments

Please sign in to leave a comment.