Introduction to COUNTER SUSHI for vendors, librarians, and consortia

“REST” basically means a URL can be created that includes all of the information needed to make a successful report request.

JSON is a format for encoding text-based (including Unicode) structured data, more similar to XML than to spreadsheet formats like csv. There are some examples in the “Content of Reports” section below.

• Request the list of all supported reports:
  https://sushi.vendorname.com/reports?customer_id=12345&requestor_id=678989&api_key=7f89dhj489f0dajk348
• Request a specific standard view (in this case, the TR_J1):
  https://sushi.vendorname.com/reports/tr_j1?customer_id=12345&requestor_id=678989&api_key=7f89dhj489f0dajk348&begin_date=2020-01&end_date=2020-07
• Request a specific Master Report with optional attributes and metrics (in this case, the TR with all possible attributes shown and limiting the stats to data type: Book):
  https://sushi.vendorname.com/reports/tr?customer_id=12345&requestor_id=678989&api_key=7f89dhj489f0dajk348&begin_date=2020-01&end_date=2020-07&data_type=Book&attributes_to_show=Data_Type|Access_Method|YOP|Access_Type|Section_Type

Note that including more attributes to show will not just add more “columns” but often results in more “rows” per database/platform/title/item. In particular, the YOP (year of publication) for title-level journal-containing Master Reports can get very large as each title/YOP combination is given its own row.

• customer_id – all vendors must require a customer_id
• requestor_id – optional, vendors can use this for whatever they like
• api_key – optional, vendors can use this for whatever they like

The customer_id, requestor_id, and api_key are embedded in the SUSHI URL itself (“parameters”), so it is recommended (but not required) that vendors assign such codes using characters that do not need to be URI-encoded (e.g., convert space characters into %20).

Other authentication options/restrictions that are not visible in the SUSHI URL itself are permitted but are problematic because they represent vendor-specific layers which are typically hidden from end customers and their software developers. These are:

• IP authentication – the vendor may choose to link a given customer_id or requestor_id to one or a range of IP addresses (e.g. the range of addresses associated with the end customer’s institution) so that even with the URL-embedded credentials, a SUSHI report request will be rejected if it originates from a non-associated IP address. This is problematic both for legitimate institutional employees working from “off campus” and for cloud-based client harvesting services who have to separately register their own servers’ IP addresses with every vendor for each of their end customers. It is not recommended, as the other three authentication controls should provide sufficient security.
• IP blocking – some vendors who do not otherwise associate customer_ids/requestor_ids with specific IP ranges may nevertheless keep their own “blacklists” of IPs from whom they will not accept connections. Typically this occurs when a vendor is maintaining such blacklists for their entire stable of company servers in order to protect their content from being illegally harvested, and their SUSHI server just ends up falling within that network protection.
  However, some vendors have broadened these blacklists to include large ranges of IP addresses that are used by commercial VPN services and that end customers working from home may be using to protect their own home networks. This is permitted but not recommended. These restrictions are generally not documented (specifically because they would not serve their purpose if the “bad guys” knew the list), they vary by vendor, and they can pose significant barriers to SUSHI client developers and their end customers. There is no evidence that SUSHI servers per se have been targeted by malicious parties.
• User agent blocking – similar to IP blocking, some vendors’ networks reject all requests if the “user agent” code (a hidden browser-type code that all web browsers and RESTful clients send automatically as part of the http protocol that is fundamental to how the web works) either is or is not part of a specific whitelist/blacklist of acceptable codes. As with IP blocking, this becomes a vendor-specific barrier to the purpose of the SUSHI API and vendors are reluctant to share their whitelist/blacklist settings.
• Limiting request volume – vendors may impose limits on the number of report requests from a given customer over a given period of time, such as “only one per second” or “only 10 per day”. Again, these limits are technically allowed but highly discouraged as they represent yet another undocumented vendor-specific barrier to SUSHI clients, whose developers have to discover these limitations and code for them specifically per vendor. The normal behavior of end customers will be to request all supported reports for a given account at the same time on a regular basis, e.g. monthly updates requested in a single client run. Since SUSHI URLs are normally being sent at software speeds, not human speeds, such limits artificially slow down the clients. A robust SUSHI server should be able to handle the typical end-customer request behavior without such throttles being necessary for their servers to operate successfully. There is a specific “exception” code that is included in the report header when these limits are exceeded (see code 1020 in Appendix F of the COP5 and further explanation about exceptions below).

The Project COUNTER team are considering including information about the use of these constraints in future versions of the “/status” API response in a future update.

If you are a consortium administrator, to access usage for your members you will usually have a fully-fledged account and thus will have your own SUSHI credentials: customer_id and, optionally, requestor_id and api_key to get the consortial level reports.

• The SUSHI “/members” API request provides consortium administrators with the list of members known to be part of that consortium on the platform along with the requestor_id and customer_id for each. The consortium will use its own api_key in place of each member’s to request the institution-level reports.
• Each member institution will be an account on its own and have its own SUSHI credential that it can use independently of the reporting done to the consortium
• A consortial account (e.g. CRKN in Canada) is a parent account with many “child” accounts representing members
• From a SUSHI/reporting perspective, “consortial” relationships never go beyond parent-child:
  • A given institution can be a “child” of many parent accounts (libraries are often members of multiple consortia)
  • A given institution can also act as “parent” for other institutions (e.g. a multi-campus university system may see cooperative buying with one institution buying materials on behalf of the group while another buys other materials for the group, so the parent-child relationships may be license-specific and not a formal hierarchy)

Example of a report header, for a Database Master Report (DR), in its raw JSON. Aside from the date range, filters and attributes for Master Reports should only be included in the header if explicitly selected by the API call – the defaults should not be listed. For standard views whose filters and attributes are pre-defined, those default values should be listed.


{
"Report_Header": {
"Created": "2020-06-29T14:20:43Z",
"Created_By": "Vendor-Name",
"Customer_ID": "12345",
"Report_ID": "DR",
"Release": "5",
"Report_Name": "Database Master Report",
"Institution_Name": "UniversityX",
"Institution_ID": [
{
"Type": "Proprietary",
"Value": "PlatformX:customer_id"
}
],
"Report_Filters": [
{
"Name": "Begin_Date",
"Value": "2020-01-01"
},
{
"Name": "End_Date",
"Value": "2020-07-31"
}
],
"Report_Attributes": [
{
"Name": "Attributes_To_Show",
"Value": "Data_Type|Access_Method"
}
]
}
}


Institution_ID: the “Type” should be one of: ISNI, ISIL, OCLC, Proprietary. These are known as “namespaces”.

Example of a single report item, for a Database Master Report (DR), in its raw JSON. Note that although the requested date range was January to July 2020, only two months, February and April, had usage for this database on this platform. Also note that the dates are not in order; as previously mentioned, JSON generally and the SUSHI standard in particular do not expect the data to be “ordered” by date, or the items to be alphabetical or in any other particular order.

This is a single entry within the “Report_Items” array of the JSON response.


{
"Platform": "Platformname",
"Item_ID": [
{
"Type": "Proprietary",
"Value": "PlatformX:databaseY"
}
],
"Database": "DatabaseName",
"Publisher": "DatabasePublisher",
"Data_Type": "Database",
"Access_Method": "Regular",
"Performance": [
{
"Period": {
"Begin_Date": "2020-04-01",
"End_Date": "2020-04-30"
},
"Instance": [
{
"Metric_Type": "Searches_Automated",
"Count": 2
}
]
},
{
"Period": {
"Begin_Date": "2020-02-01",
"End_Date": "2020-02-29"
},
"Instance": [
{
"Metric_Type": "Searches_Automated",
"Count": 4
}
]
}
]
}


• No reporting period total column by default, just the individual months requested as separate columns
• If there is no data for a given data point (think individual cell in the tabular spreadsheet), the data must be omitted entirely rather than being provided as zero (e.g., if there is no usage for a particular journal title for a given metric in a given month for the TR_J1 report).
• The tabular report should include a hidden code called the “byte order mark” (BOM) (if delivered as a CSV or TSV file) which is used by Excel to determine that the document should be handled as “UTF-8” encoding (Unicode characters for handling all language scripts), whereas the JSON report should NOT include the BOM.
• JSON is not an ordered data format, which means there is no specific “column order” for the elements within a “report item” (e.g., a given title “row” within a title report), unlike the tabular reports in which the column order is precisely specified.
• For Item Reports, the names of some elements differ between the tabular and JSON reports (the first is tabular, the second, JSON):
  • Parent_Data_Type vs. Item_Parent.Data_Type
  • Authors vs. Item_Contributors
  • Publication_Date vs. Item_Dates
  • Article_Version vs. Item_Attributes
  • Proprietary ID vs Proprietary

  Additionally there is no way to request an IR via SUSHI that looks like
  an IR_A1 because there is no parameter for the Parent_Data_Type filter, although an IR with “include_parent_details=True” requested can be used to generate an IR_A1.
  This may be fixed in a future update to the SUSHI API.

Section 4 of the Code of Practice details which attributes are optional and which mandatory for each kind of report for the tabular reports. Note that “optional” here means for the user to choose to include; providers must implement all attributes.

To maximize the attribute data breakdowns available for each of the Master Reports (ignoring custom data added by individual vendors), use the following fields:

• TR:</br />
  https://sushi.url/reports/tr?requestor_id=xxx&customer_id=xxx&begin_date=yyyy-mm&end_date=yyyy-mm&attributes_to_show=YOP|Access_Type|Data_Type|Section_Type|Access_Method
  </br />
• PR:</br />
  https://sushi.url/reports/pr?requestor_id=xxx&customer_id=xxx&begin_date=yyyy-mm&end_date=yyyy-mm&attributes_to_show=Data_Type|Access_Method
  </br />
• DR:</br />
  https://sushi.url/reports/dr?requestor_id=xxx&customer_id=xxx&begin_date=yyyy-mm&end_date=yyyy-mm&attributes_to_show=Data_Type|Access_Method
  </br />
• IR:</br />
  https://sushi.url/reports/ir?requestor_id=xxx&customer_id=xxx&begin_date=yyyy-mm&end_date=yyyy-mm&attributes_to_show=YOP|Access_Type|Data_Type|Authors|Publication_Date|Article_Version|Access_Method

The rule is: list all attributes marked with ‘O’ in tables 4.b, 4.f, 4.k
and 4.p except the Parent and Component elements in attributes_to_show,
separated by ‘|’. For including the Parent and Component elements in IR
add include_parent_details=True and include_component_details=True. This
is explained in section 3.3.8.

Custom Elements/Values

There is no rule under 5.0.1 for default inclusion/exclusion, but when incremental release 5.0.2 is official, custom values (created by specific vendors, allowed for in section 11 of the COP5) in enumerated elements like Metric_Type should be excluded by default
unless you specify a filter that includes them.

This is not specific to SUSHI but is a point of confusion worth clarifying. Note that YOP should not be confused with the range of years of the usage report itself, i.e. the years during which the usage took place.

A major change from COUNTER COP4 to COP5 was the breakdown of the rows of usage data by individual YOP when requested for Master Reports (or automatically for TR_J4).

Master Reports only: An important distinction needs to be made between filtering the results to a range of years, versus selecting YOP as an attribute to break down the report (“attributes to show”). If a range of specific years is requested but YOP is NOT selected as an attribute to show, the data will be aggregated (added up) for just those years. But if the YOP attribute is added to the request, each individual year will be a separate row (or rows if other attributes are also added). Because zero-usage rows are not provided in COP5, this can often mean that some YOP years will seem to be “missing”.

Specific to SUSHI, the standard requires vendors to support a parameter to filter to a user-selected range of YOP years.

The standard requires that vendors provide an option to request all of the report types aggregated across all of their members, in combination with all of the other options for specifying the contents of reports. Vendors implement this through the use of consortium-specific authentication credentials, as the SUSHI standard does not allow vendors to create their own proprietary parameters for the API calls.

Validation Tool

You will need to “register” for a free account, which enables the system to keep track of your tests and provide repeat access to SUSHI results for up to 7 days.

The tool does not validate the non-usage reports (server status, supported reports list, and members list (consortial)) but just gives you the JSON.

For the usage reports, the result is a simple summary delivered as an Excel spreadsheet file of whatever problems have been found. The validation report itself does not include the actual JSON response, but you can download that (and re-retrieve the validation reports by download or email) by going to the “Report History” tab at the top.

For the Master Reports, the tool allows you to select whatever mix of filtered attributes you want.

You can also use it for testing tabular reports (which you need to upload the file for).