COUNTER - Online Usage of Electronic Resources

COUNTER Code of Practice

Appendix E

I. General Auditing Requirements

a. Auditing and test-scripts

The COUNTER Auditing requirements are needed to ensure that the usage reports provided by vendors are in line with the COUNTER principles of credibility, consistency and compatibility. For this purpose COUNTER has defined specific audit test-scripts for each of the COUNTER required usage reports. As the majority of vendors will work with their own auditor, the test-scripts will guarantee that each of them will follow an identical auditing procedure and result measurement.

b. General conditions for carrying out an audit test

It can be assumed that an auditor will be given gratis online access to all content whose usage statistics are being audited.

COUNTER has defined a reporting period as a calendar month. A report pulled for any given month will reflect all activity of a customer for the entire month in question. As a consequence this applies also to auditing activity and an auditor should always finalize the audit tests within one and the same calendar month. Any activity on an audit account not related to the audit test should be prevented, as this will make the test reports unreliable.

To prevent any collision of reported data, an auditor should be allowed to set-up and maintain separate accounts for each of the audit tests. The auditor will also use a try-out account to prepare for the audit-tests. All accounts should be set up in such a way that only the auditor carrying out a test can access the vendor’s site.

c. Two types of audit tests:

The auditor will test the layout, format and delivery of a vendor’s usage report
The auditor will test the numbers reported by the vendor by carrying out detailed test-scripts.

II. The Required Audit Tests

1. Checking the report lay-out, file-format and delivery against the Code of Practice

The auditor will check whether each of the reports mentioned below will comply with the report examples and descriptions as made available in the COUNTER Code of Practice.

The following items need to be checked:

The lay-out of report (headers/footers, number of fields, field sequence, totaling field and format of reported numbers)
The required ‘save-as’ formats
The receipt and timeliness of an email alert once usage reports are updated.

2. Checking the usage numbers as reported

Journal Report 1: Number of Successful Full-Text Article Requests by Month and Journal
(Full journal name, print ISSN and online ISSN are listed.)

	Print ISSN	Online ISSN	Jan - 01	Feb - 01	Mar - 01	Calendar YTD
Total for all journals			6637	8732	7550	45897
Journal of AA	1212-3131	3225-3123	456	521	665	4532
Journal of BB	9821-3361	2312-8751	203	251	275	3465
Journal of CC	2464-2121	0154-1521	0	0	0	0
Journal of DD	5355-5444	0165-5542	203	251	275	2978

Note:

The 'Total for all journals' line is provided at the top of the Table to allow it to be stripped out without disrupting the rest of the Table, as the number of journals included may vary from one month to another.
Journals for which the number of full-text article requests is zero in every month should be included in Journal Report 1

The above report complies with the COUNTER Code of Practice for collection and reporting of usage data. For definitions of the terms used, See Section 3.

Journal Report 1: AUDITING Requirements:

An audit of this report requires the following:

The audit-test must be conducted in such a way that the auditor’s activities during the audit-test can be isolated from other activities on the vendor’s site. Depending on the site being tested, the auditor should conduct the audit-test from a computer with a unique IP address and/or using a unique account number.
The auditor should accept user/machine and session cookies when prompted.
The auditor should have access to all available journals as published on the platform of the vendor.
Audit-test JR1-1:
1. For the audit report, the auditor should perform 100 requests for Full Text Articles from a selection of journals available on the vendor’s site. N.B. the auditor should allow at least 30 seconds when requesting one and the same article.
2. The auditor must record the journals included in the audit-test and the number of requests for full text articles for each journal.
3. The audit report should show the Total for all requests, broken down by journal.
4. The vendor will pass this audit test when the Total (across all journals) on the auditor’s report is within a –8% and +2% reliability window of the total presented on the vendor’s Journal Report 1.

Audit-test JR1-2: The 10 and 30 seconds filters.
1. The auditor will audit-test the 10 and 30 seconds filter for this report. The audit-test consists of clicking links to an article full text twice in succession (double-clicks). For HTML articles, if the two clicks occur within a ten second time-span, only one full text request should be recorded, if the two clicks occur with more than 10 seconds between, then two full text requests should be counted. For articles in PDF format, the time-span is 30 seconds. The audit test should include requesting articles where double-clicking occurs within the threshold as well as requesting articles where the time between clicks exceeds the threshold.
2. The auditor should request full text for 50 different articles, performing double-clicks within 10 seconds if the format requested is HTML or within 30 seconds if the format requested is PDF. For each article requested the auditor will record just 1 full text request for each set of double-clicks, recording the activity by journal keeping track of the HTML and PDF activity separately.
3. The auditor should request full text for 50 articles, performing double-clicks with 11 or more seconds between clicks for HTML and 31 or more seconds between clicks for PDF. For each article requested, the auditor will record a full text request for each click (2 per article), recording the activity by journal keeping track of the HTML and PDF activity separately.
4. Vendors will pass the Audit-test 2 when the total of activity on the vendor’s report for the journals audited is within a threshold of -8% and +2% of the auditor’s total.

It is needed to separate audit-test JR1-1 and audit-test JR1-2 by using separate accounts to avoid collisions of numbers.

Journal Report 2: Turnaways by Month and Journal
(Full journal name, print ISSN and online ISSN are listed.)
-this report is applicable only where the user access model is based on a maximum number of concurrent users

	Print ISSN	Online ISSN	Page Type	Jan - 01	Feb - 01	Mar - 01	Calendar YTD
Total Full-text Turnaways for all Journals				453	233	318	4765
Journal of AA	1212-3131	3225-3123	Full text Turnaways	23	40	12	342
Journal of BB	9821-3361	2312-8751	Full text Turnaways	18	20	16	287

The above report complies with the COUNTER Code of Practice for collection and reporting of usage data. For definitions of the above terms, see Section 3.

Journal Report 2: AUDITING Requirements:

The audit-test must be conducted in such a way that the auditor’s activities during the audit-test can be isolated from other activities on the vendor’s site. Depending on the site being tested, the auditor should conduct the audit-test from 4 computers within a unique account number; the vendor should allow 3 registered users having simultaneous access to all available vendor databases. If the vendor system cannot allow specifically 3 simultaneous users, then the auditor must know number of registered users allowed for the test and use this number where ever the number 3 is used below. N.B. the important number for the vendor to understand is the number of sessions that are allowed to be active before the system will turn-away subsequent sessions.
The auditor should accept user/machine and session cookies when prompted.
The auditor should have access to all journals as made available on the platform of the vendor.
Audit-test JR2-1:
1. The audit-test is to have 3 active (registered) users on the site requesting full text articles for one and the same journal. This means that all available sessions are active. An additional computer will then be used to log-in and attempt to carry out an article request for that same journal. This user should be refused access because of exceeding the simultaneous user threshold. Each time access is refused, the auditor will record this as a turn-away.
2. This audit-test should be repeated between 40 and 50 times and at different periods of the day allowing at least 20 seconds between each test. The auditor should record each time a turn-away occurs and the name of the journal accessed.
3. The vendor’s report will pass this test when the total number of turnaways shown is within a –8% and +2% reliability window of the total on the auditor’s report

Database Report 1: Total Searches and Sessions by Month and Database

		Jan - 01	Feb - 01	Mar - 01	Calendar YTD
Database AA	Searches Run	2322	2520	2742	29878
Database AA	Sessions	1821	1929	2211	27654

Database BB	Searches Run	3466	3210	4459	36543
Database BB	Sessions	1987	2200	2544	24209

The above report complies with the COUNTER Code of Practice for collection and reporting of usage data. For definitions of the above terms used, see Section 3.

Database Report 1: AUDITING Requirements:

An audit of this report requires the following:

The audit-test must be conducted in such a way that the auditor’s activities during the audit-test can be isolated from other activities on the vendor’s site. Depending on the site being tested, the auditor should conduct the audit-test from a computer with a unique IP address and/or using a unique account number.
The auditor should accept user/machine and session cookies when prompted.
The auditor should have access to all databases as made available on the platform of the vendor.
Audit-test DB1-1:
1. If a vendor offers more than one database, the auditor should run 100 searches on a subset of the databases made available to them. In case there is only 1 database the number of searches should be 50. Individual searches should always be run against only one database at a time. All database searches are considered valid and, for each search, the auditor will record the database and result total number returned by the search (if applicable). If a vendor’s COUNTER reports do not include searches yielding zero results or when the number of results exceed some predefined threshold, then these categories of searches should be recorded separately and not included in the final tally. N.B. the auditor should allow at least 11 seconds between each search when repeating the same search on the same database.
2. To be able to measure the number of sessions, the tests should consist of at least 2 sessions. During the tests, the auditor can either explicitly log-out of a session and log back in to continue the test, or, if no log-out option is available, the auditor should close the browser then open a new browser and continue the test (Note that if the vendor maintains the previous session even when the browser has been closed and re-opened, the auditor will need to wait for the session inactivity time used by the vendor – which is set at 30 minutes; the auditor should allow 10 minutes additional tolerance for the audit test (which is thus 40 minutes in total), before continuing the test as a new session.)
3. Each time a new session is started, the auditor should record this fact.
4. Each time a search is conducted, the auditor will record the search and the database searched.
5. As each search is conducted, the auditor will indicate that the database was accessed during the current session. (N.B. a database will only get credit for the session if it has been searched during that session.)
6. The audit report should show a breakdown of searches and sessions by database with a Total for each.
7. A vendor will pass this audit test when the Totals for searches and sessions on the auditor’s report are within a -8% and +2% reliability window of the sum of the sessions and searches for all databases on the vendor’ Database Report 1.

Audit-Test DB1-2: Searches on multiple databases (federated search)
1. It is necessary to separate audit-test DB1-1 and audit-test DB1-2 by using separate accounts to avoid collisions of numbers
2. The auditor should run 100 searches in total and make sure that about 50 of searches are run over combinations of 2 databases and the other 50 searches are run over a combination of all databases as made available by the vendor.
3. The auditor should keep a record of the number of searches executed for both options, indicating which database each search was applied against If a vendor’s COUNTER reports do not include searches yielding zero results or when the number of results exceed some predefined threshold, then these categories of searches should be recorded separately and not included in the final tally.
4. The audit report should show the count of searches by database plus the total database/searches (E.G. if the audit procedure is followed exactly and the auditor has access to 10 databases, the total would be 600 -- 50x2 + 50x10).
5. The vendor’s report will pass this test if the sum of the searches by database matches the total on the audit report within a –8% and +2% reliability window.

Database Report 2: Turnaways by Month and Database
-this report is applicable only where the user access model is based on a maximum number of concurrent users.

		Jan - 01	Feb - 01	Mar - 01	Calendar YTD
Total Database Record Turnaways for all Databases	Database Record Turnaways	453	233	318	2435
Database AA	Database Record Turnaways	23	40	12	60
Database BB	Database Record Turnaways	18	20	16	82

The above report complies with the COUNTER Code of Practice for collection and reporting of usage data. For definitions of the terms used, see Section 3.

Database Report 2: AUDITING Requirements:

An audit of this report requires the following:

The audit-test must be conducted in such a way that the auditor’s activities during the audit-test can be isolated from other activities on the vendor’s site. Depending on the site being tested, the auditor should conduct the audit-test from 4 computers within a unique account number; the vendor should allow 3 registered users having simultaneous access to all available vendor databases. If the vendor system cannot allow specifically 3 simultaneous users, then the auditor must know number of registered users allowed for the test and use this number where ever the number 3 is used below. N.B. the important number for the vendor to understand is the number of session that can be active before the system will turn-away subsequent sessions.
The auditor should accept user/machine and session cookies when prompted.
The auditor should have access to all databases as made available on the platform of the vendor.
Audit-test DB2-1:
1. The database used for this test should be different from the one used for Database Report 1
2. The audit-test is to have 3 active (registered) users on the site carrying out searches on one and the same available database such that all available sessions are active. An additional computer will then be used to log-in and attempt to carry out a search on the same database. This user should be refused access because of exceeding the simultaneous user threshold. Each time access is refused, the auditor will record this as a turn-away.
3. This audit-test should be repeated between 40 and 50 times and at different periods of the day allowing at least 20 seconds between each test. Recording each time a turn-away occurs and the database accessed.
4. The vendor’s report will pass this test when the total number of turnaways shown on its Database Report 2 is within –8% and +2% reliability window of the total on the auditor’s report

Database Report 3: Total Searches and Sessions by Month and Service

		Jan - 01	Feb - 01	Mar - 01	Calendar YTD
Total for Service	Searches Run	16567	18643	20987	80654
Total for Service	Sessions	12007	12677	13003	65487

Database Report 3: AUDITING Requirements:

An audit of this report requires the following:

The audit-test should be conducted in conjunction with the test results for Database Report 1 as conducted in section V. With Database Report 1, the auditor has recorded the number of searches performed as well as the number of sessions and indicated which databases they apply to.
Database Report 3 only counts the discrete searches and sessions. For example, if a 100 searches run for Database Report 1 were conducted in 10 session of 10 searches each and the auditor had accesses to 10 databases, Database Report 3 would be expected to show a total of 10 sessions and 100 searches (even though the sum of the searches and sessions on Database Report 1 will equal 600 and 60 respectively.
A vendor will pass this audit test if their Database Report 3 shows totals for sessions and searches that are within the reliability window of -8% and +2% of the total of unique sessions and searches counted on the auditor’s report for Database Report 1.