Our goal was to develop and publish a set of common criteria that can be used to determine the acceptability of services and solutions for inclusion in the Academic Toolbox at the University of Toronto – including on-premises solutions, managed hosted solutions or Software-as-a-Service (SaaS) cloud-based solutions.

Thinking about using a new teaching app or tool? Answer the questions below. If you answer NO to any of them, then reach out to an educational technology professional on campus to see if it’s the right tool for you.

A. Information Risk, Security, Privacy
Question: Does the solution protect sensitive information, such as student data or intellectual property from being put at risk?

When considering services and solutions for use at the University of Toronto, it is essential to understand the risk to the University that such services and solutions present. Risk to the University through the use of information services can occur for many reasons – threats to private or personally identifiable and other sensitive information, or vulnerabilities in the software, hardware, out-sourced or built-to-order components.

All proponents will be required to participate in our Information Risk and Risk Management (IRRM) audit processes which will cover standards related to the protection of personally identifiable information, protection of intellectual property, information security practices, access control practices, monitoring practices, business continuity planning, capacity and scalability of architecture, and so on (see link to Information Security Guidelines in the Background Reading section on the right side of this page). Proponents should note that privacy policies would be made available to members of the University community.

B. Tool Interoperability and Integration
Question: Does the solution allow the University to take advantage of international standards for interoperability and integration?

In considering proposed services and solution, it is recommended that proponents should be able to demonstrate that the solution allows the University to leverage international standards regarding the interoperability of teaching and learning tools. Examples should include the Learning Tool Interoperability (LTI) standard, the IMS Common Cartridge format, the Question and Test Interoperability (QTI) standard, and the Sharable Content Object Reference Model (SCORM), etc.

In particular, software or solutions delivered through a web browser should include a secured Application Programming Interface (API) to allow authorized system to interact with the data held behind the interface.

The university is committed to the inclusion of students who may not have access to mobile devices, but is interested in leveraging the benefits of BYOD (Bring Your Own Device) where feasible, and therefore, solutions and services should ideally be Operating System-agnostic, and where applicable, they should work with all contemporary web browsers. As such, the University is also interested in leveraging mobile access (either through a responsive web interface or multiple-OS-specific apps).

C. Single Sign On / Identity Management
Question: Does the solution allow our users to have a seamless login experience, and the ability to move from one application to another within the toolbox?

The primary credential at the University is UTorID, which consists of a user ID and password pair. Passwords can be alphanumeric, and between 8 characters and 32 characters. Current authentication technologies include Shibboleth Identity Provider (SAML 2.0), Active Directory, LDAP v3, and Kerberos.
Multi-factor authentication is required for users that handle private and confidential information on behalf of others (e.g. Registrars and other University administrators who handle student data). The standard device is an eToken USB key, supplied by SafeNet, which contains X.509 certificates. One-Time-Passwords (OTP) must be used on devices that cannot accommodate USB keys (e.g. IOS devices) – the University uses key fobs or client software supplied by SafeNet. Active Directory contains user’s X.509 certificates.

D. Authorization
Question: Does the solution allow for different kinds of roles (for example, a different experience for an instructor versus a student, or between an instructor and professional staff administrator)?

Authorization gives users permitted access to what they need to see or do within the system/tool, and this access is in compliance with the defined role of the user. Access for users has to be screened with enough granularity to limit the risk to inadvertent exposure to information to users that is not intended for their use. Enterprise LDAP repositories include Active Directory and OpenLDAP, each of which is used for coarse-grained authorization. Fine-grained authorization (i.e. the fine-grained levels required by applications that are required to support privacy and confidentiality) is usually supported within applications (e.g. via database tables) or with localized LDAP repositories (e.g. a local LDAP server).
Within the risk and security parameters stated about, the University is particularly interested in solutions which allow an instructor or professional staff to have an authentic “student view” for instructional design and planning purposes.

E. Student and Human Resource Information System Compatibility (SIS and HRIS respectively)
Question: Our SIS and HRIS systems are the authoritative source for information about members of our community. Does the solution work well with our SIS and HRIS systems (can data flow properly to and from our main systems)?

Repository of Student Information (ROSI): The University has a mainframe-based custom built Student Information System. This system supports the administrative functions of the academic lifecycle. The main subsystems are: Admissions, Course and Program offerings, Registration and enrolment, Student Fees, Awards, Grading, Convocation, and Transcripts.

Much of the student data are considered private and confidential and access is provided on a need-to-know basis. Requests for non-public ROSI data are subject to senior management approval and require a signed non-disclosure agreement. Course and Program data are considered public and are available from ROSI and other sources (e.g. On-line Calendar system).

A limited number of formats are supported by ROSI (CSV, Fixed Record Length). Batch jobs are usually required to export data, which are retrieved via sftp from a secure server, however, the University is increasingly committed to moving away from batch processing. Authorized VPN access is required to establish the sftp connection. Asynchronous Record-by-record access can be supported via message-oriented middleware (WebSphere MQ) and secure (HTTPS) RESTful-style Services are beginning to be supported. SOAP is an option, with WS-Security and SAML 2.0 tokens. IBM LTPAToken2 is an alternative token option if both end-points support it.

Human Resources Information System (HRIS): Very limited content is available from the HRIS system. All data are provided on a strict need-to-know basis. Requests for HR data are subject to senior management approval.

F. Record Discovery, Curation and Preservation
Question: Where a solution creates intellectual artifacts (and related metadata) does it allow the University (and its users) to access those artifacts, for both research and operational purposes? Does the solution allow us to store the artifacts in repositories of our choosing?

Relating to the interoperability criteria, tools should not operate in silos and the information should be accessed appropriately between tools, and organized in standardized, accessible manners that support the intended use. In appropriate contexts, proposed solutions and services will need to comply with standards for harvesting records for discovery and the ability to capture preservation, rights, and descriptive metadata in standard and interoperable formats. Standard protocols for moving documents and their metadata between systems to ensure interoperability with related systems will also be important. Vendors may be expected to demonstrate that that data can be extracted under reasonable parameters, as the risks of data loss and hidden costs in preserving data must be considered. Records that are stored with open access and within internal access should be considered with respect to rights and permissions.

G. Learning Analytics and Business Intelligence
Question: Can the University access data and metadata generated by the use of the solution for both research and operational needs?

Proposed solutions and services should allow the University of Toronto to have unrestricted access to the data generated by its users without any additional costs and ideally through unrestricted APIs. The University is particularly interested in the growing field of Learning Analytics, including standards, (eg. IMS CALIPER), solutions and services that would allow us to maximize the use of learning analytic tools (see link in the Background Reading section to the right for more details). Furthermore, the University of Toronto Business Intelligence group performs extraction, transformation, loading and other types of data warehousing activity using assorted Informatica and Cognos tools, and proposed services and solutions should be compatible with this activity.

Likewise, the University seeks to limit the extent to which 3rd party vendors and providers can use our data and metadata for other purposes, save the effective running of the service, and in all cases, would require contractual and/or written consent before access to our data is granted.

H. Terms of Service
Question: In order to use the solution, are users required to click independently on a Terms of Service agreement that may contain problematic language, or is there a University-wide Terms of Service that protects the interests of our users?

It is the position of the University of Toronto that its end users should not be required to agree to Terms of Service agreements on an individual basis when accessing enterprise services and solutions (through click-throughs or any other mechanisms). Service agreements and licenses are between providers and the University and not between the provider and individuals. Proponents should note that ToS documentation would be made available to members of the University community.

I. Copyright / Intellectual Property / Content Control
Question: Does the solution’s contract or Terms of Service make claims on the intellectual property of our users, or define other restrictions on use that are not compatible with University practice or policy?

It is the position of the University of Toronto that matters related to intellectual property ownership are governed by internal university policies, and no supplier of services and solutions should make any ownership or transfer claims on intellectual property and content created using the service or uploaded to it. The University reserves the right to grant non-exclusive licenses to external suppliers of services. Likewise, any copyright compliance mechanisms in any proposed solutions or services must reflect and be consistent with Canadian copyright legislation.

Furthermore, it is the position of the University of Toronto that matters related to how content is managed and controlled are governed by internal university policies, and no supplier of services and solutions should place attempt to define those matters independent of the University.  For example, the determination of objectionable activities (for example, the uploading of ‘obscene’ material) is solely within the purview of the University, and proponents should not propose terms of service that set limits on the University’s determination in these matters.

J. UI Design, Branding
Question: Does the solution allow us to control the user interface design and/or brand the experience?

The University is particularly interested in solutions and services, which allow us to have maximum control over branding and design elements. Ideally, this could mean ‘white labelling’ so that it is the University’s brand and name that appears to end users, not the product or company name. Administrative access for User Interface control is, therefore, an asset. Administrative controls that support the use of branding that can enhance the student experience and allow for institutional “pilot testing” and rating of tools for use in different circumstances is an asset as well.

K. AODA Compliance
Question: Is the solution AODA compliant?

In considering proposed services and solution, proponents will need to demonstrate that the solution allows the University to meet its legal obligations and requirements with regards to the Accessibility for Ontarians with Disabilities Act (AODA). Responsive design is a key criterion at the University. For more, please visit: http://www.aoda.utoronto.ca

L. Classroom Technologies
Question: If the solution is meant to be used in a classroom, is it compatible with the University’s classroom technology standards?

Technologies or solutions that are meant to be used in the University’s standardized classrooms must be compatible with the specifications and configurations maintained by each of our relevant campus-based space management groups, for example, the division of Academic & Campus Events, which is responsible for standardized classroom technologies at the University’s downtown campus. This includes projection, audio, lighting and teaching station standards. Information about current specifications, configurations and standards may be found on the ACE website (http://www.osm.utoronto.ca/). Proponents should also make themselves familiar with relevant institutional policies and guidelines, including but not limited to Provostial Guidelines, as they relate to fee-based classroom technologies (for example, Audience Response Systems / ‘Clickers’).

M. Hardware Standards – On-Prem, Managed Hosting or SaaS/Cloud
Question: Does the solution meet the University’s technical standards and specifications?

Relevant University-based Information Technology Service teams reserve the right to evaluate the tools according to their current technical infrastructure standards. As mentioned above, the University of Toronto will be pleased to receive all relevant proposals, including solutions that may be on premises, in a managed hosting environment, or SaaS / cloud-based solutions. All proposals will have to meet our IRRM standards, regardless of where they are hosted (see Section A above). All proposals will need to include a Business Continuity plan and proof of scalability. Vendors are expected to demonstrate viability of their product within the institutional network, including providing local evidence of successful integration. Relating to the User Experience, the success of a third party tool may be impacted by the internal vs, external hosting party.

SaaS /cloud-based solutions should include testing, data retrieval parameters and third party software hosted at the university are subject to different criteria that internally developed solutions.

However, for on-premises solutions, using equipment that will be located in one or more of the University’s tri-campus data centres (DCs) and managed by a relevant University-based Information Technology Service, specific vendor and configuration requirements will need to be met. This includes all server, storage and networking equipment. Furthermore, application software must be compatible with a virtualized IT infrastructure. All on-premises solutions are subject to the approval of the IT departments that manages the relevant DC. The relevant University-based Information Technology Service teams should be involved during the development stages of new tools, in order to proactively identify and manage risk at the outset.

N. Exit Strategy and Change Resilience
Question: How difficult would it be to stop using the solution, and/or transition to another solution? Are there any protections in place regarding significant changes being made to a solution while it is in use?

Technology changes rather quickly, and the University recognizes that change may affect our ability to continue using a tool or solution, sometimes without notice. The University is committed to working with solution providers regarding both an exit strategy from a tool, and also to mitigating the negative effects of vendor-driven changes in functionality and/or business practice. In this regard, the University will be looking for evidence of how its content is curated (see Section F above), and for exportability of that content (not just as a theoretical construct, but actual demonstrations of exportability). As always, the University is very interested in content export solutions and methods that are standards based (e.g. Common Cartridge, SCORM, etc.).

By general principle, the University prefers to incorporate change resiliency into contracts through limits on a provider’s actions to make unannounced, unscheduled, undocumented, and/or, unapproved changes to its products or business practices (or at the very least, lets the University set the timing around upgrades) during the life of the contract. The University is also very interested in contractually accessing a vendor’s product enhancement process, particularly one where the University has the ability to provide direct, documented input into a solution’s improvement (we would like to see proof that a vendor takes our recommendations seriously).

O. Support & Professional Development
Question: Does the solution provider include professional development and a proper support strategy with their solution?|

Support for the use of educational technologies at the University of Toronto is provided by a highly diverse network of professionals who may be employed at the program, departmental, or divisional level, or in a central support unit. Proponents should have robust support systems in place to work with our professionals, and in some cases, our community members, including relevant Service Level Agreements, case tracking and resolution processes, and cost-effective professional development and training services. In the case of Open Source opportunities, solutions should be supported by an active community network or be supported by a contracting service agency.

P. Pedagogical Drivers
Question: Can the solution provider provide research into the pedagogical value of the solution?

Instructional decision-making and the assessment of pedagogical value related to the use of a particular solution is ultimately at the discretion of the University of Toronto’s instructors/departments. Furthermore, it is acceptable to consider that not all tools will be appropriate in all contexts, nor for all users, nor for all learning objectives and outcomes. However, it is highly recommended that solution providers be able to demonstrate that the intended use of a tool is grounded in education theory and evidence-based pedagogy. The educational value of a tool should be explicitly outlined (e.g., where appropriate, demonstrated through scholarly research and/or rigorous systematic design evaluation), and related to the needs of users.

Q. Cost-Benefit / Pricing
Question: Is the cost of the solution consistent with a cost-benefit analysis?

As a publicly funded institution, the University of Toronto encourages supplier competition to obtain value for money, and uses a variety of procurement tools and methodologies to ensure cost-effective solutions.

The University of Toronto recognizes that educational technologies and methodologies are important parts of pedagogical practice and can contribute to the enhancement of teaching and learning. The benefits of educational technologies are known, and proponents should target the needs of the institution from a cost-benefit analysis. Nonetheless, as a public institution, with budget constraints, the University is committed to cost-effectiveness, especially if any services or solutions involve direct-to-student costs that may be onerous. In general, the University would prefer easy-to manage licensing schemes, (for example, not on per-server basis), but based on a more inclusive and auditable set of user criteria (for example, the ability to track use by division or role). All proposals will need to include a Business Continuity plan. Proponents should be familiar not only with the needs of the marketplace, but also with comparative market pricing for educational technologies, and price their solutions accordingly, relatively, and realistically.

Proponents should also make themselves familiar with relevant institutional policies and guidelines, including but not limited to the Provostial Guidelines on the Use of Digital Learning Materials, and the policies and guidelines of our Procurement Services <http://www.procurement.utoronto.ca>.