Many instructional technologies do not achieve high levels of use in an institution. Others are adopted but quickly abandoned. Technologies fail for myriad reasons, but many failures can be avoided if a comprehensive evaluation process is used before adoption. We analyzed the reasons for failure and, from that, developed the USA2 framework to guide institutions in asking the right questions before adopting a new technology.
The USA2 framework includes six adoption criteria: utility, security, accessibility, usability, scalability, and affordability. Questions related to each of these criteria can be found at https://mycsun.box.com/v/USA2-Framework. We have found that these categories cover the critical issues that schools must be examine when deciding whether to adopt a new technology.
Utility
Utility relates to whether the
product provides features users need. A “cool” set of features isn’t worth much
if it doesn’t help users do what they want or need to do. The most important
considerations are what the tool can do and whether it improves teaching or
learning (or both) in ways that tools are already in use cannot. For example,
if faculty are happy with PowerPoint, they are unlikely to deem another
slide-analogy presentation tool worth the effort to learn unless it provides additional
features that they think unique and useful. Likewise, when replacing a tool
with a less expensive one, the school must take care to evaluate whether the
features frequently used in the original tool are available in the proposed
replacement. Even apparently minor features can be important if faculty have
grown accustomed to using them. For instance, an institution might replace its classroom
and desktop video capture system with a cheaper one that captures only classroom
video on grounds that there are many alternatives for doing screen recording. But
faculty might balk at using two systems instead of one and may give up on the
whole video capture endeavor.
Security
Instructional technology security has
three essential components: confidentiality,
integrity, and access for authorized users. Confidentiality means that users’ private
information is kept from those who have no legitimate reason to access it.
Integrity means that data is safe from malicious attack. User access, the third
component, is often sacrificed in the interests of the first two. A system is
secure when two-factor authentication, which requires the user to enter a
password and a code received on a mobile or other device, is used; however, the
user may find this process to be unacceptably slow and tedious. In this way,
security often strikes a balance between competing interests.
Accessibility
Accessibility refers to the ability
of individuals with disabilities to access electronic information. For a
technology to be accessible, it must enable all users to access information and
complete tasks with essentially the same degree of ease, regardless of the
presence of a disability or use of assistive technology. Equity and fairness
require schools to ensure that all instructional technology is accessible. Additionally,
accessibility is the law. Sections 504 and 508 of the US Rehabilitation Code,
as well as the Americans with Disabilities Act, dictate that those with
disabilities should have the same access as others.
When evaluating accessibility, a
good place to start is the Voluntary Product Accessibility Template
(VPAT), which should be available from all vendors based in the United States. The
VPAT indicates how information and communication technology products meet
standards for accessibility and identifies any areas where standards have not
been met. In addition, institutions should ask whether any accessibility
testing has been done and whether the results of testing are available. If the
school can access the tool prior to purchase, it can assess for important
accessibility features. These include ease of navigation without a mouse (many
users with vision or mobility impairments may not be able to use a mouse); the presence
of alternative text for screen reader users; closed captions for those who
cannot hear or have learning disabilities; and appropriate use of color. (Is contrast
between text or controls and backgrounds high enough? Are alternatives provided
when color is used to impart meaning?)
Usability
Usability relates to how easy and
pleasant a product is to use. Jakob Nielsen (2012) defines usability as having
five components:
- Learnability:
Is the tool easy to learn how to use?
- Efficiency:
Can tasks be accomplished quickly with the tool?
- Memorability:
Is it easy to remember how to use the tool after not using it for a while?
- Errors:
Are errors made frequently? Is it easy to recover from errors?
- Satisfaction:
Does the user enjoy using the product?
When evaluating usability, schools
can assess for such matters as adherence to common standards or conventions,
consistent navigation, and error handling.
Sustainability
Sustainability is the product’s
ongoing ability to meet users’ needs. It can include the level of support
needed over time to enable users to learn and use the tool. It can also involve
scalability—the capacity of the technology to support increased usage over
time, with increased demands on hardware, software, data storage, or bandwidth.
The vendor must be able to remain available to update and improve the product
as needs change.
Finally, when use of the tool must be discontinued—either because of a move to another tool or because the tool is no longer available—there could problems with data loss if any stored data cannot be exported in a usable format. Anyone who has done a learning management system migration understands some of the pain involved with this last point.
Affordability
Affordability includes both the
direct and indirect costs of acquiring and using the technology. Direct costs
may include licensing, data storage, and user training. Indirect costs include
any administration or instructional designer/technologist time necessary to
support the technology. Anticipated increases in cost over time must be
considered. Affordability can be evaluated by looking at the direct and
indirect costs of the tool; consideration should be given to whether the cost
is likely to increase significantly over time.
Using the USA2 framework for decision-making
When choosing a new technology, institutions
should start by identifying everyone who will have a stake in the decision.
Each group—for instance, administration, finance, IT, faculty, and students—may
have different and even competing needs and priorities. These stakeholders should
help identify the relative importance of the six criteria as they apply to the
specific technology and the context in which it will be used. Then the tool should
be evaluated on each of the six criteria in the USA2 framework.
Institutions might want to use a weighted decision matrix like the ones provided at https://mycsun.box.com/v/USA2-weighted-matrix. After assigning scores to the criteria according to their relative importance, they can evaluate the tool for how well it meets each of the criteria, again assigning a numerical score. Multiplying the two numbers and then adding for all six criteria will result in an overall score, which a school can use to evaluate a single tool or compare multiple tools. Alternatively, to take a more holistic approach, stakeholders can get together and discuss a tool and which criteria it meets or does not meet, reaching a consensus on whether to adopt it. Either way, we believe that the USA2 framework will help to improve instructional technology decisions.
Reference
Nielsen, J. (2012, January 4). Usability 101: Introduction to usability. Retrieved from https://www.nngroup.com/articles/usability-101-introduction-to-usability
Suzanne David, MA, is the e-learning technology manager and Maria Fernandez, MS, is the assistant director of distance learning at California State University, Northridge.
