Guiding Principles
The global landscape of EdTech evaluation methods is diverse. Different approaches exist, as do a number of frameworks and instruments. There exist multiple foci for evaluations ranging from the design of a product to large-scale implementation.
While the global landscape for evaluations is rich, tools and solutions for doing such evaluations in India are not available. To build solutions for the Indian context, the Tulna team has adopted a research-based approach to create frameworks that are sensitive to the local context but globally relevant as well.
Image courtesy: Neena Majumdar
To systematically approach the design of a framework that is robust, meets identified needs and yet has potential for generalizability, EdTech Tulna is guided by the following principles:
Sound foundation for effective outcomes.
The EdTech Tulna framework emphasizes understanding the mechanism of how and why a product is effective and evaluating the extent of effectiveness in the Indian context.
Mirroring the learning complexity.
Students’ learning process using EdTech is complex and its quality has multiple facets. A single numerical rating blackboxes this complexity and fails to capture this multidimensionality. Thus, the EdTech Tulna index presents the results of the evaluation in terms of a multi-dimensional scale.
Built on theoretical and empirical research.
The EdTech Tulna index is drawn from the disciplinary foundations in the Learning Sciences, Human-Computer Interaction, and Educational Technology. They are aligned to the national standards in India and informed by empirical studies with relevant stakeholders.
Approach to Evaluation
EdTech Tulna employs a tiered approach starting from the evaluation of the design of a product followed by evaluating the interaction of the learners and teachers with the product in controlled settings. Next, the learning outcomes in-situ are evaluated followed by evaluating the outcomes at scale at a program level. This approach ensures that a product has been designed with a sound foundation thus resulting in a high likelihood for the product to support effective learning at scale on the ground.
Use cases being evaluated
EdTech Tulna is offering standards and evaluations catering to five key use-cases for mathematics, english and science (grades PreK to 10).
- Personalized Adaptive Learning (PAL): There is evidence that PAL is an effective means of addressing the diverse needs of different learners. There is also a rich supply of PAL solutions in the Indian EdTech ecosystem.
- Digital Classrooms (DCR): This use-case is popular amongst large scale and institutional adopters of EdTech such as governments and school chains, who continue to highlight the need for support for undertaking high stakes procurement decisions.
- Interactive Audio Visual (Interactive AVs): Due to Covid 19, there has been an increase in the use of such solutions and multiple stakeholders in the ecosystem have highlighted the need for support in designing, building and signal quality for these solutions.
- Games-based learning (Games): This use-case is popular among product companies, with a large number of offerings providing games-based learning to children across grades. There is also evidence to show that games-based learning is effective in increasing learner engagement and subsequently learning outcomes.
- Practice and Doubt Solving (PDS): Given the large learning gaps that exist today, children require additional support in practicing, remediating and keeping up with what is being taught in the classroom. PDS products provide ‘just-in-time’ support to children to allow them to revise concepts independently and seek additional support via doubt solving.
Commencing with these use cases enables EdTech Tulna’s offering of standards, evaluation tools and evaluation reports, to be immediately relevant for multiple stakeholders. EdTech Tulna’s mandate is to extend to other use cases of EdTech gradually.
Tiered evaluation approach used in EdTech Tulna, starting from the foundational tier of product design evaluation
Process followed
The EdTech Tulna framework is informed by:
The findings and insights from these literature reviews and interviews informed the design of the EdTech Tulna tools and framework comprising multiple dimensions and criteria. This process also revealed how potential users could be supported effectively through these tools and framework.
Research Literature
Multiple evaluation standards being used by global organizations were surveyed. The gap analysis showed that the existing models and standards can not be directly applied to the needs and requirements in the Indian context. Global research literature on EdTech evaluation instruments and rubrics are continuously reviewed to determine relevant evaluation criteria.
Educational Policies
Government policy documents were also reviewed – National Curriculum Framework (NCF) (2005) and National Education Policy (2020), and government recommended assessment instruments such as Try And Measure Aptitude And Natural Abilities (TAMANNA) – to understand grade specific requirements and expectations of the learners in India.
Stakeholder Interviews
Multiple evaluation standards being used by global organizations were surveyed. The gap analysis showed that the existing models and standards can not be directly applied to the needs and requirements in the Indian context. Global research literature on EdTech evaluation instruments and rubrics are continuously reviewed to determine relevant evaluation criteria.
Requirements and scope of EdTech Tulna
- Non-generic: Customize to domain, grade, product type, use-case
- Curriculum aligned: Aligned to national standards and curricula
- Stakeholder specific: Different views of index for various types of decision-makers for differing purposes
- Robustness: Needs to be valid and reliable
- Low barrier for evaluations: High usability of index
References
- National Curriculum Framework (2005). National Council of Educational Research and Training, New Delhi, India.
- National Education Policy (2020). Ministry of Human Resource Development, Government of India.
- Try And Measure Aptitude And Natural Abilities (TAMANNA) (n.d.) Department of School Education and Literacy, Ministry of Human Resource Development, Government of India.
