Continuous improvement philosophy at the Graduate School of Management of Saint Petersburg University (GSOM SPbU) implies constant enhancement of the academic programs to ensure that the Business School alumni gain up-to-date knowledge, skills and expertise and become competitive in the international labor market. The Assurance of Learning Committee (AoL Committee) monitors the learning outcomes and makes recommendations on the academic programs improvement if needed. The AoL Committee was created in compliance with the standards of the Association to Advance Collegiate Schools of Business (AACSB). Although GSOM SPbU was accredited by AACSB in 2021, the Assurance of Learning principles have been introduced at GSOM SPbU since 2019. Olga Alkanova, Senior Lecturer of the Marketing Department, Chairman of the AoL Committee, and Andrey Zyatchin, EMBA&MBA Academic Director, talk about the methods and results of these efforts. 

Olga, in simple language, the AoL Committee should assess the students' learning outcomes and if the results are unsatisfactory, it proposes to improve the corresponding elements of the program, right?

Right. Students go to GSOM SPbU to gain knowledge. According to the AACSB terminology, every academic program has its learning goals that are decomposed into learning objectives with certain traits. Well-defined traits allow us to take measurement — assess students' task results according to certain criteria. This may be an oral or written presentation, case solution, course paper or thesis — we call them artifacts. Then, if we reveal a problem, we consider the curriculum and analyze why it has appeared: where the students lacked knowledge or practice. Then consultations with the faculty and discussion at the Program Advisory Board take place and updates are introduced: a new discipline, one or several extra topics, case study, task, etc. The sequence of topics and courses — anything — may change, too. Then the date of the next measurement is set to check whether the updates work. 

How else do you use learning goals, learning objectives and traits?

Ideally, if the system works properly, the development of any academic program starts with setting learning goals, defining learning objectives and so on. Also, we create a curriculum map — this document is often called in Russian ‘a matrix of competences’, where disciplines and learning objectives are matched up. All learning objectives cannot be included in one course as competences are formed during the whole period of study, so we should consider all steps of this process. Only then we will be able to see where and how we should update the academic program if a certain objective was not achieved.

Who are the AoL Committee members?

There are four working groups responsible for different academic program levels (bachelor, master, doctoral and EMBA/MBA programs), each consisting of a methodologist, academic directors and faculty members. Meetings are usually held from once every two months to four times a month, depending on current issues. Most of the academic directors see the Assurance of Learning procedures as an excellent tool for decision making regarding education quality improvement.

Andrey, as an academic director you actively use the Assurance of Learning. Please tell us about its practical value. 

As part of the AoL Committee's work we aligned concepts of MBA and Executive MBA programs with the GSOM SPbU mission, decomposed these concepts into key components to identify the learning goals and then projected them to the curriculum. Well-defined learning objectives, traits and scoring used for their assessment help to develop disciplines, case studies and home assignments. This is especially important when we invite business practitioners in class who professionally present their business cases to audience, but sometimes struggle to develop a methodologically comprehensive course containing knowledge that students may apply in their own businesses and get results.

How have the Assurance of Learning procedures helped to update the Executive MBA program?

According to the results of the Executive MBA program measurements, we developed and introduced a new format of the program aimed at the personalization of learning. Three disciplines are presented in this format: ‘Personal Professional Development and Learning Strategy’, ‘Psychology of Success’ and ‘Personal Career Management’. The first course runs at the beginning of the program and is dedicated to professional development and identification of student unconscious incompetence. Revealing these gaps helps students to set personal goals and strategies to achieve success. The ‘Psychology of Success’ and ‘Personal Career Management’ courses run in the second half of the program and help students to assess their personal professional goals based on the new knowledge and skills obtained.

Another example of the program update was resulted from the achievement assessment of the learning goal ‘Student builds commitment to achieve organizational goals’. As a result, practical tools that help students to build employee commitment in their companies were added to the program. Thus, we also got another point of assessment that should prove the effectiveness of these improvements and enable further program enhancement.

Has the MBA — Challenges in Digital World program been also updated?

Yes. For example, one of the learning goals for the MBA CDW program is the development of students' competence to implement the company strategy in the context of rapid business digitalization. To enhance the skill, we developed and introduced a new discipline – ‘Management in the Context of Rapid Digitalization’. The course is run by a team of business practitioners from one of the largest Russian companies: the business development manager talks about the corporate strategy, strategic goals, methods and decomposition tools, and his team — about the implementation of strategic goals. Also, based on the MBA CDW program measurements we decided that business process management competences need to be developed. The corresponding topics have been added to such disciplines as ‘Operations Management’, ‘Technologies for Building an Intelligent Enterprise’ and ‘Digital Transformation in Business’. 

Olga, what percent of students are involved in measurements and what are the performance targets?

According to the AACSB guidelines, at least 25% of the program students should be considered. However, we often assess the artifacts collected from all students of the year of study. Talking about results, it is impossible to have a 100% success rate in all learning goals, learning objectives, and traits. The Business School sets target levels for them. Thus, we agreed that 75% of the assessed bachelor students must show the results that are not lower than expected (which means that there will be those among them who exceed expectations). The first measurements of some learning objectives have shown even 90%, so we can easily increase the performance target. In some cases, we hardly reach 75%, so this is what we must work with first. For the master and MBA&EMBA programs we set the level of 85% of the sample, and for the doctoral program — 90%.

How often do the AoL measurements take place?

AACSB recommends measuring each learning objective at least once every five years, because the standard bachelor program cycle lasts 3-4 years. Sometimes the measurement takes place in the last year of study, while the root of the problem is in the first year of study. This means that even if we manage to improve something before student enrolment, we will be able to assess the results only 4 years later in student theses. The EMBA&MBA programs have short cycles, so we can take measurements often. It is important to understand that according to the logic and philosophy of continuous improvement, there is no tragedy if we fail to improve something right away. It is a normal process and an impetus for analysis and new actions. 

Do learning objectives and other indicators change over time?

The changes follow the updates of the Business School mission and the introduction of new strategic benchmarks, goals, and objectives. It would be strange to talk about the ESG without adding it to the program. Thus, the corresponding learning objectives that are reflected in certain disciplines, topics and case studies were introduced in the master programs. Today the MBA programs are additionally focused on leadership, so a corresponding set of learning goals and learning objectives has been included there. 

What are the AoL Committee's plans?

We have several key tasks for this year. The first one is to document the key processes of interaction with the programs and the GSOM SPbU Academic Council. The second one is to adjust and, if possible, automate the data analytics system. And the third one is to implement the Assurance of Learning procedures in most programs. We have already agreed upon the learning goals and learning objectives for the Public Administration programs and now we are working on the measurement system. We are also planning to make the first measurements on the joint MBA programs with Kazakhstan and Belarus. Ideally, regular corporate programs should also be assessed. But they are often company customized, so different cohorts of graduates are difficult to compare, so we must learn the international best practices on how to work in this case. It should be mentioned that what we are doing now is called direct measurements, and they are obligatory. However, there are also indirect measurements, that may include employer and alumni surveys as education quality indicators. The updated AACSB standards presented last year pay more attention to this kind of measurement than before. We have not tried that yet, but we surely plan to do it in 2023 or 2024. 

Has the pandemic affected the AoL Committee's work greatly?

Yes. Because of the pandemic we had to cancel a major part of the measurements — uncertainty had a psychological impact on students, that affected the learning outcomes, and the environment changed where students performed their tasks. However, this does not mean that learning results got worse. On the contrary, in 2020, the quality of master theses was higher because in the spring of 2020 our life almost shifted to the online format, including job interviews for future graduates. Students had more time and opportunities to focus on their theses. Also, we had to cancel the measurements of internship results — in 2020, most students had internships in various creative formats (we accepted anything that could be correlated with the internship requirements), but not the one we planned. Because of the pandemic the AACSB accreditation of GSOM SPbU shifted to 2021. Since the spring of 2021 the AoL Committee has already been working, and in the 2021/2022 academic year it has got methodological and organizational support of the Teaching Excellence Lab.