Meta-analysis of Exam-preparation, Certification-preparation or Test-preparation Courses

My current curriculum project for work is to build a course to prepare students for a exam that leads to a professional credential.

It’s not as easy as you would think it would be. I’m finding it strange to do even simple things, like apply Bloom’s Taxonomy to the course’s performance objectives. I’ve decided to take a step back and look at the body of work that already exists for cert.credential-prep.

My approach was to start by looking at technical cert prep courses offered by the big guys (IBM, Oracle, etc).  However, there really wasn’t much out there for technical working professionals (software engineers, in particular).

My next approach was to delve a bit into the world of higher ed cert prep, and I found a lot of good information when I changed my approach.

Below, I’ve documented the key findings of my research.

(WIIFM) What students liked about various cert-prep courses

  • Ability to review materials as many times as needed
  • Knowing what exam topics to focus on
  • Knowing how deep to go on any given exam topic
  • Building confidence and feeling prepared
  • Organizing information related to exam topics
  • Understanding the difficulty level of the exam
  • Accessing the prep materials in remote geographies
  • Experiencing the mock exam in a realistic way

Examples of performance goals that cert-prep courses claimed they achieved

  • NN% of students who took the cert-prep course passed the test
  • Over NN of all successful exam takers used our class to help them pass
  • Candidates who use the exam-prep see their scores improve by an average of NN%

Examples of performance objectives of cert-prep courses

  • Prepare for the exam
  • Gain the tools and skills you need to pass the exam
  • Focus your limited time and resources
  • Focus on the information most likely to appear on the exam
  • Master the exam materials
  • Prepare with a comprehensive practice test
  • Analyze your test readiness
  • Identify areas for further study with topic-focused chapter tests
  • Identify test-taking strategies

Types of mock exams and exam assets bundled with the course

  • Multi-question practice exams
  • Pre-test evaluation
  • Per-module, topical questions
  • Realistic test-taking environment
  • Explanations of why exam question answers are right or wrong
  • Weekly pop quiz
  • Proctored exams

Instructional content formats (i.e. formats of curriculum) offered with the cert-prep course

  • Static print or multi-media content (including books and workbooks)
  • Streaming video
  • Live delivery

Exam aids offered with the cert-prep course

  • Study tips
  • Printed flash cards
  • Online flash cards
  • Audio review flashcards
  • List of key terms
  • Cheat sheets

Social services offered with the cert-prep course

  • Discussion forums
  • Comments from students who have previously taken the exam
  • Email support
  • Phone support
  • Blogs
  • “Live agent” chat

Exam-prep add-on services that were optionally offered with the cert-prep course

  • Custom study strategy
  • Online tutoring
  • On-site tutoring
  • Topic-based refresher
  • Job interview prep

Other assets offered by the cert-prep vendor

  • “Insurance” – gain access to materials until you pass
  • Detailed reports highlighting strengths and weaknesses in each content area
  • Personalized study plan with daily goals
  • Self-directed learning path (choose your adventure style)
  • Instructional e-games

Examples of cert-prep courses and some etc links (you can click these)

Let me know with a “like” or a comment if you found this post useful.


Practical Instructional Design Tips: How to Teach Decision-Making

Game theory is fascinating! It’s the study of achieving the best outcomes of a “game”, like soccer or chess, where more than one person is involved and the theorectical outcomes are known or are generally predictable.

So how does Game Theory apply to Instructional Design? In one sense, the two disciplines don’t really intersect at all. After all, instructional design is a lot of things, but it rarely involves war-time-like strategy. However, if you’ve ever studied game theory, you’ll know that game theory educators have some very interesting instructional techniques to teach people decision making – and decision making can be a part of any type of instructional content, especially when geared toward advanced learners.

Let’s explore a game theory example in which two people are playing a coin-flipping gain. Each player is given a quarter to flip and the rules of the game are: you gain one point if you flip heads.

If you were to write out the outcomes in a slide, you might have the dreaded “bullets of insanity”:

  • Scenario 1: both players flip heads, so both people “win” but tie
  • Scenario 2: both players flip tails, so both people lose, but lose equally
  • Scenario 3: one player flips heads, the other flips tails, so the player who flips heads definitively wins

What I’ve observed is that game theory educators don’t use the “bullets of insanity”. They use a fairly standard outcome matrix. Let’s build one for our scenario, step by step.

First, let’s only consider the outcomes for player 1.

Player 1
Heads 1
Tails 0

Pretty straightforward, eh? If Player 1 flips heads, he gains one point. If not, Player 1 stays at 0.

Now, let’s add the “game” part of this by adding in and comparing the outcomes for the second player. The possible outcomes for Player 1 are noted in the first column; the possible outcomes for Player 2 are noted in the first row; and the actual outcomes are compared in the remaining cells. You read the cell “1, 1” as: “if Player 1 and Player 2 both flip heads, they both achieve 1 point and are tied.” You read the cell “0, 1” as “If Player 1 flips tails and Player 2 flips heads, then only Player 1 achieves 1 point and wins the game.” Notice how easy it is to spot the situations that yield the best and worst outcomes for both players.

Player 2 Heads Player 2 Tails
Player 1 Heads 1, 1 1, 0
Player 1 Tails 0, 1 0, 0

Isn’t that SO much better than the “bullets of insanity”? And it lends itself to eLearning animations as well. You can animate the filling in of each of the outcome boxes (e.g. “1, 1”) with a spoken script overlay.

So, the next time you need to discuss comparative decision making, now you have a bullet-free way to do it, courtesy of game-theory educators. To see more examples, check out this amazing website:

Jyoti’s (Not So) Far-fetched Predictions for the Future of Learning

Ahhhh… It’s just nice to just sit down with a cup of hot chocolate and have some me-time with my thoughts. I’m so excited about all the stuff I’ve been reading about lately through “The Twitter” = as an old friend calls it.

I’ve been in the learning industry for years now, and, like you, giirrrlll, I’ve lived through some hype. OMG: mLearning! OMG: MOOCs! OMG: microlearning! I’m kinda over all the hype, and as a technologist-turned-educator, I think I have a pretty good radar for what new thing is actually going to stick.

  1. AI-enabled learning paths in LMSs: Ever have to sit through a learning path discussion with a marketing team? Tee hee hee! That’s a good use of your happy place, right? Thankfully, as AI gets embedded into LMS design, a side effect can be to have the LMS look for trends and patterns in learning records related to content consumption. If the LMS can identify the same pattern of consumption over and over again, then that becomes a natural learning path. I think we will see human-made and AI-generated learning paths in the future, living side by side, in LMSs. I also think the most accurate AI-generated learning paths will come from LMSs attached to professional networking sites, such as LinkedIn. I can see a future where LinkedIn might even sell back to companies their AI-generated insights into corporate learning paths.
  2. 3D-VR Instructors: I’m boldly going to predict that learning site vendors, similar to Microtek, will begin to invest in 3D-VR-enabled technologies. Can you imagine being one instructor and teaching two classrooms in a live-ish manner? The “ish” part comes from the fact that you’ll be live in one classroom, while a live, 3D holographic projection of you is teaching in another classroom. I think there are so, so, so many benefits of using a live-but-holographic instructor over a traditional virtual classroom. First of all, its just more personal and engaging. I’d rather spend 8 hours a day being entertained by an animated hologram rather than staring at a Powerpoint deck projected from half-way around the world.
  3. Multi-lingual Speech-to-text and Text-to-speech in LMSs: I think the day is coming where we won’t need to create content in a bunch of different languages. We just need to produce content in our native language, and the LMS will translate the content on-demand, based on the audience’s preferences. I believe that you will have the option of written and/or spoken on-demand translation.

A 5-Point Framework for Evaluating Instructional Intent in a Slide

I’m mostly doing this for me and as an analysis of what I do for my job. I started with the idea: how do I know an instructional slide is a useful one? Copy/graphic issues aside, my primary concern is whether there’s any value to creating the slide in the first place. What purpose does the slide serve in the overall context of the training? What purpose does the slide serve on its own? I call this the “instructional intent”; i.e. what is the instructional intention of this slide.

So, I constructed this rubric/framework to judge instructional intent in an individual slide. I’d say that a slide that evaluates to less than an 7 should be workshopped.

Topic 2 1 0
Aligns learning objectives Information presented has is directly aligned with learning objectives Information presented is indirectly related to learning objectives Information presented has nothing to do with learning objectives
Helps meet stated performance objectives Information presented is directly related to perfomance objectives Information presented is indirectly related to perfomance objectives Information presented has nothing to do with performance objectives
Theory-into-practice Information is presented in a manner that makes effective use of an adult ed theory Information is presented in a manner that is consistent with an adult ed theory Information is presented in a manner not consistent with adult ed theory
Conveys only accurate information All information has documented, credible sources and has been SME reviewed All information has documented, credible sources Any piece of information presented is inaccurate
Scaffolds information correctly Information presented is the logical, direct progression of the slide before it Information presented is indirectly related to the slide before it Information presented has nothing to do with the slide before it

13 Types of Educational Conference Sessions

Name (clickable) Summary Max Duration (mm)
Ignite / Pecha Kucha Very fast presentations 5
Expert lecture Regular lecture 45
Round table Lecture + discussion for a small audience 45
Birds of a Feather Small, informal, discussions on a topic 60
Demonstration Just a demo 90
Panel A group of Knowledgeable Others discuss topics while others listen in 90
Skill builder Hands on training 90
Think Tank Single topic discussed in discrete breakout groups 90
Professional Development workshop Longer hands on training 720
Unconference: Dotmocracy Audience suggests, votes on and delivers topics Unlimited
Unconference: Fishbowl Hop in, hop out panel discussion Unlimited
Unconference: Knowledge Cafe An informal round table Unlimited
Unconference: Speed Geeking Inside ring presents, outside ring listens; bell rings, the outside ring moves one seat. Unlimited

5 Gifts For the Instructional Designer in Your Life

I was in PetSmart today. They had red, green, and white “lollipops” made of rawhide for the puppies, and on discount, no less – 25% off. It’s only October, people. The jingle bells music can wait.

That shopping trip got me thinking about my own “Instructional Design” wish list for the season. Here are a few of my picks.

What is it (linked) Why it’s cool Approximate Cost, USD
The StoryHow PitchDeck This is a deck of sixty (60) playing cards helps you find, develop, and tell business stories. Use it to transform messages, ideas, and presentations into compelling narratives. $50
USB Monitor Extender USB! More screen real estate! YAAASSSSS! Have you ever tried to build a Powerpoint slide with the animation pane and the selection tool both up? It leaves no room for the slide. You need all the screen real estate you can get. You might think: why not just get a bigger screen instead? Answer: Portability. If you need to have more screen real estate in multiple physical environments (work, home, etc), this USB monitor thingy might be the better solution. $180
Portrait to Landscape Monitor So, if you don’t care about portability, get a big, rotating screen. With so many of us doing work for mobile devices, it’s useful to have a portrait view to look at your mobile designs. But it’s also good to have landscape mode so you can get the most out of ID software, such as Powerpoint $170
Communication Arts Magazine A well-established and beautiful magazine about visual communication. 6 issues/year. $53 for print
Cognitoy You know how I feel about this. The intersection of NLP, AI, and personalized learning? Every ID’er needs to experience this to see if/how it works and how it can be applied for workplace learning. $120

Updates (more gifts I found useful after the Original Post).

“Instructional Design Funnies”: Peeling an Orange is a Negotiation Tactic (Social Learning Theory)

In this episode of 30 Rock “It’s Never Too Late for Now”, Alec Baldwin plays Jack, an ever-scheming corporate executive. His protege and confidante is Tina Fey, the producer of a variety show owned by the same company Jack works for. Jack needs to negotiate higher licensing fees for NBC but can’t figure out what his leverage is for the negotiation. Through an unplanned series of role-playing and reflection (social learning theory), he figures out his leverage and how peeling an orange is a priceless negotiation tactic. Adriane Lenox plays the uncompromising nanny.