Wednesday, December 24, 2008

Is student assessment for accountability purposes educationally beneficial?

What do students and teacher learn from large-scale assessments that are given for accountability purposes? Do teachers find out what specific learning difficulties their students are having? Do students learn where they stand in relation to a curricular target or goal? In general, I'd say the answers to these three questions are: not much, no, and no. These thoughts come to mind as I've been reviewing some physics questions intended for use as preparation for some of these large-scale assessments. 

Unfortunately, the target or goal of a large-scale assessment often seems to be a cut point score or a designation of "proficient." Even though "disaggregation of the data" is often performed by teachers and schools to see where their students need remediation, it seems as though the driving force for this is the "percent proficient" statistic, rather than students' reaching specific curricular goals. 

But it doesn't have to be this way. I believe classroom assessment practices that are developed, implemented, and carried out by the teacher have the potential to be used for accountability purposes. If teachers become "assessment professionals" and emphasize the importance and interdependence of curriculum, instruction ,and assessment in teaching and learning, then their own assessments could be used for accountability. We've seen a few examples of this in the research literature and work from Australia and the United Kingdom, but what would it take to implement this on a large scale in the United States? Nothing less than a cultural shift in our beliefs about assessment and accountability I suppose, not to mention ensuring that "highly qualified" teachers (forgive the use of a poorly operationalized term here) exist in all of our schools. 

What are your thoughts?

Monday, December 22, 2008

Web-based survey software and Google docs

For a few years now, I've been using QuestionPro to create, store, and administer online surveys. I started with the free version, but upgraded to the "web professional" ($15/mo) version when our research group needed to house and administer multiple surveys. We currently have about 20 surveys posted (not all active) and they are pretty easy to work with. Downloading the data is pretty easy, though it does require a bit of cleaning before you can do anything serious with it (e.g. analysis, loading into a database, etc). 

I've also used Survey Monkey and Zoomerang in the past. Some folks in our building currently use Zoomerang. I think they're all pretty similar in that they each have a free version (usually limited to one survey and/or a limited time or number of respondents) and various levels of paid services, in which you can have the ability to house multiple surveys or use built-in analysis tools.

I recently discovered the ability to create surveys in Google Docs using forms created from spreadsheets. If you haven't checked this out, see the Google Docs help section on forms. It's very cool because it is so simple- almost elegant. You can create a form based on an existing spreadsheet, or from scratch. A variety of question formats can be added to the forms, such as multiple choice, open-response, rank-order, pick from list, etc. The form can then be embedded into a web page, sent in an email, etc. The really cool thing is that as people submit completed forms, the data is written into your associated spreadsheet in Google Docs. 

One limitation I've discovered so far is the inability to build a form with branching logic. Often in a survey, you would like for certain respondents to skip questions or get extra questions based on a previous response. If it wasn't for this limitation, I'd probably move a lot of my research surveys over to Google Docs right away. Perhaps this feature will be available in the future. Another limitation is the inability to easily label scale choices in Likert-type items (e.g. "on a scale of 1-5")- you can only label the endpoints with descriptions.

Jason Morrison has a great blog post about using spreadsheets and forms in Google docs to create a survey for your blog.

So, in keeping with the theme of this post, here's a short survey. Notice that a few of the items are required, and notice the potential need for the two things noted above as limitations: branching logic and Likert scale descriptions. Anyway, I'd appreciate you feedback so enjoy the survey. 

Sunday, December 21, 2008

Reviewing manuscripts for academic journals

I'm certainly not an expert at reviewing papers for publication in academic journals, but I've done it a few times and have another one on my desk to review now. It's a pretty time intensive task. For me, I usually have to read the paper 3 times:
  1. To figure out the gist of the whole thing and to try and pick out the structure, methods, evidence, claims, etc
  2. In order to make specific notes on the above stuff so you can comment to the authors
  3. One last time to look for the details and comment on them- things like grammar, reference consistency, etc. I do a lot of this stuff in the previous reads, but I always have to do one more focused just on these tasks. 
Once I've done this and made my notes, then I have to write them up in a coherent fashion (beginning with an outline) that will be interpretable and useful for the authors and journal editors. Then I make a decision to report to the editors (accept, revise and re-submit, reject, etc) and present it in a separate cover letter. 

Like I said- I'm not an expert at this, but it is a useful and educational service to perform. Sometimes doing this makes me feel good about my own writing, and other times it really humbles me. It also helps me keep in touch with what others are doing in a different way than just reading the final version in publication.  

Now back to the one on my desk, which I've only read through once so far.

Saturday, December 20, 2008

Some of my recent diigo bookmarks and tags

Posted from Diigo. The rest of my favorite links are here.

Where I'm active on the web

Here is a list of other places where I'm active on the web:

  • Twitter. I've recently been pretty active on twitter for both professional and hobby use. It's been quite a learning experience, and I think the potential of this tool for pedagogical purposes is only just being realized. I use twitterberry from my Blackberry and Tweet Deck (very cool) from the desktop.
  • The Science Education Research ning. I started this ning a couple of weeks ago and posted recruiting announcements to the NARST and ASTE listservs. So far, there are about 62 members and a couple of managers of this ning. Hopefully the discussion and groups will take off soon.  
  • Diigo. I've also started using diigo to grab and tag webpages, again for both professional and bobby use. As I continue to use diigo and add stuff, I'll be making some of the things shared or public.
  • Facebook. Well, I guess everyone has been on Facebook for a while. I have to admit that although I've had an account for 4.5 years, I'm only now beginning to realize it's utility. I'm interested in the groups feature as a way to get some more "constrained" use out of it- I find it a bit too much of a free for all and get tired of wading through all of the crap on my main page.
  • Classroom 2.0. Just joined this one- we'll see what transpires here.
  • Google. I am deeply into Google and its apps, including Google Docs, Reader, Calendar, Gadgets, and, or course, gmail. It's amazing how easy it was to break free from MS Outlook and, to some degree, MS Office. And integration of gmail, contacts and calendar with my Blackberry was a snap. You can email me at robert.m.talbot @ 

As the list grows or become refined, I'll update this post. 


Seeing as this is my first post to this new blog, I suppose I'd better present a bit of background and rationale for creating this blog. I am a science education doctoral candidate at the University of Colorado at Boulder, and a former high school science teacher. I taught mostly physics for 7 years in rural Indiana and Suburban Houston, TX. My interests while teaching were developing new strategies using appropriate technology, exploring students' thinking about the science topics I taught, and student assessment. Since leaving the classroom 4.5 years ago, I have become more interested in assessment of students' thinking and in educational measurement. I am also active in science teacher education. My dissertation work focuses specifically on developing an instrument to measure science teachers' strategic knowledge and knowledge of students' understanding, which I see as a part of their PCK (Shulman, 1986). Until I link things up, you can read more about this work on my webpage.

So why did I set up this blog space? Very recently, I've reconnected with my interest in educational technologies and I've been exploring potential uses of social networking in the K-12 science classroom and the science teacher education environment. I've set up this blog, therefore, as a place to record and present my thoughts along these veins:

  1. Web 2.0 stuff and social networking in K-12 and higher ed
  2. Student assessment in science
  3. My dissertation research in science teacher knowledge
  4. Whatever else pops into my head

So when I get my act together, I also hope to make this space a sort of clearing house where I can post links to my twitter activity, diigo bookmarks, nings, facebook, etc, as well as my musings on the above topics.

More to come...