Making Hard Choices - UXPA 2014

By: 
Danielle Cooley
Blog
March, 2014
Abstract: 

Just how do we select which 25% of our amazing submissions to include in the program for the annual conference? Deliberately and with great care. When so many submissions meet our strict quality standards, we must evaluate each one carefully to produce an exciting program that includes a broad mixture of relevant topics that are both current and meaningful to attendees. 

Full Article: 

We received 360 submissions for the UXPA conference in London this year. That’s a lot of submissions! In fact, far more than for any of the past 22 UPA/UXPA annual conferences. (2013’s Washington, D.C. conference ranks #2 in this area, with 316 submissions.)

 

Each one of these submissions went through a blind peer-review process, with at least 4 of our fellow UX professionals weighing in on the submission’s quality, importance to the field, and specific relevance to practitioners, among other attributes. Those reviewers found many – probably even a majority – of this year’s submissions to be high-quality submissions on important topics that they’d like to see and hear more about.

 

However, our four-day conference schedule accommodates only approximately 25% of those 360 submissions. That means we need to make some hard choices. To make sure those decisions are made fairly, and without any detriment to the event itself, we are guided by several principles:

 

Blind peer review. We rely on the recommendations of the reviewers, the scores they give, and their comments about a proposal. This is hard to do after the fact when we see the names and say, “Oh! That’s so-and-so! They are AWESOME! I don’t want to reject them.” But for blind peer review to work, we can’t let names/reputations cloud our judgment. We have to trust. And remember that UXPA has always been about good, practical talks that will help people with their career/work/job. Really interesting submissions focused on theory or in niche categories may not be best-suited for a practitioner conference.

 

Creating a fresh and balanced program. This is where the conference committee has to make some hard choices and where many of the most confusing rejections come in to play. If you think for a moment about all of the different things worthy of coverage at this event, I suspect that you will – as we did - quickly end up with a list of more than 30 items. (Eye tracking. Personas. Visual design. New techniques. Ethics. Prototyping. Accessibility. Agile. The list goes on and on.) Taking again our 91 accepted submissions, that leaves us fewer than 3 slots for each topic. Some highly-rated submissions proposed nearly identical content to an even more highly-rated submission. There’s sometimes simply too much of a good thing. If we accept five or six highly-rated submissions on Agile, for example, that can mean that another topic gets little or no representation at all. We wanted and needed to include a broad mixture of relevant topics that were both current and meaningful to attendees. We often opted for fresh voices on new, emerging topics over those that have already been well-covered.

 

Speaker variety. If a single submitter has 3 or 4 (or 5) submissions near the top of the pack, we might not put them all in the program. First, that’s not fair to the presenter – giving presentations is hard work and very draining. I don’t care what anyone says: no one wants to present four talks in three days. But most importantly, it’s not fair to the attendees - we find the quality of the later talks can generally suffer as the presenter becomes increasingly exhausted, and everyone benefits from having as many perspectives as possible. So you may see a few names twice in the program: they earned it. They submitted multiple great proposals, often on different topics and in different formats. But you won’t see anyone three times.

 

Practicality. Mathematically, this issue of too few slots for too many great submissions could be resolved by creating an 8-day conference. Or having 10 sessions per time slot instead of 5. But we don’t want to run a ten track conference that comes with a doubled registration fee. We also don’t want to create a monstrous event where you are unlikely to meet many of the people at the conference or in which you miss 95% of the talks because you can’t physically attend everything. We had to make hard choices, and so do attendees, even with only 5 concurrent sessions from which to choose. And choosing which session to attend should be hard because they should all be good. In the end, all of this meant declining some very highly-rated submissions. The highest-rated submission we didn’t accept was rated 4.5 (out of 5) by its reviewers. The lowest-rated accepted submission was a 2.9 (and I can assure you that 2.9 has something really special to offer that put it over the top). Although the accepted submissions clearly skew higher than the rejected ones (average rating 3.92 vs. 3.45), all of the aforementioned factors come into play when making final decisions.

 

So that’s how you ended up with your program. As you peruse the schedule, know that it was put together with love, sweat, and the hopes and dreams of the entire conference committee. The full program will be posted shortly, and online registration will open in early April. Our early bird rates are fantastic, but they’re limited to the first 100 registrants, so don’t delay!

 

See you in London!

Danielle, Sara, & Stavros

UXPA 2014 Conference Co-Chairs

(with special thanks to 2012/2013 Co-chair Christina York, whose blog post about the 2013 program this is based on or flat-out stolen from.)