[sandy]

How do you self-evaluate your success or failure as a designer? Are these the same ways that the person you’re accountable to measures your success?

How are you, as a design volunteer held accountable, and to whom?

How and to whom is the head of design held accountable to?

My direct report on a set would be the team lead on the set, if I am not that person. Our team reports to the Design Director. He reports to the 2eD. We're all held accountable to one another, to the Board, and to the players.

I hold myself accountable in the following ways. I look back at past work constantly. I see how many cards from sets I have been involved in are making it into play. What kind of decks they are showing up in. How many aren't being played at all. I look to see if people seem to be enjoying the cards. When I travel or host events, I look to see if cards I had a hand in are making it into decks in different metas. I look to see how many cards I had a hand in have needed errata, and if that errata was tied to feedback I had access to but didn't heed. I self-evaluate by thinking about cards I consider design successes and cards I consider design failures and trying to figure out why each is what it is. I try to gauge how much buzz is generated when a set I help produce is spoiled or released. And, yes, I try to figure out how well a particular set I was involved with accomplished its goals so far.

That’s great to hear - is this a consistent standard for design? Based on your other answers, I think not. It’s unfortunate that these aren’t quantified and transparent so that the 2e director and design director can objectively evaluate sets on a common baseline instead of relying on subjective criteria.

I can't speak for how Nathan (or John before him) or Nick (or Ross before him) hold me accountable because none of them ever had a conversation about formal metrics with me. As you'll probably figure out from reading my answers, I am not sure set success is quantifiable in a formal way. I tend to think it's a more qualitative measure - though I understand that's debatable. But, if I had to hazard an educated guess based on five years and nine sets of experience as a volunteer ( six sets produced, and three in production) I would say Nathan, John, Nick, and Ross have all held me accountable by the same kinds of standards I've listed above.

I’m a little confused by this - if they don’t have any conversation about metrics, what are they holding you accountable to, exactly?

Players aren’t happy with the set. That’s the only metric I have visibility into. There’s been cases where playtesters didn’t feel listened to, and that seems like a red flag.

Frankly, it's too early to tell whether "players" are happy with this set. It's also important to take notice of which players seem the most "unhappy" right now. There has been a lot of politics in 2e this year - probably more than I've ever seen. There is a very vocal group of players - and a couple of ex-volunteers among them - who seem determined to not be happy and to have made it their crusade to attack the current design roster and production pipeline for their own reasons. They've managed to muddy the water an awful lot with this set. But, even if time proves this set to have been, on balance, a failure, one failure does not by itself indicate a systemic problem.

This is the problem with not having quantitative metrics. You can say this, and I can say “well, that’s not my experience” and we can go back and forth. What reasons would someone have to attack the current design roster and production pipeline? Or more precisely, why do you suspect that’s the case rather than the simplest explanation (that they’re unhappy with the cards)? With quantitative metrics, you could easily refute any claim I had that this set was bad by pointing to how playtesters rated each card, the number of decks created in playtesting that players felt were competitive, etc.

As far as testers not being listened to ... all I can say is that I am on the testing forums and I haven't seen evidence of that to any significant degree. Not all testing data gets incorporated into a set, but that doesn't mean a design team is ignoring that feedback. For example, if I am designing on a set and one tester group says they love card X and give me good reasons why and another group says they hate card X and give me good reasons why, I can't please both of those groups - one of them isn't going to get what they want. That's the way it goes sometimes. As a designer, I can tell you this happens a lot more than one would think. That's because testers, like any other players, have their own preferences and comfort zones. Some are, by nature, very conservative when testing cards and don't like to have their metas challenged (John used to call it "smothering babes in cribs"). Others are more liberal and really good at setting aside their own preferences or are more willing to tolerate big risks. Sometimes the conservative testers are right about a card. Sometimes the liberal testers are. Sometimes they're both right about different things for different reasons. Designers have to try as best they can to sift through the data they get and do the best they can do. When I design on a set, I make it my highest priority to make sure testers never feel ignored, and I've been told I do a pretty good job of that. Could other designers do better? Maybe. But it's not for lack of effort.

Again, you’re making a great case for quantitative metrics to justify your choices. Otherwise it just becomes a circular argument about who said what. If you read the paper I linked about measuring customer satisfaction, you’ll see that purely qualitative measurements often skew negative.

Do you want me to come up with an entire rubric for scoring sets? I would desperately hope that there is already one which you’re already using - maybe it would be more productive to start from that as a baseline and see where we disagree.

To answer your first question, and not to try to sound blunt, but yes. If you're advocating for more accountability, I am going to assume you're doing so from an educated perspective and that you already know where you think the specific shortfalls are and how you'd address them.

Why bother having a design director then? The responsibility for synthesizing a rubric should lie with the person who (presumably) has the most context for evaluating success, which in this case should be the design director. As 2e director, I’d require all departments to do this, not just design.

To answer your second question: if I am honest I'll have to say there's no written, quantitative metric. But, I think we all understand the qualitative ways in which we're supposed to be accountable. Are players, on balance, happy? Are players using the cards? Are the cards doing what we intended them to do? Are players, on balance, having fun? Are we working well together? Are we innovating? Are we taking calculated risks? Are those risks paying off? Are we challenging players? Are we setting goals and meeting those goals? Are we having fun? If the answer to most or all of those questions is yes more often than not, then I think we're doing okay. There will always be "good" sets and "bad" sets under any system or any 2eD. But, are we hitting our marks more often than missing them? If so, I think we're okay. And, I think right now (as in over the last 12-16 months, say) we are.

So you don’t have quantitative metrics, but you’re going with your gut feel on whether you’re hitting your marks? This is exactly what I want to avoid - it’s confirmation bias of the worst kind. I reject the notion that “we all understand the qualitative ways in which we’re supposed to be accountable” All of the things you mention are quantifiable and measurable. Going by how you feel is a recipe for misunderstanding and negative feedback loops.

What’s the current process? I’m happy to evaluate the current retrospective process and accountability and see whether anything needs to change.
One benefit of having clear accountability for each department is that it flows downward. It lets us identify what we can change in specific areas to execute better.

I am not aware of a formal quantitative process, and frankly I am not sure 2e production is amenable to one. So, let's take a card I designed that certain people love to beat me up with anytime I open my mouth about design theory, The Enemy of My Enemy. That card caused a lot of problems. But, I got no testing data during the production of Far Beyond the Stars that indicated such a problem. Years later, I learned that vital feedback indicating major concern about the card was withheld from the design team, and that includes me. So, who is responsible for the problems that card caused? The person who withheld the feedback? Other testers for not catching the problem? Me for pitching the card? The rest of the design team for working with me on the card? The Design Director who let it happen? The Errata Team for taking a really long time to address the problem? All of us?

I think lack of a quantitative process is exactly the problem. I also don’t think there’s a need to assign blame for any misses. Professionally, I am a big proponent of blame-free retrospectives (https://opensource.com/article/19/4/psy ... rospective) and I think a similar model would be a good fit here.

What does the selection committee feel are the ideal qualifications for a 2e director to do the things you mentioned (which all fall under the mandate of the 2e directorship)?

We're not a monolith. There's no objective set of criteria we're judging candidates by. But, I am happy to tell you - and all the candidates - mine. These are only mine. I am not, in any way, speaking for any other member of the committee. None of these is necessarily disqualifying on its own, and I am open to being wowed in an interview by someone who doesn't even meet most or any of them - but in the case of the latter I'd have to be wowed for real.

That’s totally reasonable. I do think it’s interesting that you didn’t ask for this information up front, maybe that would have framed our conversations more around things you were looking for rather than minutia of management that may be uninteresting to other selection committee members. On the other hand, I’d rather have this conversation via posts rather than in an interview where it’d be even more boring

• I look for a history of play;
• I look for a history of forum involvement;
• I look for a history of involvement in some aspect of production, preferably as a volunteer;
• I look for someone who has experience running events or leading a play group;
• I look for someone who travels outside their own meta, or plays online a lot, and therefore gets perspectives from players outside their own circle;
• I look for someone who has a history of getting along reasonably well with other players and volunteers;
• I ask myself if I think I would want to work with or for that person;
• I look for someone with management or leadership experience;
• I ask myself if I think they genuinely want what is best for the game;
• I use many of the above to try to figure out if they have what it takes to make what they want happen;
• I ask myself what kind of leader I think they'll be; and, finally,
• I ask myself whether or not I think they'll be around a year from now.

I thank you for the answers you provided. I hope these answers o your questions have helped you as well. I look forward to continued dialogue, and ultimately to your interview.

• History of Play - I played 2e at release, and played through a majority of the decipher era. I left to play Game of Thrones, and played that until it was also discontinued. I think I have both the context of the original game, and bring some fresh new ideas from other card games based on popular properties.
• Involvement in production - as I have never worked for a gaming company, and I am not a game designer, I’m not sure what aspects I could possibly be involved in. I was a playtester for Game of Thrones, and I’ve playtested for other games too, as a significant percentage of my friends are game designers
• Running events or leading a play group - in Trek, I wouldn’t say that I ran a playgroup or ran events, but I was pretty involved in planning and running games in college. In Game of Thrones, I was part of the committee (of 5 people) that put together the Northeast Championship Circuit, events that spanned 7 months and culminated in a finale in NYC. Attendance of these events varied between 30-50 each in the years I was involved. I was also involved (although not nearly as deeply) in the similar circuit for Europe, which spanned 8 countries and culminated with a 350+ tournament in Germany. These events were all “unsanctioned” and entirely player run, with minimal support from Fantasy Flight. I also helped plan and run an event in the UK that drew >120 players from 10 countries, also unsupported by fantasy flight. I hope I can carry on that tradition of running big events for Trek.
• Travelling a lot - When I played trek, I consistently travelled to regionals. One summer I think I hit 5 or 6 regionals to get NE promos, lol. When I played game of thrones, I had considerably more discretionary income and was able to travel more. Currently, despite not playing since February, I have the record for most unique opponents at tournaments (approximately 40% higher than the next highest person). I am still the only person to win tournaments on three continents, and I think I’m tied for most countries played in (I definitely have the most states played in). Even since I came back to Trek, I’ve played in three states, and met people in yet another. I’ve talked trek in London, Charlotte, Vegas, Texas, DC... That’s in under a year.
• Getting along well - This is obviously subjective, but even now, most of my social interactions are with CCG friends.
• Mangaement and Leadership Experience - see my list of personal qualifications in a previous post.
• If I’ll be around in a year - I told Armus when I first started playing that I would continue to play as long as decipher made cards (this was in like 2002). I’m happy to transfer that to the CC as long as the community around the game is growing and vibrant.

The rest of your questions are pretty subjective, but wanted to give you actual data relevant to your qualifications.

[sandy]

For full transparency, the goaling and measurement model I've used for the last 7 or so years professionally is called OKRs, and it's been implemented in various companies.

A friend of mine (and fellow ccg player!) wrote an ebook that I think explains it pretty well: https://www.amazon.com/dp/B06X963HD7/re ... tFb3QSBNGN

If there's interest, I can talk to him about giving a talk or workshop on concretely applying this to our processes.

I would like to read it, and I'd be interested. Once I know more, I'd like to know why you think it's a good fit for this organization specifically. Thanks for sharing!

EDIT: Reading the intro, it could be we're already doing some of this and we just have never given it a formal name.... interesting.

Very interesting. As I make my way through this, so far if I had to boil it down to the cocktail party summary, it might be something like: our organization has to set measurable, achievable goals that have a direct relationship to our organization's overall vision and mission.

If that's basically right, I think it would be hard to argue that we haven't been doing that, at least since Phase II anyway. Note: when I refer to "flavor" I mean an affiliation or faction's distinct identity, what it does (or how it does it) better than other factions or affiliations...

• Unnatural Selection: 2013; Make Khan decks a thing.
• Return to Grace: 2014; Enhance Bajoran and Cardassian flavors
• Balance of Terror: 2014; Give Terok Nor flavor, create Bluegills for TNG
• 20th Anniversary Collection: 2014; Honor Star Trek CCG 20th Anniversary; make us all feel old
• Strange Bedfellows: 2015; Enhance Dominion and DS9 flavors
• Lower Decks: 2015; Enhance Borg and Voyager flavors
• Face of the Enemy: 2015; Enhance Romulan and TNG-sharing flavors
• A Time to Stand: 2016; Enhance Maquis and Starfleet flavors
• Infinite Diversity Remastered: 2016; Improve 2e Sealed deck/draft play
• Hard Time: 2016; Enhance Smugglers and Thiefs, round out Phase II affiliations
• Star Trek 50: 2016; Honor the 50th Anniversary of Trek, give a little something to everybody
• Zero Hour: 2017; Conclude formal Phase II, encourage battle and combat
• The Nth Degree: 2017; push design limits, make equipment matter more
• Dead Stop: 2017; Halloween boutique
• Symbiosis: 2018; Encourage dual HQ play, "division" theme
• Far Beyond the Stars: 2018; Celebrate DS9 25th anniversary
• Warp Pack: Access Denied: 2018; Enhance control ability for "underused" factions or deck builds
• Shattered Mirror: 2018; Introduce KCA
• Qpid: 2019; Valentines Boutique
• A Less Perfect Union: 2019; Introduce Starfleet Dissidents
• Sacrifice of Angels: 2019; make battle/combat matter more
• Inheritance: 2019; nostalgia - make 2e punch-ups of 1e cards, make us feel old again
• The Omega Directive: 2020; 50th 2e Set, highlight 50 point missions
• All Our Yesterdays: 2020; encourage interaction (in different forms) through Enterprise/TOS-era stories

*Sets I personally worked on as a designer are in bold; as a creative and art volunteer, I have worked on every set since the 50th Anniversary set; as a writing and testing volunteer I have worked on every set since Nth degree. I've been an active tournament director (TD) and ambassador since Return to Grace.

Now, it's certainly possible that not everyone agrees that all of these were the measurable, achievable goals design needed to focus on right then. To be sure, some of these sets were more well-received than others. And, it's an open question whether each of these were in keeping with the CCG's overall mission of keeping Star Trek CCG 2e alive, fun, and engaging. Some of these sets were loved in some metas and loathed in others. However, I think it would be impossible to argue that 2e production staff, including design, weren't setting goals and trying to meet them, and very hard to argue that designers in each of these sets weren't making their best good-faith effort to give players fun and exciting cards.

So, I guess my question is, what is this OKRs approach supposed to add to 2e that we aren't already doing? Or, alternatively, if it's simply a new way of doing what we've already been doing (and are doing now), why is the OKRs approach better for the CC specifically?

What are the key results which each of those, and how did you measure them? I think objectives are fairly clear in mapping to goals, but the KR part is the meat of the process. Without key results, how can you tell if you met any of your objectives?

For context, professionally, my teams get objectives from above (similar to e.g. Celebrate DS9's 25th anniversary), then we break that down into team-level objectives (e.g. "Have an article series about DS9 subfactions", "Release a set raising the profile of DS9").
The important thing that seems to not be concretely defined above is what are the key results? Keep in mind "releasing a set" doesn't count as a key result, except as maybe a trivial one. Something like "Ds9/Dominion/Ferengi decks see XX% increased usage" or "XXX personnel from DS9 episodes are used YY% more frequently".

I always guide my teams to be ambitious with their key results - you ideally want to hit about 80% of them, so that you know you set appropriately difficult goals.

I'm curious what you see as the key results tied to the objectives of a given set above are, and how you confirmed your thoughts on whether the set was successful or not.

[The Prefect]

Without key results, how can you tell if you met any of your objectives?

I'm curious what you see as the key results tied to the objectives of a given set above are, and how you confirmed your thoughts on whether the set was successful or not.

I appreciate your above answers.

I would submit that lots of firms all over the world doing lots of different things have been able to measure outcomes before the concept of OKRs came along. Maybe OKRs is a better way to do it in some cases. Maybe it will be a good fit for the CC. Then again, maybe not. I look forward to learning more.

I already laid out above how we go about trying to measure success. I'm not sure what you mean by having results confirmed. Statistical measures can "lie" all the time. It takes expertise to interpret them properly. You may disagree with this or not, but I think the multi-year experience I've had as a volunteer in multiple departments, running events, playing in events, and actively reading and participating in this forum gives me some insight into how to interpret what I am seeing. But, I certainly am open to the possibility that outcomes could be measured differently or better, though experience has taught me to have a bit of healthy skepticism of one size fits all, it worked for us so it can work for you too, types of pitches. Time and access to more information will tell.

In any case, thanks again.

[The Guardian]

Since you claim your methods are time-tested, can you clarify exactly what you think have been major results or metrics that design has improved that make the current process so robust? You say you have no patience for people who can’t see more than one path to a result, but what result exactly do you think design has had under the current leadership? In other words, to borrow your example, if I were to “test your students”, what would they have learned?

I'll be honest: I don't really know what you're asking here. As for "my methods", that's the point, actually. They're not mine. They've been passed down to me from those that came before me and I share them equally with my whole team and Design Director, Nathan. You know who threw them out? John. I've definitely felt the pressure of trying to re-establish the norms that worked and have had to explain multiple times why they were in place in the first place. It's infected the culture, which is too bad. But we're on our way.

How do you feel you’ve elevated the game with recent and upcoming sets? Given the reaction to sets that just released, I’m curious to know what metrics you use to self-evaluate, because by the one metric I have access to (namely, people liking it) it doesn’t look good.

I'd step back and watch it's effect on the game afterwards. We had predictable opposition before we even launched from sources that I'm still amazed people listen to. Years ago, Face of the Enemy was derided as providing nothing of value for the Romulans and I ended up winning Nationals with it a couple years ago after fine-tuning a deck based on the new headquarters. It could have happened sooner, if shit-stirrers would just stew in private and not poison the community against it. Good stuff gets ridiculed away every set. I love our set. I think it's going to do what I was asked to do with it.

I mostly agree with you here, but let me ask the same question - what makes you think you’re qualified to judge what’s best for the game? And, why, if that’s the case, are you not running for 2e director?

I'm a member of this community and have some of the most extensive experience with shepherding new concepts and experiences for the players into reality. I have real, actionable experience stretching back over a decade. I believe that was before anyone else currently on the Design staff (assuming Charlie doesn't count), including Nathan, whose wisdom I adore. I took off time when I wouldn't work for, in my estimation, the worst person possible with which to collaborate. And that's before noting he didn't earn it. In the meantime, I've built a local player group. I continued to play that entire time, tacitly noting the errors I saw, instead of shoving them in people's faces. I have reasons and receipts.

I read your example. My position is clear on that from my multiple statements about my understanding of the job in which you have expressed interest. But if you must know, I might push back on wanting to do something else or warning against the logistics (both only if I had a problem with it, which I don't). I don't mind an assignment. A project. If I couldn't handle it, I would just tell you to find someone else, please. Also, that would never happen. You're thinking of someone like John, who would pitch a fit if Design were instructed to go in a direction he didn't like. Thank god we don't have anyone like that any more. Why would anyone want to associate with someone like that?

I have all the professional validation I need from my real job.

Maybe I should say this a new way: I'm not interested in being treated like an employee. My impression, and feel free to correct, is that you would like to translate your experience being a boss into being our boss. If not, please explain. I already don't particularly like being willfully misunderstood over and over again. If that isn't your intent, okay, but it feels that way. You know the first thing we learned about leadership at the Academy? To be "assertive followers." To listen. After all, a good leader needs to listen to the concerns of the team. I'll be honest, and again, if this isn't accurate, it's just my impression, but you don't seem to be listening, sandy.

First, I don’t think I used the word “impose” - I think that was something The Prefect implied from my posts.

From the top of my post:

As an aside, I do think Richard was disputing that I'd have the authority to impose my vision, not that it's especially relevant.

I want to make it clear my usage of “my vision” and “my authority” is a direct reflection of the fact that at the end of the day I’m taking responsibility for the outcome - the state of 2e is something that is being entrusted to *me*. As Armus put it, the buck stops with me. I am not going to make excuses for failure by blaming teams under me, and that also means I’m going to hold teams accountable.

I know exactly where blame goes. I know how that works. Do you see people blaming Ross or Nick for imagined plights? You guys can keep repeating that talking point, but it doesn't mean anything. You were more than happy to put the seemingly negative reaction to All Our Yesterdays at my feet just a second ago, so maybe not the best point for you, sandy. Sorry. How about you pledge to shield the designers from attacks? To support hard work and block out those that want to burn everything down? No one sets out to make a bad set. No one. Hell, I know John wanted to make stuff that was great for the game, as he saw it. I don't begrudge him any of that. I knew he wasn't going to be the end of the game. We have a playtesting process and errata. Any damage can be repaired. The sets even have a bunch of good stuff in them. I take issue with his attitude. He can't take any with mine, so I hear he still rants about how Design is broken.

When the designer Q&A has a total of 10 posts, compared to nearly 100 for this thread, I think that’s symptomatic of something.

Yeah, people responding to the article posts instead. You should count those. Same thing. There was lots of engagement.

More people should care about the cards, not on these political positions. One of my goals will be to put the focus back on the game and not on the internal politics, because the game is what draws people, and politics is what drives people away. What this translates to for volunteers is that hopefully they’ll have more robust foundations and processes to do their chosen jobs more effectively.

You surround yourself with the wrong people then. And if you don't think that reflects on you or that it shouldn't be a factor...whew.

[Naetor]

I'm recording my interview tomorrow night- and I'll have it up on Youtube shortly after. The last actual question I see is from The Prefect on page 3 about throwing out production. If there was an actual question buried in these last 5 pages (I'm giving each of them the harbinger keyword), let me know and I'll address them in the interview.

[Nerdopolis Prime]

I'm recording my interview tomorrow night- and I'll have it up on Youtube shortly after. The last actual question I see is from The Prefect on page 3 about throwing out production. If there was an actual question buried in these last 5 pages (I'm giving each of them the harbinger keyword), let me know and I'll address them in the interview.

I hope you asked the CC for permission as well as all the people on the interview.

[Naetor]

I'm recording my interview tomorrow night- and I'll have it up on Youtube shortly after. The last actual question I see is from The Prefect on page 3 about throwing out production. If there was an actual question buried in these last 5 pages (I'm giving each of them the harbinger keyword), let me know and I'll address them in the interview.

I hope you asked the CC for permission as well as all the people on the interview.

Ya- I agree the harbinger keyword has been a source of controversy lately.

[monty42]

Honest question to all of the candidates:

How would each of you fair in a dick measuring contest with Corbett?

[sandy]

Honest question to all of the candidates:

How would each of you fair in a dick measuring contest with Corbett?

Well, it's "fare", but that's an easy mistake to make.

[Nerdopolis Prime]

Honest question to all of the candidates:

How would each of you fair in a dick measuring contest with Corbett?

WTF Monty??? Wie zur Hölle sollen die das Teil finden? Einfach ein Büschel erwischen und hoffen dass er dabei ist???????????

Du hast vielleicht Ideen ...

[Nerdopolis Prime]

Well, it's "fare", but that's an easy mistake to make.

Farewell, somebody said to something.

[Mogor]

Honest question to all of the candidates:

How would each of you fair in a dick measuring contest with Corbett?

Well, it's "fare", but that's an easy mistake to make.

electronic or physical need to make the distinction very different criteria. Also are we going by greek standards or modern cultural

[sandy]

In fairness to the other candidates who are having their interviews soon, I'm going to stop responding between now and my interview.

Thanks for the discussions!

[WeyounsLastClone]

Good luck to all candidates.

As a relative outsider, I hope everyone gets a fair chance (both with the selection, and after the decision is made to fulfill the job) and we can build the community again, instead of all the mudslinging and questioning all sides.

[tjark]

Honest question to all of the candidates:

How would each of you fair in a dick measuring contest with Corbett?

I ll hand that to my Medical Officer and undergo plastic surgery.

Thank you for making my nightmares more real.
Think I need another therapy session.

PS: Ich habe nicht das größte Boot im Hafen - aber keiner rudert so hart wie ich.

Tj