Show notes
Discuss this episode in the Muse community Follow @MuseAppHQ on Twitter Show notes 00:00:00 - Mark McGranaghan: Uh, so I, I imagine something like that from you where you can pick, you know, your favorite black ink, your favorite highlighter, your favorite accent pen, put them in your little toolbox, and you have this small, very curated palette that you can swipe in and out when you’re actively working on a document, and you’re not confronted with like the Photoshop style 200 buttons, most of, most of which you don’t know what they do. 00:00:33 - Adam Wiggins: Hello and welcome to Meta Muse. Use the software for your iPad that helps you with ideation and problem solving. But this podcast isn’t about Muse the product, it’s about Muse the company and the small team behind it. My name is Adam Wiggins and I’m here today with my colleague, Mark McGranahan. How are you doing, Mark? 00:00:52 - Mark McGranaghan: Doing all right. You know, it’s uh interesting times over here in Seattle with the virus, but otherwise doing pretty well. 00:00:56 - Adam Wiggins: This is a good moment to be on an all remote team, right? 00:00:59 - Mark McGranaghan: Indeed. 00:01:00 - Adam Wiggins: So the topic we wanted to talk a little bit about today is tool switching. And so this is the idea that if you take your stylus, your Apple pencil, and you touch to the screen, what happens? You know, what color is it inking? Is it erasing? Is it something else? What color is the ink? Is it something else totally different, like a a lasso or a scissor tool? And this is a a deeper topic than it might seem. Uh, because it comes to some values that I think Muse has or that we try to fulfill some principles, perhaps you could say in our design, including things about modelessness and things about sort of on-screen Chrome. But it also touches maybe on our journey from being a prototype in a research lab through to a sort of an MVP of beta and hopefully on our way to a publicly released, uh, commercial product that anyone can use. 00:01:51 - Mark McGranaghan: Yeah, it it’s been a really challenging problem, much more so than I thought it would be coming in. Uh, one does not simply ink on the iPad, it seems. 00:02:01 - Adam Wiggins: Indeed, yeah. And there’s a whole set of technical challenges that maybe one of these days we can get Julia on here to talk about would be great. Um, but yeah, maybe we can go back to the beginning. Can you, can you frame up the problem for us a little bit? What, what were we trying to accomplish? Uh, why, why not just sort of have a toolbar at the top, you tap on the thing like you would in Photoshop or any procreator or something to pick a tool and then off to the races. 00:02:24 - Mark McGranaghan: Yeah, so as a reminder just mechanically what we’re trying to do here is when you touch the uh Apple Stylist to the screen, do you get ink, do you get Eraser, what type of ink are you seeing, things like that. And the very standard way to do this in iPad apps is you have a persistent toolbar, often at the top of the screen or some other palette, where if you want to erase, you tap the erase icon and if you want to red ink, you tap the red ink icon. If you want to highlight, tap the highlighter and so on. And that’s a sort of mode where that is persistent until you go back to the toolbar and tap it again. Uh, so there are two main problems with the standard approach. One is that you have that toolbar in your face all the time, uh, which is a pretty big deal on the iPad. It’s a relatively small sized device and you want, uh, we want as much space as possible for your content and for your work, and to not always be looking at like Chrome and toolbars and buttons and other stuff that isn’t what you’re actually trying to think about and do deep work on. So that’s kind of the the chromeless goal. Uh, the other thing is modelessness. So a mode is um a property of an interface whereby when you go to do some physical action, The result depends on some hidden state of the app. So in this case, that that mode, that state is like what um inking button you have pressed in your palette toolbar or whatever. And the problem with that is that these toolbars, they tend to be off to the side of the device away from where you’re working, so you have to basically have your attention in two different places. It’s you’re looking and thinking about your, your work, the text that you’re highlighting, for example, but then you got to remember constantly what’s the actual thing that I’m currently working with. Uh, this is subtly different to, for example, if you have a physical highlighter. So you have a physical highlighter and you’re going to highlight like the highlighter is thicker, it’s bright yellow, it’s very obvious what you’re doing because you’re looking at the, your hand and your instrument and your work, which are all in the same place. But again, that’s not the case with a typical toolbar. Um, and so we wanted to try to find an interface that didn’t have this modeful property that wasn’t moded like this, uh, as well as it didn’t have, um, all this chrome in your face all the time. 00:04:33 - Adam Wiggins: And a great articulation of this uh modes concept is in the Humane interface by Jeff Roskin. And he talks about the, I think the really classic example there is the caps lock. This is just confounded many, many generations of computer users where when caps lock is on, different things happen when you press keys, specifically, you get the upper case rather than the lower case. And of course, this is really confounding for something like a password field where you can’t even see that feedback immediately. But even in a uh another case where you can see what you’re typing. You type a word or two and then you realize everything’s upper case because the caps lock indicator that being on or not, you either have to remember it, or you have to kind of look down and see an LED or some kind of indicator, and you tend not to do that because your attention isn’t there, your attention is on what you’re writing as it should be. 00:05:23 - Mark McGranaghan: Yep, this also points to a third issue with the standard moded interfaces, which is that you actually need to physically do the action of moving your hand away from your work to the toolbar and back again. And if you’re constantly switching between inking and erasing or different types of inks, that actually becomes quite troublesome. 00:05:41 - Adam Wiggins: So let’s go back in the story a little bit and kind of Work through the product or design problem. So we started from this place of let’s, let’s do the Raskin thing and try to be modeless and also that we don’t want a bunch of junk on the screen or we want as little stuff on screen as possible, be focused on the user’s content, keep all the, keep all the space for your stuff and not for the applications, uh, administrative debris. And so, uh, back when we were working on this in a research context, which probably explain what that means a little bit, but Uh, we set out with this set of goals and, and how did we first approach that or what what were some of the first things that we tried to see if we could fulfill these, these goals while still letting you, of course, do lots of things with the stylus. 00:06:23 - Mark McGranaghan: Yeah, well, so I don’t think we, we fully knew what we were getting ourselves into. Pretty early on we had these two goals. We don’t want to have any Chrome and we want it to be modeless, but if you If you do both of those things in any obvious way, you basically can only have one thing that the stylist does. um, so for a while our solution was you can only do black ink, that’s it, uh, which actually got us surprisingly far, um, but then we need to try some other things and, and then we did a whole uh litany of experiments. We, we did try some standard toolbars and palettes. We tried to make them as small and minimal and nonobtrusive as possible. Uh, we tried using various uh quasi modes, which is a term that I want to introduce here. So, uh, a standard mode is when You kind of do an action to trigger the mode and then you go and do your work and then you go back to the, the mode switcher to switch it again, whereas a quasi mode is when there’s something that you’re basically holding down, it’s like when you use the shift key or the command key or holding down some other control button, you know, that sort of thing while you’re doing the action with your other hand, basically. 00:07:28 - Adam Wiggins: Yeah, so going back to the capslock example, Caps Lock and shift do the same thing. But the difference is that with shift, you are not likely to forget you’re in the mode because you’re physically holding the button down. And if you ever get confused about how to leave the mode, you basically just release, stop doing things, and you sort of go back to your default state, exactly. 00:07:47 - Mark McGranaghan: So, we, we tried all kinds of quasi modes, uh, we, we didn’t necessarily have a keyboard, which is the obvious place to invoke a quasi mode with some kind of control key, but I think we tried, um, Using a physical keyboard, which you can sometimes get with tablets. We tried pressing like the volume button on tablets, we tried putting your thumb over the camera so that it registers a black image. Um, we tried pressing on various special places on the screen like press in the bottom left corner if you’re a right-handed anchor. Um, so there was various experiments with quasi modes. 00:08:21 - Adam Wiggins: Also worth noting there that in many cases, so Muse runs on the iPad, but In the context of the research lab, we were building for a number of different tablet stylus platforms, including the Microsoft Surface, Google’s ChromoS, and, um, I think we might have even done something with Android at one point. And so those actually platforms have different affordances or different hardware capabilities. So notably the Surface, for example, has this reversible stylus where the back is a quote eraser, which actually is really nice because again, You know, you, you have that physical reminder, just like your highlighter, um, example, you’re holding the thing in your hand in a reverse position. That’s clear, you can see it, you can feel it. And so you you flip the thing around to erase and flip it back. Fortunately, the iPad doesn’t have that. Uh, yeah, other, other platforms like ChromoS, for example, have a barrel button, all of that has certain restrictions tied into the operating system. So, we tried quite a lot of crazy stuff on this. 00:09:15 - Mark McGranaghan: Yeah, uh, we also tried some stuff just using the stylist differently. So one experiment that you had done was using the stylist to write special symbols. You called them glyphs, and it was something like if you draw an X then it’s cut or something like that and if you draw a downward V then that’s paste. um, so we did that experiment. Uh, we also tried. Um, using the stylus with different attitudes towards the tablet. So typically when you’re using a stylus, it’s like pretty vertical with respect to the tablet, uh, but you can by holding a different way, you can make it almost uh parallel with the tablet, sort of like you’re doing a a pencil shading motion, uh, and there, there are sensors in most devices to detect that altitude. Um, so we actually use that angle to trigger different behaviors of the stylus. 00:10:06 - Adam Wiggins: Yeah, notably, that one was the one that we kind of found most promising. I think we published maybe our Muse design article included that as well as Yuli’s. Yuli gave a talk at a conference last year with that approach where basically when you would hold the stylus overhand, it would allow you to move cards. Uh, or resize them, but when you held it in the, uh, more the, the standard writing grip, that would give you ink. And I, I really loved that. It worked really well, uh, in a lot of ways. It was very intuitive and, uh, you know, different grips is something that comes naturally to humans and it was pretty hard to get confused between them. But ultimately actually that one was. Killed by a technical challenge, which was the iPad, and I wouldn’t be surprised if other platforms have the same problem. I suspect it’s a hardware thing, but as you get close to the edge of the screen, for us, it was, I don’t know, 50 to 80 pixels, which maybe is, I don’t know, yeah, a couple centimeters. Uh, it would basically start to produce bad values. We did a bunch of digging on this and filed some bugs and some other things, but ultimately and and saw that this is just behavior that’s systemwide, but I think no one else ran into it because who the hell cares about the exact, uh, you know, why is it critical to hold the pencil at a certain altitude when you’re near the edge of the screen? Even like typically the only apps that really make use of this data are art kind of art drawing apps, Procreate or. by 53 or something like that, and you can see when you use them, if you move towards the edge, you lose that that sensitivity or the data gets bad about the position of the stylist, but it doesn’t really matter because it just changes fairly subtly what’s happening with the brush. But for us, the difference between moving a card, which could even have the effect of deleting it if you flick it off the screen and inking is huge. And so that ended up being a To a total non-workable thing for us, and we had to step back from it. So where did we land? So, so we went through this whole process of trying different things on different platforms, again in the research context, and then later, once we had kind of resolved onto the iPad as a platform and the prototype of what would eventually become a spin out product of Muse, by the time we went, went to make this transition from the lab to A commercial product we had actually settled on this, uh, position of the stylus as the solution, but then I think it was the early MVP and the early beta testing with with real users, not the initial usability tests. I think those, you know, if you got someone and to just try the thing for 20 minutes and and taught them how to use this different grip, that worked fine. It was more in practice over longer use in the real world where the edge of the screen problem became. Uh, basically a show stopper. And so now we’re in this mindset of, OK, we need to make this more reliable for real world use and we, we had to make the transition. So what did we eventually do on that? 00:12:55 - Mark McGranaghan: So we ended up with two mechanisms. Uh, the first is for erasing. If you press on the screen with your non-writing hand, say your left hand, while writing with the stylist in your right hand, that will actually do an erase. So while you, while a finger is pressed down on the screen, You have a quasi mode to do a race with a stylus, and then when you let go of your, of your finger, then the stylus goes back to inking. And then for selecting which ink you use, currently we have three options, a standard black ink, a sort of accent, purple ink, and a highlighter in yellow. Uh, we have this uh flow where you can drag from any edge of the iPad screen. With the stylus, and when you drag out from the edge, it reveals a small subtle um ink palette where you have those three options, and then you can select among those inks like a standard ink toolbar. Uh, and then optionally you can swipe back from that toolbar back to the edge of the screen and hide it again. So this is basically the best, uh, set of compromises that we can come up. We really like the quasi mode or you’re fairly limited on the iPad with much hardware options you have. Uh, so for now we’re just using the one finger down and that works quite well for racing, but that only gives you one, you know, degree of freedom. And so for the other inks, we have this, this toolbar that you can slide out, and it is still…
Full show notes at the publisher 
Our TOPPODCAST Picks 
Stay Connected
