Show notes
Discuss this episode in the Muse community Follow @MuseAppHQ on Twitter Show notes 00:00:00 - Speaker 1: There are a lot of other projects that have very similar models to this dynamic land database, but it definitely pushed me to think a lot more in terms of having state exposed by default, ambiently, and the value of being able to make little quick debugging tools that can piggyback on this global state. That was a super influential model on the way I think about programming and the way I think about debugging, this idea of being able to make really lightweight tools or jigs to help myself as I work. 00:00:32 - Speaker 2: Hello and welcome to Meta Muse. Muse is a tool for deep work on iPad and Mac. This podcast isn’t about used product, it’s about the small team and the big ideas behind it. I’m Adam Wiggins here with my colleague Mark McCrannigan. Hey, Adam. We’re joined today by Omar Rizwan. 00:00:49 - Speaker 1: Hi. 00:00:50 - Speaker 2: And Omar, I understand you have a collection of metro cards. 00:00:56 - Speaker 1: Yeah, I mean, I was just looking at this shelf above my desk, and it turns out I have this giant, basically the only thing on the shelf is this giant plastic pencil case that looks like a giant metro card. And so, I think a lot of people do this, but I’ve just started this habit of just filling it every time I get a metro card or transit card from wherever I go. So now there’s like, I don’t know, there’s a lot, there’s a lot of cards in here. It’s pretty full. 00:01:20 - Speaker 2: So let’s see, what must you have? It’s certainly a Bay Area transit card, and maybe, I don’t know, an Oyster card from London, or what, uh, you know, does this reflect kind of like a travel log of your places you’ve been? 00:01:32 - Speaker 1: Yeah, in a way, it’s kind of a nice, I guess we can connect it to one of the themes, which is that there’s like kind of an object for each place. There’s an octopus card from Hong Kong, there’s a card from Paris. It’s sort of like, instead of entries written down in a book, it’s like I have these like little cards that I can kind of pull out and look at. 00:01:50 - Speaker 2: Nice, and then it’s sort of like, I like the idea of keeping it around because it implies you’re gonna be back, right, that you’re a globe trotting, you know, person of the world, and you never know when you’re gonna need to whip out your Hong Kong transit card. 00:02:06 - Speaker 1: I think there’s also something like comical about the like very large metro card, like very large version of anything. It’s like, uh, you know, a prank we used to do in like middle schools. If people left their laptop unattended, we would just go and make the mouse pointer really big and like not do anything else and just like. 00:02:25 - Speaker 2: And you are an independent researcher with a very diverse set of interests, lots of things that overlap with the niche interests that Mark and I, and I think a lot of the listeners have, including end user computing and embodied computing, file systems, vintage computing, and so forth. But why don’t you give us a little bit of a summary of some of the stuff you’ve worked on over the years and where your interests in the computing world lie. 00:02:51 - Speaker 1: Yeah, I mean, so my background is mostly, you know, in programming, you know, I learned to program very early, and I sort of got interested in, like, new ways to interact with computers. Like, when I was a teenager, there was all this stuff on, like, building your own multi-touch table, and then I kind of got involved with Brett Victor’s work at Dynamicland, but also did a bunch of other different projects, kind of in that space, and just in general, I’ve always been interested in like, Different ways to interact with computing, both like future looking and also historical, like, what are their operating systems that people have done, what are other interfaces that people have done. And so, that’s my background. 00:03:28 - Speaker 2: And I feel like just looking down your portfolio the right way to describe your list of of projects, your research provocations, perhaps they’re quite varied, but they seem to have in many cases a sense of less of a like, here’s a, I don’t know, a library you’re gonna use or an application you’re gonna use and more of a Almost like an art project element of like, let me make you think a little bit here. For example, one kind of near the top, at least at the moment is hijack your feed, and if I’m not mistaken, this was one you did together with uh Jason Yuan, is that right? Yeah, yeah, who we’ve had on the podcast before as well. And yeah, I feel like that’s as much uh asking questions about social media feeds and the place they fill in our life and how we can like take a little more control of our computing world. But then you’ve got, for example, TabFS which mounts the open tabs in your browsers as files and lets you basically do, you know, The kinds of shell programmatic things that you can do with normal files, but with sort of your web browsing kind of history or current open topics. So I’m not sure how much, I mean, maybe I don’t know, Tabafest is in quote unquote production and you have people using it for serious things, but as I look down this list, I feel like they’re more of a, yeah, again, it’s just kind of like art project to make you think and question assumptions about the status quo in computing. Is that a correct conclusion to draw? 00:04:52 - Speaker 1: Yeah, I think that is a lot of them. And that’s also, I think a lot of what I do, you know, on Twitter or in my writing, or whatever, is sort of try to provoke people or come up with these really striking images. And I think, like, I’m often very skeptical when people sort of try to articulate this like philosophy of like what computing should be or, you know, explicit tenets of these are the things we want. I’m much more on the side of like, we should have a few very striking, like, concrete examples of like, things you might want to do, or like, interactions that are possible, and then those will kind of drive people in a certain direction. 00:05:28 - Speaker 2: I think the project of yours that was the first one I ever came across was Screenotate, which is essentially it seems to combine a couple of your interests here, including Provenance and OCR, but essentially it’s a screenshotting tool that makes it very easy to grab the text out. Now I’m not sure how much the latest changes in MacOS and iOS where there’s some of that built into the OS. Well, maybe we’re even inspired by what you did there, but that is well’s product. You can download it, you can pay for it, presumably you’ve been maintaining it for a while, so it’s not pure research in the sense of, and I use it also, you know, all 00:06:04 - Speaker 1: the time, that’s key, like I’ve probably taken 30 or I’ve probably taken like 40,000 screenshots in it, so, wow. Yeah, and I think there is, with a lot of the projects you’ve mentioned, there’s also this theme, and this gets at this idea of folk practices a little bit. There’s this theme of like, this is very vague, but like, connecting different universes in unexpected ways, like this idea of like, there’s your browser and your file system, and you jam them together, or there’s this like social media interface, and there’s this idea of tasks or productivity, and you jam those together, or even the dynamic land stuff, I think, has a little bit of this, like, there’s the objects in your computer and there’s the objects in the real world. You kind of try to Combine those in some way where you can use operations that you are familiar with on one and apply them to the other. And I think that also something that connects really well with people, because you’re sort of familiar with both sides, and so you kind of immediately see the combination of them, and you’re like, oh, this is really cool or really interesting, or like, I can quickly imagine how it would apply, you know, to my life in some useful way. 00:07:06 - Speaker 2: So you have the anchor points of the two things that you’re familiar with, and the novelty or the provocation, or the picture of what could be comes from thinking about how those two would combine. Yeah. So our topic today is folk practices, and this is a term Mark and I use quite a bit here on the podcast and even on our team as we talk about ways to look what people do naturally with existing tools or existing features. In, you know, a product that we or others are building and then sort of extract from that what they’re trying to do and in many cases you can even shape a product or a set of features or an operating system to embrace those folk practices. And I think Screen notate, the project we just mentioned, is one good example of that because the idea that like People sometimes complained, screenshots full of text, this is so annoying. Why not have the core text, you’re spending way more data to represent it, you can’t reflow it or do other things you can do with the text, and to some extent, Folk practices, I think is a saying like, look, screenshots of texts are really here to stay, and there’s a bunch of reasons why that might be, but just empirically, this is a thing people do and they do a lot. And so maybe we should learn from that and find out how to kind of roll with it. Like, if you can’t beat them, join them kind of thing, rather than kind of, you know, basically complain that you’re not doing it right. 00:08:32 - Speaker 1: Right, or at the very least, you might not join them, but at least you should look at it and be like, OK, why do people do this, rather than lecturing people about, you know, you should do this other thing instead. 00:08:43 - Speaker 2: We were at a conference together recently and you did a little demo to the group, and this is called Screen Matcher. Can you tell us about, yep, that one? 00:08:52 - Speaker 1: Yeah, so this is a project that I’ve been working on a little bit this year, and basically the idea is it’s this Daemon, it’s this app that sort of runs in the background of your Mac continuously, and it’s constantly watching your screen, so like the screen on your computer. And so this screen matcher, you can teach it to look for patterns on your screen. It’s like you’re taking a screenshot, like, you drag out a region of your screen, and then you kind of feed that torematcher, and it’ll look for whatever you took a screenshot of from then on. The example I usually give is like, you know, in the corner of every window on your Mac, there’s these traffic lights to like close, minimize, and maximize. And so you can teach Screen Maer to look for that pattern, it’ll find it wherever it sees it. And then you can draw on top of it. So, effectively what that means is you can add like a 4th or 5th button to every window on your computer. But, you know, there’s a lot of other things you can do once you have this kind of continuous screen matching mechanic. Like, you can kind of just like add buttons or draw or scribble on anything on your machine, and have these like automatic behaviors. So the other example I usually give is like, with the screen matcher, you can build like an alarm clock without traditional programming, because what you do is you’d be like, OK. I want to wake up at 7 a.m. tomorrow. So you’d set the clock of your computer into the future. You’d be like, pretend it’s 7 a.m. tomorrow, and then you tell Screen Matcher, hey, when you see this pattern in the top right corner of the screen, when you see it say 7 a.m. I want you to play a sound and wake me up. So there’s this idea of like, you can extend the functionality of your computer in a very natural way, and there’s this idea that you can do things you might normally take like programming or scripting or whatever, just by pointing at your screen. 00:10:25 - Speaker 2: It reminds me a bit, especially that example you gave there of an if this then that or a ZAPA or something like that, which do have this element of automation without real programming, but those really rely on APIs. So you need to have an API integration that that means that the vendor, the creator of whatever the thing is, in this case would be the clock or the operating system or whatever needs to supply an API that you can consume through some probably fairly complicated procedure. And I feel like a hypothesis or a concept that’s embedded in this project and maybe some of your others is to sort of say, well, look, it’s nice to have APIs on things, but realistically the output from computers is pixels on a screen. So if we want to give some kind of end user programming capability, basic automation, rather than trying to browbeat program creators into creating an API, just sort of give up, and maybe give up isn’t the right way to put it, embrace that folk practice or embrace that reality that That GUI interface exists and by the way, computer vision is really good now, and so something like recognizing the widgets in the corner of your window or a clock value is actually relatively straightforward, so therefore maybe that should or could be the sort of an everyday API. 00:11:45 - Speaker 1: Yeah, I think that’s right, that there’s this, instead of this closed world of whatever is available via API you have this open world, much like when you take a screenshot, you know, you can take a screenshot not just of things that are selectable text, but if anything on your screen. Similarly here, you know, you can automate based on anything on your screen, not just things that happen to be an API. But I think there’s also kind of like an interaction argument for this, which is that Even if you have all the APIs available from the end user point of view, it’s like, OK, I want to do this automation. I guess I have to like read the, like, dictionary of APIs and like figure out what the right APIs are, or if they’re even available, I have to figure out like what kind of input and output they take, and that it’s always felt to me like very disconnected from the actual experience of using the computer. Like, you know, if I want to make an alarm clock, why can’t I like point at the actual clock on my screen, instead of figuring out that there’s a clock API that’s like based on the same source as the clock on the screen. Like, it feels like you should be able to point at the actual things that you’re already familiar with, instead of having some like API dictionary that’s completely separate, that feels like this like skeleton of the app. 00:12:50 - Speaker 3: Yeah, I really like it. And Omar, so the idea with Screen Matcher that you can both sort of scrape the screen for input, but then also do, I guess you would call output of typing things and clicking things, moving the mouse around. 00:13:03 - Speaker 1: I think so. You know, the current prototype, basically what you can do is you can just add but so you like can search for a pattern and then you can be like, every time you see this pattern, I want you to draw these extra scribbles next to it, and then when I click one of these scribbles, I want you to run a bash command. I see, I see, but I think it’s very easy to imagine being able to have other responses. To seeing things on the screen, whether that’s like playing a sound. I mean, someone proposed to me that you should have all the effects happen by drawing stuff on the screen and then Screen Matcher would like match those things and do the effect directly. Uh, I don’t I don’t know if that makes sense, but it has like a very nice, like, kind of aesthetic elegance to it. 00:13:40 - Speaker 3: Yeah, I also kind of like the baseline of anything that you can do as a human, whether that’s things you can see or inputs you can do with the keyboard or the mouse, you can script. Yeah, that seems like a reasonable invariant. Yeah. And as far as that’s a floor on automation, so no matter how hard the programmers try to deny you the ab…
Full show notes at the publisher 
Our TOPPODCAST Picks 
Stay Connected
