On this week’s episode we talk with Glyph Lefkowitz, creator of Twisted, the incredibly popular event-driven, Python networking engine. Glyph covers why creators have a responsibility to users of their open source projects, why he simultaneously admires and critiques open source communities, and why the open source world should look to its free software roots. He also explains how he took over the Python IRC channel to promote Twisted, how the project was born, why it has a bright future, and much more.
Listen to the podcast in the player above, or click here to download it directly. Subscribe on iTunes or do so manually by using this RSS feed.
Show notes
Transcript
Alexis: Thanks Glyph, I really appreciate it – taking time out of your schedule for joining us on the podcast.
Glyph: No problem, glad to be here.
Alexis: For folks who aren’t familiar with yourself, give us a little bit of background.
Glyph: Well, as I like to say in my Twitter biography, I’m a human being who writes computer programs. I suppose the thing that I’m most famous for is having written the Twisted framework, which is an event-driven system for writing programs in Python. It allows you to communicate with lots of different protocols on the Internet; it’s an app server, but it’s an app server that can speak DNS, IMAP and XMPP and a whole bunch of other protocols, not just the usual HTTP endpoint.
I’ve done some other things too. I currently work for Rackspace on open source technologies like Twisted. I’ve worked for Apple on an enterprise calendaring software –.
Alexis: Oh, I did not know that.
Glyph: I had a startup for many years up until 2008-2009 doing email and before that, I worked as an independent consultant on massively multiplayer games, and before that, for Electronic Arts on the ill-fated Ultima Online 2.
Alexis: I am talking with a person who had a hand in the history of MMOs.
Glyph: The unshipped original, or say the unshipped Ultima Online 2, not the original, seminal Ultima Online. Unfortunately, that project was a textbook example of second system syndrome. We could do a whole podcast about why that didn’t ship.
I’ve used Twisted or parts of it at all of those jobs, so it’s pretty versatile.
Alexis: Twisted itself is 14 years old or so?
Glyph: I think we’re coming up on 15 now, yeah.
Alexis: So when did programming come into picture in your life?
Glyph: Well I’m a member of a fairly exclusive club; I’m a second-generation programmer. My dad was always bringing home weird computers. I had an AMIGA when I was a little kid, and I had a NeXTcube for a while. I had an i386 Next machine and a SPARC IPC. I had a Mac at various points during that, but I don’t have that usual emotional association that a lot of people have with their platform of choice because I had all kinds of weird platforms and I had to keep adapting to strange things that I might have access to next.
My dad taught me a little bit about programming because I played Zork when I was a little kid. I loved books; I loved reading; I loved words. We played all of the Infocom games and I just love to wander around those worlds and have a conversation with the computer and see what was happening.
I told my dad that I wanted to write one of those, and he started showing me a little bit about programing; he taught me some APL when I was eight years old, which – I don’t know if you’ve seen APL, but it’s maybe not the best language for an eight-year-old. It’s non-ASCII characters; lots of little Greek symbols for all of the different operations. I learned how to make the computer talk back to me because it had an interactive interpreter like Python or Ruby or Lisp. You’re going to assign a variable, and then you could play a variable back by typing the name of the variable again. I learned how to do that and I had some notion that there was some way to go beyond that and that one could – if you can get it to say one thing when you type a word, you could maybe get it to say two different things.
I didn’t quite get there until a little bit later with the HyperCard. I spent years and years working on video games using HyperCard – and later, SuperCard – and this was wonderful because these were horrible proprietary technologies, that there is no hope of every recovering any of that work. And so everybody thinks that I’ve just bust under the scene with these amazing programming abilities, whereas in actuality, I have a veritable mass grave of failures that no one will ever be able to see.
Alexis: Now, this mass grave of failures [chuckles] – sorry, I liked the way you put that – it was all, maybe not all, but it was heavily-focused on gaming, primarily personally, right?
Glyph: Yeah, and I guess this is something that – another bit of wisdom from my father was that computers aren’t good for anything but entertainment. He said this because he was a programmer and development manager with decades of experience in financial institutions and industrial applications and various serious stuff like backup systems. That’s how he felt, and he felt that way because everything we do with computers, we could do more tediously with paper and pencil.
Everything that we do with computers – there are probably a couple of things that we wouldn’t be able to do fast enough that are actually enabled by computers, but we could communicate a lot of things, and we could do a lot of things with analog technologies, but it’s more fun to do it with computers.
When you’re deep in the guts of some SOAP service that’s deployed inside some ASP.NET container and you’re swearing at Bill Gates for dooming you. Yeah, remember that it’s more fun because it could be sitting from a big, metal file cabinet and just filling out these forms over and over and over again.
That was how I always approached computing, and I still tend to think of most of the systems that I work on as games in some sense, in that the computer’s simulating something and there’s an interaction, and that interaction is coming from the user who is sort of the player who’s playing the game. That’s just how I model the internal state of a system versus the world is – well, the system is a game and the real world is what it’s modeling.
Alexis: To reference a recent blog post – I don’t know how recent it was – a blog post of yours, you’re right, you could use everything with paper and deliver it with falcons.
Glyph: Yup. The falcons thing is a reference to Achewood where one of the characters was renewing his health insurance and they asked him to fax it in and he says, “No, of course I don’t have a fax machine; I also don’t have Polio and Apple IIc or a falcon.
Alexis: Twisted came out of your intent to build a game, right?
Glyph: Yes.
Alexis: Tell us a little bit about that.
Glyph: Well, I’ve been fascinated by games my whole childhood and I was sort of a little bummed out about the decline of text-based games. I kind of felt like text-based games going away as big graphical games happened was a little bit sad; it was as if we abandoned the novel when the movie came out, which didn’t happen. It’s a different medium; it’s a little bit low-refined medium but it’s an interesting medium in its own right.
And so I wanted to make a text-based game and I wanted to have an online version because the Internet was happening and I felt like one of the reasons that these older genres are dying out was that there were two ways that online games were going – or rather two ways that text-based games were going. One was that there was the offline, very story-driven ones that were very, very simple, short – mostly written by hobbyists at that point – and they were very small. They weren’t getting a lot of interest because there just wasn’t a lot of gameplay there, even though there were some interesting ideas being communicated. The other was that there were huge, sprawling online things except that they’re really huge in the sense that they had a lot of words people had typed into them. There was no story; it was just stats, right?
Alexis: Go here, collect 20 of this.
Glyph: Yup. Ironically, that’s exactly what most of the popular online games have copied, is that the “go here, collect 20 of this, increase your strength stat” kind of gameplay.
What I wanted to do with that game was I wanted it to be small group collaborative gameplay, and it turns out that’s really, really hard to actually implement in a meaningful way. One thing that I wanted to use to give that a hook and a lot of people to interact with it more immersively was, your group would have a website associated with it and a mailing list associated with it that was in-game. Characters in the game would be sending you email and characters in the game would be sending you texts and it would be webpages that you could view –.
Alexis: Yes, and this would all be fantasy-like, right? Or a Dungeons & Dragons kind of thing?
Glyph: It was more of a science fantasy-type genre. The game still has a website at divunal.com, which is going to tell you as much about the story as you can find out anywhere on the Internet, just to say.
Alexis: Yeah, I just saw it; it’s a logo.
Glyph: And it’s still going in some sense; a more useful artifact to that process. For one thing, I should mention this whole game development thing, the original game was called Twisted Reality. The game was actually called Divunal, but the game engine was called Twisted Reality because it was like we were simulating a reality but it was twisted in some interesting ways, things were different.
We released it and we had – if you go back on the Wayback Machine, you can see twistedmatrix.com, that’s what it was. It was a website for a video game. You would go and you would download it and then run it and it held a little demo.
One of the packages in it was twisted.internet, and nobody cared about twisted.reality, so we pretty much started focusing on the networking stuff. Since we wanted it to have these immersive components where you would log in and you would be able to chat with people, email people in the game world, the ability to do that was a lot more interesting to a larger audience than this bizarre, arthouse game that we were trying to build.
People started using that, and eventually we moved off Twisted Reality to another project, which is now called Imaginary. It’s actually moved back under the auspices of the Twisted project by moving under the Twisted org on GitHub.
We’re still working on it; there are still a couple of people who commit to that. I did some work on it quite recently that has moved into the – one of the problems with that small group, story-driven, immersive gameplay that I mentioned is it becomes very difficult to implement that because there are many, many interacting simulations in a game that has both story elements and stat elements. That has turned into a research project about systems integration – about how you can create these environments out of multiple parallel simulations and then integrate them into one model.
It lives. That tagline on the divunal.com website was not kidding.
Alexis: [Chuckles] As somebody who is incredibly interested in role-playing games and has toyed with the idea of making some sort of web-based role-playing games/experiences, I could do a whole podcast peppering you with questions about this. But I will, instead, sigh deeply and focus on Twisted.
Glyph: As I have done for the past decade [chuckling].
Alexis: I mean, talk about a project that comes out of another project. You hear about open source projects get broken out of bigger things fairly often, but this is just a monster in terms of how it’s grown to be so huge and take up now nearly 15 years. It has even got folks on-board such as Lucasfilm, HipChat, JustinTV – which is would assume means Twitch wound up using it as well.
Glyph: Yup. We’re not 100% sure; I have to get official confirmation that Twitch’s whole video platform is based on Twisted. That would be great. But definitely, JustinTV and several other video streaming things – I believe that at one point the video stream for the Olympics was carried by some Twisted infrastructure. Yeah, it’s been used all over the place and it’s really awesome.
And actually, some of my favorite projects are ones you never hear about. There was infrastructure project – it was Northern Africa – that was electrifying some very remote areas, and they needed a self-sustaining management software component, so they used Twisted to power these little solar receiver arrays. I have no idea why Twisted is the right hall for that; it seems like a strange choice, but they said it was exactly what they needed. And I guess one of the things is that there are many events that occur in those types of autonomous environments and they need to be able to react to them and to test the reactions to them and to test the reordering of all those events before they deploy them. More even the testing framework part of Twisted than the networking part of Twisted was what they really wanted, and it did what they needed so I was very happy to facilitate that.
Alexis: I was going to joke that the person who was in-charge of that project was a Python programmer. He loved Python so much he just went to Twisted rather than choosing another language or another kind of solution.
I say that, looking through our GitHub page, I noticed that most of your projects are in Python. Was it because you love the language so much, or you’re just – I don’t know. What’s the reason behind that?
Glyph: Well I think that readability is very important. Ironically, given that Twisted has, at times, had a reputation for being somewhat inscrutable for other programmers, I think that being able to understand programs is really important and I think that Python is one of the most understandable languages that I’ve ever used.
I actually think that Python really needs to be learning and growing beyond the way it is right now. I think that there are other languages that are becoming very popular in much of the same audience that Python has become popular like Go and Clojure. I think that there are fantastic languages and lessons that those could teach us, but if you look at those languages and you see what the process for learning them is like, it’s very programmer-y. There’s a lot of discussion of computer science concepts and it’s hard to skip that; it’s hard to just keep somebody that kind of hacks something together.
I think Python is just a great language because it makes things more accessible.
Alexis: I agree. As somebody who self-taught himself Python, I completely and totally agree. I feel the itch to break out to other languages now but I always keep coming back to Python when I need to knock something out quickly.
Glyph: I also write a lot of Emacs Lisps, so it’s not really my own little language.
One of the other reasons that I really stick to Python a lot is especially in my early career, the choice was sort of between Python and C++, and Python was always the right choice. There were a number of times, when working on video game development where someone has some code, it was running really slow, they ask me what I could do to optimize it – because at the time, I was kind of the whiz kid; I would look at it and I would say, “Oh, you should rewrite it in Python.” And then they’d say, “You’re crazy,” and I would go rewrite it in Python.
But of course I could create an algorithm that was an order of magnitude more efficient than what they had been struggling to get working in C++, and I could do it in half the time. The fact that the interpreter was a hundred times slower than the C++ runtime, it didn’t even matter.
I routinely would say that my job was optimizing C++ by rewriting it in Python, and it was true. That gave me a bunch of really, initially, very positive experiences with the language.
Then later on, the difference was Java versus Python. Java’s just boring and tedious to write, so.
Alexis: And emails to Glyph@somewhere.com?
Glyph: No, twistedmatrix.com. I’m happy to get flamed.
Alexis: Alright, so you’ve built Twisted not knowing that it would take off so well. What was its seminal moment in terms of gaining traction and what did you do to spread the word, if anything, after that?
Glyph: Well I think it was very, very intentional. It was a project that was explicitly started to gather interest.
Partially, it was that I was not actually interested in maintaining this network layer; I wanted to work on the game, and so I kept trying to attracting people to work on the networking layer. One of the ways that I did that was that there were several competing systems in Python at the time. I mean, Twisted has almost got that Wikipedia policy of no original research. We don’t have anything novel in Twisted; it’s all existing protocols, existing patterns, existing techniques. And in a way, that’s important because these techniques have all been implemented for years and years.
There was, in fact, this system called Medusa, which was a big, async I/O server in the 90s even, I think, with Python. The problem with all the systems is they were designed to solve a single problem, whereas because we wanted to dissolve an entirety of networking infrastructure so we could construct a fake version of your online world with Twisted. We wanted to have a very general piece of infrastructure where you just pull in libraries and just use them in this async environment and have the same thing that you would use in a desktop app show up server side so that it could faithfully simulate all of these clients that you and I need to visit other websites and things like that.
When we started building it, what we did was – me and the shadowy cabal of others who were involved with its genesis pretty much took over the Python IRC channel on Freenode. And even earlier than that, the OFTC was a different competing IRC network and we bridged the two Python channels with a – I believe it was a Twisted bot. We got these two groups together, and then we – every time anybody had any networking problem we said, “Have you considered using Twisted?” It was actually the right answer 99% of the time because we were trying to build this critical mass of maintenance of networking components instead of everybody just doing their one-off network infrastructure from the ground up and then having to maintain it themselves forever.
And then inevitably, they wouldn’t, right? People change jobs; people lose interest; systems stop working; code rots. With Twisted, we just wanted this big, central repository where you wouldn’t have to worry about it; you can contribute your code, you could spend a little bit of time doing some code reviews for other people’s stuff, and you wouldn’t have to own the whole thing forever.
In that regard, we’ve been very successful. There are a couple of us who stick around and put a lot of effort into doing code reviews and updates and things like that, but the project as a whole has sustained a lot of code that people otherwise, I don’t think, would have maintained.
Alexis: You automated grassroots – I guess the equivalent of that nowadays would be a Twitter bot.
Glyph: Right, yes.
Alexis: It’s probably not as effective.
Glyph: [Chuckles] Well I mean the bridge was not totally automated; it was just joining these two networks together, so it’s like we made a Twitter bot that would unify a Facebook group or something. We just brought all these people into one place even though they didn’t necessarily want to be there with each other.
Many of the people who were interested in IRC and interested in Python were also interested in Twisted because it was a natural union there because you had the –. IRC is sort of real-time, and to speak it you need to write code that is event-driven, so people who were interested in it also tend to have at least a little bit of interest in Twisted.
For a long time, that just was the Python IRC community, and now most of us have withdrawn from that. I mean, there is still a pretty heavy – you just use Twisted contingent in that IRC channel, marginally because there are lots of people who show up who have a certain background. They use Django, or they use Flask, and they just want to continue using their existing technology but add all these additional capabilities which all exist already in Twisted. Every so often people had to go through over the friction and say, “No, no, I don’t want to use Twisted; it’s so heavyweight” and then they realize that actually, the things that they think are the heavyweights are the parts of Twisted that they need to do the things they want to do and so they eventually adopt it.
That discussion is still going on today, 15 years later.
Alexis: So speaking of the different options out there like Gevent, Eventlet and Tornado, give us a 30,000-ft view of how Twisted is different.
Glyph: Twisted is mostly different from those options because Twisted is not just an event loop; Twisted is an authentication idiom. Twisted is a mail server. Twisted is a name server. Twisted is a web server. Twisted is a web client. Twisted is a plugin system for registering arbitrary string connections. Twisted is an SSH server. Twisted is a drop-in replacement for the openSSH client. Twisted is an API for making SSH client connections. Twisted is an XMPP server. Twisted is an XMPP router. Twisted, with a couple of initial components, can be an XMPP chat server. And so on and so on and so on.
With all of that stuff, it still ends up only being about four times as big as other projects, which are just an event loop. And I should mention, Twisted isn’t just the event loop; it’s also an event loop that’s supported on Windows. It’s also an event loop that has capability to integrate with the Mac GUI, with Linux GUIs like GTK and Qt, with the Windows GUI – it unifies a whole bunch of different functionality that you might need, so that when you write some code against Twisted, it just runs in all these different environments.
I’m sorry, I think I cut you off on another question there?
Alexis: No, no, that alright [chuckles].
Glyph: Okay. So this ecosystem, or all that tooling, is just built in and then Twisted also has an ecosystem around it. We have a prefix – since Python projects use .py, we use .tx to abbreviate. We have a whole bunch of .tx foo projects where foo is anything from the Rackspace API, to integrating with some Internet of Things sensors, to speaking to Thrift – I mean, there’s just a whole bunch of different libraries that you can grab.
When you compare that to – Tornado has a web server and a web framework. Tornado has a little more web framework built-in than Twisted does.
I was like, “Okay, yes, Tornado’s cool but if you need an IMAP server, it’s not going to come out of the box.” And then a system like Gevent, which is barely even a networking system – Gevent is like a thread scheduler. It has some stuff to let you schedule threads based on sockets, but it’s really about reducing resource consumption of this one very specific, concurrency primitive in your applications, whereas Twisted is about a whole way to write your applications that they’ll work together with all these different protocols and all this other code.
Alexis: You’ve mentioned you worked for EA, and now Rackspace, and you’ve done some consulting in between, or for MMO companies, and you had your own startup. What have you done to monetize, or support yourself and Twisted? In the past, at least.
Glyph: Well, I mostly just work for a living [chuckling]. I should mention that Twisted is a member project of the Software Freedom Conservancy, and probably would not nearly as good-shaped without the generous support of many of our sponsors.
Twisted was, at various points, partially maintained by a paid fellow who had not the best salary, but it was a stipend to work full time on reviewing Twisted tickets and maintaining our infrastructure. Right now, we don’t have one; we’re considering reinstating that position on the Twisted leadership committee right now.
That infrastructure has really helped us. The Software Freedom Conservancy is a 501(c)(3), so they have all of the – you can donate to it and get a tax rebate since Twisted is hopefully for the public good. That has probably been the single biggest thing in terms of supporting Twisted financially. It has been spinning up that – making it easy and possible to just donate financially to the project, and the great folks over at the SFC have helped us a lot with that over the years.
Alexis: So you can get sponsorships from, for example, Olark is one of them currently and Critsend.
Glyph: Mm-hm, yup. And we’re absolutely terrible about reminding people that they can sponsor Twisted and fund raising for Twisted, so there are probably lots of companies out there who would love to be a sponsor and just haven’t been told or haven’t been reminded they need to renew.
Alexis: I was going to ask, why did you decide to go the sponsorship model rather than maybe providing support or focusing more on that, or maybe having a licensing model where you also have a commercial license?
Glyph: My startup, Divmod, we were a startup for a very long time – thousands of years in Internet time. Almost seven years, I think, of every human time. We eventually were doing consulting; we did a whole bunch of side projects to try to keep things going. We did sort of contemplate an enterprise support model for Twisted.
There were two problems with that: one is that Twisted just works great. You don’t need support for it. In many cases we did sort of code ourselves out of contracts. Someone would tell us about a bug, we’d say, “Oh yeah, we should fix that.” We’d fix it, and then they’d show up a month later saying, “Okay, I’m finally through the PO approval process.” “Oh, you fixed it. Okay, great! Nevermind.” [Chuckling]
That’s part of the problem is that the problems that people can articulate with Twisted, the things that they can point out and say, “This is a bug and I need it fixed” are either fixed extremely quickly or are unreasonably hard and can never be fixed.
Alexis: Or which change the project in a way that’s contrary to the view of how things should be.
Glyph: Exactly, and so the enterprise support model – there’s also the fact that it’s all very much API-driven and you write code to use Twisted so the folks who would be the ones who would want the “support” are, in many cases, all pretty much as empowered to fix it as the core maintainers are.
Twisted was really about creating this ecosystem, creating this environment where people could write this type of code. It’s a commons, and trying to squeeze revenue out of it never really felt like the right model.
We had lots of value add and Divmod did lots of specialized consulting where we helped people who were using Twisted for more specific cases, and people definitely got a lot of value out of that and I hope that more people will show up in the community and start doing that sort of work again. But I think as a whole, not all open source necessarily needs to be on one company, single provider, like there’s somebody who got rich because they did x type of model.
I think that for a lot of projects, the consortium model makes a lot of sense, and I think that the problem with Twisted of course is that given it’s designed both for large enterprises and also for kids working on a network version of Asteroids with their friends, the consortium members there – if you had a 12-year-old in the board meeting, it’ll be a little weird. Since we’re trying to serve that really broad audience that is the public, and so that’s a nonprofit. That’s why we’re set up that way.
And I think more open source projects should consider doing that because frankly, I’m doing great. Authoring Twisted has helped me, personally, lots and lots. I’ve gotten to meet lots of wonderful people, had speaking opportunities, had lots of great jobs, and a lot of that is because I created Twisted.
I think that for folks who are maybe looking for a way to make open source work as a business model, think about it more in terms of personal benefit, like make something great, give it away and it’ll probably work out. I mean, obviously don’t invest your life savings in doing that if you can’t afford to, but there’s a lot of room out there for open source with no business model.
Alexis: One of the things that – or at least one of the models where that is the case, a company will approach someone and say, “Hey, do you want to work for us but still work on your project? We’ll give you a salary because we really like the project.” Is that what you’re doing with Rackspace? You mentioned you were working with them on open source technologies. Are you working with them on other open source stuff?
Glyph: I work a great deal on Twisted; a bunch of my time is dedicated exclusively to that, and I am very happy about that. That’s one of the reasons I came into Rackspace.
We have a bunch of people at Rackspace working on important community infrastructure like Donald Stufft, one of my colleagues. He works on the Python packaging infrastructure and he does that at Rackspace, on Rackspace time because he’s got the support of the company for doing that. And thank goodness he does because he used Python, you use PyPI – pretty much everybody does.
People can say, “Well, I don’t use Twisted.” Maybe that’s true. You should – you should totally use it, but you use PyPI. I think that’s a great model, if you can get it adopted by a couple of big companies. And actually if I could sort of get on a soapbox for a second –.
Alexis: Absolutely, soapbox away.
Glyph: This is one of the things I think that people don’t always realize about the free software and open source ecosystem is that you’ll often hear people say things like “cash is accepted” or “you can always feel free to fix the bug yourself” and things like that. To some extent, that’s true. People using the software should participate in its development; that’s a good, healthy thing especially for projects which are designed for programmers, and even for those that aren’t, like helping people get into programming by hacking on their graphics editor or whatever it is that they’re interested in.
But I think that there’s a mean that comes along with that, which is that I gave this to you for free, so I don’t owe you anything. And I think that that’s very much not true. I think that if you’re a volunteer-run project, you do have limited resources; you do have be realistic about that.
Believe me, Twisted is run by a team of basically four or five people maybe, and people drift in and out all the time, and so we are – for how big we appear, it’s a very small group. But I feel like even if you’re realistic about those resources, you should not feel like you don’t owe your users anything because if you are shooting for this kind of sweet gig that I’ve got right now, or you got an employer who sees the value in the open source work you’re doing and wants to empower you to do that and enable it, you also have to remember that at some point you told that person, that employer, that company to use your stuff and they did.
In doing so, they invested their time and their energy in learning your stuff, learning the code you wrote, deploying it, managing it, hiring for it – they’ve invested in your ecosystem if they’re asking for you to do something. If they’ve invested in that, that investment – if you’re the creator or you’re a maintainer or your name is heavily associated with that – is valuable to you. It’s valuable in ways that you can’t directly turn into money and spend on the work, but nevertheless, you do have a responsibility to those users. When you tell them – even in the first place, even before they’ve started using your stuff, whatever it is – “Hey, please use this thing,” it’s not like they just started using it out of nowhere. You told them, “Please use it.”
I feel very strongly that the Twisted development team, in as much as it chooses to identify itself as a unit, really owes our users something. That’s why we have such a strict compatibility policy, and that’s why we have such a strict code quality review policy. It’s because we told people that this stuff was cool and they should use it and that it was going to get better over time and that code is going to get maintained, and bugs are going to get fixed, and I want to make sure that we’re making good on that promise.
If you are the sole person who maintains open source and you’re talking to your users about using your stuff, remember you can always let them know that they should sponsor the project if it’s got a nonprofit, they should help out and file bugs, and fix bugs, because we’re all in this together. But at the same time, remember that if they fix the bug in your project, you do benefit a little bit disproportionally from that and you should be responsible for that benefit. You should feel a little bit of responsibility.
This is especially true for security bugs and flaws that really might cause harm. Even if that is really stressful for you to fix in a timely way, you should feel like you have to do that, not because those people are depending on you. Even if they’re not paying you to depend on you, there is never a less of benefit.
Glyph: None of this is going to go in.
Alexis: Through your 15 years of working with Twisted, what’s one mistake you’d rather not repeat?
Glyph: Oh, wow.
Alexis: I mean, if it’s tough to choose, you could give us three or four.
Glyph: [Chuckles] I definitely make a lot of mistakes. I suppose the one that comes to mind at first is I really would’ve inherited from less stuff. Twisted definitely has some idioms internally, which we’re very slowly trying to fix where the thing that you do when you want to use a part of it as you subclass that part. That has been a constant headache because it’s very hard to define a clean API boundary around somebody subclassing your thing, because suddenly they’re dumping stuff into the namespace itself, which is a namespace which a library code owns, but then they also own it sort of in a way.
More generally, I would’ve really put a lot more effort early on into defining the boundaries of the APIs that we provide, so that we’re easier to evolve and to add new features so that the people could get things for free without migrating to this newer, improved API. A lot of stuff now, because there’s this tight coupling, because there’s subclassing and things, that import from random places and names that are exposed by accident through various – whenever you import something into a module in Python, someone can import it from you again even if you didn’t intend that.
I would’ve been a little bit more focused on defining what exactly the external API of everything should be, and part of that’s documentation, and part of that’s convention, and part of that’s not using subclassing – I can eliminate a whole class of mistakes by saying, “narrower, more well-specified API.”
I’m trying to think of – there are definitely other mistakes that I’ve made. I would’ve probably gotten into fewer arguments. I got into a couple of famous flameouts with people. I would like to think I’ve never been personally insulting or anything like that, but in every argument I pretty much said, “Well, I’m sorry you feel that way but we’ll see who’s standing in five to ten years –.”
Alexis: And who’s still standing after 15 [chuckles].
Glyph: Yeah, and so while that is somewhat satisfying, it also highlights how much of a waste of time it was to even engage with those people who were so mad about Twisted. It’s just like – we were doing our thing and some people weren’t going to be happy about it. When I was younger, I didn’t realize that there are always going to be people who aren’t happy about it and should’ve just – there was no point to having many of those conversations, which especially in the middle part of Twisted’s growth in the 2,000, 60,000-7 era were very personally stressful to me. I don’t even know what happened to those people; they’ve all disappeared.
So it’s fine; everyone that I still interact with from the project, from Twisted is just delightful. I should’ve focused more on those people.
Alexis: On the flip side – enough focusing on the sad parts – what’s one decision that you’re particularly proud of?
Glyph: Well there are two, I guess. One was making Twisted open source in the first place. It was not originally open source; it was originally a component of this proprietary game that I was writing. I think that at the time – this was 1999 or so when I first started making some of the code available open source – I think that that was a prescient decision. I don’t think open source has really taken over the world yet, and I glad that I saw it for what it was at the time.
The other thing that I guess I’m proud of is making the license even more liberal. It was originally LGPL, and before we got too many contributors and realized this was impossible, we realized this as MIT to try to get it to be as broadly adopted as possible, which I think was a really good decision.
I guess the point at which I feel the proudest about Twisted are always things other people have done. It’s always been about enabling other people to do those things and not really about the specific decisions that I made to get us there.
Alexis: You mean in the project itself, or in terms of people who were using it in their own projects?
Glyph: Well, both. I mean, we’ve had several people who were able to really build their resume and get better jobs because of Twisted. We’ve had people who were able to build – like that system that I mentioned, the solar panels and the collectors – systems like that that just were tremendously publicly useful. It’s stuff like that that makes me happy about participating in open source, this ability to provide a good value to society.
Because often, especially because we have this charity – 501(c)(3) – you can donate to and the IRS is highly skeptical of those; they really don’t like it when people think software is a good thing, that software is a thing that mega corporations use to get rich. When some of those resources actually visibly manifest, it’s like, “No, no, this is a good thing. We’re helping.” That really makes me feel good. The fact that that keeps happening, I guess, is the thing that I’m proudest of.
Alexis: What’s the place in the world of Twisted now that – things are changing in the Python world and in other languages with evented stuff. You recently published a blog post, early this year, maybe a few months ago – it was in May, maybe – that the report of your death was an exaggeration.
Glyph: Yes, and I still think that’s true. I still think that Twisted project has a role to play.
For one thing, that was specifically in response to the event-driven I/O that is being put into the Python standard library in the 3 branch of the language. The main reason that we’re going to stick around after the advent of async I/O in the standard library is that we just have so much useful functionality. The fact that the standard library now has an event loop doesn’t – if you need IMAP in your clients and servers in your event-driven program, you still need to import Twisted somehow. And now you can use it with this standard library facility, which is great.
I think it’ll actually expand our audience quite a bit. I think it’ll make it less of an argument, because Twisted has had to simultaneously sell itself as well as our coding standard predates the Python coding standard and so it’s different in a couple of ways so it looks a little bit weird to some Python programmers. And it’s the sort of thing that programmers get very, very upset about, like the case of certain identifiers.
There’s that, and then there’s the fact that Twisted has had to sell event-driven programming, and Python hadn’t had that, and so you have to fight against the environment a little bit. You have to be careful not to use anything in the standard library, and I think that in some way this is a good thing, that Twisted is an improvement on the standard library in many ways. It’s a slightly different way to write Python.
At the same time – this criticism has been phrased by a colleague of mine, that is, the biggest problem with Twisted is you are allowed not to use it [chuckling]. And now with async I/O you kind of won’t be allowed not to use it anymore. The standard library is going to move to a more event-driven architecture. Some of those libraries that we’ve been fighting against forever, which are just terrible libraries than the standard library, like telnetlib and the ftplib and imaplib, will become at least event-driven so you could use them. You could use code that uses them, anything that wanted to just depend on the standard library; it will no longer be a major uphill battle to convince people that it’s worthwhile to rewrite a bunch of code and adopt async architecture. They’ll just have a system that uses the standard library async I/O and we’ll say, “Hey, we have the better way of modeling your authentication system and maybe you should use Twisted for that,” or “We have an SSH server and you could use Twisted for that,” or “We have these hundreds of other projects that you could pull in that use the Twisted APIs, and let it all just layer in there.”
I think in the Python community, it will become a little bit less to a different thing. Twisted also provided a lot of guidance in the process of putting that into the standard library, while it’s not quite the “we just dropped the reactor into the standard library than I would have liked.” It is very, very close and all the APIs line up, and I’m very grateful to the core development team and Guido for soliciting that advice and listening to it.
It’s when you’re trying to design something you knew yourself and trying to get your head around something, sometimes it’s very hard to listen to people who have done it before, and I think that they did a great job of doing that.
In some ways, there is a version of the narrative where Twisted is over and the whole point was just getting to the point where we could show the standard library how to do it. But in another very really way, it’s going to be around for a long, long time – graphical, desktop, GUI app integration is still a thing that they’re struggling with to implement in async I/O. It’s still a provisional API even now; it only exists for Python 3. There’s a kind of a weird backport called trollius to Python 2, and as that proceeds, we’re still five years away, easily, from Python 3 being mainstream even in the Python community.
Alexis: We could spend a whole podcast on that.
Glyph: Yeah, many have been spent [chuckling].
Alexis: I could imagine. What would you like to see happen in the open source world in general? I know this is a broad question; you might have already answered it when you got on your soapbox.
Glyph: Actually, in the open source world in general, I would really like to see –. My soapbox is what you should do, what individual people should do in the open source world, but what I would like to see happen as sort of a broader trend is I think we should look to our free software routes a little bit more.
I am not a huge fan of Richard Stallman, personally, and I think that there are a lot of problems with the way that GNU conducts its business, but I think that those problems are not what many people criticize him for. A lot of people criticize him for being extreme or for being too strident.
I think that the problem with their approach is that they failed, and they failed horribly. I think that open source right now is – free software was supposed to be about liberating users and about allowing them, the agency, to do whatever they wanted to do with their computer because it was their computer and they should have the power. That’s the whole point of the digital revolution. Open source has sort of turned into this small club for people working on backend services of systems that mostly are providing commercial services to the general public.
Lots of open source is used in these very proprietary, commercial settings – which is fine. I don’t have a problem with that. The issue that I have is, free software on the desktop – like the Linux, the GNOME desktop – is an effort that people are not paying very much attention to. Free software in phones and free software like – not just the individual like, “Oh, we’re an open source shop because our super proprietary app that will issue you a DMCA takedown if you post a copy of, contains a copy of Zlib.” That’s not really free software; that’s not even really open source. You used the thing and you got it for free – good for you.
I think that the important thing for people to look at, the thing that I would like to see change is figuring out some way to get back to more environments that are free all the way to the user, and they’re user-modifiable as a result of that freedom. We have the ability to hack on our desktops, and then we have the same desktop as we have in the cloud, and that can be stuff where –.
A lot of people working on open source right now have a Mac – that’s great. I have a ton of Apple stock here, we should keep buying Macs; that’s wonderful. But the kernel on that OS is open source, which is amazing, but there are still not that many open source tools. If you wanted to do graphic design on a Mac, you’d buy Photoshop, right? I mean, that’s where you would go. You can download the GIMP and it kinda looks okay, and you can download Xscape and run X11, but that’s just a terrible user experience.
I think that the open source user experience should not just be adequate; it should be great. I’ve tried to do what I can in terms of making Twisted work on Windows and make good support for the Mac event loop.
The place that I see this the worst is that our build toolchains are increasingly focused on servers and containers. Like Docker – fantastic technology; everyone should use it. But if you depend too heavily on Docker, if your application and particularly your libraries are architected around assuming your Docker style environment, you can no longer run on Windows and you can no longer run on a Mac desktop as part of a Mac app. I think that those are important because a lot of users are there.
If we want the next generation of open source developers to grow up with an appreciation for why they would do stuff in their spare time, why do we do this stuff for fun, I think it has to be the kind of thing that you could do outside of work and the kind of thing that you would do outside of work has to be something that you can use on your own, by yourself.
Alexis: It’s not living in a server somewhere.
Glyph: Yeah. Some of this stuff could live in a server just fine; some of this stuff could live in a browser, but if you want to prevent that dystopian future that Richard Stallman talks about where all of your books are controlled by this one weird company that hosts thousands of servers – I mean, that’s a weird idea, right? That all of your books would be sold in the Cloud and controlled remotely?
Alexis: That’ll never happen! You know what? We could make a game about it using Twisted.
Glyph: [Chuckles] Yeah, unencrypted epubs as a game mechanic.
I think that in order to facilitate that kind of thing, to keep it fun and to keep young people getting interested, and to make sure in some sense that there’s a hedge against large corporations controlling our entire computing experience.
One of the best things about the computing environment right now is that you have Apple, Google and Microsoft all in this three-way standoff where nobody really had total control over that platform anymore, and that’s important. That’s the kind of thing that I think the free software movement has ignored, is the fact that corporate interest matter a lot and the real consumer electronics environment matters a lot, and we need to pay attention to that stuff and adapt to it.
Alexis: Now a few quick questions because I know you got to go soon.
Riding on the coattails of this last question: What is an open source project that you admire, or that you’re currently really excited about?
Glyph: Wow, there’s so much stuff going on right now; it’s just so exciting. I’m learning about new stuff every day.
One that I’m very excited about is the Cryptography Project. The Python Cryptographic Authority is working on a library to make cryptography more accessible to Python programmers. I can talk at length about why that’s great, but it’s great and you should check it out: https://github.com/pyca/cryptography
Other projects that are really cool – jeez. I’m struggling to highlight just a couple. I mean, Docker is a very cool project. It’s really kind of amazing that they’ve managed to make that a company and a project at the same time and get all that code out there involved with the community so much.
Git, I guess, is another open source project that’s really cool. I have a lot of issues with the way that Git is developed and I grumble about it constantly, but it’s made distributed development fairly accessible and a lot of people who aren’t even developers are starting to use it, which is kind of a departure from where these types of systems have been in the past.
Linux – let’s remember the kernel is something that we all depend on and that is amazing. I really admire Linux in that it works at scale. They have this gigantic, globally distributed network of people; they have a bunch of companies with directly opposed interests, and yet they somehow managed to keep the whole thing running along.
Alexis: Glued together, yeah?
Glyph: Django, I think, is inspiring. Each of these projects, as soon as I mention them, I just have a litany of criticisms about what’s wrong. The reason I bring up Django is Linux has this amazing, amazing technological artifact that they’ve managed to produce, and Django has managed to produce just such an amazing community. A lot of the problems with Linux that I feel like is at risk for losing the ability to produce that technological artifact because the culture is so toxic and they really need to work on that. The Django community has taught the Twisted community many things about paying attention to minorities and to marginalized people and women, and understanding how to actually seem welcoming to those groups and not just pay up lip service. That’s resulted in a big increase in the contributions for us.
Django also has this great culture of documentation where they’re explaining everything. And again, that’s another place where Linux could really stand to learn something.
I guess this is my impression of the community is, every project – you point me at any project out there in the world and I will simultaneously admire it and critique it, because I love open source and programming so much that I really feel like everybody could get better, but I really appreciate all the hard work that everyone has put in.
Alexis: One last question here: what is your text editor of choice? I think we know the answer [chuckles].
Glyph: Oh my goodness. When I was 10 years old – we’re going back to the beginning of the interview here – my dad took me aside. This is the age where dads teach their sons to shave or go fishing; my dad took me aside and said, “Son, when you grow up, there are a lot of people who are going to tell you that vi is better than Emacs and Emacs is better than vi, and they’re all idiots. You should learn both of them because you’ll never know what’s going to be on the system that you connect to.”
He showed me how to exit vi and he showed me how to exit Emacs, and he showed me how to edit files in both of them. Ever since then, I have been a – I tend to favor Emacs because frankly, it just has more features; it just has a better architecture; it crashes less. But that is something that one only gets to know if one has used both editors a lot. I have used Emacs, I have used vi – or Vim. I’ve never actually used nvi for an extended period of time, but I have working configurations for both of them that have most of the same editor toolchain stuff that I depend on.
I think everybody should learn at least two editors – I use Sublime Text – because frankly, programming is a social activity and you should be able to sit down at somebody else’s computer and start typing in their text editor if you need to. So yeah, that’s my hopefully slightly non-traditional answer to the text editor war question.
Alexis: You should add to your – yeah, it is non-traditional, and you should add a line to your resumé that says, “Can really tell you which one is better.” [Chuckling]
It’s come to the point of the podcast where I will divulge your real, full name. It’s Glyph Not-Looking-Don’t-Contact Lefkowitz [chuckling]. I love that you put that on LinkedIn.
Glyph: It actually works. It’s the only thing that has worked. I would receive dozens of recruiter spams a week, many of which were – it was very flattering at first, I was like, “Oh, I just finally made it in the open source community. I’m recognized; I’m a household name. Look at all these recruiters! And they’re all recruiting me to work on their stupid Rails startup somewhere in Idaho for peanuts.” I put that in there out of frustration because I was tired of –. I was very politely replying to them for a while, but really started to be a drain on my time.
Alexis: Where can folks go to learn more about Twisted and follow you on Twitter?
Glyph: I am @glyph on Twitter; that’s G-L-Y-P-H. You can learn about twisted at TwistedMatrix.com or @twistedmatrix on Twitter. You can read more from me, including many of the blog posts that were referenced in this article at glyph.twistedmatrix.com.
Alexis: Alright, and for us you can find us @Binpress and myself @alexissantos. Once again, thanks for listening. And for you, thank you for being here, carving out your time at Rackspace, I imagine, maybe close-shutting the office door and putting a sign on the door saying, “Don’t enter” or something.
Glyph: “On Air.”
Alexis: On air, yes. And we will catch you listeners next week!
Glyph: Thank you very much!
Author: Alexis Santos