Show notes
Returning to the show are Python and Open Source Experts Joe Boston and Chad McIntyre. Keeping up to date on how Python on IBM Z is advancing and improving is important, and we've got an episode chock full of info. From packages to ports, AI to Flask, it's time to refresh that knowledge. Auto-Generated Transcript (may not be 100% accurate) Frank Welcome to the Terminal Talk podcast on mainframe and mainframe related topics. Jeff Frank Hi, Frank, It's really nice to meet you. I'm Jeff. Frank And with us today, we have two very important guests. We have Chad MacIntyre, who is the lead for Python and U.S., and Joe Basson, who is the lead for Open Source NZ. And so both of these guys have been on the podcast before. You probably remember the great episodes that they did. Mm hmm. We wanted to kind of get together and hear about what's been happening since the last time they went on. Yeah, some good. Jeff Sounds good to me. Frank You feel so engaged as the important thing. Jeff At least I'm not drawing this time. Frank Yeah. So last time Jeff actually created a doodle of me that was maybe not the most complimentary. So it really felt like it was. It was with me, so. Joe Yeah, I saw that, though. It was probably about the best Jeff could do, though, don't you think? Well, I. Frank Thought about the. Jeff Worst and not the. Frank Worst. I'm going to make it my I'm my profile picture for the company. Jeff For those in the room. Frank And that's why. Jeff I sent it to you, Chad. So you can see it, too. Frank Okay. So let's start I think we'll start with the Python stuff, because some of that was has really changed quite a bit since last time we talked and. So, Chad, you want to talk about, you know, your adventures since the last time we talked to you? Chad Certainly, Frank. So we've definitely been through a lot of growth and expansion, I would say, and exciting things going on in the Python universe. And just to kind of bring everybody up to speed in terms of some of the important things that are happening there. You know, Joe's on the call today for I'd say probably one of the things that's been troublesome. The most troublesome for everybody on the python front and those consuming packages. So you know, we'll get into that as we go down the line. So there's good news there. And the other one that everybody, I think from day one of release was asking was the question, you know, can Python run on Zips? And then the answer to that question was certainly no for quite a long time and has now changed within the last couple of months, too. Yes. So that's a very exciting development. And we're seeing a lot of people really interested in adopting Python workflows and running those things on the Atlas machines. Beyond that, you know, we've been doing our work to keep up with the community. So, you know, we're it's it's core to our mission to make sure that we've got the latest and greatest python supported on the platform. But not only that that we're making it take advantage of those special things that Z has to offer. Right. So in the latest release, 311, we've added support right within Python to take advantage of the enterprise data compression accelerators, accelerators. And so this is a nice feature because you get it for free, right? You don't have to do anything to kind of code to it or take advantage of it. If you've got them in there enabled, it will just work with Python. So lots of great work and lots of great developments on the Python side, and that's just a teaser from there. Frank Well, in Python 311, across the board was a better performing python. Are we seeing that on the U.S.? Chad Absolutely, yeah. So there's been some terrific work done in the community to rework some of the guts of the python the python implementation specifically to perform better. And so, you know, this was a concerted effort by the community to take a look at, you know, what are Python workflows, how do they run and why are we here, how can we do better at running them? And so there was a whole kind of re-organization and a breaking down of byte codes and stuff like that that happened in The Interpreter. And the question is, does that pan out into Z? And absolutely, it does pan out into Z. So this is kind of an across the board thing in terms of performance benefits. So I don't want to go into quoting numbers or things of that nature. But on the Net Corp performance benchmarks, we're definitely seeing benefits that match what the community was seeing there. Frank Yeah. So how long before Python takes over the Z OS world? Chad Oh, can I defer that one to Joe? Frank No. Joe It already has. Frank Happened. Joe We just don't know yet. Chad Exactly. I think it's it's one of those things where the roots are already there. It's just a matter of time for this time the show is. Frank Well, and that's that to me, a big part of this. And I kind of want both of you guys will weigh in on this. But in my perspective, though, the python work that we you know, that you guys have done, along with things like Zillow or U.S. Open automation utilities, see, I got it all. You know, it really starts to democratize the platform in a way that really has never happened on the platform. People now have a lot more choice and they don't necessarily have to do things the way IBM tells them they should. And so I that's a yeah, I think it's a really important piece. Imagine it. My, my future is always managing the platform with Python instead of JCL. Right. And I'm oh, pause for some listeners to have, you know, a complete nervous breakdown. Jeff That's obviously one of the big. Hearts of Python success on Z is going to come from packages being available for it because Python is a very extensible language. I'm curious what the process is to in identifying those important packages, because it's not just the big ones that everyone thinks about. It's these tiny little ones that you need that one weird bolt to make a project work. So how do you how do you make sure you have all those bolts in place? Joe A lot of it's trial and error and inexperience. And one of the reasons that we have gotten to the point now where, you know, between Chad's team and our team here in Poughkeepsie, that we're at least reasonably competent. And what we do these days is that we've made all the mistakes before. Right. I mean, we we originally started diving into the python pool in 2016. We came out with some of our AI related stuff and we pretty much did all the wrong stuff, right. We tried to shoehorn we tried to shoehorn code into packaging mechanisms and delivery channels that didn't make sense. And so what happened is you would you would come out with a product or a function that did something very useful, very cool, but the code would just sit there and rot over time because you had no capability to keep up with the open source community. Chad Right. Joe And so there were so many technical issues that were in our way. And over the years we kept identifying what these problems were. And one of the very key ones right from the beginning was to settle on a standardized first class python that we could make part of our regular language environment on us and rely on it and lean on it as heavily as we do the C compiler or any other language on the platform. And once, once, once we put that that group together and Chad became the lead of that, that that's when things really started to take off. That was the major inhibitor that we really that we really faced. And then from then on, it's like, okay, now we can tackle all the other kinds of things primarily how do you stay current? And even just two years ago, currency was all about how do we put the latest function out there, as everyone knows, right? More and more security has become the focus of the open source community. And now currency and security are tightly coupled with one another. So. So as we look forward to how do we best manage the open source community? Number one, we've got our good, solid current language to work from, which solves all a whole host of problems. And from my point of view, the next question is how do we keep up with the open source community when we've got hundreds of packages that we need to put out there? Right. And so you ask the question, how do we make sure that we've got the one little utility that is the, you know, the bolt that holds everything together is even. Frank Yes. Joe Yes. Something something profound like that, you know. And in the answers, we just spend time installing things and look for stuff that falls out. Right. And it's not actually even that hard. Python is such a flexible environment, and the package management system that's built into this ecosystem is so flexible and so useful that you go, Well, I've got this set of say, 15 different things that I want to do, but me install 15 and then look at everything that gets dragged in and maybe do some analysis on the dependency trees. And that's everything that I need to know. And you're not going to you're not going to get bitten by that. One thing that's missing, because in order for you to even have installed it in the first place, all these pieces had to be okay. Jeff So from a currency perspective, how much of that falls on staying up to date? How much of that falls on the developer themselves versus like the mechanism there they're using? Joe Yeah. The open source community, I saw a statistic that we put in several of the presentations that we've done, which I believe is really true when you when you look at all of the vulnerabilities out there that are exploited in the wild, they're 97% of all of them already have fixes waiting are already available. So what we have to do as a team is make it very easy for our clients to stay current as well. And so the way we used to do things prior to coming with coming up with a more comprehensive open source strategy was that, you know, we would we would go ahead and build ETFs and parse and go through integration testing and system testing and and all the testing that's necessary. And you spend your three weeks to a month trying to get something together when this vulnerability now will be open for 28 days. And. Who knows what kind of damage could be done in time in that time frame. So we're trying to get to the OR we are almost arrived at the point where we can we can keep up with the open source community through modern package management, the package management environment that is integrated with with Python as well. And by the way, every time we run into a really hard problem, I just call Chad and say. Jeff That's a good scalable solution right there. Yeah. Joe And he does it every time. Frank Every single thing that I've had problems with is when I install a package. It's when I have to deal with C code because, you know, I don't have the compiler set up, right? Or are we doing anything to make that easier? Joe Yeah. Oh, yeah. I don't know. Chad, you want to handle that or I'm also sure if you want. Chad Yeah, go ahead. I have a couple of things that chime in on that one. So first and foremost, Frank, you have a good observation, right? There's kind of a dividing line. There are packages that are written in pure Python, and I think for the most part, for the lion's share, those just where Joe, you can challenge me on that. But in general, and that's a testament to the Python implementation and just how good the how well the language is designed. So then you get to the packages that have, you know, these built in dependencies and these are usually the more interesting ones right side by side, get learn or, you know, these these little fiddly ones that have, you know, gives you a little driver to some specific thing. So in these cases, you need a C compiler and you need a C compiler that speaks the open source parlance, so to speak. And so, you know, that tends to be something that that would be at odds with our compiler. So your allusion to, you know, getting the compiler set up right and things of that nature, this is where a lot of the package enablement work would be, right? Okay. What options do I need to tell IBM's Z compiler Z COBOL, to make sure that this code will actually compile properly and do the right thing? So one of the big developments that's actually come there is this enablement of the clang in all of the M based C compilers and C++ compilers on Z OS. So these have done a great job at sort of leveling the paint, the playing field in terms of that dialect of C and C++ and the option spaces and supporting those out of the box without needing a lot of tweaking or changing. So that's one big thing that's happened from there. You know, we've also made a version of this compiler available for building Python packages directly. So this is actually a really good thing, right? In terms of, you know, one of the challenges before was, well, I don't have the C compiler, right? So how do I get this? And so that's my solution there. And then, of course, the final and ultimate answer is the Python toolkit is providing actual built versions of these packages for customers to install directly. So then that's the ultimate answer, right? I don't have to deal with any compilers at all. I just love it. Frank Just give me a we'll. Yeah. Joe I mean, just, you know, for those who may not understand the Python environment as deeply as the, you know, those of us in the room do. The idea is that you can ship something called a source distribution, and in order to do the install, it actually builds the code on your target system for you. And the Python community has been trying to move away from that to something called a wheel, which is an entirely built, self-contained, you know, as you would think of as an executable that you would run on your system. Right. And so when we have now come out with some of our open source offerings, one of the one of the primary features that we want to make available is that everything will be a built will. It doesn't matter whether the open source community puts out a source distribution or not, we will build it ourselves. You won't need the compiler environment, You won't need you need some runtime parts, you know, maybe, but you won't need the compiler. You won't need all the other build infrastructure to go along with it. And there's a twofold advantage to that. Number one, of course, is you don't have to keep all of this development infrastructure in your production environment. Right. And that's an additional sort of tax to maintain and keep all of that current. The other one, too, is it's a security measure. And one of the, you know, the vector vectors for attack from a bad actor a lot of times is they don't actually have to infect a particular package. They just have to put their malware in something that gets included. And then the build process itself will actually build it into the the deployable unit, the deployable package. And you know, from there they can sneak their code in and it gets past all the scanners. It gets past everything. Right. And so by building the wheels, we can give those to our clients and they can run their own scans if they want, and they'll know everything that's actually going on that system. Right. And they won't have to worry about somebody sneaking something in during the build process as they set up their production environment. Jeff That's something that's people are showing interest in. Joe I. I think they…
Full show notes at the publisher 
Our TOPPODCAST Picks 
Stay Connected
