Ram Rachum's blog

Can someone break my weak home-brew encryption?

Tue, 19 Mar 2013 18:50:00 +0200

There is a wide consensus among security experts that when choosing an encryption algorithm, it’s much better to choose a well-known public algorithm rather than a home-brew one. I completely agree with this approach.

However, I am curious about how easy it is to break a weak, home-brew encryption. When you’re faced with seemingly-random piece of data, and only told that it is encrypted with no clue as to how the encryption algorithm works, how would you even know where to start at decrypting it?

To shed some light on this mystery, I’ve decided to do a little experiment here on my blog.

I’ve encrypted a piece of information using a home-brew algorithm that I came up with and implemented in a few minutes. I won’t say much about the algorithm (since the point of this exercise is breaking the algorithm without knowing what it does) but I will say that it’s the result of several steps, each of which is easily breakable.

The data is posted below in hex. I’m very curious to see whether anyone would be successful at decrypting it, and even more curious as to how that was done.

Edit: Just to make it 100% clear: I will never use something like this for any real encryption need. This is just a puzzle. Never use anything like this to encrypt something even remotely valuable. I do not condone using home-brew encryption algorithms.

Without further ado, the encrypted data:

5A9E938B9A9BA0A0D5B9DB42EDD9A0F4
AFAF2780C9D941ED0BE6C8E6DD0DD082
05845029B203AE4A2C4169BFAEB6FDD9
AD461C685A700427C0E440EDE287FD31
7710ACA70B5C059D3B3EA6F58880EB70
4132D80D9F596BB1DC5922D3331BEFED
7D4C9146091F8093109EF21DE22B387C
10ADA38DF91FFBE4305B68B87A563447
AA97364600CC682F2328E8733BBA252E
3C7F93A1805B6593834F647B5BB34356
061B30DBE73734A62426816D49BB185D
C8C6D5D5FF87D5C6FFDD7DE4651FAB43
1F61AD0D4E5E5AD211FEB1D255E6D2BF
906D963C04DCC7BF2914263EF4866BFB
2A99CB704A568C0E23D66215C922F581
8CC31947C57B9F519161C3DBBA19C4AE
F18EE25281999E4C867B69838870811F
307BB3368D884E959EABC97092DD6975
232B8642424B66B67A326F6277231D98
18CBB796693D28BD92BE26956241589B
BC28C98C903D9D6AB976FB55D57B0711
681232107038820F0AA96BD7B11ED190
1897FA07905304EA128355A3BE409379
C2484D40DB45F3A5C535AF736F492B92
63E46B078D3B74092656E3D1BEC2FCC4
46895AAE93FE7CA6D281C84E7AEAEDE4
98F707F9829324AD9C3E7C9A4017B710
CC6DDBB048699E1346AADD538853F037
7B1E139212A1055F331CAF1027ED4955
3F0C2DA2CF2C1D72173B77287893146D
93BEA605DD46B6A3BD77A95F1FF66A56
2B8A2D11804907DAFA6F64DAFB1F81A1
D70187301CE1DA816C11D5E87FBED853
9788333E1426FCF6926236B42C72A6E0
39FBB728C37633A2AD4088D42E13E005
F544D0ACE4ED0AF8EDC82D0F0D5CBE07
93096E3AB1484A2E1622077E458CE9B0
D3BB4E128DB3112706E1E55EC85D41DC
164C8177D296BCCB87FF18295E54F0A7
3A09ED54D9DC0406D97B4E2529BFB3A4
22484AE361AC1970818219409B464A9B
2BD4269F4B5901A389DF30CF7A1D6CF5
8BB39C6BA76B0B0EAEF64F516C15E166
D9BE6A256985601FBD1B2FC7FA0DDFF6
7D4989B264DE32A13B6A9385E0581D90
DB10B0C71FE99C9121B0AEC58303C61B
B4F74F0E35FB4D1AE38836630C368A92
C1C4E5ACBB4B1B390246503A174ACFEA
08674C50BA7347558BC4F967BF154EDE
67ECEABEF97CE930F2539906F8A97B8C
3DEBE7C0AC8EFEDC9AC25AE1FDA9C941
3B16DECC5EDB562CE7AC386AB25D9662
275FE07DC8F59468C3C918FEC8BC6830
A3DE122885D834104F71707458D91917
6105614B1156884C5CD9D1BBCC1F25D2
6EE91F048FB9F9D3B9504B85D25CB317
36A5030C8EFD2EDA90C06FFA0AA44966
1B6259678FADFE8C27820398E867A8BA
45438A4802828ACFA8C0B6A89C32ED60
8286FD1125E99DD4B40AFFC3306E9BB9
242D7F4E026FA448A00D8D3F2E948D09

Concise explanation of why programming is fun

Sun, 03 Mar 2013 15:39:00 +0200

I’ve been asked many times by friends, acquaintances, family members and potential romantic interests the question: Why do you like programming so much? What’s so fun about programming?

Naturally, the people who usually ask this are those who do not have a technical background, and that makes it a bit difficult to answer without giving them a crash course in programming.

Here is my attempt at a concise answer that expresses the joy of computer programming without assuming any kind of technical background.

Imagine you’re building a house with brick and mortar. You lay down one brick. You lay another next to it. You lay yet another next to the second one. You keep laying more and more bricks, until you fill up the entire length of the wall. Now you start on the second line. You lay the first brick. You lay the second one. And on and on until you complete the second line. Then the third line and the fourth and the fifth, all the way up to the top. You finished building a wall! Now to build the next wall in your house. You lay down a brick. You lay another next to it…

Now, imagine the same exercise, except this time we’ll do it similar to how a software developer writes a program. You lay down one brick. You lay another one. You keep laying bricks until you finish the line. Achievement unlocked: You now have unlimited brick lines. You lay those brick lines on top of each other until the wall is complete. Achievement unlocked: You now have unlimited walls. You lay down 4 walls next to each other. Achievement unlocked: You now have unlimited houses. You build 100 houses in the same area. Achievement unlocked… Ad infinitum.

Of course, the metaphor is not perfect, and not all is happy-fun-time in programming-land. But I think this hits pretty close to what gives me the thrill about programming.

On changing the world

Sat, 09 Feb 2013 15:53:00 +0200

I’d like to share a story from my high-school days with you, and then a few thoughts that followed from it.

Tuning the telescope

When I was in high-school, we had a fantastic teacher of Physics named Bernd. I loved that guy. Bernd would routinely organize extra-curricular activities to learn about science outside of the classroom. One evening Bernd took the entire class to the rooftop of the school building to use the school telescope. (It was a boarding school so it was legitimate for there to be school activities in the evening.)

He took us to use the telescope to look at the moon. (There might have been some kind of notable astronomical phenomenon happening at that time, I forget.) Since it was a wealthy private school, we had a high-end telescope with a myriad of dials to fine-tune the focus, zoom, and God knows what else. Bernd tuned the telescope carefully, and then we all lined up to look at the moon through the eyepiece. There were a few dozen kids on one telescope so we all had to wait for our turn to have 10 seconds to look at the moon before the next student would take his turn.

The funny thing, what I remember the most from the story is not seeing the moon, but what Bernd said before he let us use the telescope. This was not Bernd’s first time taking a class up to use the telescope, and experience taught him to give the following warning: “When you get your turn with the telescope, please do not touch the dials. There’s no need to fiddle with them. I already fine-tuned it perfectly. Every change you make will only worsen the picture and the students after you will suffer for it. I’m sorry that I have to give you this warning but for some reason people always feel the need to fiddle with the dials themselves regardless of the fact that the picture is perfect.”

That warning made me think. I think that there’s a lesson to be learned here that’s bigger than dials and telescopes.

Making a difference

People have an urge to change and to control, even if the change that they’re making is for the worse. They don’t care. The urge to feel that their actions have an impact on the world around them is bigger than the desire for the world to be in a good state. Not unlike a teenager spray-painting a tasteless piece of graffiti on a wall, people want to leave their mark, even if the world ends up uglier because of it.

It makes me think about all those people out there trying to change the world and make a difference. I live in a big, fashionable city in a first-world country, surrounded by people who belong to the wealthiest 10% of the world population. It seems that you can’t find a single person who doesn’t want to change the world and make a difference in some way or another. Everybody has their own pet causes. One guy wants to save the environment, one girl wants to stop government corruption, another guy wants everyone in the world to learn how to meditate and achieve enlightenment. Then there’s world peace, universal healthcare, helping people in third world countries…

Given that practically every person I meet wants to make the world a better place, and those are relatively wealthy people by world standards, you kind of have to wonder why the world didn’t become that better place already. In fact there seem to be more people who have a strong opinion on how the world should work then there are people who know how to live their own lives happily and be content with themselves.

Is there somewhere out there an evil, chubby, mustache-wielding cigar-smoking businessman who is sociopathically apathetic to all the bad things in the world around us and who cunningly sabotages our efforts to fix the world? I don’t think so. It’s all us.

I used to want to change the world too. I cared about education. I wanted to make a worldwide revolution in education, in effect doing something similar to what Khan Academy is currently doing, except it was back around 2006, when Khan Academy was just Salman Khan posting video lectures for his cousin. My partners and I made a nice pilot in Israel but failed to get funding and the project was abandoned.

Now I don’t care anymore. I don’t want to change the world or make a difference anymore. I still like helping people, but only the people who are close to me. I care intensely about my own life and the lives of my loved ones, but I don’t try to have an impact on the world outside of that tiny circle.

Now, this is the part of the essay that I’m not sure how to write. I know that by this point, many readers are going to be angry with me for suggesting that people who try to make the world a better place are in fact a selfish bunch who are just looking to get their own personal satisfaction. I can feel the angry comments coming my way.

But yes, that’s indeed what I’m saying: Almost all the people who are trying to change the world are in it for their own satisfaction, and they don’t care if the world ends up a worse place.

That one guy who meticulously collects plastic bottles to recycle to help the environment, bought a smartphone that was manufactured by children working in terrible conditions in a third-world country. That one guy who wants everyone to meditate and become enlightened went to a strip club and fed money into an organization that degrades women and turns them into slaves. That one girl who wants to stop government corruption has her money in a savings account that’s invested in companies that abuse the environment to make a profit.

People help the world in one way and then abuse it in 99 different ways. In fact the western man is a genius; he managed to rape the world while still maintaining that he’s the one trying to save it.

What’s a perfect world?

I know that some people are going to say, “Your story about the telescope is not a good metaphor, because while the telescope in the story was tuned perfectly, our world is not even remotely perfect.” To which I answer, why not? Why don’t you think our world is perfect? Because of all the suffering and ignorance around us?

What would a perfect world be then? If a genie offered you to change the world exactly as you want it to the smallest detail, what kind of world would you ask him to create?

Perhaps a world with no suffering at all? A world in which people of all countries can make a decent living, get free education, be always kind and supportive to one another, and never hurt or be hurt by another person? Did you ever watch a movie in which nothing bad ever happens? I never have. It’s notable that even in cinema, where humans can literally create whichever fantastic world they want, nobody wants to see a world in which there’s no suffering. Even movies intended for little children always have the character go through some kind of conflict or danger. The only thing closest to a movie with no suffering that I can think of is Teletubbies, and even that one has episodes in which the characters have to deal with some kind of mishap.

Without suffering there’s no point to life. Without a possibility of failure there is no excitement.

Then you might say, “well, we don’t want a world where there is absolutely zero suffering. We just want there to be less suffering than there is today.”

So clearly, there is such a thing as too much suffering. And as I elaborated above, there is such a thing as not enough suffering. So that implies that somewhere in between there’s an optimal amount of suffering that’s just right. Why do you think that the current amount of suffering in the world is higher than the optimal amount? The current amount of suffering in the world is lower than it’s ever been in the history of civilization. Why do you think that the optimal level is still lower than this? Are you such an expert on the workings of our world that you think that you know what the optimal amount of suffering is? Sounds like a monumentally arrogant thought to me.

So the argument that the world isn’t perfect is invalid.

Conclusions

What I take away from this is: Don’t try to change the world. Live your life. Be happy. If you’re reading this you’re probably one of the lucky few who are well-off enough to have an Internet connection, not to mention the fact that you can read, which already makes you more privileged than 85% of people on the planet. Have fun with the precious few decades that you have to live.

If you enjoy being an activist, if it’s fun and satisfying for you, that’s great, keep it up then. But don’t try to pretend you’re doing it for the world. You’re doing it for yourself, and regardless of whether it’s making the world better or not. It’s just how you like to live.

First phishing attempt that targets Israeli bank

Thu, 07 Feb 2013 17:11:00 +0200

Over the years I’ve gotten thousands of phishing attempts to my mailbox, but now for the first time I get one that’s masquerading as an Israeli bank, Bank Leumi:

The copy looks surprisingly good. To those of you who don’t read Hebrew, it’s pretty standard stuff: Due to some unspecified security problems, we need you to give us all of your secret banking information, kthxbye.

But of course, if the lack of punctuation marks didn’t give this away as a phishing attempt, the link to http://adf.ly/IjI32 surely does.

That webpage, intended to collect the victim’s personal banking information, also looks surprisingly legit:

Of course, the domain mpibullion.com doesn’t belong to Bank Leumi, so there’s no doubt that this is a scam. The link to Verisign simply links to the certificate for Leumi’s actual website.

All in all, a well executed scam.

Phone companies: Please send me my voice messages via email

Tue, 06 Nov 2012 17:55:00 +0200

I really hate checking my voice mail. It’s fucking retarded. I get a text that someone left me a voice message, and then I have to call the special voice mail number, navigate the painfully slow menu, until I finally get to hear the guy’s message, which is most often to the tune of “Uh, hi… Uh… I guess… You’re not there. It’s Jeff… I wanted to talk to you about something. It’s about… Uh… Never mind, just call me back.”

Here’s what I want: I want to have a system where every time someone leaves me a voice message, I automatically get an email with an audio file of the message, the silent parts automatically removed from it, and the whole thing playing x1.2 faster than normal speed. (Speech recognition is an option but only as an extra because it’s not reliable.)

Asides from making the message-listening experience faster, and not making me call the special number, it’ll have the advantage that the voice messages will be easier to manage. One thing is read/unread: Email programs already have an excellent concept of read-vs-unread messages, which works much better than the phone companies’ system. Also, there’s metadata like the time the call was made and the caller, which in the current system are read to you before hearing the message, which is really frustrating. These belong as text in the email message, where they do not interfere with listening to the message.

Now, of course, I’m sure that some phone companies (like Google Voice maybe?) have implemented some of this. The problem is that (a) switching providers for something like that is annoying because the provider you’re switching to may have its own host of problems and (b) I live in Israel and most likely I wouldn’t be able to find an Israeli phone provider that does this.

Maybe if a startup would create a product that’s able to interface with any provider’s voice message system, and then email you the voice files… The problem though is that at any moment the phone companies could implement that feature and leave your startup dead in the water.

So my only hope seems to be waiting for the phone companies to pull their heads out of their asses and implement this feature themselves.

Goddamnit, I really want this feature…

Cute riddle: Four kids holding one toy balloon

Mon, 29 Oct 2012 14:57:00 +0200

This morning I was lying in bed for a while before getting up, and an idea for a cute logical riddle came to me.

Four kids are standing in an amusement park holding a helium toy balloon. Each kid holds his own end of a thread, but there’s only one single toy balloon for the entire group. If any of the four kids let go of their own thread, the toy balloon will fly up to the sky and be lost, even if the remaining three kids keep holding to their respective threads.

How are the threads arranged? Post your answers in the comments!

Update: Many people solved the riddle in different ways that I didn’t think of in the comments. Congrats! Now your next task: Generalize the problem to N kids and M balloons, find the function that gives the number of threads needed for each combination, and then send a link to an interactive 3D plot of it :)

Developers: When using Frecency, please consider the context

Sun, 07 Oct 2012 19:45:00 +0200

Nowadays there are many applications that use the wonderful concept of Frecency.

“Frecency” is a combination of “frequency” and “recency”. For example, say that I have a dear friend named Bob Jones to whom I write emails every day. I may have many Bobs in my address book, but when I start a new message in GMail and start typing “Bob”, GMail is smart enough to offer me Bob Jones as the first choice, because that’s the Bob to whom I’ve written the most often to recently.

This is a really awesome feature! It saves a lot of time sifting between non-popular choices. I <3 Frecency.

Many applications use this concept of frecency. When you start typing anything into Chrome’s Omnibar, it uses frecency to guess which website you want to visit. I use a launcher for Windows called Launchy. When you press Start-Space, you can start typing the name of the program that you want to launch, and Launchy will usually guess that program after 1-3 keystrokes. That’s a really fast and convenient feature, powered by frecency.

But… There’s room for improvement

There’s one thing about most frecency-using programs that I find annoying. Let me explain it with an example.

I’ve already mentioned my good friend Bob Jones to whom I write emails every day. I also have a business partner called John Schiller to whom I write frequently, though less than to Bob. Sometimes I mean to write an email to John, so I start typing jo in the address field, but then instead of getting John Schiller as the first suggestion, I get Bob Jones.

GMail is offering me Bob Jones because I write to Bob Jones more frequently than to John Schiller, and jo happens to be the first 2 letters in Bob Jones’ name.

But, if GMail had a better algorithm, than it would realize that I never start an email to Bob Jones by typing his last name. Ever. Therefore if I’m writing jo, the odds are slim that I’m writing an email to Bob.

How to fix it?

The way to fix this problem is to always check how certain selections are made. If I always start an email to Bob Jones by typing Bob, don’t offer that selection in a high position when I start typing Jones. Every time I make a selection, save the original piece of text that I typed in to make that selection, and then consult that data next time the program is trying to guess my selection.

I have a similar problem with Launchy. Sometimes I want to launch Rosetta Stone, so I start typing ro, but then Launch offers to launch Chrome, because I launch Chrome much more often than Rosetta Stone, and “Chrome” has “ro” in it. Again, a mistake. I never type ro when I’m launching Chrome. Never. So the smart thing for Launchy to do is to notice that so it could guess better at what I mean.

I hope that developers writing frecency algorithms would listen to this advice so we could all have smarter programs!

AeroFS is x87 faster than Dropbox in LAN syncing [Video]

Sun, 07 Oct 2012 12:35:00 +0200

I’m a long-time paying user of Dropbox, and for a few months now I’m also a beta user of the competing AeroFS service. Both services are excellent and revolutionary. But I’m frustrated with how slow Dropbox’s LAN sync is. When I put a file on my Dropbox on my desktop computer, I expect it to appear in my Dropbox on my laptop very quickly. But Dropbox takes a really long time to transfer the file, probably because it’s trying to first transfer it to the Dropbox servers instead of directly to my laptop. AeroFS, on the other hand, does it much faster, probably because the transfer is done directly.

In this video I’ve measured and compared the time it takes Dropbox and AeroFS to transfer an 8.5 Megabyte file from my desktop to my laptop. Dropbox took 12:28 minutes, while AeroFS took only 00:08 seconds. AeroFS is x87 faster than Dropbox in moving a file over the LAN.

Watch the video (and pardon my accent):

Why am I posting this?

Dropbox is one of the best cloud services out there, and I really hope that they understand just how frustrating their LAN Sync feature is. I’m sure that they can figure out a way to do LAN transfers as fast as AeroFS does them. (Or at least not x87 slower!)

I am not affiliated with AeroFS in any way— I’m just a fan of their service, and of course, there are many aspects in which Dropbox is better than AeroFS.

GarlicSim is dead, long live Python Toolbox!

Fri, 08 Jun 2012 18:42:00 +0300

I’ve got some bad news and I’ve got some good news.

The bad news:

GarlicSim is dead!

I’ve tried, and I’ve failed.

I wanted to make an awesome simulation framework. I had amazing ideas that I’ve never seen before in any other piece of simulation software. In the 2.5 years in which I’ve worked my ass off on GarlicSim I managed to make some of these ideas into working software. I was so proud of them— Especially the tree-related features, that allowed you to make a change in the world, see how that change affects the future, and then come back to the parallel timeline in which the change never happened. Those kinds of features have great research applications, but beyond that they are just so damn cool and it was very satisfying to work on them and to play with them.

But…

GarlicSim is an open-source project. And I am a human being who requires such things as food, water and a roof above my head to survive. Therefore I need to spend my time working a job that pays me money, since working on GarlicSim, as satisfying as it is, does not pay me anything.

I spend most of time working on the paying job, doing Python web development, and then there’s this thing called “personal life” that takes up what’s left of my time :)

I have to prioritize the things that I spend my time on, and I find that I care more about my job, my financial security and my personal life than the GarlicSim project.

I have therefore decided: I’m abandoning the GarlicSim project.

Unlike most open-source Python projects, GarlicSim is not a tool but a framework, and a big one at that. A big framework like that is something that is either executed with a lot of attention to detail, or better not executed at all. Nobody wants to use Big Framework version 0.6.3 alpha. A framework needs to be stable and feature-rich before people start migrating to it. And I don’t want to put in the huge amount of time needed to get GarlicSim to that stage, so there is no point in me spending any time on it at all.

That is why I decided to abandon GarlicSim.

The GarlicSim code will still be available online under an open source license for whoever finds it useful (or wishes to pick up the torch), but it’ll be marked as abandoned, as will the official website.

The GarlicSim blog, which has long ago served more like my own personal blog, has been appropriately renamed to Ram Rachum’s blog and now lives at http://blog.ram.rachum.com.

Now it’s time for the good news.

Long live Python Toolbox!

Out of the ruins of GarlicSim, a fresh new open source project was born. An open source project that, while less ambitious, has proven to be very helpful in my job as a Django web developer, and moreover I’m having a lot of fun hacking on it. I’m hoping that it will be as useful to other people as it was to me.

The project is called the Python Toolbox.

What is the Python Toolbox?

When developing GarlicSim, I often had a need for various general-purpose Python tools. One example, out of many, was the cache decorator. One of the things I needed while developing GarlicSim was a cache decorator that you can put in front of a function to have its results cached. So I did what every programmer does: I Googled around for an open-source caching package that I could download from PyPI and get it over with. I imagined there would be a nice implementation available, since a cache decorator, or memoize as it’s sometimes called, is so common that it’s often used as a case study when talking about decorators.

But… I couldn’t find one that I really liked! I’m very picky when it comes to the tools I use. For example, I remember I couldn’t find any caching module whose cache decorator actually understands the arguments passed to functions, and is able to understand that f(5) is the same as f(x=5) and thus they should be cached together.

So I wrote my own, and it became known as the garlicsim.general_misc.caching module. It was included with GarlicSim, despite the fact that it had nothing to do with simulations. It was there just because I needed that tool for GarlicSim.

And then I had more needs for more general-purpose tools. And again I was surprised with how difficult it is to find good general-purpose Python tools. So then came queue_tools, and binary_search, and context_managers and cute_iter_tools… All of them general-purpose Python tools that I’ve written and that I am very proud of, but which have nothing to do with computer simulations.

The Python Toolbox is this entire collection of tools, taken out of GarlicSim. It contains about 50 such modules, not counting the sub-modules that some of these modules have.

I’ve made the Python Toolbox 0.1 release today! Download it on PyPI, fork it on GitHub, use it in your Python project, and tell me what you think!

Have fun!

Silly Python riddle

Tue, 20 Dec 2011 14:16:00 +0200

Here’s a silly Python riddle for you.

Today I opened up a Python 2.7 shell, and ran two commands in it.

>>> f = lambda: g(???)
>>> f()

(Note that these are the only commands that I ran. You’re not allowed to run any other commands before them.)

The riddle: What’s the shortest thing you can put instead of ??? so my second command would not raise an exception?

Edit: I just posted it and people are already coming up with creative solutions! So now it’s time to say: The winning solution is 7 characters long. Try to match that!

———————————

Update: An anonymous commenter on Hacker News solved it!

The solution:

(Scroll down…)

The solution:

>>> f = lambda: g((yield))
>>> f()

Funny, isn’t it? I was surprised to see that the yield keyword can be used in a lambda function.

So when you type f(), it just returns a generator. If you’ll try to exhaust it, an exception will be raised because g doesn’t exist, but that’s a new line :)

It’s funny that in this case, Python seems to throw away the value of the lambda function! As we know, the yield keyword actually forms an expression whose value is None, unless you used the generator’s .send instead of .next. So you could also use .send to send in whatever value you want into the lambda function, and Python will just throw it away. Unless I’m missing something.

So that’s the only case I can think of where Python completely throws away the value of a lambda function.

The `coverage` module celebrates 10 years today!

Sun, 04 Dec 2011 21:28:00 +0200

Today, December 4^th 2011, marks the tenth birthday of the excellent coverage module! Here is the changelog entry showing it was created exactly ten years ago, on December 4^th, 2001 by Gareth Rees.

The coverage module is one of the most sturdy and reliable pillars of my development toolset. It’s a hallmark of a great tool that you never have to spend too much with it— When I want to check the statement-coverage of my tests, I just use coverage (invoked via its Nose plugin), it just works, and then I can go back to writing my tests and ensuring they cover a big part of my code. You don’t have to spend too much time with it for the simple reason that it very rarely breaks— Which is not a trivial accomplishment for a tool that acts on every, single, line of code in your project.

Kudos to Ned Batchelder for his excellent maintenance of the coverage module! I’m a relatively new user of coverage, having used it only for about 2 years, if I remember correctly. So long-time users of coverage may have a bigger perspective on this module than I do. But in the relatively short time that I’ve been using it, Ned has been an exemplary maintainer, and the few tickets that I created on the bug tracker received his attention. Ned also made sure that coverage supported Python 3.x before it was cool, and on every release he takes the time to create Windows binaries of coverage to help Windows users avoid having to compile it themselves.

If every maintainer was as dedicated as Ned, the Python community would be in a better place today, and we would have probably already seen Python 3.x going into mainstream.

Thank you, Gareth and Ned, for this wonderful module!

Deadline for GarlicSim 0.7 cancelled

Mon, 10 Oct 2011 13:30:35 +0200

Just when I finished one contract job and thought that I finally had time for some serious work on GarlicSim, I got a new contract job.

Since the deadline for the GarlicSim 0.7 release has been postponed so many times, I’m just going to cancel the deadline altogether. Sorry for the disappointment.

I’m angry at myself for not making the deadlines, but on the other hand I’m happy about having a bunch of contract jobs. This will allow me to spend a lot of time on GarlicSim later on. My contract jobs have already allowed me to do a bunch of upgrades and improvements of my work environment (from hardware upgrades, through time-saving AutoHotKey scripts, to ergonomic upgrades), and that really helps me be more productive, both in my contract work and my open-source work.

GUIs kick CLIs' asses

Tue, 06 Sep 2011 15:17:00 +0300

Or, “Hey programmers, did you know you can use a GUI with the keyboard?”

This post is part 1 of the Unpopular Opinions series. In this series I will present a few opinions of mine that go against what most other software developers think. If you disagree any of these posts, you know they’re working. Please stay civil in the comments— Criticizing my opinions is encouraged, as long as it’s done in a constructive and respectful manner.

One of the big things that have been bugging me ever since I became a professional developer is that most developers shun GUIs.

As a developer, I often get to hang out with other developers and see their tools and work processes— Sometimes in person, and sometimes online. I quickly found that almost all of them hate GUIs (Graphical User Interfaces, i.e. visual programs with windows, buttons, menus, dialogs, etc.) Most programmers much prefer working with CLIs (Command Line Interfaces, i.e. a textual shell.)[1]

I found this baffling. Why would anyone enjoy working in front of a command-line interface, where you can’t see shit and have to remember a ton of commands and flags by heart? Surely the programmer community is intelligent and passionate enough to pick the very best possible tools, so perhaps they have good reasons for preferring CLIs?

After thinking about it for a while, my conclusion is that there are two main reasons that most programmers prefer CLIs to GUIs:

Most GUIs suck. And when it comes to proper keyboard support, most GUIs suck terribly.
Most programmers are ignorant about good GUIs.

But before we get to that:

Let’s compare GUIs and CLIs

There are a bunch of parameters that we can measure GUIs against CLIs on. The two parameters that are the most important for heavy computer users are Visibility and Speed.[2]

Visibility

The first parameter we’re going to look at is visibility. This is the 800-pound gorilla that most programmers seem to ignore.

For the sake of an example, let’s take a simple everyday task: Say you want to copy a bunch of files from one folder on your computer to another.

If you were to do this with a CLI, you would do something like this:

 cd ~/source_dir cp file_1 file_2 file_3 ~/target_dir

On the other side of the ring, if you were to do this with a GUI, you would navigate to the source folder in your explorer/nautilus/whatever, select the three files, copy them to the clipboard, navigate to the the target folder and hit paste.

When you take the GUI route to copying files, you get to see a lot of stuff. Some of it may not be relevant to your current needs, but some of it might be. You will get to see all the other files in the source and target folders. You will get to see how much disk space all these files take. You will get to see any sibling folders that the source/target folder might have. You will get to see the file size of each of the files you’re copying.

When you take the CLI route to copying files, you’re not seeing anything except the next shell prompt waiting for your next command. Sure, if you got everything right, you wouldn’t really need to see anything, because your 3 files just got copied and you completed the task. So what do you care about seeing all the data you would have seen in the GUI route?

Because keeping your eyes open is important.

Maybe when looking around the source folder, you’d see that there was another file that you forgot to copy? Maybe you’ll see that for some reason the target folder has more files that it should, which might indicate a problem with your system? Maybe you’ll see that some impolite package-management tool that you previously ran left a bunch of build folders that you should delete?

When you take the GUI route, you’re exposed to all of that stuff. You’re more likely to catch possible problems, or have ideas about how to improve the system.

So the scoreboard for Visibility is:

GUI - Medium to High (depending on GUI quality)
CLI - Low to Medium (depending on CLI quality)

And when our task isn’t to copy files, but to write or debug a computer program, this point becomes crucial. There’s so much shit that’s helpful to see when you’re developing! If you just got an exception, you should see the running code, the entire stack, the values of all variables, documentation for the currently selected function, system I/O, and on and on. Each of these pieces of data might help you solve your problem in seconds instead of hours.

Now sure, you can obtain all of that data on the shell too. But having to explicitly ask for it by running a command is such a chore! And what more, every time you ask for more data, it pushes the previous data up and away, as if that data wasn’t important anymore!

Do you know what this reminds me of? It reminds me of old text-based games, where you would tell the computer “Go west”, “Look around”, “Fight monster” etc., and it would tell you things like “You have killed the monster” or “You are in a room with exits to the north and east.” Aside from the nostalgic feelings, playing a game like this totally sucks! You want to see how many exits the room has! You want to see the monster you’re killing, in all its gory detail! And this is true whether that monster is a grue or a segfault.

Speed

Now we’re getting to our second parameter: Speed. The opposite of speed, when it comes to computing tasks, can be called friction. If you have to click in seven different places to finish a task, then that’s a lot of friction, and that makes for a slow process and a sad user.

Speed is obviously a crucially important parameter. When you spend 8 hours a day doing tasks on the computer, the speed with which you do the tasks makes for a night-and-day difference. Sometimes you manage to shorten a 10-second task to a 2-second task, which is nothing short of an orgasm for a productivity-minded developer. But even cutting a task down from 3 seconds to 2 seconds is a considerable improvement. Remember— We’re sitting in front of that freaking computer all day long.

It’s important to note that speed is important not only because of the time you save, but because computer tasks are usually done in order to achieve some greater goal. That greater goal often requires a bunch of subtasks, and as each of these subtasks becomes more complicated and slow, you lose sight of your original goal. Doing tasks fast allows your train of thought to go faster.

Okay, so how do CLIs fare against GUIs in the parameter of speed?

Speed is the old standby of CLI-loving programmers. “CLIs are so must faster than GUIs! A GUI is so cumbersome, you have to click here and click there, navigate through menus… With a CLI, you just press a few keys, hit enter, and presto, it’s done!”

I beg to differ.

The scoreboard for Speed, which I will promptly explain, is:

GUI - Low to High (depending on GUI quality)
CLI - Medium to High (depending on CLI quality)

The CLIs’ score of “Medium to High” is, on average, better than the GUIs’ score of “Low to High”; but when it comes to software, where copying costs are minimal, the best program tends to make all the other alternatives extinct. So in the long run we care more about the best program rather than the average program, so GUIs and CLIs end up pretty close to each other on the all-important speed parameter.

“But wait,” the Vim fanboy cries, “how the hell could a GUI have anything close to a “high” speed rating?”

If you don’t know the answer to that question, I want you to really listen to what I’m about to say next, and try hard to remember it.

Because GUIs can be controlled with the keyboard!

Phew, that felt good! Now let’s explain.

The reason that programmers see GUIs as so much slower and more cumbersome than CLIs is that they try to operate them with the mouse. Working with the mouse is terribly, terribly slow. When you’re working with the mouse, you don’t have muscle memory. You always have to look at the screen for the next button or next menu item that you want to click. Not to mention that the mouse is more conducive to RSI than the keyboard, and this can be a serious health problem for heavy computer users.

To take a simple example: After you’ve learned to copy-paste by pressing Ctrl-C and Ctrl-V, you don’t need to think about them anymore; those movements become second nature. Your body and mind are good at memorizing these kinds of tasks, and before you know it you’ll be Ctrl-C‘ing and Ctrl-V‘ing without even thinking about it, like a pianist doesn’t even think about which piano keys he’s hitting. But if you were to try this with the mouse, it won’t work: You’ll always have to be on the lookout for the context menu after right-clicking, and then visually search for the “copy” item and click it… Friction friction friction.

Not to mention that when you’re using the mouse, you constantly have to move your right hand from the mouse to the keyboard and back, and that sucks.

And programmers often do tasks much more complicated than copy-pasting, making this point so much stronger.

So we all agree: Mice suck, keyboards rock!

But, but, but… A good GUI can be controlled with the keyboard! A good GUI can be operated completely with the keyboard, and has good keyboard shortcuts for the most common tasks.

For example, let’s look at a program I use everyday and has both a GUI and CLI interface: Git. Say I want to check-out a branch, something I do dozens of times a day. If I’d want to do that with the CLI, I’d type git checkout master. With autocompletion in place, I will have to make exactly 12 keypresses (namely, git chec{TAB}m{Tab}{Enter}.)

What about the GUI? Sure, if I’ll try using a mouse it will be an epic fail, but with the keyboard I can do {Ctrl-O}ma{Enter}{Enter}, which makes for exactly 6 keypresses. It’s twice as fast at the CLI version, i.e. half of the friction!

Now sure, some tasks can turn out to be faster on a GUI and others can be faster on a CLI, but when you pit the best GUIs against the best CLIs, they’re in the same league.

So why do programmers still prefer CLIs?

Now we can get back to the two items I mentioned at the beginning of the article, and expand on them. Those are the two main reasons that programmers prefer CLIs to GUIs:

Most GUIs suck. And when it comes to proper keyboard support, most GUIs suck terribly. It seems that most developers of GUI apps don’t care so much about keyboard-accessibility. It’s all too common to see desktop apps that simply cannot be controlled with the keyboard. Buttons that cannot be pressed, menus that cannot be opened, scrollbars that cannot be moved with the arrow keys.

This is annoying enough in mainstream software that’s mostly intended for non-technical people, but the astounding thing is that many open-source programs that are made for developers by developers are not keyboard-usable!

I find this completely shocking. The same programmers that preach about how Vim is so much more productive than an IDE because you get to use the keyboard, suddenly put on a different face when it’s time for them to make GUIs, for other programmers no less!
Most programmers are ignorant about good GUIs. Because some GUIs are good. Some GUIs have great keyboard support. Microsoft is one company that really shines in this regard. I challenge you to find one action in Windows, or in any other Microsoft product, that can be done with the mouse but not with the keyboard.

So good GUIs exist. They can be hard to find, but hey, a lot of good software is. Google Chrome has good keyboard support. Mac and Ubuntu have okay keyboard support. Wing IDE has good keyboard support. PuTTY has great keyboard support.

And of course, these GUI programs usually have much better visibility than their CLI counterparts, making them more productive programs overall.

Final words

I want to stress out that what I did here is a comparison between the very best GUIs and the very best CLIs. It takes time to find good GUI programs that can be operated with the keyboard. That’s tough. It takes time before you can surround yourself in excellent GUI tools to help you do your job. But it’s possible, and eventually, it’s the optimal way to work.

————————————————————————————————————

[1] - Actually there are some communities of programmers where most of the members like GUIs, but these are usually communities of developers that work in environments like Dot-Net and Java.

[2] - One parameter that is very important for non-hardcore users, and still somewhat important even for hardcore users, is Discoverability, i.e. being able to easily see all the different options and commands that a program gives you. GUIs obviously beat CLIs big-time in this parameter.

Quick riddle

Sun, 04 Sep 2011 22:12:00 +0300

Here’s a quick riddle:

You owe your friend two Dollars. You have a $5 bill, but your friend has only a $1 coin for change.

What would be a fair way to settle your debt that would not require you to get more change?

Answer: You toss the coin. If you win, you don’t give your friend anything. If your friend wins, you give him the $5 and he gives you back $1, so he gets $4. His expectation is $4 * 1/2 = $2, which is exactly fair.

Thanks for all the people who posted their answers in the comments!

A Git user's first (and hopefully last) foray into SVN-land

Tue, 30 Aug 2011 14:36:00 +0300

I belong to a rare breed of programmers: I was born and raised on Git. It’s the only version-control system that I’ve ever used. This is my personal story of how I had to use SVN temporarily for a contract project.

Now, I know there are a million of blog posts out there saying “Ooh, look how much better Git is than SVN!” I’m not trying to tell people something they don’t already know about Git, SVN, and the difference between them— I’ll just be telling my own personal story and my own impressions of SVN.

Beginnings

I started my programming comeback about 3 years ago. When I was a child, I programmed a lot— I was taught Basic in third grade, and I spent a lot of time of my elementary school and junior high making various little programs in C and Pascal. But I took a long hiatus from programming starting at about 10th grade, because I was frustrated with how many stupid technical details were involved in programming.

That hiatus ended about 3 years ago as I started learning Python, programming computer simulations for my Physics research, and about a year after that became a professional programmer. By the way, the main reason I chose Python is because Paul Graham mentioned it favorably in one of his essays.

Starting out with Git

How did I start out with version control? I saw the term “GitHub” being thrown around a lot on Hacker News. It seemed to be something everyone liked, so I figured I’ll try putting my project GarlicSim there. I learned Git. It took a while, because I was still not very technical, but eventually I got the hang of it enough to work in a single branch.

After a while, I understood what branches meant. I understood that I was able to create alternate versions of my code, work on them as much as I want, and decide at my leisure whether they should be merged into master and become an official part of my project.

I was happy. Branches are awesome.

I heard about SVN…

At some point I heard about SVN. I heard that it was something like a predecessor to Git.

What I imagined in my mind was that SVN was to Git what Windows 98 was to Windows XP. I imagined that SVN was just a crappier version of Git. Like, same idea, same workflow, just with more annoyances and less polish.

That perception changed when I got my first programming job.

First contact with SVN

Then I started programming professionally. I was hired by an Israeli startup to work on a Django project. I was kinda surprised to learn that they were using SVN. I figured that a technology startup would, you know, care about technology. But I said, “Meh, so I’ll just have to use Windows 98 for a while instead of XP, I could live with that.”

Oh boy.

So I was just about to learn SVN. This is a dramatization of the conversation I had with that company’s chief developer: (I ask you to read the text in italics in John Cleese’s famous surprised voice.)

Me: Hey man, I wanna make a small change to the code to add [Feature X], I’ll just make a little branch, alright?

Chief developer: What?! You’re gonna make a branch?!

Me (taken back): Eh, yeah, a branch. So, you know, I could make changes in it without affecting our master, or trunk, or whatever?

Chief developer: And how are we supposed to put the changes back into our trunk?!

Me (totally freaked out at this point): Eh, merge them back?

Chief developer: Merge? MERGE?! That’s a mighty big word for someone your size!

And then followed a conversation in which he explained to me that branching-n-merging in SVN is really fucking complicated. I don’t remember exactly what he said was required, but it changed my perception of SVN from:

“SVN is a slightly crappier version of Git.”

To:

“SVN is a version of Git in which every time you want to start a branch or merge it back, you have to sacrifice a young virgin to the source-control gods.”

I was not happy.

Luckily, the chief developer was considerate of me, and the company was small enough (only a handful of people), so he agreed that we’ll just move our entire product to Git.

I was saved at last moment from the atrocities of SVN.

Until now.

SVN strikes back

Recently I got a nice contract job. And, like any other contract job after my initial trauma with SVN, I made sure to ask that the job was in Git before I started the job. Unfortunately, despite being told that the job was in Git, it turned out that the job was in SVN, and that we’d move to Git “after the deadline.”

I was not happy. It turned out that I had to learn SVN after all.

I had my brother give me a crash course on SVN via phone. My brother’s a C++/Java programmer, and he had to use SVN for a while, so he agreed to help me with it. He has experience with Git so he was in a good position to explain the differences between the two. I primarily remember him explaining to me about branches.

Now, I remember seeing the “branches” folder in the SVN repo. I thought it was pretty weird to keep branches in a folder, but whatever. I assumed that the stuff in the “branches” folder was like the stuff in the “.git” folder in Git: Magic data that contains all your branches and allows you to switch to them at will.

Then my brother explained to me that this is not the case. I remember this part of our phone conversation pretty well, because at that point I was sitting on my knees on the carpet, with my head in my hands, looking down into the seat of my office chair. He explained that SVN doesn’t really have branches, it’s just a folder called “branches” that you manually copy your code into.

I was not happy.

Though, on the other hand, it was kind of a relief; it was relieving to know that SVN doesn’t even try to do branches. Because God knows how badly they would fuck it up if they would have tried to. Copy-pasting folders is major bullshit, but at the very least it’s predictable; when you use mv or cp, you know exactly what you’re getting, which is something you can’t say about svn.

That partial relief nonwithstanding, the whole “branches” thing in SVN is utterly ridiculous. In the past when people told me “SVN has branches too!” I knew that something wasn’t right and I just assumed that SVN branches suck, but now with this new knowledge I can answer in confidence, “No, SVN does not have branches, it merely has a folder called “branches”, which is a far cry from having actual branches.”

You know what SVN’s “branches” folder is like? It’s like taking an A4 paper from your printer, drawing a Flux Capacitor on it, hanging it in your car and declaring with a satisfied grin, “My car’s a DeLorean time machine! I can go back in time!”

SVN Ignore

One other thing I had to deal with in SVN was doing “SVN ignore” (I don’t know the SVN jargon for it.) Basically I was looking for the SVN equivalent of .gitignore: A list of file patterns that do not get source-controlled, ever. This is very useful for build products, files containing passwords, user-uploaded static content, etc.

Now in Git, you just have a file called .gitignore in the root of your repo, and in that file you write the file patterns that should be git-ignored, like *.pyc or build. You save the file, and that’s it. I assumed that there would be a similar mechanism in SVN.

You can probably guess where this is going by now.

It turned out that SVN’s ignoring mechanism is far more complicated than that. It basically holds a list of SVN-ignored file for every, single, folder in your repo, and there is no convenient way to edit all these rules for the enitre repo, or even just view them. You have to run this baffling command in order to even view the ignore rules. Also, when you edit the ignore rules for a folder, you’re not just editing a text file like in Git, you have to actually set the rules using svn. I tried actually going into the dreaded .svn folder and edit the text file that holds the ignore rules myself, but it was encoded in a really stupid encoding, and in order to write rules there manually I would have had to count the number of characters in my ruleset and type it as part of the text file.

Back to Git

Eventually my new client saw how bad of an effect SVN was having on me, and he let me move us to Git. So over the weekend I used svn2git to convert our repo to Git and now we manage all our code in Git. Uninstalling TortoiseSVN from my machine was quite a happy moment.

Goodbye SVN, may we never meet again.

Postponing GarlicSim 0.7 yet again

Mon, 08 Aug 2011 23:23:31 +0300

Yeah, the deadline for GarlicSim 0.7 has been an epic fail. A couple of weeks ago I thought I had it under control, but then I got a contract job that’s been taking all my time. Good news for mah bank account, bad news for mah open-source project.

So I’m postponing the 0.7 release to November 1st, 2011. See you then.

Postponing GarlicSim 0.7 again

Wed, 20 Jul 2011 20:43:57 +0300

I’m postponing the deadline of GarlicSim 0.7 again, this time to August 20th.

This is fucked up and I feel really bad about this. I should have been more realistic about my deadlines, but I’ve let my optimism have the better of me. I’ll try to be more accurate in future deadlines.

Why making a cool project is a good idea for an aspiring software developer

Sun, 17 Jul 2011 00:34:00 +0300

As a software developer, I get to know a lot of other software developers. Many of the software developers I know are either running a startup, working as an early employee for a startup, doing contract jobs for a startup, or dreaming of one day starting a startup. My point is that software developers are very interested in startups, either starting them or working for them.

My advice to anyone who’s interested in startups: Before you ever start a startup, try to make at least one cool and useful project. It doesn’t have to make money; in fact, it’s better if your project doesn’t even try to make any kind of profit.

Why is making a cool project important?

When you’re doing a startup, you essentially have to do two things at once:

Create some kind of product or solution that answers your users’ needs; this is usually the software ingredient of your startup; and
Create a business around that product that brings in money for your company.

Both of these things are difficult. Really difficult. It’s easy to mess either of these up. And you have to do them both at the same time. Get one right but mess the other one up, and you still failed.

Here’s a metaphor: Starting a startup is like juggling while riding a unicycle. Riding a unicycle is hard. Juggling is hard. Doing them both at the same time? Really freaking hard.

If your goal in life was to juggle while riding a unicycle, would you get on the unicycle while holding a few juggling clubs and try to juggle while riding the unicycle? Not if you have any common sense. You would first try to ride the unicycle. You’d take your time until you’re sufficiently skilled with that. Ditto for juggling. Then you’d carefully try to do both of them at once.

The same applies to startups. Making a successful product is hard enough; it’s all too common for companies to invest a lot of resources in making an ambitious product, and when it’s launched it’s just “meh”.

Building a business on top of a product is hard as well. There are many places where you can screw it up, both by making technical mistakes or strategic ones.

As you can guess, the chance of you messing either the software side or the business side is very big.

Start a project, don’t try to make money.

That’s the essence of my advice. Start a project which is free of the burden of profit. Sure, you won’t get rich off of it, and you’re gonna have to support yourself financially while you work on it, but it’s your best chance at learning how to make a successful product— It definitely has a much better ROI than going to college.

Not being burdened with having to make a profit will make it easier for you to build a successful product. You’d never have to think about limiting or exploiting your users, which is something that many software businesses have to do. Think about those crippled “starter” versions of Windows that sell for cheap, or DRM, or annoying ads on websites, or desktop software that installs annoying toolbars and spyware. These are all attempts to make money by limiting or exploiting users. Software needs to act like a butler, which is difficult enough; imagine how hard it is to have a good butler who also tries to upsell you some crap as he serves you.

As a person who’s making a project that doesn’t aim to make money, you’ll be free from all of those dangerous practices.

Making a useful product is still hard, mind you, and you still have a good chance of failing; but at least when you crash from the unicycle and try to pick yourself up, you won’t have juggling clubs falling on your head.

`SimpackSelectionDialog` almost done

Tue, 12 Jul 2011 01:13:00 +0300

Hey, only a few weeks remain until the release of GarlicSim 0.7. I just wanted to give you a sneak peak of the new SimpackSelectionDialog, incidentally on a Mac:

(Click to enlarge.)

I’m quite pleased with this widget. It’s almost finished, I might put in a few more touches before the release. The above screenshot shows it on a Mac, but as always, it’s completely cross-platform and works on Windows XP, Windows 7 and Linux as well.

This time I put in more emphasis on keyboard-accessibility, including accelerators for as many buttons as possible. The widget is not 100% keyboard accessible, but I’d say around 90%, and it will get to 100% in a couple of years as the underlying wxPython framework evolves. I also went through all the other dialogs in GarlicSim and gave them as many accelerators as possible.

Going back to work on 0.7!

DreamPie: My favorite Python shell

Sat, 11 Jun 2011 23:56:00 +0300

If you’re not familiar with it, you should really check out DreamPie, which is a graphical Python shell.

It’s a really well-done GUI shell, with great keyboard support to boot. Its most unique feature is the separation of the “history area” from the “current command area”. I think this design makes more sense than the traditional approach of having them together. It’s also notable that the ”current command area” can be used to enter multi-line Python commands (like defining a function or a class) and it’s very easy to move between lines to edit them. (This sounds obvious but many other shells make this kind of editing cumbersome.) The autocompletion is also well done; for example, DreamPie is able to autocomplete modules even if they haven’t been imported yet.

On top of that, I was delighted to find that it works on PyPy out of the box! Check it out: