Python vs. Java: An update to a subjective speed comparison

Yesterday, one of our engineers here at SnapLogic pointed me to this page here. That’s a simple speed comparison between Python and Java, done by Glyph Lefkowitz some time ago. The tests consist of small code snippets – written first in Python and then in Java – each exercising a specific area such as object creation, loop performance, etc. It’s clearly subjective, and some of the test will depend on many other factors. Still, it’s an interesting set of tests.

Two things are remarkable about those tests:

  1. The Python code is significantly more compact than the Java code. Completely independent of the actual run time performance, this means that the programmer’s productivity should be higher, with less lines of code to debug and maintain later on. I have not replicated the code snippets here on this page, but do take a look at them on Glyph’s page. It is truly impressive.
  2. The tests indicated that Python was significantly faster than Java in some cases (and also much slower in others). I was assuming that Java should be able to run faster than Pyhton in most cases, though.

The comparison was done with Python 1.5.2 and JDK 1.1. Clearly not the latest and greatest version of each. Glyph mentions on this page that Java has improved in speed since then, but he feels that the basic conclusions still hold.

I decided to redo several of the tests with updated versions of Python (2.5) and the JDK (Java 6). And indeed, my suspicions were confirmed: Java has made huge speed improvements, and is now faster than Python in almost all cases.

Also, I ran the Python tests with the help of the Psyco Python compiler. This resulted in speed improvements in all cases, and yielded a significant speedup for some tests. The only changes in the Python source were to import psyco, move the test itself into a function and call psyco.full() as the first statement of the program.

    Update (July 11th): The timing was done in such a way that startup cost and interpreter initialization time was NOT taken into account. This means that contrary to the original article by Glyph I am not trying to comment on the suitability of a language for piped, small scripts. Considering that we are working here at SnapLogic on a data integration server, we are naturally more interested in server performance. Therefore, I don’t mind the startup cost at all. This should be kept in mind. The actual timing itself was done quite naively: In the code I take a high-resolution time stamp just before the loop starts, and right after it ends. Then I print the difference. That’s all. Since such a timing is clearly influenced by other activities of the system, I repeated each run several times. I am aware that the results are not 100% accurate. However, they are indeed close enough to give us a decent impression of the run time performance. Note how the conclusion describes the speed comparison in approximate terms because of this.

Below now are the details for the tests on my Dell D820 laptop, Intel Core Duo, 2.16 GHz, 2 GBytes, Ubuntu 7.04:

Times are measured seconds

Test Java 6 Python 2.5 Psyco Comparison
Standard Output 29.47 27.96 25.65 Python and Psyco slightly faster
Hashtable* 0.22 0.46 0.12 Psyco 2X faster than Java,
Java 2X faster than Python
I/O 0.49 1.12 0.68 Psyco 1.4X slower than Java,
Java 2X faster than Python
List 0.10 0.45 0.06 Psyco almost 2X faster than Java,
Java 4.5X faster than Python
Object Allocation 0.14 6.75 3.17 Psyco 20X slower than Java,
Java 50x faster than Python
Interpreter Speed** 0.004 0.37 0.12 Psyco 30X slower than Java,
Java 90X faster than Python

* Please see the update at the bottom of this post regarding the use of more
modern Java features, such as HashMaps with generics, that can result in significant speed improvements.

** Apparently, Java optimizes away a loop without side effects. Therefore, I added
a simple counter calculation to this test. Even so, the speed difference is significant.

A few observations:

  • It can be seen here that the speedup one can obtain through Psyco is truly considerable, and can even be noticed on IO bound operations – probably due to the optimized loops around the test statements.
  • Java’s performance in tight loops is excellent. However, one of the reasons for this may be the arbitrary precision integers used by Python vs. the more ‘native’ types used in Java. If the Java data type for the for-loop variable in the SpeedTest example is changed from integer to long, the Java performance already drops significantly (about twice), indicating the impact that the counter type has on performance. Python’s arbitrary size integers are convenient, but they are definitely not helping its performance.
  • Object allocation in Python remains much slower than in Java. This is not at all helped by changing to new-style classes (the original code for the test used old-style classes). In fact, the new-style classes are even marginally slower.
  • The Hashtable test for Python seems to be mostly limited by Python’s slower loop performance. Java apparently has increased its loop performance significantly, since this test originally ran slower in Java when conducted by Glyph a while ago. But when the Python test is now run with Psyco, the speed is better than that of Java. Psyco would only have an impact of the surrounding loops that are constructed for the test, not on the core Hash object, which is written in C anyway. Therefore, we can see that the build-in Hash object for Python genuinely performs better than Java’s version, while Java shines with its loop performance. Update: Please see the update at the bottom of this post regarding newer Java features, such as HashMaps with generics that can result in performance improvements.
  • An interesting observation: For the Psyco tests the test loops were moved into a function (Psyco only optimizes functions). When I ran those tests even without Psyco, the performance was already improved – just because the code was now located inside of a function. The speedup differed from a few percent to 100% depending on the test. Apparently, we can see that the definition of a function object allows the interpreter to make some optimizing assumptions. Java’s code always runs inside of a function, and thus these optimizations can always be made. In the future, a realistic comparison of the Python performance with the Java performance would always see the Python test code inside of a function.

Conclusion

Python is really compact, and allows you to write code more quickly than Java. Therefore, it can improve programmer efficiency, which is often more of a scarce resource than CPU cycles.

For IO bound operations, the speed difference is much less important, which means that in most real world applications one does not need to be too concerned with the Python performance penalty.

Psyco is really good! It should be used more often. If you don’t use it you might want to take a look at it.

    Update (July 11th): Several people have pointed out that Java now provides different ways to do the same thing, which may result in better performance. For example, the use of HashMaps and generics. Or buffered IO for the console output test. I made a few comparisons and in general those changes do result in some significant performance improvements. However, the code used here for comparison is roughly the same as the code in Glyph’s original paper, without those modern additions. I may be mistaken, but frankly, I don’t think that the use of buffered IO for a simple console output test is what would intuitively come to mind. Mind you, if I think about buffered IO under Java for performance improvement then I should also consider that for the Python cases, which I am not doing here. Simple IO operations are used instead in both cases, and that is what is compared.

    However, the use of HashMaps under Java does provide a very significant speedup over the naive Hashtable example. Using HashMaps makes Java around 4 times faster than Python with Psyco in that test. I would agree with those who have submitted this suggestion that it is indeed the way to go these days in Java. In that case, the original HashTest here compares Python’s performance to a somewhat outdated feature in Java. This is unfortunate, but I don’t want to start changing the sources around now and redo the test. I won’t optimize the Python code to use new or better features, and also won’t change the Java code. The goal of the original test sources was to compare the code that initially comes to mind, even if it is somewhat naive. The code that comes to mind of course depends on the person that writes it, that’s understood.

  • Zac

    One thing to note is that in the Java code that he tested, it was not written as succinctly as it could have been. He seems to have taken for granted that python can have arrays of characters, but Java can’t. There a couple other examples I can think of but they’re not so relevant, just a point I thought I’d make. Also, sometimes succinct code isn’t necessarily the best code. Code should be verbose enough for one to understand it, though most of that python code is quite straight forward. Good post, thanks.

    ZLB

  • Zac, I agree that making code compact for compactness’ sake is not desirable. Readability is much more important. In the original source code, however, you can see that the Python code actually is quite readable, and does not use ‘tricks’, which might be difficult to decipher. I would say that the compactness of the Python code comes naturally, and without negatively impacting readability. In fact, I would say that the Python code generally is more easy to read than the Java code, because there is less to distract the reader from what it is you want to accomplish.

  • James Tikalsky

    You mention that “startup cost and interpreter initialization time was NOT taken into account,” but you don’t say whether you ran the Java code a number of times before measuring the time elapsed. Sun’s JVM (and Microsoft’s CLR, and others) do run-time optimizations, so it’d be interesting to see whether the Java code got faster over time. Have a look at “Beware of Microbenchmarks” from Sun:
    http://java.sun.com/performance/reference/whitepapers/tuning.html#section3.1

  • James: Good point. I redid some of the Java tests, this time running the test-function multiple times. It did indeed get faster, after getting slower initially (probably a hit for the recompilation). The observed speedups for some of the tests were in the range of 10% to 30%, which is pretty good.

  • Other than personal experience, what evidence do you have to support that Python “allows you to write code more quickly than Java?” This sounds very subjective and task bound. I did find the article interesting, but such a broad generalization without evidence could mislead readers.

  • This article was picked up on reddit.com, where a pretty lively discussion about it has ‘erupted’. You can find this discussion here: http://programming.reddit.com/info/24ynh/comments

  • Kenneth: Sure, a comment like “allows you to write code more quickly” is definitely subjective. However, this subjective impression is based on real world experience by myself and colleagues. Also, take a look at this older paper here:

    http://page.mi.fu-berlin.de/prechelt/Biblio/jccpprt_computer2000.pdf

    Figure 6 is the important one.

  • Isaac Gouy

    “Also, take a look at this older paper here … Figure 6 is the important one.”

    The important thing to realize about that paper is the data on Java, C, and C++ programs came from a controlled experiment /but/ the work time data on Perl, Python, Rexx, and Tcl programs was not measured as part of an experiment – the work time was reported by self-selected program authors who sent in programs in response to a usenet request.

    In short, the work times for the Perl, Python, Rexx, and Tcl programs /were not measured/ by the author of the paper.

    (see “Validity of this comparison” page 3)

  • Isaac: I read that validity section. In that very same section, however, the authors of the paper give the reasons why by and large they do consider the work-time estimates even for the ‘script group’ as relevant.

    There were a few other differences in the test setup for the two groups, but in the summary (also in the validity section) the authors reaffirm that while small differences may be accounted for this way, the overall conclusion should still be seen as valid.

  • Isaac Gouy

    “Large differences, however, are likely to be valid.”

    Kind of depends on whether the self reported times exaggerated by a little or a lot 🙂

    Remember to follow the reports conclusion and take the results with a grain of salt.

  • Isaac Gouy

    1) “Update – if I think about buffered IO under Java for performance improvement then I should also consider that for the Python cases, which I am not doing here”

    On the contrary I expect /you are/ using buffered io for Python – check by opening ‘scratch’ with 0 buffer:

    f=open(‘scratch’,’wb’,0)
    for i in xrange(1000000):
    f.write(str(i))
    f.close()

    and time the program. How much slower is it?

  • Good point! The IO performance of Java certainly diminishes if I use the non-buffered option you have pointed out. When I compare that to the non-buffered (!) Java version (the initial test), then Java is around 10x faster. This now surprises me a little bit.

    On my machine, Java takes its 0.49 seconds, and Python now takes around 5 seconds. Replacing the call to write() with a dummy function call, I can see that only 0.85 of those 5 seconds are spend in loop, object creation and function call overhead.

  • Isaac Gouy

    Psyco 1.4X slower than Java
    Psyco 20X slower than Java
    Psyco 30X slower than Java

    why haven’t you written these

    Java faster than …
    Java faster than …
    Java faster than …

  • It’s also worth noting the flexibility of Java for use with UI’s and web-based applications out of the box. With 3500 classes, Java has two main UI toolkits, Swing and AWT. It supports many things that Python can’t do out of the box, and that Python doesn’t even have as a module. It’s worth noting that Python scripts over 10,000 lines initialize quite slow just like Java, and that bytecode optimizations, although supported with Python, are not the norm and are not common.

    Coding for wxwidgets, using a QT or GTK bridge, or using TCL/TK is hardly an optimal solution when writing complex graphical applications, and Java wins in this area, despite there comically being many problems with the look and feel of Java applications. This has changed since 5.0, with drag-and-drop and other common native GUI aspects being available with Java now.

    Writing web-only applications is better with Python, mainly because of it being on more hosting providers, although Ruby is far more common than either Java or Python on shared hosts. Write once, run everywhere still mainly holds true for Java, minus some GUI kludge, but at least the consistancy of the kludge is the same across the platforms. I prefer Java to Python because it’s also better documented, has more books to learn from, and is also GPL just like Python, so more people can have it out of the box if they chose to be liberal and not support anything not-FOSS.

    As for writing web applications, Python wins, but it’s worth noting a good programmer will just use XML and SOAP, and integrate it using JSP or PHP on the server-side, using the same stuff on the client. Unfortunately the install base again for Java is even smaller than Python on servers, so resorting to ugly PHP XMLRPC stuff is a necessity if you can’t afford your own dedicated server.

  • Interesting, I’d like to see how those come out with Jython/JPython. I’ve often wondered whether the interpreter overhead in Jython would be worth some of the potential advantages of being able to program in a Java environment without having to USE java.

  • Pingback: SnapLogic » Python - the first and last language you’ll ever need()

  • Pingback: Java != Slow « Thermal Noise()

  • Pingback: C++ vs Java vs Python « Computing Life()