Javier Tordable Blog

P versus NP

Sat, 14 Aug 2010 16:54:43 GMT

During the past week shocking news have stormed through the world of Theoretical Computer Science. A researcher from HP Labs, Vinay Deolalikar claimed that he had a proof that P ≠ NP. The paper is available here. But so far it seems that it will not stand. Several researchers pointed out critical defects in the proof that Deolalikar proposed. It is especially interesting to follow the discussion in Dick Lipton’s blog (In 5 posts so far: 1 2 3 4 5). Or in Wikipedia for the short version. It’s amazing to see how many of the world’s most brilliant mathematicians worked so fast to analyze Deolalikar’s proof, including Terence Tao, and Timothy Gowers. The fatal blow to the attempted proof seems to be in a letter from Neil Immerman. There are plenty of references about the proposed proof and the refutations in this Wiki.

Now for anybody that is not an expert in complexity theory (myself included), here is a short explanation of what this is all about and why it matters: P and NP are two classes of algorithms. Colloquially, algorithms in P are those for which the amount of time that it takes to execute with an input of size n is a polynomial of n. For example, finding the maximum in a list of n elements takes approximately n steps because we just need to go through all n elements and keep track of the maximum. Problems for which the best possible algorithm is in P are in general solvable in reasonable time, because even for large values of the input the time that it takes to run the algorithm is not too big.

Algorithms in NP are those for which, if we have a proposed solution, we can check if it actually solves the problem in polynomial time. For example, given a set of integers {−7, −3, −2, 5, 8} of size 5 we can check if a subset like {−3, −2, 5} adds up to zero in less than 5 steps. We simply need to add the numbers. However finding a solution in general takes a much bigger number of steps. One way to solve it is to analyze all subsets, and for each one, check if it’s a solution. But the number of subsets grows exponentially with n, hence the difficulty.

Obviously all algorithms in P are also in NP, because if we can find the solution in polynomial time then we can verify the solution in polynomial time (for example by finding it again). The complex question is whether P is equal to NP or not.

These two classes of algorithms are especially interesting for their practical applications but there are not the only classes by any means. The following diagram shows some of the most important ones. The classes at the bottom are contained in the classes on top.

And there are many more. The Complexity Zoo lists 489 complexity classes.

Now that we have seen what the problem is about, why does it matter? I am going to talk about two important real world consequences of a possible proof of P=NP. You can find many more here if you are interested.

The first application is in mathematical research. As Stephen Cook says, we can verify a proof of a theorem in polynomial time so if P=NP then we would be able to create a proof in polynomial time. This would allow us to build automatic theorem provers that basically do scientific research for us 24/7, continuously discovering new knowledge. It’s obvious the tremendous repercussions that this would have in the world.

The second example is in biology. It is known that the HP protein folding model is in NP. This model basically simplifies the way that amino acids fold in space to form proteins, the essential actors in all cells. If we had an algorithm in P to solve this problem we could possibly design substances that have specific 3D shapes and properties, instead of simply finding them. For example some vaccines work because the 3D shape of a protein in the vaccine fits the 3D shape of a protein in the pathogen. Who knows how many illnesses we could cure with that technology.

High Frequency Trading Art

Sat, 31 Jul 2010 19:29:49 GMT

The folks at Zero Hedge published a very interesting post about high frequency trading and market manipulation. I am not going to discuss economy or finance now, but there is something about that post that is quite shocking. The images which show the prices and volumes of the orders routed to the market have a certain aesthetic appeal.

All the patterns below were most likely created by high frequency trading algorithms, which send thousands of orders to the financial markets and cancel them in milliseconds. The top part of the graph shows the price of the order, and the bottom part the number of shares of the bid or ask. The descriptions are copied from Zero Hedge. Click on the images to see a high resolution version.

BATS "Flag Repeater". 15,000 quotes in 11 seconds, dropping the ASK price 1 penny each quote from $9.36 to $8.58 and back up again. 07-29-10.

"The Crown". While not a large number of quotes, this NASDAQ/BATS Bidsize sequence was just too unusual to bypass. 07-29-10.

BATS "Batsicles". BATS price cycling through a large price range, each intermittent with a stub quote, drop it down and start over. 07-28-10.

NASDAQ "Blotter". One of the more unusual repeating Asksize cycles. 07-27-10.

BATS "Stubby Triangles". Drop the quote from a valid price to 0.001 and then back up to a lower price level. When the new price level hits 0.001 as well, do it all over again at approx. 380 times a second. 07-23-10.

NASDAQ "Flutter". 4000 quotes in 2 seconds, alternating the bid price/size in 3 increments and effecting the Best Bid along the way. 07-23-10.

BATS "Periscopes". 8000 quotes in 3 seconds, alternating the bid price each quote. Pop the size up 1 every second or so. 07-23-10.

NASDAQ "Double Dip". Symbol SH. 10,000 Quotes in 4 seconds, each affecting the Best Bid. 07-22-10.

NASDAQ "Racing Stripe". Symbol WYNN. 2000 Quotes in one second, each affecting the Best Ask. 07-19-10.

PACIFIC "Puzzle Pieces". Symbol IIC. 07-19-10.

NASDAQ "Blue Bandsaw". Symbol SHG. (760 quotes in 1 second, taken from a total sampling of 10,000 quotes in 12 seconds). 07-14-10.

BATS "60-Step". Symbol SAH. Take sixty steps up (a penny at a time) and one step down (0.60), reset and do it all over again (at approx. 700 times per second). 07-13-10.

NASDAQ "Ask Mountain". Symbol IAU. Over 56,000 quotes in 10 seconds, all with same Ask Price and the Ask Size increasing or decreasing by 1 (to almost 40,000!). 07-12-10.

Beauty and Truth in Science

Sat, 24 Jul 2010 00:20:05 GMT

I just watched this fascinating talk from Murray Gell-Mann. The core idea of the talk is that in Physics, as in so many other areas of Science and Mathematics, incredibly complex ideas can be expressed in a simple and concise way. This is a remarkable property of the universe in which we live, and in that fact there is an amazing beauty.

Gell-Mann gives an example from Physics,

This is the Law of Gravitation discovered by Newton. It describes both the mechanism by with objects fall to the ground when dropped, as well as why the planets move in space. For its time it was a massive unification of many seemingly unrelated phenomena. It explains so much of how the world works, yet it can be written with just a few characters.

Here is another example from Mathematics, the Prime Number Theorem,

Natural numbers are some of the most elemental constructions of the human mind, but they allow us to count from the fingers in our hand to the last atom in the entire universe. We know from basic arithmetic that all natural numbers can be factored into primes, so primes are in a sense the “building blocks” of all numbers. And even though primes are distributed in an incredibly intricate way, the Prime Number Theorem explains the very structure of these building blocks.

Edward Tufte, Presenting Data and Information

Sun, 20 Jun 2010 01:25:20 GMT

Last week I attended Edward Tufte's course on data visualization here in Seattle. For those who don't know him, Tufte is one of the world's top experts on information visualization. Here is a short description, from his website:

Edward Tufte has written seven books, including Beautiful Evidence, Visual Explanations, Envisioning Information, The Visual Display of Quantitative Information, and Data Analysis for Politics and Policy. He writes, designs, and self-publishes his books on analytical design, which have received more than 40 awards for content and design. He is Professor Emeritus at Yale University, where he taught courses in statistical evidence, information design, and interface design. His current work includes landscape sculpture, printmaking, video and a new book.

During the course Tufte made several observations that go against what some other people in the field recommend. The most notable from my point of view is to use visualizations with lots of content. And instead of trying to simplify the data, let people understand it on their own. I believe this can be appropriate in many cases, for example in an underground map:

However in some other cases the information overload hides what is really important. For example in this awful visualization in Forbes about where Americans are moving:

It's impossible to understand everything that is going on. What I got out of it is that people are moving from the most economically impacted areas (Southern California, Detroit, Miami, etc.) and into the least impacted areas (Pacific northwest, Texas, New York, Washington, etc.). Also I think I saw a minor trend of movement from colder areas to warmer areas. However with all that clutter it's hard to understand the overall picture. It is true that overload in general is a failure of design, not a problem with information, however it's definitely easier to make bad designs when one tries to present too much data.

In general the course was interesting, but for the most part it covered basic content. However they gave us a copy of Tufte's four books in visualization! Click on the following images to go to the corresponding Amazon pages:

The Visual Display of Quantitative Information

Envisioning Information

Visual Explanations

Beautiful Evidence

The Best Cities for Singles in America

Sun, 06 Jun 2010 03:38:36 GMT

A few weeks back I had a conversation with a friend about different places in the US where she was considering moving. And one of the factors was the dating scene in each one of those places. This brought to my memory a graph that probably many of you have already seen:

This map basically says the following:

If the dot is blue, there are more men than women
If the dot is orange, there are more women than men
The bigger the dot, the greater the difference

However it has a series of problems. First of all, it's too dense. Second, it is not clear that the number of single men minus the number of single women (or the opposite) is what really matters, because obviously the bigger the city the less important the difference is. For example, in a city of 1 million we could have 100K singles and 20K more single men than woman. This means there are 3 single men for every two single woman. However in a city of 10 million, with 1 million singles, the fact that there are 20K more single men, only means there are approximately 1.02 single men for every single woman. It seems that the proportion of single men to single woman is more indicative than the difference. And third and most important, I didn't know for sure if the data in the graph is reliable, as there are no sources cited. So, as you can imagine I decided to check if there is actually any truth in this chart.

Here is the list of steps necessary to get the interesting data from the Census:

Start in the Census fact finder homepage
Click on Data Sets on the left navigation bar
Select SF3 and click on Detailed Table
In the Geography Type section choose Metropolitan Statistical Area
Make sure all statistical areas are selected and click Next
Now choose the table P18. Sex by marital status for the population 15+ years
Click on Add and then Show result

From this table we want the "Never married" rows. Because the table is too big to analyze directly (280 columns) Let's download it and map the results using the Google Charts API.

You may need to clean up the downloaded file, because there are copyright notices and extra end of line characters. Also you may keep only the interesting lines, which are 3:

The header of the table, which contains the name of the metropolitan areas
The row for Male / Never Married individuals
The row for Female / Never Married individuals

After removing all other rows, it's possible to parse the file and extract the interesting data with a small python script like this:

    input_file = open('DTDownload.csv')

    # First process the header line.
    header = input_file.readline()
    header = header[5:-2]  # Remove " ", at beginning and "\n at the end.
    tokens = header.split('","')
    cities = []
    for token in tokens:
        token = token.replace('MSA', '')  # Eliminate the MSA suffix.
        token = token.replace('CMSA', '')
        if token.find('--'):  # Remove everything after --
            token = token[:token.find('--')]
        cities.append(token)

    # Now process "Never married" men.
    line = input_file.readline()
    line = line[17:-2]  # Remove "Never married" at beginning and " at end.
    tokens = line.split('","')
    single_men = []
    for token in tokens:
        token = token.replace(',', '')  # Eliminate the , for thousands.
        single_men.append(int(token))

    # Now process "Never married" women.
    line = input_file.readline()
    line = line[17:-2]  # Remove "Never married" at beginning and " at end.
    tokens = line.split('","')
    single_women = []
    for token in tokens:
        token = token.replace(',', '')  # Eliminate the , for thousands.
        single_women.append(int(token))

    # And from those two quantities compute the ratio of single men
    # to single women. The higher, the better for women. Reverse the
    # quotient to get the other side.
    ratios = []
    for i in xrange(len(single_men)):
        ratio = float(single_men[i]) / float(single_women[i])
        ratios.append(ratio)

    # Now print the name of the city and ratio in the format accepted
    # by the Google Charts API, which is:
    #   data.setValue(0, 0, 'New York');
    #   data.setValue(0, 1, 1.064382);
    for i in xrange(len(cities)):
        print 'data.setValue(%(row)d, 0, \'%(city)s\');' % \
            {'row': i, 'city': cities[i]}
        print 'data.setValue(%(row)d, 1, %(ratio)f);' % \
            {'row': i, 'ratio': ratios[i]}

And to display the data, just include it into a Google Charts visualization. Unfortunately I couldn't figure out how to make the circles bigger or smaller based on one parameter and to set the color based on another parameter. Please tell me if you know how to do it. Anyway, until I find some time to learn Protovis, bear with me. What I did is split the chart into two, one for guys and one for girls. And to have it load faster, and avoid overloading with too much detail, I just took a small number of cities (This was actually Noha's idea so the credit should go to her). Here is how the chart looks like for girls, the bigger and bluer the better. It means there are more single guys for each single girl:

It seems that the West Coast, Texas and Florida are the best places for girls. Here is how it looks like when taking all 280 cities. I did not include the visualization, but a simple image because it takes quite a while to load.

The one for guys is the opposite, the bigger and more pink, the better. It means there are more single girls for each single guy:

So the best places for guys seem to be the East Coast and the Great Lakes area. In particular the Bos-Wash corridor seems to be quite good. And here is how it looks like when taking all metropolitan areas:

Does anyone know of similar charts for other countries? I have never seen one myself. But after what we said before, if you can find census-like data now you can make one yourself!

Quantitative Ponzinomics

Fri, 07 May 2010 22:40:28 GMT

Just a quick pointer to a book that I am reading right now, Quantitative Ponzinomics, an essential guide to understand what is going on currently in the financial markets around the world.

Ok, it's not a real book, but you should really check out the source, the financial blog Zero Hedge. And in order to keep this blog on-topic, here is an interesting link about High Frequency Trading, which is an automated procedure to make financial transactions using computer algorithms to get an advantage over other investors.

Machine Learning at Stanford

Sat, 01 May 2010 23:28:57 GMT

This quarter I am leading a study group in Machine Learning at Google's Kirkland Office. And while I was looking for datasets and resources I found Andrew Ng's course in Machine Learning at Stanford. All the lectures are available online at YouTube. You can find the links in the Stanford Engineering Everywhere Machine Learning course page.

Here is the first class of the series:

Also you can check out the main Stanford Engineering Everywhere page for other courses in computer science.

Preserving the Wealth Creation Engine of America

Sun, 18 Apr 2010 04:28:46 GMT

Over the last few decades hundreds of companies, thousands of jobs and billions of dollars of real wealth were created by startups with the support of angel investors. The possibility of a small company to obtain seed funding with little more than an idea is indeed one of the most important wealth creation engines of America. Now this possibility, this engine, is at stake.

There is a bill currently under review in the Senate that includes two provisions that directly threaten the way that startups are funded. The bill is the "Restoring American Financial Stability Act of 2010” supported by Senator Dodd. Even though the intention of the bill is noble, to attempt to protect investors against fraudulent investment requests, two of its sections can have severe unintended consequences for the economy and the society of the United States. The first one is the following:

SEC. 412. ADJUSTING THE ACCREDITED INVESTOR STANDARD FOR INFLATION.
The Commission shall, by rule—
(1) increase the financial threshold for an accredited investor, as set forth in the rules of the Commission under the Securities Act of 1933, by calculating an amount that is greater than the amount in effect on the date of enactment of this Act of $200,000 income for a natural person (or $300,000 for a couple) and $1,000,000 in assets, as the Commission determines is appropriate and in the public interest, in light of price inflation since those figures were determined; and (2) adjust that threshold not less frequently than once every 5 years, to reflect the percentage increase in the cost of living.

This basically raises the minimum income and assets that an investor needs to have to be considered qualified to invest in a startup. The limit would go from $1 million of net worth to $2.3 million and from $200,000 of annual income to $449,000. This could affect more than two thirds of all angel investors.

The second section is:

SEC. 926. AUTHORITY OF STATE REGULATORS OVER REGULATION D OFFERINGS.
[...]
IN GENERAL.—The Comission shall review any filings made relating to any security issued under Commission rules or regulations under section 4(2), other than one designated as a non-covered security under subparagraph (A)(iv), not later than 120 days of the filing with the Commission.
[...]
IN GENERAL.—Nothing in subparagraph (A)(iv), (B), or (C), shall be construed to prohibit a State from imposing notice filing requirements that are substantially similar to filing requirements required by rule or regulation under section 4(4) that were in effect on September 1, 20 1996.
[...]

it essentially would require startups to make a filing with the SEC, which will have up to 120 days to review it. If the SEC doesn't review the filing then securities regulators in all states involved in the deal will have the chance to review it. What this means is that cash-strapped startups will have to wait 4 months for funds, and they will have to comply with securities laws of up to 50 states, with significant costs in legal fees.

http://www.saveregd.com/, contains more information about the significance of this sections and the impact that they may have on startups. Also you may find the full bill here.

I believe that these two sections of the bill will impact negatively startups, angel investors, and as a result the mechanism by which entrepreneurs get funding for new ideas, innovate, and generate jobs and wealth. To express your oposition to these two sections of the bill, and help preserve one of the most important wealth creation engines of America, I would like to ask you to review and sign this petition now.

How to Make an Infographic Résumé

Tue, 23 Mar 2010 04:08:19 GMT

Infographics, or information graphics are visual representations of data. In a good visualization or representation of a data set, the author expresses an idea that is deeper that the data itself. A good visualization conveys a message that is clear and helps to extract conclusions, but also a message that is precise and based on the data, without transforming or manipulating the data in dishonest ways.

In the case of a résumé, the data is obviously the education, professional experience and interests of the candidate. And the purpose of an infographic résumé is to show the greatest highlights of his or her education and professional experience, to convince a potential employer of the value of the candidate. To do that, the charts and visualization elements have to be adequate to the characteristics that we want to remark. For example:

If a candidate has a lot of international professional experience, one way to show that is to include a world map with the location of the last work assignments
Or if the candidate is a senior executive of a public corporation, it may be interesting to show the stock value of the company at the time that the executive launched new products, or took important business-wide decisions
In case of a professor with a long publication history, it may be very valuable to show a chart of important publications and their citation index, as a measure of the impact of those publications in the scientific community

Some Example Résumés

Over the last few years several designers have experimented with inforgraphic résumés. Recently Randy Krum wrote a post about it in his blog Cool Infographics. I am not going to repeat what he said, but I will mention a few of the most famous ones.

The résumé of Michael Anderson is probably the most cited one, it includes a timeline with education and experience, and marks for the most important events. It is visually attractive, but the colors and sizes of the different elements don't really mean anything. And the whole bottom left part of the chart is only meant to show some sense of humor. You can click on the thumbnail below to see the full image.

Christoper Perkins designed another résumé using the common look of a subway map. Also it is visually original, but the connections between the different elements are not very complex (only two intersections between lines), so using the look of a subway map seems to add more complexity that it removes. Also this visualization uses intersections between lines to represent the fact that two events happened at the same point in time. But the fact that two work projects or education courses happened at the same time is probably not the most important characteristic of a résumé.

Compare this use of the subway visualization with the Map of Rock of Ernesto Lago. Here what is important is how some musicians merge two different styles, or inherit from two different musical schools or traditions, and that is exactly what the subway intersections represent.

Greg Dizzia gives a different point of view. In his résumé he adds the skills involved in each professional project. I think this is very valuable to potential employers, and it's one of the main points of a résumé. In this case I think the visualization is a little bit too charged and repetitive but the main message is very well expressed.

As a last example let's take a look at the résumé of Gabriele Bozzi. In this case he forgets almost completely about the temporal element of the résumé and concentrates on the functional areas. And he divides them between general skills, and knowledge of particular applications and tools. To me this has too much detail, and way too much keyword hunting, but the visual display is very interesting.

How to Make an Infographic Résumé yourself

First of all, just like any other infographic project we need to decide what data we are going to show. In my case I will take short snippets of projects in which I have worked on. Because the main point of this post is to show how to develop the visual component I won't spend too much time in getting a great wording of each project, or using the correct keywords.

Second, it's necessary to decide what is the most important messages that we want to convey from that data. In my case these messages are:

Education: I have a thorough education in Computer Science and Mathematics. It is not too varied because I don't have courses or certifications in other areas, and I studied all my degrees in the same University. But I have spent 10 years on it. Also, I don't want to add much detail about college projects, because they are less interesting than professional projects. And I don't want to mix education and work, because I combined both over the last few years and it would just look confusing
Work: I have worked in some of the most prestigious companies in the world, in increasing degrees of responsibility and success and I definitely want to emphasize that. Also I want to mention that I worked in some particular areas in software, like machine learning. And I started doing a little bit of research lately. So I think it's worth showing that I am versatile, in terms of work skills. Finally I think it's also worth mentioning my contributions to open source and internships, because they add variety to my professional experience
Impact: Finally, because I have been lucky to work in projects that are public and other people are using, I have had a significant impact outside of my employers. I want to emphasize that as well

And third, after the message is clear, it's turn to decide what kind of visualization would fit best the data and the message. I won't go very deep into this topic, because there is so much to talk about that we could fill several books. So here are the decisions I took:

For the education I will simply show a bar with all the degrees that I have attained, in a straight line. The size of each part of the bar indicates how long it took to get that degree. The colors are only to differentiate sections. A legend at the bottom will indicate what is each section. Also as this is the weakest section, it will go at the end of the résumé
For the experience part, I will show a timeline of professional positions, and the main projects within those positions. I will add the logos of the corresponding companies on the left side. And for each project I will indicate the functional skills that I used. In order to keep it simple I will have only 5 main skills. Also to have space for all the positions, I will have a vertical timeline, instead of the most common horizontal one
Finally, the impact of my projects is probably the most differentiating characteristic of my resume, so I will put it on top. And in order to show data that is easy to check I will use as a measure of impact the number of search results in google.com for the project name, divided by the number of developers in the project. In order to indicate which project corresponds to which value in the graph I will use an annotated time series.

And without any more delay, here is the result!

Impact

Clicking in each one of the marks in the time series highlights the description of the project. For each case I have included the employer, the position and a very short description of the project. In parentheses is the number of search results divided by the number of people that worked in the project. Also clicking in the description takes to a search results page. The results are usually a reference my personal work, or work in which I participated.

Experience

In the experience table I have added basically the same projects as above, but I have classified them by the main area of expertise. The link in each one of the projects point to search results, exactly the same way as before. It would have been also a good idea to point to a short description of the project. Initially I though about using color codes to indicate each main area of expertise, but this would add a little bit of duplication, because that is exactly what the columns mean. I think it is cleaner and more understandable without color coding the skills.

Education

And finally, the education section. Here I decided to do a static image. The interactivity of the other visualizations wouldn't add anything. Also I used a similar color palette to the previous visualizations. Even though as it is there is no relationship with the visualizations above, I could have explored this idea further. Also I didn't add the university to avoid repeating the name four times.

Conclusion

To finish the post, I will repeat the core principles of how to make a good visualization:

Collect the data to display, make sure its complete and precise
Decide what is the most important message that you want to express
Choose visualizations that convey that message but respect the data

Really Simple SEO

Fri, 12 Mar 2010 00:36:31 GMT

SEO stands for Search Engine Optimization, and is the process of improving a website's structure and content in order to make it easy for search engines to gather the pages and display them in search results in the best position possible.

In this post I am going to explain a few basic principles of SEO and show examples of how I implemented them in my blog. Also, as I am the Tech Lead of Webmaster Tools backend, I am going to talk a little bit about some features of Webmaster Tools that are very helpful for SEO. Please remember that all these tips are not only good for search engines, but also for users. If you have to choose between doing something to benefit users or search engines, always choose what is best for users.

Here is the list of simple SEO tips:

Use a good URL structure, with descriptive URLs. If the URL has keywords related to the page topic it will be easier to find in search results. For example, the url of this post includes the words really simple seo, that are the topic of the post
```
 
    http://www.javiertordable.com/blog/2010/03/11/really-simple-seo
  
```
Use good page titles. Similar to the previous tip, good page titles, with appropriate keywords make it easier to find the page in search results. Even though the title of this page includes the full name of the site, the title begins with a good description of the content
```
 
    Really Simple SEO - Javier Tordable blog on Software, Mathematics and
    Technology
  
```
Have a good meta description for each page. The meta description is used by some search engines to show in the snippets in search results (the small paragraph with a description of the page). If you don't use it, you will have the risk that the search engine will generate it by itself, with unexpected results. This happened to me before I had a meta description, my snippet was taken from the RSS feed and looked awful. My current meta description for the homepage is:
```
    <meta name="description" content="Javier Tordable blog on Software,
    Mathematics and Technology. Javier Tordable is a software engineer at
    Google and Ph.D. candidate in Mathematics.">
  
```
Structure the page appropriately, using HTML header tags. The most important parts should be within a H1 tag, the second most important in H2, etc. until H6. For example in this post the blog title and subtitle are within H1 and H2 tags. The title of the particular post is in a H2, and other less important sections, the about box and the archives are within an H3 HTML tag
```
 
    <h1><a href="/">Javier Tordable</a></h1>
    <h2>A blog on Software, Mathematics and Technology</h2>
    <h3>About</h3>
    <h3>Archives</h3>
    <h2>Really Simple SEO</h2>
```
Use the simplest format possible. Currently search engines are very advanced and can process Flash, Javascript and other content types, however it's always easier to access raw HTML content. So prefer to use HTML for most content, unless Flash or Javascript are essential. Also it will be easier to access the page from old browsers or other platforms. For example iPhone users can't see Flash pages
Have a flat internal link structure. The flatter your link structure is, the easier it will be for search engines to access a page. Also the easier it will be for users to access whatever content they are looking for. In my blog I have all the main sections linked in the top navigation bar, which appears in all pages. And in the right side of most pages there is a link to all the blog posts
```
 
    <a href="/blog/all">All Posts</a>
  
```
From the homepage of the site it's possible to access any other content page in two clicks or less.

And now, some interesting pieces of information about your site that you can find in Webmaster Tools:

One of my favorites is Backlinks, which will show you all the links pointing to your site, from all over the Web. Having many quality links is important because it will be easier for people to find your site, and it will show to search engines that the site is relevant.
Another very useful tool is Top Search Queries, which will show for which queries my site appears in search results. For example, the "bundle adjustment" query has more requests than "javier tordable" and it appears in the second row in the following table, while the other query is in the third row. However my site appears in position 16 for bundle adjustment, and it appears in position one for searches of my own name
Also, another cool piece of information is Subscriber Stats which shows how many subscribers I have for my RSS feed from Google Reader. In my case I can see that I have 10 people subscribed. And I can also submit this feed as a Sitemap, which will help getting my site indexed For example, my Sitemap statistics show that I have 11 pages in this sitemap, and they are all indexed
And to check that my site is being correctly crawled, I can check the Crawl Stats feature. As you can see in the graph, the number of pages that are crawled in my site per day has been going up significantly since I implemented all this SEO tips

To finish, I am just going to point out that there are a lot of SEO resources online. Doing proper search engine optimization doesn't need to be complicated or expensive. And is not only good for search engines, but also for users. For more tips, you should check the Google SEO guide.

Collaborative Mathematics and The Future of Science

Thu, 25 Feb 2010 20:02:55 GMT

Mathematical research is traditionally seen as a one-man job. To quote Jean Dieudonné in The Music of Reason:

Research in the experimental sciences is done in laboratories, where larger and larger teams are needed to manipulate the instruments and to scrutinize the results. To do research in mathematics nothing is needed except paper and a good library. Team-work, as practiced in the experimental sciences is, then, quite unusual in mathematics, most mathematicians finding it difficult to think seriously except in silence and solitude. Collaborative work, while quite common, most often consists in putting together results that each of the collaborators has managed to obtain in isolation, albeit with mutual profit from each other's ideas, enabling them to progress form new points of departure.

In spite of that, about a year ago Tim Gowers asked himself if it would be possible to solve important mathematical problems by collaborating openly over the internet. Not a collaboration among a few colleagues, but among everybody that had any insight about the problems. He shared this question with the mathematical community through a post in his blog. That was the birth of the Polymath project.

The first problem that the Polymath group worked on is the attempt to obtain a simple proof for the Hales–Jewett theorem. This theorem is a very important result from Ramsey theory. In very gross terms Ramsey theory says that for many mathematical structures, there is no such thing as complete randomness.

For example, take a group of six people: Alice, Bob, Charles, David, Erin and Fritz. The Ramsey theorem tells us that there are either 3 people that all know each other, or 3 people that are all strangers to each other. Even in something as random as a party, if there are at least 6 people then we can find a very special subgroup of 3 people.

Here is the proof: Take Alice, imagine that she knows less than 3 people at the party. That is, she knows only Bob, or knows only Bob and Charles. Then we consider David, Erin and Fritz. If they all know each other, we have a group of 3 people that know each other. If not, two of them don't know each other, for example David doesn't know Erin. As a consequence Alice, David and Erin are all strangers to each other. If Alice knows 3 people or more, the proof is the same, say she knows Bob, Charles and David. If none of them knows each other then there we have our group of 3 strangers. But if two of them know each other, for example Bob knows Charles, the group Alice, Bob and Charles all know each other.

So far the Polymath group has discussed 5 problems, which are all shown in the Wiki and they have started to publish some of the results. Here is link to a paper from D.H.J. Polymath on arXiv, A new proof of the density Hales-Jewett theorem.

The following image is a part of a 3D Maldelbrot fractal, as described here. It has nothing to do with the rest of the post, but the Hales-Jewett theorem doesn't lend itself easily to fancy pictures.

And for comparison purposes this is a piece of romanesco broccoli.

Of course, Mathematics is not the only scientific discipline in which people collaborate openly in interesting problems. Michael Nielsen has a great blog post about doing science online. Probably the most important point is that the way that scientists work with each other is changing. And the change is driven mostly by new online collaboration tools. To finish the post I will quote Michael:

Blogs, wikis, open notebooks, InnoCentive and the like aren’t the end of online innovation. They’re just the beginning. The coming years and decades will see far more powerful tools developed. We really will enormously scale up scientific conversation; we will scale up scientific collaboration; we will, in fact, change the entire architecture of expert attention, developing entirely new ways of navigating data, making connections and inferences from data, and making connections between people.

New Google Chart Tools

Wed, 17 Feb 2010 02:03:13 GMT

Google recently released a new set of tools for graphics and interactive visualizations called Google Chart Tools. Google Chart Tools replaces the previous Charts API (for static images) and Visualization API (for dynamic graphics). And it combines both APIs within a single framework. Here is a link to the official announcement.

This is an example of the Charts API, a map with a couple of countries marked in a different color:

This map was generated with the following link:

http://chart.apis.google.com/chart?cht=t&chtm=world&chs=440x220
&chld=USES&chd=t:10,50&chco=FFFFFF,00FF00,005500&chf=bg,s,EAF7FE

Let me go over each part in that link and explain what it means:

cht=t indicates that this is a graph of type map
chtm=world says that the map should include the whole world
chs=440x220 is the size of the chart
chld=USES is the list of countries to display in a different color, US and ES
chd=t:10,50 is the intensity of the color of each country. US=10, ES=50
chco=FFFFFF,00FF00,005500 is the color gradient FFFFFF=white for the background, 00FF00 light green (US) and 005500 medium green (ES)
chf=bg,s,EAF7FE is the background color, light blue

Here is another example, but this time of an interactive visualization:

In this case the map is dynamic. Moving the mouse over the different countries will display a message, which contains the value used to select the color of the country. Now the code is a little bit longer, about 30 lines of Javascript, so I am not going to include it, but there is a detailed explanation here: Google Chart Tools, Introduction.

These tools are probably not as powerful as custom made visualizations, like the ones that I talked about in a previous post, Interesting Visualizations: Changes Over Time, but they are definitely easier to create and modify.

To finish, I am just going to quote Robert Kosara and his blog on visualization Eager Eyes, "JavaScript for visualization is clearly the way to go. It's fast, versatile, works much better than Flash or Java, and is obviously way ahead of static images". You can check the complete post here.

Nounoublog updated

Sat, 06 Feb 2010 00:00:00 GMT

Over the last few weeks this blog has changed dramatically. It looks pretty much the same as when it started but under the covers the code of the blogging platform, Nounoublog is very different. I am going to talk about three of the features that I have been working on lately:

Archives
RSS Feed
Admin console

And I will show a few snippets of the actual code that powers the blog.

For those that visit the blog for the first time, Nououblog is a small blogging platform developed in Google App Engine. I started working on it basically for two reasons. First, I wanted to learn how to develop applications for Google App Engine. And second, because I wanted a simple but highly customizable platform, with free hosting and no ads.

RSS Feed

An RSS feed is a specially formatted XML file, which includes data about the posts in a blog. It is updated automatically by the blog, so that when it changes, RSS subscribers know that there is new content. There are applications like Google Reader that are very helpful to keep track of many RSS feeds and alerting when there is new stuff to read.

In the Nounoublog blogging platform, you can access the RSS feed by clicking in the RSS link at the top of the page. In the browser you may see only a bunch of text, but if you add this link to your Google Reader subscriptions you will see a list of the most recent posts.

Same as before the RSS feed is powered by its own handler, which is very simple:

class RssFeed(webapp.RequestHandler):
  """Handler for the RSS feed.

  This feed contains a list with all the blog posts, from last to first.
  This list is subject to the maximum item retrieval limit of the DB.
  """
  def get(self):
    # Get the list of all posts.
    posts = get_all_posts()

    # Return the xml feed with the posts.
    template_values = {'posts': posts}
    self.response.out.write(template.render(template_path("rss_feed"),
                                            template_values))

Where the template provides the XML structure of the feed, and inserts the data corresponding to the posts

<?xml version="1.0" encoding="iso-8859-1"?>
<rss version="2.0">
<channel>

<title>Javier Tordable Blog</title>
<link>http://www.javiertordable.com</link>
<description>
  Javier Tordable blog on Software, Mathematics and Technology
</description>
<generator>Nounoublog</generator>
<docs>https://code.google.com/p/nounoublog/</docs>

{# Loop over all the blog posts. #}
{% for post in posts %}
<item>
<title>{{ post.title }}</title>
<link>{{ post.absolute_url }}</link>
<pubDate>{{ post.rss_pub_date }}</pubDate>
<guid>{{ post.absolute_url }}</guid>
<description>{{ post.escaped_content }}</description>
</item>
{% endfor %}

</channel>
</rss>

Notice that in the Django template the post elements appear as attributes while in fact they are method calls. Also I use as GUID the url of the post, as it is intended to be a permanent link.

Admin console

The last item that I have been working on is the administration console. This is still work in progress, but I expect that once I am done with it I will post more often.

The admin console will have options to:

Add posts
Edit posts
Add static pages
Edit static pages
Edit the CSS
Edit redirects

All these options seem very normal with the exception of the redirects. How does it work? For example, when going to:

http://www.javiertordable.com/blog/2009-12-01/my-first-blog-post

You are redirected to another url, which appears in the url bar. Notice how the dashes are now forward slash bars

http://www.javiertordable.com/blog/2009/12/01/my-first-blog-post

I added support for redirects because I changed the site several times (including the url structure), and I didn't want to serve 404 error pages for all old urls.

Keep visiting the blog or subscribe to the RSS feed for more news on Nouonublog!

The Eternal Night

Sat, 30 Jan 2010 00:00:00 GMT

My brother David is a film director. He has been making short film for a few years, he has even won a few prizes. Most of his work is at tpmpictures.com. Today I just wanted to show his last piece of work, a science fiction short about the end of the world. The short is in Spanish but with English subtitles.

There are a couple of things that are interesting in this short. First, the script is not wildly improbable. Check out this (humorous) list of the 5 scientific experiments most likely to end the world. And second, the special effects are pretty nice for a zero budget short. If you liked it, please go ahead and leave a comment at the Notodo film festival.

TRANSCEND

Fri, 15 Jan 2010 00:00:00 GMT

A couple of days ago I started reading TRANSCEND, the new book from Ray Kurzweil and Terry Grossman. The book starts from the principle that our knowledge of medicine and biology is increasing to a point where we can start to control effectively how fast our own bodies age. And even more important the amount of knowledge that we gather is increasing over time. If the trend continues we may reach a point where we can effectively reverse engineer our bodies in order to avoid aging.

Whether you believe that we will reach that point or not, the book is an interesting read. It is filled with healthy habits, complete diets and recipes, exercise programs and more. Even if you leave aside the supplements and the fancy biomedical technologies, there are plenty of actionable tips for inproving your quality of life. For example to check for food intolerances. There are millions of people out there that can't digest milk or wheat very well and are not even aware of it.

Disclaimer: I didn't get paid to write this post.

MapReduce Integer Factorization in arXiv

Thu, 07 Jan 2010 00:00:00 GMT

This Monday I published my article on MapReduce for integer factorization in arXiv. The article is essentially the same that can be downloaded in the research section of this site. So if you have already checked it out, you won't find anything new. However I am very excited because it is my first addition to arXiv.

In case that you are not familiar with arXiv, it is one if the greatest scientific sites in the web. It has over half a million articles, especially in the fields of mathematics, physics and computer science, and many relevant papers are published in the arXiv months before they appear in any peer reviewed journal.

MapReduce Integer Factorization released!

Tue, 29 Dec 2009 00:00:00 GMT

Recently I published the code of MapReduce for Integer Factorization. It is available under the Apache 2.0 License in Google Code. It includes everything necessary to run in Apache Hadoop, as well as the numerical libraries used. It has no dependencies apart from the last version of Hadoop.

This project is a proof of concept that shows how to use MapReduce, a framework for distributed computation to solve a purely numerical problem. The main conclussion is that it's possible to use MapReduce for problems that lie far ahead from its original area of application, for example number theory. Also in this case the difficulty involved in developing the MapReduce program is similar to the difficulty of creating a worksheet in a mathematical tool like Maple. But the performance of MapReduce is significantly higher.

If you have some time, please download it from here, and let me know how it works for you.

A small blogging platform in Google App Engine

Thu, 17 Dec 2009 00:00:00 GMT

If you have never made a web application it may seem daunting. There are hundreds of alternative technologies and frameworks out there. And web apps development is quite different from client applications, which is what most developers are used to.

Here is an example of a web application. Wikipedia!

Most web applications share a few common elements:

A persistence layer, for authored content or user created content
A system to connect each user request with a part of the application
A method to render and display that content to users

Traditionally the persistence layer is a SQL database, the requests are directed for example to Java servlets in an app server like Tomcat, and there is a more or less refined templating engine, in which the content is added to create the whole page returned to the user.

Another alternative is Google App Engine. Google App Engine is a platform and a set of libraries to develop web applications based in Google's own infrastructure. It is available in Java and Python, but here I will concentrate on the Python version.

The persistence layer of Google App Engine is the Datastore, a highly parallel but simple to use storage solution. The Datastore doesn't support queries as complex SQL does, however it can scale up to a level which is beyond what a normal database can do. And it can do so in a way that is trivial for the developer.

Google App Engine uses yaml and the webapp framework to answer user queries. One can set a configuration file which assigns certain url paths (via a regular expression) to instances of webapp.RequestHandler.

handlers:
- url: /.*
  script: app.py

The instance can implement a get() method which generates the response returned to the user.

class App(webapp.RequestHandler):
    def get(self):
        self.response.headers['Content-Type'] = 'text/plain'
        self.response.out.write('Hello, World!')

Finally, in order to generate HTML, Google App Engine incorporates the templating engine from Django. A template is essentially a document with variables instead of content. When the application needs to answer a user request it can load the template and replace the variables with real content, for example information from the data storage.

This is basically the way that this blog is made. It is a very simple Google App Engine application. It uses the Datastore for the blog posts, which are retrieved when it receives a request for the /blog url path. Then it replaces the blog post content into the blog template and returns that content. Here is a sample of code. It is not the actual code, but it gives a complete example:

class Blog(webapp.RequestHandler):
  def get(self):
    # Retrieve the posts from the database.
    query = 'SELECT * FROM Post WHERE public = True ORDER BY date DESC '
         'LIMIT %d ' % NUM_POSTS_IN_MAIN_PAGE
    posts = db.GqlQuery(query).fetch(NUM_POSTS_IN_MAIN_PAGE)

    # Render them into html.
    if len(posts) > 0:
      template_values = {'posts': posts }
      template_path = os.path.join(os.path.dirname(__file__), 'templates/blog')
      content = template.render(template_path, template_values)
    else:
      content = ''

    # And return a full page with the blog content.
    s = pages.FullPageGenerator()
    self.response.out.write(s.generate(content))

To sum up, Google App Engine is a great option for developing web applications. Expecially if you will require it to scale seamlessly, or to integrate with Google services. Check out the Google App Engine tutorial.

Interesting Visualizations: Changes Over Time

Thu, 03 Dec 2009 00:00:00 GMT

Visualizations are simply ways of representing data. But if they are good, they can bring us deep insights, that go well beyond what is possible to understand by simply looking at the raw data.

There are several categories of visualizations, for example:

Compare two entities based on a given set of metrics. An example of this is a benchmark between two competing companies or products
Track the value of a given metric over time. A well known visualization of this type is a financial chart, with the value of an asset
Compare the value of a single metric in different geographic locations. We have all seen maps in which the color of each region is based on the value of the metric

Another very interesting set of visualizations are those that allow us to track a particular situation over time. Here are three examples:

Tracking a change in ranking over time (by Ciaran Hughes)

Tracking a change in distribution over time (from GE). The bottom slider changes the chart based on the age. Each section of the chart represent one kind of illness

Tracking changes in volume or magnitude over time (from xach.com). Each color block is a movie, and the size represents the box office in each week.

The first visualization doesn't attempt to indicate quantity because it displays an abstract concept such as brand appreciation. However in the third case the quantity is very concrete, the total box office in dollars.

We could use also the first or third approach for the second data set. But if we went with the first kind of visualization most likely we would only remember the most important expense for each age. Also we would be constrained in the number of years to show. If we decided to go with a visualization of the third kind it would be hard to compare how the expenses change relatively to each other as all of them are likely to increase over time.

My first blog post

Tue, 01 Dec 2009 00:00:00 GMT

Hello everybody. This is the first post in my new blog. This is not your common Wordpress or Blogger blog. It runs on a custom blogging platform made from scratch, on top of Google App Engine. Soon I will add a couple of posts about how it's done, and I will release the code of the platform.

In the future I will use this blog to talk about stuff that interests me. For example:

Google App Engine, Django and other tools for rapid web application development
Sage, an open source mathematics package
Devi Prasad Shetty, and how he transformed medicine through mass production
Interesting visualizations! As information is more and more available it is becoming increasingly important how to visualize and understand easily

The subscribe links don't work yet, so I am afraid you won't be able to read this blog in your favorite RSS reader. But I hope that I will see you again soon. Thanks for coming!