Clément Farabet

(Research Scientist (New York University / YALE))

I am currently working on the architecture of a Vector/Stream Processor for Convolutional Neural Networks with Pr. Yann LeCun, at the Computational and Biological Learning Lab [CBLL], New York University and Pr. Eugenio Culurciello, e-Lab, Yale.

Before that I have been working on unmanned helicopters's visual guidance with Dr. Matthew Garratt, at the University of New South Wales, Australia.

For more information on my current projects, go to my research page.

K O B E

K Y O T O

I T H A C A | N I A G A R A

L I N Z | A R S E L E C T R O N I C A

P R A H A

N E W Y O R K

A R T I F I C I A L I N T E L L I G E N C E

My passion for AI originally appeared as a strong desire to understand human intelligence, how one learns, recognizes what he sees, what he hears, but also how one's feelings and emotions can drive him when he decides and thinks rationally.

But I am a pragmatic person, and I have an engineering background... AND that's true, I love computers, and I like programming.

So I decided that it would be great to spend a good part of my life exploring the intricacies of intelligence, and particularly vision, via computers... and hardware designs.

I was actually led to this field by several readings, the most remarkable being Gödel, Escher, Bach: an Eternal Golden Braid (GEB), by Douglas Hofstadter (thanks to my dearest friend, Thomas). Anyone interested in this field should get this masterpiece and stop sleeping for a few days! Another major book that has been of real interest to me is Descartes' Error: Emotion, Reason, and the Human Brain, by neurologist Antonio R. Damasio. While GEB explores intelligence at a rather high-level (logic), this book goes into the intricacies of the brain.

So, like other people in this field, I enjoy trying to find new ways of giving eyes to computers, to let them see, and recognize / analyze what they see. And obviously, getting inspiration from the perfect seeing machine-the brain-seems to be the right way of doing it! This is how machine learning-a broad subfield of AI-was created.

Machine learning is concerned with the design and development of algorithms and methods that allow computers to "learn". Let's take a simple example: we want our computer to recognize cars in a video stream. The "old" and classical method would be to create an algorithm to scan the images, extract some features and try to match them against a list of handcrafted features: this is explicit knowledge. The "modern" method-machine learning (yes I am biased...)-would train the computer to recognize the cars. So instead of explicitly defining the car, we define a trainable system (e.g. neural network), and we train it by showing it different examples of cars, and letting it adapt itself (backpropagating the error).

So how can these systems be trained and learn some knowledge? The high number of connections between simple elements is the key in storing implicit knowledge. That's how our brain assimilates knowledge.

By the way, the web-Internet-is a set of highly interconnected simple elements-routers, servers-and thanks to this immense number of connections, the knowledge that can be implicitly stored is infinite.

Finally, here is an interesting sentence taken out of GEB. It illustrates a famous theorem in a rather original way: "is an expression which, when it is preceded by its translation, placed in quotation marks, into the language originating on the other side of the Channel, yields a falsehood" est une expression qui, quand elle est précédée de sa traduction, mise entre guillemets, dans la langue provenant de l'autre côté de la manche, crée une fausseté.



L A N G U A G E S

Well. Right, I love real languages, and besides my native French, I like English a lot. I am also very interested in natural language processing, a field of artificial intelligence that studies the problems of automated generation and understanding of human language.

But I was not talking about those languages here. You have to pardon me, but yes, programming languages can be beautiful, and some of them are probably worth exploring for years. As I have an electrical engineering background, I found myself first attracted by computers at the hardware level. So I went into the tedious process of understanding [and designing] processors. I am still into this, but since then I have traveled the path up and down many times, going through each layer of abstraction with a new language...and more eagerness than ever before... If I remember correctly, the road has been something like: Basic (I was 10 years old, my dad probably knew I would love these machines!...), Pascal, C (here is the beginning of everything), Assembly, C++, PHP, Javascript, JAVA, VHDL, Verilog, Latex, Lisp and Python.

From all those languages, Latex and Lisp are the ones I want to talk about.

First Lisp. Lisp is one of those old languages, and I wouldn't have thought I would love it like that. But here is the thing, Lisp offers the ultimate abstraction in classical sequential programming languages: it formats both the code and the data in exactly the same way! More than any practical application, it allows you to experiment so many concepts about logic, AI and other meta programming ideas. For the skeptics (and the musicians) have a look at that!

Now Latex. Latex is something you will cherish or just despise... For those who don't know, Latex is not a programming language, rather, it is a text formatting language. It is more difficult to use than the common text editor (word for instance). No lies. It is a pain to go into it, and get it right. But the result is just amazing. I think I may be too rigorous, or even maniac, and it's probably why I love the perfect rendering that Latex provides: it is the one, the unique perfect typography. You can even feel the texture of the paper once you've written something... But you'll understand after you visit this page.



N K S | C O M P L E X I T Y | L I M I T S O F S C I E N C E

NKS stands for a New Kind of Science. Stephen Wolfram chose this name for what he considers to be a revolutionary scientific framework to describe our universe, and explain pretty much every natural phenomenon. Wolfram has been working on this project for more than 20 years now, and he is convinced that it will represent the new universal basis for all sciences.

I am particularly excited about this new framework, as it opens a lot of new areas of research. The general idea is very simple. It comes from an experiment (now cited in so many papers) that Wolfram performed a long time ago. This experiment used the simplest of initial conditions-one darkened cell on the top row. And the process of generating future states-next rows-was based on an elementary set of rules. It turned out that even with the simplest set of rules the output generated could be infinitely complex, seemingly random [I have written an ASCII version of this experiment in Lisp, try to run it and change the rules to get an idea].

Wolfram began to realize that there was something profound about how such complexity would arise from a simple program and began to wonder about the implications. Although the framework he created is very controversial, it definitely defines a new point of view on the operation of the world, and its complexity.

I am fascinated by the impact this framework can have on the way we reason about problems-and especially about computing. For many applications, there is clearly an alternative in this direction, and more pragmatically, Wolfram's math language, Mathematica, is an amazing tool for exploring any kind of ideas...

Anyway, his website is worth visiting, particularly the section about automatic music composition.

C U R R E N T W O R K

My main ongoing project is about the design of a complete hard-wired convolutional neural network processor. Yann LeCun et al. [CBLL] have demonstrated the efficiency of such networks in pattern/object recognition in images.

A very specialized processor like this one allows high speed processing on video streams (e.g. out of a camera), to be embedded in compact, light-weight unmanned vehicles with low power constraints [6,7].

To get more flexibility, I used HDLs to describe the critical operations of the system (the vector ALU, and its streaming interfaces), and higher level languages (Lisp and Python) to describe the architecture of the networks (more details in my master's thesis [4]).

I am currently working with Eugenio Culurciello's lab [e-Lab, Yale] to extend this concept, and create a processor based on a reconfigurable grid of simple processing units. Such a grid provides a way of dynamically reconfiguring the hardware to run different operations on streams of data... in parallel. This project is largely inspired by the old ideas of dataflow computing, and there's a lot of interesting questions, both at the hardware level and on the compilers.

We are also working together to produce a custom chip based on these ideas. The main advantage of that custom chip is that it will be totally self contained (memory included!), and very low-power.

I also built a custom platform to embed our concept processor in a very small, low-power system (15W peak for the whole system):

This system is currently able to recognize and track complex objects at more than 10fps... with a VGA input !

Here is a poster...

...and slides


R E L A T E D :

We are currently developing a very compact C++ learning library, inspired by EBLearn (EBLearn is our lab's modular learning library). This library is self-contained, doesn't have any external dependency, and is intended to run on our future custom chip, and any other resource-limited devices.

More info on that library soon... [it should be open-source]


R E C E N T P R O J E C T

One of my other significant projects was the development of a miniaturized tracking system, of which the purpose was to assist an unmanned helicopter in landing. This system involved a lot of real-time video processing [1, 3, 5].

A patent is currently pending!

My system was attached to our helicopter's nose:

A lot of the work there was about integrating the whole system into a single chip (FPGA). Here's a custom design I made to host the system:


P R E V I O U S W O R K

I also did a lot of image processing before. One of my projects was the automated reconstruction of a panoramic picture from several noisy images [2].

My first large software project was the development of a Linux based platform to manage security hardware (infrared detection systems) via a TCP/IP network.


C O N F E R E N C E P A P E R S

[9] C. Farabet, B. Martini, P. Akselrod, S. Talay, Y. LeCun and E. Culurciello, "Hardware Accelerated Convolutional Neural Networks for Synthetic Vision Systems", in International Symposium on Circuits and Systems (ISCAS'10), IEEE, Paris, 2010.
-BibTeX- -PDF-

@inproceedings{ farabet-iscas-10,
title = "Hardware Accelerated Convolutional Neural Networks for Synthetic Vision Systems",
author = "Farabet, Cl\'ement and Martini, Berin and Akselrod, Polina and Talay, Selcuk and LeCun, Yann and Culurciello, Eugenio",
booktitle = "International Symposium on Circuits and Systems (ISCAS'10)",
publisher = "IEEE",
address = "Paris",
month = "May",
year = "2010"
}

[8] Y. LeCun, K. Kavukcuoglu and C. Farabet, "Convolutional Networks and Applications in Vision", in International Symposium on Circuits and Systems (ISCAS'10), IEEE, Paris, 2010.
-BibTeX- -PDF-

@inproceedings{ lecun-iscas-10,
title = "Convolutional Networks and Applications in Vision",
author = "LeCun, Yann and Kavukcuoglu, Koray and Farabet, Cl\'ement",
booktitle = "International Symposium on Circuits and Systems (ISCAS'10)",
publisher = "IEEE",
address = "Paris",
month = "May",
year = "2010"
}

[7] C. Farabet, C. Poulet and Y. LeCun, "An FPGA-Based Stream Processor for Embedded Real-Time Vision with Convolutional Networks", in Proc. of the Fifth IEEE Workshop on Embedded Computer Vision (ECV'09 @ ICCV'09), IEEE, Kyoto, 2009.
-BibTeX- -PDF-

@inproceedings{ cnp-ecvw-09,
title = {An FPGA-Based Stream Processor for Embedded Real-Time Vision with Convolutional Networks},
author = {Farabet, Cl\'ement, and Poulet, Cyril and LeCun, Yann},
booktitle = {Fifth IEEE Workshop on Embedded Computer Vision (ECV'09)},
publisher = {IEEE},
address = {Kyoto},
month = {October},
year = {2009}
}

[6] C. Farabet, C. Poulet, J. Y. Han, and Y. LeCun, "CNP: An FPGA-based Processor for Convolutional Networks", in Proc. International Conference on Field Programmable Logic and Applications (FPL'09), IEEE, Prague, 2009.
-BibTeX- -PDF-

@inproceedings { cnp-fpl-09,
title = {CNP: An FPGA-based Processor for Convolutional Networks},
author = {Farabet, Cl\'ement, and Poulet, Cyril and Han, Jefferson Y. and LeCun, Yann},
booktitle = {International Conference on Field Programmable Logic and Applications},
publisher = {IEEE},
address = {Prague},
month = {September},
year = {2009}
}

[3] M. Garratt, H. Pota, A. Lambert, S. E.-Maslin, and C. Farabet, "Visual Tracking and LIDAR Relative Positioning for Automated Launch and Recovery of an Unmanned Rotorcraft from Ships at Sea", Naval Engineers Journal, vol. 121, no. 2, pp. 99-110, June 2009.
-BibTeX- -PDF-

@InProceedings{ garratt-08,
author = {M.~Garratt and H.~Pota and A.~Lambert and S.~E.-Maslin and C.~Farabet},
title = {Visual Tracking and LIDAR Relative Positioning for Automated Launch and Recovery of an Unmanned Rotorcraft from Ships at Sea},
booktitle = {ASNE Conference on Launch and Recovery of Manned and Unmanned Vehicles From Surface Platforms 2008},
year = {2008},
month = {May},
organization = {American Society of Naval Engineers},
address = {Annapolis}
}


J O U R N A L P A P E R S

[5] M. Garratt, H. Pota, A. Lambert, S. E.-Maslin, and C. Farabet, "Visual Tracking and LIDAR Relative Positioning for Automated Launch and Recovery of an Unmanned Rotorcraft from Ships at Sea", Naval Engineers Journal, vol. 121, no. 2, pp. 99-110, June 2009.
-BibTeX- -PDF-

@article{ garratt-09,
author = {M.~Garratt and H.~Pota and A.~Lambert and S.~E.-Maslin and C.~Farabet},
title = {Visual Tracking and LIDAR Relative Positioning for Automated Launch and Recovery of an Unmanned Rotorcraft from Ships at Sea},
volume = {121},
number = {2},
journal = {Naval Engineers Journal},
year = {2009},
month = {June},
pages = {99-110}
}


O T H E R W R I T I N G

[4] C. Farabet, "Hardware implementation of a convolutional neural network - design of a neural processor", Master's thesis, INSA Lyon, France - New York University, New York, USA.
-BibTeX-

@MastersThesis{ farabet-08,
author = {Farabet, C.},
year = {2008},
title = {Hardware Implementation of a Convolutional Neural Network -- Design of a Neural Processor},
school = {INSA Lyon, France | New York University},
address = {New York, USA},
month = {September},
keywords = {Convolutional neural networks, FPGA, hardware implementation, parallel computing, special purpose processor}
}

[2] C. Farabet, I. Ghizdavescu, S. Autin, and C. Revesz, "Cartography of a wall in a coke oven by reconstruction of a panoramic picture", January 2008, available in the TdSI journal, INSA Lyon, France.
-BibTeX-

@Unpublished{ farabet-autin-08,
author = {Farabet, C. and Ghizdavescu, Ileana and Autin, Sylvain and Revesz, Charly},
year = {2008},
TITLE = {Cartography of a wall in a coke oven by reconstruction of a panoramic picture},
month = {January},
note = {available in the TdSI journal, INSA Lyon, France},
keywords = { Optic distortion correction, perspective correction, shape-match reconstruction, panorama, defaults detection, industrial vision}
}

[1] C. Farabet, "Implementation of a tracking system within an fpga", University of New South Wales at ADFA, Canberra, Australia, Technical Report, July 2007.
-BibTeX-

@TechReport{ farabet-07,
author = {Farabet, C.},
year = {2007},
title = {Implementation of a Tracking System within an FPGA},
institution = {University of New South Wales at ADFA},
address = {Canberra, Australia},
type = {Technical Report},
month = {July},
keywords = {FPGA}
}