ubantu安装caffe

之前经历无数周折和电脑变砖之后终于装好cuda，现在开始安装caffe。这里,这里,这里都有教程。

先安装sudo apt-get install libprotobuf-dev libleveldb-dev libsnappy-dev libopencv-dev libhdf5-serial-dev protobuf-compiler，提示有unmet dependency，就把没有的装上就行。

然后sudo apt-get install --no-install-recommends libboost-all-dev，一开始显示有一堆问题，之后发现要sudo apt-get --purge remove libboost-all-dev libboost-dev libboost-doc，然后sudo apt-get install -f，sudo dpkg --configure -a， sudo apt-get clean，sudo apt-get install libboost1.54-dev。这时再sudo apt-get install --no-install-recommends libboost-all-dev就可以了。

接下来就是sudo apt-get install libatlas-base-dev和sudo apt-get install libgflags-dev libgoogle-glog-dev liblmdb-dev，没遇到问题。

之后make all -j8.因为我用的旧版本的caffe，出现了no rule to make target之类的，跟gcc的limits.h有关。可能是cuda的路径不能用/usr/local/cuda-6.5? 不过在usr/local/里面cuda应该也是link到了cuda6.5。反正使用cpu only之后就可以了，不过make test又无法通过了，显示utls里面一个函数无法再opt/cuda/include中找到cuda_runtime.h。经过一系列debug，我先装了nvida current，然后cuda toolkit，最最关键的一步是make clean，之后就可以make all了，后来发现是从网上下载的caffe默认路径在opt里面，第一次安装没有成功，所以如果没有make clean就还会在opt里面安装。之后装caffe又出现过gcc没找到limits.h的情况，总之要多make clean几次再重装就OK了。

之后重装caffe的时候，又说找不到cuda，发现在config里面不能把cuda的路径设置成/usr，必须是/usr/local/cuda，不懂caffe里面说ubantu 14.04的cuda路径设置成/usr的用意何在。

装好之后想在matlab里面跑caffe，出现gcc 4.8.4 not supported的问题，根据这里，给gcc降级。

caffe如何使用可以参考这里。有一些code是在MATLAB里面跑的，往往自带caffe的folder，直接在里面安装caffe就好。我运行的时候遇到了crash，起因在caffe('forward',image)那里。这时需要在terminal里面看matlab给出的具体error是啥。

一个比较令我困惑的是nvidia driver，cuda和cuda toolkit之间的关系。

重装系统

我的笔记本是thinkpad t420，用了很久之后变得很慢，所以用oneKey+GHO重新装了win7系统。注意必须要用`GHO`文件而不是`ISO`文件，不然oneKey会报错。

> 无线网卡

重新装好系统后一开始无线网卡不亮。从[官网](http://think.lenovo.com.cn/support/driver/newdriversdownlist.aspx?yt=pt&categoryid=9779&CODEName=ThinkPad%20T420&SearchType=0&wherePage=2&osid=231)下载`ThinkPad 1x1 11b/g/n无线网卡驱动程序(Windows 7)`和`[完整版]Intel (11abgn, abg, bg)无线网卡驱动程序`，并且安装好以后就可以连接无线网了。

> 显卡

从官网下载英伟达的显卡驱动之后屏幕就能正常使用了。下载英特尔的显卡驱动提示电脑没有满足安装要求。

> 热键

下载热键驱动之后就可以用`Fn`快捷键。

> 字体

一开始无法显示某些简体中文。要在控制面板中把system locale变成中文，还要在personalize->window color中把所有的字体改为Microsoft YaHei。如果用搜狗拼音和chrome的话，也要在设置中更改字体。

如何分享音乐

在微信中分享手机里面录制的歌曲，比如自己唱的，可以把它上传到media.io/zh/，然后转换成mp3格式。在自己的微信上打开转换好的音乐链接，发送给朋友就可以了。

研究生申请总结

我主要申请的cv方向的博士，这里记录下申请的一些经历。部分内容参考的别人的申请经验。

关于申请

套磁
不少申请的总结里面都提到了套磁的重要性，而且说这是有益无害的。我觉得套磁确实非常有帮助，第一能了解教授的科研背景，第二能锻炼书写邮件的能力。至于教授回不回复就不是那么重要了。不过很多情况下教授不回复估计可能还是我对提到的paper和topic理解不够深刻。

参考资料
申请研究生是一个漫长的过程。这个过程中，我参考了这本书，里面提到申请本质。PS或许很重要，但是更重要的是招生委员会第一眼就会看到的网申表格和CV，而PS就是这些内容的延续。尽量创造一个领域让自己成为第一，换句话说，就是展示自己的独特性。在对于这种独特性有需求的教授那里，你才能脱颖而出。这就涉及到了解自己，首先要了解自己的底线，而不是盲目的人云亦云。比如自己真的非藤校不上？即便如此，也是有自己的理由，而不是其他人都这么想所以我也这么想。了解自己底线并非降低自己，而是在此基础进步。
每个人都有自负的一面，也有自卑的一面。无论申请也好还是其他的事情，一个人要首先直面内心的自负和自卑。只有具备这种素质，才有了直面社会竞争的能力，而不是躲在象牙塔内只关心考试分数。总结来说，超越自我的实质就是怀疑，学习，然后自然而然的行动。

学校数量

Princeton review和一个CS PhD的申请指南中都提到，大概选择3所同一个水平的学校，2所你要高攀的学校，2所保底学校，总共7所。

加州系

我生在现代雾都，交换去了古代雾都，就连本科都难逃森林大火导致的雾霾之害，对理想天气的渴求非常迫切。去过加州的同学都提到那里舒适的天气，所以我也是花了很多时间研究加州系学校。


加州系学校名单University of California, Berkeley
University of California, Davis
University of California, Irvine
University of California, Los Angeles
University of California, Riverside
University of California, Santa Barbara
University of California, Santa Cruz
University of California, San Diego

UC Berkeley Computer Vision Group主要做deep learning，看起来非常高大上。 

UCD好像没有专门的CV实验室，而且只有Prof Yong Jae Lee在做CV和ML。

UC Irvine vision group需要通过School of Information and Computer Sciences申请。research topic比较广泛，从bio imaging到物体识别和运动识别都有。

Google UCSC的时候发现了一篇讲解申请经历的详细日志，介绍的都是各个学校CV方向，很有参考价值。还有这一篇，是个在caltech在做过毕设的申请经历。
UC Riverside的vision实验室在这里，不过好像很久没更新了。
UCSB的实验室是这个，研究集中在bio-inspired methods for computer vision。
UCSC的lab在这里，组里面中国学生不多。

UCSD是cognitive science的发源地，bio和neuro比较强。不过之前在这里的Prof Belongie去了Cornell。有两个CV实验室，SVCL和Computer Vision Laboratory。

说完加州，再看看其他学校。布朗大学的实验室做过很酷炫的project，比如3D绘图，不过组里面中国学生不多。这个网站上面有各个国家做vision的教授的总结。

Summary of useful ROS/Linux commands

sudo chmod a+wrx /dev/ttyACM0

To enable access to a USB port on the NUC

rosrun rqt_reconfigure rqt_reconfigure

To configure cameras

roslaunch pointgrey_camera_driver stereo_camera.launch

To run the pointgrey camera drivier

ps aux | grep your_project_name
kill -9 your_project_index

To kill a rosnode when CTL+C does not work

grep -r "keywords"

search for keywords

rqt_bag

To see the imu values in rosbag

Cannot add file in github

We need to be careful when using cloned repo, since git will ignore the folder that contains a .git. Therefore we have to remove the .git before we add all the files. Otherwise, git will say nothing to commit, work directory is clean.

To solve this, first delete the .git, then as discussed here, use git rm --cached path_to_submodule, then do the normal add and push. 

Git - changes not staged for commit

One problem I encounter is that "Changes not staged for commit" when using git. It turns out that I need to use git add --all :/ instead of git add .

This may be due to the fact that I have some modified and deleted files, and should use git add --all :/ to add all the changes recursively. I suspect the behavior of git add . has changed from the older version.

Google App Engine

在ubuntu上面使用Google App Engine，根据官网只要export PATH=$PATH:/path/to/google_appengine/，但是后来运行python2.7 dev_appserver.py说找不到文件，只能在google_appengine的文件夹里面运行才可以。用echo $PATH可以看到google_appengine的路径，我还尝试更改.bashrc也不行，可能是路径没有放到python2.7里面。

Graph slam

最近读到一篇讲解图优化的博客，非常浅显易懂，尤其是里面的理念通过简单的例子，理论推导，程序应用等三个方面来介绍非常值得推广。

Python permission error

I am following the Head First Python, which is a great book, to develop a webapp, and then encounter some permission error of the .py file. 

Based on this, it turns out that I should use #!/usr/bin/python instead of #!/usr/bin/python3.

Latex figure position

Normally I use \begin{figure}[thpb] to include figures in Latex. However, sometimes this results in a lot of space between figures. According to here, we can use \begin{figure}[h!] to place the figures in a compact manner, so that all of them fit tightly with text.

windows转ubuntu遇到的问题

GCC

遇到了很多_m256d does not name a type之类的，好像是immintrin.h没有link好。把gcc从4.8升级到4.9解决了这个问题

Ceres

安装的时候如果遇到

Eigen compilation error: error: partial specialization of ‘std::vector<T, Eigen::aligned_allocator<U> >
就按照这里说的把internal/numeric_diff.h里面的#include <Eigen/StdVector>去掉，然后从新build。注意build好了之后是跟新usr/local/include里面的ceres

VTK

cc1plus: warning: /usr/local/lib/cmake/vtk-7.0/UseVTK.cmake: not a directory。之后会显示fatal error: vtk/vtkSmartPointer.h: No such file or directory。

我先试着把include(${VTK_USE_FILE})改成include_directories(${VTK_INCLUDES})这时候虽然不会显示warning了，但还是显示fatal error: vtk/vtkSmartPointer.h: No such file or directory

后来我发现在usr/local/include里面安装的文件夹叫vtk-7.1，需要把它改名字ie sudo rm vtk-7.1 vtk就ok了。

还有undefined reference to vtkRenderingOpenGL_AutoInit_Construct，在target_link_libraries里面加入/usr/local/lib/libvtkRenderingOpenGL-6.1.so搞定

cc1plus: warning: /usr/local/lib/cmake/vtk-7.0/UseVTK.cmake: not a directory还有一种解决办法就是include(${VTK_USE_FILE})，不写include_directories(${VTK_INCLUDES})

vtk初始化的时候用smart pointer和New()会爆出segmentation fault的error, 需要在target_link_libraries里面加入/usr/local/lib/libvtkRenderingOpenGL2-7.1.so
/usr/local/lib/libvtkInteractionStyle-7.1.so，注意这里的版本是7.1，相应的要写VTK_MODULE_INIT(vtkRenderingOpenGL2);而不是VTK_MODULE_INIT(vtkRenderingOpenGL);

liblapack

gcc 4.9提示liblapack.so: undefined reference to `ATL_dgemoveT'，根据这里，sudo apt-get remove libopenblas-base解决了问题

build速度慢

lib里面没有用的东西都删掉可以让cmake变快，如果一个package安装在了usr/local/include里面，那么lib里面的build文件夹就没用了。此外build好之后原来的source文件夹也可以删掉。

OpenCV

undefined imread等等问题，直接link到so文件上，比如在target_link_libraries之中加入${OpenCV_LIBS}/libopencv_core.so.3.1

_ZN2cv8cvtColorERKNS_11_InputArrayERKNS_12_OutputArrayEii undefined
加入#include "opencv2/imgproc.hpp"和${OpenCV_LIBS}/libopencv_imgproc.so.3.1

targets may link only to libraries. cmake is dropping the item
不用
#set(OpenCV_INCLUDE_DIRS "/home/nusuav/ms_workspace/opencv/build/include")
#set(OpenCV_LIBS "/home/nusuav/ms_workspace/opencv/build/lib")
而是用
SET("OpenCV_DIR" "/home/nusuav/ms_workspace/opencv/build/")
find_package( OpenCV )
不过虽然没有了warning，但是编译的时候报错找不到libopencv_core.so.3.1

LCM

包含lcm/lcm-cpp.hpp的时候提示找不到lcm_create，安装好lcm后需要在target_link_libraries里面加入/usr/local/lib/liblcm.so

Boost

运行的时候提示error loading libboost_system.so.1.59.0，根据这里，需要export LD_LIBRARY_PATH=/path/to/my/library，再rosrun

我用boost 1.55的时候显示has no member named ‘serialize’，据说要加入#include <boost/serialization/utility.hpp>，不过后来还是出现error，用boost 1.59就没有这个问题

boost load的archive ibinary会提示
terminate called after throwing an instance of 'boost::archive::archive_exception'
what(): incompatible native format - size of long
去掉error的方法是std::make_shared<boost::archive::binary_iarchive>(filename, 1)，而不是std::make_shared<boost::archive::binary_iarchive>(filename)，括号里面的1表示使用unsigned int

ROS

error loading librospack.so，根据这里，需要source /opt/ros/indigo/setup.bash

[rospack] Warning: error while crawling...Permission denied when I try to tab-complete your package name，根据这里和这里，似乎应该

#umount /home/useraccount/.gvfs
#rm -rf .gvfs/

不过我试过之后没有效果，重新启动之后才好了。

visual studio

在用VS build的时候，出现了code analysis的错误，发现并没有enable code anlaysis。改了code analysis中的一些东西之后发现不能保存。后来用administrator模式开启VS才解决了这个问题。

cout is not a member of std
根据这里，加入#include <iostream>

localtime_s or localtime_r
在windows里面用的localtime_s，在ubuntu中并没有定义，根据这里，可以换成localtime_r，注意input的两个参数位置对调

timestamp问题

每次显示timesstamp都是1970年的，实际应该是2016年。
这个问题debug了好久，最后终于找到问题了。load时候存储epoch的t少了两位数，用t *= 100就得到正常的值了。这是因为在从windows转换到linux的时候，std::chrono_system_clock 分别用的100-nanoseconds和nanoseconds。

检查epoch的值到底应该是多少，可以参考转换时间的网站。

ros node运行慢的问题

cmake生成的是debug模式，可以用release模式catkin_make CMAKE_BUILD_TYPE=RelWithDebInfo

编译的时候生成
init done
opengl support available

根据这里，这是一个opencv的bug，貌似没什么卵用，不用管它。

无法roslaunch
roslaunch的时候遇到process died，用sudo chmod a+wrx /dev/ttyACM0

No module named catkin_pkg.packages
Make sure that you have installed "catkin_pkg", it is up to date and on the PYTHONPATH.
用pip install catkin_pkg，问题原因可能是有不同版本的Python

Deep learning interesting applications

Painting

Today I find this, which is exactly what I longed for! It generates a painting from a simple sketch. The style transfer using a portrait by Van Gogh is extremely interesting! A web demo is available at deepart.io

Searching

Check out this live demo for searching topics from BBC news!

Semantic segmentation

A great tutorial for semantic image segmentation from Cambridge is here.

Image annotation

An image annotation web demo is this, but the server is always busy!

Deep learning for object detection and semantic labeling

Here are some good resources I found which use deep learning for object recognition:

Darknet
R-CNN

I tried darknet and used the tiny weights, it is fun to play with but the labels are limited and the output accuracy is not very high.

As for the semantic labeling, all the available codes seem to use caffe and I could not find any `hello world` example yet.

matplotlib installation

I am using Ubuntu 14.04, python 2.7.6, and I install matplotlib by sudo apt-get python-matplotlib. But when I import matplotlib, it says "ImportError: No module named matplotlib". It turns out that I have to install all the dependencies as well.

According to here, these steps should be taken

1. Enable Universe repository in software and updates

2. sudo apt-get update 

3. sudo apt-get build-dep python-matplotlib

After these steps, matplotlib still does not work. So I install the Anaconda, which is supposed to provide all the popular dependencies. It turns out that matplotlib still can not be imported. 

After a while, matplotlib suddenly works! As far as I can remember, I performed the numpy and scipy tests, and relaunches the ipython notebook. 

Latex reference order

When I compile the latex file, I find that the references are not listed in the order of their appearance. I am not sure whether this is because there are citations in the figure caption, and change the bib style to ieee trans does not work.

According to here, this could be addressed by adding the following to the preamble.tex

\bibliographystyle{unsrtnat}
\usepackage[numbers,sort&compress]{natbib}

Matlab functions for computer vision

A great resource of matlab functions I encountered is here. Another useful blog that I often consult is this. Moreover, I also refer to this one sometimes.

Deep learning resource lists

At present, deep learning is very very hot, and there are numerous papers for this topic. Here is a resource lists that contains almost all one wants to know about deep learning.

Implicit shape model from RWTH

I have read the paper about an impressive object detector long time ago. Today I found that it was a work from RWTH, where I have visited! Here is the resource of the implicit shape model. The matlab version of the code can be found here.

Yet another issue with calcOpticalFlowPyrLK

I remember that I encountered once the problem of 

OpenCV Error: Assertion failed ((npoints = prevPtsMat.checkVector(2, CV_32F, true)) >= 0) 

when I was using calcOpticalFlowPyrLK. It turned out that the format of features passed to calcOpticalFlowPyrLK should be Point2f, instead of keypoint. Here is an example of how to use it. 

Another good CV blog

Today I found another wonderful CV blog here. One post talks about BRIEF seems to be interesting.

ROS ROVIO implementation headache

This time we encountered a huge problem, when we tried to implement ROVIO to Ubuntu 12.04 with ROS hydro, and the results turned out to be disastrous. We had to re-install Ubuntu two times to restore everything to the working condition.

Initially, in order to install ROVIO, we installed kindr, which required Eigen3. However, kindr was unable to find the cmake file for Eigen3, and we had to set the path manually in its cmakelist as in here,

find_package(Eigen 3 QUIET)
if(NOT Eigen_FOUND)
  include(${PROJ_SOURCE_DIR}/cfg/FindEigen3.cmake)
  set(Eigen_INCLUDE_DIR ${EIGEN3_INCLUDE_DIR})
endif()
include_directories(${Eigen_INCLUDE_DIR})

We also had to upgrade gcc from 4.6.3 to 4.8 as here. But later when we installed ROVIO, it prompted that yaml-cpp >= 0.5 was not found. We could not catkin make it even after we put the catkin simple and yaml_cpp_catkin in the same src folder.

Since there was no way to install ROVIO, we deleted it and catkin make to go back to the old workspace. However, now the mavros and mission control could not be compiled, complaining that the deleted boost functions were used. We checked synaptic and found that the current boost version was 1.48.0.2, while actually ROS hydro installed 1.46 as well.

Since we had no idea as to what happened, we re-installed Ubuntu. Although mavros and mavlink seemed ok, mission control complained that it could not find a library in the mavros package. We had to re-install Ubuntu again to solve this issue, but later we found out that the same error persisted. Then we used the older version of mavros, mavlink, mission control and it compiled without any error.

The trouble did not end here, though. When we tried to launch the nodes, one node reported to have error with exit code 11, which refers to segmentation fault. It turned out that this node used an absolute path to fopen some file. Since we re-installed our system, we no longer had that file, and thus we had to create that file and put it in the path specified in the node.

Finally, everything is ok again, but it took us two days to debug!

OpenMVG gcc version

在ubuntu 12.04上面装openMVG的时候，gcc用4.7.3的时候还是会fail，后来装了gcc 4.8才能成功安装。如何安装gcc 4.8可以参考这里

具体的步骤是
sudo add-apt-repository ppa:ubuntu-toolchain-r/test
sudo apt-get update; sudo apt-get install gcc-4.8 g++-4.8
sudo update-alternatives --remove-all gcc sudo update-alternatives --remove-all g++ 

sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.8 20 

sudo update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-4.8 20

sudo update-alternatives --config gcc sudo update-alternatives --config g++

Using OpenMVG

OpenMVG is a great library on multiple view geometry. It could be used as a stand alone library like OpenCV. These are the steps to follow

Install OpenMVG following the instruction here.
After installation is finished, follow the "Using OpenMVG as a third party library dependency in cmake" session in the installation guide.

Like what is done in the examples, eg here, include the header files and namespace in your source code. Note that the namespace must be used, eg for Image<RGBColor> image;, use using namespace openMVG::image;

For another instance, to use PointFeature defined in "openMVG/features/features.hpp", not only you should include the header file, but also have to add the name space of features, using namespace openMVG::features;

You also have to enable the support for c++11 by adding the code in cmakeList

IF(CMAKE_COMPILER_IS_GNUCC)
  SET(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=c++0x")
ELSE()
  SET(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=c++11")
ENDIF()

There seems to be a bug in the non free module, according to here.

ubantu teamviewer

前几天还好好的teamviewer用着用着突然挂了，显示not ready, no connection，可是网页可以上，说明电脑有internet connection。无论是关掉ubantu防火墙还是重装teamviewer都不行。后来想起来windows远程访问ubantu还可以用remote desktop，只要输入ip地址就行。不过连接进去之后发现ubantu的桌面变得非常不一样，但是不影响程序，比如matlab的使用。

使用两个显示器

现在我有两个显示器，一个Benq的，一个Dell自己原装的。dell的是白色比较宽的插口，benq的是蓝色的比较窄的插口。一开始把蓝色的插到主机上无法detect到benq的显示器，然后切换了一下主显示器之后居然连dell的显示器也无法识别了...

后来尝试用一个黑色转换插头把蓝色插头转换到了很扁平的一种插头，貌似是HDMI，接在主机上，然后重启电脑，两个屏幕才都可以在win7下被识别。不过后来在ubuntu中dell的主显示器不如原来流畅，必须在setting里面把display中dell设置为on，benq的设置为off，然后拔掉benq的黑色转换插头，才能在重新启动的时候用dell作为主显。

Recurrent neural network简介

最近看到一篇文章简单的介绍了RNN的概念。一般的CNN由input,hidden,output layer组成，每个layer之中的neuron没有联系，后一层的layer只跟前一层有关。这种方法的缺点是无法得到context信息，比如在一个视频中frame之间是有关联的，好比每个单词都要放在一个句子里面才容易理解。而RNN为了利用这种context，增加了每个layer之间的feedback，就是说hidden layer的neuron不仅跟前一层的有关，还跟上一次它自己的值有关，这样就仿佛给了它记忆的功能。RNN的应用包括语言翻译，图片注释等。

WD hard disk

Sometimes Win7 does not recognize the WD hard disk, and according to here, this can be solved based on the following steps

win+R to open Run, then enter devmgmt.msc to open device manager

in Universal Serial Bus Controllers, uninstall the unrecognized mass storage usb device, then right click the Universal Serial Bus Controllers and scan for hardware changes

Python and pyplot

Importance of virtual environment

I am trying to compile the code from cs231n, but it says numpy.core.multiarray failed to import. It is suggested on the web to update the python numpy by pip install -U numpy. However, I did not update it in the virtual environment, instead, I updated in the global space. Then when I tried to compile my ros catkin code, it seems the header file generated from msg using python no longer works.

To resolve this, I have to re-init a catkin workspace, and copy the original src folder to replace the new src folder. Also note to source ./devel/setup.bash. In conclusion, any update and change should be made in the virtual environment only.

Anyway, later I found that matplotlib.pyplot could not be used in the cs231n virtual environment, though matplotlib can be imported. Outside the virtual environment, however, matplotlib.pyplot can be used with no problem. Just note that when using import matplotlib.pyplot as plt, plt.show() need to be called at the end to visualize the result as stated here.  

Caffe and matlab

I recently installed Caffe and want to use it with matlab. However, it seems the structure of the folder named ''matlab'' has changed after some update. The caffe folder becomes +caffe folder. The mex file is contained in the private folder, and its name is changed to caffe_.mex instead of caffe.mex

Python and image classification

Installation

I am interested in learning python and want to use it in this module. However, when I try to install it with pip, it gives an exception. I have to use sudo apt-get install iphython, and then sudo apt-get install iphython-notebook to install it.

Also, if you are using Ubantu and Windows together, this command virtualenv .env can only be executed in the Ubantu directory, otherwise it says python could not run, permission denied.

To use the ./get_datasets.sh to download the dataset, you have to first give permission to that .sh file, by using chmod u+x program_name.

Another issue is when pip install -r requirements.txt, it says /usr/bin/ld: cannot find -lncurses, and based on this you have to enter sudo apt-get install libncurses5-dev. To build scipy and numpy, you need to install blas and lapack, which could be done in synaptics by installing libblas-dev and liblapack-dev.

Then I run into the error: library dfftpack has Fortran sources but no Fortran compiler found. You have to install gfortran in synaptics.

After all the hard work, run ipython notebook in the same directory and then you will see the notebooks available for the cs231n.

Load the dataset

To load the image dataset, use the following

from cs231n.data_utils import load_CIFAR10
cifar10_dir = 'cs231n/datasets/cifar-10-batches-py'
Xte, Ytr, Xte, Yte = load_CIFAR10(cifar10_dir)

You have to implement the nearest neighbor classifier yourself based on the code provided

To use the k nearest neighbor classifier, use

from cs231n.classifiers import KNearestNeighbor
nn = KNearestNeighbor()

Also note that in python 3, the print statement is replaced with the print function as discussed here.

ROS multi sensor fusion

ETH has a very good ROS package called multi sensor fusion here. To run its tutorial, we need to follow the steps below

Create a catkin workspace
Create a package called sensor_fusion_comm, and then put all the files in the msf source code into this package, and catkin make
Do the same to create packages ssf_core and ssf_updates

At the catkin workspace folder, launch the node by roslaunch ssf_updates viconpos_sensor.launch, and we need to put the viconpos_sensor_fix.yaml in this folder as well.

To open the gui, use rosrun rqt_reconfigure rqt_reconfigure.

In the script to plot, i.e. plot_relevant, rxplot is no longer supported and we have to use rqt_plot. The legend seems could not be changed anymore, and we should delete -b $T -t "position & velocity" -l px,py,pz,vx,vy,vz.

For using your own dataset, first run roslaunch, then open reconfig gui, and then play the rosbag. After some time, we need to hit the init filter for multiple times, and then open the plot to see the output.

We need to run plot_relevant every time we start to run ssf, otherwise there will be no data displayed.

Convert roll pitch yaw to quaternion

In ROS, we could use tf::createQuaternionFromRPY to convert roll pitch yaw to quaternion. We need to include tf in the find_package, and  <build_depend>tf</build_depend <run_depend>tf</run_depend> in the package.xml.

In the source file, add #include "tf/transform_datatypes.h", the use tf::Quaternion q = tf::createQuaternionFromRPY( roll, pitch, yaw); for conversion.

To publish the quaternion in IMU message, use

imu_msg.orientation.x = q.x();

imu_msg.orientation.y = q.y();

imu_msg.orientation.z = q.z();

imu_msg.orientation.w = q.w();

Deep learning and SLAM

Today I encounter several good sources for SLAM. This publishes notes on papers, while this introduces deep learning and SLAM. It is mentioned that the maps built by SLAM could be used to fuel the ConvNets in deep learning. Meanwhile, the computer vision research can be classified into two schools, namely geometry and recognition. So far SLAM is mostly focused on geometry.

In fact, Tombone's blog is really fascinating. It is also mentioned here that a great source to start learning deep learning is this.

Notes on using ROVIO

Besides SVO, ROVIO is a recently released VO. To build it, first git clone the source code and the kindr, then download the lightweight_filtering from bitbucket and replace the empty folder in the rovio source code.

Next, install kindr and create a new catkin workspace. In the src folder, create a new package called rovio, and replace it with the source code of rovio. Then catkin_make rovio -DCMAKE_BUILD_TYPE=Release, note that we do not use --cmake-args. Then catkin_make again to build the nodes.

To run the code, use roslaunch rovio_rosbag_node.launch, and the path for the rosbag can be changed.

Wired issue about git checkout, catkin make

When I try to go back to a previous version of the code, what worked before does not work anymore. It turns out that catkin make does not compile one of the .cpp file. I have to include a cout in that .cpp, which changes it, to force catkin make to include the .cpp file in the compilation.

Levenberg-Marquardt algorithm

I find one implementation of Levenberg-Marquardt that solves Ax = b using least square, which does not seem to make sense. To learn more about Levenberg-Marquardt algorithm, I find one fascinating blog here. 

Note that LM is very sensitive to the initial guess, as stated in wikipedia, and it could not deal with outliers. If the data contains a lot of outliers, LM is not able to converge. 

In cases with only one minimum, an uninformed standard guess likeβT=(1,1,...,1) will work fine; in cases with multiple minima, the algorithm converges to the global minimum only if the initial guess is already somewhat close to the final solution.

A simpler version of LM, without using lambda is .

5 point algorithm

I have a hard time using the supposedly easy to use findEssentialMat, since I could not understand the basics of 5 point algorithm. There is a chronology about the algorithm here, which mentions that 8 point algorithm is better for front looking camera, but it could not apply to planar scene. Also, check out the fundamental matrix song here!

In this post, the difference between E and F is explained. It mentions that the points to construct E are normalized, in the sense that they are centered at the principle point and divided by the focal length.

findEssentialMat threshold issue

I find a peculiar issue, that to use the findEssentialMat, one has to normalize the points based on the discussion here, by multiplying the inverse camera matrix. However, in reality, we could undistort the points first and then still use focal = 1 and pp = (0, 0) as if the points are already normalized, even though they are not, as reported here and here.

The I come across the code for monocular VO in paper named Autonomous Visual Mapping and Exploration With a Micro Aerial Vehicle here.

It includes both solve5PointRansac and solveP3PRansac. The solve5PointRansac uses findEssentialMat and recoverPose. The solveP3PRansac seems to use the RANSAC implemented by the authors.

Note that in motion_estimation/npoint/fivepoint/src/main.cpp, the focal passed to findEssentialMat is 300, while the threshold is 1.0 - 1e-10. In the findEssentialMat, the threshold is changed according to the focal as threshold /= focal; therefore it seems that we could not normalize points first and then use focal = 1, as this would leave the threshold unchanged.

Kalman filter for dummies

Here are some good resources for learning KF, such as Kalman Filter For Dummies, the video by student dave, and the book.

Several other sources include this, which talks about KF in terms of UAV and sensor fusion, and this, this, and this.